For our server application, besides the recently explained initialization, we'll need to initialize Tclhttpd and comm packages:

csa::log::debug "Initializing Tclhttpd"
package require tclhttpdinit
Httpd_Server 8981
csa::log::debug "Initializing comm interface"
package require comm
comm::comm configure -port 1991

This will cause the csa server to listen for HTTP connections on port 8981 and on comm connections on port 1991. The HTTP connection is used for communication with all clients, and the comm package is set up to allow adding requests from other applications and testing the application.

The next step is to load the additional scripts that initialize the database, set up the comm API and handle incoming HTTP requests:

csa::log::debug "Sourcing remaining files"
source [file join $starkit::topdir commapi.tcl]
source [file join $starkit::topdir database.tcl]
source [file join $starkit::topdir clientrequest.tcl]

The files we source are described later in the chapter. Finally we need to enter the main loop. If Tk is present, we'll show the UI that comes with Tclhttpd:

csa::log::debug "Entering main loop"
# Initialize Tk UI if possible
catch {package require Tk}
if {[info commands tk] != ""} {
package require httpd::srvui
SrvUI_Init "CSA server"
} else {
puts "Server started"
vwait forever
}

For the client, after setting up the logger and the data directory, we'll accept the host name to connect to as argument, to print out an error if either one is not provided:

if {$argc < 1} {
set error "Usage: client <hostname>"
log::error $error
puts stderr $error
exit 1
}
lassign $argv csa::hostname
csa::log::info "Connecting to $csa::hostname"

We will now read the additional scripts—the initialization of local database, the client to communicate with the server, and read the local application's MD5 checksum:

source [file join $starkit::topdir database.tcl]
source [file join $starkit::topdir fileinfo.tcl]
source [file join $starkit::topdir client.tcl]

Then we'll schedule the csa client to request jobs to perform as soon as it is initialized:

after idle csa::requestJobs

The command csa::requestJobs is explained later in this chapter.

Finally we need to initialize the main loop. Similar to the server part, if Tk is present, we show a minimal UI with an exit button:

csa::log::debug "Entering main loop"
if {[info commands tk] != ""} {
# basic UI
wm title . "CSA Client"
button .exit -text "Exit application" -command exit 
-width 30 -height 3
pack .exit -fill both -expand 1
} else {
vwait forever
}

Let's start off with how our server would handle requests.

Our function to process a request would start off as follows:

proc csa::handleClientProtocol {sock suffix} {
upvar #0 ::Httpd$sock s
set req [dict create]
set response [dict create]
set ok 0
if {[info exists s(query)]} {
set req [dict create {*}[encoding 
convertfrom utf-8 $s(query)]
}

This would map the HTTP request data to the variable s, initialize default values for variables, and if a query was passed then store it in req variable. We also initialize the ok and response variables, to indicate that request was not correct and response to contain no data, which we'll add later.

After this, we'd check if a client is allowed to talk to our server by trying to authorize it, if guid and password keys were sent in the request:

if {[dict exists $req guid]
&& [dict exists $req password]} {
set guid [dict get $req guid]
if {![authorizeClient $guid 
[dict get $req password]]} {
log::warn "handleClientProtocol: Access denied"
unset guid
}
}

Our clients will send passwords as raw text. However, the password will be created as random string when client first connects, and applications requiring security should rely on more sophisticated algorithms for authorizing clients or users. such as SSL and signed certificates, described in Chapter 13, SSL and Security.

Authorization itself is done by authorizeClient command. If the authorization succeeds, the guid variable remains set. Otherwise we log an error and unset it.

Next, we can provide the client with additional data only if the user is authorized. For example, providing a list of jobs from getJobs command, providing at most the number of jobs client requests, or 10 by default:

if {[info exists guid]} {
set ok 1
if {[dict exists $req joblimit]} {
set joblimit [dict get $req joblimit]
} else {
set joblimit 10
}
dict set response jobs 
[getJobs $guid $joblimit]
}

We make sure that only valid clients get the data by checking if guid variable exists. If authorization fails, it is unset and server won't check for jobs to send to client.

Finally, we need to send the response back to the client, creating a list of ok and response variables values, and converting it into UTF-8:

Httpd_ReturnData $sock application/x-csa-data 
[encoding convertto utf-8 [list $ok $response]]
}

We've used our own MIME type of application/x-csa-data, but it is not required; the type application/octet-stream, which indicates any series of bytes, can also be used.

In order for a client to be able to send results from commands, we'll create an additional small handler. We'll start also by retrieving the query sent by client:

proc csa::handleClientResult {sock suffix} {
upvar #0 ::Httpd$sock s
set req [dict create]
set ok 0
if {[info exists s(query)]} {
set req [encoding convertfrom utf-8 $s(query)]
}

If both job and result keys are sent, we store the job's result in the database.

if {[dict exists $req job]&&[dict exists $req result]} {
setJobResult [dict get $req job] 
[dict get $req result]
set ok 1
}

Finally we also send the result back to the client:

Httpd_ReturnData $sock application/x-csa-data $ok
}

As the client needs to know a valid job identifier, we can assume that it is a valid client. Otherwise, the job identifier would probably be invalid, which setJobResult command needs to handle.

Last thing we need to do is to register handler for specified prefixes in our web server:

Url_PrefixInstall /client/protocol csa::handleClientProtocol
Url_PrefixInstall /client/result csa::handleClientResult

Implementation of the client for the communication is also not a very difficult task. We'll use a polling-based approach for communication, which means that the client will periodically poll the server for new tasks to be processed.

First thing to implement is command for requesting jobs to be performed from the server. We'll start off by defining the command that will periodically be called to get new jobs. It starts off by cancelling any other scheduled invocations of the check:

proc csa::requestJobs {} {
variable hostname
after cancel csa::requestJobs

We create URL to connect to, based on the hostname provided by the user. We also set the request's key guid and password in order to authenticate the request.

set url "http://${hostname}:8981/client/protocol"
set req [dict create]
dict set req guid $::csa::guid
dict set req password $::csa::password

Values for guid and password are stored in database and described later.

Next the command tries to send request to the server; we convert request to UTF-8, send it as application/x-csa-request MIME type and specify that csa::requestJobsDone command should be invoked whenever results are retrieved.

if {[catch {
http::geturl $url -timeout 30000 
-query [encoding convertto utf-8 $req] 
-type application/x-csa-request 
-command csa::requestJobsDone
}]} {
# try again in 1 minute
after 60000 csa::requestJobs
}
}

If the request has been successfully sent, then the next steps are followed using requestJobsDone command. Otherwise, we set up a retry in 60 seconds.

At the start, we'll set up another check for commands in 1 minute from now:

proc csa::requestJobsDone {token} {
# schedule next communication in 1 minute from now
after 60000 csa::requestJobs

Then we check if the response is valid and no error was encountered. In case of any error, we log it, clean up the HTTP token and return. Otherwise we convert the response from UTF-8 and proceed.

if {[http::status $token] != "ok"} {
log::error "requestJobsDone: Invalid HTTP status"
http::cleanup $token
return
} elseif {[http::ncode $token] != 200} {
log::error "requestJobsDone: Invalid HTTP code"
http::cleanup $token
return
} else {
set response [encoding convertfrom utf-8 
[http::data $token]]
log::debug "requestJobsDone: Response: $response"
}

As the first step, we assign ok and response variables to first and second element of the response, we retrieved from the server. If the server did not send a negative status as the ok variable, we check if the response contains key jobs—if it does, and it has any elements then we invoke the runJobs command to run each of the jobs.

lassign $response ok response
if {$ok} {
if {[dict exists $response jobs]
&& ([llength [dict get $response jobs]] > 0)} {
runJobs [dict get $response jobs]
}
} else {
log::error "requestJobsDone: Server returned error"
}

Finally we clean up the HTTP token.

http::cleanup $token
}

Running the jobs themselves is done as evaluating them in the global stack frame using uplevel #0 command. The runJobs command looks as follows:

proc csa::runJobs {jobs} {
log::debug "runJobs: Running jobs"
foreach {job command} $jobs {
log::debug "runJobs: Running job $job"
set result [uplevel #0 $command]
log::debug "runJobs: Sending result"
sendResult $job $result
}
log::debug "runJobs: Running jobs complete"
}

The command runs each of the commands and then submits its result to the server.

It uses sendResult to send a result for a job, which is shown as follows.

We first create a URL and a request, setting the keys job and result accordingly. Then the command sends a request to the server:

proc csa::sendResult {job result} {
variable hostname
set url "http://${hostname}:8981/client/result"
set req [dict create job $job result $result]
set token [http::geturl $url 
-query [encoding convertto utf-8 $req]]

After finishing sending the HTTP request, we then check the response and whether the server has properly handled our request or not; if not, we log and throw an error:

if {[http::status $token] != "ok"} {
http::cleanup $token
log::error "sendResult: Invalid HTTP status"
error "Invalid HTTP status"
} elseif {[http::ncode $token] != 200} {
http::cleanup $token
log::error "sendResult: Invalid HTTP code"
error "Invalid HTTP code"
}

Finally, the command cleans up HTTP token and exits.

http::cleanup $token
}

This is a complete implementation of the client.

For the server side, our application needs to keep at least two types of items—a list of clients and the list of jobs a client should perform, along with their results.

Database initialization code should go to src-server/database.tcl, which is read from the main script created earlier. We'll need to start off with loading packages and initializing the database:

package require sqlite3
package require uuid
sqlite csa::db [file join $csa::datadirectory csa-server.db]

After this, we'll create tables for storing clients and jobs. We'll start by checking if the tables already exist:

if {[catch {csa::db eval "SELECT COUNT(*) FROM clients"}]} {

If querying a number of items in clients table throws an error, this means that the table has not been created. In this case, we'll create both clients and clientjobs tables:

csa::db transaction {
csa::db eval {
CREATE TABLE clients (
guid CHAR(36) NOT NULL PRIMARY KEY,
password VARCHAR(255) NOT NULL,
status INTEGER NOT NULL DEFAULT 1,
lastupdate INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE clientjobs (
job CHAR(36) NOT NULL PRIMARY KEY,
client CHAR(36) NOT NULL,
status INTEGER NOT NULL DEFAULT 0,
lastupdate INTEGER NOT NULL DEFAULT 0,
command TEXT,
result TEXT
);
}
}

The first table stores the GUID and password of all clients, along with the time when they last queried the system. The field status defines current status of a client: 1 means enabled; 0 means disabled.

The table clientjobs stores jobs associated with a client. The field job is a GUID identifying the job and client is the GUID of the client that should run it. The field status defines whether the job has already been run or not: 0 means it has not run; 1 means it has already run. The field command keeps the command to run and result stores the result retrieved from the client.

Now we can create functions that other parts of the application will use.

Let's start with authorizing a client. Our csa::authorizeClient command would need to check if a client exists in the database, and add it if it does not. It would need to check if the password provided matches the one in the database and that the client's status is set to enabled. If not, it should return a message stating that the authentication failed. Otherwise, it should change the lastupdate field and return message that authentication succeeded:

proc csa::authorizeClient {guid password} {
set now [clock seconds]
set result [db eval {SELECT status, password
FROM clients WHERE guid=$guid}]
# if no entry found, add a new item to database
# and assume the new agent is ok
if {[llength $result] == 0} {
db eval {INSERT INTO clients (guid, password)
VALUES($guid, $password)}
return true
}
lassign $result status dbpassword
# check if passwords match
if {![string equal $dbpassword $password]} {
return false
}
# if client is not currently enabled, return false
if {$status != 1} {
return false
}
# if everything matched, it's a valid client
db eval {UPDATE clients SET lastupdate=$now
WHERE guid=$guid}
return true
}

Our application assumes that if we do not currently know a particular client, we'll silently add it to the database. While this assumption might not always be correct, in this case there is no danger as a client only accesses his own jobs, which need to be added for new clients explicitly. Also, such an assumption makes working with the examples much easier.

In the next step, we'll need to create functions for managing jobs:

Let's start with the command for adding new entries:

proc csa::addJob {client command {callback ""}} {
variable jobResultWait

First we'll check if a specified client exists and is active:

# do not allow adding jobs for inactive clients
if {[llength [db eval 
{SELECT guid FROM clients WHERE guid=$client
AND status=1}]] == 0} {
return ""
}

Now we'll create an identifier for the job and add it:

set job [uuid::uuid generate]
set now [clock seconds]
db eval {INSERT INTO clientjobs
(job, client, lastupdate, command)
VALUES($job, $client, $now, $command)}

If the callback script was provided, we store it:

if {$callback != ""} {
set jobResultWait($job) $callback
}

Finally we return the job identifier.

return $job
}

After we've added a job, clients need to be able to retrieve those using getJobs command, which is a simple query for jobs that were not yet run:

proc csa::getJobs {client limit} {
return [db eval {SELECT job, command FROM clientjobs
WHERE client=$client AND status=0 LIMIT $limit}]
}

The next step is being able to set a job's result:

proc csa::setJobResult {job result} {
variable jobResultWait
db eval {UPDATE clientjobs SET status=1,result=$result
WHERE job=$job}

We also need to handle callback scripts that were passed to addJob; if it has to be set, we set it to be invoked by the event loop, and unset it:

if {[info exists jobResultWait($job)]} {
after idle [linsert $jobResultWait($job) end 
$result]
unset jobResultWait($job)
}
}

Finally, retrieving a job's result is also a simple query:

proc csa::getJobResult {job} { return [db eval {SELECT status, result FROM clientjobs
WHERE job=$job AND status=1}]
}

This provides us with all the database access functions needed for both clients and jobs.

The reason for creating callbacks for jobs is that for the comm interface, we will also provide a synchronous interface. This is an interface which from caller's perspective will return, once the client has finished processing the command. They are not stored in the database as whenever a server restarts, call-back scripts will no longer be valid.

The database for the client application is definitely smaller—all that we store in it is a single table with a single row, containing the guid and the password for connecting to server.

Similar to the server, we start by initializing the SQLite package and setting up a database object:

package require sqlite3
package require uuid
namespace eval csa {}
sqlite csa::db [file join $csa::datadirectory csa-client.db]

In the next step, we intialize the table if it was previously not there:

if {[catch {csa::db eval "SELECT COUNT(*) FROM
configuration"}]} {
csa::db transaction {
csa::db eval {
CREATE TABLE configuration (
guid CHAR(36) NOT NULL PRIMARY KEY,
password VARCHAR(255) NOT NULL
);
}

After setting up the table we also generate random guid and password strings, and insert them into the database:

set guid [uuid::uuid generate]
set password [uuid::uuid generate]
# insert authentication data into database
csa::db eval {INSERT INTO configuration
VALUES($guid, $password)}
}
}

Finally, regardless of the fact that the entries were just generated or were already stored in the database, we set csa::guid and csa::password to the values stored in the database:

lassign [csa::db eval {SELECT guid, password
FROM configuration}] csa::guid csa::password
csa::log::info "Local client identifier: $csa::guid"

We also log the identifier that the client has assigned each time it is run, so that it is easier to test the application.

Although just these two values do not require a complete database, our application uses SQLite so that potential extensions could also benefit from a pre-initialized database.