Toolserver Framework for Python Quickstart Guide

The Toolserver Framework for Python (TooFPy from now on) is a medusa based webserver who's main purpose is to provide access to objects via SOAP, XML-RPC (available starting with 0.2.0) and REST (available starting with 0.3.0). The server itself has a dynamic threading model that uses a monitor thread for dynamically starting and stopping worker threads. It can be configured with a simple config module and installation of namespaces into the toolserver is done by writing subclasses and registering them.

If you installed the package, you have several things on your drive:

a tsctl command line utility to control the toolserver
a Toolserver/ package in your Python path
some documentation and sample tools in /usr/share/toolserver/

If you want to start a toolserver, you have to set it up, first. Setup is very easy:

tsctl init

This will initialize a directory ~/.Toolserver/ and populate this with a default configuration.

The default configuration has default values for the following variables:

# these variables configure the IP and port to bind to

serverhostname = 'localhost'

serverip = '127.0.0.1'

serverport = 4334

# if you want to use tsctl debug, you need to set up

# the monitor port and password

monitorport = 0

monitorpassword = ''

# usually you work with iso-8859-1 strings, but you can

# use utf-8 or any other encoding, if you want to.

# The framework will pass 8bit strings, not

# unicode strings to your methods!

documentEncoding = 'iso-8859-1'

# these variables are best left alone - they are set

# by command line options, usually

daemon = 1

verbose = 0

# these variables control dynamic thread instantiation

# maxworkers says how much workers at max can run

# freecheckinterval gives seconds to wait between checks

# minfreeworkers says how much workers must be free at

#    any time, else there will be new workers be started

# maxfreeworkers says how much workers can be free, before

#    threads are stopped

# startfreeworkers says how much threads to start if

#    the number of free workers is to low

freecheckinterval = 5

minfreeworkers = 4

maxfreeworkers = 6

startfreeworkers = 4

maxworkers = 100

Usually you just set the serverport and maybe (if your service should be callable from other machines) the serverip and serverhostname variables. After setting your variables, you can now start your toolserver:

tsctl start

Now you have a minimal toolserver running. There is one default tool bound to the namespace greeting. With TooFPy versions below 0.3.5 it is installed automatically, starting with TooFPy 0.3.5 you need to copy it from the samples directory to your tools (~/.Toolserver/tools) directory and restart the toolserver.

To try your new toolserver, you can try the following code in Python:

import SOAPpy
srv = SOAPpy.SOAPProxy('http://localhost:4334/SOAP/greeting')
print srv.greeting('you', 5)

When this is run, it should connect to your toolserver (provided you didn't change the port) and call the greeting method of the GreetingTool. This method just waits the given number of seconds and then returns a string that uses the first parameter to be constructed.

You could use it from XML-RPC, too. This is very easy, too:

import xmlrpclib
srv = xmlrpclib.Server('http://localhost:4334/RPC2/greeting')
print srv.greeting('you', 5)

As you can see, there are two protocol dispatchers - SOAP and RPC2 - that decide what protocol is used. The tools namespace can be given directly in the server URI, or you can use a dotted notation and one central dispatcher. This is often used with Userland derived protocols like xmlStorageSystem. With XML-RPC the client code looks like this:

import xmlrpclib
srv = xmlrpclib.Server('http://localhost:4334/RPC2')
print srv.greeting.greeting('you', 5)

So how to write your own tool? This is very easy. Just look at the code of /usr/share/toolserver/tools/GreetingTool.py for a starter. Writing tools is as easy as subclassing:

from Toolserver.Tool import StandardTool, registerTool

class SampleTool(StandardTool):

  def myCoolMethod(self, num1, num2):

     return num1 + num2

registerTool(SampleTool, 'sample')

This code needs to be stored as ~/.Toolserver/tools/SampleTool.py and then you need to restart the toolserver:

tsctl stop

tsctl start

Afterwards you have your first own webservice up and running. All methods that don't have a underscore in their name are automatically available to XML-RPC and SOAP clients. Test it with the following code:

import SOAPpy
srv = SOAPpy.SOAPProxy('http://localhost:4334/SOAP/sample')
print srv.myCoolMethod(5,6)

That's all there is to it. Oh, one more note: if you define a method _initopts, this one will be called in the initialization process for tools. You might want to create database connections or stuff like that in that method.

Now we start to build a REST style API based on our tool. This is more complicated as with REST style APIs every method needs to have their own parameter parser and result generator helpers - REST doesn't use a standard representation like SOAP or XML-RPC do. In our sample we want to provide a very simple XML return format and parameters done with standard URL encoding. We want to provide both POST and GET access to our method. Errors should be returned as a special simple XML error message. So we rewrite our SampleTool as follows:

from Toolserver.Tool import StandardTool, registerTool

class SampleTool(StandardTool):

  _content_type = 'text/xml'

  def _error(self, request, error, detail, traceback):

     return '<error>%s: %s</error>' % (error, repr(detail))

  def myCoolMethod(self, num1, num2):

     return num1 + num2

  def myCoolMethod_parser_GET(self, request, data):

     form = parseQueryFromRequest(request)

     return ([int(form['num1']), int(form['num2'])], {})

  def myCoolMethod_parser_POST(self, request, data):

     form = parseQueryFromString(data)

     return ([int(form['num1']), int(form['num2'])], {})

  def myCoolMethod_generator(self, request, result):

     return '<result>%d</result>' % result

registerTool(SampleTool, 'sample')

As you can see this is quite easy, too. You can provide helpers for methods, for method-and-HTTP-verb combinations or provide defaults for all methods of a tool that don't have specific details. There are the following helpers:

_content_type is a class variable that returns the mime-type of the result.
_parser is a method that receives the request and payload and returns a tuple of a list and a dictionary. The method itself will be called with the args and keyword args from the _parser result.
_generator is a method that receives the request and the method result and must produce a representation that maps to the _content_type of this method.
_error is a method that receives the request, an error object, a detail object and the traceback. It should produce an informative error message. Usually you want to set up one global handler that renders exceptions in HTML or XML.
_validate is a method that is called with the request and the user and password from a basic authentication header. This method should check wether this user and password are allowed to do the request.

Remember, all helpers can be prefixed with the method names and suffixed with the HTTP verb to get specialized versions.

_validate understands an additional pseudo verb RPC. In that case the validation method is called with the original parameters of the method and can check wether a user given in the parameters is allowed to access the method. This can be used to globally deny RPC access, too (just throw ForbiddenError). Look in the GreetingTool code to see how to use most of these wrappers.

Now we test the POST and the GET handler:

curl -d 'num1=5&num2=6' http://localhost:4334/sample/myCoolMethod

curl 'http://localhost:4334/sample/myCoolMethod?num1=5&num2=6'

The next step is to change our little tool to do some processing in an asynchronous manner: instead of directly computing the result, we just push the computation into the background and use tool methods to check for this computation from the outside. This is very handy if you need to do very longrunning computations where the RPC client would timeout before the result is computed. We just add a few methods to our tool like this:

import time

from Toolserver.Tool import StandardTool, registerTool

class SampleTool(StandardTool):

  _content_type = 'text/xml'

  def _error(self, request, error, detail, traceback):

     return '<error>%s: %s</error>' % (error, repr(detail))

  def myCoolMethod(self, num1, num2):

     return num1 + num2

  def myCoolMethod_parser_GET(self, request, data):

     form = parseQueryFromRequest(request)

     return ([int(form['num1']), int(form['num2'])], {})

  def myCoolMethod_parser_POST(self, request, data):

     form = parseQueryFromString(data)

     return ([int(form['num1']), int(form['num2'])], {})

  def myCoolMethod_generator(self, request, result):

     return '<result>%d</result>' % result

  def _myBackgroundComputation(self, delay, result):

     time.sleep(delay)

     return result

  def start(self, delay, result):

     return self._async(self._myBackgroundComputation, delay, result)

  def running(self, asyncid):

     return self._asyncActive(asyncid)

  def result(self, asyncid):

     return self._asyncResult(asyncid)

registerTool(SampleTool, 'sample')

In this case the method _myBackgroundComputation is the long running computation we want to push into the background. We have an RPC method named start that will start this computation. It returns an ID we need to store away for checking wether the computation is still active and what it's result is. The checking is done with the method running. If running returns a false value, we can use result to fetch the result. Be warned that _asyncResult will fail with KeyError, though, if the background computation didn't successfully complete! As soon as you fetch your result, it will be deleted from the internal cache. You can't check for the result twice - if you need it, store it in some variable!

Now we will write a litte script that uses this new API to do some useless tests:

import time
import xmlrpclib

srv = xmlrpclib.Server('http://localhost:4334/RPC2/sample')
aid = srv.start('result', 50)

while srv.running(aid):
time.sleep(5)

print srv.result(aid)

Very simple, isn't it? You should allways put waits or other stuff between the activity checking to not put to high a load on your toolserver. Background computations will use the normal worker threads, so your toolserver might need to start additional worker threads if you make heavy use of background computation.

If you want to have a background thread that regularily checks for some condition, like changes in the filesystem, you can call _async on your background method within your background method as a last step. Put a delay of N seconds before this call to _async and your background method will be triggered every N seconds. The code might look something like this:

def _myBackgroundPoller(self, errorcount):

   try:

      # do some heavy computation, check filesystem, whatever

   except:

      errorcount -= 1

   if errorcount > 0:

      time.sleep(5)

      self._async(self._myBackgroundPoller, errorcount)

This way your call will reinsert itself into the thread queue when it is done. You need to start it once to fire it up, of course. There is one parameter that needs to be the maximum errors you accept in your background computation. If it fails as often as errorcount says, it isn't restarted any more - this prevents servers going postal.

Debugging in TooFPy can be a bit complicated, as you have a long running server process with quite a bit of state. To make debugging easier, you use a monitor client. A monitor client directly connects to your running toolserver and allows you to inject code into the context of this running server! To run a monitor clinet, you first need to set up monitorport and monitorpassword in your configuration and restart the toolserver. After restarting the toolserver, just start a debug session:

tsctl debug

This gives you a shell where you can enter (sorry, no readline, and not possible to do readline there!) python code. To look at your tool, just do the following:

from Toolserver.Tool import getTool

tool = getTool('sample')

dir(tool)

This will show you the attributes of your tool. Use the variable tool to play around with your code. Look into instance variables, try methods, patch data structures - do whatever you want. To get out of the debugging client, just press control-d. The client will close, but might hang. If it hangs, just press control-c and everything is fine.

Have fun!