Python Desktop Server Weblog 2004/2

How to use macros in strict restructured text

PyDS has a new restructured text mode named strict structured text. This comes in handy if you often write stuff that might include # or $ chars or something like this - usually this would trigger the cheetah template processor. With strict structured text this won't happen, as cheetah isn't called. The only thing that is used in this case is docutils alone. But you might have some macros or nuggets for PyDS that you want to call from your postings.

For example, I have a macro to link a wikipedia page. If I want to use this in strict structured text, I need to call a directive pyds-macro. This works as follows:

.. |httg| pyds-macro:: wikipediaLink
   :definition: The Hitchhiker's Guide to the Galaxy
   :language: en

Now I can reference this new definition as |httg| everywhere in the posting where I need the reference to this wikipedia definition.

If you read the source for the macro, you may notice an additional parameter alt - this is created automatically with docutils. So if you need to call macros from strict structured text, your macro should have this attribute defined currently. This problem is fixed in the CVS version (the additional parameter definition is only needed with 0.7.1). Another way is to use nuggets, as they can accept any parameters, even parameters they don't actually use. The needed directive is called pyds-nugget.

More experimental features in TooFPy

The CVS version of the Toolserver Framework For Python has two more experimental features:

• easy tool-level locking
• timed execution of methods
• queued execution of methods

Timed execution is much like the asynchronous method calling, only you give a time when the call should happen. The timed events are managed by one worker thread from the worker pool. As soon as there are no more timed events, this worker will be freed up, again.

Queued execution is much like timed execution, except the event is triggered as soon as there is a thread available to handle them. All queued events are processed in sequence by one worker thread, so you don't get worker usage explosion if you need to trigger many events that are not time critical. If you would use asynchronous execution, every event would be handled immediately and this would result in many worker threads being occupied. Queued events have a priority - they are processed in the order of first priority, second age. So important events can be pushed to the front, less important events can be delayed.

Tool locking is usefull to secure critical areas in your tool - for example if you write data to the local filesystem. The local filesystem got private storage, too - so you can decide wether files stored should be accessible via HTTP or not.

French language file for PyDS in CVS

Omar Rabhi translated the PyDS message file to French. You can either go to his weblog and fetch the message file, or if you use the CVS version, just update. To switch your PyDS installation to French, just add the following line to your ~/.PyDS/etc/PyDSConfig.py (create the file if it doesn't already exist):

Remember to either upgrade to the CVS version or move the pyds-fr.msgs file to the same directory where your other pyds-XX.msgs files reside.

Programming by Contract for Webservices

The CVS version of Toolserver Framework For Python has an experimental addition for programming by contract concepts for webservices. The idea is to get better spec'ed remote APIs that are much more robust to abuse and much more stable when running as a service. The title links to a short description on how that feature is used.

One nice benefit of the architecture is that database transactions that are hooked to the toolserver with the new transaction hooks will be automatically rolled back if a contract is broken, as with any other exception in the system!

Programmers ... sigh ...

TooFPy 0.3.2 released

Toolserver Framework For Python 0.3.2 was released. It brings several bugfixes in the area of asynchronous computation (triggering computations in the background) and implements (and corrects previous implementations) of transactional hooks. These transactional hooks allow you to easily integrate with transaction mechanisms of your database - just write three hooks and if your tool code throws an exception, the database transaction will automatically roll back! Other changes are better charset support, removed timeouts at start time, several fixes and speedups to the request handling and expanded infrastructure (for example there is now a ~/.Toolserver/lib path to put libraries into).

Another important change is the dependency on SOAPpy 0.11.1 (and fpconst 0.6.0) in TooFPy. You must upgrade your SOAPpy installation, as TooFPy is not compatible with older versions any more due to changes in some inner APIs in SOAPpy!

New Beta 0.7.1: many enhancements

This release is a somewhat weird release: it's not really finished. I tried to rewrite the internal threading architecture for this release, but that went wrong - I hit a wall with some weird problems with Metakit that resulted in PyDS segfaulting. So I threw out some of the new code and implemented a simpler scheme. Result is a very stable PyDS again, but currently the threading model is suboptimal. As a result, you will hit a small performance penalty when accessing PyDS the first time after a longer period of inactivity due to the main worker thread shutting down (intentionally!) after an inactivity timeout. So if you don't need the new features, you might want to stay with 0.7.0 - I needed to release 0.7.1 because of interdependencies with other projects (TooFPy in this case) and the reliance on SOAPpy 0.11.1.

The reliance on SOAPpy 0.11.1 (and fpconst 0.6.0) requires you to upgrade your SOAPpy installation - PyDS 0.7.1 (and TooFPy 0.3.2) is not compatible with SOAPpy 0.10.1 any more, due to changes in some inner API of SOAPpy!

Now on to the new features and fixes. There are many, so read them in the changelog. But there are some highlights:

• the WeblogTool now has a default Blogmarklet linked, as does the BlogmarkTool
• SilverCity is used to allow code-blocks as written in P201
• the new ArchiveTool gives you compact posting archives across all supported tools
• strict structured text now allows to better write pure reST pages, as there is no interference with cheetah macro code (no special handling for the # char anymore). To allow use of many tools, you have special directives to link images, pictures, macros and nuggets.
• speedups due to less locking, usage of cStringIO instead of string concatenation in the aggregator and several other enhancements. Besides this weird leftover from my threading reworking, PyDS is now quite fast! (and the delay will go away with 0.7.2, at least that's planned)

SOAPpy 0.10.1 has a bug with exception handling

Update: I misremembered what version of SOAPpy I had installed. Actually the bug was first listed here as a bug for SOAPpy 0.11.1, but that's wrong. It's SOAPpy 0.10.1 that has the bug. SOAPpy 0.11.1 is working fine. But if you want to upgrade to SOAPpy 0.11.1, you need to use PyDS from current CVS, as older versions are not compatible with SOAPpy 0.11! The next release will require SOAPpy 0.11.1 (and fpconst 0.6.0, as that is needed by SOAPpy 0.11.1).

If you call a function on a remote webservice and that function throws an exception, and if that exception is of type SOAPpy.Types.faultType, the client code automatically should throw a corresponding exception. This doesn't work, because the following code:

around line 297 in Client.py does check wether the result is an instance of class faultType. But Client.py is missing the necessary import statement and so doesn't know about faultType. As a result you get the faultType instance passed along as call result instead.

To fix it, just go to the head of Client.py and change it as follows:

That's all, now SOAPpy.Client should throw exceptions just fine. The correct way to don't get exceptions thrown is to pass an optional argument throw_faults=0 to the creation of your SOAPProxy instance.

Sample WikiTool Creation Tutorial

This tutorial shows how to build a very simple wiki tool for the Toolserver Framework For Python. It builds a wiki that has a web interface that maps directly to an underlying RPC API. So you not only get a wiki, you also get a simple API to manage wiki pages. The tutorial shows several aspects of the Toolserver Framework For Python.

Needed is installation of the just released Toolserver Framework For Python version 0.3.1.

Toolserver Framework for Python

This is a small project that's an offspring of my work on the Python Desktop Server. It implements a dynamic threading server for SOAP. It sports a threading model much like Apache 1.3 - so it starts and stopps threads by need. Integrating a tool into the toolserver and export it's methods via SOAP is as simple as subclassing a base class. Not much documentation yet, if you need a lightweight server for SOAP services, it might be usefull. It doesn't provide WSDL interface descriptions, though!

UPDATE: I just released version 0.2.0 wich supports XMLRPC as an alternative to SOAP. Additionally there now is a Toolserver Framework for Python Quickstart Guide.

UPDATE 2: I commited changes for 0.3.0 to CVS that implement REST style API building. It's still a bit crude, it's interfaces might change a bit, but it is already working.

UPDATE 3: I commited changes for 0.3.0 to CVS that help with easy basic authentication for methods. Another change done is conditional GET for the REST interfaces based on ETag headers.