Python Desktop Server Weblog 2004/11

Active Storage Framework for Python

If you want to know what I am currently working on (besides the Toolserver Framework For Python of course) you can follow the title link and browse the CVS repository of my newest project. Active Storage Framework for Python is a database that borrows conceptually from Userland Frontier and Mumps (yeah, I am an old fart and still know such stuff ) in that it gives the programmer a hierarchical namespace for storage objects. It has many weird features and I think it's plain cool. But then I have to think that - I am writing it

The idea is to get a storage that's as far away from standard SQL as possible but using a relational database as data storage so I don't have to reinvent transaction handling, data security and all that stuff again. It uses currently PostgreSQL but should be easy to port over to other relational databases. It might become the database for something like PyDS/NG (there, I said it, I will do a rewrite of the Python Desktop Server one day - using Toolserver Framework For Python as the server engine and Active Storage Framework for Python as the database and whatever crosses my mind for template rendering).

If you want to know more about Active Storage Framework for Python you might want to start with the README. And yes, it should already work. All you need is a PostgreSQL database, the psycopg database driver and Active Storage Framework for Python of course.

Again a bunch of changes in CVS

The WSGI support should now be much more compliant. Especially the PATH_INFO problem is fixed and there is now support for the writer returned by start_response.

This PATH_INFO support is rather nice for other REST style APIs, too: you can implement funky caching with this. If you access static content below a published tools method that doesn't exist, the method is called with the request. There you can regenerate the static content - next time you try to access the same URI, the static content is already there and will be delivered directly, without the tool needing to do any work. If the internal state that generates this static content changes, just delete the static content file - next time you access it, it will be rebuilt automatically. Cute.

WSGI directly makes use of this stuff by splitting the request path and setting SCRIPT_NAME and PATH_INFO accordingly. This should help frameworks that want to make use of PATH_INFO to produce nice looking URIs.

Another change in WSGI is the support for REMOTE_USER. The whole authentication scheme was reworked a bit - everything now uses the new $HOME/.Toolserver/etc/passwd and $HOME/.Toolserver/etc/groups based authentication (and hosts.allow/hosts.deny, of course - even RSA authentication can now be enabled for REST style access!). If the authentication recognizes a user, REMOTE_USER is set accordingly.

Another change for the authentication stuff is that now there is a usefull base class to allow easier building of tools where access is restricted to defined clients or defined groups. Of course all this can be overloaded in your tools classes to make it fit your needs.

The authentication config files (hosts.allow, hosts.deny, passwd, groups and RSA keys) are now automatically reloaded when they change. No need to restart the server just to provide new user credentials.

Due to the changes in the authentication stuff you will have to rewrite your _validate_HTTP or _validate* (REST style) handlers, as they don't get the user and password any more - instead they should check either context.client or context.groups. All password handling is now done by the framework (sure, you still could parse the auth headers from the request and do everything yourself).

Oh, and the monitor client now works again

Update: I just commited some more changes that make pycrypto optional. Now you can decide to not install it and everything should work fine - you just don't get RSA authentication and PickleRPC. But since those are both non-standard features, you can only make use of them if you have TooFPy on both the client and the server - so if you can't rely on that, you can safely ignore pycrypto.

Additionally I made tests with more current versions:

• Python 2.3.3
• SOAPpy 0.11.6
• fpconst 0.7.0

I can report that it looks like everything works fine. I will start to do work on my OS X machine under Python 2.3 so if there are niggles left in the code, they should be found.

Many changes in TooFPy CVS

Another bunch of changes in the CVS. The main two changes are in the area of authentication and in the area of WSGI support. Yes, TooFPy now has a WSGI container where you can put in your applications. If you want, you can then use direct tool calls in your WSGI application to make use of the many TooFPy features, or you can stay fully in WSGI world and so be portable to other python servers.

The changes in authentication are that now there is not only the rather heavywheight RSA authentication, but support for IP based authentication and HTTP basic authentication for RPC calls, too. Those are rather easy to set up, so if you don't need the full blown public key authentication stuff, you can now have a simpler way to restrict access. Clients can be grouped now, so you can write your tools against some groups instead of maintaining client name lists for all allowed clients within the tool itself.

Another bigger change now is that the two special medusa http handers for the /API/ and /WSDL/ namespace are gone - and reborn as real tools. This makes some stuff easier to maintain and is more logical. The WSGI handler is built as a tool, too.

A side effect of the API and WSDL change is that now the transient tools and factory tools are actually tested and actually working - nobody seems to have used them before, because nobody complained about them not working. On the other hand, maybe nobody used them because nobody knows how

TooFPy now has RSA authentication and communication encryption

The CVS version of TooFPy now has additional stuff to make communicating more secure. I added my own client modules that can be extended with protocols (supported protocols now are SOAP, XMLRPC and PickleRPC). This client modules allow additional non-standard extensions. These extensions were used to provide RSA authentication for communication: if enabled, additional HTTP headers will be passed with the request and response that carry a SHA256 hash and a RSA signature (and the signers name). That way both the server and the client can be sure they are communiating with exactly the system they expect to communicate with. This works for all RPC protocols, but of course your client library has to support it (the standard libraries don't support it directly - you might need to either use my client modules or patch the stuff into your library). Of course the RSA authentication is optional - if you don't enable it, you are still fully compatible with other libraries.

Additionally the PickleRPC protocol now can use encryption of requests and responses to prevent people from listening in. The shared secret for PickleRPC communication is used for encryption.

And another change is that I switched to only use SHA256 instead of either md5 or sha throughout the code.

All the encryption and signature stuff needs the pycrypto modules by amk installed - there are source installation packages and even windows binary installers, just read the INSTALL-FROM-SOURCE (or INSTALL-ON-WINDOWS) files to see how to get the stuff.

A next step might be to overhaul the PickleRPC protocol (hey, it's still not released, so you can't expect me to keep the interface ) to only work when you have RSA authentication set up and to provide encryption via randomly generated session secrets that are passed along RSA-encrypted. That way the shared secret can be completely removed (I never really liked it, anyway).

Update: I now did the above and changed PickleRPC to be tied to RSA authentication and making use of session keys. This removes the need for a shared secret.