Natural Order Development

OK so this took more then a while to figure out. I had to go read the FAST CGI specification (http://www.fastcgi.com/devkit/doc/fcgi-spec.html), the lighttpd FastCGI Interface docs (http://trac.lighttpd.net/trac/wiki/Docs:ModFastCGI), the Django FastCGI docs (http://www.djangoproject.com/documentation/fastcgi/) and a number of other forum posts in a number of groups.

All this just to answer the question: How do I get my Python Django applications to print to the the lighttpd error logs when using Lighttpd + FastCGI?

At last I found the answer. At least partially...

The crux of this problem is that FastCGI is a multi-threaded and multiplexed communication protocol. Just calling print is not enough. The output streams need to be encoded and directed to the right channel and in a way that the FCGI service can untangle them to know where they should be routed.

This is a messy problem due to the complexity, the poor documentation, the incomplete state of the specifications and also the incomplete state of the FCGI supporting modules on both the server and client ends.

So here is the basic trick.

You can not do this from a settings file. The streams are not available UNTIL there is a call that causes the FCGI instance to initiate a request. The streams will be available via the request call. They might be set up in the environment (there are supposed to a couple of hooks for you to look for and then snag onto). I haven't finished getting all this figured out yet. But, I have succeeded in having a multi-threaded call to a single FCGI channel successfully process all the data into responses with resulting messages ending up in the expected logs.

Once you have the request it is not too difficult to do this. For example within the views of a Django application you can do this:

if 'wsgi.errors' in request.META:
    request.META['wsgi.errors'].writelines("An error message for the server's error logs.\n")
    request.META['wsgi.errors'].flush()

Now that I know where to find these channels and I know how to use them correctly (using them correctly is the key). I can experiment with a few other tricks I know to see if I can make this as easy as:

if isFCGI():
    dup2StdStreams()

print "This should go to the access log."
print >>sys.stderr "This should go to the error log."

If not then I can at least hook into the python logging and set up the logging to go to the server's access and error logs.