Natural Order Development

OK so this took more then a while to figure out. I had to go read the FAST CGI specification (http://www.fastcgi.com/devkit/doc/fcgi-spec.html), the lighttpd FastCGI Interface docs (http://trac.lighttpd.net/trac/wiki/Docs:ModFastCGI), the Django FastCGI docs (http://www.djangoproject.com/documentation/fastcgi/) and a number of other forum posts in a number of groups.

All this just to answer the question: How do I get my Python Django applications to print to the the lighttpd error logs when using Lighttpd + FastCGI?

At last I found the answer. At least partially...

The crux of this problem is that FastCGI is a multi-threaded and multiplexed communication protocol. Just calling print is not enough. The output streams need to be encoded and directed to the right channel and in a way that the FCGI service can untangle them to know where they should be routed.

This is a messy problem due to the complexity, the poor documentation, the incomplete state of the specifications and also the incomplete state of the FCGI supporting modules on both the server and client ends.

So here is the basic trick.

You can not do this from a settings file. The streams are not available UNTIL there is a call that causes the FCGI instance to initiate a request. The streams will be available via the request call. They might be set up in the environment (there are supposed to a couple of hooks for you to look for and then snag onto). I haven't finished getting all this figured out yet. But, I have succeeded in having a multi-threaded call to a single FCGI channel successfully process all the data into responses with resulting messages ending up in the expected logs.

Once you have the request it is not too difficult to do this. For example within the views of a Django application you can do this:

if 'wsgi.errors' in request.META:
    request.META['wsgi.errors'].writelines("An error message for the server's error logs.\n")
    request.META['wsgi.errors'].flush()

Now that I know where to find these channels and I know how to use them correctly (using them correctly is the key). I can experiment with a few other tricks I know to see if I can make this as easy as:

if isFCGI():
    dup2StdStreams()

print "This should go to the access log."
print >>sys.stderr "This should go to the error log."

If not then I can at least hook into the python logging and set up the logging to go to the server's access and error logs.

With a few code reviews and tweaks to SCT Tools basic captcha support is now online. I still need to integrate the captcha call into the user registration system.

The nice part is now email addresses are hidden unless a captcha check is done. Further captcha checks time out after a short period. Is this open to some kind of mining pass? Possibly a replay attach. Need to check that out. Of course Django is supposed to be pretty solid. Need to see about coping cookies around a little.

Back in 1988 I started a BBS in San Diego. I put out a lot of money, installed 2 separate phone lines, had state of the art 2400 baud modems and an Amiga 2000 with nearly 100 MB of disk space and a eye popping 2 MB of RAM. I coded and tested for hundreds of hours. Finally it was ready and I put it online. I published in the local computer magazines that it was up and running. Then I sat and watched it for hours waiting for someone to come use what I created for them. And I waited, and I waited and ... well you get the picture.

After days of waiting I got a call, they didn't sign up for an account. Then I finally got my first user. He posted to an area and shared a file. I answered him instantly. Then I waited some more. I can remember the first 10 people on that system. A few of them are still friends today.

And again. I have a nearly complete system. But this place has far more then my original BBS every had. Yet again, I find I am sitting here waiting for that first hit. Excited by the idea of the possible community that will form.

Well in the end my BBS in San Diego had 10 phone lines 14400 baud modems and over a hundred regular members who logged in and chatted almost daily and about 500 odd total members who actually came back once a month or so. It was a great time. I expect this place will do the same.

So here I am laughing at myself because I am waiting and checking every few minutes to see what activity has happened. Even though I know almost nothing about this place is completed. I'll probably put in a few thousand more hours of work before I am really happy with this place. Still having some folks around to at least share the results with will be fun.

Hope you like the possibilities enough to come back around from time to time.

Well in a moment of testing need I decided to determine if the backups where working. So I took a backup of the database and restored it to an entirely different database service. I altered the database handler to point at the new database service and kicked the web services.

What a surprise it actually all came back up just like it is supposed to.

I am in shock. Not only are all the forums back in place, the posts are there, the blogs are there and all the test user data is there too.

I call that one heck of a test. I am feeling a lot more confident in this process now. Especially since I moved this from a testing database to a production database service.

Excellent test of the services, the automation and the disaster recovery process.

Not to mention my testing database service can now be redirected to more serious poking and prodding for the layout of all the database models used for this service.

Oh what a relief, I was trying to figure out why none of the static elements of the site were working. Well not really. I was actually trying to determine how the templates get all wrangled up when I noticed a lot of 404 error messages in the log files. Which is when I figured out why some of the site's interfaces were not working exactly right.

Turned out that although the template loaders for Django have a nice "go find the template relative to the application asking for it" feature. The static stuff like CSS pages, images and other elements do not have that ability. Instead they all ask for the /media/ directory.

So what was breaking was two fold:

I had a typo in the alias url definition I forgot the trailing slash. Curd I have been bitten by this in Apache and now in lighttpd too. If you alias a directory say "/media/" then you need to specify where to look for that directory and that needs to have a matching trailing slash. So I had this:

    alias.url = (
        "/media/" => "/srv/nodsw.com/media",
    )

when what I was supposed to do was:

    alias.url = (
        "/media/" => "/srv/nodsw.com/media/",
    )

Big time dooh! I even knew better. Oh well. It is fixed now. So thing are looking much better.

Standing there asking people for money is not an easy thing to do. Even more important is figuring out how much to charge. A lot of people feel guilty and charge far less then they should. Further you always feel like "man if I charge more I wont get hired." Well take it from me there is an upper limit, but it is higher then you probably think. I have personally seen consultants with only a few years experience charge $250 / hour and get large contracts, while really smart people charging only $50 / hour are always on the verge of starving.

So what is a fair rate for full-time consultants? Well there is a pretty simple standard formula for hourly rate which is "(current annual salary * 2) / 1000".

Then number break down like this.

• As a consultants you will have unexpected additional costs than a salaried employee. For example there is "self employment tax" which is an additional 15% on top of what you already have to pay in income tax. You also have to pay all the state employer taxes on yourself. Then of course you have to provide your buy your own benefits (I suggest putting 25% off the top in an IRA or other investment plan for yourself and then taking another 30% and putting that in a high interest savings account to cover taxes). Don't forget insurances your medical, dental, auto, home insurance are not to be skipped and also very important is umbrella or professional liability insurance (if something goes wrong you will need it). So to maintain your current standard of living, expect to have to bill double your current annual salary.
  
  
• You will need to cover sick time, vacation time, marketing time and of course time for project over runs. So you will not be able to call all 2000 working hours a year billable (especially when you're just starting out). Most consulting books will say the billable hour range is between 1200-1500 billing hours per year. Trust me and be conservative use the lower end of this formula.
  
  
• You also have office expenses, paper, business cards, invoicing materials and other professional services such as a CPA to review your books and give you advice on how to keep track of your budgets to make tax preparation easier.
  
  
• Adjust you salary to the level you want. If think you should be earning more adjust the expected salary to match what people in your profession and local area are making. Try checking out the online consulting sites and job board sites to get an idea of the salary ranges in your area.
  

Good luck!

Back at the job and code crawling. Got more done this weekend on my site project then I did in the last year. Of course a new kid will always crush your schedule for a couple of years. So not really complaining.

The main page now shows the site goals. I think I need to expand on those a little bit. But first I need to figure out how to integrate a source code control system. I want to be able to get at my source code from anywhere and I definitely want to be able to publish parts of it here.

I found a great code highlighting engine and there is a Wiki page on that now. (**Edit this with the right links**). LOL this will be a test to show I managed to get re-edit ability active for posts that YOU own. I am thinking something like a re-edit with change log should be about right.

Well it has been a cruddy day. I have been trying to figure out a problem with the fcgi implementation. Seem to have been running circles around myself here.

There is a bug in the Django source specifically in django/core/servers/fastcgi.py that was ignores the output settings unless the process is being daemonized. So I added the else clause below has fixed it so I am now getting stdout and stderr messages.

    if daemonize:
        from django.utils.daemonize import become_daemon
        become_daemon(our_home_dir=options["workdir"], **daemon_kwargs)
    else:
        if options['outlog']:
            so = open(options['outlog'], 'a+', 0)
            os.dup2(so.fileno(), sys.stdout.fileno())
        if options['errlog']:
            se = open(err_log, 'a+', 0)
            os.dup2(se.fileno(), sys.stderr.fileno())

Except that that breaks the fcgi interactions. With that change I am not able to get responses through the auto-generated named pipes. It seems I have to hand it the bind name for the pipe. But if I do that and daemonize it then lightttpd looses control and if I stop the services the python processes persist.

On the other hand I am getting a lot of code crawling experience on the python classes. Which leads me to another observation.

Why are so many open source developers so lazy about documentation. If you are going to publish something you dang well had better provide solid design docs, API references and user docs. But, no fcgi is not well documented, lighttpd is not well documented, django is not well documented and Sphene Community Tools are not well documented. Which means I have to spend hours of code crawling to figure anything out.

And it isn't just open source developers I have the same problems at work too. Major enterprise systems with cruddy docs and lots of black magic development lead to a nightmare of work. I have heard some people laugh at this and say "well that is called job security." It isn't job security it is crappy development and laziness. My customers have plenty of additional work they want me to do. I am sure they could give me a list 60 years long.

<SIGH> well back to code crawling.

Oh what a quagmire of crud to walk through. All of the Django docs on FCGI and lighttpd are pretty sparse and the examples are not what I would recommend for a production service. So first step was to improve the FCGI tie in for Django. This turned out better then I thought. I had to do a lot of guessing and testing. But now I have it set up so that when the lighttpd service is started and stopped the FCGI processes are also started and stopped. In other words the web service application actually controls its own subprocesses including the FCGI linking to python for the Django service.

I'll have to create a wiki pages on how I configured this place. But to save you time if you want to see my script here it is:

-- BEG start-fcgi.sh --

#!/usr/bin/env bash

NOW=`date +%Y-%m-%d_%T`
MY_LOG_DIR=/tmp
[ -n "${OUT_LOG}" ] && MY_LOG_DIR=${OUT_LOG%/*}
[ -n "${DJANGO_LOG_DIR}" ] && MY_LOG_DIR=${DJANGO_LOG_DIR}
[ ! -d $MY_LOG_DIR ] && mkdir -p  $MY_LOG_DIR
[ ! -d $MY_LOG_DIR ] && echo "$0 ERROR log dir $MY_LOG_DIR is not a directory" >&2 && exit -1
MY_LOG=${MY_LOG_DIR}/start_fcgi_${NOW}_$$.out
rm -rf $MY_LOG
echo "$0 process $$ started log file = $MY_LOG"
echo "$0 process $$ started" >> ${MY_LOG}
MY_DIR=${0%/*}
cd $MY_DIR
echo "command line: $0 $@" > ${MY_LOG}
echo "----- BEG ENV -----" >> ${MY_LOG}
env >> ${MY_LOG}
echo "----- END ENV -----" >> ${MY_LOG}
PIDFILE="pidfile=${MY_LOG_DIR}/django_fcgi_$$.pid"
[ -n "${DJANGO_PID}" ] && PIDFILE="pidfile=${DJANGO_PID}"

# if socket is not specified the fcgi process handling will negotiate the correct
# socket file via the fcgi API so best to NOT specify it
SOCKET=""
[ -n "${DJANGO_SOCKET}" ] && SOCKET="socket=${DJANGO_SOCKET}"
METHOD="method=prefork"
[ -n "${DJANGO_METHOD}" ] && METHOD="method=${DJANGO_METHOD}"

# stdout and stderr will go to the http server logs if not otherwise specified
OUTLOG=""
[ -n "${OUT_LOG}" ] && OUTLOG="outlog=${OUT_LOG}"
ERRLOG=""
[ -n "${ERROR_LOG}" ] && ERRLOG="errlog=${ERROR_LOG}"

CMD="./manage.py runfcgi $METHOD $SOCKET $PIDFILE $OUTLOG $ERRLOG"

echo "executing : exec ${CMD}" >> ${MY_LOG}

exec ${CMD}

-- END start-fcgi.sh --


You will note that this is controlled via the lighttpd.conf file. Here is the relevant section of mine:

-- BEG lighttpd.conf subsection --

## Special fastcgi handling for each domain under django python
# If the hostname is '*.nodsw.com' or 'nodsw.com' ...
$HTTP["host"] =~ "(^|\.)nodsw.com" {
    fastcgi.server += (
        "/mysite.fcgi" => (
            "main" => (
                "socket" => "/tmp/nodsw.socket", # This will get auto-incremented by fcgi
                "check-local" => "disable",
                "bin-path" => "/django/nodsw.com/start_fcgi.sh",
                "bin-environment" => (
                    "DJANGO_LOG_DIR"         => "/django/logs",
                    #"ERROR_LOG"              => "/django/logs/nodsw.django_error.log",
                    #"OUT_LOG"                => "/django/logs/nodsw.django_out.log",
                    #"DJANGO_PID"             => "/django/logs/nodws.django.pid"
                    #"DJANGO_SOCKET"          => "/tmp/nodsw.socket", # don't use this unless you are sure
                    #"PYTHONPATH" => "/django/greydragon/project"
                )
            )
        )
    )
    alias.url = (
        "/media/" => "/django/nodsw.com/media/",
    )

    url.rewrite-once = (
        "^(/media.*)$" => "$1",
        "^/favicon\.ico$" => "/media/favicon.ico",
        "^(/.*)$" => "/mysite.fcgi$1",
    )
}

-- END lighttpd.conf subsection --