Natural Order Development

Standing there asking people for money is not an easy thing to do. Even more important is figuring out how much to charge. A lot of people feel guilty and charge far less then they should. Further you always feel like "man if I charge more I wont get hired." Well take it from me there is an upper limit, but it is higher then you probably think. I have personally seen consultants with only a few years experience charge $250 / hour and get large contracts, while really smart people charging only $50 / hour are always on the verge of starving.

So what is a fair rate for full-time consultants? Well there is a pretty simple standard formula for hourly rate which is "(current annual salary * 2) / 1000".

Then number break down like this.

• As a consultants you will have unexpected additional costs than a salaried employee. For example there is "self employment tax" which is an additional 15% on top of what you already have to pay in income tax. You also have to pay all the state employer taxes on yourself. Then of course you have to provide your buy your own benefits (I suggest putting 25% off the top in an IRA or other investment plan for yourself and then taking another 30% and putting that in a high interest savings account to cover taxes). Don't forget insurances your medical, dental, auto, home insurance are not to be skipped and also very important is umbrella or professional liability insurance (if something goes wrong you will need it). So to maintain your current standard of living, expect to have to bill double your current annual salary.
  
  
• You will need to cover sick time, vacation time, marketing time and of course time for project over runs. So you will not be able to call all 2000 working hours a year billable (especially when you're just starting out). Most consulting books will say the billable hour range is between 1200-1500 billing hours per year. Trust me and be conservative use the lower end of this formula.
  
  
• You also have office expenses, paper, business cards, invoicing materials and other professional services such as a CPA to review your books and give you advice on how to keep track of your budgets to make tax preparation easier.
  
  
• Adjust you salary to the level you want. If think you should be earning more adjust the expected salary to match what people in your profession and local area are making. Try checking out the online consulting sites and job board sites to get an idea of the salary ranges in your area.
  

Good luck!

Back at the job and code crawling. Got more done this weekend on my site project then I did in the last year. Of course a new kid will always crush your schedule for a couple of years. So not really complaining.

The main page now shows the site goals. I think I need to expand on those a little bit. But first I need to figure out how to integrate a source code control system. I want to be able to get at my source code from anywhere and I definitely want to be able to publish parts of it here.

I found a great code highlighting engine and there is a Wiki page on that now. (**Edit this with the right links**). LOL this will be a test to show I managed to get re-edit ability active for posts that YOU own. I am thinking something like a re-edit with change log should be about right.

Well it has been a cruddy day. I have been trying to figure out a problem with the fcgi implementation. Seem to have been running circles around myself here.

There is a bug in the Django source specifically in django/core/servers/fastcgi.py that was ignores the output settings unless the process is being daemonized. So I added the else clause below has fixed it so I am now getting stdout and stderr messages.

    if daemonize:
        from django.utils.daemonize import become_daemon
        become_daemon(our_home_dir=options["workdir"], **daemon_kwargs)
    else:
        if options['outlog']:
            so = open(options['outlog'], 'a+', 0)
            os.dup2(so.fileno(), sys.stdout.fileno())
        if options['errlog']:
            se = open(err_log, 'a+', 0)
            os.dup2(se.fileno(), sys.stderr.fileno())

Except that that breaks the fcgi interactions. With that change I am not able to get responses through the auto-generated named pipes. It seems I have to hand it the bind name for the pipe. But if I do that and daemonize it then lightttpd looses control and if I stop the services the python processes persist.

On the other hand I am getting a lot of code crawling experience on the python classes. Which leads me to another observation.

Why are so many open source developers so lazy about documentation. If you are going to publish something you dang well had better provide solid design docs, API references and user docs. But, no fcgi is not well documented, lighttpd is not well documented, django is not well documented and Sphene Community Tools are not well documented. Which means I have to spend hours of code crawling to figure anything out.

And it isn't just open source developers I have the same problems at work too. Major enterprise systems with cruddy docs and lots of black magic development lead to a nightmare of work. I have heard some people laugh at this and say "well that is called job security." It isn't job security it is crappy development and laziness. My customers have plenty of additional work they want me to do. I am sure they could give me a list 60 years long.

<SIGH> well back to code crawling.

Oh what a quagmire of crud to walk through. All of the Django docs on FCGI and lighttpd are pretty sparse and the examples are not what I would recommend for a production service. So first step was to improve the FCGI tie in for Django. This turned out better then I thought. I had to do a lot of guessing and testing. But now I have it set up so that when the lighttpd service is started and stopped the FCGI processes are also started and stopped. In other words the web service application actually controls its own subprocesses including the FCGI linking to python for the Django service.

I'll have to create a wiki pages on how I configured this place. But to save you time if you want to see my script here it is:

-- BEG start-fcgi.sh --

#!/usr/bin/env bash

NOW=`date +%Y-%m-%d_%T`
MY_LOG_DIR=/tmp
[ -n "${OUT_LOG}" ] && MY_LOG_DIR=${OUT_LOG%/*}
[ -n "${DJANGO_LOG_DIR}" ] && MY_LOG_DIR=${DJANGO_LOG_DIR}
[ ! -d $MY_LOG_DIR ] && mkdir -p  $MY_LOG_DIR
[ ! -d $MY_LOG_DIR ] && echo "$0 ERROR log dir $MY_LOG_DIR is not a directory" >&2 && exit -1
MY_LOG=${MY_LOG_DIR}/start_fcgi_${NOW}_$$.out
rm -rf $MY_LOG
echo "$0 process $$ started log file = $MY_LOG"
echo "$0 process $$ started" >> ${MY_LOG}
MY_DIR=${0%/*}
cd $MY_DIR
echo "command line: $0 $@" > ${MY_LOG}
echo "----- BEG ENV -----" >> ${MY_LOG}
env >> ${MY_LOG}
echo "----- END ENV -----" >> ${MY_LOG}
PIDFILE="pidfile=${MY_LOG_DIR}/django_fcgi_$$.pid"
[ -n "${DJANGO_PID}" ] && PIDFILE="pidfile=${DJANGO_PID}"

# if socket is not specified the fcgi process handling will negotiate the correct
# socket file via the fcgi API so best to NOT specify it
SOCKET=""
[ -n "${DJANGO_SOCKET}" ] && SOCKET="socket=${DJANGO_SOCKET}"
METHOD="method=prefork"
[ -n "${DJANGO_METHOD}" ] && METHOD="method=${DJANGO_METHOD}"

# stdout and stderr will go to the http server logs if not otherwise specified
OUTLOG=""
[ -n "${OUT_LOG}" ] && OUTLOG="outlog=${OUT_LOG}"
ERRLOG=""
[ -n "${ERROR_LOG}" ] && ERRLOG="errlog=${ERROR_LOG}"

CMD="./manage.py runfcgi $METHOD $SOCKET $PIDFILE $OUTLOG $ERRLOG"

echo "executing : exec ${CMD}" >> ${MY_LOG}

exec ${CMD}

-- END start-fcgi.sh --


You will note that this is controlled via the lighttpd.conf file. Here is the relevant section of mine:

-- BEG lighttpd.conf subsection --

## Special fastcgi handling for each domain under django python
# If the hostname is '*.nodsw.com' or 'nodsw.com' ...
$HTTP["host"] =~ "(^|\.)nodsw.com" {
    fastcgi.server += (
        "/mysite.fcgi" => (
            "main" => (
                "socket" => "/tmp/nodsw.socket", # This will get auto-incremented by fcgi
                "check-local" => "disable",
                "bin-path" => "/django/nodsw.com/start_fcgi.sh",
                "bin-environment" => (
                    "DJANGO_LOG_DIR"         => "/django/logs",
                    #"ERROR_LOG"              => "/django/logs/nodsw.django_error.log",
                    #"OUT_LOG"                => "/django/logs/nodsw.django_out.log",
                    #"DJANGO_PID"             => "/django/logs/nodws.django.pid"
                    #"DJANGO_SOCKET"          => "/tmp/nodsw.socket", # don't use this unless you are sure
                    #"PYTHONPATH" => "/django/greydragon/project"
                )
            )
        )
    )
    alias.url = (
        "/media/" => "/django/nodsw.com/media/",
    )

    url.rewrite-once = (
        "^(/media.*)$" => "$1",
        "^/favicon\.ico$" => "/media/favicon.ico",
        "^(/.*)$" => "/mysite.fcgi$1",
    )
}

-- END lighttpd.conf subsection --

The database is running. Django seems happy. I have managed to get things kind of set up. Still need to figure out a lot of little things. But it is a start. I hope to have Natural Order Software up and rolling as well as Greydragon off of the same code base. We'll see.

I just did a code review on a change to a production service in Java which was having problems with left over open handles. Lets face it the change is so simple as to be ridiculous. We just added an explicit file.close() call.

However, part of the process of code review is to examine the unit tests which where added to insure the problem is indeed fixed. But, from Java how do you get a count of the open file handles? Although that by itself is a bit of a sticky problem, it is far more interesting to ask how do you write a unit test to confirm that the open handles are not increasing pre and post a particular function call? Essentially Java will happily toss an exception that there are too many open files. Looking at the Java docs on that particular IO Exception:

Exception in thread "main" java.io.IOException: Cannot open file "/tmp/tooManyFiles.65537": error=24, Too many open files

On Linux its not difficult to find out if a program is leaking file handles by either counting the open handles in the /proc directory or even easier running the command 'lsof -p <PID>'. So you can kick of any program on Linux and then while it is running run the command 'ls /proc/<PID>/fd | wc -w' or more accurately 'lsof -p <PID>'. My problem is that this will not work on Windows, or many other OSs on which Java runs.

So how to do this from Java? Well I thought about it and reasoned that it should be possible to the the handle count before calling a method, call the method and when it returns count the open handles. Do this a few hundred times and you should see little or no change in the file handle count. Which would be easy if there was a hook to get at the open file handles held by the Virtual Machine.

Well there is an open bug with Sun on this:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4404721

I wish I could say I found a really nifty Java only solution to this. But, truth is I needed to get a solution done and move on. Since the code was for a Linux target we could take the short cut and call out to Runtime.getRuntime().exec() to run the 'lsof -p <PID>' command and parse the results. BTW watch out for the 3 extra file handles opened by the above call! We tripped over that next.

Which leads me to the very pertinent side note that I also discovered another interesting and related issue with Java in that you need to explicitly close the output file handles (stdin, stdout and stderr) allocated by any exec process. The Java RunTime.exec() (e.g. 'Process p = Runtime.getRuntime().exec("/bin/true");') does not release them after the invoked program terminates.

Which is interesting and known but not mentioned in the javadoc for the process class and just about all the authoritative resources get this wrong. For example:

http://www.ibm.com/developerworks/java/library/j-tiger09304.html
http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html

There is even an acknowledged bug report that was filed on this in 2006 asking for the documentation to be improved:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6462165

This weekend turned out completely backwards.

Question: What happens when you throw a party and no one shows up?

Answer: You have a completely different party!

I had 30 friends swear they were coming to the free karaoke party I was roped into helping with Friday night. They even went so far as to tell me a day or so before hand how much they were looking forward to going AND that they had a couple friends that were coming too. So I figured at least 10 would show up. ;^)

The other people putting on the party also called around and we had confirmed about 60 people. We made sure there was enough food for about 40 figuring we could order pizzas if there wasn't enough.

Grand total of 16 people came... Which even with all the free booze was not enough to really do karaoke... So I reprogrammed the machine tossed in a bunch of 80s music, dragged out the poker table and we played poker to loud 80s music with TONS of booze, snacks, and deserts. Everyone had a hell of a good time. Lots of people were sent home with enough left overs to eat for a week.

After the poker game broke up at about 10 PM three of us headed out to the pool hall and played pool till 2 AM.

Friday night - not what I expected but fun none-the-less

What does this have to do with anything? Think about it. There is a lesson in there.