Natural Order Development

Been dealing a lot of SCRUM items and it seems to me that a lot of teams keep missing the mark because of not clear "done criteria". For the record I think a minimum done criteria is:

• Code is checked-in into a central version control system
  
• A detailed code review (including comparison to any company and/or team coding standards) has been complete, any suggested changes implemented and final code has been checked-in
  
• Unit tests with demonstratable cover coverage of more 80% for non-integration / "out of container"
  
• Automated Integration Coverage (AIC) or Fit / FitNess integration tests are written with code coverage more than 80% (aka "in container")
  
• Exploratory manual testing by someone other then the developer has been complete with no open defects
  
• Some kind of profiling (a la JProfiler) has been complete on the developer's system and results are documented in release notes
  
• Release notes and/or usage documentation has been written (and in the appropriate place)

The team needs to keep these items in mind when estimating any backlog, unless these items are complete, the backlog item should not be called done. Of course there are more done criteria specifying the story or behavior requirements which are the responsibility of the product owner to provide (and the team to question heavily on.)

Secure string implementation is very hard. A secure string is a means to keep and use confidential data. Essentially the text cannot be stored as plain text and definitely cannot be stored in a Java String object since that is immutable and there is no definable point when it will be removed from the system.

Secure String Implementations should:

• text is encrypted for privacy when being used
  
• deleted from computer memory when no longer needed
  
• not be allowed to be paged to swap memory
  
• cannot be seen via memory inspection

I was very excited about secure strings when I learned of them. But I have not been able to find an implementation for Java. I am sure better minds then me are looking into this problem. But, I have need now. So breaking the cardinal rule of "don't implement cryptographic solutions unless you have a PhD in math and another in cryptology" I am making a stab at it.

I have found this one article which is very excellent (but for .NET):

• Credential Management with the .NET Framework 2.0 http://msdn2.microsoft.com/en-us/library/aa480470.aspx
  
  "Summary: Get an introduction to the Credential Management API that includes functions for user interface handling and lesser-known functions for managing a user's credential set. Also see a .NET class library that dramatically simplifies the task of credential management, for languages such as C# and Visual Basic .NET, and provides a more elegant and robust approach to credential management for C++ developers. (26 printed pages)"

I always love a good laugh and this one is great. The developers and testers at Google have put up a great "How to Write 3v1L, Untestable Code" article with tongue in cheek and enough sarcasm to really tickle your thought processes. In reality is a list of things not to do as a developer. I love the reverse style delivery. You can find the complete post here: Google Testing Blog (http://googletesting.blogspot.com/2008/07/how-to-write-3v1l-untestable-code.html).

I work on a beefy Linux box with multiple desktops & monitors. I also have a windows laptop to do office junk and some testing. If my computers are side by side things are pretty sweet and I use Synergyc / Synergy2 (http://synergy2.sourceforge.net/) to make all my monitors flow together.

But, I also have another beefy Linux development box in a shared team space where again I have multiple desktops & monitors. No problem I just use rdesktop (http://www.rdesktop.org/)to make one of the monitors turn into my windows screen. All is fine in the world.

Until I need to use SSH from my windows box (it has access to some things that are not on the Linux network). When I run PuTTY (a window ssh client) through an rdesktop connection my mouse pointer turns from the nice white arrow I use to a black I inside my PuTTY windows. Normally not a big deal, except that I prefer color schemes which have near black, or very dark backgrounds, essentially rendering the mouse pointer invisible. So to find it I have to move it around a lot or selection some text to find it.

Wasn't sure if this was because of rdesktop graphic handling or something else. But I was pretty sure it was related since when directly on the Windows box the I-beam pointer turned white. So it had something to do with the interactions of PuTTY and rdesktop.

Naturally this eventually drove me nuts. (Officially this happened about 20 minutes before I wrote this. I wanted to paste some shell script stuff into a PuTTY session and it went... well never mind it was bad.) So annoyance became a real menace that demanded to be crushed under the totalitarian control of my domains. I alone will decide the fate of the bits in my systems. And this was a challenge to my authority.

A little poking at the mouse settings fixed. It a little more research into why it happened explained it. Turns out PuTTY uses "Text Select" pointer for all things within its borders. But it does set some internal color bits on the pointer when it does this. This is not a bug, just a surprising interaction for which there is no real way for PuTTY to know what it should do. The solution is to alter the "Text Select" pointer (Control Panel->Mouse->Pointers->Text Select) to a bit-map image, problem solved. There are actually a number of already provided images for this all called beam_XX.cur, I chose beam_rm.cur as it was nice and thick and had a white outline built in. Problem solved, order restored and I can see my mouse pointer in PuTTY when working via rdesktop.

Well 2 days of preparation and a lot of reading really paid off. The last few years I have not been very happy with my photographic results from events like the 4th of July shows. So this year I decided instead of being a rank amateur I would try and figure it out.

I do not have an expensive digital camera, it is a Canon A630 Powershot. But, I have figured out how to take enough good pictures that I am actually happy with it. I went through about 12 digital cameras since 1985 and until I bought this one I ended up giving them away to friends because I was so upset with performance I couldn't stand to look at the things. I was very happy taking pictures with my 1985 Canon Sureshot 35mm and I took fabulous almost professional quality pictures with that little work horse. Nighttime photos are just terrible. So what was I doing wrong?

Then I ran across the LifeHacker (http://lifehacker.com/387380/turn-your-point+and+shoot-into-a-super+camera)article on CHDK a bios replacement OS for digital cameras. After reading the article I decided to try it.

Then I read almost every page I could Google on CHDK, motion capture with CHDK and lightning capture with CHDK.

Armed with a lot of ideas, no notes (why I do not know) and a vague idea of what the aperture setting was about I played around and finally settled on the CHDK script Motion/Lightning-Detect for Canon A640 by Johan Van Barel http://chdk.wikia.com/wiki/UBASIC/Scripts/CanonA640:_Motion%2BLightning_Detect.

So in a 30 minute show using threshold=10, interval=5ms, delay=5, and blocksize=6 I took about 180 pictures with a manual focus set to infinity, a shutter time of 1 second and the default aperture and using the custom driver speed of taking 2 pictures every time the shutter is called. End results, WOW I am very, very happy with the results.

Here are some samples:









Pretty cool since I set it up, pushed the button and sat down. The camera sat there for more then 30 minutes before the show started and it took the pictures all by itself the whole show. All I did was sit back and enjoy. This is the way I like it. Let the computers do the work of recording and let me enjoy my life.

OK so this took more then a while to figure out. I had to go read the FAST CGI specification (http://www.fastcgi.com/devkit/doc/fcgi-spec.html), the lighttpd FastCGI Interface docs (http://trac.lighttpd.net/trac/wiki/Docs:ModFastCGI), the Django FastCGI docs (http://www.djangoproject.com/documentation/fastcgi/) and a number of other forum posts in a number of groups.

All this just to answer the question: How do I get my Python Django applications to print to the the lighttpd error logs when using Lighttpd + FastCGI?

At last I found the answer. At least partially...

The crux of this problem is that FastCGI is a multi-threaded and multiplexed communication protocol. Just calling print is not enough. The output streams need to be encoded and directed to the right channel and in a way that the FCGI service can untangle them to know where they should be routed.

This is a messy problem due to the complexity, the poor documentation, the incomplete state of the specifications and also the incomplete state of the FCGI supporting modules on both the server and client ends.

So here is the basic trick.

You can not do this from a settings file. The streams are not available UNTIL there is a call that causes the FCGI instance to initiate a request. The streams will be available via the request call. They might be set up in the environment (there are supposed to a couple of hooks for you to look for and then snag onto). I haven't finished getting all this figured out yet. But, I have succeeded in having a multi-threaded call to a single FCGI channel successfully process all the data into responses with resulting messages ending up in the expected logs.

Once you have the request it is not too difficult to do this. For example within the views of a Django application you can do this:

if 'wsgi.errors' in request.META:
    request.META['wsgi.errors'].writelines("An error message for the server's error logs.\n")
    request.META['wsgi.errors'].flush()

Now that I know where to find these channels and I know how to use them correctly (using them correctly is the key). I can experiment with a few other tricks I know to see if I can make this as easy as:

if isFCGI():
    dup2StdStreams()

print "This should go to the access log."
print >>sys.stderr "This should go to the error log."

If not then I can at least hook into the python logging and set up the logging to go to the server's access and error logs.

With a few code reviews and tweaks to SCT Tools basic captcha support is now online. I still need to integrate the captcha call into the user registration system.

The nice part is now email addresses are hidden unless a captcha check is done. Further captcha checks time out after a short period. Is this open to some kind of mining pass? Possibly a replay attach. Need to check that out. Of course Django is supposed to be pretty solid. Need to see about coping cookies around a little.

Back in 1988 I started a BBS in San Diego. I put out a lot of money, installed 2 separate phone lines, had state of the art 2400 baud modems and an Amiga 2000 with nearly 100 MB of disk space and a eye popping 2 MB of RAM. I coded and tested for hundreds of hours. Finally it was ready and I put it online. I published in the local computer magazines that it was up and running. Then I sat and watched it for hours waiting for someone to come use what I created for them. And I waited, and I waited and ... well you get the picture.

After days of waiting I got a call, they didn't sign up for an account. Then I finally got my first user. He posted to an area and shared a file. I answered him instantly. Then I waited some more. I can remember the first 10 people on that system. A few of them are still friends today.

And again. I have a nearly complete system. But this place has far more then my original BBS every had. Yet again, I find I am sitting here waiting for that first hit. Excited by the idea of the possible community that will form.

Well in the end my BBS in San Diego had 10 phone lines 14400 baud modems and over a hundred regular members who logged in and chatted almost daily and about 500 odd total members who actually came back once a month or so. It was a great time. I expect this place will do the same.

So here I am laughing at myself because I am waiting and checking every few minutes to see what activity has happened. Even though I know almost nothing about this place is completed. I'll probably put in a few thousand more hours of work before I am really happy with this place. Still having some folks around to at least share the results with will be fun.

Hope you like the possibilities enough to come back around from time to time.

Well in a moment of testing need I decided to determine if the backups where working. So I took a backup of the database and restored it to an entirely different database service. I altered the database handler to point at the new database service and kicked the web services.

What a surprise it actually all came back up just like it is supposed to.

I am in shock. Not only are all the forums back in place, the posts are there, the blogs are there and all the test user data is there too.

I call that one heck of a test. I am feeling a lot more confident in this process now. Especially since I moved this from a testing database to a production database service.

Excellent test of the services, the automation and the disaster recovery process.

Not to mention my testing database service can now be redirected to more serious poking and prodding for the layout of all the database models used for this service.

Oh what a relief, I was trying to figure out why none of the static elements of the site were working. Well not really. I was actually trying to determine how the templates get all wrangled up when I noticed a lot of 404 error messages in the log files. Which is when I figured out why some of the site's interfaces were not working exactly right.

Turned out that although the template loaders for Django have a nice "go find the template relative to the application asking for it" feature. The static stuff like CSS pages, images and other elements do not have that ability. Instead they all ask for the /media/ directory.

So what was breaking was two fold:

I had a typo in the alias url definition I forgot the trailing slash. Curd I have been bitten by this in Apache and now in lighttpd too. If you alias a directory say "/media/" then you need to specify where to look for that directory and that needs to have a matching trailing slash. So I had this:

    alias.url = (
        "/media/" => "/srv/nodsw.com/media",
    )

when what I was supposed to do was:

    alias.url = (
        "/media/" => "/srv/nodsw.com/media/",
    )

Big time dooh! I even knew better. Oh well. It is fixed now. So thing are looking much better.