Natural Order Development

Every 6 months or so I have some little loop I wrote to run a process on a long list of servers sit and hang forever by waiting for the nohup background process to end before moving on to the next host. Naturally I come back after getting lunch expecting all the boxes to be busily humming away to find 3 have finished and the 4th is still running. This of course causes me to remember that ssh hangs on processes even if they are pushed off to the background with a nohup. A while back I discovered this and after digging around I found that this is actually expected and correct (well correct for the standard) behavior. The problem is in the handling of stdin and stdout. This can be overcome by redirecting all the pipes to something NOT ssh.

ssh user@host 'nohup command </dev/null >/dev/null 2>&1 &'

Don't bother complaining about this. It is a known issue, but is not considered a bug according to the POSIX standard which ssh complies with. Essentially if a process is still attached to the tty ssh should not close the connection.