It's a jungle out there...

For my sins, I work as a System Admin for a reasonable size Department in large tertiary Institution (well, for now - but that's another story!).
One part of my job involves specifying and commissioning new servers and storage. Lately we have been using iSCSI storage servers attached to computational servers for the likes of geophysical and engineering applications such as high-rate GPS data processing. These servers are typically 8 core 48 megabyte machines and the storage attached them is measured in 10's of terabytes.

Normally, on receipt of a new machine the installed operating system is wiped, our preferred OS is put on it and it is run through our check procedures before it goes anywhere near our network. This time, I bought an appliance storage server - one controlled not from the command line, but from a web interface. Consequently, it wasn't wiped on arrival (and it had a 16 terabyte array on it that would have taken days to re-initialise). The upshot of the matter was that this appliance was not vetted as thoroughly as it should have been - I accept total blame for this, and can only plead advancing age and senility.

When the email from our security team arrived some 40 hours later to say that this machine was port scanning , I slapped my forehead, pulled the network cable and started a console session to confirm how stupid I had been.

The rootkit was, fortunately, not very smart. But you don't need to be smart when the box you infect has the root account activated with a password of 123456 - that I didn't change on arrival. The scripted login contacted a website in Rumania, downloaded a tarball, moved it into the /var/tmp directory, expanded it, then proceeded to run various commands designed to start ssh scanning sessions from this machine across the internet.
Careful checking on the contents of the rootkit and some careful monitoring of the rootkit in a virtual machine revealed that no modified binaries were installed, and the damage was limited to the ssh scanning sessions.

The machine is now on a private network, attached only to the computational server, with firewall rules to stop any errant ssh sessions - not that there has been any attempt to start any more.

The moral of the story is that any computer that is prepared for connection to the internet must be treated as if it could be infected from minute one.

Right, back to writing out 10000 times " I must change the root password on all new machines before attaching a network cable"