I think my server has been hacked

[Brian Densmore]

(this is a repost as the original never seems to have made it through.)

Well in the initial analysis I was rooted about 3am on the 
8th. The cracker installed at the least the shv5 rootkit.
He may have manipulated some/all of the log files. He definitely
trashed the login log file, but he missed some of my security
procedures. Initially it looks as though it was a remote
ssl exploit. I have an event in my apache log indicating how
he did it. But it may be he just deleted the tail of the log.
He didn't even bother changing the timestamps on the root kit
trojans he installed. It's a rather strange cracker, he wasn't
very thorough and did some odd things. Anyway, I digress.
I know what the initial rootkit was and when he did it and where he 
got it from. I don't yet know where he came from and am not
sure I can trust the logs to tell me that. I am interested in seeing
what he did with the system once he got in. He did put
the NIC in promiscuous mode and created virtual IPs for the
entire network the server was sitting on. So it looks like he
was just using it for sniffing. (Since the box really didn't have
enough space to be useful for much other than a tiny mailserver/webserver).

So, I'm interesting in hearing what you all think I should/can do
to try and track this person, and where on my disks to look. I didn't 
notice anything in the home directories of note, unless he's found a way
of hiding the files from ls -al. I was thinking about running a 
rootkit searching program on the disks to see if there is more than one.
I haven't yet gotten to the point of reading all the configs in /etc yet.
He definitely: altered the rc scripts, modified the ssh functionality,
replaced several programs, installed some nefarious libraries and scripts,
restarted inetd with a rooted version, restarted the webserver. I'm sure there's
more to discover. This server didn't have any thing of import on it, other
than my personal long-neglected website, and frequently used mail server.
Both of which are off-line until I can finish building my new debian based more secure
box. I knew eventually this box would be rooted. He's been trying very hard since
December. At least I believe it is the same person, although there has been
extensive attempts for some time. Not sure why it was so popular. It was a Mandrake
Bastille hardened system, that sadly was not properly maintained by me. But,
I have learned some since I built this machine 4 years ago. The new box will be
more secure and built by hand rather than from a package.

(sorry about the length)

Thanks,
Brian

 Brian

"Three OS's from corporate-kings in their towers of glass, 
Seven from valley-lords where orchards used to grow, 
Nine from dotcoms doomed to die, 
one from the dark lord Gates on his dark throne 
In the Land of Redmond where the Shadows lie. 
one OS to rule them all, one OS to find them, 
one OS to bring them all and in the darkness bind them, 
In the Land of Redmond where the Shadows lie."    john thrum