Ok...I did some more testing.  I think I've narrowed it down a bit.

The server has been running ok all weekend.  I have the physical
terminal here...as well as another desktop I've been using to connect. 
I'll also been ssh'ing from a machine outside this network, as well as
testing http requests.  

I'm still seeing some grind-to-a-halt...but I don't think it's the
machine now.  I am constantly watching 'top' the physical machine, and
it's always reasonable.  It's always 100% responsive too.  So...it must
be something in the networking.  

The temperatures seem to be ok, and the logs aren't filling up.

Now...the network card on that machine is a bit loose on the
connection.  It becomes disconnected if you touch the cord wrong (or
bump the desk wrong even.)  Could that be it?  That would be an easy one
to replace.  I could also try moving the card to a different pci slot I
suppose.  

Any thoughts?

Matt