Ghosting with Linux

[Garrett Goebel]

Jason Clinton wrote:
> Jason Clinton wrote:
> > I'll let everyone know how the 10 GB NTFS partition
> > with 7 GB of data on it compresses down to and how
> > long it took to make a backup accross a 100 Mbit
> > connection.
>
> The back-up file ended up being around 5GB. We had
> 100 mbits (as indicated by the switch) between the two
> systems. The backup took about 6 hours. That's less
> than a 10 mbit transfer rate. I was not able to
> ascertain the bottleneck. CPU usage was below 30%.
> The hard drive lights on both boxen were blinking
> intermittently.

Single or multi-processor system?

Things you could do to decrease the transfer time:
o defrag the drive
o zero all free drive space
http://www.sysinternals.com/ntw2k/source/sdelete.shtml
o set pagefile.sys to zero on shutdown
Hive: HKEY_LOCAL_MACHINE
Key: SYSTEMCurrentControlSetControlSessionManagerMemory Management
Name: ClearPageFileAtShutdown
Type: REG_DWORD
Value: 1

dd is copying 10GB regardless, but compression will be better

Let's decompose:

dd if=/dev/hda1 | bzip2 -9c | ssh [user at addr] dd of=/hda1.bz2

To test, I generated a random 1GB file using:

dd if=/dev/urandom of=/1GB bs=1M count=1000

Here's timing info gathered on gentoo quad xeon PIII 600MHz kona box with
scsi-II ultra hardware raid 5 and 512MB memory.

time	MB/s	cpu	cmd
2:32 6.58	~05%	dd if=/1GB of=/1GB.dd
36:31	0.46	~25%	bzip2 -9c 1GB > 1GB.bz2
20:29	0.81	~25%	bzip2 -1c 1GB > 1GB.bz2
2:55	5.71	~20%	lzop -c 1GB > 1GB.lzo
3:32	4.71	~50%	dd if=/1GB | ssh user at localhost dd of=/1GB.ssh
4:11	3.98	~40%  dd if=/1GB | lzop -c | ssh user at localhost dd
of=/1GB.ssh

I messed around with dd bs= values, but it didn't make a significant
difference. The sticking point appears to bzip compression speed. Given
these numbers, I'd guesstimate it'd take around 4 1/2 hours just to compress
a 7GB file on my system.

What to do? I looked around and found that lzop is considered one of the
fastest compression utilities. I've gone back and plugged the times for lzop
itself into the above table. I then tried the full command line substituting
lzop instead of bzip.

So given my numbers, at 3.98MB/s, I'd estimate it'd take you ~30 minutes to
backup 7GB. That'd generate at least 32Mb/sec traffic on your network. Which
100Mb ethernet should be able to handle... network activity permitting. Let
me know how it works out ;)

So in summary:

backup:
dd if=/dev/hda1 | lzop -c | ssh [user at addr] dd of=/hda1.lzo

restore:
ssh [user at addr] lzop -dc /hda1.lzo | dd of=/dev/hda1

--
Garrett Goebel
IS Development Specialist
ScriptPro                   Direct: 913.403.5261
5828 Reeds Road               Main: 913.384.1008
Mission, KS 66202              Fax: 913.384.2180
www.scriptpro.com          garrett at scriptpro dot com