From dean-list-rdiff-backup@arctic.org Sun Jun 2 02:40:13 2002 From: dean-list-rdiff-backup@arctic.org (dean gaudet) Date: Sat, 1 Jun 2002 18:40:13 -0700 (PDT) Subject: --exclude-filelist and shell patterns In-Reply-To: <200205310837.g4V8bb031039@folly.Stanford.EDU> Message-ID: hmm... is there any way to give a list of extended shell patterns in a file? i thought --exclude-filelist did this, but on re-reading the docs i see it doesn't... i want to put patterns like: /tmp/* /var/tmp/* /home/*/tmp/* /proc/* into a file and have them exclude the contents of the directories without excluding the directories themselves (so that it is easier to do a full restore in the future). these work with --exclude on the command line, but not if i put them into a --exclude-filelist file. -dean From bescoto@stanford.edu Sun Jun 2 08:16:20 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Sun, 02 Jun 2002 00:16:20 -0700 Subject: --exclude-filelist and shell patterns In-Reply-To: Your message of "Sat, 01 Jun 2002 18:40:13 PDT." Message-ID: <200206020716.g527GKs04803@folly.Stanford.EDU> --==_Exmh_-962271661P Content-Type: text/plain; charset=us-ascii >>>>> "DG" == dean gaudet >>>>> wrote the following on Sat, 1 Jun 2002 18:40:13 -0700 (PDT) DG> hmm... is there any way to give a list of extended shell DG> patterns in a file? Well, suppose you have a list of shell patterns in file FILE. You could say: EXCLUDES=`xargs -ri echo --exclude \'{}\' i want to put patterns like: DG> /tmp/* /var/tmp/* /home/*/tmp/* /proc/* DG> into a file and have them exclude the contents of the DG> directories without excluding the directories themselves (so DG> that it is easier to do a full restore in the future). DG> these work with --exclude on the command line, but not if i put DG> them into a --exclude-filelist file. I thought that file lists would usually be generated by some process like find, and quoting problems could be avoided by not interpreting the filelist. But now that you mention it, files containing newlines can still screw up the process. It seems the best way to handle this would be to scan the filelist, and assume that the first of \0 and \n is the file separator. That should allow the use of find -print0, and still work in 99.99% of all cases. Or maybe there should be a --null-separator switch, to override to \0, and this could also determine the directory_statistics separator... -- Ben Escoto --==_Exmh_-962271661P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE8+cZD+owuOvknOnURAqKPAKCB3sFymhriXguUb8GngNB5L3CNCgCfSBa2 kzhz8hSSeZvlPQeOE8H2n+k= =RVBW -----END PGP SIGNATURE----- --==_Exmh_-962271661P-- From jebc@c4solutions.net Mon Jun 3 03:38:23 2002 From: jebc@c4solutions.net (Jeb Campbell) Date: Sun, 02 Jun 2002 22:38:23 -0400 Subject: error (Index ('CSCOST.DAT',) already covered, skipping) Message-ID: <3CFAD69F.8030805@c4solutions.net> Using rdiff-backup 0.7.6 (also occurred with 0.7.4). This error occurs on a few files and prevents the file from being backed up: Error 'Index ('CSCOST.DAT',) already covered, skipping' processing CSCOST.DAT Skipping CSCOST.DAT because of previous error The file is readable by the backup user (and even owned by it) to see if that was the problem, but no luck. Could have 0.7.4 messed up some files, and I need to delete the the files it won't backup to get it going again? Thanks, Jeb From bescoto@stanford.edu Mon Jun 3 09:52:21 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Mon, 03 Jun 2002 01:52:21 -0700 Subject: error (Index ('CSCOST.DAT',) already covered, skipping) In-Reply-To: Your message of "Sun, 02 Jun 2002 22:38:23 EDT." <3CFAD69F.8030805@c4solutions.net> Message-ID: <200206030852.g538qLq25561@folly.Stanford.EDU> --==_Exmh_725370674P Content-Type: text/plain; charset=us-ascii >>>>> "JC" == Jeb Campbell >>>>> wrote the following on Sun, 02 Jun 2002 22:38:23 -0400 JC> Using rdiff-backup 0.7.6 (also occurred with 0.7.4). This error JC> occurs on a few files and prevents the file from being backed JC> up: JC> Error 'Index ('CSCOST.DAT',) already covered, skipping' JC> processing CSCOST.DAT Skipping CSCOST.DAT because of previous JC> error But the backup as a whole successfully finishes? What files are in rdiff-backup-data/* and rdiff-backup-data/increments/CSCOST.DAT*? This error is supposed to occur in the following situation: Suppose that the last successful backup is run at time t1. Then at time t2 a second backup is run, but it fails. A third backup runs at t3. The t3 session is supposed to record in the increments directory the state of the mirror directory at time t1. But sometimes this job was already done by the t2 session, so the file is skipped. If the last session succeeded you shouldn't see that message. -- Ben Escoto --==_Exmh_725370674P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE8+y5C+owuOvknOnURAnmaAJ9KDBfDdsHqyUZZQWkpjZ7ZbY2tjACgiy9k U59+8aY6W7S7QT6Cs/ovbx0= =o5BB -----END PGP SIGNATURE----- --==_Exmh_725370674P-- From nj.lee@plumtree.co.nz Wed Jun 5 23:27:59 2002 From: nj.lee@plumtree.co.nz (Nicholas Lee) Date: Thu, 6 Jun 2002 10:27:59 +1200 Subject: --bwlimit option? Message-ID: <20020605222759.GB960@inktiger.kiwa.co.nz> Is there any work on a --bwlimit (rate limiter) option in rdiff? Thanks. -- Nicholas Lee - nj.lee at plumtree.co dot nz, somewhere on the fish Maui caught. gpg. 8072 4F86 EDCD 4FC1 18EF 5BDD 07B0 9597 6D58 D70C icq. 1612865 Quixotic Eccentricity From bescoto@stanford.edu Thu Jun 6 01:48:30 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Wed, 05 Jun 2002 17:48:30 -0700 Subject: --bwlimit option? In-Reply-To: Your message of "Thu, 06 Jun 2002 10:27:59 +1200." <20020605222759.GB960@inktiger.kiwa.co.nz> Message-ID: <200206060048.g560mUl21572@folly.Stanford.EDU> --==_Exmh_26155693P Content-Type: text/plain; charset=us-ascii >>>>> "NL" == Nicholas Lee >>>>> wrote the following on Thu, 6 Jun 2002 10:27:59 +1200 NL> Is there any work on a --bwlimit (rate limiter) option in rdiff? You mean in rdiff-backup? Someone asked about this before, but there wasn't much discussion. My thought that all is required is to limit how much data passes through the pipe to ssh, and that there is nothing rdiff-backup specific about the situation at all. Given that, perhaps it might be best to use a 3rd party utility instead of building this functionality into rdiff-backup. Then, perhaps later a special --bwlimit type switch could be added which just passes some parameter to this other utility, the way --ssh-no-compression does now. But if this is the wrong approach for some reason then it seems more code and special switches should be added to rdiff-backup. -- Ben Escoto --==_Exmh_26155693P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE8/rFd+owuOvknOnURAhf2AJ9SEpVM51haCyoFBKKu1b4oPZG0MQCeNyEH DKXQ2+zW/PSniCnFArsdvJo= =LgEb -----END PGP SIGNATURE----- --==_Exmh_26155693P-- From nj.lee@plumtree.co.nz Thu Jun 6 01:57:51 2002 From: nj.lee@plumtree.co.nz (Nicholas Lee) Date: Thu, 6 Jun 2002 12:57:51 +1200 Subject: --bwlimit option? In-Reply-To: <200206060048.g560mUl21572@folly.Stanford.EDU> References: <20020605222759.GB960@inktiger.kiwa.co.nz> <200206060048.g560mUl21572@folly.Stanford.EDU> Message-ID: <20020606005751.GL960@inktiger.kiwa.co.nz> On Wed, Jun 05, 2002 at 05:48:30PM -0700, Ben Escoto wrote: > > You mean in rdiff-backup? Someone asked about this before, but there Yes. > wasn't much discussion. My thought that all is required is to limit > how much data passes through the pipe to ssh, and that there is > nothing rdiff-backup specific about the situation at all. Given that, > perhaps it might be best to use a 3rd party utility instead of Any suggestions on a 3rd party app? Two apps I know wget and rsync both do it themselves. I've looked around and haven't really seen anything. Seems to be the only generic way would be via some proxy. Which seems like a lot to add for a simple 'sleep for a period and let tcp rate limit itself'. > building this functionality into rdiff-backup. Then, perhaps later a > special --bwlimit type switch could be added which just passes some > parameter to this other utility, the way --ssh-no-compression does > now. Personally I wish ssh had this option. > But if this is the wrong approach for some reason then it seems > more code and special switches should be added to rdiff-backup. Maybe code can be borrowed from wget or rsync. -- Nicholas Lee - nj.lee at plumtree.co dot nz, somewhere on the fish Maui caught. gpg. 8072 4F86 EDCD 4FC1 18EF 5BDD 07B0 9597 6D58 D70C icq. 1612865 Quixotic Eccentricity From bescoto@stanford.edu Thu Jun 6 02:28:42 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Wed, 05 Jun 2002 18:28:42 -0700 Subject: --bwlimit option? In-Reply-To: Your message of "Thu, 06 Jun 2002 12:57:51 +1200." <20020606005751.GL960@inktiger.kiwa.co.nz> Message-ID: <200206060128.g561Sgx21880@folly.Stanford.EDU> --==_Exmh_39472363P Content-Type: text/plain; charset=us-ascii >>>>> "NL" == Nicholas Lee >>>>> wrote the following on Thu, 6 Jun 2002 12:57:51 +1200 NL> Any suggestions on a 3rd party app? Two apps I know wget and NL> rsync both do it themselves. I've looked around and haven't NL> really seen anything. Well, both of those need to do their own networking, but rdiff-backup only cares about ssh. NL> Seems to be the only generic way would be via some proxy. Which NL> seems like a lot to add for a simple 'sleep for a period and let NL> tcp rate limit itself'. The first entry on Freshmeat under "bandwidth limiting" is cstream. Homepage is at: http://www.cons.org/cracauer/cstream.html I've never tried it, but it looks like it might do the job. NL> Personally I wish ssh had this option. Yep, makes sense. (Or maybe all the ssh developers are trying to pawn the responsibility off just like I am...) NL> Maybe code can be borrowed from wget or rsync. Maybe, or at least the general idea how to do it.. -- Ben Escoto --==_Exmh_39472363P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE8/rq7+owuOvknOnURAiFbAKCNWoRoSyiUdZD09Ru2v9f/9sYZogCfaIgX 3IPmmubG8lTHe1iuGTyiLm0= =6ugq -----END PGP SIGNATURE----- --==_Exmh_39472363P-- From nj.lee@plumtree.co.nz Thu Jun 6 02:39:02 2002 From: nj.lee@plumtree.co.nz (Nicholas Lee) Date: Thu, 6 Jun 2002 13:39:02 +1200 Subject: --bwlimit option? In-Reply-To: <200206060128.g561Sgx21880@folly.Stanford.EDU> References: <20020606005751.GL960@inktiger.kiwa.co.nz> <200206060128.g561Sgx21880@folly.Stanford.EDU> Message-ID: <20020606013901.GM960@inktiger.kiwa.co.nz> On Wed, Jun 05, 2002 at 06:28:42PM -0700, Ben Escoto wrote: > The first entry on Freshmeat under "bandwidth limiting" is cstream. > Homepage is at: > > http://www.cons.org/cracauer/cstream.html Missed that one. Thanks! > NL> Maybe code can be borrowed from wget or rsync. > > Maybe, or at least the general idea how to do it.. man wget: --limit-rate=amount Limit the download speed to amount bytes per second. Amount may be expressed in bytes, kilobytes with the k suffix, or megabytes with the m suffix. For example, --limit-rate=20k will limit the retrieval rate to 20KB/s. This kind of thing is useful when, for whatever reason, you don't want Wget to consume the entire evailable bandwidth. Note that Wget implementeds the limiting by sleeping the appropriate amount of time after a net­ work read that took less time than specified by the rate. Eventually this strategy causes the TCP transfer to slow down to approximately the specified rate. However, it takes some time for this balance to be achieved, so don't be surprised if limiting the rate doesn't work with very small files. Also, the "sleeping" strategy will misfire when an extremely small bandwidth, say less than 1.5KB/s, is specified. -- Nicholas Lee - nj.lee at plumtree.co dot nz, somewhere on the fish Maui caught. gpg. 8072 4F86 EDCD 4FC1 18EF 5BDD 07B0 9597 6D58 D70C icq. 1612865 Quixotic Eccentricity From Kevin.Spicer@bmrb.co.uk Fri Jun 14 16:00:31 2002 From: Kevin.Spicer@bmrb.co.uk (Spicer, Kevin) Date: Fri, 14 Jun 2002 16:00:31 +0100 Subject: 0.7.6 slower than 0.6.1 Message-ID: This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C213B4.3A3CB214 Content-Type: text/plain; charset="iso-8859-1" Hi, I recently upgraded from 0.6.1 stable to 0.7.6 development, in the hopes it might help speed up my backup to a remote machine. 0.7.6 is taking much longer (13 hours rather than about 8 hours). The only other change I made was to recompile python (the build I was using didn't have zlib support which caused 0.7.6 to bomb). I'd previously added compression to ssh (remote-schema) and also changed the encryption to use arcfour (because its much quicker). In total this had skimmed about an hour or so off the total. I don't think it is a bandwidth issue as the link doesn't seem to be hammered at all. The load on the destination machine is hovering around 1.00, but the source machine is about half that (probably because the processor is twice the speed of the destination machine). Does anyone have any further suggestions for tuning this? If I can't sort this out I may have to go back to 0.6.1 - is this likely to cause any problems? My configuration... Source machine Sun Ultra 2 (296Mhz processor) Solaris 2.6 Source disk 18G 88% full lots of small files Destination machine Sun Ultra 1 (143Mhz processor) Solaris 2.6 BMRB International http://www.bmrb.co.uk +44 (0)20 8566 5000 ____________________________________________________________ This message (and any attachment) is intended only for the recipient and may contain confidential and/or privileged material. If you have received this in error, please contact the sender and delete this message immediately. Disclosure, copying or other action taken in respect of this email or in reliance on it is prohibited. BMRB International Limited accepts no liability in relation to any personal emails, or content of any email which does not directly relate to our business. ------_=_NextPart_001_01C213B4.3A3CB214 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable 0.7.6 slower than 0.6.1

Hi,
        I = recently upgraded from 0.6.1 stable to 0.7.6 development, in the hopes = it might help speed up my backup to a remote machine.  0.7.6 is = taking much longer (13 hours rather than about 8 hours).  The only = other change I made was to recompile python (the build I was using = didn't have zlib support which caused 0.7.6 to bomb).  I'd = previously added compression to ssh (remote-schema) and also changed = the encryption to use arcfour (because its much quicker).  In = total this had skimmed about an hour or so off the total.  I don't = think it is a bandwidth issue as the link doesn't seem to be hammered = at all.  The load on the destination machine is hovering around = 1.00, but the source machine is about half that (probably because the = processor is twice the speed of the destination machine).  Does = anyone have any further suggestions for tuning this?

If I can't sort this out I may have to go back to = 0.6.1 - is this likely to cause any problems?

My configuration...
Source machine  Sun Ultra 2 (296Mhz processor) = Solaris 2.6
Source disk  18G 88% full lots of small = files

Destination machine Sun Ultra 1 (143Mhz processor) = Solaris 2.6


BMRB International
http://www.bmrb.co.uk +44 (0)20 8566 5000

____________________________________________________________ =
This message (and any attachment) is intended only = for the recipient and may contain confidential and/or privileged = material. If you have received this in error, please contact the sender = and delete this message immediately. Disclosure, copying or other = action taken in respect of this email or in reliance on it is = prohibited. BMRB International Limited accepts no liability in relation = to any personal emails, or content of any email which does not directly = relate to our business.

------_=_NextPart_001_01C213B4.3A3CB214-- From bescoto@stanford.edu Fri Jun 14 19:48:04 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Fri, 14 Jun 2002 11:48:04 -0700 Subject: 0.7.6 slower than 0.6.1 In-Reply-To: Your message of "Fri, 14 Jun 2002 16:00:31 BST." Message-ID: <200206141848.g5EIm4r32666@folly.Stanford.EDU> --==_Exmh_1518177607P Content-Type: text/plain; charset=us-ascii >>>>> "KS" == Kevin Spicer >>>>> wrote the following on Fri, 14 Jun 2002 16:00:31 +0100 KS> Hi, I recently upgraded from 0.6.1 stable to 0.7.6 development, KS> in the hopes it might help speed up my backup to a remote KS> machine. 0.7.6 is taking much longer (13 hours rather than KS> about 8 hours). It's probably just that rdiff-backup has gotten more CPU hungry while I've been adding features. How many files are there, about 500000? I hope 0.9.x will see some speed enhancements. KS> If I can't sort this out I may have to go back to 0.6.1 - is KS> this likely to cause any problems? I don't think so, but when you restore you'll have to use 0.7.x if there were any compressed diff files, as 0.6.x won't recognize them. -- Ben Escoto --==_Exmh_1518177607P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE9Cjpj+owuOvknOnURAmU5AJ90fhycbN4RIjwmUsqdnDr9NezaoACcC/KI cHAZgwV+B7DLDGpqdAhcBHc= =WVId -----END PGP SIGNATURE----- --==_Exmh_1518177607P-- From bescoto@stanford.edu Sat Jun 15 00:00:25 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Fri, 14 Jun 2002 16:00:25 -0700 Subject: New stable version 0.8.0 released Message-ID: <200206142300.g5EN0P203899@folly.Stanford.EDU> --==_Exmh_1591840987P Content-Type: text/plain; charset=us-ascii Apparently 0.8.0 is a bit slower than 0.6.0, but I think most people should upgrade because it should be significantly more reliable and handle errors much better. It also has some new features (new restore syntax, include/exclude options, hard links, compressed increment files, statistics, etc.). Here is the changelog entry (only covers 0.7.6 to 0.8.0): New in v0.8.0 (2002/06/14) -------------------------- Added --null-separator argument so filenames can safely include newlines in an include/exclude filelist. Fixed bug that affected restoring from current mirror with the '-r now' option. -- Ben Escoto --==_Exmh_1591840987P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE9CnWH+owuOvknOnURAuTIAJ9DhGBu0LWHzCGZukSIJs2Cw5W0IACbBRiq c3cAhqsxAMTEWa1kJfWUoEI= =3FD+ -----END PGP SIGNATURE----- --==_Exmh_1591840987P-- From bescoto@stanford.edu Sat Jun 15 00:02:22 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Fri, 14 Jun 2002 16:02:22 -0700 Subject: New website: http://rdiff-backup.stanford.edu/ Message-ID: <200206142302.g5EN2Mu03939@folly.Stanford.EDU> --==_Exmh_1592161900P Content-Type: text/plain; charset=us-ascii Hopefully this address will be easier to remember and type... -- Ben Escoto --==_Exmh_1592161900P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE9CnX8+owuOvknOnURAmaUAJsERaZ1YiccTpUPd8alURxfA6gdZgCcCJS/ QEa+eT8ViZIcjL4UnG1XLng= =wfoP -----END PGP SIGNATURE----- --==_Exmh_1592161900P-- From Kevin.Spicer@bmrb.co.uk Sat Jun 15 21:42:05 2002 From: Kevin.Spicer@bmrb.co.uk (Spicer, Kevin) Date: Sat, 15 Jun 2002 21:42:05 +0100 Subject: 0.7.6 slower than 0.6.1 Message-ID: This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C214AD.1C234E2E Content-Type: text/plain; charset="iso-8859-1" >It's probably just that rdiff-backup has gotten more CPU hungry while >I've been adding features. How many files are there, about 500000? Its about 1.2 Million at the moment. but only a relatively small subset of these change daily (62000 today). Obviously these kinds of things are largely OS and hardware dependent, but how does these figures sound in relation to the time its taking? > I hope 0.9.x will see some speed enhancements. Great I'll try and keep up with the development releases to see if I can get any improvement. KS> If I can't sort this out I may have to go back to 0.6.1 - is KS> this likely to cause any problems? > I don't think so, but when you restore you'll have to use 0.7.x if > there were any compressed diff files, as 0.6.x won't recognize them. Do you think compressing the diffs is adding extra time? Can this be turned off (my destination disk is 6G bigger that the source dir of 18G and I delete increments after 7D). As a thought are the diffs compressed at source or destination end? I would hope destination, or else I'd guess that the compression on the ssh link is going to make them bigger(?) -- Ben Escoto BMRB International http://www.bmrb.co.uk +44 (0)20 8566 5000 ____________________________________________________________ This message (and any attachment) is intended only for the recipient and may contain confidential and/or privileged material. If you have received this in error, please contact the sender and delete this message immediately. Disclosure, copying or other action taken in respect of this email or in reliance on it is prohibited. BMRB International Limited accepts no liability in relation to any personal emails, or content of any email which does not directly relate to our business. ------_=_NextPart_001_01C214AD.1C234E2E Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable RE: 0.7.6 slower than 0.6.1

>It's probably just that rdiff-backup has gotten = more CPU hungry while
>I've been adding features.  How many files = are there, about 500000?
 
Its about 1.2 Million at the moment.  but only = a relatively small subset of these change daily (62000 today).  = Obviously these kinds of things are largely OS and hardware dependent, = but how does these figures sound in relation to the time its = taking?

> I hope 0.9.x will see some speed = enhancements.

Great I'll try and keep up with the development = releases to see if I can get any improvement.

    KS> If I can't sort this out I = may have to go back to 0.6.1 - is
    KS> this likely to cause any = problems?

> I don't think so, but when you restore you'll = have to use 0.7.x if
> there were any compressed diff files, as 0.6.x = won't recognize them.

Do you think compressing the diffs is adding extra = time?  Can this be turned off (my destination disk is 6G bigger = that the source dir of 18G and I delete increments after 7D).  As = a thought are the diffs compressed at source or destination end?  = I would hope destination, or else I'd guess that the compression on the = ssh link is going to make them bigger(?)

--
Ben Escoto


BMRB International
http://www.bmrb.co.uk +44 (0)20 8566 5000

____________________________________________________________ =
This message (and any attachment) is intended only = for the recipient and may contain confidential and/or privileged = material. If you have received this in error, please contact the sender = and delete this message immediately. Disclosure, copying or other = action taken in respect of this email or in reliance on it is = prohibited. BMRB International Limited accepts no liability in relation = to any personal emails, or content of any email which does not directly = relate to our business.

------_=_NextPart_001_01C214AD.1C234E2E-- From dean-list-rdiff-backup@arctic.org Sat Jun 15 22:07:47 2002 From: dean-list-rdiff-backup@arctic.org (dean gaudet) Date: Sat, 15 Jun 2002 14:07:47 -0700 (PDT) Subject: 0.7.6 slower than 0.6.1 In-Reply-To: Message-ID: On Sat, 15 Jun 2002, Spicer, Kevin wrote: > Its about 1.2 Million at the moment. but only a relatively small subset of > these change daily (62000 today). Obviously these kinds of things are > largely OS and hardware dependent, but how does these figures sound in > relation to the time its taking? a non-trivial amount of your overhead on sol2.6 is going to be needless atime modifications on the mirror server. if you can upgrade the mirror to 7 or 8 (i forget which) you'll be able to mount the mirror with noatime and eliminate these disk writes. 2.6 can also get into lots of trouble when you've got a lot of filenames longer than 31 characters in length (the basename without the path). it has in kernel caching for filenames up to 31 characters in length, but when you get a bunch of files longer than that the kernel wastes lots of i/o re-reading directory contents and such. i'm pretty sure this is fixed in 7 as well. (this last problem shows up a lot for folks using maildir -- the default filenames are really long...) if your mirror is a dedicated server doing not much else you could probably upgrade it without affecting production services ... as for changes between 0.6 and 0.7 that might affect this -- i haven't looked, but the problem with long filenames is exacerbated by doing many syscalls with paths in them. i know the strace of 0.7.x has a lot of overkill in file operations. but i haven't looked at 0.6.x to see if any more were added. ("truss" is the closest command to strace on solaris 2.6...) there's cases where patterns like: creat() close() chown() can be replaced with creat() fchown() close() which is generally faster everywhere due to the lack of extra name lookup (and more secure). dunno if rdiff-backup has any such opportunities for improvement. -dean From Kevin.Spicer@bmrb.co.uk Sun Jun 16 00:56:23 2002 From: Kevin.Spicer@bmrb.co.uk (Spicer, Kevin) Date: Sun, 16 Jun 2002 00:56:23 +0100 Subject: 0.7.6 slower than 0.6.1 Message-ID: This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C214C8.40D1BE5C Content-Type: text/plain; charset="iso-8859-1" I agree, I'd like to upgrade all to sol 8. But unfortunately both are production servers running mission critical third party software which the vendors only support under 2.51 and 2.6. Whereas rdiff-backup is 'just' there to help us get back up quickly should the worst happen at the remote site (basically saves us the day it would take to get the backup tape to our HO), so I'm not able to upgrade the OS. Interesting you mention the atime stuff (it is 7 that adds this option BTW), I noticed that 0.7.6 doesn't seem to change the atime (or is it changing it then changing it back - could that be whats slowing it up?) I'm not that convinced that its all filesystem related though, I've just had a quick glance at top (rdiff-backup is running at present) the 'top' process is python (around 80% CPU) cpu states are about 80% user and 5-10% iowait. Bearing in mind that another machine is performing a remote backup to this machines tape drive I can't really complain about the iowait stats. [This is the destination machine, the source machine is much happier] Undoubtably some of this is due to hardware limitations on the Ultra1. Filenames are almost all short - usually 8.3 DOS like. Thanks for your feedback -----Original Message----- From: dean gaudet [mailto:dean-list-rdiff-backup@arctic.org] Sent: 15 June 2002 22:08 To: Spicer, Kevin Cc: 'rdiff-backup@keywest.stanford.edu' Subject: RE: 0.7.6 slower than 0.6.1 On Sat, 15 Jun 2002, Spicer, Kevin wrote: > Its about 1.2 Million at the moment. but only a relatively small subset of > these change daily (62000 today). Obviously these kinds of things are > largely OS and hardware dependent, but how does these figures sound in > relation to the time its taking? a non-trivial amount of your overhead on sol2.6 is going to be needless atime modifications on the mirror server. if you can upgrade the mirror to 7 or 8 (i forget which) you'll be able to mount the mirror with noatime and eliminate these disk writes. 2.6 can also get into lots of trouble when you've got a lot of filenames longer than 31 characters in length (the basename without the path). it has in kernel caching for filenames up to 31 characters in length, but when you get a bunch of files longer than that the kernel wastes lots of i/o re-reading directory contents and such. i'm pretty sure this is fixed in 7 as well. (this last problem shows up a lot for folks using maildir -- the default filenames are really long...) if your mirror is a dedicated server doing not much else you could probably upgrade it without affecting production services ... as for changes between 0.6 and 0.7 that might affect this -- i haven't looked, but the problem with long filenames is exacerbated by doing many syscalls with paths in them. i know the strace of 0.7.x has a lot of overkill in file operations. but i haven't looked at 0.6.x to see if any more were added. ("truss" is the closest command to strace on solaris 2.6...) there's cases where patterns like: creat() close() chown() can be replaced with creat() fchown() close() which is generally faster everywhere due to the lack of extra name lookup (and more secure). dunno if rdiff-backup has any such opportunities for improvement. -dean BMRB International http://www.bmrb.co.uk +44 (0)20 8566 5000 ____________________________________________________________ This message (and any attachment) is intended only for the recipient and may contain confidential and/or privileged material. If you have received this in error, please contact the sender and delete this message immediately. Disclosure, copying or other action taken in respect of this email or in reliance on it is prohibited. BMRB International Limited accepts no liability in relation to any personal emails, or content of any email which does not directly relate to our business. ------_=_NextPart_001_01C214C8.40D1BE5C Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable RE: 0.7.6 slower than 0.6.1

I agree, I'd like to upgrade all to sol 8.  But = unfortunately both are production servers running mission critical = third party software which the vendors only support under 2.51 and = 2.6.  Whereas rdiff-backup is 'just' there to  help us get = back up quickly should the worst happen at the remote site (basically = saves us the day it would take to get the backup tape to our HO), so = I'm not able to upgrade the OS.

Interesting you mention the atime stuff (it is 7 that = adds this option BTW), I noticed that 0.7.6 doesn't seem to change the = atime (or is it changing it then changing it back - could that be whats = slowing it up?)

I'm not that convinced that its all filesystem = related though, I've just had a quick glance at top (rdiff-backup is = running at present) the 'top' process is python (around 80% CPU)  = cpu states are about 80% user and 5-10% iowait.  Bearing in mind = that another machine is performing a remote backup to this machines = tape drive I can't really complain about the iowait stats.  [This = is the destination machine, the source machine is much happier]  =

Undoubtably some of this is due to hardware = limitations on the Ultra1.

Filenames are almost all short - usually 8.3 DOS = like.

Thanks for your feedback


-----Original Message-----
From: dean gaudet [mailto:dean-list-rdiff= -backup@arctic.org]
Sent: 15 June 2002 22:08
To: Spicer, Kevin
Cc: 'rdiff-backup@keywest.stanford.edu'
Subject: RE: 0.7.6 slower than 0.6.1


On Sat, 15 Jun 2002, Spicer, Kevin wrote:

> Its about 1.2 Million at the moment.  but = only a relatively small subset of
> these change daily (62000 today).  = Obviously these kinds of things are
> largely OS and hardware dependent, but how does = these figures sound in
> relation to the time its taking?

a non-trivial amount of your overhead on sol2.6 is = going to be needless
atime modifications on the mirror server.  if = you can upgrade the mirror
to 7 or 8 (i forget which) you'll be able to mount = the mirror with noatime
and eliminate these disk writes.

2.6 can also get into lots of trouble when you've got = a lot of filenames
longer than 31 characters in length (the basename = without the path).  it
has in kernel caching for filenames up to 31 = characters in length, but
when you get a bunch of files longer than that the = kernel wastes lots of
i/o re-reading directory contents and such.  = i'm pretty sure this is fixed
in 7 as well.

(this last problem shows up a lot for folks using = maildir -- the default
filenames are really long...)

if your mirror is a dedicated server doing not much = else you could
probably upgrade it without affecting production = services ...

as for changes between 0.6 and 0.7 that might affect = this -- i haven't
looked, but the problem with long filenames is = exacerbated by doing many
syscalls with paths in them.  i know the strace = of 0.7.x has a lot of
overkill in file operations.  but i haven't = looked at 0.6.x to see if any
more were added.  ("truss" is the = closest command to strace on solaris
2.6...)

there's cases where patterns like:

creat()
close()
chown()

can be replaced with

creat()
fchown()
close()

which is generally faster everywhere due to the lack = of extra name lookup
(and more secure).  dunno if rdiff-backup has = any such opportunities for
improvement.

-dean


BMRB International
http://www.bmrb.co.uk +44 (0)20 8566 5000

____________________________________________________________ =
This message (and any attachment) is intended only = for the recipient and may contain confidential and/or privileged = material. If you have received this in error, please contact the sender = and delete this message immediately. Disclosure, copying or other = action taken in respect of this email or in reliance on it is = prohibited. BMRB International Limited accepts no liability in relation = to any personal emails, or content of any email which does not directly = relate to our business.

------_=_NextPart_001_01C214C8.40D1BE5C-- From bescoto@stanford.edu Sun Jun 16 02:54:19 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Sat, 15 Jun 2002 18:54:19 -0700 Subject: 0.7.6 slower than 0.6.1 In-Reply-To: Your message of "Sun, 16 Jun 2002 00:56:23 BST." Message-ID: <200206160154.g5G1sJ502938@folly.Stanford.EDU> --==_Exmh_-1002388649P Content-Type: text/plain; charset=us-ascii >>>>> "KS" == Kevin Spicer >>>>> wrote the following on Sun, 16 Jun 2002 00:56:23 +0100 KS> Interesting you mention the atime stuff (it is 7 that adds KS> this option BTW), I noticed that 0.7.6 doesn't seem to change KS> the atime I think the way it should work is if a file just has a different atime, it is not considered to have changed and won't be updated. If it is marked as changed, then its atime will be updated along with the other stuff. KS> I'm not that convinced that its all filesystem related though, KS> I've just had a quick glance at top (rdiff-backup is running at KS> present) the 'top' process is python (around 80% CPU) cpu states KS> are about 80% user and 5-10% iowait. Yes, rdiff-backup eats a lot of CPU. There are undoubtedly inefficiencies along the lines Dean mentioned, but for most systems CPU (or possibly bandwidth) will be the limiting factor. Code that would be relatively innocuous in C can take up a lot of time in Python. See http://www.bagley.org/~doug/shootout/ for a good language speed comparision, with an overview at http://www.bagley.org/~doug/shootout/craps.shtml The scripting lanuages are generally about 20x or less slower than C, when the builtin scripting functions (like Python's fileIO, splitting strings, etc.) can do a moderate share of the work. When they can't the ratio drops to 200:1 (see the tests "Method Calls" and "Nested Loops"). Anyway, 0.9.x will be distributed as a bunch of .py[c] files instead of one big script (I am working on exploding it now), and should be a lot faster, especially after I rewrite some important parts in C. -- Ben Escoto --==_Exmh_-1002388649P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE9C+/J+owuOvknOnURArELAJ9mzj6efSsiQe7bk1qm+PpmOl7DjgCdFjc3 jbjCM43IIYqtrENAnr5dJng= =GhhP -----END PGP SIGNATURE----- --==_Exmh_-1002388649P-- From bescoto@stanford.edu Sun Jun 16 02:57:35 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Sat, 15 Jun 2002 18:57:35 -0700 Subject: 0.7.6 slower than 0.6.1 In-Reply-To: Your message of "Sat, 15 Jun 2002 21:42:05 BST." Message-ID: <200206160157.g5G1vZA02996@folly.Stanford.EDU> --==_Exmh_-1001883039P Content-Type: text/plain; charset=us-ascii >>>>> "KS" == Kevin Spicer >>>>> wrote the following on Sat, 15 Jun 2002 21:42:05 +0100 KS> Do you think compressing the diffs is adding extra time? Can KS> this be turned off (my destination disk is 6G bigger that the KS> source dir of 18G and I delete increments after 7D). As a KS> thought are the diffs compressed at source or destination end? KS> I would hope destination, or else I'd guess that the compression KS> on the ssh link is going to make them bigger(?) That's probably not the problem, but it can be turned off (--no-compression-regexp '.*'). There are two separate kinds of diffs used by rdiff-backup. The first are the reverse diffs that are written to the increments directory. They never travel through ssh. The second are forwards diffs used to update the old mirror files on the destination directory. These only get compressed by ssh (or not at all, if --ssh-no-compressed in used). -- Ben Escoto --==_Exmh_-1001883039P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE9C/CH+owuOvknOnURAqabAJ0WzGC1Jry9mruQYLWsvdKjXzru2wCfeSrS IBFo2DAoejPyomCOe5nTCeQ= =fI3P -----END PGP SIGNATURE----- --==_Exmh_-1001883039P-- From dean-list-rdiff-backup@arctic.org Sun Jun 16 03:21:28 2002 From: dean-list-rdiff-backup@arctic.org (dean gaudet) Date: Sat, 15 Jun 2002 19:21:28 -0700 (PDT) Subject: 0.7.6 slower than 0.6.1 In-Reply-To: <200206160154.g5G1sJ502938@folly.Stanford.EDU> Message-ID: On Sat, 15 Jun 2002, Ben Escoto wrote: > Yes, rdiff-backup eats a lot of CPU. There are undoubtedly > inefficiencies along the lines Dean mentioned, but for most systems > CPU (or possibly bandwidth) will be the limiting factor. at times there's a cpu limitation, but i'm guessing the problem is that rdiff-backup is mostly serialized. fixing that is a chore though :) i suspect that there'd be some benefit to spawning a couple of rdiffs in parallel. basically so that while one rdiff is blocked on reading data, another is calculating. with that change i know that my bottleneck would be bandwidth -- there's only a 128kbit uplink from my mirror to my primary (1.5mbit the other way). i can watch the uplink saturate when rdiff hits a large file and read-ahead can feed it data as fast as the cpu can do the checksums. when it's in amongst small files the uplink isn't saturated at all. in terms of scaling the mirror host to handle many primaries it might be nice to have rdiff-backup cache the signature files on the mirror. perhaps a file per directory with the names and signatures of the directory contents. not bulletproof -- if someone goes about mucking in the mirror they could damage things. but this would reduce the cpu and i/o requirements on the mirror host. my particular setup is working like a charm though. nightly finishes in 3 hours, and i've got about a 6 hour window in which i don't mind having a 1.5mbit outbound load from the server, so i've got room to grow. ~50GB of data in ~600k files. -dean From jasonthomasagius@hotmail.com Sun Jun 16 06:19:44 2002 From: jasonthomasagius@hotmail.com (Jay Dylan) Date: Sun, 16 Jun 2002 05:19:44 +0000 Subject: Rdiff-Backup and Samba Message-ID: Has anyone had any sucess mirroring to a samba share?.. source dir /software (on RH7.2) destination dir /replnet/samba (mounted windows share) [root@localhost store]# /software/rdiff-backup-0.7.6/rdiff-backup --windows-time-format --no-hard-links --force /software/ /replnet/samba/ Exception '[Errno 1] Operation not permitted' raised of class 'exceptions.OSError': File "/software/rdiff-backup-0.7.6/rdiff-backup", line 5992, in patch_w_datadir_writes SaveState.checkpoint(ITR, finalizer, dsrp) File "/software/rdiff-backup-0.7.6/rdiff-backup", line 3360, in checkpoint cls.record_last_file_action(last_file_rorp)).execute() File "/software/rdiff-backup-0.7.6/rdiff-backup", line 2978, in execute raise exc ------------ Old traceback info ----------- Exception '[Errno 1] Operation not permitted' raised of class 'exceptions.OSError': File "/software/rdiff-backup-0.7.6/rdiff-backup", line 2970, in execute init_val = self.init_thunk() File "/software/rdiff-backup-0.7.6/rdiff-backup", line 2999, in init init_return_vals.append(ra.init_thunk()) File "/software/rdiff-backup-0.7.6/rdiff-backup", line 3124, in init def init(): tf.symlink(linktext) File "/software/rdiff-backup-0.7.6/rdiff-backup", line 2412, in symlink self.conn.os.symlink(linktext, self.path) ------------------------------------------- Traceback (most recent call last): File "/software/rdiff-backup-0.7.6/rdiff-backup", line 6752, in ? Globals.Main.Main(sys.argv[1:]) File "/software/rdiff-backup-0.7.6/rdiff-backup", line 6461, in Main self.take_action(rps) File "/software/rdiff-backup-0.7.6/rdiff-backup", line 6438, in take_action elif self.action == "backup": self.Backup(rps[0], rps[1]) File "/software/rdiff-backup-0.7.6/rdiff-backup", line 6495, in Backup else: HighLevel.Mirror(rpin, rpout, self.incdir, RSI) File "/software/rdiff-backup-0.7.6/rdiff-backup", line 5774, in Mirror DestS.patch_w_datadir_writes(dest_rpath, diffiter, inc_rpath) File "/software/rdiff-backup-0.7.6/rdiff-backup", line 5996, in patch_w_datadir_writes except: cls.handle_last_error(finished_dsrp, finalizer, ITR) File "/software/rdiff-backup-0.7.6/rdiff-backup", line 5992, in patch_w_datadir_writes SaveState.checkpoint(ITR, finalizer, dsrp) File "/software/rdiff-backup-0.7.6/rdiff-backup", line 3360, in checkpoint cls.record_last_file_action(last_file_rorp)).execute() File "/software/rdiff-backup-0.7.6/rdiff-backup", line 2978, in execute raise exc OSError: [Errno 1] Operation not permitted any help would be cool, cos this is so baffling me.. Regards Jay
-- "Hit User to Continue" --
_________________________________________________________________ Join the world’s largest e-mail service with MSN Hotmail. http://www.hotmail.com From bescoto@stanford.edu Sun Jun 16 07:04:02 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Sat, 15 Jun 2002 23:04:02 -0700 Subject: Rdiff-Backup and Samba In-Reply-To: Your message of "Sun, 16 Jun 2002 05:19:44 -0000." Message-ID: <200206160604.g5G642I05912@folly.Stanford.EDU> --==_Exmh_-929019745P Content-Type: text/plain; charset=us-ascii >>>>> "JD" == Jay Dylan >>>>> wrote the following on Sun, 16 Jun 2002 05:19:44 +0000 JD> '[Errno 1] Operation not permitted' raised of class JD> 'exceptions.OSError': JD> File "/software/rdiff-backup-0.7.6/rdiff-backup", line 2412, in JD> symlink self.conn.os.symlink(linktext, self.path) Looks like the problem is that a symlink couldn't be created. -- Ben Escoto --==_Exmh_-929019745P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE9DCpR+owuOvknOnURAo98AJ93QhGgou29il/pVgzEm3P2DrmpRwCbB12S gtwc9LaNpt0RMJgbL2SvUvM= =bQwb -----END PGP SIGNATURE----- --==_Exmh_-929019745P-- From jasonthomasagius@hotmail.com Sun Jun 16 07:25:07 2002 From: jasonthomasagius@hotmail.com (Jay Dylan) Date: Sun, 16 Jun 2002 06:25:07 +0000 Subject: Rdiff-Backup and Samba Message-ID: its still barfing.. have run with every option i can think of.. this is what I get now .. (i've changed the source dir to a dir with very simple files in) [root@localhost store]# rdiff-backup --windows-time-format --no-hard-links --force --no-compression -v 9 --no-resume --windows-mode --change-source-perms --exclude-device-files --null-separator /var/www/cgi-bin/ /replnet/samba Sun Jun 16 01:22:10 2002 Last backup dated Sun Jun 16 01:18:47 2002 was aborted, but we aren't resuming it. Sun Jun 16 01:22:10 2002 Processing (Index: () Data: {'nlink': 2, 'gid': 0, 'uid': 0, 'mtime': 1024180040L, 'perms': 493, 'type': 'dir', 'devloc': 18435L, 'inode': 1951L, 'size': 4096L}, Path: /replnet/samba Index: () Data: {'nlink': 1, 'gid': 0, 'uid': 0, 'mtime': 1024186768L, 'perms': 493, 'type': 'dir', 'devloc': 7L, 'inode': 2L, 'size': 512L}).() Sun Jun 16 01:22:10 2002 Copying attributes from () to /replnet/samba Sun Jun 16 01:22:10 2002 Writing checkpoint time 1024208530.37 Sun Jun 16 01:22:10 2002 Exception '[Errno 1] Operation not permitted' raised of class 'exceptions.OSError': File "/usr/bin/rdiff-backup", line 2975, in execute init_val = self.init_thunk() File "/usr/bin/rdiff-backup", line 3004, in init init_return_vals.append(ra.init_thunk()) File "/usr/bin/rdiff-backup", line 3129, in init def init(): tf.symlink(linktext) File "/usr/bin/rdiff-backup", line 2417, in symlink self.conn.os.symlink(linktext, self.path) Sun Jun 16 01:22:10 2002 Deleting /replnet/samba/rdiff-backup-data/rdiff-backup.tmp.0 Sun Jun 16 01:22:10 2002 Deleting /replnet/samba/rdiff-backup-data/rdiff-backup.tmp.1 Sun Jun 16 01:22:10 2002 Exception '[Errno 1] Operation not permitted' raised of class 'exceptions.OSError': File "/usr/bin/rdiff-backup", line 6018, in patch_w_datadir_writes SaveState.checkpoint(ITR, finalizer, dsrp) File "/usr/bin/rdiff-backup", line 3365, in checkpoint cls.record_last_file_action(last_file_rorp)).execute() File "/usr/bin/rdiff-backup", line 2983, in execute raise exc Sun Jun 16 01:22:10 2002 ------------ Old traceback info ----------- Exception '[Errno 1] Operation not permitted' raised of class 'exceptions.OSError': File "/usr/bin/rdiff-backup", line 2975, in execute init_val = self.init_thunk() File "/usr/bin/rdiff-backup", line 3004, in init init_return_vals.append(ra.init_thunk()) File "/usr/bin/rdiff-backup", line 3129, in init def init(): tf.symlink(linktext) File "/usr/bin/rdiff-backup", line 2417, in symlink self.conn.os.symlink(linktext, self.path) ------------------------------------------- Sun Jun 16 01:22:10 2002 Writing checkpoint time 1024208530.45 Sun Jun 16 01:22:10 2002 Renaming /replnet/samba/rdiff-backup-data/rdiff-backup.tmp.2 to /replnet/samba/rdiff-backup-data/checkpoint-data.2002-06-16T01_22_09-05_00.data Sun Jun 16 01:22:10 2002 Touching /replnet/samba/rdiff-backup-data/last-file-incremented.2002-06-16T01_22_09-05_00.data Sun Jun 16 01:22:10 2002 Touching /replnet/samba/rdiff-backup-data/last-file-definitive.2002-06-16T01_22_09-05_00.data Traceback (most recent call last): File "/usr/bin/rdiff-backup", line 6779, in ? Globals.Main.Main(sys.argv[1:]) File "/usr/bin/rdiff-backup", line 6488, in Main self.take_action(rps) File "/usr/bin/rdiff-backup", line 6465, in take_action elif self.action == "backup": self.Backup(rps[0], rps[1]) File "/usr/bin/rdiff-backup", line 6522, in Backup else: HighLevel.Mirror(rpin, rpout, self.incdir, RSI) File "/usr/bin/rdiff-backup", line 5800, in Mirror DestS.patch_w_datadir_writes(dest_rpath, diffiter, inc_rpath) File "/usr/bin/rdiff-backup", line 6022, in patch_w_datadir_writes except: cls.handle_last_error(finished_dsrp, finalizer, ITR) File "/usr/bin/rdiff-backup", line 6018, in patch_w_datadir_writes SaveState.checkpoint(ITR, finalizer, dsrp) File "/usr/bin/rdiff-backup", line 3365, in checkpoint cls.record_last_file_action(last_file_rorp)).execute() File "/usr/bin/rdiff-backup", line 2983, in execute raise exc OSError: [Errno 1] Operation not permitted [root@localhost store]# >From: Ben Escoto >To: "Jay Dylan" >CC: rdiff-backup@keywest.Stanford.EDU >Subject: Re: Rdiff-Backup and Samba >Date: Sat, 15 Jun 2002 23:04:02 -0700 > > >>>>> "JD" == Jay Dylan > >>>>> wrote the following on Sun, 16 Jun 2002 05:19:44 +0000 > > JD> '[Errno 1] Operation not permitted' raised of class > JD> 'exceptions.OSError': > JD> File "/software/rdiff-backup-0.7.6/rdiff-backup", line 2412, in > JD> symlink self.conn.os.symlink(linktext, self.path) > >Looks like the problem is that a symlink couldn't be created. > > >-- >Ben Escoto ><< attach3 >> _________________________________________________________________ Send and receive Hotmail on your mobile device: http://mobile.msn.com From bescoto@stanford.edu Sun Jun 16 07:35:39 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Sat, 15 Jun 2002 23:35:39 -0700 Subject: Rdiff-Backup and Samba In-Reply-To: Your message of "Sun, 16 Jun 2002 06:25:07 -0000." Message-ID: <200206160635.g5G6ZeM07252@folly.Stanford.EDU> --==_Exmh_-923903298P Content-Type: text/plain; charset=us-ascii >>>>> "JD" == Jay Dylan >>>>> wrote the following on Sun, 16 Jun 2002 06:25:07 +0000 JD> ----------- Exception '[Errno 1] Operation not permitted' raised JD> of class 'exceptions.OSError': File "/usr/bin/rdiff-backup", JD> tf.symlink(linktext) File "/usr/bin/rdiff-backup", line 2417, in JD> symlink self.conn.os.symlink(linktext, self.path) Still can't create the symlink. Can you make one manually (e.g. ln -s foo bar)? -- Ben Escoto --==_Exmh_-923903298P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE9DDG6+owuOvknOnURAlJ3AJ4yn07VLRF2zU2+FIcFLGkKtKbJJwCfR2mb OQnSapn2U/Q4MAcNtesiNKg= =Du2K -----END PGP SIGNATURE----- --==_Exmh_-923903298P-- From bescoto@stanford.edu Sun Jun 16 09:42:48 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Sun, 16 Jun 2002 01:42:48 -0700 Subject: 0.7.6 slower than 0.6.1 In-Reply-To: Your message of "Sat, 15 Jun 2002 19:21:28 PDT." Message-ID: <200206160842.g5G8gmw11485@folly.Stanford.EDU> --==_Exmh_349418402P Content-Type: text/plain; charset=us-ascii >>>>> "DG" == dean gaudet >>>>> wrote the following on Sat, 15 Jun 2002 19:21:28 -0700 (PDT) DG> at times there's a cpu limitation, but i'm guessing the problem DG> is that rdiff-backup is mostly serialized. fixing that is a DG> chore though :) DG> i suspect that there'd be some benefit to spawning a couple of DG> rdiffs in parallel. basically so that while one rdiff is DG> blocked on reading data, another is calculating. Yep, but at most a factor of 2 benefit, if I understand your suggestion right. Suppose the entire process involves x seconds of calculation and other CPU tasks, and y seconds of reading data. If they could be done in parallel, the whole thing could take (at minimum) max(x,y) seconds. Currently the session would take x+y seconds. (x+y)/max(x,y) is greatest when x=y and then it equals 2. But that's assuming that CPU time and reading time happen to be exactly the same, and that they can all be done simultaneously. Probably the actual speed to be gained is much less than a factor of 2... DG> with that change i know that my bottleneck would be bandwidth -- DG> there's only a 128kbit uplink from my mirror to my primary DG> (1.5mbit the other way). i can watch the uplink saturate when DG> rdiff hits a large file and read-ahead can feed it data as fast DG> as the cpu can do the checksums. when it's in amongst small DG> files the uplink isn't saturated at all. Is CPU usage close to 100% when the line usage drops off? If so it could just be a CPU problem, not the serialization. DG> in terms of scaling the mirror host to handle many primaries it DG> might be nice to have rdiff-backup cache the signature files on DG> the mirror. perhaps a file per directory with the names and DG> signatures of the directory contents. not bulletproof -- if DG> someone goes about mucking in the mirror they could damage DG> things. but this would reduce the cpu and i/o requirements on DG> the mirror host. Yep, that would make sense, but doesn't sound exactly trivial to implement, and it would probably be a headache to figure out everything that could go wrong at every point. I guess we'll see, after I optimize the current version a bit, how many real-world situations could require something like that. -- Ben Escoto --==_Exmh_349418402P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE9DE9++owuOvknOnURAqmwAKCKJjSXoT4Cu6hIZ5tP4JokykHb9ACbBTDW j9368RD/bzHaNqQJyqzy1Ak= =nnYg -----END PGP SIGNATURE----- --==_Exmh_349418402P-- From jasonthomasagius@hotmail.com Sun Jun 16 22:10:20 2002 From: jasonthomasagius@hotmail.com (Jay Dylan) Date: Sun, 16 Jun 2002 21:10:20 +0000 Subject: Rdiff-Backup and Samba Message-ID: Another day... another problem.. I think i sorted the symlink issue.. got a hole new barf though.. [root@localhost temp]# rdiff-backup -v 9 --no-resume --no-compression --no-hard-link --windows-time-format -m --force /software/ /replnet/temp/ Sun Jun 16 16:01:50 2002 Mirroring /software/ to /replnet/temp/ Traceback (most recent call last): File "/usr/bin/rdiff-backup", line 6779, in ? Globals.Main.Main(sys.argv[1:]) File "/usr/bin/rdiff-backup", line 6488, in Main self.take_action(rps) File "/usr/bin/rdiff-backup", line 6468, in take_action elif self.action == "mirror": self.Mirror(rps[0], rps[1]) File "/usr/bin/rdiff-backup", line 6499, in Mirror HighLevel.Mirror(src_rp, dest_rp) File "/usr/bin/rdiff-backup", line 5796, in Mirror src_init_dsiter = SourceS.split_initial_dsiter() File "/usr/bin/rdiff-backup", line 5839, in split_initial_dsiter dsiter = cls.iterate_from() File "/usr/bin/rdiff-backup", line 5833, in iterate_from if cls._session_info is None: Globals.select_source.set_iter() AttributeError: 'NoneType' object has no attribute 'set_iter' Any Ideas?... >From: Ben Escoto >To: "Jay Dylan" >Subject: Re: Rdiff-Backup and Samba >Date: Sun, 16 Jun 2002 00:00:12 -0700 > > >>>>> "JD" == Jay Dylan > >>>>> wrote the following on Sun, 16 Jun 2002 06:40:02 +0000 > > JD> i can create a symlink i'm sure.. but what do i symlink to > JD> where.. thats what i dont understand.. > >Well, I forget the actual case, but say you were backing up to >hostname.net::/backup/dir. Then the symlink it's trying to create >would be at >/backup/dir/rdiff-backup-data/last-file-incremented..data on the >hostname.net machine and it point at increments/something-or-other. > > >-- >Ben Escoto ><< attach3 >>
-- "Hit User to Continue" --
_________________________________________________________________ Send and receive Hotmail on your mobile device: http://mobile.msn.com From bescoto@stanford.edu Sun Jun 16 23:24:28 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Sun, 16 Jun 2002 15:24:28 -0700 Subject: Rdiff-Backup and Samba In-Reply-To: Your message of "Sun, 16 Jun 2002 21:10:20 -0000." Message-ID: <200206162224.g5GMOSG24054@folly.Stanford.EDU> --==_Exmh_574804661P Content-Type: text/plain; charset=us-ascii >>>>> "JD" == Jay Dylan >>>>> wrote the following on Sun, 16 Jun 2002 21:10:20 +0000 JD> Another day... another problem.. I think i sorted the symlink JD> issue.. got a hole new barf though.. JD> AttributeError: 'NoneType' object has no attribute 'set_iter' JD> Any Ideas?... Yeah, that's just a bug.. I guess I really need a test case for the '-m' switch... (But probably no one uses -m anyway, except by mistake.) JD> i think the symlinks are the root of all my problems.. is there JD> a way of running it with.. JD> A) no symlinks or B) the rdiff-backup-data dir stored other than JD> int he root of the folder (ie not on the samba share) JD> I'm thinking windows is having issues with the symlinks Well, you may want to try --checkpoint-interval 0 to turn off checkpointing, because rdiff-backup uses a symlink to point to the last file processed when checkpointing. However, this will impair somewhat rdiff-backup error handling ability, so only use this if rdiff-backup works consistently for you. If there are symlinks used in other places which I forget, then I think there is nothing else that can be currently done. Is Windows' file system fully case sensitive and can it handle long file names? If not you are going to have other, probably insurmountable, problems.. -- Ben Escoto --==_Exmh_574804661P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE9DRAb+owuOvknOnURAsjSAJ9y57q01ELmjv2dxAOww1fwWD4bPQCeP8w3 FU/5sRH0fhRu9ps4tbHgX50= =2UV5 -----END PGP SIGNATURE----- --==_Exmh_574804661P-- From bescoto@stanford.edu Mon Jun 17 09:29:42 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Mon, 17 Jun 2002 01:29:42 -0700 Subject: 0.9.0 released Message-ID: <200206170829.g5H8Tgu14579@folly.Stanford.EDU> --==_Exmh_1978302995P Content-Type: text/plain; charset=us-ascii This version is packaged entirely differently, as a number of small *.py and *.pyc files, and installs somewhere in the user's PYTHONPATH. It also has a very small C component, so requires a bit of compilation. A lot of the code had to be edited, so this version probably isn't stable enough for "production" use. But it uses much less CPU and should be 2 times faster or more when checking large directories with few changes. New in v0.9.0 (2002/06/17) -------------------------- Changed lots of the code to distribute as standard python package instead of single script. Installation procedure is also different. Speed optimizations - average user might see speed increase of 2 or more. -- Ben Escoto --==_Exmh_1978302995P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE9DZ31+owuOvknOnURAginAJ98Nuvg66aaXG2R6FjTSNXtlvZCggCfXKKG aql4GKKJqotAtU+nEvFpP/c= =rrHa -----END PGP SIGNATURE----- --==_Exmh_1978302995P-- From Kevin.Spicer@bmrb.co.uk Mon Jun 17 09:46:24 2002 From: Kevin.Spicer@bmrb.co.uk (Spicer, Kevin) Date: Mon, 17 Jun 2002 09:46:24 +0100 Subject: 0.9.0 released Message-ID: This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C215DB.762BAC70 Content-Type: text/plain; charset="iso-8859-1" I might try this out, given the time its taking me at present. Is backwards compatability with previous versions (0.7.6 specifically) maintained? (in case I need to roll back) -----Original Message----- From: Ben Escoto [mailto:bescoto@stanford.edu] Sent: 17 June 2002 09:30 To: rdiff-backup@keywest.Stanford.EDU Subject: 0.9.0 released This version is packaged entirely differently, as a number of small *.py and *.pyc files, and installs somewhere in the user's PYTHONPATH. It also has a very small C component, so requires a bit of compilation. A lot of the code had to be edited, so this version probably isn't stable enough for "production" use. But it uses much less CPU and should be 2 times faster or more when checking large directories with few changes. New in v0.9.0 (2002/06/17) -------------------------- Changed lots of the code to distribute as standard python package instead of single script. Installation procedure is also different. Speed optimizations - average user might see speed increase of 2 or more. -- Ben Escoto BMRB International http://www.bmrb.co.uk +44 (0)20 8566 5000 ____________________________________________________________ This message (and any attachment) is intended only for the recipient and may contain confidential and/or privileged material. If you have received this in error, please contact the sender and delete this message immediately. Disclosure, copying or other action taken in respect of this email or in reliance on it is prohibited. BMRB International Limited accepts no liability in relation to any personal emails, or content of any email which does not directly relate to our business. ------_=_NextPart_001_01C215DB.762BAC70 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable RE: 0.9.0 released

I might try this out, given the time its taking me at = present.  Is backwards compatability with previous versions (0.7.6 = specifically) maintained?  (in case I need to roll = back)

-----Original Message-----
From: Ben Escoto [mailto:bescoto@stanford.edu]
Sent: 17 June 2002 09:30
To: rdiff-backup@keywest.Stanford.EDU
Subject: 0.9.0 released



This version is packaged entirely differently, as a = number of small
*.py and *.pyc files, and installs somewhere in the = user's
PYTHONPATH.  It also has a very small C = component, so requires a bit
of compilation.

    A lot of the code had to be = edited, so this version probably isn't
stable enough for "production" use.

    But it uses much less CPU and = should be 2 times faster or more
when checking large directories with few = changes.


New in v0.9.0 (2002/06/17)
--------------------------

Changed lots of the code to distribute as standard = python package
instead of single script.  Installation = procedure is also different.

Speed optimizations - average user might see speed = increase of 2 or
more.


--
Ben Escoto


BMRB International
http://www.bmrb.co.uk +44 (0)20 8566 5000

____________________________________________________________ =
This message (and any attachment) is intended only = for the recipient and may contain confidential and/or privileged = material. If you have received this in error, please contact the sender = and delete this message immediately. Disclosure, copying or other = action taken in respect of this email or in reliance on it is = prohibited. BMRB International Limited accepts no liability in relation = to any personal emails, or content of any email which does not directly = relate to our business.

------_=_NextPart_001_01C215DB.762BAC70-- From bescoto@stanford.edu Mon Jun 17 10:10:53 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Mon, 17 Jun 2002 02:10:53 -0700 Subject: 0.9.0 released In-Reply-To: Your message of "Mon, 17 Jun 2002 09:46:24 BST." Message-ID: <200206170910.g5H9ArB15023@folly.Stanford.EDU> --==_Exmh_1991532036P Content-Type: text/plain; charset=us-ascii >>>>> "KS" == Kevin Spicer >>>>> wrote the following on Mon, 17 Jun 2002 09:46:24 +0100 KS> I might try this out, given the time its taking me at present. KS> Is backwards compatability with previous versions (0.7.6 KS> specifically) maintained? (in case I need to roll back) Yes, the only recent backwards incompatible change has been the compressed increment stuff (in 0.7.1 I think). -- Ben Escoto --==_Exmh_1991532036P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE9Daec+owuOvknOnURAhJWAKCPPNdkaVjAFGf6coW4MuczHQ73qACaAy9I RXEZnWn79hHt5u9F9dUZQfA= =rMzM -----END PGP SIGNATURE----- --==_Exmh_1991532036P-- From dean-list-rdiff-backup@arctic.org Mon Jun 17 20:43:24 2002 From: dean-list-rdiff-backup@arctic.org (dean gaudet) Date: Mon, 17 Jun 2002 12:43:24 -0700 (PDT) Subject: bottlenecks Message-ID: i haven't looked at 0.9.0 to see what's new... but i played around with some parallel rdiff invocations yesterday. not within rdiff-backup, but just using command lines such as: find mail1 mail2 -type f -print0 >filelist xargs -0 -i -P10 rdiff signature \{\} /dev/null /dev/null /dev/null This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C2169F.66C90E3E Content-Type: text/plain; charset="iso-8859-1" Further to my post a week or so ago about rdiff-backup 0.7.6 being v slow I just thought I'd let you all know that 0.9.0 seems very much faster. My bakups last night which were taking 15-6 hours took 8 hours last night, thats the quickest they've ever been and an hour or two faster than 0.6.0 (the first release I used) - with a disk thats now nearly twice as full. Nice going Ben! BMRB International http://www.bmrb.co.uk +44 (0)20 8566 5000 ____________________________________________________________ This message (and any attachment) is intended only for the recipient and may contain confidential and/or privileged material. If you have received this in error, please contact the sender and delete this message immediately. Disclosure, copying or other action taken in respect of this email or in reliance on it is prohibited. BMRB International Limited accepts no liability in relation to any personal emails, or content of any email which does not directly relate to our business. ------_=_NextPart_001_01C2169F.66C90E3E Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable 0.9.0 Much Faster!

Further to my post a week or so ago about = rdiff-backup 0.7.6 being v slow I just thought I'd let you all know = that 0.9.0 seems very much faster.  My bakups last night which = were taking 15-6 hours took 8 hours last night, thats the quickest = they've ever been and an hour or two faster than 0.6.0 (the first = release I used) - with a disk thats now nearly twice as = full.

Nice going Ben!


BMRB International
http://www.bmrb.co.uk +44 (0)20 8566 5000

____________________________________________________________ =
This message (and any attachment) is intended only = for the recipient and may contain confidential and/or privileged = material. If you have received this in error, please contact the sender = and delete this message immediately. Disclosure, copying or other = action taken in respect of this email or in reliance on it is = prohibited. BMRB International Limited accepts no liability in relation = to any personal emails, or content of any email which does not directly = relate to our business.

------_=_NextPart_001_01C2169F.66C90E3E-- From bescoto@stanford.edu Tue Jun 18 11:07:13 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Tue, 18 Jun 2002 03:07:13 -0700 Subject: bottlenecks In-Reply-To: Your message of "Mon, 17 Jun 2002 12:43:24 PDT." Message-ID: <200206181007.g5IA7Dj16990@folly.Stanford.EDU> --==_Exmh_-656681431P Content-Type: text/plain; charset=us-ascii >>>>> "DG" == dean gaudet >>>>> wrote the following on Mon, 17 Jun 2002 12:43:24 -0700 (PDT) DG> and here's why i suspect fork() overhead: DG> % /usr/bin/time xargs -0 -n100 cat -- >/dev/null 0.42user 11.92system 1:12.92elapsed 16%CPU (0avgtext+0avgdata DG> 0maxresident)k 0inputs+0outputs (23219major+5044minor)pagefaults DG> 0swaps DG> that puts 100 filenames into each cat rather than the 1 filename DG> i used above. Thank you for the analysis! I think you are right about the fork overhead. I decided to look into it more by comparing multiple uses of rdiff, with the same thing using Donovan Bailey's python wrapper around librsync. So, I created two directories full of 1000 files whose names were the numbers 0 to 999. Each file was 2048 long. Then I wrote the following two python programs: ---------------[ prog1 - "test2.py" ]---------------------------- #!/usr/bin/env python """Run rdiff to transform everything in one dir to another""" import sys, os dir1, dir2 = sys.argv[1:3] for i in xrange(1000): assert not os.system("rdiff signature %s/%s sig" % (dir1, i)) assert not os.system("rdiff delta sig %s/%s diff" % (dir2, i)) assert not os.system("rdiff patch %s/%s diff %s/%s.out" % (dir1, i, dir1, i)) ---------------[ prog2 - "test3.py" ]---------------------------- #!/usr/bin/env python """Use librsync to transform everything in one dir to another""" import sys, os, librsync dir1, dir2 = sys.argv[1:3] for i in xrange(1000): dir1fn = "%s/%s" % (dir1, i) dir2fn = "%s/%s" % (dir2, i) # Write signature file f1 = open(dir1fn, "rb") sigfile = open("sig", "wb") librsync.filesig(f1, sigfile, 2048) f1.close() sigfile.close() # Write delta file f2 = open(dir2fn, "r") sigfile = open("sig", "rb") deltafile = open("delta", "wb") librsync.filerdelta(sigfile, f2, deltafile) f2.close() sigfile.close() deltafile.close() # Write patched file f1 = open(dir1fn, "rb") newfile = open("%s/%s.out" % (dir1, i), "wb") deltafile = open("delta", "rb") librsync.filepatch(f1, deltafile, newfile) f1.close() deltafile.close() newfile.close() -------------------------------------------------------------------- The idea is they both do the same thing, but test2.py runs rdiff on each file, and test3.py does everything internally, so no forking. The results: ~/rd/src $ time python test3.py out1 out2 real 0m31.900s user 0m1.750s sys 0m1.190s ~/rd/src $ time python test2.py out1 out2 real 0m57.369s user 0m33.440s sys 0m23.090s The internal librsync procedure used up only about 1/20th of the CPU time as the external rdiff procedure, meaning I suppose that 95% of the CPU time was spent forking, setting up the external rdiff process, etc. Of course this is for small files; as the file sizes became large, the forking time would go to 0 as a percentage of the total. Oh, and the reason that test3.py had such a large real time is because it ended up using hundreds of MB of memory and my system started swapping like crazy. I guess all this efficiency won't be worth much if there is a huge memory leak in pysync or librsync... -- Ben Escoto --==_Exmh_-656681431P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE9DwZP+owuOvknOnURAtrdAJ9Eoq/oskE76Oh97nOevn2EG136FgCfYNLt phAz8z3Hx9Pe7DwDklWsIWc= =m1Pi -----END PGP SIGNATURE----- --==_Exmh_-656681431P-- From Robert.Weber@Colorado.EDU Tue Jun 18 11:30:48 2002 From: Robert.Weber@Colorado.EDU (Robert Weber) Date: Tue, 18 Jun 2002 04:30:48 -0600 Subject: bottlenecks In-Reply-To: Your message of Tue, 18 Jun 2002 03:07:13 PDT Message-ID: <200206181030.g5IAUo610857@refuge.Colorado.EDU> -------- > > The internal librsync procedure used up only about 1/20th of the CPU > time as the external rdiff procedure, meaning I suppose that 95% of > the CPU time was spent forking, setting up the external rdiff process, > etc. > I was just able to do some code work on librsync's md4 and crc32 code to get a reasonable speedup. Granted, this is only an issue with large files, but every little bit helps. I have posted my changes to the developers, and hopefully they will pass inspection and find their way in to the library. The speed up for a 2GB file sig was (hannah)[4:00am]:/hannah/scratch/medialib > /usr/bin/time ./rdiff.orig signature /usr/local/pc/C.diskimage foo2 real 3:19.3 user 2:07.0 sys 24.4 (hannah)[4:03am]:/hannah/scratch/medialib > /usr/bin/time ./rdiff signature /usr/local/pc/C.diskimage foo1 real 2:47.2 user 1:18.6 sys 23.7 Robert Weber UnixOps University of Colorado From ambra@ambrasoft.com Tue Jun 18 15:36:55 2002 From: ambra@ambrasoft.com (ambra) Date: Tue, 18 Jun 2002 10:36:55 -0400 Subject: 0.9.0 Much Faster! In-Reply-To: Message-ID: This is a multi-part message in MIME format. ------=_NextPart_000_000B_01C216B4.11121FC0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit 0.9.0 Much Faster!The only problem is, that when I include some stuff, so only the first include works: --include=something1 --include=something2 --include=something3 ... The something2 and something3 will not be backed up... Vlastimil Adamovsky -----Original Message----- From: rdiff-backup-admin@keywest.Stanford.EDU [mailto:rdiff-backup-admin@keywest.Stanford.EDU]On Behalf Of Spicer, Kevin Sent: Tuesday, June 18, 2002 4:09 AM To: 'rdiff-backup@keywest.stanford.edu' Subject: 0.9.0 Much Faster! Further to my post a week or so ago about rdiff-backup 0.7.6 being v slow I just thought I'd let you all know that 0.9.0 seems very much faster. My bakups last night which were taking 15-6 hours took 8 hours last night, thats the quickest they've ever been and an hour or two faster than 0.6.0 (the first release I used) - with a disk thats now nearly twice as full. Nice going Ben! BMRB International http://www.bmrb.co.uk +44 (0)20 8566 5000 ____________________________________________________________ This message (and any attachment) is intended only for the recipient and may contain confidential and/or privileged material. If you have received this in error, please contact the sender and delete this message immediately. Disclosure, copying or other action taken in respect of this email or in reliance on it is prohibited. BMRB International Limited accepts no liability in relation to any personal emails, or content of any email which does not directly relate to our business. --- Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.365 / Virus Database: 202 - Release Date: 5/24/2002 ------=_NextPart_000_000B_01C216B4.11121FC0 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable 0.9.0 Much Faster!
The=20 only problem is, that when I include some stuff, so only the first = include=20 works:
 
--include=3Dsomething1 --include=3Dsomething2 = --include=3Dsomething3 ...
 
The=20 something2 and something3 will not be backed up...
 
Vlastimil Adamovsky
-----Original Message-----
From:=20 rdiff-backup-admin@keywest.Stanford.EDU=20 [mailto:rdiff-backup-admin@keywest.Stanford.EDU]On Behalf Of = Spicer,=20 Kevin
Sent: Tuesday, June 18, 2002 4:09 AM
To:=20 'rdiff-backup@keywest.stanford.edu'
Subject: 0.9.0 Much=20 Faster!

Further to my post a week or so ago about = rdiff-backup 0.7.6=20 being v slow I just thought I'd let you all know that 0.9.0 seems very = much=20 faster.  My bakups last night which were taking 15-6 hours took 8 = hours=20 last night, thats the quickest they've ever been and an hour or two = faster=20 than 0.6.0 (the first release I used) - with a disk thats now nearly = twice as=20 full.

Nice going Ben!


BMRB International
http://www.bmrb.co.uk +44 (0)20=20 8566 5000

____________________________________________________________=20
This message (and any attachment) is = intended only for=20 the recipient and may contain confidential and/or privileged material. = If you=20 have received this in error, please contact the sender and delete this = message=20 immediately. Disclosure, copying or other action taken in respect of = this=20 email or in reliance on it is prohibited. BMRB International Limited = accepts=20 no liability in relation to any personal emails, or content of any = email which=20 does not directly relate to our = business.

------=_NextPart_000_000B_01C216B4.11121FC0-- From ambra@ambrasoft.com Wed Jun 19 04:42:41 2002 From: ambra@ambrasoft.com (ambra) Date: Tue, 18 Jun 2002 23:42:41 -0400 Subject: 0.9.0 Much Faster! In-Reply-To: Message-ID: This is a multi-part message in MIME format. ------=_NextPart_000_001B_01C21721.D67E0D60 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit 0.9.0 Much Faster!Ben, when I run something like this: rdiff-backup --force --exclude-device-files --include=/home2/*/folder1 --ex clude=/ / userid@1.2.3.4::/myback then it will DELETE /myback folder . When I run this: rdiff-backup --force --exclude-device-files --include=/home2 --exclude=/ / userid@1.2.3.4::/myback then it works fine..(I think)... Would you, please, notify me, if you will find out, what the problem is and what the solution is, so I can go and continue testing? Thanks, Vlastik --- Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.365 / Virus Database: 202 - Release Date: 5/24/2002 ------=_NextPart_000_001B_01C21721.D67E0D60 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable 0.9.0 Much Faster!
Ben,
 
when I  = run something=20 like this:
 
rdiff-backup --force  = --exclude-device-files=20 --include=3D/home2/*/folder1 = --exclude=3D/=20 / userid@1.2.3.4::/myback
 
then it will=20 DELETE   /myback  folder = .
 
When I run=20 this:
 
rdiff-backup --force  = --exclude-device-files=20 --include=3D/home2  = --exclude=3D/ /=20 userid@1.2.3.4::/myback
 
then it works = fine..(I=20 think)...
 
Would you, please, = notify me,=20 if you will find out, what the problem is and what the solution is, so I = can go=20 and continue testing?
 
Thanks,
 
     =20 Vlastik
 
 =20
------=_NextPart_000_001B_01C21721.D67E0D60-- From ambra@ambrasoft.com Wed Jun 19 04:47:54 2002 From: ambra@ambrasoft.com (ambra) Date: Tue, 18 Jun 2002 23:47:54 -0400 Subject: rdiff-backup is deleting destination folder In-Reply-To: Message-ID: This is a multi-part message in MIME format. ------=_NextPart_000_002B_01C21722.910CB050 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit 0.9.0 Much Faster! Ben, when I run something like this: rdiff-backup --force --exclude-device-files --include=/home2/*/folder1 --ex clude=/ / userid@1.2.3.4::/myback then it will DELETE /myback folder . When I run this: rdiff-backup --force --exclude-device-files --include=/home2 --exclude=/ / userid@1.2.3.4::/myback then it works fine..(I think)... Would you, please, notify me, if you will find out, what the problem is and what the solution is, so I can go and continue testing? Thanks, Vlastik --- Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.365 / Virus Database: 202 - Release Date: 5/24/2002 ------=_NextPart_000_002B_01C21722.910CB050 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable 0.9.0 Much Faster!

 
Ben,
 
when I  = run=20 something like this:
 
rdiff-backup --force  = --exclude-device-files=20 --include=3D/home2/*/folder1 = --exclude=3D/=20 / userid@1.2.3.4::/myback
 
then it will=20 DELETE   /myback  folder = .
 
When I run=20 this:
 
rdiff-backup --force  = --exclude-device-files=20 --include=3D/home2  = --exclude=3D/ /=20 userid@1.2.3.4::/myback
 
then it works = fine..(I=20 think)...
 
Would you, = please, notify me,=20 if you will find out, what the problem is and what the solution is, so = I can=20 go and continue testing?
 
Thanks,
 
      Vlastik
 
 =20
------=_NextPart_000_002B_01C21722.910CB050-- From abo@minkirri.apana.org.au Wed Jun 19 05:21:07 2002 From: abo@minkirri.apana.org.au (Donovan Baarda) Date: Wed, 19 Jun 2002 14:21:07 +1000 Subject: bottlenecks In-Reply-To: <200206181007.g5IA7Dj16990@folly.Stanford.EDU> References: <200206181007.g5IA7Dj16990@folly.Stanford.EDU> Message-ID: <20020619042107.GC23703@minkirri.apana.org.au> I've Cc'd this to the rproxy-devel list for reference. On Tue, Jun 18, 2002 at 03:07:13AM -0700, Ben Escoto wrote: > >>>>> "DG" == dean gaudet > >>>>> wrote the following on Mon, 17 Jun 2002 12:43:24 -0700 (PDT) [...] > The internal librsync procedure used up only about 1/20th of the CPU > time as the external rdiff procedure, meaning I suppose that 95% of > the CPU time was spent forking, setting up the external rdiff process, > etc. Of course this is for small files; as the file sizes became > large, the forking time would go to 0 as a percentage of the total. > > Oh, and the reason that test3.py had such a large real time is > because it ended up using hundreds of MB of memory and my system > started swapping like crazy. I guess all this efficiency won't be > worth much if there is a huge memory leak in pysync or librsync... I'm glad to see pysync's librsync wrapper being useful. 1/20th is a significant difference! The memory problem... It is quite possible that there is a memory leak either in librsync or the Python wrapper. The Python wrapper is a first cut using swig, so it is possible swig introduced the memory leak, or I did with a bad swig wrapper. Librsync is not exactly clean code either, and I've already seen some dodgey memory alloc/free stuff in there that can be cleaned up, though I'm not sure it exactly leaks. There are two things that need doing; improving librsync, and improving the wrapper. I've decided that I'm going to improve librsync a bit first. After that I'm going to fix up the Python wrapper using the development versions of swig. None of this is going to happen quickly unless someone decides to pay me for it (my wife is hassling me to get a real job so that we can buy food). However, because I'm a programming addict, there is a chance I'll slowly starve to death after my wife leaves me as I code it up anyway :-) -- ---------------------------------------------------------------------- ABO: finger abo@minkirri.apana.org.au for more info, including pgp key ---------------------------------------------------------------------- From bescoto@stanford.edu Wed Jun 19 10:51:22 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Wed, 19 Jun 2002 02:51:22 -0700 Subject: 0.9.1 released Message-ID: <200206190951.g5J9pMN14185@folly.Stanford.EDU> --==_Exmh_965196386P Content-Type: text/plain; charset=us-ascii This version mainly fixes all the bad C I wrote. 0.9.0 users should definitely upgrade. New in v0.9.1 (2002/06/19) -------------------------- Fixed some bad C. Besides being unportable and leaking memory, it may have lead to someone's backup directory getting deleted (?). Tweaked some error recovery code to make it more like 0.8.0. Improved the installation a bit. -- Ben Escoto --==_Exmh_965196386P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE9EFQW+owuOvknOnURAlLBAJ0cFCYLMKLIL6MCwtOdfovqe7Q61QCdFaD6 jf6auv1bOtZI3G3YOdOAkZo= =QLpF -----END PGP SIGNATURE----- --==_Exmh_965196386P-- From bescoto@stanford.edu Wed Jun 19 11:02:01 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Wed, 19 Jun 2002 03:02:01 -0700 Subject: 0.9.0 Much Faster! In-Reply-To: Your message of "Tue, 18 Jun 2002 10:36:55 EDT." Message-ID: <200206191002.g5JA21H14317@folly.Stanford.EDU> --==_Exmh_973485128P Content-Type: text/plain; charset=us-ascii >>>>> "VA" == ambra >>>>> wrote the following on Tue, 18 Jun 2002 10:36:55 -0400 VA> 0.9.0 Much Faster!The only problem is, that when I include some VA> stuff, so only the first include works: VA> --include=something1 --include=something2 --include=something3 VA> ... VA> The something2 and something3 will not be backed up... ... VA> when I  run something like this:   rdiff-backup --force  VA> --exclude-device-files --include=/home2/*/folder1 --exclude=/ VA> / userid@1.2.3.4::/myback   then it will DELETE   /myback  VA> folder . Ouch, I tried briefly to duplicate these errors with the current version (0.9.1) and couldn't. Could you tell me if you get the problem on all directories or just some, and whether or not the problem shows up locally, or only remotely? rdiff-backup will try to delete the destination directory when (and hopefully only when) you use the --force switch and it thinks that the destination path exists, but is not a directory. I don't know why it would have thought that in your case, when it obviously was a directory, but there were some errors in the C statting code, and I guess C type errors can lead in theory to any behavior whatsoever? So anyway if you can recreate the error with 0.9.1, let me know how. Thanks. -- Ben Escoto --==_Exmh_973485128P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE9EFaY+owuOvknOnURAgdPAJ9bAM2ZaUOKhhPltcvjSA5jxMJemwCfVVQH tcX+BLcfI2p0ytfBn1Z10wU= =4jlL -----END PGP SIGNATURE----- --==_Exmh_973485128P-- From shirish@tacitnetworks.com Wed Jun 19 16:01:31 2002 From: shirish@tacitnetworks.com (Shirish H. Phatak) Date: Wed, 19 Jun 2002 11:01:31 -0400 Subject: [rproxy-devel] Re: bottlenecks References: <200206181007.g5IA7Dj16990@folly.Stanford.EDU> <20020619042107.GC23703@minkirri.apana.org.au> Message-ID: <3D109CCB.7020807@tacitnetworks.com> Hi, I have run the librsync compiled binaries through a bunch of different memory checkers already. There is a buffer leak in rs_delta_*, which my last patch should take care of. Other than that I haver not found anything. -Shirish Donovan Baarda wrote: >I've Cc'd this to the rproxy-devel list for reference. > >On Tue, Jun 18, 2002 at 03:07:13AM -0700, Ben Escoto wrote: > > >>>>>>>"DG" == dean gaudet >>>>>>>wrote the following on Mon, 17 Jun 2002 12:43:24 -0700 (PDT) >>>>>>> >>>>>>> >[...] > > >>The internal librsync procedure used up only about 1/20th of the CPU >>time as the external rdiff procedure, meaning I suppose that 95% of >>the CPU time was spent forking, setting up the external rdiff process, >>etc. Of course this is for small files; as the file sizes became >>large, the forking time would go to 0 as a percentage of the total. >> >> Oh, and the reason that test3.py had such a large real time is >>because it ended up using hundreds of MB of memory and my system >>started swapping like crazy. I guess all this efficiency won't be >>worth much if there is a huge memory leak in pysync or librsync... >> >> > >I'm glad to see pysync's librsync wrapper being useful. 1/20th is a >significant difference! > >The memory problem... > >It is quite possible that there is a memory leak either in librsync or the >Python wrapper. The Python wrapper is a first cut using swig, so it is >possible swig introduced the memory leak, or I did with a bad swig wrapper. >Librsync is not exactly clean code either, and I've already seen some dodgey >memory alloc/free stuff in there that can be cleaned up, though I'm not sure >it exactly leaks. > >There are two things that need doing; improving librsync, and improving the >wrapper. > >I've decided that I'm going to improve librsync a bit first. After that I'm >going to fix up the Python wrapper using the development versions of swig. >None of this is going to happen quickly unless someone decides to pay me for >it (my wife is hassling me to get a real job so that we can buy food). >However, because I'm a programming addict, there is a chance I'll slowly >starve to death after my wife leaves me as I code it up anyway :-) > > > From cummingscs@netscape.net Wed Jun 19 17:23:19 2002 From: cummingscs@netscape.net (Steven Cummings) Date: Wed, 19 Jun 2002 12:23:19 -0400 Subject: Installation Message-ID: <49D11C83.0DA8B2E8.0EE20915@netscape.net> Hey all, I don't know if this has been covered before, but I can't find it in the mailing list archives, so here goes nothings. I'm trying to install rdiff-backup.rpm and I'm having a problem. I've got two versions of python installed: python1 is python-1.5.2 and python2 is python-2.2.1. The rdiff-backup installer is telling me I need python >= 2.2.1. How do I get it to find and use the installation of python2.2? Thanks. /S -- Steven Cummings Columbia, MO __________________________________________________________________ Your favorite stores, helpful shopping tools and great gift ideas. Experience the convenience of buying online with Shop@Netscape! http://shopnow.netscape.com/ Get your own FREE, personal Netscape Mail account today at http://webmail.netscape.com/ From bescoto@stanford.edu Wed Jun 19 20:27:06 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Wed, 19 Jun 2002 12:27:06 -0700 Subject: Installation In-Reply-To: Your message of "Wed, 19 Jun 2002 12:23:19 EDT." <49D11C83.0DA8B2E8.0EE20915@netscape.net> Message-ID: <200206191927.g5JJR6V25499@folly.Stanford.EDU> --==_Exmh_1124359152P Content-Type: text/plain; charset=us-ascii >>>>> "SC" == Steven Cummings >>>>> wrote the following on Wed, 19 Jun 2002 12:23:19 -0400 SC> I'm trying to install rdiff-backup.rpm and I'm having a SC> problem. I've got two versions of python installed: python1 is SC> python-1.5.2 and python2 is python-2.2.1. The rdiff-backup SC> installer is telling me I need python >= 2.2.1. How do I get it SC> to find and use the installation of python2.2? Thanks. Unfortunately different rpm-based distributions (e.g. Mandrake, Redhat) have decided to package python 2.x differently so a single RPM cannot install and operate cleanly on all simultaneously. See FAQ question 5 (http://rdiff-backup.stanford.edu/FAQ.html#redhat) about redhat installation. Now (in 0.9.x) I'm making redhat-compatible RPMs, so in the future it'll only be a problem for Mandrake and other users.. -- Ben Escoto --==_Exmh_1124359152P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE9ENsI+owuOvknOnURAvG0AJ4zp6T9CHWTYUo/vmyL+A87VgHZHgCfXRfQ APgQcFtMnvf05JPx5OlZfnk= =owh9 -----END PGP SIGNATURE----- --==_Exmh_1124359152P-- From Kevin.Spicer@bmrb.co.uk Wed Jun 19 21:05:48 2002 From: Kevin.Spicer@bmrb.co.uk (Spicer, Kevin) Date: Wed, 19 Jun 2002 21:05:48 +0100 Subject: 0.9.1 Install problem Message-ID: This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C217CC.B48F5FDC Content-Type: text/plain; charset="iso-8859-1" Hello, The setup.py in 0.9.1 tar.gz doesn't install the rdiff-backup executable, I'm not an expert but ... I'd say its because that section of the code is commented out! As the only changes from 0.9.0 are comments this isn't a problem for me but I guess it might cause problems for others installing from scratch or upgrading earlier versions. BMRB International http://www.bmrb.co.uk +44 (0)20 8566 5000 ____________________________________________________________ This message (and any attachment) is intended only for the recipient and may contain confidential and/or privileged material. If you have received this in error, please contact the sender and delete this message immediately. Disclosure, copying or other action taken in respect of this email or in reliance on it is prohibited. BMRB International Limited accepts no liability in relation to any personal emails, or content of any email which does not directly relate to our business. ------_=_NextPart_001_01C217CC.B48F5FDC Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable 0.9.1 Install problem

Hello,
The setup.py in 0.9.1 tar.gz doesn't install the = rdiff-backup executable, I'm not an expert but ... I'd say its because = that section of the code is commented out!  As the only changes = from 0.9.0 are comments this isn't a problem for me but I guess it = might cause problems for others installing from scratch or upgrading = earlier versions.


BMRB International
http://www.bmrb.co.uk +44 (0)20 8566 5000

____________________________________________________________ =
This message (and any attachment) is intended only = for the recipient and may contain confidential and/or privileged = material. If you have received this in error, please contact the sender = and delete this message immediately. Disclosure, copying or other = action taken in respect of this email or in reliance on it is = prohibited. BMRB International Limited accepts no liability in relation = to any personal emails, or content of any email which does not directly = relate to our business.

------_=_NextPart_001_01C217CC.B48F5FDC-- From ambra@ambrasoft.com Wed Jun 19 22:23:56 2002 From: ambra@ambrasoft.com (ambra) Date: Wed, 19 Jun 2002 17:23:56 -0400 Subject: Still some strange problems In-Reply-To: <200206191927.g5JJR6V25499@folly.Stanford.EDU> Message-ID: When I am trying to run this command: rdiff-backup -v7 --ssh-no-compression --change-source-perms --exclude-device -files --force --include=/etc/master.passwd --include=/home2/*/folder1/folde r2 --include=/home2/*/folder1/usr/guide --include=/home2/*/folder1/var --inc lude=/home2/*/folder1/usr/tmp --include=/home2/*/folder1/tmp --exclude=/ / dummy@1.2.3.4::/home2/backup/someserver then it will make a backup of /etc/master.passwd and exit.... Any ideas? Please........ Thanks, Vlastimil Adamovsky --- Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.365 / Virus Database: 202 - Release Date: 5/24/2002 From Kevin.Spicer@bmrb.co.uk Wed Jun 19 23:09:05 2002 From: Kevin.Spicer@bmrb.co.uk (Spicer, Kevin) Date: Wed, 19 Jun 2002 23:09:05 +0100 Subject: An Interesting thought-maybe... Message-ID: This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C217DD.ED4F8CA0 Content-Type: text/plain; charset="iso-8859-1" First some background... I use rdiff-backup to make a remote backup of a data directory from remote site. Yesterday the source disk was nearing its capacity which triggered an email alert to the group of users who 'own' the data. Helpfully they responded by logging into the machine and gzipping huge quantities of files. At 9.15pm rdiff-backup kicked off. At 8.45pm I logged onto the remote machine to upgrade it to 0.9.1 in readiness for tonights backup, quite reasonably thinking that the rdiff-backup job would have finished some 14 hours ago. It didn't. Not unsurprisingly upgrading it whilst it was running didn't go down too well & rdiff-backup crashed. Enough rambling, my point is this... When people zip files they a) change the filename and b) change (in a binary sense) the content - but they don't change (in a human sense) the files real content. I don't think it would be unreasonable to guess that rdiff backup is transferring the entire file again. However 99% of zip files consist of the original filename with a suffix, if it was possible to summise which file had been zipped and how and just replicate this action on the remote machine this would likely be much quicker. In theory a neat idea I think (even if I do say so myself). I suspect actually trying to implement this would be much more difficult. Several complicating factors I can think of... different zip formats (zip gzip bz2 etc.) different levels of compression (do zip files record this in their metadata?) file could have changed between previous backup and zipping (either unzip and compare or zip remote side then compare?) Doesn't account for moving zipped files (but then it doesn't account for files moving anyway) What about tar too? I guess there would be a lot of work here, but I just wondered whether others think there might be gains to be made here? Kevin BMRB International http://www.bmrb.co.uk +44 (0)20 8566 5000 ____________________________________________________________ This message (and any attachment) is intended only for the recipient and may contain confidential and/or privileged material. If you have received this in error, please contact the sender and delete this message immediately. Disclosure, copying or other action taken in respect of this email or in reliance on it is prohibited. BMRB International Limited accepts no liability in relation to any personal emails, or content of any email which does not directly relate to our business. ------_=_NextPart_001_01C217DD.ED4F8CA0 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable An Interesting thought-maybe...

First some background...
I use rdiff-backup to make a remote backup of a data = directory from remote site.  Yesterday the source disk was nearing = its capacity which triggered an email alert to the group of users who = 'own' the data.  Helpfully they responded by logging into the = machine and gzipping huge quantities of files.  At 9.15pm = rdiff-backup kicked off.  At 8.45pm I logged onto the remote = machine to upgrade it to 0.9.1 in readiness for tonights backup, quite = reasonably thinking that  the rdiff-backup job would have finished = some 14 hours ago.  It didn't.  Not unsurprisingly upgrading = it whilst it was running didn't go down too well & rdiff-backup = crashed.

Enough rambling, my point is this...
When people zip files they a) change the filename = and b) change (in a binary sense) the content - but they don't change = (in a human sense) the files real content.  I don't think it would = be unreasonable to guess that rdiff backup is transferring the entire = file again.

However 99% of zip files consist of the original = filename with a suffix, if it was possible to summise which file had = been zipped and how and just replicate this action on the remote = machine this would likely be much quicker.

In theory a neat idea I think (even if I do say so = myself).  I suspect actually trying to implement this would be = much more difficult.  Several complicating factors I can think = of...

different zip formats (zip gzip bz2 etc.)
different levels of compression (do zip files record = this in their metadata?)
file could have changed between previous backup and = zipping (either unzip and compare or zip remote side then = compare?)
Doesn't account for moving zipped files (but then it = doesn't account for files moving anyway)
What about tar too?

I guess there would be a lot of work here, but I just = wondered whether others think there might be gains to be made = here?


Kevin



BMRB International
http://www.bmrb.co.uk +44 (0)20 8566 5000

____________________________________________________________ =
This message (and any attachment) is intended only = for the recipient and may contain confidential and/or privileged = material. If you have received this in error, please contact the sender = and delete this message immediately. Disclosure, copying or other = action taken in respect of this email or in reliance on it is = prohibited. BMRB International Limited accepts no liability in relation = to any personal emails, or content of any email which does not directly = relate to our business.

------_=_NextPart_001_01C217DD.ED4F8CA0-- From dean-list-rdiff-backup@arctic.org Wed Jun 19 23:16:50 2002 From: dean-list-rdiff-backup@arctic.org (dean gaudet) Date: Wed, 19 Jun 2002 15:16:50 -0700 (PDT) Subject: An Interesting thought-maybe... In-Reply-To: Message-ID: On Wed, 19 Jun 2002, Spicer, Kevin wrote: > However 99% of zip files consist of the original filename with a suffix, if > it was possible to summise which file had been zipped and how and just > replicate this action on the remote machine this would likely be much > quicker. i re-read andrew tridgell's thesis regarding rsync again this weekend and he touches on this issue. (it's linked off of ). > In theory a neat idea I think (even if I do say so myself). I suspect > actually trying to implement this would be much more difficult. Several > complicating factors I can think of... > different zip formats (zip gzip bz2 etc.) > different levels of compression (do zip files record this in their > metadata?) it's not even just a gzip -5 vs. gzip -9 difference... it's varying versions of gzip (and possibly also variances due to processor architecture?). unless you can be 100% certain that you'll generate the same compressed file on both sides of the link you can't use this technique. -dean From ambra@ambrasoft.com Thu Jun 20 01:34:08 2002 From: ambra@ambrasoft.com (ambra) Date: Wed, 19 Jun 2002 20:34:08 -0400 Subject: Still some strange problems In-Reply-To: Message-ID: I forgot to write that it will back up ONLY /etc/master.passwd and ignore all those other directories to be backed up... It has worked in previous versions Vlastimil Adamovsky -----Original Message----- From: rdiff-backup-admin@keywest.Stanford.EDU [mailto:rdiff-backup-admin@keywest.Stanford.EDU]On Behalf Of ambra Sent: Wednesday, June 19, 2002 5:24 PM To: Ben Escoto Cc: rdiff-backup@keywest.Stanford.EDU Subject: Still some strange problems Importance: High When I am trying to run this command: rdiff-backup -v7 --ssh-no-compression --change-source-perms --exclude-device -files --force --include=/etc/master.passwd --include=/home2/*/folder1/folde r2 --include=/home2/*/folder1/usr/guide --include=/home2/*/folder1/var --inc lude=/home2/*/folder1/usr/tmp --include=/home2/*/folder1/tmp --exclude=/ / dummy@1.2.3.4::/home2/backup/someserver then it will make a backup of /etc/master.passwd and exit.... Any ideas? Please........ Thanks, Vlastimil Adamovsky --- Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.365 / Virus Database: 202 - Release Date: 5/24/2002 _______________________________________________ Rdiff-backup mailing list Rdiff-backup@keywest.Stanford.EDU http://keywest.Stanford.EDU/mailman/listinfo/rdiff-backup --- Incoming mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.365 / Virus Database: 202 - Release Date: 5/24/2002 --- Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.365 / Virus Database: 202 - Release Date: 5/24/2002 From Billing@smarthosting.com Fri Jun 21 08:40:29 2002 From: Billing@smarthosting.com (Billing (Smarthosting Services)) Date: Fri, 21 Jun 2002 03:40:29 -0400 Subject: Star problem Message-ID: When I am trying : rdiff-backup --include=/usr/ports --exclude=/ / /temp then it works fine... If I am trying : rdiff-backup --include=/usr/ports/*/Makefile --exclude=/ / /temp then it will throw an exception.. It worked fine in previous versions. Can you try it out and tell me what to do? I am using FreeBSD 4.4 Thank you, Vlastimil Adamovsky --- Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.365 / Virus Database: 202 - Release Date: 5/24/2002 From bescoto@stanford.edu Mon Jun 24 19:27:03 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Mon, 24 Jun 2002 11:27:03 -0700 Subject: Still some strange problems In-Reply-To: Your message of "Wed, 19 Jun 2002 20:34:08 EDT." Message-ID: <200206241827.g5OIR3v26715@folly.Stanford.EDU> --==_Exmh_1346883037P Content-Type: multipart/mixed; boundary="----- =_aaaaaaaaaa0" Content-Id: <26703.1024943210.0@folly.Stanford.EDU> ------- =_aaaaaaaaaa0 Content-Type: text/plain; charset="us-ascii" Content-ID: <26703.1024943210.1@folly.Stanford.EDU> >>>>> "VA" == ambra >>>>> wrote the following on Wed, 19 Jun 2002 20:34:08 -0400 VA> I forgot to write that it will back up ONLY /etc/master.passwd VA> and ignore all those other directories to be backed up... It VA> has worked in previous versions Ok, I found the bug. Thanks for the heads-up. It affects all the regular expression/globbing type selection statements when the source directory is root ("/"). It'll be fixed in 0.9.2 or here is a patch: -- Ben Escoto ------- =_aaaaaaaaaa0 Content-Type: text/patch; charset="us-ascii" Content-ID: <26703.1024943210.2@folly.Stanford.EDU> --- rpath.py.old Wed Jun 19 12:17:20 2002 +++ rpath.py Mon Jun 24 11:17:07 2002 @@ -433,7 +433,9 @@ self.conn = connection self.index = index self.base = base - if base is not None: self.path = "/".join((base,) + index) + if base is not None: + if base == "/": self.path = "/" + "/".join(index) + else: self.path = "/".join((base,) + index) self.file = None if data or base is None: self.data = data else: self.data = self.conn.C.make_file_dict(self.path) ------- =_aaaaaaaaaa0-- --==_Exmh_1346883037P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE9F2Rz+owuOvknOnURAi6KAJ9qkb2Hdys7NGSzNXOFRphN1ieczwCgjGeK a/0WpW6NXFuvYFMJ7TSPc4w= =Ol+g -----END PGP SIGNATURE----- --==_Exmh_1346883037P-- From bescoto@stanford.edu Mon Jun 24 19:43:18 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Mon, 24 Jun 2002 11:43:18 -0700 Subject: An Interesting thought-maybe... In-Reply-To: Your message of "Wed, 19 Jun 2002 23:09:05 BST." Message-ID: <200206241843.g5OIhIc26815@folly.Stanford.EDU> --==_Exmh_1354654083P Content-Type: text/plain; charset=us-ascii >>>>> "KS" == Kevin Spicer >>>>> wrote the following on Wed, 19 Jun 2002 23:09:05 +0100 KS> When people zip files they a) change the filename and b) change KS> (in a binary sense) the content - but they don't change (in a KS> human sense) the files real content. I don't think it would be KS> unreasonable to guess that rdiff backup is transferring the KS> entire file again. However 99% of zip files consist of the KS> original filename with a suffix, if it was possible to summise KS> which file had been zipped and how and just replicate this KS> action on the remote machine this would likely be much quicker. ... KS> I guess there would be a lot of work here, but I just wondered KS> whether others think there might be gains to be made here? I think this falls under the general category of content specific diffing/delta generation. I don't know the technical term but I bet there is a lot of academic literature on it. Recognizing that a file isn't just a bunch of bytes could theoretically improve performance on many different file types. For instance, all natural language files could be translated into Esperanto or something, so if /foo/bible on the source end was an English translation of the bible, and /foo/bible on the receiving end was a Spanish translation, it would only take few bytes (directive "Translate Spanish->English") to encode the relevant delta. Well this example is fanciful, but I hope the idea is clear... In this case I think what you suggest would be hard, because, as Dean Gaudet pointed out, zipping a file is not a bytewise 1-1 transformation (different versions, different compression levels, etc.), and it seems we should preserve byte-level identity. However, unzipping usually is a 1-1 transformation, so it would be much easier to notice when a user has unzipped a file, and just tell the writing end to do that. It would still be pretty time-consuming to implement, though. Another idea along these lines was suggested a while ago by Dean: keep track of inodes, so rdiff-backup could detect when a file was moved, and not send over the whole file again. This also seems like it could be worth it, but would be long to implement, and the inode info would take up a lot of memory. -- Ben Escoto --==_Exmh_1354654083P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE9F2hF+owuOvknOnURAonBAKCD/jUaq3mOCPPbDIBBj2/okTom1QCfbLtA cMhQn4zIIE20qpT+UHcDJZQ= =5nqf -----END PGP SIGNATURE----- --==_Exmh_1354654083P-- From dean-list-rdiff-backup@arctic.org Wed Jun 26 00:41:18 2002 From: dean-list-rdiff-backup@arctic.org (dean gaudet) Date: Tue, 25 Jun 2002 16:41:18 -0700 (PDT) Subject: directory_statistics and pathnames with spaces Message-ID: hmm ben, i thought you were using some quoting function for the pathname in the directory_statistics file? i see lots of paths with spaces in mine ... it's still mostly parseable -- i can just parse "left to right" instead of using fixed $n notations in awk. but a bit of a pain. or did this get fixed post 0.7.6? perhaps i should upgrade :) -dean From bescoto@stanford.edu Wed Jun 26 03:35:59 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Tue, 25 Jun 2002 19:35:59 -0700 Subject: directory_statistics and pathnames with spaces In-Reply-To: Your message of "Tue, 25 Jun 2002 16:41:18 PDT." Message-ID: <200206260235.g5Q2Zx327844@folly.Stanford.EDU> --==_Exmh_2118084667P Content-Type: text/plain; charset=us-ascii >>>>> "DG" == dean gaudet >>>>> wrote the following on Tue, 25 Jun 2002 16:41:18 -0700 (PDT) DG> hmm ben, i thought you were using some quoting function for the DG> pathname in the directory_statistics file? i see lots of paths DG> with spaces in mine ... it's still mostly parseable -- i can DG> just parse "left to right" instead of using fixed $n notations DG> in awk. but a bit of a pain. Do you mean parsing from "right to left"? That is the way I thought people would do it. Newlines are quoted, and that is the important case - I think everything is uniquely readable the way it is now. Some recent version has a --null-separator (?) switch, which turns off quoting all together (but also affects the expected format of any include/exclude filelists). Anyway, I didn't realize that it would be a pain the way it is now. It would be pretty trivial to replace all spaces with '\x20', so I can do that in 0.9.2. -- Ben Escoto --==_Exmh_2118084667P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE9GSiN+owuOvknOnURAh66AJ9zNNWtVbvd/6WaJEvP2X3sTP1POACZAa2v DyEcSJFjbu4CyT9qNlg8m9E= =l3xr -----END PGP SIGNATURE----- --==_Exmh_2118084667P-- From dean-list-rdiff-backup@arctic.org Wed Jun 26 03:51:16 2002 From: dean-list-rdiff-backup@arctic.org (dean gaudet) Date: Tue, 25 Jun 2002 19:51:16 -0700 (PDT) Subject: directory_statistics and pathnames with spaces In-Reply-To: <200206260235.g5Q2Zx327844@folly.Stanford.EDU> Message-ID: On Tue, 25 Jun 2002, Ben Escoto wrote: > >>>>> "DG" == dean gaudet > >>>>> wrote the following on Tue, 25 Jun 2002 16:41:18 -0700 (PDT) > > DG> hmm ben, i thought you were using some quoting function for the > DG> pathname in the directory_statistics file? i see lots of paths > DG> with spaces in mine ... it's still mostly parseable -- i can > DG> just parse "left to right" instead of using fixed $n notations > DG> in awk. but a bit of a pain. > > Do you mean parsing from "right to left"? oops, yeah :) > Anyway, I didn't realize that it would be a pain the way it is > now. It would be pretty trivial to replace all spaces with '\x20', so > I can do that in 0.9.2. an alternative is to put the path at the end of a line rather than the beginning. but it isn't too hard to parse the way it is now. i just misunderstood. -dean From bescoto@stanford.edu Thu Jun 27 21:40:34 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Thu, 27 Jun 2002 13:40:34 -0700 Subject: Version 0.9.2 released Message-ID: <200206272040.g5RKeYw20490@folly.Stanford.EDU> --==_Exmh_1712942066P Content-Type: text/plain; charset=us-ascii This version has some new optimizations and doesn't use rdiff. Backing up my hard disk locally takes about 90 minutes with 0.8.0, 55 minutes with 0.9.0/0.9.1 and 35 minutes with 0.9.2. Unfortunately the current official version of librsync leaks a lot of memory, so if you are still running librsync 0.9.5 I recommend you upgrade to the "0.9.5.1" version I just put on the rdiff-backup home page. >From the changelog: New in v0.9.2 (2002/06/27) -------------------------- Interface directly with librsync(.a|.so) instead of running "rdiff" command line utility. This can significant save fork()ing time when processing lots of smallish files that have changed. Also, rdiff is no longer required to be in the PATH. Further speed optimizations, mostly reducing CPU consumption when scanning through unchanged files. Fixed Path bug which could caused globbing and regexp include/exclude statements to malfunction when the base of the source directory was "/" (root of filesystem). Thanks to Vlastimil Adamovsky for noting this bug. Added quoting for spaces in directory_statistics file, hopefully making it easier to parse. -- Ben Escoto --==_Exmh_1712942066P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE9G3g++owuOvknOnURAoEHAJ4/uzZ/Axi4EaRs9D7QYeaoUCsrsgCcCO4b oRk8nsIW3rrasneylsWEm5M= =R7nR -----END PGP SIGNATURE----- --==_Exmh_1712942066P-- From bescoto@stanford.edu Thu Jun 27 21:50:44 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Thu, 27 Jun 2002 13:50:44 -0700 Subject: librsync v0.9.5.1 Message-ID: <200206272050.g5RKoi420571@folly.Stanford.EDU> --==_Exmh_1714607818P Content-Type: text/plain; charset=us-ascii Version 0.9.5 of librsync has some memory leaks which don't matter much for rdiff but would cripple rdiff-backup 0.9.2. So I took librsync 0.9.5 and: 1. Applied Donovan Baarda's "MSVC6 and Cygwin" patch, which does a lot of stuff including fix some memory leaks. 2. Applied patches from Wayne Davidson which fix a couple more memory leaks. 3. Enable shared library building by default. So if you are running 0.9.2, you must upgrade past 0.9.5, either by downloading my "0.9.5.1" or patching your own version. Just so there is no confusion, the librsync packages on the rdiff-backup home page do not represent an official librsync release. I just made them from 0.9.5 by applying the patches I thought were necessary for adequate performance with rdiff-backup 0.9.2. Hopefully an official 0.9.6 or later will come out soon and unofficial packages won't be necessary. -- Ben Escoto --==_Exmh_1714607818P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE9G3qj+owuOvknOnURAtjaAKCNf/ltyrJ4FiyrpYTRjli+m4IpugCfbKpS 2Ez1xRUqqNQCOrx0e/hNrJg= =x7SH -----END PGP SIGNATURE----- --==_Exmh_1714607818P-- From paul@xtdnet.nl Fri Jun 28 00:17:45 2002 From: paul@xtdnet.nl (Paul Wouters) Date: Fri, 28 Jun 2002 01:17:45 +0200 (MET DST) Subject: -mount (--one-file-system) like option for rdiff-backup? Message-ID: Hi, I'm using my own little script for running offsite rsync backups and just noticed rdiff-backup. This seems to do what I try to do myself using something like: rsync --bwlimit=0 --numeric-ids --compress --rsh=/usr/bin/ssh --recursive --archive --relative --sparse --one-file-system --compare-dest=/vol/backup/$HOSTNAME/Current $HOSTNAME:$DIRECTORY /vol/backup/$HOSTNAME/$DATE Then once a week I run this with --delete to get rid of the older files. Ofcourse, I end up with rsync rebuilding the entire structure, so even though for bandwidth this is nice, for storage, it's rather a waste of diskspace. So while checking to see if someone had hacked rsync or written a script to replace identical files with hardlinks, I noticed rdiff-backup. the only feature that seems to be missing at first glance is the --one-file-system option. I'd like to be able to backup / without a mounted /usr for example. Can rdiff-backup do this somehow, without coming up with include or exlude statements, so that I can just feed it hostname:directory entries to backup without needing to write custom rules per entry? Thanks, Paul -- "Movie scripts no longer write, George Lucas shall" From dean-list-rdiff-backup@arctic.org Fri Jun 28 01:11:22 2002 From: dean-list-rdiff-backup@arctic.org (dean gaudet) Date: Thu, 27 Jun 2002 17:11:22 -0700 (PDT) Subject: -mount (--one-file-system) like option for rdiff-backup? In-Reply-To: Message-ID: ben -- this, plus other features i've been pondering, make me wonder about external filelist generation for rdiff-backup. --include-filelist goes some of the way -- but if you include a directory in --include-filelist then rdiff-backup will recurse. also -- for some file selection problems you really need access to the complete file list on both sides of the link. it seems hard to move this outside of rdiff-backup. the problem i run into occasionally is that my users will upload a couple Gb of data which is a backup of their accounts elsewhere... and they're sometimes not nice enough to ask me beforehand, or i'd tell them how to avoid it ending up in my nightly rdiff-backup. so what i was considering is some sort of "unexpected large delta" detection prior to transmitting across the link. the most simple form of this would be "this 200MB+ file/directory is not yet on the mirror, it won't be backed up until it's whitelisted somewhere". i almost asked for a feature like that (and --one-file-system was on my list early on, but i eventually convinced myself that rdiff-backup of everything in one hierarchy was best for me). but i worried that featuritis in file selection was an indication of a need for a different design :) i don't really have an answer... but i keep thinking about multiple processes sharing data via pipes, so that we could replace some components with locally modified pieces to get new features. -dean From dean-list-rdiff-backup@arctic.org Fri Jun 28 01:12:48 2002 From: dean-list-rdiff-backup@arctic.org (dean gaudet) Date: Thu, 27 Jun 2002 17:12:48 -0700 (PDT) Subject: Version 0.9.2 released In-Reply-To: <200206272040.g5RKeYw20490@folly.Stanford.EDU> Message-ID: On Thu, 27 Jun 2002, Ben Escoto wrote: > This version has some new optimizations and doesn't use rdiff. > Backing up my hard disk locally takes about 90 minutes with 0.8.0, 55 > minutes with 0.9.0/0.9.1 and 35 minutes with 0.9.2. nice! i'll have to find you some more bottlenecks to remove, eh? :) -dean From bescoto@stanford.edu Fri Jun 28 01:50:47 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Thu, 27 Jun 2002 17:50:47 -0700 Subject: -mount (--one-file-system) like option for rdiff-backup? In-Reply-To: Your message of "Fri, 28 Jun 2002 01:17:45 +0200." Message-ID: <200206280050.g5S0ol922195@folly.Stanford.EDU> --==_Exmh_1780451110P Content-Type: text/plain; charset=us-ascii >>>>> "PW" == Paul Wouters >>>>> wrote the following on Fri, 28 Jun 2002 01:17:45 +0200 (MET DST) PW> the only feature that seems to be missing at first glance is the PW> --one-file-system option. I'd like to be able to backup / PW> without a mounted /usr for example. PW> Can rdiff-backup do this somehow, without coming up with include PW> or exlude statements, so that I can just feed it PW> hostname:directory entries to backup without needing to write PW> custom rules per entry? BTW, rdiff-backup uses two colon as in "hostname::directory". It does this because the increments have colons in them (because of the time format) so indicating a hostname with only one colon would require a lot of escaping when restoring an increment. Anyway, there is no --one-file-system option, but there's no reason it couldn't be added I suppose. Why not just use --exclude though? It is too inconvenient when hierarchies have lots of file systems in them or the mount points change frequently? Also, just to make sure I know what a "file system" is, are two files on the same file system if and only if their device numbers are the same? Or is it something else? For instance, could / and /usr be different file systems even if they are both ext2? -- Ben Escoto --==_Exmh_1780451110P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD4DBQE9G7Ll+owuOvknOnURAmvtAJQPxPeYJI5xi+2qvfNBfTXTHDQYAJ4gv3mE V4eAY5WneTSCyLeboiQnIA== =0smc -----END PGP SIGNATURE----- --==_Exmh_1780451110P-- From bescoto@stanford.edu Fri Jun 28 02:05:41 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Thu, 27 Jun 2002 18:05:41 -0700 Subject: -mount (--one-file-system) like option for rdiff-backup? In-Reply-To: Your message of "Thu, 27 Jun 2002 17:11:22 PDT." Message-ID: <200206280105.g5S15fw22301@folly.Stanford.EDU> --==_Exmh_1789494234P Content-Type: text/plain; charset=us-ascii >>>>> "DG" == dean gaudet >>>>> wrote the following on Thu, 27 Jun 2002 17:11:22 -0700 (PDT) DG> --include-filelist goes some of the way -- but if you include a DG> directory in --include-filelist then rdiff-backup will recurse. Are you sure? The man page says: Each line in a filelist is interpreted similarly to the way extended shell patterns are, with a few exceptions: 1. Globbing patterns like *, **, ?, and [...] are not expanded. 2. Include patterns do not match files in a directory that is included. So /usr/local in an include file will not match /usr/local/doc. 3. Lines starting with "+ " are interpreted as include directives, even if found in a filelist referenced by --exclude-filelist. Similarly, lines starting with "- " exclude files even if they are found within an include filelist. So that is not the intention at least... DG> also -- for some file selection problems you really need access DG> to the complete file list on both sides of the link. it seems DG> hard to move this outside of rdiff-backup. ... DG> so what i was considering is some sort of "unexpected large DG> delta" detection prior to transmitting across the link. the DG> most simple form of this would be "this 200MB+ file/directory is DG> not yet on the mirror, it won't be backed up until it's DG> whitelisted somewhere". Hmm, I can see how large differences in files could be detected, but it would be harder to detect a difference of 200MB spread out over a directory. Currently rdiff-backup just makes one pass, and it would be more complicated to have it look ahead into directories on both ends to see if the total difference would add up to 200MB, and if so not to back up the directory in the first place. Then again, it might not be so hard to just quit backing up the rest once 200MB of total changes are found... It would be like giving users a quota not on total disk space, but on how much they can change their directory. But even if this were painless to implement, it raises a lot of interface issues. Do we measure changes on how many/how large new files are? Or how many bytes get added to files? Once a directory's change quota gets exceeded, do we stop processing the directory altogether, or do we still allow file changes/deletions. Do we specify the quota by directory, with different directories have different quotas, etc.? Just today I read an interesting article on interface design by Havoc Pennington: http://www106.pair.com/rhp/free-software-ui.html he makes a strong case that there is a real cost to adding too many preferences, so that's why I'm hesitant to add something like this. And I'm sure people can think of more esoteric include/exclude rules.. Maybe if it could be brought under something general but conceptually simple, like some kind of plugin system... I don't know.. DG> i don't really have an answer... but i keep thinking about DG> multiple processes sharing data via pipes, so that we could DG> replace some components with locally modified pieces to get new DG> features. Sorry, could you elaborate on this? -- Ben Escoto --==_Exmh_1789494234P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE9G7Zj+owuOvknOnURAr4iAJ95Ke+4whgGZLEIES7OgmZ25qPTHACbBmNv 0HrQg/iKy9jnmDbTLMX6Th8= =Rr3z -----END PGP SIGNATURE----- --==_Exmh_1789494234P-- From paul@xtdnet.nl Fri Jun 28 02:24:09 2002 From: paul@xtdnet.nl (Paul Wouters) Date: Fri, 28 Jun 2002 03:24:09 +0200 (MET DST) Subject: -mount (--one-file-system) like option for rdiff-backup? In-Reply-To: <200206280050.g5S0ol922195@folly.Stanford.EDU> Message-ID: On Thu, 27 Jun 2002, Ben Escoto wrote: > Anyway, there is no --one-file-system option, but there's no > reason it couldn't be added I suppose. Why not just use --exclude > though? It is too inconvenient when hierarchies have lots of file > systems in them or the mount points change frequently? Mostly because then you can keep thinking in "disks". If I add a disk, I need to explicitely put an entry in the backup scheme. I won't ever accidently backup the same data numerous times because /usr/local/ is mounted on /usr is mounted on / Often, a mounted disk means a different subset of data, eg mail, logs, webpages, or even a shared directory over multiple machines. It's good to be able to back those up as seperate data sets. On my machines, the / filesystem is relatively small, but important (for /etc/ for example). Specifying / without 'staying on one filesystem' options would backup everything that was mounted on the machine. Another reason is to just cut down on numerous problems with mounted filesystems. For instance, I can exclude the most obvious candidates now, such as /mnt, /tmp, or proc, but I don't know what other features will be presented in OS'ses at a later stage. Or users mounting remote NFS directories temporarily (eg some ftp server) Anyway, a lot of programs have this option, because it has proven to be very useful. find, locate, rsync, cp, etc etc all have the option. It's very useful. > Also, just to make sure I know what a "file system" is, are two > files on the same file system if and only if their device numbers are > the same? Or is it something else? For instance, could / and /usr be > different file systems even if they are both ext2? with "one filesystem" I meant one instance. eg one mounted directory, or one physical disk partition. It has nothing to do with the type of filesystem used (dos,ext2,vfat,nfs). Paul -- "Movie scripts no longer write, George Lucas shall" From bescoto@stanford.edu Fri Jun 28 06:10:38 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Thu, 27 Jun 2002 22:10:38 -0700 Subject: -mount (--one-file-system) like option for rdiff-backup? In-Reply-To: Your message of "Fri, 28 Jun 2002 03:24:09 +0200." Message-ID: <200206280510.g5S5AcW23732@folly.Stanford.EDU> --==_Exmh_1856153942P Content-Type: text/plain; charset=us-ascii Ok, I see your point. I can think of two ways of implementing this. Either way, I think the option should fit into the current include/exclude priority scheme. 1. Have an --exclude-other-devices switch, that excludes files on devices other than the device the root is on. So if /usr were on a different device, "--include /usr/local --exclude-other-devices" would include /usr/local and everything under it, but not anything else in /usr. 2. Unfortunately, the above scheme doesn't have an easy way of saying "backup / and /usr/local, but don't move from one device to another given that." "--exclude-other-devices --include /usr/local --exclude-other-devices" wouldn't work, because /usr/local would be excluded first before the --include was processed. So one way would be add a --exclude-different-device-from switch, so the above could be accomplished by saying, "--exclude-different-device-from /usr/local --include /usr/local --exclude-different-device from /". Hmm, and I guess there could also be --include-same-device-as switches... But maybe the situation in (2) doesn't come up very often and (1) is good enough... -- Ben Escoto --==_Exmh_1856153942P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE9G+/N+owuOvknOnURAjhsAJ9diAE+YF4KBG1X/v0h6xYsrNg1GgCfTLAA CAljYDcFViEXZoVve/z0is4= =kW9F -----END PGP SIGNATURE----- --==_Exmh_1856153942P-- From Kevin.Spicer@bmrb.co.uk Fri Jun 28 09:33:26 2002 From: Kevin.Spicer@bmrb.co.uk (Spicer, Kevin) Date: Fri, 28 Jun 2002 09:33:26 +0100 Subject: Problems building 0.9.2 on solaris 2.6 Message-ID: This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C21E7E.78D4F4D0 Content-Type: text/plain; charset="iso-8859-1" Hello, I've had some problems building rdiff-backup 0.9.2 on Solaris - this is the tail ed of the messages... 0x17c /usr/local/lib/librsync.a(msg.o) 0x178 /usr/local/lib/librsync.a(msg.o) 0xa8 /usr/local/lib/librsync.a(msg.o) 0x174 /usr/local/lib/librsync.a(msg.o) 0x3c /usr/local/lib/librsync.a(msg.o) 0xa4 /usr/local/lib/librsync.a(msg.o) ld: fatal: relocations remain against allocatable but non-writable sections error: command 'gcc' failed with exit status 1 I've fixed this error before (on other packages) by adding -fPIC to the gcc flags in the makefile. But I can't see a Makefile here, and I don't know enough about Python to track it down. Any ideas? BMRB International http://www.bmrb.co.uk +44 (0)20 8566 5000 ____________________________________________________________ This message (and any attachment) is intended only for the recipient and may contain confidential and/or privileged material. If you have received this in error, please contact the sender and delete this message immediately. Disclosure, copying or other action taken in respect of this email or in reliance on it is prohibited. BMRB International Limited accepts no liability in relation to any personal emails, or content of any email which does not directly relate to our business. ------_=_NextPart_001_01C21E7E.78D4F4D0 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Problems building 0.9.2 on solaris 2.6

Hello,
        I've had = some problems building rdiff-backup 0.9.2 on Solaris - this is the tail = ed of the messages...

<unknown>        =             =        = 0x17c       = /usr/local/lib/librsync.a(msg.o)
<unknown>        =             =        = 0x178       = /usr/local/lib/librsync.a(msg.o)
<unknown>        =             =        = 0xa8        = /usr/local/lib/librsync.a(msg.o)
<unknown>        =             =        = 0x174       = /usr/local/lib/librsync.a(msg.o)
<unknown>        =             =        = 0x3c        = /usr/local/lib/librsync.a(msg.o)
<unknown>        =             =        = 0xa4        = /usr/local/lib/librsync.a(msg.o)
ld: fatal: relocations remain against allocatable = but non-writable sections
error: command 'gcc' failed with exit status = 1

I've fixed this error before (on other packages) by = adding -fPIC to the gcc flags in the makefile.  But I can't see a = Makefile here, and I don't know enough about Python to track it = down.  Any ideas?



BMRB International
http://www.bmrb.co.uk +44 (0)20 8566 5000

____________________________________________________________ =
This message (and any attachment) is intended only = for the recipient and may contain confidential and/or privileged = material. If you have received this in error, please contact the sender = and delete this message immediately. Disclosure, copying or other = action taken in respect of this email or in reliance on it is = prohibited. BMRB International Limited accepts no liability in relation = to any personal emails, or content of any email which does not directly = relate to our business.

------_=_NextPart_001_01C21E7E.78D4F4D0-- From ambra@ambrasoft.com Fri Jun 28 14:58:24 2002 From: ambra@ambrasoft.com (ambra) Date: Fri, 28 Jun 2002 09:58:24 -0400 Subject: Problems building 0.9.2 on solaris 2.6 In-Reply-To: Message-ID: This is a multi-part message in MIME format. ------=_NextPart_000_000F_01C21E8A.57F8D2C0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Problems building 0.9.2 on solaris 2.6I had also lot of "could not find" messages when kompiling the extension written in C.. I am running FreeBSD 4.4 and 4.5 Vlastimil Adamovsky -----Original Message----- From: rdiff-backup-admin@keywest.Stanford.EDU [mailto:rdiff-backup-admin@keywest.Stanford.EDU]On Behalf Of Spicer, Kevin Sent: Friday, June 28, 2002 4:33 AM To: 'rdiff-backup@keywest.Stanford.EDU' Subject: Problems building 0.9.2 on solaris 2.6 Hello, I've had some problems building rdiff-backup 0.9.2 on Solaris - this is the tail ed of the messages... 0x17c /usr/local/lib/librsync.a(msg.o) 0x178 /usr/local/lib/librsync.a(msg.o) 0xa8 /usr/local/lib/librsync.a(msg.o) 0x174 /usr/local/lib/librsync.a(msg.o) 0x3c /usr/local/lib/librsync.a(msg.o) 0xa4 /usr/local/lib/librsync.a(msg.o) ld: fatal: relocations remain against allocatable but non-writable sections error: command 'gcc' failed with exit status 1 I've fixed this error before (on other packages) by adding -fPIC to the gcc flags in the makefile. But I can't see a Makefile here, and I don't know enough about Python to track it down. Any ideas? BMRB International http://www.bmrb.co.uk +44 (0)20 8566 5000 ____________________________________________________________ This message (and any attachment) is intended only for the recipient and may contain confidential and/or privileged material. If you have received this in error, please contact the sender and delete this message immediately. Disclosure, copying or other action taken in respect of this email or in reliance on it is prohibited. BMRB International Limited accepts no liability in relation to any personal emails, or content of any email which does not directly relate to our business. --- Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.365 / Virus Database: 202 - Release Date: 5/24/2002 ------=_NextPart_000_000F_01C21E8A.57F8D2C0 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Problems building 0.9.2 on solaris 2.6
I had=20 also lot of "could not find" messages when kompiling the extension = written in=20 C..
 
I am=20 running FreeBSD 4.4 and 4.5
 
 
Vlastimil Adamovsky
-----Original Message-----
From:=20 rdiff-backup-admin@keywest.Stanford.EDU=20 [mailto:rdiff-backup-admin@keywest.Stanford.EDU]On Behalf Of = Spicer,=20 Kevin
Sent: Friday, June 28, 2002 4:33 AM
To:=20 'rdiff-backup@keywest.Stanford.EDU'
Subject: Problems = building 0.9.2=20 on solaris 2.6

Hello, =
       =20 I've had some problems building rdiff-backup 0.9.2 on = Solaris -=20 this is the tail ed of the messages...

<unknown>        &= nbsp;           &n= bsp;     =20 0x17c      =20 /usr/local/lib/librsync.a(msg.o)
<unknown>        &= nbsp;           &n= bsp;     =20 0x178      =20 /usr/local/lib/librsync.a(msg.o)
<unknown>        &= nbsp;           &n= bsp;     =20 0xa8       =20 /usr/local/lib/librsync.a(msg.o)
<unknown>        &= nbsp;           &n= bsp;     =20 0x174      =20 /usr/local/lib/librsync.a(msg.o)
<unknown>        &= nbsp;           &n= bsp;     =20 0x3c       =20 /usr/local/lib/librsync.a(msg.o)
<unknown>        &= nbsp;           &n= bsp;     =20 0xa4       =20 /usr/local/lib/librsync.a(msg.o)
ld: fatal:=20 relocations remain against allocatable but non-writable = sections=20
error: command 'gcc' failed with exit status = 1

I've fixed this error before (on other packages) by = adding=20 -fPIC to the gcc flags in the makefile.  But I can't see a = Makefile here,=20 and I don't know enough about Python to track it down.  Any=20 ideas?



BMRB International
http://www.bmrb.co.uk +44 (0)20=20 8566 5000

____________________________________________________________=20
This message (and any attachment) is = intended only for=20 the recipient and may contain confidential and/or privileged material. = If you=20 have received this in error, please contact the sender and delete this = message=20 immediately. Disclosure, copying or other action taken in respect of = this=20 email or in reliance on it is prohibited. BMRB International Limited = accepts=20 no liability in relation to any personal emails, or content of any = email which=20 does not directly relate to our = business.

------=_NextPart_000_000F_01C21E8A.57F8D2C0-- From bescoto@stanford.edu Fri Jun 28 16:23:15 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Fri, 28 Jun 2002 08:23:15 -0700 Subject: Problems building 0.9.2 on solaris 2.6 In-Reply-To: Your message of "Fri, 28 Jun 2002 09:33:26 BST." Message-ID: <200206281523.g5SFNFC27710@folly.Stanford.EDU> --==_Exmh_-1021156350P Content-Type: text/plain; charset=us-ascii Did you two install the librsync-0.9.5.1 package? Did it build a share library (librsync.so and not just librsync.a)? -- Ben Escoto --==_Exmh_-1021156350P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE9HH9f+owuOvknOnURAudDAKCNYOsRrHYMsUrKXkfBLODEFH0tcQCghYan swqkbxG7esjzymCD9oTTZ54= =SY21 -----END PGP SIGNATURE----- --==_Exmh_-1021156350P-- From Kevin.Spicer@bmrb.co.uk Fri Jun 28 16:36:03 2002 From: Kevin.Spicer@bmrb.co.uk (Spicer, Kevin) Date: Fri, 28 Jun 2002 16:36:03 +0100 Subject: Problems building 0.9.2 on solaris 2.6 Message-ID: This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C21EB9.82F088E0 Content-Type: text/plain; charset="iso-8859-1" I did install librsync-0.9.5.1, but only ended up with .a and .la libraries. Any idea how I can force it to build shared libs? -----Original Message----- From: Ben Escoto [mailto:bescoto@stanford.edu] Sent: 28 June 2002 16:23 To: Spicer, Kevin; ambra Cc: 'rdiff-backup@keywest.Stanford.EDU' Subject: Re: Problems building 0.9.2 on solaris 2.6 Did you two install the librsync-0.9.5.1 package? Did it build a share library (librsync.so and not just librsync.a)? -- Ben Escoto BMRB International http://www.bmrb.co.uk +44 (0)20 8566 5000 ____________________________________________________________ This message (and any attachment) is intended only for the recipient and may contain confidential and/or privileged material. If you have received this in error, please contact the sender and delete this message immediately. Disclosure, copying or other action taken in respect of this email or in reliance on it is prohibited. BMRB International Limited accepts no liability in relation to any personal emails, or content of any email which does not directly relate to our business. ------_=_NextPart_001_01C21EB9.82F088E0 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable RE: Problems building 0.9.2 on solaris 2.6

I did install librsync-0.9.5.1, but only ended up = with .a and .la libraries.  Any idea how I can force it to build = shared libs?

-----Original Message-----
From: Ben Escoto [mailto:bescoto@stanford.edu]
Sent: 28 June 2002 16:23
To: Spicer, Kevin; ambra
Cc: 'rdiff-backup@keywest.Stanford.EDU'
Subject: Re: Problems building 0.9.2 on solaris 2.6 =



Did you two install the librsync-0.9.5.1 = package?  Did it build a
share library (librsync.so and not just = librsync.a)?


--
Ben Escoto


BMRB International
http://www.bmrb.co.uk +44 (0)20 8566 5000

____________________________________________________________ =
This message (and any attachment) is intended only = for the recipient and may contain confidential and/or privileged = material. If you have received this in error, please contact the sender = and delete this message immediately. Disclosure, copying or other = action taken in respect of this email or in reliance on it is = prohibited. BMRB International Limited accepts no liability in relation = to any personal emails, or content of any email which does not directly = relate to our business.

------_=_NextPart_001_01C21EB9.82F088E0-- From Jason_Piterak@c-i-s.com Fri Jun 28 17:30:25 2002 From: Jason_Piterak@c-i-s.com (Jason Piterak) Date: Fri, 28 Jun 2002 12:30:25 -0400 Subject: Problems building 0.9.2 on solaris 2.6 Message-ID: Kevin, Did you try './configure --enable-shared'? ... Looks like the shared libraries don't compile by default...: $ ./configure --help Optional Features: ... --enable-shared=PKGS build shared libraries default=no --enable-static=PKGS build static libraries default=yes ... --- Jason Piterak System Architect CIS Technical Services 33 Main St., Suite 302 Nashua, NH 03064 (603) 889-4684 - FAX (603) 889-0534 -----Original Message----- From: rdiff-backup-admin@keywest.Stanford.EDU [mailto:rdiff-backup-admin@keywest.Stanford.EDU]On Behalf Of Spicer, Kevin Sent: Friday, June 28, 2002 11:36 AM To: 'Ben Escoto' Cc: 'rdiff-backup@keywest.stanford.edu' Subject: RE: Problems building 0.9.2 on solaris 2.6 I did install librsync-0.9.5.1, but only ended up with .a and .la libraries. Any idea how I can force it to build shared libs? -----Original Message----- From: Ben Escoto [mailto:bescoto@stanford.edu] Sent: 28 June 2002 16:23 To: Spicer, Kevin; ambra Cc: 'rdiff-backup@keywest.Stanford.EDU' Subject: Re: Problems building 0.9.2 on solaris 2.6 Did you two install the librsync-0.9.5.1 package? Did it build a share library (librsync.so and not just librsync.a)? -- Ben Escoto BMRB International http://www.bmrb.co.uk +44 (0)20 8566 5000 ____________________________________________________________ This message (and any attachment) is intended only for the recipient and may contain confidential and/or privileged material. If you have received this in error, please contact the sender and delete this message immediately. Disclosure, copying or other action taken in respect of this email or in reliance on it is prohibited. BMRB International Limited accepts no liability in relation to any personal emails, or content of any email which does not directly relate to our business. From Jason_Piterak@c-i-s.com Fri Jun 28 17:53:06 2002 From: Jason_Piterak@c-i-s.com (Jason Piterak) Date: Fri, 28 Jun 2002 12:53:06 -0400 Subject: rdiff-backup (0.9.1 or 0.9.2) will not run from the build directo ry Message-ID: Hi Ben, Found this odd... No big deal, but if you're like I am it might take a while to figure out...;-) rdiff-backup (0.9.1 or 0.9.2) will not run from the build directory. [root@sasherlinux rdiff-backup-0.9.2]# rdiff-backup --version Traceback (most recent call last): File "/usr/local/bin/rdiff-backup", line 21, in ? import rdiff_backup.Main File "/home/downloads/server/backup/rdiff-backup/rdiff-backup/rdiff-backup-0.9.2/ rdiff_backup/Main.py", line 14, in ? from log import * File "/home/downloads/server/backup/rdiff-backup/rdiff-backup/rdiff-backup-0.9.2/ rdiff_backup/log.py", line 168, in ? import Globals, Main File "/home/downloads/server/backup/rdiff-backup/rdiff-backup/rdiff-backup-0.9.2/ rdiff_backup/Globals.py", line 238, in ? from rpath import * # kludge to avoid circularity - not needed in this module File "/home/downloads/server/backup/rdiff-backup/rdiff-backup/rdiff-backup-0.9.2/ rdiff_backup/rpath.py", line 796, in ? from lazy import * File "/home/downloads/server/backup/rdiff-backup/rdiff-backup/rdiff-backup-0.9.2/ rdiff_backup/lazy.py", line 352, in ? from robust import * File "/home/downloads/server/backup/rdiff-backup/rdiff-backup/rdiff-backup-0.9.2/ rdiff_backup/robust.py", line 40, in ? import tempfile, errno, signal, cPickle, C, pickle ImportError: No module named C [root@sasherlinux rdiff-backup-0.9.2]# cd .. [root@sasherlinux rdiff-backup]# rdiff-backup --version rdiff-backup 0.9.2 [root@sasherlinux rdiff-backup]# --- Jason Piterak System Architect CIS Technical Services 33 Main St., Suite 302 Nashua, NH 03064 (603) 889-4684 - FAX (603) 889-0534 From bescoto@stanford.edu Sat Jun 29 00:57:54 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Fri, 28 Jun 2002 16:57:54 -0700 Subject: Problems building 0.9.2 on solaris 2.6 In-Reply-To: Your message of "Fri, 28 Jun 2002 12:30:25 EDT." Message-ID: <200206282357.g5SNvsh11540@folly.Stanford.EDU> --==_Exmh_-883812464P Content-Type: text/plain; charset=us-ascii >>>>> "JP" == Jason Piterak >>>>> wrote the following on Fri, 28 Jun 2002 12:30:25 -0400 JP> Kevin, Did you try './configure --enable-shared'? ... Looks JP> like the shared libraries don't compile by default...: JP> $ ./configure --help Optional Features: ... JP> --enable-shared=PKGS build shared libraries default=no JP> --enable-static=PKGS build static libraries default=yes ... Apparently I forgot to update the configure script. On my system (RH linux) it gets regenerated automatically from the autoconf file when "make" is run, so I got shared libraries, but I guess this isn't a standard. Anyway, I put up new versions of libsync-0.9.5.1 (0.9.5.1.1?) that should have the right configure script. Let me know if there is some problem with these. -- Ben Escoto --==_Exmh_-883812464P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE9HPgA+owuOvknOnURAlyZAJ4zW3FaExy8mW/2vIiNNlvkR+5nMwCfR+mB CgfVx38k3jlW82VFWGxscSQ= =/y3F -----END PGP SIGNATURE----- --==_Exmh_-883812464P-- From bescoto@stanford.edu Sat Jun 29 01:07:03 2002 From: bescoto@stanford.edu (Ben Escoto) Date: Fri, 28 Jun 2002 17:07:03 -0700 Subject: rdiff-backup (0.9.1 or 0.9.2) will not run from the build directo ry In-Reply-To: Your message of "Fri, 28 Jun 2002 12:53:06 EDT." Message-ID: <200206290007.g5T073x11638@folly.Stanford.EDU> --==_Exmh_-875664764P Content-Type: text/plain; charset=us-ascii >>>>> "JP" == Jason Piterak >>>>> wrote the following on Fri, 28 Jun 2002 12:53:06 -0400 JP> Hi Ben, Found this odd... No big deal, but if you're like I am JP> it might take a while to figure out...;-) rdiff-backup (0.9.1 or JP> 0.9.2) will not run from the build directory. JP> [root@sasherlinux rdiff-backup-0.9.2]# rdiff-backup --version Hmm, on my system: ~/work/rdiff-backup-0.9.2 $ rdiff-backup --version rdiff-backup 0.9.2 ~/work/rdiff-backup-0.9.2 $ ./rdiff-backup --version Traceback (most recent call last): File "./rdiff-backup", line 20, in ? ... So when you typed in "rdiff-backup", which script was run, a previous copy you had in /usr/local/bin or somewhere? Or do you have "." in your PATH (as root no less!)? If the problem is just that the uninstalled rdiff-backup script doesn't work, I will just make it less tempting, maybe by removing its exec permissions and/or moving it to a different directory. Thanks for the tip though; I can see how those kinds of things can be frustrating... -- Ben Escoto --==_Exmh_-875664764P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Exmh version 2.5 01/15/2001 iD8DBQE9HPom+owuOvknOnURAsM1AJ9875yCa2tKHZnCXZOVtgpK19DtswCfezTY SPTwoWyT9AhuIGcSAjVQWbo= =1rWv -----END PGP SIGNATURE----- --==_Exmh_-875664764P-- From Kevin.Spicer@bmrb.co.uk Sat Jun 29 15:59:22 2002 From: Kevin.Spicer@bmrb.co.uk (Spicer, Kevin) Date: Sat, 29 Jun 2002 15:59:22 +0100 Subject: Problems building 0.9.2 on solaris 2.6 Message-ID: This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C21F7D.8DA19E2C Content-Type: text/plain; charset="iso-8859-1" Cracked it now, thanks for the pointer. I only had static libraries for librsync, to build the shared ones I needed to do... ./configure --enable-shared; make; make install Unfortunately this option didn't seem to be documented (had to read the configure script) - I guess I should have guessed it though :) However, librsync wouldn't build shared libraries until I had rebuilt zlib with shared libararies too (./configure --shared in that case) I did have one small problem because rdiff-backup tried (and failed) to build using the old static libraries in /usr/local/lib, because I had put the new shared libs in /usr/lib. Strangely LD_LIBRARY_PATH didn't seem to have any effect, so I copied the shared libs in /usr/local/libs and all was well. -----Original Message----- From: Spicer, Kevin [mailto:Kevin.Spicer@bmrb.co.uk] Sent: 28 June 2002 16:36 To: 'Ben Escoto' Cc: 'rdiff-backup@keywest.stanford.edu' Subject: RE: Problems building 0.9.2 on solaris 2.6 I did install librsync-0.9.5.1, but only ended up with .a and .la libraries. Any idea how I can force it to build shared libs? -----Original Message----- From: Ben Escoto [ mailto:bescoto@stanford.edu ] Sent: 28 June 2002 16:23 To: Spicer, Kevin; ambra Cc: 'rdiff-backup@keywest.Stanford.EDU' Subject: Re: Problems building 0.9.2 on solaris 2.6 Did you two install the librsync-0.9.5.1 package? Did it build a share library (librsync.so and not just librsync.a)? -- Ben Escoto BMRB International http://www.bmrb.co.uk +44 (0)20 8566 5000 ____________________________________________________________ This message (and any attachment) is intended only for the recipient and may contain confidential and/or privileged material. If you have received this in error, please contact the sender and delete this message immediately. Disclosure, copying or other action taken in respect of this email or in reliance on it is prohibited. BMRB International Limited accepts no liability in relation to any personal emails, or content of any email which does not directly relate to our business. BMRB International http://www.bmrb.co.uk +44 (0)20 8566 5000 ____________________________________________________________ This message (and any attachment) is intended only for the recipient and may contain confidential and/or privileged material. If you have received this in error, please contact the sender and delete this message immediately. Disclosure, copying or other action taken in respect of this email or in reliance on it is prohibited. BMRB International Limited accepts no liability in relation to any personal emails, or content of any email which does not directly relate to our business. ------_=_NextPart_001_01C21F7D.8DA19E2C Content-Type: text/html; charset="iso-8859-1" RE: Problems building 0.9.2 on solaris 2.6
Cracked it now, thanks for the pointer.  I only had static libraries for librsync, to build the shared ones I needed to do...
./configure --enable-shared; make; make install
Unfortunately this option didn't seem to be documented (had to read the configure script) - I guess I should have guessed it though :)
However, librsync wouldn't build shared libraries until I had rebuilt zlib with shared libararies too (./configure --shared in that case)
 
I did have one small problem because rdiff-backup tried (and failed) to build using the old static libraries in /usr/local/lib, because I had put the new shared libs in /usr/lib.  Strangely LD_LIBRARY_PATH didn't seem to have any effect, so I copied the shared libs in /usr/local/libs and all was well.
 
-----Original Message-----
From: Spicer, Kevin [mailto:Kevin.Spicer@bmrb.co.uk]
Sent: 28 June 2002 16:36
To: 'Ben Escoto'
Cc: 'rdiff-backup@keywest.stanford.edu'
Subject: RE: Problems building 0.9.2 on solaris 2.6

I did install librsync-0.9.5.1, but only ended up with .a and .la libraries.  Any idea how I can force it to build shared libs?

-----Original Message-----
From: Ben Escoto [mailto:bescoto@stanford.edu]
Sent: 28 June 2002 16:23
To: Spicer, Kevin; ambra
Cc: 'rdiff-backup@keywest.Stanford.EDU'
Subject: Re: Problems building 0.9.2 on solaris 2.6



Did you two install the librsync-0.9.5.1 package?  Did it build a
share library (librsync.so and not just librsync.a)?


--
Ben Escoto


BMRB International
http://www.bmrb.co.uk +44 (0)20 8566 5000

____________________________________________________________
This message (and any attachment) is intended only for the recipient and may contain confidential and/or privileged material. If you have received this in error, please contact the sender and delete this message immediately. Disclosure, copying or other action taken in respect of this email or in reliance on it is prohibited. BMRB International Limited accepts no liability in relation to any personal emails, or content of any email which does not directly relate to our business.



BMRB International

http://www.bmrb.co.uk +44 (0)20 8566 5000


____________________________________________________________

This message (and any attachment) is intended only for the recipient and may contain confidential and/or privileged material. If you have received this in error, please contact the sender and delete this message immediately. Disclosure, copying or other action taken in respect of this email or in reliance on it is prohibited. BMRB International Limited accepts no liability in relation to any personal emails, or content of any email which does not directly relate to our business.

------_=_NextPart_001_01C21F7D.8DA19E2C--