bottlenecks

[Donovan Baarda]

I've Cc'd this to the rproxy-devel list for reference.

On Tue, Jun 18, 2002 at 03:07:13AM -0700, Ben Escoto wrote:
> >>>>> "DG" == dean gaudet <dean-list-rdiff-backup@arctic.org>
> >>>>> wrote the following on Mon, 17 Jun 2002 12:43:24 -0700 (PDT)
[...]
> The internal librsync procedure used up only about 1/20th of the CPU
> time as the external rdiff procedure, meaning I suppose that 95% of
> the CPU time was spent forking, setting up the external rdiff process,
> etc.  Of course this is for small files; as the file sizes became
> large, the forking time would go to 0 as a percentage of the total.
> 
>     Oh, and the reason that test3.py had such a large real time is
> because it ended up using hundreds of MB of memory and my system
> started swapping like crazy.  I guess all this efficiency won't be
> worth much if there is a huge memory leak in pysync or librsync...

I'm glad to see pysync's librsync wrapper being useful. 1/20th is a
significant difference!

The memory problem... 

It is quite possible that there is a memory leak either in librsync or the
Python wrapper. The Python wrapper is a first cut using swig, so it is
possible swig introduced the memory leak, or I did with a bad swig wrapper.
Librsync is not exactly clean code either, and I've already seen some dodgey
memory alloc/free stuff in there that can be cleaned up, though I'm not sure
it exactly leaks.

There are two things that need doing; improving librsync, and improving the
wrapper.

I've decided that I'm going to improve librsync a bit first. After that I'm
going to fix up the Python wrapper using the development versions of swig.
None of this is going to happen quickly unless someone decides to pay me for
it (my wife is hassling me to get a real job so that we can buy food).
However, because I'm a programming addict, there is a chance I'll slowly
starve to death after my wife leaves me as I code it up anyway :-)

-- 
----------------------------------------------------------------------
ABO: finger abo@minkirri.apana.org.au for more info, including pgp key
----------------------------------------------------------------------