[rproxy-devel] Re: bottlenecks

[Shirish H. Phatak]

Hi,

   I have run the librsync compiled binaries through a bunch of 
different memory checkers already. There is a buffer leak in rs_delta_*, 
which my last patch should take care of. Other than that I haver not 
found anything.

-Shirish

Donovan Baarda wrote:

>I've Cc'd this to the rproxy-devel list for reference.
>
>On Tue, Jun 18, 2002 at 03:07:13AM -0700, Ben Escoto wrote:
>  
>
>>>>>>>"DG" == dean gaudet <dean-list-rdiff-backup@arctic.org>
>>>>>>>wrote the following on Mon, 17 Jun 2002 12:43:24 -0700 (PDT)
>>>>>>>              
>>>>>>>
>[...]
>  
>
>>The internal librsync procedure used up only about 1/20th of the CPU
>>time as the external rdiff procedure, meaning I suppose that 95% of
>>the CPU time was spent forking, setting up the external rdiff process,
>>etc.  Of course this is for small files; as the file sizes became
>>large, the forking time would go to 0 as a percentage of the total.
>>
>>    Oh, and the reason that test3.py had such a large real time is
>>because it ended up using hundreds of MB of memory and my system
>>started swapping like crazy.  I guess all this efficiency won't be
>>worth much if there is a huge memory leak in pysync or librsync...
>>    
>>
>
>I'm glad to see pysync's librsync wrapper being useful. 1/20th is a
>significant difference!
>
>The memory problem... 
>
>It is quite possible that there is a memory leak either in librsync or the
>Python wrapper. The Python wrapper is a first cut using swig, so it is
>possible swig introduced the memory leak, or I did with a bad swig wrapper.
>Librsync is not exactly clean code either, and I've already seen some dodgey
>memory alloc/free stuff in there that can be cleaned up, though I'm not sure
>it exactly leaks.
>
>There are two things that need doing; improving librsync, and improving the
>wrapper.
>
>I've decided that I'm going to improve librsync a bit first. After that I'm
>going to fix up the Python wrapper using the development versions of swig.
>None of this is going to happen quickly unless someone decides to pay me for
>it (my wife is hassling me to get a real job so that we can buy food).
>However, because I'm a programming addict, there is a chance I'll slowly
>starve to death after my wife leaves me as I code it up anyway :-)
>
>  
>