regexp support
Ben Escoto
bescoto@stanford.edu
Thu, 11 Apr 2002 22:55:54 -0700
--==_Exmh_-539183865P
Content-Type: multipart/mixed; boundary="----- =_aaaaaaaaaa0"
Content-Id: <16247.1018590941.0@folly.Stanford.EDU>
------- =_aaaaaaaaaa0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <16247.1018590941.1@folly.Stanford.EDU>
>>>>> "DB" == Donovan Baarda <abo@minkirri.apana.org.au>
>>>>> wrote the following on Fri, 12 Apr 2002 00:07:01 +1000
DB> I'm interested to know what changes you might have made to my
DB> efnmatch.py and dirscan.py. I really want to keep my stuff as
DB> common code if I can.
Well, there were a number of changes because the rest of the
rdiff-backup code expected a generator to yield the correct paths in
the form of DSRPath objects. So mainly I made some rdiff-backup
specific changes which wouldn't really be applicable elsewhere. I'm
attaching the relevant code (called "selection.py" in the
rdiff-backup-devel package).
DB> How do you test this? It would not be simple.
I think I just test if the pattern, if it were an include pattern,
would "scan" the root directory. (See attached code.)
DB> I really don't like --include and --exclude being matched
DB> against the full local path. I'd much prefer if the --include
DB> and --exclude expressions were taken as relative to the source
DB> path. This allows you to use the same --include --exclude
DB> options for backing up a partition that happens to be mounted at
DB> a different point. This is equivalent to an implicit
DB> os.path.join of the source and each --include --exclude pattern.
I saw three advantages to matching against the path including the
root:
1. Patterns as specified by --include should be somewhat compatible
with the format of the lines in a filelist, and rdiff-backup should
accept filelists generated by the find utility. find generates full
and not relative paths.
2. When running rdiff-backup from the shell, the files included or
excluded are the ones that the shell thinks you are referring to. For
instance
rdiff-backup --exclude foo/bar foo /backup
the file excluded will be the same the one listed if you just typed in
'ls foo/bar'. So you can use shell file completion or whatever to
find the correct files.
3. A user might get the syntax wrong whichever way it goes. If the
user types in:
rdiff-backup --exclude bar foo /backup
then, the way things are now, rdiff-backup will exit with error,
knowing that a mistake has been made. If rdiff-backup used relative
patterns then the analogous mistake would be for a user to type in
rdiff-backup --exclude foo/bar foo /backup
Here, however, rdiff-backup cannot check for errors, because the
--exclude option is valid (it excludes the foo/foo/bar directory). So
by making the user include the root in the pattern we prevent the user
from making a certain kind of natural mistake.
About using the same options to back up a partition mounted at
two different points, it is possible by cd'ing to the directory first.
For instance:
cd $MOUNT_POINT
rdiff-backup --include ./whatever/**bar --exclude ./foo . /backup
would work as long as $MOUNT_POINT were updated.
--
Ben Escoto
------- =_aaaaaaaaaa0
Content-Type: text/python; charset="us-ascii"
Content-ID: <16247.1018590941.2@folly.Stanford.EDU>
Content-Description: selection.py
Content-Transfer-Encoding: quoted-printable
from __future__ import generators
execfile("destructive_stepping.py")
import re
#######################################################################
#
# selection - Provides the iterator-like DSRPIterator class
#
# Parses includes and excludes to yield correct files. More
# documentation on what this code does can be found on the man page.
#
class SelectError(Exception):
"""Some error dealing with the Select class"""
pass
class FilePrefixError(SelectError):
"""Signals that a specified file doesn't start with correct prefix"""
pass
class GlobbingError(SelectError):
"""Something has gone wrong when parsing a glob string"""
pass
class Select:
"""Iterate appropriate DSRPaths in given directory
This class acts as an iterator on account of its next() method.
Basically, it just goes through all the files in a directory in
order (depth-first) and subjects each file to a bunch of tests
(selection functions) in order. The first test that includes or
excludes the file means that the file gets included (iterated) or
excluded. The default is include, so with no tests we would just
iterate all the files in the directory in order.
The one complication to this is that sometimes we don't know
whether or not to include a directory until we examine its
contents. For instance, if we want to include all the **.py
files. If /home/ben/foo.py exists, we should also include /home
and /home/ben, but if these directories contain no **.py files,
they shouldn't be included. For this reason, a test may not
include or exclude a directory, but merely "scan" it. If later a
file in the directory gets included, so does the directory.
As mentioned above, each test takes the form of a selection
function. The selection function takes a dsrp, and returns:
None - means the test has nothing to say about the related file
0 - the file is excluded by the test
1 - the file is included
2 - the test says the file (must be directory) should be scanned
Also, a selection function f has a variable f.exclude which should
be true iff f could potentially exclude some file. This is used
to signal an error if the last function only includes, which would
be redundant and presumably isn't what the user intends.
"""
# This re should not match normal filenames, but usually just globs
glob_re =3D re.compile("(.*[*?[]|ignorecase\\:)", re.I | re.S)
def __init__(self, rpath, source):
"""DSRPIterator initializer.
rpath is the root dir. Source is true if rpath is the root of
the source directory, and false for the mirror directory
"""
assert isinstance(rpath, RPath)
self.selection_functions =3D []
self.source =3D source
if isinstance(rpath, DSRPath): self.dsrpath =3D rpath
else: self.dsrpath =3D DSRPath(rpath.conn, rpath.base,
rpath.index, rpath.data)
self.prefix =3D self.dsrpath.path
=
def set_iter(self, starting_index =3D None, sel_func =3D None):
"""Initialize more variables, get ready to iterate
Will iterate indicies greater than starting_index. Selection
function sel_func is called on each dsrp and is usually
self.Select. Returns self just for convenience.
"""
if not sel_func: sel_func =3D self.Select
self.dsrpath.setdata() # this may have changed since Select init
if starting_index is not None:
self.starting_index =3D starting_index
self.iter =3D self.iterate_starting_from(self.dsrpath,
self.iterate_starting_from, sel_func)
else: self.iter =3D self.Iterate(self.dsrpath, self.Iterate, sel_func)
self.next =3D self.iter.next
self.__iter__ =3D lambda: self
return self
def Iterate(self, dsrpath, rec_func, sel_func):
"""Return iterator yielding dsrps in dsrpath
rec_func is usually the same as this function and is what
Iterate uses to find files in subdirectories. It is used in
iterate_starting_from.
sel_func is the selection function to use on the dsrps. It is
usually self.Select.
"""
s =3D sel_func(dsrpath)
if not s or DestructiveStepping.initialize(dsrpath, self.source):
return
if s =3D=3D 1: # File is included
yield dsrpath
if dsrpath.isdir():
for dsrp in self.iterate_in_dir(dsrpath, rec_func, sel_func):
yield dsrp
elif s =3D=3D 2 and dsrpath.isdir(): # Directory is merely scanned
iid =3D self.iterate_in_dir(dsrpath, rec_func, sel_func)
try: first =3D iid.next()
except StopIteration: return # no files inside; skip dsrp
yield dsrpath
yield first
for dsrp in iid: yield dsrp
def iterate_in_dir(self, dsrpath, rec_func, sel_func):
"""Iterate the dsrps in directory dsrpath."""
dir_listing =3D dsrpath.listdir()
dir_listing.sort()
for filename in dir_listing:
for dsrp in rec_func(dsrpath.append(filename), rec_func, sel_func):
yield dsrp
def iterate_starting_from(self, dsrpath, rec_func, sel_func):
"""Like Iterate, but only yield indicies > self.starting_index"""
if DestructiveStepping.initialize(dsrpath, self.source): return
if dsrpath.index > self.starting_index: # past starting_index
for dsrp in self.Iterate(dsrpath, self.Iterate, sel_func):
yield dsrp
elif dsrpath.index =3D=3D self.starting_index[:len(dsrpath.index)]:
# May encounter starting index on this branch
for dsrp in self.iterate_in_dir(dsrpath,
self.iterate_starting_from,
sel_func): yield dsrp
def iterate_with_finalizer(self):
"""Like Iterate, but missing some options, and add finalizer"""
finalize =3D DestructiveStepping.Finalizer()
for dsrp in self:
yield dsrp
finalize(dsrp)
finalize.getresult()
def Select(self, dsrp):
"""Run through the selection functions and return dominant value"""
for sf in self.selection_functions:
result =3D sf(dsrp)
if result is not None: return result
return 1
def ParseArgs(self, argtuples):
"""Create selection functions based on list of tuples
The tuples have the form (option string, additional argument)
and are created when the initial commandline arguments are
read. The reason for the extra level of processing is that
the filelists may only be openable by the main connection, but
the selection functions need to be on the backup reader or
writer side. When the initial arguments are parsed the right
information is sent over the link.
"""
try:
for opt, arg in argtuples:
if opt =3D=3D "--exclude":
self.add_selection_func(self.glob_get_sf(arg, 0))
elif opt =3D=3D "--exclude-device-files":
self.add_selection_func(self.devfiles_get_sf())
elif opt =3D=3D "--exclude-filelist":
self.add_selection_func(self.filelist_get_sf(arg[1],
0, arg[0]))
elif opt =3D=3D "--exclude-regexp":
self.add_selection_func(self.regexp_get_sf(arg, 0))
elif opt =3D=3D "--include":
self.add_selection_func(self.glob_get_sf(arg, 1))
elif opt =3D=3D "--include-filelist":
self.add_selection_func(self.filelist_get_sf(arg[1],
1, arg[0]))
elif opt =3D=3D "--include-regexp":
self.add_selection_func(self.regexp_get_sf(arg, 1))
else: assert 0, "Bad option %s" % opt
except SelectError, e: self.parse_catch_error(e)
self.parse_last_excludes()
self.parse_rbdir_exclude()
self.parse_proc_exclude()
def parse_catch_error(self, exc):
"""Deal with selection error exc"""
if isinstance(exc, FilePrefixError):
Log.FatalError(
"""Fatal Error: The file specification
%s
cannot match any files in the base directory
%s
Useful file specifications begin with the base directory or some
pattern (such as '**') which matches the base directory.""" %
(exc, self.prefix))
elif isinstance(e, GlobbingError):
Log.FatalError("Fatal Error while processing expression\n"
"%s" % exc)
else: raise
def parse_rbdir_exclude(self):
"""Add exclusion of rdiff-backup-data dir to front of list"""
self.add_selection_func(
self.glob_get_tuple_sf(("rdiff-backup-data",), 0), 1)
def parse_proc_exclude(self):
"""Exclude the /proc directory if starting from /"""
if self.prefix =3D=3D "/":
self.add_selection_func(self.glob_get_tuple_sf(("proc",), 0), 1)
def parse_last_excludes(self):
"""Exit with error if last selection function isn't an exclude"""
if (self.selection_functions and
not self.selection_functions[-1].exclude):
Log.FatalError(
"""Last selection expression:
%s
only specifies that files be included. Because the default is to
include all files, the expression is redundant. Exiting because this
probably isn't what you meant.""" %
(self.selection_functions[-1].name, self.prefix))
def add_selection_func(self, sel_func, add_to_start =3D None):
"""Add another selection function at the end or beginning"""
if add_to_start: self.selection_functions.insert(0, sel_func)
else: self.selection_functions.append(sel_func)
def filelist_get_sf(self, filelist_fp, inc_default, filelist_name):
"""Return selection function by reading list of files
The format of the filelist is documented in the man page.
filelist_fp should be an (open) file object.
inc_default should be true if this is an include list,
false for an exclude list.
filelist_name is just a string used for logging.
"""
Log("Reading filelist %s" % filelist_name, 4)
tuple_list, something_excluded =3D \
self.filelist_read(filelist_fp, inc_default, filelist_name)
Log("Sorting filelist %s" % filelist_name, 4)
tuple_list.sort()
i =3D [0] # We have to put index in list because of stupid scoping rules
def selection_function(dsrp):
if i[0] > len(tuple_list): return inc_default
while 1:
include, move_on =3D \
self.filelist_pair_match(dsrp, tuple_list[i[0]])
if move_on:
i[0] +=3D 1
if include is None: continue # later line may match
return include
selection_function.exclude =3D something_excluded
selection_function.name =3D "Filelist: " + filelist_name
return selection_function
def filelist_read(self, filelist_fp, include, filelist_name):
"""Read filelist from fp, return (tuplelist, something_excluded)"""
something_excluded, tuple_list =3D None, []
prefix_warnings =3D 0
for line in filelist_fp:
if not line.strip(): continue # skip blanks
try: tuple =3D self.filelist_parse_line(line, include)
except FilePrefixError, exp:
prefix_warnings +=3D 1
if prefix_warnings < 6:
Log("Warning: file specification %s in filelist %s\n"
"doesn't start with correct prefix %s, ignoring." %
(exp, filelist_name, self.prefix), 2)
if prefix_warnings =3D=3D 5:
Log("Future prefix errors will not be logged.", 2)
tuple_list.append(tuple)
if not tuple[1]: something_excluded =3D 1
if filelist_fp.close():
Log("Error closing filelist %s" % filelist_name, 2)
return (tuple_list, something_excluded)
def filelist_parse_line(self, line, include):
"""Parse a single line of a filelist, returning a pair
pair will be of form (index, include), where index is another
tuple, and include is 1 if the line specifies that we are
including a file. The default is given as an argument.
prefix is the string that the index is relative to.
"""
line =3D line.strip()
if line[:2] =3D=3D "+ ": # Check for "+ "/"- " syntax
include =3D 1
line =3D line[2:]
elif line[:2] =3D=3D "- ":
include =3D 0
line =3D line[2:]
if not line.startswith(self.prefix): raise FilePrefixError(line)
line =3D line[len(self.prefix):] # Discard prefix
index =3D tuple(filter(lambda x: x, line.split("/"))) # remove empties
return (index, include)
def filelist_pair_match(self, dsrp, pair):
"""Matches a filelist tuple against a dsrp
Returns a pair (include, move_on, definitive). include is
None if the tuple doesn't match either way, and 0/1 if the
tuple excludes or includes the dsrp.
move_on is true if the tuple cannot match a later index, and
so we should move on to the next tuple in the index.
"""
index, include =3D pair
if include =3D=3D 1:
if index < dsrp.index: return (None, 1)
if index =3D=3D dsrp.index: return (1, 1)
elif index[:len(dsrp.index)] =3D=3D dsrp.index:
return (1, None) # /foo/bar implicitly includes /foo
else: return (None, None) # dsrp greater, not initial sequence
elif include =3D=3D 0:
if dsrp.index[:len(index)] =3D=3D index:
return (0, None) # /foo implicitly excludes /foo/bar
elif index < dsrp.index: return (None, 1)
else: return (None, None) # dsrp greater, not initial sequence
else: assert 0, "Include is %s, should be 0 or 1" % (include,)
def regexp_get_sf(self, regexp_string, include):
"""Return selection function given by regexp_string"""
assert include =3D=3D 0 or include =3D=3D 1
try: regexp =3D re.compile(regexp_string)
except:
Log("Error compiling regular expression %s" % regexp_string, 1)
raise
=
def sel_func(dsrp):
if regexp.search(dsrp.path): return include
else: return None
sel_func.exclude =3D not include
sel_func.name =3D "Regular expression: %s" % regexp_string
return sel_func
def devfiles_get_sf(self):
"""Return a selection function to exclude all dev files"""
if self.selection_functions:
Log("Warning: exclude-device-files is not the first "
"selector.\nThis may not be what you intended", 3)
def sel_func(dsrp):
if dsrp.isdev(): return 0
else: return None
sel_func.exclude =3D 1
sel_func.name =3D "Exclude device files"
return sel_func
def glob_get_sf(self, glob_str, include):
"""Return selection function given by glob string"""
assert include =3D=3D 0 or include =3D=3D 1
if glob_str =3D=3D "**": sel_func =3D lambda dsrp: include
elif not self.glob_re.match(glob_str): # normal file
sel_func =3D self.glob_get_filename_sf(glob_str, include)
else: sel_func =3D self.glob_get_normal_sf(glob_str, include)
sel_func.exclude =3D not include
sel_func.name =3D "Command-line glob: %s" % glob_str
return sel_func
def glob_get_filename_sf(self, filename, include):
"""Get a selection function given a normal filename
Some of the parsing is better explained in
filelist_parse_line. The reason this is split from normal
globbing is things are a lot less complicated if no special
globbing characters are used.
"""
if not filename.startswith(self.prefix):
raise FilePrefixError(filename)
index =3D tuple(filter(lambda x: x,
filename[len(self.prefix):].split("/")))
return self.glob_get_tuple_sf(index, include)
def glob_get_tuple_sf(self, tuple, include):
"""Return selection function based on tuple"""
def include_sel_func(dsrp):
if (dsrp.index =3D=3D tuple[:len(dsrp.index)] or
dsrp.index[:len(tuple)] =3D=3D tuple):
return 1 # /foo/bar implicitly matches /foo, vice-versa
else: return None
def exclude_sel_func(dsrp):
if dsrp.index[:len(tuple)] =3D=3D tuple:
return 0 # /foo excludes /foo/bar, not vice-versa
else: return None
if include =3D=3D 1: sel_func =3D include_sel_func
elif include =3D=3D 0: sel_func =3D exclude_sel_func
sel_func.exclude =3D not include
sel_func.name =3D "Tuple select %s" % (tuple,)
return sel_func
def glob_get_normal_sf(self, glob_str, include):
"""Return selection function based on glob_str
The basic idea is to turn glob_str into a regular expression,
and just use the normal regular expression. There is a
complication because the selection function should return '2'
(scan) for directories which may contain a file which matches
the glob_str. So we break up the glob string into parts, and
any file which matches an initial sequence of glob parts gets
scanned.
Thanks to Donovan Baarda who provided some code which did some
things similar to this.
"""
if glob_str.lower().startswith("ignorecase:"):
re_comp =3D lambda r: re.compile(r, re.I | re.S)
glob_str =3D glob_str[len("ignorecase:"):]
else: re_comp =3D lambda r: re.compile(r, re.S)
# matches what glob matches and any files in directory
glob_comp_re =3D re_comp("^%s($|/)" % self.glob_to_re(glob_str))
if glob_str.find("**") !=3D -1:
glob_str =3D glob_str[:glob_str.find("**")+2] # truncate after **
scan_comp_re =3D re_comp("^(%s)$" %
"|".join(self.glob_get_prefix_res(glob_str)))
def include_sel_func(dsrp):
if glob_comp_re.match(dsrp.path): return 1
elif scan_comp_re.match(dsrp.path): return 2
else: return None
def exclude_sel_func(dsrp):
if glob_comp_re.match(dsrp.path): return 0
else: return None
# Check to make sure prefix is ok
if not include_sel_func(self.dsrpath): raise FilePrefixError(glob_str)
=
if include: return include_sel_func
else: return exclude_sel_func
def glob_get_prefix_res(self, glob_str):
"""Return list of regexps equivalent to prefixes of glob_str"""
glob_parts =3D glob_str.split("/")
if "" in glob_parts[1:-1]: # "" OK if comes first or last, as in /foo/
raise GlobbingError("Consecutive '/'s found in globbing string "
+ glob_str)
prefixes =3D map(lambda i: "/".join(glob_parts[:i+1]),
range(len(glob_parts)))
# we must make exception for root "/", only dir to end in slash
if prefixes[0] =3D=3D "": prefixes[0] =3D "/"
return map(self.glob_to_re, prefixes)
def glob_to_re(self, pat):
"""Returned regular expression equivalent to shell glob pat
Currently only the ?, *, [], and ** expressions are supported.
Ranges like [a-z] are also currently unsupported. There is no
way to quote these special characters.
This function taken with minor modifications from efnmatch.py
by Donovan Baarda.
"""
i, n, res =3D 0, len(pat), ''
while i < n:
c, s =3D pat[i], pat[i:i+2]
i =3D i+1
if s =3D=3D '**':
res =3D res + '.*'
i =3D i + 1
elif c =3D=3D '*': res =3D res + '[^/]*'
elif c =3D=3D '?': res =3D res + '[^/]'
elif c =3D=3D '[':
j =3D i
if j < n and pat[j] in '!^': j =3D j+1
if j < n and pat[j] =3D=3D ']': j =3D j+1
while j < n and pat[j] !=3D ']': j =3D j+1
if j >=3D n: res =3D res + '\\[' # interpret the [ literally
else: # Deal with inside of [..]
stuff =3D pat[i:j].replace('\\','\\\\')
i =3D j+1
if stuff[0] in '!^': stuff =3D '^' + stuff[1:]
res =3D res + '[' + stuff + ']'
else: res =3D res + re.escape(c)
return res
------- =_aaaaaaaaaa0--
--==_Exmh_-539183865P
Content-Type: application/pgp-signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: Exmh version 2.5 01/15/2001
iD8DBQE8tnbm+owuOvknOnURApVAAJ9u9HJaus34JaNvApH/ksvA9MvuTACfe9/e
V8mbT0HYUim4nxYBZrWa4MA=
=RcWO
-----END PGP SIGNATURE-----
--==_Exmh_-539183865P--