Develop Reference

Perl compatible regular expression (PCRE) in Python

I have to parse some strings based on PCRE in Python, and I've no idea how to do that.
Strings I want to parse looks like:
match mysql m/^.\0\0\0\n(4\.[-.\w]+)\0...\0/s p/MySQL/ i/$1/
In this example, I have to get this different items:
"m/^.\0\0\0\n(4\.[-.\w]+)\0...\0/s" ; "p/MySQL/" ; "i/$1/"
The only thing I've found relating to PCRE manipulation in Python is this module: http://pydoc.org/2.2.3/pcre.html (but it's written it's a .so file ...)
Do you know if some Python module exists to parse this kind of string?

Be Especially Careful with non‐ASCII in Python
There are some really subtle issues with how Python deals with, or fails to deal with, non-ASCII in patterns and strings. Worse, these disparities vary substantially according, not just to which version of Python you are using, but also whether you have a “wide build”.
In general, when you’re doing Unicode stuff, Python 3 with a wide build works best and Python 2 with a narrow build works worst, but all combinations are still a pretty far cry far from how Perl regexes work vis‐à‐vis Unicode. If you’re looking for ᴘᴄʀᴇ patterns in Python, you may have to look a bit further afield than its old re module.
The vexing “wide-build” issues have finally been fixed once and for all — provided you use a sufficiently advanced release of Python. Here’s an excerpt from the v3.3 release notes:
Functionality
Changes introduced by PEP 393 are the following:
Python now always supports the full range of Unicode codepoints, including non-BMP ones (i.e. from U+0000 to U+10FFFF). The distinction between narrow and wide builds no longer exists and Python now behaves like a wide build, even under Windows.
With the death of narrow builds, the problems specific to narrow builds have also been fixed, for example:
len() now always returns 1 for non-BMP characters, so len('\U0010FFFF') == 1;
surrogate pairs are not recombined in string literals, so '\uDBFF\uDFFF' != '\U0010FFFF';
indexing or slicing non-BMP characters returns the expected value, so '\U0010FFFF'[0] now returns '\U0010FFFF' and not '\uDBFF';
all other functions in the standard library now correctly handle non-BMP codepoints.
The value of sys.maxunicode is now always 1114111 (0x10FFFF in hexadecimal). The PyUnicode_GetMax() function still returns either 0xFFFF or 0x10FFFF for backward compatibility, and it should not be used with the new Unicode API (see issue 13054).
The ./configure flag --with-wide-unicode has been removed.
The Future of Python Regexes
In contrast to what’s currently available in the standard Python distribution’s re library, Matthew Barnett’s regex module for both Python 2 and Python 3 alike is much, much better in pretty much all possible ways and will quite probably replace re eventually. Its particular relevance to your question is that his regex library is far more ᴘᴄʀᴇ (i.e. it’s much more Perl‐compatible) in every way than re now is, which will make porting Perl regexes to Python easier for you. Because it is a ground‐up rewrite (as in from‐scratch, not as in hamburger :), it was written with non-ASCII in mind, which re was not.
The regex library therefore much more closely follows the (current) recommendations of UTS#18: Unicode Regular Expressions in how it approaches things. It meets or exceeds the UTS#18 Level 1 requirements in most if not all regards, something you normally have to use the ICU regex library or Perl itself for — or if you are especially courageous, the new Java 7 update to its regexes, as that also conforms to the Level One requirements from UTS#18.
Beyond meeting those Level One requirements, which are all absolutely essential for basic Unicode support, but which are not met by Python’s current re library, the awesome regex library also meets the Level Two requirements for RL2.5 Named Characters (\N{...})), RL2.2 Extended Grapheme Clusters (\X), and the new RL2.7 on Full Properties from revision 14 of UTS#18.
Matthew’s regex module also does Unicode casefolding so that case insensitive matches work reliably on Unicode, which re does not.
The following is no longer true, because regex now supports full Unicode casefolding, like Perl and Ruby.
One super‐tiny difference is that for now, Perl’s case‐insensitive patterns use full string‐oriented casefolds while his regex module still uses simple single‐char‐oriented casefolds, but this is something he’s looking into. It’s actually a very hard problem, one which apart from Perl, only Ruby even attempts.
Under full casefolding, this means that (for example) "ß" now correct matches "SS", "ss", "ſſ", "ſs" (etc.) when case-insensitive matching is selected. (This is admittedly more important in the Greek script than the Latin one.)
See also the slides or doc source code from my third OSCON2011 talk entitled “Unicode Support Shootout: The Good, the Bad, and the (mostly) Ugly” for general issues in Unicode support across JavaScript, PHP, Go, Ruby, Python, Java, and Perl. If can’t use either Perl regexes or possibly the ICU regex library (which doesn’t have named captures, alas!), then Matthew’s regex for Python is probably your best shot.
Nᴏᴛᴀ Bᴇɴᴇ s.ᴠ.ᴘ. (= s’il vous plaît, et même s’il ne vous plaît pas :) The following unsolicited noncommercial nonadvertisement was not actually put here by the author of the Python regex library. :)
Cool regex Features
The Python regex library has a cornucopeia of superneat features, some of which are found in no other regex system anywhere. These make it very much worth checking out no matter whether you happen to be using it for its ᴘᴄʀᴇ‐ness or its stellar Unicode support.
A few of this module’s outstanding features of interest are:
Variable‐width lookbehind, a feature which is quite rare in regex engines and very frustrating not to have when you really want it. This may well be the most frequently requested feature in regexes.
Backwards searching so you don’t have to reverse your string yourself first.
Scoped ismx‐type options, so that (?i:foo) only casefolds for foo, not overall, or (?-i:foo) to turn it off just on foo. This is how Perl works (or can).
Fuzzy matching based on edit‐distance (which Udi Manber’s agrep and glimpse also have)
Implicit shortest‐to‐longest sorted named lists via \L<list> interpolation
Metacharacters that specifically match only the start or only the end of a word rather than either side (\m, \M)
Support for all Unicode line separators (Java can do this, as can Perl albeit somewhat begrudgingly with \R per RL1.6.
Full set operations — union, intersection, difference, and symmetric difference — on bracketed character classes per RL1.3, which is much easier than getting at it in Perl.
Allows for repeated capture groups like (\w+\s+)+ where you can get all separate matches of the first group not just its last match. (I believe C# might also do this.)
A more straightforward way to get at overlapping matches than sneaky capture groups in lookaheads.
Start and end positions for all groups for later slicing/substring operations, much like Perl’s #+ and #- arrays.
The branch‐reset operator via (?|...|...|...|) to reset group numbering in each branch the way it works in Perl.
Can be configured to have your coffee waiting for you in the morning.
Support for the more sophisticated word boundaries from RL2.3.
Assumes Unicode strings by default, and fully supports RL1.2a so that \w, \b, \s, and such work on Unicode.
Supports \X for graphemes.
Supports the \G continuation point assertion.
Works correctly for 64‐bit builds (re only has 32‐bit indices).
Supports multithreading.
Ok, that’s enough hype. :)
Yet Another Fine Alternate Regex Engine
One final alternative that is worth looking at if you are a regex geek is the Python library bindings to Russ Cox’s awesome RE2 library. It also supports Unicode natively, including simple char‐based casefolding, and unlike re it notably provides for both the Unicode General Category and the Unicode Script character properties, which are the two key properties you most often need for the simpler kinds of Unicode processing.
Although RE2 misses out on a few Unicode features like \N{...} named character support found in ICU, Perl, and Python, it has extremely serious computational advantages that make it the regex engine of choice whenever you’re concern with starvation‐based denial‐of‐service attacks through regexes in web queries and such. It manages this by forbidding backreferences, which cause a regex to stop being regular and risk super‐exponential explosions in time and space.
Library bindings for RE2 are available not just for C/C++ and Python, but also for Perl and most especially for Go, where it is slated to very shortly replace the standard regex library there.

You're looking for '(\w/[^/]+/\w*)'.
Used like so,
import re
x = re.compile('(\w/[^/]+/\w*)')
s = 'match mysql m/^.\0\0\0\n(4\.[-.\w]+)\0...\0/s p/MySQL/ i/$1/'
y = x.findall(s)
# y = ['m/^.\x00\x00\x00\n(4\\.[-.\\w]+)\x00...\x00/s', 'p/MySQL/', 'i/$1/']
Found it while playing with Edi Weitz's Regex Coach, so thanks to the comments to the question which made me remember its existence.

Since you want to run PCRE regexes, and Python's re module has diverged from its original PCRE origins, you may also want to check out Arkadiusz Wahlig's Python bindings for PCRE. That way you'll have access to native PCRE and won't need to translate between regex flavors.

Regular Expression GUI?

I've just started looking at regular expressions, and they are pretty cool. They are also pretty annoying looking, and I really don't want to 'learn' them if I can avoid it.
Which is why a nice gui would be great. I'm looking for something intuitive, where you can drag and drop 'condition boxes', select which conditions you want them to select for, get a list of things that your conditions go against, etc. Something which makes building regular expressions easy... heh
If anyone knows of anything, let me know!
Edit: Thanks for all the responses. After looking at some, I googled questions based on them and found this link: https://stackoverflow.com/questions/89718/is-there-anything-like-regexbuddy-in-the-open-source-world
Quickrex seems to have alot of the stuff I want (although not as good as some), plus its integrated into eclipse, which is the IDE I use at the moment.

Regexpal isn't a GUI, but it is handy for testing to see if you've built a regex that matches the correct stuff.
http://regexpal.com/
And at the bottom of regexpal.com is a link to regexbuddy, which seems to be closer to what you are looking for:
http://www.regexbuddy.com/

I use RegExRX, it is in the App Store (for MAC). It is very useful and the pulldown menus help you construct expressions. Their website is here: http://www.mactechnologies.com/index.php?page=downloads#regexrx
Nevertheless studying a bit about them will get you far. If you are on a Mac try for example "Searching with Grep" help section in TextWrangler.
=====
Addition: Of late I have being using http://regex101.com/. Really good.

Online Regular Expression Analyzer for Perl.
See also this collection of other Perl regex links: My Favourite Regex Tools

What you are looking for is RegexMagic - (From the creator of RegexBuddy)
(Although I would strongly recommend learning regex syntax - its not that hard and the time you spend will pay for itself many times over. See: regular-expressions.info)

There are a lot of GUIs for assisting you in writing regular expressions and testing them but there isn't one for writing regex for your (there are some tools with very limited scope). Asking a tool write regex for you is like asking for a tool to write Python code for you:)
A decent regex statements may get fairly complex but learning how to write them is not..

Although this is for Ruby and not python, I've found http://www.rubular.com/ to be quite good at regex testing.

I don't think there is such a program with "codition boxes" and stuff, but the regex coach is very useful for exploring regex.

If you're using a mac, the text editor Espresso from Macrabbit has really fantastic interactive regex abilities. It's not a GUI per se, but you can see results as you type.

There are regular expression testers available online, but probably no tools for generation. Usually, when you generate some code using a gui tool, you end up having much more redundant and inefficnent code than a human generated one. This is true in general: gui toolkit generation tool for scripting languages, excel-to-latex table conversion, ..., and probably for most other things, and probably the same will be true for regular expressions.
However, there are ways to reduce clumsiness in regular expressions. For example in Ruby, you can define regular expression as parts that you will repeatedly use e.g. /small_regex/, and refer to them within larger regular expressions e.g., /foo#{small_regex}bar/, Regexp.union(small_regex1, small_regex2), and so on.

One of the new players in the game makes it really easy to define tests and work in a red => green style with your regexs is http://refiddle.com. Right now it has runners for JavaScript, Ruby and .NET, bu more are on the way.

Checkout Rx Toolkit that comes with Komodo. But you'll need to type in the regular expression, some flags can be checked on/off. It's gui based.

Ruby Regex vs Python Regex

Are there any real differences between Ruby regex and Python regex?
I've been unable to find any differences in the two, but may have missed something.

The last time I checked, they differed substantially in their Unicode support. Ruby in 1.9 at least has some very limited Unicode support. I believe one or two Unicode properties might be supported by now. Probably the general categories and maybe the scripts were the two I'm thinking of.
Python has less and more Unicode support at the same time. Python does seem to make it possible to meet the requirements of RL1.2a "Compatability Properties" from UTS#18 on Unicode Regular Expressions.
That said, there is a really rather nice Python library out there by Matthew Barnett (mrab) that finally adds a couple of Unicode properties to Python regexes. He supports the two most important ones: the general categories, and the script properties. It has some other intriguing features as well. It deserves some good publicity.
I don't think either of Ruby or Python support Unicode all that terribly well, although more and more gets done every day. In particular, however, neither meets even the barebones Level 1 requirement for Unicode Regular Expressions cited above. For example, RL1.2 requires that at least 11 properties be supported: General_Category, Script, Alphabetic, Uppercase, Lowercase, White_Space, Noncharacter_Code_Point, Default_Ignorable_Code_Point, ANY, ASCII, and ASSIGNED.
I think Python only lets you get to some of those, and only in a roundabout way. Of course, there are many, many other properties beyond these 11.
When you’re looking for Unicode support, there's more than just UTS#10 on Regular Expressions of course, although that is the one that matters most to this question and neither Ruby nor Puython are Level 1 compliant. Other very important aspects of Unicode include UAX#15, UAX#14, UTS#18, UAX#11, UAX#29, and of course the crucial UAX#44. Python has libraries for at least a couple of those, I know. I don't know that they're standard.
But when it comes to regular expression support, um, there are richer alternatives than just those two, you know. :)

I like the /pattern/ syntax in Ruby, inspired from Perl, for regular expressions. Python's re.compile("pattern") is not really elegant for me. The syntatic sugar in Ruby and the fact that regular expressions are a separate re module in Python, makes me lean towards Ruby when it comes to Regular Expressions.
Apart from this, I don't see much of a difference from a normal Regular Expression programming perspective. Both the languages have pretty comprehensive and mostly similar RE support. There might be performance differences ( Python traditionally has has better performance ) and also Python has greater unicode regular expressions support.

If the question is only about regex's: neither. Use Perl.
You should choose between those languages based on the other non-regex issues that you are trying to solve and the community support in that language that is nearby your field of endeavor.
If you are truly only picking a language based on regex support -- choose Perl...

Ruby's Regexp#match method is equivalent to Python's re.search(), not re.match(). re.search() and Regexp#match look for the first match anywhere in a string. re.match() looks for a match only at the beginning of a string.
To perform the equivalent of re.match(), a Ruby regular expression will need to start with a ^, indicating matching the beginning of the string.
To perform the equivalent of Regexp#match, a Python regular expression will need to start with .*, indicating matching zero or more characters.

The regular expression libraries for Ruby and Python are developed by two completely independent teams. Even if they are identical now (and I wouldn't be certain they are), there's no guarantee that they won't diverge sometime in the future.
The safest position is to assume they're different now, and assume they will continue to be different in the future.

Listing all patterns that a regex matches

I am looking for a way to list all possible patterns from a finite regex (with no duplicates). Is there any source available?

Although it won't cover some advanced features, and has its own share of other caveats, Regexp::Genex seems to be close to what you are looking for.
There's also this thread of PerlMonks which is relevant enough (as well as explaining how Regexp::Genex might not do for you, and some roll-yourself alternatives).
Otherwise, as per Jeffrey Friedl's Mastering Regular Expressions, you could use the /g modifier, coupled with the (?{CODE}) extension and a pattern that will never match, ala:
perl -E '$_ = 'Mastering Regular Expressions'; /(\p{L}*)(?{ say qq![$^N]! })(?!)/g;'

A Haskell program based on Perl's Regexp::Genex can be found on Github and on Hackage.
According to the author, it was inspired by Regexp::Genex, but "uses a random-walk approach for character classes, instead of enumerating all possibilities."

programming language implemented in pure python

i am creating ( researching possibility of ) a highly customizable python client and would like to allow users to actually edit the code in another language to customize the running of program. ( analogous to browser which itself coded in c/c++ and run another language html/js ). so my question is , is there any programming language implemented in pure python which i can see as a reference ( or use directly ? ) -- i need simple language ( simple statements and ifs can do )
edit: sorry if i did not make myself clear but what i want is "a language to customize the running of program" , even though pypi seems a great option, what i am looking for is more simple which i can study and extend myself if need arise. my google searches pointing towards xml based langagues. ( BMEL , XForms etc ).

The question isn't completely clear on scope, but I have a hunch that PyPy, embedding other full languages, and similar solutions might be overkill. It sounds like iamgopal may really be interested in something more like Interpreter Pattern or Little Language.
If the language you want to support is really small (see the Interpreter Pattern link), then hand-coding this yourself in Python won't be too hard. You can write a simple parser (Google around; here's one example), then walk the AST and evaluate user expressions.
However, if you expect this to be used for a long time or by many people, it may be worth throwing a real language at the problem. (I'd recommend Python itself if your users are already familiar with basic Python syntax).

Ren'Py is a modification to Python syntax built on top of Python itself, using the language tools in the stdlib.

For your user's sake, don't use an XML based language - XML is an awful basis for a programming language and your users will hate you for it.
Here is a suggestion. Use a strict subset of Python for your language. Use the compiler module to convert their code into an abstract syntax tree and walk the tree to to validate that the code conforms to your subset before converting the AST into python bytecode.
N.B. I just checked the docs and see that the compiler package is deprecated in 2.6 and removed in Python 3.x. Does anyone know why that is?

Numerous template languages such as Cheetah, Django templates, Genshi, Mako, Mighty might serve as an example.

Why not Python itself? With some care you can use eval to run user code.
One of the good thing about interpreted scripting languages is that you don't need another extra scripting language!

PLY (Python Lex-Yacc)
is something of your interest.

Possibly Common Lisp (or any other Lisp) will be the best choice for that task. Because Lisp make it possible to easily extend host language with powerful macroses and construct DSL (domain specific language).

If all you need is simple if statements and expressions, I'm sure it wouldn't be an awful task to parse each line. Something like
if some flag
activate some feature
deactivate some feature
elif some other flag
activate some feature
activate some feature
else
logout
Just write a class which, while parsing takes the first word, checks if it's "if, elif, else," etc, and if so, check a flag and set a flag saying you either are or are not executing until the next conditional. If it's not a conditional, call a function based on the first keyword that would modify the program state in some way.
The class could store some local execution state (are we in an if statement? If so are we executing this branch?) and have another class containing some global application state (flags that are checkable by if statements, etc).
This is probably the wrong thing to do in your situation (it's very prone to bugs, it's dangerous if you don't treat the data in the scripts correctly), but it's at least a start if you do decide to interpret your own mini-language.
Seriously though, if you try this, be very, very, srs careful. Don't give the scripts any functionality that they don't definitely need, because you are almost certainly opening security holes by doing something like this.
Don't say I didn't warn you.