Adding a new feature in the python syntax base - python

I want to add a new functionality in python, purely for experimental purposes where I would like to extend the decorator syntax. Currently decorators can be applied on functions and classes.
I would like to also use decorators on loops (for loop for example) and also to blocks of code.
Example 1:
#foo
for i in range(20):
# do something
# and something more
Example 2:
#foo
# there's a block starting from an indent here.
# there's some code now
# do something
# and something more
Now although this is the basic idea, my requirement is to modify the body of the decorator.
For example, I want to change the loop a bit based on the decorator applied to it. I can use the AST module for this.
The problem is I do not want to completely add a new syntax and its complete implementation. I just want to parse with the new syntax, access the parse tree and the decorator's body, operate on it and insert that into the body of the program, removing the decorator thus changing the program which had a new syntax to a syntax that python has right now.
Any idea on how I would go about doing this?

You can't do that without adding new syntax. Decorators themselves don't have a "body" as such. Decorators can apply to functions or to classes and that's it. See near the top of http://docs.python.org/2/reference/grammar.html :
decorated: decorators (classdef | funcdef)
If you want something else, it can't be a decorator, it has to be your own syntax that looks like a decorator.
You could write some kind of preprocessor that parses your syntax and transforms into valid Python. One possibility is the parser module. It has facilities for parsing basic Python elements like suites (i.e., blocks). You can see a simple example in the documentation. The ast module also provides this functionality. But these modules don't provide a way to parse decorators independently of class/function defs; the decorators are essentially considered as part of the class/function defition.
Even if you manage to parse your particular construct, you will probably have to do substantial trickery to create the AST. The problem is that you can't just "access the parse tree" and "modify the AST" because the program as you've written won't have a normal Python parse tree since it can't be parsed as valid Python. So you'll have to try to stitch together your own AST by patching together your custom code with ordinary Python-parsed code.

Related

How to add code snippets to python docstring (not as doctest)?

I would like to provide code snippets to show-case how one could use a particular method or class in python. How can I do that?
In Java one could use <pre> ... </pre> to do so.
Doctest is the only way?
As I look at the existing docstrings for typical packages (e.g., pandas, numpy, etc), I never see anything other than doctest which is intended to test the method and not just to format text as python code. So, if doctest is the only way, what would be the proper way of formatting a snippet of code to look like interactive python sessions? I don't want to write my code in an interactive session each time and then c+p it in my docstring. It doesn't seem to be right.
DocStrings are not the only way. If you are using Sphinx to generate documentation then you should read this.
Example
Here is something I want to talk about:: <-- Special syntax to mark code beginning
def my_fn(foo, bar=True): # Code itself
"""A really useful function.
Returns None
"""

Regular expression to match python method

Is there a python regex which will generically match a method definition (not just the declaration but also the method body) inside a python code file?
I did my share of googling but only found something similar for Java. Python is different w.r.t. to the way scopes are entered through indentation rather than accolades. What makes this problem hard is the fact that indentation may drop inside the method body (i.e. blank lines, multiline strings, comments).
I also looked for DOM parsers but basically they're all aimed at XML or HTML.
Finally I am looking into introspection (How can I get the source code of a Python function?) but I still wonder if there is a nicer way for code analysis without execution.
EDIT: the question receives a bunch of downvotes but I think it's actually a valid and specific programming question. I elaborated the question a bit.
Err, you don't want to use regexes to parse Python. The 'nicer way for code analysis without execution' is to use the Python standard library parser and/or ast modules. Look under the heading Python Language Services, e.g. https://docs.python.org/2/library/language.html

How to store a python function within an XML file

I'm developing a GUI application in Python that stores it's documents in an XML based format. The application is a mathematical model which several pre-defined components which can be drag-and-dropped. I'd also like the user to be able to create custom components by writing a python function inside an editor provided within the application. My issue is with storing these functions in the XML.
A function might look something like this:
def func(node, timestamp):
return node.weight * timestamp.day + 4
These functions are wrapped in an object which provides a standard way of calling them (compared to the pre-defined components). If I was to create one from Python directly it would look like this:
parameter = ParameterFunction(func)
The function is then called by the model like this:
parameter.value(node=node, timestamp=timestamp)
The ParameterFunction object has a to_xml and from_xml functions which need to serialise/deserialise the object to/from an XML representation.
My question is: how do I store the Python functions in an XML document?
One solution I have thought of so far is to store the function definition as a string, eval() or exec() it for use but keep the string, then store the string in a CDATA block in the XML. Are there any issues with this that I'm not seeing?
An alternative would be to store all of the Python code in a separate file, and have the XML reference just the function names. This could be nice as it could be edited easily in an external editor. In which case what is the best way to import the code? I am envisiging fighting with the python import path...
I'm aware there are will be security concerns with running untrusted code, but I'm willing to make this tradeoff for the freedom it gives users.
The specific application I'm referring to is on github. I'm happy to provide more information if it's needed, but I've tried to keep it fairly generic here. https://github.com/snorfalorpagus/pywr/blob/120928eaacb9206701ceb9bc91a5d73740db1953/pywr/core.py#L396-L402
Nope, you have the easiest and best solution that I can think of. Just keep them as strings, as long as your not worried about running the untrusted code.
The way I'd deal with external python scripts containing tiny snippets like yours would be to treat them as plain text files and read them in as strings. This avoids all the problems with importing them. Just read them in and call exec on them, then the functions will exist in scope.
EDIT: I was going to add something on sandboxing python code, but after a bit of research it seems this will not be an easy task, it would be easier to sandbox the entire program. Another longer and harder way to restrict the untrusted code would be to create your own tiny interpreter that only did safe operations (i.e mathematical operations, calling existing functions, etc..)

Watching which function is called in Python

What is the easiest way to record function calls for debugging in Python? I'm usually interested in particular functions or all functions from a given class. Or sometimes even all functions called on a particular object attribute. Seeing the call arguments would be useful, too.
I can imagine writing decorators for all that, but then I'd still have to modify the source code in different places. And writing a class decorator which modifies all methods isn't that straightforward.
Is there a solution where I don't have to modify my source code? Ideally something which doesn't slow down Python too much.
You ought to be able to implement something that does what you want using either sys.setprofile() or perhaps sys.settrace(). They both let you define a function to be called when specific "events" occur, like function calls, and pass additional information to which can be used to to determine the function/method being called and examine its arguments.
If you look around, there's probably sample usage code to use as a good starting point.
Except decorators, for Python >= 3.0 you could use new __getattribute__ method for a class, which will be called every time you call any method of the object.
You could look through Lutz "Learning Python" chapters 31, 37 about it.

Is there a way to make a user-defined Python function act like a built-in statement?

Is it possible to make a user-defined Python function act like a statement? In other words, I'd like to be able to say:
myfunc
rather than:
myfunc()
and have it get called anyway -- the way that, say, print would.
I can already hear you all composing responses about how this is a horrible thing to do and I'm stupid for asking it and why do I want to do this and I really should do something else instead, but please take my word that it's something I need to do to debug a problem I'm having and it's not going to be checked in or used for, like, an air traffic control system.
No, it is not possible.
As you can see from the Language Reference, there is no room left for extensions of the list of simple statements in the specification.
Moreover, print as a statement no longer exists in Python 3.0 and is replaced by the print() builtin function.
If what you're looking for is to add a new statement (like print) to Python's language, then this would not be easy. You'd probably have to modify lexer, parser and then recompile Python's C sources. A lot of work to do for a questionable convenience.
I would not implement this, but if I was implementing this, I would give code with myfunc a special extension, write an import hook to parse the file, add the parenthesis to make it valid Python, and feed that into the interpreter.
Not if you want to pass in arguments. You could do something build an object that ABUSES the __str__ method, but it is highly not recommended. You can also use other operators like overload the << operator like cout does in C++.
In Python 2.x print is not a function it is a statement just as if, while and def are statements.
Not possible in a planned way, or without a lot of work.
If you are bold and adventurous, read this wikipedia article about meta circular evaluation. Python has pretty good inspection and reflection on its own compiler/evaluater objects, you may be able to cobble something together along these lines.
"""Meta-circular implementations are suited to extending the language
they are written in. They are also useful for writing tools that are
tightly integrated with the programming language, such as
sophisticated debuggers. A language designed with a meta-circular
implementation in mind is often more suited for building languages in
general, even ones completely different from the host language."""
http://en.wikipedia.org/wiki/Meta-circular_evaluator
I believe pypy is doing something similarily, you might want to look into it.
http://pypy.org
This probably isn't going to cover your problem, but I'll mention it anyway. If myfunc is part of a module, and you are using it like this:
from mymodule import myfunc
myfunc # I want this to turn into a function call
Then you could instead do this:
import mymodule
mymodule.myfunc # I want this to turn into a function call
You could then remove myfunc from mymodule and overload the module so it calls a particular function each time the myfunc member is requested.

Categories

Resources