Develop Reference

How to override python file lookup for traceback printing?

I've developed a template engine (https://github.com/hl037/tempiny) that translate a file to python code. Then I call exec().
However, when there are errors in the template, the traceback either print no information (if I don't give a filename to compile) or prints the wrong line.
Thus, I would like to change the way the file is retrieved, to use my in-ram version. (by the way, I also can change the file name).
I've checked this question : File "<string>" traceback with line preview
...But the answer does not work with the built-in traceback. (and that means that you have to hack globally sys.excepthook, which could cause conflicts with other modules)
I could also use a temporary file, but it would results in some overhead just for getting info only when something wrong happens...
Do you have a solution?

Exception while using Sphinx to document script with command line arguments

Is there any way I can use Sphinx to document the command line inputs of a python script? I am able to document inputs of functions or methods but I don't know how to document inputs of scripts. I have been trying to follow the same syntax I use for functions by adding in the source file the line .. automodule:: scriptLDOnServer where scriptLDOnServer is my python script (which corresponds to my main).
The problem is that I get an error like this:
__import__(self.modname)
File "/home/ubuntu/SVNBioinfo/trunk/Code/LD/scriptLDOnServer.py", line 10, in <module>
genotype_filename=sys.argv[7];
IndexError: list index out of range
It seems that Sphinx is trying to get the command line inputs but in my source file there are no inputs so the import fails. Is there a way to solve this? Should I use another command in the source for a script instead of a module?
Sorry for not being very clear but it is difficult to explain the problem.

It is failing while you are importing the file because you are trying to access sys.argv[7] at import time. This is not something you should be doing. Such code which can possibly fail should be either in an if __name__ == '__main__': block or in a function, so that it won't be executed when the code is imported.
This holds generally. You should never* write code which will fail to import or which has any side-effects of being imported.
* Conditions apply. But you'd better have pretty good justification before breaking this rule.

Running Another program from python

I want to call a program multiple times from a python code, and save the output of that program in a text file. My first problem right now is just calling the other code. I have to redirect to a different directory and call ./rank on output.txt. This is how Im trying to do it:
TheCommand = "~/src/rank-8-9-2011/rank output.txt"
os.system(TheCommand)
but im getting a parsing error.
[Parsing error on line ]Unknown error: 0
Im running python2.7 on Mac OS 10.5.8. Im not sure what the problem is. I also tried using subprocess:
subprocess.call(["~/src/rank-8-9-2011/rank", "output.txt"])
This does not find the directory (I have a feeling Im using the subprocess incorrectly), but I dont know what is wrong with the os.system.

the name of the program in the first argument to subprocess.Popen must not contain ~ as it doesn't pass the string to the shell for processing (which like always using parameterized queries in sql, protects one from string injection attacks, e.g. if instead of output.text one had ;rm -rf /, the system version would run rank and then run rm -rf . but the subprocess.Popen would only have rank open a file named ;rm -rf .), so one should expand it by calling os.path.expanduser:
subprocess.Popen([os.path.expanduser('~/src/rank-8-9-2011/rank'), "output.txt"])
although it is possible to turn shell processing on by passing shell=True, it is not recommended for the aforementioned reason.

you should try http://docs.python.org/library/os.path.html#os.path.expanduser
import os.path
subprocess.call([os.path.expanduser("~/src/rank-8-9-2011/rank"), "output.txt"])

I'm fairly certain your parsing error is coming from rank, not from your os.system command, as nothing there looks weird. What happens if you run rank manually?
subprocess seems to have a problem with '~', although I'm not immediately sure why. Put the full path and it should work (although you'll likely get that parsing error if it is indeed a problem with rank).

Getting python -m module to work for a module implemented in C

I have a pure C module for Python and I'd like to be able to invoke it using the python -m modulename approach. This works fine with modules implemented in Python and one obvious workaround is to add an extra file for that purpose. However I really want to keep things to my one single distributed binary and not add a second file just for this workaround.
I don't care how hacky the solution is.
If you do try to use a C module with -m then you get an error message No code object available for <modulename>.

-m implementation is in runpy._run_module_as_main . Its essence is:
mod_name, loader, code, fname = _get_module_details(mod_name)
<...>
exec code in run_globals
A compiled module has no "code object" accociated with it so the 1st statement fails with ImportError("No code object available for <module>"). You need to extend runpy - specifically, _get_module_details - to make it work for a compiled module. I suggest returning a code object constructed from the aforementioned "import mod; mod.main()":
(python 2.6.1)
code = loader.get_code(mod_name)
if code is None:
+ if loader.etc[2]==imp.C_EXTENSION:
+ code=compile("import %(mod)s; %(mod)s.main()"%{'mod':mod_name},"<extension loader wrapper>","exec")
+ else:
+ raise ImportError("No code object available for %s" % mod_name)
- raise ImportError("No code object available for %s" % mod_name)
filename = _get_filename(loader, mod_name)
(Update: fixed an error in format string)
Now...
C:\Documents and Settings\Пользователь>python -m pythoncom
C:\Documents and Settings\Пользователь>
This still won't work for builtin modules. Again, you'll need to invent some notion of "main code unit" for them.
Update:
I've looked through the internals called from _get_module_details and can say with confidence that they don't even attempt to retrieve a code object from a module of type other than imp.PY_SOURCE, imp.PY_COMPILED or imp.PKG_DIRECTORY . So you have to patch this machinery this way or another for -m to work. Python fails before retrieving anything from your module (it doesn't even check if the dll is a valid module) so you can't do anything by building it in a special way.

Does your requirement of single distributed binary allow for the use of an egg? If so, you could package your module with a __main__.py with your calling code and the usual __init__.py...
If you're really adamant, maybe you could extend pkgutil.ImpLoader.get_code to return something for C modules (e.g., maybe a special __code__ function). To do that, I think you're going to have to actually change it in the Python source. Even then, pkgutil uses exec to execute the code block, so it would have to be Python code anyway.
TL;DR: I think you're euchred. While Python modules have code at the global level that runs at import time, C modules don't; they're mostly just a dict namespace. Thus, running a C module doesn't really make sense from a conceptual standpoint. You need some real Python code to direct the action.

I think that you need to start by making a separate file in Python and getting the -m option to work. Then, turn that Python file into a code object and incorporate it into your binary in such a way that it continues to work.
Look up setuptools in PyPi, download the .egg and take a look at the file. You will see that the first few bytes contain a Python script and these are followed by a .ZIP file bytestream. Something similar may work for you.

There's a brand new thing that may solve your problems easily. I've just learnt about it and it looks preety decent to me: http://code.google.com/p/pts-mini-gpl/wiki/StaticPython

Google Appengine and Python exceptions

In my Google Appengine application I have defined a custom exception InvalidUrlException(Exception) in the module 'gvu'. Somewhere in my code I do:
try:
results = gvu.article_parser.parse(source_url)
except gvu.InvalidUrlException as e:
self.redirect('/home?message='+str(e))
...
which works fine in the local GAE development server, but raises
<type 'exceptions.SyntaxError'>: invalid syntax (translator.py, line 18)
when I upload it. (line 18 is the line starting with 'except')
The problem seems to come from the 'as e' part: if I remove it I don't get this exception anymore. However I would like to be able to access the raised exception. Have you ever encountered this issue? Is there an alternative syntax?

You probably have an older Python version on your server. except ExceptionType as varname: is a newer syntax. Previously you needed to simply use a comma: except ExceptionType, varname:.

I was getting the same error because I was using the pydoc command instead of the pydoc3 command on a python3 file that was using python3 print statements (print statements with parenthesis).

Just FYI, another possible cause for this error - especially if the line referenced is early in the script (like line 2) is line ending differences between Unix and Windows.
I was running Python on Windows from a Cygwin shell and got this error, and was really puzzled. I had created the file with "touch" before editing it.
I renamed the file to a temp file name, and copied another file (which I had downloaded from a Unix server) onto the original file name, then restored the contents via the temp file, and problem solved. Same exact file contents (on the screen anyway), only difference was where the file had originally been created.
Just wanted to post this in case anyone else ran across this error and was similarly puzzled.