I have gdata library install on my ArchLinux, and a simple application which imports atom library at the beginning, when I run gapp engine and access that web app,
$ python2.5 ./dev_appserver.py ~/myapp
It throws exception 'No module named atom'. But when I run 'import atom' in Python2.5 interactive mode, it works well. How can I import atom module in my GAppEngine applications?
Add atom.py to the same directory you keep you GAE Python sources in, and make sure it's uploaded to the server when you upload your app. (The upload happens when you do appcfg.py update myapp/ unless you go out of your way to stop it; use the --verbose flag on the command to see exactly what's being uploaded or updated).
(Or, if it's a large file, make a zipfile with it and in your handler append that zipfile to sys.path; see zipimport for example).
This assumes that you have a single file atom.py which is what you're importing; if that file in turns imports others you'll have to make those others available too in similar ways, and so on (see modulefinder in Python's standard library for ways to find all modules you need).
If atom is not a module but a package, then what you get on import is the __init__.py file in the directory that's the package; so the same advice applies (and zipimport becomes much more attractive since you can easily package up any directory structure e.g. with a zip -r command from the Linux command line).
If at any point (as modulefinder will help you discover) there is a dependency on a third party C-coded extension (a .so or .pyd file that Python can use but is not written in pure Python) that is not in the short list supplied with GAE (see here), then that Python code is not usable on GAE, as GAE supports only pure-Python. If this is the case then you must look for alternatives that are supported on GAE, i.e. pure-Python ways to obtain the same functionality you require.
Related
I am developing a plugin for a multi-platform Python program (Deluge). My plugin needs to import some modules which aren't available by default (in my case, the requests module).
On Linux, everything works flawlessly assuming the required modules are installed beforehand (e.g. via pip).
On Windows, the program makes use of python27.dll which comes as part of the installer, so importing modules - even those available on the local Python installation (verified via interpreter) - yields an import error.
I've seen the answers to this question, but I'd like to know if there is a proper way of adding module search paths for Python on Windows specifically. Is it safe to assume C:\Python27\Lib\site-packages will point me to the local Python installation's modules?
EDIT: Is there a different method I could incorporate for using "external" modules? Could I perhaps package other modules into my final .egg file? Not just plain Python, but more sophisticated modules like requests which need to be properly built and may even rely on other modules.
I need to ship a collection of Python programs that use multiple packages stored in a local Library directory: the goal is to avoid having users install packages before using my programs (the packages are shipped in the Library directory). What is the best way of importing the packages contained in Library?
I tried three methods, but none of them appears perfect: is there a simpler and robust method? or is one of these methods the best one can do?
In the first method, the Library folder is simply added to the library path:
import sys
import os
sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'Library'))
import package_from_Library
The Library folder is put at the beginning so that the packages shipped with my programs have priority over the same modules installed by the user (this way I am sure that they have the correct version to work with my programs). This method also works when the Library folder is not in the current directory, which is good. However, this approach has drawbacks. Each and every one of my programs adds a copy of the same path to sys.path, which is a waste. In addition, all programs must contain the same three path-modifying lines, which goes against the Don't Repeat Yourself principle.
An improvement over the above problems consists in trying to add the Library path only once, by doing it in an imported module:
# In module add_Library_path:
sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'Library'))
and then to use, in each of my programs:
import add_Library_path
import package_from_Library
This way, thanks to the caching mechanism of CPython, the module add_Library_path is only run once, and the Library path is added only once to sys.path. However, a drawback of this approach is that import add_Library_path has an invisible side effect, and that the order of the imports matters: this makes the code less legible, and more fragile. Also, this forces my distribution of programs to inlude an add_Library_path.py program that users will not use.
Python modules from Library can also be imported by making it a package (empty __init__.py file stored inside), which allows one to do:
from Library import module_from_Library
However, this breaks for packages in Library, as they might do something like from xlutils.filter import …, which breaks because xlutils is not found in sys.path. So, this method works, but only when including modules in Library, not packages.
All these methods have some drawback.
Is there a better way of shipping programs with a collection of packages (that they use) stored in a local Library directory? or is one of the methods above (method 1?) the best one can do?
PS: In my case, all the packages from Library are pure Python packages, but a more general solution that works for any operating system is best.
PPS: The goal is that the user be able to use my programs without having to install anything (beyond copying the directory I ship them regularly), like in the examples above.
PPPS: More precisely, the goal is to have the flexibility of easily updating both my collection of programs and their associated third-party packages from Library by having my users do a simple copy of a directory containing my programs and the Library folder of "hidden" third-party packages. (I do frequent updates, so I prefer not forcing the users to update their Python distribution too.)
Messing around with sys.path() leads to pain... The modern package template and Distribute contain a vast array of information and were in part set up to solve your problem.
What I would do is to set up setup.py to install all your packages to a specific site-packages location or if you could do it to the system's site-packages. In the former case, the local site-packages would then be added to the PYTHONPATH of the system/user. In the latter case, nothing needs to changes
You could use the batch file to set the python path as well. Or change the python executable to point to a shell script that contains a modified PYTHONPATH and then executes the python interpreter. The latter of course, means that you have to have access to the user's machine, which you do not. However, if your users only run scripts and do not import your own libraries, you could use your own wrapper for scripts:
#!/path/to/my/python
And the /path/to/my/python script would be something like:
#!/bin/sh
PYTHONPATH=/whatever/lib/path:$PYTHONPATH /usr/bin/python $*
I think you should have a look at path import hooks which allow to modify the behaviour of python when searching for modules.
For example you could try to do something like kde's scriptengine does for python plugins[1].
It adds a special token to sys.path(like "<plasmaXXXXXX>" with XXXXXX being a random number just to avoid name collisions) and then when python try to import modules and can't find them in the other paths, it will call your importer which can deal with it.
A simpler alternative is to have a main script used as launcher which simply adds the path to sys.path and execute the target file(so that you can safely avoid putting the sys.path.append(...) line on every file).
Yet an other alternative, that works on python2.6+, would be to install the library under the per-user site-packages directory.
[1] You can find the source code under /usr/share/kde4/apps/plasma_scriptengine_python in a linux installation with kde.
My python program uses plugins (python files) which I import dynamically using __import__.
I bundle my python program into a Windows exe using py2exe.
I've just spend searching 2 hours why my plugin python file couldn't be loaded properly from the .exe version. I got an ImportError: "no module named urllib2"
It appeared my plugin was using urllib2 (through an import urllib2 statement), and that standard library module was apparently not bundled into the exe. Other modules used in the plugin (re, urllib, ...) gave no problem, but perhaps they were already references in python files I statically include in my program.
How can I know which standard Python library modules py2exe bundles by default in the exe? (so I know whether I or somebody else can use them in a plugins). The py2exe documentation doesn't give an hints, except for saying that it includes a lot of modules from the standard library.
To see which modules are included look inside the library.zip (if there is no library.zip file - then try opening the EXE in any ZIP application - or rename it to .ZIP and try and open it).
You will be able to see a list of *.pyc. You can look at the list of files and directories to get an impression of which modules are included or not.
If you require a specific package to be added - add it to the 'packages' list.
As to why it doesn't include everything or how it chooses to include some and not others? My understanding is that py2exe looks in your code to figure out what you are using and includes those (and some that it probably needs itself) but maybe it also has some heuristics to add other modules too (I haven't checked :)
Currently, I'm deploying a full python distribution (the original python 2.7 msi) with my app. Which is an embedded web server made with delphi.
Reading this, I wonder if is possible to embed the necessary python files with my app, to decrease load files and avoid conflict with several python versions.
I have previous experience with python for delphi so I only need to know if only shipping the python dll + zip with the distro + own scripts will work (and if exist any caveats I must know or a sample where I can look)
zipimport should work just fine for you -- I'm not familiar with Python for Delphi, but I doubt it disables that functionality (an embedding application can do that, but it's an unusual choice). Just remember that what you can zip up and import directly are the Python-coded modules (or just their corresponding .pyc or .pyo byte codes) -- DLLs (even if renamed as .pyds;-) need to be on disk to be loaded (so if you have a zipfile with them it will need to be unzipped at the start of the app, e.g. into a temporary directory).
Moreover, you don't even need to zip up all modules, just those you actually need (by transitive closure) -- and you can easily find out exactly which modules those are, with the modulefinder module of the standard Python library. The example on the documentation page I just pointed to should clarify things. Happy zipping!
Yes it is possible.
I'm actually writing automatisation script in Python with the Zipimport library. I actually included every .py files in my zip as well as configuration or xml files needed by those script.
Then, I call a .command file targeting a __main__.py class that redirect towards the desired script according to my sys.argv parameters which is really useful!
How should I install (or where should I put and organize) usual python libraries in Google App Engine.
Some libraries require to be installed using setuptools. How can I install that libraries.
You need to unpack the libraries into a subdirectory of your app, and add the library directory to the Python path in your request handler module. Any steps required by setup scripts, you'll have to execute manually, but there generally aren't any unless the library bundles a native module (which aren't supported on App Engine anyway).
If your library contains many files, it's possible to zip them and use zipimport, but that's somewhat more complex, and has performance implications.
For example, suppose you put a library in lib/mylibrary, under your app's directory. In your request handler module, add the following before any of your other imports:
import os
import sys
sys.path.append(os.path.join(os.path.dirname(__file__), "lib/mylibrary"))
(Note that this assumes that your request handler is in the root directory of your app.)
Most of them can be installed using the pip.
Follow 3 first points from the Google wiki.