I'm trying to create a development environment where modules are
divided into libraries and applications.
The application needs to import a module that is not installed
as a package into the main python packages.
Both the application and the libraries are continuously modified.
This is the directory layout of the files. Files from one project may be reused by other projects, and thus cannot be in the same directory tree.
projA\lib\util.py
projA\lib\other.py
projB\lib\another.py
projC\src\app1\app1.py
So far, the best I could come up with is the following,
which causes problems for IDEs code completion because of the dynamic imports
# app1.py
import sys
sys.path.append('../../../projA/lib')
import util
Is there a better way of doing this?
How about using virtualenv, and installing your other projects as libraries in the virtual system path.
Most Python IDE's support virtualenv and have no problems with code-completion.
Also it is a good practice making it easy to distribute your project and managing dependencies.
If you don't want to use virtualenv or dynamic imports,
you can add your modules path to PYTHONPATH environment variable.
PYTHONPATH
Note: you might have to create this environment variable, assuming you are using Windows OS, you can use (from command line):
setx PYTHONPATH folder1;folder2;etc
setx
Related
In my particular setting, I have a set of python modules that include auxiliary functions used in many different other modules. I putted them into a LIBS folder and I have other folder at the same path level those are including other modules that are doing certain jobs by using the help of these LIBS modules. Presently, I do this for all the modules to import LIBS modules.
import sys
sys.path.insert(0, '../LIBS')
import lib_module1
import lib_module2
....
As the project getting larger, this starts to be pain in the neck. I need to write down a large set of import statements for these auxiliary LIBS modules for each new module.
Is there any way to automatically import all these LIBS modules for the other modules that are in the folders living at the same path lelvel with LIBS folder?
For this, you can use
__init__.py
Kindly refer Modules and Stackoverflow.
Indeed there is! Start treating your LIBS modules as "real" modules (or packages) that are installed into the system like any other.
This means you will have to write a setup.py script to install your code. Generally this is done inside your development directory, then your module is installed with:
$ sudo python setup.py install
This will install your module under the site-packages subdirectory of wherever Python libraries are stored on your system.
I suggest starting by copying someone else's working setup.py and supporting files, then modifying to suit your packages. For example, here is my quoter module.
Fair warning: This is a pretty big step. Not only will you learn to deploy your module locally, you can also publish it on PyPI if you wish. The step of moving to true packages will encourage you to write more and more standard documentation, to develop and run more tests, to adopt more rigorous version specifications, to more clearly identify and define code dependencies, and take many other "professionalization" steps. These all pay dividends in better, more reliable, more portable, more easily deployed code--but I'd be lying if I didn't admit the learning curve can be steep at times.
I am currently in the process of developing a module that I am using as a library to be imported in another project. I need a sane way that I can install this module in the python site packages in such a way that I don't need to reinstall it everytime I make changes to it. Currently, I am using sudo pip install --force-reinstall {BASE_FOLDER_FOR_MODULE}, but need to re run this command everytime I make any changes to the module code.
The little bit of info I've been able to find on the subject seems to indicate that while I can symlink the base folder for the module in the site-packages folder for my other project, that this may not necessarily be a good way to do it. Is symlinking the folder bad, and if so why? Is there an alternate (better) option?
Thanks
If the library is still under active development then consider adding it to environment variable PYTHONPATH. Directories in PYTHONPATH are appended to sys.path and are searched last when trying to find a module. Using PYTHONPATH means you only have to make minimal changes (set it in a config file source file or .bashrc file) for things to work. Once the library is finalised you can install it to the site-packages directory and remove it from PYTHONPATH.
I have a system python installation and a personal python installation in my home directory. My personal python comes in ahead in my $PATH and I usually run that. But the system python installation has some modules that I want to use with my personal python installation.
So, basically, I want to append the sys.path of the system python installation to the sys.path of the personal python installation.
I read up on the docs and source of site module and saw that I could use the sitecustomize or usercustomize modules to do this. But where I am stuck is how do I get the sys.path of the system python to be appended to the personal python's sys.path.
The gensitepackages function in the site modules seems to calculate the paths to be added to sys.path but it is using the PREFIXES global variable instead of taking it as an argument, so for all I know, I can't use it. Adding system python's prefixes to PREFIXES is also not an option as by the time the customize module(s) are loaded, the PREFIXES is already used to build the path.
Any ideas on how to go about this?
Also, I'm not sure if I should ask this on askubuntu/unix&linux. Comments?
Edit: Guess I wasn't clear on this part. I want the system python's path to be appended so that when I try to use modules that are not present in my personal python, it will automatically fallback to the system python's modules.
Either set the PYTHONPATH environment variable or, in your personal Python installation's site-packages directory, add one or more path configuration files (.pth files) that point to the directories you want to add from the system Python installation.
UPDATE: In general, it's not a good idea to mix modules from different Python instances. For instance, there could be differences in the configure parameters used to build the two Python that could produce incompatibilities in C extension modules built for the existing system Python but used with your personal Python. But, if you really want to go down this path, you could manipulate the path with .pth files or modify the site module. Or what might be a more-straightforward and easier-to-manage approach is to use virtualenv to create a private virtual environment using the system Python but then replace the interpreter in the environment; see, for example, the section Using Virtualenv without bin/python.
python is gonna check if there is a $PYTHONPATH environment variable set. Use that for the path of your other modules.
use export PYTHONPATH="path:locations"
I want to install python on a flash drive in a virtual environment so that I can develop code wherever I am. Is this possible to do in such a way that I can use my flash drive on windows/mac/linux computers?
For windows, head to Portable Python (http://PortablePython.com) to see various options you have,
For linux and Mac you don't need to install it on USB drive as those systems usually come with Python pre-installed. If you need specific packages for those systems, bring them on USB together with a command line script that can load them with one call in virtualenv on those systems and you are good to go !
Be aware that this is never 100% bullet proof as you are depending on Python version you are using/bringing packages for.
You could try looking at setting up something using some VirtualEnv type environments, with the various Python versions installed on your machines.
Not sure how you'd get round the different paths on the different operating systems though.
Virtualenv: http://pypi.python.org/pypi/virtualenv
As #millimoose pointed out, you could install three different versions of Python.
For each Python package you are working on, you can create a .pth file in the site-packages directory of each Python version that you would like to use the package from.
Note that, as described here:
If you put a .pth file in the site-packages directory containing a path, python searches this path for imports.
For example, if you have a package named my_package that you are working on that resides at the path C:\Users\Me\Documents\dev_packages\my_package, you can add a file with extension .pth (note that the name doesn't matter, specifically it doesn't have to have any relation to the package name), with the contents:
C:\Users\Me\Documents\dev_packages
This will add C:\Users\Me\Documents\dev_packages to the Python import search-path, causing the my_package package to be discovered. By placing this .pth file in the site-packages directory of each Python version, my_package will be available in all corresponding versions of Python.
I'm developing a Python utility module to help with file downloads, archives, etc. I have a project set up in a virtual environment along with my unit tests. When I want to use this module on the same computer (essentially as "Production"), I move the files to the mymodule directory in the ~/dev/modules/mymodule
I keep all 3rd-party modules under ~/dev/modules/contrib. This contrib path is on my PYTHONPATH, but mymodule is NOT because I've noticed that if mymodule is on my PYTHONPATH, my unit tests cannot distinguish between the "Development" version and the "Production" version. But now if I want to use this common utility module, I have to manually add it to the PYTHONPATH.
This works, but I'm sure there's a better, more automated way.
What is the best way to have a Development and Production module on the same computer? For instance, is there a way to set PYTHONPATH dynamically?
You can add/modify python paths at sys.path, just make sure that the first path is the current directory ".", because some third-party modules rely on importing from the directory of the current module.
More information on python paths:
http://djangotricks.blogspot.com/2008/09/note-on-python-paths.html
I'm guessing by virtual environment you mean the virtualenv package?
http://pypi.python.org/pypi/virtualenv
What I'd try (and apologies if I've not understood the question right) is:
Keep the source somewhere that isn't referenced by PYTHONPATH (e.g. ~/projects/myproject)
Write a simple setuptools or distutils script for installing it (see Python distutils - does anyone know how to use it?)
Use the virtualenv package to create a dev virtual environment with the --no-site-packages option - this way your "dev" version won't see any packages installed in the default python installation.
(Also make sure your PYTHONPATH doesn't have any of your source directories)
Then, for testing:
Activate dev virtual environment
Run install script, (usually something like python setup.py build install). Your package ends up in /path/to/dev_virtualenv/lib/python2.x/site-packages/
Test, break, fix, repeat
And, for production:
Make sure dev virtualenv isn't activated
Run install script
All good to go, the "dev" version is hidden away in a virtual environment that production can't see...
...And there's no (direct) messing around with PYTHONPATH
That said, I write this with the confidence of someone who's not actually tried setting using virtualenv in anger and the hope I've vaguely understood your question... ;)
You could set the PYTHONPATH as a global environment variable pointing to your Production code, and then in any shell in which you want to use the Development code, change the PYTHONPATH to point to that code.
(Is that too simplistic? Have I missed something?)