I have my own package in python and I am using it very often. what is the most elegant or conventional directory where i should put my package so it is going to be imported without playing with PYTHONPATH or sys.path?
What about site-packages for example?
/usr/lib/python2.7/site-packages.
Is it common in python to copy and paste the package there ?
I usually put the stuff i want to have ready to import in the user site directory:
~/.local/lib/pythonX.X/site-packages
To show the right directory for your platform, you can use python -m site --user-site
edit: it will show up in sys.path once you create it:
mkdir -p "`python -m site --user-site`"
So if your a novice like myself and your directories are not very well organized you may want to try this method.
Open your python terminal. Import a module that you know works such as numpy in my case and do the following.
Import numpy
numpy.__file__
which results in
'/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site- packages/numpy/__init__.py'
The result of numpy.__file__ is the location you should put the python file with your module (excluding the numpy/__init__.py) so for me that would be
/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site- packages
To do this just go to your terminal and type
mv "location of your module" "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site- packages"
Now you should be able to import your module.
This is something that works for me (I have to frequently create python packages that are uploaded to a private pip repository). elaborating on the comment by #joran on the question.
create a "build directory" which is used as a workspace to create packages. any directory of your choice will do
Copy your python package dir there, and create a setup.py file. this should help in creating the setup.py correctly.
create a virtualenv for the project you are working on. virtualenvs have a bunch of other benefits, I am not going into the details here.
create a local dist package python setup.py sdist --format=tar. the package created should ideally be in the dist folder.
Install the package on your virtualenv (after activating it). pip install <yourpackage>.tar
you can use pip install --force-reinstall if you need to play around with the libraries more and re-create the dist packages.
I've found that this method works great for me. If you do not need to package the modules for use of other systems instead of just your local, this method might be an overkill
Happy hacking.
If you're developing a module I would recommend to follow this.
import sys
sys.path.append("/home/mylinux/python-packages")
Now any module you keep in python-packages is importable by Python-interpreter.
On my Mac, I did a sudo find / -name "site-packages". That gave me a few paths like /Library/Python/2.6/site-packages, /Library/Python/2.7/site-packages, and /opt/X11/lib/python2.6/site-packages.
So, I knew where to put my modules if I was using v2.7 or v2.6.
Hope it helps.
import folders could be extracted by adding following source code:
import sys
print sys.path
automatic symlink generation example would be:
ln -s \`pip show em | grep "Location"| cut -d " " -f2\` \`python -m site --user-site\`
instead of "em" you may use other package you've "just installed but the python can't see it"
below I'll explain in more details as being requested in the comment.
suppose you've installed python module em or pyserial with the following command (examples are for ubuntu):
sudo pip install pyserial
and the output is like this:
Collecting pyserial
Downloading pyserial-3.3-py2.py3-none-any.whl (189kB)
100% |████████████████████████████████| 194kB 2.3MB/s
Installing collected packages: pyserial
Successfully installed pyserial-3.3
the question would be following - python can't see the module pyserial, why?
because the location where the module has been installed isn't the one python is looking at for your particular user account.
solution - we have to create symlink from the path where pyserial arrived to the path where your python is looking for.
symlink creation command would be:
ln -s <what_to_link> <where_to_link>
instead of typing exact location we are asking pip to tell us where it stored modules by executing command:
pip show pyserial | grep "Location"| cut -d " " -f2
instead of typing exact location we are asking python to tell us where it looks for the modules being installed by executing command:
python -m site --user-site
both commands has to be escaped with "`" character (usually on the left of your 1 button for the US keyboards)
in result following command will be provided for ln and the missing symlink would be created:
ln -s /usr/local/lib/python2.7/dist-packages /home/<your_username>/.local/lib/python2.7/site-packages
or something similar, depending on your distro and python/pip defaults.
Related
The Python Cookbook suggests the following tree structure for a "typical library package":
projectname/
README.txt
Doc/
documentation.txt
projectname/
__init__.py
foo.py
bar.py
utils/
__init__.py
spam.py
grok.py
examples/
helloworld.py
You 'll notice that the examples/ are not part of the actual package, which resides under projectname/projectname/ (that's where you 'll find the top-level __init__.py of the package).
Well, examples/helloworld.py obviously needs to import the projectname package.
I am aware that there are at least 2-3 relevant questions in StackOverflow. I do not believe that this is a duplicate because the other questions either involve intra-package imports or the general case of importing one python module from another when they do not reside in the same directory. I am specifically asking for the suggested way to do this when packaging a library.
Is there a way to achieve this without modifying the path? If modifying the path is the only way, is there a way for this to be done in an elegant manner?
Let me elaborate on that last point. In Repository Structure and Python by Kenneth Reitz, a similar structure appears, with tests/ instead of examples/. This is exactly the same problem. He suggests using "a simple (but explicit) path modification to resolve the package properly." OK, but this is the actual code:
import os
import sys
sys.path.insert(0, os.path.abspath('..'))
I really don't like the .. part. I would hope for a more general solution, hopefully one that would work from whichever directory I would choose to run the example (or the test).
While the folder tree looks alright, I believe that implicitly assuming that modules are available for import is wrong from both a developer's perspective and certainly from a user's perspective.
It's true that you can technically use sys.path.insert(0, os.path.abspath('..')) to add any path for python to allow imports from, but that means you, the developer, have to make sure that the added path is always in the right location.
It is common for users to install packages to use them. There's a clear workflow for developers:
Have pip and virtualenv installed (and even better, virtualenvwrapper)
Install the package in editable mode using pip's -e flag which means that any changes you make to the code will directly effect execution. You won't have to reinstall everytime you make changes to your code.
Since your code is always installed (specifically, under site-packages as a user but in editable mode when you develop), you can always import your package using its explicit name from any example or test.
A common workflow:
$ pip install virtualenv
...
$ virtualenv distro
New python executable in /home/nir0s/work/distro/bin/python3
Also creating executable in /home/nir0s/work/distro/bin/python
Installing setuptools, pip, wheel...done.
$ source distro/bin/activate
# install in editable mode
$ pip install -e ~/repos/nir0s/distro/
Obtaining file:///home/nir0s/repos/nir0s/distro
Installing collected packages: distro
Running setup.py develop for distro
Successfully installed distro
(distro) $ pip freeze
appdirs==1.4.3
-e git+git#github.com:nir0s/distro#e8a182f9d1dbe6391f25...#egg=distro
packaging==16.8
pyparsing==2.2.0
six==1.10.0
The package, being installed in editable mode, means that there's an egg-link file pointing to the directory of the package:
$ cat distro/lib/python3.6/site-packages/distro.egg-link
/home/nir0s/repos/nir0s/distro
Users will do the same thing without dealing with editable mode. Simply creating virtual environments and install packages in them. Then, when they're done working, they'll delete those virtual environments. Easy peasy.
After installing the Nest Neural Simulator, I keep getting the following error when trying to run any of the example python files that came in the installation. I've tried re-installing Nest, Python, and using Anaconda, but no go.
Python error:
ImportError: No module named nest
Suggestions?
At https://nest-simulator.org/documentation you now find many different install instructions and how to solve the "ImportError: no module named nest" depends on the way you installed NEST.
System Python
The problem with the nest python module not being found is usually, that NEST is installed for a specific Python version and you can not load it from another. So while many OS still use Python 2.7 you may need to explicitly run
$ python3
>>> import nest
Additionally, if you have multiple Python 3.x versions installed, modules may still be installed for a different version and you have to explicitly start python with python3.6 or python3.8, etc.
Conda package
As #nosratullah-mohammadi already mentioned, if you have a Conda flavour installed, using the pre-built package is a very quick solution. The link in his post is unfortunately broken; this one should work, then go to "Installation" in the side bar.
$ conda create --name nest -c conda-forge python3 nest-simulator
$ conda activate nest
$ python # this should load the Python from the conda env
>>> import nest # this loads nest which is installed explicitly for that Python
From Source
For every install from source, be sure to have Python and other prerequisites installed before building NEST. Then you can create your temporary build directory (can be deleted afterwards) and configure with the flags you need.
cd somewhere
mkdir nest-build
cd nest-build
cmake -DCMAKE_INSTALL_PREFIX:PATH=/install/path -Dwith-python=3 .../sources/of/nest-simulator
Replace somewhere, /install/path and .../sources/of/nest-simulator with the paths correct for your setup. (A popular choice when compiling from source in conjunction with Conda environments, for example, is to use -CMAKE_INSTALL_PREFIX=$CONDA_PREFIX, which installs NEST directly into the active environment. Conda is however in no way necessary for NEST.)
Add more -D... flags as you prefer. Possible flags you see with cmake -LA .../sources/of/nest-simulator, as pointed out here. You are probably interested in many of the with-xyz at the end. Check the aforementioned documentation for deatils.
Check that the paths and libraries reported in the Configuration Summary make sense (you may need to scroll up a bit to see). It could for example look something like this:
--------------------------------------------------------------------------------
NEST Configuration Summary
--------------------------------------------------------------------------------
[...]
Python bindings : Yes (Python 3.6.8: /home/yourname/miniconda3/envs/nest/bin/python3)
Includes : /home/yourname/miniconda3/envs/nest/include/python3.6m
Libraries : /home/yourname/miniconda3/envs/nest/lib/libpython3.6m.so
Cython bindings : Yes (Cython 0.27.3: /home/yourname/miniconda3/envs/nest/bin/cython)
[...]
--------------------------------------------------------------------------------
[...]
PyNEST will be installed to:
/home/yourname/miniconda3/envs/nest/lib/python3.6/site-packages
--------------------------------------------------------------------------------
In this example CMake configured everything for Python3.6 from my conda environment.
If you are satisfied with your settings and all the found Python versions match, run the usual
$ make # optionally with -j$(nproc)
$ make install
$ make installcheck
In case that works out fine you are done and can delete the build directory to free the space. Congratulations!
Also if things get too mixed up and it doesn't seem to do what you expect, it is sometimes useful to delete the build directory and start off clean.
there is a new method of installation added to other methods, wich is installing nest with conda package and it's in its beta version. but it works and it's really simple.
you can find the installation from here!
simply after install mini conda package run your terminal and type this :
conda create --name ENVNAME -c conda-forge nest-simulator python
then type :
conda activate ENVNAME
and you're good to go!
NEST now provide the solution to that problem and similar ones, by providing a script which automatically sets the relevant system variables:
If your operating system does not find the nest executable or Python
does not find the nest module, your path variables may not be set
correctly. This may also be the case if Python cannot load the nest
module due to missing or incompatible libraries. In this case, please
run
source </path/to/nest_install_dir>/bin/nest_vars.sh
to set the necessary environment variables. You may want to include
this line in your .bashrc file, so that the environment variables are
set automatically.
https://nest-simulator.readthedocs.io/en/latest/installation/linux_install.html
Turns out I needed to move the directory where I installed nest (Users/name/opt/nest) into a nest folder in the following directory in anaconda. Specifically, I moved the contents of the folder (from the nest installation):
/Users/name/opt/nest/lib/python2.7/site-packages/nest
Into this one:
/anaconda/lib/python2.7/site-packages/nest
Disclaimer: I could very well run into problems for not having copied all the contents of the Nest installation, but this little hack is helping me run example files now.
First of all, I should say, that I checked all links at stackoverflow, but I still can't make it work. What I want is just as simple as my nose - I want to import numpy and I want to import modules created by f2py. Now, when I do in console
$ python
>> import numpy
I get an error No module named numpy. In the same way I get an error, when I try to import a fortran module made by f2py:
>> import testmodule
My OS is Ubuntu 12.04. I should also add, that I tried to uninstall and reinstall python hundreds of times with different libs and of course I did sudo apt-get install python-numpy etc. But that didn't help. What I want to hear from you, guys, is a complete step-by-step instruction (including unintallation of the current versions of python which may be corrupt and including installation instructions - download this version, unpack it to here etc.) I guess that instruction will be extremely worthy and usefull for python newbeis like me. The problem that I face now seems to be the simplest in the world, but I wonder why it has no simple solution.
Does your Python prompt have >> as the prompt? I've always seen >>> from Python.
If uninstalling Python and reinstalling doesn't work, perhaps the problem is with your user account? I'd try:
Create a new user, sudo useradd joe
Log in as the new user sudo -u joe bash -login
See if Python and numpy work now.
Exit from joe's shell (exit, logout or ^D).
Get rid of joe, sudo userdel joe
Now at least you know if the problem is with your system setup or your user setup.
Other things to look for:
run pip freeze | grep numpy or pip freeze | less to see which numpy package is (or isn't) installed.
Do you have something odd in your environment? Try env | grep -i python to see if you have a nonstandard environment variable.
Do you have python aliased to something else in your .profile or other startup? Try alias python to see if you really are starting python when you run python.
Do you have some old python in your $PATH? You can try which python and you should see /usr/bin/python. If you get '/usr/local/bin/pythonthat should be a link pointing to the "real" python at/usr/bin/python`.
Have a look at /usr/bin/python. It should be a link to python2.7.
During your uninstall-reinstall cycles, you can run pip freeze to see the list of installed packages. You should be able to make numpy appear and disappear in the freeze list when you install and uninstall it.
I just wrote a python script to fix some database issues and it uses the psycopg2 module. For some reason the person that needs to run it (on the server) is claiming that they can't install psycopg2 on their server machine... is there a way that I can package that module within my script such that they don't have to have psycopg2 installed? Something similar to adding a lib folder to the classpath in Java?
Thx in advance,
Andre
Make a directory, put your script into it.
cd into that directory, then run:
$ easy_install -Z --prefix . psycopg2
That should result in a copy of the python module in the same directory.
Zip it all up and send the whole thing off.
As to be unable to install psycopg2, I sure it relates to security policies about who can have root access on database servers. If you can't find someone with the necessary permissions, there are two methods to skin this particular cat:
1) You could build an exe using py2exe, that way you know that all of the dependencies for your scripts are there and available. Also this will allow to you escape dealing with any python incompatibilities. You made no mention of your python version (2.5, 2.6, 2.7, 3.1?) nor the version on the server.
2) Or you could place the psycopg2 module into its own directory (c:\mypymods) and add the path to sys.path before importing.
import sys
sys.path = sys.path.append("c:/mypymods")
import psycopg2
....
The question is related to the answer to "Unable to install Python without sudo access".
I need to install python-setuptools to install python modules.
I have extracted the installation package.
I get the following error when configuring
[~/wepapps/pythonModules/setuptools-0.6c9]# ./configure --prefix=/home/masi/.local
-bash: ./configure: No such file or directory
I did not find the solution at the program's homepage.
How can I resolve this error?
As Noah states, setuptools isn't an automake package so doesn't use ‘./configure’. Instead it's a pure-Python-style ‘setup.py’ (distutils) script.
You shouldn't normally need to play with .pydistutils.cfg, as long as you run it with the right version of Python. So if you haven't added the .local/bin folder to PATH, you'd have to say explicitly:
/home/masi/.local/bin/python setup.py install
AIUI this should Just Work.
I did not find the solution at the program's homepage.
Yeah, they want you to install it from a shell script egg which uses the default version of Python. Which you don't want.
(Another approach if you can't get setuptools to work is to skip it and install each module and dependency manually. Personally I have a bit of an aversion to setuptools/egg, as it contains far too much “clever” magic for my tastes and makes a mess of my filesystem. But I'm an old curmudgeon like that. Most Python modules can be obtained as simple Python files or plain old distutils scripts, but unfortunately there are some that demand eggs.)
You might want to check http://peak.telecommunity.com/DevCenter/EasyInstall#custom-installation-locations.
EasyInstall is a python module with some shell scripts (or some shell scripts with a python module?) and does not use the unix make tool that gets configured with the "./configure" command. It looks like your best bet is to try editing ~/.pydistutils.cfg to include:
[install]
install_lib = /home/masi/.local/lib/python/site-packages/
install_scripts = /home/masi/.local/bin
You'll also presumably have made the ~/.local/bin/ folder part of your PATH so you can run the easy_install script. (I'm not sure exactly where the site-packages directory will be under .local, but it shouldn't be hard to find.)
Hope this helps.