The Python Cookbook suggests the following tree structure for a "typical library package":
projectname/
README.txt
Doc/
documentation.txt
projectname/
__init__.py
foo.py
bar.py
utils/
__init__.py
spam.py
grok.py
examples/
helloworld.py
You 'll notice that the examples/ are not part of the actual package, which resides under projectname/projectname/ (that's where you 'll find the top-level __init__.py of the package).
Well, examples/helloworld.py obviously needs to import the projectname package.
I am aware that there are at least 2-3 relevant questions in StackOverflow. I do not believe that this is a duplicate because the other questions either involve intra-package imports or the general case of importing one python module from another when they do not reside in the same directory. I am specifically asking for the suggested way to do this when packaging a library.
Is there a way to achieve this without modifying the path? If modifying the path is the only way, is there a way for this to be done in an elegant manner?
Let me elaborate on that last point. In Repository Structure and Python by Kenneth Reitz, a similar structure appears, with tests/ instead of examples/. This is exactly the same problem. He suggests using "a simple (but explicit) path modification to resolve the package properly." OK, but this is the actual code:
import os
import sys
sys.path.insert(0, os.path.abspath('..'))
I really don't like the .. part. I would hope for a more general solution, hopefully one that would work from whichever directory I would choose to run the example (or the test).
While the folder tree looks alright, I believe that implicitly assuming that modules are available for import is wrong from both a developer's perspective and certainly from a user's perspective.
It's true that you can technically use sys.path.insert(0, os.path.abspath('..')) to add any path for python to allow imports from, but that means you, the developer, have to make sure that the added path is always in the right location.
It is common for users to install packages to use them. There's a clear workflow for developers:
Have pip and virtualenv installed (and even better, virtualenvwrapper)
Install the package in editable mode using pip's -e flag which means that any changes you make to the code will directly effect execution. You won't have to reinstall everytime you make changes to your code.
Since your code is always installed (specifically, under site-packages as a user but in editable mode when you develop), you can always import your package using its explicit name from any example or test.
A common workflow:
$ pip install virtualenv
...
$ virtualenv distro
New python executable in /home/nir0s/work/distro/bin/python3
Also creating executable in /home/nir0s/work/distro/bin/python
Installing setuptools, pip, wheel...done.
$ source distro/bin/activate
# install in editable mode
$ pip install -e ~/repos/nir0s/distro/
Obtaining file:///home/nir0s/repos/nir0s/distro
Installing collected packages: distro
Running setup.py develop for distro
Successfully installed distro
(distro) $ pip freeze
appdirs==1.4.3
-e git+git#github.com:nir0s/distro#e8a182f9d1dbe6391f25...#egg=distro
packaging==16.8
pyparsing==2.2.0
six==1.10.0
The package, being installed in editable mode, means that there's an egg-link file pointing to the directory of the package:
$ cat distro/lib/python3.6/site-packages/distro.egg-link
/home/nir0s/repos/nir0s/distro
Users will do the same thing without dealing with editable mode. Simply creating virtual environments and install packages in them. Then, when they're done working, they'll delete those virtual environments. Easy peasy.
Related
I found this package of descent gradient optimization variants in python.
I installed python interpreter but i don't know how i can run the package.
I already tried to use windows cmd.
Ididn't used python before ,thanks a lot for helping me.
py-optim github
The github repository lacks a setup.py. If you want to install it, add the following code with the name setup.py to the top-level folder of the repository on your device. Then add one __init__.py file to the folder ..\PyOptim. The __init__.py can be totally empty. Try also to file an issue in the repo, stating that the setup.pyis missing.
from setuptools import setup
setup(name='pyoptim',
version='0.1',
description='optimizerTool',
url='https://github.com/schaul/py-optim',
author='None',
packages=['PyOptim',
'PyOptim.algorithms',
'PyOptim.benchmarks',
'PyOptim.core',
'PyOptim.external_libs'])
Afterwards, open a cmd in the top-level folder, and run
python setup.py install
This installs everything. You then can import everything.
Note: This is only a quick- setup.py. Please also add install-requeries and so on to install dependencies of the repo.
If you want the folders test and tools also to be installed, add an empty __init__.pyfile to these folders as well and add the names in the packages list in the setup.py.
EDIT: Use this fork of the repository were i added the missing files. Make sure you install python 2.x as this repo is not for 3.x.
I have my own package in python and I am using it very often. what is the most elegant or conventional directory where i should put my package so it is going to be imported without playing with PYTHONPATH or sys.path?
What about site-packages for example?
/usr/lib/python2.7/site-packages.
Is it common in python to copy and paste the package there ?
I usually put the stuff i want to have ready to import in the user site directory:
~/.local/lib/pythonX.X/site-packages
To show the right directory for your platform, you can use python -m site --user-site
edit: it will show up in sys.path once you create it:
mkdir -p "`python -m site --user-site`"
So if your a novice like myself and your directories are not very well organized you may want to try this method.
Open your python terminal. Import a module that you know works such as numpy in my case and do the following.
Import numpy
numpy.__file__
which results in
'/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site- packages/numpy/__init__.py'
The result of numpy.__file__ is the location you should put the python file with your module (excluding the numpy/__init__.py) so for me that would be
/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site- packages
To do this just go to your terminal and type
mv "location of your module" "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site- packages"
Now you should be able to import your module.
This is something that works for me (I have to frequently create python packages that are uploaded to a private pip repository). elaborating on the comment by #joran on the question.
create a "build directory" which is used as a workspace to create packages. any directory of your choice will do
Copy your python package dir there, and create a setup.py file. this should help in creating the setup.py correctly.
create a virtualenv for the project you are working on. virtualenvs have a bunch of other benefits, I am not going into the details here.
create a local dist package python setup.py sdist --format=tar. the package created should ideally be in the dist folder.
Install the package on your virtualenv (after activating it). pip install <yourpackage>.tar
you can use pip install --force-reinstall if you need to play around with the libraries more and re-create the dist packages.
I've found that this method works great for me. If you do not need to package the modules for use of other systems instead of just your local, this method might be an overkill
Happy hacking.
If you're developing a module I would recommend to follow this.
import sys
sys.path.append("/home/mylinux/python-packages")
Now any module you keep in python-packages is importable by Python-interpreter.
On my Mac, I did a sudo find / -name "site-packages". That gave me a few paths like /Library/Python/2.6/site-packages, /Library/Python/2.7/site-packages, and /opt/X11/lib/python2.6/site-packages.
So, I knew where to put my modules if I was using v2.7 or v2.6.
Hope it helps.
import folders could be extracted by adding following source code:
import sys
print sys.path
automatic symlink generation example would be:
ln -s \`pip show em | grep "Location"| cut -d " " -f2\` \`python -m site --user-site\`
instead of "em" you may use other package you've "just installed but the python can't see it"
below I'll explain in more details as being requested in the comment.
suppose you've installed python module em or pyserial with the following command (examples are for ubuntu):
sudo pip install pyserial
and the output is like this:
Collecting pyserial
Downloading pyserial-3.3-py2.py3-none-any.whl (189kB)
100% |████████████████████████████████| 194kB 2.3MB/s
Installing collected packages: pyserial
Successfully installed pyserial-3.3
the question would be following - python can't see the module pyserial, why?
because the location where the module has been installed isn't the one python is looking at for your particular user account.
solution - we have to create symlink from the path where pyserial arrived to the path where your python is looking for.
symlink creation command would be:
ln -s <what_to_link> <where_to_link>
instead of typing exact location we are asking pip to tell us where it stored modules by executing command:
pip show pyserial | grep "Location"| cut -d " " -f2
instead of typing exact location we are asking python to tell us where it looks for the modules being installed by executing command:
python -m site --user-site
both commands has to be escaped with "`" character (usually on the left of your 1 button for the US keyboards)
in result following command will be provided for ln and the missing symlink would be created:
ln -s /usr/local/lib/python2.7/dist-packages /home/<your_username>/.local/lib/python2.7/site-packages
or something similar, depending on your distro and python/pip defaults.
We're using a requirements.txt file to store all the external modules needed. Every module but one is gathered from internet. The other one is stored on a folder under the one holding the requirements.txt file.
BTW, this module can be easily installed with pip install
I've tried using this:
file:folder/module
or this:
file:./folder/module
or even this:
folder/module
but always throws me an error.
Does anyone know which is the right way to do this?
Thanks
In the current version of pip (1.2.1) the way relative paths in a requirements file are interpreted is ambiguous and semi-broken. There is an open issue on the pip repository which explains the various problems and ambiguities in greater detail:
https://github.com/pypa/pip/issues/328
Long story short the current implementation does not match the description in the pip documentation, so as of this writing there is no consistent and reliable way to use relative paths in requirements.txt.
THAT SAID, placing the following in my requirements.txt:
./foo/bar/mymodule
works when there is a setup.py at the top level of the mymodule directory. Note the lack of the file:: protocol designation and the inclusion of the leading ./. This path is not relative to the requirements.txt file, but rather to the current working directory. Therefore it is necessary to navigate into the same directory as the requirements.txt and then run the command:
pip install -r requirements.txt
Its based off the current working directory (find with os.getcwd() if needed) and the relative path you provide in the requirements file.
Your requirements file should look like this:
fabric==1.13.1
./some_fir/some_package.whl
packaging==16.8
Note this will only work for .whl files not .exe
Remember to keep an eye on the pip install output for errors.
For me only the file: directive worked. This even works with AWS SAM, i.e. sam build. Here is my requirements.txt and the englishapps is my own custom Python package that I need in AWS Lambda
requests
file:englishapps-0.0.1-py3-none-any.whl
As was mentioned before, the files are relative to the current working directory, not the requirement.txt.
Since v10.0 requirements files support environment variables in the format: ${VAR_NAME}. This could be used as a mechanism to specify a file location relative to the requirements.txt. For example:
# Set REQUIREMENTS_DIRECTORY outside of pip
${REQUIREMENTS_DIRECTORY}/folder/module
Another option is to use the environment manager called Pipenv to manage this use case.
The steps after you do the pipenv install for a new project:
pipenv install -e app/deps/fastai (-e is editable, and is optional)
then you will see the following line in your Pipfile:
fastai = {editable = true,path = "./app/deps/fastai"}
here's similar issues:
https://github.com/pypa/pipenv/issues/209#issuecomment-337409290
https://stackoverflow.com/a/53507226/7032846
A solution that worked for me both for local and remote files (via Windows share). Here an example of requirements.txt
file:////REMOTE_SERVER/FOLDER/myfile.whl
I have the following folder structure
lib
my_module
I have moved all the libraries I need into the lib/ folder.
In my module/__init__.py, I think I will do:
import sys
sys.path.append('../lib/')
import my_dependency
Then when I need to use this dependency, I will refer to it as
my_module.my_dependency
Is this a bad usage of Python import?
NOTE: the dependencies consists of some third-party libraries not available via pip/easy_install and some C++ stuff that I wrote.
sys.path.append('../lib/') assumes that the current working directory is the directory of your script, which may or may not be the case.
A version that doesn't depend on the working directory is:
import sys, os
sys.path.append(os.path.join(os.path.split(os.path.split(os.path.abspath(sys.argv[0]))[0])[0], "lib"))
import my_dependency
The above in plain language takes the full path to the script, chops off the last two components (script directory and script filename) and appends lib.
If the libraries you use are third-party modules—especially popular ones—namespacing them like that is going to get inconsistent pretty fast. You will end up with code that sometimes is referenced as bar and sometimes as foo.bar. Maintaining such a codebase will not be worth whatever gains you expect to get from prefixing them like that.
If you keep third-party code in your own repository, consider replacing it with a requirements.txt file that can be fed to utilities like easy_install and pip. Both these tools support a --user switch that installs to your home directory (not touching system stuff).
pip install -r requirements.txt --user
The question is related to the answer to "Unable to install Python without sudo access".
I need to install python-setuptools to install python modules.
I have extracted the installation package.
I get the following error when configuring
[~/wepapps/pythonModules/setuptools-0.6c9]# ./configure --prefix=/home/masi/.local
-bash: ./configure: No such file or directory
I did not find the solution at the program's homepage.
How can I resolve this error?
As Noah states, setuptools isn't an automake package so doesn't use ‘./configure’. Instead it's a pure-Python-style ‘setup.py’ (distutils) script.
You shouldn't normally need to play with .pydistutils.cfg, as long as you run it with the right version of Python. So if you haven't added the .local/bin folder to PATH, you'd have to say explicitly:
/home/masi/.local/bin/python setup.py install
AIUI this should Just Work.
I did not find the solution at the program's homepage.
Yeah, they want you to install it from a shell script egg which uses the default version of Python. Which you don't want.
(Another approach if you can't get setuptools to work is to skip it and install each module and dependency manually. Personally I have a bit of an aversion to setuptools/egg, as it contains far too much “clever” magic for my tastes and makes a mess of my filesystem. But I'm an old curmudgeon like that. Most Python modules can be obtained as simple Python files or plain old distutils scripts, but unfortunately there are some that demand eggs.)
You might want to check http://peak.telecommunity.com/DevCenter/EasyInstall#custom-installation-locations.
EasyInstall is a python module with some shell scripts (or some shell scripts with a python module?) and does not use the unix make tool that gets configured with the "./configure" command. It looks like your best bet is to try editing ~/.pydistutils.cfg to include:
[install]
install_lib = /home/masi/.local/lib/python/site-packages/
install_scripts = /home/masi/.local/bin
You'll also presumably have made the ~/.local/bin/ folder part of your PATH so you can run the easy_install script. (I'm not sure exactly where the site-packages directory will be under .local, but it shouldn't be hard to find.)
Hope this helps.