I'd like to collect all the functions I wrote in Python and store them in a folder. I'm an Ubuntu user, what environmental path do I have to add in my ~/.profile?
I tried
export PATH:$PATH:/home/functionFolder
or
export PYTHONPATH:/home/functionFolder
I also added an init.py file in the /home/functionFolder, but it doesn't work.
My aim is to import the functions with
from myFunctions import function1
with myFunctions located in /home/functionFolder.
Do not mess with PYTHONPATH. It will lead to hard-to-debug situations, non-portable code and general misery. (If you really, really want to, you could mess with it within your program with e.g. sys.path.insert(0, '...'), but that's also non-portable.)
If you want to maintain a personal toolbox library, it's better to just make it a package you can install. This will also pave the way to making it distributable later on should you want to.
The canonical guide to packaging Python libraries lives here at packaging.python.org but the very simplest tl;dr edition (at the time of writing) is:
This assumes
you have a project directory, e.g. /home/me/project
there is a package directory in the project directory, e.g. if your package is named myFunctions, you have a myFunctions/__init__.py
you know how to use virtualenvs for your other projects
Add pyproject.toml to /home/me/project with the contents listed below.
Add setup.cfg to /home/me/project with the contents listed below.
Now, in a project virtualenv where you want to use myFunctions, run pip install -e /home/me/project. This will install the package that lives in that path in editable mode, i.e. just link that folder into the project's environment.
pyproject.toml
[build-system]
requires = [
"setuptools>=42",
"wheel"
]
build-backend = "setuptools.build_meta"
setup.cfg
[metadata]
name = myFunctions
[options]
packages = find:
You can do it by adding the following commands to your .bashrc file which is located in ~/:
PYTHONPATH=/home/functionFolder/:$PYTHONPATH
export PYTHONPATH
once you have added the above commands in the .bashrc file, save it and type following in your console:
source ~/.bashrc
afterward, you can access and import your functions anywhere.
Python makes uses of a specific user-related directory for storing packages and modules that are installed by and for that user only (that is, not system-wide). The place for modules and packages is
$HOME/.local/lib/pythonx.y/site-packages/<module-or-package>
Where x.y denotes the relevant Python version, e.g. 3.9, and <module-or-package> is the .py file for a module, and a directory for the package. (Note that this means you can use multiple minor Python versions next to each other without them interfering, but you do have install packages for each minor Python version separately.)
This directory is automatically picked up by Python when the relevant user uses Python; no need to involve PYTHONPATH.
pip also installs into this directory, when you specify the --user flag; or when pip finds it can't install in the system directory and it will use the $HOME/.local directory instead.
If you have installable packages, you can even use pip install . --user or similar to install the packages properly in $HOME/.local
Related
I have a python project with multiple scripts (scriptA, scriptB, scriptC) that must be able to find packages located in subpath of the project that is not a python package or module. The organization is like so:
|_project
|_scriptA.py
|_scriptB.py
|_scriptC.py
|_thrift_packages
|_gen-py
|_thriftA
|__init__.py
|_thriftA.py
On a per script basis I am adding the absolute path to this directory to sys.path
Is there a way that I can alter the PYTHONPATH or sys.path every time a script is executed within the project so that I do not have to append the path to this directory to sys.path on a per-script basis?
You have an XY problem, albeit an understandable one since the "proper" way to develop a Python project is not obvious, and there aren't a lot of good guides to getting started (there are also differing opinions on this, especially in the details, though I think what I'll describe here is fairly commonplace).
First, at the root of your project, you can create a setup.py. These days this file can just be a stub; eventually the need for it should go away entirely, but some tools still require it:
$ cat setup.py
#!/usr/bin/env python
from setuptools import setup
setup()
Then create a setup.cfg. For most Python projects this should be sufficient--you only need to put additional code in setup.py for special cases. Here's a template for a more sophisticated setup.cfg, but for your project you need at a minimum:
$ cat setup.cfg
[metadata]
name = project
version = 0.1.0
[options]
package_dir =
=thrift_packages/gen-py
packages = find:
Now create and activate a virtual environment for your project (going in-depth into virtual environments will be out of scope for this answer but I'm happy to answer follow-up questions).
$ mkdir -p ~/.virtualenvs
$ python3 -m venv ~/.virtualenvs/thrift
$ source ~/.virtualenvs/thrift/bin/activate
Now you should have a prompt that look something like (thrift) $ indicating that you're in an isolated virtualenv. Here you can install any dependencies for your package, as well as the package itself, without interfering with your main Python installation.
Now install your package in "editable" mode, meaning that the path to the sources you're actively developing on will automatically be added to sys.path when you run Python (including your top-level scripts):
$ pip install -e .
If you then start Python and import your package you can see, for example, something like:
$ python -c 'import thriftA'; print(thriftA)
<module 'thriftA' from '/path/to/your/source/code/project/thrift_packages/gen-py/thriftA/__init__.py'>
If this seems like too much trouble, trust me, it isn't. Once you get the hang of this (and there are several project templates, e.g. made with cookie-cutter to take the thinking out of it) you'll see that it makes things like paths less trouble). This is how I start any non-trivial project (anything more than a single file); if you set everything up properly you'll never have to worry about fussing with sys.path or $PYTHONPATH manually.
In this guide, although the first part is a bit application-specific, if you ignore the specific purpose of this code a lot of it is actually pretty generic, especially the section titled "Packaging our package", which repeats some of this advice in more detail.
As an aside, if you're already using conda you don't need to create a virtualenv, as conda environments are just fancy virtualenvs.
An advantage to doing this the "right" way, is that when it comes time to install your package, whether by yourself, or by users, if your setup.cfg and setup.py are set up properly, then all users have to do is run pip install . (without the -e) and it should work the same way.
You should add __init__.py in each package, and then properly call all your script.
|_project
|_scriptA.py
|_scriptB.py
|_scriptC.py
|__init__.py <====== Here
|_thrift_packages
|__init__.py <====== Here
|_gen-py
|_thriftA
|__init__.py
|_thriftA.py
Assuming that, project is in your pythonpath, you can do:
from project.thrift_packages.thriftA import thriftA
yes you can.
However I think the other answers are more adapted to your current issue.
Here I just explain, how to call subprocesses with another environment, another current working directory.
This can come in handy in other situations.
You can get the current environment as a dict (os.environ) copy it to another dict, modify it and pass it to a supbprocess call. (subprocess functions have an env parameter)
import os
import subprocess
new_env = dict(os.environ)
new_env["PYTHONPATH"] = whateveryouwanthere # add here the pythonpath that you want
cmd = ["mycmd.py", "myarg1", "myarg2"] # replace with your command
subprocess.call(cmd, env=new_env) #
# or alternatively if you also want to change the current directory
# subprocess.call(cmd, env=new_env, cwd=path_of_your_choice) #
I have downloaded a python package to install, on my ubuntu machine. The package has already a setup.py file to use, but I want to change the default python installation address to something else, for this package specifically (and not for good). So what I tried is:
First in the terminal, I export that address of the new folder:
export PYTHONPATH=${PYTHONPATH}:${HOME}/Documents/testfolder/lib/python2.7/site-packages
Then I add this exported address as prefix to the installation command:
python setup.py install --prefix=~/Documents/testfolder
The installation goes through. Now to make python always look for this new path as well (next to the default installation path), I export the address in bashrc file:
export PYTHONPATH="${PYTHONPATH}:~/Documents/testfolder/lib/python2.7/site-packages"
But now whenever I open a terminal and try to import the installed package, it cannot see ("no module named..."). Only when I open a terminal in the folder where I had the installation files (namely setup.py), and run python, can it then see the package, and it works there.
Why isn't my export in bashrc making the package available from anywhere?
Is there something I have done wrong in the above?
To answer your question about the export path. Do you have $PYTHONPATH as a part of your $PATH? If not you should add it to path.
The best way to handle this scenario in my opinion is to use a virtual python environment. There are a couple to choose from, but I like virtualenv the best. The reason to take this approach is because you can manage different versions of python in separate folders. And have separate packages installed in these folders. I recommend looking into it as it is a very useful tool. If you want an examole of how to use it i can provide that https://virtualenv.pypa.io/en/stable/
In my home, I have a directory named lib/python2.7 (there are actually five directories like that, for different python versions). Since this is a shared hosting (Webfaction), that directory is fundamental to me. There, I have stuff like virtualenv and virtualenvwrapper installed, since as customer of a shared hosting, I have no access to sudo and installing global packages.
However, when I create a virtualenv:
$ mkvirtualenv myenvironment
$ workon myenvironment
$ which pip
# outputs the myenvironment's path to pip
$ pip freeze
The command shows the whole packages list under my lib/python2.7 (this includes the same virtualenv packages, and conflicting packages I have due to... legacy... reasons). This also annoys me if I want to install a package which is the name of a package in lib/python2.7 since it does not allow me to update it.
Right inside the workon environment, I try to check whether the PYTHONPATH has weird stuff, but it is empty:
$ echo $PYTHONPATH
# shows a blank line
It is also empty if I try that command out of any virtual environment.
It seems that --no-site-packages is default but solves just part of the problem. This means: pip freeze | wc -l displays a lesser value when in an environment than when executing globally, out of any environment, which tells me that there are certain already-provided packages that are being excluded and are from the hosting itself (and not installed by me since, again, the hosting is shared and I don't have access to global space).
My question is: How can I solve this? I want my virtualenv not list the packages in $HOME/lib/python2.7
Please avoid dupe-linking to this question, nothing was useful there and still does not have an accepted answer. I wrote this question after reading and trying each solution in that question
I think you need to specify python version. You can specify python version with which you want to create virtual environment using command like
virtualenv -p /usr/bin/python3.4 virt/virtname --no-site-packages
Because when you not specify a python version, virtualenv creates a environment with pythonv2.7 and hence all packages end up in the folder you mentioned.
Found the solution after deeply digging. This is a Webfaction custom but this could apply to any installation like this if the problem occurs.
The core of the problem is that Webfaction configured a sitecustomize.py file for us. The file is located at /usr/local/lib/pythonX.Y/sitecustomize.py and adds by itself the contents of ~/lib/pythonX.Y (and conditionally the contents of any python app under ~/webapps if you are working under a directory of it to run a python script).
Even when the virtualenv's python executable is a different one, it will load the said sitecustomize.py file each time it runs as the base python executable does.
The workaround here? Create an empty sitecustomize.py in your virtualenv to override the other:
touch ~/.virtualenvs/yourvenv/lib/pythonX.Y/sitecustomize.py
And it will work. Take this as reference if you are stuck here like I was
Notes: Replace X.Y on each case with the corresponding version you are working. Additionally remember: You cannot remove or edit the base sitecustomize.py since you are in a shared hosting, in this case. However, overriding will work for each case as long as you do this for every virtualenv you want.
I am currently writing a command line application in Python, which needs to be made available to end users in such a way that it is very easy to download and run. For those on Windows, who may not have Python (2.7) installed, I intend to use PyInstaller to generate a self-contained Windows executable. Users will then be able to simply download "myapp.exe" and run myapp.exe [ARGUMENTS].
I would also like to provide a (smaller) download for users (on various platforms) who already have Python installed. One option is to put all of my code into a single .py file, "myapp.py" (beginning with #! /usr/bin/env python), and make this available. This could be downloaded, then run using myapp.py [ARGUMENTS] or python myapp.py [ARGUMENTS]. However, restricting my application to a single .py file has several downsides, including limiting my ability to organize the code and making it difficult to use third-party dependencies.
Instead I would like to distribute the contents of several files of my own code, plus some (pure Python) dependencies. Are there any tools which can package all of this into a single file, which can easily be downloaded and run using an existing Python installation?
Edit: Note that I need these applications to be easy for end users to run. They are not likely to have pip installed, nor anything else which is outside the Python core. Using PyInstaller, I can generate a file which these users can download from the web and run with one command (or, if there are no arguments, simply by double-clicking). Is there a way to achieve this ease-of-use without using PyInstaller (i.e. without redundantly bundling the Python runtime)?
I don't like the single file idea because it becomes a maintenance burden. I would explore an approach like the one below.
I've become a big fan of Python's virtual environments because it allows you to silo your application dependencies from the OS's installation. Imagine a scenario where the application you are currently looking to distribute uses a Python package requests v1.0. Some time later you create another application you want to distribute that uses requests v2.3. You may end up with version conflicts on a system where you want to install both applications side-by-side. Virtual environments solve this problem as each application would have its own package location.
Creating a virtual environment is easy. Once you have virtualenv installed, it's simply a matter of running, for example, virtualenv /opt/application/env. Now you have an isolated python environment for your application. Additionally, virtual environments are very easy to clean up, simply remove the env directory and you're done.
You'll need a setup.py file to install your application into the environment. Say your application uses requests v2.3.0, your custom code is in a package called acme, and your script is called phone_home. Your directory structure looks like this:
acme/
__init__.py
models.py
actions.py
scripts/
phone_home
setup.py
The setup.py would look something like this:
from distutils.core import setup
install_requires = [
'requests==2.3.0',
]
setup(name='phone_home',
version='0.0.1',
description='Sample application to phone home',
author='John Doe',
author_email='john#doe.com',
packages=['acme'],
scripts=['scripts/phone_home'],
url='http://acme.com/phone_home',
install_requires=install_requires,
)
You can now make a tarball out of your project and host it however you wish (your own web server, S3, etc.):
tar cvzf phone_home-0.0.1.tar.gz .
Finally, you can use pip to install your package into the virtual environment you created:
/opt/application/env/bin/pip install http://acme.com/phone_home-0.0.1.tar.gz
You can then run phone_home with:
/opt/application/env/bin/phone_home
Or create a symlink in /usr/local/bin to simply call the script using phone_home:
ln -s /opt/application/env/bin/phone_home /usr/local/bin/phone_home
All of the steps above can be put in a shell script, which would make the process a single-command install.
And with slight modification this approach works really well for development environments; i.e. using pip to install / reference your development directory: pip install -e . where . refers to the current directory and you should be in your project directory alongside setup.py.
Hope this helps!
You could use pip as suggested in the comments. You need to create a MANIFEST.in and setup.py in your project to make it installable. You can also add modules as prerequisites. More info can be found in this question (not specific to Django):
How do I package a python application to make it pip-installable?
This will make your module available in Python. You can then have users run a file that runs your module, by either python path/run.py, ./path/run.py (with +x permission) or python -c "some code here" (e.g. for an alias).
You can even have users install from a git public reporitory, like this
pip install git+https://bitbucket.org/yourname/projectname.git
...in which case they also need git.
I have been programming in Python for a while now, and have created some utilities that I use a lot. Whenever I start a new project, I start writing, and as I need these utilities I copy them from where ever I think the latest version of the particular utility is. I have enough projects now that I am losing track of where the latest version is. And, I will upgrade one of these scripts to fix a problem in a specific situation, and then wish it had propagated back to all of the other projects that use that script.
I am thinking the best way to solve this problem is to create a directory in the site-packages directory, and put all of my utility modules in there. And then add this directory to the sys.path directory list.
Is this the best way to solve this problem?
How do modify my installation of Python so that this directory is always added to sys.path, and I don't have to explicitly modify sys.path at the beginning of each module that needs to use these utilities?
I'm using Python 2.5 on Windows XP, and Wing IDE.
The site-packages directory within the Python lib directory should always be added to sys.path, so you shouldn't need to modify anything to take care of that. That's actually just what I'd recommend, that you make yourself a Python package within that directory and put your code in there.
Actually, something you might consider is packaging up your utilities using distutils. All that entails is basically creating a setup.py file in the root of the folder tree where you keep your utility code. The distutils documentation that I just linked to describes what should go in setup.py. Then, from within that directory, run
python setup.py install
to install your utility code into the system site-packages directory, creating the necessary folder structure automatically. Or you can use
python setup.py install --user
to install it into a site-packages folder in your own user account.
Add your directory to the PYTHONPATH environment variable. For windows, see these directions.
If it's not in site-packages then you can add a file with the extension .pth to your site-packages directory.
The file should have one path per line, that you want included in sys.path