I'm working on a script in python that relies on several different packages and libraries. When this script is transferred to another machine, the packages it needs in order to run are sometimes not present or are older versions that do not have the same functionality and cause the script to fail.
I was considering using a virtual environment, but I can't find a way to have the script use the specific environment I design as it's default, and in order to use the environment a user must manually activate it from the command line.
I've also looked into trying to check the versions of the packages installed on the machine, and if they are not sufficient then updating them from the script as described here:
Installing python module within code
Is there any easier/surefire way to make sure that the needed packages will always be available regardless of where it's run?
The normal approach is to create an installation script and have that manage your dependencies. Then when you move your project to a new environment your installer will check that all dependencies are present.
I recommend you check out setuptools: https://setuptools.readthedocs.io/en/latest/
If you don't want to install dependencies whenever you need to use your script somewhere new, then you could package your script into a Docker container.
If the problem is ensuring the required packages are available in a new environment or virtual environment, you could use pip and generate a requirements.txt and check it in version control or use a tool to do that for you, like pipenv.
If you would prefer to generate a requirements.txt by hand, you should:
Install your depencencies using pip
Type pip freeze > requirements.txt to generate a requirements.txt file
Check requirements.txt in you source management software
When you need to setup a new environment, use pip install -m requirements.txt
The solution that I've been using has been to include a custom library (folder with all of my desired packages) in the folder with my script, and I simply import them from there:
from Customlib import pkg1, pkg2,...
As long as the custom library and script stay together in the same folder, it will always have access to the right packages and the correct versions of those packages.
I'm not sure how robust this solution actually is or what possible bugs may arise from this if it is passed from machine to machine, but for now this seems to work.
Related
I have a Jupyter notebook script that will be used to teach others how to use python.
Instead of asking each participant to install the required packages, I would like to provide a folder with the environment ready from the start.
How can I do this?
What is the easiest way to teach python without running into technical problems with packages/environments etc.?
If you just need to install Python dependencies, you can use #Aero Blue solution. However, the users would need probably to make a virtual environment, so they don't mess with other environments and versions, etc.
However, if they should need some Linux packages, this would not be enough. Therefore, I would suggest using Docker. You would need to provide them with a Dockerfile, that you should set to install any dependencies (whether is for Python or Linux), and they would just need to use docker build and docker run commands.
The easiest way I have found to package python files is to use pyinstaller which packages your python file into an executable file.
If it's a single file I usually run pyinstaller main.py --onefile
Another option is to have a requirements file
This reduces installing all packages to one command pip install -r requirements.txt
You would need to use a program such as py2exe, pyinstaller, or cx_freeze to package each the file, the modules, and a lightweight interpreter. The result will be an executable which does not require the user to have any modules or even python installed to access it; however, because of the built-in interpreter, it can get quite large (which is why Python is not commonly used to make executables).
Have you considered using Azure notebooks or another Jupyter hosting service ? Most of these have a special syntax you can use to perform pip installs. For Azure it is !pip install
https://notebooks.azure.com
I would like to easily export one Python project from one PC to other. When I created the project, I used a virtual environment in order to avoid problems with different package versions.
What I did was to just copy the project folder and paste it in the destination PC. Once I opened the project with Pycharm, I activated the virtual environment with project_path/venv/Scripts/activate, but when I tried to execute any Script, it said it didn´t find the modules.
Which is the workflow I should follow in order to create projects and be able to run them from multiple PC-s without needing to install all the dependencies?
Since you did not specify your Python version I will provide a solution working for both Python 2.x and 3.x.
My suggestion is to create a requirements.txt file containing all your requirements.
This file can be easily prepared using the output from the command:
pip freeze
Then you can paste the output in your requirements.txt file and when you are going to install your Python code on another PC you can simply:
pip install -r requirements.txt
To install your requirements again.
Depending on your project it could be possible, for example, to create a single EXE file (if you are using Windows machines) but more detailed is needed if this is the case.
In case you are using Python 3 the method that is at the moment arguably more popular in the Python community is Pipenv.
Here's its relevant documentation.
And here you can read a simple example of a workflow.
if you are using python3 then use pipenv. It will automatically create Pipfile and Pipfile.lock. That will insure reinstalling dependencies on different machine will have the same packages.
basic and helpful commands:
pipenv shell # activate virutalenv
pipenv install # will install dependencies in Pipfile
pipenv install requests # will install requests lib. and will auto update Pipfile and Pipfile.lock
I've been working on a Python project for a while which uses quite some third party libraries. I want to deploy my project to another server, but I don't know by heart which projects I use and digging through every line of source code would be too much work.
Is there a way to generate a list of third party modules in my project so I can use this with PIP's installer? Thank you for your help
pip install -r dependencies.txt # <-- I need to generate this file from a project
Provided that you're using a virtual environment to keep your dependencies separate from the globally installed pip packages, you should be able to use pip's freeze command, like so:
pip freeze > dependencies.txt
If you haven't been using a virtual environment, then you probably need to peruse the source code to find the modules. A virtual environment is a means of keeping your python project isolated from the global environment, meaning that you can only import modules that are installed within that environment, and it should only contain modules relevant to its corresponding python project. I recommend that you read up on Virtual Environments, they are very useful for larger projects.
I ended up writing a python module which does this instead as I couldn't find it. The source code is available on GitHub. You can install it like so:
$ pip install pip-module-scanner
Using it is pretty simple, for full usage examples check the GitHub repo.
$ pip-module-scanner
foo==0.0.1
bar==2.0.0
I want to contribute some changes to a python package which is using github. I have forked it.
It is a library I am using in a project (in a python 3.5.1 virtual environment).
The documentation at https://pip.pypa.io/en/latest/reference/pip_install/#vcs-support tell me how to install from a github fork, and it goes on to mention "editable installs" (https://pip.pypa.io/en/latest/reference/pip_install/#editable-installs) which basically does "development mode"
If it is a pure python package does it matter if I skip editable mode?
(since there would be no build steps necessary as I would only be changing python code. This would mean I can keep using the same import statements.)
When you pip install without editable mode, the package is copied into your Python environment (such as env/lib/python3.5/site-packages). You can, of course, edit it right there, as it’s usually just a bunch of Python files, but that is inconvenient.
When you pip install with editable mode, pip only sets up a link from your environment to wherever the source code is. So, you can clone your GitHub fork into a convenient directory like ~/projects/libraryX, then do pip install -e ~/projects/libraryX, and keep editing the code at ~/projects/libraryX while your changes are immediately reflected in the environment where you installed it.
This all applies to pure Python packages.
I installed some package via pip install something. I want to edit the source code for the package something. Where is it (on ubuntu 12.04) and how do I make it reload each time I edit the source code and run it?
Currently I am editing the source code, and then running python setup.py again and again, which turns out to be quite a hassle.
You should never edit an installed package. Instead, install a forked version of package.
If you need to edit the code frequently, DO NOT install the package via pip install something and edit the code in '.../site_packages/...'
Instead, put the source code under a development directory, and install it with
$ python setup.py develop
or
$ pip install -e path/to/SomePackage
Or use a vcs at the first place
$ pip install -e git+https://github.com/lakshmivyas/hyde.git#egg=hyde
Put your changes in a version control system, and tell pip to install it explicitly.
Reference:
Edit mode
You can edit the files installed in /usr/local/lib/python2.7/dist-packages/. Do note that you will have to use sudo or become root.
The better option would be to use virtual environment for your development. Then you can edit the files installed with your permissions inside your virtual environment and only affect the current project.
In this case the files are in ./venv/lib/pythonX.Y/site-packages
The path could be dist-packages or site-packages, you can read more in the answer to this question
Note that, as the rest of the people have mentioned, this should only be used sparingly, for small tests or debug, and being sure to revert your changes to prevent issues when upgrading the package.
To properly apply a change to the package (a fix or a new feature) go for the options described in other answers to contribute to the repo or fork it.
I too needed to change some things inside a package. Taking inspiration from the previous answers, You can do the following.
Fork the package/repo to your GitHub
clone your forked version and create a new branch of your choice
make changes and push code to the new branch on your repository
you can easily use pip install -e git+repositoryurl#branchname
There are certain things to consider if its a private repository
If you are doing the custom module that you want hot loading, you can put your running code also inside the module. Then you can use python -m package.your_running_code. In this way, you can change the module in the package and reflect the result of your running code immediately.