I'm developing a python project on my desktop (OS X) and it's running well. Now I need to run it on a computing cluster that is running Linux and I have no root access. On my Home in the computing cluster I installed Anaconda locally. Then when I run my project there I got so many strange bugs and errors. THe code runs fine on my desktop. So I thought probably the problem is that probably Anaconda has newr modules than what I have and that's probably causing errors. (backward compatibility issues)
Is there any way to use the same python modules versions I have in my desktop on the computing cluster?
I wonder if there is a command that produces all the packages and modules I have on my desktop with their versions and then import this file into some tool in that linux machine and then I get a cloned version of my setup. Is there something like that?
Use pyenv to control the python version
pyenv is designed for installing python into the home folder so you don't need root access (but even with root access it's a great idea), it lets you use a specified version of python within your home folder hierarchy rather than the OS version. You should also install pyenv virtualenv so you can use a virtual environment for your project (not strictly essential, but virtual environments are a great idea and you should always use one, with pyenv they're practically effortless).
One of the neat things about pyenv is the pyenv local command which specifies which version of python (or which virtualenv) should be used for a folder (and subfolders), once you've used pyenv local in your project folder to set the python version any time you use the python command it'll use the version set by pyenv local. It's not needed if you only install one version of python, and don't use a virtualenv (in that case you can use pyenv global to set the version for the user). The neatest thing about pyenv local/global setup is it works great with both scripts and manually invoking python, it is simply set-and-forget unlike other python environment managers which require activation.
In brief, once you've set up pyenv you control exactly what version of python will run, and as it is installed into the home folder the OS has no power to affect it.
Having installed pyenv and pyenv virtualenv, you would then use it to install the same version of python as is used on your development machine, the commands you'd run would be something like this:
pyenv install 3.4.2
pyenv virtualenv 3.4.2 my_project_env
cd my_project
pyenv local my_project_env
Install modules into python environment
To get a list of python module version use pip freeze, you can do this on your development machine:
pip freeze > requirements.txt
Now copy requirements.txt to your deployment machines (with pyenv already setup using pyenv local or pyenv global) and run:
pip install -r requirements.txt
Which will install the same modules into the python environment.
Duplicating a pyenv
While it's kind of dodgy, once you've done this, you can even copy the entire installation (i.e. at least the .bashrc file and .pyenv folder) onto other machines, and if the machines have the same OS and the home folders have the same name, the transplanted environment should work fine. It might be more responsible to use a setup script but I have copied pyenv environments without anything terrible happening.
If you have pip installed you could try this pip freeze.
If you want to get list of modules from python shell:
help('modules')
Related
Overall, pip is working fine on my server. I have just installed the package waitress and the installation seems successful. I checked it with pip freeze:
$ pip freeze | grep waitress
waitress==2.1.0
Waitress also can be imported via python3:
>>> import waitress
>>>
However, waitress-serve cannot be executed:
$ waitress-serve
Command 'waitress-serve' not found, but can be installed with:
apt install python3-waitress
Please ask your administrator.
I am not a root user on this server. Could this be a reason why the package was installed partially, or am I speculating here?
Since I am not authorized to run apt install and since the simple pip install worked in my virtualenv, I would like to be able to get this to work without using the suggested apt install python3-waitress command.
The conclusion is that, while it is installed both inside and outside virtualenv, it is only actually executable inside it.
When installable Python packages give you an actual entry point, generally it will not be on your path.
When you use a virtual environment, activating the environment puts various parts of that environment onto the path temporarily. This is intended to ensure that commands like python or python3 run the environment's Python, but it also allows those entry points to be found on the path.
A system Python installation (here I mean, not just a Python that comes with your operating system, but also one that you install manually after the fact - but not a virtual environment) will generally not have its library folders on the path by default - only enough to make python and pip work. (On Windows, often even these are not added to the path; instead, a program py is placed in the Windows installation folder, and it does the work of looking for Python executables.) Even if you are allowed to install things directly into a system Python (and you should normally not do this if you can avoid it, even if you're allowed to), they won't be findable there.
Of course, you could execute these things just fine by explicitly specifying their paths. However, the normally correct approach is to just ensure that, when you want to run the program, the same virtual environment is activated into which you installed the package.
(On my system, I have one main "sandbox" virtual environment that I use for all my projects - unless I am specifically testing the installation process, or testing how the code works on a different version of Python. Then I use a wrapper script to open a terminal window, navigate to a folder that contains all my projects, and activate the environment.)
if waitress is installed inside a virtual environment, I think you may have accidentally (or not) gotten out of said virtual environment.
If you are running a virtual environment, you can try the following commands, one after the other:
source venv/bin/activate #venv is assumed to be the name of the virtual environment you are using.
pip install waitress
waitress-serve
anytime you need to use waitress you will need to activate the virtual environment yet again:
source venv/bin/activate
waitress-serve
Please take note I am assuming you are running in a Linux environment
If this is not the issue you are facing, then feel free to expaund a bit more on your question.
Edit: Installing with pip and running it worked perfectly on my virtualenv; see the picture below, running on python 3.8.10
I need python3.6 for tensorflow installation, so I downloaded python3.6.12.tar. And I found that I should pip install tarfile. However, in this case it is an older version of python. FYI, In my computer(laptop) I installed python3.9.
My question is: can I pip install python.tar inside a virtualenv?
This is not how virtual environments work. I suggest you to do a little bit more research on virtual environments in Python.
Virtual Environments and Packages
Basically you need to install the necessary python version onto your machine. Then go ahead and use that specific python (which is version 3.6 in your case), to create a virtual environment with the command
~ /usr/bin/<path-to-python3.6> -m venv venv
This command will create a folder called venv. Now you need to source the activation script inside this folder to activate your environment.
Handy note: if you are dealing with different versions of python, a more robust way of handling such situations is via using a tool called pyenv.
After moving over to Fedora (from Windows), I realized that it came with both installations of Python 2.7.5 and Python 3.6.6.
As I familiarized myself with using Python, I learned of the great utilities of virtual environments and how organized they keep everything.
However, my current dilemma is for which Python version should I do pip(2 or 3) install virtualenv virtualenvwrapper.
From my research, I understand that the virtualenvwrapper provides the ability to create a virtual environment using a specified version of Python: mkvirtualenv -p /usr/bin/python(2 or 3) {name}.
Therefore, should I only install virtualenv and virtualenvwrapper on one of the Python versions and use the aforementioned feature? Or should I install virtualenv and virtualenvwrapper on both versions of Python.
Would there be any conflicts?
Edit
More importantly, assuming that I have virtualenv and virtualenvwrapper installed for both Python 2.7.5 and Python 3.6.6, which version's command is called when I run any of the following: workon, mkvirtualenv, rmvirtualenv, etc.?
Would there be any conflicts?
Not until you mistakenly run the default system python command with a script that's using the opposite version as compared to the more specific python2 or python3 commands.
The virtualenvs do not conflict, and must be activated to be used. You can also of course have as many virtualenv's as you wish.
To avoid any problems setting up an environment, its suggested to run python2 -m virtualenv for example, rather than simply virtualenv command itself
For the commands listed at the bottom of the question, it depends on how your PATH is configured. Personally, I use pyenv rather than virtualenv directly, which injects itself into the OS PATH variable
I am working on a django project on two separate systems, Debian Jessie and Mac El Capitan. The project is hosted on github where both systems will pull from or push to.
However, I noticed that on my Debian, when I run python --version, it gives me Python 3.4.2 but on my Mac, it gives me Python 2.7.10 despite being in the same virtual environment. Moreover, when I run django-admin --version on my Debian, it gives me 1.10 while on my Mac, 1.8.3.
This happens even when I freshly clone the projects from github and run the commands.
Why is it that the virtual environment does not keep the same version of python and django?
Now you understand that virtual environments can't be transferred easily from machine to machine. It's common to use the
pip freeze
command and store its output in a file called requirements.txt. Then anyone else can rebuild your envirnment on their machine by running
pip install -r requirements.txt
When you create a new virtual environment you can say which Python interpreter you want to use with the -p or --python switch, which should be followed by the path to the correct executable.
I'd personally recommend against modifying the system Python in any way, because system maintenance routines often rely on its integrity. It's relatively simple to install new copies for Python 2 and 3 somewhere like /usr/local/bin (Mac users often use brew for this) and have virtual environments that rely on different Python interpreters.
Thanks to #Oliver and #Daniel's comments that lead me to the answer why it did not work.
I started the virtual environment on my Debian with python 3. virtualenv made the virtual environment but it was specifically for Debian.
When I used it for mac, since it could not run the python executable in the virtual environment (since it is only compatible with Debian), hence, it used my Mac's system python, which is Python 2.7.10.
In summary, as virtualenv uses the python executable on the system, when the python executable is run on another system, it will not work.
So I used virtualenv to define environments for a number of projects I am working on. I defined the virtualenv python as being version 3.4. Eventually, my global python was upgraded from 3.4.0 to 3.4.3. This proved to be a problem because the virtualenv was dependent on the global binaries (the contents of /lib/python3.4 in my virtualenv is actually just links to the global binaries), and these aren't defined up to their minor versions. In other words, when the upgrade was done, the contents of the binary folder /usr/lib/python3.4 was replaced. This is because python doesn't install things separately in 3.4.0 and 3.4.3 but only into a single folder named /usr/lib/python3.4. Since the python executable in my virtualenv was 3.4.0, there were obviously compatibility issues with the 3.4.3 binaries (it would fail to load ctypes which prevented just about anything python dependent to run). The only fix to this I've found is to downgrade my global python installation, but this feels "dirty". What if I had one project running 3.4.0 and another running 3.4.3 ? Is there no way to make them work in parallel on the same machine given that only one binary folder can exist for any 3.4.x installation ?
I'm trying to understand if I'm missing something obvious here or if this is a common problem with virtualenv, given that I've heard quite a few people complain about issues with binares when using virtualenv.
In the future, is there anyway of telling virtualenvwrapper to copy the binaries rather than link to them ?
Virtualenvs were not desiged to be portable, both across machines or across Python versions.
This means upgrading Python versions sometimes breaks virtualenvs. You need to recreate them and reinstall everything inside of it (run this in your virtualenv root):
# Save a list of what you had installed
pip freeze > freeze.txt
# Trash the entire virtualenv
deactivate
rm -rf lib/ bin/ share/ man/ include/ .Python pip-selfcheck.json
# Create it anew
virtualenv .
# Install all libraries you had before
pip install -r freeze.txt