I'd like to make a switch from Windows to Linux (Ubuntu) writing my python programs but I just can't get things to work. Here's the problem: I can see that there are quite the number of modules pre-installed (like numpy, pandas, matplotlib, etc.) in Ubuntu. They sit nicely in the /host/Python27/Lib/site-packages directory. But when I write a test python script and try to execute it, it gives me an ImportError whenever I try to import a module (for instance import numpy as np gives me ImportError: No module named numpy). When I type which python in the commandline I get the /usr/bin/python path. I think I might need to change things related to the python path, but I don't know how to do that.
You can use the following command in your terminal to see what folders are in your PYTHONPATH.
python -c "import sys, pprint; pprint.pprint(sys.path)"
I'm guessing /host/Python27/Lib/site-packages wont be in there (it doesn't sound like a normal python path. How did you install these packages?).
If you want to add folders to your PYTHONPATH then use the following:
export PYTHONPATH=$PYTHONPATH:/host/Python27/Lib/site-packages
Personally here are some recommendations for developing with Python:
Use virtualenv. It is a very powerful tool that creates sandboxed python environments so you can install modules and keep them separate from the main interpreter.
Use pip - When you've created a virtualenv, and activated it you can use pip install to install packages for you. e.g. pip install numpy will install numpy into your virtual environment and will be accessible from only this virtualenv. This means you can also install different versions for testing etc. Very powerful. I would recommend using pip to install your python packages over using ubuntu apt-get install as you are more likely to get the newer versions of modules (apt-get relies on someone packaging the latest versions of your python libraries and may not be available for as many libraries as pip).
When writing python scripts that you will make executable (chmod +x my_python_script.py) make sure you put #!/usr/bin/env python at the top as this will pick up the python interpreter in your virtual environment. If you don't (and put #!/usr/bin/python) then running ./my_python_script.py will always use the system python interpreter.
/host/Python27/Lib/site-packages is not a default python directory on linux installations as far as I am aware.
The normal python installation (and python packages) should be found under /usr/lib or /usr/lib64 depending on your processor architecture.
If you want to check where python is searching in addition to these directories you can use a terminal with the following command:
echo $PYTHONPATH
If the /host/Python27/Lib/site-packages path is not listed, attempt to use the following command and try it again:
export PYTHONPATH=$PYTHONPATH:host/Python27/Lib/site-packages
If this should work and you do not want to write this in a terminal every time you want to use these packages, simply put it into a file called .bashrc in your home folder (normally /home/<username>).
When installing other python libraries, specify the pip version you want to install it to, if it's python2 you use, then enter this syntax:
pip2 install <package>
For python3
pip3 install <package>
Related
I've been coding with R for quite a while but I want to start learning and using python more for its machine learning applications. However, I'm quite confused as to how to properly install packages and set up the whole working environment. Unlike R where I suppose most people just use RStudio and directly install packages with install.packages(), there seems to be a variety of ways this can be done in python, including pip install conda install and there is also the issue of doing it in the command prompt or one of the IDEs. I've downloaded python 3.8.5 and anaconda3 and some of my most burning questions right now are:
When to use which command for installing packages? (and also should I always do it in the command prompt aka cmd on windows instead of inside jupyter notebook)
How to navigate the cmd syntax/coding (for example the python documentation for installing packages has this piece of code: py -m pip install "SomeProject" but I am completely unfamiliar with this syntax and how to use it - so in the long run do I also have to learn what goes on in the command prompt or does most of the operations occur in the IDE and I mostly don't have to touch the cmd?)
How to set up a working directory of sorts (like setwd() in R) such that my .ipynb files can be saved to other directories or even better if I can just directly start my IDE from another file destination?
I've tried looking at some online resources but they mostly deal with coding basics and the python language instead of these technical aspects of the set up, so I would greatly appreciate some advice on how to navigate and set up the python working environment in general. Thanks a lot!
Python uses a different way of installing packages. Python has a thing named venv which stands for Virtual Environment. You install all of your packages in venv. Usually for each new project you make a new venv.
By using Anaconda on windows you install everything within the anaconda environment that you have specified.
python -m pip install "modulename" is a command that will install modulename to your default venv. You will be able to use this module when no other venv is specified. Here is the docs page. And here is a tutorial on how to use venv
By default python uses the same directory you have your code in. e.g. C:/Users/me/home/mypythonfile.py will run in C:/Users/me/home/ and will be able to access files in this directory. However you can use ../ to navigate directories or you can specify an absolute path to file you want to open e.g. with open("C:/system32/somesystemfile.sys") as file
Going over the technical differences of conda and pip:
So Conda is a packaging tool and installer that aims to do more than what pip does; handle library dependencies outside of the Python packages as well as the Python packages themselves. Both have many similar functionalities as well, you can install packages or create virtual environments with both.
It is generally advisable to generally have both conda and pip installed since there are some packages which might not be available with conda but with pip and vice versa.
The commands to install in both the ways is easy enough, but one thing to keep in mind is that
conda stores packages in the anaconda/pkgs directory
pip stores it in directory under /usr/local/bin/ for a Unix-based system, or \Program Files\ for Windows
You can use both pip or conda inside the jupyter notebook, it will work just fine, but it may be possible that you get multiple versions of the same package.
Most of the times, you will use cmd only to install a module used in your code, or to create environments, py -m pip install "SomeProject" here basically means that the module "SomeProject" will be downloaded in base env.
You could think of conda as python with a variety of additional functionalities, such as certain pre-installed packages and tools, such as spyder and jupyter. Hence, you must be precise when you say:
I've downloaded python 3.8.5 and anaconda3
Does it mean you installed python in your computer and then also anaconda?
In general, or at least in my opinion, using anaconda has advantages for development, but typically you'll just use a simple python installation in production (if that applies to you).
Anaconda has it's own package registry/repository . When you call conda install <package>, it will search for the package there and install it if available. You would better search it first, for instance matplotlib.
pip is a package manager for the Python Package Index. pip also ships with anaconda. Hence, in an anaconda environment you may install packages from either sources (either using pip install or conda install). For instance, pandas from PyPI and pandas from conda. There is no guarantee that packages exist in both sources. You must either search it first or simply try it.
In your first steps, I would suggest you to stick to only one dev env (either simple python or anaconda, recommend the second). Because that simplifies the question: "which python and which pip is executed in the cmd line?". That said, those commands should work as expected in any terminal, it be a simple cmd or an embedded one like in PyCharm or VS Code.
You could inspect that by running (on windows and linux at least):
which python, which pip.
Honestly, this is a question/answer that falls outside the scope of SO and for more info you would better check official websites, such as for anaconda or search for python vs anaconda blogs.
[On a mac]
I know I can get packages doing pip install etc.
But I'm not entirely sure how all this works.
Does it matter which folder my terminal is in when I write this command?
What happens if I write it in a specific folder?
Does it matter if I do pip/pip3?
I'm doing a project, which had a requirements file.
So I went to the folder the requirements txt was in and did pip install requirements, but there was a specific tensorflow version, which only works for python 3.7. So I did """python3.7 -m pip install requirements""" and it worked (i'm not sure why). Then I got jupyter with brew and ran a notebook which used one of the modules in the requirements file, but it says there is no such module.
I suspect packages are linked to specific versions of python and I need to be running that version of python with my notebook, but I'm really not sure how. Is there some better way to be setting up my environment than just blindley pip installing stuff in random folders?
I'm sorry if this is not a well formed question, I will fix it if you let me know how.
Yes, there is. Setup an virtual environment.
pip install virtualenv #installs the library
virtualenv mypython #creates the environment
source mypython/bin/activate #activates the environment
Now, install your requirements through pip.
Afterwards, when your work is finished.
Just type deactivate to come out of the virtual environment.
There may be a difference between pip and pip3, depending on what you have installed on your system. pip is likely the pip used for python2 while pip3 is used for python3.
The easiest way to tell is to simply execute python and see what version starts. python will run typically run the older version 2.x python and python3 is required to run python version 3.x. If you install into the python2 environment (using pip install or python -m pip install the libraries will be available to the python version that runs when you execute python. To install them into a python3 environment, use pip3 or python3 -m pip install.
Basically, pip is writing module components into a library path, where import <module> can find them. To do this for ALL users, use python3 or pip3 from the command line. To test it out, or use it on an individual basis, use a virtual environment as #Abhishek Verma said.
I try to install Theano by Anaconda. It works, but when I enter the python -i, import theano shows No module named 'theano'. Do I need to switch another interpreter of Python, how? Also, for the packages installed by conda, if I don't double install them, can I find in Python? How is Python related to Python by Anaconda? Thanks!!!
I had have a similar issue, trying to install folium. If you are using the Anaconda:
When you install using conda install -c conda-forge folium, the package will be placed in:
./anaconda3/envs/[name env]/lib/python3.7/site-packages/folium
When you install using pip (with a anaconda env activated), pip install folium, the package will be placed in:
./anaconda3/lib/python3.7/site-packages/folium
Python use first the sites-packages as the target directory of manually built python packages. When you build and install python packages from source (using distutils, probably by executing python setup.py install ), you will find the installed modules in site-packages by default.
In this case you have two places: /anaconda3/lib/python3.7/site-packages/ and /anaconda3/envs/[name env]/lib/python3.7/site-packages/.
First the modules will be available as default in /anaconda3/lib/python3.7/site-packages/. Sometimes (and I really don't know why) the modules inside sites-packages conda env are not available to import automatically without export the PATH.
So, to solve this issue, you have 2 options:
Installing using pip install folium and import folium (don't need install by conda install), or
After conda install , run conda init, close the terminal and open a new one. So, try to import again.
Here are some tips about use a pip in a conda-environment.
You can refer to a specific version of python by using the following at the first line of your .py file
This is for python 2.7
#!/usr/bin/env python2.7
This is for python 3
#!/usr/bin/env python3
As other users already pointed out you need to check if your module is included in your sys path. Use code:
import sys
print(sys.path)
If not you can include this in your sys.path by using the command:
sys.path.append('/path/to/the/folder/of/your/module/file')
or place it in default PYTHONPATH itself.
Other great answers:
https://stackoverflow.com/a/19305076/5381704
The problem is that in the code editor you are using, you are running the default interpreter. Based on your code editor, change the python interpreter to the conda interpreter and it will work.
In my case that happened because conda screwed up the environment variables. Instead of using env-specific python and pip, it used the globally installed ones.
Solution:
conda deactivate your-env
conda activate your-env
In my workstation, I was able to solve No module named <module name> error using two different ways.
First method, I solved this temporarily by:
(1) Open a Terminal
(2) $ conda activate <Conda environment name>
(3) $ export PYTHONPATH=/home/<user name>/anaconda3/envs/<Conda environment name>/lib/<Python package version>/site-packages:$PYTHONPATH
It is a temporary solution. Whenever you run your virtual environment, you have to do this.
My runtime environment:
OS: Unbuntu 18.04
Conda version: 4.8.2
Conda-build version: 3.18,11
Python version 3.7.6.final.0
Second method, I removed the
alias python=/usr/bin/python3.6 line in bashrc file.
Somehow this line blocks using Python tools installed in Anaconda Virtual Environment if the Python version in the Virtual Environment is different.
I want to use xlrd, xlwt, xlutils with sagemath. I have installed them in my system's Python(2.7.3) but sage environment is not recognizing them. Getting following error when I try to import with sage
ImportError: No module named xlrd`
But without sage its working fine. Any idea how can I make them work inside sage?? I am using Ubuntu 12(quantal),32 bits and Sage Version 5.8 with Python 2.7.5(sage's Python).
Faced same problem with Tkinter too but solved it using sudo apt-get install tk8.5-dev got solution form http://www.sagemath.org/doc/faq/faq-usage.html#how-to-get-sage-s-python-to-recognize-my-system-s-tcl-tk-install
But what for other libraries?
I tried to install them manually inside /usr/lib/sagemath/local/lib/python2.7/site-packages/ folder by putting tar files then extract them and then sudo python setup.py install
but still not working.
Rather than installing them with the system Python, install them with Sage's Python: you can either do sage --sh: this starts a subshell with various environment variables set appropriately for use with Sage, in particular $PATH will have $SAGE_ROOT/local/bin first. Then install the packages with python setup.py install etc. Or you can just run Sage's Python directly with sage --python setup.py install ....
Sage does not use the python installed in your system. It uses its own copy of python so while working with sage programs you would have to run the programs from terminal using this command
$ sage --python <Python_File>
Similarly sage does not uses the modules installed in the system's python. The Modules need to be installed under sage libraries. You have downloaded and extracted the folder right but used a wrong command. Now go again in the xlrd extracted folder and use this command.
$ sage --python setup.py install
Now Everything should work.
Cheers
I have an RHEL5/OEL5 64 bit OS with native python-2.4 on it and rpm-python-4.4.2.3-27.0.1.el5 installed.
When doing 'import rpm' with python-2.4 everything works as expected.
I would like (must) to use the python rpm module with python-2.7.5 on the same machine and not sure what is the proper way of doing that.
Python 2.7.5 was successfully installed. When calling 'import rpm' I got an import error.
I've found few RPMs for python-2.7.5 however, they are not good for RHEL5/OEL5 64 bit
Appreciate any pointers/advise!
A good solution for Python 2.7 is to use virtualenv.
In a nutshell, virtualenv allows you to manage several versions of Python on the same computer (even the same user) without getting in each other's way. It also allows to have several "flavors" of the same Python version, each with a different set of modules.
The process is described in detail in the documentation.
In your case, create an environment and then use pip to install the RPM module into this environment. When you activate the environment, Python scripts will be able to import the RPM module as long as you start them inside the environment (usually, the same terminal or shell process).
This will not affect the existing installation of 2.7 nor the old Python 2.4.
[EDIT] There is no pip module for rpm-module. Depending on how the module works, you should try to download the source RPM (*.src.rpm) and modify the SPEC file until it takes your Python 2.7 for the build and installs into the Python 2.7 module path.
[EDIT2] Steps to fix the problem:
I opened the tar tar -zxvf rpm-4.4.2.3.tar.gz
vi configure -----> changed all instances of python2.5 to python2.7
./autogen.sh
gmake
gmake -n install > log
Check the log file to make sure it doesn't install in the wrong places.
gmake install to install for real