It took me forever to find this solution, so I want others to be able to see it.
I wanted to write a python script to create a virtual env and install modules inside it. Unfortunately, pip does not play nice with subprocess, as detailed here:
https://github.com/pypa/pip/issues/610
My answer is already on that thread, but I wanted to detail it below
Basically, the issue is that pip is still using the python executable that the original python called. To fix this, you need to delete it from the passed in environment variables. Here is the solution:
#!/usr/bin/python3
import os
import subprocess
python_env_var = {"_", "__PYVENV_LAUNCHER__"}
CMD_ENVIRONMENT = {name: value for (name, value) in os.environ.items()
if name not in python_env_var}
subprocess.call('./pip install -r requirements.txt', shell=True,
env=CMD_ENVIRONMENT)
Tested on Mac, ubuntu 14.04 and Windows with python 3
This same problem could easily exist for lots of situations -- I will be deleting this variable from now on, to prevent this kind of behavior when dealing with virtualenv's
Related
In a jupyter notebook, I can fairly easily check with python code if some libraries are installed on the current kernel of the python notebook.
However there is also the "host" kernel which has its own python env (ie the python process that was launched when jupyter notebook was called). Depending on what libraries/extensions were installed on the host, it may not be possible to do specific things on the jupyter notebook client itself.
Is there a way to query what libraries/modules/extensions are installed on the host, from the "client" notebook ? thanks
Did you test !pip list or !pip freeze?
To list locally installed packages and their version # within a pipenv environment, cd into a pipenv project and enter the following command:
pipenv lock -r
I guess you can use !pip list to show the modules/libraries installed on the current env. But I don't think you can view extensions.
Please use !pip list --local or !pip freeze --local if you're using a virtual environment.
This could get you started:
import sys
import os
for i in sys.path:
try:
print("->", i, ":")
print(repr(os.listdir(i or ".")))
except Exception:
print("can't list", i)
pass
NOTE that my code isn't a complete solution, but it is a start. I will attempt to improve it shortly, but even as it is it may be immediately useful for you to develop your own solution.
What it doesn't do:
it does not determine whether it found a python file or a directory containing a valid package or something that isn't either (i.e. is not 'import'-able)
It skips over zipped packages (such as this item that could be in sys.path: /home/pyodide/lib/python39.zip)
it does not attempt to find built-in packages (those that are compiled into Python itself). Most of those would be already loaded, so you will find them in sys.modules, though
If you need to test for individual packages, rather than get a whole list: you may be better off using 'import' inside a try/except statement:
try:
import something
except ImportError:
pass
# do what you want if the package isn't available
If I understand correctly, you can run:
help("modules")
Or you can use pydoc for your work:
import pydoc
!pydoc modules
I believe !pip list should work- if not, then pydoc most likely would, see above.
I am new to Python so this may be a stupid question, sorry in advance if it is so!
The paths Python searches for modules using Spyder 4.1.4.:
[.., 'D:\Users\Martin\Anaconda3\lib\site-packages', ..]
The paths Python searches for when used via command line in windows:
[.., 'D:\Users\Martin\Anaconda3\lib\site-packages', ..]
Thus, both include path to
'D:\Users\Martin\Anaconda3\lib\site-packages'
where I store my cv2 module installed. Yet, Spyder can import cv2 and to my surprise cmd cannot import cv2 with an error No module named 'cv2'. What am I missing please? Can anyone verify if it works for him in command line please? (I changed/edited the question again to cut unneccessary long talk, though the essence stayed the same, in hope someone could answer).
I solved it. If you have a similar problem, you can check if your case is similar to mine and follow the steps.
I use Spyder and everything works fine. However my goal is to use Python via cmd. This could not be done because Python via cmd could not find my modules.
What I did: After typing python in cmd, Python worked and output its version by default. However, I noticed the version was different than version running in Spyder - I had two Python installations (one of which I didn't know about). I found the installation cmd uses by import sys and print(sys.executable). I uninstalled that Python installation.
Now, typing python in cmd does not do anything, it is an unknown command. Then I followed this guide to add my Python installation to Path (using Windows) by:
My Computer > Properties > Advanced System Settings > Environment Variables >
in System variables, I found "Path", clicked Edit, New, then input my Python installation path D:\Users\Martin\Anaconda3 as well as D:\Users\Martin\Anaconda3\Scripts and D:\Users\Martin\Anaconda3\Library\bin. This solved it for me.
I am curious about running pip.
Everytime I ran pip in command shell in windows like that
c:\python27\script>pip install numpy
But, I wondered if I can run it in python idle.
import pip
pip.install("numpy")
Unfortunately, it is not working.
Still cannot comment so I added another answer. Pip has had several entrypoints in the past. And it's not recommended to call pip directly or in-process (if you still want to do it, "runpy" is kind of recommended):
import sys
import runpy
sys.argv=["pip", "install", "packagename"]
runpy.run_module("pip", run_name="__main__")
But this should also work:
try:
from pip._internal import main as _pip_main
except ImportError:
from pip import main as _pip_main
_pip_main(["install", "packagename"])
This question is, or should be, about how to run pip from a python program. IDLE is not directly relevant to this version of the quesiton.
To expand on J. J. Hakala's comment: a command-line such as pip install pillow is split on spaces to become sys.argv. When pip is run as a main module, it calls pip.main(sys.argv[1:]). If one imports pip, one may call pip.main(arg_line.split()), where arg_line is the part of the command line after pip.
Last September (2015) I experimented with using this unintended API from another python program and reported the initial results on tracker issue 23551. Discussion and further results followed.
The problem with executing multiple commands in one process is that some pip commands cache not only sys.path, which normally stays constant, but also the list of installed packages, which normally changes. Since pip is designed to run one command per process, and then exit, it never updates the cache. When pip.main is used to run multiple commands in one process, commands given after the caching may use a stale and no-longer-correct cache. For example, list after install shows how things were before the install.
A second problem for a program that wants to examine the output from pip is that it goes to stdout and stderr. I posted a program that captures these streams into program variables as part of running pip.
Using a subprocess call for each pip command, as suggested by L_Pav, though less efficient, solves both problems. The communicate method makes the output streams available. See the subprocess doc.
At moment there are no official way to do it, you could use pip.main but you current idle session will not 'see' this installed package.
There been a lot a discussion over how to add a "high level" programmatic API for pip, it's seems promising.
Actually, I think, you can use subprocess.Popen(apt-get numpy), not sure how to do it with PIP though.
If your on a Mac you should be able to do it like this:
Go to your IDLE.
Run help('modules').
Find the HTML module.
Run help('HTML')
There should pop up a file map, for example this/file/map/example/.
Go to the finder and do command+shift+g and paste the file map there. Please delete the last file, because then your gonna go to the modules files.
There are all the modules. If you want to add modules, download the files of the module and put them there.
I hope this helps you.
I have recently tried to use pylearn2, a deep machin learning package for Python developed at University of Montreal.
I've just installed it and tried to run a simple example, but it did not work.
I have been using a pc with an Ubuntu 13.10 system, on which I found ipython installed.
I have installed Theano and later pylearn2, by following this webpage instructions:
http://deeplearning.net/software/pylearn2/
I have also modified the .bashrc file, as suggested
I thought that everything went well, and then I tried this Quick start example:
http://deeplearning.net/software/pylearn2/tutorial/index.html
I stopped at the first command:
python make_dataset.py
My terminal states:
Traceback (most recent call last): File "make_dataset.py", line 14,
in
Do you have any ideas on why it is not working?
Do you why these errors occur?
Thanks a lot
EDIT: the 14 line is the first non-commented line of the file. It states
from pylearn2.utils import serial
Without more information, I can only guess, but my first guess is…
You haven't actually installed pylearn2, because if you follow the linked docs to grab the git repo and add a PYLEARN2_DATA_PATH variable, nothing gets installed into site-packages (or dist-packages or anywhere else on sys.path).
This means that pylearn2 will only work when you start Python from within the top-level directory of the pylearn2 repo.
So, if you run a script like this:
$ cd /path/to/pylearn2
$ cd scripts/tutorials/grbm_smd/
$ python make_dataset.py
… it won't actually work.
It looks like there is a setup.py file in the repository. Does it work? I have no idea. Even though the docs don't mention using it, you might want to try. Either this:
$ pip install .
… or, if you don't have pip or it doesn't work on this package:
$ python setup.py install
Either way, of course, you may need sudo or a flag to install to your user site-packages instead of system, etc., as with any other Python package.
If that doesn't work, you might be able to just add /path/to/pylearn2 to your sys.path in some way. The most obvious way is by doing an export PYTHONPATH=/path/to/pylearn2:$PYTHONPATH in your ~/.bashrc.
Also, you will need to either source ~/.bashrc or create a new shell to get any effects of modifying the file.
If you're wondering why the instructions and the tutorial together don't give you enough information to make this work without a lot of hassle, I think that's covered in the very top of the documentation:
Pylearn2 is still undergoing rapid development. Don’t expect a clean road without bumps!
And the very fact that there is no PyPI download yet implies that this really is not ready for novices to use. If you don't know enough about using Python packages (and bash basics) to muddle through on your own, there's a good chance you won't be able to use this package.
I'm just starting out with Python, and have found out that I can import various libraries. How do I find out what libraries exist on my Mac that I can import? How do I find out what functions they include?
I seem to remember using some web server type thing to browse through local help files, but I may have imagined that!
From the Python REPL (the command-line interpreter / Read-Eval-Print-Loop), type help("modules") to see a list of all your available libs.
Then to see functions within a module, do help("posix"), for example. If you haven't imported the library yet, you have to put quotes around the library's name.
For the web server, you can run the pydoc module that is included in the python distribution as a script:
python /path/to/pydoc.py -p 1234
where 1234 is the port you want the server to run at. You can then visit http://localhost:1234/ and browse the documentation.
Every standard python distribution has these libraries, which cover most of what you will need in a project.
In case you need to find out if a library exists at runtime, you do it like this
try:
import ObscureModule
except ImportError:
print "you need to install ObscureModule"
sys.exit(1) # or something like that
You can install another library: yolk.
yolk is a python package manager and will show you everything you have added via pypi. But it will also show you site-packages added through whatever local package manager you run.
just run the Python interpeter and type the command
import "lib_name"
if it gives an error, you don't have the lib installed...else you are good to go
On Leopard, depending on the python package you're using and the version number, the modules can be found in /Library/Python:
/Library/Python/2.5/site-packages
or in /Library/Frameworks
/Library/Frameworks/Python.framework/Versions/Current/lib/python2.6/site-packages
(it could also be 3.0 or whatever version)...
I guess it is quite the same with Tiger
Considering that in every operating system most of python's packages are installed using 'pip' (see pip documentation) you can also use the command 'pip freeze' on a terminal to print a list of all the packages you have installed through it.
Other tools like 'homebrew' for macOS (used when for some reason you can't install a package using pip) have similar commands, in this specific case 'brew list'.