I would like to know how to extract the requirements from a single python script. I tried the following way, at the beginning of my file, immediately after the imports:
try:
from pip._internal.operations import freeze
except ImportError: # pip < 10.0
from pip.operations import freeze
x = freeze.freeze()
for p in x:
print(p)
The piece of code above, however, gives me back all the Python frameworks installed locally. I would like to extract only the necessary requirements for the script, in order to be able to deploying the final application.
I hope I was clear.
pipreqs is simple to use
install:
pip install pipreqs
in linux in the same folder of your script
use:
pipreqs .
then the requirements.txt file is created
pip home page:
https://pypi.org/project/pipreqs/
You can do this easily with 'modulefinder' python module.
I think you want to print all the modules required by a script.
So, you can refer to
http://blog.rtwilson.com/how-to-find-out-what-modules-a-python-script-requires/
or for your ease the code is here:
from modulefinder import ModuleFinder
f = ModuleFinder()
# Run the main script
f.run_script('run.py')
# Get names of all the imported modules
names = list(f.modules.keys())
# Get a sorted list of the root modules imported
basemods = sorted(set([name.split('.')[0] for name in names]))
# Print it nicely
print ("\n".join(basemods))
Related
I would like to install the modules 'mutagen' and 'gTTS' for my code, but I want to have it so it will install the modules on every computer that doesn't have them, but it won't try to install them if they're already installed. I currently have:
def install(package):
pip.main(['install', package])
install('mutagen')
install('gTTS')
from gtts import gTTS
from mutagen.mp3 import MP3
However, if you already have the modules, this will just add unnecessary clutter to the start of the program whenever you open it.
EDIT - 2020/02/03
The pip module has updated quite a lot since the time I posted this answer. I've updated the snippet with the proper way to install a missing dependency, which is to use subprocess and pkg_resources, and not pip.
To hide the output, you can redirect the subprocess output to devnull:
import sys
import subprocess
import pkg_resources
required = {'mutagen', 'gTTS'}
installed = {pkg.key for pkg in pkg_resources.working_set}
missing = required - installed
if missing:
python = sys.executable
subprocess.check_call([python, '-m', 'pip', 'install', *missing], stdout=subprocess.DEVNULL)
Like #zwer mentioned, the above works, although it is not seen as a proper way of packaging your project. To look at this in better depth, read the the page How to package a Python App.
you can use simple try/except:
try:
import mutagen
print("module 'mutagen' is installed")
except ModuleNotFoundError:
print("module 'mutagen' is not installed")
# or
install("mutagen") # the install function from the question
If you want to know if a package is installed, you can check it in your terminal using the following command:
pip list | grep <module_name_you_want_to_check>
How this works:
pip list
lists all modules installed in your Python.
The vertical bar | is commonly referred to as a "pipe". It is used to pipe one command into another. That is, it directs the output from the first command into the input for the second command.
grep <module_name_you_want_to_check>
finds the keyword from the list.
Example:
pip list| grep quant
Lists all packages which start with "quant" (for example "quantstrats"). If you do not have any output, this means the library is not installed.
You can check if a package is installed using pkg_resources.get_distribution:
import pkg_resources
for package in ['mutagen', 'gTTS']:
try:
dist = pkg_resources.get_distribution(package)
print('{} ({}) is installed'.format(dist.key, dist.version))
except pkg_resources.DistributionNotFound:
print('{} is NOT installed'.format(package))
Note: You should not be directly importing the pip module as it is an unsupported use-case of the pip command.
The recommended way of using pip from your program is to execute it using subprocess:
subprocess.check_call([sys.executable, '-m', 'pip', 'install', 'my_package'])
Although #girrafish's answer might suffice, you can check package installation via importlib too:
import importlib
packages = ['mutagen', 'gTTS']
[subprocess.check_call(['pip', 'install', pkg])
for pkg in packages if not importlib.util.find_spec(pkg)]
You can use the command line :
python -m MyModule
it will say if the module exists
Else you can simply use the best practice :
pip freeze > requirements.txt
That will put the modules you've on you python installation in a file
and :
pip install -r requirements.txt
to load them
It will automatically you purposes
Have fun
Another solution it to put an import statement for whatever you're trying to import into a try/except block, so if it works it's installed, but if not it'll throw the exception and you can run the command to install it.
You can run pip show package_name
or for broad view use pip list
Reference
If you would like to preview if a specific package (or some) are installed or not maybe you can use the idle in python.
Specifically :
Open IDLE
Browse to File > Open Module > Some Module
IDLE will either display the module or will prompt an error message.
Above is tested with python 3.9.0
This is my project structure. I use virtualenv in my project but when I run it ,it has an ImportError.I use Mac.
But I can run it successfully use Pycharm
So how to run it successfully by Terminal.Because I want to run it in a Ubuntu server with cron
Thanks you for your answers.Here I show my solution.I modify my handler.py I think it may be related to The Module Search Path.
So I add the project path to the PYTHONPATH.
import os
project_home = os.path.realpath(__file__)
project_home = os.path.split(project_home)[0]
import sys
sys.path.append(os.path.split(project_home)[0])
import shutil
from modules import db, json_parse, config_out
from init_log import init as initlog
initlog()
if __name__ == '__main__':
try:
columns = json_parse.json_parse()
if not columns:
sys.exit()
is_table_has_exist = db.check_tables_exist(columns=columns)
if is_table_has_exist:
db.check_columns(columns=columns)
is_ok, config_path = config_out.output(columns)
if is_ok:
file_name = os.path.split(config_path)[1]
shutil.copy(config_path, os.path.join("/app/statics_log/config", file_name))
except Exception, e:
print e
And I run with crontab by this.
cd to/my/py_file/path && /project_path/.env/bin/python /path/to/py_file
example:
13 8 1 * * cd bulu-statics/create_config/ && /home/buka/bulu-statics/.env/bin/python /home/buka/bulu-statics/create_config/handler.py >> /app/statics_log/config/create_config.log
PyCharm automatically adds project directories marked as containing sources to the PYTHONPATH environment variable, whihc is why it works from within pycharm. On the terminal use
PYTHONPATH=${PWD}/..:${PYTHONPATH} python handler.py
You can use explicit relative imports:
from .modules import db, json_parse, config_out
The proper way to do this is to turn your project into a proper Python package by adding a setup.py file and then installing it with pip install -e .
probably because PyCharm added your project folder to the PythonPath, so you can run you app inside PyCharm.
However, when you try to run it from command line, python interpreter cannot find these libs in Python python, so what you need to do is to add your python virtualenv the python python.
there are different ways to adding python path, but I would suggest you to follow:
prepare a setup.py you'll need to specify packages and install_requires.
install your app locally in development mode via pip install -e /path/to/your-package -> it'll create a egg-link in your python virtualenv, you can run your app in your local terminal from now on;
for packing and releasing, you may want to build an artifact by following https://docs.python.org/2.7/distutils/builtdist.html
you may pip install or easy_install the artifact on your other machines. you also can release your package to PyPi if you want.
Is there any way to get pip to print the config it will attempt to use? For debugging purposes it would be very nice to know that:
config.ini files are in the correct place and pip is finding them.
The precedence of the config settings is treated in the way one would expect from the docs
For 10.0.x and higher
There is new pip config command, to list current configuration values
pip config list
(As pointed by #wmaddox in comments) To get the list of where pip looks for config files
pip config list -v
Pre 10.0.x
You can start python console and do. (If you have virtaulenv don't forget to activate it first)
from pip import create_main_parser
parser = create_main_parser()
# print all config files that it will try to read
print(parser.files)
# reads parser files that are actually found and prints their names
print(parser.config.read(parser.files))
create_main_parser is function that creates parser which pip uses to read params from command line(optparse) and loading configs(configparser)
Possible file names for configurations are generated in get_config_files. Including PIP_CONFIG_FILE environment variable if it set.
parser.config is instance of RawConfigParser so all generated file names in get_config_files are passed to parser.config.read
.
Attempt to read and parse a list of filenames, returning a list of filenames which were successfully parsed. If filenames is a string, it is treated as a single filename. If a file named in filenames cannot be opened, that file will be ignored. This is designed so that you can specify a list of potential configuration file locations (for example, the current directory, the user’s home directory, and some system-wide directory), and all existing configuration files in the list will be read. If none of the named files exist, the ConfigParser instance will contain an empty dataset. An application which requires initial values to be loaded from a file should load the required file or files using read_file() before calling read() for any optional files:
From how I see it, your question can be interpreted in three ways:
What is the configuration of the pip executable?
There is a quite extensive documentation for the configurations supported by pip, see here: https://pip.pypa.io/en/stable/user_guide/#configuration
What is the configuration that pip uses when configuring and subsequently building code required by a Python module?
This is specified by the package that is being installed. The package maintainer is responsible for producing a configuration script. For example, Numpy has a Configuration class (https://github.com/numpy/numpy/blob/master/numpy/distutils/misc_util.py) that they use to configure their Cython build.
What are the current modules installed with pip so I can reproduce a specific environment configuration?
This is easy, pip freeze > requirements.txt. This will produce a file of all currently installed pip modules along with their exact versions. You can then do pip install -r requirements.txt to reproduce that exact environment configuration on another machine.
I hope this helps.
You can run pip in pdb. Here's an example inside ipython:
>>> import pip
>>> import pdb
>>> pdb.run("pip.main()", globals())
(Pdb) s
--Call--
> /usr/lib/python3.5/site-packages/pip/__init__.py(197)main()
-> def main(args=None):
(Pdb) b /usr/lib/python3.5/site-packages/pip/baseparser.py:146
Breakpoint 1 at /usr/lib/python3.5/site-packages/pip/baseparser.py:146
(Pdb) c
> /usr/lib/python3.5/site-packages/pip/baseparser.py(146)__init__()
-> if self.files:
(Pdb) p self.files
['/etc/xdg/pip/pip.conf', '/etc/pip.conf', '/home/andre/.pip/pip.conf', '/home/andre/.config/pip/pip.conf']
The only trick here was looking up the path of the baseparser (and knowing that the files are in there). If you don't know this already you can simply step through the program or read the source. This type of exploration should work for most Python programs.
Suppose I have the following setup:
mkdir test && cd test
virtualenv .venv
source .venv/bin/activate
pip install django
mkdir mod1
touch mod1/__init__.py
echo "a = 1" > mod1/mod2.py
Which gives me:
test/.venv
test/mod1/__init__.py
test/mod1/mod2.py
How would I write this function:
def get_module(module_name, root_path, virtualenv_path=None)
In order for this to work:
project_root_path = "./test"
project_virtualenv_path = "./test/.venv"
get_module("mod1.mod2", project_root_path, project_virtualenv_path)
get_module("django.contrib.auth", project_root_path, project_virtualenv_path)
Assuming I don't have ./test/.venv activated.
The reason I want to do this, is because I'm working on a vim plugin which would implement gf functionality in a python file on an import statement. I'm trying to support virtualenvs as well.
EDIT:
Also, the script should not alter the current runtime, by adding or appending to sys.path. This should run inside vim, via the vim python bindings, and I don't think altering the vim python runtime would be a good idea.
get_module could either return a module object, or the path to the module, which is what I'm basically looking for.
You can add your virtualenv on python path like:
import site
site.addsitedir('/home/user/.virtualenvs/myapp1/lib/python2.7/site-packages')
and then import should work
The only practical solution I could find here is to run the virtualenv's activate_this.py script, look for what I need, then remove it's changes from sys.path.
import sys
import os
old_sys_path = list(sys.path)
virtualenv_path = "/path/to/venv"
activate_this_path = os.path.join(virtualenv_path, "bin", "activate_this.py")
execfile(activate_this_path, dict(__file__=activate_this_path))
# get my module here
# restore sys.path
sys.path = old_sys_path
If you have a better answer, please add it, and I'll change the accepted answer gladly.
I didn't want to install python modules using easy install, symlinks in site-packages or PYTHONPATH.
So, I am trying something that I do wants system wide, then any application installation is done locally. Note, the root password is required only once here.
First create a symblink of.../pythonX.Y/site-packages/mymodules -> /home/me/lib/python_related
So, I create a directory called
/home/me/lib/python_related/
In there:
/home/me/lib/python_related
/home/me/lib/python_related/__init__.py
/home/me/lib/python_related/django_related/
/home/me/lib/python_related/django_related/core
/home/me/lib/python_related/django_related/core/Django1.0
/home/me/lib/python_related/django_related/core/Django1.1
/home/me/lib/python_related/django_related/core/mycurrent_django -> Django1.1/django
/home/me/lib/python_related/django_related/apps
/home/me/lib/python_related/django_related/apps/tagging
/home/me/lib/python_related/django_related/apps/tagging/django-tagging-0.2
/home/me/lib/python_related/django_related/apps/tagging/django-tagging-0.3
/home/me/lib/python_related/django_related/apps/tagging/mycurrent_tagging -> django-tagging-0.3
Now, here is the content of:
/home/me/lib/python_related/__init__.py
==========================================
import sys, os
# tell us where you keep all your modules and this didn't work as it gave me
# the location of the site-packages
#PYTHON_MODULE_PATH = os.path.dirname(__file__)
PYTHON_MODULE_PATH = "/home/me/libs/python_bucket"
def run_cmd(cmd):
"""
Given a command name, this function will run the command and returns the output
in a list.
"""
output = []
phdl = os.popen(cmd)
while 1:
line = phdl.readline()
if line == "":
break
output.append(line.replace("\n", ""))
return output
def install():
"""
A cheesy way of installing and managing your python apps locally without
a need to install them in the site-package. All you'd need is to install
the directory containing this file in the site-package and that's it.
Anytime you have a python package you want to install, just put it in a
proper sub-directory and make a symlink to that directory called mycurrent_xyz
and you are done. (e.g. mycurrent_django, mycurrent_tagging .. etc)
"""
cmd = "find %s -name mycurrent_*" % PYTHON_MODULE_PATH
modules_to_be_installed = run_cmd(cmd)
sys.path += modules_to_be_installed
install()
=======================================================
Now in any new python project, just import your mymodules and that pulls in any apps that you have in the above directory with the proper symbolic link. This way you can have multiple copies of apps and just use the mycurrent_xyz to the one you want to use.
Now here is question. Is this a good way of doing it?
Have a look at virtualenv.
It may do what you are after.