Develop Reference

Import python from different folder without using .py extension

Import python from different folder without using .py extension
Below is my python script (feed_prg) that calls the python script (distribute)
Please note that my script are at different location
feed_prg is at location /opt/callscript
#!/usr/bin/env python
import sys
sys.dont_write_bytecode = True
sys.path.append('/home/username')
import distribute
# Main method start execution
def main(argv):
something = 'some'
distribute.feed(something)
if __name__ == "__main__":
main(sys.argv[1:])
distribute is at location /home/username
#!/usr/bin/env python
import sys
def feed(something):
print something
def main():
something= "some"
feed(something)
if __name__ == "__main__":
main()
I am seeing the below error while executing ./feed_prg , and only when my distribute filename is just distribute and not distribute.py
Traceback (most recent call last):
File "./feed_prg", line XX, in <module>
import distribute
ImportError: No module named distribute
the distribute also has the execute privilege, as below
-rwxr-xr-x 1 username username 3028 Dec 16 21:05 distribute
How can I fix this. ? Any inputs will be helpful
Thanks !

It is not possible to do this using import directly. It's best to simply rename the file to .py
That being said, it's possible to load the module into a variable using importlib or imp depending on your Python version.
Given the following file at path ./distribute (relative to where python is run):
# distribute
print("Imported!")
a_var = 5
Python 2
# python2import
from imp import load_source
distribute = load_source("distribute", "./distribute")
print(distribute.a_var)
Usage:
$ python python2import
Imported!
5
Python 3
#python3import
from importlib.machinery import SourceFileLoader
distribute = SourceFileLoader("distribute", "./distribute").load_module()
print(distribute.a_var)
Usage:
$ python3 python3import
Imported!
5

What is the most efficient way of checking for specific python modules and installing the ones not found?

I would like to check if a list of third party python modules are installed before my script proceeds. For example, here is a list of modules that my script would need in order to run:
modulesList = ['numpy', 'PIL', 'boto']
I want the script to check if each of those modules are installed and if not, automatically install them. Is there a way to do that without having to write what I have below every single time for each item in the list?
try:
import foo
except ImportError:
if extraModules == True:
os.system(os.path.join(os.path.dirname(sys.executable),'scripts','pip install foo'))
import foo
else:
print 'ERROR: The script cannot run without the foo module installed.\n'
sys.exit()

This script should do the job:
import pip
pkgs = ['numpy', 'pil', 'boto']
for package in pkgs:
try:
import package
except ImportError, e:
pip.main(['install', package])

How to use virtualenv and pip in python

I'd like to have a python script which in the beginning create a virtual environment, install required modules (e.g. cherrypy) and then continues with the rest of the code.
What I found so far is as follow:
import os, virtualenv
HOME_DIRECTORY = "venv"
virtualenv.create_environment(HOME_DIRECTORY)
execfile(os.path.join(HOME_DIRECTORY, "bin", "activate_this.py"))
import pip
pip.main(["install", "--prefix", HOME_DIRECTORY, "cherrypy"])
import cherrypy
class Root(object):
#cherrypy.expose
def index(self):
return "Hello World!"
if __name__ == '__main__':
cherrypy.quickstart(Root(), '/')
The script creates virtual environment and install cherrypy (based on the log), but still I get ImportError: No module named cherrypy error.
I also tried the following, but got the same error:
import importlib
CHERRYPY = 'cherrypy'
try:
importlib.import_module(CHERRYPY)
except ImportError:
import pip
pip.main(["install", "--prefix", HOME_DIRECTORY, CHERRYPY])
finally:
globals()[CHERRYPY] = importlib.import_module(CHERRYPY)
Apart from the above problem, also, I'd like to know how to specify the python version (e.g. python 3.6) while creating virtual environment. Thanks.

For the record the issue was with the virtual environment activate (the line to execute activate_this.py). For Python 3 the following worked for me:
ACTIVATE_THIS = 'venv/bin/activate_this.py'
exec(Path(ACTIVATE_THIS).read_text(), dict(__file__ = ACTIVATE_THIS))

How to install and import Python modules at runtime

I want to write a script to automatically setup a brand new ubuntu installation and install a django-based app. Since the script will be run on a new server, the Python script needs to automatically install some required modules.
Here is the script.
#!/usr/bin/env python
import subprocess
import os
import sys
def pip_install(mod):
print subprocess.check_output("pip install %s" % mod, shell=True)
if __name__ == "__main__":
if os.getuid() != 0:
print "Sorry, you need to run the script as root."
sys.exit()
try:
import pexpect
except:
pip_install('pexpect')
import pexpect
# More code here...
The installation of pexpect is success, however the next line import pexpect is failed. I think its because at runtime the code doesn't aware about the newly installed pexpect.
How to install and import Python modules at runtime? I'm open to another approaches.

You can import pip instead of using subprocess:
import pip
def install(package):
pip.main(['install', package])
# Example
if __name__ == '__main__':
try:
import pexpect
except ImportError:
install('pexpect')
import pexpect
Another take:
import pip
def import_with_auto_install(package):
try:
return __import__(package)
except ImportError:
pip.main(['install', package])
return __import__(package)
# Example
if __name__ == '__main__':
pexpect = import_with_auto_install('pexpect')
print(pexpect)
[edit]
You should consider using a requirements.txt along with pip. Seems like you are trying to automate deployments (and this is good!), in my tool belt I have also virtualenvwrapper, vagrant and ansible.
This is the output for me:
(test)root#vagrant:~/test# pip uninstall pexpect
Uninstalling pexpect:
/usr/lib/python-environments/test/lib/python2.6/site-packages/ANSI.py
/usr/lib/python-environments/test/lib/python2.6/site-packages/ANSI.pyc
/usr/lib/python-environments/test/lib/python2.6/site-packages/FSM.py
/usr/lib/python-environments/test/lib/python2.6/site-packages/FSM.pyc
/usr/lib/python-environments/test/lib/python2.6/site-packages/fdpexpect.py
/usr/lib/python-environments/test/lib/python2.6/site-packages/fdpexpect.pyc
/usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect-2.4-py2.6.egg-info
/usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect.py
/usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect.pyc
/usr/lib/python-environments/test/lib/python2.6/site-packages/pxssh.py
/usr/lib/python-environments/test/lib/python2.6/site-packages/pxssh.pyc
/usr/lib/python-environments/test/lib/python2.6/site-packages/screen.py
/usr/lib/python-environments/test/lib/python2.6/site-packages/screen.pyc
Proceed (y/n)? y
Successfully uninstalled pexpect
(test)root#vagrant:~/test# python test.py
Downloading/unpacking pexpect
Downloading pexpect-2.4.tar.gz (113Kb): 113Kb downloaded
Running setup.py egg_info for package pexpect
Installing collected packages: pexpect
Running setup.py install for pexpect
Successfully installed pexpect
Cleaning up...
<module 'pexpect' from '/usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect.pyc'>
(test)root#vagrant:~/test#

For those who are using pip version greater than 10.x, there is no main function for pip so the alternative approach is using import pip._internal as pip instead of import pip like :
Updated answer of Paulo
import pip._internal as pip
def install(package):
pip.main(['install', package])
if __name__ == '__main__':
try:
import pexpect
except ImportError:
install('pexpect')
import pexpect

I actually made a module for this exact purpose (impstall)
It's really easy to use:
import impstall
impstall.now('pexpect')
impstall.now('wx', pipName='wxPython')
Github link for issues/contributions

I solved my problem using the imp module.
#!/usr/bin/env python
import pip
import imp
def install_and_load(package):
pip.main(['install', package])
path = '/usr/local/lib/python2.7/dist-packages'
if path not in sys.path:
sys.path.append(path)
f, fname, desc = imp.find_module(package)
return imp.load(package, f, fname, desc)
if __name__ == "__main__":
try:
import pexpect
except:
pexpect = install_and_load('pexpect')
# More code...
Actually the code is less than ideal, since I need to hardcode the Python module directory. But since the script is intended for a known target system, I think that is ok.

I had the same issue but none of Google's searches helped. After hours debugging, I found that it may be because the sys.path is not reloaded with new installation directory.
In my case on my Ubuntu Docker, I want to import dns.resolver at runtime for Python3.8 (pre-installed). I also created ubuntu user and run all things with this user (including my Python script).
Before installing, sys.path doesn't have /home/ubuntu/.local/lib/python3.8/site-packages since I didn't install anything.
While installing with subprocess or pip.main like above, it creates /home/ubuntu/.local/lib/python3.8/site-packages (as user installation).
After installing , the sys.path should be refreshed to include this new location.
Since the sys.path is managed by site module, we should reload it (ref HERE):
import site
from importlib import reload
reload(site)
The full block for anyone that needs:
import subprocess
import sys
try:
import dns.resolver
except ImportError:
subprocess.check_call([sys.executable, "-m", "pip", "install", "dnspython"])
import site
from importlib import reload
reload(site)
import dns.resolver
I'm not Python developer so these code can be simplify more. This may help in cases such as fresh CI/CD environment for DevOps engineers like me.

Check if module exists, if not install it

I want to check if a module exists, if it doesn't I want to install it.
How should I do this?
So far I have this code which correctly prints f if the module doesn't exist.
try:
import keyring
except ImportError:
print 'f'

import pip
def import_or_install(package):
try:
__import__(package)
except ImportError:
pip.main(['install', package])
This code simply attempt to import a package, where package is of type str, and if it is unable to, calls pip and attempt to install it from there.

Here is how it should be done, and if I am wrong, please correct me. However, Noufal seems to confirm it in another answer to this question, so I guess it's right.
When writing the setup.py script for some scripts I wrote, I was dependent on the package manager of my distribution to install the required library for me.
So, in my setup.py file, I did this:
package = 'package_name'
try:
return __import__(package)
except ImportError:
return None
So if package_name was installed, fine, continue. Else, install it via the package manager which I called using subprocess.

This approach of dynamic import work really well in cases you just want to print a message if module is not installed. Automatically installing a module SHOULDN'T be done like issuing pip via subprocess. That's why we have setuptools (or distribute).
We have some great tutorials on packaging, and the task of dependencies detection/installation is as simple as providing install_requires=[ 'FancyDependency', 'otherFancy>=1.0' ]. That's just it!
But, if you really NEED to do by hand, you can use setuptools to help you.
from pkg_resources import WorkingSet , DistributionNotFound
working_set = WorkingSet()
# Printing all installed modules
print tuple(working_set)
# Detecting if module is installed
try:
dep = working_set.require('paramiko>=1.0')
except DistributionNotFound:
pass
# Installing it (anyone knows a better way?)
from setuptools.command.easy_install import main as install
install(['django>=1.2'])

NOTE: Ipython / Jupyter specific solution.
While using notebooks / online kernels, I usually do it using systems call.
try:
import keyring
except:
!pip install keyring
import keyring
P.S. One may wish to call conda install or mamba install instead.

You can use os.system as follows:
import os
package = "package_name"
try:
__import__package
except:
os.system("pip install "+ package)

You can launch pip install %s"%keyring in the except part to do this but I don't recommend it. The correct way is to package your application using distutils so that when it's installed, dependencies will be pulled in.

Not all modules can be installed so easily. Not all of them have easy-install support, some can only be installed by building them.. others require some non-python prerequisites, like gcc, which makes things even more complicated (and forget about it working well on Windows).
So I would say you could probably make it work for some predetermined modules, but there's no chance it'll be something generic that works for any module.

I made an import_neccessary_modules() function to fix this common issue.
# ======================================================================================
# == Fix any missing Module, that need to be installed with PIP.exe. [Windows System] ==
# ======================================================================================
import importlib, os
def import_neccessary_modules(modname:str)->None:
'''
Import a Module,
and if that fails, try to use the Command Window PIP.exe to install it,
if that fails, because PIP in not in the Path,
try find the location of PIP.exe and again attempt to install from the Command Window.
'''
try:
# If Module it is already installed, try to Import it
importlib.import_module(modname)
print(f"Importing {modname}")
except ImportError:
# Error if Module is not installed Yet, the '\033[93m' is just code to print in certain colors
print(f"\033[93mSince you don't have the Python Module [{modname}] installed!")
print("I will need to install it using Python's PIP.exe command.\033[0m")
if os.system('PIP --version') == 0:
# No error from running PIP in the Command Window, therefor PIP.exe is in the %PATH%
os.system(f'PIP install {modname}')
else:
# Error, PIP.exe is NOT in the Path!! So I'll try to find it.
pip_location_attempt_1 = sys.executable.replace("python.exe", "") + "pip.exe"
pip_location_attempt_2 = sys.executable.replace("python.exe", "") + "scripts\pip.exe"
if os.path.exists(pip_location_attempt_1):
# The Attempt #1 File exists!!!
os.system(pip_location_attempt_1 + " install " + modname)
elif os.path.exists(pip_location_attempt_2):
# The Attempt #2 File exists!!!
os.system(pip_location_attempt_2 + " install " + modname)
else:
# Neither Attempts found the PIP.exe file, So i Fail...
print(f"\033[91mAbort!!! I can't find PIP.exe program!")
print(f"You'll need to manually install the Module: {modname} in order for this program to work.")
print(f"Find the PIP.exe file on your computer and in the CMD Command window...")
print(f" in that directory, type PIP.exe install {modname}\033[0m")
exit()
import_neccessary_modules('art')
import_neccessary_modules('pyperclip')
import_neccessary_modules('winsound')

Here is my approach. The idea is loop until python has already installed all modules by built in module as "pip" .
import pip
while True:
try:
#import your modules here. !
import seaborn
import bokeh
break
except ImportError as err_mdl:
print((err_mdl.name))
pip.main(['install', err_mdl.name])

I tried installing transformers using the below method and it worked fine. Similarly, you can just replace your library name instead of "transformers".
import pip
try:
from transformers import pipeline
except ModuleNotFoundError:
pip.main(['install', "transformers"])
from transformers import pipeline

I tried this in a new virtual envoirnment with no packages installed and it installed the necessary package i.e. opencv-python
Example is given below
import os
try:
import cv2
except ImportError:
os.system('pip install opencv-python')