Develop Reference

When I try to generate files for protobuf I get error ModuleNotFoundError

When I try to generate files with the command
python -m grpc_tools.protoc -I./ --python_out=. --grpc_python_out=. service.proto
I get error.
Traceback:
test_client.py:11: in <module>
from tests.remote.grpc_wrapper.service_pb2_grpc import TestServiceServicer, add_TestServiceServicer_to_server, \
service_pb2_grpc.py:4: in <module>
import service_pb2 as service__pb2
E ModuleNotFoundError: No module named 'service_pb2'
How can I fix it? I truing reinstall protobuf but it don’t help me.
pip uninstall protobuf
pip install -U protobuf
P.S. I use conda, I truing use
conda install protobuf
but it don’t help me too.

There's talk of specifying this at the point of generation in the .proto file in this issue. As far as I know you have two options currently:
1) Change your line 4 to have . in front (this signifies a relative import):
from . import service_pb2 as service__pb2
2) Change the __init__.py file in the "grpc_wrapper" folder to include:
import os
import sys
sys.path.insert(0, os.path.abspath(os.path.dirname(__file__)))

Python 3.x - How to get the directory where user installed package

Suppose that I am building a Python package with the following folder tree:
main
|-- setup.py
|-- the_package
|--globar_vars.py
|--the_main_script.py
When the user performs anyway to install it, like:
sudo python setup.py install
python setup.py install --user
python setup.py install --prefix=/usr/local
Or well, using PIP:
pip install 'SomeProject'
I want that the folder where the package was installed be saved on the global_vars.py, in any variable, Eg:
globarl_vars.py
#!/usr/bin/env python3
user_installed_pkg = '/some/path/where/the/package/was/installed'
There is someway to get it? Thanks in advance.

Assuming your SomeProject package is a well-formed Python package (which can be done using setuptools), a user can derive the location simply use the pkg_resources module provided by setuptools package to get this information. For example:
>>> from pkg_resources import working_set
>>> from pkg_resources import Requirement
>>> working_set.find(Requirement.parse('requests'))
requests 2.2.1 (/usr/lib/python3/dist-packages)
>>> working_set.find(Requirement.parse('requests')).location
'/usr/lib/python3/dist-packages'
However, the path returned could be something inside an egg which means it will not be a path directly usable through standard filesystem tools. You generally want to use the Resource manager API to access resources in those cases.
>>> import pkg_resources
>>> api_src = pkg_resources.resource_string('requests', 'api.py')
>>> api_src[:25]
b'# -*- coding: utf-8 -*-\n\n'

import wtv_module
print(wtv_module.__file__)
import os
print(os.path.dirname(wtv_module.__file__))

How to install and import Python modules at runtime

I want to write a script to automatically setup a brand new ubuntu installation and install a django-based app. Since the script will be run on a new server, the Python script needs to automatically install some required modules.
Here is the script.
#!/usr/bin/env python
import subprocess
import os
import sys
def pip_install(mod):
print subprocess.check_output("pip install %s" % mod, shell=True)
if __name__ == "__main__":
if os.getuid() != 0:
print "Sorry, you need to run the script as root."
sys.exit()
try:
import pexpect
except:
pip_install('pexpect')
import pexpect
# More code here...
The installation of pexpect is success, however the next line import pexpect is failed. I think its because at runtime the code doesn't aware about the newly installed pexpect.
How to install and import Python modules at runtime? I'm open to another approaches.

You can import pip instead of using subprocess:
import pip
def install(package):
pip.main(['install', package])
# Example
if __name__ == '__main__':
try:
import pexpect
except ImportError:
install('pexpect')
import pexpect
Another take:
import pip
def import_with_auto_install(package):
try:
return __import__(package)
except ImportError:
pip.main(['install', package])
return __import__(package)
# Example
if __name__ == '__main__':
pexpect = import_with_auto_install('pexpect')
print(pexpect)
[edit]
You should consider using a requirements.txt along with pip. Seems like you are trying to automate deployments (and this is good!), in my tool belt I have also virtualenvwrapper, vagrant and ansible.
This is the output for me:
(test)root#vagrant:~/test# pip uninstall pexpect
Uninstalling pexpect:
/usr/lib/python-environments/test/lib/python2.6/site-packages/ANSI.py
/usr/lib/python-environments/test/lib/python2.6/site-packages/ANSI.pyc
/usr/lib/python-environments/test/lib/python2.6/site-packages/FSM.py
/usr/lib/python-environments/test/lib/python2.6/site-packages/FSM.pyc
/usr/lib/python-environments/test/lib/python2.6/site-packages/fdpexpect.py
/usr/lib/python-environments/test/lib/python2.6/site-packages/fdpexpect.pyc
/usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect-2.4-py2.6.egg-info
/usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect.py
/usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect.pyc
/usr/lib/python-environments/test/lib/python2.6/site-packages/pxssh.py
/usr/lib/python-environments/test/lib/python2.6/site-packages/pxssh.pyc
/usr/lib/python-environments/test/lib/python2.6/site-packages/screen.py
/usr/lib/python-environments/test/lib/python2.6/site-packages/screen.pyc
Proceed (y/n)? y
Successfully uninstalled pexpect
(test)root#vagrant:~/test# python test.py
Downloading/unpacking pexpect
Downloading pexpect-2.4.tar.gz (113Kb): 113Kb downloaded
Running setup.py egg_info for package pexpect
Installing collected packages: pexpect
Running setup.py install for pexpect
Successfully installed pexpect
Cleaning up...
<module 'pexpect' from '/usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect.pyc'>
(test)root#vagrant:~/test#

For those who are using pip version greater than 10.x, there is no main function for pip so the alternative approach is using import pip._internal as pip instead of import pip like :
Updated answer of Paulo
import pip._internal as pip
def install(package):
pip.main(['install', package])
if __name__ == '__main__':
try:
import pexpect
except ImportError:
install('pexpect')
import pexpect

I actually made a module for this exact purpose (impstall)
It's really easy to use:
import impstall
impstall.now('pexpect')
impstall.now('wx', pipName='wxPython')
Github link for issues/contributions

I solved my problem using the imp module.
#!/usr/bin/env python
import pip
import imp
def install_and_load(package):
pip.main(['install', package])
path = '/usr/local/lib/python2.7/dist-packages'
if path not in sys.path:
sys.path.append(path)
f, fname, desc = imp.find_module(package)
return imp.load(package, f, fname, desc)
if __name__ == "__main__":
try:
import pexpect
except:
pexpect = install_and_load('pexpect')
# More code...
Actually the code is less than ideal, since I need to hardcode the Python module directory. But since the script is intended for a known target system, I think that is ok.

I had the same issue but none of Google's searches helped. After hours debugging, I found that it may be because the sys.path is not reloaded with new installation directory.
In my case on my Ubuntu Docker, I want to import dns.resolver at runtime for Python3.8 (pre-installed). I also created ubuntu user and run all things with this user (including my Python script).
Before installing, sys.path doesn't have /home/ubuntu/.local/lib/python3.8/site-packages since I didn't install anything.
While installing with subprocess or pip.main like above, it creates /home/ubuntu/.local/lib/python3.8/site-packages (as user installation).
After installing , the sys.path should be refreshed to include this new location.
Since the sys.path is managed by site module, we should reload it (ref HERE):
import site
from importlib import reload
reload(site)
The full block for anyone that needs:
import subprocess
import sys
try:
import dns.resolver
except ImportError:
subprocess.check_call([sys.executable, "-m", "pip", "install", "dnspython"])
import site
from importlib import reload
reload(site)
import dns.resolver
I'm not Python developer so these code can be simplify more. This may help in cases such as fresh CI/CD environment for DevOps engineers like me.

Can I define optional packages in setuptools?

Currently one of my packages requires a JSON parser/encoder, and is designed to use simplejson if available falling back to the json module (in the standard library) if necessary (as benchmarks show simplejson is faster).
However, recently it's been hit or miss as to whether simplejson will install when using zc.buildout - something with the move to github, I believe. Which got me wondering; is it possible to define optional packages in my setup.py file which, if unavailable, won't stop the installation of my package?

optional packages at installation time.
I am assuming you are talking about your setup.py script.
You could change it to have:
# mypackage/setup.py
extras = {
'with_simplejson': ['simplejson>=3.5.3']
}
setup(
...
extras_require=extras,
...)
then you can do either of:
pip install mypackage,
pip install mypackage[with_simplejson]
with the latter installing simplejson>=3.5.3.
Instead of trying to install everything and fallback to a known good version,
you would want to install the subset of packages you know work.
optional packages at execution time.
Once you have two different sets of packages that could be installed, you need
to make sure you can use them if they are available. E.g. for your json import:
try:
# helpful comment saying this should be faster.
import simplejson as json
except ImportError:
import json
Another more complex example:
try:
# xml is dangerous
from defusedxml.cElementTree import parse
except ImportError:
try:
# cElementTree is not available in older python
from xml.cElementTree import parse
except ImportError:
from xml.ElementTree import parse
But you can also find this pattern in some packages:
try:
optional_package = None
import optional.package as optional_package
except ImportError:
pass
...
if optional_package:
# do addtional behavior

AFAIK there is no way to define an optional package and there would be no use to do so. What do you expect when you define an optional package? That it is installed when it is not yet available? (that would somehow make it mandatory)
No, IMHO the correct way to address this is in your imports where you want to use the package. E.g:
try:
from somespecialpackage import someapi as myapi
except ImportError:
from basepackage import theapi as myapi
This of course requires that the two APIs are compatible, but this is the case with simplejson and the standard library json package.

Check if module exists, if not install it

I want to check if a module exists, if it doesn't I want to install it.
How should I do this?
So far I have this code which correctly prints f if the module doesn't exist.
try:
import keyring
except ImportError:
print 'f'

import pip
def import_or_install(package):
try:
__import__(package)
except ImportError:
pip.main(['install', package])
This code simply attempt to import a package, where package is of type str, and if it is unable to, calls pip and attempt to install it from there.

Here is how it should be done, and if I am wrong, please correct me. However, Noufal seems to confirm it in another answer to this question, so I guess it's right.
When writing the setup.py script for some scripts I wrote, I was dependent on the package manager of my distribution to install the required library for me.
So, in my setup.py file, I did this:
package = 'package_name'
try:
return __import__(package)
except ImportError:
return None
So if package_name was installed, fine, continue. Else, install it via the package manager which I called using subprocess.

This approach of dynamic import work really well in cases you just want to print a message if module is not installed. Automatically installing a module SHOULDN'T be done like issuing pip via subprocess. That's why we have setuptools (or distribute).
We have some great tutorials on packaging, and the task of dependencies detection/installation is as simple as providing install_requires=[ 'FancyDependency', 'otherFancy>=1.0' ]. That's just it!
But, if you really NEED to do by hand, you can use setuptools to help you.
from pkg_resources import WorkingSet , DistributionNotFound
working_set = WorkingSet()
# Printing all installed modules
print tuple(working_set)
# Detecting if module is installed
try:
dep = working_set.require('paramiko>=1.0')
except DistributionNotFound:
pass
# Installing it (anyone knows a better way?)
from setuptools.command.easy_install import main as install
install(['django>=1.2'])

NOTE: Ipython / Jupyter specific solution.
While using notebooks / online kernels, I usually do it using systems call.
try:
import keyring
except:
!pip install keyring
import keyring
P.S. One may wish to call conda install or mamba install instead.

You can use os.system as follows:
import os
package = "package_name"
try:
__import__package
except:
os.system("pip install "+ package)

You can launch pip install %s"%keyring in the except part to do this but I don't recommend it. The correct way is to package your application using distutils so that when it's installed, dependencies will be pulled in.

Not all modules can be installed so easily. Not all of them have easy-install support, some can only be installed by building them.. others require some non-python prerequisites, like gcc, which makes things even more complicated (and forget about it working well on Windows).
So I would say you could probably make it work for some predetermined modules, but there's no chance it'll be something generic that works for any module.

I made an import_neccessary_modules() function to fix this common issue.
# ======================================================================================
# == Fix any missing Module, that need to be installed with PIP.exe. [Windows System] ==
# ======================================================================================
import importlib, os
def import_neccessary_modules(modname:str)->None:
'''
Import a Module,
and if that fails, try to use the Command Window PIP.exe to install it,
if that fails, because PIP in not in the Path,
try find the location of PIP.exe and again attempt to install from the Command Window.
'''
try:
# If Module it is already installed, try to Import it
importlib.import_module(modname)
print(f"Importing {modname}")
except ImportError:
# Error if Module is not installed Yet, the '\033[93m' is just code to print in certain colors
print(f"\033[93mSince you don't have the Python Module [{modname}] installed!")
print("I will need to install it using Python's PIP.exe command.\033[0m")
if os.system('PIP --version') == 0:
# No error from running PIP in the Command Window, therefor PIP.exe is in the %PATH%
os.system(f'PIP install {modname}')
else:
# Error, PIP.exe is NOT in the Path!! So I'll try to find it.
pip_location_attempt_1 = sys.executable.replace("python.exe", "") + "pip.exe"
pip_location_attempt_2 = sys.executable.replace("python.exe", "") + "scripts\pip.exe"
if os.path.exists(pip_location_attempt_1):
# The Attempt #1 File exists!!!
os.system(pip_location_attempt_1 + " install " + modname)
elif os.path.exists(pip_location_attempt_2):
# The Attempt #2 File exists!!!
os.system(pip_location_attempt_2 + " install " + modname)
else:
# Neither Attempts found the PIP.exe file, So i Fail...
print(f"\033[91mAbort!!! I can't find PIP.exe program!")
print(f"You'll need to manually install the Module: {modname} in order for this program to work.")
print(f"Find the PIP.exe file on your computer and in the CMD Command window...")
print(f" in that directory, type PIP.exe install {modname}\033[0m")
exit()
import_neccessary_modules('art')
import_neccessary_modules('pyperclip')
import_neccessary_modules('winsound')

Here is my approach. The idea is loop until python has already installed all modules by built in module as "pip" .
import pip
while True:
try:
#import your modules here. !
import seaborn
import bokeh
break
except ImportError as err_mdl:
print((err_mdl.name))
pip.main(['install', err_mdl.name])

I tried installing transformers using the below method and it worked fine. Similarly, you can just replace your library name instead of "transformers".
import pip
try:
from transformers import pipeline
except ModuleNotFoundError:
pip.main(['install', "transformers"])
from transformers import pipeline

I tried this in a new virtual envoirnment with no packages installed and it installed the necessary package i.e. opencv-python
Example is given below
import os
try:
import cv2
except ImportError:
os.system('pip install opencv-python')