I am trying to use the google API client to download something every day for something but I have deduced that for some reason it won't let me import googleapiclient when I run it in crontab.
For example, if I run this in crontab
crontab: * * * * * python3 test.py >> cron.log
test.py:
print("Hello")
cron.log outputs:
Hello
But if I then once I import it so the file look like this
from googleapiclient import discovery
print("Hello")
then the cron.log looks like this:
Its just empty. I can not figure out why this is true. The google API client I believe is installed correctly because when I run it manually then it works perfectly with no issues.
The operating system I am using is macos.
The way I fixed it is first by changing the crontab to python3 test.py > cron.log 2>&1. This showed that the errors where. Then I realized that it was not able to import googleapiclient into python. I still did not understand why it does not, but a workaround I found was to first install the libraries into a folder and then accessing the libraries from there.
To install the libraries into a folder the command I used in terminal is below:
pip3 install --upgrade google-api-python-client google-auth-httplib2 google-auth-oauthlib -t ./lib
Then inside the python I added whats below to the beginning of the file so it would know where to import the libraries from.
import os
import sys
file_path = os.path.dirname(__file__)
module_path = os.path.join(file_path, "lib")
sys.path.append(module_path)
This does work.
Not sure if this is more google-cloud-related or pytest-related. See files below.
When I run either python app/my_script.py or python -m app.my_script, the script runs fine.
But when I run pytest, the line in the script from google.cloud import vision throws "ModuleNotFoundError: No module named 'google.cloud'".
I have tried unsuccessfully to add various package names into the requirements.txt file and/or run pip install google-cloud and pip install google-cloud-language with and without --upgrade flags. What steps can I take to overcome this error?
conftest.py: (empty)
requirements.txt:
google-cloud-vision
app/my_script.py:
from google.cloud import vision
from google.cloud.vision import types
def new_client():
client = vision.ImageAnnotatorClient()
return client
if __name__ == "__main__":
client = new_client()
# etc...
test/test_my_script.py:
from app.my_script import new_client
# tests here...
I'd like to have a python script which in the beginning create a virtual environment, install required modules (e.g. cherrypy) and then continues with the rest of the code.
What I found so far is as follow:
import os, virtualenv
HOME_DIRECTORY = "venv"
virtualenv.create_environment(HOME_DIRECTORY)
execfile(os.path.join(HOME_DIRECTORY, "bin", "activate_this.py"))
import pip
pip.main(["install", "--prefix", HOME_DIRECTORY, "cherrypy"])
import cherrypy
class Root(object):
#cherrypy.expose
def index(self):
return "Hello World!"
if __name__ == '__main__':
cherrypy.quickstart(Root(), '/')
The script creates virtual environment and install cherrypy (based on the log), but still I get ImportError: No module named cherrypy error.
I also tried the following, but got the same error:
import importlib
CHERRYPY = 'cherrypy'
try:
importlib.import_module(CHERRYPY)
except ImportError:
import pip
pip.main(["install", "--prefix", HOME_DIRECTORY, CHERRYPY])
finally:
globals()[CHERRYPY] = importlib.import_module(CHERRYPY)
Apart from the above problem, also, I'd like to know how to specify the python version (e.g. python 3.6) while creating virtual environment. Thanks.
For the record the issue was with the virtual environment activate (the line to execute activate_this.py). For Python 3 the following worked for me:
ACTIVATE_THIS = 'venv/bin/activate_this.py'
exec(Path(ACTIVATE_THIS).read_text(), dict(__file__ = ACTIVATE_THIS))
I tried the following simple code,
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
return "Hello World!"
if __name__ == "__main__":
app.run()
it is running fine with,
python hello.py
but it gives an error when i try with python3
ImportError: cannot import name 'Flask'
A package is installed against a specific Python version/location. Installing Flask for Python 2 (which is probably what the python and pip commands are aliased to), doesn't install it for Python 3.
You should really just use virtualenv to control exactly what versions and packages you are using.
This creates a Python 3 environment and installs Flask:
virtualenv -p /usr/bin/python3 my_py3_env
source my_py3_env/bin/activate
pip install flask
When you open a new terminal, just source the activate script again to keep using the environment.
I want to write a script to automatically setup a brand new ubuntu installation and install a django-based app. Since the script will be run on a new server, the Python script needs to automatically install some required modules.
Here is the script.
#!/usr/bin/env python
import subprocess
import os
import sys
def pip_install(mod):
print subprocess.check_output("pip install %s" % mod, shell=True)
if __name__ == "__main__":
if os.getuid() != 0:
print "Sorry, you need to run the script as root."
sys.exit()
try:
import pexpect
except:
pip_install('pexpect')
import pexpect
# More code here...
The installation of pexpect is success, however the next line import pexpect is failed. I think its because at runtime the code doesn't aware about the newly installed pexpect.
How to install and import Python modules at runtime? I'm open to another approaches.
You can import pip instead of using subprocess:
import pip
def install(package):
pip.main(['install', package])
# Example
if __name__ == '__main__':
try:
import pexpect
except ImportError:
install('pexpect')
import pexpect
Another take:
import pip
def import_with_auto_install(package):
try:
return __import__(package)
except ImportError:
pip.main(['install', package])
return __import__(package)
# Example
if __name__ == '__main__':
pexpect = import_with_auto_install('pexpect')
print(pexpect)
[edit]
You should consider using a requirements.txt along with pip. Seems like you are trying to automate deployments (and this is good!), in my tool belt I have also virtualenvwrapper, vagrant and ansible.
This is the output for me:
(test)root#vagrant:~/test# pip uninstall pexpect
Uninstalling pexpect:
/usr/lib/python-environments/test/lib/python2.6/site-packages/ANSI.py
/usr/lib/python-environments/test/lib/python2.6/site-packages/ANSI.pyc
/usr/lib/python-environments/test/lib/python2.6/site-packages/FSM.py
/usr/lib/python-environments/test/lib/python2.6/site-packages/FSM.pyc
/usr/lib/python-environments/test/lib/python2.6/site-packages/fdpexpect.py
/usr/lib/python-environments/test/lib/python2.6/site-packages/fdpexpect.pyc
/usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect-2.4-py2.6.egg-info
/usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect.py
/usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect.pyc
/usr/lib/python-environments/test/lib/python2.6/site-packages/pxssh.py
/usr/lib/python-environments/test/lib/python2.6/site-packages/pxssh.pyc
/usr/lib/python-environments/test/lib/python2.6/site-packages/screen.py
/usr/lib/python-environments/test/lib/python2.6/site-packages/screen.pyc
Proceed (y/n)? y
Successfully uninstalled pexpect
(test)root#vagrant:~/test# python test.py
Downloading/unpacking pexpect
Downloading pexpect-2.4.tar.gz (113Kb): 113Kb downloaded
Running setup.py egg_info for package pexpect
Installing collected packages: pexpect
Running setup.py install for pexpect
Successfully installed pexpect
Cleaning up...
<module 'pexpect' from '/usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect.pyc'>
(test)root#vagrant:~/test#
For those who are using pip version greater than 10.x, there is no main function for pip so the alternative approach is using import pip._internal as pip instead of import pip like :
Updated answer of Paulo
import pip._internal as pip
def install(package):
pip.main(['install', package])
if __name__ == '__main__':
try:
import pexpect
except ImportError:
install('pexpect')
import pexpect
I actually made a module for this exact purpose (impstall)
It's really easy to use:
import impstall
impstall.now('pexpect')
impstall.now('wx', pipName='wxPython')
Github link for issues/contributions
I solved my problem using the imp module.
#!/usr/bin/env python
import pip
import imp
def install_and_load(package):
pip.main(['install', package])
path = '/usr/local/lib/python2.7/dist-packages'
if path not in sys.path:
sys.path.append(path)
f, fname, desc = imp.find_module(package)
return imp.load(package, f, fname, desc)
if __name__ == "__main__":
try:
import pexpect
except:
pexpect = install_and_load('pexpect')
# More code...
Actually the code is less than ideal, since I need to hardcode the Python module directory. But since the script is intended for a known target system, I think that is ok.
I had the same issue but none of Google's searches helped. After hours debugging, I found that it may be because the sys.path is not reloaded with new installation directory.
In my case on my Ubuntu Docker, I want to import dns.resolver at runtime for Python3.8 (pre-installed). I also created ubuntu user and run all things with this user (including my Python script).
Before installing, sys.path doesn't have /home/ubuntu/.local/lib/python3.8/site-packages since I didn't install anything.
While installing with subprocess or pip.main like above, it creates /home/ubuntu/.local/lib/python3.8/site-packages (as user installation).
After installing , the sys.path should be refreshed to include this new location.
Since the sys.path is managed by site module, we should reload it (ref HERE):
import site
from importlib import reload
reload(site)
The full block for anyone that needs:
import subprocess
import sys
try:
import dns.resolver
except ImportError:
subprocess.check_call([sys.executable, "-m", "pip", "install", "dnspython"])
import site
from importlib import reload
reload(site)
import dns.resolver
I'm not Python developer so these code can be simplify more. This may help in cases such as fresh CI/CD environment for DevOps engineers like me.