Develop Reference

python run ImportError:No module named

This is my project structure. I use virtualenv in my project but when I run it ,it has an ImportError.I use Mac.
But I can run it successfully use Pycharm
So how to run it successfully by Terminal.Because I want to run it in a Ubuntu server with cron
Thanks you for your answers.Here I show my solution.I modify my handler.py I think it may be related to The Module Search Path.
So I add the project path to the PYTHONPATH.
import os
project_home = os.path.realpath(__file__)
project_home = os.path.split(project_home)[0]
import sys
sys.path.append(os.path.split(project_home)[0])
import shutil
from modules import db, json_parse, config_out
from init_log import init as initlog
initlog()
if __name__ == '__main__':
try:
columns = json_parse.json_parse()
if not columns:
sys.exit()
is_table_has_exist = db.check_tables_exist(columns=columns)
if is_table_has_exist:
db.check_columns(columns=columns)
is_ok, config_path = config_out.output(columns)
if is_ok:
file_name = os.path.split(config_path)[1]
shutil.copy(config_path, os.path.join("/app/statics_log/config", file_name))
except Exception, e:
print e
And I run with crontab by this.
cd to/my/py_file/path && /project_path/.env/bin/python /path/to/py_file
example:
13 8 1 * * cd bulu-statics/create_config/ && /home/buka/bulu-statics/.env/bin/python /home/buka/bulu-statics/create_config/handler.py >> /app/statics_log/config/create_config.log

PyCharm automatically adds project directories marked as containing sources to the PYTHONPATH environment variable, whihc is why it works from within pycharm. On the terminal use
PYTHONPATH=${PWD}/..:${PYTHONPATH} python handler.py

You can use explicit relative imports:
from .modules import db, json_parse, config_out
The proper way to do this is to turn your project into a proper Python package by adding a setup.py file and then installing it with pip install -e .

probably because PyCharm added your project folder to the PythonPath, so you can run you app inside PyCharm.
However, when you try to run it from command line, python interpreter cannot find these libs in Python python, so what you need to do is to add your python virtualenv the python python.
there are different ways to adding python path, but I would suggest you to follow:
prepare a setup.py you'll need to specify packages and install_requires.
install your app locally in development mode via pip install -e /path/to/your-package -> it'll create a egg-link in your python virtualenv, you can run your app in your local terminal from now on;
for packing and releasing, you may want to build an artifact by following https://docs.python.org/2.7/distutils/builtdist.html
you may pip install or easy_install the artifact on your other machines. you also can release your package to PyPi if you want.

Using PythonService.exe to host python service while using virtualenv

I've got a Windows 7 environment where I need to develop a Python Windows Service using Python 3.4. I'm using pywin32's win32service module to setup the service and most of the hooks seem to be working ok.
The problem is when I attempt to run the service from source code (using python service.py install followed by python service.py start). This uses PythonService.exe to host service.py - but I'm using a venv virtual environment and the script can't find it's modules (error message discovered with python service.py debug).
Pywin32 is installed in the virtualenv and in looking at the source code of PythonService.exe, it dynamically links in Python34.dll, imports my service.py and invokes it.
How can I get PythonService.exe to use my virtualenv when running my service.py?

Thanks very much for posting this question and a solution. I took a slightly different approach which might also be useful. It is pretty difficult to find working tips for Python services, let alone doing it with a virtualenv. Anyway...
Steps
This is using Windows 7 x64, Python 3.5.1 x64, pywin32-220 (or pypiwin32-219).
Open an Administrator command prompt.
Create a virtualenv. C:\Python35\python -m venv myvenv
Activate the virtualenv. call myvenv\scripts\activate.bat
Install pywin32, either:
From Pypi: pip install pypiwin32,
From http://www.lfd.uci.edu/~gohlke/pythonlibs/: pip install path\to\pywin32.whl
Run the post-install script python myvenv\Scripts\pywin32_postinstall.py -install.
This script registers the DLL's in the system, and copies them to C:\Windows\System32. The DLL's are named pythoncom35.dll and pywintypes35.dll. So virtual environments on the same machine on the same major Python point release will share these... it's a minor tradeoff :)
Copy myvenv\Lib\site-packages\win32\pythonservice.exe to myvenv\Scripts\pythonservice.exe
On the service class (whatever subclasses win32serviceutil.ServiceFramework), set the class property _exe_path_ to point to this relocated exe. This will become the service binPath. For example: _exe_path_ = os.path.join(*[os.environ['VIRTUAL_ENV'], 'Scripts', 'pythonservice.exe']).
Discussion
I think why this works is that Python looks upwards to figure out where the Libs folders are and based on that sets package import paths, similar to the accepted answer. When pythonservice.exe is in the original location, that doesn't seem to work smoothly.
It also resolves DLL linking problems (discoverable with depends.exe from http://www.dependencywalker.com/). Without the DLL business sorted out, it won't be possible to import from the *.pyd files from venv\Lib\site-packages\win32 as modules in your scripts. For example it's needed allow import servicemanager; as servicemanager.pyd is not in the package as a .py file, and has some cool Windows Event Log capabilities.
One of the problems I had with the accepted answer is that I couldn't figure out how to get it to accurately pick up on package.egg-link paths that are created when using setup.py develop. These .egg-link files include the path to the package when it's not located in the virtualenv under myvenv\Lib\site-packages.
If it all went smoothly, it should be possible to install, start and test the example win32 service (from an Admin prompt in the activated virtualenv):
python venv\Lib\site-packages\win32\Demos\service\pipeTestService.py install
python venv\Lib\site-packages\win32\Demos\service\pipeTestService.py start
python venv\Lib\site-packages\win32\Demos\service\pipeTestServiceClient.py
The Service Environment
Another important note in all this is that the service will execute the python code in a completely separate environment to the one you might run python myservice.py debug. So for example os.environ['VIRTUAL_ENV'] will be empty when running the service. This can be handled by either:
Setting environment variables from inside the script, e.g.
Find current path starting from the sys.executable, as described in the accepted answer.
Use that path to locate a config file.
Read the config file and put them in the environment with os.environ.
Add registry keys to the service with the environment variables.
See Accessing Environment Variables from Windows Services for doing this manually with regedit.exe
See REG ADD a REG_MULTI_SZ Multi-Line Registry Value for doing this from the command line.

I read all the answers, but no solution can fix my problem.
After carefully researched David K. Hess's code, I made some change, and it finally works.
But my reputation doesn't enough, so I just post the code here.
# 1. Custom your Project's name and Virtual Environment folder's name
# 2. Import this before all third part models
# 3. If you still failed, check the link below:
# https://stackoverflow.com/questions/34696815/using-pythonservice-exe-to-host-python-service-while-using-virtualenv
# 2019-05-29 by oraant, modified from David K. Hess's answer.
import os, sys, site
project_name = "PythonService" # Change this for your own project !!!!!!!!!!!!!!
venv_folder_name = "venv" # Change this for your own venv path !!!!!!!!!!!!!!
if sys.executable.lower().endswith("pythonservice.exe"):
# Get root path for the project
service_directory = os.path.abspath(os.path.dirname(__file__))
project_directory = service_directory[:service_directory.find(project_name)+len(project_name)]
# Get venv path for the project
def file_path(x): return os.path.join(project_directory, x)
venv_base = file_path(venv_folder_name)
venv_scripts = os.path.join(venv_base, "Scripts")
venv_packages = os.path.join(venv_base, 'Lib', 'site-packages')
# Change current working directory from PythonService.exe location to something better.
os.chdir(project_directory)
sys.path.append(".")
prev_sys_path = list(sys.path)
# Manually activate a virtual environment inside an already initialized interpreter.
os.environ['PATH'] = venv_scripts + os.pathsep + os.environ['PATH']
site.addsitedir(venv_packages)
sys.real_prefix = sys.prefix
sys.prefix = venv_base
# Move some sys path in front of others
new_sys_path = []
for item in list(sys.path):
if item not in prev_sys_path:
new_sys_path.append(item)
sys.path.remove(item)
sys.path[:0] = new_sys_path
How to use it? It's simple, just paste it into a new python file, and import it before any third part model like this:
import service_in_venv # import at top
import win32serviceutil
import win32service
import win32event
import servicemanager
import time
import sys, os
........
And now you should fix your problem.

It appears this used to work correctly with the virtualenv module before virtual environments were added to Python 3.3. There's anecdotal evidence (see this answer: https://stackoverflow.com/a/12424980/1055722) that Python's site.py used to look upward from the executable file until it found a directory that would satisfy imports. It would then use that for sys.prefix and this was sufficient for PythonService.exe to find the virtualenv it was inside of and use it.
If that was the behavior, it appears that site.py no longer does that with the introduction of the venv module. Instead, it looks one level up for a pyvenv.cfg file and configures for a virtual environment in that case only. This of course doesn't work for PythonService.exe which is buried down in the pywin32 module under site-packages.
To work around it, I adapted the activate_this.py code that comes with the original virtualenv module (see this answer: https://stackoverflow.com/a/33637378/1055722). It is used to bootstrap an interpreter embedded in an executable (which is the case with PythonService.exe) into using a virtualenv. Unfortunately, venv does not include this.
Here's what worked for me. Note, this assumes the virtual environment is named my-venv and is located one level above the source code location.
import os
import sys
if sys.executable.endswith("PythonService.exe"):
# Change current working directory from PythonService.exe location to something better.
service_directory = os.path.dirname(__file__)
source_directory = os.path.abspath(os.path.join(service_directory, ".."))
os.chdir(source_directory)
sys.path.append(".")
# Adapted from virtualenv's activate_this.py
# Manually activate a virtual environment inside an already initialized interpreter.
old_os_path = os.environ['PATH']
venv_base = os.path.abspath(os.path.join(source_directory, "..", "my-venv"))
os.environ['PATH'] = os.path.join(venv_base, "Scripts") + os.pathsep + old_os_path
site_packages = os.path.join(venv_base, 'Lib', 'site-packages')
prev_sys_path = list(sys.path)
import site
site.addsitedir(site_packages)
sys.real_prefix = sys.prefix
sys.prefix = venv_base
new_sys_path = []
for item in list(sys.path):
if item not in prev_sys_path:
new_sys_path.append(item)
sys.path.remove(item)
sys.path[:0] = new_sys_path
One other factor in my troubles - there is a new pypi wheel for pywin32 that is provided by the Twisted folks that makes it easier to install with pip. The PythonService.exe in that package was acting oddly (couldn't find a pywin32 dll when invoked) compared to the one you get when installing the official win32 exe package into the virtual env using easy_install.

For anyone reading in 2018, I didn't have any luck with either solution above (Win10, Python 3.6) - so this is what I did to get it working. The working directory is in site-packages/win32 on launch, so you need to change the working directory and fix the sys.path before you try and import any project code. This assumed venv sits in your project dir, otherwise you may just need to hard code some paths:
import sys
import os
if sys.executable.lower().endswith("pythonservice.exe"):
for i in range(4): # goes up 4 directories to project folder
os.chdir("..")
# insert site-packages 2nd in path (behind project folder)
sys.path.insert(1, os.path.join("venv",'Lib','site-packages'))
[REST OF IMPORTS]
class TestService(win32serviceutil.ServiceFramework):
[...]

Not use "pythonservice.exe", register python.exe to services directly:
import win32serviceutil
import win32service
import servicemanager
import sys
import os
import os.path
import multiprocessing
#
def main():
import time
time.sleep(600)
class ProcessService(win32serviceutil.ServiceFramework):
_svc_name_ = "SleepService"
_svc_display_name_ = "Sleep Service"
_svc_description_ = "Sleeps for 600"
_exe_name_ = sys.executable # python.exe from venv
_exe_args_ = '-u -E "' + os.path.abspath(__file__) + '"'
proc = None
def SvcStop(self):
self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
if self.proc:
self.proc.terminate()
def SvcRun(self):
self.proc = multiprocessing.Process(target=main)
self.proc.start()
self.ReportServiceStatus(win32service.SERVICE_RUNNING)
self.SvcDoRun()
self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
def SvcDoRun(self):
self.proc.join()
def start():
if len(sys.argv)==1:
import win32traceutil
servicemanager.Initialize()
servicemanager.PrepareToHostSingle(ProcessService)
servicemanager.StartServiceCtrlDispatcher()
elif '--fg' in sys.argv:
main()
else:
win32serviceutil.HandleCommandLine(ProcessService)
if __name__ == '__main__':
try:
start()
except (SystemExit, KeyboardInterrupt):
raise
except:
import traceback
traceback.print_exc()
Its make python 3.5+ virtualenv support to work by pointing right iterpreter with service install.

Creating virtualenv in a Python script and running the rest in that env

Based on this link I created and activated a virtual environment in a Python script but what I want is after activating the virtualenv the rest of the code should run in the env, install few items and then process.
Code:
#!/usr/bin/python
import commands
import os
import time
import datetime
import sys
import json
import requests
out = commands.getoutput("wget <url>/s.sh")
new = commands.getoutput("chmod 755 s.sh")
env = "virtualenv " + "test"
#checking if env present if not create and activate it
try:
l=commands.getoutput('/bin/bash --rcfile s.sh')
except:
print env + " not present"
m = commands.getoutput(env)
print env + " not present so creating"
os.system('/bin/bash --rcfile s.sh')
v = commands.getoutput("which python")
print v + " python to be used"
v = commands.getoutput("pip install wget")
s.sh file code:-
#!/bin/sh
. test/bin/activate
Basically instead of a shell script to create virtualenv, activate it, and run a few steps I want to use python script.
What am I missing? Is it right use case?

I've found the virtual env activation stuff to all be a bit too much like hard work in scripts. In your case you're doing it from python instead of the shell, but I guess it's the same basic principal (haven't seen commands before - are they all running in the same subprocess)?
Instead, I normally just use the full path to the executables in the venv. It's all more explicit, but then again, that is the python way :)
One thing you have to watch out for is permissions.
sudo -u whatever_user /path_to_myvenv/bin/pip install -r /some_path/requirements.txt
Not sure if that's helpful at all, but I'd be looking to call the binary in the venv directly.
EDIT just had a play with commands - each of them is independent (so that's probably the root cause of your issue)
commands.getoutput('pwd') # /somepath
commands.getoutput('cd /tmp')
commands.getoutput('pwd') # still /somepath
I'd be looking to do something like:
commands.getoutput('virtualenv test')
commands.getoutput('test/bin/pip install wget')

Automatically load a virtualenv when running a script

I have a python script that needs dependencies from a virtualenv. I was wondering if there was some way I could add it to my path and have it auto start it's virtualenv, run and then go back to the system's python.
I've try playing around with autoenv and .env but that doesn't seem to do exactly what I'm looking for. I also thought about changing the shabang to point to the virtualenv path but that seems fragile.

There are two ways to do this:
Put the name of the virtual env python into first line of the script. Like this
#!/your/virtual/env/path/bin/python
Add virtual environment directories to the sys.path. Note that you need to import sys library. Like this
import sys
sys.path.append('/path/to/virtual/env/lib')
If you go with the second option you might need to add multiple paths to the sys.path (site etc). The best way to get it is to run your virtual env python interpreter and fish out the sys.path value. Like this:
/your/virtual/env/bin/python
Python blah blah blah
> import sys
> print sys.path
[ 'blah', 'blah' , 'blah' ]
Copy the value of sys.path into the snippet above.

I'm surprised that nobody has mentioned this yet, but this is why there is a file called activate_this.py in the virtualenv's bin directory. You can pass that to execfile() to alter the module search path for the currently running interpreter.
# doing execfile() on this file will alter the current interpreter's
# environment so you can import libraries in the virtualenv
activate_this_file = "/path/to/virtualenv/bin/activate_this.py"
execfile(activate_this_file, dict(__file__=activate_this_file))
You can put this file at the top of your script to force the script to always run in that virtualenv. Unlike the modifying hashbang, you can use relative path with by doing:
script_directory = os.path.dirname(os.path.abspath(__file__))
activate_this_file = os.path.join(script_directory, '../../relative/path/to/env/bin/activate_this.py')

From the virtualenv documentation:
If you directly run a script or the python interpreter from the
virtualenv’s bin/ directory (e.g. path/to/env/bin/pip or
/path/to/env/bin/python script.py) there’s no need for activation.
So if you just call the python executable in your virtualenv, your virtualenv will be 'active'. So you can create a script like this:
#!/bin/bash
PATH_TO_MY_VENV=/opt/django/ev_scraper/venv/bin
$PATH_TO_MY_VENV/python -c 'import sys; print(sys.version_info)'
python -c 'import sys; print(sys.version_info)'
When I run this script on my system, the two calls to python print what you see below. (Python 3.2.3 is in my virtualenv, and 2.7.3 is my system Python.)
sys.version_info(major=3, minor=2, micro=3, releaselevel='final', serial=0)
sys.version_info(major=2, minor=7, micro=3, releaselevel='final', serial=0)
So any libraries you have installed in your virtualenv will be available when you call $PATH_TO_MY_VENV/python. Calls to your regular system python will of course be unaware of whatever is in the virtualenv.

I think the best answer here is to create a simple script and install it inside your virtualenv. Then you can either directly use the script, or create a symlink, or whatever.
Here's an example:
$ mkdir my-tool
$ cd my-tool
$ mkdir scripts
$ touch setup.py
$ mkdir scripts
$ touch scripts/crunchy-frog
$ chmod +x scripts/crunchy-frog
crunchy-frog
#!/usr/bin/env python
print("Constable Parrot ate one of those!")
setup.py
from setuptools import setup
setup(name="my-cool-tool",
scripts=['scripts/crunchy-frog'],
)
Now:
$ source /path/to/my/env/bin/activate
(env) $ python setup.py develop
(env) $ deactivate
$ cd ~
$ ln -s /path/to/my/env/bin/crunchy-frog crunchy-frog
$ ./crunchy-frog
Constable Parrot ate one of those!
When you install your script (via setup.py install or setup.py develop) then it will replace the first line of the scripts with a shebang line for the env python (which you can verify with $ head /path/to/my/env/bin/crunchy-frog). So whenever you run that particular script, it will use that specific Python env.

Does this help?
import site
site.addsitedir('/path/to/virtualenv/lib/python2.7/site-packages/')

I had this problem before and I made a simple script to look for a virtualenv folder recursively just importing and calling a function:
script_autoenv.py
# -*- coding:utf-8 -*-
import os, site
def locate_env(path, env_name):
"""search for a env directory name in each directory in the path"""
if os.path.isdir(path + "/env"):
env_26_path = '%s/%s/lib/python2.6/site-packages/' % (path, env_name)
env_27_path = '%s/%s/lib/python2.7/site-packages/' % (path, env_name)
if os.path.isdir(env_26_path):
site.addsitedir(env_26_path)
print "Virtualenv 2.6 founding"
elif os.path.isdir(env_27_path):
site.addsitedir(env_27_path)
print "Virtualenv 2.7 founding"
else:
new_path, old_dir = os.path.split(path)
if old_dir:
locate_env(new_path, env_name)
else:
print "No envs found"
You just need to specify the script directory and the env name folder and the script do the rest:
test.py
# -*- coding:utf-8 -*-
import os
import script_autoenv
script_autoenv.locate_env(os.path.realpath(__file__), 'env')
import django
print django.VERSION
I hope it's works for you

The answer may be pipenv (https://pipenv.readthedocs.io/en/latest/).
It will allow you to do something like:
pipenv run python main.py
to run main.py in the python environment with the specified libraries.
You can give it a try here https://rootnroll.com/d/pipenv/
...Maybe is not exactly what you are looking for, but it may be worth taking a look before reinventing it.

easy way of installing python apps without using PYTHON path or muli symlink in site-package

I didn't want to install python modules using easy install, symlinks in site-packages or PYTHONPATH.
So, I am trying something that I do wants system wide, then any application installation is done locally. Note, the root password is required only once here.
First create a symblink of.../pythonX.Y/site-packages/mymodules -> /home/me/lib/python_related
So, I create a directory called
/home/me/lib/python_related/
In there:
/home/me/lib/python_related
/home/me/lib/python_related/__init__.py
/home/me/lib/python_related/django_related/
/home/me/lib/python_related/django_related/core
/home/me/lib/python_related/django_related/core/Django1.0
/home/me/lib/python_related/django_related/core/Django1.1
/home/me/lib/python_related/django_related/core/mycurrent_django -> Django1.1/django
/home/me/lib/python_related/django_related/apps
/home/me/lib/python_related/django_related/apps/tagging
/home/me/lib/python_related/django_related/apps/tagging/django-tagging-0.2
/home/me/lib/python_related/django_related/apps/tagging/django-tagging-0.3
/home/me/lib/python_related/django_related/apps/tagging/mycurrent_tagging -> django-tagging-0.3
Now, here is the content of:
/home/me/lib/python_related/__init__.py
==========================================
import sys, os
# tell us where you keep all your modules and this didn't work as it gave me
# the location of the site-packages
#PYTHON_MODULE_PATH = os.path.dirname(__file__)
PYTHON_MODULE_PATH = "/home/me/libs/python_bucket"
def run_cmd(cmd):
"""
Given a command name, this function will run the command and returns the output
in a list.
"""
output = []
phdl = os.popen(cmd)
while 1:
line = phdl.readline()
if line == "":
break
output.append(line.replace("\n", ""))
return output
def install():
"""
A cheesy way of installing and managing your python apps locally without
a need to install them in the site-package. All you'd need is to install
the directory containing this file in the site-package and that's it.
Anytime you have a python package you want to install, just put it in a
proper sub-directory and make a symlink to that directory called mycurrent_xyz
and you are done. (e.g. mycurrent_django, mycurrent_tagging .. etc)
"""
cmd = "find %s -name mycurrent_*" % PYTHON_MODULE_PATH
modules_to_be_installed = run_cmd(cmd)
sys.path += modules_to_be_installed
install()
=======================================================
Now in any new python project, just import your mymodules and that pulls in any apps that you have in the above directory with the proper symbolic link. This way you can have multiple copies of apps and just use the mycurrent_xyz to the one you want to use.
Now here is question. Is this a good way of doing it?

Have a look at virtualenv.
It may do what you are after.