I've got a Windows 7 environment where I need to develop a Python Windows Service using Python 3.4. I'm using pywin32's win32service module to setup the service and most of the hooks seem to be working ok.
The problem is when I attempt to run the service from source code (using python service.py install followed by python service.py start). This uses PythonService.exe to host service.py - but I'm using a venv virtual environment and the script can't find it's modules (error message discovered with python service.py debug).
Pywin32 is installed in the virtualenv and in looking at the source code of PythonService.exe, it dynamically links in Python34.dll, imports my service.py and invokes it.
How can I get PythonService.exe to use my virtualenv when running my service.py?
Thanks very much for posting this question and a solution. I took a slightly different approach which might also be useful. It is pretty difficult to find working tips for Python services, let alone doing it with a virtualenv. Anyway...
Steps
This is using Windows 7 x64, Python 3.5.1 x64, pywin32-220 (or pypiwin32-219).
Open an Administrator command prompt.
Create a virtualenv. C:\Python35\python -m venv myvenv
Activate the virtualenv. call myvenv\scripts\activate.bat
Install pywin32, either:
From Pypi: pip install pypiwin32,
From http://www.lfd.uci.edu/~gohlke/pythonlibs/: pip install path\to\pywin32.whl
Run the post-install script python myvenv\Scripts\pywin32_postinstall.py -install.
This script registers the DLL's in the system, and copies them to C:\Windows\System32. The DLL's are named pythoncom35.dll and pywintypes35.dll. So virtual environments on the same machine on the same major Python point release will share these... it's a minor tradeoff :)
Copy myvenv\Lib\site-packages\win32\pythonservice.exe to myvenv\Scripts\pythonservice.exe
On the service class (whatever subclasses win32serviceutil.ServiceFramework), set the class property _exe_path_ to point to this relocated exe. This will become the service binPath. For example: _exe_path_ = os.path.join(*[os.environ['VIRTUAL_ENV'], 'Scripts', 'pythonservice.exe']).
Discussion
I think why this works is that Python looks upwards to figure out where the Libs folders are and based on that sets package import paths, similar to the accepted answer. When pythonservice.exe is in the original location, that doesn't seem to work smoothly.
It also resolves DLL linking problems (discoverable with depends.exe from http://www.dependencywalker.com/). Without the DLL business sorted out, it won't be possible to import from the *.pyd files from venv\Lib\site-packages\win32 as modules in your scripts. For example it's needed allow import servicemanager; as servicemanager.pyd is not in the package as a .py file, and has some cool Windows Event Log capabilities.
One of the problems I had with the accepted answer is that I couldn't figure out how to get it to accurately pick up on package.egg-link paths that are created when using setup.py develop. These .egg-link files include the path to the package when it's not located in the virtualenv under myvenv\Lib\site-packages.
If it all went smoothly, it should be possible to install, start and test the example win32 service (from an Admin prompt in the activated virtualenv):
python venv\Lib\site-packages\win32\Demos\service\pipeTestService.py install
python venv\Lib\site-packages\win32\Demos\service\pipeTestService.py start
python venv\Lib\site-packages\win32\Demos\service\pipeTestServiceClient.py
The Service Environment
Another important note in all this is that the service will execute the python code in a completely separate environment to the one you might run python myservice.py debug. So for example os.environ['VIRTUAL_ENV'] will be empty when running the service. This can be handled by either:
Setting environment variables from inside the script, e.g.
Find current path starting from the sys.executable, as described in the accepted answer.
Use that path to locate a config file.
Read the config file and put them in the environment with os.environ.
Add registry keys to the service with the environment variables.
See Accessing Environment Variables from Windows Services for doing this manually with regedit.exe
See REG ADD a REG_MULTI_SZ Multi-Line Registry Value for doing this from the command line.
I read all the answers, but no solution can fix my problem.
After carefully researched David K. Hess's code, I made some change, and it finally works.
But my reputation doesn't enough, so I just post the code here.
# 1. Custom your Project's name and Virtual Environment folder's name
# 2. Import this before all third part models
# 3. If you still failed, check the link below:
# https://stackoverflow.com/questions/34696815/using-pythonservice-exe-to-host-python-service-while-using-virtualenv
# 2019-05-29 by oraant, modified from David K. Hess's answer.
import os, sys, site
project_name = "PythonService" # Change this for your own project !!!!!!!!!!!!!!
venv_folder_name = "venv" # Change this for your own venv path !!!!!!!!!!!!!!
if sys.executable.lower().endswith("pythonservice.exe"):
# Get root path for the project
service_directory = os.path.abspath(os.path.dirname(__file__))
project_directory = service_directory[:service_directory.find(project_name)+len(project_name)]
# Get venv path for the project
def file_path(x): return os.path.join(project_directory, x)
venv_base = file_path(venv_folder_name)
venv_scripts = os.path.join(venv_base, "Scripts")
venv_packages = os.path.join(venv_base, 'Lib', 'site-packages')
# Change current working directory from PythonService.exe location to something better.
os.chdir(project_directory)
sys.path.append(".")
prev_sys_path = list(sys.path)
# Manually activate a virtual environment inside an already initialized interpreter.
os.environ['PATH'] = venv_scripts + os.pathsep + os.environ['PATH']
site.addsitedir(venv_packages)
sys.real_prefix = sys.prefix
sys.prefix = venv_base
# Move some sys path in front of others
new_sys_path = []
for item in list(sys.path):
if item not in prev_sys_path:
new_sys_path.append(item)
sys.path.remove(item)
sys.path[:0] = new_sys_path
How to use it? It's simple, just paste it into a new python file, and import it before any third part model like this:
import service_in_venv # import at top
import win32serviceutil
import win32service
import win32event
import servicemanager
import time
import sys, os
........
And now you should fix your problem.
It appears this used to work correctly with the virtualenv module before virtual environments were added to Python 3.3. There's anecdotal evidence (see this answer: https://stackoverflow.com/a/12424980/1055722) that Python's site.py used to look upward from the executable file until it found a directory that would satisfy imports. It would then use that for sys.prefix and this was sufficient for PythonService.exe to find the virtualenv it was inside of and use it.
If that was the behavior, it appears that site.py no longer does that with the introduction of the venv module. Instead, it looks one level up for a pyvenv.cfg file and configures for a virtual environment in that case only. This of course doesn't work for PythonService.exe which is buried down in the pywin32 module under site-packages.
To work around it, I adapted the activate_this.py code that comes with the original virtualenv module (see this answer: https://stackoverflow.com/a/33637378/1055722). It is used to bootstrap an interpreter embedded in an executable (which is the case with PythonService.exe) into using a virtualenv. Unfortunately, venv does not include this.
Here's what worked for me. Note, this assumes the virtual environment is named my-venv and is located one level above the source code location.
import os
import sys
if sys.executable.endswith("PythonService.exe"):
# Change current working directory from PythonService.exe location to something better.
service_directory = os.path.dirname(__file__)
source_directory = os.path.abspath(os.path.join(service_directory, ".."))
os.chdir(source_directory)
sys.path.append(".")
# Adapted from virtualenv's activate_this.py
# Manually activate a virtual environment inside an already initialized interpreter.
old_os_path = os.environ['PATH']
venv_base = os.path.abspath(os.path.join(source_directory, "..", "my-venv"))
os.environ['PATH'] = os.path.join(venv_base, "Scripts") + os.pathsep + old_os_path
site_packages = os.path.join(venv_base, 'Lib', 'site-packages')
prev_sys_path = list(sys.path)
import site
site.addsitedir(site_packages)
sys.real_prefix = sys.prefix
sys.prefix = venv_base
new_sys_path = []
for item in list(sys.path):
if item not in prev_sys_path:
new_sys_path.append(item)
sys.path.remove(item)
sys.path[:0] = new_sys_path
One other factor in my troubles - there is a new pypi wheel for pywin32 that is provided by the Twisted folks that makes it easier to install with pip. The PythonService.exe in that package was acting oddly (couldn't find a pywin32 dll when invoked) compared to the one you get when installing the official win32 exe package into the virtual env using easy_install.
For anyone reading in 2018, I didn't have any luck with either solution above (Win10, Python 3.6) - so this is what I did to get it working. The working directory is in site-packages/win32 on launch, so you need to change the working directory and fix the sys.path before you try and import any project code. This assumed venv sits in your project dir, otherwise you may just need to hard code some paths:
import sys
import os
if sys.executable.lower().endswith("pythonservice.exe"):
for i in range(4): # goes up 4 directories to project folder
os.chdir("..")
# insert site-packages 2nd in path (behind project folder)
sys.path.insert(1, os.path.join("venv",'Lib','site-packages'))
[REST OF IMPORTS]
class TestService(win32serviceutil.ServiceFramework):
[...]
Not use "pythonservice.exe", register python.exe to services directly:
import win32serviceutil
import win32service
import servicemanager
import sys
import os
import os.path
import multiprocessing
#
def main():
import time
time.sleep(600)
class ProcessService(win32serviceutil.ServiceFramework):
_svc_name_ = "SleepService"
_svc_display_name_ = "Sleep Service"
_svc_description_ = "Sleeps for 600"
_exe_name_ = sys.executable # python.exe from venv
_exe_args_ = '-u -E "' + os.path.abspath(__file__) + '"'
proc = None
def SvcStop(self):
self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
if self.proc:
self.proc.terminate()
def SvcRun(self):
self.proc = multiprocessing.Process(target=main)
self.proc.start()
self.ReportServiceStatus(win32service.SERVICE_RUNNING)
self.SvcDoRun()
self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
def SvcDoRun(self):
self.proc.join()
def start():
if len(sys.argv)==1:
import win32traceutil
servicemanager.Initialize()
servicemanager.PrepareToHostSingle(ProcessService)
servicemanager.StartServiceCtrlDispatcher()
elif '--fg' in sys.argv:
main()
else:
win32serviceutil.HandleCommandLine(ProcessService)
if __name__ == '__main__':
try:
start()
except (SystemExit, KeyboardInterrupt):
raise
except:
import traceback
traceback.print_exc()
Its make python 3.5+ virtualenv support to work by pointing right iterpreter with service install.
Related
I need a python script that will activate a virtualenv, run another python program inside the virtualenv, and then close the virutalenv after the second python program closes. Here is my code:
import os
import subprocess
from subprocess import Popen
activate_dir = "C:/Users/JohnDoe/theprogram/Scripts/"
os.chdir(activate_dir)
subprocess.Popen(["activate.bat"])
cal_dir = "C:/Users/JohnDoe/theprogram/"
os.chdir(cal_dir)
os.system('python program_file.py')
However, when this code run, I get an import error from the program_file which means the virtualenv is not activated. How can I fix this?
Thanks
Edit:
This is on a Windows environment.
The issue is that you are creating a new process with subprocess.Popen(["activate.bat"]) that is using that virtual environment, you're not changing your environment. What you need to do is to either call the python script in the same process you span:
os.system("source activate;python -V")
Or you could write a shell script that starts the virtual environment and calls any python script you send to it. In bash (on linux) this would be:
#!/bin/bash
# start a virtual environment and call a python module
# usage: ./runVirenvPythonModule module.py
source activate
python $1 # this is the first cmd line argument passed in
I've found a method to detect, activate, and create (if needed) a virtual environment inside a Python script and run inside that virtual environment while remaining inside that script and without the use of shell commands issued from that script (except to display installed packages via pip list). Without the use of shell commands, the script becomes OS agnostic.
Here is some example code. You simply run it from whatever OS shell you are using (Windows, Linux, MacOS):
import os
import sys
import venv
def isvirtualenv():
return sys.prefix != sys.base_prefix
def findfile(startdir, pattern):
for root, dirs, files in os.walk(startdir):
for name in files:
if name.find(pattern) >= 0:
return root + os.sep + name
return None
venv_path = 'venv' # This is an example path
if isvirtualenv():
print('Already in virtual environment.')
else:
if findfile(os.getcwd(), 'activate') is None:
print('No virtual environment found. Creating one.')
env = venv.EnvBuilder(with_pip = True)
env.create(venv_path)
else:
print('Not in virtual environment. Virtual environment directory found.')
# This is the heart of this script that puts you inside the virtual environment.
# There is no need to undo this. When this script ends, your original path will
# be restored.
os.environ['PATH'] = os.path.dirname(findfile(os.getcwd(), 'activate')) + os.pathsep + os.environ['PATH']
sys.path.insert(1, os.path.dirname(findfile(venv_path, 'easy_install.py')))
# Run your script inside the virtual environment from here
print(os.environ['PATH'])
os.system('pip list')
This is my project structure. I use virtualenv in my project but when I run it ,it has an ImportError.I use Mac.
But I can run it successfully use Pycharm
So how to run it successfully by Terminal.Because I want to run it in a Ubuntu server with cron
Thanks you for your answers.Here I show my solution.I modify my handler.py I think it may be related to The Module Search Path.
So I add the project path to the PYTHONPATH.
import os
project_home = os.path.realpath(__file__)
project_home = os.path.split(project_home)[0]
import sys
sys.path.append(os.path.split(project_home)[0])
import shutil
from modules import db, json_parse, config_out
from init_log import init as initlog
initlog()
if __name__ == '__main__':
try:
columns = json_parse.json_parse()
if not columns:
sys.exit()
is_table_has_exist = db.check_tables_exist(columns=columns)
if is_table_has_exist:
db.check_columns(columns=columns)
is_ok, config_path = config_out.output(columns)
if is_ok:
file_name = os.path.split(config_path)[1]
shutil.copy(config_path, os.path.join("/app/statics_log/config", file_name))
except Exception, e:
print e
And I run with crontab by this.
cd to/my/py_file/path && /project_path/.env/bin/python /path/to/py_file
example:
13 8 1 * * cd bulu-statics/create_config/ && /home/buka/bulu-statics/.env/bin/python /home/buka/bulu-statics/create_config/handler.py >> /app/statics_log/config/create_config.log
PyCharm automatically adds project directories marked as containing sources to the PYTHONPATH environment variable, whihc is why it works from within pycharm. On the terminal use
PYTHONPATH=${PWD}/..:${PYTHONPATH} python handler.py
You can use explicit relative imports:
from .modules import db, json_parse, config_out
The proper way to do this is to turn your project into a proper Python package by adding a setup.py file and then installing it with pip install -e .
probably because PyCharm added your project folder to the PythonPath, so you can run you app inside PyCharm.
However, when you try to run it from command line, python interpreter cannot find these libs in Python python, so what you need to do is to add your python virtualenv the python python.
there are different ways to adding python path, but I would suggest you to follow:
prepare a setup.py you'll need to specify packages and install_requires.
install your app locally in development mode via pip install -e /path/to/your-package -> it'll create a egg-link in your python virtualenv, you can run your app in your local terminal from now on;
for packing and releasing, you may want to build an artifact by following https://docs.python.org/2.7/distutils/builtdist.html
you may pip install or easy_install the artifact on your other machines. you also can release your package to PyPi if you want.
Suppose I have the following setup:
mkdir test && cd test
virtualenv .venv
source .venv/bin/activate
pip install django
mkdir mod1
touch mod1/__init__.py
echo "a = 1" > mod1/mod2.py
Which gives me:
test/.venv
test/mod1/__init__.py
test/mod1/mod2.py
How would I write this function:
def get_module(module_name, root_path, virtualenv_path=None)
In order for this to work:
project_root_path = "./test"
project_virtualenv_path = "./test/.venv"
get_module("mod1.mod2", project_root_path, project_virtualenv_path)
get_module("django.contrib.auth", project_root_path, project_virtualenv_path)
Assuming I don't have ./test/.venv activated.
The reason I want to do this, is because I'm working on a vim plugin which would implement gf functionality in a python file on an import statement. I'm trying to support virtualenvs as well.
EDIT:
Also, the script should not alter the current runtime, by adding or appending to sys.path. This should run inside vim, via the vim python bindings, and I don't think altering the vim python runtime would be a good idea.
get_module could either return a module object, or the path to the module, which is what I'm basically looking for.
You can add your virtualenv on python path like:
import site
site.addsitedir('/home/user/.virtualenvs/myapp1/lib/python2.7/site-packages')
and then import should work
The only practical solution I could find here is to run the virtualenv's activate_this.py script, look for what I need, then remove it's changes from sys.path.
import sys
import os
old_sys_path = list(sys.path)
virtualenv_path = "/path/to/venv"
activate_this_path = os.path.join(virtualenv_path, "bin", "activate_this.py")
execfile(activate_this_path, dict(__file__=activate_this_path))
# get my module here
# restore sys.path
sys.path = old_sys_path
If you have a better answer, please add it, and I'll change the accepted answer gladly.
I have a Fabric task that needs to access the settings of my Django project.
On Windows, I'm unable to install Fabric into the project's virtualenv (issues with Paramiko + pycrypto deps). However, I am able to install Fabric in my system-wide site-packages, no problem.
I have installed Django into the project's virtualenv and I am able to use all the "> python manage.py" commands easily when I activate the virtualenv with the "VIRTUALENV\Scripts\activate.bat" script.
I have a fabric tasks file (fabfile.py) in my project that provides tasks for setup, test, deploy, etc. Some of the tasks in my fabfile need to access the settings of my django project through "from django.conf import settings".
Since the only usable Fabric install I have is in my system-wide site-packages, I need to activate the virtualenv within my fabfile so django becomes available. To do this, I use the "activate_this" module of the project's virtualenv in order to have access to the project settings and such. Using "print sys.path" before and after I execute activate_this.py, I can tell the python path changes to point to the virtualenv for the project. However, I still cannot import django.conf.settings.
I have been able to successfully do this on *nix (Ubuntu and CentOS) and in Cygwin. Do you use this setup/workflow on Windows? If so Can you help me figure out why this wont work on Windows or provide any tips and tricks to get around this issue?
Thanks and Cheers.
REF:
http://virtualenv.openplans.org/#id9 | Using Virtualenv without
bin/python
Local development environment:
Python 2.5.4
Virtualenv 1.4.6
Fabric 0.9.0
Pip 0.6.1
Django 1.1.1
Windows XP (SP3)
After some digging, I found out that this is an issue with the activate_this.py script. In it's current state, virtualenv<=1.4.6, this script assumes that the path to the site-packages directory is the same for all platforms. However, the path to the site-packages directory differs between *nix like platforms and Windows.
In this case the activate_this.py script adds the *nix style path:
VIRTUALENV_BASE/lib/python2.5/site-packages/
to the python path instead of the Windows specific path:
VIRTUALENV_BASE\Lib\site-packages\
I have created an issue in the virtualenv issue tracker which outlines the problem and the solution. If you are interested, you may check on the issue here: http://bitbucket.org/ianb/virtualenv/issue/31/windows-activate_this-assumes-nix-path-to-site
Hopefully the fix will be made available in an upcomming release of virtualenv.
If you need a fix for this problem right now, and the virtualenv package has not yet been patched, you may "fix" your own activate_this.py as shown below.
Edit your VIRTUALENV\Scripts\activate_this.py file. Change the line (17 ?):
site_packages = os.path.join(base, 'lib', 'python%s' % sys.version[:3], 'site-packages')
to
if sys.platform == 'win32':
site_packages = os.path.join(base, 'Lib', 'site-packages')
else:
site_packages = os.path.join(base, 'lib', 'python%s' % sys.version[:3], 'site-packages')
With this in place, your activate_this.py script would first check which platform it is running on and then tailor the path to the site-packages directory to fit.
Enjoy!
You will have to execute the activate this, from within the fab file. Altho' I have not tested it, I believe following should work:
activate_this = '/path/to/env/bin/activate_this.py'
execfile(activate_this, dict(__file__=activate_this))
I didn't want to install python modules using easy install, symlinks in site-packages or PYTHONPATH.
So, I am trying something that I do wants system wide, then any application installation is done locally. Note, the root password is required only once here.
First create a symblink of.../pythonX.Y/site-packages/mymodules -> /home/me/lib/python_related
So, I create a directory called
/home/me/lib/python_related/
In there:
/home/me/lib/python_related
/home/me/lib/python_related/__init__.py
/home/me/lib/python_related/django_related/
/home/me/lib/python_related/django_related/core
/home/me/lib/python_related/django_related/core/Django1.0
/home/me/lib/python_related/django_related/core/Django1.1
/home/me/lib/python_related/django_related/core/mycurrent_django -> Django1.1/django
/home/me/lib/python_related/django_related/apps
/home/me/lib/python_related/django_related/apps/tagging
/home/me/lib/python_related/django_related/apps/tagging/django-tagging-0.2
/home/me/lib/python_related/django_related/apps/tagging/django-tagging-0.3
/home/me/lib/python_related/django_related/apps/tagging/mycurrent_tagging -> django-tagging-0.3
Now, here is the content of:
/home/me/lib/python_related/__init__.py
==========================================
import sys, os
# tell us where you keep all your modules and this didn't work as it gave me
# the location of the site-packages
#PYTHON_MODULE_PATH = os.path.dirname(__file__)
PYTHON_MODULE_PATH = "/home/me/libs/python_bucket"
def run_cmd(cmd):
"""
Given a command name, this function will run the command and returns the output
in a list.
"""
output = []
phdl = os.popen(cmd)
while 1:
line = phdl.readline()
if line == "":
break
output.append(line.replace("\n", ""))
return output
def install():
"""
A cheesy way of installing and managing your python apps locally without
a need to install them in the site-package. All you'd need is to install
the directory containing this file in the site-package and that's it.
Anytime you have a python package you want to install, just put it in a
proper sub-directory and make a symlink to that directory called mycurrent_xyz
and you are done. (e.g. mycurrent_django, mycurrent_tagging .. etc)
"""
cmd = "find %s -name mycurrent_*" % PYTHON_MODULE_PATH
modules_to_be_installed = run_cmd(cmd)
sys.path += modules_to_be_installed
install()
=======================================================
Now in any new python project, just import your mymodules and that pulls in any apps that you have in the above directory with the proper symbolic link. This way you can have multiple copies of apps and just use the mycurrent_xyz to the one you want to use.
Now here is question. Is this a good way of doing it?
Have a look at virtualenv.
It may do what you are after.