Develop Reference

How to distribute and or package a python file with all dependencies pre-downloaded without building an executable

I have written a python script which depends on paramiko to work. The system that will run my script has the following limitations:
No internet connectivity (so it can't download dependencies on the fly).
Volumes are mounted with 'noexec' (so I cannot run my code as a binary file generated with something like 'pyInstaller')
End-user cannot be expected to install any dependencies.
Vanilla python is installed (without paramiko)
Python version is 2.7.5
Pip is not installed either and cannot be installed on the box
I however, have access to pip on my development box (if that helps in any way).
So, what is the way to deploy my script so that I am able to provide it to the end-user with the required dependencies, i.e paramiko (and sub-dependencies of paramiko), so that the user is able to run the script out-of-the-box?
I have already tried the pyinstaller 'one folder' approach but then faced the 'noexec' issue. I have also tried to directly copy paramiko (and sub-dependencies of paramiko) to a place from where my script is able to find it with no success.

pip is usually installed with python installation. You could use it to install the dependencies on the machine as follows:
import os, sys
def selfInstallParamiko():
# assuming paramiko tar-ball/wheel is under the current working directory
currentDir = os.path.abspath(os.path.dirname(__file__))
# paramikoFileName: name of the already downloaded paramiko tar-ball,
# which you'll have to ship with your scripts
paramikoPath = os.path.join(currentDir, paramikoFileName)
print("\nInstalling {} ...".format(paramikoPath))
# To check for which pip to use (pip for python2, pip3 for python3)
pipName = "pip"
if sys.version_info[0] >= 3:
pipName = "pip3"
p = subprocess.Popen("{} install --user {}".format(pipName, paramikoPath).split())
out, err= p.communicate("")
if err or p.returncode != 0:
print("Unable to self-install {}\n".format(paramikoFileName))
sys.exit(1)
# Needed, because sometimes windows command shell does not pick up changes so good
print("\n\nPython paramiko module installed successfully.\n"
"Please start the script again from a new command shell\n\n")
sys.exit()
You can invoke it when your script starts and an ImportError occurs:
# Try to self install on import failure:
try:
import paramiko
except ImportError:
selfInstallParamiko()

pdfminer - ImportError: No module named pdfminer.pdfdocument

I am trying to install pdfMiner to work with CollectiveAccess. My host (pair.com) has given me the following information to help in this quest:
When compiling, it will likely be necessary to instruct the
installation to use your account space above, and not try to install
into the operating system directories. Typically, using "--
home=/usr/home/username/pdfminer" at the end of the install command
should allow for that.
I followed this instruction when trying to install.
The result was:
running install
running build
running build_py
running build_scripts
running install_lib
running install_scripts
changing mode of /usr/home/username/pdfminer/bin/latin2ascii.py to 755
changing mode of /usr/home/username/pdfminer/bin/pdf2txt.py to 755
changing mode of /usr/home/username/pdfminer/bin/dumppdf.py to 755
running install_egg_info
Removing /usr/home/username/pdfminer/lib/python/pdfminer-20140328.egg-info
Writing /usr/home/username/pdfminer/lib/python/pdfminer-20140328.egg-info
I don't see anything wrong with that (I'm very new to python), but when I try to run the sample command $ pdf2txt.py samples/simple1.pdf I get this error:
Traceback (most recent call last): File "pdf2txt.py", line 3, in <module>
from pdfminer.pdfdocument import PDFDocument ImportError: No module named pdfminer.pdfdocument
I'm running python 2.7.3. I can't install from root (shared hosting). The most recent version of pdfminer, which is 2014/03/28.
I've seen some posts on similar issues ("no module named. . . " but nothing exactly the same. The proposed solutions either don't help (such as installing with sudo - not an option; specifying the path for python (which doesn't seem to be the issue), etc.).
Or is this a question for my host? (i.e., something amiss or different about their setup)

I had an error like this:
No module named 'pdfminer.pdfinterp'; 'pdfminer' is not a package
My problem was that I had named my script pdfminer.py which for the reasons that I don't know, Python took it for the original pdfminer package files and tried to compiled it.
I renamed my script to something else, deleted all the *.pyc file and __pycache__ directory and my problem was solved.

use this command worked for me and removed the error
pip install pdfminer.six

Since the package pdfminer is installed to a non-standard/non-default location, Python won't be be able to find it. In order to use it, you will need to add it to your 'pythonpath'. Three ways:
At run time, put this in your script pdf2txt.py:
import sys
# if there are no conflicting packages in the default Python Libs =>
sys.path.append("/usr/home/username/pdfminer")
or
import sys
# to always use your package lib before the system's =>
sys.path.insert(1, "/usr/home/username/pdfminer")
Note: The install path specified with --home is used as the Lib for all packages which you might want to install, not just this one. You should delete that folder and re-install with --
home=/usr/home/username/myPyLibs (or any generic name) so that when you install other packages with that install path, you would only need the one path to add to your local Lib to be able to import them:
import sys
sys.path.insert(1, "/usr/home/username/myPyLibs")
Add it to PYTHONPATH before executing your script:
export PYTHONPATH="${PYTHONPATH}:/usr/home/username/myPyLibs"
And then put that in your ~/.bashrc file (/usr/home/username/.bashrc) or .profile as applicable. This may not work for programs which are not executed from the console.
Create a VirtualEnv and install the packages you need to that.

I have a virtual environment and I had to activate it before I did a pip3 install to have the venv see it.
source ~/venv/bin/activate

Using PythonService.exe to host python service while using virtualenv

I've got a Windows 7 environment where I need to develop a Python Windows Service using Python 3.4. I'm using pywin32's win32service module to setup the service and most of the hooks seem to be working ok.
The problem is when I attempt to run the service from source code (using python service.py install followed by python service.py start). This uses PythonService.exe to host service.py - but I'm using a venv virtual environment and the script can't find it's modules (error message discovered with python service.py debug).
Pywin32 is installed in the virtualenv and in looking at the source code of PythonService.exe, it dynamically links in Python34.dll, imports my service.py and invokes it.
How can I get PythonService.exe to use my virtualenv when running my service.py?

Thanks very much for posting this question and a solution. I took a slightly different approach which might also be useful. It is pretty difficult to find working tips for Python services, let alone doing it with a virtualenv. Anyway...
Steps
This is using Windows 7 x64, Python 3.5.1 x64, pywin32-220 (or pypiwin32-219).
Open an Administrator command prompt.
Create a virtualenv. C:\Python35\python -m venv myvenv
Activate the virtualenv. call myvenv\scripts\activate.bat
Install pywin32, either:
From Pypi: pip install pypiwin32,
From http://www.lfd.uci.edu/~gohlke/pythonlibs/: pip install path\to\pywin32.whl
Run the post-install script python myvenv\Scripts\pywin32_postinstall.py -install.
This script registers the DLL's in the system, and copies them to C:\Windows\System32. The DLL's are named pythoncom35.dll and pywintypes35.dll. So virtual environments on the same machine on the same major Python point release will share these... it's a minor tradeoff :)
Copy myvenv\Lib\site-packages\win32\pythonservice.exe to myvenv\Scripts\pythonservice.exe
On the service class (whatever subclasses win32serviceutil.ServiceFramework), set the class property _exe_path_ to point to this relocated exe. This will become the service binPath. For example: _exe_path_ = os.path.join(*[os.environ['VIRTUAL_ENV'], 'Scripts', 'pythonservice.exe']).
Discussion
I think why this works is that Python looks upwards to figure out where the Libs folders are and based on that sets package import paths, similar to the accepted answer. When pythonservice.exe is in the original location, that doesn't seem to work smoothly.
It also resolves DLL linking problems (discoverable with depends.exe from http://www.dependencywalker.com/). Without the DLL business sorted out, it won't be possible to import from the *.pyd files from venv\Lib\site-packages\win32 as modules in your scripts. For example it's needed allow import servicemanager; as servicemanager.pyd is not in the package as a .py file, and has some cool Windows Event Log capabilities.
One of the problems I had with the accepted answer is that I couldn't figure out how to get it to accurately pick up on package.egg-link paths that are created when using setup.py develop. These .egg-link files include the path to the package when it's not located in the virtualenv under myvenv\Lib\site-packages.
If it all went smoothly, it should be possible to install, start and test the example win32 service (from an Admin prompt in the activated virtualenv):
python venv\Lib\site-packages\win32\Demos\service\pipeTestService.py install
python venv\Lib\site-packages\win32\Demos\service\pipeTestService.py start
python venv\Lib\site-packages\win32\Demos\service\pipeTestServiceClient.py
The Service Environment
Another important note in all this is that the service will execute the python code in a completely separate environment to the one you might run python myservice.py debug. So for example os.environ['VIRTUAL_ENV'] will be empty when running the service. This can be handled by either:
Setting environment variables from inside the script, e.g.
Find current path starting from the sys.executable, as described in the accepted answer.
Use that path to locate a config file.
Read the config file and put them in the environment with os.environ.
Add registry keys to the service with the environment variables.
See Accessing Environment Variables from Windows Services for doing this manually with regedit.exe
See REG ADD a REG_MULTI_SZ Multi-Line Registry Value for doing this from the command line.

I read all the answers, but no solution can fix my problem.
After carefully researched David K. Hess's code, I made some change, and it finally works.
But my reputation doesn't enough, so I just post the code here.
# 1. Custom your Project's name and Virtual Environment folder's name
# 2. Import this before all third part models
# 3. If you still failed, check the link below:
# https://stackoverflow.com/questions/34696815/using-pythonservice-exe-to-host-python-service-while-using-virtualenv
# 2019-05-29 by oraant, modified from David K. Hess's answer.
import os, sys, site
project_name = "PythonService" # Change this for your own project !!!!!!!!!!!!!!
venv_folder_name = "venv" # Change this for your own venv path !!!!!!!!!!!!!!
if sys.executable.lower().endswith("pythonservice.exe"):
# Get root path for the project
service_directory = os.path.abspath(os.path.dirname(__file__))
project_directory = service_directory[:service_directory.find(project_name)+len(project_name)]
# Get venv path for the project
def file_path(x): return os.path.join(project_directory, x)
venv_base = file_path(venv_folder_name)
venv_scripts = os.path.join(venv_base, "Scripts")
venv_packages = os.path.join(venv_base, 'Lib', 'site-packages')
# Change current working directory from PythonService.exe location to something better.
os.chdir(project_directory)
sys.path.append(".")
prev_sys_path = list(sys.path)
# Manually activate a virtual environment inside an already initialized interpreter.
os.environ['PATH'] = venv_scripts + os.pathsep + os.environ['PATH']
site.addsitedir(venv_packages)
sys.real_prefix = sys.prefix
sys.prefix = venv_base
# Move some sys path in front of others
new_sys_path = []
for item in list(sys.path):
if item not in prev_sys_path:
new_sys_path.append(item)
sys.path.remove(item)
sys.path[:0] = new_sys_path
How to use it? It's simple, just paste it into a new python file, and import it before any third part model like this:
import service_in_venv # import at top
import win32serviceutil
import win32service
import win32event
import servicemanager
import time
import sys, os
........
And now you should fix your problem.

It appears this used to work correctly with the virtualenv module before virtual environments were added to Python 3.3. There's anecdotal evidence (see this answer: https://stackoverflow.com/a/12424980/1055722) that Python's site.py used to look upward from the executable file until it found a directory that would satisfy imports. It would then use that for sys.prefix and this was sufficient for PythonService.exe to find the virtualenv it was inside of and use it.
If that was the behavior, it appears that site.py no longer does that with the introduction of the venv module. Instead, it looks one level up for a pyvenv.cfg file and configures for a virtual environment in that case only. This of course doesn't work for PythonService.exe which is buried down in the pywin32 module under site-packages.
To work around it, I adapted the activate_this.py code that comes with the original virtualenv module (see this answer: https://stackoverflow.com/a/33637378/1055722). It is used to bootstrap an interpreter embedded in an executable (which is the case with PythonService.exe) into using a virtualenv. Unfortunately, venv does not include this.
Here's what worked for me. Note, this assumes the virtual environment is named my-venv and is located one level above the source code location.
import os
import sys
if sys.executable.endswith("PythonService.exe"):
# Change current working directory from PythonService.exe location to something better.
service_directory = os.path.dirname(__file__)
source_directory = os.path.abspath(os.path.join(service_directory, ".."))
os.chdir(source_directory)
sys.path.append(".")
# Adapted from virtualenv's activate_this.py
# Manually activate a virtual environment inside an already initialized interpreter.
old_os_path = os.environ['PATH']
venv_base = os.path.abspath(os.path.join(source_directory, "..", "my-venv"))
os.environ['PATH'] = os.path.join(venv_base, "Scripts") + os.pathsep + old_os_path
site_packages = os.path.join(venv_base, 'Lib', 'site-packages')
prev_sys_path = list(sys.path)
import site
site.addsitedir(site_packages)
sys.real_prefix = sys.prefix
sys.prefix = venv_base
new_sys_path = []
for item in list(sys.path):
if item not in prev_sys_path:
new_sys_path.append(item)
sys.path.remove(item)
sys.path[:0] = new_sys_path
One other factor in my troubles - there is a new pypi wheel for pywin32 that is provided by the Twisted folks that makes it easier to install with pip. The PythonService.exe in that package was acting oddly (couldn't find a pywin32 dll when invoked) compared to the one you get when installing the official win32 exe package into the virtual env using easy_install.

For anyone reading in 2018, I didn't have any luck with either solution above (Win10, Python 3.6) - so this is what I did to get it working. The working directory is in site-packages/win32 on launch, so you need to change the working directory and fix the sys.path before you try and import any project code. This assumed venv sits in your project dir, otherwise you may just need to hard code some paths:
import sys
import os
if sys.executable.lower().endswith("pythonservice.exe"):
for i in range(4): # goes up 4 directories to project folder
os.chdir("..")
# insert site-packages 2nd in path (behind project folder)
sys.path.insert(1, os.path.join("venv",'Lib','site-packages'))
[REST OF IMPORTS]
class TestService(win32serviceutil.ServiceFramework):
[...]

Not use "pythonservice.exe", register python.exe to services directly:
import win32serviceutil
import win32service
import servicemanager
import sys
import os
import os.path
import multiprocessing
#
def main():
import time
time.sleep(600)
class ProcessService(win32serviceutil.ServiceFramework):
_svc_name_ = "SleepService"
_svc_display_name_ = "Sleep Service"
_svc_description_ = "Sleeps for 600"
_exe_name_ = sys.executable # python.exe from venv
_exe_args_ = '-u -E "' + os.path.abspath(__file__) + '"'
proc = None
def SvcStop(self):
self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
if self.proc:
self.proc.terminate()
def SvcRun(self):
self.proc = multiprocessing.Process(target=main)
self.proc.start()
self.ReportServiceStatus(win32service.SERVICE_RUNNING)
self.SvcDoRun()
self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
def SvcDoRun(self):
self.proc.join()
def start():
if len(sys.argv)==1:
import win32traceutil
servicemanager.Initialize()
servicemanager.PrepareToHostSingle(ProcessService)
servicemanager.StartServiceCtrlDispatcher()
elif '--fg' in sys.argv:
main()
else:
win32serviceutil.HandleCommandLine(ProcessService)
if __name__ == '__main__':
try:
start()
except (SystemExit, KeyboardInterrupt):
raise
except:
import traceback
traceback.print_exc()
Its make python 3.5+ virtualenv support to work by pointing right iterpreter with service install.

How can I install Python modules programmatically / through a Python script? [duplicate]

This question already has answers here:
How can I Install a Python module within code?
(12 answers)
Closed 2 years ago.
Can I download and install Python modules from PyPi strictly inside a script, without using a shell at all?
I use a non-standard Python environment, Autodesk Maya's Python interpreter. This does not come with "easy_install," and there is no "shell," only a python script interpreter invoked by the main Maya executable. Copying and pasting ez_setup.py's contents into the script editor window and running it correctly installs an easy_install somewhere into Maya's directory, but the script incorrectly records the Python interpreter as "...maya.exe" instead of "...mayapy.exe" Furthermore, using easy_install requires a shell.
The objective is to deliver a Python script that, e.g., installs NumPy into the Maya Python system. This could be accomplished by dropping eggs into the site-packages directory, but that requires manual user intervention. Anything an end user has to do outside the Maya environment is essentially untouchable, especially messing with the file system. But messing with the filesystem through a script? That's fine.
Is there something more elegant than ez_setup.py + editing the resulting easy_install...py's + subprocess calls? I feel like this is a basic feature. I see documentation online for programmatic module installation through pip... but pip needs to be installed first!
What is the most elegant way to install a module strictly within the confines of a script?

Installing easy_install for Maya on windows.
Download ez_setup.py.
open windows cmd elevated (start, type cmd, rmb click on it ->run as administrator)
change the cmd directory to x:\maya install dir\bin
example: cd c:\Program Files\MayaXX\bin
execute following command mayapy x:\WhereYouSaved\ez_setup.py
Now easy install should be set up properly. You may want to still do following steps:
cd x:\maya install dir\python\scripts
rename all files in this folder to start with ma
example: for %i in (*) do ren %i ma%i
add this folder to your path
hit win+e
rmb my computer and choose properties
Advanced system settings -> Environment variables
search variable path edit it and append ;x:\maya install dir\python\scripts
Now you can call maeasy_install pythonModule from cmd for installing stuff. Also you can call following inside Maya to install modules:
from setuptools.command import easy_install
easy_install.main( ["pythonModule"] )
NOTE: If Maya is installed in program files then you can not really install stuff without elevating. Unless you change disk permissions to the Maya python directory.

#!/usr/bin/env python
from __future__ import print_function
REQUIREMENTS = [ 'distribute', 'version', 'Cython', 'sortedcollection' ]
try:
from setuptools import find_packages
from distutils.core import setup
from Cython.Distutils import build_ext as cython_build
import sortedcollection
except:
import os, pip
pip_args = [ '-vvv' ]
proxy = os.environ['http_proxy']
if proxy:
pip_args.append('--proxy')
pip_args.append(proxy)
pip_args.append('install')
for req in REQUIREMENTS:
pip_args.append( req )
print('Installing requirements: ' + str(REQUIREMENTS))
pip.main(initial_args = pip_args)
# do it again
from setuptools import find_packages
from distutils.core import setup
from Cython.Distutils import build_ext as cython_build
import sortedcollection

To make it work, open the ez_setup.py file and simply add an s after http at this line:
DEFAULT_URL = "http://pypi.python.org/packages/%s/s/setuptools/" % sys.version[:3]
so that it becomes
DEFAULT_URL = "https://pypi.python.org/packages/%s/s/setuptools/" % sys.version[:3]

How to find a module in a virtualenv without activating said virtualenv?

Suppose I have the following setup:
mkdir test && cd test
virtualenv .venv
source .venv/bin/activate
pip install django
mkdir mod1
touch mod1/__init__.py
echo "a = 1" > mod1/mod2.py
Which gives me:
test/.venv
test/mod1/__init__.py
test/mod1/mod2.py
How would I write this function:
def get_module(module_name, root_path, virtualenv_path=None)
In order for this to work:
project_root_path = "./test"
project_virtualenv_path = "./test/.venv"
get_module("mod1.mod2", project_root_path, project_virtualenv_path)
get_module("django.contrib.auth", project_root_path, project_virtualenv_path)
Assuming I don't have ./test/.venv activated.
The reason I want to do this, is because I'm working on a vim plugin which would implement gf functionality in a python file on an import statement. I'm trying to support virtualenvs as well.
EDIT:
Also, the script should not alter the current runtime, by adding or appending to sys.path. This should run inside vim, via the vim python bindings, and I don't think altering the vim python runtime would be a good idea.
get_module could either return a module object, or the path to the module, which is what I'm basically looking for.

You can add your virtualenv on python path like:
import site
site.addsitedir('/home/user/.virtualenvs/myapp1/lib/python2.7/site-packages')
and then import should work

The only practical solution I could find here is to run the virtualenv's activate_this.py script, look for what I need, then remove it's changes from sys.path.
import sys
import os
old_sys_path = list(sys.path)
virtualenv_path = "/path/to/venv"
activate_this_path = os.path.join(virtualenv_path, "bin", "activate_this.py")
execfile(activate_this_path, dict(__file__=activate_this_path))
# get my module here
# restore sys.path
sys.path = old_sys_path
If you have a better answer, please add it, and I'll change the accepted answer gladly.