I have this script that creates and activates a virtual environment
import os
import platform
import subprocess
import venv
# Determine the user's platform
current_platform = platform.system()
# Define the name of the virtual environment
venv_name = "my_venv"
# Create the virtual environment using EnvBuilder
builder = venv.EnvBuilder(system_site_packages=False, clear=True, symlinks=False, upgrade=False, with_pip=True)
builder.create(venv_name)
# Activate the virtual environment
if current_platform == "Windows":
activate_path = os.path.join(venv_name, "Scripts", "activate.ps1")
subprocess.call([f"powershell.exe", f"{activate_path}"])
else:
activate_path = os.path.join(venv_name, "bin", "activate")
subprocess.call([f"bash", f"source {activate_path}"])
It works on Windows, however the venv does not remain active when the script finishes executing. This means that I cannot access the venv in the terminal, something which is desirable. If I write ".\my_venv\Scripts\Activate.ps1 " then I see that it is activated. Does anybody have a solution to this?
Executed the script, expected that the venv would remain active in terminal which wasn't the case.
Related
I want to activate a virtualenv instance from a Python script.
I know it's quite easy to do, but all the examples I've seen use it to run commands within the env and then close the subprocess.
I simply want to activate the virtualenv and return to the shell, the same way that bin/activate does.
Something like this:
$me: my-script.py -d env-name
$(env-name)me:
Is this possible?
Relevant:
virtualenv › Invoking an env from a script
If you want to run a Python subprocess under the virtualenv, you can do that by running the script using the Python interpreter that lives inside virtualenv's /bin/ directory:
import subprocess
# Path to a Python interpreter that runs any Python script
# under the virtualenv /path/to/virtualenv/
python_bin = "/path/to/virtualenv/bin/python"
# Path to the script that must run under the virtualenv
script_file = "must/run/under/virtualenv/script.py"
subprocess.Popen([python_bin, script_file])
However, if you want to activate the virtualenv under the current Python interpreter instead of a subprocess, you can use the activate_this.py script:
# Doing execfile() on this file will alter the current interpreter's
# environment so you can import libraries in the virtualenv
activate_this_file = "/path/to/virtualenv/bin/activate_this.py"
execfile(activate_this_file, dict(__file__=activate_this_file))
The simplest solution to run your script under virtualenv's interpreter is to replace the default shebang line with path to your virtualenv's interpreter like so at the beginning of the script:
#!/path/to/project/venv/bin/python
Make the script executable:
chmod u+x script.py
Run the script:
./script.py
Voila!
It turns out that, yes, the problem is not simple, but the solution is.
First I had to create a shell script to wrap the "source" command. That said I used the "." instead, because I've read that it's better to use it than source for Bash scripts.
#!/bin/bash
. /path/to/env/bin/activate
Then from my Python script I can simply do this:
import os
os.system('/bin/bash --rcfile /path/to/myscript.sh')
The whole trick lies within the --rcfile argument.
When the Python interpreter exits it leaves the current shell in the activated environment.
Win!
To run another Python environment according to the official Virtualenv documentation, in the command line you can specify the full path to the executable Python binary, just that (no need to active the virtualenv before):
/path/to/virtualenv/bin/python
The same applies if you want to invoke a script from the command line with your virtualenv. You don't need to activate it before:
me$ /path/to/virtualenv/bin/python myscript.py
The same for a Windows environment (whether it is from the command line or from a script):
> \path\to\env\Scripts\python.exe myscript.py
Just a simple solution that works for me. I don't know why you need the Bash script which basically does a useless step (am I wrong ?)
import os
os.system('/bin/bash --rcfile flask/bin/activate')
Which basically does what you need:
[hellsing#silence Foundation]$ python2.7 pythonvenv.py
(flask)[hellsing#silence Foundation]$
Then instead of deactivating the virtual environment, just Ctrl + D or exit. Is that a possible solution or isn't that what you wanted?
The top answer only works for Python 2.x
For Python 3.x, use this:
activate_this_file = "/path/to/virtualenv/bin/activate_this.py"
exec(compile(open(activate_this_file, "rb").read(), activate_this_file, 'exec'), dict(__file__=activate_this_file))
Reference: What is an alternative to execfile in Python 3?
The child process environment is lost in the moment it ceases to exist, and moving the environment content from there to the parent is somewhat tricky.
You probably need to spawn a shell script (you can generate one dynamically to /tmp) which will output the virtualenv environment variables to a file, which you then read in the parent Python process and put in os.environ.
Or you simply parse the activate script in using for the line in open("bin/activate"), manually extract stuff, and put in os.environ. It is tricky, but not impossible.
For python2/3, Using below code snippet we can activate virtual env.
activate_this = "/home/<--path-->/<--virtual env name -->/bin/activate_this.py" #for ubuntu
activate_this = "D:\<-- path -->\<--virtual env name -->\Scripts\\activate_this.py" #for windows
with open(activate_this) as f:
code = compile(f.read(), activate_this, 'exec')
exec(code, dict(__file__=activate_this))
I had the same issue and there was no activate_this.py in the Scripts directory of my environment.
activate_this.py
"""By using execfile(this_file, dict(__file__=this_file)) you will
activate this virtualenv environment.
This can be used when you must use an existing Python interpreter, not
the virtualenv bin/python
"""
try:
__file__
except NameError:
raise AssertionError(
"You must run this like execfile('path/to/active_this.py', dict(__file__='path/to/activate_this.py'))")
import sys
import os
base = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
if(sys.platform=='win32'):
site_packages = os.path.join(base, 'Lib', 'site-packages')
else:
site_packages = os.path.join(base, 'lib', 'python%s' % sys.version[:3], 'site-packages')
prev_sys_path = list(sys.path)
import site
site.addsitedir(site_packages)
sys.real_prefix = sys.prefix
sys.prefix = base
# Move the added items to the front of the path:
new_sys_path = []
for item in list(sys.path):
if item not in prev_sys_path:
new_sys_path.append(item)
sys.path.remove(item)
sys.path[:0] = new_sys_path
Copy the file to the Scripts directory of your environment and use it like this:
def activate_virtual_environment(environment_root):
"""Configures the virtual environment starting at ``environment_root``."""
activate_script = os.path.join(
environment_root, 'Scripts', 'activate_this.py')
execfile(activate_script, {'__file__': activate_script})
activate_virtual_environment('path/to/your/venv')
Refrence: https://github.com/dcreager/virtualenv/blob/master/virtualenv_support/activate_this.py
You should create all your virtualenvs in one folder, such as virt.
Assuming your virtualenv folder name is virt, if not change it
cd
mkdir custom
Copy the below lines...
#!/usr/bin/env bash
ENV_PATH="$HOME/virt/$1/bin/activate"
bash --rcfile $ENV_PATH -i
Create a shell script file and paste the above lines...
touch custom/vhelper
nano custom/vhelper
Grant executable permission to your file:
sudo chmod +x custom/vhelper
Now export that custom folder path so that you can find it on the command-line by clicking tab...
export PATH=$PATH:"$HOME/custom"
Now you can use it from anywhere by just typing the below command...
vhelper YOUR_VIRTUAL_ENV_FOLDER_NAME
Suppose it is abc then...
vhelper abc
The scenario is I have a main virtual environment and I want to implement a functionality where basically user is able to create a new virtual environment extending that main virtual environment (user's environment will be created while main environment is activated ) .
What I mean by extending a virtual environment is that the main environment's packages should be usable in user's environment i.e. there should be no copying and if a user installs a package that already exist in main environment with a different version then this version should get used in user's environment instead of main env's version .
I tried using virtualenv --system-site-packages option and though there were some positive result I landed up in some problem described below .
The main environment also has a base environment and whenever I use virtualenv --system-site-packages option , instead of main env's path , base env's path gets added to the sys.path of user's env .
I'll show 3 attributes of sys path that may help understanding the problem
Main Environment : sys path
sys.real_prefix : A this is base of main env path
sys.base_prefix : B
sys.prefix = B : this is main env's path
User's Environment :
sys.real_prefix : invalid attribute
sys.base_prefix : A this is base of main env's path
sys.prefix = C : this is user env's path
The user's env was created while keeping main env activated . Python Version : 3.5.3
I need a python script that will activate a virtualenv, run another python program inside the virtualenv, and then close the virutalenv after the second python program closes. Here is my code:
import os
import subprocess
from subprocess import Popen
activate_dir = "C:/Users/JohnDoe/theprogram/Scripts/"
os.chdir(activate_dir)
subprocess.Popen(["activate.bat"])
cal_dir = "C:/Users/JohnDoe/theprogram/"
os.chdir(cal_dir)
os.system('python program_file.py')
However, when this code run, I get an import error from the program_file which means the virtualenv is not activated. How can I fix this?
Thanks
Edit:
This is on a Windows environment.
The issue is that you are creating a new process with subprocess.Popen(["activate.bat"]) that is using that virtual environment, you're not changing your environment. What you need to do is to either call the python script in the same process you span:
os.system("source activate;python -V")
Or you could write a shell script that starts the virtual environment and calls any python script you send to it. In bash (on linux) this would be:
#!/bin/bash
# start a virtual environment and call a python module
# usage: ./runVirenvPythonModule module.py
source activate
python $1 # this is the first cmd line argument passed in
I've found a method to detect, activate, and create (if needed) a virtual environment inside a Python script and run inside that virtual environment while remaining inside that script and without the use of shell commands issued from that script (except to display installed packages via pip list). Without the use of shell commands, the script becomes OS agnostic.
Here is some example code. You simply run it from whatever OS shell you are using (Windows, Linux, MacOS):
import os
import sys
import venv
def isvirtualenv():
return sys.prefix != sys.base_prefix
def findfile(startdir, pattern):
for root, dirs, files in os.walk(startdir):
for name in files:
if name.find(pattern) >= 0:
return root + os.sep + name
return None
venv_path = 'venv' # This is an example path
if isvirtualenv():
print('Already in virtual environment.')
else:
if findfile(os.getcwd(), 'activate') is None:
print('No virtual environment found. Creating one.')
env = venv.EnvBuilder(with_pip = True)
env.create(venv_path)
else:
print('Not in virtual environment. Virtual environment directory found.')
# This is the heart of this script that puts you inside the virtual environment.
# There is no need to undo this. When this script ends, your original path will
# be restored.
os.environ['PATH'] = os.path.dirname(findfile(os.getcwd(), 'activate')) + os.pathsep + os.environ['PATH']
sys.path.insert(1, os.path.dirname(findfile(venv_path, 'easy_install.py')))
# Run your script inside the virtual environment from here
print(os.environ['PATH'])
os.system('pip list')
I've got a Windows 7 environment where I need to develop a Python Windows Service using Python 3.4. I'm using pywin32's win32service module to setup the service and most of the hooks seem to be working ok.
The problem is when I attempt to run the service from source code (using python service.py install followed by python service.py start). This uses PythonService.exe to host service.py - but I'm using a venv virtual environment and the script can't find it's modules (error message discovered with python service.py debug).
Pywin32 is installed in the virtualenv and in looking at the source code of PythonService.exe, it dynamically links in Python34.dll, imports my service.py and invokes it.
How can I get PythonService.exe to use my virtualenv when running my service.py?
Thanks very much for posting this question and a solution. I took a slightly different approach which might also be useful. It is pretty difficult to find working tips for Python services, let alone doing it with a virtualenv. Anyway...
Steps
This is using Windows 7 x64, Python 3.5.1 x64, pywin32-220 (or pypiwin32-219).
Open an Administrator command prompt.
Create a virtualenv. C:\Python35\python -m venv myvenv
Activate the virtualenv. call myvenv\scripts\activate.bat
Install pywin32, either:
From Pypi: pip install pypiwin32,
From http://www.lfd.uci.edu/~gohlke/pythonlibs/: pip install path\to\pywin32.whl
Run the post-install script python myvenv\Scripts\pywin32_postinstall.py -install.
This script registers the DLL's in the system, and copies them to C:\Windows\System32. The DLL's are named pythoncom35.dll and pywintypes35.dll. So virtual environments on the same machine on the same major Python point release will share these... it's a minor tradeoff :)
Copy myvenv\Lib\site-packages\win32\pythonservice.exe to myvenv\Scripts\pythonservice.exe
On the service class (whatever subclasses win32serviceutil.ServiceFramework), set the class property _exe_path_ to point to this relocated exe. This will become the service binPath. For example: _exe_path_ = os.path.join(*[os.environ['VIRTUAL_ENV'], 'Scripts', 'pythonservice.exe']).
Discussion
I think why this works is that Python looks upwards to figure out where the Libs folders are and based on that sets package import paths, similar to the accepted answer. When pythonservice.exe is in the original location, that doesn't seem to work smoothly.
It also resolves DLL linking problems (discoverable with depends.exe from http://www.dependencywalker.com/). Without the DLL business sorted out, it won't be possible to import from the *.pyd files from venv\Lib\site-packages\win32 as modules in your scripts. For example it's needed allow import servicemanager; as servicemanager.pyd is not in the package as a .py file, and has some cool Windows Event Log capabilities.
One of the problems I had with the accepted answer is that I couldn't figure out how to get it to accurately pick up on package.egg-link paths that are created when using setup.py develop. These .egg-link files include the path to the package when it's not located in the virtualenv under myvenv\Lib\site-packages.
If it all went smoothly, it should be possible to install, start and test the example win32 service (from an Admin prompt in the activated virtualenv):
python venv\Lib\site-packages\win32\Demos\service\pipeTestService.py install
python venv\Lib\site-packages\win32\Demos\service\pipeTestService.py start
python venv\Lib\site-packages\win32\Demos\service\pipeTestServiceClient.py
The Service Environment
Another important note in all this is that the service will execute the python code in a completely separate environment to the one you might run python myservice.py debug. So for example os.environ['VIRTUAL_ENV'] will be empty when running the service. This can be handled by either:
Setting environment variables from inside the script, e.g.
Find current path starting from the sys.executable, as described in the accepted answer.
Use that path to locate a config file.
Read the config file and put them in the environment with os.environ.
Add registry keys to the service with the environment variables.
See Accessing Environment Variables from Windows Services for doing this manually with regedit.exe
See REG ADD a REG_MULTI_SZ Multi-Line Registry Value for doing this from the command line.
I read all the answers, but no solution can fix my problem.
After carefully researched David K. Hess's code, I made some change, and it finally works.
But my reputation doesn't enough, so I just post the code here.
# 1. Custom your Project's name and Virtual Environment folder's name
# 2. Import this before all third part models
# 3. If you still failed, check the link below:
# https://stackoverflow.com/questions/34696815/using-pythonservice-exe-to-host-python-service-while-using-virtualenv
# 2019-05-29 by oraant, modified from David K. Hess's answer.
import os, sys, site
project_name = "PythonService" # Change this for your own project !!!!!!!!!!!!!!
venv_folder_name = "venv" # Change this for your own venv path !!!!!!!!!!!!!!
if sys.executable.lower().endswith("pythonservice.exe"):
# Get root path for the project
service_directory = os.path.abspath(os.path.dirname(__file__))
project_directory = service_directory[:service_directory.find(project_name)+len(project_name)]
# Get venv path for the project
def file_path(x): return os.path.join(project_directory, x)
venv_base = file_path(venv_folder_name)
venv_scripts = os.path.join(venv_base, "Scripts")
venv_packages = os.path.join(venv_base, 'Lib', 'site-packages')
# Change current working directory from PythonService.exe location to something better.
os.chdir(project_directory)
sys.path.append(".")
prev_sys_path = list(sys.path)
# Manually activate a virtual environment inside an already initialized interpreter.
os.environ['PATH'] = venv_scripts + os.pathsep + os.environ['PATH']
site.addsitedir(venv_packages)
sys.real_prefix = sys.prefix
sys.prefix = venv_base
# Move some sys path in front of others
new_sys_path = []
for item in list(sys.path):
if item not in prev_sys_path:
new_sys_path.append(item)
sys.path.remove(item)
sys.path[:0] = new_sys_path
How to use it? It's simple, just paste it into a new python file, and import it before any third part model like this:
import service_in_venv # import at top
import win32serviceutil
import win32service
import win32event
import servicemanager
import time
import sys, os
........
And now you should fix your problem.
It appears this used to work correctly with the virtualenv module before virtual environments were added to Python 3.3. There's anecdotal evidence (see this answer: https://stackoverflow.com/a/12424980/1055722) that Python's site.py used to look upward from the executable file until it found a directory that would satisfy imports. It would then use that for sys.prefix and this was sufficient for PythonService.exe to find the virtualenv it was inside of and use it.
If that was the behavior, it appears that site.py no longer does that with the introduction of the venv module. Instead, it looks one level up for a pyvenv.cfg file and configures for a virtual environment in that case only. This of course doesn't work for PythonService.exe which is buried down in the pywin32 module under site-packages.
To work around it, I adapted the activate_this.py code that comes with the original virtualenv module (see this answer: https://stackoverflow.com/a/33637378/1055722). It is used to bootstrap an interpreter embedded in an executable (which is the case with PythonService.exe) into using a virtualenv. Unfortunately, venv does not include this.
Here's what worked for me. Note, this assumes the virtual environment is named my-venv and is located one level above the source code location.
import os
import sys
if sys.executable.endswith("PythonService.exe"):
# Change current working directory from PythonService.exe location to something better.
service_directory = os.path.dirname(__file__)
source_directory = os.path.abspath(os.path.join(service_directory, ".."))
os.chdir(source_directory)
sys.path.append(".")
# Adapted from virtualenv's activate_this.py
# Manually activate a virtual environment inside an already initialized interpreter.
old_os_path = os.environ['PATH']
venv_base = os.path.abspath(os.path.join(source_directory, "..", "my-venv"))
os.environ['PATH'] = os.path.join(venv_base, "Scripts") + os.pathsep + old_os_path
site_packages = os.path.join(venv_base, 'Lib', 'site-packages')
prev_sys_path = list(sys.path)
import site
site.addsitedir(site_packages)
sys.real_prefix = sys.prefix
sys.prefix = venv_base
new_sys_path = []
for item in list(sys.path):
if item not in prev_sys_path:
new_sys_path.append(item)
sys.path.remove(item)
sys.path[:0] = new_sys_path
One other factor in my troubles - there is a new pypi wheel for pywin32 that is provided by the Twisted folks that makes it easier to install with pip. The PythonService.exe in that package was acting oddly (couldn't find a pywin32 dll when invoked) compared to the one you get when installing the official win32 exe package into the virtual env using easy_install.
For anyone reading in 2018, I didn't have any luck with either solution above (Win10, Python 3.6) - so this is what I did to get it working. The working directory is in site-packages/win32 on launch, so you need to change the working directory and fix the sys.path before you try and import any project code. This assumed venv sits in your project dir, otherwise you may just need to hard code some paths:
import sys
import os
if sys.executable.lower().endswith("pythonservice.exe"):
for i in range(4): # goes up 4 directories to project folder
os.chdir("..")
# insert site-packages 2nd in path (behind project folder)
sys.path.insert(1, os.path.join("venv",'Lib','site-packages'))
[REST OF IMPORTS]
class TestService(win32serviceutil.ServiceFramework):
[...]
Not use "pythonservice.exe", register python.exe to services directly:
import win32serviceutil
import win32service
import servicemanager
import sys
import os
import os.path
import multiprocessing
#
def main():
import time
time.sleep(600)
class ProcessService(win32serviceutil.ServiceFramework):
_svc_name_ = "SleepService"
_svc_display_name_ = "Sleep Service"
_svc_description_ = "Sleeps for 600"
_exe_name_ = sys.executable # python.exe from venv
_exe_args_ = '-u -E "' + os.path.abspath(__file__) + '"'
proc = None
def SvcStop(self):
self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
if self.proc:
self.proc.terminate()
def SvcRun(self):
self.proc = multiprocessing.Process(target=main)
self.proc.start()
self.ReportServiceStatus(win32service.SERVICE_RUNNING)
self.SvcDoRun()
self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
def SvcDoRun(self):
self.proc.join()
def start():
if len(sys.argv)==1:
import win32traceutil
servicemanager.Initialize()
servicemanager.PrepareToHostSingle(ProcessService)
servicemanager.StartServiceCtrlDispatcher()
elif '--fg' in sys.argv:
main()
else:
win32serviceutil.HandleCommandLine(ProcessService)
if __name__ == '__main__':
try:
start()
except (SystemExit, KeyboardInterrupt):
raise
except:
import traceback
traceback.print_exc()
Its make python 3.5+ virtualenv support to work by pointing right iterpreter with service install.
Based on this link I created and activated a virtual environment in a Python script but what I want is after activating the virtualenv the rest of the code should run in the env, install few items and then process.
Code:
#!/usr/bin/python
import commands
import os
import time
import datetime
import sys
import json
import requests
out = commands.getoutput("wget <url>/s.sh")
new = commands.getoutput("chmod 755 s.sh")
env = "virtualenv " + "test"
#checking if env present if not create and activate it
try:
l=commands.getoutput('/bin/bash --rcfile s.sh')
except:
print env + " not present"
m = commands.getoutput(env)
print env + " not present so creating"
os.system('/bin/bash --rcfile s.sh')
v = commands.getoutput("which python")
print v + " python to be used"
v = commands.getoutput("pip install wget")
s.sh file code:-
#!/bin/sh
. test/bin/activate
Basically instead of a shell script to create virtualenv, activate it, and run a few steps I want to use python script.
What am I missing? Is it right use case?
I've found the virtual env activation stuff to all be a bit too much like hard work in scripts. In your case you're doing it from python instead of the shell, but I guess it's the same basic principal (haven't seen commands before - are they all running in the same subprocess)?
Instead, I normally just use the full path to the executables in the venv. It's all more explicit, but then again, that is the python way :)
One thing you have to watch out for is permissions.
sudo -u whatever_user /path_to_myvenv/bin/pip install -r /some_path/requirements.txt
Not sure if that's helpful at all, but I'd be looking to call the binary in the venv directly.
EDIT just had a play with commands - each of them is independent (so that's probably the root cause of your issue)
commands.getoutput('pwd') # /somepath
commands.getoutput('cd /tmp')
commands.getoutput('pwd') # still /somepath
I'd be looking to do something like:
commands.getoutput('virtualenv test')
commands.getoutput('test/bin/pip install wget')