Develop Reference

How do I add a directory to system environment variable using python script [duplicate]

From what I've read, any changes to the environment variables in a Python instance are only available within that instance, and disappear once the instance is closed. Is there any way to make them stick by committing them to the system?
The reason I need to do this is because at the studio where I work, tools like Maya rely heavily on environment variables to configure paths across multiple platforms.
My test code is
import os
os.environ['FAKE'] = 'C:\\'
Opening another instance of Python and requesting os.environ['FAKE'] yields a KeyError.
NOTE: Portability will be an issue, but the small API I'm writing will be able to check OS version and trigger different commands if necessary.
That said, I've gone the route of using the Windows registry technique and will simply write alternative methods that will call shell scripts on other platforms as they become requirements.

You can using SETX at the command-line.
By default these actions go on the USER env vars.
To set and modify SYSTEM vars use the /M flag
import os
env_var = "BUILD_NUMBER"
env_val = "3.1.3.3.7"
os.system("SETX {0} {1} /M".format(env_var,env_val))

make them stick by committing them to
the system?
I think you are a bit confused here. There is no 'system' environment. Each process has their own environment as part its memory. A process can only change its own environment. A process can set the initial environment for processes it creates.
If you really do think you need to set environment variables for the system you will need to look at changing them in the location they get initially loaded from like the registry on windows or your shell configuration file on Linux.

Under Windows it's possible for you to make changes to environment variables persistent via the registry with this recipe, though it seems like overkill.
To echo Brian's question, what are you trying to accomplish? There is probably an easier way.

Seems like there is simplier solution for Windows
import subprocess
subprocess.call(['setx', 'Hello', 'World!'], shell=True)

I don't believe you can do this; there are two work-arounds I can think of.
The os.putenv function sets the environment for processes you start with, i.e. os.system, popen, etc. Depending on what you're trying to do, perhaps you could have one master Python instance that sets the variable, and then spawns new instances.
You could run a shell script or batch file to set it for you, but that becomes much less portable. See this article:
http://code.activestate.com/recipes/159462/

Think about it this way.
You're not setting shell environment variables.
You're spawning a subshell with some given environment variable settings; this subshell runs your application with the modified environment.

According to this discussion, you cannot do it. What are you trying to accomplish?

You are forking a new process and cannot change the environment of the parent process as you cannot do if you start a new shell process from the shell

You might want to try Python Win32 Extensions, developed by Mark Hammond, which is included in the ActivePython (or can be installed separately). You can learn how to perform many Windows related tasks in Hammond's and Robinson's book.
Using PyWin32 to access windows COM objects, a Python program can use the Environment Property (a collection of environment variables) of the WScript.Shell object.

Try to use py-setenv that will allow you to set variable via registry
python -m pip install py-setenv

From within Python? No, it can't be done!
If you are not bound to Python, you should consider using shell scripts (sh, bash, etc). The "source" command allows you to run a script that modifies the environment and will "stick" like you want to the shell you "sourced" the script in. What's going on here is that the shell executes the script directly rather creating a sub-process to execute the script.
This will be quite portable - you can use cygwin on windows to do this.

In case someone might need this info. I realize this was asked 7 yrs ago, but even I forget how sometimes. .
Yes there is a way to make them "stick" in windows. Simply go control panel, system, advanced system settings,when the system properties window opens you should see an option (button) for Environment Variables. .The process for getting to this is a little different depending on what OS you're using (google it).
Choose that (click button), then the Environment Variables window will open. It has 2 split windows, the top one should be your "User Variables For yourusername". . .choose "new", then simply set the variable. For instance one of mine is "Database_Password = mypassword".
Then in your app you can access them like this: import os, os.environ.get('Database_Password'). You can do something like pass = os.environ.get('Database_Password').

How do you I add a module into my PYTHONPATH?

I am attempting to download a code from github which contains the library "ee" - Google Earth Engine. GitBash is giving me an error:
ModuleNotFoundError: No module named 'fcntl'
fcntl is a module within the library Google Earth Engine. I have Windows and it seems Linux is required. I was directed to add the module (fcntl) to the PYTHONPATH. Any other suggestions for this error would be helpful as well! The code I intend to add in PYTHONPATH is below.
def fcntl(fd, op, arg=0):
return 0
def ioctl(fd, op, arg=0, mutable_flag=True):
if mutable_flag:
return 0
else:
return ""
def flock(fd, op):
return
def lockf(fd, operation, length=0, start=0, whence=0):
return

First, this is probably not going to work for you.
You can't turn Windows to Linux just by adding modules to your Python library. The reason you don't have the fcntl module on your path is that fcntl isn't included on Windows. And the reason it isn't included on Windows is that the Windows OS doesn't support the syscalls that module wraps, or anything close enough to reasonably emulate those syscalls.
If you have code that requires fcntl, that code cannot run on Windows (unless you do some significant work to port it to not require fcntl in the first place).
If you have code that doesn't require fcntl but uses it anyway, or if you just need something for temporary development purposes so you can catch and fix file sharing errors while porting the code to not require fcntl, then you can use msoliman's dummy code, which I'll explain how to do below. But you seem to be expecting it to do magic, and it won't do that.
You may not be sure. Maybe you're using code that uses other code that uses other code that uses fcntl in some scenarios but not others, it may not actually need fcntl to do any of the things you're actually trying to do with it.
If you want to test that, you can take msoliman's dummy code, and change each function body to this:
raise RuntimeError('Oops, using fcntl!')
Then run the program and see if it fails with that error. If not, you don't actually need fcntl after all. (Or at least you don't need it for any of the things you tested—it's always possible that some other thing you need to do with the app that you didn't think to test will need it.)
If your code actually needs fcntl, and you don't want to/can't port that code to Windows code that uses Win32 API calls (or a cross-platform library like portalocker), then what you probably need to do is install Linux and run the program there.
There are multiple ways to run Linux on top of Windows, rather than instead of Windows. For example, you could install Docker for Windows and then build a linux docker container with the app. Or you could use VMWare Player to, in effect, run a Linux image as an application under Windows, and then do your work inside that image. And so on.
Finally, msoliman's "Place this module in your PYTHONPATH" is a little misleading.
What you actually need to do is get it into your sys.path. PYTHONPATH is just one way of doing that, and probably not the one you want here.
The options are:
Just put it in the same directory as your script. As the docs say, "As initialized upon program startup, the first item of this list, path[0], is the directory containing the script that was used to invoke the Python interpreter."
Put it in your user or system site packages, or some other directory that's already on your default sys.path. You can import sys; print(sys.path) to get a list of these directories. If you see something inside your home directory, that's a good place to put it; if not, look for something with site-packages in the name.
Put it in some other directory somewhere else, and set the PYTHONPATH environment variable to be the full path to directory. You can set an environment variable in the Windows cmd command prompt by writing SET PYTHONPATH C:\Path\To\Directory. This will only persist as long as the current command prompt window. If you want to set it permanently, there's a setting somewhere in Control Panel (it changes with each Windows version; Super User should have good up-to-date answers for each version) where you can set System and User environment variables. Any User environment variable will take effect in every new command prompt window you open from now on.

Setting up a specific Python in Jenkins

I am quite new with configuring Jenkins or Python but I have to set up a unitary test in Jenkins. My program is in Python, but only works on Python 2.6 whereas the Jenkins version I should be using is 2.7, so I'm trying to set up Jenkins to set some environment variables so that it prepares launching the accurate Python for that specific test (it is part of a greater project that will successfully run several other tests that work well).
The idea I had was to set in the command to execute several environment variables like PATH, LD_LIBRARY_PATH and PYTHONPATH such as following in the "Execute shell" command line interpreter:
PYTHONPATH=/path/to/python2.6/lib:$PYTHONPATH
PATH=/path/to/python2.6/bin:$PATH
LD_LIBRARY_PATH=/path/to/python2.6/lib:$LD_LIBRARY_PATH
... however, it was still calling the wrong version of Python. Therefore, I forced these variables to:
PYTHONPATH=/path/to/python2.6/lib
PATH=/path/to/python2.6/bin
LD_LIBRARY_PATH=/path/to/python2.6/lib
... and I still get errors because the old version of Python is called instead, even if it should not appear in the PATH ... It appears Jenkins will remember the location of the old libraries however and will try loading them first.
How would I correctly set the environment in a "subproject" in Jenkins so that I can call a different version of Python?
Thank you and best regards,
~Stéphane

If you want your program to run with a specific version of the python interpreter, you indicate it in the shebang
#!/usr/bin/python2.6
#your code here

What i did in my Jenkins shell script using a specific python version was something like this when calling my unit test:
python3 src/test/unit_test.py
I was using it to use Python 3.X but it should work with 2.6 as well using:
python2.6 src/test/unit_test.py

Stupid me... I was indeed doing things correctly, I just had a part of my code that was overriding the PYTHONPATH value, so the solution I had found previously was good.
FYI, I modified my shebang, if it's of any help to anyone ;)

Some powershell cmd-lets not available if run from python?

So this is an unusual one, and perhaps I am simply missing the obvious, but I have the following python code that creates a powershell script and runs it.
# Create the PowerShell file
f = open("getKey.ps1", "w")
f.write('$c = Get-BitlockerVolume -MountPoint C:\n')
f.write('$c.KeyProtector[1].RecoveryPassword | Out-File C:\\Temp\\recovery.key\n')
# Invoke Script
startPS = subprocess.Popen([r'C:\Windows\system32\WindowsPowerShell\v1.0\powershell.exe',
'-ExecutionPolicy', 'Unrestricted', './getKey.ps1'], cwd=os.getcwd())
result = startPS.wait()
When this is run, it gives me the following error:
The term 'Get-BitlockerVolume' is not recognized as the name of a cmdlet, function, script file, or operable program.
However, if I then go and manually run the generated script, it works perfectly. To add to the oddity, if I run the same command exactly as above ie:
C:\Windows\system32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy Unrestricted ./getKey.ps1
it also works exactly as expected.
Clearly, the above error is a powershell error, so it is successfully running the script. It almost seems like powershell somehow knows that this is being run from python and has some restricted library of commands when a script is run from a particular source. I grant that that idea makes no real sense, but it's certainly how things appear.
I don't think this is a permissions issue, because when you run the same command from an unelevated powershell prompt, you get an Access is denied type error, rather than a command doesn't exist kind of error.
Anyway, any help would be greatly appreciated!
Edits
Edit: New evidence to help figure this out:
It's definitely an issue of cmdlets being loaded properly. If I programmatically run a script to dump the list of all available commands to a text file, it is only about 2/3's as big as if I do so through a powershell prompt directly

I bet Python is running as a 32-bit process on 64-bit Windows. In this case, you'll end up running 32-bit PowerShell, which in practice is a Bad Thing since many PowerShell modules depend on native binaries that may not have 32-bit equivalents. I hit this with IIS Manager commandlets--the commandlets themselves are registered in 32-bit PowerShell, but the underlying COM objects they rely on are not.
If you need to run 64-bit PowerShell from a 32-bit process, specify the path as %SystemRoot%\SysNative\WindowsPowerShell\v1.0\PowerShell.exe instead of System32.
System32 is actually virtualized for 32-bit processes and refers to the 32-bit binaries in %SystemRoot%\SysWow64. This is why your paths (and PSMODULEPATH) will look the same, but aren't. (SysNative is also a virtualized path that only exists in virtualized 32-bit processes.)

Adding to what #jbsmith said in the comment, also check to make sure that the environment variable that PowerShell relies on to know where it's modules are is populated correctly when python starts the process.
%PSMODULEPATH% is the environment variable in question, and it works the same way the %PATH% variable does, multiple directories separated by ;. Based on what you say your observed behavior is, it seems that you are using PowerShell 3.0, and cmdlet autoloading is in effect.

The solution here: Run a powershell script from python that uses Web-Administration module got me the cmdlet I needed, however there are still missing cmdlets even when using this method. I'm still at a loss as to why some are loaded and others are not, but for the time being, my script does what I need it to and I can't spend any more time to figure it out.
For reference here is the code that worked for me
startPS = subprocess.Popen([r'C:\Windows\sysnative\cmd.exe', '/c', 'powershell',
'-ExecutionPolicy', 'Unrestricted', './getKey.ps1'], cwd=os.getcwd())

I had the same issue, and it was simply that the BitLocker feature was not installed, hence the module wasn't present.
I fixed it by installing the Bitlocker feature:
Windows Server:
Install-WindowsFeature BitLocker -IncludeAllSubFeature -IncludeManagementTools -Restart
Windows Desktop:
Enable-WindowsOptionalFeature -Online -FeatureName BitLocker -All

How to properly use PyDev with two different Python versions with scripts that are recalling other python scripts?

The story began with a very strange error while I was running my script from PyDev. Running the same script from outside will not encounter the same problem.
Fatal Python error: Py_Initialize: can't initialize sys standard streams
File "C:\Python26\lib\encodings\__init__.py", line 123
raise CodecRegistryError,\
^
SyntaxError: invalid syntax
This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.
I was able to find why this is happening: In PyDev I use two different Python versions: 3.1 that is the default installation and 2.6 as the alternative one.
My Windows Environment does not contains PYTHONHOME, CLASSPATH, PYTHONPATH but PyDev does add them.
Now the problem is at one stage my python script does execute another python script using os.system(python second.py) and the second script will fail with the above error.
Now I'm looking to find a way to prevent this issue, issue that is happening because it will run the execute the default python using the settings for the non-default one (added by PyDev).
I do not want to change the standard call (python file.py) but I want to be able to run my script from pydev without problem and being able to use default or alternative python environment.
Any ideas?

I found a solution that seams acceptable specially because it will not interfere with running the scripts on other systems, just to run python -E second.py - this will force Python to ignore PYTHON* environment variables.

I may not be understanding this quite right, but I think you're invoking a script from pydev that works okay, but this script executes another script which requires a different version.
While this would unfortunately be installation-specific, you could use os.system("c:\absolute\path\to\proper\version\of\python.exe second.py").
If PyDev is setting up conflicting environmental variables, you may want to look into subprocess over os.system.
http://docs.python.org/library/subprocess.html#using-the-subprocess-module
This will allow you to invoke a process with a handle, so you can optionally wait for it to terminate. It will also allow you to pass environment variables upon execution.

I believe your call should be:
import sys
os.system(sys.executable+ ' second.py')
So that you guarantee you're using the same interpreter you're currently running and not launching the other one (or did you really mean to use the other interpreter?)