Develop Reference

How to modify windows 10 path variable directly from a python script

I am looking to permanently modify the path variable inside windows from a python script. The script is a wrapper to help automate the installation of applications and I want to enable the API to add applications to the path.
So for example I want to install a program called micro which has an install path of C:\Users\USERNAME\Path\to\micro and then add that install path to my path variable so that I can just run micro in my terminal.
I've been made aware of 2 possible solutions, which both do not work:
1. Using os.environ
In python the os module let's you read environment variables, but not actually modify them. so for example:
program_path = "C:\\Users\\USERNAME\\Path\\to\\micro"
new_path = f"{os.environ['PATH']};{program_path}"
os.environ["PATH"] = new_path
This would update the path variable in the python script, but it does not actually modify it on the system which is what I want.
2. setx
I was made aware that it is possible to update your path using the setx command in windows, but for some reason on windows 10 this destroys your path variable.
The idea is that you can call the setx command from python and use it to update the path variable. You should be able to type setx path "%path%;C:\Users\USERNAME\Path\to\micro" and have it update correctly.
So for example, in python code that would be:
program_path = "C:\\Users\\USERNAME\\Path\\to\\micro"
subprocess.Popen(f'setx path "%path%;{program_path}"')
This should take the current path variable and append the program path to it, but instead it just wipes your entire path and replaces it with a literal %path% and then the program path.
So now my path looks like this:
%path%
C:\Users\USERNAME\Path\to\micro
Any ideas on how to get this to work would be appreciated.

Okay, so after a long (and disgusting) amount of research I found a solution. Here is the method I came up with for a cross-platform system of adding to PATH variable:
def add_to_path(program_path:str):
"""Takes in a path to a program and adds it to the system path"""
if os.name == "nt": # Windows systems
import winreg # Allows access to the windows registry
import ctypes # Allows interface with low-level C API's
with winreg.ConnectRegistry(None, winreg.HKEY_CURRENT_USER) as root: # Get the current user registry
with winreg.OpenKey(root, "Environment", 0, winreg.KEY_ALL_ACCESS) as key: # Go to the environment key
existing_path_value = winreg.EnumValue(key, 3)[1] # Grab the current path value
new_path_value = existing_path_value + program_path + ";" # Takes the current path value and appends the new program path
winreg.SetValueEx(key, "PATH", 0, winreg.REG_EXPAND_SZ, new_path_value) # Updated the path with the updated path
# Tell other processes to update their environment
HWND_BROADCAST = 0xFFFF
WM_SETTINGCHANGE = 0x1A
SMTO_ABORTIFHUNG = 0x0002
result = ctypes.c_long()
SendMessageTimeoutW = ctypes.windll.user32.SendMessageTimeoutW
SendMessageTimeoutW(HWND_BROADCAST, WM_SETTINGCHANGE, 0, u"Environment", SMTO_ABORTIFHUNG, 5000, ctypes.byref(result),)
else: # If system is *nix
with open(f"{os.getenv('HOME')}/.bashrc", "a") as bash_file: # Open bashrc file
bash_file.write(f'\nexport PATH="{program_path}:$PATH"\n') # Add program path to Path variable
os.system(f". {os.getenv('HOME')}/.bashrc") # Update bash source
print(f"Added {program_path} to path, please restart shell for changes to take effect")
Neither are pretty, but it does actually work. You do need to restart running shells for it to take effect, but other than that it's perfect.

winreg parameters to retrieve PATH in Windows 10

To retrieve the contents of the %PATH% variable on a Windows 10 Home edition machine, what values should be used for the key, subkey, and name parameters in the following Python script?
import winreg
def _get_reg_value(key, subkey, name):
"""Return registry value specified by key, subkey, and name.
Environment variables in values of type REG_EXPAND_SZ are expanded
if possible.
"""
key = _winreg.OpenKey(key, subkey)
try:
ret = _winreg.QueryValueEx(key, name)
except WindowsError:
return None
else:
key.Close()
if ret[1] == _winreg.REG_EXPAND_SZ:
return expandvars(ret[0])
else:
return ret[0]
Note that we are not using os.environ here because we need to interact with the permanent path values, not just with the runtime values that are exposed by os.environ.
The code example above is from this link . I am using it to study how to interact with the Windows Registry programmatically using Python.
Failed Attempt:
When I call the above function using the following syntax in a .py file run from Windows CMD, all that is returned is a blank line in Windows CMD, followed by a command prompt. As if nothing has happened.
key = winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE, r'SYSTEM\CurrentControlSet\Control\Session Manager')
_get_reg_value(key, 'Environment', 'Path')

If Python's winreg library simply wraps the corresponding Windows APIs, then you have to open each component of the key. You can't say:
winreg.OpenKey(winreg::HKEY_LOCAL_MACHINE, "FOO\\BAR")
Instead, you first have to open FOO and then open BAR:
foo_key = winreg.OpenKey(winreg::HKEY_LOCAL_MACHINE, "FOO")
bar_key = winreg.OpenKey(foo_key, "BAR")
This is likely the root of your problem.
But there may be more to it. After making such a change, you're supposed to broadcast a WM_SETTINGCHANGE so that other processes (like the shell) know to invalidate their caches and reread the system settings. It's likely the PowerShell commandlet does that automatically.
There can be other issues, too. For example, if you're running a 32-bit Python on a 64-bit OS, you might run into Registry redirection, depending exactly on which part(s) of the hive you're trying to access.
To clarify, what you're calling the "permanent path values" is part of the system environment variables. If you have privileges (e.g., running as Administrator), you are correct that you can modify the path from the system environment block at those keys.
The system merges the user environment variables with the system environment variables. If PATH is defined in both environments, the final PATH is the concatenation of the system values followed by the user values. For other variables with both a system and a user definition (e.g., TMP), the user values are used.
Child processes inherit a copy of their parent process's environment. So, even with a WM_SETTINGCHANGE broadcast, they probably won't update their path to reflect changes you've made to the system environment block.

Methods to avoid hard-coding file paths in Python

Working with scientific data, specifically climate data, I am constantly hard-coding paths to data directories in my Python code. Even if I were to write the most extensible code in the world, the hard-coded file paths prevent it from ever being truly portable. I also feel like having information about the file system of your machine coded in your programs could be security issue.
What solutions are out there for handling the configuration of paths in Python to avoid having to code them out explicitly?

One of the solution rely on using configuration files.
You can store all your path in a json file like so :
{
"base_path" : "/home/bob/base_folder",
"low_temp_area_path" : "/home/bob/base/folder/low_temp"
}
and then in your python code, you could just do :
import json
with open("conf.json") as json_conf :
CONF = json.load(json_conf)
and then you can use your path (or any configuration variable you like) like so :
print "The base path is {}".format(CONF["base_path"])

First off its always good practise to add a main function to go with each class to test that class or functions in the file. Along with this you determine the current working directory. This becomes incredibly important when running python from a cron job or from a directory that is not the current working directory. No JSON files or environment variables are then needed and you will obtain interoperation across Mac, RHEL and Debian distributions.
This is how you do it, and it will work on windows also if you use '\' instead of '/' (if that is even necessary, in your case).
if "__main__" == __name__:
workingDirectory = os.path.realpath(sys.argv[0])
As you can see when you run your command, the working directory is calculated if you provide a full path or relative path, meaning it will work in a cron job automatically.
After that if you want to work with data that is stored in the current directory use:
fileName = os.path.join( workingDirectory, './sub-folder-of-current-directory/filename.csv' )
fp = open( fileName,'r')
or in the case of the above working directory (parallel to your project directory):
fileName = os.path.join( workingDirectory, '../folder-at-same-level-as-my-project/filename.csv' )
fp = open( fileName,'r')

I believe there are many ways around this, but here is what I would do:
Create a JSON config file with all the paths I need defined.
For even more portability, I'd have a default path where I look for this config file but also have a command line input to change it.

In my opinion passing arguments from command line would be best solution. You should take a look at argparse . This allows you to create nice way to handle arguments from the command line. for example:
myDataScript.py /home/userName/datasource1/

Detect where Python code is running (e.g., in Spyder interpreter vs. IDLE vs. cmd)

Is there a way in Python to detect, within a process, where that process is being executed? I have some code that includes the getpass.getpass() function, which is broken in Spyder, and it's annoying to go back and forth between the command line and the IDE all the time. It would be useful if I could add code like:
if not being run from Spyder:
use getpass
else:
use alternative

Here is the solution I ended up using. After reading Markus's answer, I noticed that Spyder adds half a dozen or so environment variables to os.environ with names like SPYDER_ENCODING, SPYDER_SHELL_ID, etc. Detecting the presence of any of these seems relatively unambiguous, compared to detecting the absence of a variable with as generic a name as 'PYTHONSTARTUP'. The code is simple, and works independently of Spyder's startup script (as far as I can tell):
if any('SPYDER' in name for name in os.environ)
# use alternative
else:
# use getpass
Since the string is at the beginning of each environment variable name, you could also use str.startswith, but it's less flexible, and a little bit slower (I was curious):
>>> import timeit
>>> s = timeit.Timer("[name.startswith('SPYDER') for name in os.environ]", "import os")
>>> i = timeit.Timer("['SPYDER' in name for name in os.environ]", "import os")
>>> s.timeit()
16.18333065883474
>>> i.timeit()
6.156869294143846
The sys.executable method may or may not be useful depending on your installation. I have a couple WinPython installations and a separate Python 2.7 installation, so I was able to check the condition sys.executable.find('WinPy') == -1 to detect a folder name in the path of the executable Spyder uses. Since the warning that shows in IDLE when you try to use getpass is less "loud" than it could be, in my opinion, I ended up also checking the condition sys.executable.find('pythonw.exe') == -1 to make it slightly louder. Using sys.executable only, that method looks like:
if sys.executable.find('pythonw.exe') == sys.executable.find('WinPy') == -1:
# use getpass
else:
# use alternative
But since I want this to work on other machines, and it's much more likely that another user would modify their WinPython installation folder name than that they would rename their IDLE executable, my final code uses sys.executable to detect IDLE and os.environ to detect Spyder, providing a "louder" warning in either case and keeping the code from breaking in the latter.
if any('SPYDER' in name for name in os.environ) \
or 'pythonw.exe' in sys.executable:
password = raw_input('WARNING: PASSWORD WILL BE SHOWN ON SCREEN\n\n' * 3
+ 'Please enter your password: ')
else:
password = getpass.getpass("Please enter your password: ")

By default, Spyder uses a startup scrip, see Preferences -> Console -> Adanced setting. This option is usually set to the scientific_startup.py file that loads pylab et al.
The easiest solution is to just add a global variable to the file and then use that in your if statement, e.g. add this line at the end of scientific_startup.py:
SPYDER_IDE_ACTIVE = True
In your script:
if not 'SPYDER_IDE_ACTIVE' in globals():
use getpass
else:
use alternative
This will work without throwing an error. You can also use exceptions if you like that more.
A second solution would be (if you cannot modify that file for some reason) to just check if the environment variable PYTHONSTARTUP is set. On my machine (using the Anaconda Python stack), it is not set for a regular Python shell. You could do
import os
if not 'PYTHONSTARTUP' in os.environ:
use getpass
else:
use alternative

Spyder provides the option of executing the current editor script in a native system terminal. This would produce identical behavior as if you were running from the command line. To set this up, open the Run Settings dialog by hitting F6. Then select the radio button "Execute in an external System terminal". Now run the script as usual by hitting F5. You should be able to use getpass in the normal fashion with this approach.

You could add env variable when running in Spyder and check it in code.

How to find the real user home directory using python?

I see that if we change the HOME (linux) or USERPROFILE (windows) environmental variable and run a python script, it returns the new value as the user home when I try
os.environ['HOME']
os.exp
Is there any way to find the real user home directory without relying on the environmental variable?
edit:
Here is a way to find userhome in windows by reading in the registry,
http://mail.python.org/pipermail/python-win32/2008-January/006677.html
edit:
One way to find windows home using pywin32,
from win32com.shell import shell,shellcon
home = shell.SHGetFolderPath(0, shellcon.CSIDL_PROFILE, None, 0)

I think os.path.expanduser(path) could be helpful.
On Unix and Windows, return the argument with an initial component of ~ or ~user replaced by that user‘s home directory.
On Unix, an initial ~ is replaced by the environment variable HOME if it is set; otherwise the current user’s home directory is looked up in the password directory through the built-in module pwd. An initial ~user is looked up directly in the password directory.
On Windows, HOME and USERPROFILE will be used if set, otherwise a combination of HOMEPATH and HOMEDRIVE will be used. An initial ~user is handled by stripping the last directory component from the created user path derived above.
If the expansion fails or if the path does not begin with a tilde, the path is returned unchanged.
So you could just do:
os.path.expanduser('~user')

from pathlib import Path
str(Path.home())
works in Python 3.5 and above. Path.home() returns a Path object providing an API I find very useful.

I think os.path.expanduser(path) is the best answer to your question, but there's an alternative that may be worth mentioning in the Unix world: the pwd package. e.g.
import os, pwd
pwd.getpwuid(os.getuid()).pw_dir

For windows;
import os
homepath = os.path.expanduser(os.getenv('USERPROFILE'))
will give you a handle to current user's home directory and
filepath = os.path.expanduser(os.getenv('USERPROFILE'))+'\\Documents\\myfile.txt'
will give you a handle to below file;
C:\Users\urUserName\Documents\myfile.txt

home_folder = os.getenv('HOME')
This should work on Windows and Mac OS too, works well on Linux.

Really, a change in environment variable indicates that the home must be changed. So every program/script should have the new home in context; also the consequences are up to the person who changed it.
I would still stick with
home = os.getenv('USERPROFILE') or os.getenv('HOME')
what exactly is required?

I realize that this is an old question that has been answered but I thought I would add my two cents. The accepted answer was not working for me. I needed to find the user directory and I wanted it to work with and without sudo. In Linux, my user directory is "/home/someuser" but my root directory is "/root/". However, on my Mac, the user directory is "/Users/someuser". Here is what I ended up doing:
_USERNAME = os.getenv("SUDO_USER") or os.getenv("USER")
_HOME = os.path.expanduser('~'+_USERNAME)
This worked both with and without sudo on Mac and Linux.

get (translated) user folder names on Linux:
from gi.repository import GLib
docs = GLib.get_user_special_dir(GLib.USER_DIRECTORY_DOCUMENTS)
desktop = GLib.get_user_special_dir(GLib.USER_DIRECTORY_DESKTOP)
pics = GLib.get_user_special_dir(GLib.USER_DIRECTORY_PICTURES)
videos = GLib.get_user_special_dir(GLib.USER_DIRECTORY_VIDEOS)
music = GLib.get_user_special_dir(GLib.USER_DIRECTORY_MUSIC)
downloads = GLib.get_user_special_dir(GLib.USER_DIRECTORY_DOWNLOAD)
public = GLib.get_user_special_dir(GLib.USER_DIRECTORY_PUBLIC_SHARE)
templates = GLib.get_user_special_dir(GLib.USER_DIRECTORY_TEMPLATES)
print(docs)
print(desktop)
print(pics)
print(videos)
print(music)
print(downloads)
print(public)
print(templates)

On Linux and other UNIXoids you can always take a peek in /etc/passwd. The home directory is the sixth colon-separated field in there. No idea on how to do better than the environment variable on Windows though. There'll be a system call for it, but if it's available from Python, ...