Develop Reference

python subprocess behaviour changed between ubuntu 20 and 22?

I upgraded my ubuntu installation to 22.04 from 18.04 yesterday. Now I notice that python virtual environment is no longer working as expect.
I use python to run a lot of tools, and hence am highly dependent on subprocess library.
However to my "horror" I notice that it changed quite a lot, even when I keep using python 3.8. Mostly I notice I can no longer interact and the output is no longer piped to the shell that executes the python script.
import subprocess
def main():
proc = subprocess.run(["echo", "test"], check=True, stdout=subprocess.PIPE)
print('-------')
print(proc.stdout)
print("FINISH")
if __name__ == "__main__":
main()
If I call this with python3.8 test.py I notice that the output isn't displayed in the shell. However it is displayed when the prints happen.
What changed and how do I fix this? So output of the subprocess is piped to the output and can be seen?
Especially since a lot of tools are just running dockers (which in turn run git/javascript programs) and having output while the process is busy is kind of useful.

Python Popen hangs, yet same command in Windows cmd window runs fine

I am pulling my hair out here. I am spawning a process which I need the feedback from in Python.
When I run the command in the cmd window it runs fine, but when I try to run it via Python the terminal hangs.
p = subprocess.Popen(startcmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(out, err) = p.communicate()
Where startcmd is a string which when printed in the Python console looks like this:
"C:/Program Files/GRASS GIS 7.2.1/grass72.bat" --version
If I copy and paste this into a Windows cmd, it shows the version information and returns control to the command prompt about a second later, but in Python it freezes up.
I should point out, if I replace the startcmd string with something like "dir" or even "python --version", it works fine!
Additional: I have tried shell=True, this has the same result.
Additional: I have tried sending the cmd and arguments through as an array as suggested in an answer below given that shell=False, but this also hangs the same.
Additional: I have added the GRASS path to the system PATH, so that now I can simply call grass72 --version in the cmd window to get a result, however this also still freezes in Python but works fine in cmd.
Additional: I have created a basic .bat file to test if .bat files run ok via Python, here is what I created:
#echo off
title Test Batch Script
echo I should see this message
This runs fine both in cmd, and in Python.
Problem found but not solved!
So, I'm running the script which spawns the process using subprocess.Popen using Python 3.6. The .bat file which is spawned launches a Python script using a version of Python (based on 2.7) which comes shipped with GRASS:
%GRASS_PYTHON% "\BLAH\BLAH\grass72.py"
What is interesting, is that if I launch the subprocess.Popen script with Python 2.7, it works fine. Ahah, you may think, solved! But this doesn't solve my problem - because I really need Python 3.6 to be launching the process, also why does it matter what version of Python launches the batch file? The new Python script which is spawned is launched with Python 2.7 anyway.
Since I started re-directing stdout I can see that there is an error when I use Python 3.6 to launch the process:
File "C:\ProgramData\Anaconda3\lib\site.py", line 177
file=sys.stderr)
^
SyntaxError: invalid syntax
Notice its reverting to Anaconda3! Even though it is launched using python.exe from 2.7!

I experienced the same issue with Python 3.6 and 3.7 on Windows hanging for subprocess calls:
p = subprocess.Popen(startcmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(out, err) = p.communicate()
Upon closer investigation I noticed this occurs only if the process writes more than about 4 KB (4096 bytes) of output which might explain why your short script does not reproduce this.
A workaround I found is using tempfile in the standard library:
# Write to a temporary file because pipe redirection seems broken
with tempfile.NamedTemporaryFile(mode="w+") as tmp_out,
tempfile.NamedTemporaryFile(mode="w+") as tmp_err:
p = subprocess.Popen(startcmd, stdout=tmp_out, stderr=tmp_err,
universal_newlines=True)
# `run` waits for command to complete, `Popen` continues Python program
while p.poll() is None:
time.sleep(.1)
# Cursor is after the last write call, reset to read output
tmp_out.seek(0)
tmp_err.seek(0)
out = tmp_out.read()
err = tmp_err.read()

You don't specify shell=True in your arguments to Popen. The recommended usage in that case is to specify a sequence of arguments instead of a string. So you should set startcmd equal to ["C:/Program Files/GRASS GIS 7.2.1/grass72.bat", "--version"].

Try this:
p = subprocess.Popen(startcmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)

.pyw and pythonw does not run under Windows 7

Running a simple .py or .pyw python file causes python.exe to show up under Task Manager.
python myApp.py
python myApp.pyw
However when we try to run it without using the console, the script does not appear to run, nor does python.exe or pythonw.exe appears under Task Manager
pythonw myApp.pyw
pythonw myApp.py
How do we troubleshoot the problem? The system is running Python 2.7.8 x64.

tl;dr
To troubleshoot, use output redirection on invocation:
pythonw myApp.py 1>stdout.txt 2>stderr.txt
This will capture stdout output, such as from print(), in file stdout.txt, and stderr output (such as from unhandled exceptions), in file stderr.txt; from PowerShell, use
cmd /c pythonw myApp.py 1>stdout.txt 2>stderr.txt).
Note that the very act of redirecting stdout may actually make your script work again, if the only reason for its failure with pythonw was the use of print (in Python 2.x - see below).
Caveat: This output redirection technique seemingly does not work when invoking *.pyw scripts directly (as opposed to by passing the script file path to pythonw.exe). Do let me know if you know why and/or if it does work for you.
To fix your script:
Place the following at the top of any Python 2.x or 3.x script that you want to run with pythonw.exe:
import sys, os
if sys.executable.endswith("pythonw.exe"):
sys.stdout = open(os.devnull, "w");
sys.stderr = open(os.path.join(os.getenv("TEMP"), "stderr-"+os.path.basename(sys.argv[0])), "w")
This ensures the following when a script is run with pythonw.exe:
print() calls and explicit calls to sys.stdout() are effectively ignored (are no-ops).
Stderr output, including from an unhandled fatal exception, is sent to file
%TEMP%\stderr-<scriptFileName>; %TEMP% is a standard Windows environment variable that points to the current user's folder for temporary files.
In other words: With the above code in place, check file %TEMP%\stderr-<scriptFileName> after your script has failed silently when invoked with pythonw.exe.
For an explanation, read on.
On Windows, pythonw.exe is for launching GUI/no-UI-at-all scripts, which means that the
standard in- and output streams - sys.stdin, sys.stdout, sys.stderr are NOT available.
This has two nasty side effects:
Using print() - which targets sys.stdout by default - causes an exception in Python 2.x.
This problem has been fixed in Python 3.x.
Any unhandled exception - including one triggered by print() in 2.x - causes the script to abort silently.
Exception error messages go to sys.stderr by default, which is the very thing not available in this scenario.
The above code fixes these problems by:
sending stdout output to the null device, effectively ignoring any attempt to output to sys.stdout - whether explicitly, or implicitly via print().
sending all stderr output to a temporary file.
Differences between Python 2.x and Python 3.x:
When a script is run with pythonw.exe, sys.stdin, sys.stdout, and sys.stderr:
in Python 2.x: have invalid file descriptors
The eventual result when trying to write to sys.stdout or sys.stderr is the following exception: IOError: [Errno 9] Bad file descriptor
Pitfall: Due to output buffering, this exception may not surface until you've output, say, 4K bytes; you can provoke it instantly by invoking pythonw.exe with -u (for unbuffered output).
print() blindly tries to sys.stdout (by default), so it provokes this exception sooner or later.
in Python 3.x: are set to None
This is complemented with the 3.x print() function performing a no-op (doing nothing) when it finds that sys.stdout is None, so that print() statements can by default safely be used - they'll simply be ignored when run with pythonw.exe
However, it follows that trying to use sys.stdout.write() and sys.stderr.write() still results in an exception.
See here for more background.

Try adding the line import sys; sys.stderr = open("errlog.txt", "w") to the start of myApp.py. Then look in errlog.txt for a traceback or any other error messages.

I faced the same problem on a script of my own and found that when adding the output from Ross' answer the script would actually run.
It appears that for some reason that redirecting output fixes the problem. Since I'm not interested in writing the output to disk I've instead written it to /dev/null (or the platform equivalent) with:
if ( sys.platform == 'win32' and sys.executable.split( '\\' )[-1] == 'pythonw.exe'):
sys.stdout = open(os.devnull, 'w')
sys.stderr = open(os.devnull, 'w')
The if statement ensures it only happens when the script is launched from pythonw.exe. I'm not sure if it is related but it was important to do this before other imports (including e.g. import logging).

I was having similar problem.
After debugging step by step by writing to a log file, I discovered that pythonw.exe crashed after a statement that tried to use the call: sys.stdout.write(). It turns out, when run with pythonw.exe, sys.stdout is None.
If you are using functions of sys.stdout/stderr/stdin, and intend to use your program with pythonw.exe, adding a check for "None" is a good idea.

Im not sure I understand your problem but I think this is what you need to know
you need to right click on a py or pyw file and select open with ... find python.exe (probably C:\Python27\python.exe) .. check the box that says always open ... now you can just double click it if you want to run it
(usually the installer sets this up for you ...)

This is an old answer, but I want to leave my solution here also:
Open CMD (with elevated privileges or not - depends on your needs)
Change to directory of .py / .pyw script - this is important
Run pythonw with script as argument
cd E:\my\script\folder\
pythonw script.py

I had a similar problem after an upgrade to my computer RAM. Turns out I had to reinstall Pillow (library used for image processing). So make sure it is installed and if it's not, install it using "pip install Pillow" in cmd.

Why does running a PowerShell script from Python seem to delay?

When I launch a PowerShell script from Python, the delay seems to be approximately 45s, and I cannot figure out why.
I'm trying to run a PowerShell script (accessing some APIs only available to PowerShell) from a Python script.
I've tried a lot of permutations, and all incur ~45 second delay compared to just running the script from a command prompt, using an identical command line.
For example - sample.ps1 might say:
echo foo
And runner.py might say:
import subprocess
p = subprocess.Popen([POWERSHELL, '-File', 'sample.ps1'], stdout=subprocess.STDOUT)
d = p.stdout.read()
Running the .ps1 script directly is fast, running it via runner.py (Python 2.7, 32bit on a 64bit machine) incurs 45 second delay.
The exact same thing occurs if I use "os.system", or Twisted's built-in process tools. So I suspect it's some subtle interaction between the Python interpreter and the Powershell interpreter, possibly related to creation of console windows, or handling of stdin/out/err streams? (which I know don't "really exist" in the same way on Windows)

I do not see any such delays. It is pretty snappy. ( that will also depend on what your script actually does.) Try using call:
from subprocess import call
call(["powershell", "sample.ps1"])

PowerShell loads your user's profile by default. Use the -NoProfile argument to turn that behavior off:
import subprocess
p = subprocess.Popen([POWERSHELL, '-NoProfile', '-File', 'sample.ps1'], stdout=subprocess.STDOUT)
d = p.stdout.read()

python as a "batch" script (i.e. run commands from python)

I'm working in a windows environment (my laptop!) and I need a couple of scripts that run other programs, pretty much like a windows batch file.
how can I run a command from python such that the program when run, will replace the script? The program is interactive (for instance, unison) and keeps printing lines and asking for user input all the time.
So, just running a program and printing the output won't suffice. The program has to takeover the script's input/output, pretty mcuh like running the command from a .bat file.
I tried os.execl but it keeps telling me "invalid arguments", also, it doesn't find the program name (doesn't search the PATH variable); I have to give it the full path ..?!
basically, in a batch script I can write:
unison profile
how can I achieve the same effect in python?
EDIT:
I found out it can be done with os.system( ... ) and since I cannot accept my own answer, I'm closing the question.
EDIT: this was supposed to be a comment, but when I posted it I didn't have much points.
Thanks Claudiu, that's pretty much what I want, except for a little thing: I want the function to end when the program exits, but when I try it on unison, it doesn't return control to the python script, but to the windows command line environment
>>> os.execlp("unison")
C:\>Usage: unison [options]
or unison root1 root2 [options]
or unison profilename [options]
For a list of options, type "unison -help".
For a tutorial on basic usage, type "unison -doc tutorial".
For other documentation, type "unison -doc topics".
C:\>
C:\>
C:\>
how to get around this?

You should create a new processess using the subprocess module.
I'm not fluent in windows processes but its Popen function is cross-platform, and should be preffered to OS specific solutions.
EDIT: I maintain that you should prefer the Subprocess module to os.* OS specific functions, it is cross-platform and more pythonic (just google it). You can wait for the result easily, and cleanly:
import os
import subprocess
unison = os.path.join(os.path.curdir, "unison")
p = subprocess.Popen(unison)
p.wait()

I found out that os.system does what I want,
Thanks for all that tried to help.
os.system("dir")
runs the command just as if it was run from a batch file

import subprocess
proc = subprocess.Popen(['unison', 'profile'], stderr=subprocess.PIPE,
stdout=subprocess.PIPE, stdin=subprocess.PIPE)
proc.stdin.write('user input')
print proc.stdout.read()
This should help you get started. Please edit your question with more information if you want a more detailed answer!

os.execlp should work. This will search your path for the command. Don't give it any args if they're not necessary:
>>> import os
>>> os.execlp("cmd")
D:\Documents and Settings\Claudiu>Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.
D:\Documents and Settings\Claudiu>