I am pulling my hair out here. I am spawning a process which I need the feedback from in Python.
When I run the command in the cmd window it runs fine, but when I try to run it via Python the terminal hangs.
p = subprocess.Popen(startcmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(out, err) = p.communicate()
Where startcmd is a string which when printed in the Python console looks like this:
"C:/Program Files/GRASS GIS 7.2.1/grass72.bat" --version
If I copy and paste this into a Windows cmd, it shows the version information and returns control to the command prompt about a second later, but in Python it freezes up.
I should point out, if I replace the startcmd string with something like "dir" or even "python --version", it works fine!
Additional: I have tried shell=True, this has the same result.
Additional: I have tried sending the cmd and arguments through as an array as suggested in an answer below given that shell=False, but this also hangs the same.
Additional: I have added the GRASS path to the system PATH, so that now I can simply call grass72 --version in the cmd window to get a result, however this also still freezes in Python but works fine in cmd.
Additional: I have created a basic .bat file to test if .bat files run ok via Python, here is what I created:
#echo off
title Test Batch Script
echo I should see this message
This runs fine both in cmd, and in Python.
Problem found but not solved!
So, I'm running the script which spawns the process using subprocess.Popen using Python 3.6. The .bat file which is spawned launches a Python script using a version of Python (based on 2.7) which comes shipped with GRASS:
%GRASS_PYTHON% "\BLAH\BLAH\grass72.py"
What is interesting, is that if I launch the subprocess.Popen script with Python 2.7, it works fine. Ahah, you may think, solved! But this doesn't solve my problem - because I really need Python 3.6 to be launching the process, also why does it matter what version of Python launches the batch file? The new Python script which is spawned is launched with Python 2.7 anyway.
Since I started re-directing stdout I can see that there is an error when I use Python 3.6 to launch the process:
File "C:\ProgramData\Anaconda3\lib\site.py", line 177
file=sys.stderr)
^
SyntaxError: invalid syntax
Notice its reverting to Anaconda3! Even though it is launched using python.exe from 2.7!
I experienced the same issue with Python 3.6 and 3.7 on Windows hanging for subprocess calls:
p = subprocess.Popen(startcmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(out, err) = p.communicate()
Upon closer investigation I noticed this occurs only if the process writes more than about 4 KB (4096 bytes) of output which might explain why your short script does not reproduce this.
A workaround I found is using tempfile in the standard library:
# Write to a temporary file because pipe redirection seems broken
with tempfile.NamedTemporaryFile(mode="w+") as tmp_out,
tempfile.NamedTemporaryFile(mode="w+") as tmp_err:
p = subprocess.Popen(startcmd, stdout=tmp_out, stderr=tmp_err,
universal_newlines=True)
# `run` waits for command to complete, `Popen` continues Python program
while p.poll() is None:
time.sleep(.1)
# Cursor is after the last write call, reset to read output
tmp_out.seek(0)
tmp_err.seek(0)
out = tmp_out.read()
err = tmp_err.read()
You don't specify shell=True in your arguments to Popen. The recommended usage in that case is to specify a sequence of arguments instead of a string. So you should set startcmd equal to ["C:/Program Files/GRASS GIS 7.2.1/grass72.bat", "--version"].
Try this:
p = subprocess.Popen(startcmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
Related
I am trying to launch a python subprocess from excel using PYXLL, but it seems to have trouble launching the cmd window and running commands.
Below is a sample of what I am trying to run:
#xl_macro()
def test():
if 1 == 1:
xlcAlert("Next line nothing happens") #Popup appears
p = subprocess.Popen(r'start cmd /k', shell=True, creationflags=subprocess.CREATE_NEW_CONSOLE, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
xlcAlert("{}".format(p.pid)) #p was never launched
I am trying to capture values from excel and pass them in a subprocess. This works when executing in my IDE: data is read from excel and then subprocess launches window. However, once adding the decorator to have it run as macro in EXCEL, the script will just stop once subprocess.Popen line is reached. Is there any way to launch a subprocess from pyxll?
After investigation, and thanks to Charles Duffy, Microsoft Office SandBoxing kills the shell subprocess. This has been implemented for security reasons in latest versions.
The simple solution is to run subprocess with shell=False and pass the args in a list:
p1 = subprocess.Popen(cmdlist, shell=False)
The Sandboxing will not terminate the process - python window will open while script is running.
I am trying to run the following simple command in Python which returns a directory associated with the Node.js
import subprocess
import shutil
cmd = [shutil.which("npm"), 'bin']
subprocess.run(cmd, stdout=subprocess.PIPE)
I've tested this on two different systems, on my Mac it runs almost instantly whereas on my Windows 10 system it hangs for a long time even though running the command npm bin in the terminal completes immediately. Running other commands, say cmd = [shutil.which("condo"), "info"] works fine. If I don't specify the stdout argument it will also work OK. I would like to figure why it is hanging and how to fix that.
UPDATE
Also tried
p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
p.communicate()
On Windows 10 under Python 3.6.10, same result as running subprocess.run().
I am trying to run a simple Python script which runs the ipconfig /all command as a proof of concept.
You can find it below:
from subprocess import PIPE, run
my_command = "ipconfig /all"
result = run(my_command, stdout=PIPE, stderr=PIPE, universal_newlines=True)
print(result.stdout, result.stderr)
But I didn't suceed to run it, I tryed both with the command line and by clicking on it but it open a cmd window for 1 second, and then close it so I cannot even read it.
Edit: I am using Python 3.7 and my script is called ipconfig.py
Apparently, your problem is not related to the script itself, but rather to Python interpreter invocation. Check [Python 3.Docs]: How do I run a Python program under Windows?. A general approach would be to:
Open a cmd (PS) window in your script directory
Launch Python (using its full path: check [Python 3.Docs]: Using Python on Windows for more details) on your module (e.g.):
"C:\Program Files\Python37-64\python.exe" ipconfig.py
Of course, there are many ways to improve things, like adding its installation directory in %PATH% (if not already there) in order to avoid specifying its full path every time 1, but take one step at a time.
On the script side: check [Python 3.Docs]: subprocess.run(args, *, stdin=None, input=None, stdout=None, stderr=None, capture_output=False, shell=False, cwd=None, timeout=None, check=False, encoding=None, errors=None, text=None, env=None, universal_newlines=None) (and the examples):
Pass the arguments as a list:
my_command = ["ipconfig", "/all"]
You might also want to check the command termination status (result.returncode)
1: If you didn't check Add Python 3.7 to PATH when installing it (check image from 2nd URL), you have to add Python's path (C:\Users\USER\AppData\Local\Programs\Python\Python37) manually. There are many resources on the web, here are 3:
[SuperUser]: How do I add Python to the Windows PATH?
[Geek University]: Add Python to the Windows Path
[RaspberryPi.Projects]: Is Python in your PATH?
Your code is working good.
The problem is that the cmd closes the window too fast and you can't see the result.
Just add a command to wait for your interaction before closing the window.
You can add this at the end of your code:
input("Press Enter to finish...")
Or pause the execution after completion:
import time
[at the end of the code pause for 5 seconds....]
time.sleep(5)
I am using the subprocess module in Python 2.7. My code looks like
cmd = 'unrar l -p%s %s' % (pwd, filename)
proc = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
(output, err) = proc.communicate()
This code works fine on Windows 7/10, but it does not work on Ubuntu 16.04. I debugged and saw the variable err is empty, which on Windows contains the error message. And the variable output contains the command's help message. It looks like the command was run without any options.
If this is not the correct way to run a command and get its stdout/stderr output, what is the correct way?
Thanks to the tip provided by Jean-François Fabre, I dropped the shell=True parameter and the program works fine. I guess I will need to read the documentation on subprocess.Popen() to figure out why on Windows I need shell=True while on Ubuntu I don't, but for now I am happy.
I have a Perl script that I need to run with Python, and I've been trying to use subprocess to do it, unsuccessfully. I'm able to get the command to run just fine on the command line, but subprocess isn't able to get it to work.
If I have a Perl script like this:
#!/usr/bin/perl
use strict;
use warnings;
my $name = shift;
print "Hello $name!\n";
I am able to successfully run the command on the command line like so
C:\current\path> perl test.pl world
>>>Hello world!
But when I try and invoke the same command with subprocess, I get this error
cmd = 'perl test.pl world'
pipe = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE)
>>>"'perl' is not recognized as an internal or external command, operable program or batch file."
I've tried different combinations of creating cmd as a list
cmd = ['perl','test.pl','world']
and of giving subprocess the absolute path to Perl
cmd = ['C:\path\to\perl.exe','test.pl','world']
but nothing gets it to work. I'm able to get subprocess to play well without Perl
pipe = subprocess.Popen('dir',shell=True,stdout=subprocess.PIPE)
pipe.stdout.read()
>>>Volume in drive C is Windows7_OS....
I'm absolutely sure that Perl is in my PATH, because like I said I can invoke the command just fine in the command line. I've looked around at different posts that suggested to check os.environ["COMSPEC"], adding Perl to the subprocess env, and nothing has worked.
EDIT: I also can't get the command to work with any other subprocess methods: check_output() returns an empty byte string, and call() returns 1.
Any sort of fix or alternative solution would be immensely appreciated. I'm running Python 3.4 on Windows 7 64bit. I've tried 32- and 64-bit Python as well, to no avail.
UPDATE:
I've been able to get this to work on Mac, but with one difference: I can't have shell=True. Other than that, any subprocess function I want will work, including Popen, check_output, and call.
So I guess this is a Windows problem more than anything. I've tried not setting shell=True on my Windows machine, but I always get this error:
WindowsError: [Error 193] %1 is not a valid Win32 application
UPDATE 2:
I've also tried creating a .bat file that runs the command, but I get the same error as well I try to just call Perl.
pipe = subprocess.Popen('test.bat',shell=True,stdout=subprocess.PIPE)
>>>"'test.bat' is not recognized as an internal or external command, operable program or batch file."
Where test.bat has only one line:
echo Hello, World
you could try using os.system('command') instead. Apparently its not well regarded for some reason, or at least not as well regarded as subprocess though. Alternately, you could try some of the other subprocess methods, such as call, check_call, or check_output. I've had a similar problem with Popen in the past and switching to one of these methods helped.
I got the same error while trying to run a perl script from Python.
I managed to get it work by adding the full path of the perl.exe (with an extra escape character, as '\') to the command I was trying to run as below:
pipe = subprocess.Popen('C:\\Perl64\\bin\\perl myscrip.pl',shell=True,stdout=subprocess.PIPE)
pipe.stdout.read()
This works fine for me. I know it's been a while and things may have just fixed themselves.
I'm using activestate perl 5.8.9
and python 3.9.5 on Windows 10.
from subprocess import Popen, PIPE
proc = Popen(['perl', 'test.pl', 'world'], stdout=PIPE, stderr=PIPE)
stdout, stderr = proc.communicate()
if proc.returncode == 0:
print(stdout.decode())
else:
print(stderr.decode())