I have a very basic subprocess.Popen command like:
cmd = ['docker', 'run', '--name=test','server']
p = subprocess.Popen(cmd, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
When I run this in a Python script from terminal, this correctly launches the docker image and the image is persisted until I docker stop it.
However, when I run the same code from PyCharm 5, the docker image is immediately stopped when the above line is completed.
I have verified this in the debugger (running docker ps -a shows this image immediately dies when launched from PyCharm).
What do I have to do in order to keep my subprocess opened when running with PyCharm? Not being able to run the debugger is quite annoying.
Are you printing out the output from the Popen call (stdout and stderr)? What does it say?
Launching subprocesses from an IDE can be tricky, because certain environment variables (like PATH) may not be set in the same way as your shell.
Related
I am trying to launch a python subprocess from excel using PYXLL, but it seems to have trouble launching the cmd window and running commands.
Below is a sample of what I am trying to run:
#xl_macro()
def test():
if 1 == 1:
xlcAlert("Next line nothing happens") #Popup appears
p = subprocess.Popen(r'start cmd /k', shell=True, creationflags=subprocess.CREATE_NEW_CONSOLE, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
xlcAlert("{}".format(p.pid)) #p was never launched
I am trying to capture values from excel and pass them in a subprocess. This works when executing in my IDE: data is read from excel and then subprocess launches window. However, once adding the decorator to have it run as macro in EXCEL, the script will just stop once subprocess.Popen line is reached. Is there any way to launch a subprocess from pyxll?
After investigation, and thanks to Charles Duffy, Microsoft Office SandBoxing kills the shell subprocess. This has been implemented for security reasons in latest versions.
The simple solution is to run subprocess with shell=False and pass the args in a list:
p1 = subprocess.Popen(cmdlist, shell=False)
The Sandboxing will not terminate the process - python window will open while script is running.
I am trying to run the following simple command in Python which returns a directory associated with the Node.js
import subprocess
import shutil
cmd = [shutil.which("npm"), 'bin']
subprocess.run(cmd, stdout=subprocess.PIPE)
I've tested this on two different systems, on my Mac it runs almost instantly whereas on my Windows 10 system it hangs for a long time even though running the command npm bin in the terminal completes immediately. Running other commands, say cmd = [shutil.which("condo"), "info"] works fine. If I don't specify the stdout argument it will also work OK. I would like to figure why it is hanging and how to fix that.
UPDATE
Also tried
p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
p.communicate()
On Windows 10 under Python 3.6.10, same result as running subprocess.run().
I have a shell script (test.sh -> example shown below) which has a infinte while loop and prints some data to screen.
I am calling all my .sh scripts from python and I need to stop the test.sh before calling my other commands
I am using python 2.7 and linux system is on propritary hardware where I cannot install any python modules.
Here is my test.sh
#!/bin/sh
while :
do
echo "this code is in infinite while loop"
sleep 1
done
Here is my python Scripts
import subprocess as SP
SP.call(['./test.sh']) # I need to stop the test.sh in order for python to
# go and execute more commands and call
# another_script.sh
# some code statements
SP.call(['./another_script.sh'])
Well, quick google search made me look into subprocess call and Popen modules . and Popen has a terminate option and it doesn't work for me (or) I'm doing something wrong here
cmd=['test.sh']
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
p.terminate()
Any other suggestions on how I can stop the test.sh from python are highly appreciated
PS: I don't mind to run the test.sh for like T seconds and then stop it
I use tmux for these type of processes, python has a good package libtmux which should solve your problem.
Basically you create a tmux session:
import libtmux
server = libtmux.Server()
session = server.new_session(session_name='my_session_name')
then you create a window to run the command in
window = session.new_window(attach=False, window_name='my_window_name')
command = './my_bash_file.sh'
window.select_pane('0').send_keys(command, enter=True)
You'll be able to run subsequent commands right after this one. To access the tmux session from your bash terminal use tmux attach -t my_session_name you'll then be in a tmux window, the one which ran your bash script.
To kill the tmux window use window.kill_window() there's a lot of options look at the libtmux docs.
The project aileen has some useful tmux commands if you want to see some more implementations.
I am pulling my hair out here. I am spawning a process which I need the feedback from in Python.
When I run the command in the cmd window it runs fine, but when I try to run it via Python the terminal hangs.
p = subprocess.Popen(startcmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(out, err) = p.communicate()
Where startcmd is a string which when printed in the Python console looks like this:
"C:/Program Files/GRASS GIS 7.2.1/grass72.bat" --version
If I copy and paste this into a Windows cmd, it shows the version information and returns control to the command prompt about a second later, but in Python it freezes up.
I should point out, if I replace the startcmd string with something like "dir" or even "python --version", it works fine!
Additional: I have tried shell=True, this has the same result.
Additional: I have tried sending the cmd and arguments through as an array as suggested in an answer below given that shell=False, but this also hangs the same.
Additional: I have added the GRASS path to the system PATH, so that now I can simply call grass72 --version in the cmd window to get a result, however this also still freezes in Python but works fine in cmd.
Additional: I have created a basic .bat file to test if .bat files run ok via Python, here is what I created:
#echo off
title Test Batch Script
echo I should see this message
This runs fine both in cmd, and in Python.
Problem found but not solved!
So, I'm running the script which spawns the process using subprocess.Popen using Python 3.6. The .bat file which is spawned launches a Python script using a version of Python (based on 2.7) which comes shipped with GRASS:
%GRASS_PYTHON% "\BLAH\BLAH\grass72.py"
What is interesting, is that if I launch the subprocess.Popen script with Python 2.7, it works fine. Ahah, you may think, solved! But this doesn't solve my problem - because I really need Python 3.6 to be launching the process, also why does it matter what version of Python launches the batch file? The new Python script which is spawned is launched with Python 2.7 anyway.
Since I started re-directing stdout I can see that there is an error when I use Python 3.6 to launch the process:
File "C:\ProgramData\Anaconda3\lib\site.py", line 177
file=sys.stderr)
^
SyntaxError: invalid syntax
Notice its reverting to Anaconda3! Even though it is launched using python.exe from 2.7!
I experienced the same issue with Python 3.6 and 3.7 on Windows hanging for subprocess calls:
p = subprocess.Popen(startcmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(out, err) = p.communicate()
Upon closer investigation I noticed this occurs only if the process writes more than about 4 KB (4096 bytes) of output which might explain why your short script does not reproduce this.
A workaround I found is using tempfile in the standard library:
# Write to a temporary file because pipe redirection seems broken
with tempfile.NamedTemporaryFile(mode="w+") as tmp_out,
tempfile.NamedTemporaryFile(mode="w+") as tmp_err:
p = subprocess.Popen(startcmd, stdout=tmp_out, stderr=tmp_err,
universal_newlines=True)
# `run` waits for command to complete, `Popen` continues Python program
while p.poll() is None:
time.sleep(.1)
# Cursor is after the last write call, reset to read output
tmp_out.seek(0)
tmp_err.seek(0)
out = tmp_out.read()
err = tmp_err.read()
You don't specify shell=True in your arguments to Popen. The recommended usage in that case is to specify a sequence of arguments instead of a string. So you should set startcmd equal to ["C:/Program Files/GRASS GIS 7.2.1/grass72.bat", "--version"].
Try this:
p = subprocess.Popen(startcmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
I'm using Python 2.6 on Windows 7. I have Windows .cmd file which invokes Python to run the CherryPy Web server (version 3.1.2). I start this .cmd file by executing it at the prompt in a Windows CMD shell. When CherryPy notices that one of its dependent files has changed, it restarts. There are a couple problems that arise in this configuration on Windows, because the invocation that CherryPy uses to restart itself is
os.execv(sys.executable, args)
but you can't invoke the Python executable on the .cmd file. I've managed to bypass that problem through various Python gymnastics, and now I've got it restarting by calling (essentially)
os.execv(sys.argv[0], args)
But the restarted process is detached from the keyboard; it will catch Ctrl-C, and then I'm asked
Terminate batch job (Y/N)?
but the Y or N I type is caught not by the terminating .cmd file, but by the Windows CMD shell, and it responds
'Y' is not recognized as an internal or external command, operable program or batch file.
So how do I set things up in Windows to keep the restarted process in what I'd call "the foreground" in Unix?
you can use subprocess.call method like this example
import subprocess
retcode = subprocess.call(["ls", "-l"])
see this link for more information about subprocess class:
subprocess — Subprocess management