Develop Reference

Python: subprocess.Popen() returns None

I need to execute a CLI binary with args, keep the process alive and run multiple commands throughout the python script. So I am using Python and subprocess.Popen() in the following way:
from subprocess import Popen, PIPE
cmd = ["/full/path/to/binary","--arg1"]
process = Popen(cmd,stdin=PIPE, stdout=None)
process.stdin.write(f"command-for-the-CLI-tool".encode())
process.stdin.flush()
However, no matter how I call Popen(), the returned process object is None.
If I run process = Popen(cmd), without specifying stdin and stdout, I can see the process running correctly in the output console, meaning that the binary path and args are correct, but the process object is still None, meaning that I cannot issue other commands afterwards.
EDIT: The point of this is that I want to execute the following:
command = (
f"cat << EOF | {cmd}\n"
f"use {dbname};\n"
"set optimizer_switch='hypergraph_optimizer=on';\n"
f"SET forced_plan='{forced_plan}';\n"
f"{query_text}\n"
"EOF"
)
runtimes = []
for _ in trange(runs):
start = time.time()
subprocess.run(command, shell=True, stdout=sys.stdout)
runtimes.append(time.time() - start)
But this clearly measures the time of all the commands, whereas I am only interested in measuring the "query_text" command.
This is why I am looking for a solution where I can send the commands separately and time only the one I am interested in.
If I use multiple subprocess.run(), then the process instances will be different. I want the instance to be the same because the query depends on the previous commands.

With subprocess.run you can pass the entire input as ... input.
command = f"""\
use {dbname};
set optimizer_switch='hypergraph_optimizer=on';
SET forced_plan='{forced_plan}';
{query_text}
"""
runtimes = []
for _ in trange(runs):
start = time.time()
subprocess.run([cmd], text=true, input=command, stdout=sys.stdout)
runtimes.append(time.time() - start)
I took out shell=True; perhaps see also Actual meaning of shell=True in subprocess as well as perhaps Running Bash commands in Python which elaborates on several of the changes here.

Try using subprocess.run() instead of subprocess.Popen()
If you still use subprocess.Popen(), then you can use the .poll() method
But subprocess.Popen() will always return None if the execution of the command has not yet completed, or an exit code if the command has finished its execution.

Use subprocess to open an exe file and interact with it

I am using Python to script running an exe program.
If we open the exe program in the shell, we could enter different command such as "a", "b", "c" in the program. These commands can not be passed as flags into the exe program. I want to use Python to script running this exe program for many times, with custom exe-program specific input.
But if I run the "program.exe" with
p = subprocess.call(['program.exe'],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
)
Python won't terminate. Can I achieve this purpose with subprocess in Python?

Beware: subprocess.call will not return before the child process has terminated. So you have no possibility to write anything to the standard input of the child.
If you can prepare the bunch of commands in advance, and if output has no risk to fill the system buffer, you can still use call that way:
cmds = "a\nb\nc\n"
p = subprocess.call(['program.exe'],
stdin=io.StringIO(cmds),
stdout=subprocess.PIPE,
)
But the more robust way is to directly use the Popen constructor, and then feed the input:
p = subprocess.Popen(['program.exe'],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
)
p.stdin.write("a\n");
p.stdin.write("b\n");
...
p.stdin.close();
p.wait();
If you know that one subcommand will generate very large output you can read it before sending the next one. Beware to avoid being blocked waiting an input that the child has still not sent...

First, you have to use p = subprocess.Popen(…) in order to get the subprocess object. subprocess.call(…) would give you just the return status, and that's not enough.
If p is your connection object, you can send your commands to p.stdin, such as p.stdin.write("a\n"), and then read out p.stdout() until the next indication that the command output is finished. How you detect this depends on said program.
Then you can send the next command and read its output.
At the end, you can do p.stdin.close() in order to signal an EOF ot the other process, and then it should terminate.

python subprocess.Popen stdin.write

I'm new to python and would like to open a windows cmd prompt, start a process, leave the process running and then issue commands to the same running process.
The commands will change so i cant just include these commands in the cmdline variable below. Also, the process takes 10-15 seconds to start so i dont want to waste time waiting for the process to start and run commands each time. just want to start process once. and run quick commands as needed in the same process
I was hoping to use subprocess.Popen to make this work, though i am open to better methods. Note that my process to run is not cmd, but im just using this as example
import subprocess
cmdline = ['cmd', '/k']
cmd = subprocess.Popen(cmdline, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
cmd.stdin.write("echo hi") #would like this to be written to the cmd prompt
print cmd.stdout.readline() #would like to see 'hi' readback
cmd.stdin.write("echo hi again") #would like this to be written to the cmd prompt
print cmd.stdout.readline() #would like to see 'hi again' readback
The results arent what i expect. Seems as though the stdin.write commands arent actually getting in and the readline freezes up with nothing to read.
I have tried the popen.communicate() instead of write/readline, but it kills the process. I have tried setting bufsize in the Popen line, but that didn't make too much difference

Your comments suggest that you are confusing command-line arguments with input via stdin. Namely, the fact that system-console.exe program accepts script=filename parameter does not imply that you can send it the same string as a command via stdin e.g., python executable accepts -c "print(1)" command-line arguments but it is a SyntaxError if you pass it as a command to Python shell.
Therefore, the first step is to use the correct syntax. Suppose the system-console.exe accepts a filename by itself:
#!/usr/bin/env python3
import time
from subprocess import Popen, PIPE
with Popen(r'C:\full\path\to\system-console.exe -cli -',
stdin=PIPE, bufsize=1, universal_newlines=True) as shell:
for _ in range(10):
print('capture.tcl', file=shell.stdin, flush=True)
time.sleep(5)
Note: if you've redirected more than one stream e.g., stdin, stdout then you should read/write both streams concurrently (e.g., using multiple threads) otherwise it is very easy to deadlock your program.
Related:
Q: Why not just use a pipe (popen())? -- mandatory reading for Unix environment but it might also be applicable for some programs on Windows
subprocess readline hangs waiting for EOF -- code example on how to pass multiple inputs, read multiple outputs using subprocess, pexpect modules.
The second and the following steps might have to deal with buffering issues on the side of the child process (out of your hands on Windows), whether system-console allows to redirect its stdin/stdout or whether it works with a console directly, and character encoding issues (how various commands in the pipeline encode text).

Here is some code that I tested and is working on Windows 10, Quartus Prime 15.1 and Python 3.5
import subprocess
class altera_system_console:
def __init__(self):
sc_path = r'C:\altera_lite\15.1\quartus\sopc_builder\bin\system-console.exe --cli --disable_readline'
self.console = subprocess.Popen(sc_path, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
def read_output(self):
rtn = ""
loop = True
i = 0
match = '% '
while loop:
out = self.console.stdout.read1(1)
if bytes(match[i],'utf-8') == out:
i = i+1
if i==len(match):
loop=False
else:
rtn = rtn + out.decode('utf-8')
return rtn
def cmd(self,cmd_string):
self.console.stdin.write(bytes(cmd_string+'\n','utf-8'))
self.console.stdin.flush()
c = altera_system_console()
print(c.read_output())
c.cmd('set jtag_master [lindex [get_service_paths master] 0]')
print(c.read_output())
c.cmd('open_service master $jtag_master')
print(c.read_output())
c.cmd('master_write_8 $jtag_master 0x00 0xFF')
print(c.read_output())

You need to use iter if you want to see the output in real time:
import subprocess
cmdline = ['cmd', '/k']
cmd = subprocess.Popen(cmdline, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
cmd.stdin.write("echo hi\n")#would like this to be written to the cmd prompt
for line in iter(cmd.stdout.readline,""):
print line
cmd.stdin.write("echo hi again\n")#would like this to be written to the cmd prompt
Not sure exactly what you are trying to do but if you want to input certain data when you get certain output then I would recommend using pexpect

Popen in Python 3

HERE is a code snippet from my program.
I am trying to open cmd.exe on Windows and pass commands to a separate program and capture the output and parse it WITHOUT having to load cmd.exe every time.
All the examples to do this I found where using Python 2, and there are several changes in Python 3 about PIPES, making me unsure what is going wrong.
#DOScmd is a list of command line parameters to type into command shell.
p = subprocess.Popen('cmd.exe',
stdout=subprocess.PIPE,
stdin=subprocess.PIPE,
shell=True,
bufsize=0)
myCall = ' '.join(DOScmd) + '\n'
p.stdin.write( bytes(myCall, 'UTF-8') )
searchLines = p.stdout.readlines()
print(searchLines)
I am calling a program bowtie.exe. Now, bowtie.exe crashes when I do this. I think I might be angering the I/O gods. Any help appreciate.

I am trying to open cmd.exe on Windows and pass commands to a separate program and capture the output and parse it WITHOUT having to load cmd.exe every time.
Unless you want to run commands that are builtin to cmd.exe such as dir then you don't need to start cmd.exe at all:
from subprocess import check_output
for cmd in ["first.exe", "arg1", "arg2"], ["second.exe", ".."]:
output = check_output(cmd)
do_whatever_you_like_with(output)

How to execute a command prompt command from python

I tried something like this, but with no effect:
command = "cmd.exe"
proc = subprocess.Popen(command, stdin = subprocess.PIPE, stdout = subprocess.PIPE)
proc.stdin.write("dir c:\\")

how about simply:
import os
os.system('dir c:\\')

You probably want to try something like this:
command = "cmd.exe /C dir C:\\"
I don't think you can pipe into cmd.exe... If you are coming from a unix background, well, cmd.exe has some ugly warts!
EDIT: According to Sven Marnach, you can pipe to cmd.exe. I tried following in a python shell:
>>> import subprocess
>>> proc = subprocess.Popen('cmd.exe', stdin = subprocess.PIPE, stdout = subprocess.PIPE)
>>> stdout, stderr = proc.communicate('dir c:\\')
>>> stdout
'Microsoft Windows [Version 6.1.7600]\r\nCopyright (c) 2009 Microsoft Corporatio
n. All rights reserved.\r\n\r\nC:\\Python25>More? '
As you can see, you still have a bit of work to do (only the first line is returned), but you might be able to get this to work...

Try:
import os
os.popen("Your command here")

Using ' and " at the same time works great for me (Windows 10, python 3)
import os
os.system('"some cmd command here"')
for example to open my web browser I can use this:
os.system(r'"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe"')
(Edit)
for an easier way to open your browser I can use this:
import webbrowser
webbrowser.open('website or leave it alone if you only want to open the
browser')

Try adding a call to proc.stdin.flush() after writing to the pipe and see if things start behaving more as you expect. Explicitly flushing the pipe means you don't need to worry about exactly how the buffering is set up.
Also, don't forget to include a "\n" at the end of your command or your child shell will sit there at the prompt waiting for completion of the command entry.
I wrote about using Popen to manipulate an external shell instance in more detail at: Running three commands in the same process with Python
As was the case in that question, this trick can be valuable if you need to maintain shell state across multiple out-of-process invocations on a Windows machine.

Taking some inspiration from Daren Thomas's answer (and edit), try this:
proc = subprocess.Popen('dir C:\\', shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
out, err = proc.communicate()
out will now contain the text output.
They key nugget here is that the subprocess module already provides you shell integration with shell=True, so you don't need to call cmd.exe directly.
As a reminder, if you're in Python 3, this is going to be bytes, so you may want to do out.decode() to convert to a string.

Why do you want to call cmd.exe ? cmd.exe is a command line (shell). If you want to change directory, use os.chdir("C:\\"). Try not to call external commands if Python can provide it. In fact, most operating system commands are provide through the os module (and sys). I suggest you take a look at os module documentation to see the various methods available.

It's very simple. You need just two lines of code with just using the built-in function and also it takes the input and runs forever until you stop it. Also that 'cmd' in quotes, leave it and don't change it. Here is the code:
import os
os.system('cmd')
Now just run this code and see the whole windows command prompt in your python project!

Here's a way to just execute a command line command and get its output using the subprocess module:
import subprocess
# You can put the parts of your command in the list below or just use a string directly.
command_to_execute = ["echo", "Test"]
run = subprocess.run(command_to_execute, capture_output=True)
print(run.stdout) # the output "Test"
print(run.stderr) # the error part of the output
Just don't forget the capture_output=True argument and you're fine. Also, you will get the output as a binary string (b"something" in Python), but you can easily convert it using run.stdout.decode().

In Python, you can use CMD commands using these lines :
import os
os.system("YOUR_COMMAND_HERE")
Just replace YOUR_COMMAND_HERE with the command you like.

From Python you can do directly using below code
import subprocess
proc = subprocess.check_output('C:\Windows\System32\cmd.exe /k %windir%\System32\\reg.exe ADD HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System /v EnableLUA /t REG_DWORD /d 0 /f' ,stderr=subprocess.STDOUT,shell=True)
print(str(proc))
in first parameter just executed User Account setting you may customize with yours.