Develop Reference

How to debug code run using Popen in Pycharm

I am running a codebase I have inherited from someone else which makes extensive use of user input. For this reason I am running it using subprocess.Popen. Here is an example. The following script (caller.py) calls the third-party code.
from subprocess import Popen, PIPE, STDOUT
import sys
user_input = ['John', '555']
communicate_argument = '\n'.join(user_input)
p = Popen([sys.executable, 'example2.py'], stdout=PIPE, stdin=PIPE, stderr=STDOUT, encoding='utf-8')
stdout, stderr = p.communicate(communicate_argument)
print(stdout)
The following script (example.py) emulates behavior of the source code I was provided, by accepting a couple of input arguments from the user:
name = input('What is your name\n')
age = input('What is your age\n')
print('You are {}, and you are {} years old'.format(name, age))
Running the code works fine, and I get the expected output.
Debugging the code partially works, but partially doesn't. The debugger successfully attaches to the child process p such that any breakpoints placed in example.py will work. However it seems that the debug console does not successfully attached to the child process. When I try to enter some variables in the debug console, they do not print out, even if they appear as active variables in my debug session.
EDIT
It turns out this might be a bug. I have asked the same question in pycharm's official forum and they made an issue out of it:
https://intellij-support.jetbrains.com/hc/en-us/community/posts/360009571680-Debugging-code-run-through-subprocess-Popen
So I guess what I'd be looking for is an effective workaround which would allow me to use a regular python interpreter in combination with the debugger to inspect and run operations on variables.

I have had a similar issue with python and docker logs. What solved was running python with the -u flag as described here. Perhaps try changing your Popen call to use python and then include the -u flag. e.g. Popen(["python", "-u", "example2.py"], stdout=PIPE, stdin=PIPE, stderr=STDOUT, encoding='utf-8')

Python subprocess.run C Program not working

I am trying to write the codes to run a C executable using Python.
The C program can be run in the terminal just by calling ./myprogram and it will prompt a selection menu, as shown below:
1. Login
2. Register
Now, using Python and subprocess, I write the following codes:
import subprocess
subprocess.run(["./myprogram"])
The Python program runs but it shows nothing (No errors too!). Any ideas why it is happening?
When I tried:
import subprocess
subprocess.run(["ls"])
All the files in that particular directory are showing. So I assume this is right.

You have to open the subprocess like this:
import subprocess
cmd = subprocess.Popen(['./myprogram'], stdin=subprocess.PIPE)
This means that cmd will have a .stdin you can write to; print by default sends output to your Python script's stdout, which has no connection with the subprocess' stdin. So do that:
cmd.stdin.write('1\n') # tell myprogram to select 1
and then quite probably you should:
cmd.stdin.flush() # don't let your input stay in in-memory-buffers
or
cmd.stdin.close() # if you're done with writing to the subprocess.
PS If your Python script is a long-running process on a *nix system and you notice your subprocess has ended but is still displayed as a Z (zombie) process, please check that answer.

Maybe flush stdout?
print("", flush=True,end="")

Wait for subprocess .exe to finish before proceeding in Python

I'm running an application from within my code, and it rewrites files which I need to read later on in the code. There is no output the goes directly into my program. I can't get my code to wait until the subprocess has finished, it just goes ahead and reads the unchanged files.
I've tried subprocess.Popen.wait(), subprocess.call(), and subprocess.check_call(), but none of them work for my problem. Does anyone have any idea how to make this work? Thanks.
Edit: Here is the relevant part of my code:
os.chdir('C:\Users\Jeremy\Documents\FORCAST\dusty')
t = subprocess.Popen('start dusty.exe', shell=True)
t.wait()
os.chdir('C:\Users\Jeremy\Documents\FORCAST')

Do you use the return object of subprocess.Popen()?
p = subprocess.Popen(command)
p.wait()
should work.
Are you sure that the command does not end instantly?

If you execute a program with
t = subprocess.Popen(prog, Shell=True)
Python won't thrown an error, regardless whether the program exists or not. If you try to start an non-existing program with Popen and Shell=False, you will get an error. My guess would be that your program either doesn't exist in the folder or doesn't execute. Try to execute in the Python IDLE environment with Shell=False and see if you get a new window.

How to execute a command prompt command from python

I tried something like this, but with no effect:
command = "cmd.exe"
proc = subprocess.Popen(command, stdin = subprocess.PIPE, stdout = subprocess.PIPE)
proc.stdin.write("dir c:\\")

how about simply:
import os
os.system('dir c:\\')

You probably want to try something like this:
command = "cmd.exe /C dir C:\\"
I don't think you can pipe into cmd.exe... If you are coming from a unix background, well, cmd.exe has some ugly warts!
EDIT: According to Sven Marnach, you can pipe to cmd.exe. I tried following in a python shell:
>>> import subprocess
>>> proc = subprocess.Popen('cmd.exe', stdin = subprocess.PIPE, stdout = subprocess.PIPE)
>>> stdout, stderr = proc.communicate('dir c:\\')
>>> stdout
'Microsoft Windows [Version 6.1.7600]\r\nCopyright (c) 2009 Microsoft Corporatio
n. All rights reserved.\r\n\r\nC:\\Python25>More? '
As you can see, you still have a bit of work to do (only the first line is returned), but you might be able to get this to work...

Try:
import os
os.popen("Your command here")

Using ' and " at the same time works great for me (Windows 10, python 3)
import os
os.system('"some cmd command here"')
for example to open my web browser I can use this:
os.system(r'"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe"')
(Edit)
for an easier way to open your browser I can use this:
import webbrowser
webbrowser.open('website or leave it alone if you only want to open the
browser')

Try adding a call to proc.stdin.flush() after writing to the pipe and see if things start behaving more as you expect. Explicitly flushing the pipe means you don't need to worry about exactly how the buffering is set up.
Also, don't forget to include a "\n" at the end of your command or your child shell will sit there at the prompt waiting for completion of the command entry.
I wrote about using Popen to manipulate an external shell instance in more detail at: Running three commands in the same process with Python
As was the case in that question, this trick can be valuable if you need to maintain shell state across multiple out-of-process invocations on a Windows machine.

Taking some inspiration from Daren Thomas's answer (and edit), try this:
proc = subprocess.Popen('dir C:\\', shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
out, err = proc.communicate()
out will now contain the text output.
They key nugget here is that the subprocess module already provides you shell integration with shell=True, so you don't need to call cmd.exe directly.
As a reminder, if you're in Python 3, this is going to be bytes, so you may want to do out.decode() to convert to a string.

Why do you want to call cmd.exe ? cmd.exe is a command line (shell). If you want to change directory, use os.chdir("C:\\"). Try not to call external commands if Python can provide it. In fact, most operating system commands are provide through the os module (and sys). I suggest you take a look at os module documentation to see the various methods available.

It's very simple. You need just two lines of code with just using the built-in function and also it takes the input and runs forever until you stop it. Also that 'cmd' in quotes, leave it and don't change it. Here is the code:
import os
os.system('cmd')
Now just run this code and see the whole windows command prompt in your python project!

Here's a way to just execute a command line command and get its output using the subprocess module:
import subprocess
# You can put the parts of your command in the list below or just use a string directly.
command_to_execute = ["echo", "Test"]
run = subprocess.run(command_to_execute, capture_output=True)
print(run.stdout) # the output "Test"
print(run.stderr) # the error part of the output
Just don't forget the capture_output=True argument and you're fine. Also, you will get the output as a binary string (b"something" in Python), but you can easily convert it using run.stdout.decode().

In Python, you can use CMD commands using these lines :
import os
os.system("YOUR_COMMAND_HERE")
Just replace YOUR_COMMAND_HERE with the command you like.

From Python you can do directly using below code
import subprocess
proc = subprocess.check_output('C:\Windows\System32\cmd.exe /k %windir%\System32\\reg.exe ADD HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System /v EnableLUA /t REG_DWORD /d 0 /f' ,stderr=subprocess.STDOUT,shell=True)
print(str(proc))
in first parameter just executed User Account setting you may customize with yours.

What's the problem with executing commands in Windows CMD from Python?

I'm having huge trouble with passing commands to CMD from Python.
First, I open a CMD process:
cmdprocess = subprocess.Popen("cmd",
stdin = subprocess.PIPE,
stdout = subprocess.PIPE,
stderr = subprocess.PIPE)
Then, I do something, for example:
for i in range(500):
#time.sleep(1)
command = ("dir > " + os.path.join("C:\\", str(i)) + "\r\n").encode("utf-8")
print(command)
cmdprocess.stdin.write(command)
So this is supposed to create 500 small text files in a folder. I tested it in Python 3.2 x64 and 3.2 x86 and the result for both is: it counts up to about 250-350 in the Python shell, and then just stops. No error, nothing. There are then the files 1-80 in the specified folder.
Now, I thought that maybe Python is too fast and so had it sleep(1) for 1 second between the commands. Now, it counts up to about 200 before the first file appears in the folder! and then stops at about 270.
What happens here and how can I force CMD to execute a command immediately?

Are you handling the output in the PIPES? They might be filling. If you fill up the stdout or stderror buffers from the process, it will stop execution.

I think you'd better to use pywin32 package. there are win32pipe and win32process modules.
I also had same issue but I could not resolve it without pywin32-site-package...
So now I am using them... If you need sample code and you're using windows, I will attach it.
if you mean linux... it's same but you need another one like IO select.