Develop Reference

Running sequential command line arguments from python and retrieving output

I've been banging my head against the wall long enough, throwing in the towel here.
I am trying to use Python (specifically 3.8.2) to interface with a tool that has an ugly command line interface. I have the below command, which works. However, I've been reading up and it seems like this is a deprecated method, and they recommend using subprocess.run now. I've been trying convert my code over and having a lot of trouble, so hoping to find some help. Code below, along with an explanation.
os.system(rf'cmd /k "{ExecDrive}: & cd {ExecDirectory} & {command}"')
The first part of this is changing the drive letter and directory to a place where the programs executable is stored. Given a user could run this from any location, I have to ensure that they are in the right directory before running the command in the f-string below (which is essentially targetApp.exe -Arg1 Val1 -Arg2 Val2 etc.).
Second, I need to capture the output so I can parse it for some messages. I think I can figure that part out on my own if I can get the first part working, but if you're a subprocess.run pro, any help would be appreciated!

I was actually able to use the cwd command to accomplish what I needed. The new code is below.
subprocess.run(command, cwd=rf"{ExecDrive}:{ExecDirectory}", shell=True)

There is
subprocess.check_output(args)
for capturing output.
Reference: https://docs.python.org/3/library/subprocess.html#subprocess.check_output

Shell command works in shell, but not when when fired from Python subprocess

I am trying to execute a shell command from a Python script.
I have tried the usual suspects, subprocess.call, Popen, os.system etc.
The command i am trying to execute is admittedly rather long (7k characters), since one of the parameters is a json string. From what I've read length should not be the issue here.
The command looks like this:
phantomjs /some/path/visualizer_interface.js -path /another/path/chart.svg -type chart_pie -id 0 -language de -data '{...}'
The visualizer interface is a script i wrote myself, that basically renders a requested chart in a Phantom JS context, grabs the svg and writes it to the specified path. When i execute the exact same command i get a flawless chart, but in Python the subprocess never return, and i don't get any form of feedback, not even on the subprocesses stdout.
with open('/home/max/stdout.txt', 'w') as out:
res = subprocess.Popen(command, shell=True, stdout=out)
res.wait()
I am able to execute other shell commands, so it's not a fundamental Python problem.
Any ideas very much appreciated.

Turns out i had a very slight error in the phantomjs script, that behaved differently depending on where it was executed.

Strange difference between calling command from windows cmd line and from python

I have a weird behaviour while managing a third party executable in my python code. Conceptually I have the following code in python:
import subprocess
p = subprocess.Popen([r'c:\path\to\programme.exe', '-d'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
print p.returncode, out, err
And the tool crashes showing in out its traceback, and returning an error code that means "unhandled exception". I have tried with a simple os.system(...) with the same results.
But, here comes the fun part, when I just paste the command in the windows shell, it works perfectly...
C:\> c:\path\to\programme.exe -d
The python interpreter is a 32bit 2.7.2 version.
So... what can be the difference between these two calls that leads to the crash? thanks in advance.
Extra info
I am not quite sure if this helps, but this external tool connects to a database and performs some operations. With some RDBMS it works when called from python code, but when it connects to an Oracle DB, it crashes. So the python code seems to be right, there is just a factor or difference that I don't know.

Well, you really don´t provide much info. I will make a guess based on my own experience dealing with situations like this.
Make sure you´re running the Python app as admin if the third party app require priveleges.
Check there is no problem with the working dir. Meaning, if the program opens some file or in any way it references to some relative path, you must change your working directory when executing from python. See code below for how to do this.
If the programm you're executing is a builtin windows shell app (dir, copy, etc...) consider using shell=True when creating the Popen object. See Popen constructor reference.
Python sets or modifies some environment variable needed/used by your third party application.
Code for changing working directory within the running Python app.
import os
os.chdir('/path_you_need/python/work_from')

You are supposed to use raw string.
p = subprocess.Popen([r'c:\path\to\programme.exe', '-d'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
else you can use \\ instead of using raw string like this:-
p = subprocess.Popen(['c:\\path\\to\\programme.exe', '-d'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)

python subprocess terminal mac osx

longtime lurker, first time poster.
I know there are quite a few examples throughout the interweb on using subprocess, however I am yet to find one that explains the steps I need to take to birth a new terminal window, and send it commands. There are plenty of posts that give workarounds to launch tools and scripts via a direct subprocess call, but I have not found any that actually answer the original questions of how to send a command properly to terminal.
In my case, I need to open a new terminal window, then send the path to a particular version of an application, and finally the path to the file I wish to open in that application.
I know how to use subprocess to call the applications needed directly (without opening a visible terminal), how to open a new terminal with subprocess, and how to call either the application path OR the file path (have not been able to get both to execute together using --args for open() or any other workaround I have found).
I have been unable to send terminal a command once I have opened it. The following is a simple version of opening a new instance of terminal and sending it ls, which does not work.
from subprocess import Popen, PIPE, STDOUT
p = Popen(['open', '-a', 'Terminal', '-n'], stdout=PIPE, stdin=PIPE, stderr=STDOUT)
output = p.communicate(input='ls')
print(output)
This is most likely a trivial issue and I am simply missing something, but I have been unable to find the information or an example that illustrates what I need and I am beginning to get frustrated with it, so I figured I would ask for help.
Any assistance is greatly appreciated! TIA

First, I doubt that command you are trying to run will run at all.
Did you try it in terminal first? open -an Terminal will give you
an error. It probably should be something like open -n
/Applications/Utilities/Terminal.app
Second, #korylprince is right: open itself will create new process
of Terminal and exit. So you are linking pipe with wrong process.
Third, at the moment of passing ls to the stdin that process
doesn't exist already (unless you will pass -W option to the open,
but it certainly will not help due to the 2 problem).
So I see only one opportunity to do this: via AppleScript. You can create an AppleScript string, something like next:
tell application "System Events"
tell process "Terminal"
keystroke "ls"
keystroke return
end tell
end tell
and then run this script via osascript -e '<your_script>' via Popen.
Yes it is quite tricky (I'd say it is a hack)
Yes there probably will be problems with passing multiline string to Popen and with determining correct Terminal window.
But it is possible.

#cody
My response to your answer was too long, therefore I am making an answer to respond:
You are correct, if you enter it the way you offered, it flags an error, and if you put the -n before Terminal it still flags an error. However, if you enter it the way I showed in the first example (-n after Terminal) "open" calls a new instance of app bundle Terminal, even if one is already open.
As for 2-3, that was kind of what my research was leading me to believe, but I was hoping I was wrong or missed something somewhere and someone here could clarify. Sadly, I wasn't mistaken…
I should probably expand on what I am trying to do, as maybe it will help generate a better way to accomplish it via Python.
I have created a tool that launches application files based on the movie, scene, and shot an artist is working on. For some applications, like Nuke and Houdini, opening from Terminal gives you a wealth of information that the artist would be blind to otherwise, so we would like to give the artist the option to launch the file they have chosen in a Terminal. That terminal has to be standalone, and a new instance of Terminal, because the app I have created must persist after the launch in order to open other shots in different applications without making the user routinely open the app.
Parsing the necessary info, building the commands, and launching a new Terminal that launches the desired application were all trivial. Doing the same with the desired file was trivial as well. The issue arises when a particular version of the app is chosen, and I have not been able to pass the newly birthed instance of Terminal with more than a single command (honestly the syntax of my OSX command may be the issue as well, will post further down).
I can get the following two commands to work without issue:
p = Popen(['open', '-a', 'Terminal', '-n', '--args', '/Applications/Nuke6.3v8/Nuke6.3v8.app/Nuke6.3v8'])
p = Popen(['open', '-a', 'Terminal', '-n', '--args', '/Path/to/Nuke/File.nk'])
I cannot get the following to work properly:
p = Popen(['open', '-a', 'Terminal', '-n', '--args', '/Applications/Nuke6.3v8/Nuke6.3v8.app/Nuke6.3v8', '/Path/to/Nuke/File.nk'])
From there my thought was possibly that I should launch the Terminal in the Popen, then pass the commands I needed. That did not work, and then I came here lol
Thanks again for any help! Just knowing that I cannot send to the commands I want to Terminal is saving me a ton of time that would have been spent on continuos frustrated research.

running a python script indefinitely (as a process, pretty much)

i have tests that i ran which can take up to 15m at a time. during these 15m, a log file is periodically written to. however, most of the content is useless.
in response to this i have a python script that parses out the useless text and displays the relevant data.
what i'm trying to achieve is similar to what tail -f log_file, constantly updating the terminal with the newest additions to a file. i was thinking that if a python script ran as a process, it could parse the log file whenever the tests write to it, then the python script can go to sleep until interrupted again once the log file is written to.
any ideas how one can achieve this?
i already have a script that does the parsing, i just don't know how to make it do it continually and efficiently.

You could just have the script filter standard input, and pipe tail -f through it. When you're waiting on stdin, your script will sleep, so it's plenty efficient.
Eg.
python long_running_script.py && tail -f log_file | python filter_logs.py
Your script can be something like
while true:
line = sys.stdin.readline()
if filter_line(line): print line

looks like you need something like "pytailer":
http://code.google.com/p/pytailer/
While I never used it myself, last example looks like what you want.

any ideas how one can achieve this?
This should be pretty easy to do. Most of what you want is already part of your OS.
python test.py | python log_parser.py
Be sure your tests write their log to stdout instead of some other file. This is often easy to do with small changes to the logging configuration.

Having implemented almost this exact tool, I had great success using the inotify capability in twisted