There is a git command that I am using
git log --format=%H 3c2232a5583711aa5f37d0f21014934f67913202
Here the long string at the end is the commit id. This command gives the list of previously commit ids of the branch. The output is like,
3c2232a5583711aa5f37d0f21014934f67913202
9i45e2a5583711aa5f37d0f21014934f679132de
I am trying to issue the same command in python and trying to store the output in a string as the following,
import subprocess
result = subprocess.run(
[
"cd",
"/Users/XYZ/Desktop/gitrepo",
"git",
"log",
"3c2232a5583711aa5f37d0f21014934f67913202",
],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
)
print(result.stdout.decode("utf-8"), type(result.stdout.decode("utf-8")))
But the output of the print is empty! I tried subprocess.run with ["-ls", "-l"] and it worked well. The git command works on command line but I am not able to capture it in a string. When I print the result alone,
CompletedProcess(args=['cd', '/Users/XYZ/Desktop/gitrepo', 'git', 'log', '3c2232a5583711aa5f37d0f21014934f67913202'], returncode=0, stdout=b'')
How can I save the git command's output in a string? I am issuing two commands in one line. Should I issue the commands separately? If I should then, how can I (a) navigate to git folder and (b) issue git command there?
Your code runs cd "/Users/XYZ/Desktop/gitrepo" "git" "log" "3c2232a5583711aa5f37d0f21014934f67913202" which is probably not what you intended.
The best way is not to interpret changing the working directory as a separate command but as part of the setup of the environment to run the git command. The subprocess module has the keyword argument cwd for that.
If cwd is not None, the function changes the working directory to cwd
before executing the child. In particular, the function looks for
executable (or for the first item in args) relative to cwd if the
executable path is a relative path.
This is only documented for the Popen constructor but the subprocess.run documentation has this paragraph:
The arguments shown above are merely the most common ones, described
below in Frequently Used Arguments (hence the use of keyword-only
notation in the abbreviated signature). The full function signature is
largely the same as that of the Popen constructor - apart from
timeout, input and check, all the arguments to this function are
passed through to that interface.
So you can rewrite your code like this:
import subprocess
result = subprocess.run(
[
"git",
"log",
"3c2232a5583711aa5f37d0f21014934f67913202",
],
cwd="/Users/XYZ/Desktop/gitrepo"
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
)
print(result.stdout.decode("utf-8"), type(result.stdout.decode("utf-8")))
Related
I have the following paths:
airflow_home = os.path.join("opt", "airflow")
airflow_docs = os.path.join(airflow_home, "docs")
And I want airflow_docs path to be used within a bash command. For that, I have used the following code:
subprocess.run([f"sphinx-apidoc -o ./ ../plugins"],
shell=True,
cwd=airflow_docs)
And I get an error FileNotFoundError.
However, this does work:
subprocess.run([f"sphinx-apidoc -o ./ ../{doc_module}"],
shell=True,
cwd="/opt/airflow/docs")
So it seems that a missing leading slash is causing the problem. I have searched in google about adding a leading slash to a path with no success. So, is it possible to use os.path package for subprocess.run, or do I have to use a hardcoded string?
If you want a slash, put a slash.
airflow_home = os.path.join("/opt", "airflow")
But of course, having Python glue together the strings is not really useful. Indeed, the result of os.path.join is simply a string, equivalent to a hard-coded string. So just write it out:
airflow_home = "/opt/airflow"
Or if you want to do this in Python, perhaps prefer pathlib:
airflow_home = pathlib.Path("/opt") / "airflow"
As an aside, your subprocess code is broken; you want to pass either a string, with shell=True, or a list of tokens, without shell=True. (Windows "helpfully" hides this error but it's still wrong.)
subprocess.run(
["sphinx-apidoc", "-o", "./", "../plugins"],
cwd=airflow_docs)
subprocess conveniently allows you to pass in a pathlib.Path object as the value of cwd, though this might not always have been the case, if you need to support older versions of Python.
You probably want to add check=True to have Python raise an error if the subprocess fails. Perhaps see also Running Bash commands in Python
I am trying to execute the following command programatically:
~$ obabel -:"ccco" -O /home/user/output.png
obabel is a chemistry library, and basically if the string in the "" is complete nonsense chemically, it won't be able to generate the PNG file, and if it is a legitimate chemical structure the PNG file will be generated. This works in the terminal.
However, if I call the same command with Python, PNG files are generated for complete nonsense input strings which don't generate a PNG when the command is executed in the terminal.
I'm using subprocess like this:
cmd = 'obabel -:"ccco" -O /home/user/output.png'
proc = sub.Popen([cmd], shell=True, stderr=sub.PIPE)
res = proc.communicate()
I have also tried this:
os.system(cmd)
And tried Python2 and Python3. This happens when running scripts from the terminal or iPython.
I have also tried using C++ and running the cmd like this:
std::string cmd = "obabel -:\"ccco\" -O /home/user/output.png";
system(cmd.c_str());
By default Popen expects a list of string arguments, but if you pass shell=True, you can supply the command as a simple string (it will be executed in a shell). Currently you are passing in a list with one string that contains the entirety of the command, instead you can use either of these:
proc = subprocess.Popen('obabel -:"ccco" -O output.png', shell=True, stderr=subprocess.PIPE)
proc = subprocess.Popen(['obabel', '-:ccco', '-O', 'output.png'], stderr=subprocess.PIPE)
Escaping the SMILES string with quotes seems to be done to protect it from the shell, and you don't need it when passing the input directly (otherwise the " characters will be a part of the string and cause invalid syntax).
For some reason, no matter how many variations I've tried, I can't seem to execute a bash script I've written. The command words 100% fine in Terminal, but when I try calling it with a subprocess, it returns nothing.
from os import listdir
import subprocess
computer_name = 'homedirectoryname'
moviefolder = '/Users/{}/Documents/Programming/Voicer/Movies'.format(computer_name)
string = 'The lion king'
for i in listdir(moviefolder):
title = i.split('.')
formatted_title = title[0].replace(' ', '\ ')
if string.lower() == title[0].lower():
command = 'vlc {}/{}.{}'.format(moviefolder, formatted_title, title[1])
subprocess.call(["/usr/local/bin",'-i','-c', command], stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True)
else:
continue
The bash executable file looks like this:
#/bin/bash
func() {
open -a /Applications/VLC.app/Contents/MacOS/VLC $1
}
Where have I gone wrong?
You should call open directly:
import os
import subprocess
computer_name = 'homedirectoryname'
moviefolder = '/Users/{}/Documents/Programming/Voicer/Movies'.format(computer_name)
string = 'The lion king'
for filename in os.listdir(moviefolder):
title = filename.split('.')
if string.lower() == title[0].lower():
subprocess.call(['open', '-a', '/Applications/VLC.app/Contents/MacOS/VLC', os.path.join(moviefolder, filename)])
Since you are using shell=True, the command must be a string:
On Unix with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
This means that the string must be formatted exactly as it would be
when typed at the shell prompt. This includes, for example, quoting or
backslash escaping filenames with spaces in them. If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself. (docs)
Like you even mentioned in a comment, you get /usr/local/bin: is a directory when you properly capture the error from the shell (and take out the erroneous shell=True; or correspondingly refactor the command line to be suitable for this usage, i.e. pass a string instead of a list).
Just to spell this out, you are attempting to run the command /usr/local/bin with some options; but of course, it's not a valid command; so this fails.
The actual script you seem to want to run will declare a function and then exit, which results in the function's definition being lost again, because the subprocess which ran the shell in which this function declaration was executed has now terminated and released all its resources back to the system.
Perhaps you should take more than just a few steps back and explain what you actually want to accomplish; but really, that should be a new, separate question.
Assuming you are actually trying to run vlc, and guessing some other things, too, perhaps you actually want
subprocess.call(['vlc','{}/{}.{}'.format(moviefolder, formatted_title, title[1]),
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
If your PATH is correct, you should not need to specify /usr/local/bin/ explicitly (and if your PATH is wrong, correct it in the code before, instead of hardcoding a directory for the executable you want to call).
/usr/local/bin is a directory. You can't run a directory as if it were a command.
Anyhow, there's no point to having /usr/local/bin anywhere in your command at all. Leave out the shell=True, and explicitly call vlc:
subprocess.call([
'vlc',
'{}/{}.{}'.format(moviefolder, formatted_title, title[1])
])
When shell=True is used in subprocess.call, if the command arguments is a sequence, then the first element of the sequence needs to be the command, and the rest are treated as argument(s) to the shell itself.
So, this should do:
subprocess.call(["/usr/local/bin/{}".format(command), '-i','-c'], shell=True, ...)
Otherwise, you can make the command a string.
Example:
In [20]: subprocess.call(["cat spamegg", "-i", "-c"], shell=True)
foobar
I have a a file structure like the following (Windows):
D:\
dir_1\
batch_1.bat
dir_1a\
batch_2.bat
dir_2\
main.py
For the sake of this question, batch_1.bat simply calls batch_2.bat, and looks like:
cd dir_1a
start batch_2.bat %*
Opening batch_1.bat from a command prompt indeed opens batch_2.bat as it's supposed to, and from there on, everything is golden.
Now I want my Python file, D:\dir_2\main.py, to spawn a new process which starts batch_1.bat, which in turn should start batch_2.bat. So I figured the following Python code should work:
import subprocess
subprocess.Popen(['cd "D:/dir_1"', "start batch_1.bat"], shell=True)
This results in "The system cannot find the path specified" being printed to my Python console. (No error is raised, of course.) This is due to the first command. I get the same result even if I cut it down to:
subprocess.Popen(['cd "D:/"'], shell=True)
I also tried starting the batch file directly, like so:
subprocess.Popen("start D:/dir_1/batch_1.bat", shell=True)
For reasons that I don't entirely get, this seems to just open a windows command prompt, in dir_2.
If I forego the start part of this command, then my Python process is going to end up waiting for batch_1 to finish, which I don't want. But it does get a little further:
subprocess.Popen("D:/dir_1/batch_1.bat", shell=True)
This results in batch_1.bat successfully executing... in dir_2, the directory of the Python script, rather than the directory of batch_1.bat, which results in it not being able to find dir_1a\ and hence, batch_2.bat is not executed at all.
I am left highly confused. What am I doing wrong, and what should I be doing instead?
Your question is answered here: Python specify popen working directory via argument
In a nutshell, just pass an optional cwd argument to Popen:
subprocess.Popen(["batch_1.bat"], shell=True, cwd=r'd:\<your path>\dir1')
I ran into an error while executing one of our devops scripts. The script uses the sh package (for executing common unix commands, pypi link). However, the commands that are executed are truncated in the messages printed by sh. How can I see the whole command that was executed?
example:
import sh
sh.ssh(host,
'rsync -av {src} {dst}'.format(src=src,
dst=dst),
_out=sys.stdout
)
Produces output like:
INFO:sh.command:<Command '/bin/ssh dbw#ny...(77 more)' call_args {'bg': False, 'timeo...(522 more)>: starting process
I'd like to see the full command executed, and all of the call_args.
sh.ssh returns an sh.RunningCommand object, which you can query to find the call args and the cmd:
import sh
a=sh.ssh(host,
'rsync -av {src} {dst}'.format(src=src,
dst=dst),
_out=sys.stdout
)
print(a.cmd)
print(a.call_args)
After peeking into the source code, it looks like this is controlled by the max_len parameter of the friendly_truncate function, so one option may be to edit the sh.py code directly and set a higher int value:
https://github.com/amoffat/sh/blob/master/sh.py#L424
https://github.com/amoffat/sh/blob/master/sh.py#L425
Or, possibly just remove points where that function is called.