I have a problem with my python script. I want to download files from my server to my NAS. My script is downloading every file, except for files containing single quotes/apastrophes and/or spaces. I already know where the problem is, but I can't fix it. The problem is with the shlex.split() command. It just deletes the single quotes. I also looked into paramiko, but it's kinda buggy with big files, so that won't work for me. I am also open for a completly different approach.
I was also wondering if it is possible to get a return for that command, so that I know that scp succesfully downloaded my file or if it failed.
import subprocess
import shlex
def download(download_path, remote_path):
command = "sshpass -p %s scp -o StrictHostKeyChecking=no -r %s#%s:%s %s" %\
(ssh_password, ssh_user, host, remote_path, download_path)
args = shlex.split(command)
# call scp
p = subprocess.Popen(args, stdout=subprocess.PIPE, bufsize=1)
for line in iter(p.stdout.readline, b''):
print line,
p.stdout.close()
p.wait()
P.S.: There already is a topic with a similar problem, which I can't get working. Passing a filename with an apostrophe into scp using python
You aren't getting an arbitrary string that needs to be split; you are basically starting out with the individual words and concatenating them yourself. Don't do that; just make command a list to begin with.
command = [
"sshpass", "-p", ssh_password,
"scp", "-o", "StrictHostKeyChecking=no", "-r",
"%s#%s:%s" % (ssh_user, host, remote_path),
download_path
]
p = subprocess.Popen(command, stdout=subprocess.PIPE, bufsize=1)
Related
I'm looking for a simple way to run a shell command in python 3, get its output in real-time and finally store all output to a variable.
I searched the web for a possible solution but couldn't find one.
I also found a similar question on this site, but no one provided a clear answer.
I end up with this code, but it's more of a workaround then a clear solution
def get_os_cmd(command):
proc_file = '/tmp/proc.tmp'
if os.path.isfile(proc_file):
os.remove(proc_file) # remove temporary file
proc = subprocess.Popen(
command,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
shell=True, encoding='utf-8')
with open(proc_file, 'a+') as f:
for line in iter(proc.stdout.readline, ''):
string = line.rstrip()
print(string)
f.write(f'{string}\n')
return proc.stdout, proc.returncode
doki.al
This program is good and will work good enough i think.
import os
command = ['dir', 'echo program working']
for i in command:
stream = os.popen(i) # Execute the command in command list
output = stream.read() # Read the output of the executed program
print(output) # Print the output
This program first executes a command in cmd and then reads it and displays it.
This program returns the variable output as a string.
If you want a list you can use
output = stream.readline()
This will include all the newlines.
I hope this solves your problem. If not feel free to get back in comment section.
I have a string, which is a framed command that should be executed by in command line
cmdToExecute = "TRAPTOOL -a STRING "ABC" -o STRING 'XYZ'"
I am considering the string to have the entire command that should be triggered from command prompt. If you take a closer look at the string cmdToExecute, you can see the option o with value XYZ enclosed in SINGLE QUOTE. There is a reason that this needs to be given in single quote orelse my tool TRAPTOOL will not be able to process the command.
I am using subprocess.Popen to execute the entire command. Before executing my command in a shell, I am printing the content
print "Cmd to be exectued: %r" % cmdToExecute
myProcess = subprocess.Popen(cmdToExecute, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=False)
(stdOut, stdErr) = myProcess.communicate()
The output of the above command is,
Cmd to be executed: TRAPTOOL -a STRING "ABC" -o \'XYZ\'.
You can see that the output shows a BACKWARD SLASH added automatically while printing. Actually, the \ is not there in the string, which I tested using a regex. But, when the script is run on my box, the TRAPTOOL truncates the part of the string XYZ on the receiving server. I manually copy pasted the print output and tried sending it, I saw the same error on the receiving server. However, when I removed the backward slash, it sent the trap without any truncation.
Can anyone say why this happens?
Is there anyway where we can see what command is actually executed in subprocess.Popen?
Is there any other way I can execute my command other that subprocess.Popen that might solve this problem?
Try using shlex to split your command string:
>>> import shlex
>>> argv = shlex.split("TRAPTOOL -a STRING \"ABC\" -o STRING 'XYZ'")
>>> myProcess = subprocess.Popen(argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=False)
>>> (stdOut, stdErr) = myProcess.communicate()
The first parameter to the Popen constructor can be an argument list for your shell command or a string, but an argument list might be easier to work with because of all the quotes involved. (See the Python subprocess documentation.)
If you want to see the commands being written, you could probably do something like:
>>> argv = shlex.split("bash -x -c 'TRAPTOOL -a STRING \"ABC\" -o STRING \'XYZ\''")
This makes bash echo the commands to the shell by means of the -x option.
You asked for the repr representation of the string, not the str representation. Basically, what would you have to type at the Python interactive interpreter to get the same output? That's what %r displays. Change that to %s to see the value as it's actually stored:
print "Cmd to be exectued: %s" % cmdToExecute
I've been trying to pass a command that works only with literal double quotes in the commandline around the "concat:file1|file2" argument for ffmpeg.
I cant however make this work from python with subprocess.Popen(). Anyone have an idea how one passes quotes into subprocess.Popen?
Here is the code:
command = "ffmpeg -i "concat:1.ts|2.ts" -vcodec copy -acodec copy temp.mp4"
output,error = subprocess.Popen(command, universal_newlines=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE).communicate()
When I do this, ffmpeg won't take it any other way other than quotes around the concat segement. Is there a way to successfully pass this line to subprocess.Popen command?
I'd suggest using the list form of invocation rather than the quoted string version:
command = ["ffmpeg", "-i", "concat:1.ts|2.ts", "-vcodec", "copy",
"-acodec", "copy", "temp.mp4"]
output,error = subprocess.Popen(
command, universal_newlines=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
This more accurately represents the exact set of parameters that are going to be passed to the end process and eliminates the need to mess around with shell quoting.
That said, if you absolutely want to use the plain string version, just use different quotes (and shell=True):
command = 'ffmpeg -i "concat:1.ts|2.ts" -vcodec copy -acodec copy temp.mp4'
output,error = subprocess.Popen(
command, universal_newlines=True, shell=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
Either use single quotes 'around the "whole pattern"' to automatically escape the doubles or explicitly "escape the \"double quotes\"". Your problem has nothing to do with Popen as such.
Just for the record, I had a problem particularly with a list-based command passed to Popen that would not preserve proper double quotes around a glob pattern (i.e. what was suggested in the accepted answer) under Windows. Joining the list into a string with ' '.join(cmd) before passing it to Popen solved the problem.
This works with python 2.7.3 The command to pipe stderr to stdout has changed since older versions of python:
Put this in a file called test.py:
#!/usr/bin/python
import subprocess
command = 'php -r "echo gethostname();"'
p = subprocess.Popen(command, universal_newlines=True, shell=True,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
text = p.stdout.read()
retcode = p.wait()
print text
Invoke it:
python test.py
It prints my hostname, which is apollo:
apollo
Read up on the manual for subprocess: http://docs.python.org/2/library/subprocess.html
I have been working with a similar issue, with running a relatively complex
command over ssh. It also had multiple double quotes and single quotes. Because
I was piping the command through python, ssh, powershell etc.
If you can instead just convert the command into a shell script, and run the
shell script through subprocess.call/Popen/run, these issues will go away.
So depending on whether you are on windows or on linux or mac, put the
following in a shell script either (script.sh or script.bat)
ffmpeg -i "concat:1.ts|2.ts" -vcodec copy -acodec copy temp.mp4
Then you can run
import subprocess; subprocess.call(`./script.sh`; shell=True)
Without having to worry about single quotes, etc.
This line of code in your question isn't valid Python syntax:
command = "ffmpeg -i "concat:1.ts|2.ts" -vcodec copy -acodec copy temp.mp4"
If you had a Python file with just this line in it, you would get a syntax error. A string literal surrounded with double quotes can't have double quotes in them unless they are escaped with a backslash. So you could fix that line by replacing it with:
command = "ffmpeg -i \"concat:1.ts|2.ts\" -vcodec copy -acodec copy temp.mp4"
Another way to fix this line is to use single quotes for the string literal in Python, that way Python is not confused when the string itself contains a double quote:
command = 'ffmpeg -i "concat:1.ts|2.ts" -vcodec copy -acodec copy temp.mp4'
Once you have fixed the syntax error, you can then tackle the issue with using subprocess, as explained in this answer. I also wrote this answer to explain a helpful mental model for subprocess in general.
Also struggling with a string argument containing spaces and not wanting to use the shell=True.
The solution was to use double quotes for the inside strings.
args = ['salt', '-G', 'environment:DEV', 'grains.setvals', '{"man_version": "man-dev-2.3"}']
try:
p = subprocess.Popen(args, stdin=subprocess.PIPE
, stdout=subprocess.PIPE
, stderr=subprocess.PIPE
)
(stdin,stderr) = p.communicate()
except (subprocess.CalledProcessError, OSError ) as err:
exit(1)
if p.returncode != 0:
print("Failure in returncode of command:")
Anybody suffering from this pain. It also works with params enclosed with quotation marks.
params = ["ls", "-la"]
subprocess.check_output(" ".join(params), shell=True)
I'm running Python 3.3 on Mac OS 10.6.8. I am writing a script that runs several subprocesses, and I want to capture the output of each one and record it in a file. I'm having trouble with this.
I first tried the following:
import subprocess
logFile = open("log.txt", 'w')
proc = subprocess.Popen(args, stdout=logFile, stderr=logFile)
proc.wait()
This produced an empty log.txt. After poking around on the internet for a bit, I tried this instead
import subprocess
proc = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output, err = proc.communicate()
logFile = open("log.txt", 'w')
logFile.write(output)
This, too, produced an empty log.txt. So instead of writing to the file, I tried to just print the output to the command line:
output, err = proc.communicate()
print(output)
print(err)
That produced this:
b''
b''
The process I'm trying to run is fastq_quality_trimmer. It takes an input file, filters it, and saves the result to a new file. It only writes a few lines to stdout, like so
Minimum Quality Threshold: 20
Minimum Length: 20
Input: 750000 reads.
Output: 750000 reads.
discarded 0 (0%) too-short reads.
If I run it from the command line and redirect the output like this
fastq_quality_trimmer -Q 33 -v -t 50 -l 20 -i in.fq -o in_trimmed.fq > log.txt
the output is successfully written to log.txt.
I thought perhaps that fastq_quality_trimmer was somehow failing to run when I called it with Popen, but my script produces a filtered file that is identical to the one produced when I run fastq_quality_trimmer from the command line. So it's working; I just can't capture the output. To make matters more confusing, I can successfully capture the output of other processes (echo, other Python scripts) using code that is essentially identical to what I've posted.
Any thoughts? Am I missing something blindingly obvious?
You forgot a comma:
["fastq_quality_trimmer", "-Q", "33" "-v", "-t", "50", "-l", "20", "-i", leftInitial, "-o", leftTrimmed]
add it between "33" and "-v".
You are essentially passing in the arguments -Q 33-v instead of -Q 33 -v.
Python will concatenate two adjacent strings if there is only whitespace between them:
>>> "33", "-v"
('33', '-v')
>>> "33" "-v"
'33-v'
Since -v is the verbose switch that is required to make fastq_quality_trimmer produce output at all, it'll remain silent with it missing.
Whenever you encounter problems with calling a subprocess, triple check the command line created. Pre-pending args with ['echo'] can help in that:
proc = subprocess.Popen(['echo'] + args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output, err = proc.communicate()
print(output)
Am using subprocess on Windows and Python 2.6 as follows. I am trying to parse a text file using a legacy parser application (assume parser.py) as follows:
import subprocess
k = subprocess.Popen(['python', 'parser.py', '-f C:\Report1\2011-03-14.txt'],
shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
print k.communicate()
The issue here is with the way filename gets passed to the legacy application where I cannot change the code but only can access it using Python.
It generates with the following error:
IOError: [Errno 22] invalid mode (\'r\') or filename: C:\\Report1\\2011-03-14.txt
When I copy the modified filename(with double forward slashes) from the traceback to check the existence, the system is not able to find it.
Question: How can I pass the path as argument so that it gets treated without getting changed to double slashes so that the system can read the file?
NOTE: os.sep also does not resolve the issue.
EDIT: Executing using os.system works perfectly, but the issue there is to grab the output for later use. Am currently using os.sytem in a module(run_parser.py) and then using subprocess in another module(get_parse_status.py) that Popens run_parser.py to grab the output. Would appreciate anything that is better than this.
Thanks for the time.
Change your parameter list to encode the path as a raw string:
k = subprocess.Popen(['python', 'parser.py', '-f', r'C:\Report1\2011-03-14.txt'],
shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
a simple program that reads a file and reports the length:
import sys
import os
userinput = sys.argv[1]
data = open(userinput, 'rb').read()
datalength = len(data)
fname = os.path.basename(userinput)
print "%s datasize = %s" % (fname, datalength)
Then to call it through the interpreter:
>>> k = subprocess.Popen(['python', 'test2.py', 'w:\bin\test2.py'], shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
>>> k.communicate()
5: ('Traceback (most recent call last):\r\n File "w:\\bin\\test2.py", line 4, in <module>
data = open(userinput, \'rb\').read()
IOError: [Errno 22] invalid mode (\'rb\') or filename: 'w:\\x08in\\test2.py', None)
>>> k = subprocess.Popen(['python', r'w:\bin\test2.py', r'w:\bin\test2.py'], shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
>>> k.communicate()
6: ('test2.py datasize = 194\n', None)
"C:\Report1\2011-03-14.txt" isn't the same as the path C:\Report1\2011-03-14.txt. It's actually some bytestring, 'C:\\Report1\x811-03-14.txt'. Strangely enough it doesn't sound like this is your issue, but it might be related. r"C:\Report1\2011-03-14.txt" fixes this.
But be aware that double backslashes in the printed representation doesn't necessarily mean that there are actually two backslashes. '\\' is a Python string of length 1.
"C:\Report1\2011-03-14.txt" isn't the same as the path C:\Report1\2011-03-14.txt. It's actually some bytestring, 'C:\Report1\x811-03-14.txt'. Strangely enough it doesn't sound like this is your issue, but it might be related. r"C:\Report1\2011-03-14.txt" fixes this.
But be aware that double backslashes in the printed representation doesn't necessarily mean that there are actually two backslashes. '\' is a Python string of length 1.
Have you tried:
from subprocess import Popen, PIPE
k = Popen(r'python parser.py -f "C:\Report1\2011-03-14.txt"',
shell=True,
stdout=PIPE,
stderr=STDOUT)
print k.communicate()
I find that often when passing args on the command line via Popen, enclosing the parameters in double-quotes is the only reliable way to get it to behave. I also don't always trust the list method of calling Popen and usually form the command myself. Notice also the raw indicator (r'').