Develop Reference

Python os.system or subprocess calls for command line automation

I would like to be able to call some executables that take in parameters and then dump the output to a file. I've attempted to use both os.system and subprocess calls to no avail. Here is a sample of what I'd like python to execute for me...
c:\directory\executable_program.exe -f w:\directory\input_file.txt > Z\directory\output_file.txt
Notice the absolute paths as I will be traversing hundreds of various directories to act on files etc..
Many thanks ahead of time!
Some examples that I've tried:
subprocess.run(['c:\directory\executable_program.exe -f w:\directory\input_file.txt > Z\directory\output_file.txt']
subprocess.call(r'"c:\directory\executable_program.exe -f w:\directory\input_file.txt > Z\directory\output_file.txt"']
subprocess.call(r'"c:\directory\executable_program.exe" -f "w:\directory\input_file.txt > Z\directory\output_file.txt"']

Your attempts contain various amounts of quoting errors.
subprocess.run(r'c:\directory\executable_program.exe -f w:\directory\input_file.txt > Z\directory\output_file.txt', shell=True)
should work, where the r prefix protects the backslashes from being interpreted and removed by Python before the subprocess runs, and the absence of [...] around the value passes it verbatim to the shell (hence, shell=True).
On Windows you could get away with putting the command in square brackets even though it's not a list, and omitting shell=True in some circumstances.
If you wanted to avoid the shell, try
with open(r'Z\directory\output_file.txt', 'wb') as dest:
subprocess.run(
[r'c:\directory\executable_program.exe', '-f', r'w:\directory\input_file.txt'],
stdout=dest)
which also illustrates how to properly pass a list of strings in square brackets as the first argument to subprocess.run.

Python: how do I read (not run) a shell script, inserting arguments along the way

I have a simple shell script script.sh:
echo "ubuntu:$1" | sudo chpasswd
I need to open the script, read it, insert the argument, and save it as a string like so: 'echo "ubuntu:arg_passed_when_opening" | sudo chpasswd' using Python.
All the options suggested here actually execute the script, which is not what I want.
Any suggestions?

You would do this the same way that you read any text file, and we can use sys.argv to get the argument passed when running the python script.
Ex:
import sys
with open('script.sh', 'r') as sfile:
modified_file_contents = sfile.read().replace('$1', sys.argv[1])
With this method, modified_file_contents is a string containing the text of the file, but with the specified variable replaced with the argument passed to the python script when it was run.

Python script doesn't delete file from archive -- printed command via terminal works fine

I'm creating an archive in Python using this code:
#Creates archive using string like [proxy_16-08-15_08.57.07.tar]
proxyArchiveLabel = 'proxy_%s' % EXECUTION_START_TIME + '.tar'
log.info('Packaging %s ...' % proxyArchiveLabel)
#Removes .tar from label during creation
shutil.make_archive(proxyArchiveLabel.rsplit('.',1)[0], 'tar', verbose=True)
So this creates an archive fine in the local directory. The problem is, there's a specific directory in my archive I want to remove, due to it's size and lack of necessity for this task.
ExecWithLogging('tar -vf %s --delete ./roles/jobs/*' % proxyArchiveLabel)
# ------------
def ExecWithLogging(cmd):
print cmd
p = subprocess.Popen(cmd.split(' '), env=os.environ, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while(True):
log.info(p.stdout.readline().strip())
if(p.poll() is not None):
break
However, this seems to do basically nothing. The size remains the same. If I print cmd inside of the ExecWithLogging, and copy/past that command to a terminal in the working directory of the script, it works fine. Just to be sure, I also tried hard-coding the full path to where the archive is created as part of the tar -vf %s --delete command, but still nothing seemed to happen.
I do get this output in my INFO log: tar: Pattern matching characters used in file names, so I'm kind of thinking Popen is interpreting my command incorrectly somehow... (or rather, I'm more likely passing in something incorrectly).
Am I doing something wrong? What else can I try?

You may have to use the --wildcards option in the tar command, which enables pattern matching. This may well be what you are seeing in your log, be it somewhat cryptically.
Edit:
In answer to your question Why? I suspect that the shell is performing the wildcard expansion whilst the command proffered through Popen is not. The --wildcard option for tar, forces tar to perform the wildcard expansion.
For a more detailed explanation see here:
Tar and wildcards

python subprocess module can't parse filename with special characters "("

I have a Python program that reads files and then tars them into tar balls of a certain size.
One of my files not only has spaces in it but also contains parentheses. I have the following code:
cmd = "/bin/tar -cvf " + tmpname + " '" + filename + "'"
NOTE: Those are single quotes inside double quotes outside of the filename variable. It's a little difficult to see.
Where tmpname and filename are variables in a for-loop that are subject to change each iteration (irrelevant).
As you can see the filename I'm tarballing contains single quotes around the file name so that the shell (bash) interprets it literally as is and doesn't try to do variable substitution which "" will do or program execution which ` will do.
As far as I can see, the cmd variable contains the exact syntax for the shell to interpret the command as I want it to. However when I run the following subprocess command substituting the cmd variable:
cmdobj = call(cmd, shell=True)
I get the following output/error:
/bin/tar: 237-r Property Transport Request (PTR) 012314.pdf: Cannot stat: No such file or directory
/bin/tar: Exiting with failure status due to previous errors
unable to tar: 237-r Property Transport Request (PTR) 012314.pdf
I even print the command out to the console before running the subprocess command to see what it will look when running in the shell and it's:
cmd: /bin/tar -cvf tempname0.tar '237-r Property Transport Request (PTR) 012314.pdf'
When I run the above command in the shell as is it works just fine. Not really sure what's going on here. Help please!

Pass a list of args without shell=True and the full path to the file if running from a different directory:
from subprocess import check_call
check_call(["tar","-cvf",tmpname ,"Property Transport Request (PTR) 012314.pdf"])
Also use tar not 'bin/tar'. check_call will raise a CalledProcessError if the command returns a non-zero exit status.

The call method that is part of the subprocess module should have an array of strings passed.
On the command line you would call
tar -cvf "file folder with space/"
The following is equivalent in python
call(["tar", "-cvf", "file folder with space/"])
You are making this call in the shell
"tar -cvf 'file folder with space/'"
Which causes the shell to look for a program with the exact name as `tar -cvf 'file folder with space/'
This avoids string concatenation, which makes for cleaner code.

Using cat command in Python for printing

In the Linux kernel, I can send a file to the printer using the following command
cat file.txt > /dev/usb/lp0
From what I understand, this redirects the contents in file.txt into the printing location. I tried using the following command
>>os.system('cat file.txt > /dev/usb/lp0')
I thought this command would achieve the same thing, but it gave me a "Permission Denied" error. In the command line, I would run the following command prior to concatenating.
sudo chown root:lpadmin /dev/usb/lp0
Is there a better way to do this?

While there's no reason your code shouldn't work, this probably isn't the way you want to do this. If you just want to run shell commands, bash is much better than python. On the other hand, if you want to use Python, there are better ways to copy files than shell redirection.
The simplest way to copy one file to another is to use shutil:
shutil.copyfile('file.txt', '/dev/usb/lp0')
(Of course if you have permissions problems that prevent redirect from working, you'll have the same permissions problems with copying.)
You want a program that reads input from the keyboard, and when it gets a certain input, it prints a certain file. That's easy:
import shutil
while True:
line = raw_input() # or just input() if you're on Python 3.x
if line == 'certain input':
shutil.copyfile('file.txt', '/dev/usb/lp0')
Obviously a real program will be a bit more complex—it'll do different things with different commands, and maybe take arguments that tell it which file to print, and so on. If you want to go that way, the cmd module is a great help.

Remember, in UNIX - everything is a file. Even devices.
So, you can just use basic (or anything else, e.g. shutil.copyfile) files methods (http://docs.python.org/2/tutorial/inputoutput.html#reading-and-writing-files).
In your case code may (just a way) be like that:
# Read file.txt
with open('file.txt', 'r') as content_file:
content = content_file.read()
with open('/dev/usb/lp0', 'w') as target_device:
target_device.write(content)
P. S. Please, don't use system() call (or similar) to solve your issue.

under windows OS there is no cat command you should usetype instead of cat under windows
(**if you want to run cat command under windows please look at: https://stackoverflow.com/a/71998867/2723298 )
import os
os.system('type a.txt > copy.txt')
..or if your OS is linux and cat command didn't work anyway here are other methods to copy file..
with grep:
import os
os.system('grep "" a.txt > b.txt')
*' ' are important!
copy file with sed:
os.system('sed "" a.txt > sed.txt')
copy file with awk:
os.system('awk "{print $0}" a.txt > awk.txt')