Develop Reference

Subprocess in Python: File Name too long

I try to call a shellscript via the subprocess module in Python 2.6.
import subprocess
shellFile = open("linksNetCdf.txt", "r")
for row in shellFile:
subprocess.call([str(row)])
My filenames have a length ranging between 400 and 430 characters.
When calling the script I get the error:
File "/usr/lib64/python2.6/subprocess.py", line 444, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib64/python2.6/subprocess.py", line 595, in __init__
errread, errwrite)
File "/usr/lib64/python2.6/subprocess.py", line 1106, in _execute_child
raise child_exception
OSError: [Errno 36] File name too long
An example of the lines within linksNetCdf.txt is
./ShellScript 'Title' 'Sometehing else' 'InfoInfo' 'MoreInformation' inputfiile outputfile.txt 3 2
Any ideas how to still run the script?

subprocess.call can take the command to run in two ways - either a single string like you'd type into a shell, or a list of the executable name followed by the arguments.
You want the first, but were using the second
import subprocess
shellFile = open("linksNetCdf.txt", "r")
for row in shellFile:
subprocess.call(row, shell=True)
By converting your row into a list containing a single string, you're saying something like "Run the command named echo these were supposed to be arguments with no arguments"

You need to tell subprocess to execute the line as full command including arguments, not just one program.
This is done by passing shell=True to call
import subprocess
cmd = "ls " + "/tmp/ " * 30
subprocess.call(cmd, shell=True)

subprocess.check_output(): OSError file not found in Python

Executing following command and its variations always results in an error, which I just cannot figure out:
command = "/bin/dd if=/dev/sda8 count=100 skip=$(expr 19868431049 / 512)"
print subprocess.check_output([command])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/subprocess.py", line 566, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
WHich file it is referring to ? other commands like ls,wc are running correctly though, the command is also running well on terminal but not python script.

Your command is a list with one element. Imagine if you tried to run this at the shell:
/bin/'dd if='/dev/'sda8 count=100 skip=$(expr 19868431049 '/' 512)'
That's effectively what you're doing. There's almost certainly no directory named dd if= in your bin directory, and there's even more almost certainly no dev directory under that with an sd8 count=100 skip=$(expr 19868431049 directory with a program named 512 in it.
What you want is a list where each argument is its own element:
command = ['/bin/dd', 'if=/dev/sda8', 'count=100', 'skip=$(expr 19868431049 / 512)']
print subprocess.check_output(command) # notice no []
But that brings us to your second problem: $(expr 19868431049 / 512) isn't going to be parsed by Python or by dd; that's bash syntax. You can, of course, just do the same thing in Python instead of in bash:
command = ['/bin/dd', 'if=/dev/sda8', 'count=100',
'skip={}'.format(19868431049 // 512)]
print subprocess.check_output(command)
Or, if you really want to use bash for no good reason, pass a string, rather than a list, and use shell=True:
command = "/bin/dd if=/dev/sda8 count=100 skip=$(expr 19868431049 / 512)"
print subprocess.check_output(command, shell=True) # still no []
Although that still isn't going to work portably, because the default shell is /bin/sh, which may not know how to handle bashisms like $(…) (and expr, although I think POSIX requires that expr exist as a separate process…). So:
command = "/bin/dd if=/dev/sda8 count=100 skip=$(expr 19868431049 / 512)"
print subprocess.check_output(command, shell=True, executable='/bin/bash')

This worked for me using subprocess.popen
command = "echo $JAVA_HOME"
proc = subprocess.Popen(command,stdout=subprocess.PIPE,shell=True)

data stream python subprocess.check_output exe from another location

I would like to run an exe from this directory:/home/pi/pi_sensors-master/bin/Release/
This exe is then run by tying mono i2c.exe and it runs fine.
I would like to get this output in python which is in a completely different directory.
I know that I should use subprocess.check_output to take the output as a string.
I tried to implement this in python:
import subprocess
import os
cmd = "/home/pi/pi_sensors-master/bin/Release/"
os.chdir(cmd)
process=subprocess.check_output(['mono i2c.exe'])
print process
However, I received this error:
The output would usually be a data stream with a new number each time, is it possible to capture this output and store it as a constantly changing variable?
Any help would be greatly appreciated.

Your command syntax is incorrect, which is actually generating the exception. You want to call mono i2c.exe, so your command list should look like:
subprocess.check_output(['mono', 'i2c.exe']) # Notice the comma separation.
Try the following:
import subprocess
import os
executable = "/home/pi/pi_sensors-master/bin/Release/i2c.exe"
print subprocess.check_output(['mono', executable])
The sudo is not a problem as long as you give the full path to the file and you are sure that running the mono command as sudo works.
I can generate the same error by doing a ls -l:
>>> subprocess.check_output(['ls -l'])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/subprocess.py", line 537, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
However when you separate the command from the options:
>>> subprocess.check_output(['ls', '-l'])
# outputs my entire folder contents which are quite large.
I strongly advice you to use the subprocess.Popen -object to deal with external processes. Use Popen.communicate() to get the data from both stdout and stderr. This way you should not run into blocking problems.
import os
import subprocess
executable = "/home/pi/pi_sensors-master/bin/Release/i2c.exe"
proc = subprocess.Popen(['mono', executable])
try:
outs, errs = proc.communicate(timeout=15) # Times out after 15 seconds.
except TimeoutExpired:
proc.kill()
outs, errs = proc.communicate()
Or you can call the communicate in a loop if you want a 'data-stream' of sort, an answer from this question:
from subprocess import Popen, PIPE
executable = "/home/pi/pi_sensors-master/bin/Release/i2c.exe"
p = Popen(["mono", executable], stdout=PIPE, bufsize=1)
for line in iter(p.stdout.readline, b''):
print line,
p.communicate() # close p.stdout, wait for the subprocess to exit

Executing a vbs file with arguments created by python

I would like to convert dozens of excel sheets to csv files at once. I have a working .vbs file which makes the conversion, and I would like to execute this .vbs file on the different sheets with the help of a python code. I have the following 2 versions of the python code:
Version 1:
import os
import sys
import subprocess
FolderName=sys.argv[1]
FileList=os.listdir(FolderName)
NewList=[]
for i in FileList:
NewItem=i.split('.xls')
NewXls=FolderName+"\\"+NewItem[0]+".xlsx "
NewCsv=FolderName+"\\"+NewItem[0]+".csv"
NewCommand="C:\\Users\\user\\XlsToCsv.vbs "+sys.argv[2]+" "+NewXls+NewCsv
subprocess.call(NewCommand)
Version 2:
import os
import sys
import subprocess
def main(directory,extension,sheet):
for filename in os.listdir(directory):
if filename.endswith(extension):
path = os.path.join(directory, filename)
base = os.path.join(directory, filename[:len(filename)-len(extension)])
print base
new_xls = base + extension
new_csv = base + '.csv'
subprocess.call(['C:\\Users\\user\\XlsToCsv.vbs', sheet, new_xls, new_csv])
main(sys.argv[1],sys.argv[2],sys.argv[3])
It does not matter, which I try, I get the same error message:
Traceback (most recent call last):
File "C:/Users/user/Desktop/Work/XlsDir.py", line 16, in <module>
subprocess.call(NewCommand)
File "C:\Python27\lib\subprocess.py", line 524, in call
return Popen(*popenargs, **kwargs).wait()
File "C:\Python27\lib\subprocess.py", line 711, in __init__
errread, errwrite)
File "C:\Python27\lib\subprocess.py", line 948, in _execute_child
startupinfo)
WindowsError: [Error 193] %1 er ikke et gyldigt Win32-program
The last line of the error message means approximately, that it is not a valid Win32-program.
What I have tried so far:
If I run the .vbs file from command prompt with the right arguments (sheet, name of the .xls file and name of the .csv file) then it works fine.
If I print the commands that python generates and copy them into command prompt, they work fine.
I tried every combinations of '\' and '\' within the different paths, and nothing got any better.
I tried to execute the programs with replacing the sys.argv[i] arguments with specific arguments and then execute the .py file from command prompt. I get the same error message.
I hope some of you can help me. Thanks a lot!

To elaborate on Ansgar's remedy:
Starting a .vbs from the command line 'works', because the shell associates the extension .vbs with an application (e.g. cscript/wscript; see ftype, assoc, cscript //E, cescript //S).
subprocess.call() does not open a shell, so either specify the application (c|wscript.exe) or start the shell yourself:
import subprocess
#subprocess.call("notepad") # works
#subprocess.call("dir") # [Error 2] The system cannot find the file specified
# no shell, no intrinsics
#subprocess.call("19112944.vbs") # [Error 193] %1 is not a valid Win32 application
# no shell, can't associate .vbs with c|wscript.exe
subprocess.call("cscript 19112944.vbs") # works
subprocess.call("cmd /c 19112944.vbs") # works
# have shell, can associate .vbs with c|wscript.exe

Try running the script with cscript.exe:
subprocess.call(['cscript.exe', 'C:\\Users\\user\\XlsToCsv.vbs', sheet, new_xls, new_csv])

Answering the terminal questions from a GUI window

I want to run terminal commands within a python file. It is working fine and I can also get the terminal messages on a gui window using subprocess.Popen.
import subprocess
import wx
import os
def main():
p = subprocess.Popen(['ls'], stdout = subprocess.PIPE)
text = p.stdout.readlines()
text = "".join(text)
wx.MessageBox("file names:\n%s" % text, "info")
if __name__ == '__main__':
app = wx.PySimpleApp()
main()
But when I run a command for which terminal should ask answers of some questions, I am getting error?
Traceback (most recent call last):
File "to_make_new_project_folder.py", line 19, in <module> main()
File "to_make_new_project_folder.py", line 10, in main p = subprocess.Popen(['gr_modtool add -t general square_ff'], stdout = subprocess.PIPE)
File "/usr/lib/python2.7/subprocess.py", line 711, in init errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1308, in _execute_child raise child_exception
OSError: [Errno 2] No such file or directory
Does someone have idea how to answer the question from terminal using a gui window?

You should try passing in stdin=PIPE as well to popen

Based on your stack trace, the error you're receiving is OSError: No such file or directory, coming up from subprocess. It looks to me like Popen can't find the file that you're trying to execute, and is therefore failing.