I am trying to run this command in the terminal using Python:
./Pascal --set=settings/1_settings.txt --runpathway=on
--genescoring=sum --pval=1_snp_values.txt.gz
I need to run this script 180 times using a different pval everytime. Hence, automating it via Python saves me a lot of time.
Currently I have a Python subprocess like this:
subprocess.call("./Pascal --set=settings/1_settings.txt
--runpathway=on --genescoring=sum --pval=1_snp_values.txt.gz")
Although, I am getting this error:
Traceback (most recent call last):
File "test_automation.py", line 4, in <module>
subprocess.call("./Pascal --set=settings/1_settings.txt --runpathway=on --genescoring=sum --pval=1_snp_values.txt.gz")
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 522, in call
return Popen(*popenargs, **kwargs).wait()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child
raise child_exception
The problem is when I execute the exact same command in terminal (outside of the Python code) it works fine. Am I using the syntax incorrectly?
Using os.system...
You could try using this to run your terminal command like so:
import os
os.system("./Pascal --set=settings/1_settings.txt --runpathway=on --genescoring=sum --pval=1_snp_values.txt.gz")
And if this works, it is simple to execute in a for-loop to get it to run 180 times:
import os
for _ in range(180):
os.system("./Pascal --set=settings/1_settings.txt --runpathway=on --genescoring=sum --pval=1_snp_values.txt.gz")
Additionally, if Pascal is in the same directory that you are running your script then the path of ./ is not necessary as by default, it will search the current working directory. So you can change it to just Pascal --set-settings...
Using subprocess.call...
Personally, I think using os.system is a cleaner solution, however you can use subprocess.call to do the same thing in one of two ways, either:
call Pascal and the other arguments separately from a list like:
import subprocess as s
s.call(['./Pascal', '--set=settings/1_settings.txt', '--runpathway=on', '--genescoring=sum', '--pval=1_snp_values.txt.gz'])
or just set shell to true and pass in the full line as a string like:
import subprocess as s
s.call('./Pascal --set=settings/1_settings.txt --runpathway=on --genescoring=sum --pval=1_snp_values.txt.gz', shell=True)
Hope this works for you!
Related
I try to call a shellscript via the subprocess module in Python 2.6.
import subprocess
shellFile = open("linksNetCdf.txt", "r")
for row in shellFile:
subprocess.call([str(row)])
My filenames have a length ranging between 400 and 430 characters.
When calling the script I get the error:
File "/usr/lib64/python2.6/subprocess.py", line 444, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib64/python2.6/subprocess.py", line 595, in __init__
errread, errwrite)
File "/usr/lib64/python2.6/subprocess.py", line 1106, in _execute_child
raise child_exception
OSError: [Errno 36] File name too long
An example of the lines within linksNetCdf.txt is
./ShellScript 'Title' 'Sometehing else' 'InfoInfo' 'MoreInformation' inputfiile outputfile.txt 3 2
Any ideas how to still run the script?
subprocess.call can take the command to run in two ways - either a single string like you'd type into a shell, or a list of the executable name followed by the arguments.
You want the first, but were using the second
import subprocess
shellFile = open("linksNetCdf.txt", "r")
for row in shellFile:
subprocess.call(row, shell=True)
By converting your row into a list containing a single string, you're saying something like "Run the command named echo these were supposed to be arguments with no arguments"
You need to tell subprocess to execute the line as full command including arguments, not just one program.
This is done by passing shell=True to call
import subprocess
cmd = "ls " + "/tmp/ " * 30
subprocess.call(cmd, shell=True)
Executing following command and its variations always results in an error, which I just cannot figure out:
command = "/bin/dd if=/dev/sda8 count=100 skip=$(expr 19868431049 / 512)"
print subprocess.check_output([command])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/subprocess.py", line 566, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
WHich file it is referring to ? other commands like ls,wc are running correctly though, the command is also running well on terminal but not python script.
Your command is a list with one element. Imagine if you tried to run this at the shell:
/bin/'dd if='/dev/'sda8 count=100 skip=$(expr 19868431049 '/' 512)'
That's effectively what you're doing. There's almost certainly no directory named dd if= in your bin directory, and there's even more almost certainly no dev directory under that with an sd8 count=100 skip=$(expr 19868431049 directory with a program named 512 in it.
What you want is a list where each argument is its own element:
command = ['/bin/dd', 'if=/dev/sda8', 'count=100', 'skip=$(expr 19868431049 / 512)']
print subprocess.check_output(command) # notice no []
But that brings us to your second problem: $(expr 19868431049 / 512) isn't going to be parsed by Python or by dd; that's bash syntax. You can, of course, just do the same thing in Python instead of in bash:
command = ['/bin/dd', 'if=/dev/sda8', 'count=100',
'skip={}'.format(19868431049 // 512)]
print subprocess.check_output(command)
Or, if you really want to use bash for no good reason, pass a string, rather than a list, and use shell=True:
command = "/bin/dd if=/dev/sda8 count=100 skip=$(expr 19868431049 / 512)"
print subprocess.check_output(command, shell=True) # still no []
Although that still isn't going to work portably, because the default shell is /bin/sh, which may not know how to handle bashisms like $(…) (and expr, although I think POSIX requires that expr exist as a separate process…). So:
command = "/bin/dd if=/dev/sda8 count=100 skip=$(expr 19868431049 / 512)"
print subprocess.check_output(command, shell=True, executable='/bin/bash')
This worked for me using subprocess.popen
command = "echo $JAVA_HOME"
proc = subprocess.Popen(command,stdout=subprocess.PIPE,shell=True)
I am writing a python script which checks various conditions and runs a powershell script accordingly to help me automate migration from windows XP to windows 7. The powershell script gives its own output giving the user updates as to what is happening. I would like to take the output of the powershell script and print it as output of the python script. I have looked around at some questions which seem to want to do the same thing but they don't seem to be working for me. Initially I tried using
import subprocess
subprocess.call(["C:\Users\gu2124\Desktop\helloworld.ps1"])
As was suggested here Run PowerShell function from Python script but I found out that this waits for the program to execute first and does not give output so I found out I need to use subprocess.Popen() as was suggusted here Use Popen to execute a Powershell script in Python, how can I get the Powershell script's output and update it to web page? so I tried this
import subprocess
subprocess.Popen(["C:\Users\gu2124\Desktop\helloworld.ps1"], stdout=sys.stdout)
and I get this error
Traceback (most recent call last):
File "C:\Users\gu2124\Desktop\pstest.py", line 5, in <module>
subprocess.Popen(["C:\Users\gu2124\Desktop\helloworld.py1"], stdout=sys.stdout)
File "C:\Python27\lib\subprocess.py", line 701, in __init__
errread, errwrite), to_close = self._get_handles(stdin, stdout, stderr)
File "C:\Python27\lib\subprocess.py", line 848, in _get_handles
c2pwrite = msvcrt.get_osfhandle(stdout.fileno())
File "<string>", line 523, in __getattr__
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\core\netref.py", line 150, in __getattr__
return syncreq(self, consts.HANDLE_GETATTR, name)
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\core\netref.py", line 71, in syncreq
return conn.sync_request(handler, oid, *args)
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\core\protocol.py", line 434, in sync_request
raise obj
AttributeError: DebugOutput instance has no attribute 'fileno'
I'm not completely sure what this means but from what I think I understand after reading this AttributeError: StringIO instance has no attribute 'fileno' is that it is because I am messing with the stdout incorrectly. I looked a around more and I found this Why won't my python subprocess code work? where the answers said to use stdout=subprocess.PIPE so I tried this
import subprocess
subprocess.Popen(["C:\Users\gu2124\Desktop\helloworld.ps1"], stdout=subprocess.PIPE)
which also does not give me output
Finally I saw this http://www.pythonforbeginners.com/os/subprocess-for-system-administrators and changed my code to this
import subprocess
p = subprocess.Popen(["powershell","C:\Users\gu2124\Desktop\helloworld.ps1"], stdout=subprocess.PIPE)
print p.communicate
I thought that it may because I am initially trying to run a powershell script from the command line so I have to open powershell first. When I type these commands directly into the command line it works the way it should but when I run it through the python script it gives this
<bound method Popen.communicate of <subprocess.Popen object at 0x00000000026E4A90>>
which is an improvement I guess but not the "Hello world" I was expecting.
I have no idea what I should try to do next to get this to work. Any help would be greatly appreciated
Also if the powershell script I am using is needed here it is
$strString = "Hello World"
write-host $strString
function ftest{
$test = "Test"
write-host $test
}
EDIT: I tried upgrading to python 3.3 like was suggested in the first answer but I still can't get it to work. I used the command p = subprocess.Popen(['powershell.exe', "C:\\Users\\gu2124\\Desktop\\helloworld.ps1"], stdout=sys.stdout) and am sure the file is there but am getting this error:
Traceback (most recent call last):
File "<pyshell#3>", line 1, in <module>
p = subprocess.Popen(['powershell.exe', "C:\\Users\\gu2124\\Desktop\\helloworld.ps1"], stdout=sys.stdout)
File "C:\Python27\lib\subprocess.py", line 701, in __init__
errread, errwrite), to_close = self._get_handles(stdin, stdout, stderr)
File "C:\Python27\lib\subprocess.py", line 848, in _get_handles
c2pwrite = msvcrt.get_osfhandle(stdout.fileno())
UnsupportedOperation: fileno
Make sure you can run powershell scripts (it is disabled by default). Likely you have already done this. http://technet.microsoft.com/en-us/library/ee176949.aspx
Set-ExecutionPolicy RemoteSigned
Run this python script on your powershell script helloworld.py:
# -*- coding: iso-8859-1 -*-
import subprocess, sys
p = subprocess.Popen(["powershell.exe",
"C:\\Users\\USER\\Desktop\\helloworld.ps1"],
stdout=sys.stdout)
p.communicate()
This code is based on python3.4 (or any 3.x series interpreter), though it should work on python2.x series as well.
C:\Users\MacEwin\Desktop>python helloworld.py
Hello World
I don't have Python 2.7 installed, but in Python 3.3 calling Popen with stdout set to sys.stdout worked just fine. Not before I had escaped the backslashes in the path, though.
>>> import subprocess
>>> import sys
>>> p = subprocess.Popen(['powershell.exe', 'C:\\Temp\\test.ps1'], stdout=sys.stdout)
>>> Hello World
_
In addition to the previous answers, I have some suggestions which makes your code more portable.
Instead of setting ExecutionPolicy globally to RemoteSigned (which imposes some security issues) you can use this to set it only for the PowerShell instance created by your Python script:
import subprocess, sys
p = subprocess.Popen('powershell.exe -ExecutionPolicy RemoteSigned -file "hello world.ps1"', stdout=sys.stdout)
p.communicate()
Note the quotes which allows your PowerShell-script's path/filename to contain spaces.
Furthermore, as shown in the above example, you can use a relative path to call your PowerShell script. The path is relative to your Python workspace directory.
This is how I get the output from Popen
p = subprocess.Popen(["powershell","C:\Users\gu2124\Desktop\helloworld.ps1"], stdout=subprocess.PIPE)
p_out, p_err = p.communicate()
print(p_out)
From the docs on Popen.communicate(). The function returns a tuple (stdout_data, stderr_data).
I need to execute a shell script using python. Output of shell program is a text file. No inputs to the script. Help me to resolve this.
def invokescript( shfile ):
s=subprocess.Popen(["./Script1.sh"],stderr=subprocess.PIPE,stdin=subprocess.PIPE);
return;
invokescript("Script1.sh");
On using above code., I receive the following error.
Traceback (most recent call last):
File "./test4.py", line 12, in <module>
invokescript("Script1.sh");
File "./test4.py", line 8, in invokescript
s=subprocess.Popen(["./Script1.sh"],stderr=subprocess.PIPE,stdin=subprocess.PIPE);
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 8] Exec format error
Thanks in advance...
Try this:
import shlex
def invokescript(shfile):
return subprocess.Popen(
shlex.split(shfile),
stderr=subprocess.PIPE,
stdin=subprocess.PIPE
)
invokescript("Script1.sh");
And add #!/usr/bin/env bash to your bash file of course.
I used os.system() to call shell script. This do what i expected. Make sure that you have imported os module in your python code.
invokescript( "Script1.sh" ) // Calling Function
function invokescript( shfile ): // Function Defenition
os.system("/root/Saranya/Script1.sh")
return;
following is also executable:
invokescript( "Script1.sh" ) // Calling Function
function invokescript( shfile ): // Function Defenition
os.system(shfile)
return;
Thanks for your immediate response guys.
I have some custom commands.
This works:
subprocess.Popen(['python'], stdout=subprocess.PIPE)
But if I have my own system commands like deactivate, I get that error
Traceback (most recent call last):
File "runner2.py", line 21, in <module>
main()
File "runner2.py", line 18, in main
subprocess.Popen(['deactivate',], stdout=subprocess.PIPE)
File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
errread, errwrite)
File "/usr/lib/python2.6/subprocess.py", line 1139, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Let alone I need to execute this under my sandbox virtualenv.
Try add an extra parameter shell=True to the Popen call.
Just a note. shell=True was likely the correct solution to the o.p., since they did not make the following mistake, but you can also get the "No such file or directory" error if you do not split up your executable from its arguments.
import subprocess as sp, shlex
sp.Popen(['echo 1']) # FAILS with "No such file or directory"
sp.Popen(['echo', '1']) # SUCCEEDS
sp.Popen(['echo 1'], shell=True) # SUCCEEDS, but extra overhead
sp.Popen(shlex.split('echo 1')) # SUCCEEDS, equivalent to #2
Without shell=True, Popen expects the executable to be the first element of args, which is why it fails, there is no "echo 1" executable. Adding shell=True invokes your system shell and passes the first element of args to the shell. i.e. for linux, Popen(['echo 1'], shell=True) is equivalent to Popen('/bin/sh', '-c', 'echo 1') which is more overhead than you may need. See Popen() documentation for cases when shell=True is actually useful.
You have to give the full path to your program deactivate and then it the subprocess module should be able to find it.
I'm spawning subprocesses like that:
SHUTDOWN_CMD = os.path.sep.join(["c:", "windows", "system32", "shutdown.exe"])
def abortShutdown():
os.spawnv(os.P_NOWAIT, SHUTDOWN_CMD,
[SHUTDOWN_CMD, '/A'])
time.sleep(3)
I'm not using subprocess since Python 2.5 does not support it. I had to use use the FULL path to have it working and I guess you also have to use the full path to your custom commands.