I'm trying to run the following command using subprocess.run, but I think the hyphen in the command ir-keytable, is confusing the parser. What's the best way to solve this?
ir-keytable -t -s rc0
The error:
stdout:
stderr: Traceback (most recent call last):
File "<string>", line 1, in <module>
NameError: name 'ir' is not defined
My code:
import subprocess
import sys
result = subprocess.run(
[sys.executable, "-c", "ir-keytable('-t -s rc0')"], capture_output=True, text=True
)
print("stdout:", result.stdout)
print("stderr:", result.stderr)
Your Syntax is giving error for any command, you should just try this :
import subprocess
# If you just want to run command then this :
result=subprocess.run("<>")
# But if you want to get result of your command the this:
result=subprocess.check_output("<>")
print(result.decode("utf-8")) # It returns byte type data, so we are converting that into utf-8
Related
I would like to get a full, descriptive error message from failed Python script executed with subprocess module.
I have a following script sub_script.py which fails and produces IndexError: pop from empty list error when executed on it's own:
# sub_script.py
empty_list = []
empty_list.pop()
I am calling sub_script.py from sub_test.py as follows:
# sub_test.py
import subprocess
import sys
print(str(subprocess.run([sys.executable, 'sub_script.py'],
check=True,
capture_output=True)))
However I am only getting subprocess.CalledProcessError error.
Traceback (most recent call last):
File "/Users/my_user/Desktop/my_dir/sub_test.py", line 4, in <module>
print(str(subprocess.run([sys.executable, 'sub_script.py'],
File "/usr/local/Cellar/python#3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/local/opt/python#3.9/bin/python3.9', 'sub_script.py']' returned non-zero exit status 1.
I would like to see a full description of the error returned from sub_script.py (IndexError: pop from empty list) when it's executed with a subprocess in sub_test.py.
Is it possible to get full error when script executes and fails within subprocess.run?
Keep the return value of the subprocess.run call instead of immediately converting it to a str, and do not have check=True.
# main.py
import subprocess
import sys
command = [sys.executable, 'task.py']
outcome = subprocess.run(command, check=False, capture_output=True)
print(f"returncode = {outcome.returncode}")
if outcome.returncode != 0:
# there was an error, we assume the traceback was printed to stderr
print("there was an error :\n")
print(outcome.stderr.decode("utf-8"))
# task.py
empty_list = []
empty_list.pop()
output :
returncode = 1
there was an error :
Traceback (most recent call last):
File "task.py", line 2, in <module>
empty_list.pop()
IndexError: pop from empty list
Hi I have spent a few days reading at stackoverflow for an answer why I get the output listed below instead of seeing the loop in hello.py count forever. Still unclear for me and nothing works. How to start a few scripts in POpen subprocesses and get their output continously. Not when they have finished, because they will run forever. In case this is considered already answered, I'd be glad if someone can link to something functional with Python3.
I run this .py code in the hope to see hello.py execute:
import subprocess
import sys
command = [sys.executable, 'hello.py']
process_handle = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)
while process_handle.poll() is None:
for line in iter(process_handle.stdout.readline, ""):
output = line.decode('utf-8')
if output != '':
print('Last number is: ' + output)
Here is what I get. No running of hello.py:
Python 3.7.3 (default, Apr 3 2019, 05:39:12) [GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
hello.py contains:
i = 0
while True:
print(i)
i=i+1
sleep(1)
I tried the first variant from the second duplicate suggestion from live output from subprocess command
It does not compile/execute:
import subprocess
import sys
with open('test.log', 'w') as f: # replace 'w' with 'wb' for Python 3
process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
for c in iter(lambda: process.stdout.read(1), ''): # replace '' with b'' for Python 3
sys.stdout.write(c)
f.write(c)
Result:
Traceback (most recent call last): File "hello.py", line 11, in
Traceback (most recent call last): File "hello5.py", line
7, in
sys.stdout.flush() NameError: name 'sys' is not defined
sys.stdout.write(c) TypeError: write() argument must be str, not bytes
Now I tried the second variant from above, modified like this:
import subprocess
import sys
command = [sys.executable, 'hello.py']
process = subprocess.Popen(command, stdout=subprocess.PIPE)
for line in iter(process.stdout.readline, b''): # replace '' with b'' for Python 3
sys.stdout.write(line)
Result:
Traceback (most recent call last): File "hello.py", line 11, in
Traceback (most recent call last): File "hello5.py", line
8, in
sys.stdout.flush() NameError: name 'sys' is not defined
sys.stdout.write(line) TypeError: write() argument must be str, not bytes
After advice from ShadowRanger I added b'' for binary, sys was already imported however:
import subprocess
import sys
command = [sys.executable, 'hello.py']
process = subprocess.Popen(command, stdout=subprocess.PIPE)
for c in iter(lambda: process.stdout.read(1), b''): # replace '' with b'' for Python 3
sys.stdout.write(c)
Result, despite adding import sys, sys is not defined? Plus now it complains about binary and wants string.
Traceback (most recent call last): File "hello.py", line 11, in
sys.stdout.flush() NameError: name 'sys' is not defined Traceback (most recent call last): File "hello5.py", line 8, in
sys.stdout.write(c) TypeError: write() argument must be str, not bytes
OK, I went with ShadowRanger's info and converted the binary output to text again. hello.py is now printing text.
sys.stdout.write(c.decode('ASCII'))
BUT its coming all at once and thats not what I need. Every printed line in hello.py must show up in realtime and not just when the script finishes.
Got it working, thanks for all help. In order to get output in realtime instead of all of it at the end of script, flushing of stdout must be done in the subprocess script (hello.py in this case) Also, like ShadowRanger said, if b'' for binary is used there will be a complaint from stdout.write. That can be handled by decoding to text.
hello.py
import sys
from time import sleep
i = 0
while True:
print(i)
sys.stdout.flush()
i=i+1
sleep(0.1)
mainscript.py
import sys
import subprocess
command = [sys.executable, 'hello.py']
process = subprocess.Popen(command, stdout=subprocess.PIPE)
for c in iter(lambda: process.stdout.read(1), b''): # replace '' with b'' for Python 3
sys.stdout.write(c.decode('utf-8')) # decode b'' back to text
I have tried this code in pyhthon shell prompt using windows. but I am getting the error as follows
>>> python -m twobitreader hg19.2bit < example.bed
SyntaxError: invalid syntax
I have also tried with the code you have sent
import twobitreader
with open("fas.fa", "w") as output, open('example.bed') as bedfile:
twobit = twobitreader.TwoBitFile('hg19.2bit')
twobitreader.twobit_reader(twobit, input_stream=bedfile, write=output)
When I try to execute the above code I am getting error as
Traceback (most recent call last):
File "D:/genome/sample6.py", line 3, in <module>
with open("fas.fa", "w") as output, open('example.bed') as bedfile:
IOError: [Errno 2] No such file or directory: 'example.bed'`Filed:
I unable to trace the error exactly.
You are trying to execute a shell command in the Python interpreter. The interpreter is not a shell.
Inside the shell you can still execute the same code the command line code would. Both a 2bit file and the input stream is be required for twobit_reader() to do anything. The first line of the function reads:
if input_stream is None: return
The library takes the hg19.2bit file as a TwoBitFile object; the input_stream argument must be a file or other iterable in using the BED format, according to the documentation string for the command line tool:
Regions should be given in BED format on stdin
chrom start(0-based) end(0-based, not included)
To use a BED file of regions, do
python -m twobitreader example.2bit < example.bed
From a Python script, the example.bed input should be an open file passed in as input_stream:
import twobitreader
with open("fas.fa", "w") as output, open('example.bed') as bedfile:
twobit = twobitreader.TwoBitFile('hg19.2bit')
twobitreader.twobit_reader(twobit, input_stream=bedfile, write=output)
The project documentation provides a link for the BED format: http://genome.ucsc.edu/FAQ/FAQformat#format1
I am writing a python script which checks various conditions and runs a powershell script accordingly to help me automate migration from windows XP to windows 7. The powershell script gives its own output giving the user updates as to what is happening. I would like to take the output of the powershell script and print it as output of the python script. I have looked around at some questions which seem to want to do the same thing but they don't seem to be working for me. Initially I tried using
import subprocess
subprocess.call(["C:\Users\gu2124\Desktop\helloworld.ps1"])
As was suggested here Run PowerShell function from Python script but I found out that this waits for the program to execute first and does not give output so I found out I need to use subprocess.Popen() as was suggusted here Use Popen to execute a Powershell script in Python, how can I get the Powershell script's output and update it to web page? so I tried this
import subprocess
subprocess.Popen(["C:\Users\gu2124\Desktop\helloworld.ps1"], stdout=sys.stdout)
and I get this error
Traceback (most recent call last):
File "C:\Users\gu2124\Desktop\pstest.py", line 5, in <module>
subprocess.Popen(["C:\Users\gu2124\Desktop\helloworld.py1"], stdout=sys.stdout)
File "C:\Python27\lib\subprocess.py", line 701, in __init__
errread, errwrite), to_close = self._get_handles(stdin, stdout, stderr)
File "C:\Python27\lib\subprocess.py", line 848, in _get_handles
c2pwrite = msvcrt.get_osfhandle(stdout.fileno())
File "<string>", line 523, in __getattr__
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\core\netref.py", line 150, in __getattr__
return syncreq(self, consts.HANDLE_GETATTR, name)
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\core\netref.py", line 71, in syncreq
return conn.sync_request(handler, oid, *args)
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\core\protocol.py", line 434, in sync_request
raise obj
AttributeError: DebugOutput instance has no attribute 'fileno'
I'm not completely sure what this means but from what I think I understand after reading this AttributeError: StringIO instance has no attribute 'fileno' is that it is because I am messing with the stdout incorrectly. I looked a around more and I found this Why won't my python subprocess code work? where the answers said to use stdout=subprocess.PIPE so I tried this
import subprocess
subprocess.Popen(["C:\Users\gu2124\Desktop\helloworld.ps1"], stdout=subprocess.PIPE)
which also does not give me output
Finally I saw this http://www.pythonforbeginners.com/os/subprocess-for-system-administrators and changed my code to this
import subprocess
p = subprocess.Popen(["powershell","C:\Users\gu2124\Desktop\helloworld.ps1"], stdout=subprocess.PIPE)
print p.communicate
I thought that it may because I am initially trying to run a powershell script from the command line so I have to open powershell first. When I type these commands directly into the command line it works the way it should but when I run it through the python script it gives this
<bound method Popen.communicate of <subprocess.Popen object at 0x00000000026E4A90>>
which is an improvement I guess but not the "Hello world" I was expecting.
I have no idea what I should try to do next to get this to work. Any help would be greatly appreciated
Also if the powershell script I am using is needed here it is
$strString = "Hello World"
write-host $strString
function ftest{
$test = "Test"
write-host $test
}
EDIT: I tried upgrading to python 3.3 like was suggested in the first answer but I still can't get it to work. I used the command p = subprocess.Popen(['powershell.exe', "C:\\Users\\gu2124\\Desktop\\helloworld.ps1"], stdout=sys.stdout) and am sure the file is there but am getting this error:
Traceback (most recent call last):
File "<pyshell#3>", line 1, in <module>
p = subprocess.Popen(['powershell.exe', "C:\\Users\\gu2124\\Desktop\\helloworld.ps1"], stdout=sys.stdout)
File "C:\Python27\lib\subprocess.py", line 701, in __init__
errread, errwrite), to_close = self._get_handles(stdin, stdout, stderr)
File "C:\Python27\lib\subprocess.py", line 848, in _get_handles
c2pwrite = msvcrt.get_osfhandle(stdout.fileno())
UnsupportedOperation: fileno
Make sure you can run powershell scripts (it is disabled by default). Likely you have already done this. http://technet.microsoft.com/en-us/library/ee176949.aspx
Set-ExecutionPolicy RemoteSigned
Run this python script on your powershell script helloworld.py:
# -*- coding: iso-8859-1 -*-
import subprocess, sys
p = subprocess.Popen(["powershell.exe",
"C:\\Users\\USER\\Desktop\\helloworld.ps1"],
stdout=sys.stdout)
p.communicate()
This code is based on python3.4 (or any 3.x series interpreter), though it should work on python2.x series as well.
C:\Users\MacEwin\Desktop>python helloworld.py
Hello World
I don't have Python 2.7 installed, but in Python 3.3 calling Popen with stdout set to sys.stdout worked just fine. Not before I had escaped the backslashes in the path, though.
>>> import subprocess
>>> import sys
>>> p = subprocess.Popen(['powershell.exe', 'C:\\Temp\\test.ps1'], stdout=sys.stdout)
>>> Hello World
_
In addition to the previous answers, I have some suggestions which makes your code more portable.
Instead of setting ExecutionPolicy globally to RemoteSigned (which imposes some security issues) you can use this to set it only for the PowerShell instance created by your Python script:
import subprocess, sys
p = subprocess.Popen('powershell.exe -ExecutionPolicy RemoteSigned -file "hello world.ps1"', stdout=sys.stdout)
p.communicate()
Note the quotes which allows your PowerShell-script's path/filename to contain spaces.
Furthermore, as shown in the above example, you can use a relative path to call your PowerShell script. The path is relative to your Python workspace directory.
This is how I get the output from Popen
p = subprocess.Popen(["powershell","C:\Users\gu2124\Desktop\helloworld.ps1"], stdout=subprocess.PIPE)
p_out, p_err = p.communicate()
print(p_out)
From the docs on Popen.communicate(). The function returns a tuple (stdout_data, stderr_data).
I've got this piece of code that works fine on Linux but fails on Windows. Process is created fine, but I get an error and nothing is read from pipe:
p = subprocess.Popen(['python', '-u', self.file_to_run,
'-s', '-g', '-i', self.input_file],
universal_newlines=True,
stdout=subprocess.PIPE)
...
out = p.stdout.readline().rstrip()
Error I get is
Traceback (most recent call last):
File "bench.py", line 59, in <module>
multi.add_process()
File "bench.py", line 47, in add_process
stdout=subprocess.PIPE)
File "c:\python\v2.5.1-ast3\...\lib\subprocess.py", line 615, in __init__
self.stdout = os.fdopen(c2pread, 'rU', bufsize)
OSError: [Errno 22] Invalid argument
I actually create multiple such processes and based on their output calculate some values, but that is irrelevant. What I need to do is, run the script with certain arguments multiple times and parse the data piped from stdout of each process.
try using sys.executable instead of 'python' in your subprocess args. I think this is because Python is not in the PATH on Windows.
Also check the value of self.file_to_run and self.input_file which must be strings and not None or strange stuff, but this probably won't cause an OSError.