Develop Reference

how to catch subprocess call exception in python?

I have a python code as follows:
try:
print("Running code " + str(sub.id))
r = subprocess.call("node codes.js > outputs.txt", shell=True)
except:
print("Error running submission code id " + str(sub.id))
The code is running node command using subprocess.call. The node command is running codes.js file. Sometimes if there is error in code like if there is document. command then the code throws error.
With try and except it is not catching the error thrown when the node command fails.
The error thrown is as follows
There is document. line in the code so node cannot understand that line so it throws error.
/home/kofhearts/homework/codes.js:5
document.getElementById("outputalert").innerHTML = "Hacked";
^
ReferenceError: document is not defined
at solve (/home/kofhearts/homework/codes.js:5:3)
at Object.<anonymous> (/home/kofhearts/homework/codes.js:13:28)
at Module._compile (internal/modules/cjs/loader.js:1068:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1097:10)
at Module.load (internal/modules/cjs/loader.js:933:32)
at Function.Module._load (internal/modules/cjs/loader.js:774:14)
at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)
at internal/main/run_main_module.js:17:47
Traceback (most recent call last):
File "manage.py", line 22, in <module>
main()
File "manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "/home/kofhearts/.virtualenvs/myenv/lib/python3.7/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
utility.execute()
File "/home/kofhearts/.virtualenvs/myenv/lib/python3.7/site-packages/django/core/management/__init__.py", line 395, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/kofhearts/.virtualenvs/myenv/lib/python3.7/site-packages/django/core/management/base.py", line 330, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/kofhearts/.virtualenvs/myenv/lib/python3.7/site-packages/django/core/management/base.py", line 371, in execute
output = self.handle(*args, **options)
File "/home/kofhearts/homework/assignments/management/commands/police.py", line 73, in handle
if isCorrect(data.strip()[:-1], sub.question.outputs, sub.question, sub.code):
File "/home/kofhearts/homework/assignments/views.py", line 566, in isCorrect
givenans = [json.loads(e.strip()) for e in received.split('|')]
File "/home/kofhearts/homework/assignments/views.py",
How is it possible to catch the error when subprocess.call fails? Thanks for the help!

How is it possible to catch the error when subprocess.call fails?
The 'standard' way to do this is to use subprocess.run:
from subprocess import run, CalledProcessError
cmd = ["node", "code.js"]
try:
r = run(cmd, check=True, capture_output=True, encoding="utf8")
with open("outputs.txt", "w") as f:
f.write(r.stdout)
except CalledProcessError as e:
print("oh no!")
print(e.stderr)
Note that I have dropped the redirect and done it in python. You might be able to redirect with shell=True, but it's a whole security hole you don't need just for sending stdout to a file.
check=True ensures it will throw with non-zero return state.
capture_output=True is handy, because stderr and stdout are passed through to the exception, allowing you to retrieve them there. Thank to #OlvinRoght for pointing that out.
Lastly, it is possible to check manually:
r = run(cmd, capture_output=True, encoding="utf8")
if r.returncode:
print("Failed", r.stderr, r.stdout)
else:
print("Success", r.stdout)
I would generally avoid this pattern as
try is free for success (and we expect this to succeed)
catching exceptions is how we normally handle problems, so it's the Right Way (TM)
but YMMV.

How to save the last output in variable when hit KeyboardInterrupt subprocess

I am completely new to the subprocess module. And I was trying to automate the deauthentication attack commands. When I run airodump-ng wlan0mon as you know it looks for the APs nearby and the connected clients to it.
Now when I try to run this command using lets suppose p = subprocess.run(["airmon-ng","wlan0mon"], capture_output=True) in Python as you know this command runs until the user hits Ctrl+C, so it should save the last output when user hits Ctrl+C in the variable but instead I get error which is this:
Traceback (most recent call last):
File "Deauth.py", line 9, in <module>
p3 = subprocess.run(["airodump-ng","wlan0"], capture_output=True)
File "/usr/lib/python3.8/subprocess.py", line 491, in run
stdout, stderr = process.communicate(input, timeout=timeout)
File "/usr/lib/python3.8/subprocess.py", line 1024, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "/usr/lib/python3.8/subprocess.py", line 1866, in _communicate
ready = selector.select(timeout)
File "/usr/lib/python3.8/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
KeyboardInterrupt
What can I try to resolve this?

Just use Python's error handling. Catch any KeyboardInnterrupts (within your subprocess function) using try and except statements like so:
def stuff(things):
try:
# do stuff
except KeyboardInterrupt:
return last_value

How to run a blob data transfer through .py

I have tried to create a program to run in my VM to make it so I can do a transfer of data from a directory to my azure blob storage account. Whenever I run the command outside of the program (On the command line) it works, however, if I run the program that contains a subprocess that runs the command, it does not work.
Here is what I send over the command line that works:
sudo ./azcopy cp "/directory/subdirectory" "https://myblob.blob.core.windows.net/container[SAS]" --recursive=true
This completes the data transfer.
When I put it into a program, I ran into many issues.
Current code:
import subprocess
import os
import sys
try:
key = ('SAS')
file_path = ('/directory/subdirectory')
full_link = ('"https://myblob.blob.core.windows.net/' + key + '"')
transfer = subprocess.check_output(['azcopy', 'cp', file_path,
full_link,
'--recursive=true'], stderr=subprocess.STDOUT)
print('Transfer Complete.')
# except subprocess.CalledProcessError as e:
# raise RuntimeError("command '{}' return with error (code {}): {}".format(e.cmd, e.returncode, e.output))
except EOFError as error:
#Output for EOF error, would be caused by missing SAS
print('Error with SAS')
except Exception as e:
#When an unexpected error has occured.
print(str(e) + 'Unknown error has occured')
exit(0)
Output:
Command '['azcopy', 'cp', '/directory/subdirectory', '"https://myblob.blob.core.windows.net/[SAS]"', '--recursive=true']'
returned non-zero exit status 1Unknown error has occured
If I re-add the except statement I have in the code that is currently commented out, I get this error:
Traceback (most recent call last):
File "data_transfer.py", line 11, in <module>
'--recursive=true'], stderr=subprocess.STDOUT)
File "/usr/lib/python3.5/subprocess.py", line 626, in check_output
**kwargs).stdout
File "/usr/lib/python3.5/subprocess.py", line 708, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['azcopy', 'cp', 'datadrive/peeled-images', '"https://myblob.blob.core.windows.net[SAS]"', '--recursive=true']' returned non-zero exit status 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "data_transfer.py", line 14, in <module>
raise RuntimeError("command '{}' return with error (code {}): {}".format(e.cmd, e.returncode, e.output))
RuntimeError: command '['azcopy', 'cp', '/directory/subdirectory', '"https://myblob.blob.core.windows.net/[SAS]"', '--recursive=true']' return with error (code 1): b'\nfailed to parse user input due to error: the inferred source/destination combination is currently not supported. Please post an issue on Github if support for this scenario is desired\n'
All help is much appreciated

The answer to this was to change the following line:
subprocess.check_output(['azcopy', 'cp', '/directory/subdirectory',
full_link, '--recursive=true'], stderr=subprocess.STDOUT)
The change needed was:
subprocess.call(['azcopy', 'cp', '/directory/subdirectory',
full_link, '--recursive=true'], stderr=subprocess.STDOUT)
This is because this was meant to run and execute a program, not necessarily provide a specific output.

Getting git fetch output to file through python

I am trying to save git fetch output to file through python, using:
subprocess.check_output(["git", "fetch", "origin", ">>", "C:/bitbucket_backup/backup.log", "2>&1"], cwd='C:/bitbucket_backup/loopx')
but I believe there is something missing in subprocess.check_output args because when adding >> C:/bitbucket_backup/backup.log 2>&1 I receive this error:
Traceback (most recent call last):
File "<pyshell#28>", line 1, in <module>
subprocess.check_output(["git", "fetch", "origin", ">>", "C://bitbucket_backup//backup.log", "2>&1"], cwd='C://bitbucket_backup//loopx')
File "C:\Users\fabio\AppData\Local\Programs\Python\Python36-32\lib\subprocess.py", line 336, in check_output
**kwargs).stdout
File "C:\Users\fabio\AppData\Local\Programs\Python\Python36-32\lib\subprocess.py", line 418, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['git', 'fetch', 'origin', '>>', 'C://bitbucket_backup//backup.log', '2>&1']' returned non-zero exit status 128.

Quickfix: enable shell features to handle redirection arguments:
subprocess.check_output(["git", "fetch", "origin", ">>", "C:/bitbucket_backup/backup.log", "2>&1"], cwd='C:/bitbucket_backup/loopx', shell=True)
But that's really dirty as python is able to do that really nicely:
output = subprocess.check_output(["git", "fetch", "origin"], stderr=subprocess.STDOUT, cwd='C:/bitbucket_backup/loopx')
with open("C:/bitbucket_backup/backup.log","ab") as f: # append to file
f.write(output)
That said, if you're to rewrite all git commands in python, maybe you should use a git python API like GitPython for instance.

Python error No such file or directory but the file is there

I'm trying to run a python script. The first steps of the scripts run ok, but at some point I have this message:
Traceback (most recent call last):
File "/home/chodar/Descargas/00.Comprimidos/REPET_linux-x64-2.2/bin/LaunchTRF.py", line 154, in <module>
iLaunchTRF.run()
File "/home/chodar/Descargas/00.Comprimidos/REPET_linux-x64-2.2/bin/LaunchTRF.py", line 143, in run
self._launchTRF()
File "/home/chodar/Descargas/00.Comprimidos/REPET_linux-x64-2.2/bin/LaunchTRF.py", line 101, in _launchTRF
process = subprocess.Popen(cmd.split(' '), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
2015-08-28 12:38:05 - DetectTEFeatures - ERROR - ERROR when launching 'LaunchTRF.py -i TEs.60bp.fa -o TEs.60bp.fa.SSR.set -m 15 -c -v 0 > launchTRF.log'
Traceback (most recent call last):
File "PASTEClassifier.py", line 199, in <module>
iLaunch.run()
File "PASTEClassifier.py", line 165, in run
iDF.run()
File "/home/chodar/Descargas/00.Comprimidos/REPET_linux-x64-2.2/commons/tools/DetectTEFeatures.py", line 185, in run
self._detectFeatures()
File "/home/chodar/Descargas/00.Comprimidos/REPET_linux-x64-2.2/commons/tools/DetectTEFeatures.py", line 204, in _detectFeatures
self._detectTRF()
File "/home/chodar/Descargas/00.Comprimidos/REPET_linux-x64-2.2/commons/tools/DetectTEFeatures.py", line 267, in _detectTRF
self._logAndRaise("ERROR when launching '%s'" % cmd)
File "/home/chodar/Descargas/00.Comprimidos/REPET_linux-x64-2.2/commons/tools/DetectTEFeatures.py", line 176, in _logAndRaise
raise Exception(errorMsg)
Exception: ERROR when launching 'LaunchTRF.py -i TEs.60bp.fa -o TEs.60bp.fa.SSR.set -m 15 -c -v 0 > launchTRF.log'
I read some other posts with this type of error and based on them, I looked if the LaunchTRF.py existed and it is there. In fact, when I run only LaunchTRF.py without any option, I can see the list of options and the help. I was thinking that the missing file is TEs.60bp.fa, but it is also there.
Here are the lines 98 to 103 from LaunchTRF.py in case that may help:
def _launchTRF(self):
cmd = "trf %s 2 3 5 80 10 20 %d -h -d" % (self.inFileName, self.maxPeriod)
self._log.debug("Running : %s" % cmd)
process = subprocess.Popen(cmd.split(' '), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output = process.communicate()
self._log.debug("Output:\n%s" % output[0])
Any help is welcome.

print(self.inFileName)
should be equal to
"C:\blah\something\test.txt"
if it isn't just prepend the name in your command
fullFile = "C:\blah\hooray\"+self.inFileName
cmd = "trf %s 2 3 5 80 10 20 %d -h -d" % (fullFile, self.maxPeriod)

This seems to be a issue with your working directory, maybe the parent script is changing it, or you are executing the parent script from another directory.
What you could do to fix that is to change de working directory by passing an additional parameter to popen:
subprocess.Popen('./LaunchTRF.py ...', cwd='/path/to/')
Hope it helps