I have tried to create a program to run in my VM to make it so I can do a transfer of data from a directory to my azure blob storage account. Whenever I run the command outside of the program (On the command line) it works, however, if I run the program that contains a subprocess that runs the command, it does not work.
Here is what I send over the command line that works:
sudo ./azcopy cp "/directory/subdirectory" "https://myblob.blob.core.windows.net/container[SAS]" --recursive=true
This completes the data transfer.
When I put it into a program, I ran into many issues.
Current code:
import subprocess
import os
import sys
try:
key = ('SAS')
file_path = ('/directory/subdirectory')
full_link = ('"https://myblob.blob.core.windows.net/' + key + '"')
transfer = subprocess.check_output(['azcopy', 'cp', file_path,
full_link,
'--recursive=true'], stderr=subprocess.STDOUT)
print('Transfer Complete.')
# except subprocess.CalledProcessError as e:
# raise RuntimeError("command '{}' return with error (code {}): {}".format(e.cmd, e.returncode, e.output))
except EOFError as error:
#Output for EOF error, would be caused by missing SAS
print('Error with SAS')
except Exception as e:
#When an unexpected error has occured.
print(str(e) + 'Unknown error has occured')
exit(0)
Output:
Command '['azcopy', 'cp', '/directory/subdirectory', '"https://myblob.blob.core.windows.net/[SAS]"', '--recursive=true']'
returned non-zero exit status 1Unknown error has occured
If I re-add the except statement I have in the code that is currently commented out, I get this error:
Traceback (most recent call last):
File "data_transfer.py", line 11, in <module>
'--recursive=true'], stderr=subprocess.STDOUT)
File "/usr/lib/python3.5/subprocess.py", line 626, in check_output
**kwargs).stdout
File "/usr/lib/python3.5/subprocess.py", line 708, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['azcopy', 'cp', 'datadrive/peeled-images', '"https://myblob.blob.core.windows.net[SAS]"', '--recursive=true']' returned non-zero exit status 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "data_transfer.py", line 14, in <module>
raise RuntimeError("command '{}' return with error (code {}): {}".format(e.cmd, e.returncode, e.output))
RuntimeError: command '['azcopy', 'cp', '/directory/subdirectory', '"https://myblob.blob.core.windows.net/[SAS]"', '--recursive=true']' return with error (code 1): b'\nfailed to parse user input due to error: the inferred source/destination combination is currently not supported. Please post an issue on Github if support for this scenario is desired\n'
All help is much appreciated
The answer to this was to change the following line:
subprocess.check_output(['azcopy', 'cp', '/directory/subdirectory',
full_link, '--recursive=true'], stderr=subprocess.STDOUT)
The change needed was:
subprocess.call(['azcopy', 'cp', '/directory/subdirectory',
full_link, '--recursive=true'], stderr=subprocess.STDOUT)
This is because this was meant to run and execute a program, not necessarily provide a specific output.
Related
I have a python code as follows:
try:
print("Running code " + str(sub.id))
r = subprocess.call("node codes.js > outputs.txt", shell=True)
except:
print("Error running submission code id " + str(sub.id))
The code is running node command using subprocess.call. The node command is running codes.js file. Sometimes if there is error in code like if there is document. command then the code throws error.
With try and except it is not catching the error thrown when the node command fails.
The error thrown is as follows
There is document. line in the code so node cannot understand that line so it throws error.
/home/kofhearts/homework/codes.js:5
document.getElementById("outputalert").innerHTML = "Hacked";
^
ReferenceError: document is not defined
at solve (/home/kofhearts/homework/codes.js:5:3)
at Object.<anonymous> (/home/kofhearts/homework/codes.js:13:28)
at Module._compile (internal/modules/cjs/loader.js:1068:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1097:10)
at Module.load (internal/modules/cjs/loader.js:933:32)
at Function.Module._load (internal/modules/cjs/loader.js:774:14)
at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)
at internal/main/run_main_module.js:17:47
Traceback (most recent call last):
File "manage.py", line 22, in <module>
main()
File "manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "/home/kofhearts/.virtualenvs/myenv/lib/python3.7/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
utility.execute()
File "/home/kofhearts/.virtualenvs/myenv/lib/python3.7/site-packages/django/core/management/__init__.py", line 395, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/kofhearts/.virtualenvs/myenv/lib/python3.7/site-packages/django/core/management/base.py", line 330, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/kofhearts/.virtualenvs/myenv/lib/python3.7/site-packages/django/core/management/base.py", line 371, in execute
output = self.handle(*args, **options)
File "/home/kofhearts/homework/assignments/management/commands/police.py", line 73, in handle
if isCorrect(data.strip()[:-1], sub.question.outputs, sub.question, sub.code):
File "/home/kofhearts/homework/assignments/views.py", line 566, in isCorrect
givenans = [json.loads(e.strip()) for e in received.split('|')]
File "/home/kofhearts/homework/assignments/views.py",
How is it possible to catch the error when subprocess.call fails? Thanks for the help!
How is it possible to catch the error when subprocess.call fails?
The 'standard' way to do this is to use subprocess.run:
from subprocess import run, CalledProcessError
cmd = ["node", "code.js"]
try:
r = run(cmd, check=True, capture_output=True, encoding="utf8")
with open("outputs.txt", "w") as f:
f.write(r.stdout)
except CalledProcessError as e:
print("oh no!")
print(e.stderr)
Note that I have dropped the redirect and done it in python. You might be able to redirect with shell=True, but it's a whole security hole you don't need just for sending stdout to a file.
check=True ensures it will throw with non-zero return state.
capture_output=True is handy, because stderr and stdout are passed through to the exception, allowing you to retrieve them there. Thank to #OlvinRoght for pointing that out.
Lastly, it is possible to check manually:
r = run(cmd, capture_output=True, encoding="utf8")
if r.returncode:
print("Failed", r.stderr, r.stdout)
else:
print("Success", r.stdout)
I would generally avoid this pattern as
try is free for success (and we expect this to succeed)
catching exceptions is how we normally handle problems, so it's the Right Way (TM)
but YMMV.
I would like to get a full, descriptive error message from failed Python script executed with subprocess module.
I have a following script sub_script.py which fails and produces IndexError: pop from empty list error when executed on it's own:
# sub_script.py
empty_list = []
empty_list.pop()
I am calling sub_script.py from sub_test.py as follows:
# sub_test.py
import subprocess
import sys
print(str(subprocess.run([sys.executable, 'sub_script.py'],
check=True,
capture_output=True)))
However I am only getting subprocess.CalledProcessError error.
Traceback (most recent call last):
File "/Users/my_user/Desktop/my_dir/sub_test.py", line 4, in <module>
print(str(subprocess.run([sys.executable, 'sub_script.py'],
File "/usr/local/Cellar/python#3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/local/opt/python#3.9/bin/python3.9', 'sub_script.py']' returned non-zero exit status 1.
I would like to see a full description of the error returned from sub_script.py (IndexError: pop from empty list) when it's executed with a subprocess in sub_test.py.
Is it possible to get full error when script executes and fails within subprocess.run?
Keep the return value of the subprocess.run call instead of immediately converting it to a str, and do not have check=True.
# main.py
import subprocess
import sys
command = [sys.executable, 'task.py']
outcome = subprocess.run(command, check=False, capture_output=True)
print(f"returncode = {outcome.returncode}")
if outcome.returncode != 0:
# there was an error, we assume the traceback was printed to stderr
print("there was an error :\n")
print(outcome.stderr.decode("utf-8"))
# task.py
empty_list = []
empty_list.pop()
output :
returncode = 1
there was an error :
Traceback (most recent call last):
File "task.py", line 2, in <module>
empty_list.pop()
IndexError: pop from empty list
I use python and I want to create a database source controller like liquibase.
I find the python version of liquibase call pyquibase
but get subprocess.CalledProcessError
this is my simple code:
from pyquibase.pyquibase import Pyquibase
if __name__ == '__main__':
pyquibase = Pyquibase.sqlite('test.sqlite', 'db-changelog-1.xml')
pyquibase.update()
and I got these errors:
Traceback (most recent call last):
File "/home/ali/dev/project/python/DatabaseSourceContoller/DatabaseSourceContoller/main.py", line 5, in <module>
pyquibase.update()
File "/home/ali/dev/project/python/DatabaseSourceContoller/venv/lib/python3.5/site-packages/pyquibase/pyquibase.py", line 69, in update
output = self.liquibase.execute(self.change_log_file, "update")
File "/home/ali/dev/project/python/DatabaseSourceContoller/venv/lib/python3.5/site-packages/pyquibase/liquibase_executor.py", line 103, in execute
shell = True
File "/usr/lib/python3.5/subprocess.py", line 316, in check_output
**kwargs).stdout
File "/usr/lib/python3.5/subprocess.py", line 398, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'java -jar /home/ali/dev/project/python/DatabaseSourceContoller/venv/lib/python3.5/site-packages/pyquibase/liquibase/liquibase.jar --driver=org.sqlite.JDBC --classpath=/home/ali/dev/project/python/DatabaseSourceContoller/venv/lib/python3.5/site-packages/pyquibase/db-connectors/sqlite-jdbc-3.18.0.jar --changeLogFile=db-changelog-1.xml --url="jdbc:sqlite:test.sqlite" update' returned non-zero exit status 255
pyquibase forks a child process to execute the liquibase changelog update. And the subprocess.CalledProcessError means that the liquibase changelog update has failed.
Now, in order to find out why it failed, you can run the liquibase command manually to see the actual error messages:
java -jar /home/ali/dev/project/python/DatabaseSourceContoller/venv/lib/python3.5/site-packages/pyquibase/liquibase/liquibase.jar --driver=org.sqlite.JDBC --classpath=/home/ali/dev/project/python/DatabaseSourceContoller/venv/lib/python3.5/site-packages/pyquibase/db-connectors/sqlite-jdbc-3.18.0.jar --changeLogFile=db-changelog-1.xml --url="jdbc:sqlite:test.sqlite" update
pyquibase doesn't print the actual error messages for you yet. The next version upgrade should have that feature.
Sorry if this is a simple question and has been answered before, but I couldn't find it anywhere.
I'm trying to listen to UDP packets and if they are certain packets, run different batch scripts. I have this working correctly, but I have found that if the Popen command doesn't find the file it triggers an exception and the script stops running. Ideally, I want this to print a message and then continue listening for other packets and act upon them, just giving us a message for debugging. Here is the code I have used, how could I do this?
if rxdata == "Camera 1":
from subprocess import Popen
try:
p = Popen("Camera 1.bat", cwd=r"C:\xxx")
stdout, stderr = p.communicate()
except FileNotFoundError():
print('Camera 1.bat not found')
elif rxdata == "Camera 2":
from subprocess import Popen
p = Popen("Camera 2.bat", cwd=r"C:\xxx")
stdout, stderr = p.communicate()
In both examples, I receive the following and the script closes.
Traceback (most recent call last):
File "C:\UDP Listener.py", line 42, in <module>
p = Popen("Camera 1.bat", cwd=r"C:\xxx")
File "C:\Python34\lib\subprocess.py", line 858, in __init__
restore_signals, start_new_session)
File "C:\Python34\lib\subprocess.py", line 1111, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
Thanks in advance
Matt
You must not use the brackets behind the FileNotFoundError (don't call it, just "name" it). Test (with Python 2):
try:
b = a
except NameError():
print "NameError caught."
Execution:
Traceback (most recent call last):
File "test.py", line 2, in <module>
b = a
NameError: name 'a' is not defined
For instance, OSError is a type, whereas OSError() creates an instance of this type:
>>> type(OSError)
<type 'type'>
>>> type(OSError())
<type 'exceptions.OSError'>
Strangely, after re-installing python on my PC everything is now working correctly. Not sure what went wrong but when I run the code now and an exception is found then the code prints as expected.
Thanks for your help!
I'm attempting to get the PID of a process on Windows 7 using psutil, but I'm running into a permissions error. I've tried running the Command Prompt which is running the script as administrator, but this doesn't seem to have any effect. Both the error and the relevant code is below. The line that the error occurs on is when trying to access the process name using proc.name. Any suggestions on how I might fix this? Thank you much!
Error:
Traceback (most recent call last):
File "C:\Python33\lib\site-packages\psutil\_psmswindows.py", line 190, in wrapper
return fun(self, *args, **kwargs)
File "C:\Python33\lib\site-packages\psutil\_psmswindows.py", line 229, in get_process_exe
return _convert_raw_path(_psutil_mswindows.get_process_exe(self.pid))
PermissionError: [WinError 5] Access is denied
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "simple_address_retrieve.py", line 14, in <module>
if proc.name == PROCNAME:
File "C:\Python33\lib\site-packages\psutil\_common.py", line 48, in __get__
ret = self.func(instance)
File "C:\Python33\lib\site-packages\psutil\__init__.py", line 341, in name
name = self._platform_impl.get_process_name()
File "C:\Python33\lib\site-packages\psutil\_psmswindows.py", line 190, in wrapper
return fun(self, *args, **kwargs)
File "C:\Python33\lib\site-packages\psutil\_psmswindows.py", line 222, in get_process_name
return os.path.basename(self.get_process_exe())
File "C:\Python33\lib\site-packages\psutil\_psmswindows.py", line 194, in wrapper
raise AccessDenied(self.pid, self._process_name)
psutil._error.AccessDenied: (pid=128)
Code:
PROCNAME = "MyProcessName.exe"
for proc in psutil.process_iter():
if proc.name == PROCNAME:
print(proc)
get_process_list() is deprecated use psutil.process_iter() 0.6.0 .
Also in newest psutil this problem seems to be fixed.
You can also continue iterating over processes:
for proc in psutil.process_iter():
try:
if proc.name == PROCNAME:
print(proc)
except (PermissionError, AccessDenied):
print "Permission error or access denied on process" # can't display name or id here
From comments:
...and searching more, it appears that this is an issue that the author won't fix (too complex): http://groups.google.com/forum/#!topic/psutil/EbdkIGlb4ls. This answer looks like the best way to do this. There is no PermissionError though, so just catch AccessDenied
except psutil.AccessDenied: # windows
Example
def test_children_duplicates(self):
# find the process which has the highest number of children
table = collections.defaultdict(int)
for p in psutil.process_iter():
try:
table[p.ppid()] += 1
except psutil.Error:
pass
# this is the one, now let's make sure there are no duplicates
pid = sorted(table.items(), key=lambda x: x[1])[-1][0]
p = psutil.Process(pid)
try:
c = p.children(recursive=True)
except psutil.AccessDenied: # windows
pass
else:
self.assertEqual(len(c), len(set(c)))
Ref: https://www.programcreek.com/python/example/53869/psutil.process_iter
def find_process(regex):
"If 'regex' match on cmdline return number and list of processes with his pid, name, cmdline."
process_cmd_name = re.compile(regex)
ls = []
for proc in psutil.process_iter(attrs=['pid','name','cmdline']):
try:
if process_cmd_name.search(str(" ".join(proc.cmdline()))):
ls.append(proc.info)
except psutil.AccessDenied: # windows
pass
return (ls)
There is possible usage in list comprehensions in conjuction of psutil.AccessDenied ?