Lets say I have a python project, where using a command in my local terminal I can do something like:
python3 somePythonFile.py --someFlag True --someOtherFlag False -f https://www.somexmlfile/thefile.xml
If I wanted to run this in a python Lambda, would it be possible to run it the same way, without importing as a module.
For example, I have tried this:
import subprocess
from subprocess import Popen
cmd = "python3 somePythonFile.py --someFlag True --someOtherFlag False -f https://www.somexmlfile/thefile.xml"
returned_output = Popen(cmd)
# using decode() function to convert byte string to string
print('Converted result:', returned_output.decode("utf-8"))
However this gives me an error:
[ERROR] FileNotFoundError: [Errno 2] No such file or directory: 'python3 somePythonFile.py --someFlag True --someOtherFlag False -f https://www.somexmlfile/thefile.xml'
Traceback (most recent call last):
File "/var/task/app.py", line 32, in lambda_handler
returned_output = Popen(cmd)
File "/var/lang/lib/python3.8/subprocess.py", line 854, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/var/lang/lib/python3.8/subprocess.py", line 1702, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
The command for Popen() needs to have the command line parameters as separated in a list. https://docs.python.org/3/library/subprocess.html#popen-constructor
Change your line
returned_output = Popen(cmd)
to this
returned_output = Popen(cmd.split())
As far as the lambda goes. I would highly suggest you put this code in a module (might save you lots of money). The AWS Lambda environment will be terminated as soon as the handler function returns. You would need to code your Lambda function to wait for the subprocess to complete.
Thanks to Mark B's answer to this question Independent python subprocess from AWS Lambda function
Related
I'm trying to run and kill site.py python script every 5 minutes using webreset.py script.
webreset.py:
from psutil import Process, Popen
from time import sleep
pid = 0
while True:
if pid:
Process(pid).terminate()
process = Popen('python3 site.py')
pid = process.pid
sleep(300)
But when I run webreset.py on Ubuntu 20.04, I get the following Error:
Traceback (most recent call last):
File "webreset.py", line 7, in <module>
process = Popen('python3 site.py')
File "/usr/local/lib/python3.8/dist-packages/psutil/__init__.py", line 1316, in __init__
self.__subproc = subprocess.Popen(*args, **kwargs)
File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'python3 site.py'
I run the script using this command :
sudo python3 webreset.py
You need to separate the command and arguments.
process = Popen(['python3', 'site.py'])
Otherwise it is looking for a command with the filename "python3 site.py"
using this:
Popen(['python3', 'site.py'])
instead of:
process = Popen('python3 site.py')
fixed my problem.
This question already has answers here:
Subprocess.Call fails "The System Cannot Find the File Specified"
(2 answers)
Closed 25 days ago.
I want to connect and use the functions of adb in a python environment.
In the terminal, adb is connected well as shown below.
$ adb --version
Android Debug Bridge version 1.0.41
Version 31.0.3-7562133
Installed as /opt/homebrew/bin/adb
$ command -v adb
/opt/homebrew/bin/adb
To use this in the python environment, subprocess.check_output was used,
I wrote python code as below.
import subprocess
cmd = f"adb --version"
print(f"cmd:{cmd}")
res = subprocess.check_output(cmd).decode("utf-8")
print(f"res:{res}")
I got an error as below.
$ python test.py
cmd:adb --version
Traceback (most recent call last):
File "/Users/hhd/project/hhdpy/test.py", line 5, in <module>
res = subprocess.check_output(cmd).decode("utf-8")
File "/opt/homebrew/anaconda3/envs/hhdpy/lib/python3.9/subprocess.py", line 424, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/opt/homebrew/anaconda3/envs/hhdpy/lib/python3.9/subprocess.py", line 505, in run
with Popen(*popenargs, **kwargs) as process:
File "/opt/homebrew/anaconda3/envs/hhdpy/lib/python3.9/subprocess.py", line 951, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/opt/homebrew/anaconda3/envs/hhdpy/lib/python3.9/subprocess.py", line 1821, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'adb --version'
In my opinion, subprocess.check_output does not seem to be able to import the PATH environment variable,
How should I solve it?
As suggested by John, one possible solution would be to provide --version as an argument.
To do so, you have to split the cmd command into a list, for instance:
res = subprocess.check_output(["adb", "--version"]).decode("utf-8")
tested with Python3.
I tried to setup "airflow worker" to run after system start via rc.local(centos 7).
I have installed python and airflow as root. Path is /root/airflow and /root/anaconda3.
I added this to rc.local:
#!/bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
# It is highly advisable to create own systemd services or udev rules
# to run scripts during boot instead of using this file.
#
# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.
#
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# that this script will be executed during boot.
touch /var/lock/subsys/local
exec 2> /home/centos/rc.local.log # send stderr from rc.local to a log file
exec 1>&2 # send stdout to the same log file
set -x # tell sh to display commands before execution
export C_FORCE_ROOT="true"
/root/anaconda3/bin/python /root/anaconda3/bin/airflow worker
exit 0
When I try it to run manually it works (sh /etc/rc.local)
But when it runs after boot, it crashes with this error in log file.
It seems like it can't find path to airflow, but I have written it in full.
+ export C_FORCE_ROOT=true
+ C_FORCE_ROOT=true
+ /root/anaconda3/bin/python /root/anaconda3/bin/airflow worker
Traceback (most recent call last):
File "/root/anaconda3/bin/airflow", line 37, in <module>
args.func(args)
File "/root/anaconda3/lib/python3.7/site-packages/airflow/utils/cli.py", line 75, in wrapper
return f(*args, **kwargs)
File "/root/anaconda3/lib/python3.7/site-packages/airflow/bin/cli.py", line 1129, in worker
sp = _serve_logs(env, skip_serve_logs)
File "/root/anaconda3/lib/python3.7/site-packages/airflow/bin/cli.py", line 1065, in _serve_logs
sub_proc = subprocess.Popen(['airflow', 'serve_logs'], env=env, close_fds=True)
File "/root/anaconda3/lib/python3.7/subprocess.py", line 800, in __init__
restore_signals, start_new_session)
File "/root/anaconda3/lib/python3.7/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'airflow': 'airflow'
A place to start is to change this line...
/root/anaconda3/bin/python /root/anaconda3/bin/airflow worker
to
/root/anaconda3/bin/airflow worker
As you only need to invoke the airflow bin you need and pass it a single service. Bear in mind you can pass more arguments. But calling a version of Python doesn't feel necessary.
I have a bash script which helps establish a local SimpleHTTPServer.
python -m SimpleHTTPServer 8080
I have put this inside my project folder. While I am running the program by using:
subprocess.call('./setup.sh')
an error message comes out:
Traceback (most recent call last):
File "test.py", line 2, in <module>
subprocess.call('./setup.sh')
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 522, in call
return Popen(*popenargs, **kwargs).wait()
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 709, in __init__
errread, errwrite)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1326, in _execute_child
raise child_exception
OSError: [Errno 13] Permission denied
I retried this in terminal
localhost:Desktop XXXX$ sh setup.sh
Serving HTTP on 0.0.0.0 port 8080 ...
It is working fine.
I remember there are a few times where the terminal has popped up a window ask me about the permission for python about something related to firewall and I allowed it. Can you help me?
Run it exactly as you would on the shell, i.e., as sh ./setup.sh:
subprocess.call('sh ./setup.sh', shell=True)
That should do the trick. Most likely, your setup.sh is not set to executable or is missing the first #! line that marks its interpreter.
EDIT:
Make sure to set shell=True to execute it via the shell, if you pass it as a single string, or separate the parameters into a list, as you might with execve:
subprocess.call(['sh', './setup.sh'])
Give subprocess.Popen() a try, with cwd param:
subprocess.Popen(['sh', './setup.sh'], cwd='/dir/contains/setup.sh/')
I have OSX and am running the python script out of the Unix shell
I'm running a python code that should open an application. I've been testing with Firefox.app and have been getting
Traceback (most recent call last):
File "/Users/brennan/Desktop/Coding/Wilix/wilix.py", line 453, in <module>
subprocess.call(["open -a "+cp2])
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 493, in call
return Popen(*popenargs, **kwargs).wait()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1228, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
My code is:
subprocess.call(["open -a "+cp2])
where cp2 is user input. (Firefox in this case)
if I cd into the programs directory and then do
open -a Firefox
firefox opens fine.
if I change my code to
subprocess.call(["open -a Firefox"])
I still get the error message.
You're passing open -a Firefox as one argument, as if you ran this in the shell:
$ "open -a Firefox"
You need to split up the items:
subprocess.call(['open', '-a', 'Firefox'])
Try giving the full path of firefox app.
It's wrong to use subprocess.call without shell=True or providing command as a list. Please, take a look at first examples in the docs:
http://docs.python.org/2/library/subprocess.html
Full path to Firefox may be needed or may be not needed.