I tried to setup "airflow worker" to run after system start via rc.local(centos 7).
I have installed python and airflow as root. Path is /root/airflow and /root/anaconda3.
I added this to rc.local:
#!/bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
# It is highly advisable to create own systemd services or udev rules
# to run scripts during boot instead of using this file.
#
# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.
#
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# that this script will be executed during boot.
touch /var/lock/subsys/local
exec 2> /home/centos/rc.local.log # send stderr from rc.local to a log file
exec 1>&2 # send stdout to the same log file
set -x # tell sh to display commands before execution
export C_FORCE_ROOT="true"
/root/anaconda3/bin/python /root/anaconda3/bin/airflow worker
exit 0
When I try it to run manually it works (sh /etc/rc.local)
But when it runs after boot, it crashes with this error in log file.
It seems like it can't find path to airflow, but I have written it in full.
+ export C_FORCE_ROOT=true
+ C_FORCE_ROOT=true
+ /root/anaconda3/bin/python /root/anaconda3/bin/airflow worker
Traceback (most recent call last):
File "/root/anaconda3/bin/airflow", line 37, in <module>
args.func(args)
File "/root/anaconda3/lib/python3.7/site-packages/airflow/utils/cli.py", line 75, in wrapper
return f(*args, **kwargs)
File "/root/anaconda3/lib/python3.7/site-packages/airflow/bin/cli.py", line 1129, in worker
sp = _serve_logs(env, skip_serve_logs)
File "/root/anaconda3/lib/python3.7/site-packages/airflow/bin/cli.py", line 1065, in _serve_logs
sub_proc = subprocess.Popen(['airflow', 'serve_logs'], env=env, close_fds=True)
File "/root/anaconda3/lib/python3.7/subprocess.py", line 800, in __init__
restore_signals, start_new_session)
File "/root/anaconda3/lib/python3.7/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'airflow': 'airflow'
A place to start is to change this line...
/root/anaconda3/bin/python /root/anaconda3/bin/airflow worker
to
/root/anaconda3/bin/airflow worker
As you only need to invoke the airflow bin you need and pass it a single service. Bear in mind you can pass more arguments. But calling a version of Python doesn't feel necessary.
Related
Lets say I have a python project, where using a command in my local terminal I can do something like:
python3 somePythonFile.py --someFlag True --someOtherFlag False -f https://www.somexmlfile/thefile.xml
If I wanted to run this in a python Lambda, would it be possible to run it the same way, without importing as a module.
For example, I have tried this:
import subprocess
from subprocess import Popen
cmd = "python3 somePythonFile.py --someFlag True --someOtherFlag False -f https://www.somexmlfile/thefile.xml"
returned_output = Popen(cmd)
# using decode() function to convert byte string to string
print('Converted result:', returned_output.decode("utf-8"))
However this gives me an error:
[ERROR] FileNotFoundError: [Errno 2] No such file or directory: 'python3 somePythonFile.py --someFlag True --someOtherFlag False -f https://www.somexmlfile/thefile.xml'
Traceback (most recent call last):
File "/var/task/app.py", line 32, in lambda_handler
returned_output = Popen(cmd)
File "/var/lang/lib/python3.8/subprocess.py", line 854, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/var/lang/lib/python3.8/subprocess.py", line 1702, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
The command for Popen() needs to have the command line parameters as separated in a list. https://docs.python.org/3/library/subprocess.html#popen-constructor
Change your line
returned_output = Popen(cmd)
to this
returned_output = Popen(cmd.split())
As far as the lambda goes. I would highly suggest you put this code in a module (might save you lots of money). The AWS Lambda environment will be terminated as soon as the handler function returns. You would need to code your Lambda function to wait for the subprocess to complete.
Thanks to Mark B's answer to this question Independent python subprocess from AWS Lambda function
I have a task where I have to SSH to different unix servers to run some command and get output of each server in one single file. The script is in python 2 as follows-
import subprocess
server_list = ['server1', 'server2', 'server3']
for srv in server_list:
subprocess.call(['ssh', srv])
# DO PROCESS
The problem is that when I run the script from command line on server myserver, it ssh to server1 and remains there until I manually run exit on command line. Then it goes to server2, again it remains there and I have to run exit. This process goes on until I return to myserver.
Unless I run the final exit, the file is not completed. So, I tried to give "exit" as command using subprocess module (to automate whole process), as follows
import subprocess
server_list = ['server1', 'server2', 'server3']
for srv in server_list:
subprocess.call(['ssh', srv])
# DO PROCESS
subprocess.call(['exit'])
When I run this, It ssh to 'server1', again it does not return to myserver. When I run exit, I get error-
Traceback (most recent call last):
File "test_return_ssh.py", line 9, in <module>
subprocess.call(['exit'])
File "/usr/lib64/python2.7/subprocess.py", line 524, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
And the execution stops.
What should I do to make this script work? I want to automate the whole process - ssh into servers, run the command and then after execution of whole script, exit and return to myserver.
I also have searched for this, but no link seems to provide the answer.
I am using Centos 7.0 and have installed Eclipse Kepler in the Pydev environment. I want to run a simple c shell script through Python using subprocess as follows:
import subprocess
subprocess.call(["./test1.csh"])
This c shell script executes in the terminal and also if I run command like "ls" or ""pwd then I get the correct output e.g.
subprocess.call(["ls"]) # give me the list of all files
subprocess.call(["pwd"]) # gives me the location of current directory.
But when I run subprocess.call(["./test1.csh"]), I get the following error:
Traceback (most recent call last):
File "/home/nishant/workspace/codec_implement/src/NTTool/raw2waveconvert.py", line 8, in <module>
subprocess.call(["./test1.csh"])
File "/usr/lib64/python2.7/subprocess.py", line 524, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib64/python2.7/subprocess.py", line 1308, in _execute_child
raise child_exception
OSError: [Errno 13] Permission denied
Where am I going wrong? Please suggest
Make sure that the file test1.csh is executable. As Lukas Graf commented, also check the shebang (#!...) in the first line.
To confirm that, before run it through Python, run it in the shell.
$ ls -l test1.csh
...
$ ./test1.csh
The current working directory will be different from when you run it in the terminal. Specify the full path of the shell script. Or change the working directory configuration in the PyDev.
UPDATE
Prepend the shell executable:
import subprocess
subprocess.call(["csh", "./test1.csh"])
I have a bash script which helps establish a local SimpleHTTPServer.
python -m SimpleHTTPServer 8080
I have put this inside my project folder. While I am running the program by using:
subprocess.call('./setup.sh')
an error message comes out:
Traceback (most recent call last):
File "test.py", line 2, in <module>
subprocess.call('./setup.sh')
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 522, in call
return Popen(*popenargs, **kwargs).wait()
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 709, in __init__
errread, errwrite)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1326, in _execute_child
raise child_exception
OSError: [Errno 13] Permission denied
I retried this in terminal
localhost:Desktop XXXX$ sh setup.sh
Serving HTTP on 0.0.0.0 port 8080 ...
It is working fine.
I remember there are a few times where the terminal has popped up a window ask me about the permission for python about something related to firewall and I allowed it. Can you help me?
Run it exactly as you would on the shell, i.e., as sh ./setup.sh:
subprocess.call('sh ./setup.sh', shell=True)
That should do the trick. Most likely, your setup.sh is not set to executable or is missing the first #! line that marks its interpreter.
EDIT:
Make sure to set shell=True to execute it via the shell, if you pass it as a single string, or separate the parameters into a list, as you might with execve:
subprocess.call(['sh', './setup.sh'])
Give subprocess.Popen() a try, with cwd param:
subprocess.Popen(['sh', './setup.sh'], cwd='/dir/contains/setup.sh/')
I have OSX and am running the python script out of the Unix shell
I'm running a python code that should open an application. I've been testing with Firefox.app and have been getting
Traceback (most recent call last):
File "/Users/brennan/Desktop/Coding/Wilix/wilix.py", line 453, in <module>
subprocess.call(["open -a "+cp2])
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 493, in call
return Popen(*popenargs, **kwargs).wait()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1228, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
My code is:
subprocess.call(["open -a "+cp2])
where cp2 is user input. (Firefox in this case)
if I cd into the programs directory and then do
open -a Firefox
firefox opens fine.
if I change my code to
subprocess.call(["open -a Firefox"])
I still get the error message.
You're passing open -a Firefox as one argument, as if you ran this in the shell:
$ "open -a Firefox"
You need to split up the items:
subprocess.call(['open', '-a', 'Firefox'])
Try giving the full path of firefox app.
It's wrong to use subprocess.call without shell=True or providing command as a list. Please, take a look at first examples in the docs:
http://docs.python.org/2/library/subprocess.html
Full path to Firefox may be needed or may be not needed.