I want to kill a python script that runs on my system from another python script.
I followed this answer and tweaked the code a bit, but got an error:
Traceback (most recent call last): File "/home/pi/base.py", line 13, in <module>
check_call(["pkill", "-9", "-f", script])
File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['pkill', '-9', '-f', '/home/pi/MotionDetector.py']' returned non-zero exit status 1
Code:
from subprocess import check_call
import sys
import time
script = '/home/pi/MotionDetector.py'
check_call(["pkill", "-9", "-f", script])
This means the pkill call has failed. Two possible reasons that come to mind:
it did not actually match any process. pkill would in this case not generate any output and return 1. You can verify this by trying to run pgrep instead of pkill and see what did it return on stdout (should be one or more lines with a PID in case of a match) and/or whether it also returned a non-zero status.
it did match, but user under which the pkill has been executed cannot kill the process(s) matched. In that case pkill should generate output on stderr similar to: pkill: killing pid 3244 failed: Operation not permitted
From the pkill(1) man page:
EXIT STATUS
...
1 No processes matched or none of them could be signalled.
...
It turns out it was just a bug.
The solution was simple, I copied my script to a new file, deleted the old one and it worked, simple as that.
Related
This is a direct clarification question to the answer in here which I thought it worked, but it does not!
I have the following test bash script (testbash.sh) which just creates some output and a lot of errors for testing purposes (running on Red Hat Enterprise Linux Server release 7.6 (Maipo) and also Ubuntu 16.04.6 LTS):
export MAX_SEED=2
echo "Start test"
pids=""
for seed in `seq 1 ${MAX_SEED}`
do
python -c "raise ValueError('test')" &
pids="${pids} $!"
done
echo "pids: ${pids}"
wait $pids
echo "End test"
If I run this script I get the following output:
Start test
pids: 68322 68323
Traceback (most recent call last):
File "<string>", line 1, in <module>
ValueError: test
Traceback (most recent call last):
File "<string>", line 1, in <module>
ValueError: test
[1]- Exit 1 python -c "raise ValueError('test')"
[2]+ Exit 1 python -c "raise ValueError('test')"
End test
That is the expected outcome. That is fine. I want to get errors!
Now here is the python code that is supposed to catch all the output:
from __future__ import print_function
import sys
import time
from subprocess import PIPE, Popen, STDOUT
from threading import Thread
try:
from queue import Queue, Empty
except ImportError:
from Queue import Queue, Empty # python 2.x
ON_POSIX = 'posix' in sys.builtin_module_names
def enqueue_output(out, queue):
for line in iter(out.readline, b''):
queue.put(line.decode('ascii'))
out.close()
p = Popen(['. testbash.sh'], stdout=PIPE, stderr=STDOUT, bufsize=1, close_fds=ON_POSIX, shell=True)
q = Queue()
t = Thread(target=enqueue_output, args=(p.stdout, q))
t.daemon = True # thread dies with the program
t.start()
# read line without blocking
while t.is_alive():
#time.sleep(1)
try:
line = q.get(timeout=.1)
except Empty:
print(line)
pass
else:
# got line
print(line, end='')
p.wait()
print('returncode = {}'.format(p.returncode))
But when I run this code I only get the following output:
Start test
pids: 70191 70192
Traceback (most recent call last):
returncode = 0
or this output (without the line End test):
Start test
pids: 10180 10181
Traceback (most recent call last):
File "<string>", line 1, in <module>
ValueError: test
Traceback (most recent call last):
File "<string>", line 1, in <module>
ValueError: test
returncode = 0
Most of the above output is missing! How can I fix this? Also, I need some way to check if any command in the bash script did not succeed. In the example this is the case, but the errorcode printed out is still 0. I expect an errorcode != 0.
It is not important to immediately get the output. A delay of some seconds is fine. Also if the output order is a bit mixed up this is of no concern. The important thing is to get all the output (stdout and stderr).
Maybe there is a simpler way to just get the output of a bash script which is started from python?
To be run with python3
from __future__ import print_function
import os
import stat
import sys
import time
from subprocess import PIPE, Popen, STDOUT
from threading import Thread
try:
from queue import Queue, Empty
except ImportError:
from Queue import Queue, Empty # python 2.x
ON_POSIX = 'posix' in sys.builtin_module_names
TESTBASH = '/tmp/testbash.sh'
def create_bashtest():
with open(TESTBASH, 'wt') as file_desc:
file_desc.write("""#!/usr/bin/env bash
export MAX_SEED=2
echo "Start test"
pids=""
for seed in `seq 1 ${MAX_SEED}`
do
python -c "raise ValueError('test')" &
pids="${pids} $!"
sleep .1 # Wait so that error messages don't get out of order.
done
wait $pids; return_code=$?
sleep 0.2 # Wait for background messages to be processed.
echo "pids: ${pids}"
echo "End test"
sleep 1 # Wait for main process to handle all the output
exit $return_code
""")
os.chmod(TESTBASH, stat.S_IEXEC|stat.S_IRUSR|stat.S_IWUSR)
def enqueue_output(queue):
pipe = Popen([TESTBASH], stdout=PIPE, stderr=STDOUT,
bufsize=1, close_fds=ON_POSIX, shell=True)
out = pipe.stdout
while pipe.poll() is None:
line = out.readline()
if line:
queue.put(line.decode('ascii'))
time.sleep(.1)
print('returncode = {}'.format(pipe.returncode))
create_bashtest()
C_CHANNEL = Queue()
THREAD = Thread(target=enqueue_output, args=(C_CHANNEL,))
THREAD.daemon = True
THREAD.start()
while THREAD.is_alive():
time.sleep(0.1)
try:
line = C_CHANNEL.get_nowait()
except Empty:
pass # print("no output")
else:
print(line, end='')
Hope this helps :
First, looks like buffers are not being flushed. Redirecting (and, to be safe, appending) stdout/stderr to a file(s) rather than to the terminal, may help. You can always use tee (or tee -a) if you really want both. Using context managers 'might' help.
As far as the zero return code, $!
https://unix.stackexchange.com/questions/386196/doesnt-work-on-command-line
! may be invoking history invoking history, thereby $! resulting in an empty value.
If you somehow end up with just a bare wait the return code will be a zero. Regardless, return codes can be tricky, and you might be picking a successful return code from elsewhere.
Take a look at stdbuf command to change the buffer sizes for stdout and stderr:
Is there a way to flush stdout of a running process
That may also help with getting the rest of your expected output.
Rewrite the while block this way:
# read line without blocking
while t.is_alive():
try:
line = q.get(block=False)
except Empty:
# print(line)
pass
else:
# got line
print(line, end='')
You don't want to block on getting a line from the Queue when there's none, and you don't need a timeout in this case, as it's only used when blocking the thread is required. Consequently, if the Queue.get() throws Empty, there's no line to print, and we just pass.
===
Also, let's clarify the script execution logic.
Since you're using Bash expressions, and the default shell used by Popen is /bin/sh, you'd probably want to rewrite the invokation line this way:
p = Popen(['/usr/bin/bash','-c', './testbash.sh'], stdout=PIPE, stderr=STDOUT, bufsize=1, close_fds=ON_POSIX)
It won't hurt to add a shebang to your shell script, too:
#!/usr/bin/env bash
<... rest of the script ...>
If you're looking for these lines:
[1]- Exit 1 python -c "raise ValueError('test')"
[2]+ Exit 1 python -c "raise ValueError('test')"
This is a function of the bash shell that's typically only available in interactive mode, i.e. when you're typing commands into a terminal. If you check the bash source code, you can see that it explicitly checks the mode before printing to stdout/stderr.
In the more recent versions of bash, you can't set this inside a script: see https://unix.stackexchange.com/a/364618 . However, you can set this yourself when starting the script:
p = Popen(['/bin/bash -i ./testbash.sh'], stdout=PIPE, stderr=STDOUT, bufsize=1, close_fds=ON_POSIX, shell=True)
I will note that this is only working for me on Python3 - Python2 is only getting part of the output. It isn't clear version of Python you're using, but considering Python2 is end of life now we should probably all be trying to switch to Python3.
As for the bash script, even with interactive mode set it seems you have to change how you wait to get that output:
#!/bin/bash
export MAX_SEED=2
echo "Start test"
pids=""
for seed in `seq 1 ${MAX_SEED}`
do
python -c "raise ValueError('test')" &
pids="${pids} $!"
done
echo "pids: ${pids}"
wait -n $pids
wait -n $pids
ret=$?
echo "End test"
exit $ret
Normal wait wasn't working for me (Ubuntu 18.04), but wait -n seemed to work - but as it only waits for the next job to complete, I had inconsistent output just calling it once. Calling wait -n for each job launched seems to do the trick, but the program flow should probably be refactored to loop over the wait the same number of times you spin up the job.
Also note that to change the return code of the script, Philippe's answer has the right approach - the $? variable has the return code of the latest command that failed, which you can then pass to exit. (Yet another difference in Python versions: Python2 is returning 127 while Python3 returns 1 for me.) If you need the return values for each job, one way might be to parse out the values in the interactive job exit lines.
Just guessing - could it be that a line that starts with an empty character / space is not recognized as a line by your logic.
Maybe this indent is the issue. Another option is, that there is a tab or something like that and the ascii decode might fail.
This is how I usually use subprocess:
import subprocess
with subprocess.Popen(["./test.sh"], shell=True, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE) as p:
error = p.stderr.read().decode()
std_out = p.stdout.read().decode()
if std_out:
print(std_out)
if error:
print("Error message: {}".format(error))
Here you decode and read both the stdout and the stderr. You get everything but not in the same order, I don't if that's an issue.
i am trying to get the output of a command in my python program by using "check_output" method. but i'm getting this error:
out = check_output(command5 , shell=True)
File "/usr/lib64/python3.6/subprocess.py", line 336, in check_output
**kwargs).stdout
File "/usr/lib64/python3.6/subprocess.py", line 418, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'oscap xccdf eval --profile xccdf_org.ssgproject.content_profile_rht-ccp --results-arf arf.xml /usr/share/xml/scap/ssg/content/ssg-centos7-ds.xml' returned non-zero exit status 2.
this is the part of my program that is related:
command4 = "oscap xccdf eval --profile xccdf_org.ssgproject.content_profile_rht-ccp --results-arf arf.xml /usr/share/xml/scap/ssg/content/ssg-centos7-ds.xml"
out = check_output(command4 , shell=True)
I am sure that the command is alright because I get the results when I write:
subprocess.call(command5,shell=True)
I am using python 3.6, and work in centos 7.
any idea why the check_output can not get the result?
This is entirely normal, because the command you ran produced a non-zero exit code. It means that the command you ran is signalling that something may be wrong.
See the subprocess.check_output() documentation:
If the return code was non-zero it raises a CalledProcessError.
and
This is equivalent to:
run(..., check=True, stdout=PIPE).stdout
where the check=True flag tells run() to raise an exception when return_value is not 0:
If check is true, and the process exits with a non-zero exit code, a CalledProcessError exception will be raised.
The other function you used, subprocess.call(), does not set check=True:
Run the command described by args. Wait for command to complete, then return the returncode attribute.
This is equivalent to:
run(...).returncode
So either don't use check_output(), or catch the exception thrown, or fix the command you are running. That call() worked is no indication that the process actually produced a successful result.
For example, you could use subprocess.run() directly:
proc = subprocess.run(
command5, shell=True, text=True
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
if proc.returncode:
print(f'Issue reported, exit code {proc.returncode}, stderr:')
print(proc.stderr)
else:
print(proc.stdout)
I have been working with a short python script that executes bash commands. The program worked fine for about a month. Recently, I tried running the script passing it this command:
my_launcher.py -c /path/to/config/file.json
(BTW, when the command is entered in terminal, I causes no error and runs fine) and I get the following message:
RuntimeError: Command '['bash', '-c', 'my_launcher.py -c /path/to/config/file.json']' returns non-zero exit status (code5)
After looking on Google, I found definitions for return codes 0, 1, and 2, but nothing for code 5. Wondering if any of you knows anything about it. What it means? How can it be resolved? etc.
This is the python code that causes the error:
try :
#check_output produces byte string
#raises exception if command returns a non-zero exit status (error occurs during processing of command)
string_of_text_rc = subprocess.check_output(['bash', '-c', bashCommand])
except subprocess.CalledProcessError as e:
raise RuntimeError("Command '{}' returns non-zero exit status (code{})".format(e.cmd, e.returncode))
When removing try/except, here is the taceback:
Traceback (most recent call last):
File "bash_cmd.py", line 27, in <module>
run_cmd('my_launcher.py -c /path/to/config/file.json')
File "bash_cmd.py", line 17, in run_cmd
string_of_text_rc = subprocess.check_output(['bash', '-c', bashCommand])
File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['bash', '-c', 'my_launcher.py -c /path/to/config/file.json']' returned non-zero exit status 5
**EDIT: The correct output is contained into e.output. Which means that the command is ran and returns correct output. I really don't know why I get this error code.
For the record, here's how you should run your .py file:
result = subprocess.check_output([
sys.executable, 'my_launcher.py', '-c', path_to_json])
And here is how you run shell commands:
result = subprocess.check_output(bash_command, shell=True)
For your problem - can you remove the try/except from your code, so we can see the full error traceback? Some good information might be hidden in there.
What I am trying to do is record the output of airodump-ng every 10 seconds.
First attempt:
Going through the airodump-ng documentation they mention such a command --write-interval
When I tried using it:sudo airodump-ng mon0 -w testOutput --write-interval 10 -o csv, I got the error that --write-interval is an unrecognized option.
Second attempt:
I tried doing this myself in Python. I then came accross the issue of trying to stop the process. The closest I got was this solution.
airodump = subprocess.Popen(['sudo', 'airodump-ng', 'mon0', '-w', 'pythonTest'],
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
o_airodump, unused_stderr = airodump.communicate(timeout=15)
airodump.kill()
This does stop the process, and I do have the relevant output files, BUT what happens is that all my running programs close and I get logged out of Ubuntu.
Additional Info:
Just before everything closes and I got logged out, I saw an error message on the terminal. I quickly screenshot it to see what it said:
Traceback (most recent call last):
File "airodump-call.py", line 3, in <module> o_airodump, unused_stderr = airodump.communicate(timeout=15)
File "/usr/lib/python3.5/subprocess.py", line 1072, in communicate stdout, stderr = self.communicate(input, endtime, timeout)
File "usr/lib/python3.5/subprocess.py", line 1713, in _communicate raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['airodump-ng', 'mon0', '-w', 'pythonTest']' timed out after 15 seconds
I've run into the same problem. Despite this being an old post, I post my solution since it could help someone searching for this.
Let's say I run airodump-ng like the OP:
proc = subprocess.Popen(['airodump-ng', 'wlan0mon'])
This can be terminated by sending a SIGINT signal for the pid of the process:
os.kill(proc.pid, signal.SIGINT)
Note: you need import os and import signal
i'm trying to catch the output of airodump-ng, that has a continuous output, and process every line searching for a string. but that doesn't work. so i try the same thing with "htop" command that has the same kind of output, and it still doesn't work.
i'm trying this with python 3.4 and python 2.7, both on arch linux and osx mavericks. here's the code (not every import is necessary but nevermind):
import subprocess
import sys
import os
import time
command = ["htop"]
proc = subprocess.Popen(command, stdout = subprocess.PIPE)
outs, errs = proc.communicate(timeout=3)
proc.kill()
and it gives me:
Traceback (most recent call last):
File "/Users/andrei/Dropbox/python/file_prova.py", line 8, in <module>
outs, errs = proc.communicate(timeout=3)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/subprocess.py", line 960, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/subprocess.py", line 1618, in _communicate
self._check_timeout(endtime, orig_timeout)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/subprocess.py", line 986, in _check_timeout
raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['htop']' timed out after 3 seconds
seems like it crashes at proc.communicate() and doesn't execute the lines under that. i also tried to handle the exception but no way to make it work...
[EDIT]
ok so it's for 4 am, i learned the try - exception handling, and after a looong time a managed to make it work with htop, following the tips hardly found here (the 2nd solution doesn't seem to work):
this is how it looks
from subprocess import Popen, PIPE
from time import sleep
from fcntl import fcntl, F_GETFL, F_SETFL
from os import O_NONBLOCK, read
# run the shell as a subprocess:
p = Popen(['htop'], stdout = PIPE)
# set the O_NONBLOCK flag of p.stdout file descriptor:
flags = fcntl(p.stdout, F_GETFL) # get current p.stdout flags
fcntl(p.stdout, F_SETFL, flags | O_NONBLOCK)
# let the shell output the result:
# get the output
while True:
sleep(1)
try:
print (read(p.stdout.fileno(), 1024).decode("utf-8")),
except OSError:
# the os throws an exception if there is no data
print ('[No more data]')
continue
it works flawlessly. with htop.
but not with airodump-ng. it prints on the terminal its output and every 1 second (the sleep() in the while loop) prints [No more data], like the stream is going elsewhere...
EDIT 2:
solved! the thing was just that airodump-ng dumps data to stderr, not stdout. pretty straight forward try ahah :D
From the documentation:
The timeout argument is passed to Popen.wait(). If the timeout
expires, the child process will be killed and then waited for again.
The TimeoutExpired exception will be re-raised after the child process
has terminated.
That seems to describe exactly the behavior you are seeing. You will need to learn about exception handling using try/except.