I'm facing a strange situation, I've searched on google without any good results.
I'm running a python script as a subprocess from a parent subprocess with nohup using subprocess package:
cmd = list()
cmd.append("nohup")
cmd.append(sys.executable)
cmd.append(os.path.abspath(script))
cmd.append(os.path.abspath(conf_path))
_env = os.environ.copy()
if env:
_env.update({k: str(v) for k, v in env.items()})
p = subprocess.Popen(cmd, env=_env, cwd=os.getcwd())
After some time the parent process exists and the subprocess (the one with the nohup continues to run).
After another minute or two the process with the nohup exits, and with obvious reasons, becomes a zombie.
When running it on local PC with python3.6 and ubuntu 18.04, I manage to run the following code and everything works like a charm:
comp_process = psutil.Process(pid)
if comp_process.status() == "zombie":
comp_status_code = comp_process.wait(timeout=10)
As I said, everything works like a charm, The zombie process removed and I got the status code of the mentioned process.
But for some reason, when doing the SAME at docker container with the SAME python version and Ubuntu version, It fails after the timeout (Doesn't matter if its 10 seconds or 10 minutes)
The error:
psutil.TimeoutExpired timeout after 60 seconds (pid=779)
Traceback (most recent call last): File
"/usr/local/lib/python3.6/dist-packages/psutil/_psposix.py", line 84,
in wait_pid
retpid, status = waitcall() File "/usr/local/lib/python3.6/dist-packages/psutil/_psposix.py", line 75,
in waitcall
return os.waitpid(pid, os.WNOHANG) ChildProcessError: [Errno 10] No child processes
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File ".py", line 41, in
run
comp_status_code = comp_process.wait(timeout=60) File "/usr/local/lib/python3.6/dist-packages/psutil/init.py", line
1383, in wait
return self._proc.wait(timeout) File "/usr/local/lib/python3.6/dist-packages/psutil/_pslinux.py", line
1517, in wrapper
return fun(self, *args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/psutil/_pslinux.py", line
1725, in wait
return _psposix.wait_pid(self.pid, timeout, self._name) File "/usr/local/lib/python3.6/dist-packages/psutil/_psposix.py", line 96,
in wait_pid
delay = check_timeout(delay) File "/usr/local/lib/python3.6/dist-packages/psutil/_psposix.py", line 68,
in check_timeout
raise TimeoutExpired(timeout, pid=pid, name=proc_name) psutil.TimeoutExpired: psutil.TimeoutExpired timeout after 60 seconds
(pid=779)
One possibility may be the lack of an init process to reap zombies. You can fix this by running with docker run --init, or using e.g. tini. See https://hynek.me/articles/docker-signals/
Related
For context:
I have an HP DL380p and the fans are annoyingly loud because of PCIe cards and non HP-drives. So I flashed the iLO (HPE Integrated Lights-Out) on the server with a modded version which allows to edit fan curves. I already tried it manually and it works fine, PCIe cards and CPU never go over 60 °C with my usage. There is problem with the mod however, which makes to if you're SSH connection turns off, the fan command (used to change the fan curves) doesn't work anymore. You have to reset the iLO with the "reset /map1" command, which resets previous fan curves adjustments. Which is why I found fabric and decided to use for this instead of a bash script.
I wrote this code:
from fabric import Connection
import time
connection = Connection(host = "username#ip", connect_kwargs = {"password" : "password"})
connection.run("reset /map1")
time.sleep(40)
connection = Connection(host = "username#ip", connect_kwargs = {"password" : "password"})
connection.run("fan info g")
But every time I run it, I get this error:
reset /map1
status=0
status_tag=COMMAND COMPLETED
Tue Jan 31 11:35:12 2023
Resetting iLO.
CLI session stopped
Traceback (most recent call last):
File "C:\Users\ewenlau\Desktop\silence fan.py", line 4, in <module>
connection.run("reset /map1")
File "<decorator-gen-3>", line 2, in run
File "C:\Users\ewenlau\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\fabric\connection.py", line 27, in opens
return method(self, *args, **kwargs)
File "C:\Users\ewenlau\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\fabric\connection.py", line 739, in run
return self._run(self._remote_runner(), command, **kwargs)
File "C:\Users\ewenlau\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\invoke\context.py", line 98, in _run
return runner.run(command, **kwargs)
File "C:\Users\ewenlau\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\fabric\runners.py", line 75, in run
return super().run(command, **kwargs)
File "C:\Users\ewenlau\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\invoke\runners.py", line 376, in run
return self._run_body(command, **kwargs)
File "C:\Users\ewenlau\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\invoke\runners.py", line 432, in _run_body
return self.make_promise() if self._asynchronous else self._finish()
File "C:\Users\ewenlau\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\invoke\runners.py", line 499, in _finish
raise UnexpectedExit(result)
invoke.exceptions.UnexpectedExit: Encountered a bad command exit code!
Command: 'reset /map1'
Exit code: -1
Stdout: already printed
Stderr: already printed
What I was expecting to happen is this script to return me the current fan speed (fan info g). Instead it outputted the error above. I tried removing the sleep command, nothing changed. I have to note that the reset /map1 command closes the connection host side, and I think the problem comes from there, but I have no idea how to fix it.
I'm trying to use the Python debugger on VS Code 2019 on my Mac, but it looks like it is causing the program to hang indefinitely (the print statement never gets executed). If I run the program without the debugger, everything works. I don't have any breakpoints configured either.
Here is my code:
#!/usr/bin/env python3
dummy = input("Enter something: ")
print(dummy)
Edit: I'm not sure if it's related, but when I click the stop button "square", the program doesn't immediately quit. It takes a few seconds, and then I see this in my terminal window:
E+01487.808: Failed to kill Debuggee[PID=2791]
Traceback (most recent call last):
File "/Users/XXXXX/.vscode/extensions/ms-python.python-2020.8.101144/pythonFiles/lib/python/debugpy/launcher/../../debugpy/launcher/debuggee.py", line 160, in kill
os.killpg(process.pid, signal.SIGKILL)
PermissionError: [Errno 1] Operation not permitted
Stack where logged:
File "/Users/XXXXX/.vscode/extensions/ms-python.python-2020.8.101144/pythonFiles/lib/python/debugpy/launcher/../../debugpy/launcher/debuggee.py", line 162, in kill
log.swallow_exception("Failed to kill {0}", describe())
File "/Users/XXXXX/.vscode/extensions/ms-python.python-2020.8.101144/pythonFiles/lib/python/debugpy/launcher/../../debugpy/common/log.py", line 215, in swallow_exception
_exception(format_string, *args, **kwargs)
(.venv) Scripts %
I have a bash script, which is running perfectly:
gvim --servername "servername" $1
if [ -f ${1%.tex}.pdf ];
then
evince ${1%.tex}.pdf &
fi
evince_vim_dbus.py GVIM servername ${1%.tex}.pdf $1 &
I am trying to convert it to python as:
#!/usr/bin/env python3
from subprocess import call
import sys, os
inp_tex = sys.argv[1]
oup_pdf = os.path.splitext(sys.argv[1])[0]+".pdf"
print(oup_pdf)
call(["gvim", "--servername", "servername", sys.argv[1]])
if os.path.exists(oup_pdf):
call(["evince", oup_pdf])
call(["evince_vim_dbus.py", "GVIM", "servername", oup_pdf, inp_tex])
in the python, both gvim and evince window is open, but evince_vim_dbus.py line is not working. Not that it is giving any error, but it is not showing intended result, as it should, and is doing with the bash script.
trying with check_call (I have to kill it after a while, here's the traceback):
Traceback (most recent call last):
File "/home/rudra/vims.py", line 28, in <module>
check_call(["python","/home/rudra/bin/evince_vim_dbus.py", "GVIM", "servername", oup_pdf, inp_tex])
File "/usr/lib64/python3.5/subprocess.py", line 576, in check_call
retcode = call(*popenargs, **kwargs)
File "/usr/lib64/python3.5/subprocess.py", line 559, in call
return p.wait(timeout=timeout)
File "/usr/lib64/python3.5/subprocess.py", line 1658, in wait
(pid, sts) = self._try_wait(0)
File "/usr/lib64/python3.5/subprocess.py", line 1608, in _try_wait
(pid, sts) = os.waitpid(self.pid, wait_flags)
KeyboardInterrupt
I'm going to have a guess that your real problem isn't the evince_vim_dbus.py line itself, but rather the gvim line, because you pass it the server name 'servername' instead of simply servername, and so doesn't match the name on the line that runs evince_vim_dbus.py.
I'm not familiar with gvim or its server functionality, but I'm guessing the evince_vim_dbus.py program connects to gvim using the given name, in which case it's going to fail since the server of the right name isn't running.
If that's not it, then maybe the problem is that subprocess.call() runs the given program and waits for it to exit, whereas in your original bash script, you run evince with an ampersand, causing bash not to wait for it, so maybe the problem is that evince_vim_dbus.py never runs at all until you exit Evince.
Given a pexpect spawned process that's opened with sudo, like so:
#!/usr/bin/env python
import pexpect
cmd = ['sudo', 'bash', '-c', '"some long-running sudo command"']
cmd = ' '.join(cmd)
child = pexpect.spawn(cmd, timeout=60)
i = child.expect([
'success',
'error'])
if i == 0:
print('ok')
else:
print('fail')
# insert code here
How would I kill this process on fail (or success, for that matter)?
I've tried the following (replacing # insert code here):
child.kill(0)
child.close(force=True)
Both give the following error, which makes sense as the Python script is not a root process, and it's trying to kill something that is a root process.
Traceback (most recent call last):
File "./myscript.py", line 85, in <module>
requires_qemu()
File "./myscript.py", line 82, in requires_qemu
child.close(0)
File "/usr/lib/python2.7/site-packages/pexpect/__init__.py", line 747, in close
raise ExceptionPexpect('Could not terminate the child.')
pexpect.ExceptionPexpect: Could not terminate the child.
It is not possible to run the script as root, due to file permissions (run from a shared NFS drive where root access is blocked)
Use sudo to kill it as root:
subprocess.call(['sudo', 'kill', str(child.pid)])
In my python code I use grab python lib and try to open 3 sites in ope python script
My current code:
from grab import Grab
...
g = Grab(interface=ip, headers=headers)
a = g.go('http://site1.com');
g = Grab(interface=ip, headers=headers)
b = g.go('http://site2.com');
g = Grab(interface=ip, headers=headers)
c = g.go('http://site3.com');
And this code working fine if I run even 10 python scripts
But I decide that better for me to open all connections in same time, (no wait when site "a" will be loaded before open site "b") And I tried to make processes:
pa = Process(target=m_a, args=(ip))
pb = Process(target=m_b, args=(ip))
pc = Process(target=m_c, args=(ip))
pa.start()
pb.start()
pc.start()
But when I try to run more than 5 python processes I see "could not allocate memory" message.
Why this code working in one python file, and "could not allocate memory" when I try to run it by processes for each site request?
Actuality I already use python process for running this python script,
and my name != 'main' .
In first python (which run this script) I use this code:
if __name__ == '__main__':
jobs = []
for f in [exit_error, exit_ok, return_value, raises, terminated]:
print 'Starting process for', f.func_name
j = multiprocessing.Process(target=f, name=f.func_name)
jobs.append(j)
j.start()
I use VPS OpenVZ 512
Error Report:
Process Process-18:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/root/_scripts/bf/check_current.py", line 140, in worker
p.start()
File "/usr/lib/python2.7/multiprocessing/process.py", line 130, in start
self._popen = Popen(self)
File "/usr/lib/python2.7/multiprocessing/forking.py", line 120, in __init__
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
If the processes are running in parallel then you may indeed be running out of RAM. Open Task Manager or its equivalent and check your total allocated memory as you run.
I found good command for set limit of memory using for python processes
ulimit -s 2000