I have a Python script which runs on a server with RHEL5. The server has 32GB memory and 8 Intel Xeon CPUs at 2.83GHz. I think the hardware resource should not be a problem, but when I attempt to upload and process a 15 million line text file, the program gives me an error message:
Traceback (most recent call last):
File "./test.py", line 511, in <module>
startup()
File "./test.py", line 249, in startup
cmdoutput = commands.getoutput(cmd_file)
File "/usr/local/lib/python2.6/commands.py", line 46, in getoutput
return getstatusoutput(cmd)[1]
File "/usr/local/lib/python2.6/commands.py", line 55, in getstatusoutput
pipe = os.popen('{ ' + cmd + '; } 2>&1', 'r')
OSError: [Errno 12] Cannot allocate memory
I have investigated this problem and did not found any answers that exactly match my problem. Those answers were focused on the "popen" subroutine, but I do not use this routine. I just use the "commands.getoutput()" to display the file type of a document.
It should be noted that if I try to process a 10 million line text, this problem does not happen. It only happens when the text file is large.
Can any people help me out on this issue? The answer could be a better module other than the "command.getoutput()". Thanks!
your command might consume too much memory. To check, run it with the large file from a console without python to see if you get any errors
your command might generate too much output. To check, run:
subprocess.check_call(["cmd", "arg1", "arg2"])
if it succeeds then you should read output incrementally and discard the processed output e.g. line by line:
p = subprocess.Popen(["cmd", "arg1", "arg2"], stdout=subprocess.PIPE)
for line in iter(p.stdout.readline, ''):
# do something with line
print line,
p.stdout.close()
exit_code = p.wait() # wait for the process to exit
Related
I'm having multiple problems with a python (v3.7) script using multiprocessing (as mp hereafter). One of them is that my computations end with an "OSError: [Errno 24] Too many open files". My scripts and modules are complex, so I've broken down the problem to the following code:
def worker(n):
time.sleep(1)
n = 2000
procs = [mp.Process(target=worker, args=(i,)) for i in range(n)]
nprocs = 40
i = 0
while i<n:
if (len(mp.active_children())<=nprocs):
print('Starting proc {:d}'.format(i))
procs[i].start()
i += 1
else:
time.sleep(1)
[p.join() for p in procs]
This code fails when approx ~ 1020 processes have been excecuted. I've always used multiprocessing in a similar fashion without running into this problem, I'm running this on a serveur with ~ 120 CPU. Lately I've switch from Python 2.7 to 3.7, I don't know if that can be an issue.
Here's the full trace:
Traceback (most recent call last):
File "test_toomanyopen.py", line 18, in <module>
procs[i].start()
File "/p/jqueryrel/local_install/conda_envs/trois/lib/python3.7/multiprocessing/process.py", line 112, in start
self._popen = self._Popen(self)
File "/p/jqueryrel/local_install/conda_envs/trois/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/p/jqueryrel/local_install/conda_envs/trois/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
return Popen(process_obj)
File "/p/jqueryrel/local_install/conda_envs/trois/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
self._launch(process_obj)
File "/p/jqueryrel/local_install/conda_envs/trois/lib/python3.7/multiprocessing/popen_fork.py", line 69, in _launch
parent_r, child_w = os.pipe()
OSError: [Errno 24] Too many open files
I've seen a similar issue here, but I don't see how I can solve this.
Thanks
To put the comments into an answer, several options to fix this:
Increase the limit of possible open file handles. Edit /etc/security/limits.conf. E.g. see here.
Don't spawn so many processes. If you have 120 CPUs, it doesn't really make sense to spawn more than 120 procs.
Maybe using Pool might be helpful to restructure your code.
I'm facing a strange situation, I've searched on google without any good results.
I'm running a python script as a subprocess from a parent subprocess with nohup using subprocess package:
cmd = list()
cmd.append("nohup")
cmd.append(sys.executable)
cmd.append(os.path.abspath(script))
cmd.append(os.path.abspath(conf_path))
_env = os.environ.copy()
if env:
_env.update({k: str(v) for k, v in env.items()})
p = subprocess.Popen(cmd, env=_env, cwd=os.getcwd())
After some time the parent process exists and the subprocess (the one with the nohup continues to run).
After another minute or two the process with the nohup exits, and with obvious reasons, becomes a zombie.
When running it on local PC with python3.6 and ubuntu 18.04, I manage to run the following code and everything works like a charm:
comp_process = psutil.Process(pid)
if comp_process.status() == "zombie":
comp_status_code = comp_process.wait(timeout=10)
As I said, everything works like a charm, The zombie process removed and I got the status code of the mentioned process.
But for some reason, when doing the SAME at docker container with the SAME python version and Ubuntu version, It fails after the timeout (Doesn't matter if its 10 seconds or 10 minutes)
The error:
psutil.TimeoutExpired timeout after 60 seconds (pid=779)
Traceback (most recent call last): File
"/usr/local/lib/python3.6/dist-packages/psutil/_psposix.py", line 84,
in wait_pid
retpid, status = waitcall() File "/usr/local/lib/python3.6/dist-packages/psutil/_psposix.py", line 75,
in waitcall
return os.waitpid(pid, os.WNOHANG) ChildProcessError: [Errno 10] No child processes
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File ".py", line 41, in
run
comp_status_code = comp_process.wait(timeout=60) File "/usr/local/lib/python3.6/dist-packages/psutil/init.py", line
1383, in wait
return self._proc.wait(timeout) File "/usr/local/lib/python3.6/dist-packages/psutil/_pslinux.py", line
1517, in wrapper
return fun(self, *args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/psutil/_pslinux.py", line
1725, in wait
return _psposix.wait_pid(self.pid, timeout, self._name) File "/usr/local/lib/python3.6/dist-packages/psutil/_psposix.py", line 96,
in wait_pid
delay = check_timeout(delay) File "/usr/local/lib/python3.6/dist-packages/psutil/_psposix.py", line 68,
in check_timeout
raise TimeoutExpired(timeout, pid=pid, name=proc_name) psutil.TimeoutExpired: psutil.TimeoutExpired timeout after 60 seconds
(pid=779)
One possibility may be the lack of an init process to reap zombies. You can fix this by running with docker run --init, or using e.g. tini. See https://hynek.me/articles/docker-signals/
I have a script called client that does a few things around the OS and it's failing when it comes to /proc/1/cwd
I'm trying to ignore these types of files but the script is still crashing when it comes to this file. I get Memory Map output and also a Stack Trace before the script crashes.
I did a file /proc/1/cwd command on the file and I can see it's a symlink, where I then included an if os.path.islink(file) is False: IF statement to skip it and proceed.
However, I seem to still be hitting it.
The below is the Traceback after the crash.
Traceback (most recent call last):
File "client.py", line 235, in <module>
File "client.py", line 120, in runner
OSError: [Errno 13] Permission denied: '/proc/1/cwd'
Failed to execute script client
The error suggests the script client is working on /proc/1/cwd and then right after seems to say that it can't execute itself?
Any help would be appreciated.
-----Additional -------
Code lines from error:
like 235 is literally the line runner(path)
runner() represents a function that has a for loop that cycles through files in a folder to find the correct file. During this process, runner will do the following:
do line 120 mode = os.stat(fullfilename).st_mode
check if os.path.isdir(fullfilename) is False:
then if stat.S_ISREG(mode) is True:
then if os.path.islink(fullfilename) is False:
then if fullfilename not in blacklist:
Line 120 is mode = os.stat(fullfilename).st_mode
When opening 4GB+ file on AIX Python 2.6.2 I receive IOError:
>>> fd = open('/mnt/t/MY_BIG_4GB_FILE', 'r')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IOError: [Errno 127] Value too large to be stored in data type: '/mnt/t/MY_BIG_4GB_FILE'
Any solutions? I didn't find information in Google.
EDIT:
To read BIG file I do something line that (I know that it is rubbish but sometimes You simply can't change python version):
from subprocess import Popen, PIPE
p = Popen(["cat", source_file], stdout=PIPE, bufsize=BUFFER_SIZE)
try:
for line in iter(p.stdout.readline, ''):
# process line
pass
finally:
p.communicate() # closing Popen
You're looking for "large file support." There is a decent brief here: http://docs.python.org/2/library/posix.html#large-file-support . You likely need to recompile your Python interpreter with the appropriate options, or find a pre-built one that has those options. Try a build of Python 2.7 if you can, while you're at it.
In my python code I use grab python lib and try to open 3 sites in ope python script
My current code:
from grab import Grab
...
g = Grab(interface=ip, headers=headers)
a = g.go('http://site1.com');
g = Grab(interface=ip, headers=headers)
b = g.go('http://site2.com');
g = Grab(interface=ip, headers=headers)
c = g.go('http://site3.com');
And this code working fine if I run even 10 python scripts
But I decide that better for me to open all connections in same time, (no wait when site "a" will be loaded before open site "b") And I tried to make processes:
pa = Process(target=m_a, args=(ip))
pb = Process(target=m_b, args=(ip))
pc = Process(target=m_c, args=(ip))
pa.start()
pb.start()
pc.start()
But when I try to run more than 5 python processes I see "could not allocate memory" message.
Why this code working in one python file, and "could not allocate memory" when I try to run it by processes for each site request?
Actuality I already use python process for running this python script,
and my name != 'main' .
In first python (which run this script) I use this code:
if __name__ == '__main__':
jobs = []
for f in [exit_error, exit_ok, return_value, raises, terminated]:
print 'Starting process for', f.func_name
j = multiprocessing.Process(target=f, name=f.func_name)
jobs.append(j)
j.start()
I use VPS OpenVZ 512
Error Report:
Process Process-18:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/root/_scripts/bf/check_current.py", line 140, in worker
p.start()
File "/usr/lib/python2.7/multiprocessing/process.py", line 130, in start
self._popen = Popen(self)
File "/usr/lib/python2.7/multiprocessing/forking.py", line 120, in __init__
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
If the processes are running in parallel then you may indeed be running out of RAM. Open Task Manager or its equivalent and check your total allocated memory as you run.
I found good command for set limit of memory using for python processes
ulimit -s 2000