I'm using map_asyn to share my work load, however I'm finding that I'm getting a MemoryError, but I'm not finding a solution or a work around. Here is the error I get:
Exception in thread Thread-3:
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
self.run()
File "C:\Python27\lib\threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "C:\Python27\lib\multiprocessing\pool.py", line 380, in _handle_results
task = get()
MemoryError
Here is the code:
pool = Pool(maxtasksperchild=2)
fixed_args = (targetdirectorytxt, value_dict)
varg = ((filename,) + fixed_args for filename in readinfiles)
op_list = pool.map_async(PPD_star, list(varg), chunksize=1)
while not op_list.ready():
print("Number of files left to process: {}".format(op_list._number_left))
time.sleep(600)
op_list = op_list.get()
pool.close()
pool.join()
Here is what I have tried:
Reducing the number of threads
Limiting maxtasksperchild
appply_sync instead of map_sync
Are there anymore suggestions to avoid this error?
I'm reading in the file with:
with open(os.path.join(txtdatapath,pathfilename), "r") as data:
datalines = (line.rstrip('\r\n') for line in data)
for record in datalines:
I agree with #AndréLaszio, it is likely that the files are too large to be kept in memory. Altering your logic to only keep one line in memory at a time should alleviate the memory pressure unless each line is also huge.
Here is an alternate approach to opening the file and working with one line at a time. Keeping the file contents available in memory as an array is an expensive operation.
readingfiles.py:
from memory_profiler import profile
#profile
def open_file_read_all_then_parse(filename):
"""
Open a file and remove all the new lines characters from each line then
parse the resulting array of clean lines.
"""
with open(filename, "r") as data:
datalines = (line.rstrip('\r\n') for line in data)
for record in datalines:
pass
#profile
def open_file_read_and_parse(filename):
"""
Open a file and iterate over each line of the file while striping the record
of any newline characters.
"""
with open(filename, "r") as data:
for record in data:
record.rstrip('\r\n')
if __name__ == '__main__':
# input.dat is a roughly 4m file with 10000 lines
open_file_read_all_then_parse("./input.dat")
open_file_read_and_parse("./input.dat")
I used an additional module to help track down my memory usage, the module is named memory profiler. This module helped me verify where my memory issues were coming from and may be useful for your debugging. It will list out memory usage of a program and the area the memory is being used.
For more in depth performance analysis I recommend this article by Huy Nguyen.
Related
I'm offloading a task to a separate process to protect my memory space. The process runs a cythonized C library that tends to not fully clean up after itself. The result is then returned through multiprocessing.Queue. However, once the item being return reaches a certain size, the Queue.get method stalls.
I'm using the processify wrapper from https://gist.github.com/schlamar/2311116 which wraps the function call in a Process.
My test function is
#processify
def do_something(size: int):
return np.zeros(shape=size, dtype=np.uint8)
My test code is
if __name__ == '__main__':
for k in range(1, 11):
print(f"{k*256}MB")
t0 = time()
do_something(k * 256 * 1024 * 1024)
print(f"Took {time()-t0:0.1f}s")
This runs smoothly until 2048MB where it just stays for minutes (no CPU activity) until I cancel the process:
256MB
Took 0.7s
512MB
Took 1.6s
768MB
Took 2.1s
1024MB
Took 2.7s
1280MB
Took 3.4s
1536MB
Took 4.0s
1792MB
Took 4.6s
2048MB
^CTraceback (most recent call last):
File "processify.py", line 63, in <module>
do_something(k * 256 * 1024 * 1024)
File "processify.py", line 40, in wrapper
ret, error = q.get()
File "/home/.../python3.7/multiprocessing/queues.py", line 94, in get
res = self._recv_bytes()
File "/home/.../python3.7/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/home/.../python3.7/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/home/.../python3.7/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt
From the stack trace it becomes evident that the Queue.get function is waiting. If I add print statements, I can see that Queue.put has already finished at this time so the return value should be inside the Queue. I also tried to run without Process.join as suggested by a commend in the GitHub gist, but that didn't help either.
I know that this kind of design is suboptimal and I should probably fix the cython library so that I don't need to offload in the first place. Yet, I would like to know if there's an inherent limitation in python's multiprocessing that doesn't allow for objects of a certain size to pass through the Queue.
Thank you all in advance!
I'm using python to do some processing on text files and am having issues with MemoryErrors. Sometimes the file being processed is quite large which means that too much RAM is being used by a multiprocessing Process.
Here is a snippet of my code:
import multiprocessing as mp
import os
def preprocess_file(file_path):
with open(file_path, "r+") as f:
file_contents = f.read()
# modify the file_contents
# ...
# overwrite file
f.seek(0)
f.write(file_contents)
f.truncate()
if __name__ == "main":
with mp.Pool(mp.cpu_count()) as pool:
pool_processes = []
# for all files in dir
for root, dirs, files in os.walk(some_path):
for f in files:
pool_processes.append(os.path.join(root, f))
# start the processes
pool.map(preprocess_file, pool_processes)
I have tried to use the resource package to set a limit to how much RAM each process can use as shown below but this hasn't fixed the issue, and I still get MemoryErrors being raised which leads me to believe it's the pool.map which is causing issues. I was hoping to have each process deal with the exception individually so that the file could be skipped rather than crashing the whole program.
import resource
def preprocess_file(file_path):
try:
hard = os.sysconf("SC_PAGE_SIZE") * os.sysconf("SC_PHYS_PAGES") # total bytes of RAM in machine
soft = (hard - 512 * 1024 * 1024) // mp.cpu_count() # split between each cpu and save 512MB for the system
resource.setrlimit(resource.RLIMIT_AS, (soft, hard)) # apply limit
with open(file_path, "r+") as f:
# ...
except Exception as e: # bad practice - should be more specific but just a placeholder
# ...
How can I let an individual process run out of memory while letting the other processes continue unaffected? Ideally I want to catch the exception within the preprocess_file file so that I can log exactly which file caused the error.
Edit: The preprocess_file function does not share data with any other processes so there is no need for shared memory. The function also needs to read the entire file at once as the file is reformatted which cannot be done line by line.
Edit 2: The traceback from the program is below. As you can see, the error doesn't actually point to the file being run, and instead comes from the package's files.
Process ForkPoolWorker-2:
Traceback (most recent call last):
File "/usr/lib64/python3.6/multiprocessing/pool.py", line 125, in worker
File "/usr/lib64/python3.6/multiprocessing/queues.py", line 341, in put
File "/usr/lib64/python3.6/multiprocessing/reduction.py", line 51, in dumps
File "/usr/lib64/python3.6/multiprocessing/reduction.py", line 39, in __init__
MemoryError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
File "/usr/lib64/python3.6/multiprocessing/pool.py", line 130, in worker
File "/usr/lib64/python3.6/multiprocessing/queues.py", line 341, in put
File "/usr/lib64/python3.6/multiprocessing/reduction.py", line 51, in dumps
File "/usr/lib64/python3.6/multiprocessing/reduction.py", line 39, in __init__
MemoryError
If MemoryError is raised, the worker process may or may not be able to recover from the situation. If it do, as #Thomas suggest, catch the MemoryError somewhere.
import multiprocessing as mp
from time import sleep
def initializer():
# Probably set the memory limit here
pass
def worker(i):
sleep(1)
try:
if i % 2 == 0:
raise MemoryError
except MemoryError as ex:
return str(ex)
return i
if __name__ == '__main__':
with mp.Pool(2, initializer=initializer) as pool:
tasks = range(10)
results = pool.map(worker, tasks)
print(results)
If the worker cannot recover, the whole pool is unlikely working. For example, change worker to do a force exit:
def worker(i):
sleep(1)
try:
if i % 2 == 0:
raise MemoryError
elif i == 5:
import sys
sys.exit()
except MemoryError as ex:
return str(ex)
return i
the Pool.map never return and block forever.
I wrote a script on a linux platform using the multiprocessing module of python. When I tried running the program on Windows this was not working directly which I found out is related to the fact how child-processes are generated on Windows. It seems to be crucial that the objects which are used can be pickled.
My main problem is, that I am using large numpy arrays. It seems that with a certain size they are not pickable any more. To break it down to a simple script, I want to do something like that:
### Import modules
import numpy as np
import multiprocessing as mp
number_of_processes = 4
if __name__ == '__main__':
def reverse_np_array(arr):
arr = arr + 1
return arr
a = np.ndarray((200,1024,1280),dtype=np.uint16)
def put_into_queue(_Queue,arr):
_Queue.put(reverse_np_array(arr))
Queue_list = []
Process_list = []
list_of_arrays = []
for i in range(number_of_processes):
Queue_list.append(mp.Queue())
for i in range(number_of_processes):
Process_list.append(mp.Process(target=put_into_queue, args=(Queue_list[i],a)))
for i in range(number_of_processes):
Process_list[i].start()
for i in range(number_of_processes):
list_of_arrays.append(Queue_list[i].get())
for i in range(number_of_processes):
Process_list[i].join()
I get the following error message:
Traceback (most recent call last):
File "Windows_multi.py", line 34, in <module>
Process_list[i].start()
File "C:\Program Files\Anaconda32\lib\multiprocessing\process.py", line 130, i
n start
self._popen = Popen(self)
File "C:\Program Files\Anaconda32\lib\multiprocessing\forking.py", line 277, i
n __init__
dump(process_obj, to_child, HIGHEST_PROTOCOL)
File "C:\Program Files\Anaconda32\lib\multiprocessing\forking.py", line 199, i
n dump
ForkingPickler(file, protocol).dump(obj)
File "C:\Program Files\Anaconda32\lib\pickle.py", line 224, in dump
self.save(obj)
File "C:\Program Files\Anaconda32\lib\pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "C:\Program Files\Anaconda32\lib\pickle.py", line 419, in save_reduce
save(state)
File "C:\Program Files\Anaconda32\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Program Files\Anaconda32\lib\pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
So I am basically creating a large array which I need in all processes to do calculations with this array and return it.
One important thing seems to be to write the definitions of the functions before the statement if __name__ = '__main__':
The whole thing is working if I reduce the array to (50,1024,1280).
However even if 4 processes are started and 4 cores are working, it is slower than writing the code without multiprocessing for one core only (on windows). So I think I have another problem here.
The function in my real program later on is in a cython module.
I am using the anaconda package with python 32-bit since I could not get my cython package compiled with the 64-bit version (I'll ask about that in a different thread).
Any help is welcome!!
Thanks!
Philipp
UPDATE:
First mistake I did was haveing the a "put_into_queue" function definition in the __main__.
Then I introduced shared arrays as suggested, however, uses a lot of memory and the used memory scales with the processes I use (which should of course not be the case).
Any ideas what I am doing wrong here? It seems not to be important where I place the definition of the shared array (in or outside __main__), though, I think it should be in the __main__. Got this from this post: Is shared readonly data copied to different processes for Python multiprocessing?
import numpy as np
import multiprocessing as mp
import ctypes
shared_array_base = mp.Array(ctypes.c_uint, 1280*1024*20)
shared_array = np.ctypeslib.as_array(shared_array_base.get_obj())
#print shared_array
shared_array = shared_array.reshape(20,1024,1280)
number_of_processes = 4
def put_into_queue(_Queue,arr):
_Queue.put(reverse_np_array(arr))
def reverse_np_array(arr):
arr = arr + 1 + np.random.rand()
return arr
if __name__ == '__main__':
#print shared_arra
#a = np.ndarray((50,1024,1280),dtype=np.uint16)
Queue_list = []
Process_list = []
list_of_arrays = []
for i in range(number_of_processes):
Queue_list.append(mp.Queue())
for i in range(number_of_processes):
Process_list.append(mp.Process(target=put_into_queue, args=(Queue_list[i],shared_array)))
for i in range(number_of_processes):
Process_list[i].start()
for i in range(number_of_processes):
list_of_arrays.append(Queue_list[i].get())
for i in range(number_of_processes):
Process_list[i].join()
You didn't include the full traceback; the end is most important. On my 32-bit Python I get the same traceback that finally ends in
File "C:\Python27\lib\pickle.py", line 486, in save_string
self.write(BINSTRING + pack("<i", n) + obj)
MemoryError
MemoryError is the exception and it says you ran out of memory.
64-bit Python would get around this, but sending large amounts of data between processes can easily become a serious bottleneck in multiprocessing.
I have a Python script which runs on a server with RHEL5. The server has 32GB memory and 8 Intel Xeon CPUs at 2.83GHz. I think the hardware resource should not be a problem, but when I attempt to upload and process a 15 million line text file, the program gives me an error message:
Traceback (most recent call last):
File "./test.py", line 511, in <module>
startup()
File "./test.py", line 249, in startup
cmdoutput = commands.getoutput(cmd_file)
File "/usr/local/lib/python2.6/commands.py", line 46, in getoutput
return getstatusoutput(cmd)[1]
File "/usr/local/lib/python2.6/commands.py", line 55, in getstatusoutput
pipe = os.popen('{ ' + cmd + '; } 2>&1', 'r')
OSError: [Errno 12] Cannot allocate memory
I have investigated this problem and did not found any answers that exactly match my problem. Those answers were focused on the "popen" subroutine, but I do not use this routine. I just use the "commands.getoutput()" to display the file type of a document.
It should be noted that if I try to process a 10 million line text, this problem does not happen. It only happens when the text file is large.
Can any people help me out on this issue? The answer could be a better module other than the "command.getoutput()". Thanks!
your command might consume too much memory. To check, run it with the large file from a console without python to see if you get any errors
your command might generate too much output. To check, run:
subprocess.check_call(["cmd", "arg1", "arg2"])
if it succeeds then you should read output incrementally and discard the processed output e.g. line by line:
p = subprocess.Popen(["cmd", "arg1", "arg2"], stdout=subprocess.PIPE)
for line in iter(p.stdout.readline, ''):
# do something with line
print line,
p.stdout.close()
exit_code = p.wait() # wait for the process to exit
I recently found out about pickle, which is amazing. But it errors on me when used for my actual script, testing it with a one item dictionary it worked fine. My real script is thousands of lines of code storing various objects within maya into it. I do not know if it has anything to do with the size, I have read around a lot of threads here but none are specific to my error.
I have tried writing with all priorities. No luck.
This is my output code:
output = open('locatorsDump.pkl', 'wb')
pickle.dump(l.locators, output, -1)
output.close()
This is my read code:
jntdump = open('locatorsDump.pkl', 'rb')
test = pickle.load(jntdump)
jntdump.close()
This is the error:
# Error: Error in maya.utils._guiExceptHook:
# File "C:\Program Files\Autodesk\Maya2011\Python\lib\site-packages\pymel-1.0.0-py2.6.egg\maya\utils.py", line 277, in formatGuiException
# exceptionMsg = excLines[-1].split(':',1)[1].strip()
# IndexError: list index out of range
#
# Original exception was:
# Traceback (most recent call last):
# File "<maya console>", line 3, in <module>
# File "C:\Program Files\Autodesk\Maya2011\bin\python26.zip\pickle.py", line 1370, in load
# return Unpickler(file).load()
# File "C:\Program Files\Autodesk\Maya2011\bin\python26.zip\pickle.py", line 858, in load
# dispatch[key](self)
# File "C:\Program Files\Autodesk\Maya2011\bin\python26.zip\pickle.py", line 880, in load_eof
# raise EOFError
# EOFError #
Try using pickle.dumps() and pickle.loads() as a test.
If you don't recieve the same error, you know it is related to the file write.