Some strange problems occurred when executing the codes:
# a lot more
bank_info = p.map(itr_banks, range(len(bank_sites)))
bank_info = sum(bank_info, [])
# a lot more
Stack trace:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/local/Cellar/python#3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
......
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
......
What makes it strange is that the almost same code run a year ago doesn't throw any error. The project was created a year ago and this time is a slight refactor. The python version may be changed but still using PyCharm.
Any suggestions?
Related
I know that this questions seems repeated but the existing questions didn't apply to my case. I don't have any file named multiprocess.pool in my directory and I still get this error when trying to run the traffic generator.
Traceback (most recent call last):
File "run.py", line 1, in <module>
import generator
File "/home/alaa/synthetic_traffic_generator-master/synthetic_traffic_generator-master/generator.py", line 13, in <module>
from multiprocessing.pool import Pool, Process
ImportError: cannot import name 'Process' from 'multiprocessing.pool' (/usr/lib/python3.8/multiprocessing/pool.py)
This is the part of the code where it uses Process:
def generate_synthethic_users_and_traffic(number_of_users=NUMBER_OF_SYNTHETIC_USERS):
# Makes the random numbers predictable, to make the experiment reproducible.
seed(1234)
user_generator = UserDistribution(number_of_users)
# generate_and_write_synthetic_traffic is a very light method, thus it
# is not necessary to create a pool of processes and join them later on
for user in user_generator.users():
Process(target=user.generate_and_write_synthetic_traffic).start()
I believe this parts needs to be updated by I have no idea how. Any help with this issue is appreciated.
Thanks in advance.
EDIT:
I followed the first answer and now the error changed to this:
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/alaa/synthetic_traffic_generator-master/synthetic_traffic_generator-master/generator.py", line 348, in generate_and_write_synthetic_traffic
self.generate_synthetic_traffic()
File "/home/alaa/synthetic_traffic_generator-master/synthetic_traffic_generator-master/generator.py", line 265, in generate_synthetic_traffic
for hour, traffic_model in self.traffic_model_per_hour.iteritems():
AttributeError: 'dict' object has no attribute 'iteritems'
EDIT 2:
I followed this question to solve the second issue and now it works.
There is no multiprocessing.pool.Process And the Github Repository you are following is 7 Years Old! Which hasn't been updated since then. And not compatible with the current version of python So, it's obvious to expect errors like this. But,
You can try replacing that import code block in generator.py line 13 which is from multiprocessing.pool import Pool, Process delete that line and add the followings:
from multiprocessing import Pool, Process
I'm facing a strange situation, I've searched on google without any good results.
I'm running a python script as a subprocess from a parent subprocess with nohup using subprocess package:
cmd = list()
cmd.append("nohup")
cmd.append(sys.executable)
cmd.append(os.path.abspath(script))
cmd.append(os.path.abspath(conf_path))
_env = os.environ.copy()
if env:
_env.update({k: str(v) for k, v in env.items()})
p = subprocess.Popen(cmd, env=_env, cwd=os.getcwd())
After some time the parent process exists and the subprocess (the one with the nohup continues to run).
After another minute or two the process with the nohup exits, and with obvious reasons, becomes a zombie.
When running it on local PC with python3.6 and ubuntu 18.04, I manage to run the following code and everything works like a charm:
comp_process = psutil.Process(pid)
if comp_process.status() == "zombie":
comp_status_code = comp_process.wait(timeout=10)
As I said, everything works like a charm, The zombie process removed and I got the status code of the mentioned process.
But for some reason, when doing the SAME at docker container with the SAME python version and Ubuntu version, It fails after the timeout (Doesn't matter if its 10 seconds or 10 minutes)
The error:
psutil.TimeoutExpired timeout after 60 seconds (pid=779)
Traceback (most recent call last): File
"/usr/local/lib/python3.6/dist-packages/psutil/_psposix.py", line 84,
in wait_pid
retpid, status = waitcall() File "/usr/local/lib/python3.6/dist-packages/psutil/_psposix.py", line 75,
in waitcall
return os.waitpid(pid, os.WNOHANG) ChildProcessError: [Errno 10] No child processes
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File ".py", line 41, in
run
comp_status_code = comp_process.wait(timeout=60) File "/usr/local/lib/python3.6/dist-packages/psutil/init.py", line
1383, in wait
return self._proc.wait(timeout) File "/usr/local/lib/python3.6/dist-packages/psutil/_pslinux.py", line
1517, in wrapper
return fun(self, *args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/psutil/_pslinux.py", line
1725, in wait
return _psposix.wait_pid(self.pid, timeout, self._name) File "/usr/local/lib/python3.6/dist-packages/psutil/_psposix.py", line 96,
in wait_pid
delay = check_timeout(delay) File "/usr/local/lib/python3.6/dist-packages/psutil/_psposix.py", line 68,
in check_timeout
raise TimeoutExpired(timeout, pid=pid, name=proc_name) psutil.TimeoutExpired: psutil.TimeoutExpired timeout after 60 seconds
(pid=779)
One possibility may be the lack of an init process to reap zombies. You can fix this by running with docker run --init, or using e.g. tini. See https://hynek.me/articles/docker-signals/
I am working on a multiprocessing module in python. I have a code similar to below which works perfectly in Ubuntu but not in Windows. As i want to run it on windows server, I have to make it work on windows.
# Importing libraries
from multiprocessing import Process
def testFunc(value=0):
print("I am in test function", value)
class testClass1(Process):
def __init__(self):
super(testClass1, self).__init__()
def run(self):
testFunc(1)
print("Done ran testFunc and back to testClass1")
if __name__ == "__main__":
t1 = testClass1()
t1.start()
Below is the error/warning message i am getting on windows platform
Traceback (most recent call last):
File "", line 1, in
File "c:\users\user1\appdata\local\continuum\anaconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "c:\users\user1\appdata\local\continuum\anaconda3\lib\multiprocessing\spawn.py", line 115, in _main
self = reduction.pickle.load(from_parent)
AttributeError: Can't get attribute 'testClass1' on
Let me know how to resolve this error in windows.
I tried searching for similar issues on web but could not find solution which resolves this.
Also, If you think this question is duplicate point me to the original question and let me get this resolved before closing the question.
I have a script called client that does a few things around the OS and it's failing when it comes to /proc/1/cwd
I'm trying to ignore these types of files but the script is still crashing when it comes to this file. I get Memory Map output and also a Stack Trace before the script crashes.
I did a file /proc/1/cwd command on the file and I can see it's a symlink, where I then included an if os.path.islink(file) is False: IF statement to skip it and proceed.
However, I seem to still be hitting it.
The below is the Traceback after the crash.
Traceback (most recent call last):
File "client.py", line 235, in <module>
File "client.py", line 120, in runner
OSError: [Errno 13] Permission denied: '/proc/1/cwd'
Failed to execute script client
The error suggests the script client is working on /proc/1/cwd and then right after seems to say that it can't execute itself?
Any help would be appreciated.
-----Additional -------
Code lines from error:
like 235 is literally the line runner(path)
runner() represents a function that has a for loop that cycles through files in a folder to find the correct file. During this process, runner will do the following:
do line 120 mode = os.stat(fullfilename).st_mode
check if os.path.isdir(fullfilename) is False:
then if stat.S_ISREG(mode) is True:
then if os.path.islink(fullfilename) is False:
then if fullfilename not in blacklist:
Line 120 is mode = os.stat(fullfilename).st_mode
I'm going to dump the error code I got while try a python script :
Preprocess validation data upfront
Using gpu device 0: Tesla K20c
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\SciSoft\WinPython-64bit-2.7.6.4\python-2.7.6.amd64\lib\multiprocessing\forking.py", line 380, in main
prepare(preparation_data)
File "C:\SciSoft\WinPython-64bit-2.7.6.4\python-2.7.6.amd64\lib\multiprocessing\forking.py", line 495, in prepare
'__parents_main__', file, path_name, etc
File "C:\Users\Administrator\Desktop\Galaxy Data\kaggle-galaxies-master\kaggle-galaxies-master\try_convnet_cc_multirotflip_3x69r45_maxout2048_extradense.py", line 133, in <module>
for data, length in create_valid_gen():
File "load_data.py", line 572, in buffered_gen_mp
process.start()
`File "C:\SciSoft\WinPython-64bit-2.7.6.4\python-2.7.6.amd64\lib\multiprocessing\process.py", line 130, in start
self._popen = Popen(self)
File "C:\SciSoft\WinPython-64bit-2.7.6.4\python-2.7.6.amd64\lib\multiprocessing\forking.py", line 258, in init
cmd = get_command_line() + [rhandle]
File "C:\SciSoft\WinPython-64bit-2.7.6.4\python-2.7.6.amd64\lib\multiprocessing\forking.py", line 358, in get_command_line`
is not going to be frozen to produce a Windows executable.''')
RuntimeError:
Attempt to start a new process before the current process
has finished its bootstrapping phase.
This probably means that you are on Windows and you have
forgotten to use the proper idiom in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce a Windows executable.
As I understand I have to insert a line
if __name__ == '__main__':
Some where to get it to work
Can anyone tell me in which File I should insert it ? I have included the affected files list in the initial error logs
The affected file :
https://github.com/benanne/kaggle-galaxies/blob/master/try_convnet_cc_multirotflip_3x69r45_maxout2048_extradense.py
Lines 131-134
and
https://github.com/benanne/kaggle-galaxies/blob/master/load_data.py
line 572
Python documentation is quite clear in this case.
The important part is Safe importing of main module.
Your try_convnet_cc_multirotflip_3x69r45_maxout2048_extradense.py script is doing lots of things on module level. Without reading it in details I can already say that you should wrap the workflow with a function and use it like this:
if __name__ == '__main__':
freeze_support() # Optional under circumstances described in docs
your_workflow_function()
Besides the problem you have, it's a good habit not to surprise possible user of your script with side effects, if the user just wants to import it and reuse some of it's functionality.
So don't put your code on module level. It's ok to have constants on module level but the workflow should be in functions and classes.
If Python module is intended to be used as a script (like in your case), you simply put the if __name__ == '__main__' in the very end of this module, calling your_workflow_function() only if the module is the entry point for the interpreter - so called main module.