I am trying to run a preprocessing pipeline using nipype and I get the following error message:
Traceback (most recent call last):
File "preprocscript.py", line 211, in <module>
preproc.run('MultiProc', plugin_args={'n_procs': 8})
File "/sw/anaconda/3/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 579, in run
runner = plugin_mod(plugin_args=plugin_args)
File "/sw/anaconda/3/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 162, in __init__
initargs=(self._cwd,)
File "/sw/anaconda/3/lib/python3.6/multiprocessing/pool.py", line 175, in __init__
self._repopulate_pool()
File "/sw/anaconda/3/lib/python3.6/multiprocessing/pool.py", line 236, in _repopulate_pool
self._wrap_exception)
File "/sw/anaconda/3/lib/python3.6/multiprocessing/pool.py", line 250, in _repopulate_pool_static
wrap_exception)
File "/sw/anaconda/3/lib/python3.6/multiprocessing/process.py", line 73, in __init__
assert group is None, 'group argument must be None for now'
AssertionError: group argument must be None for now
and I am not sure what exactly might be wrong in my code that leads to this or if this is an issue with my software.
I am on a linux system and use python 3.6.
The module you are using has a ProcessPoolExecuter being used in it. In Python 3.7 they added some additional arguments to that class, namely initargs which is what is being called in nipype multiprocess module you are using. Unfortunately it is not backwards compatible to 3.6 and they did not write in another way to use that module.
Your options are to upgrade or not use the multiprocessing portion of nipype.
Related
I am trying to run a simple multiprocessing example in python3.6 in a zeppelin notebook(in windows) but I am not able to execute it. Below is the code that i used:
def sqrt(x):
return x**0.5
numbers = [i for i in range(1000000)]
with Pool() as pool:
sqrt_ls = pool.map(sqrt, numbers)
After running this code I am getting the following error:
Traceback (most recent call last):
File "/tmp/zeppelin_python-3196160128578820301.py", line 315, in <module>
exec(code, _zcUserQueryNameSpace)
File "<stdin>", line 6, in <module>
File "/usr/lib64/python3.6/multiprocessing/pool.py", line 266, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/lib64/python3.6/multiprocessing/pool.py", line 644, in get
raise self._value
File "/usr/lib64/python3.6/multiprocessing/pool.py", line 424, in _handle_tasks
put(task)
File "/usr/lib64/python3.6/multiprocessing/connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/lib64/python3.6/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function sqrt at 0x7f6f84f1a620>: attribute lookup sqrt on __main__ failed
I am not sure if its just me who is facing the issue. As i have seen so many articles where people can run the code easily. If you know the solution please help
Thanks
From the multiprocessing documentation:
Note: Functionality within this package requires that the main module be importable by the children. This is covered in Programming guidelines however it is worth pointing out here. This means that some examples, such as the Pool examples will not work in the interactive interpreter.
Notebooks are running Python interactive interpreters behind the scene, so it's probably why you get this error. You can try to run your code from within a if __name__ == '__main__': statement.
A Zeppelin notebook does not emulate a normal module well enough to support the pickling that is used to identify the correct operation to another process. You can put all the functions you want to call into a proper module that you import in the usual fashion.
I am running under Python 3.7 on Linux Ubuntu 18.04 under Eclipse 4.8 and Pydev.
The declaration:
args:Dict[str: Optional[Any]] = {}
is in a module that is imported from my testing code. and it is flagged with the following error message from typing.py:
TypeError: Parameters to generic types must be types. Got slice(<class 'str'>, typing.Union[typing.Any, NoneType], None). The stack trace follows: Finding files... done. Importing test modules ... Traceback (most recent call last): File "/Data/WiseOldBird/Eclipse/pool/plugins/org.python.pydev.core_7.0.3.201811082356/pysrc/_pydev_runfiles/pydev_runfiles.py", line 468, in __get_module_from_str
mod = __import__(modname) File "/Data/WiseOldBird/Workspaces/WikimediaAccess/WikimediaLoader/Tests/wiseoldbird/loaders/TestWikimediaLoader.py", line 10, in <module>
from wiseoldbird.application_controller import main File "/Data/WiseOldBird/Workspaces/WikimediaAccess/WikimediaLoader/src/wiseoldbird/application_controller.py", line 36, in <module>
args:Dict[str: Optional[Any]] = {} File "/usr/local/lib/python3.7/typing.py", line 251, in inner
return func(*args, **kwds) File "/usr/local/lib/python3.7/typing.py", line 626, in __getitem__
params = tuple(_type_check(p, msg) for p in params) File "/usr/local/lib/python3.7/typing.py", line 626, in <genexpr>
params = tuple(_type_check(p, msg) for p in params) File "/usr/local/lib/python3.7/typing.py", line 139, in _type_check
raise TypeError(f"{msg} Got {arg!r:.100}.") TypeError: Parameters
This prevents my testing module from being imported.
What am I doing wrong?
The proper syntax for a dict's type is
Dict[str, Optional[Any]]
When you write [a: b], Python interprets this as a slice, i.e. the thing that makes taking parts of arrays work, like a[1:10]. You can see this in the error message: Got slice(<class 'str'>, typing.Union[typing.Any, NoneType], None).
I am trying to run the file l2cap_infra.py with Python 2, but I am getting the following error:
Traceback (most recent call last):
File "l2cap_infra.py", line 524, in <module>
main(*sys.argv[1:])
File "l2cap_infra.py", line 508, in main
l2cap_loop, _ = create_l2cap_connection(src_hci, dst_bdaddr, pcap_path=pcap_path)
File "l2cap_infra.py", line 489, in create_l2cap_connection
handle_information_negotiation_process(l2cap_loop)
File "l2cap_infra.py", line 425, in handle_information_negotiation_process
l2cap_loop.send(info_req)
File "l2cap_infra.py", line 142, in send
self._sock.send(packet)
File "l2cap_infra.py", line 213, in send
self.send_fragment(Raw(str(l2cap)[i:i+L2CAP_DEFAULT_MTU]), i == 0)
File "l2cap_infra.py", line 223, in send_fragment
hci = HCI_Hdr() / HCI_ACL_Hdr(handle=scapy_handle, flags=scapy_flags) / frag
File "/usr/local/lib/python2.7/dist-packages/scapy/base_classes.py", line 227, in __call__
i.__init__(*args, **kargs)
File "/usr/local/lib/python2.7/dist-packages/scapy/packet.py", line 135, in __init__
self.fields[f] = self.get_field(f).any2i(self, v)
File "/usr/local/lib/python2.7/dist-packages/scapy/packet.py", line 170, in get_field
return self.fieldtype[fld]
KeyError: 'flags'
This might be a version conflict; I had a similar problem and I had to edit a file in /usr/local/lib/python2.7/.
What code do I have to change in that linked file or in one of my pip libraries to make this code work?
It seems it's a compatibility issue between BlueBorne and Scapy.
You (most likely) have installed the latest Scapy version (v2.4.0), which dropped the fields kwarg from scapy.layers.bluetooth.HCI_ACL_Hdr's initializer, while BlueBorne (l2cap_infra.py, and possibly others) was not updated (or branched) accordingly.
The latest version that still has it is v2.3.3 ([GitHub]: secdev/scapy - (v2.3.3) scapy/scapy/layers/bluetooth.py).
Possible solutions:
Uninstall your current Scapy version (pip uninstall scapy) and install v2.3.3 (pip install scapy==2.3.3). Probably, this is the simplest (and most suitable) for you ([PyPI]: scapy 2.3.3)
Submit a bug to BlueBorne and wait for them to add support for newer Scapy versions
Fix it yourself ("fields" (v2.3.3) to "PB" + "BC" (v2.4.0) kwargs conversion), and maybe submit a patch :)
Is it possible to use the antlr4 python runtime with python 2.6 or is the minimun required version python 2.7? I want to use it on CentOS 6.3 that comes with python 2.6.6. If it is not possible, is it known what features of python 2.7 that is used?
Finally I can answer this myself. It is not possible to use antlr4-python2 runtime on centos6.3 (that has python 2.6.6), I have installed it and tried this example. The result is as follows:
[lg#localhost py]$ python test.py data.txt
Traceback (most recent call last):
File "test.py", line 15, in <module>
main(sys.argv)
File "test.py", line 10, in main
tree = parser.r()
File "/home/lg/antlr/py/HelloParser.py", line 74, in r
self.enterRule(localctx, 0, self.RULE_r)
File "/usr/lib/python2.6/site-packages/antlr4/Parser.py", line 367, in enterRule
self._ctx.start = self._input.LT(1)
File "/usr/lib/python2.6/site-packages/antlr4/CommonTokenStream.py", line 85, in LT
self.lazyInit()
File "/usr/lib/python2.6/site-packages/antlr4/BufferedTokenStream.py", line 209, in lazyInit
self.setup()
File "/usr/lib/python2.6/site-packages/antlr4/BufferedTokenStream.py", line 212, in setup
self.sync(0)
File "/usr/lib/python2.6/site-packages/antlr4/BufferedTokenStream.py", line 134, in sync
fetched = self.fetch(n)
File "/usr/lib/python2.6/site-packages/antlr4/BufferedTokenStream.py", line 146, in fetch
t = self.tokenSource.nextToken()
File "/usr/lib/python2.6/site-packages/antlr4/Lexer.py", line 131, in nextToken
tokenStartMarker = self._input.mark()
AttributeError: 'builtin_function_or_method' object has no attribute 'mark'
[lg#localhost py]$
The example you used was badly written. input is the name of a built-in Python function. Give the Lexer a FileStream and it might work.
I'm using parallel python to run multiple dynamic simulations using a module called OrcFxAPI. The program works perfectly if it is run as a python program on my machine however if I convert it to an exe file using py2exe and then run I get the following error:
Traceback (most recent call last):
File "Analysis.pyc", Line 500, in multiprocessor
File "pp.pyc", Line 342, in __init__
File "pp.pyc", Line 506, in set_ncpus
File "pp.pyc", Line 140, in __init__
File "pp.pyc", Line 152, in start
File "pptransport.pyc", Line 140, in receive
RuntimeError: Communication pipe read error
It is failing at this line in my program:
job_server = pp.Server(ppservers=ppservers)
but I think it might be something to do with the path used to import the OrcFxAPI module when submitting the job:
job = job_server.submit(max_seastate, (gui_vars, case_list, case, line_info, output_vars), (), ("OrcFxAPI",), callback=finished, callbackargs=(case_no, no_of_cases,))