Trying to write a command line function, and I've been stymied by this AttributeError. I know that other people have asked similar questions but I haven't seen any using plac so I figured I'd write this out.
#plac.annotations(
training_file=("The filename containing the text you wish to annotate", "option", "-tf", Path),
entity_type=("The name of the entity you wish to annotate", "option", "-e", str)
)
def main(training_file=None, entity_type=None):
"""Script to more easily annotate spaCy NER training examples"""
if not training_file:
training_file = input("Please enter the filename of the data you wish to annotate: ")
with open(training_file, 'r') as training_file:
list_to_annotate = training_file.read()
print(list_to_annotate)
and where it's run:
if __name__ == "__main__":
plac.call(main)
There's more to my actual command, but whenever I run this I get the same error message:
Traceback (most recent call last):
File "C:\Users\Steve\PycharmProjects\GroceryListMaker\model_scripts\training_data_maker.py", line 79, in <module>
plac.call(main)
File "C:\Users\Steve\PycharmProjects\GroceryListMaker\lib\site-packages\plac_core.py", line 367, in call
cmd, result = parser.consume(arglist)
File "C:\Users\Steve\PycharmProjects\GroceryListMaker\lib\site-packages\plac_core.py", line 230, in consume
args = [getattr(ns, a) for a in self.argspec.args]
File "C:\Users\Steve\PycharmProjects\GroceryListMaker\lib\site-packages\plac_core.py", line 230, in <listcomp>
args = [getattr(ns, a) for a in self.argspec.args]
AttributeError: 'Namespace' object has no attribute 'training_file'
I'm really not sure what's wrong, and it's making me tear my hair out here. Any help greatly appreciated, thank you.
If you replace it with:
#plac.annotations(
training_file=("The filename containing the text you wish to annotate",
"option", "tf", Path),
entity_type=("The name of the entity you wish to annotate", "option", "e", str)
)
it works (note that I removed the - in the abbreviations).
In the future you can use pdb to track down problems like this more quickly. Here's what I did:
$ python -m pdb main.py
> /home/embray/src/junk/so/60005716/main.py(1)<module>()
-> import plac
(Pdb) cont
Traceback (most recent call last):
File "/usr/lib/python3.6/pdb.py", line 1667, in main
pdb._runscript(mainpyfile)
File "/usr/lib/python3.6/pdb.py", line 1548, in _runscript
self.run(statement)
File "/usr/lib/python3.6/bdb.py", line 434, in run
exec(cmd, globals, locals)
File "<string>", line 1, in <module>
File "/home/embray/src/junk/so/60005716/main.py", line 1, in <module>
import plac
File "/home/embray/.virtualenvs/tmp-954ecd64f7669c29/lib/python3.6/site-packages/plac_core.py", line 367, in call
cmd, result = parser.consume(arglist)
File "/home/embray/.virtualenvs/tmp-954ecd64f7669c29/lib/python3.6/site-packages/plac_core.py", line 230, in consume
args = [getattr(ns, a) for a in self.argspec.args]
File "/home/embray/.virtualenvs/tmp-954ecd64f7669c29/lib/python3.6/site-packages/plac_core.py", line 230, in <listcomp>
args = [getattr(ns, a) for a in self.argspec.args]
AttributeError: 'Namespace' object has no attribute 'training_file'
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /home/embray/.virtualenvs/tmp-954ecd64f7669c29/lib/python3.6/site-packages/plac_core.py(230)<listcomp>()
-> args = [getattr(ns, a) for a in self.argspec.args]
(Pdb) up
> /home/embray/.virtualenvs/tmp-954ecd64f7669c29/lib/python3.6/site-packages/plac_core.py(230)consume()
-> args = [getattr(ns, a) for a in self.argspec.args]
(Pdb) p ns
Namespace(e=None, tf=None)
Here you can see that your argument namespace was replaced with e and tf, suggesting that somehow putting a - in the abbreviation actually replaces the argument name (this was just a guess on my part but it turned out to be correct).
I'd consider that a bit of a bug on plac's part--it's very confusing and the documentation doesn't indicate anything about this.
Related
OmegaConf allows you to register a custom resolver. Here is an example of resolving a tuple.
def resolve_tuple(*args):
return tuple(args)
OmegaConf.register_new_resolver("tuple", resolve_tuple)
This can be used to resolve a value in a config file with a structure like ${tuple:1,2} to a tuple (1, 2). Along with hydra.utils.instantiate this can be used to create objects that contain or utilize tuples. For example:
config.yaml
obj:
tuple: ${tuple:1,2}
test.py
import hydra
import hydra.utils as hu
from omegaconf import OmegaConf
def resolve_tuple(*args):
return tuple(args)
OmegaConf.register_new_resolver('tuple', resolve_tuple)
#hydra.main(config_path='conf', config_name='config_test')
def main(cfg):
obj = hu.instantiate(cfg.obj, _convert_='partial')
print(obj)
if __name__ == '__main__':
main()
Running this example returns:
$ python test.py
{'tuple': (1, 2)}
However, imagine you had a much more complex config structure. You may want to use interpolation to bring in configs from other files like so.
tuple/base.yaml
tuple: ${tuple:1,2}
config.yaml
defaults:
- tuple: base
- _self_
obj:
tuple: ${tuple}
Running this example you get an error:
$ python test.py
Error executing job with overrides: []
Traceback (most recent call last):
File "test.py", line 16, in main
obj = hu.instantiate(cfg.obj, _convert_='partial')
File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 175, in instantiate
OmegaConf.resolve(config)
omegaconf.errors.UnsupportedValueType: Value 'tuple' is not a supported primitive type
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
The full traceback from hydra is:
Error executing job with overrides: []
Traceback (most recent call last):
File "test.py", line 21, in <module>
main()
File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/hydra/main.py", line 52, in decorated_main
config_name=config_name,
File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/hydra/_internal/utils.py", line 378, in _run_hydra
lambda: hydra.run(
File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/hydra/_internal/utils.py", line 214, in run_and_report
raise ex
File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/hydra/_internal/utils.py", line 211, in run_and_report
return func()
File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/hydra/_internal/utils.py", line 381, in <lambda>
overrides=args.overrides,
File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/hydra/_internal/hydra.py", line 111, in run
_ = ret.return_value
File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/hydra/core/utils.py", line 233, in return_value
raise self._return_value
File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/hydra/core/utils.py", line 160, in run_job
ret.return_value = task_function(task_cfg)
File "test.py", line 17, in main
model = hu.instantiate(cfg.obj, _convert_='partial')
File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 175, in instantiate
OmegaConf.resolve(config)
File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/omegaconf/omegaconf.py", line 792, in resolve
omegaconf._impl._resolve(cfg)
File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/omegaconf/_impl.py", line 40, in _resolve
_resolve_container_value(cfg, k)
File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/omegaconf/_impl.py", line 19, in _resolve_container_value
_resolve(resolved)
File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/omegaconf/_impl.py", line 40, in _resolve
_resolve_container_value(cfg, k)
File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/omegaconf/_impl.py", line 23, in _resolve_container_value
node._set_value(resolved._value())
File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/omegaconf/nodes.py", line 44, in _set_value
self._val = self.validate_and_convert(value)
File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/omegaconf/nodes.py", line 57, in validate_and_convert
return self._validate_and_convert_impl(value)
File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/omegaconf/nodes.py", line 134, in _validate_and_convert_impl
f"Value '{t.__name__}' is not a supported primitive type"
omegaconf.errors.UnsupportedValueType: Value 'tuple' is not a supported primitive type
If you really dig around in the omegaconf code in the trace you will find that there is a flag for the config object allow_objects that is True in the example that passes and None in the example that does not. What is interesting is that in the _instantaite2.py file just before calling Omegaconf.resolve(config) several flags are set, one being allow_objects as True.
Is the intended behavior for these interpolated/resolved values populated from separate files to override this flag? If so, is there some way to ensure that the allow_objects flag is (or remains) true for all resolved and interpolated values?
I think there is some confusion because you are using the word tuple for multiple different purposes :)
Here is an example that works for me:
# my_app.py
import hydra
import hydra.utils as hu
from omegaconf import OmegaConf
def resolve_tuple(*args):
return tuple(args)
OmegaConf.register_new_resolver('as_tuple', resolve_tuple)
#hydra.main(config_path='conf', config_name='config')
def main(cfg):
obj = hu.instantiate(cfg.obj, _convert_='partial')
print(obj)
if __name__ == '__main__':
main()
# conf/config.yaml
defaults:
- subdir: base
- _self_
obj:
a_tuple: ${subdir.my_tuple}
# conf/subdir/base.yaml
my_tuple: ${as_tuple:1,2}
$ python my_app.py # at the command line:
{'a_tuple': (1, 2)}
The main difference here is that we've got a_tuple: ${subdir.my_tuple} instead of a_tuple: ${my_tuple}.
Notes:
Tuples may be supported by OmegaConf as a first-class type at some point in the future. Here's the relevant issue: https://github.com/omry/omegaconf/issues/392
The allow_objects flag that you mentioned is undocumented and it's behavior is subject to change.
I'm trying to sniff an existing pcap file, filter it and save it to a new file, but I have this exceptions popping up when I'm running my code.
How can I fix this?
Code:
from scapy.all import *
def write(pcap):
for pkt in pcap:
wrpcap('filtered.pcap', pkt, append=True)
else:
pass
def load_pcap(path, filter_str):
pcap = sniff(offline=path, filter=filter_str)
write(pcap)
def main():
load_pcap("file.pcap", 'icmp')
main()
Exceptions:
Traceback (most recent call last):
File "C:\Users\myUser\AppData\Local\Programs\Python\Python37\lib\site-packages\scapy\utils.py", line 1663, in tcpdump
stderr=stderr,
File "C:\Users\myUser\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 756, in __init__
restore_signals, start_new_session)
File "C:\Users\myUser\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 1100, in _execute_child
args = list2cmdline(args)
File "C:\Users\myUser\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 511, in list2cmdline
needquote = (" " in arg) or ("\t" in arg) or not arg
TypeError: argument of type 'NoneType' is not iterable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "sharkscript.py", line 140, in <module>
main()
File "sharkscript.py", line 137, in main
funcs()
File "sharkscript.py", line 130, in funcs
options()
File "sharkscript.py", line 95, in options
load_pcap(get_filter(), path)
File "sharkscript.py", line 33, in load_pcap
pcap = sniff(offline=path, filter=filter_str)
File "C:\Users\myUser\AppData\Local\Programs\Python\Python37\lib\site-packages\scapy\sendrecv.py", line 972, in sniff
sniffer._run(*args, **kwargs)
File "C:\Users\myUser\AppData\Local\Programs\Python\Python37\lib\site-packages\scapy\sendrecv.py", line 824, in _run
)] = offline
File "C:\Users\myUser\AppData\Local\Programs\Python\Python37\lib\site-packages\scapy\utils.py", line 1663, in tcpdump
stderr=stderr,
File "C:\Users\myUser\AppData\Local\Programs\Python\Python37\lib\site-packages\scapy\utils.py", line 555, in __exit__
raise OSError(msg)
OSError: Could not execute windump(), is it installed ?
I tried searching for windump and how to install it, but couldn't find anything.
Is there another way to filter an 'offline' pcap?
I tried to run your code and got the same error. I think this is a bug in the sniff function, as removing the filter parameter made it work (and it seems to work to others in the past, for example here.
Anyway, if you know the filter in advance you can replace it with haslayer function, something like this-
def load_pcap(path):
f = PcapWriter("out.pcap", append=True, sync=True)
sniff(offline=path, prn=lambda p: f.write(p) if ICMP in p else None)
If you don't know the exact filter but it will be a simple protocol name you can make a mapping between string and Scapy layer, and use it in the same way.
If the filter is more complicated (for example something like tcp.srcport==1234) I'm afraid you'll need to get the parameters from the user separately (for example load_pcap(path, src_mac, dst_mac, src_ip, dst_ip, src_port, dst_port, protocol,...) or find a way to parse the BPF string into parameters.
Hope this helps :)
I am running under Python 3.7 on Linux Ubuntu 18.04 under Eclipse 4.8 and Pydev.
The declaration:
args:Dict[str: Optional[Any]] = {}
is in a module that is imported from my testing code. and it is flagged with the following error message from typing.py:
TypeError: Parameters to generic types must be types. Got slice(<class 'str'>, typing.Union[typing.Any, NoneType], None). The stack trace follows: Finding files... done. Importing test modules ... Traceback (most recent call last): File "/Data/WiseOldBird/Eclipse/pool/plugins/org.python.pydev.core_7.0.3.201811082356/pysrc/_pydev_runfiles/pydev_runfiles.py", line 468, in __get_module_from_str
mod = __import__(modname) File "/Data/WiseOldBird/Workspaces/WikimediaAccess/WikimediaLoader/Tests/wiseoldbird/loaders/TestWikimediaLoader.py", line 10, in <module>
from wiseoldbird.application_controller import main File "/Data/WiseOldBird/Workspaces/WikimediaAccess/WikimediaLoader/src/wiseoldbird/application_controller.py", line 36, in <module>
args:Dict[str: Optional[Any]] = {} File "/usr/local/lib/python3.7/typing.py", line 251, in inner
return func(*args, **kwds) File "/usr/local/lib/python3.7/typing.py", line 626, in __getitem__
params = tuple(_type_check(p, msg) for p in params) File "/usr/local/lib/python3.7/typing.py", line 626, in <genexpr>
params = tuple(_type_check(p, msg) for p in params) File "/usr/local/lib/python3.7/typing.py", line 139, in _type_check
raise TypeError(f"{msg} Got {arg!r:.100}.") TypeError: Parameters
This prevents my testing module from being imported.
What am I doing wrong?
The proper syntax for a dict's type is
Dict[str, Optional[Any]]
When you write [a: b], Python interprets this as a slice, i.e. the thing that makes taking parts of arrays work, like a[1:10]. You can see this in the error message: Got slice(<class 'str'>, typing.Union[typing.Any, NoneType], None).
When I executed in Python command
rssa = importr('Rssa')
I got eroor
Traceback (most recent call last):
File "<pyshell#3>", line 1, in <module>
rssa = importr('Rssa')
File "C:\Python34\lib\site-packages\rpy2\robjects\packages.py", line 412, in importr
version = version)
File "C:\Python34\lib\site-packages\rpy2\robjects\packages.py", line 178, in __init__
self.__fill_rpy2r__(on_conflict = on_conflict)
File "C:\Python34\lib\site-packages\rpy2\robjects\packages.py", line 280, in __fill_rpy2r__
super(SignatureTranslatedPackage, self).__fill_rpy2r__(on_conflict = on_conflict)
File "C:\Python34\lib\site-packages\rpy2\robjects\packages.py", line 233, in __fill_rpy2r__
rpyobj = conversion.ri2ro(riobj)
File "C:\Python34\lib\functools.py", line 707, in wrapper
return dispatch(args[0].__class__)(*args, **kw)
File "C:\Python34\lib\site-packages\rpy2\robjects\__init__.py", line 101, in _
return SignatureTranslatedFunction(obj)
File "C:\Python34\lib\site-packages\rpy2\robjects\functions.py", line 150, in __init__
raise ValueError("Error: '%s' already in the translation table. This means that the signature of the R function contains the parameters '%s' and/or '%s' <sigh> in multiple copies." %(r_param, r_param, prm_translate[py_param]))
ValueError: Error: '...' already in the translation table. This means that the signature of the R function contains the parameters '...' and/or '...' <sigh> in multiple copies.
Other packages are imported without problem, as example
stats = importr('stats')
tseries = importr('tseries')
forecast = importr('forecast')
I was looking for such problem, but I could not find nothing close. Please, suggest some decision of this problem.
This was caused by bug in Rssa when it tries to take control over 'decompose' function (which is exported from 'stats' package). In particular, in the list of formals '...' is added twice and this is what rpy2 is complaining about.
This will be fixed in the subsequent Rssa releases.
The only workaround is to comment out in R/common.R the following line:
formals(decompose.default) <- c(formals(decompose.default), alist(... = ))
And reinstall Rssa from source. This might be non-trivial on Windows, though R windows builder can help here.
Issue 1: When sys.stdout.write is not wrapped in a separate function, the code below fails.
Issue 2: When ssys.stdout.write is wrapped in a separate function, the code prints spaces between each letter.
Code (v1):
#!/usr/bin/env python
import pp
import sys
def main():
server = pp.Server()
for c in "Hello World!\n":
server.submit(sys.stdout.write, (c,), (), ("sys",))()
if __name__=="__main__":
main()
Trace:
$ ./parhello.py
Traceback (most recent call last):
File "./parhello.py", line 15, in <module>
main()
File "./parhello.py", line 12, in main
server.submit(write, (c,), (), ("sys",))()
File "/Library/Python/2.7/site-packages/pp.py", line 461, in submit
sfunc = self.__dumpsfunc((func, ) + depfuncs, modules)
File "/Library/Python/2.7/site-packages/pp.py", line 639, in __dumpsfunc
sources = [self.__get_source(func) for func in funcs]
File "/Library/Python/2.7/site-packages/pp.py", line 706, in __get_source
sourcelines = inspect.getsourcelines(func)[0]
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 688, in getsourcelines
lines, lnum = findsource(object)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 527, in findsource
file = getsourcefile(object)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 446, in getsourcefile
filename = getfile(object)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 422, in getfile
'function, traceback, frame, or code object'.format(object))
TypeError: <built-in method write of file object at 0x1002811e0> is not a module, class, method, function, traceback, frame, or code object
make: *** [test] Error 1
Code (v2):
#!/usr/bin/env python
import pp
import sys
def hello(c):
sys.stdout.write(c)
def main():
server = pp.Server()
for c in "Hello World!\n":
server.submit(hello, (c,), (), ("sys",))()
if __name__=="__main__":
main()
Trace:
$ ./parhello.py
H e l l o W o r l d !
For the first part, pp wasn't designed to handle built-ins as arguments to submit. The second problem is more complicated. Before pp calls the submitted function, it redirects stdout and stderr to a StringIO object. On completing the task, it prints the value from the StringIO object with
print sout,
This means that it appends a space the contents of sout before printing it. To get around this, don't have your functions use sys.stdout directly, but print either to a file or a queue you manage and handle the printing of in a better way.