According to this: http://code.activestate.com/lists/python-list/413540/, tokenize.generate_tokens should be used and not tokenize.tokenize.
This works perfectly fine in Python 2.6. But it does not work anymore in Python 3:
>>> a = list(tokenize.generate_tokens(io.BytesIO("1\n".encode()).readline))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.2/tokenize.py", line 439, in _tokenize
if line[pos] in '#\r\n': # skip comments or blank lines
However, also in Python 3, this works (and returns also the desired output):
a = list(tokenize.tokenize(io.BytesIO("1\n".encode()).readline))
According to the documentation, it seems like tokenize.tokenize is the new way to use this module: http://docs.python.org/py3k/library/tokenize.html. tokenize.generate_tokens isn't even documented anymore.
But, why is there still a generate_tokens function in this module, if it's not documented? I haven't found any PEP regarding this.
I'm trying to maintain a code base for Python 2.5-3.2, should I call generate_tokens for Python 2 and tokenize for Python 3? Aren't there any better ways?
generate_tokens seems to be really a strange thing in Python 3. It doesn't work like in Python 2. However, tokenize.tokenize behaves like the old Python 2 tokenize.generate_tokens. Therefore I wrote a little workaround:
import tokenize
if sys.hexversion >= 0x03000000d:
tokenize_func = tokenize.tokenize
else:
tokenize_func = tokenize.generate_tokens
Now I just use tokenize_func, which works without problems.
generate_tokens in python3 is undocumented but not uncommented. it's there for backward compatibility, so you can use it, but it's probably better to use the changed tokenize instead...
Related
This question already has an answer here:
Why is pyauto gui and Pycharm not working for me?
(1 answer)
Closed last year.
I dont understand why this gives back an error and i'm really confused.
>>> import pyautogui
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/connorgooch/Library/Python/2.7/lib/python/site-packages/pyautogui/__init__.py", line 249, in <module>
import mouseinfo
File "/Library/Python/2.7/site-packages/mouseinfo/__init__.py", line 100, in <module>
from rubicon.objc import ObjCClass, CGPoint
File "/Library/Python/2.7/site-packages/rubicon/objc/__init__.py", line 3, in <module>
from .runtime import ( # noqa: F401
File "/Library/Python/2.7/site-packages/rubicon/objc/runtime.py", line 785
self.restype, *self.argtypes = ctypes_for_method_encoding(self.encoding)
^
SyntaxError: invalid syntax
How do I fix this?
Diagnosis
You're running the library in a version of Python that's too old.
You can tell because assignment of the form ..., *whatever = some_function() became valid only in a relatively recent version of Python.
Prior to that, having a unary * (splat operator) like that on the left side of an assignment was a syntax error.
More specifically, you're using Python 2, which we can tell by both the fact that this feature became available in Python 3.0, and the fact that the traceback has paths like this: /Library/Python/2.7/... (that 2.7 is the Python version).
Solution
Use a newer version of Python.
I can't tell you how to do that because that totally depends on how you're currently running Python.
On some systems, the python command is still Python 2, and you need to call python3 to get Python 3. You might also have a situation where you have separate commands like python27 for Python 2.7, python38 for Python 3.8, and python39 for Python 3.9, but python3 and python point to Python 3.8 instead of either the latest 2 or the latest 3.
Also, on some systems, each Python version is installed in a different directory, and if you specify the right path you don't need to worry about the exact command name.
So if you're calling Python from a shell, you may need to change the PATH or the command or both. If you're calling Python from an IDE, there might be a setting for it. And so on.
Similarly, you might need to actually install a newer version of Python, but the right way to do that depends on your system.
Maybe the package manager provided by your operating system like apt or yum or any number of others, or a third-party package manager like brew or chocolatey, or maybe you're using a tool that manages multiple Python versions. Or maybe you just download it manually from the Python webpage.
Based on your traceback, the fact that some of the paths start with /Library/... suggests that you're on MacOS, which means you can probably just get Python 3 by using the command python3, or changing the path your tool is using to Library/Python/3.? (replace ? with the right minor version). And if you don't already have it installed, you can get it from the Python website or the "homebrew" package manager if you're using that or whatever else works on Mac.
Solution (if you really can't use a newer Python)
You can change the line that's raising the syntax error.
There are many equivalent ways to do this once you understand what's going on.
Here is one pattern which is less readable than most, but which you can use in all cases with no understanding or review or changes:
self.restype, self.argtypes = (lambda r: (r[0], list(r[1:])))(ctypes_for_method_encoding(self.encoding))
Background
In Python, a function can return a tuple or something similar like a list.
This is how returning multiple values is implemented.
In order to make this work on the assignment side, Python can do something called "iterable unpacking": foo, bar = my_function() (this is more generally known by other names too, such as "de-structuring assignment").
(This is also what's happening when you "swap" values: foo, bar = bar, foo is actually semantically equivalent to temp = (bar, foo); foo = temp[0]; bar = temp[1]; del temp. First a tuple is created, and then it goes through de-structuring assignment.)
In Python 3, they added a feature where an assignment with the "splat" operator (*) on one of the values will capture "the rest" of the de-structured values. So if you do temp = (1, 2, 3, 4, 5), then foo, bar, *qux = temp will result in foo = 1, bar = 2, and qux = (3, 4, 5).
But in earlier versions of Python that feature didn't exist.
So the line
self.restype, *self.argtypes = ctypes_for_method_encoding(self.encoding)
is taking the return value from ctypes_for_method_encoding, and assigning the first element of it into self.restype and a tuple holding the rest of the elements from it into self.argtypes.
First try:-
python -m pip install --upgrade pyautogui
If it does not work, then in File "/Library/Python/2.7/site-packages/rubicon/objc/runtime.py", line 785
self.restype, *self.argtypes = ctypes_for_method_encoding(self.encoding)
Try correcting the syntax because you have a syntax error in this line. You can find the syntax on google.
I think you have an error in the asterisk symbol *. But I am not sure.
I was trying to use the os.mknod function in Python 3.5.0 in Windows 7, however I find the error:
Traceback (most recent call last):
File "<pyshell#1>", line 1, in <module>
os.mknod
AttributeError: module 'os' has no attribute 'mknod'
I guess it's supposed to be there, since https://docs.python.org/3/library/os.html doesn't say anything about limited availability. Is there another option to use for a similar function in Windows? I'm just looking to create an empty file in a specific path, and I was thinking calling open(path, 'w') is kinda ugly for this.
I don't know if this might be a version specific problem since I've never used Python in Windows before.
Since commit in 2016, this is now documented:
Availability: Unix.
I'm new here and on Python World (although learning kind of quickly...), and just stumbled uppon the same issue.
My suggestion: for now, I would just go with the following and turn a blind eye on it...
with open('name_your_file.extention', 'w') as an_alias_for_it:
pass
In the end, it's not neat, but will be naturally "portable" among POSIX and NT systems.
Hello everyone i am new to the python language and i have chosen learn python the hard way to learn it and to better my understanding... I am stumped on exercise 25 , When we import the code directly into the terminal
>>> import ex25
>>> sentence = "All good things come to those who wait."
>>> words = ex25.break_words(sentence)
And then I get an attribute error
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'break_words'
I am using python 2.7 on
windows 7 please help..... http://learnpythonthehardway.org/book/ex25.html
It appears to me that the exercise does not instruct the learner to save the file prior to the import. In order for this to work, you've got to save the code that defines the break_words function in a file called ex25.py using your text editor. Then, from the same directory open the python interpreter by typing:
python
and you should be able to import ex25 and run the break_words function which the ex25.py module has defined.
The code in your link for ex25.py does include that function - that yours doesn't suggests that you've somehow missed it when you transcribed the code into your file. Check that your ex25.py includes all the code from the page, and in particular contains this function (it's the very top one):
def break_words(stuff):
"""This function will break up words for us."""
words = stuff.split(' ')
return words
Consider pasting the code into your editor in preference to transcribing it in order to avoid errors like this.
I found (after another question here on StackOverflow) this interesting library written in Python which goal is the grammar parsing.
http://code.google.com/p/modgrammar/
And I also found this tutorial regarding it:
http://packages.python.org/modgrammar/tutorial.html
So, after reading all the tutorial, I understood that this is what I'm looking for! I tried to write the first example in the tutorial:
from modgrammar import *
class MyGrammar (Grammar):
grammar = (LITERAL("Hello,"), LITERAL("world!"))
but I ran into this error:
Traceback (most recent call last):
File "test.py", line 1, in <module>
from modgrammar import *
File "/Users/tesi/Desktop/Prova/modgrammar/__init__.py", line 503
class Grammar(metaclass=GrammarClass):
^
SyntaxError: invalid syntax
The problem seems to be in the metaclass declaration. Might be I have to add a "compilation flag" when I call the python interpreter? Some good news?! :) Thx
class Grammar(metaclass=GrammarClass)
is using Python3 syntax. The equivalent Python2 syntax would be
class Grammar(object):
__metaclass__=GrammarClass
but since there may be lots of other Python3-isms, you may have to use Python3 to use modgrammar.
I've backport modgrammar to Python 2.6+. Backport will also work with Python 3.x. You can find it here: https://github.com/don-ramon/modgrammar-py2.
Yeah, this is Python3 library http://code.google.com/p/modgrammar/source/browse/setup.py#32
Python2x doesn't support class Grammar(metaclass=GrammarClass)
I've come across a bug in Python (at least in 2.6.1) for the bytearray.fromhex function. This is what happens if you try the example from the docstring:
>>> bytearray.fromhex('B9 01EF')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: fromhex() argument 1 must be unicode, not str
This example works fine in Python 2.7, and I want to know the best way of coding around the problem. I don't want to always convert to unicode as it's a performance hit, and testing which Python version is being used feels wrong.
So is there a better way to code around this sort of problem so that it will work for all versions, preferably without slowing it down for the working Pythons?
For cases like this it's good to remember that a try block is very cheap if no exception is thrown. So I'd use:
try:
x = bytearray.fromhex(some_str)
except TypeError:
# Work-around for Python 2.6 bug
x = bytearray.fromhex(unicode(some_str))
This lets Python 2.6 work with a small performance hit, but 2.7 shouldn't suffer at all. It's certainly preferable to checking Python version explicitly!
The bug itself (and it certainly does seem to be one) is still present in Python 2.6.5, but I couldn't find any mention of it at bugs.python.org, so maybe it was fixed by accident in 2.7! It looks like a back-ported Python 3 feature that wasn't tested properly in 2.6.
You can also create your own function to do the work, conditionalized on what you need:
def my_fromhex(s):
return bytearray.fromhex(s)
try:
my_fromhex('hello')
except TypeError:
def my_fromhex(s):
return bytearray.fromhex(unicode(s))
and then use my_fromhex in your code. This way, the exception only happens once, and during your runtime, the correct function is used without excess unicode casting or exception machinery.