Develop Reference

Error in python asterisk syntax. No idea what is wrong

I'm trying to locally minimize a complicated function using scipy.optimize.minimize. Since I require good gradients in order for the local optimization to be smoothily performed, but the function is so very much complicated in order for the derivatives to be written by hand, I decided to use Autoptim as the middle-man to handle my optimization using the automatic differentiation package Autograd to obtain the gradients.
After I installed the package (as well as Autograd), I opened my python terminal in order to run a few preliminary tests to check whether the installation and the package integration between scipy, autograd and autoptim went smoothily. Then Autoptim raised an error immediately upon import (at the line import autoptim). Since the interpreter gives the full stack of Exceptions raised, I went to the deeper layers to see what line initiated the cascade that halted the interpreter.
The line I found was line 88 of autoptim.py:
87. optim_vars = _convert_to_tuple(optim_vars)
88. precon_optim_vars = precon_fwd(*optim_vars,*args)
89. n_args = len(args)
Python interpreter raised an Invalid Syntax Exception, which means that something in that line is not written "in Python". I checked to see whether there was some unclosed parenthesis and that was not the case. I am using Python 3 so I figured that maybe something on that line was written in Python 2 syntax and it registers wrong for a Python 3 interpreter but as I understand it, the differences between the two versions is quite small and there is some (although not complete) retrocompatibility between the two.
So what gives? What am I missing here?
What is wrong with that line?
Here is the traceback of the import line in the python interpreter
>>> import autoptim
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.4/dist-packages/autoptim/__init__.py", line 6, in <module>
from .autoptim import minimize # noqa
File "/usr/local/lib/python3.4/dist-packages/autoptim/autoptim.py", line 95
return objective_function(*optim_vars, *args)
^
SyntaxError: invalid syntax

The syntax being used wasn't introduced until Python 3.5 (see PEP 448). You are using Python 3.4.
As a workaround, you could explicitly build the required list to unpack:
return objective_function(*list(optim_vars + args))

pympler raises TypeError

In python2.7, following the pympler example:
from anotherfile import somefunction, somecustomclass
from os import path, listdir
import pandas as pd
import gc
from pympler import tracker, muppy, summary
all_objects = muppy.get_objects()
print 'all objects: ', len(all_objects)
sum1 = summary.summarize(all_objects)
summary.print_(sum1)
This is the first code after the imports. It results in
/usr/bin/python2.7 /myprog.py
all objects: 98755
Traceback (most recent call last):
File "/myprog.py", line 12, in <module>
sum1 = summary.summarize(all_objects)
File "/usr/local/lib/python2.7/dist-packages/pympler/summary.py", line 131, in summarize
total_size[otype] = _getsizeof(o)
File "/usr/local/lib/python2.7/dist-packages/pandas/core/base.py", line 130, in __sizeof__
return super(self, PandasObject).__sizeof__()
TypeError: super() argument 1 must be type, not FrozenList
Process finished with exit code 1
I get the same error when I try to initialize a SummaryTracker object.
It looks like a bug in Pympler, but the fact that I can't find any mentions of it contradicts this. According to the official documentation, "Pympler is written entirely in Python, with no dependencies to external libraries. It has been tested with Python 2.5, 2.6, 2.7, 3.1, 3.2, 3.3, 3.4 on Linux, Windows and MacOS X." In fact, running only the following code with python 2.7 in a new python file does not produce any errors and works as expected:
from pympler import muppy, tracker
tr = tracker.SummaryTracker()
tr.print_diff()
So what am I missing?

it seems to be a problem in pandas library. I solved it by editing the library code. The track of the error indicates you which line is wrong:
File "/usr/local/lib/python2.7/dist-packages/pandas/core/base.py", line 130, in __sizeof__
return super(self, PandasObject).__sizeof__()
You just have to change the order of the parameters like this:
return super(PandasObject, self).__sizeof__()
I did it and I was able to run normally my program.

This is Pandas issue #12924. PandasObject.__sizeof__ had the arguments in the wrong order for the super call. The fix has been pulled, and it should be available in the next release. In the meantime, you could edit pandas/core/base.py to switch the argument order, or you could test for the bug's presence and monkey-patch the method with a corrected version.

Probably the code is supposed to work with Python3, only. In this question, there's the same error for python2 and if you look at the accepted response it says:
super() (without arguments) was introduced in python3:
super() -> same as super(class, )
so that would be the python2 equivalent for new-style classes:
super(CurrentClass, self)
You'll probably have to use python3 if you don't want to alter the library code.

Code not compiling in Python3

I wrote a code and it is getting compiled on my PC with Python3. But Showing error while uploading to Codechef server. Please suggest, I am coding for the first time in Python 3.
Traceback (most recent call last):
File "/run-ls7W2DcLmzUs9GNKbLGN/solution.py", line 41, in <module>
l,r,k=map(int,input().split())
File "<string>", line 1
9 23 1
^
SyntaxError: invalid syntax

You're using python2.x which evaluates the string you enter for input. Change the function from input to raw_input and you should be all set.
If you want the code to work for both python2.x and python3.x, you could do a simple little hack like this at the top of your script:
try:
#This raises `NameError` on python3.x since `raw_input` is renamed to `input`
input = raw_input
except NameError:
pass
It's not pretty, but it works (and I've used things like this on occasion). Ultimately, this shadows the builtin input on python2.x, but that's really not a big deal. You probably don't want to be using that builtin for any serious coding anyway.

Python 3: tokenize library changes

According to this: http://code.activestate.com/lists/python-list/413540/, tokenize.generate_tokens should be used and not tokenize.tokenize.
This works perfectly fine in Python 2.6. But it does not work anymore in Python 3:
>>> a = list(tokenize.generate_tokens(io.BytesIO("1\n".encode()).readline))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.2/tokenize.py", line 439, in _tokenize
if line[pos] in '#\r\n': # skip comments or blank lines
However, also in Python 3, this works (and returns also the desired output):
a = list(tokenize.tokenize(io.BytesIO("1\n".encode()).readline))
According to the documentation, it seems like tokenize.tokenize is the new way to use this module: http://docs.python.org/py3k/library/tokenize.html. tokenize.generate_tokens isn't even documented anymore.
But, why is there still a generate_tokens function in this module, if it's not documented? I haven't found any PEP regarding this.
I'm trying to maintain a code base for Python 2.5-3.2, should I call generate_tokens for Python 2 and tokenize for Python 3? Aren't there any better ways?

generate_tokens seems to be really a strange thing in Python 3. It doesn't work like in Python 2. However, tokenize.tokenize behaves like the old Python 2 tokenize.generate_tokens. Therefore I wrote a little workaround:
import tokenize
if sys.hexversion >= 0x03000000d:
tokenize_func = tokenize.tokenize
else:
tokenize_func = tokenize.generate_tokens
Now I just use tokenize_func, which works without problems.

generate_tokens in python3 is undocumented but not uncommented. it's there for backward compatibility, so you can use it, but it's probably better to use the changed tokenize instead...

There is no spawnl function in python 2.6?

I just noticed that my old codes written in python 2.5 does not work now. I am in python 2.6 btw.
>>> os.spawnl(os.P_NOWAIT,"setup.exe")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "c:\python26\lib\os.py", line 612, in spawnl
return spawnv(mode, file, args)
OSError: [Errno 22] Invalid argument
>>>
Any clue? or do you have any working sample of os.spawn* with NOWAIT option.
Update:
Even I put full path in os.spawnl(), Its still error.

thrope is right about subprocess being preferred. But the spawn* stuff is still there in 2.6. In fact, you can see that in your error message. Your first arg seems to be valid. I'd check the second arg, which is the path.

I got it work by adding DUMMY parameter finally, a bit funky though
This is not working
os.spawnl(os.P_NOWAIT,"Setup.exe")
This is also not working
os.spawnl(os.P_NOWAIT,"Setup.exe","")
But this is working
os.spawnl(os.P_NOWAIT,"Setup.exe","DUMMY")
Thanks all anyway.

I think its recommended to use the subprocess module these days rather than the os.spawn* functions. (I can't reproduce your problem, but I'm not on windows).

A Google search brings up this page about the same problem happening when there is a space in the Python installation path. I couldn't reproduce it here, but maybe it's the problem?
In any case, according to MS documentation this error value (EINVAL) should only be returned if the mode argument is invalid, which isn't the case here.

os.spawnl() requires full path to executable, while os.spawnlp() uses PATH environment variable to find it.
Update: Also it's common error to use unescaped backslashes in the path literal (try printing it to see whether it's interpreted right).