For the following Python code
def add_func(a,b):
print(a+b)
a = 2
b = 3
add_func(a,b)
pylint will state
W0621: Redefining name 'a' from outer scope (line 4) (redefined-outer-name)
...
I can rename it as (perhaps due to a and b outside function will interfere add_func)
a_input = 2
b_input = 3
add_func(a_input,b_input)
to get rid of the message from pylint. But, _input looks somehow lengthy. Is there any recommended coding practice to get rid of the outer scope message from pylint?
the main usecase of a function is to pass arguments directly into it, so in this case the simplest way to input those would be:
add_func(2,3)
If you want to pass in varibales, name them after their usecase for readability purposes (It is generally not recommended to have a single char as variable name, with a few exceptions like i for loops)
Related
This question is similar to others asked on here, but after reading the answers I'm not grasping it and would appreciate further guidance.
While sketching new code I find myself adding a lot of statements like:
print('var=')
pprint(var)
It became tedious always writing that, so I thought I could make it into a function. Since I want to print the variable name on the preceding line, I tried:
def dbp(var):
eval('print(\'{0}=\')'.format(var))
eval('pprint({0})'.format(var))
so then I do do things like:
foo = 'bar'
dbp('foo')
which prints
foo=
'bar'
This is all great, but when I go to use it in a function things get messed up. For example, doing
def f():
a = ['123']
dbp('a')
f()
raises a NameError (NameError: name 'a' is not defined).
My expectation was that dbp() would have read access to anything in f()'s scope, but clearly it doesn't. Can someone explain why?
Also, better ways of printing a variable's name followed by its formatted contents are also appreciated.
You really should look at other ways to doing this.
The logging module is a really good habit to get into, and you can turn off and on debug output.
Python 3.6 has f'' strings so you would simplify this to:
pprint(f'var=\n{var}`)`
However, here's an example (not recommended) using locals():
In []:
def dbp(var, l):
print('{}='.format(var))
pprint(l[var])
def f():
a = 1
dbp('a', locals())
f()
Out[]:
a=
1
first of all, id like to say that eval is a high security risk for whoever is going to be running that code.
However, if you absolutely must, you can do this.
def dbp(var):
env = {'var': var}
# Adding global variables to the enviroment
env.update(globals())
eval("print('{0}=')".format(var))
eval('pprint(var)', env)
def f():
a = ['123']
dbp('a')
you can then do
>>> f()
a=
'a'
I have found some vaguely related questions to this question, but not any clean and specific solution for CPython. And I assume that a "valid" solution is interpreter specific.
First the things I think I understand:
locals() gives a non-modifiable dictionary.
A function may (and indeed does) use some kind of optimization to access its local variables
frame.f_locals gives a locals() like dictionary, but less prone to hackish things through exec. Or at least I have been less able to do hackish undocumented things like the locals()['var'] = value ; exec ""
exec is capable to do weird things to the local variables, but it is not reliable --e.g. I read somewhere that it doesn't work in Python 3. Haven't tested.
So I understand that, given those limitations, it will never be safe to add extra variables to the locals, because it breaks the interpreter structure.
However, it should be possible to change a variable already existing, isn't it?
Things that I considered
In a function f, one can access the f.func_code.co_nlocals and f.func_code.co_varnames.
In a frame, the variables can be accessed / checked / read through the frame.f_locals. This is in the use case of setting a tracer through sys.settrace.
One can easily access the function in which a frame is --cosidering the use case of setting a trace and using it to "do things" in with the local variables given a certain trigger or whatever.
The variables should be somewhere, preferably writeable... but I am not capable of finding it. Even if it is an array (for interpreter efficient access), or I need some extra C-specific wiring, I am ready to commit to it.
How can I achieve that modification of variables from a tracer function or from a decorated wrapped function or something like that?
A full solution will be of course appreciated, but even some pointers will help me greatly, because I'm stuck here with lots of non writeable dictionaries :-/
Edit: Hackish exec is doing things like this or this
It exists an undocumented C-API call for doing things like that:
PyFrame_LocalsToFast
There is some more discussion in this PyDev blog post. The basic idea seems to be:
import ctypes
...
frame.f_locals.update({
'a': 'newvalue',
'b': other_local_value,
})
ctypes.pythonapi.PyFrame_LocalsToFast(
ctypes.py_object(frame), ctypes.c_int(0))
I have yet to test if this works as expected.
Note that there might be some way to access the Fast directly, to avoid an indirection if the requirements is only modification of existing variable. But, as this seems to be mostly non-documented API, source code is the documentation resource.
Based on the notes from MariusSiuram, I wrote a recipe that show the behavior.
The conclusions are:
we can modify an existing variable
we can delete an existing variable
we can NOT add a new variable.
So, here is the code:
import inspect
import ctypes
def parent():
a = 1
z = 'foo'
print('- Trying to add a new variable ---------------')
hack(case=0) # just try to add a new variable 'b'
print(a)
print(z)
assert a == 1
assert z == 'foo'
try:
print (b)
assert False # never is going to reach this point
except NameError, why:
print("ok, global name 'b' is not defined")
print('- Trying to remove an existing variable ------')
hack(case=1)
print(a)
assert a == 2
try:
print (z)
except NameError, why:
print("ok, we've removed the 'z' var")
print('- Trying to update an existing variable ------')
hack(case=2)
print(a)
assert a == 3
def hack(case=0):
frame = inspect.stack()[1][0]
if case == 0:
frame.f_locals['b'] = "don't work"
elif case == 1:
frame.f_locals.pop('z')
frame.f_locals['a'] += 1
else:
frame.f_locals['a'] += 1
# passing c_int(1) will remove and update variables as well
# passing c_int(0) will only update
ctypes.pythonapi.PyFrame_LocalsToFast(
ctypes.py_object(frame),
ctypes.c_int(1))
if __name__ == '__main__':
parent()
The output would be like:
- Trying to add a new variable ---------------
1
foo
ok, global name 'b' is not defined
- Trying to remove an existing variable ------
2
foo
- Trying to update an existing variable ------
3
I am writing some program using python and the z3py module.
What I am trying to do is the following: I extract a constraint of an if or a while statement from a function which is located in some other file. Additionally I extract the used variables in the statement as well as their types.
As I do not want to parse the constraint by hand into a z3py friendly form, I tried to use evaluate to do this for me. Therefore I used the tip of the following page: Z3 with string expressions
Now the problem is: I do not know how the variables in the constraint are called. But it seems as I have to name the handle of each variable like the actual variable. Otherwise evaluate won't find it. My code looks like this:
solver = Solver()
# Look up the constraint:
branch = bd.getBranchNum(0)
constr = branch.code
# Create handle for each variable, depending on its type:
for k in mapper.getVariables():
var = mapper.getVariables()[k]
if k in constr:
if var.type == "intNum":
Int(k)
else:
Real(k)
# Evaluate constraint, insert the result and solve it:
f = eval(constr)
solver.insert(f)
solve(f)
As you can see I saved the variables and constraints in classes. When executing this code I get the following error:
NameError: name 'real_x' is not defined
If I do not use the looping over the variables, but instead the following code, everything works fine:
solver = Solver()
branch = bd.getBranchNum(0)
constr = branch.code
print(constr)
real_x = Real('real_x')
int_y = Int('int_y')
f = eval(constr)
print(f)
solver.insert(f)
solve(f)
The problem is: I do not know, that the variables are called "real_x" or "int_y". Furthermore I do not know how many variables there are used, which means I have to use some dynamic thing like a loop.
Now my question is: Is there a way around this? What can I do to tell python that the handles already exist, but have a different name? Or is my approach completely wrong and I have to do something totally different?
This kind of thing is almost always a bad idea (see Why eval/exec is bad for more details), but "almost always" isn't "always", and it looks like you're using a library that was specifically designed to be used this way, in which case you've found one of the exceptions.
And at first glance, it seems like you've also hit one of the rare exceptions to the Keep data out of your variable names guideline (also see Why you don't want to dynamically create variables). But you haven't.
The only reason you need these variables like real_x to exist is so that eval can see them, right? But the eval function already knows how to look for variables in a dictionary instead of in your global namespace. And it looks like what you're getting back from mapper.getVariables() is a dictionary.
So, skip that whole messy loop, and just do this:
variables = mapper.getVariables()
f = eval(constr, globals=variables)
(In earlier versions of Python, globals is a positional-only argument, so just drop the globals= if you get an error about that.)
As the documentation explains, this gives the eval function access to your actual variables, plus the ones the mapper wants to generate, and it can do all kinds of unsafe things. If you want to prevent unsafe things, do this:
variables = dict(mapper.getVariables())
variables['__builtins__'] = {}
f = eval(constr, globals=variables)
I have a question that I am sure has been on the mind of every intermediate-level Python programmer at some point: that is, how to fix/prevent/avoid/work around those ever-so-persistent and equally frustrating NameErrors. I'm not talking about actual errors (like typos, etc.), but a bizarre problem that basically say a global name was not defined, when in reality it was defined further down. For whatever reason, Python seems to be extremely needy in this area: every single variable absolutely positively has to hast to be defined above and only above anything that refers to it (or so it seems).
For example:
condition = True
if condition == True:
doStuff()
def doStuff():
it_worked = True
Causes Python to give me this:
Traceback (most recent call last):
File "C:\Users\Owner\Desktop\Python projects\test7.py", line 4, in <module>
doStuff()
NameError: name 'doStuff' is not defined
However, the name WAS defined, just not where Python apparently wanted it. So for a cheesy little function like doStuff() it's no big deal; just cut and paste the function into an area that satisfies the system's requirement for a certain order. But when you try to actually design something with it it makes organizing code practically impossible (I've had to "un-organize" tons of code to accomodate this bug). I have never encountered this problem with any of the other languages I've written in, so it seems to be specific to Python... but anyway I've researched this in the docs and haven't found any solutions (or even potential leads to a possible solution) so I'd appreciate any tips, tricks, workarounds or other suggestions.
It may be as simple as learning a specific organizational structure (like some kind of "Pythonic" and very strategic approach to working around the bug), or maybe just use a lot of import statements so it'll be easier to organize those in a specific order that will keep the system from acting up...
Avoid writing code (other than declarations) at top-level, use a main() function in files meant to be executed directly:
def main():
condition = True
if condition:
do_stuff()
def do_stuff():
it_worked = True
if __name__ == '__main__':
main()
This way you only need to make sure that the if..main construct follows the main() function (e.g. place it at the end of the file), the rest can be in any order. The file will be fully parsed (and thus all the names defined in the module can be resolved) by the time main() is executed.
As a rule of thumb: For most cases define all your functions first and then use them later in your code.
It is just the way it is: every name has to be defined at the time it is used.
This is especially true at code being executed at top level:
func()
def func():
func2()
def func2():
print "OK"
func()
The first func() will fail, because it is not defined yet.
But if I call func() at the end, everything will be OK, although func2() is defined after func().
Why? Because at the time of calling, func2() exists.
In short, the code of func() says "Call whatever is defined as func2 at the time of calling".
In Python defining a function is an act which happens at runtime, not at compile time. During that act, the code compiled at compile time is assigned to the name of the function. This name then is a variable in the current scope. It can be overwritten later as any other variable can:
def f():
print 42
f() # will print 42
def f():
print 23
f() # will print 23
You can even assign functions like other values to variables:
def f():
print 42
g = 23
f() # will print 42
g # will print 23
f, g = g, f
f # will print 23
g() # will print 42
When you say that you didn't come across this in other languages, it's because the other languages you are referring to aren't interpreted as a script. Try similar things in bash for instance and you will find that things can be as in Python in other languages as well.
There are a few things to say about this:
If your code is so complex that you can't organize it in one file, think about using many files and import them into one smaller main file
I you put your function in a class it will work. example:
class test():
def __init__(self):
self.do_something()
def do_something(self):
print 'test'
As said in the comment from Volatility that is an characteristic of interpreted languages
Consider this example:
def outer():
s_outer = "outer\n"
def inner():
s_inner = "inner\n"
do_something()
inner()
I want the code in do_something to be able to access the variables of the calling functions further up the call stack, in this case s_outer and s_inner. More generally, I want to call it from various other functions, but always execute it in their respective context and access their respective scopes (implement dynamic scoping).
I know that in Python 3.x, the nonlocal keyword allows access to s_outer from within inner. Unfortunately, that only helps with do_something if it's defined within inner. Otherwise, inner isn't a lexically enclosing scope (similarly, neither is outer, unless do_something is defined within outer).
I figured out how to inspect stack frames with the standard library inspect, and made a small accessor that I can call from within do_something() like this:
def reach(name):
for f in inspect.stack():
if name in f[0].f_locals:
return f[0].f_locals[name]
return None
and then
def do_something():
print( reach("s_outer"), reach("s_inner") )
works just fine.
Can reach be implemented more simply? How else can I solve the problem?
There is no and, in my opinion, should be no elegant way of implementing reach since that introduces a new non-standard indirection which is really hard to comprehend, debug, test and maintain. As the Python mantra (try import this) says:
Explicit is better than implicit.
So, just pass the arguments. You-from-the-future will be really grateful to you-from-today.
What I ended up doing was
scope = locals()
and make scope accessible from do_something. That way I don't have to reach, but I can still access the dictionary of local variables of the caller. This is quite similar to building a dictionary myself and passing it on.
We can get naughtier.
This is an answer to the "Is there a more elegant/shortened way to implement the reach() function?" half of the question.
We can give better syntax for the user: instead of reach("foo"), outer.foo.
This is nicer to type, and the language itself immediately tells you if you used a name that can't be a valid variable (attribute names and variable names have the same constraints).
We can raise an error, to properly distinguish "this doesn't exist" from "this was set to None".
If we actually want to smudge those cases together, we can getattr with the default parameter, or try-except AttributeError.
We can optimize: no need to pessimistically build a list big enough for all the frames at once.
In most cases we probably won't need to go all the way to the root of the call stack.
Just because we're inappropriately reaching up stack frames, violating one of the most important rules of programming to not have things far away invisibly effecting behavior, doesn't mean we can't be civilized.
If someone is trying to use this Serious API for Real Work on a Python without stack frame inspection support, we should helpfully let them know.
import inspect
class OuterScopeGetter(object):
def __getattribute__(self, name):
frame = inspect.currentframe()
if frame is None:
raise RuntimeError('cannot inspect stack frames')
sentinel = object()
frame = frame.f_back
while frame is not None:
value = frame.f_locals.get(name, sentinel)
if value is not sentinel:
return value
frame = frame.f_back
raise AttributeError(repr(name) + ' not found in any outer scope')
outer = OuterScopeGetter()
Excellent. Now we can just do:
>>> def f():
... return outer.x
...
>>> f()
Traceback (most recent call last):
...
AttributeError: 'x' not found in any outer scope
>>>
>>> x = 1
>>> f()
1
>>> x = 2
>>> f()
2
>>>
>>> def do_something():
... print(outer.y)
... print(outer.z)
...
>>> def g():
... y = 3
... def h():
... z = 4
... do_something()
... h()
...
>>> g()
3
4
Perversion elegantly achieved.
Is there a better way to solve this problem? (Other than wrapping the respective data into dicts and pass these dicts explicitly to do_something())
Passing the dicts explicitly is a better way.
What you're proposing sounds very unconventional. When code increases in size, you have to break down the code into a modular architecture, with clean APIs between modules. It also has to be something that is easy to comprehend, easy to explain, and easy to hand over to another programmer to modify/improve/debug it. What you're proposing sounds like it is not a clean API, unconventional, with a non-obvious data flow. I suspect it would probably make many programmers grumpy when they saw it. :)
Another option would be to make the functions members of a class, with the data being in the class instance. That could work well if your problem can be modelled as several functions operating on the data object.