I have found some vaguely related questions to this question, but not any clean and specific solution for CPython. And I assume that a "valid" solution is interpreter specific.
First the things I think I understand:
locals() gives a non-modifiable dictionary.
A function may (and indeed does) use some kind of optimization to access its local variables
frame.f_locals gives a locals() like dictionary, but less prone to hackish things through exec. Or at least I have been less able to do hackish undocumented things like the locals()['var'] = value ; exec ""
exec is capable to do weird things to the local variables, but it is not reliable --e.g. I read somewhere that it doesn't work in Python 3. Haven't tested.
So I understand that, given those limitations, it will never be safe to add extra variables to the locals, because it breaks the interpreter structure.
However, it should be possible to change a variable already existing, isn't it?
Things that I considered
In a function f, one can access the f.func_code.co_nlocals and f.func_code.co_varnames.
In a frame, the variables can be accessed / checked / read through the frame.f_locals. This is in the use case of setting a tracer through sys.settrace.
One can easily access the function in which a frame is --cosidering the use case of setting a trace and using it to "do things" in with the local variables given a certain trigger or whatever.
The variables should be somewhere, preferably writeable... but I am not capable of finding it. Even if it is an array (for interpreter efficient access), or I need some extra C-specific wiring, I am ready to commit to it.
How can I achieve that modification of variables from a tracer function or from a decorated wrapped function or something like that?
A full solution will be of course appreciated, but even some pointers will help me greatly, because I'm stuck here with lots of non writeable dictionaries :-/
Edit: Hackish exec is doing things like this or this
It exists an undocumented C-API call for doing things like that:
PyFrame_LocalsToFast
There is some more discussion in this PyDev blog post. The basic idea seems to be:
import ctypes
...
frame.f_locals.update({
'a': 'newvalue',
'b': other_local_value,
})
ctypes.pythonapi.PyFrame_LocalsToFast(
ctypes.py_object(frame), ctypes.c_int(0))
I have yet to test if this works as expected.
Note that there might be some way to access the Fast directly, to avoid an indirection if the requirements is only modification of existing variable. But, as this seems to be mostly non-documented API, source code is the documentation resource.
Based on the notes from MariusSiuram, I wrote a recipe that show the behavior.
The conclusions are:
we can modify an existing variable
we can delete an existing variable
we can NOT add a new variable.
So, here is the code:
import inspect
import ctypes
def parent():
a = 1
z = 'foo'
print('- Trying to add a new variable ---------------')
hack(case=0) # just try to add a new variable 'b'
print(a)
print(z)
assert a == 1
assert z == 'foo'
try:
print (b)
assert False # never is going to reach this point
except NameError, why:
print("ok, global name 'b' is not defined")
print('- Trying to remove an existing variable ------')
hack(case=1)
print(a)
assert a == 2
try:
print (z)
except NameError, why:
print("ok, we've removed the 'z' var")
print('- Trying to update an existing variable ------')
hack(case=2)
print(a)
assert a == 3
def hack(case=0):
frame = inspect.stack()[1][0]
if case == 0:
frame.f_locals['b'] = "don't work"
elif case == 1:
frame.f_locals.pop('z')
frame.f_locals['a'] += 1
else:
frame.f_locals['a'] += 1
# passing c_int(1) will remove and update variables as well
# passing c_int(0) will only update
ctypes.pythonapi.PyFrame_LocalsToFast(
ctypes.py_object(frame),
ctypes.c_int(1))
if __name__ == '__main__':
parent()
The output would be like:
- Trying to add a new variable ---------------
1
foo
ok, global name 'b' is not defined
- Trying to remove an existing variable ------
2
foo
- Trying to update an existing variable ------
3
Related
I realize the question is a bit specific, sorry.
I'm trying to solve a python puzzle where what I input get exec'ed. My aim is to store something of arbitrary length in the global name space (e.g, change the variable target). You'll see below that my input is limited to 35 chars.
Here's the program:
#!/usr/bin/env python2
from sys import modules
modules.clear()
del modules
raw_input2 = raw_input
exception2 = Exception
__builtins__.__dict__.clear()
__builtins__ = None
target = None # change this !
while 1:
try:
scope = {'r':0}
exec 'r = ' + raw_input2()[:35] in scope
print 'Result:', scope['r']
except exception2, e:
print "Error: ", e
As said, my point is to store data somewhere, since vars get reset each loop.
I know I could use globals()['target']=xxx, but the builtins were disabled.
In theory I could use global target;target=xxx but this executes in scope, not in the global scope (also I think the global keyword must come at the beginning of the anonymous function)
I know all variables are stored in some __dict__ object, but the only way I know to write in it is via globals()['target']=xxx which is disabled
I know you can painfully access the super-object with ().__class__.__base__, but with the limit of 35 characters, it doesn't seem to be the way (this string alone is 21 chars already, you need two more to start with 0;, so only 11 char remaining to assign something...).
Any ideas ?
Thanks!
The answer is: write in __builtins__.
Example input:
1;__builtins__['a']="string1"
Result: 1
1;__builtins__['a']+="string2"
... which actually seems very simple since it is right above the variable marked # change this in the question. I do not fully understand yet why the __builtins__ var is passed in the exec scope, especially since it should be None (and hence not a dictionary), but is assignable like a dictionary.
But it works, tested in python 2.7
I'm writing a small app in Django and I'm keeping the state saved in a few variables I declare out of the methods in views.py. Here is the important part of this file:
from app.playerlist import fullList
auc_unsold = fullList[:]
auc_teams = []
auc_in_progress = []
auc_current_turn = -1
print(auc_in_progress)
def auc_action(request):
data = json.loads(request.GET["data"])
# ...
elif data[0] == "start":
random.shuffle(auc_teams)
print(auc_unsold)
print(auc_in_progress)
auc_in_progress = [None, 0, None]
print(auc_in_progress)
The auc_unsold and auc_teams variables work fine; the auc_in_progress variable is not seen by this method, though, giving the error in the title. If I take out the print statement and let this code assign a value to it, the exception will be thrown somewhere else in the code as soon as I use that variable again.
I have tried making another variable and this new one seems to suffer from this problem as well.
What is happening?
Edit: I found a solution: if I write global auc_in_progress just before the print statements, then everything works fine. If I try writing that as I declare the variable above it doesn't work, though, for some reason.
I am unsatisfied with this, because I don't know why this happens and because I dislike using global like that, but eh. Someone has an explanation?
You should absolutely not be doing this, either your original code or your proposed solution with global.
Anything at module level will be shared across requests, not only for the current user but for all users for that process. So everyone will see the same auction, etc.
The reason for your error is because you assign to that variable within your function, which automatically makes it a local variable: see this question for more details. But the solution recommended there, which is the same as your workaround - ie use global - is not appropriate here; you should store the data somewhere specifically associated with the user, eg the session.
I am writing some program using python and the z3py module.
What I am trying to do is the following: I extract a constraint of an if or a while statement from a function which is located in some other file. Additionally I extract the used variables in the statement as well as their types.
As I do not want to parse the constraint by hand into a z3py friendly form, I tried to use evaluate to do this for me. Therefore I used the tip of the following page: Z3 with string expressions
Now the problem is: I do not know how the variables in the constraint are called. But it seems as I have to name the handle of each variable like the actual variable. Otherwise evaluate won't find it. My code looks like this:
solver = Solver()
# Look up the constraint:
branch = bd.getBranchNum(0)
constr = branch.code
# Create handle for each variable, depending on its type:
for k in mapper.getVariables():
var = mapper.getVariables()[k]
if k in constr:
if var.type == "intNum":
Int(k)
else:
Real(k)
# Evaluate constraint, insert the result and solve it:
f = eval(constr)
solver.insert(f)
solve(f)
As you can see I saved the variables and constraints in classes. When executing this code I get the following error:
NameError: name 'real_x' is not defined
If I do not use the looping over the variables, but instead the following code, everything works fine:
solver = Solver()
branch = bd.getBranchNum(0)
constr = branch.code
print(constr)
real_x = Real('real_x')
int_y = Int('int_y')
f = eval(constr)
print(f)
solver.insert(f)
solve(f)
The problem is: I do not know, that the variables are called "real_x" or "int_y". Furthermore I do not know how many variables there are used, which means I have to use some dynamic thing like a loop.
Now my question is: Is there a way around this? What can I do to tell python that the handles already exist, but have a different name? Or is my approach completely wrong and I have to do something totally different?
This kind of thing is almost always a bad idea (see Why eval/exec is bad for more details), but "almost always" isn't "always", and it looks like you're using a library that was specifically designed to be used this way, in which case you've found one of the exceptions.
And at first glance, it seems like you've also hit one of the rare exceptions to the Keep data out of your variable names guideline (also see Why you don't want to dynamically create variables). But you haven't.
The only reason you need these variables like real_x to exist is so that eval can see them, right? But the eval function already knows how to look for variables in a dictionary instead of in your global namespace. And it looks like what you're getting back from mapper.getVariables() is a dictionary.
So, skip that whole messy loop, and just do this:
variables = mapper.getVariables()
f = eval(constr, globals=variables)
(In earlier versions of Python, globals is a positional-only argument, so just drop the globals= if you get an error about that.)
As the documentation explains, this gives the eval function access to your actual variables, plus the ones the mapper wants to generate, and it can do all kinds of unsafe things. If you want to prevent unsafe things, do this:
variables = dict(mapper.getVariables())
variables['__builtins__'] = {}
f = eval(constr, globals=variables)
I find myself doing something like this constantly to pull GET args into vars:
some_var = self.request.get('some_var', None)
other_var = self.request.get('other_var', None)
if None in [some_var, other_var]:
logging.error("some arg was missing in " + self.request.path)
exit()
What I would really want to do is:
pull_args('some_var', 'other_var')
And that would somehow pull these variables to be available in current scope, or log an error and exit if not (or return to calling method if possible). Is this possible in Python?
First, a disclaimer: "pulling" variables into the local scope in any way other than var = something is really really really not recommended. It tends to make your code really confusing for someone who isn't intimately familiar with what you're doing (i.e. anyone who isn't you, or who is you 6 months in the future, etc.)
That being said, for educational purposes only, there is a way. Your pull_args function could be implemented like this:
def pull_args(request, *args):
pulled = {}
try:
for a in args:
pulled[a] = request[a]
except AttributeError:
logging.error("some arg was missing in " + self.request.path)
exit()
else:
caller = inspect.stack()[1][0]
caller.f_locals.update(pulled)
At least, something to that effect worked when I came up with it probably about a year ago. I wouldn't necessarily count on it continuing to work in future Python versions. (Yet another reason not to do it) I personally have never found a good reason to use this code snippet.
No it's not and also pointless. Writing to outer namespaces completely destroys the purpose of namespaces, which is having only the things around that you explicitly set. Use lists!
def pull_args(*names):
return [self.request.get(name, None) for name in names]
print None in pull_args('some_var', 'other_var')
Probably this works too, to check if all _var are set:
print all(name in self.request for name in ('some_var', 'other_var'))
I know this must be a trivial question, but I've tried many different ways, and searched quie a bit for a solution, but how do I create and reference subfunctions in the current module?
For example, I am writing a program to parse through a text file, and for each of the 300 different names in it, I want to assign to a category.
There are 300 of these, and I have a list of these structured to create a dict, so of the form lookup[key]=value (bonus question; any more efficient or sensible way to do this than a massive dict?).
I would like to keep all of this in the same module, but with the functions (dict initialisation, etc) at the
end of the file, so I dont have to scroll down 300 lines to see the code, i.e. as laid out as in the example below.
When I run it as below, I get the error 'initlookups is not defined'. When I structure is so that it is initialisation, then function definition, then function use, no problem.
I'm sure there must be an obvious way to initialise the functions and associated dict without keeping the code inline, but have tried quite a few so far without success. I can put it in an external module and import this, but would prefer not to for simplicity.
What should I be doing in terms of module structure? Is there any better way than using a dict to store this lookup table (It is 300 unique text keys mapping on to approx 10 categories?
Thanks,
Brendan
import ..... (initialisation code,etc )
initLookups() # **Should create the dict - How should this be referenced?**
print getlookup(KEY) # **How should this be referenced?**
def initLookups():
global lookup
lookup={}
lookup["A"]="AA"
lookup["B"]="BB"
(etc etc etc....)
def getlookup(value)
if name in lookup.keys():
getlookup=lookup[name]
else:
getlookup=""
return getlookup
A function needs to be defined before it can be called. If you want to have the code that needs to be executed at the top of the file, just define a main function and call it from the bottom:
import sys
def main(args):
pass
# All your other function definitions here
if __name__ == '__main__':
exit(main(sys.argv[1:]))
This way, whatever you reference in main will have been parsed and is hence known already. The reason for testing __name__ is that in this way the main method will only be run when the script is executed directly, not when it is imported by another file.
Side note: a dict with 300 keys is by no means massive, but you may want to either move the code that fills the dict to a separate module, or (perhaps more fancy) store the key/value pairs in a format like JSON and load it when the program starts.
Here's a more pythonic ways to do this. There aren't a lot of choices, BTW.
A function must be defined before it can be used. Period.
However, you don't have to strictly order all functions for the compiler's benefit. You merely have to put your execution of the functions last.
import # (initialisation code,etc )
def initLookups(): # Definitions must come before actual use
lookup={}
lookup["A"]="AA"
lookup["B"]="BB"
(etc etc etc....)
return lookup
# Any functions initLookups uses, can be define here.
# As long as they're findable in the same module.
if __name__ == "__main__": # Use comes last
lookup= initLookups()
print lookup.get("Key","")
Note that you don't need the getlookup function, it's a built-in feature of a dict, named get.
Also, "initialisation code" is suspicious. An import should not "do" anything. It should define functions and classes, but not actually provide any executable code. In the long run, executable code that is processed by an import can become a maintenance nightmare.
The most notable exception is a module-level Singleton object that gets created by default. Even then, be sure that the mystery object which makes a module work is clearly identified in the documentation.
If your lookup dict is unchanging, the simplest way is to just make it a module scope variable. ie:
lookup = {
'A' : 'AA',
'B' : 'BB',
...
}
If you may need to make changes, and later re-initialise it, you can do this in an initialisation function:
def initLookups():
global lookup
lookup = {
'A' : 'AA',
'B' : 'BB',
...
}
(Alternatively, lookup.update({'A':'AA', ...}) to change the dict in-place, affecting all callers with access to the old binding.)
However, if you've got these lookups in some standard format, it may be simpler simply to load it from a file and create the dictionary from that.
You can arrange your functions as you wish. The only rule about ordering is that the accessed variables must exist at the time the function is called - it's fine if the function has references to variables in the body that don't exist yet, so long as nothing actually tries to use that function. ie:
def foo():
print greeting, "World" # Note that greeting is not yet defined when foo() is created
greeting = "Hello"
foo() # Prints "Hello World"
But:
def foo():
print greeting, "World"
foo() # Gives an error - greeting not yet defined.
greeting = "Hello"
One further thing to note: your getlookup function is very inefficient. Using "if name in lookup.keys()" is actually getting a list of the keys from the dict, and then iterating over this list to find the item. This loses all the performance benefit the dict gives. Instead, "if name in lookup" would avoid this, or even better, use the fact that .get can be given a default to return if the key is not in the dictionary:
def getlookup(name)
return lookup.get(name, "")
I think that keeping the names in a flat text file, and loading them at runtime would be a good alternative. I try to stick to the lowest level of complexity possible with my data, starting with plain text and working up to a RDMS (I lifted this idea from The Pragmatic Programmer).
Dictionaries are very efficient in python. It's essentially what the whole language is built on. 300 items is well within the bounds of sane dict usage.
names.txt:
A = AAA
B = BBB
C = CCC
getname.py:
import sys
FILENAME = "names.txt"
def main(key):
pairs = (line.split("=") for line in open(FILENAME))
names = dict((x.strip(), y.strip()) for x,y in pairs)
return names.get(key, "Not found")
if __name__ == "__main__":
print main(sys.argv[-1])
If you really want to keep it all in one module for some reason, you could just stick a string at the top of the module. I think that a big swath of text is less distracting than a huge mess of dict initialization code (and easier to edit later):
import sys
LINES = """
A = AAA
B = BBB
C = CCC
D = DDD
E = EEE""".strip().splitlines()
PAIRS = (line.split("=") for line in LINES)
NAMES = dict((x.strip(), y.strip()) for x,y in PAIRS)
def main(key):
return NAMES.get(key, "Not found")
if __name__ == "__main__":
print main(sys.argv[-1])