I am writing some program using python and the z3py module.
What I am trying to do is the following: I extract a constraint of an if or a while statement from a function which is located in some other file. Additionally I extract the used variables in the statement as well as their types.
As I do not want to parse the constraint by hand into a z3py friendly form, I tried to use evaluate to do this for me. Therefore I used the tip of the following page: Z3 with string expressions
Now the problem is: I do not know how the variables in the constraint are called. But it seems as I have to name the handle of each variable like the actual variable. Otherwise evaluate won't find it. My code looks like this:
solver = Solver()
# Look up the constraint:
branch = bd.getBranchNum(0)
constr = branch.code
# Create handle for each variable, depending on its type:
for k in mapper.getVariables():
var = mapper.getVariables()[k]
if k in constr:
if var.type == "intNum":
Int(k)
else:
Real(k)
# Evaluate constraint, insert the result and solve it:
f = eval(constr)
solver.insert(f)
solve(f)
As you can see I saved the variables and constraints in classes. When executing this code I get the following error:
NameError: name 'real_x' is not defined
If I do not use the looping over the variables, but instead the following code, everything works fine:
solver = Solver()
branch = bd.getBranchNum(0)
constr = branch.code
print(constr)
real_x = Real('real_x')
int_y = Int('int_y')
f = eval(constr)
print(f)
solver.insert(f)
solve(f)
The problem is: I do not know, that the variables are called "real_x" or "int_y". Furthermore I do not know how many variables there are used, which means I have to use some dynamic thing like a loop.
Now my question is: Is there a way around this? What can I do to tell python that the handles already exist, but have a different name? Or is my approach completely wrong and I have to do something totally different?
This kind of thing is almost always a bad idea (see Why eval/exec is bad for more details), but "almost always" isn't "always", and it looks like you're using a library that was specifically designed to be used this way, in which case you've found one of the exceptions.
And at first glance, it seems like you've also hit one of the rare exceptions to the Keep data out of your variable names guideline (also see Why you don't want to dynamically create variables). But you haven't.
The only reason you need these variables like real_x to exist is so that eval can see them, right? But the eval function already knows how to look for variables in a dictionary instead of in your global namespace. And it looks like what you're getting back from mapper.getVariables() is a dictionary.
So, skip that whole messy loop, and just do this:
variables = mapper.getVariables()
f = eval(constr, globals=variables)
(In earlier versions of Python, globals is a positional-only argument, so just drop the globals= if you get an error about that.)
As the documentation explains, this gives the eval function access to your actual variables, plus the ones the mapper wants to generate, and it can do all kinds of unsafe things. If you want to prevent unsafe things, do this:
variables = dict(mapper.getVariables())
variables['__builtins__'] = {}
f = eval(constr, globals=variables)
Related
Can you run a for loop over the names of multiple subsets?
For instance, I now have subsets dfVC1 up until dfVC20 and I would like to do something like:
for x in range(20):
print(dfVC[x])
I get this doesn't work... but wonder if there is a way to do this.
I'm going to assume your 'subsets' in this case are variables, named dbVC0, dbVC1, etc. Then, your problem is that you want to print all of them by number, but since they're variables, you can't.
One way to solve this would be to change how the 'subsets' are declared. Instead of
dfVC0 = ...
dfVC1 = ...
you could make one dfVC variable that's a dict, that holds all the others at their proper indices.
dfVC = {}
dfVC[0] = ...
dfVC[1] = ...
which would then allow you to access the various dbVC subsets in the way you're currently trying to.
But changing such a large part of the program isn't always possible. What you might be able to do instead is to figure out which object the dfVCs are attached to, and grab them by string.
If they're in the local namespace (i.e. were declared in the same function as you're currently executing in), you can call the built-in locals() to get a dict that you can then try to find your key in:
for x in range(20):
sname = f'dfVC{x}'
print(locals()[sname])
globals() can be used similarly, if your 'subsets' are in the global scope (i.e. declared outside of the current function).
And if your dfVC variables are attached to a class or module (or something else that behaves like a namespace), you can retrieve them using the built-in getattr() function:
for x in range(20):
sname = f'dfVC{x}'
print(getattr(self, sname)) # replace self with whichever object has the dbVC attached to it
I would like to use a function's parameter to create dynamic names of dataframes and/or objects in Python. I have about 40 different names so it would be really elegant to do this in a function. Is there a way to do this or do I need to do this via 'dict'? I read that 'exec' is dangerous (not that I could get this to work). SAS has this feature for their macros which is where I am coming from. Here is an example of what I am trying to do (using '#' for illustrative purposes):
def TrainModels (mtype):
model_#mtype = ExtraTreesClassifier()
model_#mtype.fit(X_#mtype, Y_#mtype)
TrainModels ('FirstModel')
TrainModels ('SecondModel')
You could use a dictionary for this:
models = {}
def TrainModels (mtype):
models[mtype] = ExtraTreesClassifier()
models[mtype].fit()
First of all, any name you define within your TrainModels function will be local to that function, so won't be accessible in the rest of your program. So you have to define a global name.
Everything in Python is a dictionary, including the global namespace. You can define a new global name dynamically as follows:
my_name = 'foo'
globals()[my_name] = 'bar'
This is terrible and you should never do it. It adds too much indirection to your code. When someone else (or yourself in 3 months when the code is no longer fresh in your mind) reads the code and see 'foo' used elsewhere, they'll have a hard time figuring out where it came from. Code analysis tools will not be able to help you.
I would use a dict as Milkboat suggested.
I'm writing a small app in Django and I'm keeping the state saved in a few variables I declare out of the methods in views.py. Here is the important part of this file:
from app.playerlist import fullList
auc_unsold = fullList[:]
auc_teams = []
auc_in_progress = []
auc_current_turn = -1
print(auc_in_progress)
def auc_action(request):
data = json.loads(request.GET["data"])
# ...
elif data[0] == "start":
random.shuffle(auc_teams)
print(auc_unsold)
print(auc_in_progress)
auc_in_progress = [None, 0, None]
print(auc_in_progress)
The auc_unsold and auc_teams variables work fine; the auc_in_progress variable is not seen by this method, though, giving the error in the title. If I take out the print statement and let this code assign a value to it, the exception will be thrown somewhere else in the code as soon as I use that variable again.
I have tried making another variable and this new one seems to suffer from this problem as well.
What is happening?
Edit: I found a solution: if I write global auc_in_progress just before the print statements, then everything works fine. If I try writing that as I declare the variable above it doesn't work, though, for some reason.
I am unsatisfied with this, because I don't know why this happens and because I dislike using global like that, but eh. Someone has an explanation?
You should absolutely not be doing this, either your original code or your proposed solution with global.
Anything at module level will be shared across requests, not only for the current user but for all users for that process. So everyone will see the same auction, etc.
The reason for your error is because you assign to that variable within your function, which automatically makes it a local variable: see this question for more details. But the solution recommended there, which is the same as your workaround - ie use global - is not appropriate here; you should store the data somewhere specifically associated with the user, eg the session.
I have found some vaguely related questions to this question, but not any clean and specific solution for CPython. And I assume that a "valid" solution is interpreter specific.
First the things I think I understand:
locals() gives a non-modifiable dictionary.
A function may (and indeed does) use some kind of optimization to access its local variables
frame.f_locals gives a locals() like dictionary, but less prone to hackish things through exec. Or at least I have been less able to do hackish undocumented things like the locals()['var'] = value ; exec ""
exec is capable to do weird things to the local variables, but it is not reliable --e.g. I read somewhere that it doesn't work in Python 3. Haven't tested.
So I understand that, given those limitations, it will never be safe to add extra variables to the locals, because it breaks the interpreter structure.
However, it should be possible to change a variable already existing, isn't it?
Things that I considered
In a function f, one can access the f.func_code.co_nlocals and f.func_code.co_varnames.
In a frame, the variables can be accessed / checked / read through the frame.f_locals. This is in the use case of setting a tracer through sys.settrace.
One can easily access the function in which a frame is --cosidering the use case of setting a trace and using it to "do things" in with the local variables given a certain trigger or whatever.
The variables should be somewhere, preferably writeable... but I am not capable of finding it. Even if it is an array (for interpreter efficient access), or I need some extra C-specific wiring, I am ready to commit to it.
How can I achieve that modification of variables from a tracer function or from a decorated wrapped function or something like that?
A full solution will be of course appreciated, but even some pointers will help me greatly, because I'm stuck here with lots of non writeable dictionaries :-/
Edit: Hackish exec is doing things like this or this
It exists an undocumented C-API call for doing things like that:
PyFrame_LocalsToFast
There is some more discussion in this PyDev blog post. The basic idea seems to be:
import ctypes
...
frame.f_locals.update({
'a': 'newvalue',
'b': other_local_value,
})
ctypes.pythonapi.PyFrame_LocalsToFast(
ctypes.py_object(frame), ctypes.c_int(0))
I have yet to test if this works as expected.
Note that there might be some way to access the Fast directly, to avoid an indirection if the requirements is only modification of existing variable. But, as this seems to be mostly non-documented API, source code is the documentation resource.
Based on the notes from MariusSiuram, I wrote a recipe that show the behavior.
The conclusions are:
we can modify an existing variable
we can delete an existing variable
we can NOT add a new variable.
So, here is the code:
import inspect
import ctypes
def parent():
a = 1
z = 'foo'
print('- Trying to add a new variable ---------------')
hack(case=0) # just try to add a new variable 'b'
print(a)
print(z)
assert a == 1
assert z == 'foo'
try:
print (b)
assert False # never is going to reach this point
except NameError, why:
print("ok, global name 'b' is not defined")
print('- Trying to remove an existing variable ------')
hack(case=1)
print(a)
assert a == 2
try:
print (z)
except NameError, why:
print("ok, we've removed the 'z' var")
print('- Trying to update an existing variable ------')
hack(case=2)
print(a)
assert a == 3
def hack(case=0):
frame = inspect.stack()[1][0]
if case == 0:
frame.f_locals['b'] = "don't work"
elif case == 1:
frame.f_locals.pop('z')
frame.f_locals['a'] += 1
else:
frame.f_locals['a'] += 1
# passing c_int(1) will remove and update variables as well
# passing c_int(0) will only update
ctypes.pythonapi.PyFrame_LocalsToFast(
ctypes.py_object(frame),
ctypes.c_int(1))
if __name__ == '__main__':
parent()
The output would be like:
- Trying to add a new variable ---------------
1
foo
ok, global name 'b' is not defined
- Trying to remove an existing variable ------
2
foo
- Trying to update an existing variable ------
3
I have a whole series of arrays with similar names mcmcdata.rho0, mcmcdata.rho1, ... and I want to be able to loop through them while updating their values. I can't figure out how this might be done or even what such a thing might be called.
I read my data in from file like this:
names1='l b rho0 rho1 rho2 rho3 rho4 rho5 rho6 rho7 rho8 rho9 rho10 rho11 rho12 rho13 rho14 rho15 rho16 rho17 rho18 rho19 rho20 rho21 rho22 rho23'.split()
mcmcdata=np.genfromtxt(filename,names=names1,dtype=None).view(np.recarray)
and I want to update the "rho" arrays later on after I do some calculations.
for jj in range(dbins):
mcmc_x, mcmc_y, mcmc_z = wf.lbd_to_xyz(mcmcdata.l,mcmcdata.b,d[jj],R_sun)
rho, thindisk, thickdisk, halo = wf.total_density_fithRthinhRthickhzthinhzthickhrfRiA( mcmc_x, mcmc_y, mcmc_z, R_sun,params)
eval("mcmcdata."+names1[2+jj]) = copy.deepcopy(rho)
eval("mcmcthin."+names1[2+jj]) = copy.deepcopy(thindisk)
eval("mcmcthick."+names1[2+jj]) = copy.deepcopy(thickdisk)
eval("mcmchalo."+names1[2+jj]) = copy.deepcopy(halo)
But the eval command is giving an error:
File "<ipython-input-133-30322c5e633d>", line 13
eval("mcmcdata."+names1[2+jj]) = copy.deepcopy(rho)
SyntaxError: can't assign to function call
How can I loop through my existing arrays and update their values?
or
How can identify the arrays by name so I can update them?
The eval command doesn't work the way you seem to think it does. You appear to be using it like a text-replacement macro, hoping that Python will read the given string and then pretend you wrote that text in the original source code. Instead, it receives a string, and then it executes that code. You're giving it an expression that refers to an attribute of an object, which is fine, but the result of evaluating that expression does not yield a thing you can assign to. It yields the value of that attribute.
Although Python provides eval, it also provides many other things that often obviate the need for eval. In the case of your code, Python provides setattr. You give it an object, the name of an attribute on that object, and a value, and it assigns that object's attribute to refer to the given value.
setattr(mcmcdata, names1[2+jj], copy.deepcopy(rho))
It might make the code more readable to get rid of the names1 portion, too. I might write the code like this:
setattr(mcmcdata, 'rho' + str(jj), copy.deepcopy(rho))
That way, it's clear that I'm assigning the rho-related attributes of the object without having to go look at what's held in the names1 list; the name names1 doesn't offer much information about what's in it.