Develop Reference

Dynamically creating variables without using a dict

Use case - I am taking python code created in another system, and breaking it up into individual functions and connecting them together. The entire point of this work is to break up large python functions that we did not write into smaller python functions for many business reasons.
I COULD take the code, parse for variables, and arbitrarily put them in a dict when doing this, but that is more than a teeny bit of work, and I'd like to run this to ground before I do.
I understand we should almost never but I need to because I am code generating wrappers for functions I did not write, I need to dynamically create variables inside a function. I also can't use exec because the value could be a complex structure (e.g., a dict).
So, the point of what we're doing is to ask the original authors to make no changes to the incoming code while still executing it across several independent entities.
Just like in the example listed here - we're capturing as much state as we can with the first exit (ideally functions, lambdas and all variables), and re-instating them in the second function so that two functions which formerly had the same scope and context can execute with no changes.
Here is a single block of reproducible code (everything not related to b is code that I can use to wrap the assignment:
Original:
def original_function():
b = 100
b = b + 20
Resulting generated function:
def fun_1() -> str:
import dill
from base64 import urlsafe_b64decode, urlsafe_b64encode
from types import ModuleType
b = 100
locals_keys = frozenset(locals().keys())
global_keys = frozenset(globals().keys())
__context_export = {}
for val in locals_keys:
if not val.startswith("_") and not isinstance(val, ModuleType):
__context_export[val] = dill.dumps(locals()[val])
for val in global_keys:
if not val.startswith("_") and not isinstance(val, ModuleType):
__context_export[val] = dill.dumps(globals()[val])
b64_string = str(urlsafe_b64encode(dill.dumps(__context_export)), encoding="ascii")
from collections import namedtuple
output = namedtuple("FuncOutput", ["context"])
return output(b64_string)
def fun_2(context):
import dill
from base64 import urlsafe_b64encode, urlsafe_b64decode
from types import ModuleType
__base64_decode = urlsafe_b64decode(context)
__context_import_dict = dill.loads(__base64_decode)
for k in __context_import_dict:
val = dill.loads(__context_import_dict[k])
if globals().get(k) is None and not isinstance(val, ModuleType):
globals()[k] = val
b = b + 20
output = fun_1()
fun_2(output[0])
The error I get when I run this is:
UnboundLocalError: local variable 'b' referenced before assignment
Thank you all for the help!

Ok, this was a pretty easy solve after I understood the issues. To be honest, this makes even more sense - since I'm getting the code from externally (as a string), it makes sense that I should mount in the necessary global variables and exec inside closed environment.
TO BE CLEAR - this is executing inside the USER'S environment, so security is not an issue. But this works!
def fun_1() -> str:
import dill
from base64 import urlsafe_b64decode, urlsafe_b64encode
from types import ModuleType, FunctionType
# CODE FROM EXTERNAL
b = 100
# END CODE
locals_keys = frozenset(locals().keys())
global_keys = frozenset(globals().keys())
__context_export = {}
for val in locals_keys:
if (
not val.startswith("_")
and not isinstance(val, ModuleType)
and not isinstance(val, FunctionType)
):
__context_export[val] = dill.dumps(locals()[val])
for val in global_keys:
if (
not val.startswith("_")
and not isinstance(val, ModuleType)
and not isinstance(val, FunctionType)
):
__context_export[val] = dill.dumps(globals()[val])
b64_string = str(urlsafe_b64encode(dill.dumps(__context_export)), encoding="ascii")
from collections import namedtuple
output = namedtuple("FuncOutput", ["context"])
return output(b64_string)
def fun_2(context):
import dill
from base64 import urlsafe_b64encode, urlsafe_b64decode
from types import ModuleType, FunctionType
from pprint import pprint as pp
__base64_decode = urlsafe_b64decode(context)
__context_import_dict = dill.loads(__base64_decode)
variables = {}
for k in __context_import_dict:
variables[k] = dill.loads(__context_import_dict[k])
loc = {}
# CODE FROM EXTERNAL
inner_code_to_execute "b = b + 20"
# END CODE
exec(inner_code_to_execute, variables, loc)
print(loc["b"])
output = fun_1()
fun_2(output[0])

Python - How to obtain a dictionary(or even a list) of variables in a specific scope; more specific than locals/globals()

So, the title pretty much says it all.
for instance let's look at the below example code:
## How can I obtain a dict/list (like locals()) of all the variables in second and/or third layer scopes via a command
# coming from the first layer?
## Or another example would be how could I obtain the variables "locals() style" from the thirdlayer via a
# command from the second layer?
# essentially can a parent function/class access a list/dict of a child function
# or class??
def firstLayer():
a = 4.7
q = locals()
print(q)
# local vars of 1st layer
def secondlayer():
b = 7
r = locals()
print(r)
# local vars of 2nd layer
def thirdlayer():
c = False
s = locals()
i = globals()
print('c:\n', c, "\nglobals from 3rd layer:\n\t", i)
# local vars of 3rd layer
thirdlayer()
secondlayer()
firstLayer()
sample_var = globals()
print(sample_var)
# returns the list of global variables
to reiterate what I said in the comments in the code, essentially is their any way I can get a list of all the variables local to a 'child' scope? I know functions are shut off, but if their is no way to do this is their any more complicated code that could achieve this and I could integrate it into a function or class if necessary.
EDIT:
To elaborate further; here's the situation i'm in.
def varsfunc():
font1 = "Harlow Solid"
grey = '#454545'
font2 = 'Mistral'
font3 = 'Italic 35px Times New Roman'
pnk = 'pink'
grn = 'green'
return locals()
Essentially, I am creating a module and the user must create some type of function that they list all of they variables they would like to declare to be used to modify a css file. Essentially, I would like to allow the user to not have to type "return locals()". I want to achieve it by having the end-users wrap the above example function in a decorator that will do the equivalent of returning locals() of the exact scope I want. The decorator does not work for me because it is in an outer scope.
TO BE EVEN MORE CLEAR:
I need a decorator/function that wraps another function(i.e. a decorator), that can access and create a list of a child element.
def module_decorator_func_thing():
r = command_that_acts_like_locals()_but_for_child_scopes
def user_var_list():
font1 = 'green'
font2 = 'pink'
# back in "module_decorator_func_thing"'s scope
print(r) # this variable should contain only a dict/list containing the
# the following:
# r = {'font1': 'green', 'font2': 'pink')
currently users need to do this:
def vars_func_container():
font1 = 'green'
font2 = 'pink'
return locals() # <---- I want the user to not have to type this and for
# a function decorator to take care of it instead possibly.
Info for #aguy and others wishing for more info.
The dictionary/list that I am obtaining via your guys' tips will be sent to this function to do the real job of the program.
(If I were to start using lists, i'd need to convert to a dictionary but that's no problem for me to solve.)
The dict of variables is used with this function to "compile/compyle"(Pun on the word 'Python' + 'compile) and is insert in the "variables" parameter. e.g. you execute the function like this.
compyle("My sample title", return_stylesheet_from_func(*insert .css filename),
return_variables_from_function(*insert function containing variables*), "**True/False to turn on compilation**",
"**True/False to turn on annotations/suggestions**")
def compyle(title, style_sheet, variables, boolean=False, boolean2=True):
"""
:param title: The name you wish your .css file to be named.
:param style_sheet: The name of the multi-line string that will compose your .css file
:param variables: The name of the dictionary containing your .pcss variables
:param boolean: A.K.A the "Compiler Parameter" - Turns the compiler on or off
:param boolean2: A.K.A the "Annotation Parameter" - Turns annotations on or off
:return: returns compiled .pcss text as normal .css style text to be utilized with .html
"""
# -----------------------------------
file_name = title + ".css"
replace_num = len(variables.keys())
counter = replace_num
content = style_sheet
# -----------------------------------
# add theme support with namedtuple's formatted to mimic structs in C/C++
# this will be a major feature update as well as a nice way to allow the future prospect of integrating C/C++ into
# the compiler. Info: https://stackoverflow.com/questions/35988/c-like-structures-in-python
for k, v in variables.items():
counter -= 1
content = content.replace(k, v, replace_num)
if counter == 0:
break
else:
pass
looped_content = str(content)
id_content = looped_content.replace("hash_", "#")
output = id_content.replace("dot_", ".")
if boolean is True:
if boolean2 is True:
output = " /* --- Pyle Sheet --- */\n" + output
with open(file_name, 'w') as writ:
writ.write(output)
writ.close()
print('compiled successfully; The file was saved as ' + "\"" + file_name + "\".")
elif boolean2 is False:
pass
else:
logging.warning("An Error Occurred - see module, documentation, or online Q&A for assistance.")
elif boolean is False:
if boolean2 is True:
print('compiled successfully; The file ' + "\"" + file_name + "\"" + "was not saved/created.")
elif boolean2 is False:
pass
else:
logging.warning("An Error Occurred - see module, documentation, or online Q&A for assistance.")
else:
logging.warning('An Error Occurred with the Compile Parameter (See: boolean in pyle_sheets source file) - \ '
'see module, documentation, or online Q&A for assistance.')

I can't see any way to do this without getting pretty deep; what follows is the simplest solution I've come up with.
how it works
Using the ast module, we go through the code of the given function and find all the assignments. These are evaluated in a given namespace and this namespace is returned.
the code
import ast
import functools
import inspect
def returnAssignments(f):
#functools.wraps(f)
def returner():
assignments = dict()
for node in ast.walk(ast.parse(inspect.getsource(f))):
if isinstance(node, ast.Assign):
exec(compile(ast.Module([node]), '<ast>', 'exec'),
globals(),
assignments)
return assignments
return returner
usage
from ra import returnAssignments
#returnAssignments
def foo():
this = 'something'
that = 37
the_other = object()
print(foo())
output
rat#pandion:~/tmp$ python test.py
{'this': 'something', 'that': 37, 'the_other': <object object at 0x10205b130>}

I wonder if such a crude solution as I provide here might be useful to you. Note that I haven't tested it on all cases, so it might be a bit rough. Also, it returns everything as a string, a behavior which you might want to further change.
Define the function:
def get_local_vars_from_function(f):
import inspect
s = inspect.getsourcelines(f)[0]
d = {}
for l in s:
if '=' in l:
var, val = l.split('=')
var = var.strip()
val = val.strip()
d[var] = val
return d
Then to use it:
In[91]: get_local_vars_from_function(user_var_list)
Out[91]: {'font1': "'green'", 'font2': "'pink'"}

Export function that names the export file after the input variable

I'm looking to get a function to export Numpy arrays, but to use the name of the variable input to the function as the name of the exported file. Something like:
MyArray = [some numbers]
def export(Varb):
return np.savetxt("%s.dat" %Varb.name, Varb)
export(MyArray)
that will output a file called 'MyArray.dat' filled with [some numbers]. I can't work out how to do the 'Varb.name' bit. Any suggestions?
I'm new to Python and programming so I hope there is something simple I've missed!
Thanks.

I don't recommend such a code style, but you can do it this way:
import copy
myarray = range(4)
for k, v in iter(copy.copy(locals()).items()):
if myarray == v:
print k
This gives myarray as output. To do this in a function useful for exports use:
import copy
def export_with_name(arg):
""" Export variable's string representation with it's name as filename """
for k, v in iter(copy.copy(globals()).items()):
if arg == v:
with open(k, 'w') as handle:
handle.writelines(repr(arg))
locals() and globals() both give dictionaries holding the variable names as keys and the variable values as values.
Use the function the following way:
some_data = list(range(4))
export_with_name(some_data)
gives a file called some_data with
[0, 1, 2, 3]
as content.
Tested and compatible with Python 2.7 and 3.3

You can't. Python objects don't know what named variables happen to be referencing them at any particular time. By the time the variable has been dereferenced and sent to the function, you don't know where it came from. Consider this bit of code:
map(export, myvarbs)
Here, the varbs were in some sort of container and didn't even have a named variable referencing them.

import os
import inspect
import re
number_array = [8,3,90]
def varname(p):
for line in inspect.getframeinfo(inspect.currentframe().f_back)[3]:
m = re.search(r'\bvarname\s*\(\s*([A-Za-z_][A-Za-z0-9_]*)\s*\)', line)
if m:
return m.group(1)
def export(arg):
file_name = "%s.dat" % varname(arg)
fd = open(file_name, "w")
fd.writelines(str(arg))
fd.close()
export(number_array)
Refer How can you print a variable name in python? to get more details about def varname()

How to create an array in Python by using as name the elements of another array

I am trying to do the following in python: I have a list which contains some string values. The list looks like this:
parameters = ['RH', 'WindSp_Avg', 'WindDir']
What I want to do - and I really hope the question is not ridiculous - is to create three lists which each of them have the name of the "parameters" list.
For example:
RH = []
WindSp_Avg = []
WindDir = []
I am new in python and although I had searched a bit online I couldn't find some elegant way to do something like this.
I am trying something like this:
for i in parameters:
parameters[0] = []
But it doesn't seem to work. Any suggestions?
Thanks
Dimitris

What you are trying to do is very unsafe and is against all good practices. Is there a problem in simply creating a dictionary?
myVars = {}
for param in parameters:
myVars[param] = []
WARNING The following code is for educational purposes! DO NOT USE IT IN REAL CODE!
You can do a hard hack to add dynamically a variable to the local variables inside a function. Normally locals() represent all local variables. However simply adding to that dictionary won't solve the problem. There is a hack to force Python to reevaluate locals by using exec, for example:
def test():
for param im parameters:
locals()[param] = []
exec ""
print WindSp_Avg
and result:
>>> test()
[]

parameters = ['RH', 'WindSp_Avg', 'WindDir']
for i in parameters:
vars()[i] = [];
print locals()

To create a variable you can do so:
parameters = ['RH', 'WindSp_Avg', 'WindDir']
for i in parameters:
exec("%s = []" % i);
print vars()

you could something like this:
code = "{0} = []"
for i in parameters:
codeobj = compile(code.format(i), "/dev/null", "single")
eval(codeobj)
but i think tha's very unsafe. Because it could be something in parameters and that will be excuted by the eval. So please only use this if it's only really necessary and security is less important.

Capturing **vars() pattern in string formatting

I frequently find myself using the following pattern for string formatting.
a = 3
b = 'foo'
c = dict(mykey='myval')
#prints a is 3, b is foo, mykey is myval
print('a is {a}, b is {b}, mykey is {c[mykey]}'.format(**vars()))
That is, I often have the values I need to print in the local namespace, represented by a call to vars(). As I look over my code, however, it seems awfully unpythonic to be constantly repeating the .format(**vars()) pattern.
I'd like to create a function that will capture this pattern. It would be something like the following.
# doesn't work
def lfmt(s):
"""
lfmt (local format) will format the string using variables
in the caller's local namespace.
"""
return s.format(**vars())
Except that by the time I'm in the lfmt namespace, vars() is no longer what I want.
How can I write lfmt so that it executes vars() in the caller's namespace such that the following code would work as the example above?
print(lfmt('a is {a}, b is {b}, mykey is {c[mykey]}'))

Edit: In order for lfmt to work when called from different namespaces, you'll need the inspect module. Note, as the documentation warns, the inspect module may not be suitable for production code since it may not work with all implementations of Python
import inspect
def lfmt(s):
caller = inspect.currentframe().f_back
return s.format(**caller.f_locals)
a = 3
b = 'foo'
c = dict(mykey='myval')
print(lfmt('a is {a}, b is {b}, mykey is {c[mykey]}'))
# a is 3, b is foo, mykey is myval

You have to inspect the variables from the calling frames.
This will get you started:
import inspect
import pprint
def lfmt(s):
for frame in inspect.getouterframes(inspect.currentframe()):
f = frame[0]
print pprint.pformat(f.f_locals)
return '???'
if __name__ == '__main__':
a = 10
b = 20
c = 30
lfmt('test')

Here you are:
import sys
def lfmt(s):
"""
lfmt (local format) will format the string using variables
in the caller's local namespace.
"""
if hasattr(sys, "tracebacklimit") and sys.tracebacklimit == 0:
raise Exception, "failfailfail"
try:
raise ZeroDivisionError
except ZeroDivisionError:
f = sys.exc_info()[2].tb_frame.f_back
return s.format(**f.f_locals)
a = 5
somestring = "text"
print lfmt("{a} {somestring}")
The fact that it works doesn't mean you should use it. This is what developers call "major hack", usually shipped with a comment "XXX fix me XXX".

Is it so bad to type ,vars each time you call the function?
def lfmt(s,v):
"""
lfmt (local format) will format the string using variables
from the dict returned by calling v()"""
return s.format(**v())
print(lfmt('a is {a}, b is {b}, mykey is {c[mykey]}',vars))

You could also use sys instead of inspect, but I don't know if it has the same problem with different implementations as inspect has.
import sys
def lfmt(s):
caller = sys._getframe(1)
return s.format(**caller.f_locals)
This is as far as I got: Python string interpolation implementation