Imagine the following problem: you have a dictionary of some content in python and want to generate python code that would create this dict. (which is like eval but in reverse)
Is there something that can do this?
Scenario:
I am working with a remote python interpreter. I can give source files to it but no input. So I am now looking for a way to encode my input data into a python source file.
Example:
d = {'a': [1,4,7]}
str_d = reverse_eval(d)
# "{'a': [1, 4, 7]}"
eval(str_d) == d
repr(thing)
will output text that when executed will (in most cases) reproduce the dictionary.
Actually, it's important for which types of data do you want this reverse function to exist. If you're talking about built-in/standard classes, usually their .__repr__() method returns the code you want to access. But if your goal is to save something in a human-readable format, but to use an eval-like function to use this data in python, there is a json library.
It's better to use json for this reason because using eval is not safe.
Json's problem is that it can't save any type of data, it can save only standard objects, but if we're talking about not built-in types of data, you never know, what is at their .__repr__(), so there's no way to use repr-eval with this kind of data
So, there is no reverse function for all types of data, you can use repr-eval for built-in, but for built-in data the json library is better at least because it's safe
I have a list of strings:
A = ['a','b']
I want to parse this list and write following expressions:
a = x**2
b = a*x
where x is a sympy symbol and later on I will use these expressions for other operations like differentiation and so on. The problem is that a and b are strings inside the list. I am not being able to use them as expressions! How can I do this?
In general, you don't. Trying to set up dynamic variable names is usually a sign of poor design. If you do need those symbols to represent things to the outside world, try keeping a label and a value. For instance, a dictionary can do something like this for you.
symbol = { 'a': x**2; 'b': x**3 }
You can add symbols from there, change values, etc. For instance,
symbol = { 'a': x**2 }
symbol['b'] = symbol['a'] * x
Granted, you can build an expression string and use eval on the contents, but this is generally dangerous and hard to maintain well.
A wider possibility is to manipulate string values and write the Python script you'd like to run. Write it to a file and then use the os or subprocess commands to execute it.
Does that get you moving?
For debugging purposes I need to know the values of some variables in Python. But, I don't want create a dictionary with all variables and don’t want to add every new variable that I want to test at some point to a dictionary, especially when it comes to lists and their content, which is often procedurally generated.
So is there an alternative to just taking input as a string and then executing it by exec() or using print() and hardcoded variables?
Yes, there is a way. You can use the locals() function to get a dictionary of all variables. For example:
a=5
b=locals()["a"]
Now b will get the value of a, i.e. 5.
However, while you can do this doesn't mean you should do this. There may be something wrong in the structure of your program if you want to access variables by using their name stored in a string.
Well to directly explain it, I wish to "parse" user input. - Simple input string -> output string.
However there isn't any "logical" way to parse this apart from simply checking the input string against a dictionary (regex check). Now this isn't difficult at all - I'd just create a dictionary with regex search string keys & regex/function pointer values.
However the problem is: there will be around ~100-200 "keys" probably. And I can easily see myself wishing to add/remove keys (maybe merge) in the future.
So is there a way that creating such a dictionary looks "structured"? Keeping the "data" away from the "code". (Data would be the regex-functionname pairs)?
Store the dictionary in the JSON format in file, with the function names as ordinary strings. Demo how to load a JSON file:
Content of sample file:
{"somestring":"myfunction"}
Code:
import json
d = json.load(open('very_small_dic.txt', 'r'))
print(d) # {'somestring': 'myfunction'}
How to get the string:function mapping:
First you load the dictionary from a file as illustrated in the code above. After that, you build a new dictionary where the strings of the function names are replaced by the actual functions. Demo:
def myfunction(x):
return 2*x
d = {'somestring': 'myfunction'} # in the real code this came from json.load
d = {k:globals()[v] for k,v in d.items()}
print(d) # {'somestring': <function myfunction at 0x7f36e69d8c20>}
print(d['somestring'](42)) # 84
You could also store your functions in a separate file myfunctions.py and use getattr. This is probably a cleaner way than using globals.
import myfunctions # for this demo, this module only contains the function myfunction
d = {'somestring': 'myfunction'} # in the real code this came from json.load
d = {k:getattr(myfunctions,v) for k,v in d.items()}
print(d) # {'somestring': <function myfunction at 0x7f36e69d8c20>}
print(d['somestring'](42)) # 84
You could also use JsonSchema (http://json-schema.org/example1.html).
I think in order to transform values belonging to certain keys you'd certainly have to write a function to perform the conversions.
If you'd just like to sanitize the input based on the presence of certain keys - It would be ideal to convert it into a dictionary, define a schema and then check for the (required/optional fields)/the type or validate a field against a list of enums.
Currently expensively parsing a file, which generates a dictionary of ~400 key, value pairs, which is seldomly updated. Previously had a function which parsed the file, wrote it to a text file in dictionary syntax (ie. dict = {'Adam': 'Room 430', 'Bob': 'Room 404'}) etc, and copied and pasted it into another function whose sole purpose was to return that parsed dictionary.
Hence, in every file where I would use that dictionary, I would import that function, and assign it to a variable, which is now that dictionary. Wondering if there's a more elegant way to do this, which does not involve explicitly copying and pasting code around? Using a database kind of seems unnecessary, and the text file gave me the benefit of seeing whether the parsing was done correctly before adding it to the function. But I'm open to suggestions.
Why not dump it to a JSON file, and then load it from there where you need it?
import json
with open('my_dict.json', 'w') as f:
json.dump(my_dict, f)
# elsewhere...
with open('my_dict.json') as f:
my_dict = json.load(f)
Loading from JSON is fairly efficient.
Another option would be to use pickle, but unlike JSON, the files it generates aren't human-readable so you lose out on the visual verification you liked from your old method.
Why mess with all these serialization methods? It's already written to a file as a Python dict (although with the unfortunate name 'dict'). Change your program to write out the data with a better variable name - maybe 'data', or 'catalog', and save the file as a Python file, say data.py. Then you can just import the data directly at runtime without any clumsy copy/pasting or JSON/shelve/etc. parsing:
from data import catalog
JSON is probably the right way to go in many cases; but there might be an alternative. It looks like your keys and your values are always strings, is that right? You might consider using dbm/anydbm. These are "databases" but they act almost exactly like dictionaries. They're great for cheap data persistence.
>>> import anydbm
>>> dict_of_strings = anydbm.open('data', 'c')
>>> dict_of_strings['foo'] = 'bar'
>>> dict_of_strings.close()
>>> dict_of_strings = anydbm.open('data')
>>> dict_of_strings['foo']
'bar'
If the keys are all strings, you can use the shelve module
A shelf is a persistent, dictionary-like object. The difference with
“dbm” databases is that the values (not the keys!) in a shelf can be
essentially arbitrary Python objects — anything that the pickle module
can handle. This includes most class instances, recursive data types,
and objects containing lots of shared sub-objects. The keys are
ordinary strings.
json would be a good choice if you need to use the data from other languages
If storage efficiency matters, use Pickle or CPickle(for execution performance gain). As Amber pointed out, you can also dump/load via Json. It will be human-readable, but takes more disk.
I suggest you consider using the shelve module since your data-structure is a mapping.
That was my answer to a similar question titled If I want to build a custom database, how could I? There's also a bit of sample code in another answer of mine promoting its use for the question How to get a object database?
ActiveState has a highly rated PersistentDict recipe which supports csv, json, and pickle output file formats. It's pretty fast since all three of those formats are implement in C (although the recipe itself is pure Python), so the fact that it reads the whole file into memory when it's opened might be acceptable.
JSON (or YAML, or whatever) serialisation is probably better, but if you're already writing the dictionary to a text file in python syntax, complete with a variable name binding, you could just write that to a .py file instead. Then that python file would be importable and usable as is. There's no need for the "function which returns a dictionary" approach, since you can directly use it as a global in that file. e.g.
# generated.py
please_dont_use_dict_as_a_variable_name = {'Adam': 'Room 430', 'Bob': 'Room 404'}
rather than:
# manually_copied.py
def get_dict():
return {'Adam': 'Room 430', 'Bob': 'Room 404'}
The only difference is that manually_copied.get_dict gives you a fresh copy of the dictionary every time, whereas generated.please_dont_use_dict_as_a_variable_name[1] is a single shared object. This may matter if you're modifying the dictionary in your program after retrieving it, but you can always use copy.copy or copy.deepcopy to create a new copy if you need to modify one independently of the others.
[1] dict, list, str, int, map, etc are generally viewed as bad variable names. The reason is that these are already defined as built-ins, and are used very commonly. So if you give something a name like that, at the least it's going to cause cognitive-dissonance for people reading your code (including you after you've been away for a while) as they have to keep in mind that "dict doesn't mean what it normally does here". It's also quite likely that at some point you'll get an infuriating-to-solve bug reporting that dict objects aren't callable (or something), because some piece of code is trying to use the type dict, but is getting the dictionary object you bound to the name dict instead.
on the JSON direction there is also something called simpleJSON. My first time using json in python the json library didnt work for me/ i couldnt figure it out. simpleJSON was...easier to use