Develop Reference

Python: Obtaining index of an element within a value list of a dictionary

I have a dictionary with key:value list pairings, and I intend to find the index of the value list that contains the desired element.
E.g., if the dictionary is:
my_dict = {"key1":['v1'], "key2":None, "key3":['v2','v3'], "key4":['v4','v5','v6']}
Then, given element 'v2' I should be able to get index 2.
For a value list with one element, the index can be obtained with: list(my_dict.values()).index(['v1']) , however this approach does not work with lists containing multiple elements.
Using for loop, it can be obtained via:
for key, value in my_dict.items():
if value is None:
continue
if 'v2' in value:
print (list(my_dict.keys()).index(key))
Is there a neater (pythonic) way to obtain the same?

You've got an XY problem. You want to know the key that points to a value, and you think you need to find the enumeration index iterating the values so you can then use it to find the key by iteration as well. You don't need all that. Just find the key directly:
my_dict = {"key1":['v1'], "key2":None, "key3":['v2','v3'], "key4":['v4','v5','v6']}
value = 'v2'
# Iterate key/vals pairs in genexpr; if the vals contains value, yield the key,
# next stops immediately for the first key yielded so you don't iterate the whole dict
# when the value is found on an early key
key_for_value = next(key for key, vals in my_dict.items() if vals and value in vals)
print(key_for_value)
Try it online!
That'll raise StopIteration if the value doesn't exist, otherwise it directly retrieves the first key where the values list for that key contains the desired value.
If you don't really have an XY problem, and the index is important (it shouldn't be, that's a misuse of dicts) it's trivial to produce it as well, changing the extraction of the key to get both, e.g.:
index, key_for_value = next((i, key) for i, (key, vals) in enumerate(my_dict.items()) if vals and value in vals)
Mind you, this is a terrible solution if you need to perform these lookups a lot and my_dict isn't trivially small; it's O(n) on the total number of values, so a large dict would take quite a while to check (relative to the cost of just looking up an arbitrary key, which is average-case O(1)). In that case, ideally, if my_dict doesn't change much/at all, you'd construct a reversed dictionary up-front to find the key(s) associated with a value, e.g.:
from collections import defaultdict
my_dict = {"key1":['v1'], "key2":None, "key3":['v2','v3'], "key4":['v4','v5','v6']}
reversed_my_dict = defaultdict(set)
for key, vals in my_dict:
for val in vals:
reversed_my_dict[val].add(key)
reversed_my_dict = dict(reversed_my_dict) # Optional: Prevents future autovivification of keys
# by converting back to plain dict
after which you can cheaply determine the key(s) associated with a given value with:
reversed_my_dict.get(value, ()) # Using .get prevents autovivification of keys even if still a defaultdict
which returns the set of all keys that map to that value, if any, or the empty tuple if not (if you convert back to dict above, reversed_my_dict[value] would also work if you'd prefer to get a KeyError when the value is missing entirely; leaving it a defaultdict(set) would silently construct a new empty set, map it to the key and return it, which is fine if this happens rarely, but a problem if you test thousands of unmapped values and create a corresponding thousands of empty sets for no benefit, consuming memory wastefully).
Which you choose depends on how big my_dict is (for small my_dict, O(n) work doesn't really matter that much), how many times you need to search it (fewer searches mean less gain from reversed dict), and whether it's regularly modified. For that last point, if it's never modified, or rarely modified between lookups, rebuilding the reversed dict from scratch after each modification might be worth it for simplicity (assuming you perform many lookups per rebuild); if it's frequently modified, the reversed dict might still be worth it, you'd just have to update both the forward and reversed dicts rather than just one, e.g., expanding:
# New key
my_dict[newkey] = [newval1, newval2]
# Add value
my_dict[existingkey].append(newval)
# Delete value
my_dict[existingkey].remove(badval)
# Delete key
del my_dict[existingkey]
to:
# New key
newvals = my_dict[newkey] = [newval1, newval2]
for newval in newvals:
reversed_my_dict[newval].add(newkey) # reversed_my_dict.setdefault(newval, set()).add(newkey) if not defaultdict(set) anymore
# Add value
my_dict[existingkey].append(newval)
reversed_my_dict[newval].add(existingkey) # reversed_my_dict.setdefault(newval, set()).add(existingkey) if not defaultdict(set) anymore
# Delete value
my_dict[existingkey].remove(badval)
if badval not in my_dict[existingkey]: # Removed last copy; test only needed if one key can hold same value more than once
reversed_my_dict[badval].discard(existingkey)
# Optional delete badval from reverse mapping if last key removed:
if not reversed_my_dict[badval]:
del reversed_my_dict[badval]
# Delete key
# set() conversion not needed if my_dict's value lists guaranteed not to contain duplicates
for badval in set(my_dict.pop(existingkey)):
reversed_my_dict[badval].discard(existingkey)
# Optional delete badval from reverse mapping if last key removed:
if not reversed_my_dict[badval]:
del reversed_my_dict[badval]
respectively, roughly doubling the work incurred by modifications, in exchange for always getting O(1) lookups in either direction.

If you are looking for the key corresponding to a value, you can reverse the dictionary like so:
reverse_dict = {e: k for k, v in my_dict.items() if v for e in v}
Careful with duplicate values though. The last occurence will override the previous ones.

Don't know if it's the best solution but this works:
value = 'v2'
list(map(lambda x : value in x, list(map(lambda x : x[1] or [], list(my_dict.items()))))).index(True)

Optimization of comparisons between each key in a dictionary (Python)

So I want to compare each key of a dictionary to each other and if a key happens to be too similar to another key (based on fuzzy similarity), I want to merge those 2 entries together into a single key (so one key gets deleted whereas the values will be added up). Is there a more efficient way to do this?
d is a dictionary with {"labels": [list of sentences]}
# First I create a new dictionary that has a dictionary as value which includes the key and value
# of the old dictionary. It feels a bit redundant but afaik you can't loop through a dictionary and
# edit it at the same time + you can't edit keys themselves?
d_new = {}
for key, value in d.items():
d_new[key] = {"label": key, "sentences": value}
for key1, key2 in itertools.combinations(d, 2):
if fuzz.token_set_ratio(key1, key2) >= similarity:
if len(d_new[key1]["sentences"]) > len(d_new[key2]["sentences"]):
d_new[key2]["label"] = key1
# Merge keys
d_new[key2]["sentences"] = list(set(d_new[key1]["sentences"] + d_new[key2]["sentences"]))
del d_new[key1]
continue
# Prepare output
result = {}
for key, value in d_new.items():
result[value["label"]] = list(set(value["sentences"]))
return result

Since you are using itertools.combinations, this will result in a non-linear time complexity to loop over the dictionary. but your code seems to be optimized in the way you want it to be and there probably isn't any better way to find similar keys unless you know what similar things you are looking for.
As the other answer said, further optimization depends on token_set_ratio.
You can also experience with dict comprehension to see if you can make the dict creation faster.
for example:
{value['lable']:list(set(value['sentences'])) for (key,value) in d_new.items()}
suggestion:
due to limited data on what values are there in your keys its hard to say, but if your keys are structured and are based on a pattern and your dict length is large enough, you can brute force for keys since accessing dict keys is O(1) but there is the cost of 'guessing' similar things (for example if your keys are words, you can probably know what words are similar.). but as I said the dict should be huge for this method to be efficient.

Whether or not this can be optimized depends on the implementation of token_set_ratio. If token_set_ratio is your only heuristic of similarity between the two tokens, and the implementation of token_set_ratio is a black box, then this is the most Big O efficient way to calculate similarity. The reasoning here is that because your only heuristic is a black box, you must test every binary combination to get a similarity score.
If instead, you have a different heuristic that works on a single token, then you may be able to precompute a feature score that can help you cluster more efficiently. You would expect some false positives.
Alternatively, if you know how the original heuristic is computed, then there may be a way to optimize based on the heuristic. In that case, you should post it here.

If there is a large percentage of the keys that will be merged, it may be worthwhile to implement the combination logic yourself so that keys that have been merged are not checked again.
There is no data to test this but, here's how it could look like:
from collections import deque
def mergeKeys(d):
result = {k:{v} for k,v in d.items()} # start with nonthing merged
keys = deque(d.keys()) # list of remaining keys
while keys:
key1 = keys.popleft() # consume one key
nextKeys = deque() # track unmerged for next pass
while keys:
key1 = keys.popleft() # consume paired key
if key2 not in result: continue
if fuzz.token_set_ratio(key1, key2) >= similarity:
result[key1].update(result.pop(key2)) # merge
else:
nextKeys.append(key2) # not merged, add to next pass
keys = nextKeys # remaining keys for next pass
return result

What's the fastest way to identify the 'name' of a dictionary that contains a specific key-value pair?

I'd like to identify the dictionary within the following list that contains the key-value pair 'Keya':'123a', which is ID1 in this case.
lst = {ID1:{'Keya':'123a','Keyb':456,'Keyc':789},ID2:{'Keya':'132a','Keyb':654,'Keyc':987},ID3:{'Keya':'5433a','Keyb':222,'Keyc':333},ID4:{'Keya':'444a','Keyb':777,'Keyc':666}}
It's safe to assume all dictionaries have the same key's, but have different values.
I currently have the following to identify which dictionary has the value '123a' for the key 'Keya', but is there a shorter and faster way?
DictionaryNames = map(lambda Dict: str(Dict),lst)
Dictionaries = [i[1] for i in lst.items()]
Dictionaries = map(lambda Dict: str(Dict),Dictionaries)
Dict = filter(lambda item:'123a' in item,Dictionaries)
val = DictionaryNames[Dictionaries.index(Dict[0])]
return val

If you actually had a list of dictionaries, this would be:
next(d for d in list_o_dicts if d[key]==value)
Since you actually have a dictionary of dictionaries, and you want the key associated with the dictionary, it's:
next(k for k, d in dict_o_dicts.items() if d[key]==value)
This returns the first matching value. If you're absolutely sure there is exactly one, or if you don't care which you get if there are more than one, and if you're happy with a StopIteration exception if you were wrong and there isn't one, that's exactly what you want.
If you need all matching values, just do the same with a list comprehension:
[k for k, d in dict_o_dicts.items() if d[key]==value]
That list can of course have 0, 1, or 17 values.

You can just do [name for name, d in lst.iteritems() if d['Keya']=='123a'] to get a list of all the dictionaries in lst that have that value for that key. If you know there is only one, you can get it with [name for name, d in lst.iteritems() if d['Keya']=='123a'][0]. (As Andy mentions in a comment, your name lst is misleading, since lst is actually a dictionary of dictionaries, not a list.)

Since you want the fastest, you should short-cut your search as soon as you find the data you are after. Iterating through the whole list is not necessary, nor is producing any temporary dictionary:
for key,data in lst.iteritems():
if data['Keya']=='132a':
return key #or break is not in a function

Å different way to do this is to use the appropriate data structure: Keep a "reverse map" of key-value pairs to names. If your dictionary of dictionaries is static after being built, you can build the reverse dictionary like this:
revdict = {(key, value): name
for name, subdict in dictodicts.items()
for key, value in subdict.items()}
If not, you just need to add revdict[key, value] = name for each d[name][key] = value statement and build them up in parallel.
Either way, to find the name of the dict that maps key to value, it's just:
revdict[key, value]
For (a whole lot) more information (than you actually want), and some sample code for wrapping things up in different ways… I dug up an unfinished blog post, considered editing it, and decided to not bother and just clicked Publish instead, so: Reverse dictionary lookup and more, on beyond z.

Value-sorted dict for Python?

I am interested in a dict implementation for Python that provides an iterating interface to sorted values. I.e., a dict with a "sortedvalues()" function.
Naively one can do sorted(dict.values()) but that's not what I want. Every time items are inserted or deleted, one has to run a full sorting which isn't efficient.
Note that I am not asking about key-sorted dict either (for that question, there are excellent answers in Key-ordered dict in Python and Python 2.6 TreeMap/SortedDictionary?).

One solution is to write a class that inherits from dict but also maintains a list of keys sorted by their value (sorted_keys), along with the list of corresponding (sorted) values (sorted_values).
You can then define a __setitem__() method that uses the bisect module in order to know quickly the position k where the new (key, value) pair should be inserted in the two lists. You can then insert the new key and the new value both in the dictionary itself, and in the two lists that you maintain, with sorted_values[k:k] = [new_value] and sorted_keys[k:k] = [new_key]; unfortunately, the time complexity of such an insertion is O(n) (so O(n^2) for the whole dictionary).
Another approach to the ordered element insertion would be to use the heapq module and insert (value, key) pairs in it. This works in O(log n) instead of the list-based approach of the previous paragraph.
Iterating over the dictionary can then simply done by iterating over the list of keys (sorted_keys) that you maintain.
This method saves you the time it would take to sort the keys each time you want to iterate over the dictionary (with sorted values), by basically shifting (and increasing, unfortunately) this time cost to the construction of the sorted lists of keys and values.

The problem is that you need to sort or hash it by keys to get reasonable insert and lookup performance. A naive way of implementing it would be a value-sorted tree structure of entries, and a dict to lookup the tree position for a key. You need to get deep into updating the tree though, as this lookup dictionary needs to be kept correct. Essentially, as you would do for an updatable heap.
I figure there are too many options to make a resonable standard library option out of such a structure, while it is too rarely needed.
Update: a trick that might work for you is to use a dual structure:
a regular dict storing the key-value pairs as usual
any kind of sorted list, for example using bisect
Then you have to implement the common operations on both: a new value is inserted into both structures. The tricky part are the update and delete operations. You use the first structure to look up the old value, delete the old value from the second structure, then (when updating) reinsert as before.
If you need to know the keys too, store (value, key) pairs in your b list.
Update 2: Try this class:
import bisect
class dictvs(dict):
def __init__(self):
self._list = []
def __setitem__(self, key, value):
old = self.get(key)
if old is None:
bisect.insort(self._list, value)
dict.__setitem__(self, key, value)
else:
oldpos = bisect.bisect_left(self._list, old)
newpos = bisect.bisect_left(self._list, value)
if newpos > oldpos:
newpos -= 1
for i in xrange(oldpos, newpos):
self._list[i] = self._list[i + 1]
else:
for i in xrange(oldpos, newpos, -1):
self._list[i] = self._list[i - 1]
self._list[newpos] = value
dict.__setitem__(self, key, value)
def __delitem__(self, key):
old = self.get(key)
if old is not None:
oldpos = bisect.bisect(self._list, old)
del self._list[oldpos]
dict.__delitem__(self, key)
def values(self):
return list(self._list)
It's not a complete dict yet I guess. I havn't tested deletions, and just a tiny update set. You should make a larger unit test for it, and compare the return of values() with that of sorted(dict.values(instance)) there. This is just to show how to update the sorted list with bisect

Here is another, simpler idea:
You create a class that inherits from dict.
You use a cache: you only sort the keys when iterating over the dictionary, and you mark the dictionary as being sorted; insertions should simply append to the list of keys.
kindall mention in a comment that sorting lists that are almost sorted is fast, so this approach should be quite fast.

You can use a skip dict. It is a Python dictionary that is permanently sorted by value.
Insertion is slightly more expensive than a regular dictionary, but it is well worth the cost if you frequently need to iterate in order, or perform value-based queries such as:
What's the highest / lowest item?
Which items have a value between X and Y?

A data-structure for 1:1 mappings in python?

I have a problem which requires a reversable 1:1 mapping of keys to values.
That means sometimes I want to find the value given a key, but at other times I want to find the key given the value. Both keys and values are guaranteed unique.
x = D[y]
y == D.inverse[x]
The obvious solution is to simply invert the dictionary every time I want a reverse-lookup: Inverting a dictionary is very easy, there's a recipe here but for a large dictionary it can be very slow.
The other alternative is to make a new class which unites two dictionaries, one for each kind of lookup. That would most likely be fast but would use up twice as much memory as a single dict.
So is there a better structure I can use?
My application requires that this should be very fast and use as little as possible memory.
The structure must be mutable, and it's strongly desirable that mutating the object should not cause it to be slower (e.g. to force a complete re-index)
We can guarantee that either the key or the value (or both) will be an integer
It's likely that the structure will be needed to store thousands or possibly millions of items.
Keys & Valus are guaranteed to be unique, i.e. len(set(x)) == len(x) for for x in [D.keys(), D.valuies()]

The other alternative is to make a new
class which unites two dictionaries,
one for each kind of lookup. That
would most likely be fast but would
use up twice as much memory as a
single dict.
Not really. Have you measured that? Since both dictionaries would use references to the same objects as keys and values, then the memory spent would be just the dictionary structure. That's a lot less than twice and is a fixed ammount regardless of your data size.
What I mean is that the actual data wouldn't be copied. So you'd spend little extra memory.
Example:
a = "some really really big text spending a lot of memory"
number_to_text = {1: a}
text_to_number = {a: 1}
Only a single copy of the "really big" string exists, so you end up spending just a little more memory. That's generally affordable.
I can't imagine a solution where you'd have the key lookup speed when looking by value, if you don't spend at least enough memory to store a reverse lookup hash table (which is exactly what's being done in your "unite two dicts" solution).

class TwoWay:
def __init__(self):
self.d = {}
def add(self, k, v):
self.d[k] = v
self.d[v] = k
def remove(self, k):
self.d.pop(self.d.pop(k))
def get(self, k):
return self.d[k]

The other alternative is to make a new class which unites two dictionaries, one for each > kind of lookup. That would most likely use up twice as much memory as a single dict.
Not really, since they would just be holding two references to the same data. In my mind, this is not a bad solution.
Have you considered an in-memory database lookup? I am not sure how it will compare in speed, but lookups in relational databases can be very fast.

Here is my own solution to this problem: http://github.com/spenthil/pymathmap/blob/master/pymathmap.py
The goal is to make it as transparent to the user as possible. The only introduced significant attribute is partner.
OneToOneDict subclasses from dict - I know that isn't generally recommended, but I think I have the common use cases covered. The backend is pretty simple, it (dict1) keeps a weakref to a 'partner' OneToOneDict (dict2) which is its inverse. When dict1 is modified dict2 is updated accordingly as well and vice versa.
From the docstring:
>>> dict1 = OneToOneDict()
>>> dict2 = OneToOneDict()
>>> dict1.partner = dict2
>>> assert(dict1 is dict2.partner)
>>> assert(dict2 is dict1.partner)
>>> dict1['one'] = '1'
>>> dict2['2'] = '1'
>>> dict1['one'] = 'wow'
>>> assert(dict1 == dict((v,k) for k,v in dict2.items()))
>>> dict1['one'] = '1'
>>> assert(dict1 == dict((v,k) for k,v in dict2.items()))
>>> dict1.update({'three': '3', 'four': '4'})
>>> assert(dict1 == dict((v,k) for k,v in dict2.items()))
>>> dict3 = OneToOneDict({'4':'four'})
>>> assert(dict3.partner is None)
>>> assert(dict3 == {'4':'four'})
>>> dict1.partner = dict3
>>> assert(dict1.partner is not dict2)
>>> assert(dict2.partner is None)
>>> assert(dict1.partner is dict3)
>>> assert(dict3.partner is dict1)
>>> dict1.setdefault('five', '5')
>>> dict1['five']
'5'
>>> dict1.setdefault('five', '0')
>>> dict1['five']
'5'
When I get some free time, I intend to make a version that doesn't store things twice. No clue when that'll be though :)

Assuming that you have a key with which you look up a more complex mutable object, just make the key a property of that object. It does seem you might be better off thinking about the data model a bit.

"We can guarantee that either the key or the value (or both) will be an integer"
That's weirdly written -- "key or the value (or both)" doesn't feel right. Either they're all integers, or they're not all integers.
It sounds like they're all integers.
Or, it sounds like you're thinking of replacing the target object with an integer value so you only have one copy referenced by an integer. This is a false economy. Just keep the target object. All Python objects are -- in effect -- references. Very little actual copying gets done.
Let's pretend that you simply have two integers and can do a lookup on either one of the pair. One way to do this is to use heap queues or the bisect module to maintain ordered lists of integer key-value tuples.
See http://docs.python.org/library/heapq.html#module-heapq
See http://docs.python.org/library/bisect.html#module-bisect
You have one heapq (key,value) tuples. Or, if your underlying object is more complex, the (key,object) tuples.
You have another heapq (value,key) tuples. Or, if your underlying object is more complex, (otherkey,object) tuples.
An "insert" becomes two inserts, one to each heapq-structured list.
A key lookup is in one queue; a value lookup is in the other queue. Do the lookups using bisect(list,item).

It so happens that I find myself asking this question all the time (yesterday in particular). I agree with the approach of making two dictionaries. Do some benchmarking to see how much memory it's taking. I've never needed to make it mutable, but here's how I abstract it, if it's of any use:
class BiDict(list):
def __init__(self,*pairs):
super(list,self).__init__(pairs)
self._first_access = {}
self._second_access = {}
for pair in pairs:
self._first_access[pair[0]] = pair[1]
self._second_access[pair[1]] = pair[0]
self.append(pair)
def _get_by_first(self,key):
return self._first_access[key]
def _get_by_second(self,key):
return self._second_access[key]
# You'll have to do some overrides to make it mutable
# Methods such as append, __add__, __del__, __iadd__
# to name a few will have to maintain ._*_access
class Constants(BiDict):
# An implementation expecting an integer and a string
get_by_name = BiDict._get_by_second
get_by_number = BiDict._get_by_first
t = Constants(
( 1, 'foo'),
( 5, 'bar'),
( 8, 'baz'),
)
>>> print t.get_by_number(5)
bar
>>> print t.get_by_name('baz')
8
>>> print t
[(1, 'foo'), (5, 'bar'), (8, 'baz')]

How about using sqlite? Just create a :memory: database with a two-column table. You can even add indexes, then query by either one. Wrap it in a class if it's something you're going to use a lot.