Develop Reference

Finding how a key's value compares to the rest of a dictionary of values

I'm creating a virtual race, and for each frame of the simulation, I have a data file containing each racer's distance from the lead point, meaning the racer with the lowest value is winning.
After each race, I need to go back to the data for the midpoint, and find what race position the eventual winner was in. It's easy to SEE in the example below the winner (1) was in 3rd position, with a distance of 600, but I'm struggling how to do this in Python (there will be hundreds of races and many more racers in each race).
RaceDistances = {1:600,2:300,3:450,4:1000,5:750}

We can generate the rankings using .items() to get a list of tuples containing the key-value pairs in the dictionary. Then, we sort by distance using sorted() and the key parameter. We then read off the rankings into a list using a list comprehension.
If you only need to get the rank of one specific player, you can then use .index([<id of player>]) + 1:
race_distances = {1:600,2:300,3:450,4:1000,5:750}
player_ranks = [x for x, _ in sorted(race_distances.items(), key=lambda x: x[1])]
result = player_ranks.index(1) + 1
print(result)
If you're going to be looking up the ranks for a lot of players, it is much better to create a dictionary instead, which has faster lookup:
result = {elem: idx + 1 for idx, elem in enumerate(player_ranks)}
print(result[1])
In both cases, this prints:
3

Counting how many smaller distances there are, then add 1:
print(sum(d < RaceDistances[1]
for d in RaceDistances.values()) + 1)

Python: Removing lower versions from a 2D list

Assume I have the following data structure (a list of lists):
myList = [['Something','1'], ['Something','2'], ['Something Else','5'], ['Yet Another Something','1'], ['Yet ANOTHER Something','0'], ['Yet Another Something','2']]
I have a function that will remove duplicates from that list, choosing the highest number for the 2nd value. However, it seems to choke on very large data sets (150+ entries in myList). For this small data set, I expect the returned list to be:
[['Something','2'], ['Something Else','5'], ['Yet Another Something','2']]
What kind of optimization can be implemented using standard python (without including custom, external modules) into this function so that it returns the same result set without issues on large data sets?
Here is my function:
def remove_duplicates(duplicate):
final_list = []
final_list_upper = []
for k,v in duplicate:
found = False
for x in range(len(final_list)):
if k in final_list[x] or k.upper() in final_list_upper[x]:
if k == final_list[x][0] or k.upper() == final_list_upper[x][0]:
if int(v) >= int(final_list[x][1]):
final_list.pop(final_list.index(final_list[x]))
final_list_upper.pop(final_list_upper.index(final_list_upper[x]))
break
else:
found = True
break
if not found:
final_list.append([k,v])
final_list_upper.append([k.upper(),v])
final_list_upper = [] # clear the list
return final_list

You're using a second loop to check if the current "key" that you're checking exists in the list. This is slowing down your code.
Why? Because, as your code demonstrates, checking for membership in lists is slow. Really slow, because you need to iterate over the entire list, which means it's an O(N) operation, so the time depends linearly on the size of the list.
Instead, you could simply change the list to a dictionary. Lookup in a dictionary is an O(1) operation, so the lookup happens in constant (or nearly constant) time regardless of the size of the dictionary.
When you do this, there's no longer a need for two loops. Here's an idea:
def remove_duplicates_new(duplicate):
final_dict = {}
case_sensitive_keys = {}
for k, v in duplicate:
klower = k.lower()
vint = int(v)
old_val = final_dict.get(klower, 0) # Get the key k, with a default of zero if the key doesn't exist
if vint > old_val:
# Replace if current value is greater than old value
final_dict[klower] = vint
case_sensitive_keys[klower] = k
# Now we're done looping, so create the list
final_list = [[case_sensitive_keys[k], str(v)] for k, v in final_dict.items()]
return final_list
To compare, let's make a test list with 10000 elements. The "keys" are random numbers between 1 and 100, so we're bound to get a whole bunch of duplicates.:
import random
import timeit
testList = [[str(random.randint(1, 100)), str(random.randint(1, 10))] for i in range(10000)]
timeit.timeit('remove_duplicates(testList)', setup='from __main__ import testList, remove_duplicates', number=10)
# Output: 1.1064800999999989
timeit.timeit('remove_duplicates_new(testList)', setup='from __main__ import testList, remove_duplicates_new', number=10)
# Output: 0.03743689999998878
Hot damn! That's a ~30x speedup!

Cython dictionary / map

I have a list of element, label pairs like this: [(e1, l1), (e2, l2), (e3, l1)]
I have to count how many labels two element have in common - ie. in the list above e1and e3have the label l1 in common and thus 1 label in common.
I have this Python implementation:
def common_count(e_l_list):
count = defaultdict(int)
l_list = defaultdict(set)
for e1, l in e_l_list:
for e2 in l_list[l]:
if e1 == e2:
continue
elif e1 > e2:
count[e1,e2] += 1
else:
count[e2,e1] += 1
l_list[l].add(e1)
return count
It takes a list like the one above and computes a dictionary of element pairs and counts. The result for the list above should give {(e1, e2): 1}
Now i have to scale this to millions of elements and labels and i though Cython would be a good solution to save CPU time and memory. But i can't find docs on how to use maps in Cython.
How would i implement the above in pure Cython?
It can be asumed that all elements and labels are unsigned integers.
Thanks in advance :-)

I think you are trying to over complicate this by creating pairs of elements and storing all common labels as the value when you can create a dict with the element as the key and have a list of all values associated with that element. When you want to find common labels convert the lists to a set and perform an intersection on them, the resulting set will have the common labels between the two. The average time of the intersection, checked with ~20000 lists, is roughly 0.006 or very fast
I tested this with this code
from collections import *
import random
import time
l =[]
for i in xrange(10000000):
#With element range 0-10000000 the dictionary creation time increases to ~16 seconds
l.append((random.randrange(0,50000),random.randrange(0,50000)))
start = time.clock()
d = defaultdict(list)
for i in l: #O(n)
d[i[0]].append(i[1]) #O(n)
print time.clock()-start
times = []
for i in xrange(10000):
start = time.clock()
tmp = set(d[random.randrange(0,50000)]) #picks a random list of labels
tmp2 = set(d[random.randrange(0,50000)]) #not guaranteed to be a different list but more than likely
times.append(time.clock()-start)
common_elements = tmp.intersection(tmp2)
print sum(times)/100.0
18.6747529999 #creation of list
4.17812619876 #creation of dictionary
0.00633531142994 #intersection
Note: The times do change slightly depending on number of labels. Also creating the dict might be too long for your situation but that is only a one time operation.
I would also highly not recommend creating all pairs of elements. If you have 5,000,000 elements and they all share at least one label, which is worst case, then you are looking at 1.24e+13 pairs or, more bluntly, 12.5 trillion. That would be ~1700 terabytes or ~1.7 petabytes

Efficient way of searching a list of lists according to their hash value?

I have a list of tuples with 3 members in each tuple as seen below:
[(-5092793511388848640, 'test1', 1),
(-5092793511388848639, 'test0', 0),
(-5092793511388848638, 'test3', 3),
(-5092793511388848637, 'test2', 2),
(-5092793511388848636, 'test5', 5)]
The tuples are ordered in ascending according to first element of each tuple - the hash value of each key (e.g 'test0'). I want to find a quick way of searching through these tuples using binary search of their hash values to find a specific key. Problem is the quickest way I have found is using a for loop:
def get(key, D, hasher=hash):
'''
Returns the value in the dictionary corresponding to the given key.
Arguements:
key -- desired key to retrieve the value of.
D -- intended dictionary to retrieve value from.
hasher -- the hash function to be used on the key.
'''
for item in D:
if item[0] == hash(key):
return item[2]
raise TypeError('Key not found in the dictionary.')
The function I have written above seems to be very slow at searching through a much longer list of tuples, lets say a list of 6000 different tuples. It also breaks if there are any hash collisions. I was wondering if there was a more efficient/quick way of searching the list for the correct tuple?
Side note: I know using dictionaries will be a much quicker and easier way to solve my problem but I'd like to avoid using them.

First, prehash the key, don't do it over and over. Second, you can use next with an unpacking generator expression to optimize a bit:
def get(key, D, hasher=hash):
keyhash = hasher(key)
try:
return next(v for hsh, k, v in D if keyhash == hsh and key == k)
except StopIteration:
raise TypeError('Key not found in the dictionary.')
That said, you claim you want to do a binary search, but the above is still a linear search, just optimized to avoid redundant work and to stop when the desired key is found (it checks hash first, assuming key comparison is expensive, then checks key equality only on hash match, since you complained about issues with duplicates). If the goal is binary search, and D is sorted by hash code, you'd want to use the bisect module. It's not trivial to do so (because bisect doesn't take a key argument like sorted), but if you could split D into two parts, one with just hash codes, and one with codes, keys and values, you could do:
import bisect
def get(key, Dhashes, D, hasher=hash):
keyhash = hasher(key)
# Search whole list of hashes for beginning of range with correct hash
start = bisect.bisect_left(Dhashes, keyhash)
# Search for end point of correct hashes (limit to entries after start for speed)
end = bisect.bisect_right(Dhashes, keyhash, start)
try:
# Linear search of only start->end indices for exact key
return next(v for hsh, k, v in D[start:end] if key == k)
except StopIteration:
raise TypeError('Key not found in the dictionary.')
That gets you true binary search, but as noted, requires that the hash codes be separated from the complete tuples of hashcode, key, value ahead of time, before the search. Splitting the hash codes at the time of each search wouldn't be worth it since the loop that split them off could have just found your desired value directly (it would only be worth splitting if you were performing many searches at once).
As Padraic notes in his answer, at the expense of giving up the C accelerator code, you could copy and modify the pure Python implementation of bisect.bisect_right and bisect.bisect_left changing each use of a[mid] to a[mid][0] which would get you bisect code that doesn't require you to maintain a separate list of hashes. The memory savings might be worth the higher lookup costs. Don't use itertools.islice to perform the slicing though, as islice with a start index iterates the whole list up to that point; true slicing only reads and copies what you care about. If you want to avoid the second bisect operation though, you could always write your own Sequence-optimized islice and combine it with itertools.takewhile to get a similar effect without having to calculate the end index up front. Code for that might be something like:
from itertools import takewhile
# Copied from bisect.bisect_left, with unused arguments removed and only
# index 0 of each tuple checked
def bisect_idx0_left(a, x):
lo, hi = 0, len(a)
while lo < hi:
mid = (lo+hi)//2
if a[mid][0] < x: lo = mid+1
else: hi = mid
return lo
def sequence_skipper(seq, start):
return (seq[i] for i in xrange(start, len(seq)))
def get(key, D, hasher=hash):
keyhash = hasher(key)
# Search whole list of hashes for beginning of range with correct hash
start = bisect_idx0_left(D, keyhash)
# Make lazy iterator that skips start values in the list
# and stops producing values when the hash stops matching
hashmatches = takewhile(lambda x: keyhash == x[0], sequence_skipper(D, start))
try:
# Linear search of only indices with matching hashes for exact key
return next(v for hsh, k, v in hashmatches if key == k)
except StopIteration:
raise TypeError('Key not found in the dictionary.')
Note: You could save even more work at the expense of more memory, by having Dhashes actually be (hashcode, key) pairs; assuming uniqueness, this would mean a single bisect.bisect* call, not two, and no need for a scan between indices for a key match; you either found it in the binary search or you didn't. Just for example, I generated 1000 key value pairs, storing them as either (hashcode, key, value) tuples in a list (which I sorted on the hashcode), or a dict mapping keys->values. The keys were all 65 bit ints (long enough that the hash code wasn't a trivial self-mapping). Using the linear search code I provided up top, it took ~15 microseconds to find the value located at index 321; with binary search (having copied hashes only to a separate list) it took just over 2 microseconds. Looking it up in the equivalent dict took ~55 _nano_seconds; the run time overhead even for binary search worked out to ~37x, and linear search ran ~270x higher. And that's before we get into the increased memory costs, increased code complexity, and increased overhead to maintain sorted order (assuming D is ever modified).
Lastly: You say "I'd like to avoid using [dicts]", but give no explanation as to why. dicts are the correct way to solve a problem like this; assuming no self-hashing (i.e. key is an int that hashes to itself, possibly saving the cost of the hash code), the memory overhead just for the list of tuples (not including a separate list of hash codes) would be (roughly) twice that of a simple dict mapping keys to values. dict would also prevent accidentally storing duplicates, have ~O(1) insertion cost (even with bisect, insertion maintaining sorted order would have ~O(log n) lookup and O(n) memory move costs), ~O(1) lookup cost (vs. ~O(log n) with bisect), and beyond the big-O differences, would do all the work using C built-in functions that are heavily optimized, so the real savings would be greater.

You could modify bisect to just check the first element:
def bisect_left(a, x, lo=0, hi=None):
if lo < 0:
raise ValueError('lo must be non-negative')
if hi is None:
hi = len(a)
while lo < hi:
mid = (lo+hi) // 2
if a[mid][0] < x:
lo = mid+1
else: hi = mid
return lo
def get_bis(key, d):
h = hash(key)
ind = bisect_left(d, h)
if ind == -1:
raise KeyError()
for i in xrange(ind, len(d)):
if d[i][0] != h:
raise KeyError()
if d[i][1] == key:
return d[i][2]
raise KeyError()
replicating some collisions, it does what it should:
In [41]: l = [(-5092793511388848640, 'test1', 1), (-5092793511388848639, 'test9', 0), (-5092793511388848639, 'test0', 3), (-5092793511388848637, 'test2', 2), (-5092793511388848636, 'test5', 5)]
In [42]: get("test0", l)
Out[42]: 3
In [43]: get("test1", l)
Out[43]: 1
In [44]: get(-5092793511388848639, l)
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-44-81e928da1ac8> in <module>()
----> 1 get(-5092793511388848639, l)
<ipython-input-30-499e71432196> in get(key, d)
6 for sub in islice(d, ind, None):
7 if sub[0] != h:
----> 8 raise KeyError()
9 if sub[1] == key:
10 return sub
KeyError:
Some timings:
In [91]: l = sorted((hash(s), s,randint(1,100000)) for s in ("".join(sample(ascii_letters,randint(10,26))) for _ in xrange(1000000)))
In [92]: l[-1]
Out[92]: (9223342880888029755, 'FocWPinpYZXjHhBqRkJxQeGMa', 43768)
In [93]: timeit get_bis(l[-1][1],l)hed
100000 loops, best of 3: 5.29 µs per loop
In [94]: l[250000]
Out[94]: (-4616437486317828880, 'qXsybdhFPLczWwCQkm', 86136)
In [95]: timeit get_bis(l[250000][1],l)
100000 loops, best of 3: 4.4 µs per loop
In [96]: l[750000]
Out[96]: (4623630109115829672, 'dlQewhpMoBGmn', 39904)
In [97]: timeit get_bis(l[750000][1],l)
100000 loops, best of 3: 4.46 µs per loop
To get a better idea you would have to throw in collisions but to find the section that the hash may belong is pretty efficient.
Just type casting a few variables and compiling with cython:
def cython_bisect_left(a, long x, long lo=0):
if lo < 0:
raise ValueError('lo must be non-negative')
cdef long hi = len(a)
while lo < hi:
mid = (lo + hi) // 2
if a[mid][0] < x:
lo = mid + 1
else:
hi = mid
return lo
def cython_get(str key, d):
cdef long h = hash(key)
cdef ind = cython_bisect_left(d, h)
if ind == -1:
raise KeyError()
for i in xrange(ind, len(d)):
if d[i][0] != h:
raise KeyError()
if d[i][1] == key:
return d[i][2]
raise KeyError()
Gets us almost down to 1 microsecond:
In [13]: timeit cython_get(l[-1][1],l)
The slowest run took 40.77 times longer than the fastest. This could mean that an intermediate result is being cached
1000000 loops, best of 3: 1.44 µs per loop
In [14]: timeit cython_get(l[250000][1],l)
1000000 loops, best of 3: 1.33 µs per loop
In [15]: timeit cython_get(l[750000][1],l)
1000000 loops, best of 3: 1.33 µs per loop

Try using list comprehensions. I'm not sure if it's the most efficient way, but it's the pythonic way and quite effective!
[ x for x in D if x[0] == hash(key) ]

Better algorithm (than using a dict) for enumerating pairs with a given sum.

Given a number, I have to find out all possible index-pairs in a given array whose sum equals that number. I am currently using the following algo:
def myfunc(array,num):
dic = {}
for x in xrange(len(array)): # if 6 is the current key,
if dic.has_key(num-array[x]): #look at whether num-x is there in dic
for y in dic[num-array[x]]: #if yes, print all key-pair values
print (x,y),
if dic.has_key(array[x]): #check whether the current keyed value exists
dic[array[x]].append(x) #if so, append the index to the list of indexes for that keyed value
else:
dic[array[x]] = [x] #else create a new array
Will this run in O(N) time? If not, then what should be done to make it so? And in any case, will it be possible to make it run in O(N) time without using any auxiliary data structure?

Will this run in O(N) time?
Yes and no. The complexity is actually O(N + M) where M is the output size.
Unfortunately, the output size is in O(N^2) worst case, for example the array [3,3,3,3,3,...,3] and number == 6 - it will result in quadric number of elements needed to be produced.
However - asymptotically speaking - it cannot be done better then this, because it is linear in the input size and output size.

Very, very simple solution that actually does run in O(N) time by using array references. If you want to enumerate all the output pairs, then of course (as amit notes) it must take O(N^2) in the worst case.
from collections import defaultdict
def findpairs(arr, target):
flip = defaultdict(list)
for i, j in enumerate(arr):
flip[j].append(i)
for i, j in enumerate(arr):
if target-j in flip:
yield i, flip[target-j]
Postprocessing to get all of the output values (and filter out (i,i) answers):
def allpairs(arr, target):
for i, js in findpairs(arr, target):
for j in js:
if i < j: yield (i, j)

This might help - Optimal Algorithm needed for finding pairs divisible by a given integer k
(With a slight modification, there we are seeing for all pairs divisible by given number and not necessarily just equal to given number)