Develop Reference

How to sort a list of 1billion elements in python

How to sort a list of 1 billion elements in python
Please elaborate
Assuming we have unlimited space.
Thanks for the answers, but, this question is asked in the perspective of optimizing algorithm to sort, not to work on python. This question is asked in an interview, in the context of, having large number of elements, may be integers or strings, this probably wont be used in real world as we have techniques like pagination.

Dictionaries are unordered. They are hash tables and you are not guaranteed the order of keys in a hash table.
If you require the keys to be ordered, try the ordered dict class in collections.
If you need to sort the keys you could place them in a list, and sort the list.
my_dict = {key: value for key, value in zip(keys, values)} # Example dict
keys = [i for i in my_dict]
keys.sort()

A dictionary on its own does not store a key order. For this, you will have to use the OrderedDict which stores the order of insertion.
If you just need to iterate through the sorted keys, you can use sorted:
for key in sorted(my_dict):
# output is already sorted by dictionary key
print key, my_dict[key]
if you need to specify a special key or method, you could pass this as information to sorted. The following example sorts by value:
for key, value in sorted(my_dict.items(), key=lambda x: x[1]):
# output is sorted by value
print key, value
Sort orders can be reversed by using reversed:
for key in reversed(sorted(my_dict)):
# output is already sorted descending by dictionary key
print key, my_dict[key]
Finally, this code snippet would fill an OrderedDict with sorted key/value pairs:
from collections import OrderedDict
my_ordered_dict = OrderedDict(sorted(my_dict.items(), key=lambda t: t[0]))
Since you updated your question from dictionary to list
Sorting lists is even easier, just use sorted (again, provide a small method, if you have a different sorting key):
sorted_list = sorted(unsorted_list)

Comparing multiple values within a dictionary and returning key

I have a dictionary where the keys are an arbitrary name and the values are an mtime of a file. Example:
{'server_1': 1506286408.854673, 'server_2': 1506286219.1254442, 'server_3':1506472359.154043}
I wish to iterate over comparing two of the values from the dictionary finding the largest of the two, and returning the key of said large value and continuing to do this until there is only a single key:val pair left.
I know there is a way of "ordering" dictionaries by value with some tricks provided by outside libraries like operator and defaultdict. However, I was curious if there was an easier way to accomplish this goal and avoid trying to sort a naturally unordered structure.
So the end result I would be looking for is the first iteration to return server_3, then server_1 and then stop there.

It looks like you want to sort dictionary based on values but ignore the last one.
def keys_sorted_by_values(d):
return [ k for k, v in sorted(d.items(), key=lambda item: item[1], reverse=True) ][:-1]
server_to_mtime = {'server_1': 1506286408.854673, 'server_2': 1506286219.1254442, 'server_3':1506472359.154043}
for server in keys_sorted_by_values(server_to_mtime):
print(server)
Output
server_3
server_1

What's the fastest way to identify the 'name' of a dictionary that contains a specific key-value pair?

I'd like to identify the dictionary within the following list that contains the key-value pair 'Keya':'123a', which is ID1 in this case.
lst = {ID1:{'Keya':'123a','Keyb':456,'Keyc':789},ID2:{'Keya':'132a','Keyb':654,'Keyc':987},ID3:{'Keya':'5433a','Keyb':222,'Keyc':333},ID4:{'Keya':'444a','Keyb':777,'Keyc':666}}
It's safe to assume all dictionaries have the same key's, but have different values.
I currently have the following to identify which dictionary has the value '123a' for the key 'Keya', but is there a shorter and faster way?
DictionaryNames = map(lambda Dict: str(Dict),lst)
Dictionaries = [i[1] for i in lst.items()]
Dictionaries = map(lambda Dict: str(Dict),Dictionaries)
Dict = filter(lambda item:'123a' in item,Dictionaries)
val = DictionaryNames[Dictionaries.index(Dict[0])]
return val

If you actually had a list of dictionaries, this would be:
next(d for d in list_o_dicts if d[key]==value)
Since you actually have a dictionary of dictionaries, and you want the key associated with the dictionary, it's:
next(k for k, d in dict_o_dicts.items() if d[key]==value)
This returns the first matching value. If you're absolutely sure there is exactly one, or if you don't care which you get if there are more than one, and if you're happy with a StopIteration exception if you were wrong and there isn't one, that's exactly what you want.
If you need all matching values, just do the same with a list comprehension:
[k for k, d in dict_o_dicts.items() if d[key]==value]
That list can of course have 0, 1, or 17 values.

You can just do [name for name, d in lst.iteritems() if d['Keya']=='123a'] to get a list of all the dictionaries in lst that have that value for that key. If you know there is only one, you can get it with [name for name, d in lst.iteritems() if d['Keya']=='123a'][0]. (As Andy mentions in a comment, your name lst is misleading, since lst is actually a dictionary of dictionaries, not a list.)

Since you want the fastest, you should short-cut your search as soon as you find the data you are after. Iterating through the whole list is not necessary, nor is producing any temporary dictionary:
for key,data in lst.iteritems():
if data['Keya']=='132a':
return key #or break is not in a function

Å different way to do this is to use the appropriate data structure: Keep a "reverse map" of key-value pairs to names. If your dictionary of dictionaries is static after being built, you can build the reverse dictionary like this:
revdict = {(key, value): name
for name, subdict in dictodicts.items()
for key, value in subdict.items()}
If not, you just need to add revdict[key, value] = name for each d[name][key] = value statement and build them up in parallel.
Either way, to find the name of the dict that maps key to value, it's just:
revdict[key, value]
For (a whole lot) more information (than you actually want), and some sample code for wrapping things up in different ways… I dug up an unfinished blog post, considered editing it, and decided to not bother and just clicked Publish instead, so: Reverse dictionary lookup and more, on beyond z.

Python3: Conditional extraction of keys from a dictionary with comprehension

I need to extract those keys of a dictionary whose values pass a certain condition.
Basically, I want to do this, only in a shorter, more pythony way:
keys=[]
for key in dict:
if dict[key]==True:
keys.append(key)
This was my original idea, but it raises a ValueError:
[key for (key,val) in map if val==True]
I came up with this for the moment, but I can't help feeling it's not very nice:
[key for key in map.keys() if map[key]==True]
Is there a less messy way to it? Perhaps something obvious that I'm clearly missing?
Thanks!

Here is a way to get the keys with true values that is a lot shorter and cleaner than a comprehension (not that comprehensions are bad though):
>>> dct = {0:False, 1:True, 2:False, 3:True}
>>> list(filter(dct.get, dct))
[1, 3]
>>>

Use dict.items()
[key for key, val in dct.items() if val]
If you want to take only the keys with True values, rather than any true-ish value, you can use an equality check:
[key for key, val in dct.items() if val==True]
It is noted in PEP8 though, that one shouldn't compare boolean values using == - so don't use it unless you absolutely need to.
Also, please don't name variables dict or map (even if it's for demonstration purposes only) because they shadow the bulitins.

Iterating over a mapping yields only keys. Use map.items() instead.

python dictionary float search within range

Suppose I have some kind of dictionary structure like this (or another data structure representing the same thing.
d = {
42.123231:'X',
42.1432423:'Y',
45.3213213:'Z',
..etc
}
I want to create a function like this:
f(n,d,e):
'''Return a list with the values in dictionary d corresponding to the float n
within (+/-) the float error term e'''
So if I called the function like this with the above dictionary:
f(42,d,2)
It would return
['X','Y']
However, while it is straightforward to write this function with a loop, I don't want to do something that goes through every value in the dictionary and checks it exhaustively, but I want it to take advantage of the indexed structure somehow (or a even a sorted list could be used) to make the search much faster.

Dictionary is a wrong data structure for this. Write a search tree.

Python dictionary is a hashmap implementation. Its keys can't be compared and traversed as in search tree. So you simply can't do it using python dictionary without actually checking all keys.

Dictionaries with numeric keys are usually sorted - by key values. But you may - to be on the safe side - rearrange it as OrderedDictionary - you do it once
from collections import OrderedDict
d_ordered = OrderedDict(sorted(d.items(), key =lambda i:i[0]))
Then filtering values is rather simple - and it will stop at the upper border
import itertools
values = [val for k, val in
itertools.takewhile(lambda (k,v): k<upper, d_ordered.iteritems())
if k > lower]
As I've already stated, ordering dictionary is not really necessary - but some will say that this assumption is based on the current implementation and may change in the future.