what are my options there? I need to call a lot of appends (to the right end) and poplefts (from the left end, naturally), but also to read from the middle of the storage, which will steadily grow, by the nature of the algorithm. I would like to have all these operations in O(1).
I could implement it in C easy enough on circularly-addressed array (what's the word?) which would grow automatically when it's full; but what about Python? Pointers to other languages are appreciated too (I realize the "collections" tag is more Java etc. oriented and would appreciate the comparison, but as a secondary goal).
I come from a Lisp background and was amazed to learn that in Python removing a head element from a list is an O(n) operation. A deque could be an answer except the documentation says access is O(n) in the middle. Is there anything else, pre-built?
You can get an amortized O(1) data structure by using two python lists, one holding the left half of the deque and the other holding the right half. The front half is stored reversed so the left end of the deque is at the back of the list. Something like this:
class mydeque(object):
def __init__(self):
self.left = []
self.right = []
def pushleft(self, v):
self.left.append(v)
def pushright(self, v):
self.right.append(v)
def popleft(self):
if not self.left:
self.__fill_left()
return self.left.pop()
def popright(self):
if not self.right:
self.__fill_right()
return self.right.pop()
def __len__(self):
return len(self.left) + len(self.right)
def __getitem__(self, i):
if i >= len(self.left):
return self.right[i-len(self.left)]
else:
return self.left[-(i+1)]
def __fill_right(self):
x = len(self.left)//2
self.right.extend(self.left[0:x])
self.right.reverse()
del self.left[0:x]
def __fill_left(self):
x = len(self.right)//2
self.left.extend(self.right[0:x])
self.left.reverse()
del self.right[0:x]
I'm not 100% sure if the interaction between this code and the amortized performance of python's lists actually result in O(1) for each operation, but my gut says so.
Accessing the middle of a lisp list is also O(n).
Python lists are array lists, which is why popping the head is expensive (popping the tail is constant time).
What you are looking for is an array with (amortised) constant time deletions at the head; that basically means that you are going to have to build a datastructure on top of list that uses lazy deletion, and is able to recycle lazily-deleted slots when the queue is empty.
Alternatively, use a hashtable, and a couple of integers to keep track of the current contiguous range of keys.
Python's Queue Module may help you, although I'm not sure if access is O(1).
Related
What python data structure do you recommend with this requirements:
Fixed size defined at init
Ordered data
Adding a data at the beginning of the structure remove the data at it's end (as a queue structure)
Adding the data at the beginning return the data at it's end
Any data could be accessible, but not removed
Structure can be cleaned
I looked structure like lists but it does not provide me what I want. dequeue seems great but I don't specifically need to add or remove element at both sides.
I would like something like that:
last_object = my_struct.add(new_first_object)
Any ideas?
collections.deque initialized with maxlen is exactly what you need, it can do the operations you need in O(1), and the "removing at the end" is taken care of:
If maxlen is not specified or is None, deques may grow to an arbitrary length. Otherwise, the deque is bounded to the specified maximum length. Once a bounded length deque is full, when new items are added, a corresponding number of items are discarded from the opposite end.
just use .append and nothing else.
as per the docs, it also supports "peeking" at [0] or [-1] in O(1) (for the "Adding the data at the beginning return the data at it's end" requirement)
If you really don't want any other method to exist on your class (e.g. so IDEs don't auto-complete stuff other than your add) you can wrap deque in your own custom class the has just an add method that calls the deque's append.
example:
from collections import deque
class MyCollection:
def __init__(self, maxlen):
self.d = deque(maxlen=maxlen)
def add(self, new_first_object):
result = None if len(self.d)==0 else self.d[0]
self.d.append(new_first_object)
return result
my_struct = MyCollection(3)
my_struct.add(1)
my_struct.add(2)
my_struct.add(3) # my_struct is now full
print(my_struct.add(4))
print(my_struct.add(5))
print(my_struct.add(6))
Output:
1
2
3
I'm looking for an object in a python Heap. Technically, looking for the absence of it, but I assume the logic works similarly.
heap = []
heapq.heappush(heap, (10, object))
if object not in [k for v, k in heap]:
## code goes here ##
However, this check is the longest (most processor-intensive) part of my program at large numbers of elements in the heap.
Can this search be optimized? And if so, how?
heapq is a binary heap implementation of a priority queue. A binary heap makes a pretty efficient priority queue but, as you've discovered, finding an item requires a sequential search.
If all you need to know is whether an item is in the queue, then your best bet is probably to maintain a dictionary along with the queue. So when you add something to the queue, your code is something like:
"""
I'm not really a python guy, so the code probably has syntax errors.
But I think you get the idea.
"""
theQueue = [];
queueIndex = {};
queueInsert(item)
if (item.key in queueIndex)
// item already in queue. Exit.
heapq.heappush(theQueue, item);
queueIndex[item.key] = 1
queuePop()
result = heapq.heappop();
del queueIndex[result.key];
return result;
Note that if the item you're putting on the heap is a primitive like a number or a string, then you'd replace item.key with item.
Also, note that this will not work correctly if you can place duplicates in the queue. You could modify it to allow that, though. You'd just need to maintain the count of items so that you don't remove from the index until the count goes to 0.
You can't do it with heapq, but here's a compatible implementation that works as long as the heap will not contain multiple copies of the same element.
https://github.com/elplatt/python-priorityq
I have implemented LFU cache in python with the help of Priority Queue Implementation given at
https://docs.python.org/2/library/heapq.html#priority-queue-implementation-notes
I have given code in the end of the post.
But I feel that code has some serious problems:
1. To give a scenario, suppose there is only one page is continuously getting visited (say 50 times). But this code will always mark the already added node as "removed" and add it to heap again. So basically it will have 50 different nodes for the same page. Hence increasing heap size enormously.
2. This question is almost similar to Q1 of Telephonic Interview of
http://www.geeksforgeeks.org/flipkart-interview-set-2-sde-2/
And the person mentioned that doubly linked list can give better efficiency as compared to heap. Can anyone explain me, how?
from llist import dllist
import sys
from heapq import heappush, heappop
class LFUCache:
heap = []
cache_map = {}
REMOVED = "<removed-task>"
def __init__(self, cache_size):
self.cache_size = cache_size
def get_page_content(self, page_no):
if self.cache_map.has_key(page_no):
self.update_frequency_of_page_in_cache(page_no)
else:
self.add_page_in_cache(page_no)
return self.cache_map[page_no][2]
def add_page_in_cache(self, page_no):
if (len(self.cache_map) == self.cache_size):
self.delete_page_from_cache()
heap_node = [1, page_no, "content of page " + str(page_no)]
heappush(self.heap, heap_node)
self.cache_map[page_no] = heap_node
def delete_page_from_cache(self):
while self.heap:
count, page_no, page_content = heappop(self.heap)
if page_content is not self.REMOVED:
del self.cache_map[page_no]
return
def update_frequency_of_page_in_cache(self, page_no):
heap_node = self.cache_map[page_no]
heap_node[2] = self.REMOVED
count = heap_node[0]
heap_node = [count+1, page_no, "content of page " + str(page_no)]
heappush(self.heap, heap_node)
self.cache_map[page_no] = heap_node
def main():
cache_size = int(raw_input("Enter cache size "))
cache = LFUCache(cache_size)
while 1:
page_no = int(raw_input("Enter page no needed "))
print cache.get_page_content(page_no)
print cache.heap, cache.cache_map, "\n"
if __name__ == "__main__":
main()
Efficiency is a tricky thing. In real-world applications, it's often a good idea to use the simplest and easiest algorithm, and only start to optimize when that's measurably slow. And then you optimize by doing profiling to figure out where the code is slow.
If you are using CPython, it gets especially tricky, as even an inefficient algorithm implemented in C can beat an efficient algorithm implemented in Python due to the large constant factors; e.g. a double-linked list implemented in Python tends to be a lot slower than simply using the normal Python list, even for cases where in theory it should be faster.
Simple algorithm:
For an LFU, the simplest algorithm is to use a dictionary that maps keys to (item, frequency) objects, and update the frequency on each access. This makes access very fast (O(1)), but pruning the cache is slower as you need to sort by frequency to cut off the least-used elements. For certain usage characteristics, this is actually faster than other "smarter" solutions, though.
You can optimize for this pattern by not simply pruning your LFU cache to the maximum length, but to prune it to, say, 50% of the maximum length when it grows too large. That means your prune operation is called infrequently, so it can be inefficient compared to the read operation.
Using a heap:
In (1), you used a heap because that's an efficient way of storing a priority queue. But you are not implementing a priority queue. The resulting algorithm is optimized for pruning, but not access: You can easily find the n smallest elements, but it's not quite as obvious how to update the priority of an existing element. In theory, you'd have to rebalance the heap after every access, which is highly inefficient.
To avoid that, you added a trick by keeping elements around even if they are deleted. But this trades in space for time.
If you don't want to trade in time, you could update the frequencies in-place, and simply rebalance the heap before pruning the cache. You regain fast access times at the expense of slower pruning time, like the simple algorithm above. (I doubt there is any speed difference between the two, but I have not measured this.)
Using a double-linked list:
The double-linked list mentioned in (2) takes advantage of the nature of the possible changes here: An element is either added as the lowest priority (0 accesses), or an existing element's priority is incremented exactly by 1. You can use these attributes to your advantage if you design your data structures like this:
You have a double-linked list of elements which is ordered by the frequency of the elements. In addition, you have a dictionary that maps items to elements within that list.
Accessing an element then means:
Either it's not in the dictionary, that is, it's a new item, in which case you can simply append it to the end of the double-linked list (O(1))
or it's in the dictionary, in which case you increment the frequency in the element and move it leftwards through the double-linked list until the list is ordered again (O(n) worst-case, but usually closer to O(1)).
To prune the cache, you simply cut off n elements from the end of the list (O(n)).
The Python class has six requirements as listed below. Only bold terms are to be read as requirements.
Close to O(1) performance for as many of the following four operations.
Maintaining sorted order while inserting an object into the container.
Ability to peek at last value (the largest value) contained in the object.
Allowing for pops on both sides (getting the smallest or largest values).
Capability of getting the total size or number of objects being stored.
Being a ready made solution like the code in Python's standard library.
What follows is left here for historical reasons (help the curious and prove that research was conducted).
After looking through Python's Standard Library (specifically the section on Data Types), I still have not found a class that fulfills the requirements requirements of a fragmentation table. collections.deque is close to what is required, but it does not support keeping the data contained in it sorted. It provides:
Efficient append and pops on either side of a deque with O(1) performance.
Pops on both sides for the data contained within the object.
Getting the total size or count of objects contained within.
Implementing an inefficient solution using lists would be trivial, but finding a class that performs well would be far more desirable. In a growing memory simulation with no upper limit, such a class could keep indexes of empty (deleted) cells and keep fragmentation levels down. The bisect module may help:
Helps keep an array in sorted order while inserting new objects in array.
Ready made solution for keeping lists sorted as objects are added.
Would allow executing array[-1] to peek at last value in the array.
The final candidate that failed to fully satisfy the requirements and appeared least promising was the heapq module. While supporting what looked like efficient insertions and assuring that array[0] was the smallest value, the array is not always in a fully sorted state. Nothing else was found to be as helpful.
Does anyone know of a class or data structure in Python that comes close to these six requirements?
Your requirements seem to be:
O(1) pop from each end
Efficient len
Sorted order
Peek at last value
for which you can use a deque with a custom insert method which rotates the deque, appends to one end, and unrotates.
>>> from collections import deque
>>> import bisect
>>> class FunkyDeque(deque):
... def _insert(self, index, value):
... self.rotate(-index)
... self.appendleft(value)
... self.rotate(index)
...
... def insert(self, value):
... self._insert(bisect.bisect_left(self, value), value)
...
... def __init__(self, iterable):
... super(FunkyDeque, self).__init__(sorted(iterable))
...
>>> foo = FunkyDeque([3,2,1])
>>> foo
deque([1, 2, 3])
>>> foo.insert(2.5)
>>> foo
deque([1, 2, 2.5, 3])
Notice that requirements 1, 2, and 4 all follow directly from the fact that the underlying data structure is a deque, and requirement 3 holds because of the way data is inserted. (Note of course that you could bypass the sorting requirement by calling e.g. _insert, but that's beside the point.)
Many thanks go out to katrielalex for providing the inspiration that led to the following Python class:
import collections
import bisect
class FastTable:
def __init__(self):
self.__deque = collections.deque()
def __len__(self):
return len(self.__deque)
def head(self):
return self.__deque.popleft()
def tail(self):
return self.__deque.pop()
def peek(self):
return self.__deque[-1]
def insert(self, obj):
index = bisect.bisect_left(self.__deque, obj)
self.__deque.rotate(-index)
self.__deque.appendleft(obj)
self.__deque.rotate(index)
blist.sortedlist
Close to O(1) performance for as many of the following four operations.
Maintaining sorted order while inserting an object into the container.
Ability to peek at last value (the largest value) contained in the object.
Allowing for pops on both sides (getting the smallest or largest values).
Capability of getting the total size or number of objects being stored.
Being a ready made solution like the code in Python's standard library.
It's a B+ Tree.
What is your preferred method of traversing a tree data structure, since recursive method calls can be pretty inefficient in some circumstances. I am simply using a generator like the one above. Do you have any hints to make it faster?
def children(self):
stack = [self.entities]
while stack:
for e in stack.pop():
yield e
if e.entities:
stack.append(e.entities)
Here is some test data.
The first one is recursive, the second uses the generator:
s = time.time()
for i in range(100000):
e.inc_counter()
print time.time() - s
s = time.time()
for i in range(100000):
for e in e.children():
e.inc_counter_s()
print time.time() - s
Results:
0.416000127792
0.298999786377
Test code:
import random
class Entity():
def __init__(self, name):
self.entities = []
self.name = name
self.counter = 1
self.depth = 0
def add_entity(self, e):
e.depth = self.depth + 1
self.entities.append(e)
def inc_counter_r(self):
for e in self.entities:
e.counter += 1
e.inc_counter_r()
def children(self):
stack = [self.entities]
while stack:
for e in stack.pop():
yield e
if e.entities:
stack.append(e.entities)
root = Entity("main")
def fill_node(root, max_depth):
if root.depth <= max_depth:
for i in range(random.randint(10, 15)):
e = Entity("node_%s_%s" % (root.depth, i))
root.add_entity(e)
fill_node(e, max_depth)
fill_node(root, 3)
import time
s = time.time()
for i in range(100):
root.inc_counter_r()
print "recursive:", time.time() - s
s = time.time()
for i in range(100):
for e in root.children():
e.counter += 1
print "generator:", time.time() - s
I can't think of any big algorithmic improvements, but a simple microoptimisation you can make is to bind frequently called methods (such as stack.append / stack.pop) to locals (this saves a dictionary lookup)
def children(self):
stack = [self.entities]
push = stack.append
pop = stack.pop
while stack:
for e in pop():
yield e
if e.entities:
push(e.entities)
This gives a small (~15%) speedup by my tests (using 100 traversals of an 8-deep tree with 4 children at each node gives me the below timings:)
children : 5.53942348004
children_bind: 4.77636131253
Not huge, but worth doing if speed is important.
Unless your tree is really large or you have really high (real) requirements for speed, I would choose the recursive method. Easier to read, easier to code.
Recursive function calls are not incredibly inefficient, that is an old programming myth. (If they're badly implemented, they may incur a larger overhead than necessary, but calling them "incredibly inefficient" is plain wrong.)
Remember: don't optimize prematurely, and never optimize without benchmarking first.
I'm not sure if you can reduce the overhead much on a full in-order traversal of a tree, if you use recursion the call stack will grow some, otherwise you must manually use a stack to push references of the children while visiting each node. Which way is fastest and uses less memory, depends on the expensiveness of the call stack vs. a normal stack. (I would guess the callstack is faster since it should be optimized for its use, and recursion is much easier to implement)
If you don't care about the order you visit the nodes, some implementations of trees is actually stored in a dynamic array or linked list or stack wich you can traverse linearly if you don't care about the order it's traversed.
But why is it important to have a fast traversal anyway? Trees are good for searching, arrays/linked lists is good for full traversal. If you often need full in-order traversal but few searches and insertions/deletions, an ordered linked list might be best, if searching is what you do most you use a tree. If the data is really massive, so that memory overhead may render recursion impossible, you should use a database.
If you have a lot of RAM and the tree doesn't change often, you can cache the result of the call:
def children(self):
if self._children_cache is not None:
return self._children_cache
# Put your code into collectChildren()
self._children_cache = self.collectChildren()
return self._children_cache
Whenever the tree changes, set the cache to None. In this case, using recursive calls might be more effective since the results will accumulate faster.
I've written iterative tree-traversal code in the past: it's very ugly, and not fast, unless you know exactly how many children not only each subtree will have, but how many levels there are.
I don't know too much about Python internals of function calls, but I really can't imagine that your code snippet is faster than recursively traversing the tree.
The call stack (used for function calls, including recursive ones) is typically very fast. Going to the next object will only cost you a single function call. But in your snippet - where you use a stack object, going to the next object will cost you a stack.append (possibly allocating memory on heap), a stack.push (possibly freeing memory from heap), and a yield.
The main problem with recursive calls is that you might blow the stack if your tree gets too deep. This isn't likely to happen.
Here's a pair of small corrections.
def children(self):
stack = [self.entities]
for e in stack:
yield e
if e.entities:
stack.extend(e.entities)
I actually think the generator, using append, isn't visiting all the nodes. I think you mean to extend the stack with all entities, not append a simple list of entities to the stack.
Also, when the for loop terminates, the while loop in your original example will also terminate because there's no change to the empty stack after the for loop.