Develop Reference

How to reduce/optimize memory usage when calculating area of skyline?

I'm trying to calculate the area of skyline (overlapping rectangles with same baseline)
building_count = int(input())
items = {} # dictionary, location on x axis is the key, height is the value
count = 0 # total area
for j in range(building_count):
line = input().split(' ')
H = int(line[0]) # height
L = int(line[1]) # left point (start of the building)
R = int(line[2]) # right point (end of the building)
for k in range(R - L):
if not (L+k in items): # if it's not there, add it
items[L+k] = H
elif H > items[L+k]: # if we have a higher building on that index
items[L+k] = H
for value in items.values(): # we add each column basically
count += value
print(count)
sample input would be:
5
3 -3 0
2 -1 1
4 2 4
2 3 7
3 6 8
and output is 29.
The issue is memory efficiency, when there are lots of values, the script simply throws MemoryError. Anyone have some ideas for optimizing memory usage?

You are allocating a separate key-value pair for every single integer value in your range. Imagine the case where R = 1 and L = 100000. Your items dictionary will be filled with 1000000 items. Your basic idea of processing/removing overlaps is is sound, but the way you do it is massive overkill.
Like so much else in life, this is a graph problem in disguise. Imaging the vertices being the rectangles you are trying to process and the (weighted) edges being the overlaps. The complication is that you can not just add up the areas of the vertices and subtract the areas of the overlaps, because many of the overlaps overlap each other as well. The overlap issue can be resolved by applying a transformation that converts two overlapping rectangles into non-overlapping rectangles, effectively cutting the edge that connects them. The transformation is shown in the image below. Notice that in some cases one of the vertices will be removed as well, simplifying the graph, while in another case a new vertex is added:
Green: overlap to be chopped out.
Normally, if we have m rectangles and n overlaps between them, constructing the graph would be an O(m2) operation because we would have to check all vertices for overlaps against each other. However, we can bypass a construction of the input graph entirely to get a O(m + n) traversal algorithm, which is going to be optimal since we will only analyze each rectangle once, and construct the output graph with no overlaps as efficiently as possible. O(m + n) assumes that your input rectangles are sorted according to their left edges in ascending order. If that is not the case, the algorithm will be O(mlog(m) + n) to account for the initial sorting step. Note that as the graph density increases, n will go from ~m to ~m2. This confirms the intuitive idea that the fewer overlaps there are, them more you would expect the process will run in O(m) time, while the more overlaps there are, the closer you will run to O(m2) time.
The space complexity of the proposed algorithm will be O(m): each rectangle in the input will result in at most two rectangles in the output, and 2m = O(m).
Enough about complexity analysis and on to the algorithm itself. The input will be a sequence of rectangles defined by L, R, H as you have now. I will assume that the input is sorted by the leftmost edge L. The output graph will be a linked list of rectangles defined by the same parameters, sorted in descending order by the rightmost edge. The head of the list will be the rightmost rectangle. The output will have no overlaps between any rectangles, so the total area of the skyline will just be the sum of H * (R - L) for each of the ~m output rectangles.
The reason for picking a linked list is that the only two operations we need is iteration from the head node and the cheapest insertion possible to maintain the list in sorted order. The sorting will be done as part of overlap checking, so we do not need to do any kind of binary searches through the list or anything like that.
Since the input list is ordered by increasing left edge and the output list is ordered by decreasing right edge, we can guarantee that each rectangle added will be checked only against the rectangles it actually overlaps1. We will do overlap checking and removal as shown in the diagram above until we reach a rectangle whose left edge is less than or equal to the left edge of the new rectangle. All further rectangles in the output list are guaranteed not to overlap with the new rectangle. This check-and-chop operation guarantees that each overlap is visited at most once, and that no non-overlapping rectangles are processed unnecessarily, making the algorithm optimal.
Before I show code, here is a diagram of the algorithm in action. Red rectangles are new rectangles; note that their left edges progress to the right. Blue rectangles are ones that are already added and have overlap with the new rectangle. Black rectangles are already added and have no overlap with the new one. The numbering represents the order of the output list. It is always done from the right. A linked list is a perfect structure to maintain this progression since it allows cheap insertions and replacements:
Here is an implementation of the algorithm which assumes that the input coordinates are passed in as an iterable of objects having the attributes l, r, and h. The iteration order is assumed to be sorted by the left edge. If that is not the case, apply sorted or list.sort to the input first:
from collections import namedtuple
# Defined in this order so you can sort a list by left edge without a custom key
Rect = namedtuple('Rect', ['l', 'r', 'h'])
class LinkedList:
__slots__ = ['value', 'next']
"""
Implements a singly-linked list with mutable nodes and an iterator.
"""
def __init__(self, value=None, next=None):
self.value = value
self.next = next
def __iter__(self):
"""
Iterate over the *nodes* in the list, starting with this one.
The `value` and `next` attribute of any node may be modified
during iteration.
"""
while self:
yield self
self = self.next
def __str__(self):
"""
Provided for inspection purposes.
Works well with `namedtuple` values.
"""
return ' -> '.join(repr(x.value) for x in self)
def process_skyline(skyline):
"""
Turns an iterable of rectangles sharing a common baseline into a
`LinkedList` of rectangles containing no overlaps.
The input is assumed to be sorted in ascending order by left edge.
Each element of the input must have the attributes `l`, r`, `h`.
The output will be sorted in descending order by right edge.
Return `None` if the input is empty.
"""
def intersect(r1, r2, default=None):
"""
Return (1) a flag indicating the order of `r1` and `r2`,
(2) a linked list of between one and three non-overlapping
rectangles covering the exact same area as `r1` and `r2`,
and (3) a pointer to the last nodes (4) a pointer to the
second-to-last node, or `default` if there is only one node.
The flag is set to True if the left edge of `r2` is strictly less
than the left edge of `r1`. That would indicate that the left-most
(last) chunk of the tuple came from `r2` instead of `r1`. For the
algorithm as a whole, that means that we need to keep checking for
overlaps.
The resulting list is always returned sorted descending by the
right edge. The input rectangles will not be modified. If they are
not returned as-is, a `Rect` object will be used instead.
"""
# Swap so left edge of r1 < left edge of r2
if r1.l > r2.l:
r1, r2 = r2, r1
swapped = True
else:
swapped = False
if r2.l >= r1.r:
# case 0: no overlap at all
last = LinkedList(r1)
s2l = result = LinkedList(r2, last)
elif r1.r < r2.r:
# case 1: simple overlap
if r1.h > r2.h:
# Chop r2
r2 = Rect(r1.r, r2.r, r2.h)
else:
r1 = Rect(r1.l, r2.l, r1.h)
last = LinkedList(r1)
s2l = result = LinkedList(r2, last)
elif r1.h < r2.h:
# case 2: split into 3
r1a = Rect(r1.l, r2.l, r1.h)
r1b = Rect(r2.r, r1.r, r1.h)
last = LinkedList(r1a)
s2l = LinkedList(r2, last)
result = LinkedList(r1b, s2l)
else:
# case 3: complete containment
result = LinkedList(r1)
last = result
s2l = default
return swapped, result, last, s2l
root = LinkedList()
skyline = iter(skyline)
try:
# Add the first node as-is
root.next = LinkedList(next(skyline))
except StopIteration:
# Empty input iterator
return None
for new_rect in skyline:
prev = root
for rect in root.next:
need_to_continue, replacement, last, second2last = \
intersect(rect.value, new_rect, prev)
# Replace the rectangle with the de-overlapped regions
prev.next = replacement
if not need_to_continue:
# Retain the remainder of the list
last.next = rect.next
break
# Force the iterator to move on to the last node
new_rect = last.value
prev = second2last
return root.next
Computing the total area is now trivial:
skyline = [
Rect(-3, 0, 3), Rect(-1, 1, 2), Rect(2, 4, 4),
Rect(3, 7, 2), Rect(6, 8, 3),
]
processed = process_skyline(skyline)
area = sum((x.value.r - x.value.l) * x.value.h for x in processed) if processed else None
Notice the altered order of the input parameters (h moved to the end). The resulting area is 29. This matches with what I get by doing the computation by hand. You can also do
>>> print(processed)
Rect(l=6, r=8, h=3) -> Rect(l=4, r=6, h=2) -> Rect(l=2, r=4, h=4) ->
Rect(l=0, r=1, h=2) -> Rect(l=-3, r=0, h=3)
This is to be expected from the diagram of the inputs/output shown below:
As an additional verification, I added a new building, Rect(-4, 9, 1) to the start of the list. It overlaps all the others and adds three units to area, or a final result of 32. processed comes out as:
Rect(l=8, r=9, h=1) -> Rect(l=6, r=8, h=3) -> Rect(l=4, r=6, h=2) ->
Rect(l=2, r=4, h=4) -> Rect(l=1, r=2, h=1) -> Rect(l=0, r=1, h=2) ->
Rect(l=-3, r=0, h=3) -> Rect(l=-4, r=-3, h=1)
Note:
While I am sure that this problem has been solved many times over, the solution I present here is entirely my own work, done without consulting any other references. The idea of using an implicit graph representation and the resulting analysis is inspired by a recent reading of Steven Skiena's Algorithm Design Manual, Second Edition. It is one of the best comp-sci books I have ever come across.
1 Technically, if a new rectangle does not overlap any other rectangles, it will be checked against one rectangle it does not overlap. If that extra check was always the case, the algorithm would have an additional m - 1 comparisons to do. Fortunately, m + m + n - 1 = O(m + n) even if we always had to check one extra rectangle (which we don't).

The reason for getting MemoryError is huge size of the dictionary being created. In the worst case, the dict can have 10^10 keys, which would end up taking all your memory. If there really is a need, shelve is a possible solution to make use of such large dict.
Let's say there is a building with 10 0 100 and another with 20 50 150, then that list might have info like [(-10^9, 0), (0, 10), (50, 20), (150, 0), (10^9, 0)]. As you come across more buildings, you can add more entries in this list. This will be O(n^2).
This might help you further.

recursion and multi dimension matrix in python

this is famous path counting problem , i am trying to solve it using memoization.
Enlighten me!
def pathCounter(a,b):
matrix = [[0 for i in xrange(a)] for i in xrange(b)]
if a==0 or b==0:
return 1
if matrix[a][b]:
return matrix[a][b]
print matrix[a][b]
matrix[a][b]=pathCounter(a,b-1)+pathCounter(a-1,b)
return matrix[2][2]
if __name__=='__main__':
k=pathCounter(2,2)
print k

I believe your trying to solve this problem.
If that is the case, then you are correct that it would be sensible to solve with recursion.
If you imagine each corner of the grid as a node, then you want to a recursive function that simply takes a parameter of the node it is at (x, y). In the function, it first needs to check if the position that it was called at is the bottom right vertex of the grid. If it is, the function adds one to the path count (as a path is finished when it reaches this corner) and then returns. Otherwise, this function just calls two more of itself (this is the recursion), one to its right (so y+1) and one to its left (x+1). An added step is to check that the coordinates are in the grid before calling them as a node in the middle of the bottom row for instance shouldn't call a node below it as that would be off the grid.
Now you have the recursive function defined, all you need to do now is declare a variable to store the path count. And call the recursive function from the coordinate (0,0).
However, as I am sure you have seen, this solution does not complete in reasonable time so it is necessary that you use memoization - speeding it up by caching the nodes so that the same sections of paths aren't calculated twice.
It also makes coding it more simple if as you have done, we work from the bottom right corner up to the top left corner. One last thing is that if you use a dictionary then the code becomes clearer.
The final code should look something like:
cache = {}
def pathCounter(x, y):
if x == 0 or y == 0:
return 1
if (x,y) in cache:
return cache[(x,y)]
cache[(x,y)] = pathCounter(x, y-1) + pathCounter(x-1, y)
return cache[(x,y)]
print(pathCounter(2,2))
this gives the expected result of 6.
I'll leave you to do the 20x20 grid. Hope this helps!

You made a few errors in your implementation of the algorithm. If your using a recursive approach you do not have to use the grid because you want require any of the stored data, actually. You only need to return the two possible sub-paths from your current position - that's it! Therefore, you need to make some changes in the main idea of your code.
I tried to keep as much of your original code as possible, but still make it working:
def pathCounterNaive(width, height, startX = 0, startY = 0):
if startX >= width or startY >= height:
return 0
if startX == width-1 and startY == height-1:
return 1
return pathCounter(width,height, startX+1, startY) + pathCounter(width,height, startX, startY+1)
slowK=pathCounterNaive(3,3)
print(slowK)
Please keep in mind, that the parameters width and height represent the number of vertices, and are therefore not 2 but 3 for a 2x2 grid. As this code is using pure recursion it is very slow. If you want to use your memorization approach, you have to modify your code like this:
import numpy as np
def pathCounter(width, height):
grid = np.zeros((height+1, width+1))
def pathCounterInternal(x, y):
if x==0 or y==0:
return 1
grid[x, y] = pathCounterInternal(x,y-1)+pathCounterInternal(x-1,y)
return grid[x, y]
grid[width, height] = pathCounterInternal(width, height)
return grid[width, height]
k=pathCounter(2,2)
print(k)
Here you have to call it with 2 as the parameter for a 2x2 grid. This code is much faster due to the caching of already calculated paths.

How to manage memory in recursive function that creates new lists with each recursive function call?

I'm writing a function to count the possible paths of length M in an N-dimensions grid starting from points specified in list X_n where the dimensions of the grid are given in list D_n.
I am passing the current coordinates and dimensions of the grid to the Solve function with lists. However, this means I am making many list copies, as can be seen below.
def Solve(X_n, D_n, M):
"""Find number of paths of length M in grid of dimensions D_n we can get starting at X_n"""
#using dynamic methods to avoid unnecessary calculations by storing number of paths
# from a given grid member that are available. This will be 2^N except for edge cases
if M == 1:
return solve_grid[tuple(X_n)] if solve_grid[tuple(X_n)] else count_em(X_n, D_n)
#identify which current coordinates are at an edge and don't modify these in wrong direction
diff = np.subtract(D_n, X_n)
good_add = [i for i, d in enumerate(diff) if d!= 0]
good_subtract = [i for i, x in enumerate(X_n) if x != 0]
return_val = 0
#here's my concern: what's going to happen with all these lists?
#is there a better way to handle this?
for idx in good_add:
new_x = list(X_n)
new_x[idx] += 1
return_val += Solve(new_x, D_n, M-1)
for idx in good_subtract:
new_x = list(X_n)
new_x[idx] -= 1
return_val += Solve(new_x, D_n, M-1)
return return_val
My understanding is that Python has a write-only type memory management system, so when I am creating many duplicate lists (as this function certainly will), these duplicate objects will all point to the same item so long as they are not modified. This seems like the above should be not so bad...
Is this ok memory-wise or will it blow up?
If it will blow up, what's a smart way to do this with lists (they're so convenient)?
If not, is using an f(*args) style function the best way to recursively pass coordinates and dimensions?

Map Color Algorithm in Python

I have a 2D array in Python (version 3.2) that looks like this:
...AAA....
...AAABB..
..BBBBBCC.
.....CCCC.
.DDD..CC..
.DDD......
It represents a kind of map with areas painted different colors. The above example shows four distinct regions, A, B, C, and D.
Here's an example of indexing the array:
map[1][5] == 'A' would return True.
I'm trying to write a function that takes in an array like this, and a row/col index, and returns the number of adjoining spaces that are of the same "color". So using that example above, here are some return values (the arguments are the array, row, and column number respectively:
6 <-- countArea(map, 5, 2)
8 <-- countArea(map, 2, 8)
I'd like to implement this as a recursive function, but I can't figure it out. Here's what I have so far:
def countArea(map, row, col):
key = map[row][col]
if (map[row-1][col] == key):
return 1 + countArea(map, row-1, col)
elif (map[row+1][col] == key):
return 1 + countArea(map, row+1, col)
elif (map[row][col+1] == key):
return 1 + countArea(map, row, col+1)
elif (map[row][col-1] == key):
return 1 + countArea(map, row, col-1)
else:
return 1
I know I'm missing something basic here. I'm basically saying "here is the current character, now look in each direction to see if it has the same character."
My question is, what am I missing in this recursive definition?
Thanks for your help.

My question is, what am I missing in this recursive definition?
Once a grid square has been counted, it must not be counted again (this includes counting by recursive invocations of countArea()!)
Your current algorithm goes as far north as it can, and then keeps taking one step to the south followed by one step to the north. This two-step sequence repeats until you run out of stack space.
If you like, you could read up on algorithms for this problem in Wikipedia.

In your code the algorithm would look one field left of a given input field and in the recursive call would again call the function on the initial field. (What you obviously don't want since it would lead to an infinite recursion)
Approach 1
A method to overcome this problem while still using recursion would be to specify a direction where the recursion should look for more fields of the same type. For example the call to the field directly north (or above) of the initial one could look recursively farer to the north or east (or right), the one to the east go south (below) and east and so on.
By intelligently choosing the first step you can ensure, that there is no overlap in the scanned regions. However it needs some adaptions to specify the directions the recursive call should scan. BUT: Note that this algorithm would not work if the area is overhanging so if not every field northeast of the starting point can be reached by just moving right and up.
There exist more algorithms like this that are also capable to solve the the mentioned problem. Have a look at Flood Filling on wikipedia.
Approach 2
You can also save the already visited fields in some way and directly return from the recursive call if the field was already visited.

The following implementation should work:
def countArea(map, row, col, key=None, seen=None):
if key is None:
key = map[row][col]
if seen is None:
seen = set()
seen.add((row, col)) # mark this location as visited
n = 1
for dy, dx in [(0, 1), (1, 0), (-1, 0), (0, -1)]:
r, c = row + dy, col + dx
if r < 0 or r >= len(map) or c < 0 or c >= len(map[0]): # check boundaries
continue
# only increment and recurse if key matches and we haven't already visited
if map[r][c] == key and (r, c) not in seen:
n += countArea(map, r, c, key, seen)
return n
Example:
>>> print '\n'.join(''.join(row) for row in map)
...AAA....
...AAABB..
..BBBBBCC.
.....CCCC.
.DDD..CC..
.DDD......
>>> countArea(map, 5, 2)
6
>>> countArea(map, 2, 8)
8
Note that this assumes that areas with the same key that are only touching at a diagonal should be considered separate, for example for the following map countArea(map, 0, 0) and countArea(map, 1, 1) would both return 1:
A.
.A
As a side note, you should not use map as a variable name, as it will mask the builtin map() function.

Converting a few python lines in pseudocde

Goal: Trying to convert some of the lines of an algorithm written in python to pseudocode.
Goal of the given algorithm: Find all cycles in a directed graph with cycles.
Where I stand: I well understand the theory behind the algorithm, I have also coded different versions on my own, however I cannot write an algorithm that small, efficient and correct on my own.
Source: stackoverflow
What I have done so far: I cannot describe enough how many weeks spent on it, have coded Tarjan, various versions DFS, Flloyd etc in php but unfortunately they are partial solutions only and one have to extend them more.
In addition: I have run this algorithm online and it worked, I need it for a school project that I am stack and cannot proceed further.
This is the algorithm:
def paths_rec(path,edges):
if len(path) > 0 and path[0][0] == path[-1][1]:
print "cycle", path
return #cut processing when find a cycle
if len(edges) == 0:
return
if len(path) == 0:
#path is empty so all edges are candidates for next step
next_edges = edges
else:
#only edges starting where the last one finishes are candidates
next_edges = filter(lambda x: path[-1][1] == x[0], edges)
for edge in next_edges:
edges_recursive = list(edges)
edges_recursive.remove(edge)
#recursive call to keep on permuting possible path combinations
paths_rec(list(path) + [edge], edges_recursive)
def all_paths(edges):
paths_rec(list(),edges)
if __name__ == "__main__":
#edges are represented as (node,node)
# so (1,2) represents 1->2 the edge from node 1 to node 2.
edges = [(1,2),(2,3),(3,4),(4,2),(2,1)]
all_paths(edges)
This is what I have managed to write in pseudocode from it, I have marked with #? the lines I do not understand. Once I have them in pseudocode I can code them in php with which I am a lot familiar.
procedure paths_rec (path, edges)
if size(path) > 0 and path[0][0] equals path[-1][1]
print "cycle"
for each element in path
print element
end of for
return
end of if
if size(edges) equals 0
return
end of if
if size(path) equals 0
next_edges equals edges
else
next edges equals filter(lambda x: path[-1][1] == x[0], edges) #?
end of else
for each edge in next_edges
edges_recursive = list(edges) #?
edges_recursive.remove(edge)#?
#recursive call to keep on permuting possible path combinations
paths_rec(list(path) + [edge], edges_recursive)#?

The line next_edges = filter(lambda x: path[-1][1] == x[0], edges) creates a new list containing those edges in edges whose first point is the same as the second point of the last element of the current path, and is equivalent to
next_edges = []
for x in edges:
if path[len(path) - 1][1] == x[0]
next_edges.append[x]
The lines create a new copy of the list of edges edges, so that when edge is removed from this copy, it doesn't change the list edges
edges_recursive = list(edges)
edges_recursive.remove(edge)
paths_rec(list(path) + [edge], edges_recursive) is just the recursive call with the path with edge added to the end and the new list of edges that has edge removed from it.