Develop Reference

How to evaluate columns that contain lists in pandas?

Say I have a dataframe which describes the dimensions of hundreds of cardboard boxes:
df = [[17829292, (13, 14, 20)], [17739292, (20, 10, 15)], [17827792, (10, 10, 12)]]
df = pd.DataFrame(df, columns = ['Serial Number', 'Box Dimensions'])
Given that the 'Box Dimensions' column contains a list of numbers, is there a simple way to search through the dataframe for all boxes with dimensions less than or equal to certain values, <= (15, 15, 15)? Or do I need to separate out each box's (x, y, z) values into three separate columns to evaluate them this way?
I've tried df.loc[df['Box Dimensions'] <= (15, 15, 15)], but for some reason this only evaluates the first number in the list. The function will return boxes whose x values are <=15, but whose y and z dimensions exceed these parameters.

You can try:
m=pd.DataFrame(df['Box Dimensions'].tolist()).le(15).all(1)
#OR(If needed to check dimensions seperately above method can be modified as well)
m=df['Box Dimensions'].map(lambda x:x[0]<=15 and x[1]<=15 and x[2]<=15)
#Finally:
df[m]
#OR
df.loc[m]
output of above code:
Serial Number Box Dimensions
2 17827792 (10, 10, 12)

Alternatively if you need to test box dimensions independently
df.loc[(df['Box Dimensions'].str[0] <= 15) & (df['Box Dimensions'].str[1] <= 15) & (df['Box Dimensions'].str[2] <= 15)]
Serial Number Box Dimensions
2 17827792 (10, 10, 12)

python nested for loop zip

I have the following list:
grid = [[0] *50 for n in range(50)]
I want to replace the values in grid (with 1) for each coordinate contained in the list:
area = [(30, 28), (27, 32), (32, 34), (43,23), (43, 2) ...] # Continues on
Is there any simple method that this can be done?

A simple for loop is what is needed.
for i,j in area:
grid[i][j] = 1

How to check if a set of coordinates matches a tetris piece in Python

I’m working with tetris pieces.
The pieces are defined with coordinates, where each piece has an origin block (0,0)
So an L piece could be defined as [(0,0), (0,1), (0,2), (1,2)] as well as [(0,-1), (0,0), (0,1), (1,1)] depending on where you place the origin block.
I want to check whether a set of coordinates A e.g. [(50,50), (50,51), (50,52), (51,52)] matches the shape of a given tetris piece B.
I’m currently using numpy to take away one of the A values from every value in A to reach relative coordinates, then compare with B. The ordering of A will always been in increasing order, but is not guarenteed to match the ordering of B. B is stored in a list with other tetris pieces, and throughout the program, it's origin block will remain the same. This method below seems inefficient and doesn’t account for rotations / reflections of B.
def isAinB(A,B): # A and B are numpy arrays
for i in range(len(A)):
matchCoords = A - A[i]
setM = set([tuple(x) for x in matchCoords])
setB = set([tuple(x) for x in B])
if setM == setB: # Sets are used here because the ordering of M and B are not guarenteed to match
return True
return False
Is there an efficient method / function to implement this? (Accounting for rotations and reflections aswell if possible)

This is one way to approach it. The idea is to first build all the set of variations of a piece in some canonical coordinates (you can do this once per piece kind and reuse it), then put the given piece in the same canonical coordinates and compare.
# Rotates a piece by 90 degrees
def rotate_coords(coords):
return [(y, -x) for x, y in coords]
# Returns a canonical coordinates representation of a piece as a frozen set
def canonical_coords(coords):
x_min = min(x for x, _ in coords)
y_min = min(y for _, y in coords)
return frozenset((y - y_min, x - x_min) for x, y in coords)
# Makes all possible variations of a piece (optionally including reflections)
# as a set of canonical representations
def make_piece_variations(piece, reflections=True):
variations = {canonical_coords(piece)}
for i in range(3):
piece = rotate_coords(piece)
variations.add(canonical_coords(piece))
if reflections:
piece_reflected = [(y, x) for x, y in piece]
variations.update(make_piece_variations(piece_reflected, False))
return variations
# Checks if a given piece is in a set of variations
def matches_piece(piece, variations):
return canonical_coords(piece) in variations
These are some tests:
# L-shaped piece
l_piece = [(0, 0), (0, 1), (0, 2), (1, 2)]
l_piece_variations = make_piece_variations(l_piece, reflections=True)
# Same orientation
print(matches_piece([(50, 50), (50, 51), (50, 52), (51, 52)], l_piece_variations))
# True
# Rotated
print(matches_piece([(50, 50), (51, 50), (52, 50), (52, 49)], l_piece_variations))
# True
# Reflected and rotated
print(matches_piece([(50, 50), (49, 50), (48, 50), (48, 49)], l_piece_variations))
# True
# Rotated and different order of coordinates
print(matches_piece([(50, 48), (50, 50), (49, 48), (50, 49)], l_piece_variations))
# True
# Different piece
print(matches_piece([(50, 50), (50, 51), (50, 52), (50, 53)], l_piece_variations))
# False
This is not a particularly smart algorithm, but it works with minimal constraints.
EDIT: Since in your case you say that the first block and the relative order will always be the same, you can redefine the canonical coordinates as follows to make it just a bit more optimal (although the performance difference will probably be negligible and its use will be more restricted):
def canonical_coords(coords):
return tuple((y - coords[0][0], x - coords[0][1]) for x, y in coords[1:])
The first coordinate will always be (0, 0), so you can skip that and use it as reference point for the rest, and instead of a frozenset you can use a tuple for the sequence of coordinates.

find closest value pair in python list

I have a dictionary as so:
d = {'ID_1':[(10, 20), (40, 60), (125, 200)], 'ID_2': [(75, 100), (250, 300)]}
and a position and ID:
pos = 70
IDed = ID_1
output = (40, 60)
pos = 90
IDed = ID_2
expected output = (75, 100)
pos = 39
IDed = ID_1
expected output = (40, 60)
I would like to find the value pair in the list that is closest to the test pos.
I've tried this below:
if IDed in d:
y = d[IDed]
closest = min(y, key=lambda x:abs(x-pos))
This doesn't work because it's not a list with single values. Is there another way to do this using a similar method. If not, I can find a long way around the problem by indexing through the list and calculating the distance between each value pair. However, I don't think this would be very efficient.

You were really close. Bellow is a working solution.
d = {'ID_1': [(10, 20), (40, 60), (125, 200)], 'ID_2': [(75, 100), (250, 300)]}
pos = 70
IDed = 'ID_1'
closest = min(d[IDed], key=lambda x: min(abs(y - pos) for y in x)) if IDed in d else None
print(closest)
# (40, 60)
The problem with your code was that you were trying to do x - pos with x being the entire tuple (e.g., (40, 60)) and pos being the integer-target value.
You can consider wraping that in a function to avoid code repetition in case you need to run it multiple times.
def find_nearest(point_dict, id, stigma):
try:
return min(point_dict[id], key=lambda x: min(abs(w - stigma) for w in x))
except:
return None
d = {'ID_1': [(10, 20), (40, 60), (125, 200)], 'ID_2': [(75, 100), (250, 300)]}
print(find_nearest(d, 'ID_1', 70))
# (40, 60)
Note that the order in which the terms appear in the list nested in the initial dictionary is important in cases like d = {'ID_1': [(10, 20), (40, 69), (71, 200)], ...} for example. Terms 69 and 71 are equidistant from given target 70 but the code returns (40, 69) since it finds that first.

I think that you want to find the couple which has the closest average with pos value...
So this is the answer:
d = {'ID_1':[(10, 20), (40, 60), (125, 200)], 'ID_2': [(75, 100), (250, 300)]}
pos = 70
closest = (0, 0)
IDed = "ID_1"
for i in d.items():
if IDed == i[0]:
for x in i[1]:
avg = (x[0]+x[1])/2
avg_closest = (closest[0]+closest[1])/2
if abs(pos-avg) < abs(pos-avg_closest):
closest = x
print closest

Is there a standard Python data structure that keeps things in sorted order?

I have a set of ranges that might look something like this:
[(0, 100), (150, 220), (500, 1000)]
I would then add a range, say (250, 400) and the list would look like this:
[(0, 100), (150, 220), (250, 400), (500, 1000)]
I would then try to add the range (399, 450), and it would error out because that overlapped (250, 400).
When I add a new range, I need to search to make sure the new range does not overlap an existing range. And no range will ever be in the list that overlaps another range in the list.
To this end, I would like a data structure that cheaply maintained its elements in sorted order, and quickly allowed me to find the element before or after a given element.
Is there a better way to solve this problem? Is there a data structure like that available in Python?
I know the bisect module exists, and that's likely what I will use. But I was hoping there was something better.
EDIT: I solved this using the bisect module. I had a link to the code since it was a bit longish. Unfortunately, paste.list.org turned out to be a bad place to put it because it's not there anymore.

It looks like you want something like bisect's insort_right/insort_left. The bisect module works with lists and tuples.
import bisect
l = [(0, 100), (150, 300), (500, 1000)]
bisect.insort_right(l, (250, 400))
print l # [(0, 100), (150, 300), (250, 400), (500, 1000)]
bisect.insort_right(l, (399, 450))
print l # [(0, 100), (150, 300), (250, 400), (399, 450), (500, 1000)]
You can write your own overlaps function, which you can use to check before using insort.
I assume you made a mistake with your numbers as (250, 400) overlaps (150, 300).
overlaps() can be written like so:
def overlaps(inlist, inrange):
for min, max in inlist:
if min < inrange[0] < max and max < inrange[1]:
return True
return False

Use SortedDict from the SortedCollection.
A SortedDict provides the same methods as a dict. Additionally, a SortedDict efficiently maintains its keys in sorted order. Consequently, the keys method will return the keys in sorted order, the popitem method will remove the item with the highest key, etc.
I've used it - it works. Unfortunately I don't have the time now to do a proper performance comparison, but subjectively it seems to have become faster than the bisect module.

Cheap searching and cheap insertion tend to be at odds. You could use a linked list for the data structure. Then searching to find the insertion point for a new element is O(n), and the subsequent insertion of the new element in the correct location is O(1).
But you're probably better off just using a straightforward Python list. Random access (i.e. finding your spot) takes constant time. Insertion in the correct location to maintain the sort is theoretically more expensive, but that depends on how the dynamic array is implemented. You don't really pay the big price for insertions until reallocation of the underlying array takes place.
Regarding checking for date range overlaps, I happen to have had the same problem in the past. Here's the code I use. I originally found it in a blog post, linked from an SO answer, but that site no longer appears to exist. I actually use datetimes in my ranges, but it will work equally well with your numeric values.
def dt_windows_intersect(dt1start, dt1end, dt2start, dt2end):
'''Returns true if two ranges intersect. Note that if two
ranges are adjacent, they do not intersect.
Code based on:
http://beautifulisbetterthanugly.com/posts/2009/oct/7/datetime-intersection-python/
http://stackoverflow.com/questions/143552/comparing-date-ranges
'''
if dt2end <= dt1start or dt2start >= dt1end:
return False
return dt1start <= dt2end and dt1end >= dt2start
Here are the unit tests to prove it works:
from nose.tools import eq_, assert_equal, raises
class test_dt_windows_intersect():
"""
test_dt_windows_intersect
Code based on:
http://beautifulisbetterthanugly.com/posts/2009/oct/7/datetime-intersection-python/
http://stackoverflow.com/questions/143552/comparing-date-ranges
|-------------------| compare to this one
1 |---------| contained within
2 |----------| contained within, equal start
3 |-----------| contained within, equal end
4 |-------------------| contained within, equal start+end
5 |------------| overlaps start but not end
6 |-----------| overlaps end but not start
7 |------------------------| overlaps start, but equal end
8 |-----------------------| overlaps end, but equal start
9 |------------------------------| overlaps entire range
10 |---| not overlap, less than
11 |-------| not overlap, end equal
12 |---| not overlap, bigger than
13 |---| not overlap, start equal
"""
def test_contained_within(self):
assert dt_windows_intersect(
datetime(2009,10,1,6,0), datetime(2009,10,1,7,0),
datetime(2009,10,1,6,30), datetime(2009,10,1,6,40),
)
def test_contained_within_equal_start(self):
assert dt_windows_intersect(
datetime(2009,10,1,6,0), datetime(2009,10,1,7,0),
datetime(2009,10,1,6,0), datetime(2009,10,1,6,30),
)
def test_contained_within_equal_end(self):
assert dt_windows_intersect(
datetime(2009,10,1,6,0), datetime(2009,10,1,7,0),
datetime(2009,10,1,6,30), datetime(2009,10,1,7,0),
)
def test_contained_within_equal_start_and_end(self):
assert dt_windows_intersect(
datetime(2009,10,1,6,0), datetime(2009,10,1,7,0),
datetime(2009,10,1,6,0), datetime(2009,10,1,7,0),
)
def test_overlaps_start_but_not_end(self):
assert dt_windows_intersect(
datetime(2009,10,1,6,0), datetime(2009,10,1,7,0),
datetime(2009,10,1,5,30), datetime(2009,10,1,6,30),
)
def test_overlaps_end_but_not_start(self):
assert dt_windows_intersect(
datetime(2009,10,1,6,0), datetime(2009,10,1,7,0),
datetime(2009,10,1,6,30), datetime(2009,10,1,7,30),
)
def test_overlaps_start_equal_end(self):
assert dt_windows_intersect(
datetime(2009,10,1,6,0), datetime(2009,10,1,7,0),
datetime(2009,10,1,5,30), datetime(2009,10,1,7,0),
)
def test_equal_start_overlaps_end(self):
assert dt_windows_intersect(
datetime(2009,10,1,6,0), datetime(2009,10,1,7,0),
datetime(2009,10,1,6,0), datetime(2009,10,1,7,30),
)
def test_overlaps_entire_range(self):
assert dt_windows_intersect(
datetime(2009,10,1,6,0), datetime(2009,10,1,7,0),
datetime(2009,10,1,5,0), datetime(2009,10,1,8,0),
)
def test_not_overlap_less_than(self):
assert not dt_windows_intersect(
datetime(2009,10,1,6,0), datetime(2009,10,1,7,0),
datetime(2009,10,1,5,0), datetime(2009,10,1,5,30),
)
def test_not_overlap_end_equal(self):
assert not dt_windows_intersect(
datetime(2009,10,1,6,0), datetime(2009,10,1,7,0),
datetime(2009,10,1,5,0), datetime(2009,10,1,6,0),
)
def test_not_overlap_greater_than(self):
assert not dt_windows_intersect(
datetime(2009,10,1,6,0), datetime(2009,10,1,7,0),
datetime(2009,10,1,7,30), datetime(2009,10,1,8,0),
)
def test_not_overlap_start_equal(self):
assert not dt_windows_intersect(
datetime(2009,10,1,6,0), datetime(2009,10,1,7,0),
datetime(2009,10,1,7,0), datetime(2009,10,1,8,0),
)

Maybe the module bisect could be better than the simple following function ? :
li = [(0, 100), (150, 220), (250, 400), (500, 1000)]
def verified_insertion(x,L):
u,v = x
if v<L[0][0]:
return [x] + L
elif u>L[-1][0]:
return L + [x]
else:
for i,(a,b) in enumerate(L[0:-1]):
if a<u and v<L[i+1][0]:
return L[0:i+1] + [x] + L[i+1:]
return L
lo = verified_insertion((-10,-2),li)
lu = verified_insertion((102,140),li)
le = verified_insertion((222,230),li)
lee = verified_insertion((234,236),le) # <== le
la = verified_insertion((408,450),li)
ly = verified_insertion((2000,3000),li)
for w in (lo,lu,le,lee,la,ly):
print li,'\n',w,'\n'
The function returns a list without modifying the list passed as argument.
result
[(0, 100), (150, 220), (250, 400), (500, 1000)]
[(-10, -2), (0, 100), (150, 220), (250, 400), (500, 1000)]
[(0, 100), (150, 220), (250, 400), (500, 1000)]
[(0, 100), (102, 140), (150, 220), (250, 400), (500, 1000)]
[(0, 100), (150, 220), (250, 400), (500, 1000)]
[(0, 100), (150, 220), (222, 230), (250, 400), (500, 1000)]
[(0, 100), (150, 220), (250, 400), (500, 1000)]
[(0, 100), (150, 220), (222, 230), (234, 236), (250, 400), (500, 1000)]
[(0, 100), (150, 220), (250, 400), (500, 1000)]
[(0, 100), (150, 220), (250, 400), (408, 450), (500, 1000)]
[(0, 100), (150, 220), (250, 400), (500, 1000)]
[(0, 100), (150, 220), (250, 400), (500, 1000), (2000, 3000)]

To answer your question:
Is there a data structure like that available in Python?
No there is not. But you can easily build one yourself using a list as the basic structure and code from the bisect module to keep the list in order and check for overlaps.
class RangeList(list):
"""Maintain ordered list of non-overlapping ranges"""
def add(self, range):
"""Add a range if no overlap else reject it"""
lo = 0; hi = len(self)
while lo < hi:
mid = (lo + hi)//2
if range < self[mid]: hi = mid
else: lo = mid + 1
if overlaps(range, self[lo]):
print("range overlap, not added")
else:
self.insert(lo, range)
I leave the overlaps function as an exercise.
(This code is untested and may need some tweeking)