Develop Reference

pandas groupby to list of dicts

this is my data:
data = [
{'shape': 'circle', 'width': 10, 'height': 8},
{'shape': 'circle', 'width': 7, 'height': 2},
{'shape': 'square', 'width': 4, 'height': 6}
]
I am trying to group by shapes that will hold the x, y
my final output should be a dict in the following format:
{
'circle': [
{'x': 10, 'y': 8},
{'x': 7, 'y': 2}
],
'square': [
{'x': 4, 'y': 6}
],
}
here is what I tried, which does not work
df = pd.DataFrame(data)
df = df.rename({'width': 'x', 'height': 'y'}, axis='columns')
df.groupby('shape').apply(
lambda s: s.do_dict()).to_dict()
what is the correct way to do it? also is there a way to do it with out renaming the columns before, something like:
df.groupby('shape').apply(
lambda s: {'x': s['width'], 'y': s['height']}).to_dict()

I could not do without renaming the column but something like this?
(df.rename(columns={'width': 'x', 'height': 'y'})
.groupby('shape')
.apply(lambda s: s[['x', 'y']].to_dict(orient='records'))
.to_dict())

It can be done with a dict comprehension:
res = {i:df[df['shape']==i][['x', 'y']].to_dict(orient='records') for i in set(df['shape'])}
>>>print(res)
{'circle': [{'x': 10, 'y': 8}, {'x': 7, 'y': 2}], 'square': [{'x': 4, 'y': 6}]}

How to return line of latest field in text file with Python?

I'm writing a Python script. I need to return lines that contain latest 'timestamp': field from a text file. For example, in the below text file example:
{'uid': 3167, 'user_id': '6', 'timestamp': datetime.datetime(2021, 3, 10, 18, 7, 13), 'status': 1, 'punch': 1}, {'uid': 3168, 'user_id': '198', 'timestamp': datetime.datetime(2021, 3, 10, 18, 10, 42), 'status': 2, 'punch': 1}, {'uid': 3169, 'user_id': '3', 'timestamp': datetime.datetime(2021, 3, 10, 18, 13, 53), 'status': 1, 'punch': 1}, {'uid': 3170, 'user_id': '13', 'timestamp': datetime.datetime(2021, 3, 10, 18, 22, 2), 'status': 1, 'punch': 1}, {'uid': 3171, 'user_id': '9', 'timestamp': datetime.datetime(2021, 3, 10, 18, 22, 43), 'status': 1, 'punch': 1}, {'uid': 3172, 'user_id': '15', 'timestamp': datetime.datetime(2021, 3, 10, 18, 32, 30), 'status': 2, 'punch': 1}, {'uid': 3173, 'user_id': '4', 'timestamp': datetime.datetime(2021, 3, 10, 19, 42, 26), 'status': 1, 'punch': 1}, {'uid': 3174, 'user_id': '1', 'timestamp': datetime.datetime(2021, 3, 10, 19, 42, 34), 'status': 1, 'punch': 1}, {'uid': 3175, 'user_id': '3', 'timestamp': datetime.datetime(2021, 3, 11, 8, 48, 6), 'status': 1, 'punch': 1}, {'uid': 3176, 'user_id': '7', 'timestamp': datetime.datetime(2021, 3, 11, 9, 2, 30), 'status': 2, 'punch': 1}, {'uid': 3177, 'user_id': '5', 'timestamp': datetime.datetime(2021, 3, 11, 9, 12, 40), 'status': 1, 'punch': 1}, {'uid': 3178, 'user_id': '6', 'timestamp': datetime.datetime(2021, 3, 11, 9, 40, 47), 'status': 1, 'punch': 1}, {'uid': 3179, 'user_id': '15', 'timestamp': datetime.datetime(2021, 3, 11, 9, 49, 59), 'status': 2, 'punch': 1},
Return Text File 'today's date 11/3/2021' ex:
{'uid': 3175, 'user_id': '3', 'timestamp': datetime.datetime(2021, 3, 11, 8, 48, 6), 'status': 1, 'punch': 1}, {'uid': 3176, 'user_id': '7', 'timestamp': datetime.datetime(2021, 3, 11, 9, 2, 30), 'status': 2, 'punch': 1}, {'uid': 3177, 'user_id': '5', 'timestamp': datetime.datetime(2021, 3, 11, 9, 12, 40), 'status': 1, 'punch': 1}, {'uid': 3178, 'user_id': '6', 'timestamp': datetime.datetime(2021, 3, 11, 9, 40, 47), 'status': 1, 'punch': 1}, {'uid': 3179, 'user_id': '15', 'timestamp': datetime.datetime(2021, 3, 11, 9, 49, 59), 'status': 2, 'punch': 1},

It seems you're dealing with tabular data and pandas is very natural for that.
import datetime
import pandas as pd
df = pd.DataFrame([{'uid': 3167, 'user_id': '6', 'timestamp': datetime.datetime(2021, 3, 10, 18, 7, 13), 'status': 1, 'punch': 1}, {'uid': 3168, 'user_id': '198', 'timestamp': datetime.datetime(2021, 3, 10, 18, 10, 42), 'status': 2, 'punch': 1}, {'uid': 3169, 'user_id': '3', 'timestamp': datetime.datetime(2021, 3, 10, 18, 13, 53), 'status': 1, 'punch': 1}, {'uid': 3170, 'user_id': '13', 'timestamp': datetime.datetime(2021, 3, 10, 18, 22, 2), 'status': 1, 'punch': 1}, {'uid': 3171, 'user_id': '9', 'timestamp': datetime.datetime(2021, 3, 10, 18, 22, 43), 'status': 1, 'punch': 1}, {'uid': 3172, 'user_id': '15', 'timestamp': datetime.datetime(2021, 3, 10, 18, 32, 30), 'status': 2, 'punch': 1}, {'uid': 3173, 'user_id': '4', 'timestamp': datetime.datetime(2021, 3, 10, 19, 42, 26), 'status': 1, 'punch': 1}, {'uid': 3174, 'user_id': '1', 'timestamp': datetime.datetime(2021, 3, 10, 19, 42, 34), 'status': 1, 'punch': 1}, {'uid': 3175, 'user_id': '3', 'timestamp': datetime.datetime(2021, 3, 11, 8, 48, 6), 'status': 1, 'punch': 1}, {'uid': 3176, 'user_id': '7', 'timestamp': datetime.datetime(2021, 3, 11, 9, 2, 30), 'status': 2, 'punch': 1}, {'uid': 3177, 'user_id': '5', 'timestamp': datetime.datetime(2021, 3, 11, 9, 12, 40), 'status': 1, 'punch': 1}, {'uid': 3178, 'user_id': '6', 'timestamp': datetime.datetime(2021, 3, 11, 9, 40, 47), 'status': 1, 'punch': 1}, {'uid': 3179, 'user_id': '15', 'timestamp': datetime.datetime(2021, 3, 11, 9, 49, 59), 'status': 2, 'punch': 1},])
today = pd.to_datetime('today').normalize()
rows = df[df['timestamp'] >= today]
Which gives
uid user_id timestamp status punch
8 3175 3 2021-03-11 08:48:06 1 1
9 3176 7 2021-03-11 09:02:30 2 1
10 3177 5 2021-03-11 09:12:40 1 1
11 3178 6 2021-03-11 09:40:47 1 1
12 3179 15 2021-03-11 09:49:59 2 1
If you want the result in a list of dicts, you can then do rows.to_dict('records').
Without pandas it'd be a similar approach of getting today's datetime and iterating over your data to filter them.
lines = [{'uid': 3167, 'user_id': '6', 'timestamp': datetime.datetime(2021, 3, 10, 18, 7, 13), 'status': 1, 'punch': 1}, {'uid': 3168, 'user_id': '198', 'timestamp': datetime.datetime(2021, 3, 10, 18, 10, 42), 'status': 2, 'punch': 1}, {'uid': 3169, 'user_id': '3', 'timestamp': datetime.datetime(2021, 3, 10, 18, 13, 53), 'status': 1, 'punch': 1}, {'uid': 3170, 'user_id': '13', 'timestamp': datetime.datetime(2021, 3, 10, 18, 22, 2), 'status': 1, 'punch': 1}, {'uid': 3171, 'user_id': '9', 'timestamp': datetime.datetime(2021, 3, 10, 18, 22, 43), 'status': 1, 'punch': 1}, {'uid': 3172, 'user_id': '15', 'timestamp': datetime.datetime(2021, 3, 10, 18, 32, 30), 'status': 2, 'punch': 1}, {'uid': 3173, 'user_id': '4', 'timestamp': datetime.datetime(2021, 3, 10, 19, 42, 26), 'status': 1, 'punch': 1}, {'uid': 3174, 'user_id': '1', 'timestamp': datetime.datetime(2021, 3, 10, 19, 42, 34), 'status': 1, 'punch': 1}, {'uid': 3175, 'user_id': '3', 'timestamp': datetime.datetime(2021, 3, 11, 8, 48, 6), 'status': 1, 'punch': 1}, {'uid': 3176, 'user_id': '7', 'timestamp': datetime.datetime(2021, 3, 11, 9, 2, 30), 'status': 2, 'punch': 1}, {'uid': 3177, 'user_id': '5', 'timestamp': datetime.datetime(2021, 3, 11, 9, 12, 40), 'status': 1, 'punch': 1}, {'uid': 3178, 'user_id': '6', 'timestamp': datetime.datetime(2021, 3, 11, 9, 40, 47), 'status': 1, 'punch': 1}, {'uid': 3179, 'user_id': '15', 'timestamp': datetime.datetime(2021, 3, 11, 9, 49, 59), 'status': 2, 'punch': 1},]
today = datetime.date.today()
today = datetime.datetime(today.year, today.month, today.day)
result = [line for line in lines if line['timestamp'] >= today]

Scatter plot line to out of order data

I am tracking the movements of an avian animal. I have detection points on an xy plot. I want to connect the previous detected point to the next detection, regardless of direction. This will assist with removing extraneous detections.
Data Sample:
Sample input
The goal is to have a line from the previous data point to the next point.
Sample output
Unsuccessful method 1:
plt.figure('Frame',figsize=(16,12))
plt.imshow(frame)
plt.plot(x, y, '-ro', 'd',markersize=2.5, color='orange')
Method 1 output
Unsuccessful method 2:
plt.plot(np.sort(x), y[np.argsort(x)], '-bo', ms = 2)
Method 2 output

I used your sample data and make a plot with method 1 (but with pandas) and the output was as you expected. I don't understand why you have an unsuccessful result.
data = [{'frame': 1, 'x': 5, 'y': 15},
{'frame': 4, 'x': 10, 'y': 15},
{'frame': 5, 'x': 15, 'y': 15},
{'frame': 6, 'x': 20, 'y': 15},
{'frame': 7, 'x': 23, 'y': 20},
{'frame': 8, 'x': 25, 'y': 25},
{'frame': 11, 'x': 20, 'y': 23},
{'frame': 15, 'x': 15, 'y': 20},
{'frame': 18, 'x': 8, 'y': 18},
{'frame': 19, 'x': 8, 'y': 10},
{'frame': 20, 'x': 12, 'y': 7}]
df = pd.DataFrame(data).sort_values('frame')
df.plot(x='x', y='y')

Increasing opacity of area plot in plotly

How do I increase the opacity or alpha of the 'filled' area of my plot? I tried:
import pandas as pd
import plotly.offline as py
import plotly.graph_objs as go
import cufflinks as cf
from plotly import tools
plotly.offline.init_notebook_mode()
cf.go_offline()
df = pd.DataFrame(np.random.rand(10, 4), columns=['a', 'b', 'c', 'd'])
df.iplot(kind='area', fill=True, filename='cuflinks/stacked-area', opacity=.1)
but it doesn't seem to work.

There doesn't seem to be an easy built-in way of achieving this. However, a work-around is to first get the figure object of the plot, modify it to change the opacity, and then plot it.
You can get the figure object using the asFigure attribute like this:
figure = df.iplot(asFigure=True, kind='area', fill=True, filename='cuflinks/stacked-area')
The figure object in this case looks like:
Figure({
'data': [{'fill': 'tonexty',
'fillcolor': 'rgba(255, 153, 51, 0.3)',
'line': {'color': 'rgba(255, 153, 51, 1.0)', 'dash': 'solid', 'shape': 'linear', 'width': 1.3},
'mode': 'lines',
'name': 'a',
'text': '',
'type': 'scatter',
'uid': '4dcc1a3e-fba3-4a32-bb2a-40925b4fae5b',
'x': array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int64),
'y': array([0.91229144, 0.63049138, 0.22855077, 0.13470399, 0.9114691 , 0.39640368,
0.46534334, 0.20508211, 0.00203548, 0.41343938])},
{'fill': 'tonexty',
'fillcolor': 'rgba(55, 128, 191, 0.3)',
'line': {'color': 'rgba(55, 128, 191, 1.0)', 'dash': 'solid', 'shape': 'linear', 'width': 1.3},
'mode': 'lines',
'name': 'b',
'text': '',
'type': 'scatter',
'uid': '1015b30d-7c09-456c-875c-8a211a6ebdeb',
'x': array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int64),
'y': array([1.81115175, 1.57534372, 0.41288126, 0.38068805, 1.72268856, 0.87778503,
1.32714727, 0.848242 , 0.51605283, 0.58190402])},
{'fill': 'tonexty',
'fillcolor': 'rgba(50, 171, 96, 0.3)',
'line': {'color': 'rgba(50, 171, 96, 1.0)', 'dash': 'solid', 'shape': 'linear', 'width': 1.3},
'mode': 'lines',
'name': 'c',
'text': '',
'type': 'scatter',
'uid': '7d1852ac-b8e7-44e6-ae69-54229d7e2c83',
'x': array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int64),
'y': array([2.79222081, 1.58812634, 1.1439478 , 1.30453731, 2.50881795, 1.67681961,
1.85609861, 1.36657712, 0.89024486, 0.82749039])},
{'fill': 'tonexty',
'fillcolor': 'rgba(128, 0, 128, 0.3)',
'line': {'color': 'rgba(128, 0, 128, 1.0)', 'dash': 'solid', 'shape': 'linear', 'width': 1.3},
'mode': 'lines',
'name': 'd',
'text': '',
'type': 'scatter',
'uid': '89b85012-fc95-487c-b7ba-9cb6c249b768',
'x': array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int64),
'y': array([3.54740551, 1.79856232, 2.1326556 , 2.10560567, 2.64867039, 2.55519564,
2.73888819, 2.23274393, 1.16987343, 1.42794202])}],
'layout': {'legend': {'bgcolor': '#F5F6F9', 'font': {'color': '#4D5663'}},
'paper_bgcolor': '#F5F6F9',
'plot_bgcolor': '#F5F6F9',
'title': {'font': {'color': '#4D5663'}},
'xaxis': {'gridcolor': '#E1E5ED',
'showgrid': True,
'tickfont': {'color': '#4D5663'},
'title': {'font': {'color': '#4D5663'}, 'text': ''},
'zerolinecolor': '#E1E5ED'},
'yaxis': {'gridcolor': '#E1E5ED',
'showgrid': True,
'tickfont': {'color': '#4D5663'},
'title': {'font': {'color': '#4D5663'}, 'text': ''},
'zerolinecolor': '#E1E5ED'}}
})
You'll note that each trace in the data has a fillcolor attribute: 'fillcolor': 'rgba(255, 153, 51, 0.3)'. The last number is the alpha value, which you want to modify. I've made a hacky little function to update the fillcolor attribute of all traces in a figure object:
def update_opacity(figure,opacity):
for trace in range(len(figure['data'])):
# print(figure['data'][trace]['fillcolor'],'-> ',end='')
rgba_split = figure['data'][trace]['fillcolor'].split(',')
figure['data'][trace]['fillcolor'] = ','.join(rgba_split[:-1] + [' {})'.format(opacity)])
# print(figure['data'][trace]['fillcolor'])
return figure
For full opacity, you can do:
figure = update_opacity(figure,1)
Then, simply plot the result with
py.iplot(figure)
Output:

Python dict values to list of integers

I'm fairly new to Python. I have a list of dictionaries.
[{"x": "23"}, {"x": "14"}, {"x": "9"}, {"x": "19"}, {"x": "21"}, {"x": "14"}, {"x": "16"}, {"x": "11"}, {"x": "9"}, {"x": "6"}, {"x": "24"}, {"x": "13"}]
I want to extract the values associated with the x key and plot them. To do that the list must look like this, and the values must be integers (or floats).
[23, 14, 9, 19, 21, 14, 16, 11, 9, 6, 24, 13]
My question is: how do I
extract these values, and
convert them to integers?

This can be done using a list comprehension using the dictionary's get method.
>>> l = [{"x": "23"}, {"x": "14"}, {"x": "9"}, {"x": "19"}, {"x": "21"}, {"x": "14"}, {"x": "16"}, {"x": "11"}, {"x": "9"}, {"x": "6"}, {"x": "24"}, {"x": "13"}]
>>> [i.get('x') for i in l]
['23', '14', '9', '19', '21', '14', '16', '11', '9', '6', '24', '13']
If you'd like them as integers, convert them in the list comprehension
>>> [int(i.get('x')) for i in l]
[23, 14, 9, 19, 21, 14, 16, 11, 9, 6, 24, 13]

Use a list comprehension. For each item get "x" and cast to int:
[int(item['x']) for item in mylist]
If you need floats change int to float.

You can use list comprehension
>>> mylist = [{'x': '23'}, {'x': '14'}, {'x': '9'}, {'x': '19'}, {'x': '21'}, {'x': '14'}, {'x': '16'}, {'x': '11'}, {'x': '9'}, {'x': '6'}, {'x': '24'}, {'x': '13'}]
>>> [ int(item.values()[0]) for item in mylist]
[23, 14, 9, 19, 21, 14, 16, 11, 9, 6, 24, 13]
or map
>>> map(lambda x: int(x['x']), mylist)
[23, 14, 9, 19, 21, 14, 16, 11, 9, 6, 24, 13]