How to import list of cookies to requests? - python

I have list of dicts with full cookies of such type:
<class 'list'>: [{'name': 'c_user', 'value': '100041356816620', 'expires': 'Sun, 06 Sep 2020 20:12:51 GMT', 'expires_timestamp': 1599423171, 'domain': '.facebook.com', 'path': '/', 'secure': True}, {'name': 'xs', 'value': '40:KIuQ_QG_K0JALw:2:1567887172:-1:-1', 'expires': 'Sun, 06 Sep 2020 20:12:51 GMT', 'expires_timestamp': 1599423171, 'domain': '.facebook.com', 'path': '/', 'secure': True, 'httponly': True}, {'name': 'fr', 'value': '5N1aV12Rlv9dmM9f7.AWVvFfCaNn4mFDxKf5WU3HVX3Fg.BddA9D..AAA.0.0.BddA9D.AWW5fJZ-', 'expires': 'Sun, 06 Sep 2020 20:12:48 GMT', 'expires_timestamp': 1599423168, 'domain': '.facebook.com', 'path': '/', 'secure': True, 'httponly': True}, {'name': 'datr', 'value': 'Qw90X2hNUueymhbnR9mUeYd_', 'expires': 'Mon, 06 Sep 2021 20:12:51 GMT', 'expires_timestamp': 1630959171, 'domain': '.facebook.com', 'path': '/', 'secure': True, 'httponly': True}]
How is it possible to load them to requests?
I tried smth like this:
cookies = {key: morsel.value for key, morsel in session_cookies}
requests.cookies = cookies
But received error

you can easily do so by using the following code:
requests.cookies.update(cookies)
hope this helps!

Related

reading Tweepy data value from tweepy.models.Status object in python is not working

I am trying to get information on retweeters for a specific tweet using Tweepy and fetch the in_reply_to_status_id from the returned Tweepy response.
Here is the code
retweets_list = api.get_retweets(id=tweetid)
for retweet in retweets_list:
retweet_json = json.dumps(retweet._json, indent=2)
retweet_json = json.loads(retweet_json)
print(retweet_json)
The code about above produces the data response below
{'created_at': 'Sat Jun 18 06:38:49 +0000 2022', 'id': 1538048568782688256, 'id_str': '1538048568782688256', 'text': 'RT #gyfboxAI: #isle_mcelroy Some mentioned items in thread \n\n#AllisonPDavis The Governesses => httpsurl The Ob…', 'truncated': False, 'entities': {'hashtags': [], 'symbols': [], 'user_mentions': [{'screen_name': 'gyfboxAI', 'name': 'Gyfbox', 'id': 1521109812032978946, 'id_str': '1521109812032978946', 'indices': [3, 12]}, {'screen_name': 'isle_mcelroy', 'name': 'Isle McElroy', 'id': 868462820, 'id_str': '868462820', 'indices': [14, 27]}, {'screen_name': 'AllisonPDavis', 'name': 'Allison P Davis', 'id': 15088579, 'id_str': '15088579', 'indices': [61, 75]}, {'screen_name': 'kvargs93', 'name': 'Katherine Varga', 'id': 885284552897429504, 'id_str': '885284552897429504', 'indices': [125, 134]}], 'urls': [{'url': 'httpsurl', 'expanded_url': 'httpsurlamzn.to/3MUM0mI', 'display_url': 'amzn.to/3MUM0mI', 'indices': [100, 123]}]}, 'source': 'Twitter for iPhone', 'in_reply_to_status_id': None, 'in_reply_to_status_id_str': None, 'in_reply_to_user_id': None, 'in_reply_to_user_id_str': None, 'in_reply_to_screen_name': None, 'user': {'id': 1003173584, 'id_str': '1003173584', 'name': 'Elaine Showalter', 'screen_name': 'ecshowalter', 'location': 'Washington, D.C./London', 'description': 'Professor Emerita Princeton U; Anglophile, feminist, theatre fanatic, “The Civil Wars of Julia Ward Howe.” watercolor by Vanessa Bell, “The Queen’s Tea Party”', 'url': None, 'entities': {'description': {'urls': []}}, 'protected': False, 'followers_count': 8142, 'friends_count': 1049, 'listed_count': 104, 'created_at': 'Tue Dec 11 03:08:17 +0000 2012', 'favourites_count': 24912, 'utc_offset': None, 'time_zone': None, 'geo_enabled': True, 'verified': False, 'statuses_count': 26489, 'lang': None, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': 'C0DEED', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_image_url_https': 'httpsurlabs.twimg.com/images/themes/theme1/bg.png', 'profile_background_tile': True, 'profile_image_url': 'http://pbs.twimg.com/profile_images/968862619699425281/CKzdSRf6_normal.jpg', 'profile_image_url_https': 'httpsurlpbs.twimg.com/profile_images/968862619699425281/CKzdSRf6_normal.jpg', 'profile_banner_url': 'httpsurlpbs.twimg.com/profile_banners/1003173584/1569562029', 'profile_link_color': '0084B4', 'profile_sidebar_border_color': 'FFFFFF', 'profile_sidebar_fill_color': 'DDEEF6', 'profile_text_color': '333333', 'profile_use_background_image': True, 'has_extended_profile': False, 'default_profile': False, 'default_profile_image': False, 'following': False, 'follow_request_sent': False, 'notifications': False, 'translator_type': 'none', 'withheld_in_countries': []}, 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'retweeted_status': {'created_at': 'Fri Jun 17 17:55:18 +0000 2022', 'id': 1537856423740198913, 'id_str': '1537856423740198913', 'text': '#isle_mcelroy Some mentioned items in thread \n\n#AllisonPDavis The Governesses => httpsurl… httpsurl', 'truncated': True, 'entities': {'hashtags': [], 'symbols': [], 'user_mentions': [{'screen_name': 'isle_mcelroy', 'name': 'Isle McElroy', 'id': 868462820, 'id_str': '868462820', 'indices': [0, 13]}, {'screen_name': 'AllisonPDavis', 'name': 'Allison P Davis', 'id': 15088579, 'id_str': '15088579', 'indices': [47, 61]}], 'urls': [{'url': 'httpsurl', 'expanded_url': 'httpsurlamzn.to/3MUM0mI', 'display_url': 'amzn.to/3MUM0mI', 'indices': [86, 109]}, {'url': 'httpsurl’, 'expanded_url': 'httpsurltwitter.com/i/web/status/1537856423740198913', 'display_url': 'twitter.com/i/web/status/1…', 'indices': [111, 134]}]}, 'source': 'gyfbox', 'in_reply_to_status_id': 1537835837542604801, 'in_reply_to_status_id_str': '1537835837542604801', 'in_reply_to_user_id': 868462820, 'in_reply_to_user_id_str': '868462820', 'in_reply_to_screen_name': 'isle_mcelroy', 'user': {'id': 1521109812032978946, 'id_str': '1521109812032978946', 'name': 'Gyfbox', 'screen_name': 'gyfboxAI', 'location': '', 'description': 'Tag "#GyfboxAI find item" \n\n#GyfboxAI will reply with link for items mentioned in the thread\n\nCOMING SOON !', 'url': 'httpsurlt.co/u7fGrxh24Y', 'entities': {'url': {'urls': [{'url': 'httpsurlt.co/u7fGrxh24Y', 'expanded_url': 'httpsurlwww.gyfbox.com', 'display_url': 'gyfbox.com', 'indices': [0, 23]}]}, 'description': {'urls': []}}, 'protected': False, 'followers_count': 1, 'friends_count': 6, 'listed_count': 0, 'created_at': 'Mon May 02 12:50:32 +0000 2022', 'favourites_count': 1, 'utc_offset': None, 'time_zone': None, 'geo_enabled': False, 'verified': False, 'statuses_count': 49, 'lang': None, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': 'F5F8FA', 'profile_background_image_url': None, 'profile_background_image_url_https': None, 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1521109885827661824/iTrlR67U_normal.png', 'profile_image_url_https': 'httpsurlpbs.twimg.com/profile_images/1521109885827661824/iTrlR67U_normal.png', 'profile_link_color': '1DA1F2', 'profile_sidebar_border_color': 'C0DEED', 'profile_sidebar_fill_color': 'DDEEF6', 'profile_text_color': '333333', 'profile_use_background_image': True, 'has_extended_profile': True, 'default_profile': True, 'default_profile_image': False, 'following': False, 'follow_request_sent': False, 'notifications': False, 'translator_type': 'none', 'withheld_in_countries': []}, 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 1, 'favorite_count': 0, 'favorited': False, 'retweeted': False, 'possibly_sensitive': False, 'lang': 'en'}, 'is_quote_status': False, 'retweet_count': 1, 'favorite_count': 0, 'favorited': False, 'retweeted': False, 'possibly_sensitive': False, 'lang': 'en'}
Multiple attempts to extract the in_reply_to_status_id always return "None"
Sample attempt that returned none
retweet_json['in_reply_to_status_id']
retweet.in_reply_to_status_id
The data return above shows
'in_reply_to_status_id': 1537835837542604801,
so I should be getting 1537835837542604801 for in_reply_to_status_id
what am i doing wrong and how can I obtain the in_reply_to_status_id ?
According to your Json structure,
the in_reply_status_id is None, the id is in retweeted_status so based on the Json structure
retweet_json['retweeted_status']['in_reply_to_status_id']
should give
1537835837542604801

Convert nested dictionary, list, and dictionary into a pandas data frame in python

So, I am trying to work with a rest API, and it is giving me the following data:
{'sports': [{'id': '20',
'uid': 's:20',
'name': 'Football',
'slug': 'football',
'leagues': [{'id': '28',
'uid': 's:20~l:28',
'name': 'National Football League',
'abbreviation': 'NFL',
'shortName': 'NFL',
'slug': 'nfl',
'teams': [{'team': {'id': '22',
'uid': 's:20~l:28~t:22',
'slug': 'arizona-cardinals',
'location': 'Arizona',
'name': 'Cardinals',
'nickname': 'Cardinals',
'abbreviation': 'ARI',
'displayName': 'Arizona Cardinals',
'shortDisplayName': 'Cardinals',
'color': 'A40227',
'alternateColor': '000000',
'isActive': True,
'isAllStar': False,
'logos': [{'href': 'https://a.espncdn.com/i/teamlogos/nfl/500/ari.png',
'width': 500,
'height': 500,
'alt': '',
'rel': ['full', 'default'],
'lastUpdated': '2018-06-05T12:11Z'},
{'href': 'https://a.espncdn.com/i/teamlogos/nfl/500-dark/ari.png',
'width': 500,
'height': 500,
'alt': '',
'rel': ['full', 'dark'],
'lastUpdated': '2018-06-05T12:11Z'},
{'href': 'https://a.espncdn.com/i/teamlogos/nfl/500/scoreboard/ari.png',
'width': 500,
'height': 500,
'alt': '',
'rel': ['full', 'scoreboard'],
'lastUpdated': '2018-06-05T12:11Z'},
...
I'm just interested in the teams data. However, I try to slice the pie, I'm having trouble extracting the desired information into the dataframe properly.
Here is my code:
url = 'http://site.api.espn.com/apis/site/v2/sports/football/nfl/teams'
r = requests.get(url)
teams_json = r.json()
nfl = []
for teams in teams_json.items():
for x in teams:
for row in x:
print(row['teams'])
I keep getting errors.
Any assistance is greatly appreciated.
I'd suggest looking into how to navigate lists and dictionaries in python (that's all that json files are). It's just a matter of knowing the path, or how to iterate through those.
To get into a dataframe, pandas has a nice .json_normalize() method. I'm not sure what data you want exactly, as the root teams key data is also nested. So depending what you are after, you may need to do a little extra work to extract what you want. But this is the general dataframe for teams.
import requests
import pandas as pd
url = 'http://site.api.espn.com/apis/site/v2/sports/football/nfl/teams'
jsonData = requests.get(url).json()
teams_json = jsonData['sports'][0]['leagues'][0]['teams']
df = pd.json_normalize(teams_json)
Output:
print(df.head().to_string())
team.id team.uid team.slug team.location team.name team.nickname team.abbreviation team.displayName team.shortDisplayName team.color team.alternateColor team.isActive team.isAllStar team.logos team.record.items team.links
0 22 s:20~l:28~t:22 arizona-cardinals Arizona Cardinals Cardinals ARI Arizona Cardinals Cardinals A40227 000000 True False [{'href': 'https://a.espncdn.com/i/teamlogos/nfl/500/ari.png', 'width': 500, 'height': 500, 'alt': '', 'rel': ['full', 'default'], 'lastUpdated': '2018-06-05T12:11Z'}, {'href': 'https://a.espncdn.com/i/teamlogos/nfl/500-dark/ari.png', 'width': 500, 'height': 500, 'alt': '', 'rel': ['full', 'dark'], 'lastUpdated': '2018-06-05T12:11Z'}, {'href': 'https://a.espncdn.com/i/teamlogos/nfl/500/scoreboard/ari.png', 'width': 500, 'height': 500, 'alt': '', 'rel': ['full', 'scoreboard'], 'lastUpdated': '2018-06-05T12:11Z'}, {'href': 'https://a.espncdn.com/i/teamlogos/nfl/500-dark/scoreboard/ari.png', 'width': 500, 'height': 500, 'alt': '', 'rel': ['full', 'scoreboard', 'dark'], 'lastUpdated': '2018-06-05T12:11Z'}] [{'summary': '11-6', 'stats': [{'name': 'playoffSeed', 'value': 5.0}, {'name': 'wins', 'value': 11.0}, {'name': 'losses', 'value': 6.0}, {'name': 'winPercent', 'value': 0.6470588445663452}, {'name': 'gamesBehind', 'value': 0.0}, {'name': 'ties', 'value': 0.0}, {'name': 'OTWins', 'value': 0.0}, {'name': 'OTLosses', 'value': 0.0}, {'name': 'gamesPlayed', 'value': 17.0}, {'name': 'pointsFor', 'value': 449.0}, {'name': 'pointsAgainst', 'value': 366.0}, {'name': 'avgPointsFor', 'value': 26.41176414489746}, {'name': 'avgPointsAgainst', 'value': 21.52941131591797}, {'name': 'points', 'value': 2.5}, {'name': 'differential', 'value': 83.0}, {'name': 'streak', 'value': -1.0}, {'name': 'clincher', 'value': 0.0}, {'name': 'divisionWinPercent', 'value': 0.6666666865348816}, {'name': 'leagueWinPercent', 'value': 0.5833333134651184}, {'name': 'divisionRecord', 'value': 0.0}, {'name': 'divisionWins', 'value': 4.0}, {'name': 'divisionTies', 'value': 0.0}, {'name': 'divisionLosses', 'value': 2.0}]}] [{'language': 'en-US', 'rel': ['clubhouse', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/_/name/ari/arizona-cardinals', 'text': 'Clubhouse', 'shortText': 'Clubhouse', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['roster', 'desktop', 'team'], 'href': 'http://www.espn.com/nfl/team/roster/_/name/ari/arizona-cardinals', 'text': 'Roster', 'shortText': 'Roster', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['stats', 'desktop', 'team'], 'href': 'http://www.espn.com/nfl/team/stats/_/name/ari/arizona-cardinals', 'text': 'Statistics', 'shortText': 'Statistics', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['schedule', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/schedule/_/name/ari', 'text': 'Schedule', 'shortText': 'Schedule', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['photos', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/photos/_/name/ari', 'text': 'photos', 'shortText': 'photos', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['scores', 'sportscenter', 'app', 'team'], 'href': 'sportscenter://x-callback-url/showClubhouse?uid=s:20~l:28~t:22&section=scores', 'text': 'Scores', 'shortText': 'Scores', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['draftpicks', 'desktop', 'team'], 'href': 'http://www.espn.com/nfl/draft/teams/_/name/ari/arizona-cardinals', 'text': 'Draft Picks', 'shortText': 'Draft Picks', 'isExternal': False, 'isPremium': True}, {'language': 'en-US', 'rel': ['transactions', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/transactions/_/name/ari', 'text': 'Transactions', 'shortText': 'Transactions', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['injuries', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/injuries/_/name/ari', 'text': 'Injuries', 'shortText': 'Injuries', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['depthchart', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/depth/_/name/ari', 'text': 'Depth Chart', 'shortText': 'Depth Chart', 'isExternal': False, 'isPremium': False}, {'language': 'en', 'rel': ['tickets', 'desktop', 'team'], 'href': 'https://www.vividseats.com/nfl-football/arizona-cardinals-tickets.html?wsUser=717', 'text': 'Tickets', 'isExternal': True, 'isPremium': False}]
1 1 s:20~l:28~t:1 atlanta-falcons Atlanta Falcons Falcons ATL Atlanta Falcons Falcons 000000 000000 True False [{'href': 'https://a.espncdn.com/i/teamlogos/nfl/500/atl.png', 'width': 500, 'height': 500, 'alt': '', 'rel': ['full', 'default'], 'lastUpdated': '2018-06-05T12:11Z'}, {'href': 'https://a.espncdn.com/i/teamlogos/nfl/500-dark/atl.png', 'width': 500, 'height': 500, 'alt': '', 'rel': ['full', 'dark'], 'lastUpdated': '2018-06-05T12:11Z'}, {'href': 'https://a.espncdn.com/i/teamlogos/nfl/500/scoreboard/atl.png', 'width': 500, 'height': 500, 'alt': '', 'rel': ['full', 'scoreboard'], 'lastUpdated': '2018-06-05T12:11Z'}, {'href': 'https://a.espncdn.com/i/teamlogos/nfl/500-dark/scoreboard/atl.png', 'width': 500, 'height': 500, 'alt': '', 'rel': ['full', 'scoreboard', 'dark'], 'lastUpdated': '2018-06-05T12:11Z'}] [{'summary': '7-10', 'stats': [{'name': 'playoffSeed', 'value': 12.0}, {'name': 'wins', 'value': 7.0}, {'name': 'losses', 'value': 10.0}, {'name': 'winPercent', 'value': 0.4117647111415863}, {'name': 'gamesBehind', 'value': 0.0}, {'name': 'ties', 'value': 0.0}, {'name': 'OTWins', 'value': 0.0}, {'name': 'OTLosses', 'value': 0.0}, {'name': 'gamesPlayed', 'value': 17.0}, {'name': 'pointsFor', 'value': 313.0}, {'name': 'pointsAgainst', 'value': 459.0}, {'name': 'avgPointsFor', 'value': 18.41176414489746}, {'name': 'avgPointsAgainst', 'value': 27.0}, {'name': 'points', 'value': -1.5}, {'name': 'differential', 'value': -146.0}, {'name': 'streak', 'value': -2.0}, {'name': 'clincher', 'value': 0.0}, {'name': 'divisionWinPercent', 'value': 0.3333333432674408}, {'name': 'leagueWinPercent', 'value': 0.3333333432674408}, {'name': 'divisionRecord', 'value': 0.0}, {'name': 'divisionWins', 'value': 2.0}, {'name': 'divisionTies', 'value': 0.0}, {'name': 'divisionLosses', 'value': 4.0}]}] [{'language': 'en-US', 'rel': ['clubhouse', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/_/name/atl/atlanta-falcons', 'text': 'Clubhouse', 'shortText': 'Clubhouse', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['roster', 'desktop', 'team'], 'href': 'http://www.espn.com/nfl/team/roster/_/name/atl/atlanta-falcons', 'text': 'Roster', 'shortText': 'Roster', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['stats', 'desktop', 'team'], 'href': 'http://www.espn.com/nfl/team/stats/_/name/atl/atlanta-falcons', 'text': 'Statistics', 'shortText': 'Statistics', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['schedule', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/schedule/_/name/atl', 'text': 'Schedule', 'shortText': 'Schedule', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['photos', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/photos/_/name/atl', 'text': 'photos', 'shortText': 'photos', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['scores', 'sportscenter', 'app', 'team'], 'href': 'sportscenter://x-callback-url/showClubhouse?uid=s:20~l:28~t:1&section=scores', 'text': 'Scores', 'shortText': 'Scores', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['draftpicks', 'desktop', 'team'], 'href': 'http://www.espn.com/nfl/draft/teams/_/name/atl/atlanta-falcons', 'text': 'Draft Picks', 'shortText': 'Draft Picks', 'isExternal': False, 'isPremium': True}, {'language': 'en-US', 'rel': ['transactions', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/transactions/_/name/atl', 'text': 'Transactions', 'shortText': 'Transactions', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['injuries', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/injuries/_/name/atl', 'text': 'Injuries', 'shortText': 'Injuries', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['depthchart', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/depth/_/name/atl', 'text': 'Depth Chart', 'shortText': 'Depth Chart', 'isExternal': False, 'isPremium': False}, {'language': 'en', 'rel': ['tickets', 'desktop', 'team'], 'href': 'https://www.vividseats.com/nfl-football/atlanta-falcons-tickets.html?wsUser=717', 'text': 'Tickets', 'isExternal': True, 'isPremium': False}]
2 2 s:20~l:28~t:2 buffalo-bills Buffalo Bills Bills BUF Buffalo Bills Bills 04407F c60c30 True False [{'href': 'https://a.espncdn.com/i/teamlogos/nfl/500/buf.png', 'width': 500, 'height': 500, 'alt': '', 'rel': ['full', 'default'], 'lastUpdated': '2018-06-05T12:11Z'}, {'href': 'https://a.espncdn.com/i/teamlogos/nfl/500-dark/buf.png', 'width': 500, 'height': 500, 'alt': '', 'rel': ['full', 'dark'], 'lastUpdated': '2018-06-05T12:11Z'}, {'href': 'https://a.espncdn.com/i/teamlogos/nfl/500/scoreboard/buf.png', 'width': 500, 'height': 500, 'alt': '', 'rel': ['full', 'scoreboard'], 'lastUpdated': '2018-06-05T12:11Z'}, {'href': 'https://a.espncdn.com/i/teamlogos/nfl/500-dark/scoreboard/buf.png', 'width': 500, 'height': 500, 'alt': '', 'rel': ['full', 'scoreboard', 'dark'], 'lastUpdated': '2018-06-05T12:11Z'}] [{'summary': '11-6', 'stats': [{'name': 'playoffSeed', 'value': 3.0}, {'name': 'wins', 'value': 11.0}, {'name': 'losses', 'value': 6.0}, {'name': 'winPercent', 'value': 0.6470588445663452}, {'name': 'gamesBehind', 'value': 0.0}, {'name': 'ties', 'value': 0.0}, {'name': 'OTWins', 'value': 0.0}, {'name': 'OTLosses', 'value': 1.0}, {'name': 'gamesPlayed', 'value': 17.0}, {'name': 'pointsFor', 'value': 483.0}, {'name': 'pointsAgainst', 'value': 289.0}, {'name': 'avgPointsFor', 'value': 28.41176414489746}, {'name': 'avgPointsAgainst', 'value': 17.0}, {'name': 'points', 'value': 2.5}, {'name': 'differential', 'value': 194.0}, {'name': 'streak', 'value': 4.0}, {'name': 'clincher', 'value': 0.0}, {'name': 'divisionWinPercent', 'value': 0.8333333134651184}, {'name': 'leagueWinPercent', 'value': 0.5833333134651184}, {'name': 'divisionRecord', 'value': 0.0}, {'name': 'divisionWins', 'value': 5.0}, {'name': 'divisionTies', 'value': 0.0}, {'name': 'divisionLosses', 'value': 1.0}]}] [{'language': 'en-US', 'rel': ['clubhouse', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/_/name/buf/buffalo-bills', 'text': 'Clubhouse', 'shortText': 'Clubhouse', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['roster', 'desktop', 'team'], 'href': 'http://www.espn.com/nfl/team/roster/_/name/buf/buffalo-bills', 'text': 'Roster', 'shortText': 'Roster', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['stats', 'desktop', 'team'], 'href': 'http://www.espn.com/nfl/team/stats/_/name/buf/buffalo-bills', 'text': 'Statistics', 'shortText': 'Statistics', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['schedule', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/schedule/_/name/buf', 'text': 'Schedule', 'shortText': 'Schedule', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['photos', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/photos/_/name/buf', 'text': 'photos', 'shortText': 'photos', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['scores', 'sportscenter', 'app', 'team'], 'href': 'sportscenter://x-callback-url/showClubhouse?uid=s:20~l:28~t:2&section=scores', 'text': 'Scores', 'shortText': 'Scores', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['draftpicks', 'desktop', 'team'], 'href': 'http://www.espn.com/nfl/draft/teams/_/name/buf/buffalo-bills', 'text': 'Draft Picks', 'shortText': 'Draft Picks', 'isExternal': False, 'isPremium': True}, {'language': 'en-US', 'rel': ['transactions', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/transactions/_/name/buf', 'text': 'Transactions', 'shortText': 'Transactions', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['injuries', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/injuries/_/name/buf', 'text': 'Injuries', 'shortText': 'Injuries', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['depthchart', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/depth/_/name/buf', 'text': 'Depth Chart', 'shortText': 'Depth Chart', 'isExternal': False, 'isPremium': False}, {'language': 'en', 'rel': ['tickets', 'desktop', 'team'], 'href': 'https://www.vividseats.com/nfl-football/buffalo-bills-tickets.html?wsUser=717', 'text': 'Tickets', 'isExternal': True, 'isPremium': False}]
3 3 s:20~l:28~t:3 chicago-bears Chicago Bears Bears CHI Chicago Bears Bears 152644 0b162a True False [{'href': 'https://a.espncdn.com/i/teamlogos/nfl/500/chi.png', 'width': 500, 'height': 500, 'alt': '', 'rel': ['full', 'default'], 'lastUpdated': '2018-06-05T12:11Z'}, {'href': 'https://a.espncdn.com/i/teamlogos/nfl/500-dark/chi.png', 'width': 500, 'height': 500, 'alt': '', 'rel': ['full', 'dark'], 'lastUpdated': '2018-06-05T12:11Z'}, {'href': 'https://a.espncdn.com/i/teamlogos/nfl/500/scoreboard/chi.png', 'width': 500, 'height': 500, 'alt': '', 'rel': ['full', 'scoreboard'], 'lastUpdated': '2018-06-05T12:11Z'}, {'href': 'https://a.espncdn.com/i/teamlogos/nfl/500-dark/scoreboard/chi.png', 'width': 500, 'height': 500, 'alt': '', 'rel': ['full', 'scoreboard', 'dark'], 'lastUpdated': '2018-06-05T12:11Z'}] [{'summary': '6-11', 'stats': [{'name': 'playoffSeed', 'value': 13.0}, {'name': 'wins', 'value': 6.0}, {'name': 'losses', 'value': 11.0}, {'name': 'winPercent', 'value': 0.3529411852359772}, {'name': 'gamesBehind', 'value': 0.0}, {'name': 'ties', 'value': 0.0}, {'name': 'OTWins', 'value': 0.0}, {'name': 'OTLosses', 'value': 0.0}, {'name': 'gamesPlayed', 'value': 17.0}, {'name': 'pointsFor', 'value': 311.0}, {'name': 'pointsAgainst', 'value': 407.0}, {'name': 'avgPointsFor', 'value': 18.294116973876953}, {'name': 'avgPointsAgainst', 'value': 23.941177368164062}, {'name': 'points', 'value': -2.5}, {'name': 'differential', 'value': -96.0}, {'name': 'streak', 'value': -1.0}, {'name': 'clincher', 'value': 0.0}, {'name': 'divisionWinPercent', 'value': 0.3333333432674408}, {'name': 'leagueWinPercent', 'value': 0.3333333432674408}, {'name': 'divisionRecord', 'value': 0.0}, {'name': 'divisionWins', 'value': 2.0}, {'name': 'divisionTies', 'value': 0.0}, {'name': 'divisionLosses', 'value': 4.0}]}] [{'language': 'en-US', 'rel': ['clubhouse', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/_/name/chi/chicago-bears', 'text': 'Clubhouse', 'shortText': 'Clubhouse', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['roster', 'desktop', 'team'], 'href': 'http://www.espn.com/nfl/team/roster/_/name/chi/chicago-bears', 'text': 'Roster', 'shortText': 'Roster', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['stats', 'desktop', 'team'], 'href': 'http://www.espn.com/nfl/team/stats/_/name/chi/chicago-bears', 'text': 'Statistics', 'shortText': 'Statistics', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['schedule', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/schedule/_/name/chi', 'text': 'Schedule', 'shortText': 'Schedule', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['photos', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/photos/_/name/chi', 'text': 'photos', 'shortText': 'photos', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['scores', 'sportscenter', 'app', 'team'], 'href': 'sportscenter://x-callback-url/showClubhouse?uid=s:20~l:28~t:3&section=scores', 'text': 'Scores', 'shortText': 'Scores', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['draftpicks', 'desktop', 'team'], 'href': 'http://www.espn.com/nfl/draft/teams/_/name/chi/chicago-bears', 'text': 'Draft Picks', 'shortText': 'Draft Picks', 'isExternal': False, 'isPremium': True}, {'language': 'en-US', 'rel': ['transactions', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/transactions/_/name/chi', 'text': 'Transactions', 'shortText': 'Transactions', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['injuries', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/injuries/_/name/chi', 'text': 'Injuries', 'shortText': 'Injuries', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['depthchart', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/depth/_/name/chi', 'text': 'Depth Chart', 'shortText': 'Depth Chart', 'isExternal': False, 'isPremium': False}, {'language': 'en', 'rel': ['tickets', 'desktop', 'team'], 'href': 'https://www.vividseats.com/nfl-football/chicago-bears-tickets.html?wsUser=717', 'text': 'Tickets', 'isExternal': True, 'isPremium': False}]
4 4 s:20~l:28~t:4 cincinnati-bengals Cincinnati Bengals Bengals CIN Cincinnati Bengals Bengals FF2700 000000 True False [{'href': 'https://a.espncdn.com/i/teamlogos/nfl/500/cin.png', 'width': 500, 'height': 500, 'alt': '', 'rel': ['full', 'default'], 'lastUpdated': '2018-06-05T12:11Z'}, {'href': 'https://a.espncdn.com/i/teamlogos/nfl/500-dark/cin.png', 'width': 500, 'height': 500, 'alt': '', 'rel': ['full', 'dark'], 'lastUpdated': '2018-06-05T12:11Z'}, {'href': 'https://a.espncdn.com/i/teamlogos/nfl/500/scoreboard/cin.png', 'width': 500, 'height': 500, 'alt': '', 'rel': ['full', 'scoreboard'], 'lastUpdated': '2018-06-05T12:11Z'}, {'href': 'https://a.espncdn.com/i/teamlogos/nfl/500-dark/scoreboard/cin.png', 'width': 500, 'height': 500, 'alt': '', 'rel': ['full', 'scoreboard', 'dark'], 'lastUpdated': '2018-06-05T12:11Z'}] [{'summary': '10-7', 'stats': [{'name': 'playoffSeed', 'value': 4.0}, {'name': 'wins', 'value': 10.0}, {'name': 'losses', 'value': 7.0}, {'name': 'winPercent', 'value': 0.5882353186607361}, {'name': 'gamesBehind', 'value': 0.0}, {'name': 'ties', 'value': 0.0}, {'name': 'OTWins', 'value': 1.0}, {'name': 'OTLosses', 'value': 2.0}, {'name': 'gamesPlayed', 'value': 17.0}, {'name': 'pointsFor', 'value': 460.0}, {'name': 'pointsAgainst', 'value': 376.0}, {'name': 'avgPointsFor', 'value': 27.058822631835938}, {'name': 'avgPointsAgainst', 'value': 22.117647171020508}, {'name': 'points', 'value': 1.5}, {'name': 'differential', 'value': 84.0}, {'name': 'streak', 'value': -1.0}, {'name': 'clincher', 'value': 0.0}, {'name': 'divisionWinPercent', 'value': 0.6666666865348816}, {'name': 'leagueWinPercent', 'value': 0.6666666865348816}, {'name': 'divisionRecord', 'value': 0.0}, {'name': 'divisionWins', 'value': 4.0}, {'name': 'divisionTies', 'value': 0.0}, {'name': 'divisionLosses', 'value': 2.0}]}] [{'language': 'en-US', 'rel': ['clubhouse', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/_/name/cin/cincinnati-bengals', 'text': 'Clubhouse', 'shortText': 'Clubhouse', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['roster', 'desktop', 'team'], 'href': 'http://www.espn.com/nfl/team/roster/_/name/cin/cincinnati-bengals', 'text': 'Roster', 'shortText': 'Roster', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['stats', 'desktop', 'team'], 'href': 'http://www.espn.com/nfl/team/stats/_/name/cin/cincinnati-bengals', 'text': 'Statistics', 'shortText': 'Statistics', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['schedule', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/schedule/_/name/cin', 'text': 'Schedule', 'shortText': 'Schedule', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['photos', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/photos/_/name/cin', 'text': 'photos', 'shortText': 'photos', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['scores', 'sportscenter', 'app', 'team'], 'href': 'sportscenter://x-callback-url/showClubhouse?uid=s:20~l:28~t:4&section=scores', 'text': 'Scores', 'shortText': 'Scores', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['draftpicks', 'desktop', 'team'], 'href': 'http://www.espn.com/nfl/draft/teams/_/name/cin/cincinnati-bengals', 'text': 'Draft Picks', 'shortText': 'Draft Picks', 'isExternal': False, 'isPremium': True}, {'language': 'en-US', 'rel': ['transactions', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/transactions/_/name/cin', 'text': 'Transactions', 'shortText': 'Transactions', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['injuries', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/injuries/_/name/cin', 'text': 'Injuries', 'shortText': 'Injuries', 'isExternal': False, 'isPremium': False}, {'language': 'en-US', 'rel': ['depthchart', 'desktop', 'team'], 'href': 'https://www.espn.com/nfl/team/depth/_/name/cin', 'text': 'Depth Chart', 'shortText': 'Depth Chart', 'isExternal': False, 'isPremium': False}, {'language': 'en', 'rel': ['tickets', 'desktop', 'team'], 'href': 'https://www.vividseats.com/nfl-football/cincinnati-bengals-tickets.html?wsUser=717', 'text': 'Tickets', 'isExternal': True, 'isPremium': False}]
...
[25 rows x 16 columns]
json.items() returns not just the values but the keys also, meaning that the first value in teams is the string "sports" and the second value is what you are looking for, the list. That's what is causing the error.
Edit: You want to do for key, teams in teams_json.items() not for teams in teams_json.items()

Simple filtering by date and user who created the event

How would I filter all events by the user who are the attendees but not the creator of the event(email) and only the current month's events. So start date this month to end date this month last day.
https://developers.google.com/calendar/api/v3/reference/events
events_result = service.events().list(calendarId=CAL_ID, maxResults=2500).execute()
events = events_result.get('items', [])
for e in events:
print(e)
An example result:
[{'kind': 'calendar#event', 'etag': '"-', 'id': '-', 'status':
'confirmed', 'htmlLink': '-', 'created': '2022-01-12T04:06:25.000Z',
'updated': '2022-01-16T20:47:31.134Z', 'summary': 'First day',
'description': 'This is just here as a placeholder', 'creator':
{'email': 'x#hotmail.com'}, 'organizer': {'email':
'-#group.calendar.google.com', 'displayName': 'HerokuApp', 'self':
True}, 'start': {'dateTime': '2022-01-11T02:30:00-08:00', 'timeZone':
'America/Vancouver'}, 'end': {'dateTime': '2022-01-11T03:30:00-08:00',
'timeZone': 'America/Vancouver'}, 'iCalUID': '-#google.com',
'sequence': 0, 'reminders': {'useDefault': True}, 'eventType':
'default'}, {'kind': 'calendar#event', 'etag': '"-"', 'id': '-',
'status': 'confirmed', 'htmlLink': '-', 'created':
'2022-01-12T04:15:09.000Z', 'updated': '2022-01-16T20:48:08.061Z',
'summary': '2nd', 'description': 'Testing sort', 'creator': {'email':
'#hotmail.com'}, 'organizer': {'email': '-#group.calendar.google.com',
'displayName': 'HerokuApp', 'self': True}, 'start': {'dateTime':
'2022-01-11T04:00:00-08:00', 'timeZone': 'America/Vancouver'}, 'end':
{'dateTime': '2022-01-11T05:00:00-08:00', 'timeZone':
'America/Vancouver'}, 'iCalUID': '-#google.com', 'sequence': 0,
'reminders': {'useDefault': True}, 'eventType': 'default'}]
Using this for month
today = datetime.today()
monthAgo = today - relativedelta(months=1)
tmax = today.isoformat('T') + "Z"
tmin = monthAgo.isoformat('T') + "Z"
events_result = service.events().list(
calendarId=CAL_ID,
timeMin=tmin,
timeMax=tmax,
maxResults=2500,
singleEvents=True,
orderBy='startTime',
).execute()

Dataframe not showing twitter sources from Android

I am trying to try to do some analysis on a twitter account, but I am having trouble trying to show sources from Android. What I did was merged two json files and I think I merged it correctly, but incase I got that wrong here is the code I used.
old_tweets = load_tweets("real_tweets/real_old_tweets.json")
print(len(old_tweets))
for aLis1 in old_tweets:
if aLis1 not in tweets:
tweets.append(aLis1)
load_tweets is a custom function that simply opens and loads a json file given a specific path
with open(path, "rb") as f:
import json
return json.load(f)
After merging the two json files of tweets I then called this function to create the data frame and clean it up to only display the information I want.
df_tweets1 = pd.DataFrame(tweets)
df_tweets2 = df_tweets1[['id','created_at','source','full_text','retweet_count']]
df_tweets = df_tweets2.drop_duplicates('id', keep=False
df_tweets.set_index('id', inplace=True)
df_tweets = df_tweets.rename(columns={"created_at": "time", "full_text": "text"})
df_tweets["time"] = pd.to_datetime(df_tweets["time"])
The problem is that when i call df_tweets["source"].unique() I don't see any tweets coming from android
array(['Twitter for iPhone',
'Twitter for iPad',
'Twitter Media Studio',
'Media Studio',
'Twitter Web Client'],
dtype=object)
Did I do something wrong when merging the two sets of Twitter data? Or did I do something wrong when trying to create the data frame?
EDIT**Here is a sample output from real_old_tweets.json to give a sense of the format. I am only going to post one because there is a lot of information contained in one tweet.
[{'created_at': 'Tue Oct 16 16:22:11 +0000 2018',
'id': 1052233253040640001,
'id_str': '1052233253040640001',
'full_text': 'REGISTER TO https://url/0pWiwCHGbh! #MAGA🇺🇸 https://url/ACTMe53TZU',
'truncated': False,
'display_text_range': [0, 44],
'entities': {'hashtags': [{'text': 'MAGA', 'indices': [37, 42]}],
'symbols': [],
'user_mentions': [],
'urls': [{'url': 'url/0pWiwCHGbh',
'expanded_url': 'linkVote.GOP',
'display_url': 'Vote.GOP',
'indices': [12, 35]},
{'url': 'url/ACTMe53TZU',
'expanded_url': 'linktwitter.com/erictrump/status/1052174007708147714',
'display_url': 'twitter.com/erictrump/stat…',
'indices': [45, 68]}]},
'source': 'Twitter for iPhone',
'in_reply_to_status_id': None,
'in_reply_to_status_id_str': None,
'in_reply_to_user_id': None,
'in_reply_to_user_id_str': None,
'in_reply_to_screen_name': None,
'user': {'id': 25073877,
'id_str': '25073877',
'name': 'Donald J. Trump',
'screen_name': 'realDonaldTrump',
'location': 'Washington, DC',
'description': '45th President of the United States of America🇺🇸',
'url': 'url/OMxB0x7xC5',
'entities': {'url': {'urls': [{'url': 'url/OMxB0x7xC5',
'expanded_url': 'linkwww.Instagram.com/realDonaldTrump',
'display_url': 'Instagram.com/realDonaldTrump',
'indices': [0, 23]}]},
'description': {'urls': []}},
'protected': False,
'followers_count': 55165024,
'friends_count': 47,
'listed_count': 94709,
'created_at': 'Wed Mar 18 13:46:38 +0000 2009',
'favourites_count': 25,
'utc_offset': None,
'time_zone': None,
'geo_enabled': True,
'verified': True,
'statuses_count': 39296,
'lang': 'en',
'contributors_enabled': False,
'is_translator': False,
'is_translation_enabled': True,
'profile_background_color': '6D5C18',
'profile_background_image_url': 'linkabs.twimg.com/images/themes/theme1/bg.png',
'profile_background_image_url_https': 'linkabs.twimg.com/images/themes/theme1/bg.png',
'profile_background_tile': True,
'profile_image_url': 'linkpbs.twimg.com/profile_images/874276197357596672/kUuht00m_normal.jpg',
'profile_image_url_https': 'linkpbs.twimg.com/profile_images/874276197357596672/kUuht00m_normal.jpg',
'profile_banner_url': 'linkpbs.twimg.com/profile_banners/25073877/1539493274',
'profile_link_color': '1B95E0',
'profile_sidebar_border_color': 'BDDCAD',
'profile_sidebar_fill_color': 'C5CEC0',
'profile_text_color': '333333',
'profile_use_background_image': True,
'has_extended_profile': False,
'default_profile': False,
'default_profile_image': False,
'following': False,
'follow_request_sent': False,
'notifications': False,
'translator_type': 'regular'},
'geo': None,
'coordinates': None,
'place': None,
'contributors': None,
'is_quote_status': True,
'quoted_status_id': 1052174007708147714,
'quoted_status_id_str': '1052174007708147714',
'quoted_status_permalink': {'url': 'url/ACTMe53TZU',
'expanded': 'linktwitter.com/erictrump/status/1052174007708147714',
'display': 'twitter.com/erictrump/stat…'},
'quoted_status': {'created_at': 'Tue Oct 16 12:26:46 +0000 2018',
'id': 1052174007708147714,
'id_str': '1052174007708147714',
'full_text': 'Friends: Quick reminder that today is that last day to register to vote in Oregon, Kansas, Louisiana, West Virginia, New Jersey and Maryland. It is very quick and easy - simply go to url/GE5BO5ONN1! Let’s #MakeAmericaGreatAgain 🇺🇸🇺🇸🇺🇸',
'truncated': False,
'display_text_range': [0, 243],
'entities': {'hashtags': [{'text': 'MakeAmericaGreatAgain',
'indices': [214, 236]}],
'symbols': [],
'user_mentions': [],
'urls': [{'url': 'url/GE5BO5ONN1',
'expanded_url': 'linkwww.Vote.GOP',
'display_url': 'Vote.GOP',
'indices': [183, 206]}]},
'source': 'Twitter for iPhone',
'in_reply_to_status_id': None,
'in_reply_to_status_id_str': None,
'in_reply_to_user_id': None,
'in_reply_to_user_id_str': None,
'in_reply_to_screen_name': None,
'user': {'id': 39349894,
'id_str': '39349894',
'name': 'Eric Trump',
'screen_name': 'EricTrump',
'location': '',
'description': "Executive Vice President of The #Trump Organization. Husband to #LaraLeaTrump. Large advocate of #StJude Children's Research Hospital. #MakeAmericaGreatAgain",
'url': 'url/uwwNiWyamR',
'entities': {'url': {'urls': [{'url': 'url/uwwNiWyamR',
'expanded_url': 'linkwww.Trump.com',
'display_url': 'Trump.com',
'indices': [0, 23]}]},
'description': {'urls': []}},
'protected': False,
'followers_count': 2191617,
'friends_count': 715,
'listed_count': 5736,
'created_at': 'Mon May 11 21:42:30 +0000 2009',
'favourites_count': 8638,
'utc_offset': None,
'time_zone': None,
'geo_enabled': True,
'verified': True,
'statuses_count': 5601,
'lang': 'en',
'contributors_enabled': False,
'is_translator': False,
'is_translation_enabled': False,
'profile_background_color': '000000',
'profile_background_image_url': 'linkabs.twimg.com/images/themes/theme1/bg.png',
'profile_background_image_url_link': 'linkabs.twimg.com/images/themes/theme1/bg.png',
'profile_background_tile': True,
'profile_image_url': 'linkpbs.twimg.com/profile_images/974045997268529152/R0CuVYHM_normal.jpg',
'profile_image_url_link': 'linkpbs.twimg.com/profile_images/974045997268529152/R0CuVYHM_normal.jpg',
'profile_banner_url': 'linkpbs.twimg.com/profile_banners/39349894/1516709628',
'profile_link_color': '116AB8',
'profile_sidebar_border_color': '000000',
'profile_sidebar_fill_color': '616161',
'profile_text_color': '000000',
'profile_use_background_image': True,
'has_extended_profile': False,
'default_profile': False,
'default_profile_image': False,
'following': False,
'follow_request_sent': False,
'notifications': False,
'translator_type': 'none'},
'geo': None,
'coordinates': None,
'place': None,
'contributors': None,
'is_quote_status': False,
'retweet_count': 1945,
'favorite_count': 3828,
'favorited': False,
'retweeted': False,
'possibly_sensitive': False,
'lang': 'en'},
'retweet_count': 5415,
'favorite_count': 16565,
'favorited': False,
'retweeted': False,
'possibly_sensitive': False,
'lang': 'en'},
I am assuming that you are having "android" sources and I don't have a clear idea of how your data looks like and what is the relation between the "id" and source. Having said that, there is a bug when you are preparing your data. You are dropping all the duplicates.
For example:
>>> import pandas as pd
>>> df = pd.DataFrame(data={'col1':[1,2,2],'col2':[3,4,3],'col3':[1,4,1]})
>>> df
col1 col2 col3
0 1 3 1
1 2 4 4
2 2 3 1
>>> df.drop_duplicates('col1',keep=False)
col1 col2 col3
0 1 3 1
In the above code, you can see that it is dropping all the duplicate rows if you use "keep=False".
>>> df.drop_duplicates('col1',keep='first')
col1 col2 col3
0 1 3 1
1 2 4 4
Instead, use keep='first' or keep='last' and see if there is any improvement. Also, it would be great if I can get more sense of data, to figure out where it is going wrong.
EDIT
After some time, I took your JSON object and saved to a "me.json" file in the format of:
[{},{}]
Where the first object's source is an iPhone and second object's source is android. I used your code for loading in the data:
Python 2.7.15rc1 (default, Nov 12 2018, 14:31:15)
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> import json
>>> with open('me.json','rb') as file:
... json_list = json.load(file)
...
>>> len(json_list)
2
>>> df = pd.DataFrame(json_list)
>>> df1 = df[['id','source']]
>>> df1['source'].value_counts()
Twitter for Android 1
Twitter for iPhone 1
Name: source, dtype: int64
In the above output, you can see that I am able to see the "Android". My conclusion is that in your data, there might be no "Android" at all, in the df['source'] column.
Please see it carefully as there are two "source" keys inside each JSON object, one key is inside the "quoted_status". There are chances that you might have seen "Android" in this key.

Python how to delete empty row from list of dictionaries in a for loop?

I have a list of dictionaries called "rows." However, this list contains some dictionaries with empty values (labeled as "None"). How could I write a for loop that goes through every row and deletes the entire row if it finds "None" in the key named 'lat'?
This is my list of dictionaries ("rows"). I'd like to write a for loop that deletes first row since 'lat' value is 'None.' Any help would be greatly appreciated!
[{'created_at': None, 'userid': None, 'long': None, 'tweetid': None, 'timestamp': None, 'lat': None},
{'created_at': 'Mon May 04 04:35:34 +0000 2015', 'userid': '113503286', 'long': '-87.624387', 'tweetid': '595084326489956352', 'timestamp': '1430714134224', 'lat': '41.852653'},
{'created_at': 'Mon May 04 04:35:46 +0000 2015', 'userid': '2421024865', 'long': '-87.629798', 'tweetid': '595084376632729600', 'timestamp': '1430714146179', 'lat': '41.878114'}]
List Comprehensions is the way to go.
rows = [{'created_at': None, 'userid': None, 'long': None, 'tweetid': None, 'timestamp': None, 'lat': None},
{'created_at': 'Mon May 04 04:35:34 +0000 2015', 'userid': '113503286', 'long': '-87.624387', 'tweetid': '595084326489956352', 'timestamp': '1430714134224', 'lat': '41.852653'},
{'created_at': 'Mon May 04 04:35:46 +0000 2015', 'userid': '2421024865', 'long': '-87.629798', 'tweetid': '595084376632729600', 'timestamp': '1430714146179', 'lat': '41.878114'}]
rows = [ row for row in rows if row['lat'] is not None ]
You can read the
list comprehensions documentation to learn more.
If you are a beginner, an easy way to understand it is to define a function and remove them.
def deleteNone(rows):
ret = []
for item in rows:
if item.get('lat', None) is None:
pass
else:
ret.append(item)
return ret
rows = deleteNone(rows)
A universal way is to iterate it by index in reverse order. This makes sure that the indices will still be correct after removing items. By universal I mean that the same technique will work in any programming language.
for n in range(len(list) - 1,-1,-1):
if list[n]['lat'] == None:
list.pop(n)
You can filter on having a non-Falsish lat attribute:
d = [{'created_at': None, 'userid': None, 'long': None, 'tweetid': None, 'timestamp': None, 'lat': None},
{'created_at': 'Mon May 04 04:35:34 +0000 2015', 'userid': '113503286', 'long': '-87.624387', 'tweetid': '595084326489956352', 'timestamp': '1430714134224', 'lat': '41.852653'},
{'created_at': 'Mon May 04 04:35:46 +0000 2015', 'userid': '2421024865', 'long': '-87.629798', 'tweetid': '595084376632729600', 'timestamp': '1430714146179', 'lat': '41.878114'}]
d = list(filter(lambda x: x["lat"], d))
Perfect job for filter:
rows = filter(lambda d: d['lat'] is not None, rows)
Although if you can, it would be better to prevent getting those bad rows into your list in the first place!

Categories

Resources