Develop Reference

Python ValueError: Length of values does not match length of index

I am carrying out a API search on scaleserp and for each search I do i want to put the output in the column in my dataframe.
#Matches the GET request
api_result = requests.get('https://api.scaleserp.com/search', params, verify=False)
print(type(api_result))
#stores the result in JSON
result = api_result.json()
print(type(result))
#Creates a new DataFrame with 'Organic_Results' From the JSON output.
Results_df = (result['organic_results'])
#FOR loop to look at each result and select which output from the JSON is wanted.
for res in Results_df:
StartingDataFrame['JSONDump'] = res
api_result is a requests.models.Response
result is a dict.
RES is a dict.
I want the RES to be put into the column Dump. is this possible?
Updated Code
#Matches the GET request
api_result = requests.get('https://api.scaleserp.com/search', params, verify=False)
#stores the result in JSON
result = api_result.json()
#Creates a new DataFrame with 'Organic_Results' From the JSON output.
Results_df = (result['organic_results'])
#FOR loop to look at each result and select which output from the JSON is wanted.
for res in Results_df:
Extracted_data = {key: res[key] for key in res.keys()
& {'title', 'link', 'snippet_matched', 'date', 'snippet'}}
Extracted_data is a dict and contains the info i need.
{'title': '25 Jun 1914 - Advertising - Trove', 'link': 'https://trove.nla.gov.au/newspaper/article/7280119', 'snippet_matched': ['()', 'charge', 'Dan Whit'], 'snippet': 'I Iron roof, riltibcd II (),. Line 0.139.5. wai at r ar ... Propertb-« entired free of charge. Line 2.130.0 ... AT Dan Whit",\'»\', 6il 02 sturt »L, Prlnce\'»~Brti\'»e,. Line 3.12.0.'}
{'snippet': "Mary Bardwell is in charge of ... I() •. Al'companit'd by: Choppf'd Chitkf'n Li\\f>r Palt·. 1h!iiSC'o Gret'n Salad g iii ... of the overtime as Dan Whit-.", 'title': 'October 16,1980 - Bethlehem Public Library',
'link': 'http://www.bethlehempubliclibrary.org/webapps/spotlight/years/1980/1980-10-16.pdf', 'snippet_matched': ['charge', '()', 'Dan Whit'], 'date': '16 Oct 1980'}
{'snippet': 'CONGRATULATIONS TO DAN WHIT-. TLE ON THE ... jailed and beaten dozens of times. In one of ... ern p()rts ceased. The MIF is not only\xa0...', 'title': 'extensions of remarks - US Government Publishing Office', 'link': 'https://www.gpo.gov/fdsys/pkg/GPO-CRECB-1996-pt5/pdf/GPO-CRECB-1996-pt5-7-3.pdf', 'snippet_matched': ['DAN WHIT', 'jailed', '()'], 'date': '26 Apr 1986'}
{'snippet': 'ILLUSTRATION BY DAN WHIT! By Matt Manning ... ()n the one hand, there are doctors on both ... self-serving will go to jail at the beginning of\xa0...', 'title': 'The BG News May 23, 2007 - ScholarWorks#BGSU - Bowling ...', 'link': 'https://scholarworks.bgsu.edu/cgi/viewcontent.cgi?article=8766&context=bg-news', 'snippet_matched': ['DAN WHIT', '()', 'jail'], 'date': '23 May 2007'}
{'snippet': '$19.95 Charge card number SERVICE HOURS: ... Explorer Advisor Dan Whit- ... lhrr %(OnrwflC or ()utuflrueonlinelfmarketing (arnpaigfl%? 0I - .',
'title': '<%BANNER%> TABLE OF CONTENTS HIDE Section A: Main ...', 'link': 'https://ufdc.ufl.edu/UF00028295/00194', 'snippet_matched': ['Charge', 'Dan Whit', '()'], 'date': 'Listings 1 - 800'}
{'title': 'Lledo Promotional,Bull Nose Morris,Dandy,Desperate Dan ...', 'link': 'https://www.ebay.co.uk/itm/Lledo-Promotional-Bull-Nose-Morris-Dandy-Desperate-Dan-White-Van-/233817683840', 'snippet_matched': ['charges'], 'snippet': 'No additional import charges on delivery. This item will be sent through the Global Shipping Programme and includes international tracking. Learn more- opens\xa0...'}

The problem looks like the length of your organic_results is not the same as the length of your dataframe.
StartingDataFrame['Dump'] = (result['organic_results'])
Here your setting the whole column dump to equal the organic_results which is smaller or larger than your already defined dataframe. I'm not sure what your dataframe already has but you could do this by iterating through the rows and stashing them in that row if you have values you want them to add up with like this:
StartingDataFrame['Dump'] = []*len(StartingDataFrame)
for i,row in StartingDataFrame.iterrows():
StartingDataFrame.at[i,'Dump'] = result['organic_results']
Depending on what your data looks like you could maybe just append it to the dataframe
StartingDataFrame = StartingDataFrame.append(result['organic_results'],ignore_index=True)
Could you show us a sample of what both data source looks like?

Iterating through JSON Array in Python Match Array of Values

I'm working with the songkick api found here and am very close to finishing a program I'm working on to pull info about a few specific artists upcoming shows. I have inputted an array of metro_areas that I want to track and output ONLY shows within THESE cities and their accompanying ID's. Other info I'm pulling is date of show, artist name, venue name. Basically right now my program is able to pull every show from the list of artist_ids I've inputted iterating through the id's in the request url for the range of dates in my parameters. My current output looks something like this:
['Date', 'Artist Name', 'Venue Name', 'City', 'metroArea ID']
['FEB - 7', 'Rosalia', 'NOTO', 'Philadelphia, PA, US', 5202]
['FEB - 8', 'Rosalia', 'Audio', 'San Francisco, CA, US', 26330]
['FEB - 8', 'Kid Cudi', 'Shady Park', 'Tempe, AZ, US', 23068]
['FEB - 8', 'Kid Cudi', 'Madison Square Garden', 'New York City, NY, US', 7644]
I want the output to be like this:
['FEB - 8', 'Rosalia', 'Audio', 'San Francisco, CA, US', 26330]
['FEB - 8', 'Kid Cudi', 'Madison Square Garden', 'New York City, NY, US', 7644]
Based on this array I've defined at the beginning of my program to match with the metro_areas in the songkick object array.
metro_areas = [
('Los Angeles', '17835'),
('San Francisco', '26330'),
('New York City', '7644'),
('Seattle', '2846'),
('Nashville', '11104')
]
Here is the json object array that I pull for each artist_id:
{
"resultsPage": {
"results": {
"event": [
{
"id":11129128,
"type":"Concert",
"uri":"http://www.songkick.com/concerts/11129128-wild-flag-at-fillmore?utm_source=PARTNER_ID&utm_medium=partner",
"displayName":"Wild Flag at The Fillmore (April 18, 2012)",
"start": {
"time":"20:00:00",
"date":"2012-04-18",
"datetime":"2012-04-18T20:00:00-0800"
},
"performance": [
{
"artist": {
"id":29835,
"uri":"http://www.songkick.com/artists/29835-wild-flag?utm_source=PARTNER_ID&utm_medium=partner",
"displayName":"Wild Flag",
"identifier": []
},
"id":21579303,
"displayName":"Wild Flag",
"billingIndex":1,
"billing":"headline"
}
],
"location": {
"city":"San Francisco, CA, US",
"lng":-122.4332937,
"lat":37.7842398
},
"venue": {
"id":6239,
"displayName":"The Fillmore",
"uri":"http://www.songkick.com/venues/6239-fillmore?utm_source=PARTNER_ID&utm_medium=partner",
"lng":-122.4332937,
"lat":37.7842398,
"metroArea": {
"id":26330,
"uri":"http://www.songkick.com/metro-areas/26330-us-sf-bay-area?utm_source=PARTNER_ID&utm_medium=partner",
"displayName":"SF Bay Area",
"country": { "displayName":"US" },
"state": { "displayName":"CA" }
}
},
"status":"ok",
"popularity":0.012763
}, ....
]
},
"totalEntries":24,
"perPage":50,
"page":1,
"status":"ok"
}
}
Some more code to see how I'm getting to my output from the JSON in the songkick requests.
metro_areas = [
('Los Angeles','17835'),
('San Francisco', '26330'),
('New York City','7644'),
('Seattle','2846'),
('Nashville','11104')
]
# artists we want to track
artist_ids = [
('Rosalia', '4610868'), ('EARTHGANG', '5720759'), ('Kid Cudi', '8630279'), ('Kanye West', '5566863'),
('Ludacris', '398291'), ('Hayley Williams', '10087966')
]
# Fetch existing events in each metro area
for artist_id in artist_ids:
params = {
'apikey': 'API_KEY',
'min_date': '2020-02-01',
'max_date': '2020-02-08',
# 'type': 'Concert'
}
r = requests.get('https://api.songkick.com/api/3.0/artists/' + artist_id[1] + '/calendar.json', params=params)
response = r.json()
shows = response['resultsPage']['results']
for show in shows:
try:
shows = shows['event']
formatted_shows = [{
'artistID': [perf['artist']['id'] for perf in s['performance']],
'date': s['start']['date'],
'name': [perf['artist']['displayName'] for perf in s['performance']],
'metroArea': s['venue']['metroArea']['id'],
'city': s['location']['city'],
'venue': s['venue']['displayName']
}
for s in shows if len(s['performance']) > 0
]
for sub in formatted_shows:
if sub['artistID'] == artist_id[1]:
sub['name'] = artist_id[0]
new_show = artist_id[1]
new_show_name = artist_id[0]
new_date = sub['date']
new_date_time = new_date = datetime.strptime(new_date, '%Y-%m-%d')
date_time_fin = new_date_time.strftime('%b - %-d').upper()
formatted_show_final = [date_time_fin, new_show_name, sub['venue'], sub['city'], sub['metroArea']
print(formatted_show_final)
Long story short, I need to find a way to iterate through each of my listed metro_areas id's (LA, SF, NYC, Seattle, Nashville) only and only output the shows that match with 'metroArea': s['venue']['metroArea']['id'] for each request iteration.

If I understood well the question, add inside the second for loop: if sub['metroArea'] in [area[1] for area in metro_areas]:
for show in shows:
try:
shows = shows['event']
formatted_shows = [{
'artistID': [perf['artist']['id'] for perf in s['performance']],
'date': s['start']['date'],
'name': [perf['artist']['displayName'] for perf in s['performance']],
'metroArea': s['venue']['metroArea']['id'],
'city': s['location']['city'],
'venue': s['venue']['displayName']
}
for s in shows if len(s['performance']) > 0
]
for sub in formatted_shows:
#Modified here to apply str() function to transform #sub['metroArea'] to string instead of int value
if str(sub['metroArea']) in [area[1] for area in metro_areas]:
if sub['artistID'] == artist_id[1]:
sub['name'] = artist_id[0]
new_show = artist_id[1]
new_show_name = artist_id[0]
new_date = sub['date']
new_date_time = new_date = datetime.strptime(new_date, '%Y-%m-%d')
date_time_fin = new_date_time.strftime('%b - %-d').upper()
formatted_show_final = [date_time_fin, new_show_name, sub['venue'], sub['city'], sub['metroArea']]
print(formatted_show_final)

Sequentially filescraping text files - a smarter way?

I'm trying to scrape some text files into a DB - the format is similar to this with a couple of 1000 segments like this :
Posted By
Date
John Keys
31.08.2019, 10:10 AM
Peter Hall 200 150
Ed Parker 14 1
Posted By
Date
John Keys
31.08.2019, 10:15 AM
Rose Stone 200 150
Travis Anderson 14 1
The records that are important are the records that are coming right after "Date" - so the logic is :
inside_match_flag =0
for line in ins:
if inside_match_flag == 1:
inside_match_flag = 2 # add one to it as we will get all lines
if line == "Posted By": # until we see Posted By again (or EOF)
inside_match_flag =0 # we are now outside the segment
if line == "Date" : # lines after Dates are the ones we want
inside_match_flag =1 # the following lines are to be stored
So this is the way I've done it (the above is not the running code) before doing this by keeping track of a flag and depending on the flag_value I know what lines are most likely coming next.
The issue is of course this about 'the lines coming next' - as I'm reading line per line, I can't easy grab out these segments as I don't want to rely on loading the complete file into memory (as it can go huge).
But the code always gets ugly when I implement something like this - and thinking anyone here that would have a lot smarter approach to do this ?
And note - I am also interested if there would be a super-smart compact way to do this if it requires to load all into memory where code doesn't get so ugly, if all is in memory I guess I can just look for DATE field and save all lines between until it sees Posted By again.
Edit 1
Note the number of players can be more than 2 per game, so a record could also look like this :
Posted By
Date
John Keys
31.08.2019, 10:10 AM
Peter Hall 200 150
Ed Parker 54 1
Rose Stone 20 15
Travis Anderson 1 150
Posted By
...
....
My dream format would be to have an object like this - example based on the match above with 4 players :
{
"Game 1:"
{
"posted by" : "john keys"
"date" : "31.08.2019, 10:10 AM"
"players" : {
{ 1, "Peter Hall, "200", "150" }
{ 2, Ed Parker, "54", "1" }
{ 3 , Rose Stone, "20", "15" }
{ 4, Travis Anderson, "1", "150" }
}
}
}
Note : not 100% correct json format there - and doesn't have to be json, just an object as I will throw them into a SQLite database where it's stored per game which should be illustrated above.

Optimized and memory-efficient generator function approach which yields records on demand:
import pprint
def extract_records(fname):
def prepare_record(rec):
return {'posted by': rec[0], 'date': rec[1],
'players': [[i] + p.rsplit(maxsplit=2)
for i, p in enumerate(rec[2:], 1)]}
with open(fname) as f:
record = []
add_item = False
for line in f:
line = line.strip()
if line == 'Date':
add_item = True
continue
elif line == 'Posted By':
add_item = False
if record:
yield prepare_record(record)
record = []
continue
if add_item:
record.append(line)
if record:
yield prepare_record(record)
records_gen = extract_records('datafile.txt') # generator
for rec in records_gen:
pprint.pprint(rec) # further processing, ex. inserting into DB
The output (2 sample records):
{'date': '31.08.2019, 10:10 AM',
'players': [[1, 'Peter Hall', '200', '150'],
[2, 'Ed Parker', '14', '1'],
[3, 'Rose Stone', '20', '15'],
[4, 'Travis Anderson', '1', '150']],
'posted by': 'John Keys'}
{'date': '31.08.2019, 10:15 AM',
'players': [[1, 'Rose Stone', '200', '150'],
[2, 'Travis Anderson', '14', '1']],
'posted by': 'John Keys'}

There is no magic method for this specific case. Here is an example solution:
buf_size = ...
start_marker = "Posted by\n"
date_marker = "Date\n"
def parse_game(filename)
fh = open(filename)
page = ""
buffer = True # just the start value
while buffer:
buffer = fh.read(buf_size)
page += buffer
records = page.split(start_marker)
if buffer:
page = records.pop()
for record in records:
# skip everything before "Date" and split by lines
chunks = record.split(date_marker, 1)[-1].split("\n")
posted_by, date = chunks[:2]
players = [chunk.split() for chunk in chunks[2:]]
yield {
"posted_by": posted_by,
"date": date,
"players": players
}
If you can read the whole file into memory, it will be just:
def read_game(filename):
for record in open(filename).read().split(start_marker):
# skip everything before "Date" and split by lines
chunks = record.split(date_marker, 1)[-1].split("\n")
posted_by, date = chunks[:2]
players = [chunk.split() for chunk in chunks[2:]]
yield {
"posted_by": posted_by,
"date": date,
"players": players
}
This solution is very similar to Roman's. It is slightly less memory efficient (assuming you have buf_size of memory), but will result in less IO

How to extract first item in this json list

I have this json that im trying to extract the first element/list in this list of lists in python... How do you extract the first item in ads_list?
Are there any built in functions i can use to extract first item in a json file? I need something other than simply iterating through this like an array....
P.S. i shortened the json data
Here is the list.
{
u'data':{
u'ad_list':[
{
u'data':{
u'require_feedback_score':0,
u'hidden_by_opening_hours':False,
u'trusted_required':False,
u'currency':u'EGP',
u'require_identification':False,
u'is_local_office':False,
u'first_time_limit_btc':None,
u'city':u'',
u'location_string':u'Egypt',
u'countrycode':u'EG',
u'max_amount':u'20000',
u'lon':0.0,
u'sms_verification_required':False,
u'require_trade_volume':0.0,
u'online_provider':u'SPECIFIC_BANK',
u'max_amount_available':u'20000',
u'msg': u" \u2605\u2605\u2605\u2605\u2605 \u0645\u0631\u062d\u0628\u0627 \u2605\u2605\u2605\u2605\u2605\r\n\r\n\u0625\u0630\u0627 \u0643\u0646\u062a \u062a\u0631\u063a\u0628 \u0641\u064a \u0628\u064a\u0639 \u0627\u0648 \u0634\u0631\u0627\u0621 \u0627\u0644\u0628\u062a\u0643\u0648\u064a\u0646 \u062a\u0648\u0627\u0635\u0644 \u0645\u0639\u064a \u0648\u0633\u0623\u0642\u0648\u0645 \u0628\u062e\u062f\u0645\u062a\u0643\r\n\u0644\u0644\u062a\u0648\u0627\u0635\u0644: https: //tawk.to/hanyibrahim\r\n \u0627\u0644\u062e\u064a\u0627\u0631 \u0644\u0644\u062a\u062d\u0648\u064a\u0644: \u0627\u0644\u0628\u0646\u0643 \u0627\u0644\u062a\u062c\u0627\u0631\u064a \u0627\u0644\u062f\u0648\u0644\u064a \u0627\u0648\u0641\u0648\u062f\u0627\u0641\u0648\u0646 \u0643\u0627\u0634 \u0627\u0648 \u0627\u062a\u0635\u0627\u0644\u0627\u062a \u0641\u0644\u0648\u0633 \u0627\u0648 \u0627\u0648\u0631\u0627\u0646\u062c \u0645\u0648\u0646\u064a\r\n\r\n'' \u0634\u0643\u0631\u0627 ''\r\n\r\n\r\n \u2605\u2605\u2605\u2605\u2605 Hello \u2605\u2605\u2605\u2605\u2605\r\n\r\nIf you would like to trade Bitcoins please let me know and I will help you\r\nconnect: https: //tawk.to/hanyibrahim\r\nOption transfer:Bank CIB Or Vodafone Cash Or Etisalat Flous Or Orange Money\r\n'' Thank ''",
u'volume_coefficient_btc':u'1.50',
u'profile':{
u'username':u'hanyibrahim11',
u'feedback_score':100,
u'trade_count':u'3000+',
u'name':u'hanyibrahim11 (3000+; 100%)',
u'last_online': u'2019-01-14T17:54:52+00:00 '}, u' bank_name':u'CIB_Vodafone Cash_Etisalat Flous_Orange Money',
u'trade_type':u'ONLINE_BUY',
u'ad_id':803036,
u'temp_price':u'67079.44',
u'payment_window_minutes':90,
u'min_amount':u'50',
u'limit_to_fiat_amounts':u'',
u'require_trusted_by_advertiser':False,
u'temp_price_usd':u'3738.54',
u'lat':0.0,
u'visible':True,
u'created_at': u'2018-07-25T08:12:21+00:00 ', u' atm_model':None,
u'is_low_risk':True
},
u'actions':{
u'public_view': u'https://localbitcoins.com/ad/803036'
}
},
{
u'data':{
u'require_feedback_score':0,
u'hidden_by_opening_hours':False,
u'trusted_required':False,
u'currency':u'EGP',
u'require_identification':False,
u'is_local_office':False,
u'first_time_limit_btc':None,
u'city':u'',
u'location_string':u'Egypt',
u'countrycode':u'EG',
u'max_amount':u'20000',
u'lon':0.0,
u'sms_verification_required':False,
u'require_trade_volume':0.0,
u'online_provider':u'CASH_DEPOSIT',
u'max_amount_available':u'20000',
u'msg':u'QNB,
CIB deposite- Vodafone Cash - Etisalat Felous - Orange Money - Western Union - Money Gram \r\n- Please do not entiate a new trade request if you are not serious to finalize it.',
u'volume_coefficient_btc':u'1.50',
u'profile':{
u'username':u'Haboush',
u'feedback_score':99,
u'trade_count':u'500+',
u'name':u'Haboush (500+; 99%)',
u'last_online': u'2019-01-14T16:48:52+00:00 '}, u' bank_name':u'QNB\u2714CIB\u2714Vodafone\u2714Orange\u2714Etisalat\u2714WU',
u'trade_type':u'ONLINE_BUY',
u'ad_id':719807,
u'temp_price':u'66860.18',
u'payment_window_minutes':270,
u'min_amount':u'100',
u'limit_to_fiat_amounts':u'',
u'require_trusted_by_advertiser':False,
u'temp_price_usd':u'3726.32',
u'lat':0.0,
u'visible':True,
u'created_at': u'2018-03-24T19:29:08+00:00 ', u' atm_model':None,
u'is_low_risk':True
},
u'actions':{
u'public_view': u'https://localbitcoins.com/ad/719807'
}
},
}
],
u'ad_count':17
}
}

Assuming your data structure is stored in the variable j, you can use j['data']['ad_list'][0] to extract the first item from the ad_list key. Use a try-except block to catch a possible IndexError exception if ad_list can ever be empty.

Return particular string from response

I am trying to return a particular string value after getting response from request URL.
Ex.
response =
{
'assets': [
{
'VEG': True,
'CONTACT': '12345',
'CLASS': 'SIX',
'ROLLNO': 'A101',
'CITY': 'CHANDI',
}
],
"body": "**Trip**: 2017\r\n** Date**: 15th Jan 2015\r\n**Count**: 501\r\n\r\n"
}
This is the response which i am getting, from this I need only Date: 15th Jan 2015. I am not sure how to do it.
Any help would be appreciated.

assuming it is a dictionary
a={'body': '**Trip**: 2017\r\n** Date**: 15th Jan 2015\r\n**Count**: 501\r\n\r\n', 'assets': [{'VEG': True, 'CONTACT': '12345', 'CLASS': 'SIX', 'ROLLNO': 'A101', 'CITY': 'CHANDI'}]}
then
required=a['body'].split('\r\n')[1].replace('**','')
print required
result:
' Date: 15th Jan 2015'
Access the key body of the dictionary a
split through \r\n to get a list ['**Trip**: 2017', '** Date**:
15th Jan 2015', '**Count**: 501', '', '']
access it's first index and replace ** with empty('')

a = str(yourdictionary)
print([e for e in a.split("\r\n") if "Date" in e][0].remove("**").strip())
Try this

>>> body = response['body']
>>> bodylist = body.split('\r\n')
>>> for value in bodylist:
... value = value.split(':')
...
>>> for i,value in enumerate(bodylist):
... bodylist[i] = value.split(':')
...
>>> for i, value in enumerate(bodylist):
... if bodylist[i][0] == '** Date**':
... print(bodylist[i][1])
...
15th Jan 2015
I have trown it in the interpreter and this works. I don't know if it is the best code around, but it works ;-)