I'm working on a project that needs to access an API that returns a JSON with the data from a checklist that the user has completed, it contains all the data, but the user can also evaluate some points such as: choosing several options or an option, answering a text, answer a number, etc. These assessments come in another endpoint by passing the assessment ID.
So, I created the following code. It takes the result of the "general" endpoint and takes only the ID to be used as an index to use as a parameter in the endpoint link.
evaluations_array = df_evaluations_results.evaluationId.unique()
df_itens = pd.DataFrame()
df = pd.DataFrame()
for index in evaluations_array:
response = requests. Get(f'https://.../{index}/results', headers=headers)
texto = response.text
data = json.loads(texto)
df = pd.json_normalize(data['data'])
df_itens = df_itens.append(df, ignore_index=True)
df_itens
It returns the following error:
TypeError: list indices must be integers or slices, not str
The type of JSON that is returned is in the following format:
[
{
'resultId': <value>,
'evaluationId': <value>,
'categoryId': <value>,
'itemId': <value>,
'scaleId': 1010,
'answeredAt': <value>,
'evaluative': None,
'text': None,
'number': None,
'stateId': None,
'cityId': None,
'product': None,
'competencePeriodId': None,
'selectedOptions': [
{'optionId': <value>, 'optionText': 'text', 'optionValue': None},
{'optionId': <value>, 'optionText': 'text', 'optionValue': None},
{'optionId': <value>, 'optionText': 'text', 'optionValue': None}
],
'index': None,
'originalWeight': <value>,
'maximumWeight': 0,
'obtainedWeight': 0,
'comment': '',
'countAttachments': 0,
'countSignatures': 0
}
]
I think the problem may be in the 'selectedOptions' field which is multivalued and from what I'm understanding, it's understanding the keys of 'selectedOptions' as indices.
What I can't understand is that when I return small data records it doesn't return errors. When it returns a large number of records, it generates this type of error.
Please try the code below. The data you get error will be added to the error_json list. Add the error data (if there is any private data, please replace it) to the question. This way we can fix the problem.
evaluations_array = df_evaluations_results.evaluationId.unique()
df_itens = pd.DataFrame()
df = pd.DataFrame()
error_json=[] # we will add the jsons that we got an error to this list
for index in evaluations_array:
response = requests. Get(f'https://.../{index}/results', headers=headers)
texto = response.text
data = json.loads(texto)
try: # if there is no error, continue
df = pd.json_normalize(data['data'])
df_itens = df_itens.append(df, ignore_index=True)
except: #If we got an error in the json normalize function, let's add this data to the list.
error_json.append(data)
df_itens
I wrote a code that takes 9 keys from API.
The authors, isbn_one, isbn_two, thumbinail, page_count fields may not always be retrievable, and if any of them are missing, I would like it to be None. Unfortunately, if, or even nested, doesn't work. Because that leads to a lot of loops. I also tried try and except KeyError etc. because each key has a different error and it is not known which to assign none to. Here is an example of logic when a photo is missing:
th = result['volumeInfo'].get('imageLinks')
if th is not None:
book_exists_thumbinail = {
'thumbinail': result['volumeInfo']['imageLinks']['thumbnail']
}
dnew = {**book_data, **book_exists_thumbinail}
book_import.append(dnew)
else:
book_exists_thumbinail_n = {
'thumbinail': None
}
dnew_none = {**book_data, **book_exists_thumbinail_n}
book_import.append(dnew_none)
When I use logic, you know when one condition is met, e.g. for thumbinail, the rest is not even checked.
When I use try and except, it's similar. There's also an ISBN in the keys, but there's a list in the dictionary over there, and I need to use something like this:
isbn_zer = result['volumeInfo']['industryIdentifiers']
dic = collections.defaultdict(list)
for d in isbn_zer:
for k, v in d.items():
dic[k].append(v)
Output data: [{'type': 'ISBN_10', 'identifier': '8320717507'}, {'type': 'ISBN_13', 'identifier': '9788320717501'}]
I don't know what to use anymore to check each key separately and in the case of its absence or lack of one ISBN (identifier) assign the value None. I have already tried many ideas.
The rest of the code:
book_import = []
if request.method == 'POST':
filter_ch = BookFilterForm(request.POST)
if filter_ch.is_valid():
cd = filter_ch.cleaned_data
filter_choice = cd['choose_v']
filter_search = cd['search']
search_url = "https://www.googleapis.com/books/v1/volumes?"
params = {
'q': '{}{}'.format(filter_choice, filter_search),
'key': settings.BOOK_DATA_API_KEY,
'maxResults': 2,
'printType': 'books'
}
r = requests.get(search_url, params=params)
results = r.json()['items']
for result in results:
book_data = {
'title': result['volumeInfo']['title'],
'authors': result['volumeInfo']['authors'][0],
'publish_date': result['volumeInfo']['publishedDate'],
'isbn_one': result['volumeInfo']['industryIdentifiers'][0]['identifier'],
'isbn_two': result['volumeInfo']['industryIdentifiers'][1]['identifier'],
'page_count': result['volumeInfo']['pageCount'],
'thumbnail': result['volumeInfo']['imageLinks']['thumbnail'],
'country': result['saleInfo']['country']
}
book_import.append(book_data)
else:
filter_ch = BookFilterForm()
return render(request, "BookApp/book_import.html", {'book_import': book_import,
'filter_ch': filter_ch})```
How can i get needed value, because i send post request to other site and cant edit answer from site.
I have this dict from responded content:
{'username': 'DeadFinder', 'subscriptions': [{'subscription': 'default', 'expiry': '1635683460'}], 'ip': 'not at this life'}
How you can see in this dict there is a key subscriptions, i'm need value expiry(this is timestamp) but how can i get this value if when i'm trying to call this value i'm not see any results (code not gives needed value), maybe any variants how to get this value? I'm not finded anything like this.
Maybe my small part of code can smally help you but i doubt.
data1 = {f"hwid":"", "type":"login", "username": {username}, "pass": {password},
"sessionid":f"{response_cut2}", "name":"test_app", "ownerid":"5OLbm5S3fS"}
url1 = "nope"
response1 = requests.post(url1, data1)
data = response1.json()
#get = data.get('expiry')
file_write = open("test.txt", "w")
file_write.write(str(data))
file_write.close()
for key in data.keys():
if key == 'info':
print (data[key])
Are you trying to achieve this as result ?
data = {'username': 'DeadFinder', 'subscriptions': [{'subscription': 'default', 'expiry': '1635683460'}], 'ip': 'not at this life'}
print(data['subscriptions'][0]['expiry'])
# first get 'subscriptions' which returns an array,
# so use [0] to get this dict {'subscription': 'default', 'expiry': '1635683460'}
# then get 'expiry'
EDIT : In case subscriptions has multiple values then use for loop
subscriptions = data['subscriptions']
for subscription in subscriptions:
print(subscription['expiry'])
Output
1635683460
I am trying to run this code where data of a dictionary is saved in a separate csv file.
Here is the dict:
body = {
'dont-ask-for-email': 0,
'action': 'submit_user_review',
'post_id': 76196,
'email': email_random(),
'subscribe': 1,
'previous_hosting_id': prev_hosting_comp_random(),
'fb_token': '',
'title': review_title_random(),
'summary': summary_random(),
'score_pricing': star_random(),
'score_userfriendly': star_random(),
'score_support': star_random(),
'score_features': star_random(),
'hosting_type': hosting_type_random(),
'author': name_random(),
'social_link': '',
'site': '',
'screenshot[image][]': '',
'screenshot[description][]': '',
'user_data_process_agreement': 1,
'user_email_popup': '',
'subscribe_popup': 1,
'email_asked': 1
}
Now this is the code to write in a CSV file and finally save it:
columns = []
rows = []
chunks = body.split('}')
for chunk in chunks:
row = []
if len(chunk)>1:
entry = chunk.replace('{','').strip().split(',')
for e in entry:
item = e.strip().split(':')
if len(item)==2:
row.append(item[1])
if chunks.index(chunk)==0:
columns.append(item[0])
rows.append(row)
df = pd.DataFrame(rows, columns = columns)
df.head()
df.to_csv ('r3edata.csv', index = False, header = True)
but this is the error I get:
Traceback (most recent call last):
File "codeOffshoreupdated.py", line 125, in <module>
chunks = body.split('}')
AttributeError: 'dict' object has no attribute 'split'
I know that dict has no attribute named split but how do I fix it?
Edit:
format of the CSV I want:
dont-ask-for-email, action, post_id, email, subscribe, previous_hosting_id, fb_token, title, summary, score_pricing, score_userfriendly, score_support, score_features, hosting_type,author, social_link, site, screenshot[image][],screenshot[description][],user_data_process_agreement,user_email_popup,subscribe_popup,email_asked
0,'submit_user_review',76196,email_random(),1,prev_hosting_comp_random(),,review_title_random(),summary_random(),star_random(),star_random(),star_random(),star_random(),hosting_type_random(),name_random(),,,,,1,,1,1
Note: all these functions mentioned are return values
Edit2:
I am picking emails from the email_random() function like this:
def email_random():
with open('emaillist.txt') as emails:
read_emails = csv.reader(emails, delimiter = '\n')
return random.choice(list(read_emails))[0]
and the emaillist.txt is like this:
xyz#gmail.com
xya#gmail.com
xyb#gmail.com
xyc#gmail.com
xyd#gmail.com
other functions are also picking the data from the files like this too.
Since body is a dictionary, you don't have to a any manual parsing to get it into a CSV format.
If you want the function calls (like email_random()) to be written into the CSV as such, you need to wrap them into quotes (as I have done below). If you want them to resolve as function calls and write the results, you can keep them as they are.
import csv
def email_random():
return "john#example.com"
body = {
'dont-ask-for-email': 0,
'action': 'submit_user_review',
'post_id': 76196,
'email': email_random(),
'subscribe': 1,
'previous_hosting_id': "prev_hosting_comp_random()",
'fb_token': '',
'title': "review_title_random()",
'summary': "summary_random()",
'score_pricing': "star_random()",
'score_userfriendly': "star_random()",
'score_support': "star_random()",
'score_features': "star_random()",
'hosting_type': "hosting_type_random()",
'author': "name_random()",
'social_link': '',
'site': '',
'screenshot[image][]': '',
'screenshot[description][]': '',
'user_data_process_agreement': 1,
'user_email_popup': '',
'subscribe_popup': 1,
'email_asked': 1
}
with open('example.csv', 'w') as fhandle:
writer = csv.writer(fhandle)
items = body.items()
writer.writerow([key for key, value in items])
writer.writerow([value for key, value in items])
What we do here is:
with open('example.csv', 'w') as fhandle:
this opens a new file (named example.csv) with writing permissions ('w') and stores the reference into variable fhandle. If using with is not familiar to you, you can learn more about them from this PEP.
body.items() will return an iterable of tuples (this is done to guarantee dictionary items are returned in the same order). The output of this will look like [('dont-ask-for-email', 0), ('action', 'submit_user_review'), ...].
We can then write first all the keys using a list comprehension and to the next row, we write all the values.
This results in
dont-ask-for-email,action,post_id,email,subscribe,previous_hosting_id,fb_token,title,summary,score_pricing,score_userfriendly,score_support,score_features,hosting_type,author,social_link,site,screenshot[image][],screenshot[description][],user_data_process_agreement,user_email_popup,subscribe_popup,email_asked
0,submit_user_review,76196,john#example.com,1,prev_hosting_comp_random(),,review_title_random(),summary_random(),star_random(),star_random(),star_random(),star_random(),hosting_type_random(),name_random(),,,,,1,,1,1
I want to save an array of objects passed from javascript through ajax to me database. This is my view code:
data2 = json.loads(request.raw_get_data)
for i in data2:
print(key)
obj = ShoppingCart(quantity = i.quantity , user_id = 3, datetime = datetime.now(), product_id = i.pk)
obj.save()
return render_to_response("HTML.html",RequestContext(request))
After the first line, i get this in my dictionary:
[{'model': 'Phase_2.product', 'fields': {'name': 'Bata', 'category': 2, 'quantity': 1, 'subcategory': 1, 'count': 2, 'price': 50}, 'imageSource': None, 'pk': 1}]
(Only one object in the array right now)
I want to be able access individual fields like quantity, id, etc in order to save the data to my database. When i debug this code, it gives a name error on 'i'. I also tried accessing the fields like this: data2[0].quantity but it gives this error: {AttributeError}dict object has no attribute quantity.
Edited code:
for i in data2:
name = i["fields"]["name"]
obj = ShoppingCart(quantity = i["fields"]["quantity"] , user_id = 3, datetime = datetime.now(), product_id = i["fields"]["pk"])
obj.save()
It might help you to visualise the returned dict with proper formatting:
[
{
'model': 'Phase_2.product',
'fields': {
'name': 'Bata',
'category': 2,
'quantity': 1,
'subcategory': 1,
'count': 2,
'price': 50
},
'imageSource': None,
'pk': 1
}
]
The most likely reason for your error is that you are trying to access values of of the inner 'fields' dictionary as if they belong to the outer i dictionary.
i.e.
# Incorrect
i["quantity"]
# Gives KeyError
# Correct
i["fields"]["quantity"]
Edit
You have the same problem in your update:
# Incorrect
i["fields"]["pk"]
# Correct
i["pk"]
The "pk" field is in the outer dictionary, not the inner "fields" dictionary.
You may try:
i['fields']['quantity']
The json.loads() returns you a dictionary, which should be accessed by key.