Develop Reference

Import api url data more than 10 000 rows with Python

i try to import an json url (api), the file have "nhits":20843
So this is my url : https://opendata.reseaux-energies.fr/api/records/1.0/search/?dataset=injections-regionales-quotidiennes-consolidees-rpt&q=&rows=20843&facet=date&facet=region&facet=filiere&facet=plage_de_puissance
This is my code :
import requests
site = "https://opendata.reseaux-energies.fr/api/records/1.0/search/?dataset=injections-regionales-quotidiennes-consolidees-rpt&q=&rows=20843&facet=date&facet=region&facet=filiere&facet=plage_de_puissance"
r = requests.get(site)
data = r.json()
And i have a error message :
'raw_params': {'expected': '-1 <= rows <= 10000', 'field_value': 20843, 'field_name': 'rows'}, 'raw_message': 'Invalid field in API request: {field_name} with value {field_value}. Expected: {expected}', 'error_key': 'InvalidFieldInAPIRequestExpectedException'}
How can I avoid the error
ps: sorry for my bad english

You are requesting to many resuklts at once. The error says that you may only request 10K reusults at once. So you need to split your request into multiple parts if you want to get every single result.
For my code i reduced the results per request to 1000 so that the request does not time out. Then for each subsequent request I increment the start-parameter of you api by adding start=" + str(i) + "& to the url. See Documentation https://help.opendatasoft.com/apis/ods-search-v1/#available-apis
import requests
import math
datas = []
for i in range(math.ceil(20843/1000)):
site = "https://opendata.reseaux-energies.fr/api/records/1.0/search/?dataset=injections-regionales-quotidiennes-consolidees-rpt&q=&rows=1000&start="+str(i*1000)+"&facet=date&facet=region&facet=filiere&facet=plage_de_puissance"
r = requests.get(site)
datas.append(r.json())

I am unable to connect with the pwnedpasswords API from python

I am unable to connect with the pwnedpasswords API when I define a function and I get error "400", which is a bad request. I am using the following code:
import requests
def request_api_data(char):
url = 'https://api.pwnedpasswords.com/range/' + char
res = requests.get(url)
print(res)
request_api_data('123')
However, when I use this code without function then I get a response of "200" which is a good response.
import requests
url = 'https://api.pwnedpasswords.com/range/' + 'CBFDA'
res = requests.get(url)
print(res)
I don't understand why it becomes a bad request when I try to make it a function.

The problem is with the 123, if you try CBFDA in the first code it will work, it gives this error with 123:
"The hash prefix was not in a valid format"
This happens because 123 is not a valid hash prefix

Django http request to api error

As this is the first time I'm trying this out, I do not know what is wrong with the problem. So it would be great if someone can help me solve this problem
The code I'm using is at the bottom page of this website: https://www.twilio.com/blog/2014/11/build-your-own-pokedex-with-django-mms-and-pokeapi.html
Where it give example on how you can make HTTP request function and retrieve database on your query.
The code on the website is this.
query.py
import requests
import json
BASE_URL = 'http://pokeapi.co'
def query_pokeapi(resource_url):
url = '{0}{1}'.format(BASE_URL, resource_url)
response = requests.get(url)
if response.status_code == 200:
return json.loads(response.text)
return None
charizard = query_pokeapi('/api/v1/pokemon/charizard/')
sprite_uri = charizard['sprites'][0]['resource_uri']
description_uri = charizard['descriptions'][0]['resource_uri']
sprite = query_pokeapi(sprite_uri)
description = query_pokeapi(description_uri)
print
charizard['name']
print
description['description']
print
BASE_URL + sprite['image']
In my edit, I only change these print line at the bottom of this
query.py
print(charizard['name'])
print(description['description'])
print(BASE_URL + sprite['image'])
But i got this error instead
Traceback (most recent call last): File "query2.py", line 46, in
sprite_uri = charizard['sprites'][0]['resource_uri'] TypeError: 'NoneType' object is not subscriptable

query_pokeapi must be returning None, which would mean that your API call is not receiving a 200 HTTP response. I'd check your URL, to make sure it's properly formed. Test it in your web browser.
best practice would be to try-except your API call with an error message letting you know that your API call failed and otherwise routing the thread.
Update: reread and the sub scripting issue could be in any layer of your nested object.
Evaluate charizard['sprites'][0]['resource_uri']
step by step in your debugger.

When you call api requests.get(url) then its response is
More than one resource is found at this URI
you are using charizard['sprites'][0]['resource_uri'] on result and it's raising exception.
When I tried to get response then status code is 300 so
def query_pokeapi(resource_url) returning None value.
'{0}{1}'.format(BASE_URL, resource_url)
Update
it means at {0} BASE_URL will be places and at {1} resource_url will be places.
Complete url will be
url = '{0}{1}'.format(BASE_URL, resource_url)
url = 'http://pokeapi.co/api/v1/pokemon/charizard/'.
update
you can try
import json
charizard = query_pokeapi('/api/v1/pokemon/')
data = json.loads(charizard.content)
print data['objects'][0]['descriptions'][0]
result will be
{u'name': u'ekans_gen_1', u'resource_uri': u'/api/v1/description/353/'}
Update with complete code
import requests
import json
BASE_URL = 'http://pokeapi.co'
def query_pokeapi(resource_url):
url = '{0}{1}'.format(BASE_URL, resource_url)
response = requests.get(url)
if response.status_code == 200:
return json.loads(response.text)
return None
charizard = query_pokeapi('/api/v1/pokemon/')
print charizard['objects'][0]['descriptions'][0]
result will be:
{u'name': u'ekans_gen_1', u'resource_uri': u'/api/v1/description/353/'}

Reading website data automatically with POST requests using Python

I am trying to automatically read data from a website where first I need to fill in some fields, submit the form and then read the data that appears. I am new to this but I wrote a code which obviously doesn't work and the result is HTTP Error 500. What am I missing here? or How do I fix this?
Also, I am happy to do this using BS4 as well because I will need to build upon this code.
Website: http://www.mlindex.ml.com/GISPublic/bin/SnapShot.asp
Inputs required: Index Ticker = H0A0 , Base Curr = LOC , Date = 09/22/2017
I checked the source code and went through the js form that submits the POST request and created the code and payload accordingly:
import requests
post_data = {'hdnDate':'1/1/2016', 'hdnAction':'SS', 'hdnSelCurr':'0,LOC', 'hdnCurrDesc':'USD', 'hdnSelTitle':'Hedged', 'txtSSCUSIP':'H0A0'}
# POST some form-encoded data:
post_response = requests.post(url='http://www.mlindex.ml.com/GISPublic/bin/Snapshot.asp', data=post_data)
print post_response

You are missing 'cboSnapCurr': 0, 'cboSSHedge' : 1 from the payload data, as the server that handles the request is expecting those values.
post_data = {'hdnDate':'1/1/2016', 'hdnAction':'SS', 'hdnSelCurr':'0,LOC', 'hdnCurrDesc':'USD', 'hdnSelTitle':'Hedged', 'txtSSCUSIP':'H0A0', 'cboSnapCurr': 0, 'cboSSHedge' : 1}

requests.post with Python

I'm connecting to a login protected API with a Python script here below.
import requests
url = 'https://api.json'
header = {'Content-Type': 'application/x-www-form-urlencoded'}
login = ('kjji#snm.com', 'xxxxx')
mnem = 'inputRequests':'{'inputRequests':'[{'function':'GDSP','identifier':'ibm','mnemonic':'IQ_TOTAL_REV'}]}}
r = requests.post(url, auth=login, data=mnem, headers=header)
print(r.json())
The connection is established but I am getting an error from the API because of the format of the data request.The original format is here below. I cannot find a way to enter this in the mnem here above:
inputRequests={inputRequests:
[
{function:"xxx",identifier:"xxx",mnemonic:"xxx"},
]
}
The error given is
C:\Users\xxx\Desktop>pie.py
File "C:\Users\xxx\Desktop\pie.py", line 6
mnem={'inputRequests':'{'inputRequests':'[{'function':'xxx','identifier':'xx','mnemonic':'xxx'}]}}
^
SyntaxError: invalid syntax
I am unsure on how to proceed from here. I cannot find anything in the requests documentation that points to how to insert several variables in the data field.

The requests module in Python receive protogenic Python dict as the JSON data in post request but not a string. Therefore, you may try to define mnem like this:
mnem = {
'inputRequests':[
{'function':'GDSP',
'identifier':'ibm',
'mnemonic':'IQ_TOTAL_REV'
}
]}

the data parameter should be a dictionary.
therefore to pass the three parameters try using:
mnem = {'function':'GDSP','identifier':'ibm','mnemonic':'IQ_TOTAL_REV'}