I am trying to get all request of API with rate limit of 100 request per minute. This API request is in json format. I used import urllib, import requests and import json and from time import sleep. My code below returns the same data
import urllib
import requests
import json
import urllib.request as urllib2
from time import sleep
data = []
#resp
for i in range(4):
url = "https://api.leadfeed.com/accounts/*?start_date=2018-01-01&end_date=2020-06-01"
header = {"Authorization": 'Bearer GnHbmjtUEQDXCFOlwKAyy9hJedZtDg'}
resp = requests.get(url, headers = header)
#print(i)
data.append(resp.text)
sleep(60)
print(data)
It returns 4 the same json data as shown below:
[
"{\"data\":[{\"id\":\"b50b0cf4-06f1-11ea_p6q3nDbB2C4m9KjK9gndy\",\"type\.....}"
"{\"data\":[{\"id\":\"b50b0cf4-06f1-11ea_p6q3nDbB2C4m9KjK9gndy\",\"type\.....}"
"{\"data\":[{\"id\":\"b50b0cf4-06f1-11ea_p6q3nDbB2C4m9KjK9gndy\",\"type\.....}"
"{\"data\":[{\"id\":\"b50b0cf4-06f1-11ea_p6q3nDbB2C4m9KjK9gndy\",\"type\.....}"
]
I don't know the length of the json that why i used range(4) to return 4 json data.
How can i get data from API with rate limit request?
Related
I did this code to connect to an API and read the data. (I removed the Cookie and URL)
import requests
import urllib3
import json
import sseclient
import pprint
requests.packages.urllib3.disable_warnings()
requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS += ':HIGH:!DH:!aNULL'
parameter = {
"Cookie" : ""
}
response2 = requests.get("", headers=parameter, verify=False, stream=True)
print(response2.status_code)
for a in response2.iter_lines(chunk_size=1024, decode_unicode=True):
print(a)
It works but the data are not very clean. Example :
data: {"id":"1660609778692","timestamp":1660636693,"heure":35893,"sens":1,"action":"A"}
event: heartbeat
data:
data: {"id":"1660604578692","timestamp":1660636665,"heure":35864,"sens":2,"action":"A"}
Can you tell me how to clean the data ? maybe have a json format.
Is there a good way to register the data in a file ?
If you have advice to improve the code, it would be great.
Thanks.
I need [0] to increase everytime and fetch the data when index change. from 0 to 13
import requests as r
import json
url = "https://services6.arcgis.com/bKYAIlQgwHslVRaK/arcgis/rest/services/CasesByRegion_ViewLayer/FeatureServer/0/query?where=1%3D1&outFields=*&outSR=4326&f=json"
response = urlopen(url)
Data= json.load(response )
for index in Data:
list = Data['features'][0]['attributes']
[0]+1
print(list)
Here is another simple approach without using urllib:
import requests as r
import json
url = "https://jsonplaceholder.typicode.com/todos/1"
response = r.get(url)
data = response.json()
print(data)
requests.get().json() delivers the complete dict from the response payload:
import requests as r
response = r.get(url)
Data = response.json()
Your json.load() doesn't work as expected because response is a dictionary from the requests module, containing some HTTP stuff like status code, reason, encoding. For API calls, this is not what you want (HTTP errors should be handled with exceptions). What you want is response.json() or response.text.
Also, you imported requests but didn't use it? I don't know about urlopen(). Use requests.get().
I am trying to pull the data using get request through the below URL and I'm getting below error.
I'm not understanding what is the issue with the URL.
Any help would be appreciated
import requests
from requests.auth import HTTPBasicAuth
import json
import urllib.parse
url = """https://msi.abc.com/admin/ui/feedbackCSV/reports
/5d14de32309baf0001501fb7 ?reports[]=pageviews&reports[]=
searches&from=Oct 1 2020&to=Oct 2 2020"""
encode_url = urllib.parse.quote(url,encoding='utf-8')
response = requests.get(encode_url,auth = HTTPBasicAuth('admin#abc.com', 'Summer2020'))
print(response.content)
Error
raise MissingSchema(error)
requests.exceptions.MissingSchema: Invalid URL 'https%3A%2F%2Fmsi.abc.com%2Fadmin%2Fui%2FfeedbackCSV%2Freports%0D%0A%2F5d14de32309baf0001501fb7%20%3Freports%5B%5D%3Dpageviews%26reports%5B%5D%3D%0D%0Asearches%26from%3DOct%201%202020%26to%3DOct%202%202020': No schema supplied. Perhaps you meant http:https%3A%2F%2Fmsi.abc.com%2Fadmin%2Fui%2FfeedbackCSV%2Freports%0D%0A%2F5d14de32309baf0001501fb7%20%3Freports%5B%5D%3Dpageviews%26reports%5B%5D%3D%0D%0Asearches%26from%3DOct%201%202020%26to%3DOct%202%202020?
This might be due to that you pass the whole URL to the urllib.parse.quote function. Try passing the params only to the urllib.parse.quote or use requests params like in the below example:
import requests
from requests.auth import HTTPBasicAuth
import json
import urllib.parse
url = "https://msi.abc.com/admin/ui/feedbackCSV/reports/5d14de32309baf0001501fb7"
payload = {'reports[]':'pageviews', 'reports[]':'searches','from':'Oct 1 2020', 'to':'Oct 2 2020' }
authParams = HTTPBasicAuth('admin#abc.com', 'Summer2020')
response = requests.get(url,params=payload, auth =authParams )
print(response.content)
I have the following code using urllib in Python 2.7 and its working. I'm trying to do the same request using the requests library but I cant get it to work.
import urllib
import urllib2
import json
req = urllib2.Request(url='https://testone.limequery.com/index.php/admin/remotecontrol',\
data='{\"method\":\"get_session_key\",\"params\":[\"username\",\"password\"],\"id\":1}')
req.add_header('content-type', 'application/json')
req.add_header('connection', 'Keep-Alive')
f = urllib2.urlopen(req)
myretun = f.read()
j=json.loads(myretun)
print(j['result'])
Using requests library( Doesn't work)
import requests
import json
d= {"method":"get_session_key","params":["username","password"],"id":"1"}
headers = {'content-type' :'application/json','connection': 'Keep-Alive'}
req2 = requests.get(url='https://testone.limequery.com/index.php/admin/remotecontrol',data=d,headers=headers)
json_data = json.loads(req2.text)
print(json data['result'])
I'm getting an error JSONDecodeError: Expecting value: line 1 column 1 (char 0) How can I make the code work with the requests library?
First, you're sending the wrong type of request. You're sending a GET request, but you need to send a POST, with requests.post.
Second, passing a dict as data will form-encode the data rather than JSON-encoding it. If you want to use JSON in your request body, use the json argument, not data:
requests.post(url=..., json=d)
Reference Link: http://docs.python-requests.org/en/master/api/
You can use requests module of python like so
import requests
Req = requests.request(
method = "GET", # or "POST", "PUT", "DELETE", "PATCH" etcetera
url = "http(s)://*",
params = {"key": "value"}, # IF GET Request (Optional)
data = {"key": "value"}, # IF POST Request (Optional)
headers = {"header_name": "header_value"} # (Optional)
)
print Req.content
You can surround the code with try::catch block like below to catch any exception thrown by requests module
try:
# requests.request(** Arguments)
except requests.exceptions.RequestException as e:
print e
For full argument list, please check reference link.
I am trying to send a POST request with a Neo4j transaction query. Although I get a response 200 the node is not created. This is my Python script:
import requests
import json
import csv
headers = {'content-type': 'application/json'}
url = "http://localhost:7474/db/data/transaction/commit"
checkNode = {"query" : '{"statements": [{"statement":"CREATE (n:test) RETURN n"}]}'}
mkr =requests.post(url, data=json.dumps(checkNode), headers=headers)
print(mkr)
I haven't used transactions before and nver tried to create one through the Rest Api. What am I doing wrong here?
It seems unlikely to me that you're receiving a response code of 200; you should be getting a 500 as the transactional endpoint doesn't accept a query parameter. Try this:
import requests
import json
import csv
headers = {'content-type': 'application/json'}
url = "http://localhost:7474/db/data/transaction/commit"
checkNode = {"statements":[{"statement":"CREATE n RETURN n"}]}
mkr = requests.post(url, data=json.dumps(checkNode), headers=headers)
print(mkr.text)