I'm having an issue getting a response from an rest API into a json format with Pandas. Here's my code:
import pandas as pd
import numpy as np
import requests
import json
headers = {
'Authorization': 'Token token="mytoken"',
'Content-Type': 'application/json',
'Cache-Control': 'no-cache--url',
}
response = requests.get('https://www.someurl.com/api/path', headers=headers)
data = response.json()
This is my issue: Whenever I check the data type that I get in response, it is in a dictionary format, like so:
In[2]: type(data)
Out[2]: dict
Because it returns text that's in a JSON format, but it comes into my code as a dictionary, I cannot use panda's .read_json() command because it appears that it's expecting a JSON datatype. Whenever I try to do that, it returns this:
In[3]: pd.read_json(data)
Out[3]: ValueError: Invalid file path or buffer object type: <class 'dict'>
I think the main issue is that my response returns in the form of a dictionary instead of pure JSON, and not because of the syntax within the JSON data itself, but by no means am I an expert in this subject. Let me know what you guys think.
Here's the documentation to the API I'm using:
https://documentation.joinhandshake.com/v1.0/reference#introduction
Any help would be greatly appreciated.
Your request does not return a Python dictionary -- the requests module's .json() method returns a dictionary. If your response is in json and you just want to use the string to load it in to pandas, use response.text.
Related
I need [0] to increase everytime and fetch the data when index change. from 0 to 13
import requests as r
import json
url = "https://services6.arcgis.com/bKYAIlQgwHslVRaK/arcgis/rest/services/CasesByRegion_ViewLayer/FeatureServer/0/query?where=1%3D1&outFields=*&outSR=4326&f=json"
response = urlopen(url)
Data= json.load(response )
for index in Data:
list = Data['features'][0]['attributes']
[0]+1
print(list)
Here is another simple approach without using urllib:
import requests as r
import json
url = "https://jsonplaceholder.typicode.com/todos/1"
response = r.get(url)
data = response.json()
print(data)
requests.get().json() delivers the complete dict from the response payload:
import requests as r
response = r.get(url)
Data = response.json()
Your json.load() doesn't work as expected because response is a dictionary from the requests module, containing some HTTP stuff like status code, reason, encoding. For API calls, this is not what you want (HTTP errors should be handled with exceptions). What you want is response.json() or response.text.
Also, you imported requests but didn't use it? I don't know about urlopen(). Use requests.get().
new to coding so apologies for the basic q.
I am working with an API for a cyber security tool - Virus Total. I am trying to write a program that will call the API to get the report of the IP address, and then save that report to a file. I would like each API call to be saved in a separate file with a different name (with the format 'report[number of report]-[DDMMYYYY].txt'
I have tried to accomplish this with the open and write commands, but I am getting error: TypeError: write() argument must be str, not bytes
I have successfully got the API response but I do not know how to save it to a file with an automatically changing filename.
Any ideas?
I will post my code below (with my API key redacted).
Thanks
url = "https://www.virustotal.com/api/v3/ip_addresses/192.169.69.25"
headers = {
"Accept": "application/json",
"x-apikey": "REDACTED"
}
response = requests.request("GET", url, headers=headers)
with open("testoutput1.txt", "w") as f:
f.write(response)
This is very late, so you likely already have this solved, but for future searches, you likely want to save the json response (f.write(response.json())) or raw text (f.write(response.text)) rather than the response directly, which is what the TypeError: write() argument must be str, not bytes is indicating.
Here is an example with minor changes to use pathlib and format the filename based on your request:
import json
from datetime import datetime
from pathlib import Path
import requests
url = "https://catfact.ninja/fact"
headers = {
"Accept": "application/json",
}
response = requests.request("GET", url, headers=headers)
# # As a tip, breakpoint can be very helpful when debugging. Try:
# breakpoint()
idx = 1 # Hypothetical index from a for-loop
date = datetime.now().strftime("%d%m%Y")
Path(f"report{idx}-{date}.json").write_text(json.dumps(response.json()))
As a side-note, with Python 3.10, you can now get a much more helpful error message: TypeError: write() argument must be str, not Response
I wrote a simple script to check if a URL return me a JSON.
I'm using request to check if return 200 and is working.
But now i need to check if the URL return me a JSON. I don't need to open it. Just check if is a JSON or not, because even if the URL returns me a 200 doesn't mean that is JSON file I need.
How can I check if result.json() is true? I tried to check len but if the site don't have JSON my script crashes.
import pandas as pd
from requests import get
lista = pd.read_csv('sites.csv', sep=',')
df = pd.DataFrame(lista, columns=['Site', 'Json'])
newdf = df.assign(Site=df['Site'].map(str) + 'Json')
for i in newdf['Site']:
result = get(i)
result.json()
An option could be to check the response headers, which means you don't need to try and parse the response at all:
'application/json' in result.headers.get('Content-Type')
I'm implemented a code to return Json value like this
{ "key1":"1", "key2":"2", ......}
But I want to get only value of key1. so I first defined info to get Json.
info = requests.get(url, headers=headers)
And use info.text['key1'] to get value of key1. but i got error. could anyone suggest any solution?
JSON is a format, but the data are included as text. Inside the info.text you have a string.
If you want to access the json data you can do info.json()['key1']. This will work only if the response content type is defined as JSON, to check that do
info.headers['content-type'] should be application/json; charset=utf8
http://docs.python-requests.org/en/master/
Otherwise you will have to manually load the text to json, with json library for example
import json
response = requests.get(url, headers=headers)
data = json.loads(response.text)
import json
info = requests.get(url, headers=headers)
jsonObject = json.loads(info)
key1= jsonObject['key1']
or
jsonObject = info.json()
key1= jsonObject['key1']
I am using Python, and I sent a request to a URL and received a reply using httplib2. The reply I got was in JSon, how do I access a specific parameter. What I have at the moment is:
resp, content = parser.request(access_token_uri, method = 'POST', body = params, headers = headers)
raise Exception(content['access_token'])
and I get the error
string indices must be integers, not str
How do I do it?
Thanks
Well if the response type is json and it comes in type str.
If you are running 2.4 of Python use simplejson if 2.6 use json:
import json
# Your code
retdict = json.loads(content)
Then treat it like a dictionary.
accesstoken = retdict['access_token']
You can use dump;
result = json.dumps(content)
result = json.loads(result)