I am pretty new to python and I am trying to create a script that will pull data from a ticketing platform.
I got the list of agents and their ids but when I try to pull the data it's giving me this error:
KeyError: 'data'
Is there a way for me to have the parameter "agents": to automatically update using the agent_id list?
Here is the code, I removed the links and the API key for privacy reasons:
import requests
import json
from cgitb import text
from openpyxl import Workbook
import openpyxl
import requests
from datetime import date
from datetime import timedelta
#Agents list
agents_list = ["Agent1", "Agent2", "Agent3"]
agent_id = []
agents_names = []
today = date.today()
yesterday = today - timedelta(days = 1)
start_date = str(yesterday)
end_date = str(yesterday)
def extragere_date_agenti():
url = "https://x.gorgias.com/api/users?limit=100&order_by=name%3Aasc&roles=agent&roles=admin"
headers = {
"accept": "application/json",
"authorization": "Basic"
}
response = requests.get(url, headers=headers)
text_name_id = json.loads(response.text)
for names in text_name_id["data"]:
agent_name = names["firstname"]
agents_id = names["id"]
if agent_name in agents_list:
agents_names.append(agent_name)
agent_id.append(agents_id)
extragere_date_agenti()
def extragere_numere():
url = "https://x.gorgias.com/api/stats/total-messages-sent"
payload = {"filters": {
"period": {
"start_datetime": start_date + "T00:00:00-05:00",
"end_datetime": end_date + "T23:59:59-05:00"
},
"agents": [agent_id], #This is the value that I want to modify
"channels": ["email"]
}}
headers = {
"accept": "application/json",
"content-type": "application/json",
"authorization": "Basic"
}
response = requests.post(url, json=payload, headers=headers)
text_numere = json.loads(response.text)
numere_finale = text_numere["data"]["data"]["value"]
print(numere_finale)
I've tried to do a for loop but it's giving me the same error. Any suggestions?
First, add the condition to check the response status code
Also, add another condition to prevent this type of key error:
if "data" in text_name_id:
Your Error:
KeyError: 'data'
Means that in text_name_id is no Key named "data".
Difficult to tell you how to fix it without any more info...
Are you sure, that request returns a positiv status?? I see no ErrorHandling, if respone.status_code == 200: should be enough to check.
Are you sure that the response json has a Key named "data"? Try this to set a default if key is missing:
text_name_id.get("data", [{"firstname": "error", "id": 0}])
--- Edit ---
Okay, is that the right one I don't see a "id" or "firstname" key. But if it is the right JSON, than you can't iterate over dict like you did in Python.
To do so you would want to do this:
for key, value in text_name_id['data']['data'].items():
...
Related
I have been trying to export offer file using Python for Bol Retailer API
According to the official docs on request an offer file export
I have include all the headers and formats but it throws a 400 Bad Request
400
Bad Request
b'{\n "type" : "https://api.bol.com/problems",\n "title" : "Bad Request",\n "status" : 400,\n "detail" : "The supplied content-type media type is not supported.",\n "host" : "Instance-111",\n "instance" : "https://api.bol.com/retailer/offers/export"\n}'
Here is a minimal example from my code
import base64
import requests
import json
import time
class BolService:
def __init__(self, _id, secret):
self.host = "https://api.bol.com"
self.__header = {
"Accept": "application/vnd.retailer.v7+json",
"Content-Type": "N/A",
"Authorization": "Bearer " + self.__get_token(_id, secret)
}
def __get_token(self, _id, secret) -> str:
creds = (_id + ":" + secret).encode('ascii') #creds : ascii bytes
creds_b64_b = base64.b64encode(creds) #creds : base64 bytes
creds_b64 = creds_b64_b.decode('ascii') #creds : base64 string
header = {
"Authorization":"Basic " + creds_b64
}
link = "https://login.bol.com/token?grant_type=client_credentials"
response = requests.post(link, headers=header)
response_data = json.loads(response.content.decode())
return response_data['access_token']
def get_offer_file(self):
path = f"/retailer/offers/export"
new_header = self.__header.copy()
new_header["format"] = "CSV"
response = requests.post(self.host + path, headers=new_header)
return response
Note: I have also tried changing the "Content-Type" in self.__header to "application/vnd.retailer.v7+json", I have also changed the same to add csv using "application/vnd.retailer.v7+json+csv" or "application/vnd.retailer.v7+csv". I have also tried adding self.__header['Content-Type'] = 'text/csv' but nothing seems to work it keeps on throwing the same Bad Request. I have also tried using the v6 of the API instead of v7 but same issue.
I know this is something that should be dealt with the customer service of Bol but they their service is too pathetic to even give a simple reply. Also as of August 2022 their site which details API issues is down. Maybe if someone with experience can help here.
I don't think I am missing anything here. Please let me know.
So I was able to sucessfully make the POST request.
1st what I did was change the "Content-Type" in self.__header to "application/vnd.retailer.v7+json"
so the header now looks like this
self.__header = {
"Accept": "application/vnd.retailer.v7+json",
"Content-Type": "application/vnd.retailer.v7+json",
"Authorization": "Bearer " + self.__get_token(_id, secret)
}
Since we require the content type in JSON format so we have to include a JSON body by dumping our dictionary content using json.dumps
So the get_offer_file method now looks like with {"format":"CSV"} as the body
def get_offer_file(self):
path = f"/retailer/offers/export"
response = requests.post(self.host + path, headers=self.__header, data=json.dumps({"format":"CSV"}))
return response
Here is the full code:
import base64
import requests
import json
class BolService:
def __init__(self, _id, secret):
self.host = "https://api.bol.com"
self.__header = {
"Accept": "application/vnd.retailer.v7+json",
"Content-Type": "application/vnd.retailer.v7+json",
"Authorization": "Bearer " + self.__get_token(_id, secret)
}
def __get_token(self, _id, secret) -> str:
creds = (_id + ":" + secret).encode('ascii') #creds : ascii bytes
creds_b64_b = base64.b64encode(creds) #creds : base64 bytes
creds_b64 = creds_b64_b.decode('ascii') #creds : base64 string
header = {
"Authorization":"Basic " + creds_b64
}
link = "https://login.bol.com/token?grant_type=client_credentials"
response = requests.post(link, headers=header)
response_data = json.loads(response.content.decode())
return response_data['access_token']
def get_offer_file(self):
path = f"/retailer/offers/export"
response = requests.post(self.host + path, headers=self.__header, data=json.dumps({"format":"CSV"}))
return response
I am trying to get a python script to say whether a twitch channel is live but haven't been able to do it, any and all help would be appreciated.
here are the docs I've been able to find
https://dev.twitch.tv/docs/api/guide
This is what I have atm but I keep on getting "'set' object has no attribute 'items'". This is modified code from "Is There Any Way To Check if a Twitch Stream Is Live Using Python?" however it is now outdated because of the new API.
import requests
def checkUser():
API_HEADERS = {
'Client-ID : [client id here from dev portal]',
'Accept : application/vnd.twitchtv.v5+json',
}
url = "https://api.twitch.tv/helix/streams/[streamer here]"
req = requests.Session().get(url, headers=API_HEADERS)
jsondata = req.json()
print(jsondata)
checkUser()
The answer to your problem of "'set' object has no attribute 'items'" is just a simple typo. It should be
API_HEADERS = {
'Client-ID' : '[client id here from dev portal]',
'Accept' : 'application/vnd.twitchtv.v5+json'
}
Notice how the Colon's aren't part of the text now
And to answer your overarching question of how to tell if a channel is online you can look at this sample code I made.
import requests
URL = 'https://api.twitch.tv/helix/streams?user_login=[Channel_Name_Here]'
authURL = 'https://id.twitch.tv/oauth2/token'
Client_ID = [Your_client_ID]
Secret = [Your Client_Secret]
AutParams = {'client_id': Client_ID,
'client_secret': Secret,
'grant_type': 'client_credentials'
}
def Check():
AutCall = requests.post(url=authURL, params=AutParams)
access_token = AutCall.json()['access_token']
head = {
'Client-ID' : Client_ID,
'Authorization' : "Bearer " + access_token
}
r = requests.get(URL, headers = head).json()['data']
if r:
r = r[0]
if r['type'] == 'live':
return True
else:
return False
else:
return False
print(Check())
I have been struggling to send a signed request to binance future using signature.
I found that example code on StackOverflow ("Binance API call with SHA56 and Python requests") and an answer has been given to it mentioning to use hmac
as below: but unfortunately i still don't see how to write this example. Could anyone show how the code of this example should look like? i am really uncomfortable with signed request. Thanks a lot for your understanding and your help advice given:
params = urlencode({
"signature" : hashedsig,
"timestamp" : servertimeint,
})
hashedsig = hmac.new(secret.encode('utf-8'), params.encode('utf-8'), hashlib.sha256).hexdigest()
Original example:
import requests, json, time, hashlib
apikey = "myactualapikey"
secret = "myrealsecret"
test = requests.get("https://api.binance.com/api/v1/ping")
servertime = requests.get("https://api.binance.com/api/v1/time")
servertimeobject = json.loads(servertime.text)
servertimeint = servertimeobject['serverTime']
hashedsig = hashlib.sha256(secret)
userdata = requests.get("https://api.binance.com/api/v3/account",
params = {
"signature" : hashedsig,
"timestamp" : servertimeint,
},
headers = {
"X-MBX-APIKEY" : apikey,
}
)
print(userdata)
The proper way would be:
apikey = "myKey"
secret = "mySecret"
servertime = requests.get("https://api.binance.com/api/v1/time")
servertimeobject = json.loads(servertime.text)
servertimeint = servertimeobject['serverTime']
params = urlencode({
"timestamp" : servertimeint,
})
hashedsig = hmac.new(secret.encode('utf-8'), params.encode('utf-8'),
hashlib.sha256).hexdigest()
userdata = requests.get("https://api.binance.com/api/v3/account",
params = {
"timestamp" : servertimeint,
"signature" : hashedsig,
},
headers = {
"X-MBX-APIKEY" : apikey,
}
)
print(userdata)
print(userdata.text)
Make sure to put the signature as the last parameter or the request will return [400]...
Incorrect:
params = {
"signature" : hashedsig,
"timestamp" : servertimeint,
}
Correct:
params = {
"timestamp" : servertimeint,
"signature" : hashedsig,
}
At the time of writing, Binance themselves are mainting a repo with some examples*, using the requests library. Here is a sample in case the link goes down or is moved:
import hmac
import time
import hashlib
import requests
from urllib.parse import urlencode
KEY = ''
SECRET = ''
# BASE_URL = 'https://fapi.binance.com' # production base url
BASE_URL = 'https://testnet.binancefuture.com' # testnet base url
''' ====== begin of functions, you don't need to touch ====== '''
def hashing(query_string):
return hmac.new(SECRET.encode('utf-8'), query_string.encode('utf-8'), hashlib.sha256).hexdigest()
def get_timestamp():
return int(time.time() * 1000)
def dispatch_request(http_method):
session = requests.Session()
session.headers.update({
'Content-Type': 'application/json;charset=utf-8',
'X-MBX-APIKEY': KEY
})
return {
'GET': session.get,
'DELETE': session.delete,
'PUT': session.put,
'POST': session.post,
}.get(http_method, 'GET')
# used for sending request requires the signature
def send_signed_request(http_method, url_path, payload={}):
query_string = urlencode(payload)
# replace single quote to double quote
query_string = query_string.replace('%27', '%22')
if query_string:
query_string = "{}×tamp={}".format(query_string, get_timestamp())
else:
query_string = 'timestamp={}'.format(get_timestamp())
url = BASE_URL + url_path + '?' + query_string + '&signature=' + hashing(query_string)
print("{} {}".format(http_method, url))
params = {'url': url, 'params': {}}
response = dispatch_request(http_method)(**params)
return response.json()
# used for sending public data request
def send_public_request(url_path, payload={}):
query_string = urlencode(payload, True)
url = BASE_URL + url_path
if query_string:
url = url + '?' + query_string
print("{}".format(url))
response = dispatch_request('GET')(url=url)
return response.json()
response = send_signed_request('POST', '/fapi/v1/order', params)
print(response)
Some additional thoughts from myself:
You can also use a new library also from Binance called Binance connector. It is a bit new, it has some issues, but it can do the basic operations without you worrying about signed requests.
I wouldn't use serverTime because that means you need to make an additional request and networks can be slow, I'd follow this example and use the int(time.time() * 1000) you may not even need the function.
I purposedly used the POST example, because this is more complicated as you need to also encode and hash your custom parameters
At the time of writing, v3 is the latest version
Hope it helps.
* https://github.com/binance/binance-signature-examples/blob/master/python/futures.py
I have dynamic API URL using the which each URL is getting data in response as JSON which is as following.
{
"#type":"connection",
"id":"001ZOZ0B00000000006Z",
"orgId":"001ZOZ",
"name":"WWW3",
"description":"Test connection2",
"createTime":"2018-07-20T18:28:05.000Z",
"updateTime":"2018-07-20T18:28:53.000Z",
"createdBy":"xx.xx#xx.com.dev",
"updatedBy":"xx.xx#xx.com.dev",
"agentId":"001ZOZ08000000000007",
"runtimeEnvironmentId":"001ZOZ25000000000007",
"instanceName":"ShareConsumer",
"shortDescription":"Test connection2",
"type":"TOOLKIT",
"port":0,
"majorUpdateTime":"2018-07-20T18:28:05.000Z",
"timeout":60,
"connParams":{
"WSDL URL":"https://xxxservices1.work.com/xxx/service/xxport2/n5/Integration%20System/API__Data?wsdl",
"Must Understand":"true",
"DOMAIN":"n5",
"agentId":"001ZOZ0800XXX0007",
"agentGroupId":"001ZOZ25000XXX0007",
"AUTHENTICATION_TYPE":"Auto",
"HTTP Password":"********",
"Encrypt password":"false",
"orgId":"001Z9Z",
"PRIVATE_KEY_FILE":"",
"KEY_FILE_TYPE":"PEM",
"mode":"UPDATE",
"CERTIFICATE_FILE_PASSWORD":null,
"CERTIFICATE_FILE":null,
"TRUST_CERTIFICATES_FILE":null,
"Username":"xxx#xxx",
"CERTIFICATE_FILE_TYPE":"PEM",
"KEY_PASSWORD":null,
"TIMEOUT":"60",
"Endpoint URL":"https://wxxservices1.xx.com/xxx/service/xxport2/n5/Integration%20System/API__Data",
"connectionTypes":"NOAUTH",
"HTTP Username":"API#n5",
"Password":"********"
}
}
Now catch over here is i have close around 50 URLs which gives this type JSON data. I am iterating it using the following code but i am not able to store in Python pandas dataframe as each response from each URL.
It will be either last response only stored there.
I would also like to convert this whole dataframe to CSV.
What is best method to append response of each result of URL response to dataframe and then convert to CSV?
Python Code as following:
import requests
from urllib.request import Request, urlopen
from urllib.request import urlopen, URLError, HTTPError
import urllib.error
import json
import pandas as pd
from pandas.io.json import json_normalize
import os
import csv
#This CSV file where we are getting ID and iterating over it for each url for get JSON data for the each URL
ConnID_data_read=pd.read_csv('ConnID.csv', delimiter = ',')
df = pd.DataFrame(ConnID_data_read)
user_iics_loginURL='https://xx-us.xxx.com/ma/api/v2/user/login'
headers = {
'Content-Type': "application/json",
'Accept': "application/json",
'cache-control': "no-cache"
}
payload = "{\r\n\"#type\": \"login\",\r\n\"username\": \"xx#xx.com.xx\",\r\n\"password\": \"xxxx\"\r\n}"
response = requests.request("POST", user_iics_loginURL, data=payload, headers=headers)
resp_obj = json.loads(response.text)
session_id = resp_obj['SessionId']
server_URL = resp_obj['serverUrl']
print(session_id)
Finaldf = pd.DataFrame()
for index, row in df.iterrows():
api_ver="/api/v2/connection/"+row['id']
#https://xx-us.xxx.com/saas/api/v2/connection/001ZOZ0B000000000066
conndetails_url = server_URL+api_ver
print(conndetails_url)
act_headers = {
'icSessionId': session_id,
'Content-Type': "application/json",
'cache-control': "no-cache",
}
act_response = requests.get(conndetails_url.strip(),headers=act_headers)
print(act_response.text)
print("Creating Data Frame on this***********************")
act_json_data= json.loads(act_response.text)
flat_json = json_normalize(act_json_data)
print(flat_json)
Conndf = pd.DataFrame(flat_json)
Finaldf.append(Conndf)
Finaldf.to_csv('NewTest.csv')
first thing I notice is:
flat_json = json_normalize(act_json_data)
print(flat_json)
Conndf = pd.DataFrame(flat_json)
when you do flat_json = json_normalize(act_json_data), flat_json is already a dataframe. Doing Conndf = pd.DataFrame(flat_json) is unnecessary and redundant, although shouldn't cause a problem, it's just extra code you don't need.
Secondly here's the issue. When you append the dataframe, you need to set it equal to itself. So change:
Finaldf.append(Conndf)
to
Finaldf = Finaldf.append(Conndf)
I'd also just rest the index, as that's just a habit of mine when I append dataframes:
Finaldf = Finaldf.append(Conndf).reset_index(drop=True)
Other than that 1 line, it looks fine and you should get the full dataframe saved to csv with Finaldf.to_csv('NewTest.csv')
I need to use Python to do a POST request using JSON format. What I have right now is
url = 'http://mysurl.org'
data = {my data }
headers = {'content-type': 'application/json'}
r = requests.post(url,data= json.dumps(data, headers=headers)
The issue come when my data is not one line but 500 lines of :
[
{
"Id" : "abc123",
"usr": "u1",
"pwd" : "p1"
},
{
"Id" : "abc124",
"usr": "u2",
"pwd" : "p2"
},
{
"Id" : "abc125",
"usr": "u3",
"pwd" : "p3"
}
.......
]
This really threw me off because "Id" field come from a random generater: id = gennum()
usr is from a query: usr = sqlout[0][0], and pwd is from pwd = sqlout[0][1].
I really do not have an idea how to read 500 line of data into my file data=....
I try to use data.append but do not know how to continue after that.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[update] sorry that question is not specific. my data comes from three different area:
(1) id row come from an random number generator: gennum()
(2) from query my database. sqlout variable will have 500 lines of out put wiht :
user, and pwd. so basically user = sqlout[0][0], and pwd will = sqlout[0][1] and they need to be in the post request body all together, in one request. so when I send the post request, my request bodywill contain 500 entries of json data like stated below. Hope this will clean the question up a little bit.
Read content of the file using open and file.read:
with open('/path/to/json_file') as f:
data = f.read()
url = 'http://mysurl.org'
headers = {'content-type': 'application/json'}
r = requests.post(url, data=data, headers=headers)
UPDATE after reading comments.
You can make dictionaries from multiple data sources using zip and list comprehension:
data = [{'id': id, 'usr': usr, 'pwd': pwd} for id,usr,pwd in
zip(id_data_generator, usr_data_generator, pwd_data_generator)]