I have a yaml file : file.yaml structured as follows :
index:
- uid: "uid"
name: "name"
headline: "headline"
overview: "overview"
features: "features"
instructions: "instructions"
callback_url: "https://some-url.com/params"
edit_url: "https://edit-url/params"
uninstall_hook: "https://uninstall-url/params"
svg:
screenshot1:
screenshot2:
screenshot3:
I have to upload those informations to an api endpoint by performing a PUT request. I managed to do it first using the register.py following script that I just run python register.py:
import json
import requests
from pathlib import Path
import base64
import yaml
BASE_URL = "https://url.com" # API Host
FILE_FOLDER = Path.cwd() # Current working directory
if __name__ == "__main__":
public_key = <public_key>
private_key = <private_key>
auth_key = "{}:{}".format(public_key, private_key).encode("utf-8")
encodedKey = base64.b64encode(auth_key).decode("utf-8")
headers = {"Authorization": f"Basic {encodedKey}", "Content-type": "application/json"}
def update_app_info():
infos_file = FILE_FOLDER / "file.yaml"
with open(infos_file) as infos_file_data:
yamlcontent = yaml.safe_load(infos_file_data) # Parse file.yaml and produce a dictionary of it
file_infos = yamlcontent["index"][0] # retrieve actual configuration informations
response = requests.put(
f"{BASE_URL}/path/to/api_endpoint/{public_key}", data=json.dumps(file_infos), headers=headers
)
print(response)
print(response.json())
update_app_info()
That gives a 202 success response.
As you may observe, I tried to get content of the yaml file as a dicitonary and send that in data. I proceeded that way regarding format of data at GET https://url.com/path/to/api_endpoint (mock example for illustration...) . Having the dictionary file_infos seemed more appropriate and gets me a success response. Sending directly the file itself or 'infos_file_data' gave me some errors I got over with the above script.
The issue is when I update svg, screenshot1, screenshot2 & screenshot3 so that file.yaml is now :
index:
- uid: "uid"
name: "name"
headline: "headline"
overview: "overview"
features: "features"
instructions: "instructions"
callback_url: "https://some-url.com/params"
edit_url: "https://edit-url/params"
uninstall_hook: "https://uninstall-url/params"
svg: "icon.svg"
screenshot1: "screenshot1.png"
screenshot2: "screenshot2.png"
screenshot3: "screenshot3.png"
That gives now :
<Response [400]>
{'error': {'message': {'svg': ['The submitted data was not a file. Check the encoding type on the form.'], 'screenshot1': ['The submitted data was not a file. Check the encoding type on the form.'], 'screenshot2': ['The submitted data was not a file. Check the encoding type on the form.'], 'screenshot3': ['The submitted data was not a file. Check the encoding type on the form.']}, 'code': 400}}
I've done multiple searches (1 , 2 , 3 , 4 , 5...) but their application and few other errors, eventually get me to this :
import base64
import json
from pathlib import Path
import requests
import yaml
from requests_toolbelt.multipart.encoder import MultipartEncoder
BASE_URL = "https://url.com" # API Host
FILE_FOLDER = Path.cwd() # Current working directory
if __name__ == "__main__":
public_key = <public_key>
private_key = <private_key>
auth_key = "{}:{}".format(public_key, private_key).encode("utf-8")
encodedKey = base64.b64encode(auth_key).decode("utf-8")
def update_app_info():
infos_file = FILE_FOLDER / "file.yaml"
with open(infos_file) as infos_file_data:
yamlcontent = yaml.safe_load(infos_file_data) # Parse file.yaml and produce a dictionary of it
file_infos = yamlcontent["index"][0] # retrieve actual configuration informations
m = MultipartEncoder(fields=file_infos)
#print(m.content_type)
headers = {
"Authorization": f"Basic {encodedKey}",
"Content-Type": m.content_type,
}
response = requests.put(
f"{BASE_URL}/path/to/api_endpoint/{public_key}",
data=json.dumps(file_infos),
headers=headers
)
print(response)
print(response.json())
update_app_info()
That is also giving me the 202 success response but the file svg, screenshot1, screenshot2 & screenshot3 fields are not updated.
I'll share more informations where needed. Your help is very welcome.
I've got additional resources that helped.
As I was trying to solve my issue, I found this. It happens I didn't wrote files part as it should, plus I was having data as a JSON string. That causes a ValueError: Data must not be a string. error. This was useful to get it fixed.
Now, for what it's worth, here's the working script :
import base64
from pathlib import Path
import requests
import yaml
BASE_URL = "https://url.com" # API Host
FILE_FOLDER = Path.cwd() # Current working directory
if __name__ == "__main__":
public_key = <public_key>
private_key = <private_key>
auth_key = "{}:{}".format(public_key, private_key).encode("utf-8")
encodedKey = base64.b64encode(auth_key).decode("utf-8")
def update_app_info():
infos_file = FILE_FOLDER / "file.yaml"
with open(infos_file) as infos_file_data:
yamlcontent = yaml.safe_load(infos_file_data) # Parse file.yaml and produce a dictionary of it
if "index" in yamlcontent:
file_infos = yamlcontent["index"][0] # retrieve actual configuration informations
headers = {
"Authorization": f"Basic {encodedKey}",
}
files = {
"svg": open("icon.svg", "rb"),
"screenshot1": open("screenshot1.png", "rb"),
"screenshot2": open("screenshot2.png", "rb"),
"screenshot3": open("screenshot3.png", "rb"),
}
response = requests.put(
f"{BASE_URL}/path/to/api_endpoint/{public_key}", data=file_infos, files=files, headers=headers
)
print("\n", response)
print("\n", response.headers)
print("\n", response.json())
update_app_info()
Related
This is my code for uploading to google drive with python requests using google-drive-api.
import sys
import json
import requests
from tqdm import tqdm
import requests_toolbelt
from requests.exceptions import JSONDecodeError
class ProgressBar(tqdm):
def update_to(self, n: int) -> None:
self.update(n - self.n)
def upload_file(access_token:str, filename:str, filedirectory:str):
metadata = {
"title": filename,
}
files = {}
session = requests.session()
with open(filedirectory, "rb") as fp:
files["file"] = fp
files["data"] = ('metadata', json.dumps(metadata), 'application/json')
encoder = requests_toolbelt.MultipartEncoder(files)
with ProgressBar(
total=encoder.len,
unit="B",
unit_scale=True,
unit_divisor=1024,
miniters=1,
file=sys.stdout,
) as bar:
monitor = requests_toolbelt.MultipartEncoderMonitor(
encoder, lambda monitor: bar.update_to(monitor.bytes_read)
)
r = session.post(
"https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart",
data=monitor,
allow_redirects=False,
headers={"Authorization": "Bearer " + access_token},
)
try:
resp = r.json()
print(resp)
except JSONDecodeError:
sys.exit(r.text)
upload_file("access_token", "test.txt", "test.txt")
When i am trying send file with data attribute in post request then file name did not send and with files attribute in post request then requests-toolbelt not working. How to fix this error ?
When I saw your script, I thought that the content type is not included in the request header. In this case, I think that the request body is directly shown in the uploaded file. I thought that this might be the reason for your current issue. In order to remove this issue, how about the following modification?
From:
r = session.post(
url,
data=monitor,
allow_redirects=False,
headers={"Authorization": "Bearer " + access_token},
)
To:
r = session.post(
url,
data=monitor,
allow_redirects=False,
headers={
"Authorization": "Bearer " + access_token,
"Content-Type": monitor.content_type,
},
)
In this case, from metadata = { "title": filename }, it supposes that url is https://www.googleapis.com/upload/drive/v2/files?uploadType=multipart. Please be careful about this.
When you want to use Drive API v3, please modify metadata = { "title": filename } to metadata = { "name": filename }, and use the endpoint of https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart.
When the file is uploaded with Drive API v3, the value of {'kind': 'drive#file', 'id': '###', 'name': 'test.txt', 'mimeType': 'text/plain'} is returned.
By the way, when an error like badContent occurs in your testing, please try to test the following modification. When in the request body of multipart/form-data the file content is put before the file metadata, it seems that an error occurs. I'm not sure whether this is the current specification. But, I didn't know the order of request body is required to be checked.
From
files = {}
files["file"] = fp
files["data"] = ('metadata', json.dumps(metadata), 'application/json')
To
files = collections.OrderedDict(data=("metadata", json.dumps(metadata), "application/json"), file=fp)
Note:
I thought that in your script, an error might occur at file_size = os.path.getsize(filename). Please confirm this again.
When I tested your script by modifying the above modifications, I could confirm that a test file could be uploaded to Google Drive with the expected filename. In this case, I also modified it as follows.
files = collections.OrderedDict(data=("metadata", json.dumps(metadata), "application/json"), file=fp)
References:
Files: insert of Drive API v2
Files: create of Drive API v3
Upload file data
Metadata needs to be sent in the post body as json.
Python Requests post() Method
data = Optional. A dictionary, list of tuples, bytes or a file object to send to the specified url
json = Optional. A JSON object to send to the specified url
metadata = {
"name": filename,
}
r = session.post(
url,
json=json.dumps(metadata),
allow_redirects=False,
headers={"Authorization": "Bearer " + access_token},
)
Future readers can find below a complete script that also contains details on how to get access to the bearer token for HTTP authentication.
Most of the credit goes to the OP and answers to the OPs question.
"""
Goal: For one time upload of a large file (as the GDrive UI hangs up)
Step 1 - Create OAuth 2.0 Client ID + Client Secret
- by following the "Authentication" part of https://pythonhosted.org/PyDrive/quickstart.html
Step 2 - Get Access Token
- from the OAuth playground -> https://developers.google.com/oauthplayground/
--> Select Drive API v3 -> www.googleapis.com/auth/drive --> Click on "Authorize APIs"
--> Click on "Exchange authorization code for tokens" --> "Copy paste the access token"
--> Use it in the script below
Step 3 - Run file as daemon process
- nohup python -u upload_gdrive.py > upload_gdrive.log 2>&1 &
- tail -f upload_gdrive.log
"""
import sys
import json
import requests
from tqdm import tqdm
import requests_toolbelt # pip install requests_toolbelt
from requests.exceptions import JSONDecodeError
import collections
class ProgressBar(tqdm):
def update_to(self, n: int) -> None:
self.update(n - self.n)
def upload_file(access_token:str, filename:str, filepath:str):
metadata = {
"name": filename,
}
files = {}
session = requests.session()
with open(filepath, "rb") as fp:
files = collections.OrderedDict(data=("metadata", json.dumps(metadata), "application/json"), file=fp)
encoder = requests_toolbelt.MultipartEncoder(files)
with ProgressBar(
total=encoder.len,
unit="B",
unit_scale=True,
unit_divisor=1024,
miniters=1,
file=sys.stdout,
) as bar:
monitor = requests_toolbelt.MultipartEncoderMonitor(
encoder, lambda monitor: bar.update_to(monitor.bytes_read)
)
r = session.post(
"https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart",
data=monitor,
allow_redirects=False,
headers={
"Authorization": "Bearer " + access_token
, "Content-Type": monitor.content_type
},
)
try:
resp = r.json()
print(resp)
except JSONDecodeError:
sys.exit(r.text)
upload_file("<access_token>"
, "<upload_filename>", "<path_to_file>")
I have a model endpoint running in Azure Kubernetes Service and I am not using Django or Flask. I am sending local png files to score as follows:
import base64
import json
import cv2
import requests
img_path = 'C:/path/to/exampleImage.png'
link = aks_service.scoring_uri
api_key = aks_service.get_keys()[0]
def send2score(img_path, score_url, api_key):
headers = {'Content-Type': 'application/json',
'Authorization': ('Bearer ' + api_key)
}
img = cv2.imread(img_path)
string = base64.b64encode(cv2.imencode('.png', img)[1]).decode()
dict = {
'img': string
}
jsonimg2 = json.dumps(dict, ensure_ascii=False, indent=4)
resp = requests.post(url=link, data=jsonimg2, headers=headers)
print(resp.text)
send2score(img_path=img_path, score_url=link, api_key=api_key)
My question is: how can I get the file name (exampleImage.png) in the score script in Azure Kubernetes after I do the request.post? Please no Django or Flask specific methods
Bonus question: Feel free to suggest improvements in the way I am uploading the data (send2score function), this function is working, I get the score back, I just can't get the file name in the score script. Thank you!
According to your code, you send your image as base64 string. It cannot contain the file name. I think you need to define a parameter to store the file name in request body. Besides, you also can post the file as a Multipart-Encoded File with the requests module.
For example
send file
import requests
import magic
import os
url = ''
path = "D:/sample/image/faces.jpg"
mime = magic.Magic(mime=True)
headers = {
'Authorization': ('Bearer ' + 'cdff')
}
files = {'file': (os.path.basename(path), open(path, 'rb'), mime.from_file(path), {'Expires': '0'})}
res = requests.post(url, files=files, headers=headers)
print(res.content.decode('utf-8'))
My backend
from http.server import BaseHTTPRequestHandler, HTTPServer
import cgi
hostName =
hostPort =
class MyServer(BaseHTTPRequestHandler):
def do_POST(self):
try:
// use cgi to read file
form = cgi.FieldStorage(fp=self.rfile, headers=self.headers, environ={
'REQUEST_METHOD': 'POST', 'CONTENT_TYPE': self.headers['Content-Type'], })
file = form.list[0]
data =file.file.read()
#process data
.............
self.send_response(200)
self.send_header("Content-type", "text/html")
self.end_headers()
self.wfile.write(bytes(
f"<html><head></head><body><h2>fileName : {file.filename}</h2>/html>", "utf-8"))
except Exception as e:
httperror = e.httperror if hasattr(e, "httperror") else 500
self.send_error(httperror, str(e)) # Send an error response
myServer = HTTPServer((hostName, hostPort), MyServer)
myServer.serve_forever()
I was overcomplicating things, I realized that I am sending the encoded image as a json in a dictionary. I can include other information, like the file name, in the dictionary:
import base64
import json
import cv2
import requests
img_path = 'C:/path/to/exampleImage.png'
link = aks_service.scoring_uri
api_key = aks_service.get_keys()[0]
def send2score(img_path, score_url, api_key):
headers = {'Content-Type': 'application/json',
'Authorization': ('Bearer ' + api_key)
}
img = cv2.imread(img_path)
string = base64.b64encode(cv2.imencode('.png', img)[1]).decode()
dict = {
'imgname': os.path.basename(img_path),
'img': string
}
jsonimg2 = json.dumps(dict, ensure_ascii=False, indent=4)
resp = requests.post(url=link, data=jsonimg2, headers=headers)
print(resp.text)
send2score(img_path=img_path, score_url=link, api_key=api_key)
And I can get the image and file name in the score script:
# Lots of code before
response = json.loads(path)
string = response['img']
jpg_original = base64.b64decode(string) # decode
jpg_as_np = np.frombuffer(jpg_original, dtype=np.uint8)
img0 = cv2.imdecode(jpg_as_np, flags=1) # image
img0name = response['imgname'] # file name
# Lots of code after
I try to acess data from an Exchange Platform with API.
I have my API_Key and my SECRET_KEY, and I have the documentation of that Platform here:
https://apidocs.exir.io/
I generate the "signature-key" just as described in the documentation under "Authentication" chapter and then try to test it with a sample GET request with only one parameter.
Now if I run the Code I get "message": "Access Denied: Invalid API Signature"
Can you please help me to find the wrong thing in this code?
I think I do something wrong with params because if I use other GET orders without parameters it works!
Thank you in advance!
import time
import json
import hmac
import hashlib
import requests
API_KEY = '*****'
SECRET_KEY = '*****'
BASE_URL = 'https://api.exir.io'
timestamp = str(int(time.time()+10))
headers = {
'api-key': API_KEY,
'api-expires': timestamp} # see documentation under "Authentication"
PATH = '/v1/user/orders' # This ist just a simple example, which uses "params". See Exir documentation under "Get All Orders"
params = {"symbol":"btc-irt"}
string = 'GET'+timestamp+str(params) # see Exir API doumentation under "Authentication"
headers['api-signature'] = hmac.new(SECRET_KEY.encode('utf-8'), string.encode('utf-8'), hashlib.sha256).hexdigest()
url = 'https://api.exir.io/v1/user/orders?symbol=btc-irt'
r = requests.get(url, headers=headers)
data = r.json()
print(json.dumps(data, indent=2))
A lot of what you are doing is unnecessary, actually. The requests library handles most of what you are trying to do.
import requests
API_KEY = '*****'
SECRET_KEY = '*****'
BASE_URL = 'https://api.exir.io'
timestamp = str(int(time.time()+10))
headers = {
'api-key': API_KEY,
'api-expires': timestamp
}
url = 'https://api.exir.io/v1/user/orders?symbol=btc-irt'
r = requests.get(url, headers=headers)
The library will do the encoding for you.
You mixed the post and get request parameters. For the get request, you only need to include the params in the URL to sign. In your case it will be:
PATH = '/v1/user/orders?symbol=btc-irt'
string = 'GET/' + PATH + timestamp
I am not able to get JSON response when I try to run python core-backup.py file: In code FB_SHORT_ACCESS_TOKEN and FB_LONG_ACCESS_TOKEN are same.
core-backup.py :
import os
from os.path import join
import requests
def refresh_short_token():
"""
Refresh short access token
"""
request_url = FB_URL + 'oauth/access_token'
request_payload = {
'grant_type': 'fb_exchange_token',
'client_id': FB_APP_ID,
'client_secret': FB_APP_SECRET,
'fb_exchange_token': FB_SHORT_ACCESS_TOKEN
}
response = REQ_SESSION.get(request_url, params=request_payload).json()
# dotenvfile = find_dotenv()
# load_dotenv(dotenvfile)
# dotenv.set_key(dotenvfile, "FB_LONG_ACCESS_TOKEN", response['access_token'])
FB_LONG_ACCESS_TOKEN = response["access_token"]
# PAYLOAD['access_token'] = dotenv.get_key(dotenvfile, "FB_LONG_ACCESS_TOKEN")
PAYLOAD['access_token'] = FB_LONG_ACCESS_TOKEN
'''
TODO: refresh_long_token()
A function to refresh the long term access token
Current validity: 60 days
'''
def get_feed():
"""
Fetch feed
"""
request_url = FB_URL + LTTK_GROUP_ID + '/feed'
response = REQ_SESSION.get(request_url, params=PAYLOAD)
if response.status_code == 400:
refresh_short_token()
print(response.json())
return response.json()
def main():
"""
Fetch posts from a Facebook group and populate in database
"""
get_feed()
if __name__ == "__main__":
main()
I am getting UnicodeDecodeError in windows7 after running core-backup.py
file. How to fix this issue.
See screenshot for more clarity:
Entire code of file can be found out here:
https://gist.github.com/anonymous/2ab9e023d631a7cc4dad15237104ee34
It appears that your code page is set to cp437. Try setting python output to utf-8 by entering the following line in your terminal before running your python script.
set PYTHONIOENCODING=UTF-8
python core-backup.py
Try changing response encoding to UTF-8:
response.encoding = 'UTF-8'
print(response.json())
I'm in the process of moving learning how to interact with APIs in my programming training. The following script is used on the sandbox environment for a software I'm trying to interact with.
For some reason, I cannot login to the API to run the second function. Does anyone see the issue?
# import requests library
import requests
#import json library
import json
controller='apicontroller#test.com'
def getToken():
# put the ip address or dns of your apic-em controller in this url
url = "https://" + controller + "/api/aaaLogin.json"
json_object = {
"aaaUser" : {
"attributes" : {
"name" : "******",
"pwd" : "*******"
}
}
}
#Content type must be included in the header
header = {"content-type": "application/json"}
#Performs a POST on the specified url to get the service ticket
response= requests.post(url,data=json.dumps(json_object), headers=header, verify=False)
#convert response to json format
r_json=response.json()
#parse the json to get the service ticket
token = r_json["response"]["token"]
return token
def pushtenant():
# URL for network device REST API call to get list of existing devices on the network.
url = "https://" + controller + "/api/api/mo/uni"
#Content type must be included in the header as well as the ticket
headers = {"content-type": "application/json", "X-Auth-Token":Token}
json_tenant = [{
"fvTenant" : {
"attributes" : {
"name" : "ExampleCorp"
}
}
}]
# this statement performs a GET on the specified network-device url
response = requests.post(url, json.dumps(json_tenant), headers=headers,verify=False)
# json.dumps serializes the json into a string and allows us to
# print the response in a 'pretty' format with indentation etc.
print ("Response = ")
print (json.dumps(response.json(), indent=4, separators=(',', ': ')))
enter code here
#convert data to json format.
r_json=response.json()
theToken=getToken()
pushtenant(theToken)