Develop Reference

Python requests PUT not updating data

I have a yaml file : file.yaml structured as follows :
index:
- uid: "uid"
name: "name"
headline: "headline"
overview: "overview"
features: "features"
instructions: "instructions"
callback_url: "https://some-url.com/params"
edit_url: "https://edit-url/params"
uninstall_hook: "https://uninstall-url/params"
svg:
screenshot1:
screenshot2:
screenshot3:
I have to upload those informations to an api endpoint by performing a PUT request. I managed to do it first using the register.py following script that I just run python register.py:
import json
import requests
from pathlib import Path
import base64
import yaml
BASE_URL = "https://url.com" # API Host
FILE_FOLDER = Path.cwd() # Current working directory
if __name__ == "__main__":
public_key = <public_key>
private_key = <private_key>
auth_key = "{}:{}".format(public_key, private_key).encode("utf-8")
encodedKey = base64.b64encode(auth_key).decode("utf-8")
headers = {"Authorization": f"Basic {encodedKey}", "Content-type": "application/json"}
def update_app_info():
infos_file = FILE_FOLDER / "file.yaml"
with open(infos_file) as infos_file_data:
yamlcontent = yaml.safe_load(infos_file_data) # Parse file.yaml and produce a dictionary of it
file_infos = yamlcontent["index"][0] # retrieve actual configuration informations
response = requests.put(
f"{BASE_URL}/path/to/api_endpoint/{public_key}", data=json.dumps(file_infos), headers=headers
)
print(response)
print(response.json())
update_app_info()
That gives a 202 success response.
As you may observe, I tried to get content of the yaml file as a dicitonary and send that in data. I proceeded that way regarding format of data at GET https://url.com/path/to/api_endpoint (mock example for illustration...) . Having the dictionary file_infos seemed more appropriate and gets me a success response. Sending directly the file itself or 'infos_file_data' gave me some errors I got over with the above script.
The issue is when I update svg, screenshot1, screenshot2 & screenshot3 so that file.yaml is now :
index:
- uid: "uid"
name: "name"
headline: "headline"
overview: "overview"
features: "features"
instructions: "instructions"
callback_url: "https://some-url.com/params"
edit_url: "https://edit-url/params"
uninstall_hook: "https://uninstall-url/params"
svg: "icon.svg"
screenshot1: "screenshot1.png"
screenshot2: "screenshot2.png"
screenshot3: "screenshot3.png"
That gives now :
<Response [400]>
{'error': {'message': {'svg': ['The submitted data was not a file. Check the encoding type on the form.'], 'screenshot1': ['The submitted data was not a file. Check the encoding type on the form.'], 'screenshot2': ['The submitted data was not a file. Check the encoding type on the form.'], 'screenshot3': ['The submitted data was not a file. Check the encoding type on the form.']}, 'code': 400}}
I've done multiple searches (1 , 2 , 3 , 4 , 5...) but their application and few other errors, eventually get me to this :
import base64
import json
from pathlib import Path
import requests
import yaml
from requests_toolbelt.multipart.encoder import MultipartEncoder
BASE_URL = "https://url.com" # API Host
FILE_FOLDER = Path.cwd() # Current working directory
if __name__ == "__main__":
public_key = <public_key>
private_key = <private_key>
auth_key = "{}:{}".format(public_key, private_key).encode("utf-8")
encodedKey = base64.b64encode(auth_key).decode("utf-8")
def update_app_info():
infos_file = FILE_FOLDER / "file.yaml"
with open(infos_file) as infos_file_data:
yamlcontent = yaml.safe_load(infos_file_data) # Parse file.yaml and produce a dictionary of it
file_infos = yamlcontent["index"][0] # retrieve actual configuration informations
m = MultipartEncoder(fields=file_infos)
#print(m.content_type)
headers = {
"Authorization": f"Basic {encodedKey}",
"Content-Type": m.content_type,
}
response = requests.put(
f"{BASE_URL}/path/to/api_endpoint/{public_key}",
data=json.dumps(file_infos),
headers=headers
)
print(response)
print(response.json())
update_app_info()
That is also giving me the 202 success response but the file svg, screenshot1, screenshot2 & screenshot3 fields are not updated.
I'll share more informations where needed. Your help is very welcome.

I've got additional resources that helped.
As I was trying to solve my issue, I found this. It happens I didn't wrote files part as it should, plus I was having data as a JSON string. That causes a ValueError: Data must not be a string. error. This was useful to get it fixed.
Now, for what it's worth, here's the working script :
import base64
from pathlib import Path
import requests
import yaml
BASE_URL = "https://url.com" # API Host
FILE_FOLDER = Path.cwd() # Current working directory
if __name__ == "__main__":
public_key = <public_key>
private_key = <private_key>
auth_key = "{}:{}".format(public_key, private_key).encode("utf-8")
encodedKey = base64.b64encode(auth_key).decode("utf-8")
def update_app_info():
infos_file = FILE_FOLDER / "file.yaml"
with open(infos_file) as infos_file_data:
yamlcontent = yaml.safe_load(infos_file_data) # Parse file.yaml and produce a dictionary of it
if "index" in yamlcontent:
file_infos = yamlcontent["index"][0] # retrieve actual configuration informations
headers = {
"Authorization": f"Basic {encodedKey}",
}
files = {
"svg": open("icon.svg", "rb"),
"screenshot1": open("screenshot1.png", "rb"),
"screenshot2": open("screenshot2.png", "rb"),
"screenshot3": open("screenshot3.png", "rb"),
}
response = requests.put(
f"{BASE_URL}/path/to/api_endpoint/{public_key}", data=file_infos, files=files, headers=headers
)
print("\n", response)
print("\n", response.headers)
print("\n", response.json())
update_app_info()

Replacing the existing file in google drive using python

I am trying to replace an existing file with the same id in the google drive.
import json
import requests
headers = {"Authorization": "Bearer Token"}
para = {
"name": "video1.mp4",
}
files = {
'data': ('metadata', json.dumps(para), 'application/json; charset=UTF-8'),
'file': open("./video1.mp4", "rb")
}
r = requests.post(
"https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart",
headers=headers,
files=files
)
print(r.text)
It creates files of similar names with different id with this code. Can I replace an existing file with the same id by mentioning it somewhere in this code?

Although I'm not sure whether I could correctly understand Can I replace an existing file with the same id by mentioning it somewhere in this code?, if you want to update an existing file on Google Drive, how about the following modified script?
From your showing script, I understood that you wanted to achieve your goal using requests instead of googleapis for python.
Modified script 1:
If you want to update both the file content and file metadata, how about the following modification?
import json
import requests
fileId = "###" # Please set the file ID you want to update.
headers = {"Authorization": "Bearer Token"}
para = {"name": "video1.mp4"}
files = {
"data": ("metadata", json.dumps(para), "application/json; charset=UTF-8"),
"file": open("./video1.mp4", "rb"),
}
r = requests.patch("https://www.googleapis.com/upload/drive/v3/files/" + fileId + "?uploadType=multipart",
headers=headers,
files=files,
)
print(r.text)
If you want to update only the file content, please modify para = {"name": "video1.mp4"} to para = {}.
Modified script 2:
If you want to update only the file metadata, how about the following modification?
import json
import requests
fileId = "###" # Please set the file ID you want to update.
headers = {"Authorization": "Bearer Token"}
para = {"name": "Updated filename video1.mp4"}
r = requests.patch(
"https://www.googleapis.com/drive/v3/files/" + fileId,
headers=headers,
data=json.dumps(para),
)
print(r.text)
Note:
In this answer, it supposes that your access token can be used for upload and update the file. Please be careful about this.
Reference:
Files: update

Python Requests - POST file with additional paramas

So I have this code that send API request to upload a file to a server, after searching the web I came up with something like that where 'payload' is additional params I need to send and 'files' is a path to the file I want to upload :
def sendAPIRequest2(self, env, object, method, params, files):
apiPath = "/api/secure/jsonws"
headers = {'content-type': 'multipart/form-data'}
url = env + apiPath + object
payload = {
"method": method,
"params": params,
"jsonrpc": "2.0"
}
data = json.dumps(payload)
myFile = {'file': open(files,'rb')}
try:
r = requests.post(url, files=myFile, data=data, headers=headers,
auth=HTTPBasicAuth(self.user, self.password), verify=False)
print r.text
resjson = json.loads(r.text)
except:
print "no Valid JSON has returned from"+object+" API"
sys.exit(1)
return resjson
I get:
ValueError: Data must not be a string.
What am I doing wrong?

How to upload file with multipart/form POST having only URL to file that need to be uploaded (chunks) [duplicate]

This question already has answers here:
Python file upload from url using requests library
(3 answers)
Closed 5 years ago.
Is there a chance to upload file by API endpoint which is taking multipart/form-data as a content-type having only URL of that file?
Rule:
Download the whole file into memory and then upload by this endpoint is no option (There is no guarantee that the box will ever be big enough to hold a temporary file).
Question:
I want to stream file in chunks from one server (GET) to another (multipart/form-data POST). Is this possible? How to achieve that?
Flow:
file_server <-GET- my_script.py -POST-> upload server
here is a simple example of downloading into memory (RAM) option (but it's against the rule):
from io import BytesIO
import requests
from requests_toolbelt.multipart.encoder import MultipartEncoder
file_url = 'https://www.sysaid.com/wp-content/uploads/features/itam/image-banner-asset.png'
requested_file_response = requests.get(file_url, stream=True)
TOKEN_PAYLOAD = {
'grant_type': 'password',
'client_id': '#########',
'client_secret': '#########',
'username': '#########',
'password': '#########'
}
def get_token():
response = requests.post(
'https://upload_server/oauth/token',
params=TOKEN_PAYLOAD)
response_data = response.json()
token = response_data.get('access_token')
if not token:
print("token error!")
return token
token = get_token()
file_object = BytesIO()
file_object.write(requested_file_response.content)
# Form conctent
multipart_data = MultipartEncoder(
fields={
'--': (
'test.png',
file_object # AttributeError: 'generator' object has no attribute 'encode' when I try to pass generator here.
),
'id': '2217',
'fileFieldDefId': '4258',
}
)
# Create headers
headers = {
"Authorization": "Bearer {}".format(token),
'Content-Type': multipart_data.content_type
}
session = requests.Session()
response = session.post(
'https://upload_server/multipartUpdate',
headers=headers,
data=multipart_data,
)
the answer is in a file like object creation for stream purposes
Thank You very much for any help. Cheers!

If I read requests_toolbelt source code right than it requires not only a ability to .read() the file (which we could get just by passing requests.get(..., stream=True).raw), but also that there is someway to determine how much data is left in the stream.
Assuming you are CONFIDENT that you always have a valid content-length header, this would be the solution I would suggest:
import requests
from requests_toolbelt.multipart.encoder import MultipartEncoder
file_url = 'https://www.sysaid.com/wp-content/uploads/features/itam/image-banner-asset.png'
target = 'http://localhost:5000/test'
class PinocchioFile:
"""I wish I was a real file"""
def __init__(self, url):
self.req = requests.get(url, stream=True)
length = self.req.headers.get('content-length')
self.len = None if length is None else int(length)
self._raw = self.req.raw
def read(self, chunk_size):
chunk = self._raw.read(chunk_size) or b''
self.len -= len(chunk)
if not chunk:
self.len = 0
return chunk
multipart_data = MultipartEncoder(
fields={
'--': (
'test.png',
PinocchioFile(file_url),
),
'id': '2217',
'fileFieldDefId': '4258',
}
)
# Create headers
headers = {
'Content-Type': multipart_data.content_type
}
response = requests.post(
target,
data=multipart_data,
headers=headers,
)

python-requests and EchoSign

I am trying to post a file into EchoSign APIs and it works everywhere for me except for python-requests.
I have here the CURL command that works perfectly
curl -H "Access-Token: API_KEY" \
-F File=#/home/user/Desktop/test123.pdf \
https://secure.echosign.com/api/rest/v2/transientDocuments
and this is my requests function. It will upload the PDF file but with garbage whereas CURL works perfectly.
api_url = 'https://secure.echosign.com/api/rest/v2'
def send_document(file_path, access_token=access_token):
"""Uploads document to EchoSign and returns its ID
:param access_token: EchoSign Access Token
:param file_path: Absolute or relative path to File
:return string: Document ID
"""
headers = {'Access-Token': access_token}
url = api_url + '/transientDocuments'
with open(file_path, 'rb') as f:
files = {
'File': f,
}
return requests.post(url, headers=headers, files=files).json().get('transientDocumentId')
What am I doing wrong?? I have tried posting the file as data instead of files too and still no different result
Thanks
EDIT
It worked when I added
data = {
'Mime-Type': 'application/pdf',
'File-Name': 'abc.pdf'
}
So, my new function would be:
def send_document(file_path, access_token=access_token):
"""Uploads document to EchoSign and returns its ID
:param access_token: EchoSign Access Token
:param file_path: Absolute or relative path to File
:return string: Document ID
"""
headers = {
'Access-Token': access_token,
}
data = {
'Mime-Type': 'application/pdf',
'File-Name': 'abc.pdf'
}
url = api_url + '/transientDocuments'
files = {'File': open(file_path, 'rb')}
return requests.post(url, headers=headers, data=data,
files=files).json().get('transientDocumentId')

This is how it works
def send_document(file_path, access_token=access_token):
"""Uploads document to EchoSign and returns its ID
:param access_token: EchoSign Access Token
:param file_path: Absolute or relative path to File
:return string: Document ID
"""
headers = {
'Access-Token': access_token,
}
data = {
'Mime-Type': 'application/pdf',
'File-Name': 'abc.pdf'
}
url = api_url + '/transientDocuments'
files = {'File': open(file_path, 'rb')}
return requests.post(url, headers=headers, data=data,
files=files).json().get('transientDocumentId')

Try passing in the filename and mime-type, like so:
files = {
'File': (
os.path.basename(file_path),
f,
'application/pdf',
)
}
References:
http://docs.python-requests.org/en/latest/user/quickstart/#post-a-multipart-encoded-file
man curl, see the --trace-file FILE argumet
Proper MIME media type for PDF files