I have a REST API that I am trying to upload data to , which is basically this : https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/update
Now, since the only option I have is PATCH, what are the options for me to have an optimized data load. I have been able to upload files by using the data parameter and using the read() function, but I don't think it's optimal as the entire file is read into memory I guess. I have tried using the files parameter (the multipaprt form encoding) and also looked at the toolbelt package but that does not seem to work for PATCH
This is the Sample Code that works but is not optimal
files={'file':('Sample',open('D:/FilePath/Demo.txt','rb'))}
length=os.stat('D:/FilePath/Demo.txt')
filesize=str(length.st_size)
with open('D:/File|Path/Demo.txt','rb') as f:
file_data = f.read()
leng=len(file_data)
header = {
'Authorization': "Bearer " + auth_t
}
header_append = {
'Content-Length': filesize,
'Authorization': "Bearer " + auth_t
#'If-None-Match': "*" #Conditional HTTP Header
}
header_flush = {
'Content-Length': '0',
'Authorization': "Bearer " + auth_t
}
header_read = {
'Authorization': "Bearer " + auth_t
}
try:
init_put=requests.put('https://adlstorageacc.dfs.core.windows.net/adobe/2019/02/DemoStreamFile4.txt?resource=file&recursive=True', headers=header_flush, proxies=proxies,verify=False)
init_write=requests.patch('https://adlstorageacc.dfs.core.windows.net/adobe/2019/02/DemoStreamFile4.txt?action=append&position=0', headers=header_append, proxies=proxies,verify=False,data=file_data)
flush_url='https://adlstorageacc.dfs.core.windows.net/adobe/2019/02/DemoStreamFile4.txt?action=flush&position=' + str(leng)
init_flush=requests.patch(flush_url, headers=header_flush, proxies=proxies,verify=False)
Problem is the line
init_write=requests.patch('https://adlstorageacc.dfs.core.windows.net/adobe/2019/02/DemoStreamFile4.txt?action=append&position=0', headers=header_append, proxies=proxies,verify=False,data=file_data)
It only seems to take take the data parameter. If I change it to
init_write=requests.patch('https://adlstorageacc.dfs.core.windows.net/adobe/2019/02/DemoStreamFile4.txt?action=append&position=0', headers=header_append, proxies=proxies,verify=False,file=files)
I get an empty file.
Same is the case when I use the requestToolbelt package.
Does patch not recognize the file parameter? Nothing on the requests documents says any of it.
Also, if data parameter is the only way out, what is the best way to loading a file without doing a f.read() or iteratively specifying number of characters to read using f.read(n). Isn't there a better way?
After also looking through Postman, was able to find the problem. Here's the solution. Problem was the with open statement and particularly the position parameter for the flush part, because the Content Length was overridden automatically, so had to get the Content Length from the request of the response.
files={'file':('Sample',open('D:/FilePath/Demo.txt','rb'))}
length=os.stat('D:/FilePath/Demo.txt')
filesize=str(length.st_size)
header = {
# 'Content-Type': 'text/plain',
'Authorization': "Bearer " + auth_t
#'If-None-Match': "*" #Conditional HTTP Header
}
header_append = {
'Content-Length': filesize,
'Authorization': "Bearer " + auth_t
#'If-None-Match': "*" #Conditional HTTP Header
}
header_flush = {
'Content-Type': "application/x-www-form-urlencoded",
'Content-Length': '0',
'Authorization': "Bearer " + auth_t,
#'If-None-Match': "*" #Conditional HTTP Header
}
header_read = {
# 'Content-Type': 'text/plain',
'Authorization': "Bearer " + auth_t,
#'Range': 'bytes=300000-302591'
#'If-None-Match': "*" #Conditional HTTP Header
}
try:
init_put=requests.put('https://adlstorageacc.dfs.core.windows.net/adobe/2019/02/DemoStreamFile4.txt?resource=file&recursive=True', headers=header_flush, proxies=proxies,verify=False)
init_write=requests.patch('https://adlstorageacc.dfs.core.windows.net/adobe/2019/02/DemoStreamFile4.txt?action=append&position=0', headers=header_append, proxies=proxies,verify=False,files=files)
flush_length=init_write.request.headers['Content-Length']
flush_url='https://adlstorageacc.dfs.core.windows.net/adobe/2019/02/DemoStreamFile4.txt?action=flush&position=' + str(flush_length)
init_flush=requests.patch(flush_url, headers=header_flush, proxies=proxies,verify=False)
except Exception as e:
print("In Error")
print(e)
Related
I've been trying (and failing miserably) to use google's urlfetch module (python within app engine's local server) to retrieve a token from paypal. It works as follows using the "requests" module outside of app engine:
url = base + "/v1/oauth2/token"
payload = {
'grant_type': 'client_credentials',
}
auth_encoded = APP_CLIENT_ID + ":" + APP_SECRET
auth_encoded = base64.b64encode(auth_encoded)
##headers = {'Content-Type': 'application/json', 'Authorization': 'Basic ' + auth_encoded}
headers = {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic ' + auth_encoded}
r = requests.post(url,headers=headers,params=payload)
print r.text
... but I get this message when trying the same thing with urlfetch:
{"error":"unsupported_grant_type","error_description":"Grant Type is NULL"}
... here's the code that I'm using:
url = base + "/v1/oauth2/token"
payload = {"grant_type": "client_credentials"}
auth_encoded = APP_CLIENT_ID + ":" + APP_SECRET
auth_encoded = base64.b64encode(auth_encoded)
headers = {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic ' + auth_encoded}
result = urlfetch.fetch(
url=url,
method=urlfetch.POST,
headers=headers,
payload = payload
)
... I've tried everything that I can think of. Should be a simple thing.
This API call is formatted as application/x-www-form-urlencoded , not JSON.
Therefore:
payload = "grant_type=client_credentials"
or
import urllib
payload = urllib.urlencode({"grant_type": "client_credentials"})
I want to have a python request and add pagination, but for some reason it does not work,
this is my request. Does someone know how to add 'paging': '{"page":0,"size":100}' corretly? The following isn't working?
url = self.get_url_for_endpoint(Constants.PATH_STATISTICS_CUSTOMERS)
payload = {}
params = {
'paging': '{"page":0,"size":100}'
}
headers = {
'Authorization': 'Bearer ' + self.access_token,
'Cookie': 'JSESSIONID=AB52DV8260C*****************',
'Content-Type': 'application/json',
}
r = requests.request("GET", url, headers=headers, data=payload, params=params)
This is working though via postman:
url + ?paging=%7B%22page%22:0,%22size%22:400%7D
So the endpoint has pagination!
I wouldn't recommend using an object as your paging value in the params, instead break down your page and size into individual parameters, like ?page=0&size=100.
But if you want to use an object, your params should be like this:
params = {
"paging": {"page":0,"size":100}
}
I am trying to upload a file via Python requests and i receive an error code 400 (Bad request)
#Update ticket with upload of CSV file
header_upload_file = {
'Authorization': 'TOKEN id="' + token + '"',
'Content-Type': 'multipart/form-data'
}
files = {
'name': 'file',
'filename': open(main_path + '/temp/test.txt', 'rb'),
'Content-Disposition': 'form-data'
}
response = requests.post(baseurl + '/incidents/number/' + ticket_number + '/attachments/', headers=header_upload_file, data=files, verify=certificate)
If i try via Postman this is successfull using following code.
url = "https://<url>"
payload = "------WebKitFormBoundary7MA4YWxkTrZu0gW\r\nContent-Disposition: form-data; name=\"file\"; filename=\"C:\\Users\\<filename>\"\r\nContent-Type: text/csv\r\n\r\n\r\n------WebKitFormBoundary7MA4YWxkTrZu0gW--"
headers = {
'content-type': "multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW",
'Authorization': "TOKEN id="3e9d095d-a47b-48b5-a0b8-ae8b8ad9ae74"",
'cache-control': "no-cache",
'Postman-Token': "bb155176-b1b8-47a6-8fb3-46f5740cf9e0"
}
response = requests.request("POST", url, data=payload, headers=headers)
print(response.text)
What do i wrong?
You should use the files parameter instead. Also, don't explicitly set Content-Type in the header so that requests can set it for you with a proper boundary:
header_upload_file = {
'Authorization': 'TOKEN id="' + token + '"'
}
response = requests.post(
baseurl + '/incidents/number/' + ticket_number + '/attachments/',
headers=header_upload_file,
files={'file': ('file', open(main_path + '/temp/test.txt', 'rb'), 'text/csv')},
verify=certificate
)
I'm not sure how to get the headers in an alphabetical order as the in the example in python-requests. Do the brackets and the punctuation such as ' effect the request headers when they are sent?
This is my code:
headers = {'oauth_callback': "http://www.website-tm-access.co.nz%2Ftrademe-callback",
'oauth_consumer_key' : "5C82CC6BC7C6472154FBC9CAB24A29A2" ,
'oauth_version': "1.0",
'oauth_timestamp': time, #int(time.time()),
'oauth_nonce' : nonce,
'oauth_signature_method' : "HMAC-SHA1",
'oauth_signature' : signature
}
authorization = ', '.join([key + '="' + urllib.parse.quote_plus(str(value)) + '"' for key, value in headers.items()])
http_headers = {'Authorization': authorization}
The Output is:
{'Authorization': 'oauth_nonce="62942100", oauth_callback="http%3A%2F%2Fwww.website-tm-access.co.nz%252Ftrademe-callback", oauth_consumer_key="5C82CC6BC7C6472154FBC9CAB24A29A2", oauth_version="1.0", oauth_timestamp="1457345962", oauth_signature="b%27o0rNPtba78EQ3ALsg2mX1Y4vIQw%3D%27", oauth_signature_method="HMAC-SHA1"'}
The output as defined in the example:
To make the Authorization header, you simply append all the values
starting with “OAuth”. Each value must be URL encoded.
Authorization: OAuth oauth_callback="http%3A%2F%2Fwww.website-tm-access.co.nz%2Ftrademe-callback", oauth_consumer_key="C74CD73FDBE37D29BDD21BAB54BC70E422", oauth_version="1.0", oauth_timestamp="1285532322", oauth_nonce="7O3kEe", oauth_signature_method="HMAC-SHA1", oauth_signature="5s3%2Bel078a5AXGi43FBDyfg5yWY%3D"
The order shouldn't matter when passing the headers to the requests module.
You can just do:
headers = {'oauth_callback': "http://www.website-tm-access.co.nz%2Ftrademe-callback",
'oauth_consumer_key' : "5C82CC6BC7C6472154FBC9CAB24A29A2" ,
'oauth_version': "1.0",
'oauth_timestamp': int(time.time()),
'oauth_nonce' : nonce,
'oauth_signature_method' : "HMAC-SHA1",
'oauth_signature' : signature
}
r = requests.get(url, headers=headers)
I am new to python & POST message.
I am trying to call an API with JSON POST message and expecting a JSON response but my initial code not able to make the call as required .
using the URL,Headers & Postdata in chrome browser POST extension works fine.
#!/usr/bin/python
import requests
import json
url = 'http://xxxxx:111/batches'
postdata = {
"active": "true",
"size": "2",
"ctr": {
"user": "Admin",
"id": "1234"}}
#headers = {'content-type': 'application/json',
#'Authorization': 'Basic xyz879jjkhhnm',
#'Accept-Encoding': '0'}
headers = {'Authorization': 'Basic xyz879jjkhhnm', 'Accept-Encoding': '0'}
print headers
post_call = requests.post(url, headers=headers, data=json.dumps(postdata))
print post_call, "POST call"
print post_call.text, "TEXT"
print post_call.content, "CONTENT"
post_call.status_code, "STATUS CODE"
Error:
{'Accept-Encoding': '0', 'Authorization': 'Basic xyz879jjkhhnm'}
<Response [500]> POST call
[{"code":"server_error","description":"com.sun.jersey.api.MessageException: A message body reader for Java class com.hide.cpn.rest.v1.entity.CouponCodeBatchResourceEntity,
and Java type class com.hide.cpn.rest.v1.entity.CouponCodeBatchResourceEntity, and MIME media type application/octet-stream was not found.
\nThe registered message body readers compatible with the MIME media type are:\napplication/octet-stream ->\n
com.sun.jersey.core.impl.provider.entity.ByteArrayProvider\n com.sun.jersey.core.impl.provider.entity.FileProvider\n
com.sun.jersey.core.impl.provider.entity.InputStreamProvider\n com.sun.jersey.core.impl.provider.entity.DataSourceProvider\n
com.sun.jersey.core.impl.provider.entity.RenderedImageProvider\n*/* ->\n com.sun.jersey.core.impl.provider.entity.FormProvider\n
com.sun.jersey.core.impl.provider.entity.StringProvider\n com.sun.jersey.core.impl.provider.entity.ByteArrayProvider\n
com.sun.jersey.core.impl.provider.entity.FileProvider\n com.sun.jersey.core.impl.provider.entity.InputStreamProvider\n
com.sun.jersey.core.impl.provider.entity.DataSourceProvider\n com.sun.jersey.core.impl.provider.entity.XMLJAXBElementProvider$General\n
com.sun.jersey.core.impl.provider.entity.ReaderProvider\n com.sun.jersey.core.impl.provider.entity.DocumentProvider\n
com.sun.jersey.core.impl.provider.entity.SourceProvider$StreamSourceReader\n com.sun.jersey.core.impl.provider.entity.SourceProvider$SAXSourceReader\n
com.sun.jersey.core.impl.provider.entity.SourceProvider$DOMSourceReader\n com.sun.jersey.json.impl.provider.entity.JSONJAXBElementProvider$General\n
com.sun.jersey.core.impl.provider.entity.XMLRootElementProvider$General\n com.sun.jersey.core.impl.provider.entity.XMLListElementProvider$General\n
com.sun.jersey.core.impl.provider.entity.XMLRootObjectProvider$General\n com.sun.jersey.core.impl.provider.entity.EntityHolderReader\n
com.sun.jersey.json.impl.provider.entity.JSONRootElementProvider$General\n com.sun.jersey.json.impl.provider.entity.JSONListElementProvider$General\n
com.sun.jersey.json.impl.provider.entity.JacksonProviderProxy\n","errorValues":null}] TEXT
**Edit:1 (changing Headers seems to get ridoff Java error now ,but getting below error now)**
headers = {'Authorization': 'Basic xyz879jjkhhnm', 'Accept-Encoding': '0', 'content-type': 'application/json'}
New error:
{'content-type': 'application/json', 'Accept-Encoding': '0', 'Authorization': 'Basic xyz879jjkhhnm'}
<Response [400]> POST call
[{"code":"invalid_attribute_of_request","description":"Attribute value type is not Integer. actual value = (\"2\")","errorValues":null,"field":"size"}] TEXT
[{"code":"invalid_attribute_of_request","description":"Attribute value type is not Integer. actual value = (\"2\")","errorValues":null,"field":"size"}] CONTENT
Ok,as per the latest change Header change in (EDIT 1) i had to remove the double quotes from post data "SIZE",that solved my error.
headers = {'Authorization': 'Basic xyz879jjkhhnm', 'Accept-Encoding': '0', 'content-type': 'application/json'}
postdata = {
"active": "true",
"size": "2", --> Changed to "size": 2,
"ctr": {
"user": "Admin",
"id": "1234"}}