This bounty has ended. Answers to this question are eligible for a +50 reputation bounty. Bounty grace period ends in 19 hours.
user19926715 is looking for an up-to-date answer to this question:
I would like an example that shows the before and after of the packet before it gets sent and the response after and showing proof of it going into a repository and explain what was different from the code given by me.
I have a file called w.txt that holds 4 random numbers
I am trying to push it to a git repo and I have the following code:
aa=open(fileName,'r').read()
print(aa)
text= base64.b64encode(aa.encode("utf-8"))
api="https://api.github.com/repos/Ixonblitz-MatOS/Worm/contents/w.txt"
headers = {
"Authorization": f'''Bearer {token}''',
"Content-type": "application/vnd.github+json"
}
data = {
"message": msg, # Put your commit message here.
"content": text.decode("utf-8")
}
r = requests.put(api, headers=headers, json=data)
print(r.text)
In this case fileName is "w.txt" and the api is the api for my repo and the correct token is being used. What am I missing? It is leaving the file on my repo empty instead of putting the numbers there.
You need to provide the SHA of the file as the error message says.
To get the SHA, you can get the file like this:
import requests
url = "https://api.github.com/repos/Ixonblitz-MatOS/Worm/contents/w.txt"
headers = {
'Accept': 'application/vnd.github+json',
'Authorization': 'Bearer <token>',
}
response = requests.request("GET", url, headers=headers)
data = response.json()
The SHA will be at data['sha'].
Then you need to modify your request to include the SHA:
data = {
"message": msg, # Put your commit message here.
"content": text.decode("utf-8"),
"sha": data['sha']
}
The /repos/{owner}/{repo}/contents/{path} API route to fetch files has some size limits. See below:
If the requested file's size is:
1 MB or smaller: All features of this endpoint are supported.
Between
1-100 MB: Only the raw or object custom media types are supported.
Both will work as normal, except that when using the object media
type, the content field will be an empty string and the encoding field
will be "none". To get the contents of these larger files, use the raw
media type.
Greater than 100 MB: This endpoint is not supported.
For larger files, you will have to clone the repository and then calculate the SHA locally.
Why don't you do it this way.
from git import Repo
PATH_OF_GIT_REPO = r'path\to\your\project\folder\.git' # make sure .git folder is properly configured
COMMIT_MESSAGE = 'comment from python script'
def git_push():
try:
repo = Repo(PATH_OF_GIT_REPO)
repo.git.add(update=True)
repo.index.commit(COMMIT_MESSAGE)
origin = repo.remote(name='origin')
origin.push()
except:
print('Some error occured while pushing the code')
git_push()
Based on the code you've provided, it seems that you are encoding the content of the file as base64 before pushing it to your Git repository. While this is a valid approach, you need to make sure that you are decoding the content before writing it to the file on the Git repository. Here's a modified version of your code that should work:
import base64
import requests
fileName = "w.txt"
msg = "Updated w.txt with new numbers"
token = "<your GitHub access token>"
with open(fileName, 'r') as f:
content = f.read()
encoded_content = base64.b64encode(content.encode("utf- 8")).decode("utf-8")
api_url = "https://api.github.com/repos/Ixonblitz-MatOS/Worm/contents/w.txt"
headers = {
"Authorization": f"Bearer {token}",
"Content-Type": "application/json",
"Accept": "application/vnd.github+json"
}
data = {
"message": msg,
"content": encoded_content
}
response = requests.put(api_url, headers=headers, json=data)
if response.status_code == 200:
print("Successfully pushed the file to Git repository.")
else:
print(f"Error pushing file to Git repository. Response status code: {response.status_code}")
The above code reads the content of the file into the content variable and then encodes it as base64 using the base64.b64encode method. It then sends a PUT request to the GitHub API to create or update the file in your Git repository. The data payload contains the encoded content of the file along with a commit message. The response status code is then checked to verify whether the operation was successful.
Note that you may also want to make sure that the headers dictionary contains the correct User-Agent field as per GitHub API requirements.
Related
It's my first time posting here, and I want to ask you if you can help me to find a way to send a picture in a channel with the Discord API with Python Requests. (I'm not talking here about discord.py or Discord Bots).
To write the code, I first checked in the Network Tab and upload a picture to see what request it's actually sending.
In the form data, I can see that it's sending two informations:
file: (binary)
payload_json: {"content": "", "tts": false}
The last thing who is needed is to send our Discord Token in the header
When I tried to write the code, the response they gave me is
{"message": "Cannot send an empty message", "code": 50006}
Even if I add a text in the content, they give me the same error.
I guess I didn't put the right information in the data.
Here's the code:
import requests
# User's Token
header = {
'authorization': "token",
}
# Data
payload = {
"file" : open("picture.jpg", "rb").read(), # The picture that we want to send in binary
"payload_json": {"content":"","tts":False},
}
channel_id = "829628868860051480" # Channel where we send the picture
r = requests.post(f"https://discord.com/api/v9/channels/{channel_id}/messages", data=payload, headers=header).text
print(r)
Thanks to isopach, I figured out what to write, I post it here in case someone need to do the same thing as me
import requests
# User's Token
header = {
'authorization': "token",
}
# File
files = {
"file" : ("./picture.jpg", open("./picture.jpg", 'rb')) # The picture that we want to send in binary
}
# Optional message to send with the picture
payload = {
"content":"message"
}
channel_id = "channel_id" # Channel where we send the picture
r = requests.post(f"https://discord.com/api/v9/channels/{channel_id}/messages", data=payload, headers=header, files=files)
You'll have to send the file as a multipart/form-data according to the docs.
Put them in a tuple nested in a dict.
files = {
'file': ('./picture.jpg', open('./picture.jpg', 'rb')),
}
r = requests.post(f"https://discord.com/api/v9/channels/{channel_id}/messages", data=payload, headers=header, files=files).text
I'm trying to upload a image to google drive using requests, but it's not working because the requests keep giving me a status 401 (wrong credentials). I am using the access token that was given to me, so I don't know what is going on.
There's my code:
tokendrive = TOKEN_DRIVE
url = "site_url"
myid = MY_ID
subid = MY_SUB_ID
r = requests.get(url)
headers = {'Authorization': 'Bearer ' + tokendrive}
para = {"name": submission.id + ".png",
"parents": [myid]}
files = {"data": ("metadata", json.dumps(para), "application/json; charset=UTF-8"),
"file": io.BytesIO(requests.get(url).content)}
response = requests.post("https://www.googleapis.com/upload/drive/v3/files",headers=headers,files=files)
print(response.text)
I believe your goal and your current situation as follows.
You want to upload a file to Google Drive.
You want to delete a file on Google Drive.
You want to achieve this using request module of python.
From your endpoint, you want to use Drive API v3.
Your access token can be used for uploading and deleting the file using Drive API.
Modification points:
If your access token can be used for uploading a file to Google Drive, it is required to modify the script for using the access token. In this case, please modify Token to Bearer. I thought that the reason of your error might be due to this.
When Drive API v3 is used, the property of parents is "parents": [myid]. And in the current stage, please use one folder ID here.
In the case of Drive API v3, the filename can be given with name instead of title. title is used for Drive API v2.
When above points are reflected to your script, it becomes as follows.
Modified script:
This modified script uploads a file of to Google Drive. Before you use this script, please confirm the variables you want to use, again.
headers = {'Authorization': f'Bearer {tokendrive}'} # or 'Bearer ' + tokendrive
para = {
"name": "image_url.jpg",
"parents": [myid]
}
files = {
"data": ("metadata", json.dumps(para), "application/json; charset=UTF-8"),
"file": io.BytesIO(requests.get(submission.url).content)
}
response = requests.post(
"https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart",
headers=headers,
files=files
)
print(response.text)
In this script, the following value is returned.
{
"kind": "drive#file",
"id": "###",
"name": "image_url.jpg",
"mimeType": "image/jpeg"
}
In this modified script, the maximum file size for uploading is 5 MB. When you want to upload a file over 5 MB, please use resumable upload. In that case, I think that this thread might be useful. Ref
Sample script:
This sample script deletes a file on Google Drive.
fileId = '###' # Please set the file ID you want to delete.
headers = {'Authorization': f'Bearer {tokendrive}'} # or 'Bearer ' + tokendrive
response = requests.delete(
"https://www.googleapis.com/drive/v3/files/" + fileId,
headers=headers,
)
print(response.text)
In this case, no value is returned. This is the current specification.
IMPORTANT: This script completely deletes the file of fileId. So please be careful this. I would like to recommend to test using a sample file.
References:
Upload file data
Files: delete
I am having the problem that the requests library for some reason is making the payload bigger and causing issues. I enabled http logging, and in in the output I can see the content length being 50569, not 50349, as the actual file size should indicate.
send: b'POST /api/1/images HTTP/1.1\r\nHost: localhost:8000\r\nUser-Agent: python-requests/2.21.0\r\nAccept-Encoding: gzip, deflate\r\nAc
cept: application/json\r\nConnection: keep-alive\r\nAuthorization: Bearer 28956340ba9c7e25b49085b4d273522b\r\ncontent-type: image/png\r\n
Content-Length: 50569\r\n\r\n'
send: b'--ac9e15d6d3aa3a77506c2daccca2ee47\r\nContent-Disposition: form-data; name="0007-Alternate-Lateral-Pulldown_back-STEP1"; filename
="0007-Alternate-Lateral-Pulldown_back-STEP1.png"\r\n\r\n\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\xf0\x00\x00\x00\xf0\x08\x06\x00\
x00\x00>U\xe9\x92\x00\x00\x00\tpHYs\x00\x00\x0b\x13\x00\x00\x0b\x13\x01\x00\x9a\x9c\x18\x00\x00FMiTXtXML:com.adobe.xmp\x00\x00\x00\x00\x0
0<?xpacket begin="\xef\xbb\xbf" id="W5M0MpCehiHzreSzNTczkc9d"?>\n<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="Adobe XMP Core 5.5-c021 79.
155772, 2014/01/13-19:44:00 ">\n <rdf:RDF xmlns:rdf="http:
Chrome has exactly the same headers when sending this, but the correct content length, so I am assuming this is why my server complains of a invalid image being sent.
This is my code
url = self.server + "/api/1/images";
headers = self.default_headers()
headers['content-type'] = 'image/png'
# neither of these are actually used for anything
filename = os.path.basename(image)
field_name = os.path.splitext(filename)[0]
files = {field_name: (filename, open(image, 'rb'), '')}
# Post image
r = requests.post(url, headers=headers, files=files, timeout = 5.0)
As can be seen, I am using the b flag when opening the file to preserve the binary content, so it should not change.
File size is 50349
$ ls -l 0007-Alternate-Lateral-Pulldown_back-STEP1.png
-rw-rw-r--# 1 carlerik staff 50349 Nov 26 2019 0007-Alternate-Lateral-Pulldown_back-STEP1.png
I used Charles proxy to dig into this and I now have gotten to the bottom of this. There are basically two things to note:
The difference in content length between the request sent by Chrome and Requests is exactly the length of the boundary fields (a multipart form concept) before and after the file + the six CRLF (\r\n) sequences + the Content-Disposition header.
echo '50569-178-36-6' | bc
50349
The boundary field looks like this: --ac9e15d6d3aa3a77506c2daccca2ee47\r\n
You can also see from the HTTP header and body dump that the header is actually in the body and after the boundary field, not as part of the normal headers. This was important and led me on the right path.
The second part of the answer is that the guys that wrote the server API I am interfacing with did not understand/read the HTTP spec for the exact bits they ask for: the Content-Disposition header.
The API docs for .../images state that this header must be present always, as they use (well, used in the past) its content to extract filenames and such. The problem is that the way they use it is not in accordance with how it is intended to be used: in a multipart HTTP request it is part of the body of the HTTP request, describing the part (form field) of the request it precedes.
This is, of course, also how Requests uses it, but I did not have this information before venturing into this abyss, so I was misinformed by the code in the controller that states this in its doc. So I assumed that Requests would put the header in the header section, which it did not, and not the body, which it did. After all, I saw that Chrome "did the right thing", but it turned out that was only because these requests were handcrafted in javascript:
apiService.js
/**
* Upload image
* #param file
* #returns {*}
* #private
*/
_api.postImage = function (file) {
if (typeof file != 'object') {
throw Error('ApiService: Object expected');
}
file.fileName = (/(.*)\./).exec(file.name)[1];
var ContentDisposition = 'form-data; name="' + encodeURI(file.fileName) + '"; filename="' + encodeURI(file.name) + '"';
return Upload.http({
url: Routing.generate('api_post_image'),
headers: {
'Content-Disposition': ContentDisposition,
'Content-Type': file.type
},
data: file
});
};
So the Content-Disposition header here is basically a proprietary header to convey information about the filename, that shares its appearance with the general in-body header from the spec. That means all it takes to fix this is to create a request with a custom body read from file and set this header.
To round this off, this was how it was all simplified down to:
headers = dict()
headers['authorization'] = "<something>"
headers['content-type'] = 'image/png'
with open(image, 'rb') as imagefile:
# POST image
r = requests.post(url, headers=headers, data=imagefile)
I am trying to upload a large file to google storage using chunking as outlined in their tutorial here. I am using Python(Flask) and their JSON REST api since my use case cant work with the existing python packages that are not so well documented. The file chunks are coming from dropzone on the browser frontend.
Below is the code I have (Partial code)
from google.oauth2 import service_account
credentials = service_account.Credentials.from_service_account_file(
filename=os.environ['GOOGLE_APPLICATION_CREDENTIALS'],
scopes=['https://www.googleapis.com/auth/cloud-platform'])
def start_resumable_upload_session(name, mime_type):
"""
Name is the filename for the new object being uploaded
"""
url = f"https://storage.googleapis.com/upload/storage/v1/b/test-bucket-alpha-1/o?uploadType=resumable&name={name}"
headers = {
"X-Upload-Content-Type":mime_type
}
# "X-Upload-Content-Length":"262144"
#prep an authenticated session to make requests
authed_session = AuthorizedSession(credentials)
resp = authed_session.post(url, headers=headers)
if resp.status_code == 200:
return resp.headers.get('Location',None)
else:
return None
authed_session = AuthorizedSession(credentials)
sess_uri = start_resumable_upload_session(file_chunk.filename, file_chunk.content_type)
cn_length = len(file_chunk.read())
tot_size = int(request.form.get("dztotalfilesize"))
headers = {
"Content-Length": str(cn_length),
"Content-Range": f"bytes 0-{str(cn_length-1)}/{str(tot_size-1)}"
}
resp = authed_session.put(sess_uri,data=file_chunk.read(), headers=headers)
The response text is Failed to parse Content-Range header or even when I tried adjusting inputs to debug, no response is produced and the request just times out.
What may I be doing wrong in my logic? I also appreciate links to code snippets that may shed light.
UPDATE - RESOLVED
As pointed out in the comment below, the correct header should be:
headers = {
"Content-Length": str(cn_length),
"Content-Range": f"bytes 0-{str(cn_length-1)}/{str(tot_size)}"
}
i.e if an object is 1000 bytes, your ranges will go from 0-999 but overall size should still be 1000.
Here you can find a sample application which includes a Resumable Uploads in Python, using the google-api-python-client
You might be interested in this line:
media = MediaFileUpload(filename, chunksize=CHUNKSIZE, resumable=True)
if not media.mimetype():
media = MediaFileUpload(filename, DEFAULT_MIMETYPE, resumable=True)
request = service.objects().insert(bucket=bucket_name, name=object_name,
media_body=media)
Additionally, this is another example:
# Create a Resumable Upload
url = (
f'https://www.googleapis.com/upload/storage/v1/b/'
f'{bucket.name}/o?uploadType=resumable'
)
upload = ResumableUpload(
upload_url=url,
chunk_size=chunk_size
)
transport = AuthorizedSession(credentials=client._credentials)
# Start using the Resumable Upload
upload.initiate(
transport=transport,
content_type='application/octet-stream',
stream=stream,
metadata={'name': blob.name}
)
Finally, this similar question was solved by setting right the Content-Length.
I am trying to refer a local jpg file for using in Azure Emotion API.
To do this, I refer my file through "file:///" like below.
body = "{'url': 'file:///Users/jonghkim/dev_jhk/Research/Crowdfunding/Face_Analysis/me.jpg'}"
But the response says "Invalid image URL." How could I fix it?
{"error":{"code":"InvalidUrl","message":"Invalid image URL."}}
Whole code looks like below.
########### Python 2.7 #############
import httplib, urllib, base64
headers = {
# Request headers. Replace the placeholder key below with your subscription key.
'Content-Type': 'application/json',
'Ocp-Apim-Subscription-Key': '***********************',
}
params = urllib.urlencode({
})
# Replace the example URL below with the URL of the image you want to analyze.
body = "{'url': 'file:///Users/jonghkim/dev_jhk/Research/Crowdfunding/Face_Analysis/me.jpg'}"
try:
conn = httplib.HTTPSConnection('westus.api.cognitive.microsoft.com')
conn.request("POST", "/emotion/v1.0/recognize?%s" % params, body, headers)
response = conn.getresponse()
data = response.read()
print(data)
conn.close()
except Exception as e:
print("[Errno {0}] {1}".format(e.errno, e.strerror))
I solved this problem. The true reason was two fold. At first, when we refer local file, we should use 'Content-Type': 'application/octet-stream' in a header.
The second problem is that the image should satisfy the condition of Azure (learn.microsoft.com/ko-kr/azure/cognitive-services/emotion/faq).
Full code is here:
########### Python 2.7 #############
import httplib, urllib, base64
headers = {
# Request headers. Replace the placeholder key below with your subscription key.
'Content-Type': 'application/octet-stream',
'Ocp-Apim-Subscription-Key': '**************************',
}
params = urllib.urlencode({
})
# Replace the example URL below with the URL of the image you want to analyze.
body = open('test.jpg','rb').read()
conn = httplib.HTTPSConnection('westus.api.cognitive.microsoft.com')
conn.request("POST", "/emotion/v1.0/recognize?%s" % params, body, headers)
response = conn.getresponse()
data = response.read()
print(data)
conn.close()
You're executing the emotion API within Cognitive Services - just take a look at the URI. This code is not being executed locally. It's a service. Being run somewhere else.
So, when the service gets the URL (via url in the body), it then needs to reach out to that resource, which is impossible to do if the resource is on your computer. And file:// is going to be an invalid scheme because the service won't be reading from its own file system.
You'll need to have your resource in an accessible place (e.g. a public or SAS-signed blob, an image link from a website, etc).