Develop Reference

Upload secure files to GitLab using requests python module

I'm trying to upload a secure file to my repository in GitLab.
While I am able to upload a secure file with curl, I encounter an error when using requests in Python.
my python code:
r = requests.post("https://gitlab.com/api/v4/projects/10186699/secure_files",
headers={"PRIVATE-TOKEN": "glpat-TH7FM3nThKmHgOp"},
files={"file": open("/Users/me/Desktop/dev/web-server/utils/a.txt", "r"),
"name": "a.txt"})
print(r.status_code,r.json())
Response:
400 {'error': 'name is invalid'}
The equivalent curl command I use that actually works:
curl --request POST --header "PRIVATE-TOKEN: glpat-TH7FM3nThKmHgOp" https://gitlab.com/api/v4/projects/10186699/secure_files --form "name=a.txt" --form "file=#/Users/me/Desktop/dev/web-server/utils/a.txt"

The equivalent call will be
import requests
resp = requests.post(
"https://gitlab.com/api/v4/projects/10186699/secure_files",
headers={"PRIVATE-TOKEN": "glpat-TH7FM3nThKmHgOp"},
files={"file": open("/Users/me/Desktop/dev/web-server/utils/a.txt", "rb")},
data={"name": "a.txt"}
)
print(resp.status_code,resp.json())
This is because the file= parameter is intended only for uploading files. On the other hand, name is your form data (you need to pass in the data= parameter).
It's also recommended to open files in binary mode. (docs)

Trying to upload a file via api

I want to programmatically upload a pdf file to an APP via API, from a Python + Django APP.
Their documentation says:
“In order to upload an invoice you must do a POST as from.data with key type “file” and value fiscal_document that refers to the file you are attaching.”
curl -X POST https://api.mercadolibre.com/packs/$PACK_ID/fiscal_documents
-H 'Content-Type: multipart/form-data' \
-H 'Authorization: Bearer $ACCESS_TOKEN' \
-F 'fiscal_document=#/home/user/.../Factura_adjunta.pdf'
I´m trying to achieve this using requests as follows:
response = requests.request("POST", url, headers=headers, files={'fiscal_document': my_file_url}).json()
But I get the following response
{
"message": "File cannot be empty",
"error": "bad_request",
"status": 400,
"cause": []
}
Is my request call ok?
In the example there is a local path declared, but I need to get the file from an URL. Should I keep the “#” in the -F 'fiscal_document=#/home/user/.../Factura_adjunta.pdf'?
Should I upload a byte type?
And clues welcome. Thanks in advance.
UPDATE 1
Following the suggestions I tried the following:
with open(esta_operacion.factura_pdf.url, 'r') as myFile:
response = requests.request("POST", url, headers=headers, files={'fiscal_document': myFile}).json()
myFile.close()
But I get [Errno 2] No such file or directory.......
The file exists and I can download it.

I can't reproduce the problem, but you are sending the URL string not the content of it. You should read the URL first and then upload the content.
The following code will provide this:
from urllib.request import urlopen
with urlopen(my_file_url) as f:
my_file = f.read()
response = requests.request("POST", url, headers=headers, files={'fiscal_document': my_file})

Convert Grobid curl command to requests in Python

I'm trying to convert curl script to parse pdf file from grobid server to requests in Python.
Basically, if I run the grobid server as follows,
./gradlew run
I can use the following curl to get the output of parsed XML of an academic paper example.pdf as below
curl -v --form input=#example.pdf localhost:8070/api/processHeaderDocument
However, I don't know the way to convert this script into Python. Here is my attempt to use requests:
GROBID_URL = 'http://localhost:8070'
url = '%s/processHeaderDocument' % GROBID_URL
pdf = 'example.pdf'
xml = requests.post(url, files=[pdf]).text

I got the answer. Basically, I missed api in the GROBID_URL and also the input files should be a dictionary instead of a list.
GROBID_URL = 'http://localhost:8070'
url = '%s/api/processHeaderDocument' % GROBID_URL
pdf = 'example.pdf'
xml = requests.post(url, files={'input': open(pdf, 'rb')}).text

Here is an example bash script from http://ceur-ws.bitplan.com/index.php/Grobid. Please note that there is also a ready to use python client available. See https://github.com/kermitt2/grobid_client_python
#!/bin/bash
# WF 2020-08-04
# call grobid service with paper from ceur-ws
v=2644
p=44
vol=Vol-$v
pdf=paper$p.pdf
if [ ! -f $pdf ]
then
wget http://ceur-ws.org/$vol/$pdf
else
echo "paper $p from volume $v already downloaded"
fi
curl -v --form input=#./$pdf http://grobid.bitplan.com/api/processFulltextDocument > $p.tei

Python requests zip upload makes zipfile unreadable in Windows

I'm trying to upload a zipfile to a Server using Python requests. The upload works fine. However the uploaded file cannot be opened using Windows Explorer or ark. I suppose there's some problem with mime-type or content-Length.
Oddly, uploading the file using curl, does not seem to cause the same problem.
Here is my python code for the request:
s = requests.Session()
headers = {'Content-Type': 'application/zip'}
zip = open('file.zip', 'rb')
files = {'file': ('file.zip', zip, 'application/zip')}
fc = {'Content-Disposition': 'attachment; filename=file.zip'}
headers.update(fc)
r = requests.Request('POST', url, files=files, headers=headers, auth=(user, password))
prepared = r.prepare()
resp = s.send(prepared)
This is the curl code, which works flawlessly:
curl -X POST \
-ik \
-u user:password \
--data-binary '#file.zip' \
-H 'Content-Type: application/zip' \
-H "Content-Disposition: attachment; filename=file.zip" \
url
Uploading the file works in both, the Server also seems to recognize the content-type. However the file is rendered invalid when re-downloading. The zifile is readable before sending via requests or after sending with normal curl, using --data-binary.
Opening the downloaded zifile with unip or file-roller works either way.
EDIT:
I was uploading two files successively. Oddly the error was fixed when uploading the exact same files in reverse order.
This has NOT been a python problem. When trying with standard curl
I must have accidentally reversed the order, which is why it has been working.
I can not explain this behavior nor do I have a fix for it.
In conclusion: Uploading the bigger file first did the trick.
All of the above seems to be applicable in curl, pycurl and python requests, so I assume it's some kind of bug in one of the curl libraries.

equivalent python requests call to CURL request to upload image

I'm using Python requests module, but whatever I've tried to upload image, it succeeds, but image has errors when opening/reading.
I encode the image as base64, set content-type headers (image/png, image/jpeg etc...) etc...
Anyhow, I do the following using CURL and it works:
curl -u test#test.ca:test -H 'Content-Type: image/jpeg' --data-binary #test.jpeg -X POST 'https://test.test.com/api/upload.json?filename=test.jpeg'
What would be the equivalent of this request with the requests module in python (headers etc...)?

To reproduce your curl command, you don't need to encode the image in base64: --data-binary #test.jpeg curl option sends test.jpeg file as is:
import requests
r = requests.post('https://example.com/api/upload.json?filename=test.jpeg',
data=open('test.jpeg', 'rb'),
headers={'Content-Type': 'image/jpeg'},
auth=('test#test.ca', 'test')) # username, password

headers = {'Content-Type' : 'image/jpeg'}
params = {'filename' : 'test.jpg'}
r = requests.post("https://test.test.com/api/upload.json",
auth=('user','pw'), headers=headers, params=params)