Develop Reference

Upload secure files to GitLab using requests python module

I'm trying to upload a secure file to my repository in GitLab.
While I am able to upload a secure file with curl, I encounter an error when using requests in Python.
my python code:
r = requests.post("https://gitlab.com/api/v4/projects/10186699/secure_files",
headers={"PRIVATE-TOKEN": "glpat-TH7FM3nThKmHgOp"},
files={"file": open("/Users/me/Desktop/dev/web-server/utils/a.txt", "r"),
"name": "a.txt"})
print(r.status_code,r.json())
Response:
400 {'error': 'name is invalid'}
The equivalent curl command I use that actually works:
curl --request POST --header "PRIVATE-TOKEN: glpat-TH7FM3nThKmHgOp" https://gitlab.com/api/v4/projects/10186699/secure_files --form "name=a.txt" --form "file=#/Users/me/Desktop/dev/web-server/utils/a.txt"

The equivalent call will be
import requests
resp = requests.post(
"https://gitlab.com/api/v4/projects/10186699/secure_files",
headers={"PRIVATE-TOKEN": "glpat-TH7FM3nThKmHgOp"},
files={"file": open("/Users/me/Desktop/dev/web-server/utils/a.txt", "rb")},
data={"name": "a.txt"}
)
print(resp.status_code,resp.json())
This is because the file= parameter is intended only for uploading files. On the other hand, name is your form data (you need to pass in the data= parameter).
It's also recommended to open files in binary mode. (docs)

Trying to upload a file via api

I want to programmatically upload a pdf file to an APP via API, from a Python + Django APP.
Their documentation says:
“In order to upload an invoice you must do a POST as from.data with key type “file” and value fiscal_document that refers to the file you are attaching.”
curl -X POST https://api.mercadolibre.com/packs/$PACK_ID/fiscal_documents
-H 'Content-Type: multipart/form-data' \
-H 'Authorization: Bearer $ACCESS_TOKEN' \
-F 'fiscal_document=#/home/user/.../Factura_adjunta.pdf'
I´m trying to achieve this using requests as follows:
response = requests.request("POST", url, headers=headers, files={'fiscal_document': my_file_url}).json()
But I get the following response
{
"message": "File cannot be empty",
"error": "bad_request",
"status": 400,
"cause": []
}
Is my request call ok?
In the example there is a local path declared, but I need to get the file from an URL. Should I keep the “#” in the -F 'fiscal_document=#/home/user/.../Factura_adjunta.pdf'?
Should I upload a byte type?
And clues welcome. Thanks in advance.
UPDATE 1
Following the suggestions I tried the following:
with open(esta_operacion.factura_pdf.url, 'r') as myFile:
response = requests.request("POST", url, headers=headers, files={'fiscal_document': myFile}).json()
myFile.close()
But I get [Errno 2] No such file or directory.......
The file exists and I can download it.

I can't reproduce the problem, but you are sending the URL string not the content of it. You should read the URL first and then upload the content.
The following code will provide this:
from urllib.request import urlopen
with urlopen(my_file_url) as f:
my_file = f.read()
response = requests.request("POST", url, headers=headers, files={'fiscal_document': my_file})

How to implement a CURL -o in Python using request or aoihttp

I need to do a get to a URL with certain Headers and Parameters in my python code and then write to an excel file. I am trying the below code but getting the below error when I try to open the excel file. Also given below is a redacted version of a CURL I use to test this API endpoint with success that I cannot replicate in my python code
Python Code:
import requests
rheaders = {
"Content-type": "application/json",
"Accept": "application/json",
"Authorization": "Token XXX",
}
url = "https://api.YYY"
r = requests.get(url, headers=rheaders)
with open("test.xlsx" , mode="wb") as output:
output.write(r.content)
Error: Excel cannot open the file 'test.xlsx' because the file format or file extension is not valid. Verify that the file has not been corrupted and that the file extension matches the format of the file.
The assosciated CURL that runs fine in osx terminal and writes the output file without errors
curl -H "Accept: application/json; indent=4" -H 'Content-Type: application/json' -H 'Authorization: Token XXX' https://api.YYY --output ~/Downloads/output.xlsx
I have tried replacing the content-type header with this:
Content-Type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
No luck. Any suggestions? The above is my attempt with python library requests. I have also tried using aiohttp

The issue is that Excel cannot understand the format of the data that is being written into the file, because Python by itself does not support the Excel format. Try using a library such as xlrd to write to it. Alternatively, you could write to a .csv file instead, which wouldn't require any extra libraries and would still be readable by Excel

Python requests zip upload makes zipfile unreadable in Windows

I'm trying to upload a zipfile to a Server using Python requests. The upload works fine. However the uploaded file cannot be opened using Windows Explorer or ark. I suppose there's some problem with mime-type or content-Length.
Oddly, uploading the file using curl, does not seem to cause the same problem.
Here is my python code for the request:
s = requests.Session()
headers = {'Content-Type': 'application/zip'}
zip = open('file.zip', 'rb')
files = {'file': ('file.zip', zip, 'application/zip')}
fc = {'Content-Disposition': 'attachment; filename=file.zip'}
headers.update(fc)
r = requests.Request('POST', url, files=files, headers=headers, auth=(user, password))
prepared = r.prepare()
resp = s.send(prepared)
This is the curl code, which works flawlessly:
curl -X POST \
-ik \
-u user:password \
--data-binary '#file.zip' \
-H 'Content-Type: application/zip' \
-H "Content-Disposition: attachment; filename=file.zip" \
url
Uploading the file works in both, the Server also seems to recognize the content-type. However the file is rendered invalid when re-downloading. The zifile is readable before sending via requests or after sending with normal curl, using --data-binary.
Opening the downloaded zifile with unip or file-roller works either way.
EDIT:
I was uploading two files successively. Oddly the error was fixed when uploading the exact same files in reverse order.
This has NOT been a python problem. When trying with standard curl
I must have accidentally reversed the order, which is why it has been working.
I can not explain this behavior nor do I have a fix for it.
In conclusion: Uploading the bigger file first did the trick.
All of the above seems to be applicable in curl, pycurl and python requests, so I assume it's some kind of bug in one of the curl libraries.

How to POST a local file using urllib2 in Python?

I am a complete Python noob and am trying to run cURL equivalents using urllib2. What I want is a Python script that, when run, will do the exact same thing as the following cURL command in Terminal:
curl -k -F docfile=#myLocalFile.csv http://myWebsite.com/extension/extension/extension
I found the following template on a tutorial page:
import urllib
import urllib2
url = "https://uploadWebsiteHere.com"
data = "{From: 'sender#email.com', To: 'recipient#email.com', Subject: 'Postmark test', HtmlBody: 'Hello dear Postmark user.'}"
headers = { "Accept" : "application/json",
"Conthent-Type": "application/json",
"X-Postmark-Server-Token": "abcdef-1234-46cc-b2ab-38e3a208ab2b"}
req = urllib2.Request(url, data, headers)
response = urllib2.urlopen(req)
the_page = response.read()
but I am completely lost on the 'data' and 'headers' vars. The urllib2 documentation (https://docs.python.org/2/library/urllib2.html) defines the 'data' input as "a string specifying additional data to send to the server" and the 'headers' input as "a dictionary". I am totally out of my depth in trying to follow this documentation and do not see why a dictionary is necessary when I could accomplish this same task in terminal by only specifying the file and URL. Thoughts, please?

The data you are posting doesn't appear to be valid JSON. Assuming the server is expecting valid JSON, you should change that.
Your curl invocation does not pass any optional headers, so you shouldn't need to provide much in the request. If you want to verify the exact headers you could add -vi to the curl invocation and directly match them in the Python code. Alternatively, this works for me:
import urllib2
url = "http://localhost:8888/"
data = '{"From": "sender#email.com", "To": "recipient#email.com", "Subject": "Postmark test", "HtmlBody": "Hello dear Postmark user."}'
headers = {
"Content-Type": "application/json"
}
req = urllib2.Request(url, data, headers)
response = urllib2.urlopen(req)
the_page = response.read()
It probably is in your best interest to switch over to using requests, but for something this simple the standard library urllib2 can be made to work.

What I want is a Python script that, when run, will do the exact same thing as the following cURL command in Terminal:
$ curl -k -F docfile=#myLocalFile.csv https://myWebsite.com/extension...
curl -F sends the file using multipart/form-data content type. You could reproduce it easily using requests library:
import requests # $ pip install requests
with open('myLocalFile.csv','rb') as input_file:
r = requests.post('https://myWebsite.com/extension/...',
files={'docfile': input_file}, verify=False)
verify=False is to emulate curl -k.