Using CURL I can post a file like
CURL -X POST -d "pxeconfig=`cat boot.txt`" https://ip:8443/tftp/syslinux
My file looks like
$ cat boot.txt
line 1
line 2
line 3
I am trying to achieve the same thing using requests module in python
r=requests.post(url, files={'pxeconfig': open('boot.txt','rb')})
When I open the file on server side, the file contains
{:filename=>"boot.txt", :type=>nil, :name=>"pxeconfig",
:tempfile=>#<Tempfile:/tmp/RackMultipart20170405-19742-1cylrpm.txt>,
:head=>"Content-Disposition: form-data; name=\"pxeconfig\";
filename=\"boot.txt\"\r\n"}
Please suggest how I can achieve this.
Your curl request sends the file contents as form data, as opposed to an actual file! You probably want something like
with open('boot.txt', 'rb') as f:
r = requests.post(url, data={'pxeconfig': f.read()})
The two actions you are performing are not the same.
In the first: you explicitly read the file using cat and pass it to curl instructing it to use it as the value of a header pxeconfig.
Whereas, in the second example you are using multipart file uploading which is a completely different thing. The server is supposed to parse the received file in that case.
To obtain the same behavior as the curl command you should do:
requests.post(url, data={'pxeconfig': open('file.txt').read()})
For contrast the curl request if you actually wanted to send the file multipart encoded is like this:
curl -F "header=#filepath" url
with open('boot.txt', 'rb') as f: r = requests.post(url, files={'boot.txt': f})
You would probably want to do something like that, so that the files closes afterwards also.
Check here for more: Send file using POST from a Python script
Related
I'm trying to upload a secure file to my repository in GitLab.
While I am able to upload a secure file with curl, I encounter an error when using requests in Python.
my python code:
r = requests.post("https://gitlab.com/api/v4/projects/10186699/secure_files",
headers={"PRIVATE-TOKEN": "glpat-TH7FM3nThKmHgOp"},
files={"file": open("/Users/me/Desktop/dev/web-server/utils/a.txt", "r"),
"name": "a.txt"})
print(r.status_code,r.json())
Response:
400 {'error': 'name is invalid'}
The equivalent curl command I use that actually works:
curl --request POST --header "PRIVATE-TOKEN: glpat-TH7FM3nThKmHgOp" https://gitlab.com/api/v4/projects/10186699/secure_files --form "name=a.txt" --form "file=#/Users/me/Desktop/dev/web-server/utils/a.txt"
The equivalent call will be
import requests
resp = requests.post(
"https://gitlab.com/api/v4/projects/10186699/secure_files",
headers={"PRIVATE-TOKEN": "glpat-TH7FM3nThKmHgOp"},
files={"file": open("/Users/me/Desktop/dev/web-server/utils/a.txt", "rb")},
data={"name": "a.txt"}
)
print(resp.status_code,resp.json())
This is because the file= parameter is intended only for uploading files. On the other hand, name is your form data (you need to pass in the data= parameter).
It's also recommended to open files in binary mode. (docs)
What I am trying to accomplish is to download a specific portion of a video file using python. Sort of what a browser will do when playing a video. If the file is 1000 Bytes, I want to download from byte 200 to 700. I know that I can download the file in parts using the method below:
file_ = open(filename, 'wb')
res = requests.get(url, stream=True)
for chunk in res.iter_content(amount):
file_.write(chunk)
file_.close()
How can I modify this code to accomplish that?
The server has to support this:
If Accept-Ranges is present in HTTP responses (and its value isn't
none), the server supports range requests. You can check this by
issuing a HEAD request.
If the server supports it you can request the part as
curl http://i.imgur.com/z4d4kWk.jpg -i -H "Range: bytes=0-1023"
Source: https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests
Also take a look at this.
I am sending file via requests.post method in python 3.7. The code is something as simple as the following,
with open('filename','rb') as data:
r = requests.post(url, data)
The request is sent to a handler created on AWS Lambda, where the file should then be stored in other services. The body of the event seems to be an encoded string of the file object and I can't find a way to decode it.
Thanks guys!
What you're trying to do is not a great ideia. Lambda has Invocation payload limit of 6MB, so you can't send large files like this.
The best way is to use boto3 appropriate function to upload files directly to S3
If you really want to use requests.post, open the file as a string and send It via post, something like that:
with open('file.txt', 'r') as file:
STRING_FILE = file.read().replace('\n', '')
r = requests.post(<URL>, data = {'key':STRING_FILE})
I am trying to use the requests library in Python to upload a file into Fedora commons repository on localhost. I'm fairly certain my main problem is not understanding open() / read() and what I need to do to send data with an http request.
def postBinary(fileName,dirPath,url):
path = dirPath+'/'+fileName
print('to ' + url + '\n' + path)
openBin = {'file':(fileName,open(path,'rb').read())}
headers = {'Slug': fileName} #not important
r = requests.put(url, files=openBin,headers=headers, auth=HTTPBasicAuth('username', 'pass'))
print(r.text)
print("and the url used:")
print(r.url)
This will successfully upload a file in the repository, but it will be slightly larger and corrupted after. For example an image that was 6.6kb became 6.75kb and was not openable anymore.
So how should I properly open and upload a file using put in python?
###Extra details:###
When I replace files=openBin with data=openBin I end up with my dictionary and I presume the data as a string. I don't know if that information is helpful or not.
"file=FILE_NAME.extension&file=TYPE89a%24%02Q%03%E7%FF%00E%5B%19%FC%....
and the size of the file increases to a number of megabytes
I am using specifically put because the Fedora RESTful HTTP API end point says to use put.
The following command does work:
curl -u username:password -H "Content-Type: text/plain" -X PUT -T /path/to/someFile.jpeg http://localhost:8080/fcrepo/rest/someFile.jpeg
Updated
Using requests.put() with the files parameter sends a multipart/form-data encoded request which the server does not seem to be able to handle without corrupting the data, even when the correct content type is declared.
The curl command simply performs a PUT with the raw data contained in the body of the request. You can create a similar request by passing the file data in the data parameter. Specify the content type in the header:
headers = {'Content-type': 'image/jpeg', 'Slug': fileName}
r = requests.put(url, data=open(path, 'rb'), headers=headers, auth=('username', 'pass'))
You can vary the Content-type header to suit the payload as required.
Try setting the Content-type for the file.
If you are sure that it is a text file then try text/plain which you used in your curl command - even though you would appear to be uploading a jpeg file? However, for a jpeg image, you should use image/jpeg.
Otherwise for arbitrary binary data you can use application/octet-stream:
openBin = {'file': (fileName, open(path,'rb'), 'image/jpeg' )}
Also it is not necessary to explicitly read the file contents in your code, requests will do that for you, so just pass the open file handle as shown above.
I'm tying to write some simple app on python3 and tornado for server, and requests for client, and I'm getting some headers in 'self.request.body', which I can't dispose of. For instance, for file 'blahblahblah', I get:
--cb5f6ba84bdf42d382dfd3204f6307c7\r\nContent-Disposition: form-data; name="file"; filename="1.bin"\r\n\r\nblahblahblah\n\r\n--cb5f6ba84bdf42d382dfd3204f6307c7--\r\n
Files are sent by
f = {'file': open(FILE, 'rb')}
requests.post(URL_UPLOAD, files=f)
and received by
class UploadHandler(tornado.web.RequestHandler):
def post(self, filename):
with open(Dir + filename, 'wb') as f:
f.write(self.request.body)
My full code can be seen here
When I send the file by curl with curl -X POST -d $(cat ./1.bin) http://localhost:8080/upload/1.bin I get the correct file, but without \n.
There must be something I missed. Please can someone help me with that? Thank You.
There are two ways to upload files: simply using the file as the request body (usually, but not necessarily, with the HTTP PUT method), or using a multipart wrapper (usually with the HTTP POST method). If you upload the file from an HTML form, it will usually use the multipart wrapper. Your requests example is using a multipart wrapper and the curl one is not; your server is not expecting the wrapper.
To use a multipart wrapper: in requests, pass files= as you've done here. With curl, see this answer: Using curl to upload POST data with files. On the server, use self.request.files instead of self.request.body: http://www.tornadoweb.org/en/stable/httpserver.html#tornado.httpserver.HTTPRequest.files
To not use the multipart wrapper, use data=open(FILE, 'rb').read() from requests, and keep the other two components the same.
It is possible to support both styles simultaneously on the server: use self.requests.files when self.request.headers['Content-Type'] == 'multipart/form-data' and self.request.body otherwise.