I have a GitLab API (v4) that I need to call to get a project sub-directory (something apparently new in v.14.4, it seems not yet included python-gitlab libs), which in curl can be done with the following command:
curl --header "PRIVATE-TOKEN: A_Token001" http://192.168.156.55/api/v4/projects/10/repository/archive?path=ProjectSubDirectory --output ~./temp/ProjectSubDirectory.tar.gz
The issue is in the last part, the --output ~./GitLab/some_project_files/ProjectSubDirectory.tar.gz
I tried different methods (.content, .text) which failed, as:
...
response = requests.get(url=url, headers=headers, params=params).content
# and save the respon content with with open(...)
but in all the cases it saved a non-valid tar.gz file, or other issues.
I even tried https://curlconverter.com/, but the code it generates does not work as well, it seems ignoring precisely the --output parameter, not showing anything about the file itself:
headers = {'PRIVATE-TOKEN': 'A_Token001',}
params = (('path', 'ProjectSubDirectory'),)
response = requests.get('http://192.168.156.55/api/v4/projects/10/repository/archive', headers=headers, params=params)
For now, I just created a script and call it with sub-process, but I don't like much this approach due to Python has libraries, as requests, that I guess should have some way to do the same...
2 key things.
Allow redirects
Use raise_for_status() to make sure the request was successful before writing the file. This will help uncover other potential issues, like failed authentication.
After that write response.content to a file opened in binary mode for writing ('wb')
import requests
url = "https://..."
headers = {} # ...
paramus = {} # ...
output_path = 'path/to/local/file.tar.gz'
response = requests.get(url, headers=headers, params=params, allow_redirects=True)
response.raise_for_status() # make sure the request is successful
with open(output_path, 'wb') as f:
f.write(response.content)
Related
There's a Python-Eve installation that serves files using endpoints.
The endpoint looks like /smartapi/datafiles/5f4deca38cc01b4384f68529
The Python-Eve installation is not maintained by me.
I can download the file using my credentials successfully using curl.
curl -u username:password https://thepythoneve.installation/smartapi/datafiles/5f4deca38cc01b4384f68529 --output test.xlsx
However, when I tried to use python requests, I keep getting 401 unauthorized access
auth = HTTPBasicAuth(settings.username, settings.password)
with requests.get(url, auth=auth, stream=True) as response:
response.raise_for_status()
with open(file_path, "wb") as out_file:
for chunk in response.iter_content(chunk_size=1024 * 1024): # 1MB chunks
out_file.write(chunk)
logger.info("Download finished successfully")
return (response, file_path)
I will keep failing at the raise_for_status
I hope not to use pycurl. I like to still use requests What am I missing from my code to make this work?
I'm trying to upload a zipfile to a Server using Python requests. The upload works fine. However the uploaded file cannot be opened using Windows Explorer or ark. I suppose there's some problem with mime-type or content-Length.
Oddly, uploading the file using curl, does not seem to cause the same problem.
Here is my python code for the request:
s = requests.Session()
headers = {'Content-Type': 'application/zip'}
zip = open('file.zip', 'rb')
files = {'file': ('file.zip', zip, 'application/zip')}
fc = {'Content-Disposition': 'attachment; filename=file.zip'}
headers.update(fc)
r = requests.Request('POST', url, files=files, headers=headers, auth=(user, password))
prepared = r.prepare()
resp = s.send(prepared)
This is the curl code, which works flawlessly:
curl -X POST \
-ik \
-u user:password \
--data-binary '#file.zip' \
-H 'Content-Type: application/zip' \
-H "Content-Disposition: attachment; filename=file.zip" \
url
Uploading the file works in both, the Server also seems to recognize the content-type. However the file is rendered invalid when re-downloading. The zifile is readable before sending via requests or after sending with normal curl, using --data-binary.
Opening the downloaded zifile with unip or file-roller works either way.
EDIT:
I was uploading two files successively. Oddly the error was fixed when uploading the exact same files in reverse order.
This has NOT been a python problem. When trying with standard curl
I must have accidentally reversed the order, which is why it has been working.
I can not explain this behavior nor do I have a fix for it.
In conclusion: Uploading the bigger file first did the trick.
All of the above seems to be applicable in curl, pycurl and python requests, so I assume it's some kind of bug in one of the curl libraries.
I am trying to use the requests library in Python to upload a file into Fedora commons repository on localhost. I'm fairly certain my main problem is not understanding open() / read() and what I need to do to send data with an http request.
def postBinary(fileName,dirPath,url):
path = dirPath+'/'+fileName
print('to ' + url + '\n' + path)
openBin = {'file':(fileName,open(path,'rb').read())}
headers = {'Slug': fileName} #not important
r = requests.put(url, files=openBin,headers=headers, auth=HTTPBasicAuth('username', 'pass'))
print(r.text)
print("and the url used:")
print(r.url)
This will successfully upload a file in the repository, but it will be slightly larger and corrupted after. For example an image that was 6.6kb became 6.75kb and was not openable anymore.
So how should I properly open and upload a file using put in python?
###Extra details:###
When I replace files=openBin with data=openBin I end up with my dictionary and I presume the data as a string. I don't know if that information is helpful or not.
"file=FILE_NAME.extension&file=TYPE89a%24%02Q%03%E7%FF%00E%5B%19%FC%....
and the size of the file increases to a number of megabytes
I am using specifically put because the Fedora RESTful HTTP API end point says to use put.
The following command does work:
curl -u username:password -H "Content-Type: text/plain" -X PUT -T /path/to/someFile.jpeg http://localhost:8080/fcrepo/rest/someFile.jpeg
Updated
Using requests.put() with the files parameter sends a multipart/form-data encoded request which the server does not seem to be able to handle without corrupting the data, even when the correct content type is declared.
The curl command simply performs a PUT with the raw data contained in the body of the request. You can create a similar request by passing the file data in the data parameter. Specify the content type in the header:
headers = {'Content-type': 'image/jpeg', 'Slug': fileName}
r = requests.put(url, data=open(path, 'rb'), headers=headers, auth=('username', 'pass'))
You can vary the Content-type header to suit the payload as required.
Try setting the Content-type for the file.
If you are sure that it is a text file then try text/plain which you used in your curl command - even though you would appear to be uploading a jpeg file? However, for a jpeg image, you should use image/jpeg.
Otherwise for arbitrary binary data you can use application/octet-stream:
openBin = {'file': (fileName, open(path,'rb'), 'image/jpeg' )}
Also it is not necessary to explicitly read the file contents in your code, requests will do that for you, so just pass the open file handle as shown above.
I am uploading a file to server using requests lib in Python. I read its documentation and some stackoverflow questions and implemented following code:
url = "http://example.com/file.csv"
id = "user-id"
password = "password"
headers = {'content-type': 'application/x-www-form-urlencoded'}
with open(file_path, 'rb') as f:
response = requests.post(url=url, files={'file':f}, auth=HTTPBasicAuth(username=id, password=password),headers=headers)
But this code is not working, response.status_code returning 405 and response.reason returning Method Not Allowed. When i upload file using curl command on terminal it works fine and file gets uploaded:
curl -u user-id:password -T file/path/on/local/machine/file.csv "http://example.com/file.csv"
Can someone please help here.
Related question here. In reality, curl --upload-file performs a PUT not a POST. If you want to mimic what curl does, you might want to try:
with open(file_path, 'rb') as f:
response = requests.put(url=url, files={'file':f}, auth=HTTPBasicAuth(username=id, password=password), headers=headers)
I am trying to trigger some builds using a shell script by doing the following :
export url='http://test.com';
export job_name='MY_JOB_NAME';
jso="{\"parameter\": [{\"name\":\"BRANCH\",\"value\":\"master\"}, {\"name\":\"GITURL\",\"value\":\"https://github.test.com/test/test.git\"}]}";
curl $url/job/$job_name/build --data-urlencode json="$jso";
This works fine, but when I try to convert it to a python equivalent, it doesn't seem to trigger the URL:
import requests
import json
url='http://test.com/job/MY_JOB_NAME/build'
params={'name':'release_1.5', 'GITURL':'https://github.test.com/test/test.git'}
payload = json.dumps(params)
resp = requests.get(url=url, data=payload)
This executes without any error, but it does not trigger a build on my CI machine.
There are quite a few things you're doing wrong here. The first thing which I hope is apparent is that the JSON data you're sending is completely different.
Beyond that, the primary issue you're having here is that your curl is doing a POST with urlencoded data in the BODY, and your python request is doing a GET with urlencoded data as separate parameters in the url. Change your .get to a .post, and the params= to data= and you should be a whole lot closer to your intended goal.
resp = requests.post(url=url, data={'json':payload})
Also note, I embedded your payload into a key as json, as that's what is happening in your curl. I'm not fully aware of your implementation details, but I hope that this helped put you on the right track.
Add the content-type to the header of the request, i.e.:
headers = {'content-type': 'application/json'}
...
resp = requests.get(url=url, params=payload, headers=headers)