Hi am attempting to upload a file to a server when making a post with requests but everytime I get errors. If do the same with cURL it goes through but Im not familiar with cURL or uploading files really, mostly get requests, so I have no clue what its doing differently. I am on windows and am running the below. This is being uploaded to mcafee epo so Im not sure if their api just super picky or what the difference is but every python example ive tried for uploading a file via requests module has failed for me.
url = "https://server.url.com:1234/remote/repository.checkInPackage.do?&allowUnsignedPackages=True&option=Normal&branch=Evaluation"
user = "domain\user"
password = "mypass"
filepath = "C:\\my\\folder\\with\\afile.zip"
with open(filepath, "rb") as f:
file_dict = {"file": f}
response = requests.post(url, auth=(user, password), files=file_dict)
I usually get a error as follows:
'Error 0 :\r\njava.lang.reflect.InvocationTargetException\r\n'
if I use cURL it works though
curl.exe -k -s -u "domain\username:mypass" "https://server.url.com:1234/remote/repository.checkInPackage.do?&allowUnsignedPackages=True&option=Normal&branch=Evaluation" -F file=#"C:\my\folder\with\afile.zip"
I cant really see the difference though and am wondering what is being done differently on the backend for cURL or what I could be doing wrong when using python.
Related
I'm trying to upload a secure file to my repository in GitLab.
While I am able to upload a secure file with curl, I encounter an error when using requests in Python.
my python code:
r = requests.post("https://gitlab.com/api/v4/projects/10186699/secure_files",
headers={"PRIVATE-TOKEN": "glpat-TH7FM3nThKmHgOp"},
files={"file": open("/Users/me/Desktop/dev/web-server/utils/a.txt", "r"),
"name": "a.txt"})
print(r.status_code,r.json())
Response:
400 {'error': 'name is invalid'}
The equivalent curl command I use that actually works:
curl --request POST --header "PRIVATE-TOKEN: glpat-TH7FM3nThKmHgOp" https://gitlab.com/api/v4/projects/10186699/secure_files --form "name=a.txt" --form "file=#/Users/me/Desktop/dev/web-server/utils/a.txt"
The equivalent call will be
import requests
resp = requests.post(
"https://gitlab.com/api/v4/projects/10186699/secure_files",
headers={"PRIVATE-TOKEN": "glpat-TH7FM3nThKmHgOp"},
files={"file": open("/Users/me/Desktop/dev/web-server/utils/a.txt", "rb")},
data={"name": "a.txt"}
)
print(resp.status_code,resp.json())
This is because the file= parameter is intended only for uploading files. On the other hand, name is your form data (you need to pass in the data= parameter).
It's also recommended to open files in binary mode. (docs)
I have a GitLab API (v4) that I need to call to get a project sub-directory (something apparently new in v.14.4, it seems not yet included python-gitlab libs), which in curl can be done with the following command:
curl --header "PRIVATE-TOKEN: A_Token001" http://192.168.156.55/api/v4/projects/10/repository/archive?path=ProjectSubDirectory --output ~./temp/ProjectSubDirectory.tar.gz
The issue is in the last part, the --output ~./GitLab/some_project_files/ProjectSubDirectory.tar.gz
I tried different methods (.content, .text) which failed, as:
...
response = requests.get(url=url, headers=headers, params=params).content
# and save the respon content with with open(...)
but in all the cases it saved a non-valid tar.gz file, or other issues.
I even tried https://curlconverter.com/, but the code it generates does not work as well, it seems ignoring precisely the --output parameter, not showing anything about the file itself:
headers = {'PRIVATE-TOKEN': 'A_Token001',}
params = (('path', 'ProjectSubDirectory'),)
response = requests.get('http://192.168.156.55/api/v4/projects/10/repository/archive', headers=headers, params=params)
For now, I just created a script and call it with sub-process, but I don't like much this approach due to Python has libraries, as requests, that I guess should have some way to do the same...
2 key things.
Allow redirects
Use raise_for_status() to make sure the request was successful before writing the file. This will help uncover other potential issues, like failed authentication.
After that write response.content to a file opened in binary mode for writing ('wb')
import requests
url = "https://..."
headers = {} # ...
paramus = {} # ...
output_path = 'path/to/local/file.tar.gz'
response = requests.get(url, headers=headers, params=params, allow_redirects=True)
response.raise_for_status() # make sure the request is successful
with open(output_path, 'wb') as f:
f.write(response.content)
There's a Python-Eve installation that serves files using endpoints.
The endpoint looks like /smartapi/datafiles/5f4deca38cc01b4384f68529
The Python-Eve installation is not maintained by me.
I can download the file using my credentials successfully using curl.
curl -u username:password https://thepythoneve.installation/smartapi/datafiles/5f4deca38cc01b4384f68529 --output test.xlsx
However, when I tried to use python requests, I keep getting 401 unauthorized access
auth = HTTPBasicAuth(settings.username, settings.password)
with requests.get(url, auth=auth, stream=True) as response:
response.raise_for_status()
with open(file_path, "wb") as out_file:
for chunk in response.iter_content(chunk_size=1024 * 1024): # 1MB chunks
out_file.write(chunk)
logger.info("Download finished successfully")
return (response, file_path)
I will keep failing at the raise_for_status
I hope not to use pycurl. I like to still use requests What am I missing from my code to make this work?
So I'm trying to get the output of the IBM Visual Recognition Service, but always get the same Error: {"code":401, "error": "Unauthorized"}
It works if I try it with cURL:
$ curl -X POST -u "apikey: ------------" -F "images_file=#bobross.jpg" "https://gateway.watsonplatform.net/visual-recognition/api/v3/detect_faces?version=2018-03-19"
{ facerecognition data }
My python code so far:
import json
import sys
import requests
header= { 'apikey': '---------', 'Content-Type': 'FaceCharacteristics'}
url= "https://gateway.watsonplatform.net/visual-recognition/api/v3/detect_faces?version=2018-03-19"
file ={image:open('bobross.jpg','rb')}
r = requests.post(url, headers=header, files=file)
print(r.text)
I tried my code in other variants, but it always led to "Unauthorized".
Btw, I am very little experienced with python, I'm still trying to learn.
In your curl example you are passing authentication with the -u flag while in python you are passing it in the header as is. The server is ignoring the authentication in the header and you are being returned a 401 as we expect.
To make life easier we can pass our auth details into the request itself with
auth=('apikey', '[An API Key]') as a named parameter.
It would also be worth removing the Content-Type: FaceCharacteristics from the header - not really sure where this was picked up.
import requests
url = 'https://gateway.watsonplatform.net/visual-recognition/api/v3/classify?version=2018-03-19'
files = {'images_file': open('fruitbowl.jpg','rb')}
resp = requests.post(url, auth=('apikey', '[An API Key]'), files=files)
print(resp.content)
Finally add the file and you should be all set.
More info on requests here
However if you are doing anything more than this..
You probably want to have a look at the Python SDK that IBM provides.
It has more documentation and sample code that you can use.
For example, this is provided.
import json
from watson_developer_cloud import VisualRecognitionV3
visual_recognition = = VisualRecognitionV3(
version='{version}',
api_key='{api_key}'
)
with open('./fruitbowl.jpg', 'rb') as images_file:
classes = visual_recognition.classify(
images_file,
threshold='0.6',
classifier_ids='dogsx2018x03x17_1725181949,Connectors_424118776')
print(json.dumps(classes, indent=2))
I'm trying to upload a zipfile to a Server using Python requests. The upload works fine. However the uploaded file cannot be opened using Windows Explorer or ark. I suppose there's some problem with mime-type or content-Length.
Oddly, uploading the file using curl, does not seem to cause the same problem.
Here is my python code for the request:
s = requests.Session()
headers = {'Content-Type': 'application/zip'}
zip = open('file.zip', 'rb')
files = {'file': ('file.zip', zip, 'application/zip')}
fc = {'Content-Disposition': 'attachment; filename=file.zip'}
headers.update(fc)
r = requests.Request('POST', url, files=files, headers=headers, auth=(user, password))
prepared = r.prepare()
resp = s.send(prepared)
This is the curl code, which works flawlessly:
curl -X POST \
-ik \
-u user:password \
--data-binary '#file.zip' \
-H 'Content-Type: application/zip' \
-H "Content-Disposition: attachment; filename=file.zip" \
url
Uploading the file works in both, the Server also seems to recognize the content-type. However the file is rendered invalid when re-downloading. The zifile is readable before sending via requests or after sending with normal curl, using --data-binary.
Opening the downloaded zifile with unip or file-roller works either way.
EDIT:
I was uploading two files successively. Oddly the error was fixed when uploading the exact same files in reverse order.
This has NOT been a python problem. When trying with standard curl
I must have accidentally reversed the order, which is why it has been working.
I can not explain this behavior nor do I have a fix for it.
In conclusion: Uploading the bigger file first did the trick.
All of the above seems to be applicable in curl, pycurl and python requests, so I assume it's some kind of bug in one of the curl libraries.