How can I download a password protected file inside python?
The file is shared via Owncloud and the access is protected with a password by Owncloud.
I know it works with curl by using:
curl -u "FileId:FilePw" -H 'X-Requested-With: XMLHttpRequest' "https://exampledomain.com/public.php/webdav/" >output_file
The field file id FileId is extracted from the shared link.
There are web pages which can convert curl command to many different languages and modules - even to Python and requests - ie. Curl Converter
import requests
headers = {
'X-Requested-With': 'XMLHttpRequest',
}
response = requests.get('https://exampledomain.com/public.php/webdav/',
headers=headers,
auth=('FileId', 'FilePw'))
And this needs only to save response in binary mode
with open('filename.ext', 'wb') as fh:
fh.write( response.content )
You can nest the command into a system call with os module
system_object = os.system('your command')
or fork a new process and use the subprocess run
myProcess = subprocess.run()
requests module allows you to use http commands
import requests
headers = {}
response = requests.method(params)
the important part is that you assign an object variable to the instance method so that you can work with the file object
Related
I'm trying to upload a secure file to my repository in GitLab.
While I am able to upload a secure file with curl, I encounter an error when using requests in Python.
my python code:
r = requests.post("https://gitlab.com/api/v4/projects/10186699/secure_files",
headers={"PRIVATE-TOKEN": "glpat-TH7FM3nThKmHgOp"},
files={"file": open("/Users/me/Desktop/dev/web-server/utils/a.txt", "r"),
"name": "a.txt"})
print(r.status_code,r.json())
Response:
400 {'error': 'name is invalid'}
The equivalent curl command I use that actually works:
curl --request POST --header "PRIVATE-TOKEN: glpat-TH7FM3nThKmHgOp" https://gitlab.com/api/v4/projects/10186699/secure_files --form "name=a.txt" --form "file=#/Users/me/Desktop/dev/web-server/utils/a.txt"
The equivalent call will be
import requests
resp = requests.post(
"https://gitlab.com/api/v4/projects/10186699/secure_files",
headers={"PRIVATE-TOKEN": "glpat-TH7FM3nThKmHgOp"},
files={"file": open("/Users/me/Desktop/dev/web-server/utils/a.txt", "rb")},
data={"name": "a.txt"}
)
print(resp.status_code,resp.json())
This is because the file= parameter is intended only for uploading files. On the other hand, name is your form data (you need to pass in the data= parameter).
It's also recommended to open files in binary mode. (docs)
I have a GitLab API (v4) that I need to call to get a project sub-directory (something apparently new in v.14.4, it seems not yet included python-gitlab libs), which in curl can be done with the following command:
curl --header "PRIVATE-TOKEN: A_Token001" http://192.168.156.55/api/v4/projects/10/repository/archive?path=ProjectSubDirectory --output ~./temp/ProjectSubDirectory.tar.gz
The issue is in the last part, the --output ~./GitLab/some_project_files/ProjectSubDirectory.tar.gz
I tried different methods (.content, .text) which failed, as:
...
response = requests.get(url=url, headers=headers, params=params).content
# and save the respon content with with open(...)
but in all the cases it saved a non-valid tar.gz file, or other issues.
I even tried https://curlconverter.com/, but the code it generates does not work as well, it seems ignoring precisely the --output parameter, not showing anything about the file itself:
headers = {'PRIVATE-TOKEN': 'A_Token001',}
params = (('path', 'ProjectSubDirectory'),)
response = requests.get('http://192.168.156.55/api/v4/projects/10/repository/archive', headers=headers, params=params)
For now, I just created a script and call it with sub-process, but I don't like much this approach due to Python has libraries, as requests, that I guess should have some way to do the same...
2 key things.
Allow redirects
Use raise_for_status() to make sure the request was successful before writing the file. This will help uncover other potential issues, like failed authentication.
After that write response.content to a file opened in binary mode for writing ('wb')
import requests
url = "https://..."
headers = {} # ...
paramus = {} # ...
output_path = 'path/to/local/file.tar.gz'
response = requests.get(url, headers=headers, params=params, allow_redirects=True)
response.raise_for_status() # make sure the request is successful
with open(output_path, 'wb') as f:
f.write(response.content)
So I'm trying to get the output of the IBM Visual Recognition Service, but always get the same Error: {"code":401, "error": "Unauthorized"}
It works if I try it with cURL:
$ curl -X POST -u "apikey: ------------" -F "images_file=#bobross.jpg" "https://gateway.watsonplatform.net/visual-recognition/api/v3/detect_faces?version=2018-03-19"
{ facerecognition data }
My python code so far:
import json
import sys
import requests
header= { 'apikey': '---------', 'Content-Type': 'FaceCharacteristics'}
url= "https://gateway.watsonplatform.net/visual-recognition/api/v3/detect_faces?version=2018-03-19"
file ={image:open('bobross.jpg','rb')}
r = requests.post(url, headers=header, files=file)
print(r.text)
I tried my code in other variants, but it always led to "Unauthorized".
Btw, I am very little experienced with python, I'm still trying to learn.
In your curl example you are passing authentication with the -u flag while in python you are passing it in the header as is. The server is ignoring the authentication in the header and you are being returned a 401 as we expect.
To make life easier we can pass our auth details into the request itself with
auth=('apikey', '[An API Key]') as a named parameter.
It would also be worth removing the Content-Type: FaceCharacteristics from the header - not really sure where this was picked up.
import requests
url = 'https://gateway.watsonplatform.net/visual-recognition/api/v3/classify?version=2018-03-19'
files = {'images_file': open('fruitbowl.jpg','rb')}
resp = requests.post(url, auth=('apikey', '[An API Key]'), files=files)
print(resp.content)
Finally add the file and you should be all set.
More info on requests here
However if you are doing anything more than this..
You probably want to have a look at the Python SDK that IBM provides.
It has more documentation and sample code that you can use.
For example, this is provided.
import json
from watson_developer_cloud import VisualRecognitionV3
visual_recognition = = VisualRecognitionV3(
version='{version}',
api_key='{api_key}'
)
with open('./fruitbowl.jpg', 'rb') as images_file:
classes = visual_recognition.classify(
images_file,
threshold='0.6',
classifier_ids='dogsx2018x03x17_1725181949,Connectors_424118776')
print(json.dumps(classes, indent=2))
Sorry for this basic question again, still in learning stages of Python. I am writing a Python script that makes a Rest call which will have basic authentication headers included. In this example, the user is luke and password is mypasswd. Since the password is written in clear text, is there a way to encrypt the password within the script or move authentication outside the script in a more secure way? What is the recommended way of authenticatiion when using Rest with Python?
import urllib2
import base64
import xml.etree.ElementTree as ET
weblink = "https://192.168.1.1/user"
auth = base64.b64encode("luke:mypasswd")
headers = {"Authorization":"Basic " + auth}
You'll have to put somewhere the credentials, so I think you are worried about distributing the credentials with your script. This could be solved by
1) Using a configuration file where you'd store the credentials (https://docs.python.org/2/library/configparser.html)
2) Specify them at the command line
3) Specify them through environment variables.
my recommendation is to use requests package.(pip install requests).
http://docs.python-requests.org/en/latest/
Regarding the security of passwords, you can use Global variables perhaps, or some text file with adequate permissions.
In linux terminal or .bashrc file: export mypasswd="*******"
import os
import base64
import requests
weblink = "https://192.168.1.1/user"
mypasswd = os.getenv("mypasswd")
auth = base64.b64encode("luke:"+str(mypasswd))
headers = {"Authorization":"Basic " + auth}
#In headers you can have some more properties as Content-Type or so on...
#next would be to call the http method you need(GET,POST,PUT,DELETE)
resp = requests.get(weblink,headers=headers)
print resp.text
print resp.status_code
I am a complete Python noob and am trying to run cURL equivalents using urllib2. What I want is a Python script that, when run, will do the exact same thing as the following cURL command in Terminal:
curl -k -F docfile=#myLocalFile.csv http://myWebsite.com/extension/extension/extension
I found the following template on a tutorial page:
import urllib
import urllib2
url = "https://uploadWebsiteHere.com"
data = "{From: 'sender#email.com', To: 'recipient#email.com', Subject: 'Postmark test', HtmlBody: 'Hello dear Postmark user.'}"
headers = { "Accept" : "application/json",
"Conthent-Type": "application/json",
"X-Postmark-Server-Token": "abcdef-1234-46cc-b2ab-38e3a208ab2b"}
req = urllib2.Request(url, data, headers)
response = urllib2.urlopen(req)
the_page = response.read()
but I am completely lost on the 'data' and 'headers' vars. The urllib2 documentation (https://docs.python.org/2/library/urllib2.html) defines the 'data' input as "a string specifying additional data to send to the server" and the 'headers' input as "a dictionary". I am totally out of my depth in trying to follow this documentation and do not see why a dictionary is necessary when I could accomplish this same task in terminal by only specifying the file and URL. Thoughts, please?
The data you are posting doesn't appear to be valid JSON. Assuming the server is expecting valid JSON, you should change that.
Your curl invocation does not pass any optional headers, so you shouldn't need to provide much in the request. If you want to verify the exact headers you could add -vi to the curl invocation and directly match them in the Python code. Alternatively, this works for me:
import urllib2
url = "http://localhost:8888/"
data = '{"From": "sender#email.com", "To": "recipient#email.com", "Subject": "Postmark test", "HtmlBody": "Hello dear Postmark user."}'
headers = {
"Content-Type": "application/json"
}
req = urllib2.Request(url, data, headers)
response = urllib2.urlopen(req)
the_page = response.read()
It probably is in your best interest to switch over to using requests, but for something this simple the standard library urllib2 can be made to work.
What I want is a Python script that, when run, will do the exact same thing as the following cURL command in Terminal:
$ curl -k -F docfile=#myLocalFile.csv https://myWebsite.com/extension...
curl -F sends the file using multipart/form-data content type. You could reproduce it easily using requests library:
import requests # $ pip install requests
with open('myLocalFile.csv','rb') as input_file:
r = requests.post('https://myWebsite.com/extension/...',
files={'docfile': input_file}, verify=False)
verify=False is to emulate curl -k.