I am trying to access the Azure Kudu Web API interface to get historical information about my WebJobs. The URL is https://myfakewebappname.scm.azurewebsites.net/api/triggeredwebjobs/HubSpot/history
This works just fine in a browser with a one time (first time you login) user and password authentication.
When I call it with a python script using the requests library, I get a 401 response code and 'WWW-Authenticate': 'Basic realm="site"'. I then send another request:
resp = requests.get('https://myfakewebappname.scm.azurewebsites.net/api/triggeredwebjobs/HubSpot/history', auth=('actualuser', 'actualpassword')). I use the user and Password that work using my browser.
I get the same 401 response code again with the same response.headers. What am I doing wrong?
The Authentication of the WebJobs Kudu API is via basic auth, to call the API successfully in python, please follow the steps below.
1.Navigate to the Azure portal -> your web app which has the webjob -> click Get publish profile.
2.Open the downloaded file with the format joyweb11.PublishSettings in step 1, note down the userName and userPWD in this file.
3.Then use the code below, replace the value of username and password with the values in step 2, also replace the webapp name in the url with yours, it works fine on my side.
import requests
from base64 import b64encode
username = '$joyweb11'
password = '7pWclexxxxxJlHwoLRwtrneE'
base64AuthInfo = b64encode((username+':'+password).encode('ascii')).decode('ascii')
url = 'https://joyweb11.scm.azurewebsites.net/api/triggeredwebjobs/HubSpot/history'
headers = {'Authorization': 'Basic ' + base64AuthInfo}
response = requests.get(url=url, headers=headers)
print(response.status_code)
print(response.text)
I've built a Python application that generates a inventory CSV file, and I want to upload that file to my store through BigCommerce's WebDAV application. I'm using the following Python client to access the WebDAV.
https://pypi.org/project/webdavclient3/
I can access my store and add files to the content folder with CyberDuck, but I get a HTTP 401 error when I try to access it from my Python script. Here is what I'm using to connect with WebDAV.
# webDAV upload to BigCommerce
options = {
'webdav_hostname': "https://mystore.com",
'webdav_login': "email#email.com",
'webdav_password': "password",
'webdav_root': "/dav/",
}
client = Client(options)
print("Exist:", client.check("/content/mytest")) # returns "Exist: False"
print(client.list())
print(client.free())
print("HERE")
I get an error at client.list() that reads
Request to https://mystore.com/dav/ failed with code 401 and message:
<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns"><s:exception>Sabre\\DAV\\Exception\\NotAuthenticated</s:exception><s:message>No 'Authorization: Digest' header found. Either the client didn't send one, or the server is misconfigured</s:message>
</d:error>
I guess it's saying my login and/or password is incorrect or there is no authentication? But how come I could log in through CyberDuck with the same credentials?
I saw someone asking about a similar problem in the following link, and I've tried the suggestions from Karen. None of them worked.
https://support.bigcommerce.com/s/question/0D51B00004G4XfYSAV/unable-to-access-upload-files-or-create-directory-through-webdav-api
I know this is 6 months old, but I figured I'd still post the solution for visibility when others try to do this.
The webdavclient library does not support HTTP Digest Authentication which is required to upload to the WebDAV. You can achieve this using the basic Python Requests library combined with the HTTPDigestAuth library.
Sample code:
import requests
from requests.auth import HTTPDigestAuth
# Below, put the URL of your BigCommerce WebDAV, including the file name you want to create/upload
# If you're uploading a CSV that you want to use as a product import, you will put it in the /dav/import_files/ directory.
url='https://store-abcdefg123.mybigcommerce.com/dav/import_files/products_upload_filename.csv' # example
# Local filename relative to this python script to upload
files = open('products_upload_filename.csv', 'rb')
# BigCommerce WebDAV login credentials.
# Found under Server Settings > File Access (WebDAV)
usern = 'youremail#email.com' # username
passw = 'password123' # password
# Make the file upload request
r = requests.request('PUT', url=url, data=files, auth=HTTPDigestAuth(usern, passw))
r.status_code
print(r.headers)
print(r.status_code)
I had the same issue and it was from the AuthType in the VirtualHost (/etc/httpd/conf.d/webdav.conf). I switched from Digest to Basic to fix it.
I'm aware seems a pretty repetitive issue. From bash and curl I make calls to the API Github this way
# curl -u myuser https://api.github.com/repos/myuser/somerepo/pulls?state=all -H "Authorization: token randomtokenhere" -d '{"title":"Pull Request develop to master","base":"master", "head":"develop"}'
and works like a charm. However, from Python 3(3.5.2) with json and requests, I got an error no matter what.
Example:
user = "myuser"
password = "sompassword"
url = "https://api.github.com/repos/myuser/somerepo/pulls?state=all"
token = "randomtokenhere"
title = "Pull Request develop to master"
base = "master"
head = "develop"
headers = {'Authorization': 'token ' + token}
content = {"title":title,"base":base, "head":head}
req = requests.post(url,json=json.dumps(content),auth=(user,password),headers=headers)
print("response posts is {} and status code is {}".format(req.text,req.status_code))
the response of requests is
response posts is {"message":"Must specify two-factor authentication OTP code.","documentation_url":"https://developer.github.com/v3/auth#working-with-two-factor-authentication"} and status code is 401
so it seems the call is just missing the token in some way. But I'm not able to know why. Can I debug this in some way? Or did I miss something very obvious?
Thanks
Well, sorry for bother, my call was not correct. This actually worked:
//req = requests.post(url,json=json.dumps(content),auth=(user,password),headers=headers)
req = requests.post(url,json=content,auth=(user,token))
I've removed the json.dumps function, removed the headers parameter, and instead use password I set up the token with the auth.
Regards
I have been regularly accessing an API at work using curl. But now i need to do a bit of automation for the same.
Was trying to translate what i do with curl into python using the requests module.
But I keep receiving a 401 error.
My curl requests that i regularly are as below:
Step1: Session Authentication with cookies:
curl -b cookies -c cookies -v -X POST -H "Content-Type: application/json" --data-binary '{"auth":{"username":"aaa","password":"bbb"}}' http://api.xyz.at/auth
Step2: Access API URL for data retrieval
curl -b cookies -c cookies http://api.xyz.at/somepage?per_id=556677
Now using Python Requests, here is what I am doing
Step1: For Authentication
username = 'aaa'
password = 'bbb'
s = requests.Session()
s.post('http://api.xyz.at/auth',auth=('username','pasword'))
This "i think" works fine, and give me the below response
<Response [200]>
Step2: Access API URL for data retrieval
s.get('http://api.xyz.at/somepage?per_id=556677')
but this Step 2 keeps returning an error
<Response [401]>
The code is failing and not sure where.
My Python skills are visibly pedestrian. I have been looking at the Requests website. But unfortunately haven't been able to decipher.
Guidance would be appreciated :)
import urllib2, urllib
url = 'http://api.xyz.at/auth'
pm = urllib2.HTTPPasswordMgrWithDefaultRealm()
pm.add_password(None, url, 'user', 'password')
auth = urllib2.HTTPBasicAuthHandler(pm)
opener = urllib2.build_opener(auth)
urllib2.install_opener(opener)
request = urllib2.Request('http://api.xyz.at/somepage?per_id=556677', None)
handler = urllib2.urlopen(request)
handler.read()
Since you are getting a 401 error I guess you are not passing the authentication token which you get as response from login API. Use the same auth token to perform other options - Get-post-Delete.
I'm playing around, trying to write some code to use the tr.im
APIs to shorten a URL.
After reading http://docs.python.org/library/urllib2.html, I tried:
TRIM_API_URL = 'http://api.tr.im/api'
auth_handler = urllib2.HTTPBasicAuthHandler()
auth_handler.add_password(realm='tr.im',
uri=TRIM_API_URL,
user=USERNAME,
passwd=PASSWORD)
opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)
response = urllib2.urlopen('%s/trim_simple?url=%s'
% (TRIM_API_URL, url_to_trim))
url = response.read().strip()
response.code is 200 (I think it should be 202). url is valid, but
the basic HTTP authentication doesn't seem to have worked, because the
shortened URL isn't in my list of URLs (at http://tr.im/?page=1).
After reading http://www.voidspace.org.uk/python/articles/authentication.shtml#doing-it-properly
I also tried:
TRIM_API_URL = 'api.tr.im/api'
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_mgr.add_password(None, TRIM_API_URL, USERNAME, PASSWORD)
auth_handler = urllib2.HTTPBasicAuthHandler(password_mgr)
opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)
response = urllib2.urlopen('http://%s/trim_simple?url=%s'
% (TRIM_API_URL, url_to_trim))
url = response.read().strip()
But I get the same results. (response.code is 200 and url is valid,
but not recorded in my account at http://tr.im/.)
If I use query string parameters instead of basic HTTP authentication,
like this:
TRIM_API_URL = 'http://api.tr.im/api'
response = urllib2.urlopen('%s/trim_simple?url=%s&username=%s&password=%s'
% (TRIM_API_URL,
url_to_trim,
USERNAME,
PASSWORD))
url = response.read().strip()
...then not only is url valid but it's recorded in my tr.im account.
(Though response.code is still 200.)
There must be something wrong with my code though (and not tr.im's API), because
$ curl -u yacitus:xxxx http://api.tr.im/api/trim_url.json?url=http://www.google.co.uk
...returns:
{"trimpath":"hfhb","reference":"nH45bftZDWOX0QpVojeDbOvPDnaRaJ","trimmed":"11\/03\/2009","destination":"http:\/\/www.google.co.uk\/","trim_path":"hfhb","domain":"google.co.uk","url":"http:\/\/tr.im\/hfhb","visits":0,"status":{"result":"OK","code":"200","message":"tr.im URL Added."},"date_time":"2009-03-11T10:15:35-04:00"}
...and the URL does appear in my list of URLs on http://tr.im/?page=1.
And if I run:
$ curl -u yacitus:xxxx http://api.tr.im/api/trim_url.json?url=http://www.google.co.uk
...again, I get:
{"trimpath":"hfhb","reference":"nH45bftZDWOX0QpVojeDbOvPDnaRaJ","trimmed":"11\/03\/2009","destination":"http:\/\/www.google.co.uk\/","trim_path":"hfhb","domain":"google.co.uk","url":"http:\/\/tr.im\/hfhb","visits":0,"status":{"result":"OK","code":"201","message":"tr.im URL Already Created [yacitus]."},"date_time":"2009-03-11T10:15:35-04:00"}
Note code is 201, and message is "tr.im URL Already Created [yacitus]."
I must not be doing the basic HTTP authentication correctly (in either attempt). Can you spot my problem? Perhaps I should look and see what's being sent over the wire? I've never done that before. Are there Python APIs I can use (perhaps in pdb)? Or is there another tool (preferably for Mac OS X) I can use?
This seems to work really well (taken from another thread)
import urllib2, base64
request = urllib2.Request("http://api.foursquare.com/v1/user")
base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '')
request.add_header("Authorization", "Basic %s" % base64string)
result = urllib2.urlopen(request)
Really cheap solution:
urllib.urlopen('http://user:xxxx#api.tr.im/api')
(which you may decide is not suitable for a number of reasons, like security of the url)
Github API example:
>>> import urllib, json
>>> result = urllib.urlopen('https://personal-access-token:x-oauth-basic#api.github.com/repos/:owner/:repo')
>>> r = json.load(result.fp)
>>> result.close()
Take a look at this SO post answer and also look at this basic authentication tutorial from the urllib2 missing manual.
In order for urllib2 basic authentication to work, the http response must contain HTTP code 401 Unauthorized and a key "WWW-Authenticate" with the value "Basic" otherwise, Python won't send your login info, and you will need to either use Requests, or urllib.urlopen(url) with your login in the url, or add a the header like in #Flowpoke's answer.
You can view your error by putting your urlopen in a try block:
try:
urllib2.urlopen(urllib2.Request(url))
except urllib2.HTTPError, e:
print e.headers
print e.headers.has_key('WWW-Authenticate')
The recommended way is to use requests module:
#!/usr/bin/env python
import requests # $ python -m pip install requests
####from pip._vendor import requests # bundled with python
url = 'https://httpbin.org/hidden-basic-auth/user/passwd'
user, password = 'user', 'passwd'
r = requests.get(url, auth=(user, password)) # send auth unconditionally
r.raise_for_status() # raise an exception if the authentication fails
Here's a single source Python 2/3 compatible urllib2-based variant:
#!/usr/bin/env python
import base64
try:
from urllib.request import Request, urlopen
except ImportError: # Python 2
from urllib2 import Request, urlopen
credentials = '{user}:{password}'.format(**vars()).encode()
urlopen(Request(url, headers={'Authorization': # send auth unconditionally
b'Basic ' + base64.b64encode(credentials)})).close()
Python 3.5+ introduces HTTPPasswordMgrWithPriorAuth() that allows:
..to eliminate unnecessary 401 response handling, or to unconditionally send credentials on the first request in order to communicate with servers that return a 404 response instead of a 401 if the Authorization header is not sent..
#!/usr/bin/env python3
import urllib.request as urllib2
password_manager = urllib2.HTTPPasswordMgrWithPriorAuth()
password_manager.add_password(None, url, user, password,
is_authenticated=True) # to handle 404 variant
auth_manager = urllib2.HTTPBasicAuthHandler(password_manager)
opener = urllib2.build_opener(auth_manager)
opener.open(url).close()
It is easy to replace HTTPBasicAuthHandler() with ProxyBasicAuthHandler() if necessary in this case.
I would suggest that the current solution is to use my package urllib2_prior_auth which solves this pretty nicely (I work on inclusion to the standard lib.
Same solutions as Python urllib2 Basic Auth Problem apply.
see https://stackoverflow.com/a/24048852/1733117; you can subclass urllib2.HTTPBasicAuthHandler to add the Authorization header to each request that matches the known url.
class PreemptiveBasicAuthHandler(urllib2.HTTPBasicAuthHandler):
'''Preemptive basic auth.
Instead of waiting for a 403 to then retry with the credentials,
send the credentials if the url is handled by the password manager.
Note: please use realm=None when calling add_password.'''
def http_request(self, req):
url = req.get_full_url()
realm = None
# this is very similar to the code from retry_http_basic_auth()
# but returns a request object.
user, pw = self.passwd.find_user_password(realm, url)
if pw:
raw = "%s:%s" % (user, pw)
auth = 'Basic %s' % base64.b64encode(raw).strip()
req.add_unredirected_header(self.auth_header, auth)
return req
https_request = http_request
Try python-request or python-grab