Im using requests Python module to access Github's API. Github responses 404 Not Found, as expected.
This is my code:
import requests
import sys
username = sys.argv[1]
password = sys.argv[2]
url = 'https://api.github.com/{}'.format(username)
print(url)
response = requests.get(url, data={'password': password})
print(response)
How this header must be done?
I need to get the user's id!
Thanks!
I'm trying to use python urllib2 to access the files in a SharePoint 2013 folder using the SharePoint REST API. https://msdn.microsoft.com/en-us/library/office/dn450841.aspx
I can't seem to get the response as a json object even after adding the header 'accept':'application/json;odata=verbose'.
I am using Python 2.7.5, urllib2, and can not use the python requests module due to restrictions and permission constraints.
If I use the .../_api/web/GetFolderByServerRelativeUrl('/File Name')/Files url I get an AttributeError("'str' object has no attribute'read'"). If I just use the normal url to the site I get an error that there is not a 'JSON object'in the response.
I don't know what I am doing wrong and why I can't get at any json from this request?
import sys
import urllib
import urllib2
import json
from ntlm import HTTPNtlmAuthHandler
try:
url = 'https://sharepointsite/subfolder/subfolder/subfolder/Forms/AllItems.aspx/_api/web/GetFolderByServerRelativeUrl(/Folder Name)/Files'
user = r'DOMAIN\USER'
password = '*********'
req = urllib2.Request(url)
req.add_header('accept', 'application/json;odata=verbose')
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, url, user, password)
handler = HTTPNtlmAuthHandler.HTTPNtlmAuthHandler(passman)
opener = urllib2.build_opener(handler)
urllib2.install_opener(opener)
response = urllib2.urlopen(req)
data_json = json.load(response.read())
except Exception as e:
print '{0}'.format(e)
What the request actually returns is a string representation of the json object.
You can use use the json.loads() function instead of the json.load() function in order to convert the string into a json object.
I have a Python 2.6 script that downloades a file from a web server. I want this this script to pass a username and password(for authenrication before fetching the file) and I am passing them as part of the url as follows:
import urllib2
response = urllib2.urlopen("http://'user1':'password'#server_name/file")
However, I am getting syntax error in this case. Is this the correct way to go about it? I am pretty new to Python and coding in general.
Can anybody help me out?
Thanks!
If you can use the requests library, it's insanely easy. I'd highly recommend using it if possible:
import requests
url = 'http://somewebsite.org'
user, password = 'bob', 'I love cats'
resp = requests.get(url, auth=(user, password))
I suppose you are trying to pass through a Basic Authentication. In this case, you can handle it this way:
import urllib2
username = 'user1'
password = '123456'
#This should be the base url you wanted to access.
baseurl = 'http://server_name.com'
#Create a password manager
manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
manager.add_password(None, baseurl, username, password)
#Create an authentication handler using the password manager
auth = urllib2.HTTPBasicAuthHandler(manager)
#Create an opener that will replace the default urlopen method on further calls
opener = urllib2.build_opener(auth)
urllib2.install_opener(opener)
#Here you should access the full url you wanted to open
response = urllib2.urlopen(baseurl + "/file")
Use requests library and just put the credentials inside your .netrc file.
The library will load them from there and you will be able to commit the code to your SCM of choice without any security worries.
I'm trying to download a zip artifact from teamcity using python 3 and i'm not having much luck.
From the browser I would normally do this:
http://USERNAME:PWD#SERVER/httpAuth/repository/downloadAll/dood_dad/latest.lastSuccessful
But if I try this with urllib.request.urlretrieve I get an exception about an invalid port - because It doesn't know about the username and password appended to the front of the url and is parsing after the ':' as a port - fair enough.
So I guess I need to use teamcitys httpAuth stuff and use the url
http://SERVERNAME/httpAuth/repository/downloadAll/dood_dad/latest.lastSuccessful
When I try that I get a 404 Unauthorized which I expected because I need to supply the username and password.
But I can't work out how.
I've added this:
auth_handler = urllib.request.HTTPBasicAuthHandler()
auth_handler.add_password(None,
uri=url_to_open,
user='userame',
passwd='password')
opener = urllib.request.build_opener(auth_handler)
urllib.request.install_opener(opener)
local_filename, headers = urllib.request.urlretrieve(url)
But I'm still getting HTTP Error 401: Unauthorized
TIA.
You can use a lib like requests that let's you put the basic auth as a param, see more here: http://docs.python-requests.org/en/latest/user/authentication/#basic-authentication
import requests
from requests.auth import HTTPBasicAuth
import shutil
response = requests.get('http://...', auth=HTTPBasicAuth('user', 'pass'), stream=True)
with open('filename.zip', 'wb') as out_file:
shutil.copyfileobj(response.raw, out_file)
this works ok:
import urllib
from urllib.request import HTTPPasswordMgrWithDefaultRealm
pwdmgr = HTTPPasswordMgrWithDefaultRealm()
pwdmgr.add_password(None, uri=url, user='XXXX', passwd='XXXX')
auth_handler = urllib.request.HTTPBasicAuthHandler(pwdmgr)
opener = urllib.request.build_opener(auth_handler)
urllib.request.install_opener(opener)
local_filename, headers = urllib.request.urlretrieve(url)
I'm not entirely sure why the newer code works over the older stuff.
FYI: the request code never worked either
response = requests.get('http://...', auth=HTTPBasicAuth('user', 'pass'), stream=True)
I kept getting an Unauthorized http errors
Obtaining Artifacts from a Build Script
import getpass
import subprocess
USERNAME = getpass.getuser()
PWD = getpass.getpass(prompt='PWD:', stream=None)
subprocess.run(['wget','http://'+USERNAME+':'+'PWD'+'#SERVER/httpAuth/repository/downloadAll/dood_dad/latest.lastSuccessful'])
I'm playing around, trying to write some code to use the tr.im
APIs to shorten a URL.
After reading http://docs.python.org/library/urllib2.html, I tried:
TRIM_API_URL = 'http://api.tr.im/api'
auth_handler = urllib2.HTTPBasicAuthHandler()
auth_handler.add_password(realm='tr.im',
uri=TRIM_API_URL,
user=USERNAME,
passwd=PASSWORD)
opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)
response = urllib2.urlopen('%s/trim_simple?url=%s'
% (TRIM_API_URL, url_to_trim))
url = response.read().strip()
response.code is 200 (I think it should be 202). url is valid, but
the basic HTTP authentication doesn't seem to have worked, because the
shortened URL isn't in my list of URLs (at http://tr.im/?page=1).
After reading http://www.voidspace.org.uk/python/articles/authentication.shtml#doing-it-properly
I also tried:
TRIM_API_URL = 'api.tr.im/api'
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_mgr.add_password(None, TRIM_API_URL, USERNAME, PASSWORD)
auth_handler = urllib2.HTTPBasicAuthHandler(password_mgr)
opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)
response = urllib2.urlopen('http://%s/trim_simple?url=%s'
% (TRIM_API_URL, url_to_trim))
url = response.read().strip()
But I get the same results. (response.code is 200 and url is valid,
but not recorded in my account at http://tr.im/.)
If I use query string parameters instead of basic HTTP authentication,
like this:
TRIM_API_URL = 'http://api.tr.im/api'
response = urllib2.urlopen('%s/trim_simple?url=%s&username=%s&password=%s'
% (TRIM_API_URL,
url_to_trim,
USERNAME,
PASSWORD))
url = response.read().strip()
...then not only is url valid but it's recorded in my tr.im account.
(Though response.code is still 200.)
There must be something wrong with my code though (and not tr.im's API), because
$ curl -u yacitus:xxxx http://api.tr.im/api/trim_url.json?url=http://www.google.co.uk
...returns:
{"trimpath":"hfhb","reference":"nH45bftZDWOX0QpVojeDbOvPDnaRaJ","trimmed":"11\/03\/2009","destination":"http:\/\/www.google.co.uk\/","trim_path":"hfhb","domain":"google.co.uk","url":"http:\/\/tr.im\/hfhb","visits":0,"status":{"result":"OK","code":"200","message":"tr.im URL Added."},"date_time":"2009-03-11T10:15:35-04:00"}
...and the URL does appear in my list of URLs on http://tr.im/?page=1.
And if I run:
$ curl -u yacitus:xxxx http://api.tr.im/api/trim_url.json?url=http://www.google.co.uk
...again, I get:
{"trimpath":"hfhb","reference":"nH45bftZDWOX0QpVojeDbOvPDnaRaJ","trimmed":"11\/03\/2009","destination":"http:\/\/www.google.co.uk\/","trim_path":"hfhb","domain":"google.co.uk","url":"http:\/\/tr.im\/hfhb","visits":0,"status":{"result":"OK","code":"201","message":"tr.im URL Already Created [yacitus]."},"date_time":"2009-03-11T10:15:35-04:00"}
Note code is 201, and message is "tr.im URL Already Created [yacitus]."
I must not be doing the basic HTTP authentication correctly (in either attempt). Can you spot my problem? Perhaps I should look and see what's being sent over the wire? I've never done that before. Are there Python APIs I can use (perhaps in pdb)? Or is there another tool (preferably for Mac OS X) I can use?
This seems to work really well (taken from another thread)
import urllib2, base64
request = urllib2.Request("http://api.foursquare.com/v1/user")
base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '')
request.add_header("Authorization", "Basic %s" % base64string)
result = urllib2.urlopen(request)
Really cheap solution:
urllib.urlopen('http://user:xxxx#api.tr.im/api')
(which you may decide is not suitable for a number of reasons, like security of the url)
Github API example:
>>> import urllib, json
>>> result = urllib.urlopen('https://personal-access-token:x-oauth-basic#api.github.com/repos/:owner/:repo')
>>> r = json.load(result.fp)
>>> result.close()
Take a look at this SO post answer and also look at this basic authentication tutorial from the urllib2 missing manual.
In order for urllib2 basic authentication to work, the http response must contain HTTP code 401 Unauthorized and a key "WWW-Authenticate" with the value "Basic" otherwise, Python won't send your login info, and you will need to either use Requests, or urllib.urlopen(url) with your login in the url, or add a the header like in #Flowpoke's answer.
You can view your error by putting your urlopen in a try block:
try:
urllib2.urlopen(urllib2.Request(url))
except urllib2.HTTPError, e:
print e.headers
print e.headers.has_key('WWW-Authenticate')
The recommended way is to use requests module:
#!/usr/bin/env python
import requests # $ python -m pip install requests
####from pip._vendor import requests # bundled with python
url = 'https://httpbin.org/hidden-basic-auth/user/passwd'
user, password = 'user', 'passwd'
r = requests.get(url, auth=(user, password)) # send auth unconditionally
r.raise_for_status() # raise an exception if the authentication fails
Here's a single source Python 2/3 compatible urllib2-based variant:
#!/usr/bin/env python
import base64
try:
from urllib.request import Request, urlopen
except ImportError: # Python 2
from urllib2 import Request, urlopen
credentials = '{user}:{password}'.format(**vars()).encode()
urlopen(Request(url, headers={'Authorization': # send auth unconditionally
b'Basic ' + base64.b64encode(credentials)})).close()
Python 3.5+ introduces HTTPPasswordMgrWithPriorAuth() that allows:
..to eliminate unnecessary 401 response handling, or to unconditionally send credentials on the first request in order to communicate with servers that return a 404 response instead of a 401 if the Authorization header is not sent..
#!/usr/bin/env python3
import urllib.request as urllib2
password_manager = urllib2.HTTPPasswordMgrWithPriorAuth()
password_manager.add_password(None, url, user, password,
is_authenticated=True) # to handle 404 variant
auth_manager = urllib2.HTTPBasicAuthHandler(password_manager)
opener = urllib2.build_opener(auth_manager)
opener.open(url).close()
It is easy to replace HTTPBasicAuthHandler() with ProxyBasicAuthHandler() if necessary in this case.
I would suggest that the current solution is to use my package urllib2_prior_auth which solves this pretty nicely (I work on inclusion to the standard lib.
Same solutions as Python urllib2 Basic Auth Problem apply.
see https://stackoverflow.com/a/24048852/1733117; you can subclass urllib2.HTTPBasicAuthHandler to add the Authorization header to each request that matches the known url.
class PreemptiveBasicAuthHandler(urllib2.HTTPBasicAuthHandler):
'''Preemptive basic auth.
Instead of waiting for a 403 to then retry with the credentials,
send the credentials if the url is handled by the password manager.
Note: please use realm=None when calling add_password.'''
def http_request(self, req):
url = req.get_full_url()
realm = None
# this is very similar to the code from retry_http_basic_auth()
# but returns a request object.
user, pw = self.passwd.find_user_password(realm, url)
if pw:
raw = "%s:%s" % (user, pw)
auth = 'Basic %s' % base64.b64encode(raw).strip()
req.add_unredirected_header(self.auth_header, auth)
return req
https_request = http_request
Try python-request or python-grab