I have a service, that exposes API for logging in. In my browser if I do
<host>:<port>/myservice/api/login?u=admin&pw=admin
The above url, returns a ticket, that I can pass along for my successive requests.
More details about the same, here.
Below is my python script.
import urllib
url = 'http://<host>:<port>/myservice/api/login?u=admin&pw=admin'
print 'Retrieving', url
uh = urllib.urlopen(url)
data = uh.read()
print 'Retrieved',len(data),'characters'
print data
When I run this I get
IOError: ('http error', 401, 'Authorization Required', <httplib.HTTPMessage instance at 0x<somenumber>>)
Now, I am not sure what am I supposed to do. So I went to my browser, and opened the developer's console.
Apparently, the url has moved to something else. I see two requests.
first one hits the url that I am hitting. Response Header has a Location:Parameter.
The second request hits the url that is returned as Location parameter. the Authorization header has 'Negotiation
It also has a setcookie in the response header.
Now, I am not sure what exactly to do with this information, but if someone can help. Thanks
I believe you problem is having the wrong URL for the login service
If I change you code to instead be:
import urllib, json
url = 'http://localhost:8080/alfresco/service/api/login?u=admin&pw=admin&format=json'
print "Retrieving %s" % url
uh = urllib.urlopen(url)
data = uh.read()
print "Retrieved %d characters" % len(data)
print "Data is %s" % data
ticket = json.loads(data)["data"]["ticket"]
print "Ticket is %s" % ticket
Then against a freshly installed Alfresco 4.2 server, I get back a login Ticket for the admin user.
Note the use of the json format of the login API - much easier to parse from JSON, and of the correct path to the login api - /alfresco/service/api/login
try this two small changes may be it will help:
1) use urllib.urlencode while passing parameters to request url
import urllib
params = urllib.urlencode({'u': 'admin', 'pw': 'admin'})
uh = urllib.urlopen("http://<host>:<port>/myservice/api/login?%s" % params')
2) Stimulate a web browser while making a request using urllib2
import urllib2
req = urllib2.Request('http://<host>:<port>/myservice/api/login?u=admin&pw=admin', headers={ 'User-Agent': 'Mozilla/5.0' })
uh = urllib2.urlopen(req)
401 is unauthorized error! it means you are not authorized to access API. Did you already signup for API keys and access tokens?
Check detailed description of 401 Error:
http://techproblems.org/http-error-401/
Related
i want to automation testing tool using api.
at first, i login the site and get a cookie.
my code is python3
import urllib
import urllib3
from bs4 import BeautifulSoup
url ='http://ip:port/api/login'
login_req = urllib.parse.urlencode(login_form)
http = urllib3.PoolManager()
r= http.request('POST',url,fields={'userName':'id','password':'passoword'})
soup = BeautifulSoup(r.data.decode('utf-8'),'lxml')
cookie = r.getheaders().get('Set-Cookie')
str1 = r.getheaders().get('Set-Cookie')
str2 = 'JSESSIONID' +str1.split('JSESSIONID')[1]
str2 = str2[0:-2]
print(str2)
-- JSESSIONID=df0010cf-1273-4add-9158-70d817a182f7; Path=/; HttpOnly
and then, i add cookie on head another web site api.
but it is not working!
url2 = 'http://ip:port/api/notebook/job/paragraph'
r2 = http.request('POST',url2)
r2.headers['Set-Cookie']=str2
r2.headers['Cookie']=str2
http.request('POST',url2, headers=r2.headers)
why is not working? it shows another cookie
if you know this situation, please explain to me..
error contents is that.
HTTP ERROR 500
Problem accessing /api/login;JSESSIONID=b8f6d236-494b-4646-8723-ccd0d7ef832f.
Reason: Server Error
Caused by:</h3><pre>javax.servlet.ServletException: Filtered request failed.
ProtocolError: ('Connection aborted.', BadStatusLine('<html>\n',))
thanks a lot!
Use requests module in python 3.x. You have to create a session which you are not doing now that's why you are facing problems.
import requests
s=requests.Session()
url ='http://ip:port/api/login'
r=s.get(url)
dct=s.cookies.get_dict() #it will return cookies(if any) and save it in dict
Take which ever cookie is wanted by the server and all the headers requested and pass it in header
jid=dct["JSESSIONID"]
head = {JSESSIONID="+jid,.....}
payload = {'userName':'id','password':'passoword'}
r = s.post(url, data=payload,headers=head)
r = s.get('whatever url after login')
To get info about which specific headers you have to pass and all the parameters required for POST
Open link in google chrome.
Open Developers Console(fn + F12).
There search for login doc (if cannot find, input wrong details and submit).
You will get info about request headers and POST parameters.
I tried to connect HP QC using python to create defects and attach files, but I am not able to connect with HP QC. Here is my code:
domain='DEFAULT_773497139'
project='773497139_DEMO'
import requests
url = "https://almalm1250saastrial.saas.hpe.com/qcbin/"
querystring = {"username":"user#gmail.com","password":"password"}
headers = {
'cache-control': "no-cache",
'token': "5d33d0b7-1d04-4989-3349-3005b847ab7f"
}
response = requests.request("POST", url, headers=headers, params=querystring)
#~ print(response.text)
print response.headers
new_header = response.headers
new_url = url+ u'rest/domains/'+domain+u'/projects/'+project
new_querystring = {
"username":"user#gmail.com",
"password":"password",
"domain":'DEFAULT_773497139',
"project":'773497139_DEMO'
}
print new_url
response = requests.request("POST", new_url, headers=new_header, params=new_querystring)
print(response.text)
Now login works fine, but when try other API it asks for, I would get this message:
Authentication failed. Browser based integrations - to login append '?login-form-required=y' to the url you tried to access
If the parameter has been added, then it goes back to login page.
Seems that your urls are not well builded:
base_url ='https://server.saas.hpe.com/qcbin/'
base_url + '/qcbin/rest/domains/
you will get:
..../qcbin/qcbin/...
qcbin twice
The way I do it is to based on python request Sessions. First I create a session, then post my credentials to ../authentication-point/alm-authenticate/ (or sth like this, you should check it) and then using this session I can get, post or do whatever I want.
So:
s = requests.Session()
s.post(`../authentication-point/alm-authenticate/`, data=credentials)
# now session object is authenticated and recognized
# you can s.post, s.get or whatever
I think it's a good url, but I can't check it right now :)
Session issue has beensolved by LWSSO cookie (LWSSO_COOKIE_KEY).
Just send a unicode string to your server and use the header for the basic Authorization as specified by the HP REST API:
login_url = u'https://almalm1250saastrial.saas.hpe.com/qcbin/authentication-point/authenticate'
username,password = user,passwd
logs = base64.b64encode("{0}:{1}".format(username, password))
header['Authorization'] = "Basic {}".format(logs)
POST by using the requests module in python is quite easy:
requests.post(login_url, headers=header)
That's it...now you are authenticated and you can proceed with next action :-) To check on that you can "GET" on:
login_auth = u'https://almalm1250saastrial.saas.hpe.com/qcbin/rest/is-authenticated
you should get a code 200 --> That means you are authenticated.
Hope this help you. Have a nice day and let me know if something is still not clear.
p.s.: to send REST msg in python I am using requests module. It is really easy! You can create a session if you want to send multiple actions--> then work with that sessions--> ALM = requests.session(), then use ALM.post(whatever) and so on :-)
To perform screen-scraping upon iTunes Connect data, I am trying to get past its login using Python, Requests, BeautifulSoup.
This is iTunes Connect login page:
https://itunesconnect.apple.com/itc/static/login
Typically, to begin screen-scraping upon other websites, I am able to get through the login by grabbing a token from webpage's hidden contents and then I am able to perform login with credentials allowing the website to think login request is coming through a valid browser.
For example, this has been my rough steps of performing login access using pseudo-python coding:
session = requests.Session()
response = session(GET, [URL LOGIN], ...)
soup = BeautifulSoup(response.text, 'html.parser')
token_tag = soup.find_all(...)
TOKEN = token_tag.get(...)
response = session(POST, [URL LOGIN], [CREDENTIALS + TOKEN])
login_html = response.text
login_soup = BeautifulSoup(login_html, 'html.parser')
However, I am difficulty with iTune Connects' login.
Have others tried, and what is the trick?
Thanks
I'm spit-balling here, but the problem is likely a lack of base64 encoding on the login credentials and token as they are passed on through the POST.
Your request should look like something along the lines of:
import requests
import base64
r = requests.post(<url login>,
headers={"Authorization": "Basic " + base64.b64encode(b'username:password'),
data=payload)
I want to fetch the page 192.168.1.1 /basic/home_dhcplist.htm
from the router but it asks for username and password at the start.
I am fetching the page in Python through urllib2
import urllib2
response = urllib2.urlopen('http://192.168.1.1/basic/home_dhcplist.htm')
html = response.read()
str="Prasads"
value= html.find(str)
print value
if value!=-1 :
print "found"
else:
print "not found"
response.close()
Every home router I have seen uses basic auth for authentication. This is simply another header that you send along with the request. Each time you request a page the username and password are sent as headers to the server, where they are verified each and every request.
I would suggest the requests library over urllib2.
import requests
r = requests.get('http://192.168.1.1/basic/home_dhcplist.htm', auth=('username', 'password'))
if 'Prasads' in r.text():
print "found"
else:
print "not found"
Basically you need to set the cookie which maintains the session, most probably.
Access the page via a browser(Firefox) enter the login pass when prompted to do so.
Press Ctrl-Shift-k, and reload the page and click on any of the most recent GET requests, you'll get a window showing the GET request details. Note the Request Headers and set the cookie accordingly.
The key-value which would be the most useful is Authorization.
I'm playing around, trying to write some code to use the tr.im
APIs to shorten a URL.
After reading http://docs.python.org/library/urllib2.html, I tried:
TRIM_API_URL = 'http://api.tr.im/api'
auth_handler = urllib2.HTTPBasicAuthHandler()
auth_handler.add_password(realm='tr.im',
uri=TRIM_API_URL,
user=USERNAME,
passwd=PASSWORD)
opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)
response = urllib2.urlopen('%s/trim_simple?url=%s'
% (TRIM_API_URL, url_to_trim))
url = response.read().strip()
response.code is 200 (I think it should be 202). url is valid, but
the basic HTTP authentication doesn't seem to have worked, because the
shortened URL isn't in my list of URLs (at http://tr.im/?page=1).
After reading http://www.voidspace.org.uk/python/articles/authentication.shtml#doing-it-properly
I also tried:
TRIM_API_URL = 'api.tr.im/api'
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_mgr.add_password(None, TRIM_API_URL, USERNAME, PASSWORD)
auth_handler = urllib2.HTTPBasicAuthHandler(password_mgr)
opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)
response = urllib2.urlopen('http://%s/trim_simple?url=%s'
% (TRIM_API_URL, url_to_trim))
url = response.read().strip()
But I get the same results. (response.code is 200 and url is valid,
but not recorded in my account at http://tr.im/.)
If I use query string parameters instead of basic HTTP authentication,
like this:
TRIM_API_URL = 'http://api.tr.im/api'
response = urllib2.urlopen('%s/trim_simple?url=%s&username=%s&password=%s'
% (TRIM_API_URL,
url_to_trim,
USERNAME,
PASSWORD))
url = response.read().strip()
...then not only is url valid but it's recorded in my tr.im account.
(Though response.code is still 200.)
There must be something wrong with my code though (and not tr.im's API), because
$ curl -u yacitus:xxxx http://api.tr.im/api/trim_url.json?url=http://www.google.co.uk
...returns:
{"trimpath":"hfhb","reference":"nH45bftZDWOX0QpVojeDbOvPDnaRaJ","trimmed":"11\/03\/2009","destination":"http:\/\/www.google.co.uk\/","trim_path":"hfhb","domain":"google.co.uk","url":"http:\/\/tr.im\/hfhb","visits":0,"status":{"result":"OK","code":"200","message":"tr.im URL Added."},"date_time":"2009-03-11T10:15:35-04:00"}
...and the URL does appear in my list of URLs on http://tr.im/?page=1.
And if I run:
$ curl -u yacitus:xxxx http://api.tr.im/api/trim_url.json?url=http://www.google.co.uk
...again, I get:
{"trimpath":"hfhb","reference":"nH45bftZDWOX0QpVojeDbOvPDnaRaJ","trimmed":"11\/03\/2009","destination":"http:\/\/www.google.co.uk\/","trim_path":"hfhb","domain":"google.co.uk","url":"http:\/\/tr.im\/hfhb","visits":0,"status":{"result":"OK","code":"201","message":"tr.im URL Already Created [yacitus]."},"date_time":"2009-03-11T10:15:35-04:00"}
Note code is 201, and message is "tr.im URL Already Created [yacitus]."
I must not be doing the basic HTTP authentication correctly (in either attempt). Can you spot my problem? Perhaps I should look and see what's being sent over the wire? I've never done that before. Are there Python APIs I can use (perhaps in pdb)? Or is there another tool (preferably for Mac OS X) I can use?
This seems to work really well (taken from another thread)
import urllib2, base64
request = urllib2.Request("http://api.foursquare.com/v1/user")
base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '')
request.add_header("Authorization", "Basic %s" % base64string)
result = urllib2.urlopen(request)
Really cheap solution:
urllib.urlopen('http://user:xxxx#api.tr.im/api')
(which you may decide is not suitable for a number of reasons, like security of the url)
Github API example:
>>> import urllib, json
>>> result = urllib.urlopen('https://personal-access-token:x-oauth-basic#api.github.com/repos/:owner/:repo')
>>> r = json.load(result.fp)
>>> result.close()
Take a look at this SO post answer and also look at this basic authentication tutorial from the urllib2 missing manual.
In order for urllib2 basic authentication to work, the http response must contain HTTP code 401 Unauthorized and a key "WWW-Authenticate" with the value "Basic" otherwise, Python won't send your login info, and you will need to either use Requests, or urllib.urlopen(url) with your login in the url, or add a the header like in #Flowpoke's answer.
You can view your error by putting your urlopen in a try block:
try:
urllib2.urlopen(urllib2.Request(url))
except urllib2.HTTPError, e:
print e.headers
print e.headers.has_key('WWW-Authenticate')
The recommended way is to use requests module:
#!/usr/bin/env python
import requests # $ python -m pip install requests
####from pip._vendor import requests # bundled with python
url = 'https://httpbin.org/hidden-basic-auth/user/passwd'
user, password = 'user', 'passwd'
r = requests.get(url, auth=(user, password)) # send auth unconditionally
r.raise_for_status() # raise an exception if the authentication fails
Here's a single source Python 2/3 compatible urllib2-based variant:
#!/usr/bin/env python
import base64
try:
from urllib.request import Request, urlopen
except ImportError: # Python 2
from urllib2 import Request, urlopen
credentials = '{user}:{password}'.format(**vars()).encode()
urlopen(Request(url, headers={'Authorization': # send auth unconditionally
b'Basic ' + base64.b64encode(credentials)})).close()
Python 3.5+ introduces HTTPPasswordMgrWithPriorAuth() that allows:
..to eliminate unnecessary 401 response handling, or to unconditionally send credentials on the first request in order to communicate with servers that return a 404 response instead of a 401 if the Authorization header is not sent..
#!/usr/bin/env python3
import urllib.request as urllib2
password_manager = urllib2.HTTPPasswordMgrWithPriorAuth()
password_manager.add_password(None, url, user, password,
is_authenticated=True) # to handle 404 variant
auth_manager = urllib2.HTTPBasicAuthHandler(password_manager)
opener = urllib2.build_opener(auth_manager)
opener.open(url).close()
It is easy to replace HTTPBasicAuthHandler() with ProxyBasicAuthHandler() if necessary in this case.
I would suggest that the current solution is to use my package urllib2_prior_auth which solves this pretty nicely (I work on inclusion to the standard lib.
Same solutions as Python urllib2 Basic Auth Problem apply.
see https://stackoverflow.com/a/24048852/1733117; you can subclass urllib2.HTTPBasicAuthHandler to add the Authorization header to each request that matches the known url.
class PreemptiveBasicAuthHandler(urllib2.HTTPBasicAuthHandler):
'''Preemptive basic auth.
Instead of waiting for a 403 to then retry with the credentials,
send the credentials if the url is handled by the password manager.
Note: please use realm=None when calling add_password.'''
def http_request(self, req):
url = req.get_full_url()
realm = None
# this is very similar to the code from retry_http_basic_auth()
# but returns a request object.
user, pw = self.passwd.find_user_password(realm, url)
if pw:
raw = "%s:%s" % (user, pw)
auth = 'Basic %s' % base64.b64encode(raw).strip()
req.add_unredirected_header(self.auth_header, auth)
return req
https_request = http_request
Try python-request or python-grab