I am trying to access the workday report through python. i am able to access this through browser with userid and passwd. But when i run through python i am getting the below error.
import os
import platform
import ssl
import urllib.request
import urllib.parse
ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS)
ssl_context.verify_mode = ssl.CERT_REQUIRED
ssl_context.check_hostname = True
ssl_context.load_default_certs()
if platform.system().lower() == 'windows':
import certifi
print(os.path.relpath(certifi.where()))
ssl_context.load_verify_locations(
#cafile=os.path.relpath(certifi.where()),
cafile="C:\\abc_Tools\\TDX_INT166\\Lib\\site-packages\\certifi\\workday.pem",
capath=None,
cadata=None)
print(platform.system().lower())
url = 'https://wd5-impl-services1.workday.com/ccx/service/customreport2/xxx/ISU_INT167/CR_-
_FIN_Report'
username = 'XXXXXXXXX' # 10 digit ID
password = 'XXXXXXXXX'
values ={'username' : username, 'password':password}
data = urllib.parse.urlencode(values).encode("utf-8")
#cookies = cookielib.CookieJar()
https_handler = urllib.request.HTTPSHandler(context=ssl_context)
opener = urllib.request.build_opener(https_handler)
ret = opener.open(url, timeout=2)
print(ret)
I am getting the below error.
site-packages\certifi\cacert.pem
windows
Traceback (most recent call last):
File "C:\SLU_Tools\TDX_INT166\Lib\Certificate_testing.py", line 38, in <module>
ret = opener.open(url, timeout=2)
File
"C:\Users\prasannakumaravel\AppData\Local\Programs\Python\Python39\lib\urllib\request.py",
line 523, in open
response = meth(req, response)
File
"C:\Users\prasannakumaravel\AppData\Local\Programs\Python\Python39\lib\urllib\request.py",
line 632, in http_response
response = self.parent.error(
File
"C:\Users\prasannakumaravel\AppData\Local\Programs\Python\Python39\lib\urllib\request.py",
line 561, in error
return self._call_chain(*args)
File
"C:\Users\prasannakumaravel\AppData\Local\Programs\Python\Python39\lib\urllib\request.py",
line 494, in _call_chain
result = func(*args)
File
"C:\Users\prasannakumaravel\AppData\Local\Programs\Python\Python39\lib\urllib\request.py",
line 641, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Unauthorized
I tried other ways as well. But nothing worked. so far. Is this something doable?
When calling a RAAS this way, you need to authenticate using Basic Auth. The username/password needs to be Base64 encoded and the result is added as an HTTP header. See this answer for an example.
You will need to set up the authorizations using the Basic Auth type, and you will need to encode your username/password following the Basic Auth schemes -- converting the credentials to base64 first, then back to a string format and using it in the authorization string.
I used code similar to the snippet below in our web applications:
import requests, base64
url = '<Workday_RaaS_API_Endpoint>'
username = '<your_RaaS_user_username>'
password = '<your_RaaS_user_password>' # replace with the password here
auth = 'Basic %s' % base64.b64encode(bytes('%s:%s' % (username, password), 'utf-8')).decode('utf-8')
headers = { 'Authorization': auth }
res = requests.get(url=url, headers=headers)
print(res.json())
Related
I need to use the urllib for this, not urllib2, urllib3, or Requests.
The curl equivalent...
curl -H "Content-Type: application/json" -X POST -d '{"title":"My New Title","content":"blahblah","excerpt":"blah"}' http://localhost/wp-json/wp/v2/posts --user 'user:xxx'
... and the following code work fine:
r = requests.post(url + '/wp-json/wp/v2/posts',
auth=HTTPBasicAuth(username, password),
data=json.dumps(params),
headers={'content-type': 'application/json'})
if(not(r.ok)):
raise Exception('failed with rc=' + r.status_code)
return json.loads(r.text)
But this fails:
params = {
'title': 'The Title',
'content': content,
'exerpt': 'blah'
}
postparam = json.dumps(params).encode('utf-8')
password_mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm()
password_mgr.add_password(None, url, username, password)
handler = urllib.request.HTTPBasicAuthHandler(password_mgr)
opener = urllib.request.build_opener(handler)
req = urllib.request.Request(url + '/wp-json/wp/v2/posts', method='POST', data=postparam)
req.add_header('Content-Type', 'application/json')
r = opener.open(req)
if(r.getcode() != 200):
raise Exception('failed with rc=' + r.getcode())
Traceback:
Traceback (most recent call last):
File "test_wp_api.py", line 10, in <module>
rs = create(endpoint, "test")
File "C:\...\wordpress.py", line 28, in create
r = opener.open(req)
File "c:\usr\Python37-32\lib\urllib\request.py", line 531, in open
response = meth(req, response)
File "c:\usr\Python37-32\lib\urllib\request.py", line 641, in http_response
'http', request, response, code, msg, hdrs)
File "c:\usr\Python37-32\lib\urllib\request.py", line 569, in error
return self._call_chain(*args)
File "c:\usr\Python37-32\lib\urllib\request.py", line 503, in _call_chain
result = func(*args)
File "c:\usr\Python37-32\lib\urllib\request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Unauthorized
What's the cause, and how to fix it?
I will venture a guess. Please, try this out and if it does not address your issue, I can ditch the answer (but it won't easily fit into a comment as is).
Not sure about requests. but curl when provided with credentials does not wait for challenge (HTTP 401 and info on supported authentication methods) to try (respond with) authentication (attempt). Some servers do not send such challenge and just expect pre-authenticated session or a force (lucky guess a bit) authentication. I have to talk to some hosts like that too. If that is the case, you cannot change server setup and you are OK to just assume HTTP basic is supported, you can try forcing your credentials into each request. Reusing your opener,username, and password and you also need to import base64 for this snippet which does just that:
credentials = '{}:{}'.format(username, password).encode('ascii')
credentials = base64.b64encode(credentials)
credentials = b"Basic " + credentials
opener.addheaders.append(("Authorization", credentials))
I have been searching for a way to implement 2 legged oauth in python 3 to work with the brightcloud api. They offer several code examples using java, php, ruby, .NET C# here: https://bcws.brightcloud.com/code-samples.php. I tried following the same logic to convert the java example into python, however, I'm relatively new to python and I quickly came unstuck.
I have tried implementing using rauth, however, the basic setup utilises a request_token_url which is not provided by brightcloud. I also tried implementing with the following code which was based on this answer -
How do I send a POST using 2-legged oauth2 in python?
import oauth2
import urllib #for url-encode
import urllib.request
import time #Unix timestamp import oauth2
# construct request url
base_url = "http://thor.brightcloud.com/rest"
uri_info_path = "/uris"
url = urllib.parse.quote_plus("http://www.booking.com")
# api key and secret
consumer_key = 'MY_CONSUMER_KEY'
consumer_secret = 'MY_CONSUMER_SECRET'
# contruct endpoint
endpoint = rest_endpoint + uri_info_path + '/' + url
# build request
def build_request(url, method):
params = {
'oauth_version': "1.0",
'oauth_nonce': oauth2.generate_nonce(),
'oauth_timestamp': int(time.time())
}
consumer = oauth2.Consumer(key=consumer_key, secret=consumer_secret)
params['oauth_consumer_key'] = consumer.key
req = oauth2.Request(method=method, url=url, parameters=params)
signature_method = oauth2.SignatureMethod_HMAC_SHA1()
req.sign_request(signature_method, consumer, None)
return req
# call
request = build_request(endpoint,'GET')
u = urllib.request.urlopen(request.to_url())
data = u.read()
print (data)
There is a problem with this line:
u = urllib.request.urlopen(request.to_url())
Which generates the following traceback:
Traceback (most recent call last):
File "bright.py", line 37, in
u = urllib.request.urlopen(request.to_url())
File "/usr/lib/python3.5/urllib/request.py", line 163, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.5/urllib/request.py", line 472, in open
response = meth(req, response)
File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.5/urllib/request.py", line 510, in error
return self._call_chain(*args)
File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
result = func(*args)
File "/usr/lib/python3.5/urllib/request.py", line 590, in >http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Unauthorized
Any help would be much appreciated.
Usually I've been able to get around 403 Errors once I've added a known User Agent but I'm now trying to login and then eventually scrape and cannot figure out how to bypass this error.
Code:
import urllib
import http.cookiejar
cj = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
urllib.request.install_opener(opener)
authentication_url = 'https://www.linkedin.com/'
payload = {
'session_key': 'email',
'session_password': 'password'
}
data = urllib.parse.urlencode(payload)
binary_data = data.encode('UTF-8')
req = urllib.request.Request(authentication_url, binary_data)
resp = urllib.request.urlopen(req)
contents = resp.read()
Traceback:
Traceback (most recent call last):
File "C:/Python34/loginLinked.py", line 16, in <module>
resp = urllib.request.urlopen(req)
File "C:\Python34\lib\urllib\request.py", line 161, in urlopen
return opener.open(url, data, timeout)
File "C:\Python34\lib\urllib\request.py", line 469, in open
response = meth(req, response)
File "C:\Python34\lib\urllib\request.py", line 579, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python34\lib\urllib\request.py", line 507, in error
return self._call_chain(*args)
File "C:\Python34\lib\urllib\request.py", line 441, in _call_chain
result = func(*args)
File "C:\Python34\lib\urllib\request.py", line 587, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
See my answer to this question:
why isn't Requests not signing into a website correctly?
I should start with stating that you really should use their API:
http://developer.linkedin.com/apis
There does not seem to be any POST login on the frontpage of linkedin using those parameters?
This is the login URL you must POST to:
https://www.linkedin.com/uas/login-submit
Be aware that this probably wont work either, as you need at least the csrfToken parameter from the login form.
You probably need the loginCsrfParam too, also from the login form on the frontpage.
Something like this might work. Not tested, you might need to add the other POST parameters.
import requests
s = requests.session()
def get_csrf_tokens():
url = "https://www.linkedin.com/"
req = s.get(url).text
csrf_token = req.split('name="csrfToken" value=')[1].split('" id="')[0]
login_csrf_token = req.split('name="loginCsrfParam" value="')[1].split('" id="')[0]
return csrf_token, login_csrf_token
def login(username, password):
url = "https://www.linkedin.com/uas/login-submit"
csrfToken, loginCsrfParam = get_csrf_tokens()
data = {
'session_key': username,
'session_password': password,
'csrfToken': csrfToken,
'loginCsrfParam': loginCsrfParams
}
req = s.post(url, data=data)
login('username', 'password')
I try to write a simple python3 script that gets some playlist informations via the youtube API. However I always get a 401 Error whereas it works perfectly when I enter the request string in a browser or making a request with w-get. I'm relatively new to python and I guess I'm missing some important point here.
This is my script. Of course I actually use a real API-Key.
from urllib.request import Request, urlopen
from urllib.parse import urlencode
api_key = "myApiKey"
playlist_id = input('Enter playlist id: ')
output_file = input('Enter name of output file (default is playlist id')
if output_file == '':
output_file = playlist_id
url = 'https://www.googleapis.com/youtube/v3/playlistItems'
params = {'part': 'snippet',
'playlistId': playlist_id,
'key': api_key,
'fields': 'items/snippet(title,description,position,resourceId/videoId),nextPageToken,pageInfo/totalResults',
'maxResults': 50,
'pageToken': '', }
data = urlencode(params)
request = Request(url, data.encode('utf-8'))
response = urlopen(request)
content = response.read()
print(content)
Unfortunately it rises a error at response = urlopen(request)
Traceback (most recent call last):
File "gpd-helper.py", line 35, in <module>
response = urlopen(request)
File "/usr/lib/python3.4/urllib/request.py", line 153, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.4/urllib/request.py", line 461, in open
response = meth(req, response)
File "/usr/lib/python3.4/urllib/request.py", line 571, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.4/urllib/request.py", line 499, in error
return self._call_chain(*args)
File "/usr/lib/python3.4/urllib/request.py", line 433, in _call_chain
result = func(*args)
File "/usr/lib/python3.4/urllib/request.py", line 579, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Unauthorized
I looked up the documentation but couldn't find any hint. According to the docs other authentication than the api key is not required for listing a public playlist.
After diving deeper into the docs of python and google I found the solution to my problem.
Pythons Request object automatically creates a POST request when the data parameter is given but the youtube api expects GET (with post params)
The Solution is to ether supply the GET argument for the method parameter in python 3.4
request = Request(url, data.encode('utf-8'), method='GET')
or concatenate the url with the urlencoded post data
request = Request(url + '?' + data)
edit - as I couldn't tag this with Strava here are the docs if you're interested - http://strava.github.io/api/
I get through the authentication fine and gain the access_token (and my athlete info) in a response.read.
I'm having problems with the next step:
I want to return information about a specific activity.
import urllib2
import urllib
access_token = str(tp[3]) #this comes from the response not shown
print access_token
ath_url = 'https://www.strava.com/api/v3/activities/108838256'
ath_val = values={'access_token':access_token}
ath_data = urllib.urlencode (ath_val)
ath_req = urllib2.Request(ath_url, ath_data)
ath_response = urllib2.urlopen(ath_req)
the_page = ath_response.read()
print the_page
error is
Traceback (most recent call last):
File "C:\Users\JordanR\Python2.6\documents\strava\auth.py", line 30, in <module>
ath_response = urllib2.urlopen(ath_req)
File "C:\Users\JordanR\Python2.6\lib\urllib2.py", line 124, in urlopen
return _opener.open(url, data, timeout)
File "C:\Users\JordanR\Python2.6\lib\urllib2.py", line 389, in open
response = meth(req, response)
File "C:\Users\JordanR\Python2.6\lib\urllib2.py", line 502, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Users\JordanR\Python2.6\lib\urllib2.py", line 427, in error
return self._call_chain(*args)
File "C:\Users\JordanR\Python2.6\lib\urllib2.py", line 361, in _call_chain
result = func(*args)
File "C:\Users\JordanR\Python2.6\lib\urllib2.py", line 510, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 404: Not Found
the 404 is a puzzle as I know this activity exists?
Is 'access_token' the correct header information?
the documentation (http://strava.github.io/api/v3/activities/#get-details) uses Authorization: Bearer ? I'm not sure how liburl would encode the Bearer part of the info?
sorry if some of my terminology is a bit off, I'm new to this.
answered this bad boy.
import requests as r
access_token = tp[3]
ath_url = 'https://www.strava.com/api/v3/activities/108838256'
header = {'Authorization': 'Bearer 4b1d12006c51b685fd1a260490_example_jklfds'}
data = r.get(ath_url, headers=header).json()
it needed the "Bearer" part adding in the Dict.
thanks for the help idClark
I prefer using the third-party Requests module. You do indeed need to follow the documentation and use the Authorization: Header documented in the API
Requests has an arg for header data. We can then create a dict where the key is Authorization and its value is a single string Bearer access_token
#install requests from pip if you want
import requests as r
url = 'https://www.strava.com/api/v3/activities/108838256'
header = {'Authorization': 'Bearer access_token'}
r.get(url, headers=header).json()
If you really want to use urllib2
#using urllib2
import urllib2
req = urllib.Request(url)
req.add_header('Authorization', 'Bearer access_token')
resp = urllib2.urlopen(req)
content = resp.read()
Just remember access_token needs to be the literal string value, eg acc09cds09c097d9c097v9