How to get Authorization token from a webpage using python requests, i have used requests basicAuth to login, it was worked, but subsequent pages are not accpting te basicAuth, it returns "Authuser is not validated"
There is a login url where i have successfully logged in using python requests's basicAuth. then succeeding pages didn't accept basicAuth credential but it needed authorization header. after looking into browser inspect tool, found out that, this authorization header's value is generated as a part of session local storage. is there any way to get this session value without using webdriver API?
Sounds like what you need is a requests persistent session
import requests
s=requests.Session()
#then simply make the request like you already are
r=s.get(r'https://stackoverflow.com/')
#the cookies are persisted
s.cookies.get_dict()
>{'prov':......}
i can't really get more specific without more info about the site you're using.
Related
I have a problem, maybe is my lack of knowledge on the subject.
I'm building a script to connect to the Jira API via cookie auth.
When I use on my script:
session = requests.Session()
a_session.get('https://drf05.ad.local/')
session_cookies = a_session.cookies
cookies_dictionary = session_cookies.get_dict()
jira = Jira(
url='https://drf05.ad.local/',
cookies = cookies_dictionary )
The cookies that I get are different from the cookies that I inspect directly on the browser.
If I use the cookies that I get from the request, it does not work, but if I use the cookies that I get when I inspect the browser network it works.
For what I have read this request lib would get me the current session cookies, but i guess is not doing that.
For privacy concerns, I cannot distribute the url publicly.
I have been able to access this site successfully using python requests session = requests.Session(); r = session.post(url, auth = HttpNtlmAuth(USERNAME, PASSWORD), proxies = proxies) which works great and I can parse the webpage with bs4. I have tried to return cookies using session.cookies.get_dict() but it returns an empty dict (assuming b/c site is hosted using sharepoint). My original thought was to retrieve cookies then use them to access the site.
The issue that I'm facing is when you redirect to the url, a box comes up asking for credentials - which when entered directs you to the url. You can not inspect the page that the box is on- which means that I can't use send.keys() etc. to login using selenium/chromedriver.
I read through some documentation but was unable to find a way to enter pass/username when calling driver = webdriver.Chrome(path_driver) or following calls.
Any help/thoughts would be appreciated.
When right clicking the below - no option to inspect webpage.
I'm trying to log in this website using my credentials running python script but the problem is that the xhr requests visible as login in chrome dev tools stays for a moment and then vanishes, so I can't see the appropriate parameters (supposed to be recorded) necessary to log in. However, I do find that login in xhr if I put my password wrong. The form then looks incomplete, though.
I've tried so far (an incomplete payload because of chrome dev tools):
import requests
url = "https://member.angieslist.com/gateway/platform/v1/session/login"
payload = {"identifier":"username","token":"sometoken"}
res = requests.post(url,json=payload,headers={
"User-Agent":"Mozilla/5.0",
"Referer":"https://member.angieslist.com/member/login"
})
print(res.url)
How can I log in that site filling in appropriate parameters issuing a post http requests?
There is a checkbox called Persist logs in the Network tab and if its switched on the data about the post request remains. I think you should requests a session if you need to keep the script logged in. It may be done with:
import requests
url = 'https://member.angieslist.com/gateway/platform/v1/session/login'
s = requests.session()
payload = {"identifier":"youremail","token":"your password"}
res = s.post(url,json=payload,headers={"User-Agent":"Mozilla/5.0",'Referer': 'https://member.angieslist.com/member/login?redirect=%2Fapp%2Faccount'}).text
print(res)
the post requests returns a json file with all details of user.
I'm trying to get contest data from the url: "https://www.draftkings.com/contest/gamecenter/32947401"
If you go to this URL and aren't logged in, it'll just re-direct you to the lobby. If you're logged in, it'll actually show you the contest results.
Here's some things I tried:
-First, I used Chrome's Dev networking tools to watch requests while I manually logged in
-I then tried copying the cookie that I thought contained the authentication info, it was of the form:
'ajs_anonymous_id=%123123123123123, mlc=true; optimizelyEndUserId'
-I then stored that cookie as an Evironment variable and ran this code:
HEADERS= {'cookie': os.environ['MY_COOKIE'] }
requests.get(draft_kings_url, headers= HEADERS)
No luck, this just gave me the lobby.
I then tried request's built in:
HTTPBasicAuth
HTTPDigestAuth
No luck here either.
I'm no python expert by far, and I've pretty much exhausted what I know and the search results I've found. Any ideas?
The tool that you want is selenium. Something along the lines of:
from selenium import webdriver
browser = webdriver.Firefox()
browser.get(r"https://www.draftkings.com/contest/gamecenter/32947401" )
username = browser.find_element_by_id("user")
username.send_keys("username")
password = browser.find_element_by_id("password")
password.send_keys("top_secret")
login = selenium.find_element_by_name("login")
login.click()
Use fiddler to see the exact request they are making when you try to log in. Then use Session class in requests package.
import requests
session = requests.Session()
session.get('YOUR_URL_LOGIN_PAGE')
this will save all the cookies from your url in your session variable (Like when you use a browser).
Then make a post request to the login url with appropriate data.
You dont have to manually pass cookie data as it is auto generated when you first visit a website. However you can set some header explicitly like UserAgent etc by:
session.headers.update({'header_name':'header_value'})
HTTPBasicAuth & HTTPDigestAuth might not work based on the website.
I have been googling for this problem for a week now.
The thing I want to achive is the following:
Send a POST request to the URL including the correct credentials.
Save the session (not cookie since my website is not using cookies at the moment)
With the saved session open a session protected URL and grab the contents.
I have seen alot of topics on this with cookies but not with sessions, I tried sessions with requests but seems to fail everytime.
You want to use a URL opener. Here's a sample of how I've managed to do it. If you just want a default opener, use opener=urllib.request.build_opener(), otherwise use the custom opener. This worked when I had to log into a website and keep a session, using URL as your URL, user as user, password as password, all changed as appropriate.
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(http.cookiejar.CookieJar()))
pData=urllib.parse.urlencode({"identity":user,"password":password})
req=urllib.request.Request(URL,pData.encode('utf-8'))
opener.open(req)
req=urllib.request.Request(url)
response= opener.open(req)