Develop Reference

How to scrape a website which requires login using python

i want to webscrpe this website and i need to login to get the account datail (https://online.cspension.com.hk/hkpensionweb/#/login?source=corporateHK&vgnLocale=en_CA), i don't want to use selenium
how to login and get the account datail

You can use a requests.Session() instance to make a post request to the login url with your login details as a payload. Making requests from a session instance is essentially the same as using requests normally, it simply adds persistence, allowing you to store and use cookies etc.
import requests
# Fill in your details here to be posted to the login form.
payload = {
'inUserName': 'username',
'inUserPass': 'password'
}
# Use 'with' to ensure the session context is closed after use.
with requests.Session() as s:
p = s.post('LOGIN_URL', data=payload)
# print the html returned or something more intelligent to see if it's a successful login page.
print p.text
# An authorised request.
r = s.get('A protected web page url')
print r.text
# etc...

Remote login to decoupled website with python and requests

I am trying to login to a website www.seek.com.au. I am trying to test the possibility to remote login using Python request module. The site is Front end is designed using React and hence I don't see any form action component in www.seek.com.au/sign-in
When I run the below code, I see the response code as 200 indicating success, but I doubt if it's actually successful. The main concern is which URL to use in case if there is no action element in the login submit form.
import requests
payload = {'email': <username>, 'password': <password>}
url = 'https://www.seek.com.au'
with requests.Session() as s:
response_op = s.post(url, data=payload)
# print the response status code
print(response_op.status_code)
print(response_op.text)
When i examine the output data (response_op.text), i see word 'Sign in' and 'Register' in output which indicate the login failed. If its successful, the users first name will be shown in the place. What am I doing wrong here ?
P.S: I am not trying to scrape data from this website but I am trying to login to a similar website.

Try this code:
import requests
payload={"email": "test#test.com", "password": "passwordtest", "rememberMe": True}
url = "https://www.seek.com.au:443/userapi/login"
with requests.Session() as s:
response_op = s.post(url, json=payload)
# print the response status code
print(response_op.status_code)
print(response_op.text)
You are sending the request to the wrong url.
Hope this helps

Using python requests module to login on an Wordpress based website

I'm trying to login to a Wordpress based website using python's request module and beautifulsoup4. It seems like the code fails to sucessfully login. Also, there is no csrf token on the website. How do I sucessfully login to the website?
import requests
import bs4 as bs
with requests.session() as c:
link="https://gpldl.com/sign-in/" #link of the webpage to be logged in
initial=c.get(link) #passing the get request
login_data={"log":"*****","pwd":"******"} #the login data from any account on the site. Stars must be replaced with username and password
page_login=c.post(link, data=login_data) #posting the login data into the link
print(page_login) #checking status of requested page
page=c.get("https://gpldl.com/my-gpldl-account/") #requesting source code of logged in page
good_data = bs.BeautifulSoup(page.content, "lxml") #parsing it with BS4
print(good_data.title) #printing this gives the title that is got from the page when accessed from an logged-out account

You are sending your POST request to a wrong URL, the correct one should be https://gpldl.com/wp-login.php, also there're 5 parameters for the payload which are log, pwd, rememberme, redirect_to, redirect_to_automatic.
So it should be:
login_data = {"log": "*****","pwd": "******",
"rememberme": "forever",
"redirect_to": "https://gpldl.com/my-gpldl-account/",
"redirect_to_automatic": "1"
}
page_login = c.post('https://gpldl.com/wp-login.php', data=login_data)
Edit:
You could use Chrome Dev tool to find out all these info while logging in, it's like this:
As to rememberme key, I would suggest you to do exact same thing a browser does, also add some headers for your request, especially User-Agent, because for some websites they just don't welcome you got login this way.

Download a CSV file with python requests from an authenticated URL

For one of my project I need to fetch a CSV file from a website authenticated URL. Unfortunately there is no API to do it. So that is why I decided to use requests with sessions to fetch it.
The login page is:
http://extranet.ffbb.com/fbi/identification.do
I checked on the page for the login and password names in the HTML code. Login is "identificationBean.identifiant" and password is "identificationBean.mdp".
So I try to connect and to display the returned HTML with the following code but it returns the HTML code of a wrong login/password (like if I type a wrong login/pass from my browser). And I am sure my credentials are right.
login = "my_login"
password = "my_password"
with requests.Session() as session:
data = {
'identificationBean.identifiant': '{}'.format(config.login),
'identificationBean.mdp': '{}'.format(config.password)
}
url = 'http://extranet.ffbb.com/fbi/identification.do'
response = session.post(url, data=data)
print(response.text)
Thank you for your help

Logging with python into a subpage (from another post)

I came across this question: How to use Python to login to a webpage and retrieve cookies for later usage?
So, I'm trying to log into a page, (using the request method, second answer).
When I print the HTML using
print request.text
It will print the HTML of the login page, but not the subpage that I put on request.
Is there a problem with the code (which I don't think) or is it mine?
The code is similar to the one on that question, with different pages and usernames.
Thanks!
from requests import session
USERNAME = 'myuser'
PASSWORD = 'mypwd'
payload = {
'action': 'login',
'username': USERNAME,
'password': PASSWORD
}
with session() as c:
c.post('https://www.bricklink.com/login.asp', data=payload) #Login page
request = c.get('http://www.bricklink.com/orderExcelFinal.asp?') #Page I want to access
print request.headers
print request.text
Output
HTML code for the Login page, but not the page I want to access

Your code isn't sending the correct data on the login request.
Each web page is different, and sends different data in order to log in. Yours should be structured like this:
from requests import session
USERNAME = 'myuser'
PASSWORD = 'mypwd'
query = {
'logInTo': '',
'logFolder': 'p',
'logSub': 'w',
}
payload = {
'a': 'a',
'logFrmFlag': 'Y',
'frmUsername': USERNAME,
'frmPassword': PASSWORD,
}
with session() as c:
c.post('https://www.bricklink.com/login.asp', params=query, data=payload) #Login page
request = c.get('http://www.bricklink.com/orderExcelFinal.asp') #Page I want to access
print request.headers
print request.text
In the future, when you need to work out what data needs to be send on an attempt to submit a form, you should use Chrome or Firefox's Developer Tools. Use these to record your login attempt, and then structure the data accordingly. Getting started with using Chrome's developer tools is a bit beyond the scope of this answer, but there are lots of good resources on the web for finding out how you get this information.