Is it possible to sign into a website (and allow that website to be visited with any browser without the user having to sign in) via a background process without user interaction and allow the user to browse the site without logging in from any browser?
I'd guess that I would need to register the created session with each web browser on the user's system, but is there any other (possibly simpler) way of doing this?
Think of it like automatically signing into Gmail in the background and being able to browse it without ever seeing a login page.
yes is possible.I suggest two ways to solve your problem. Both of them uses HTTP requests. You should check more info about HTTP request.
1) the easiest way and recommended one for only login Requests: HTTP for Humans
2) python scrapy, but scrapy is for crawling or screen scraping.
check this example:
Login spider example
class LoginSpider(BaseSpider):
name = 'example.com'
start_urls = ['http://www.example.com/users/login.php']
def parse(self, response):
return [FormRequest.from_response(response,
formdata={'username': 'john', 'password': 'secret'},
callback=self.after_login)]
def after_login(self, response):
# check login succeed before going on
if "authentication failed" in response.body:
self.log("Login failed", level=log.ERROR)
return
# continue scraping with authenticated session...
more info here
There is a library in python called urllib2 which will let you do what you need. Look in the python docs or here:
http://www.voidspace.org.uk/python/articles/urllib2.shtml#openers-and-handlers
or this:
http://www.doughellmann.com/PyMOTW/urllib2/
Related
I need to parse site, which is hidden by ADFS service.
and struggling with authentication to it.
Is there any options to get in?
what i can see, most of solutions for backend applications, or for "system users"(with app_id, app_secret).
in my case, i can't use it, only login and password.
example of problem:
in chrome I open www.example.com and it redirects me to to https://login.microsoftonline.com/ and then to https://federation-sts.example.com/adfs/ls/?blabla with login and password form.
and how to get access into it with python3?
ADFS uses complicated redirection and CSRF protection techniques. Thus, it is better to use a browser automation tool to perform the authentication and parse the webpage afterwards. I recommend the selenium toolkit with python bindings. Here is a working example:
from selenium import webdriver
def MS_login(usrname, passwd): # call this with username and password
driver = webdriver.Edge() # change to your browser (supporting Firefox, Chrome, ...)
driver.delete_all_cookies() # clean up the prior login sessions
driver.get('https://login.microsoftonline.com/') # change the url to your website
time.sleep(5) # wait for redirection and rendering
driver.find_element_by_xpath("//input[#name='loginfmt'").send_keys(usrname)
driver.find_element_by_xpath("//input[#type='submit']").click()
time.sleep(5)
driver.find_element_by_xpath("//input[#name='passwd'").send_keys(passwd)
driver.find_element_by_xpath("//input[#name='KMSI' and #type='checkbox'").click()
driver.find_element_by_xpath("//input[#type='submit']").click()
time.sleep(5)
driver.find_element_by_xpath("//input[#type='submit']").click()
# Successfully login
# parse the site ...
driver.close() # close the browser
return driver
This script calls Microsoft Edge to open the website. It injects the username and password to the correct DOM elements and then let the browser to handle the rest. It has been tested on the webpage "https://login.microsoftonline.com". You may need to modify it to suit your website.
To Answer your question "How to Get in with python" i am assuming you want perform some web scraping operation on the pages which is secured by Azure AD authentication.
In these kind of scenario, you have to do the following steps.
For this script we will only need to import the following:
import requests
from lxml import html
First, we would like to create our session object. This object will allow us to persist the login session across all our requests.
session_requests = requests.session()
Second, we would like to extract the csrf token from the web page, this token is used during login. For this example we are using lxml and xpath, we could have used regular expression or any other method that will extract this data.
login_url = "https://bitbucket.org/account/signin/?next=/"
result = session_requests.get(login_url)
tree = html.fromstring(result.text)
authenticity_token = list(set(tree.xpath("//input[#name='csrfmiddlewaretoken']/#value")))[0]
Next, we would like to perform the login phase. In this phase, we send a POST request to the login url. We use the payload that we created in the previous step as the data. We also use a header for the request and add a referer key to it for the same url.
result = session_requests.post(
login_url,
data = payload,
headers = dict(referer=login_url)
)
Payload would be a dictionary object of user name and password etc.
payload = {
"username": "<USER NAME>",
"password": "<PASSWORD>",
"csrfmiddlewaretoken": "<CSRF_TOKEN>"
}
Note:- This is just an example.
Step 2:
Scrape content
Now, that we were able to successfully login, we will perform the actual scraping
url = 'https://bitbucket.org/dashboard/overview'
result = session_requests.get(
url,
headers = dict(referer = url)
)
So in other words, you need to get the request details payload from Azure AD and then create a session object using logged in method and then finally do the scraping.
Here is a very good example of Web scraping of a secured website.
Hope it helps.
Is it possible for Scrapy to crawl an alert message?
The link for example, http://domainhere/admin, once loaded in an actual browser, an alert message with form is present to fill up the username and password.
Or is there a way to inspect the form in an alert message to know what parameters to be filled up?
PS: I do have credentials for this website, I just want to automate processes through web crawling.
Thanks.
What I did to achieved this, was by doing the following:
Observed what after authentication data needed to proceed with the page.
Using Chrome's developers' tool in the Network tab, I checked the Request Headers. Upon observation, Authorization is needed.
To verify step #2, I used Postman. Using the Authorization in Postman, Basic Auth type, filling up the username and password will generate the same value for the Authorization header. After sending a POST request, it loaded the desired page and bypassed the authentication.
Having the same value for the Authorization under Request Headers, store the value in the Scraper class.
Use the scrapy.Request function with headers parameter.
Code:
import scrapy
class TestScraper(scrapy.Spider):
handle_httpstatus_list = [401]
name = "Test"
allowed_domains = ["xxx.xx.xx"]
start_urls = ["http://testdomain/test"]
auth = "Basic [Key Here]"
def parse(self, response):
return scrapy.Request(
"http://testdomain/test",
headers={'Authorization': self.auth},
callback=self.after_login
)
def after_login(self, response):
self.log(response.body)
Now, you can crawl the page after authentication process.
I've write a Scrapy crawler for parse data from https://www.mysonicwall.com/.
They change their login page, before username and password were in the same form on the same page and i was able to login well with this code:
def parse(self, response):
return scrapy.FormRequest.from_response(
response,
formdata={'ctl00$_mainContent$USERNAME': 'dummymail', 'ctl00$_mainContent$Password': 'dummypassword'}, #
cookies={'cookie': 'dummydatacookie',}, #
callback=self.after_login
)
Now login and password are on two separated form on two webpage (first page for username and second for password).
How can I do ?
Thanks !
You can use Splash and scrapy-splash to automate the login process.
It does require a fair bit of setup and use of Lua language.
Otherwise I would try two FormRequests in succession - one for the username and other for the password.
I'm using a Scrapy spider that authenticates with a login form upon launching. It then scrapes with this authenticated session.
During development I usually run the spider many times to test it out. Authenticating at the beginning of each run spams the login form of the website. The website will often force a password reset in response and I suspect it will ban the account if this continues.
Because the cookies last a number of hours, there's no good reason to log in this often during development. To get around the password reset problem, what would be the best way to re-use an authenticated session/cookies between runs while developing? Ideally the spider would only attempt to authenticate if the persisted session has expired.
Edit:
My structure is like:
def start_requests(self):
yield scrapy.Request(self.base, callback=self.log_in)
def log_in(self, response):
#response.headers includes 'Set-Cookie': 'JSESSIONID=xx'; Path=/cas/; Secure; HttpOnly'
yield scrapy.FormRequest.from_response(response,
formdata={'username': 'xxx',
'password':''},
callback=self.logged_in)
def logged_in(self, response):
#request.headers and subsequent requests all have headers fields 'Cookie': 'JSESSIONID=xxx';
#response.headers has no mention of cookies
#request.cookies is empty
When I run the same page request in Chrome, under the 'Cookies' tab there are ~20 fields listed.
The documentation seems thin here. I've tried setting a field 'Cookie': 'JSESSIONID=xxx' on the headers dict of all outgoing requests based on the values returned by a successful login, but this bounces back to the login screen
Turns out that for an ad-hoc development solution, this is easier to do than I thought. Get the cookie string with cookieString = request.headers['Cookie'], save, then on subsequent outgoing requests load it up and do:
request.headers.appendlist('Cookie', cookieString)
I am scraping a site that has an accept terms form that I need to click through. When I click the button I am redirected to the resource that needs to be scraped. I have the basic mechanics working, that is the initial click through works and I get a session and all goes well until the session times out. Then for some reason Scrapy does get redirected but the response URL doesn't get updated so I get duplicate items since I am using the URL to check for duplication.
For example the URL I am requesting is:
https://some-internal-web-page/Records/Details/119ce2b7-35b4-4c63-8bd2-2bfbf77299a8
But when the session expires I get:
https://some-internal-web-page/?returnUrl=%2FRecords%2FDetails%2F119ce2b7-35b4-4c63-8bd2-2bfbf77299a8
Here is my code:
# function to get through accept dialog
def parse(self, response):
yield FormRequest.from_response(response, formdata={"value":"Accept"}, callback=self.after_accept)
# function to parse markup
def after_accept(self, response):
global latest_inspection_date
urls = ['http://some-internal-web-page/Records?SearchText=&SortMode=MostRecentlyHired&page=%s&PageSize=25' % page for page in xrange(1,500)]
for u in urls:
yield Request( u, callback=self.parse_list )
So my question is, how do I persist and/or refresh the session cookie so that I don't get the redirect URL instead of the URL I need.
Cookies are enabled by default and passed through every callback, make sure you have it enabled with COOKIES_ENABLED=True in settings.py.
you can also enable debugging logs for it with COOKIES_DEBUG=True (False by default), and check if the cookies are being passed correctly, so maybe your problem is about something else.