I think I'm on the right track for ASP.NET authentication. I'm trying to use requests to pass credentials to a website. Here are the headers and network info I pulled from chrome:
Remote Address: REMOVED
Request URL: https://REMOVED/default.aspx
Request Method: POST
Status Code: 302 Found
Request Headers:
POST /default.aspx HTTP/1.1
Host: REMOVED
Connection: keep-alive
Content-Length: 928
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: https://REMOVED
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Referer: https://REMOVED/default.aspx
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Cookie: ASP.NET_SessionId=REMOVED; BIGipServerpool_REMOVED_dmz_80=REMOVED.REMOVED.0000; AUTHCDB=**REMOVED**
Form Data:
__EVENTTARGET:
__EVENTARGUMENT:
__VIEWSTATE: /wEP**REMAINDER REMOVED**
__EVENTVALIDATION: /wEd**REMAINDER REMOVED**
jsCheck:
ddlEngine:REMOVED:13008
Username:
Password:
btnLogin.x: 42
btnLogin.y: 9
btnLogin: Login
Response Headers:
Cache-Control: private
Content-Length: 132
Content-Type: text/html; charset=utf-8
Date: Fri, 13 Jun 2014 00:59:13 GMT
Location: /Dashboard.aspx
Server: Microsoft-IIS/7.5
Set-Cookie: AUTHCDB=**REMOVED**; path=/; HttpOnly
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Here is the script I wrote:
import requests
FORM_DATA = {
"__EVENTTARGET:":,
"__EVENTARGUMENT:",
"__VIEWSTATE:/wEPDwUKMTA5NTA5ODU1MQ9kFgJmD2QWAgIGDxBkDxYFZgIBAgICAwIEF***REMAINDER REMOVED***",
"__EVENTVALIDATION:/wEdAAp4d3BHvSTs+Kv6cxGP3xEbBr8xrgRYad2tj4YCyRIw5qUAjimf****REMAINDER REMOVED****",
"jsCheck:",
"ddlEngine: REMOVED:13008",
"Username: ****",
"Password: ****",
"btnLogin.x: 42",
"btnLogin.y: 9",
"btnLogin: Login",
}
HEADER = {
"Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Encoding":"gzip,deflate,sdch",
"Accept-Language":"en-US,en;q=0.8",
"Cache-Control":"max-age=0",
"Connection":"keep-alive",
"Content-Type":"application/x-www-form-urlencoded",
"Host":"REMOVED",
"Origin":"REMOVED",
"Referer":"REMOVED",
"User-Agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36"
}
LOGIN_URL = "REMOVED"
#requests session to handle cookies.
s = requests.Session()
#Send a POST request with the form data/header info
r = s.post(LOGIN_URL, data=FORM_DATA, headers=HEADER)
if r.status_code == 302:
print "Successfully logged in."
else:
print "Error logging in."
Am I able to use Python Requests to log into a webpage that uses ASP.NET? If so, is this the correct way to pass the credentials into the website? For reference, the website I'm trying to log into is a company server monitor.
Looks like a similar issue was posted here and here. I've been using RoboBrowser and it's made messing with ASPX so much simpler.
from robobrowser import RoboBrowser
login_url = 'http://example.com/Login.aspx'
username = 'JohnDoe'
password = 'passwd'
browser = RoboBrowser(history=True)
# This gets all the ASPX stuff, __VIEWSTATE and friends
browser.open(login_url)
signin = brower.get_form(id='aspnetForm')
signin["jsCheck"].value = ''
signin["ddlEngine"].value = "REMOVED:13008"
signin["Username"].value = username
signin["Password"].value = password
signin["btnLogin.x"].value = "42"
signin["btnLogin.y"].value = "9"
signin["btnLogin"].value = "Login"
browser.submit_form(signin)
I'm also working with some ASP.net pages right now, and being somewhat familiar with the requests module, I thought I'd try to help out a bit.
It is my understanding that requests supports basic authentication in this fashion:
from requests.auth import HTTPBasicAuth
requests.get('https://api.github.com/user', auth=HTTPBasicAuth('user', 'pass'))
It could be the case that you'll need to import a different authentication library that works with ASP.net and plug that directly into the auth function of requests.
Hope this helps!
Related
I want to get the data points from the graph below (https://index.mysteel.com/price/getChartMultiCity_1_0.html)
I found out from Developer Tools -> Network -> XHR that a request is made when I explore the graph. The response has all the data needed, like date and price.
If I just copy the request URL and fetch the data with python's requests, I get the correct response. Example:
import requests
r = requests.get('https://index.mysteel.com/zs/newprice/getBaiduChartMultiCity.ms?catalog=%25E8%259E%25BA%25E7%25BA%25B9%25E9%2592%25A2_%3A_%25E8%259E%25BA%25E7%25BA%25B9%25E9%2592%25A2&city=%25E4%25B8%258A%25E6%25B5%25B7%3A15278&spec=HRB400E%252020MM_%3A_HRB400E_20MM&startTime=2021-08-10&endTime=2021-08-12&callback=json')
r.json()
{'marketDatas': .........}
However, if I change the values of endTime or startTime query parameters, I get a 401 error:
Missing authentication parameters!
It looks like I can only send requests that are already sent from my browser (I'm running a jupyter notebook in the same browser). Even if I add request headers like in the Network tab, I still get the same error:
params= {'catalog':'%E8%9E%BA%E7%BA%B9%E9%92%A2_:_%E8%9E%BA%E7%BA%B9%E9%92%A2' ,'city':'%E4%B8%8A%E6%B5%B7:15278' ,
'spec':'HRB400E%2020MM_:_HRB400E_20MM' ,'startTime':'2021-08-10', 'endTime':'2021-08-12', 'callback':'json'}
headers = {'Host': 'index.mysteel.com',
'User-Agent': 'Mozilla/5.0',
'Referer': 'https://index.mysteel.com/price/getChartMultiCity_1_0.html',
'appKey': '47EE3F12CF0C443F851EFDA73AC815',
'Cookie':'href=https%3A%2F%2Findex.mysteel.com%2Fprice%2FgetChartMultiCity_1_0.html; accessId=5d36a9e0-919c-11e9-903c-ab24d11b; pageViewNum=2'}
url = 'https://index.mysteel.com/zs/newprice/getBaiduChartMultiCity.ms'
r = requests.get(url, params=params, headers=headers)
What am I missing here? I am not authenticating myself in the browser, so there shouldn't be any authentication issues. Is there some other parameter missing in my headers dict?
PS: These are all the request headers:
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,sq;q=0.8,de;q=0.7
appKey: 47EE3F12CF0C443F851EFDA73AC815
Connection: keep-alive
Cookie: href=https%3A%2F%2Findex.mysteel.com%2Fprice%2FgetChartMultiCity_1_0.html; accessId=5d36a9e0-919c-11e9-903c-ab24d11b; pageViewNum=2
dnt: 1
Host: index.mysteel.com
Referer: https://index.mysteel.com/price/getChartMultiCity_1_0.html
sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"
sec-ch-ua-mobile: ?0
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
sec-gpc: 1
sign: D404184969BF3B8C081A9F0C913AF68E
timestamp: 1629136315665
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36
version: 1.0.0
X-Requested-With: XMLHttpRequest
I'm trying to log in to a website using requests module. While creating a script to do so, I could notice that the payload used in there is completely different from the conventional approach. This is exactly how the payload +åEMAIL"PASSWORD(0 looks like. This is the content type parameters content-type: application/grpc-web+proto.
The following is what I see in dev tools when I log in to that site manually:
General
--------------------------------------------------------
Request URL: https://grips-web.aboutyou.com/checkout.CheckoutV1/logInWithEmail
Request Method: POST
Status Code: 200
Remote Address: 104.18.9.228:443
Response Headers
--------------------------------------------------------
Referrer Policy: strict-origin-when-cross-origin
access-control-allow-credentials: true
access-control-allow-origin: https://www.aboutyou.cz
access-control-expose-headers: Content-Encoding, Vary, Access-Control-Allow-Origin, Access-Control-Allow-Credentials, Date, Content-Type, grpc-status, grpc-message
cf-cache-status: DYNAMIC
cf-ray: 67d009674f604a4d-SIN
content-encoding: gzip
content-type: application/grpc-web+proto
date: Wed, 11 Aug 2021 08:19:04 GMT
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
set-cookie: __cf_bm=a45185d4acac45725b46236884673503104a9473-1628669944-1800-Ab2Aos6ocz7q8B8v53oEsSK5QiImY/zqlTba/Y0FqpdsaQt2c10FJylcwTacmdovm6tjGd8hLdy/LidfFCtOj70=; path=/; expires=Wed, 11-Aug-21 08:49:04 GMT; domain=.aboutyou.com; HttpOnly; Secure; SameSite=None
vary: Origin
Request Headers
--------------------------------------------------------
:authority: grips-web.aboutyou.com
:method: POST
:path: /checkout.CheckoutV1/logInWithEmail
:scheme: https
accept: */*
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
cache-control: no-cache
content-length: 48
content-type: application/grpc-web+proto
origin: https://www.aboutyou.cz
pragma: no-cache
referer: https://www.aboutyou.cz/
sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"
sec-ch-ua-mobile: ?0
sec-fetch-dest: empty
sec-fetch-mode: cors
sec-fetch-site: cross-site
user-agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36
x-grpc-web: 1
Request Payload
--------------------------------------------------------
+åEMAIL"PASSWORD(0
This is what I've created so far (can't find any way to fill in the payload):
import requests
from bs4 import BeautifulSoup
start_url = 'https://www.aboutyou.cz/'
post_link = 'https://grips-web.aboutyou.com/checkout.CheckoutV1/logInWithEmail'
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.3',
'content-type': 'application/grpc-web+proto',
'origin': 'https://www.aboutyou.cz',
'referer': 'https://www.aboutyou.cz/',
'x-grpc-web': '1'
}
payload = {
}
with requests.Session() as s:
s.headers.update(headers)
r = s.post(post_link,data=payload)
print(r.status_code)
print(r.url)
Steps to log in to that site manually:
Go to this site
This is how to get the login form
Login form looks like this
How can I log in to that site using requests module?
I don't think that you'll be able to use Python Requests to login to your target site.
Your post_link url:
post_link = 'https://grips-web.aboutyou.com/checkout.CheckoutV1/logInWithEmail'
states that it is: gRPC requires HTTP/2 and Python Requests send HTTP/1.1 requests only.
Additionally, I noted that the target site also uses CloudFlare, which is difficult to bypass with Python, especially when using Python Requests
'Set-Cookie': '__cf_bm=11d867459fe0951da4157b475cf88eb3ab7658fb-1629229293-1800-AeFomlmROcmUYcRosxxcSnoJkGOW/WXjUe1WxK6SkM2eXIbnAqXRlpwOkpvOfONrbApJd4Qwj+a8+kOzLAfpHIE=; path=/; expires=Tue, 17-Aug-21 20:11:33 GMT; domain=.aboutyou.com; HttpOnly; Secure; SameSite=None', 'Vary': 'Accept-Encoding', 'Server': 'cloudflare', 'CF-RAY': '6805616b8facf1b2-ATL', 'Content-Encoding': 'gzip'}
Here are previous Stack Overflow questions on Python Requests with gRPC
Can't make gRPC work with python requests rest api call
Send plain JSON to a gRPC server using python
I looked through the GitHub repository for Python Requests and saw that HTTP/2 has been a requested feature for almost 7 years.
During my research, I discovered HTTPX, which is a HTTP client for Python 3, which provides sync and async APIs, and support for both HTTP/1.1 and HTTP/2. The documentation states that the package is stable, but is still considered a beta at this point. I would recommend trying HTTPX to see if it solves your issue with logging into your target site.
I want to make a program where I can send HTTP post requests and respond.
So, I want to send THIS post:
POST https: //example.com/index.php?s=&&app=box&module=ajax§ion=coreAjax&secure_key=&type=submit&lastid=87311&global=1 HTTP/1.1
Host: example.com
Connection: keep-alive
Content-Length: 10
Accept: text/javascript, text/html, application/xml, text/xml, */*
X-Prototype-Version: 1.7.2
Origin: https://example.com
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
Content-type: application/x-www-form-urlencoded; charset=UTF-8
Referer: https://x.com/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8
Cookie: cookieconsent_status=dismiss;
And then enter the request body:
message= # Which I will make: "message= %s" % (messagex))
But I do not know how to send them and can´t seem to find any way online, could someone help please?
The main parts are:
import requests # you have to install this library, with pip for example
# define your custom headers (as many as you want)
headers = {
'X-Prototype-Version': '1.7.2'
}
# define your URL params (!= of the body of the POST request)
params = {
'your_first_param': 'its_value',
'your_second_param': 'its_value'
}
# define the body of the POST request
data = {
'message' : 'your message'
}
# send the POST request
response = requests.post('https://example.com/index.php', params=params, data=data, headers=headers)
# here is the response
print response.text
Hope that helps.
I am currently trying to get data off http://www.spotrac.com/ that requires being signed in. My current attempt uses this code(which I got by going through a bunch of other stack overflow questions on a similar topic)
from bs4 import BeautifulSoup as bs
from requests import session
payload = {
'id': 'contactForm',
'cmd': 'http://www.spotrac.com/signin/submit/',
'email': '*****',
'password': '*****'
}
with session() as c:
r_login = c.post('http://www.spotrac.com/signin/', data=payload)
print(r_login.headers)
response = c.get('http://www.spotrac.com/nba/cleveland-cavaliers/lebron-james')
print(response.cookies)
soup=bs(response.text, 'html.parser')
with open('ex.html','w') as f:
f.write(soup.prettify())
My current code does everything right, except I am not logged in when I'm making the request.
Thanks
You're sending POST request to a wrong URL, and with an incorrect payload as well.
POST http://www.spotrac.com/signin/submit/ HTTP/1.1
Host: www.spotrac.com
Connection: keep-alive
Content-Length: 86
Cache-Control: max-age=0
Origin: http://www.spotrac.com
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Referer: http://www.spotrac.com/signin/
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8
Cookie: cisession=a%3A5%3A%7Bs%3A10%3A%22session_id%22%3Bs%3A32%3A%2206021e191bdbbaf955f111f67b961056%22%3Bs%3A10%3A%22ip_address%22%3Bs%3A11%3A%22119.9.105.6%22%3Bs%3A10%3A%22user_agent%22%3Bs%3A108%3A%22Mozilla%2F5.0+%28Windows+NT+6.1%3B+WOW64%29+AppleWebKit%2F537.36+%28KHTML%2C+like+Gecko%29+Chrome%2F55.0.2883.87+Safari%2F537.36%22%3Bs%3A13%3A%22last_activity%22%3Bi%3A1485487245%3Bs%3A9%3A%22user_data%22%3Bs%3A0%3A%22%22%3B%7Dd6089620b21ecce6837161605055ae04; _ga=GA1.2.910256341.1481865346; _gali=contactForm
redirect=http%3A%2F%2Fwww.spotrac.com%2F&email=sdfs%40gmail.com&password=lkasjdflksjad
HTTP/1.1 302 Found
Server: nginx
Date: Fri, 27 Jan 2017 04:21:16 GMT
Content-Type: text/html
Content-Length: 0
Connection: keep-alive
Set-Cookie: cisession=a%3A5%3A%7Bs%3A10%3A%22session_id%22%3Bs%3A32%3A%22badb1275aee1cdad6736a6b4bb1ce809%22%3Bs%3A10%3A%22ip_address%22%3Bs%3A11%3A%22119.9.105.6%22%3Bs%3A10%3A%22user_agent%22%3Bs%3A108%3A%22Mozilla%2F5.0+%28Windows+NT+6.1%3B+WOW64%29+AppleWebKit%2F537.36+%28KHTML%2C+like+Gecko%29+Chrome%2F55.0.2883.87+Safari%2F537.36%22%3Bs%3A13%3A%22last_activity%22%3Bi%3A1485490876%3Bs%3A9%3A%22user_data%22%3Bs%3A0%3A%22%22%3B%7Dad486866c32cac526487707cea85b8a9; expires=Fri, 10-Feb-2017 04:21:16 GMT; path=/
Location: http://www.spotrac.com/register/
X-Powered-By: PleskLin
MS-Author-Via: DAV
As you can see from above session, the correct url should be http://www.spotrac.com/signin/submit/, and payload string is redirect=http%3A%2F%2Fwww.spotrac.com%2F&email=sdfs%40gmail.com&password=lkasjdflksjad, which is basically:
payload = {'redirect': 'http://www.spotrac.com/',
'email': mail_address,
'password': password}
Also make sure simulate headers with correct parameters, then you're good to go.
I want to parse a data from a page but this data is showing only registered users. So, I have to login first then parse it. There is no problem with the parsing side but I have problem on login side. Here is my only login code:
I have changed private domain name to domain.com
import requests
with requests.Session() as c:
url = 'https://domain.com/giris?returnUrl=https://domain.com/'
USERNAME = 'xxxxxxx#gmail.com'
PASSWORD = '1111111'
c.get(url)
__RequestVerificationToken = c.cookies['__RequestVerificationToken']
login_data = dict(__RequestVerificationToken=__RequestVerificationToken, UserName=USERNAME, Password=PASSWORD, ReturnUrl='https://domain.com/', RememberMe='false')
c.post(url, data=login_data, headers={"Referer": "https://domain.com/giris?returnUrl=https%3A%2F%2Fdomain.com%2F"})
page = c.get('https://domain.com/mesaj/')
print page.content
If login is succeeded I need to see https://domain.com/mesaj/ page but it redirects login page again because of unsuccessful login.
Also, here is the Header of login page that is captures from Google Chrome:
General
Request URL:https://domain.com/giris
Request Method:POST
Status Code:302 Found
Remote Address:176.53.43.2:443
Response Headers
view source
Response Headers
Cache-Control:no-cache
Content-Length:140
Content-Type:text/html; charset=utf-8
Date:Tue, 24 Jan 2017 16:51:30 GMT
Expires:-1
Location:https://domain.com/
Pragma:no-cache
Set-Cookie:a=vjFFqBh+ZZMKr71K2XrBCr5SutOMAOpWjv1RAS5hRYMrR2RojaTV/wgIP8HiUOjdMU7x28DpfxRsCnfSvLeLHPvGTBKjwF0O5W99julK7w23vdctrnE5FDBlhXSSB9nCQm+DB3vNgGjxEr+DNRMrWNwMZWbSQID+klPDtUnReAJQA/GfLdoo2izsD0HP6tir; path=/; HttpOnly
Strict-Transport-Security:max-age=31536000; includeSubDomains; preload
X-Content-Type-Options:nosniff
X-Frame-Options:DENY
X-XSS-Protection:1; mode=block
Request Headers
view source
Request Headers
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip, deflate, br
Accept-Language:tr-TR,tr;q=0.8,en-US;q=0.6,en;q=0.4
Cache-Control:max-age=0
Connection:keep-alive
Content-Length:251
Content-Type:application/x-www-form-urlencoded
Cookie:iq=c3bae8d44a624f03a431c8df6741af36; __gfp_64b=0icQApqT.NFM3ZR0rZLAyXumlDI4n2eStqQJ74n4H...U7; __gads=ID=bb39b708ca7de25d:T=1484676131:S=ALNI_MYxcRsRoDIaQmsY859bPz_jriFRDA; ASP.NET_SessionId=efsub3xz101y542pyjhugvi2; cookies_info_viewed=yes; notheme=1; __adm_int_sc=1; __adm_int=1; __RequestVerificationToken=xzayOzscFJ3a_m4C5jF8eaKrW7F_Yen7umGMm_nZxDPKmO5rUKacPc4yHK63wVqQwd2S_H2mLiKt_ROW2pCG1B5ZTEtytYF-GU0khK2BlnM1; _gat=1; _ga=GA1.2.633914475.1484676130; __asc=14f63ba5159d14e81061abc20c9; __auc=328f9ce5159ad97e58cfeb70218
Host:domain.com
Origin:https://domain.com
Referer:https://domain.com/giris?returnUrl=https%3A%2F%2Fdomain.com%2F
Upgrade-Insecure-Requests:1
User-Agent:Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36
Form Data
view source
view URL encoded
Form Data
__RequestVerificationToken:F2ZCmIge7rSV4A4Xoelf3aweaDQ9vNHew16Bfb6GDSlpeFQeQ_cfmV6UrFhNRWRBqvGPXzXrxVLAIXgbKI-08Q0fD3Vfttezq5hTkMFYTwo1
ReturnUrl:https://domain.com/
UserName:xxxxxxx#gmail.com
Password:1111111
RememberMe:false