I have written a web server in python, the client uses web-browser to send the requests like this:
http://localhost:13555/ChessBoard_x16_y16.bmp
On server side when I am printing the client-request it is like this:
GET /ChessBoard_x16_y16.bmp HTTP/1.1
Host: localhost:13555
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0
.8
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like
Gecko) Chrome/43.0.2357.81 Safari/537.36
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
GET /favicon.ico HTTP/1.1
Host: localhost:13555
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like
Gecko) Chrome/43.0.2357.81 Safari/537.36
Accept: */*
Referer: http://localhost:13555/ChessBoard_x16_y16.bmp
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
But I only want to get & print the actual URL like:
Referer: http://localhost:13555/ChessBoard_x16_y16.bmp
please tell how can I do this?
Do a regular reg-ex search in result set within python:
import re
for line in <your results set>:
line = line.rstrip()
if re.search('^Referer:', line) :
print line
If you don't have reg-ex:
for line in <your results set>:
line = line.rstrip()
if '^Referer:' in line) :
print line
Related
I need your help in putting together a post request.
The output I get is html, but the plan was to get the following:
Below are all the data for the desired item:
General
Request URL: https://dgslivebetting.betonline.ag/ngwbet.aspx/gvFrameHtml
Request Method: POST
Status Code: 200
Remote Address: 104.17.64.19:443
Referrer Policy: strict-origin-when-cross-origin
Response Headers
cache-control: no-cache
cf-cache-status: DYNAMIC
cf-ray: 76800ae95afc35b3-DME
content-encoding: br
content-type: application/json; charset=utf-8
date: Thu, 10 Nov 2022 16:07:42 GMT
expires: -1
pragma: no-cache
server: cloudflare
set-cookie: server_persistent=!zk3OrErnBetHZkiKJcby5Il79pzHsf7dxKD0PcVuB54Z2dznuEbqgGAVDWLDvoqpVSDnVq+Jtf91LHo=; path=/; Httponly; Secure
x-newrelic-app-data: PxQFUFRTDQMHR1NRBQkOVVABDhFORDQHUjZKA1ZLVVFHDFYPHjZWADdTRRcPAF0cXgMWAFJFaAcXQU4cBRAlEFEPXSpMVVgQH1UXUR1RHVBUAA9QVloUHgFIQ1YCAg9fAAgFAFZXUFYDUQBAFF5VXkAAZA==
Request Headers
:authority: dgslivebetting.betonline.ag
:method: POST
:path: /ngwbet.aspx/gvFrameHtml
:scheme: https
accept: */*
accept-encoding: gzip, deflate, br
accept-language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7
content-length: 12
content-type: application/json; charset=UTF-8
cookie: \_xpid=574830729; \_xpkey=K_F3GRHECOTdjT306mOafHByLTxopGhY; LPVID=MxZmQyM2Q5OTFlOTU0ZTJk; \_hjSessionUser_2115245=eyJpZCI6IjQ3MzAxYmQwLTQ4ODgtNWNjMC1hZGZjLWJlZDBmNDgwZDJjZCIsImNyZWF0ZWQiOjE2NjY1NTY0MjQwOTIsImV4aXN0aW5nIjp0cnVlfQ==; CT.CONTENT.NA.STATUS=1; \_gid=GA1.2.1666042031.1667883501; PreviousUrlNav=%2Fsportsbook%2Flive-betting; chQuickBet=undefined; inputAmount=100.00; kameleoonVisitorCode=\_js_ti27yqxpj7dd4k1x; DD-LINK-NAREDIRECT=0; ASP.NET_SessionId=5acflzzgqtjdvsnjc5wtwuys; tz=Eastern%20Standard%20Time; btpdb.1PR3l09.dGZjLjY2ODI2ODU=U0VTU0lPTg; oddsfmt=dec; \_hjSession_2115245=eyJpZCI6Ijk2NzBiMjNkLWY4MGQtNDM5OS1hYWNhLWQyODBjNmZlYzNkMSIsImNyZWF0ZWQiOjE2NjgwOTM2NzY4OTUsImluU2FtcGxlIjpmYWxzZX0=; \_hjAbsoluteSessionInProgress=0; \_hjIncludedInSessionSample=0; LPSID-90263191=bLgFHbiuTjOcwCg1FgR16g; \__cf_bm=5LozQOf4P4COCn1rVD5emsVzukFSNbWdS7kvBVodzJ4-1668096251-0-AQ+nY5HeihIwV+gAI1oaFKJJxOtgXWs5czIr198Ffrh18P1q4nriEcszp/j7dwjuDjVuki1jlT6IByy2ewOCcXSUWavF+3MCcBF4Yb8sfDPVkvoSufxJ46feYuPiCiPcw0eW9oTUnrmZNcEkZ1732RDx6LWq1OElUvT0Uk6sk1n1; \_gat_UA-190679354-1=1; \_ga_KC6V6402HY=GS1.1.1668096234.18.1.1668096460.0.0.0; \_ga=GA1.1.1142263304.1666556424; server_persistent=!Tdbrpsz3tJ8jlNmKJcby5Il79pzHsfLVz91fFnDrXObiJE45d6idCUAVcW4Qmd/g598vNFaqTVuVRvk=
origin: https://dgslivebetting.betonline.ag
referer: https://dgslivebetting.betonline.ag/ngwbet.aspx
sec-ch-ua: "Google Chrome";v="107", "Chromium";v="107", "Not=A?Brand";v="24"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
sec-fetch-dest: empty
sec-fetch-mode: cors
sec-fetch-site: same-origin
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36
x-newrelic-id: VgcFUVNTDxACV1NaDgIDVlw=
x-requested-with: XMLHttpRequest
Please help me figure out how I can get what I want.
My code:
import requests
import cloudscraper
scraper = cloudscraper.create_scraper()
url = 'https://dgslivebetting.betonline.ag/ngwbet.aspx/gvFrameHtml'
data = {"gameID":0}
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36",
'Referer': "https://dgslivebetting.betonline.ag/ngwbet.aspx/gvFrameHtml"
}
r = requests.post(url, data=data, headers=headers)
print(r.text)
In order to get JSON back, you need to add the Content-Type header to your request.
Your current examples shows you are only sending these headers:
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36",
'Referer': "https://dgslivebetting.betonline.ag/ngwbet.aspx/gvFrameHtml"
}
At the very least, you'll need to add Content-Type: application/json; charset=UTF-8 to the request, otherwise, requests is doing an application/x-www-form-urlencoded form post which is why you're getting back HTML from this site instead of JSON.
I'm developing a proxy and I came across this problem where I can't resend the request and get the HTML output(ofc I thought about splitting and these things but it won't help later)
GET http://www.example.com/ HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Connection: keep-alive
Upgrade-Insecure-Requests: 1
now I wanna know what do we call this and I wanna know if I can use it without extracting the URL and headers and requesting it
i found a solution by using socket
request="""
GET http://www.example.com/ HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Connection: keep-alive
Upgrade-Insecure-Requests: 1
"""
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
webserver=re.findall("(?P<url>https?://[^\s]+)", request)[0]
s.connect((webserver, 80))
s.send(request)
html = s.recv(1024)
conn.send(html)
I have a list of named locations and a set of lat/lng bounds that I want to plug in to the Google Maps API and have it find the locations for me. The names can be very vague, like simply "the boarding school". Using the lat/lng bounds, is there a way I can get GMaps to find these vaguely named locations within the coordinates provided?
My application is web-based and powered by Python Flask in the backend. I've tried looking into Maps' Place Search, but it seems like it can only 'prefer' a certain area to search in, and with my vague place names, it doesn't do well:
https://maps.googleapis.com/maps/api/place/findplacefromtext/xml?input=beach&inputtype=textquery&fields=formatted_address,geometry&locationbias=rectangle:43.3145,-79.8236|43.3490,-79.7741&key=XXXXXXX
This query has a bias covering part of Burlington, ON, but the result is in the neighbouring town of Oakville, significantly out of bounds. If you perform the search with the term "Burlington beach" instead however, it finds the beach that is within bounds.
I need the query to find the beach in Burlington, simply given the term "beach", and bounds that said beach falls within.
Edit: here are my HTTP requests+headers in both Chrome and Edge when testing the original query and Evan's smaller query (URL in comments):
==== Google Chrome: Original Request ====
:authority: maps.googleapis.com
:method: GET
:path: /maps/api/place/findplacefromtext/json?input=beach&inputtype=textquery&fields=formatted_address,geometry&locationbias=rectangle:43.3145,-79.8236|43.3490,-79.7741&key=
:scheme: https
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
accept-encoding: gzip, deflate, br
accept-language: en-CA,en;q=0.9,it;q=0.8,el-GR;q=0.7,el;q=0.6
cache-control: max-age=0
dnt: 1
sec-fetch-mode: navigate
sec-fetch-site: none
sec-fetch-user: ?1
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36
x-client-data: CIi2yQEIprbJAQjBtskBCKmdygEIup/KAQioo8oBCOKoygEIl63KAQjNrcoBCMqvygEIh7TKARjwsMoB
==== Google Chrome: Evan's Request ====
:authority: maps.googleapis.com
:method: GET
:path: /maps/api/place/findplacefromtext/json?input=beach&inputtype=textquery&fields=formatted_address,geometry&locationbias=rectangle:43.3145,-79.8236|43.3490,-79.800879&key=
:scheme: https
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
accept-encoding: gzip, deflate, br
accept-language: en-CA,en;q=0.9,it;q=0.8,el-GR;q=0.7,el;q=0.6
cache-control: max-age=0
dnt: 1
sec-fetch-mode: navigate
sec-fetch-site: none
sec-fetch-user: ?1
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36
x-client-data: CIi2yQEIprbJAQjBtskBCKmdygEIup/KAQioo8oBCOKoygEIl63KAQjNrcoBCMqvygEIh7TKARjwsMoB
==== Edge: Original Request ====
Request URL: https://maps.googleapis.com/maps/api/place/findplacefromtext/json?input=beach&inputtype=textquery&fields=formatted_address,geometry&locationbias=rectangle:43.3145,-79.8236|43.3490,-79.7741&key=
Request Method: GET
Status Code: 200 /
Accept: text/html, application/xhtml+xml, application/xml; q=0.9, */*; q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: en-CA
Cache-Control: max-age=0
Host: maps.googleapis.com
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362
==== Edge: Evan's Request ====
Request URL: https://maps.googleapis.com/maps/api/place/findplacefromtext/json?input=beach&inputtype=textquery&fields=formatted_address,geometry&locationbias=rectangle:43.3145,-79.8236|43.3490,-79.800879&key=
Request Method: GET
Status Code: 200 /
Accept: text/html, application/xhtml+xml, application/xml; q=0.9, */*; q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: en-CA
Host: maps.googleapis.com
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362
This inconsistency appears to be related to the following issue, reported in Google's Issue Tracker:
https://issuetracker.google.com/issues/35822155
The language parameter influences the results you get from Place Search requests, and even more so with such generic/broad queries. So it's intended behavior for the API.
Potential alternatives to Find Place include using the Nearby Search or the Text Search services. These are more appropriate for ambiguous queries and you can filter out results that are outside a given radius+location.
Hope this helps!
POST /somelink
Host: hostname
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:47.0) Gecko/20100101
Firefox/47.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Content-Type: application/x-www-form-urlencoded
Content-Length: 683
Cookie: xxx
Connection: keep-alive
access_token=xxx
How can I form an api call with these details?
I already tried:
payload={ "POST":"somelink",
"Host":"hostname",
"User-Agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0",
"Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language":"en-US,en;q=0.5",
"Accept-Encoding":"gzip, deflate, br",
"Content-Type": "application/x-www-form-urlencoded",
"Content-Length": "683",
"Cookie":" xxx",
"Connection":"keep-alive",
"access_token":"xxx"}
headers = {}
r = requests.post(url, data=json.dumps(payload), headers=headers)
But if I try this way I get 'Connection aborted.', gaierror(-2, 'Name or service not known'.
Can anyone help me with this ?
gaierror means that the DNS lookup is failing: What does this socket.gaierror mean?
Which means Python is unable to resolve the host name to an IP address to initiate the request. Example:
>>> from socket import gethostbyname
>>> gethostbyname('www.google.com')
'172.217.26.196'
>>> gethostbyname('www.wrongurlthatdoesnotexist.com')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
socket.gaierror: [Errno -2] Name or service not known
Here is what my XHR data looks like when captured in chrome
Request Header
POST my_url?X-Progress-ID=ee821652321919bc7ae61fbe0b625990&userpkgname=file_name.tar.gz HTTP/1.1
Host: 10.110.134.28
Connection: keep-alive
Content-Length: 17461
Accept: application/json, text/plain, */*
Origin: https://10.110.134.28
X-XSRF-TOKEN: bGIwdfFE-oaL_1yVrCzw0iHvv4yUHLC28xjw
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36
Content-Type: multipart/form-data; boundary=----WebKitFormBoundarybfK9jSdLoc2Mpj0i
Referer: https://10.110.134.28/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8
Cookie: XSRF-TOKEN=bGIwdfFE-oaL_1yVrCzw0iHvv4yUHLC28xjw; sid=s%3AXglfJNLQ9zzp3eHjQ2QOpk19kFKDDMvJ.ZMKyZd1Gx13lz2MnJgty5WncnilySzfoThGktkhlk4w
Payload
------WebKitFormBoundarybfK9jSdLoc2Mpj0i
Content-Disposition: form-data; name="package"; filename="file_name.tar.gz"
Content-Type: application/x-gzip
------WebKitFormBoundarybfK9jSdLoc2Mpj0i--
And this is how I am building my request.
files = {'package': (<file_name>, open(config_path, 'rb'), 'application/x-gzip')}
request.post(url, files=files)
This is how my request header looks like
{
'Content-Length' : '17449',
'Accept-Encoding' : 'gzip, deflate',
'Accept' : '*/*',
'User-Agent' : 'python-requests/2.10.0',
'Connection' : 'keep-alive',
'Cookie' : 'XSRF-TOKEN=zmklLEL0-gDJOfNBk113MuTpBkLo0j6MAzw0; sid=s%3AC2JZDCfpg_CgkU7qSlS5YTvWXwpgMX35.5nU7W02TPNYtMkIQ4W%2B1bjd87A7KyJbh3shoNqqADXE',
'Content-Type' : 'multipart/form-data; boundary=270d9e02bf214dc7a09c3081cba5b0e0',
'XSRF-TOKEN' : 'zmklLEL0-gDJOfNBk113MuTpBkLo0j6MAzw0'
}
When I make the request I get 502 bad gateway response that too after few seconds while on chrome I get 200 OK instantly
So most probably I am not building my request correctly. Any suggestions?