Develop Reference

How to bypass Cloudflare with Python on GET requests?

I want to bypass Cloudflare on a GET request I have tried using Cloudscraper which worked for me in the past but now seems decreped.
I tried:
import cloudscraper
import requests
ses = requests.Session()
ses.headers = {
'referer': 'https://magiceden.io/',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36',
'accept': 'application/json'
}
scraper = cloudscraper.create_scraper(sess=ses)
hookLink = f"https://magiceden.io/launchpad/planetarians"
meG = scraper.get("https://api-mainnet.magiceden.io/launchpads/planetarians")
print(meG.status_code)
print(meG.text)
The issue seems to be that I'm getting a captcha on the request

The python library works well (I never knew about it), the issue is your user agent. Cloudflare uses some sort of extra checks to determine whether you're faking it.
For me, any of the following works:
ses.headers = {
'referer': 'https://magiceden.io/',
'accept': 'application/json'
}
ses.headers = {
'accept': 'application/json'
}
And also just:
scraper = cloudscraper.create_scraper()
meG = scraper.get("https://api-mainnet.magiceden.io/launchpads/planetarians")
EDIT:
You can use this dict syntax instead to fake the user agent (as per the manual)
scraper = cloudscraper.create_scraper(
browser={
'browser': 'chrome',
'platform': 'windows',
'desktop': True
}
)

request from python get different response from nodejs

I am trying to do the same request from nodejs.
The python code is
import requests
r = requests.post(url,
data=data,
headers={
'User-Agent': self.ua,
'Content-Type': 'application/x-www-form-urlencoded'
}
)
and in node I tried node-fetch and Axios and request but not getting the same response, I also tried CURL from bash but getting the same response of node, I tried to print python headers print(r.request.headers) and copy paste it in node but getting different response
Axios.post(url, {
data,
headers: {
"User-Agent":
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36",
"Content-Type": "application/x-www-form-urlencoded",
}
})
.then(text => console.log(text.data))
.catch(err => {
console.log(err);
});
I am getting different result in python I get what I expect but in node I am getting html response
Sorry, could not complete request because: <div class="tk-intro" style="font-size: 14px;color:#ff090f;">application information was not supplied.</div>
but in python works fine

I tried to print request headers and url and data and found I should to to convert data to query string like that
"appleId=email#gmail.com&accountPassword=xxxxxx"
instead of passing it as JSON
{
"appleID": "email#gmail.com",
"accountPassword": "xxxx"
}

Can't find text from page using Python BS4

I am trying to learn how to use BS4 but I ran into this problem. I try to find the text in the Google Search results page showing the number of results for the search but I can't find no text 'results' neither in the html_page nor in the soup HTML parser. This is the code:
from bs4 import BeautifulSoup
import requests
url = 'https://www.google.com/search?q=stack'
res = requests.get(url)
html_page = res.content
soup = BeautifulSoup(html_page, 'html.parser')
print(b'results' in html_page)
print('results' in soup)
Both prints return False, what am I doing wrong? How to fix that?
EDIT:
Turns out the language of the webpage was a problem, adding &hl=en to the URL almost fixed it.
url = 'https://www.google.com/search?q=stack&hl=en'
The first print is now True but the second is still False.

requests library when returning the response in form of response.content usually returns in a raw format. So to answer your second question, replace the res.content with res.text.
from bs4 import BeautifulSoup
import requests
url = 'https://www.google.com/search?q=stack'
res = requests.get(url)
html_page = res.text
soup = BeautifulSoup(html_page, 'html.parser')
print('results' in soup)
Output: True
Keep in mind, Google is usually very active in handling scrapers. To avoid getting blocked/captcha'ed, you can add a user agent to emulate a browser. :
# This is a standard user-agent of Chrome browser running on Windows 10
headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36' }
Example:
from bs4 import BeautifulSoup
import requests
headers={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'}
resp = requests.get('https://www.amazon.com', headers=headers).text
soup = BeautifulSoup(resp, 'html.parser')
...
<your code here>
Additionally, you can add another set of headers to pretend like a legitimate browser. Add some more headers like this:
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',
'Accept' : 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language' : 'en-US,en;q=0.5',
'Accept-Encoding' : 'gzip',
'DNT' : '1', # Do Not Track Request Header
'Connection' : 'close'
}

It's not because res.content should be changed to res.text as 0xInfection mentioned, it would still return the result.
However, in some cases, it will only return bytes content if it's not gzip or deflate transfer-encodings, which are automatically decoded by requests to a readable format (correct me in the comments or edit this answer if I'm wrong).
It's because there's no user-agent specified thus Google will block a request eventually because default requests user-agent is python-requests and Google understands that it's a bot/script. Learn more about request headers.
Pass user-agent into request headers:
headers = {
"User-Agent":
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}
request.get('YOUR_URL', headers=headers)
Code and example in the online IDE:
import requests, lxml
from bs4 import BeautifulSoup
headers = {
"User-Agent":
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}
params = {
"q": "fus ro dah definition", # query
"gl": "us", # country to make request from
"hl": "en" # language
}
response = requests.get('https://www.google.com/search',
headers=headers,
params=params).content
soup = BeautifulSoup(response, 'lxml')
number_of_results = soup.select_one('#result-stats nobr').previous_sibling
print(number_of_results)
# About 114,000 results
Alternatively, you can achieve the same thing by using Google Direct Answer Box API from SerpApi. It's a paid API with a free plan.
The difference in your case is that you only need to extract the data you want without thinking about how to extract stuff or figure out how to bypass blocks from Google or other search engines since it's already done for the end-user.
import os
from serpapi import GoogleSearch
params = {
"engine": "google",
"q": "fus ro dah definition",
"api_key": os.getenv("API_KEY"),
}
search = GoogleSearch(params)
results = search.get_dict()
result = results["search_information"]['total_results']
print(result)
# 112000
Disclaimer, I work for SerpApi.

How to scrape from XHR in Chrome ( Python )

I'm trying to scrape information from the facebook game "Coin Master"
Inspect Element > Network > XHR
This brings up "Balance" which i need to access since it contains the information i need to track
Picture example
Coin Master FB Link to Test
But I do not know what module I need to achieve this. I've used BeautifulSoup, Requests in the past but this isn't as straight forward for me.
Any help/insight to my issue would be much appreciated!
Thanks & kind regards

You need to inspect the requests and under Form data find your data for the requests.
import requests
import json
data = {
"Device[udid]": "",
"API_KEY": "",
"API_SECRET": "",
"Device[change]": "",
"fbToken": ""
}
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36"
}
url = "https://vik-game.moonactive.net/api/v1/users/rof4__cjsvfw2s604xrw1lg5ex42qwc/balance"
r = requests.post(url, data=data, headers=headers)
data = r.json()
print(data["coins"])

Google Finance recognizes my Python script as a bot and blocks it

I wrote a script that retrieves stock data on google finance and prints it out, nice and simple. It always worked, but since this morning I only get a page that tells me that I'm probably an automated script instead of the stock data. Of course, being a script, I can't pass the captcha. What can I do?

well, you finally reached a quite challenging realm. decode the captcha.
there do exist OCR approaches to decode simple captcha into code. not seems to work for google captcha.
I heard there are some companies provide manual captcha decoding services, you can try to use some. ^_^ LOL
ok, to be serious, if google don't want you to do it that way, then it is not easy to decode those captchas. After all, why google on finance data, there are a lot other providers, right? try to scrape those websites.

You can try to solve the blocking issue by adding headers where your user-agent will be specified, this is necessary for Google to recognize the request as from a user, and not as from a bot, and not block it:
# https://docs.python-requests.org/en/master/user/quickstart/#custom-headers
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
}
An additional step could be to rotate user-agents:
import requests, random
user_agent_list = [
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Safari/605.1.15',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko/20100101 Firefox/77.0',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36',
]
for _ in user_agent_list:
#Pick a random user agent
user_agent = random.choice(user_agent_list)
#Set the headers
headers = {'User-Agent': user_agent}
requests.get('URL', headers=headers)
In addition to the rotate user-agent, you can rotate proxies (ideally residential) that can be used in combination with CAPTCHA solver to bypass CAPTCHA.
To parse dynamic websites using web browser automation, you can use curl-impersonate or selenium-stealth which can bypass most CAPTCHAs, but the option using browser automation could be CPU, RAM expensive and could be difficult to run in parallel.
There's a reducing the chance of being blocked while web scraping blog post if you need a little bit more info.
As an alternative you can use Google Finance Markets API from SerpApi. It's a paid API with the free plan.
The difference is that it will bypass blocks (including CAPTCHA) from Google, no need to create the parser and maintain it.
Example of SerpApi code for extracting Most Active on the main page of Google Finance in the online IDE.
from serpapi import GoogleSearch
import json, os
params = {
"engine": "google_finance_markets", # serpapi parser engine. Or engine to parse ticker page: https://serpapi.com/google-finance-api
"trend": "most-active", # parameter is used for retrieving different market trends: https://serpapi.com/google-finance-markets#api-parameters-search-query-trend
"api_key": "..." # serpapi key, https://serpapi.com/manage-api-key
}
market_trends_data = []
search = GoogleSearch(params)
results = search.get_dict()
most_active = results["markets"]
print(json.dumps(most_active, indent=2, ensure_ascii=False))
Example output:
[
{
"stock": ".DJI:INDEXDJX",
"link": "https://www.google.com/finance/quote/.DJI:INDEXDJX",
"serpapi_link": "https://serpapi.com/search.json?engine=google_finance&hl=en&q=.DJI%3AINDEXDJX",
"name": "Dow Jones",
"price": 34089.27,
"price_movement": {
"percentage": 0.4574563,
"value": 156.66016,
"movement": "Down"
}
},
{
"stock": ".INX:INDEXSP",
"link": "https://www.google.com/finance/quote/.INX:INDEXSP",
"serpapi_link": "https://serpapi.com/search.json?engine=google_finance&hl=en&q=.INX%3AINDEXSP",
"name": "S&P 500",
"price": 4136.13,
"price_movement": {
"percentage": 0.028041454,
"value": 1.1601563,
"movement": "Down"
}
},
other results ...
]