Develop Reference

How does Cloudflare differentiate Selenium and Requests traffic?

Context
I am currently attempting to build a small-scale bot using Selenium and Requests module in Python.
However, the webpage I want to interact with is running behind Cloudflare.
My python script is running over Tor using stem module.
My traffic analysis is based on Firefox's "Developer options->Network" using Persist Logs.
My findings so far:
Selenium's Firefox webdriver can often access the webpage without going through "checking browser page" (return code 503) and "captcha page" (return code 403).
Requests session object with the same user agent always results in "captcha page" (return code 403).
If Cloudflare was checking my Javascript functionality, shouldn't my requests module return 503 ?
Code Example
driver = webdriver.Firefox(firefox_profile=fp, options=fOptions)
driver.get("https://www.cloudflare.com") # usually returns code 200 without verifying the browser
session = requests.Session()
# ... applied socks5 proxy for both http and https ... #
session.headers.update({"user-agent": driver.execute_script("return navigator.userAgent;")})
page = session.get("https://www.cloudflare.com")
print(page.status_code) # return code 403
print(page.text) # returns "captcha page"
Both Selenium and Requests modules are using the same user agent and ip.
Both are using GET without any parameters.
How does Cloudflare distinguish these traffic?
Am I missing something?
I tried to transfer cookies from the webdriver to the requests session to see if a bypass is possible but had no luck.
Here is the used code:
for c in driver.get_cookies():
session.cookies.set(c['name'], c['value'], domain=c['domain'])

There are additional JavaScript APIs exposed to the webpage when using Selenium. If you can disable them, you may be able to fix the problem.

Cloudflare doesn't only check HTTP headers or javascript — it also analyses the TLS header. I'm not sure exactly how it does it, but I've found that it can be circumvented by using NSS instead of OpenSSL (though it's not well integrated into Requests).

The captcha response depends on the browser fingerprint. It's not about just sending Cookies and User-agent.
Copy all the headers from Network Tab in Developers console, and send all the key value pairs as headers in request library.
This method should work logically.

“No 'Access-Control-Allow-Origin' header is present” error for some requests but not others

I have a VueJS app running on S3 that is served by a Flask-powered API running on AWS Elastic Beastalk.
The problem
When making some requests, I get the following:
Access to XMLHttpRequest at 'https://api.myflaskapi.net/blueprint/get_info?date=2019-01-01' from origin 'https://app.myvuejsapp.net' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
What I've done so far
In my __init__.py file, initialized by the app factory in Flask I have set CORS(app, support_credentials=True) as the example on this link. With that, I would hope that basically any request would have the CORS heading, so I wouldn't have any policy blocking my frontend requests.
The ideal scenario would be to only allow requests from https://app.myvuejsapp.net, so I tried using CORS(app, support_credentials=True, origins=['https://app.myvuejsapp.net']) but also without success.
I've also tried making one CORS instance for each of my blueprints with CORS(myblueprint) for each .py route file (I have one for each blueprint), also without success.
The strange thing is, I do have one function on Vue that runs when the app is mounted that works just fine. I can't see any difference between this and other functions that won't work.
Example of working function (returns true or false from the backend):
checkDB() {
const path = this.base_url + 'blueprint/check_db'
axios.get(path, this.token)
.then(checkupd => {
this.isupdated = Boolean(checkupd.data);
if (this.isupdated == true) {
this.update_msg = "Database up to date."
this.loading = false
this.finished = true
} else {
this.update_msg = "WARNING: Check DB status."
}
})
.catch(error => {
console.log(error)
})
},
Example of non-working function (returns a XLS from the backend):
getSalesFrom(file) {
const fs = require('fs')
const FileDownload = require('js-file-download');
const path = this.base_url + `blueprint/get_sales?date=${this.date}`
axios.get(path, {
headers:
{
"X-Access-Token": "mytoken",
"Content-Type": "application/json"
},
responseType: 'blob'
})
.then(response => {
const content = response.headers['content-type'];
download(response.data, 'text.xlsx', content)
})
.catch(error => {
console.log(error)
})
this.export_dialog = false
}
S3 CORS Configuration XML
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<AllowedMethod>POST</AllowedMethod>
<MaxAgeSeconds>3000</MaxAgeSeconds>
<AllowedHeader>Authorization</AllowedHeader>
<AllowedHeader>Content-Length</AllowedHeader>
<AllowedHeader>Access-Control-Allow-Origin</AllowedHeader>
<AllowedHeader>X-Access-Token</AllowedHeader>
</CORSRule>
</CORSConfiguration>
Any ideas on what I might be doing wrong?
I've been reading for a while, but can't seem to find a solution for what it seems a pretty simple problem... Maybe I should mess with the S3 bucket permission configuration?
Thanks.

Any time you run into a case where you’re seeing a “No 'Access-Control-Allow-Origin' header is present” message for some requests but not others, use the Network pane in browser devtools to check the HTTP status code of the response.
Almost always what you’re going to find is that the HTTP status code in those cases is a 4xx or 5xx error instead of the expected 200 OK success response.
The only reason you see the “No 'Access-Control-Allow-Origin' header is present” message is those cases is, the Access-Control-Allow-Origin header typically isn’t going to get added to 4xx and 5xx errors. Part of the reason is, in the 5xx case especially, the cause can be a server failure that occurs before the server ever gets around to running your application code. Along with that, most servers by default won’t add application-set response headers to 4xx and 5xx errors; instead by default they only add them to 2xx success responses.
So anyway, what you want to do is, look at the server logs on the server side (for the server you’re sending the request to) and see what messages the server is logging about the cause of whatever problem it is that makes the server end up sending that 4xx or 5xx error response.

As I understand correctly you host your Vue.js application from S3.
You need to include CORS headers in your S3 buckets. Without them the browser will block all requests to your Flask application. You are making requests to api.myflaskapi.net from app.myvuejsapp.net so you need configure CORS in app.myvuejsapp.net.
You can read documentation how to set CORS in S3 here and here.
To configure your bucket to allow cross-origin requests, you create a CORS configuration, which is an XML document with rules that identify the origins that you will allow to access your bucket, the operations (HTTP methods) that will support for each origin, and other operation-specific information.
You can add up to 100 rules to the configuration. You add the XML document as the cors subresource to the bucket either programmatically or by using the Amazon S3 console. For more information, see Enabling Cross-Origin Resource Sharing (CORS).

COOKIES not available on sub-domain in DJANGO inspite using CORS_ALLOW_CREDENTIALS = True

I am trying to access the cookies that are set by my domain in my sub-domain site.
I am using django-cors-headers to achieve the same.
I have been able to make request to my sub-domain.
I have set the CORS_ALLOW_CREDENTIALS = True in my settings.py and still not able to access cookies on my sub-domain.
While setting the cookie using set_cookie I set the to domain = 'domain.com'

try with
ALLOWED_HOSTS = ['.domain.com']
SESSION_COOKIE_DOMAIN = ".domain.com"
by this, session cookies are set in all subdomains including www.

I was having the same issues. I tried every possible configuration of the cookies until I thought that the issue might be with the frontend.
Long story short — if you are using axios make sure you set withCredentials: true. How to force credentials to every Axios request
Other libraries might have something similar.

https site with Django in text browser throws CSRF verification failed

I have a Django site that works well on a server using HTTPS protocol, I can use it with no problem with all kind of browsers.
The thing is that every time I try to use a text browser, I get a
Forbidden (403)
CSRF verification failed. Request aborted.
You are seeing this message because this HTTPS site requires a 'Referer header' to be sent by your Web browser, but none was sent.
This header is required for security reasons, to ensure that your browser is not being hijacked by third parties.
If you have configured your browser to disable 'Referer' headers, please re-enable them, at least for this site, or for HTTPS
connections, or for 'same-origin' requests.
Help
Reason given for failure:
Referer checking failed - no Referer.
I have tried links, lynx, even w3m and eww on emacs, to no avail.
When I use a HTTP site (like when I'm using the manage.py runserver) I can use the site on text browsers with no problem, but my production server needs a HTTPS protocol and that's when I get this error.
[ EDIT: just for testing purposes, I deployed an HTTP server for my django site on the production server. It works well on text browsers... ]
[ EDIT: given the message the server throws, why are Referer headers not been given? ]

Lynx is likely configured to not send the Referer header. Check /etc/lynx.cfg for "REFERER".
There are entries like NO_REFERER_HEADER. Make sure that's set to false. If that's not it, check around in that config for any other disabled referer headers.
Also related, the CSRF and Referer header debate: https://code.djangoproject.com/ticket/16870

Are you setting SECURE_PROXY_SSL_HEADER, SESSION_COOKIE_SECURE and CSRF_COOKIE_SECURE in your settings?
https://docs.djangoproject.com/en/1.7/topics/security/#ssl-https

AngularJS cookies not set

I am developing an AngularJS website that uses an API backend on a different domain.
The front-end website is hosted at: www.example.com
The API is hosted at: api.example.com
I use Angular's $http.post to make an authentication request to the API which sets a cookie. I then make a secondary $http.get call to the API and the cookie that was set from the POST request isn't being sent back to the server. It looks like the cookie is getting lost somewhere.
The API is a Flask Python app and I'm using flask-cors to enable cross-domain calls. The Access-Control-Allow-Origin header is set to http://www.example.com The domain on the cookie being set is api.example.com
I have setup the application to run under one domain using nginx and url rewriting. So the front-end website is located at www.example.com and the API is accessed by www.example.com/api/ and the cookies are being saved/used as expected.
I can't tell whether this is a problem with my front-end or API website configuration.

Since you are sending the http requests from another domain, you need to make sure that your $http is able to send cookies. In your app's config, add:
$httpProvider.defaults.withCredentials = true
This will allow AngularJS to send your browser's cookies to the server.