Develop Reference

CSRF validation works from Postman but not React/Axios

It's honestly a catastrophic blow to the ego I haven't already figured this out--spent 6 hours so far...
I have a React app running off a Django Rest Framework backend. For the password reset functionality I am using Django's builtin view (auth_views.PasswordResetView). I have a form in React that accepts an email and sends a post request to reset the users password. The way I send the data in postman is:
Url: http://192.168.0.85:8000/reset_password
Body: {'email': 'blabla#bla.com'}
Headers: {'X-CSRFToken': OGH9iUEtGPqntMYifQ5kiin2ufV9tK39tbp9Wmh6tLST0DXCXSkY8mOvyq5AjjnZ}
...and it works like a charm! Until I try to replicate the same exact call using axios in React:
axios.defaults.xsrfCookieName = 'csrftoken';
axios.defaults.xsrfHeaderName = 'X-CSRFToken';
axios.post('http://192.168.0.85:8000/reset_password',{'email': 'blabla#bla.com'},{headers}))
...this results in a 403 error with this message from the backend:
WARNING:django.security.csrf:Forbidden (CSRF cookie not set.): /reset_password
I've spent 6 hours googling and experimenting and can't figure out for the life of me why the Axios request would be any different than the Postman one. This problem is particularly infuriating because #csrf_exempt won't work on the django builtin views.
Here are the headers from a successful request in Postman:
X-CSRFToken: OGH9iUEtGPqntMYifQ5kiin2ufV9tK39tbp9Wmh6tLST0DXCXSkY8mOvyq5AjjnZ
User-Agent: PostmanRuntime/7.28.0
Accept: */*
Postman-Token: 91cbbfe7-b434-45fc-97ec-7d442c091070
Host: 192.168.0.85:8000
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Content-Length: 33
Cookie: csrftoken=gawXPEZQfN4qC4aKPhiIvK5qpjvkp4WCVFeXt6Ct2JwW9V94xjxmlOwTtuFLfDgs
...and from the failed request using Axios from React:
baseURL: "http://192.168.0.85:8000"
data: "{\"email\":\"blabla#bla.com\"}"
headers:
Accept: "*/*"
Authorization: ""
Content-Type: "application/json"
X-CSRFToken: "udd4yAGCJuTxw95kCdHYwwEQmGwu1bwxqEQjZhI56v1qWY143S4IPCrUSvk9xkV8"
__proto__: Object
maxBodyLength: -1
maxContentLength: -1
method: "post"
timeout: 0
transformRequest: [ƒ]
transformResponse: [ƒ]
url: "/reset_password"
validateStatus: ƒ validateStatus(status)
xsrfCookieName: "csrftoken"
xsrfHeaderName: "X-CSRFToken"
__proto__: Object
isAxiosError: true

Does this answer your question, https://stackoverflow.com/a/66550363/7838574 ?
With emphasis on the settings.py portion. That is what tripped me up about a week ago.
Honestly great job keeping at this and asking for help! Programming can be insanely frustrating and once the hurdle is behind you very rewarding!

How do i login to this site with scrapy shell and python - 401 Error?

Im trying to login to this website, seeking.com/login through scrapy shell. i also installed burp suite to analyze its url and headers, etc.
from scrapy.http import FormRequest
frmdata = {"captcha":"","email":"MYEMAIL.com","password":"MY_PASSWORD","is_rememberme":"0","locale":"en_US","auth_type":"bearer_token","date":"2018-12-13T09:56:22.957Z"}
url = "https://www.seeking.com/v3/auth/login"
r = FormRequest(url, formdata=frmdata)
fetch(r)
with this code i get a HTTP 401 Error, as far as i can tell essentially an authentication error.
I forwarded the calls through burpsuite and got the following intercept.
POST /v3/auth/login HTTP/1.1
Host: www.seeking.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:63.0)
Gecko/20100101 Firefox/63.0
Accept: application/json, text/plain, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: https://www.seeking.com/login?fromLogout=1
Content-Type: application/json;charset=utf-8
Web-Version: 3.59
Authorization: Basic NTI2ZTIwYzExMDI0NDYzNTk5OTI5MzUwZThiNWEzMTI6bHN0emd4ZzpSRzRzS3VmdEJMRTQxMm92TnMxbDR6L0ZkZ1dESHZuM2wwZWxtYWhyMGtnPQ==
Content-Length: 166
Connection: close
Cookie: __cfduid=dcf9fd66583d55382f362c18a83d904ca1544519479;
_gcl_au=1.1.2035701377.1544519485; _ga=GA1.2.1740241044.1544519486;
com.silverpop.iMAWebCookie=e88c45d1-3c24-11c6-089e-e287aae2c678;
__cfruid=3eebbdc1e401ed560c23a7c474c41e59b2e93018-1544520179;
device_cookie=1; __gads=ID=a1e437c03ddad1b3:T=1544519579:S=ALNI_MYb30xY4z76J4NniCK_ZtOyOdPMKA;_lb_user=gfpuzje6kg; seeking_session=eyJpdiI6Im4yMTNJNVNRZjkxbnZzMmNpYnQ4dkE9PSIsInZhbHVlIjoiVGhGVUJDejc1dElJbEwxekh5d2hXUnhjeDlpVWR2dW9IWWJqeDZvRmI3VU9Pc1lpZXZGWGJxejQ1alNXbGVXUGJqaEpORU9LNFJITVh0N3IwR1E0bUE9PSIsIm1hYyI6IjUyODU3MWIxYjM3MGU3M2E0YjI1YzM2MzNmNDc5ZDMzZDdjYTg1ZWMxYWU2ODJjY2JlMTJmZWJlNmUyZDkyNWMifQ%3D%3D {"captcha":"","email":"MYEMAIL","password":"MYPASS","is_rememberme":0,"locale":"en_US","auth_type":"bearer_token","date":"2018-12-14T09:15:56.016Z"}
I am completely new to this, and have spent 2 days trying to figure out what i need to pass to this POST to login.
My question is
1) based on this intercept what should my request via FormRequest look like?
2) I see there are cookies/authorization (Authorization token, that changes with each POST, session cookies, etc) tokens that are being passed in to the post... Where do they come from? How do i get them when i am scraping so that i can successfully login?
3) Do i need to store these session variables when scraping other pages on the site after login? Anything special i need to do to stay logged in to access other pages?

It looks like the login page is expecting to be passed soon data, and not a url-encoded string (which is what FormRequest will create).
Something like this should work:
r = scrapy.Request(
url=url,
method='POST',
body=json.dumps(frmdata),
headers={'Content-Type': 'application/json'},
)
The tokens, cookies, etc. are probably created when you initially request the login page, so you might need to request the login page before trying to log in.
It is possible that some of it is generated with javascript (haven't checked), so you might need to dig through the js code to figure out what's going on, or even execute the js yourself (e.g. using a browser).
Scrapy will keep track of your session for you, so there's nothing you need to do to stay logged in.

Why does the requests library add extra headers to the ones I set?

I am trying to do a post request on python utilizing the requests library, when I set my custom headers which are the following:
User-Agent: MBAM-C
Content-Type: application/json
Authorization: True
Content-Length: 619
Connection: Close
However when It send the request with the custom headers it adds its own headers which give a bad request response from the server..
User-Agent: MBAM-C
Accept-Encoding: gzip, deflate
Accept: */*
Connection: Close
Content-Type: application/json
Authorization: True
Content-Length: 559

It is due to the design goals of the requests project.
This behavior is documented here. You may want to use a lower level library, if it is problematic for the library to be correcting content length or adding desirable headers. Requests bills itself as: "an elegant and simple HTTP library for Python, built for human beings." and a part of that is advertising that it can accept compressed content and all mime types.
Note: Custom headers are given less precedence than more specific sources of information. For instance:
Authorization headers set with headers= will be overridden if credentials are specified in .netrc, which in turn will be overridden by the auth= parameter.
Authorization headers will be removed if you get redirected off-host.
Proxy-Authorization headers will be overridden by proxy credentials provided in the URL.
Content-Length headers will be overridden when we can determine the length of the content.

Django rest framework ignores missing csrf tokens with httpie

I am using httpie to play with my api written in django 1.7 and django rest framework 2.4.
Today I was trying to delete an object:
$ http DELETE :8000/api/items/8/ --verbose
DELETE /api/items/8/ HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate, compress
Content-Length: 0
Host: 127.0.0.1:8000
User-Agent: HTTPie/0.8.0
HTTP/1.0 204 NO CONTENT
Allow: GET, PUT, PATCH, DELETE, HEAD, OPTIONS
Content-Language: cs
Content-Length: 0
Date: Wed, 07 Jan 2015 21:47:06 GMT
Server: WSGIServer/0.1 Python/2.7.6
Vary: Accept, Accept-Language, Cookie
Which was successful even though it should require CSRF token. When I try to delete the object from Chrome with following code:
$.ajax({
type: "DELETE",
url: "http://127.0.0.1:8000/api/items/6/"
});
I get a following request:
DELETE /api/items/6/ HTTP/1.1
Host: 127.0.0.1:8000
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Accept: */*
Origin: http://127.0.0.1:8000
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36
DNT: 1
Referer: http://127.0.0.1:8000/inventory
Accept-Encoding: gzip, deflate, sdch
Accept-Language: cs,en-US;q=0.8,en;q=0.6,es;q=0.4,pt;q=0.2,sk;q=0.2
Cookie: cc_csrf=bd9fbbc8f75cffa2e1e3d2c95c2185c5; _ga=GA1.1.2038400685.1386436341; __utma=96992031.2038400685.1386436341.1417173095.1417428975.79; __utmz=96992031.1409752584.3.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __zlcmid=MpdRtV3vZuf3D9; djdt=hide; sessionid=kiihjh6m77jm8v9ol7xrryip89sny55i; csrftoken=FtnnEWPLhMh0CAGMRMH77nB0AAno93uW
Response:
HTTP/1.0 403 FORBIDDEN
Date: Wed, 07 Jan 2015 21:57:40 GMT
Server: WSGIServer/0.1 Python/2.7.6
Vary: Accept, Accept-Language, Cookie
Content-Type: application/json
Content-Language: en
Allow: GET, PUT, PATCH, DELETE, HEAD, OPTIONS
{"detail": "CSRF Failed: CSRF token missing or incorrect."}
My settings:
REST_FRAMEWORK = {
# Use hyperlinked styles by default.
# Only used if the `serializer_class` attribute is not set on a view.
'DEFAULT_MODEL_SERIALIZER_CLASS': 'rest_framework.serializers.HyperlinkedModelSerializer',
# Use Django's standard `django.contrib.auth` permissions,
# or allow read-only access for unauthenticated users.
'DEFAULT_PERMISSION_CLASSES': [
'rest_framework.permissions.DjangoModelPermissionsOrAnonReadOnly'
],
'DEFAULT_FILTER_BACKENDS': ('rest_framework.filters.DjangoFilterBackend',),
'DATETIME_FORMAT': "%B %d, %Y"
}
MIDDLEWARE_CLASSES = (
'django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.locale.LocaleMiddleware',
'debug_toolbar.middleware.DebugToolbarMiddleware',
)
So my question is: what is the difference between sending a DELETE request with JS ajax and with sending the request with http?

It's because the CSRF check is only performed when authenticated using SessionAuthentication, (i.e. using the sessionid cookie set by django.contrib.auth):
If you're using SessionAuthentication you'll need to include valid CSRF tokens for any POST, PUT, PATCH or DELETE operations. (source)
I assume you use another auth method for the HTTPie-sent request, and therefore the CSRF check doesn't get applied there.
https://github.com/tomchristie/django-rest-framework/blob/master/tests/test_authentication.py

When the request is made through the browser, it is including the sessionid token in the Cookie header. This header is automatically set by the browser, and includes other cookies that have been set (like djdt=hide by the Django Debug Toolbar).
Cookie: ...; sessionid=kiihjh6m77jm8v9ol7xrryip89sny55i; ...
Because of this, Django is authenticating the request automatically (just like it normally would), which is triggering the SessionAuthentication provided by Django REST framework. SessionAuthentication requires that the CSRF token is being validated, which is included in the csrftoken cookie and X-CSRFToken header, to ensure that nothing suspicious is happening.
This means that you must set the X-CSRFToken header when making your request in the browser. Django includes some useful snippets of code for popular libraries in their documentation on CSRF.
Now, when making requests through HTTPie, you are typically using a different form of authentication such as basic authentication. By default, Django REST framework enables BasicAuthentication and SessionAuhentication, unless you override them, and most of the documentation expects that you are using basic authentication.
HTTPie supports basic authentication through the -a username:password parameter. This would explain why you are not getting any permission issues when making the DELETE request, as without authentication you should be getting a 403 error. The DjangoModelPermissionsOrAnonReadOnly should not allow you to make the request you have provided without being authenticated.

EDITED & UPDATED
OK apart from the explanations others already mentioned, after all that, we can conclude the reason why Httpie allows your DELETE but not Javascript:
1) Since you have actually disabled your authentication, in theory all METHODS will be allowed from individual HTTP call, hence your Httpie works (just like when you use Curl), because Restframework doesn't require you to.
2) Ajax call from Javascript, however is slightly different because you're using your browser console to do the call, which is in fact within the browser session. Further to this, your cookie has stored your previous GET's CSRF token which afterward when you perform the Ajax call, the CSRF token has been extracted from Django/Restframework which doesn't MATCH (because CSRF token will be automatically re-generated on each request. So this is a matter of INCORRECT rather than *MISSING** token.
Hence like in my above comment, removing your browser's cookie / using a Private session has indeed resolved the issue, and successfully allowed you to perform Ajax style DELETE.
Hope this helps, and thanks for everyone's guidance and hints leading me to this conclusion.

What are some possible sources for an intermittent 404 error in Flask?

I'm building a database (Postgresql) driven site using Flask on Webfaction and I'm getting some strange 404 errors. Here's what happens: after clicking through 4-5 pages on the site, there is usually a 404 error. Reloading the page (either Ctrl-R, selecting the URL and pressing Enter or clicking the refresh icon) makes the error go away and the page displays correctly. After visiting another 4-5 pages, the same problem occurs. Suprisingly enough, it is not always the same pages giving the 404.
I'd like to have people's opinions on what could be causing these intermittent errors...
Caching?
Unhandled database connection errors?
Other types of unhandled exceptions?
Background info (feel free to ask for more):
Flask on Python 2.7
Flask-Bootstrap
Hosted on Webfaction
Here are the headers from a successful request (after reloadign after getting a 404):
Response headers
HTTP/1.1 200 OK
Server: nginx
Date: Sun, 26 Jan 2014 11:46:49 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
Content-Encoding: gzip
Request Headers
GET /product/333947 HTTP/1.1
Host: [mysubdomain].webfactional.com
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive

It turns out it was because of server overloading due to another user on the shared server, so nothing to do with my code or configuration. Thanks for the help anyway!