Django has an awesome debug page that shows up whenever there's an exception in the code. That page shows all levels in the stack compactly, and you can expand any level you're interested in. It shows only in debug mode.
Django also has an awesome feature for sending email reports when errors are encountered on the production server by actual users. These reports have a stacktrace with much less information than the styled debug page.
There's an awesome optional setting 'include_html': True that makes the email include all the information of the debug page, which is really useful. The problem with this setting is that the HTML comes apparently unstyled, so all those levels of the stack are expanded to show all the data they contain.
This results in such a long email that GMail usually can't even display it without sending you to a dedicated view. But the real problem is that it's too big to navigate in and find the stack level you want.
What I want: I want Django to send that detailed stacktrace, but I want the levels of the stack to be collapsable just like in the debug page. How can I do that?
(And no, I don't want to use Sentry.)
We use a middleware to process any exceptions that happen in production. The basic idea is save the debug html to a location on the file system that can be served via a password protected view and then send a link to the generated view to whoever needs it.
Here is a basic implementation:
from django.conf import settings
from django.core.mail import send_mail
from django.views import debug
import traceback
import hashlib
import sys
class ExceptionMiddleware(object):
def process_exception(self, request, exception):
if isinstance(exception, Http404):
return
traceback = traceback.format_exc()
traceback_hash = hashlib.md5(traceback).hexdigest()
traceback_name = '%s.html' % traceback_hash
traceback_path = os.path.join(settings.EXCEPTIONS_DIR, traceback_name)
reporter = debug.ExceptionReporter(request, *sys.exc_info())
with open(traceback_path, 'w') as f:
f.write(reporter.get_traceback_html().encode("utf-8"))
send_mail(
'Error at %s' % request.path,
request.build_absolute_uri(reverse('exception', args=(traceback_name, ))),
FROM_EMAIL,
TO_EMAIL,
)
And the view
from django.conf import settings
from django.http import HttpResponse
def exception(request, traceback_name):
traceback_path = os.path.join(settings.EXCEPTIONS_DIR, traceback_name)
with open(traceback_path, 'r') as f:
response = HttpResponse(f.read())
return response
The full middleware is tailored to our needs but the basics are there. You should probably password protect the view somehow. Unless you return a response Django's own error handling will kick in and still send you an email but I'll leave that to you.
The tracebacks in the debug page are stackable because they are done in javascript. I doubt you can have what you want since HTML emails should not accept javascript.
As J. C. Leitão pointed out, the django error debug page has javascript and css(most css doesn't work in email). But all these css and js code are inline. The debug page is a single html file that has no external resources.
In my company, we include the html as a attachment in the report email. When we feel the plain text traceback is not clear enough, we download the html page and open it. The user experience is not as good as Sentry, but much better than the plain-text only version.
Related
Flask-Login recommends having an is_safe_url() function after user login:
Here is a link to the part of the documentation that discusses this:
https://flask-login.readthedocs.io/en/latest/#login-example
They link to this snippet but I don't understand how it implements is_safe_url():
https://palletsprojects.com/p/flask/
next = request.args.get('next')
if not is_safe_url(next):
return abort(400)
This doesn't seem to come with Flask. I'm relatively new to coding. I want to understand:
What exactly is happening when request gets the next argument?
What does the is_safe_url() function do to ensure the URL is safe?
Does the next URL need to be checked on login only? Or are there other places and times when it is important to include this security measure?
And most importantly: is there a reliable is_safe_url() function that I can use with Flask?
Edit: Added link to Flask-Login documentation and included snippet.
As mentioned in the comments, Flask-Login today had a dead link in the documentation (issue on GitHub). Please note the warning in the original flask snippets documentation:
Snippets are unofficial and unmaintained. No Flask maintainer has curated or checked the snippets for security, correctness, or design.
The snippet is
from urllib.parse import urlparse, urljoin
def is_safe_url(target):
ref_url = urlparse(request.host_url)
test_url = urlparse(urljoin(request.host_url, target))
return test_url.scheme in ('http', 'https') and \
ref_url.netloc == test_url.netloc
Now to address your questions:
What exactly is happening when request gets the next argument?
Part of the code we are focusing on here is
next = request.args.get('next')
return redirect(next or url_for('dashboard'))
which redirects user to dashboard (e.g. after successful login) by default. However, if user tried to reach for e.g. endpoint profile and wasn't logged in we would want to redirect him to the login page. After logging in default redirect would redirect user to dashboard and not to profile where he intended to go. To provide better user experience we can redirect user to his profile page by building URL /login?next=profile, which enables flask to redirect to profile instead of the default dashboard.
Since user can abuse URLs we want to check if URL is safe, or abort otherwise.
What does the is_safe_url() function do to ensure the URL is safe?
The snippet in question is a function that ensures that a redirect target will lead to the same server.
Does the next URL need to be checked on login only? Or are there other places and times when it is important to include this security measure?
No, you should check all dangerous URLs. Example of safe URL redirect would be redirect(url_for('index')), since its hardcoded in your program. See examples of safe and dangerous URLs on Unvalidated Redirects and Forwards - OWASP cheatsheet.
And most importantly: is there a reliable is_safe_url() function that I can use with Flask?
There is Django's is_safe_url() bundled as a standalone package on pypi.
The accepted answer covers the theory well. Here is one way to deploy a 'safe URL' strategy throughout your Flask project, involving no additional libraries:
util.py:
def is_safe_url(target):
ref_url = urlparse(request.host_url)
test_url = urlparse(urljoin(request.host_url, target))
return test_url.scheme in ('http', 'https') and ref_url.netloc == test_url.netloc
def get_redirect_target():
for target in request.values.get('next'), request.args.get('next'):
if not target:
continue
if is_safe_url(target):
return target
def redirect_back(endpoint, **values):
target = request.form['next'] if request.form and 'next' in request.form else request.args.get('next')
if not target or not is_safe_url(target):
target = url_for(endpoint, **values)
return redirect(target)
example routes.py (one of many):
from util import get_redirect_target, redirect_back
#bp.route('/routepath', methods=['GET', 'POST'], strict_slashes=False)
#login_required
def my_route():
# use these styles (both or separately) as needed
if not (some_condition):
return redirect_back('someother_route')
return_url = get_redirect_target() or url_for('another_route')
...
return redirect(return_url)
is_safe_url is a python module.
Your code is ok just import after you install the module:
#install:
python3 -m pip install is_safe_url
#import:
from is_safe_url import is_safe_url
I'm having some trouble understanding and implementing the Google Directory API's users watch function and push notification system (https://developers.google.com/admin-sdk/reports/v1/guides/push#creating-notification-channels) in my Python GAE app. What I'm trying to achieve is that any user (admin) who uses my app would be able to watch user changes within his own domain.
I've verified the domain I want to use for notifications and implemented the watch request as follows:
directoryauthdecorator = OAuth2Decorator(
approval_prompt='force',
client_id='my_client_id',
client_secret='my_client_secret',
callback_path='/oauth2callback',
scope=['https://www.googleapis.com/auth/admin.directory.user'])
class PushNotifications(webapp.RequestHandler):
#directoryauthdecorator.oauth_required
def get(self):
auth_http = directoryauthdecorator.http()
service = build("admin", "directory_v1", http=auth_http)
uu_id=str(uuid.uuid4())
param={}
param['customer']='my_customer'
param['event']='add'
param['body']={'type':'web_hook','id':uu_id,'address':'https://my-domain.com/pushNotifications'}
watchUsers = service.users().watch(**param).execute()
application = webapp.WSGIApplication(
[
('/pushNotifications',PushNotifications),
(directoryauthdecorator.callback_path, directoryauthdecorator.callback_handler())],
debug=True)
Now, the receiving part is what I don't understand. When I add a user on my domain and check the app's request logs I see some activity, but there's no usable data. How should I approach this part?
Any help would be appreciated. Thanks.
The problem
It seems like there's been some confusion in implementing the handler. Your handler actually sets up the notifications channel by sending a POST request to the Reports API endpoint. As the docs say:
To set up a notification channel for messages about changes to a particular resource, send a POST request to the watch method for the resource.
source
You should only need to send this request one time to set up the channel, and the "address" parameter should be the URL on your app that will receive the notifications.
Also, it's not clear what is happening with the following code:
param={}
param['customer']='my_customer'
param['event']='add'
Are you just breaking the code in order to post it here? Or is it actually written that way in the file? You should actually preserve, as much as possible, the code that your app is running so that we can reason about it.
The solution
It seems from the docs you linked - in the "Receiving Notifications" section, that you should have code inside the "address" specified to receive notifications that will inspect the POST request body and headers on the notification push request, and then do something with that data (like store it in BigQuery or send an email to the admin, etc.)
Managed to figure it out. In the App Engine logs I noticed that each time I make a change, which is being 'watched', on my domain I get a POST request from Google's API, but with a 302 code. I discovered that this was due to the fact I had login: required configured in my app.yaml for the script, which was handling the requests and the POST request was being redirected to the login page, instead of the processing script.
I am writing test cases for my Django application but I am using requests package in Python to access the status code, and then use the assert statement. This test case is for Staff Users only:
class StaffClientServerTestCase(TestCase):
def test_login_staff(self):
User.objects.create_superuser('abc', 'abc#gmail.com', '1234')
self.client.login(username='abc', password='1234')
self.client.get('/dashboard/')
s = requests.get('http://localhost:8000/dashboard/')
print s.status_code
self.assertEqual(200, s.status_code)
Only staff users have access to the dashboard, so I created a staff user object. The following line
self.client.login(username='abc', password='1234') is coming to be true.
self.client.get('/dashboard/') if I print out this line, it shows the html content of the page which means that my staff user can access the dashboard.
But when I use the request module to see the status code of dashboard url it shows that status code = 500, and my test is failing.
s = requests.get('http://localhost:8000/dashboard/')
print s.status_code
Could anyone tell me where I am going wrong here? Why my status code is coming out be 500, even though the staff user can access the dashboard and print out its contents using print self.client.get('/dashboard/'). Help!
you can test the case in other way:
protect your views with decorator:
#user_passes_test(lambda u: u.is_staff)
def dashboard(request, ...):
# ...
and then request with requests to see if you are banned to see the page. if you are banned (403 forbidden), then your views is working correctly. To verify if your views works correctly for staff users, remove the decorator and request again, if you go thru this time, then everything is fine, the only thing to test is then the logic inside the views.
but right now, you are getting 500 instead of 403, which means you have backend errors. fix them first
My website receives payments through paypal.
My users are complaining they only see the login screen in portuguese, which they not understand.
I've tried the solution proposed here (as the code below shows), but no luck so far.
Here's a little test I'm making:
The code below uses paypalx, and is available as a gist
from django.conf import settings
from paypalx import AdaptivePayments
from urllib import urlencode
from urllib2 import urlopen, Request
import logging
paypal = AdaptivePayments(settings.PAYPAL_API_USERNAME,
settings.PAYPAL_API_PASSWORD,
settings.PAYPAL_API_SIGNATURE,
settings.PAYPAL_API_APPLICATION_ID,
settings.PAYPAL_API_EMAIL,
sandbox=settings.PAYPAL_USE_SANDBOX)
receivers = []
receivers.append({'amount' : '10.00', 'email' : 'penepoleb#gmail.com'})
response = paypal.pay(
actionType = 'PAY',
cancelUrl = settings.PAYPAL_CANCEL_URL,
currencyCode = 'USD',
feesPayer = 'EACHRECEIVER',
receiverList = { 'receiver': receivers},
returnUrl = settings.PAYPAL_RETURN_URL,
ipnNotificationUrl = settings.PAYPAL_IPNNOTIFY_URL,
errorLanguage = "en_US"
)
paykey = response['payKey']
print('https://www.paypal.com/webapps/adaptivepayment/flow/pay?paykey=%s&expType=light&country.x=us&locale=en_US&change_locale=1' % paykey)
Here's one such paykey generated with my paypal credentials: AP-2AG393088X128224K
Now, you can try to open the generated URL and see that it is in portuguese:
https://www.paypal.com/webapps/adaptivepayment/flow/pay?paykey=AP-2AG393088X128224K&expType=light&country.x=us&locale=en_US&change_locale=1
But if you try a different paykey, then the locale parameters work
(I'm using the paykey I got from this example: AP-7LB18162J0578713L)
If you go to the link below you'll see that it's in english.
https://www.paypal.com/webapps/adaptivepayment/flow/pay?paykey=AP-7LB18162J0578713L&expType=light&country.x=us&locale=en_US&change_locale=1
What do I need to do to show users a login page in english?
Please check this post - it seems to refer to the same problem you have.
Thanks,
Arthur
Following the tip by #casals, I found out that there's another paypal URL that I can use.
[1] - https://www.paypal.com/webapps/adaptivepayment/flow/pay?paykey=...
[2] - https://www.paypal.com/webscr?cmd=_ap-payment&paykey=...
I was using [1] inside an iframe.
Now I'm using [2] with a browser redirect. Redirecting to [2] will make paypal present the login page in english, as desired.
This changes the payment experience a little bit - the user actually has to leave my site and then is redirected back to it. But that's fine, I think users actually prefer it this way because they can see Paypal's url when they type their credentials.
I still believe the current behaviour with [1] is a bug in paypal.
My application - http://www.freedomsponsors.org - is open source on Github and you can see the piece of changed code here)
I'm currently playing around with tipfy on Google's Appengine and just recently ran into a problem: I can't for the life of me find any documentation on how to use GET variables in my application, I've tried sifting through both tipfy and Werkzeug's documentations with no success. I know that I can use request.form.get('variable') to get POST variables and **kwargs in my handlers for URL variables, but that's as much as the documentation will tell me. Any ideas?
request.args.get('variable') should work for what I think you mean by "GET data".
Source: http://www.tipfy.org/wiki/guide/request/
The Request object contains all the information transmitted by the client of the application. You will retrieve from it GET and POST values, uploaded files, cookies and header information and more. All these things are so common that you will be very used to it.
To access the Request object, simply import the request variable from tipfy:
from tipfy import request
# GET
request.args.get('foo')
# POST
request.form.get('bar')
# FILES
image = request.files.get('image_upload')
if image:
# User uploaded a file. Process it.
# This is the filename as uploaded by the user.
filename = image.filename
# This is the file data to process and/or save.
filedata = image.read()
else:
# User didn't select any file. Show an error if it is required.
pass
this works for me (tipfy 0.6):
from tipfy import RequestHandler, Response
from tipfy.ext.session import SessionMiddleware, SessionMixin
from tipfy.ext.jinja2 import render_response
from tipfy import Tipfy
class I18nHandler(RequestHandler, SessionMixin):
middleware = [SessionMiddleware]
def get(self):
language = Tipfy.request.args.get('lang')
return render_response('hello_world.html', message=language)