Let's say I have a try and catch and there is an exception ... What is the proper way to deal with those exceptions/errors on a live production (django) site?
So I have
try:
create_response = wepay.call('/account/create',
{'name': name, 'description': desc})
self.wepay_account_id = create_response['account_id']
self.save()
except WePay.WePayError as e:
..... (what do I put here?
You can set up e-mail error reporting through Django: https://docs.djangoproject.com/en/dev/howto/error-reporting/
Or you can use a service like Rollbar (has a free account) to track error occurances.
Or you could use self-hosted Greylog (like suggested in comments), here's a good guide for django: http://www.caktusgroup.com/blog/2013/09/18/central-logging-django-graylog2-and-graypy/
Respond with (optionally a redirect to) a appropriate page explaining the problem to the user and if possible, provide a solution. Serving a 500 to your users in production is something you want to avoid, so catching the exception is a good idea.
So:
except WePay.WePayError as e:
return render_to_response('wepay_error_page.html')
or:
except WePay.WePayError as e:
return HttpResponseRedirect('/errors/wepay/') # Note: better use urlresolvers
(note this particular code will only work if it's in a view)
Then (optionally), make sure you get a copy of the error, by for example sending yourself an email.
A suggestion for this particular case (if I interpret the code succesfully) may be to notify yourself, and repond with a page explaining to the user their payment went wrong. Tell them this might occur because of their actions (maybe they cancelled their payment), and provide contact details for when users think it was not their fault.
Django by default mails (when mail is properly configured) all 500 errors to settings.ADMINS, but these only occur on uncaught exceptions, so in this particular question services like Rollbar or a central logging solution will only work if you re-raise the exception (will result in a 500) or send the error to one of these manually in the catch block.
I would recommend the above solution of redirecting over to a page that explains WePay error, combined with using django-wepay app available on pypi that features logging of all errors, and optionally all calls.
Related
Is there any way to detect if a link is invalid using webbot?
I need to tell the user that the link they provided was unreachable.
The only way to be completely sure that a url sends you to a valid page is to fetch that page and check it works. You could try making a request other than GET to try to avoid the wasted bandwith downloading the page, but not all servers will respond: the only way to be absolutely sure is to GET and see what happens. Something like:
import requests
from requests.exceptions import ConnectionError
def check_url(url):
try:
r = requests.get(url, timeout=1)
return r.status_code == 200
except ConnectionError:
return False
Is this a good idea? It's only a GET request, and get is supposed to idempotent, so you shouldn't cause anybody any harm. On the other hand, what if a user sets up a script to add a new link every second pointing to the same website? Then you're DDOSing that website. So when you allow users to cause your server to do things like this, you need to think how you might protect it. (In this case: you could keep a cache of valid links expiring every n seconds, and only look up if the cache doesn't hold the link.)
Note that if you just want to check the link points to a valid domain it's a bit easier: you can just do a dns query. (The same point about caching and avoiding abuse probably applies.)
Note that I used requests, because it is easy, but you likely want to do this in the bacground, either with requests in a thread, or with one of the asyncio http libraries and an asyncio event loop. Otherwise your code will block for at least timeout seconds.
(Another attack: this gets the whole page. What if a user links to a massive page? See this question for a discussion of protecting from oversize responses. For your use case you likely just want to get a few bytes. I've deliberately not complicated the example code here because I wanted to illustrate the principle.)
Note that this just checks that something is available on that page. What if it's one of the many dead links which redirects to a domain-name website? You could enforce 'no redirects'---but then some redirects are valid. (Likewise, you could try to detect redirects up to the main domain or to a blacklist of venders' domains, but this will always be imperfect.) There is a tradeoff here to consider, which depends on your concrete use case, but it's worth being aware of.
You could try sending an HTTP request, opening the result, and have a list of known error codes, 404, etc. You can easily implement this in Python and is efficient and quick. Be warned that SOMETIMES (quite rarely) a website might detect your scraper and artificially return an Error Code to confuse you.
So I'm trying to port some old Pylons code to Pyramid, and I'd like to be able to improve on the Auth setup - specifically support better RBAC, and Pyramid has good support for this.
However, I'd like to offer unauthorised users better info when they try illegal pages:
"Sorry, in order to view [page] you ([user]) need [group] privileges - please contact [admin]"
However I don't see how that's practical in Pyramid - I can do stuff in the forbidden_view_config page, however I can't easily find all the info needed from the page which was attempted - is it possible to get the exception or similar with the actual reason why permission was not granted?
The request object itself should have all the bits you need.
Specifically, security-related pieces lists some of the request attributes that you can retrieve. Also the request.exception attribute will be available when an exception is raised. There are several URL-related pieces available to get the "page", including application_url.
Technologies and Applications used: Rollbar, Django 1.7, Python 3.4
So, I'm following the official documentation found here for integrating Rollbar into a python and Django based application: https://github.com/rollbar/pyrollbar. Which includes: pip installing rollbar, adding the middleware class and creating the Rollbar dictionary configuration in a settings file, etc.
Just to test things out I added the example they provided in their docs to one of my views, and Rollbar/Django works fine (i.e. Rollbar registers the exception and the exception is sent to my Rollbar account in the cloud):
try:
main_app_loop()
except IOError:
rollbar.report_message('Got an IOError in the main loop', 'warning')
except:
# catch-all
rollbar.report_exc_info()
But, for example, in one of my template files I misspell a block tag and get an error via Django's default error logging system. However, Rollbar doesn't record the error and/or it isn't sent to my Rollbar account in the cloud. Is that because Rollbar has to be integrated manually via some kind of try, catch scenario? Or can Rollbar just grab errors by default without having to write a try, catch?
There is no other documentation I kind find for integrating Rollbar into a Django project other than what is found on the above link, so I'm not sure what to do next. Anyone else run into this or know what the issue might be?
Is it possible to hook into Django's built in error reporting emails from a try-except code block? In other words, email the default error report and stack trace to the ADMINS/MANAGERS while still having situation specific error handling.
Specific example:
In a project performing complex calculations and generating large reports, the view displaying the report page does all the calculations and generates a long html page with lots of pretty tables and graphs and also generates downloadable PDFs from sections of that same HTML.
Recently we had errors in the PDF generation from issues with storage on S3. Now this is obviously an error we need to track down and attend, but most users are happy if they can just see the report on screen. If the PDFs download links just weren't displayed the issue could go entirely unnoticed for hours or even days - but the dev team should be notified.
Ideally, but not necessarily, I would love a solution that is logger agnostic, where it will use whatever error logger is used and trigger the default 500 error handler, and return back to the finally block or after the except block.
All you need to do is to use Python's logging framework to raise an appropriate message at the appropriate level. In your settings.py there is a LOGGING variable that defines how things are logged. By default I believe Django has any ERROR in django.request will be handled by mail_admins.
So in your code, all you need to do is
import logging
logger = logging.getLogger(__name__) # this will create a logger with the module being the logger name
try:
#do stuff you watch to catch
except:
# we're going to catch and just log it
logger.error('Some error title', exc_info=True) # exc_info=True will include the stacktrace
finally:
# what you want to do in your finally block.
Note, this will swallow the exception and won't bubble it up. Your response will return as a 200. If you want to bubble up the exception, just call raise in your except block. However, if all you care about is logging the error, but the view is still functional, then just log and swallow it.
In your LOGGING variable, you can add additional entries to loggers for the different logger names. You can have an app log at a different logging status, say INFO if you want to debug a certain code path. As long as you create a logger with the module name, you have a lot of flexibility of segmenting your logging to different handlers such as mail_admins.
Lastly, I'd recommend to look into sentry, as it's a really great error logging tool.
I started playing around with web2py the other day for a new project. I really like the structure and the whole concept which feels like a breath of fresh air after spending a few years with PHP frameworks.
The only thing (currently) that is bothering me is the ticketing system. Each time I make a misstake a page with a link to a ticket is presented. I guess I could live with that if the link worked. It currently points to an admin page with http as protocol instead of https. I've done a bit of reading and the forced https for admin seems to be a security measure, but this makes debugging a pain.
Whats the standard solution here? Alter the error page, allow http for admin och use logs for debugging?
Best regards
Fredrik
I was in the same boat as you, I did not like the default mechanism. Luckily, customized exception handling with web2py is very straightforward. Take a look at routes.py in the root of your web2py directory. I've added the following to mine:
routes_onerror = [('application_name/*','/application_name/error/index')]
This routes any exceptions to my error handler controller (application_name/controllers/error.py) in which I defined my def index as:
def index():
if request.vars.code == '400':
return(dict(app=request.application,
ticket=None,
traceback="A 400 error was raised, this is controller/method path not found",
code=None,
layer=None,
wasEmailed=False))
elif request.vars.code == '404':
return(dict(app=request.application,
ticket=None,
traceback="A 404 error was raised, this is bad.",
code=None,
layer=None,
wasEmailed=False))
else:
fH = file('applications/%s/errors/%s' % (request.application,request.vars.ticket.split("/")[1]))
e = cPickle.load(fH)
fH.close()
__sendEmail(request.application,e['layer'],e['traceback'],e['code'])
return(dict(app=request.application,
ticket=request.vars.ticket,
traceback=e['traceback'],
code=e['code'],
layer=e['layer'],
wasEmailed=True))
As you can see for non-400 and 404 errors, I'm emailing the traceback to myself and then invoking the corresponding views/error/index.html. In production, this view gives a generic "I'm sorry an error has occurred, developers have been emailed". On my development server, it displays the formatted traceback.
Normally, I just use http://127.0.0.1/ (if you are local or over ssh) or edit/navigate using https://...
So, you will logon the admin app the first time, but always will the show the tickets after.