I have setup the logging module for my new python script. I have two handlers, one sending stuff to a file, and one for email alerts. The SMTPHandler is setup to mail anything at the ERROR level or above.
Everything works great, unless the SMTP connection fails. If the SMTP server does not respond or authentication fails (it requires SMTP auth), then the whole script dies.
I am fairly new to python, so I am trying to figure out how to capture the exception that the SMTPHandler is raising so that any problems sending the log message via email won't bring down my entire script. Since I am also writing errors to a log file, if the SMTP alert fails, I just want to keep going, not halt anything.
If I need a "try:" statement, would it go around the logging.handlers.SMTPHandler setup, or around the individual calls to my_logger.error()?
Exceptions which occur during logging should not stop your script, though they may cause a traceback to be printed to sys.stderr. In order to prevent this printout, do the following:
logging.raiseExceptions = 0
This is not the default (because in development you typically want to know about failures) but in production, raiseExceptions should not be set.
You should find that the SMTPHandler will attempt a re-connection the next time an ERROR (or higher) is logged.
logging does pass through SystemExit and KeyboardInterrupt exceptions, but all others should be handled so that they do not cause termination of an application which uses logging. If you find that this is not the case, please post specific details of the exceptions which are getting through and causing your script to terminate, and about your version of Python/operating system. Bear in mind that the script may appear to hang if there is a network timeout which is causing the handler to block (e.g. if a DNS lookup takes a long time, or if the SMTP connection takes a long time).
You probably need to do both. To figure this out, I suggest to install a local mail server and use that. This way, you can shut it down while your script runs and note down the error message.
To keep the code maintainable, you should extends SMTPHandler in such a way that you can handle the exceptions in a single place (instead of wrapping every logger call with try-except).
Related
I'm using Bottle server to implement my own server using an implementation not so far away from the simple "hello world" here , my own implementation is (without the routing section of course):
bottleApp =bottle.app()
bottleApp.run(host='0.0.0.0',port=80, debug=true)
My server is keep getting unresponsive all the time and then I get in the Browser: Connection reset by peer, broken pipe errno 32
The logs give me almost exactly the same stack traces such as in question.
Here are my own logs:
What I tried so far, without success:
Wrapping the server run line with try except, something like, shown here the answer of "mhawke".
This stopped the error messages in logs, apparently because I caught them in except clause, but problem is that when catching the exception like that it means that we have been thrown out of the run method context, and I want to catch it in a way it will not cause my server to fall.
I don't know if its possible without touching the inner implementations files of bottle.
Adding this before server run line:
from signal import signal, SIGPIPE, SIG_DFL
signal(SIGPIPE,SIG_DFL)
As suggested here, but it seems that it didn't had any impact on not getting Broken pipe\connection reset errors and server responsiveness.
I thought of trying also the second answer here, but I don't have any idea where to locate this code in the context of the bottle server.
This sounds like a permissions issue or a firewall.
if you really need to listen on port 80, then you need to run with a privileged account. Also you will probably need to open port 80 for tcp traffic.
I can see your using something that appears to be Posix (Linux/Unix/OSx) If you post what OS you are using I can edit this answer to be more specific as to how to open the firewall and execute privileged commands (probably sudo but who knows).
The log output of my python program (using the builtin logging module, but occurs even when using simple prints) is partially messed up, as you can see in the following image. Note the first line, first word still being correct and then it gets mixed up:
I tried to visualize the situation where this happens:
Basically in my main thread/program I start a simple socketserver.TCPServer to listen for incoming messages. That server runs on its own thread (QtCore.QThread) so my program is not blocked. If some other application sends a message the request handler of the TCPServer will simply forward the message to the main thread using a QtCore.SIGNAL like:
self.emit(QtCore.SIGNAL('received(const QString)'), receivedMessage)
The program then does some parsing and computation with that message and logs those, thereby producing the gibberish seen above. At some point the logging returns back to working normally.
I am not sure if this is related to sockets or threading or both, but I guess it may be a common issue and therefore I am thankful for any hints why this occurs.
I think I have located the problem:
When the external application wants to send a message it will always create a new client socket, connect to the server, send the message and then close the client socket.
The sock.close() does not seem to close immediately, the docs say I should call sock.shutdown(how) first, but unfortunately this did not help as well. I can use a small time.sleep(0.5) after the close to fix the logging issue, but instead I did something like this:
def ensure_closed(self):
while True:
try:
self.sock.recv(1024)
except:
break
def close_connection(self):
self.sock.close()
self.ensure_closed()
# Continue with other stuff.
# Now the logging behaves normally.
There might be better ways to do it.
I have a Twisted application (Python 2.7) that uses the kombu module to communicate with a RabbitMQ message server.
We're having problems with closing connections (probably firewall related) and I'm trying to implement the heartbeat_check() method to handle this. I've got a heartbeat value of 10 set on the connection and I've got a Twisted LoopingCall that calls the heartbeat_check(rate=2) method every 5 seconds.
However, once things get rolling I'm getting an exception thrown every other call to heartbeat_check() (based on the logging information I've got in the function that LoopingCall calls, which includes the call to heartbeat_check). I've tried all kinds of variations of heartbeat and rate values and nothing seems to help. When I debug into the heartbeat_call() it looks like some minor message traffic is being sent back and forth, am I supposed to respond to that in my message consumer?
Any hints or pointers would be very helpful, thanks in advance!!
Doug
so I'm implementing a log server with twisted (python-loggingserver) and I added simple authentication to the server. If the authentication fails, I wanna close the connection to the client. The class in the log server code already has a function called handle_quit(). Is that the right way to close the connection? Here's a code snippet:
if password != log_password:
self._logger.warning("Authentication failed. Connection closed.")
self.handle_quit()
If the handle_quit message you're referring to is this one, then that should work fine. The only thing the method does is self.transport.loseConnection(), which closes the connection. You could also just do self.transport.loseConnection() yourself, which will accomplish the same thing (since it is, of course, the same thing). I would select between these two options by thinking about whether failed authentication should just close the connection or if it should always be treated the same way a quit command is treated. In the current code this makes no difference, but you might imagine the quit command having extra processing at some future point (cleaning up some resources or something).
So far my networking code works fine, but I'm a bit worried about something I hid under the carpet:
The man pages for accept, close, connect, recv and send mention that errno.EINTR can show up when a system call was interrupted by a signal.
I am quite clueless here.
What does python do with that ? Does it automatically retry the call, does it raise a socket.error with that errno ? What is the appropriate thing I should do if that exception is raised ? Can I generate these signals myself in my unittests ?
Python simply retries the call and hides the signal from the user (helps with cross-platform consistency where -EINTR doesn't exist). You can safely ignore the EINTR issue but if you'd like to test it anyway, it's easy to do. Just set up a blocking operation that will not return (such as a socket.accept with no incoming connection) and send the process a signal.