Develop Reference

Testing python smtp email service

So I wrote a simple email service using smtplib to send welcome emails when user first logs in. I was wondering what's the best way to test out my code.
def send_email(nickname, email):
msg = MIMEMultipart('alternative')
config = read_config_file(MAIL_CONFIG_PATH)
sen = config.get('gmail_credentials', 'email')
pwd = config.get('gmail_credentials', 'password')
server = None
try:
server = smtplib.SMTP('smtp.gmail.com', 587)
server.ehlo()
server.starttls()
server.login(sen, pwd)
server.sendmail(sen, [email], msg.as_string())
return 'Sent!'
finally:
if server:
server.quit()
I found out that you can use a fake smtp server:
sudo python -m smtpd -n -c DebuggingServer localhost:25
But I am still not sure how I should implement something like this into my code. How do I change my API so that it runs the SMTP server during testing and connects to the GMAIL server for the actual calls? And are there init scripts which will run the SMTP server on our servers (this will be deployed on AWS EC2).

To anyone who comes across this old ticket with the same issue, I wanted to write integration tests with the standard library, so the above solutions didn't work for me. Setting up a debugging server locally within the unit testing code was also a bit tricky, and allegedly doesn't support starttls. In the end I went with this;
import smtplib
from unittest import TestCase
class TestSMTP(TestCase)
...
setup etc.
def test_smtp_connection(self):
# connect to actual host on actual port
smtp = smtp.SMTP(host, port)
smtp.starttls()
# check we have an open socket
self.assertIsNotNone(smtp.sock)
# run a no-operation, which is basically a server-side pass-through
self.assertEqual(smtp.noop(), (250, '2.0.0 OK'))
# assert disconnected
self.assertEqual(smtp.quit(), (221, '2.0.0 Service closing transmission channel'))
self.assertIsNone(smtp.sock)
Note, these are integration tests, which is all you need to do to verify your email server is working. Everything else can be done by mocking smtp.SMTP and capturing whatever's passed into sendmail and verifying the contents there. You're not unit testing smtp, you're unit testing your application.

Disclaimer: I worked on the product in question, please note this answer may contain bias.
I co-founded a hosted service called Mailosaur, that allows developers (us included) to test this kind of process in a real-world setting (i.e. over the net, using SMTP).
You get an SMTP endpoint (also unlimited test email addresses, but you're using SMTP so shouldn't need this).
The service then converts emails into a JSON representation, which you can get via an HTTP API.
To make the querying a little easier use can use client bindings (available for Java, .NET and Ruby), but you can use regular curl or any other HTTP client (or get in touch and we'd be happy to put together a Python client).
The final test process would be:-
Send email via SMTP to smtp.mailosaur.in (no need to change to from/to addresses)
Query the API:
curl https://mailosaur.com/v2/emails?mailbox={mailbox}&key={api_key}
Do your regular asserts (pseudo code):
assert.equal(3, email.attachments.length, 'Expected 3 attachments');
assert.contains('https://verify-url', email.html.links, 'Expected link to verification url');

I had the same problem: applications that send email, and I want to check that they do so correctly without spamming a bunch of real people with email of various forms of incompletenese.
I use PostHoc which receives email using SMTP, but then simply displays the email that was sent in a web browser.
It is on GitHub at: https://github.com/agilepro/posthoc
Here is a blog post that describes it: https://agiletribe.purplehillsbooks.com/2017/07/24/posthoc-testing-apps-that-send-email/
The reason I don't run a real SMTP server is that those servers are designed to actually find the and deliver the mail to the recipients. I want to test the application with real email address and real mailing lists. I want to see that the real situation can be handled, and I want to verify that each email sent looks the way it should. I want all this, but I want to be sure that no email is ever actually delivered.
For years I had a switch in the application that redirected email to a different address, and I could see the email there. But this is still problematic since the application is running in a different mode than it will in production. With PostHoc the application runs exactly as it will in production, and I still can safely see all the email sent.
PostHoc requires TomCat to run it, but nothing else.
I like the way it automatically deletes all the old messages after 5 days. I have never needed to see one of these emails after 5 days, and so I just never need to do any clean up around it.

Start local SMTP server

I'm going to transfer crash dumps from clients to me through the mail mechanism. Therefore, I can't use any public SMTP servers, as packing any account's credentials with the application is unacceptable.
Therefore, I need to send mails through my application directly to the destination mail server.
How can I achieve this in python? (I'm using windows so sendmail is not an option)

Just use smtplib in the standard library.
Trying to write code that can send mail to anyone is problematic, because smtplib connects to servers client-to-server style rather than server-to-server-relay style.
But if you only need to send mail to one particular server, which you control, it's trivial. Just configure your server at 'mail.example.com' to accept any mail from 'crash-reports#example.com' to 'crash-reports#example.com'.
Your code will look something like this:
import smtplib
addr = 'crash-reports#example.com'
def send_crash_report(crash_report):
msg = ('From: {}\r\nTo: {}\r\n\r\n{}'.format(
addr, addr, crash_report)
server = smtplib.SMTP('mail.example.com')
server.sendmail(addr, [addr], msg)
server.quit()
As a side note, if you're just starting on a crash report collector, you may want to consider using a web service instead of a mail address. You're going to run into problems with people who can't access port 25 through their corporate firewall/proxy, write code that extracts the crash reports from an inbox (and/or searches via IMAP or mbox or whatever), deal with spammers who somehow find crash-reports#example.com and flood it with 900 messages about Cialis for each actual crash report, etc.

How to receive an email on server? Better using Python or Perl

I've checked so many articles, but can't find one for server to server email receiving. I want to write a program or some code just acts as an email receiver, not SMTP server or something else.
Let's suppose I have a domain named example.com, and a gmail user user#gmail.com sends me an email to admin#example.com, or a yahoo user user#yahoo.com sends me an email to test#example.com. Now, what do I do to receive this email? I prefer to write this code in Python or Perl.
Regards,
David

http://docs.python.org/library/smtpd.html
http://www.doughellmann.com/PyMOTW/smtpd/

In Perl:
Net::SMTP libraries (including Net::SMTP::Server).
Here's an example of using it: http://wiki.nil.com/Simple_SMTP_server_in_PERL

"reveive" is not a word. I'm really not sure if you mean "receive" or "retrieve".
If you mean "receive" then you probably do want an SMTP server, despite your claim. An SMTP server running on a computer is responsible for listening for network requests from other SMTP servers that wish to deliver mail to that computer.
The SMTP server then, typically, deposits the mail in a directory where it can be read by the recipient. They can usually be configured (often in combination with tools such as Procmail) to do stuff to incoming email (such as pass it to a program for manipulation along the way, this allows you to avoid having to write a full blown SMTP server in order to capture some emails).
If, on the other hand, you mean "retrieve", then you are probably looking to find a library that will let your program act as an IMAP or POP client. These protocols are used to allow remote access to a mailbox.

Good article at http://muffinresearch.co.uk/archives/2010/10/15/fake-smtp-server-with-python/ showing how to subclass smtpd.SMTPserver and how to run it.

Parsing emails as soon as they are received

I have users sending emails with some text I need to extract. Each user's email is mapped to a single mailbox. I'm currently using a cron job that polls the mailbox (postfix) every 5 minutes, checks for new messages, and sends it to a queue where I have workers parse them. I have two main questions:
Is there a way I can parse the email as soon as it's received instead of
polling the server? Also, how could
I implement this to be scalable? For
example, if there are 50 incoming
messages per second.
I'm programatically writing each user's email address to point to mailbox in the postfix configuration file. Would it be better to create a catch all account, so I don't have to write each email address? However, I know catch-all accounts are more susceptible to spam.

Use a pipe alias to catch the email, then use celery to dump it into a MQ for processing.

Yes, this can be done quite easily. All you need to do is configure the postfix to forward email to a script instead of to a mailbox. It does not really have to be a catch-all, you can configure postfix to forward specific emails to a script. The script can be written in any language. I wrote such script in php a couple of times. Another possibility for a very busy server, like 50 emails per second is to write your own filter server, then configure postfix to pass each message to your filter.
TO forward email to a script, in aliases file put a line like this: the path must point to this file
someaccount |/usr/local/bin/emailParser.php
To forward emails to a filter, it has to be configured in master.cf, a little more difficult.

I would recommend using Procmail for this. It is specifically designed to process your incoming mail and you can pass all mail with a certain property to your app.
http://www.procmail.org/
The spam problem with catchall addresses can usually be solved quite easily by monitoring all mail on the machine. If multiple addresses recieve the same mail, than there's a high probability that it's spam.

Deleting the most recently received email via Python script?

I use Gmail and an application that notifies me if I've received a new email, containing its title in a tooltip. (GmailNotifier with Miranda-IM) Most of the emails I receive are ones I don't want to read, and it's annoying having to login to Gmail on a slow connection just to delete said email. I believe plugin is closed source.
I've been (unsuccessfully) trying to write a script that will login and delete the 'top' email (the one most recently received). However this is not as easy I thought it would be.
I first tried using imaplib, but discovered that it doesn't contain any of the methods I hoped it would. It's a bit like the dbapi spec, containing only minimal functionality incase the imap spec is changed. I then tried reading the imap RFC (rfc3501). Halfway through it, I realized I didn't want to write an entire mail client, so decided to try using pop3 instead.
poplib is also minimal but seemingly has what I need. However pop3 doesn't appear to sort the messages in any order I'm familiar with. I have to either call top() or retr() on every single email to read the headers if I want to see the date received.
I could probably iterate through every single message header, searching for the most recent date, but that's ugly. I want to avoid parsing my entire mailbox if possible. I also don't want to 'pop' the mailbox and download any other messages.
It's been 6 hours now and I feel no closer to a solution than when I started. Am I overlooking something simple? Is there another library I could try? (I found a 'chilkat' one, but it's bloated to hell, and I was hoping to do this with the standard library)

import poplib
#connect to server
mailserver = poplib.POP3_SSL('pop.gmail.com')
mailserver.user('recent:YOURUSERNAME') #use 'recent mode'
mailserver.pass_('YOURPASSWORD') #consider not storing in plaintext!
#newest email has the highest message number
numMessages = len(mailserver.list()[1])
#confirm this is the right one, can comment these out later
newestEmail = mailserver.retr(numMessages)
print newestEmail
#most servers will not delete until you quit
mailserver.dele(numMessages)
mailserver.quit()
I worked with the poplib recently, writing a very primitive email client. I tested this with my email server (not gmail) on some test emails and it seemed to work correctly. I would send yourself a few dummy emails to test it out first.
Caveats:
Make sure you are using 'recent
mode':
http://mail.google.com/support/bin/answer.py?answer=47948
Make sure your Gmail account has POP3
enabled: Gmail > Settings >
Forwarding and POP/IMAP > "Enable POP
for all mail"
Hope this helps, it should be enough to get you going!