I have a Pyramid web application that needs to send emails such as confirmation emails after registration, newsletters and so forth. I know how to send emails using smtplib in python and I decided on an smtp service (I think sendgrid will do the trick).
The real problem is the scheduling and delay sending of the emails - for example, when a user registers, the email is to be sent on the form post view. But, I don't want to block the request, and therefore would like to "schedule" the email in a non-blocking way.
Other than implementing this myself (probably with a DB and a worker), is there an existing solution to email queue and scheduling?
Thanks!
The existing solution to which you refer is to run your own SMTP server on the machine, bound only to localhost to prevent any other machines from connecting to it. Since you're the only one using it, submitting a message to it should be close to instantaneous, and the server will handle queuing, retries, etc. If you are running on a UNIX/Linux box, there's probably already such a server installed.
In my app, I insert emails in DB table, and I have python script running under cron that checks this table and sends email updating record as sent.
You can also use Redis Lists as a Queue to send emails. Create couple of worker processes which listens a Redis List, and
publish a Email job via RPUSH or LPUSH
receive the job at your worker via LPOP or RPOP
so that your web app worker process won't be affected, or not even feel the overhead for email sending operations.
This design allows you not to care how long does it take to send an email. The email service might be local or a external email service, however you want.
Related
I have a question about implementing passing data between two apps that I have running.
I have a Flask backend, which receives a user email via a POST request and stores it in a variable. I need to pass this variable into TelegramBotAPI.
What I need to happen is when Flask receives a new user email, it will pass the data into the TelegramBotAPI and trigger a function which will send it in a message to the user.
How do I go about implementing this?
Assuming your architecture separates the TelegramBot from the Flask server, you would treat your flask server as the client for your Telegram Bot which is the server in this scenario.
Armed with this information there are numerous ways to go about this. TelegramBot could provide an http server through which clients could send requests. Another option which requires less availability from the Bot but ensures eventual consistency is using a message broker like RabbitMQ.
Lastly, if you wish to store the email permanently you should consider using a storage like a Database (for robustness) or a File system (if you have only one server)
I'm using django-sendgrid-v5 and I read somewhere that it isn't good to send emails from the main webserver. Should I process emails from Celery? Or is it fine to call from the main app since I'm using an external service like Sendgrid anyways?
I don't know in which context you've read that, but I would guess it has something to do with reliability, spam and security in general.
Short answer: Yes, this should be fully okay as you are using an external email service.
Another option is to set up a Smart host on your webserver and let your main email server deliver it to the final recipients.
Long answer: Nowadays sending emails from a (web)server, which is not fully set up as an email server might be difficult in means of reliably sending emails.
Due to the massive amounts of spam and malware sent, most (or at least a lot) receiving email servers (Mail Exchangers) are trying to check if the emails they should deliver to their users, are legit.
This is done by several settings mostly on the server itself. To name only a few: RDNS, DKIM, Greylisting, etc.
In general a (web) server whos main purpose is not sending emails, does not have all these settings. This might result in difficulties to reach certain email addresses.
I have a use case where I have to send_email to user in my views. Now the user who submitted the form will not receive an HTTP response until the email has been sent . I do not want to make the user wait on the send_mail. So i want to send the mail asynchronously without caring of the email error. I am using using celery for sending mail async but i have read that it may be a overkill for simpler tasks like this. How can i achieve the above task without using celery
I'm assuming you don't want to wait because you are using an external service (outside of your control) for sending email. If that's the case then setup a local SMTP server as a relay. Many services such as Amazon SES, SendGrid, Mandrill/Mailchimp have directions on how to do it. The application will only have to wait on the delivery to localhost (which should be fast and is within your control). The final delivery will be forwarded on asynchronously to the request/response. STMP servers are already built to handle delivery failures with retries which is what you might gain by moving to Celery.
I have a django-rest-framework API, and am trying to understand how sending email from it works. Suppose I'm using django.core.mail.backends.smtp.EmailBackend as email backend to send emails. Sending of an email is quite slow, and I'm wondering, if the django main thread will be blocked somehow during that time so that the other APIs would be unusable during it? Is that true? Would it be a good call to send the email in a background process created by celery for example?
Yes. Django thread is blocked for that particular user. You might want to use Celery along with Rabbit Mq for sending mail in background.
I confirm the thread handling the request will be blocked until the email is sent. In a typical Django setup one thread is created per request.
I have a live site, but would like to run a periodic task that would periodically check if various pages and email sending work.
e.g. register page, forgot password page, etc...
I'm having quite a problem figuring out how to do this...
Any ideas? :)
At my workplace we use nagios to monitor, among many things, that the smtp service and web apps are running as it should. For crucial pages, we use the check_http nagios plugin to ensure that the pages are accessible and contain specific keywords.
You can set each check to run as frequently as you like, and have nagios send you an email if anything goes amiss.
As for checking whether email sending from django is working, that's a little tricky. None of my sites are mission critical, so I'm happy to assume that if my code base doesn't change and if the web apps and SMTP servers are working, then emails from django should be ok.
If it is critical that you check that django can send emails, you could have a view (restricted by source IP and perhaps a password in POST/GET) that triggers an outgoing email. You can then write your own plugin to access said view and check that the email is sent.
Here's a nice write-up on how to check your email service using nagios: http://www.linuxjournal.com/content/monitoring-email-nagios