I'm currently using the following views.py function sendsmss to allow a user to do a bulk sms message to their list of subscribers, after the user has completed an html form with the sms they want to send to their subscribers:
def sendsmss(request):
if request.method == "POST":
subscribers = Subscriber.objects.all()
sms = request.POST['sms']
mytwilionum = "+13421234567"
ACCOUNT_SID = TWILIO_ACCOUNT_SID
AUTH_TOKEN = TWILIO_AUTH_TOKEN
client = Client(ACCOUNT_SID, AUTH_TOKEN)
for subscriber in subscribers:
subscriber_num = subscriber.phone_number
client.messages.create(
to= subscriber_num,
from_=mytwilionum,
body=sms
)
return redirect('homepage')
This function works, but I have only tested the bulk send with 3 subscribers. If there were 100s or 1000s of subscribers how long would this take .. if it takes long then would user be waiting for task to complete before redirect to homepage happens? Is there a better way to do this in Django?
The questions are very subjective and I will try to answer those accordingly:
If there were 100s or 1000s of subscribers how long would this take
This is totally dependent on performance of Twilio. The API client is using the requests library and it is creating the messages one by one for each subscriber. In an ideal scenario the time taken seems proportional to the number of subscribers.
if it takes long then would user be waiting for task to complete before redirect to homepage happens?
Based on your current implementation, Yes. The return redirect('homepage') will be executed only after the message has been sent to all the subscribers. In case, there is an error it will be thrown and the page won't redirect to the home page.
Is there a better way to do this in Django?
Yes, there are. You can use an asynchronous job queue e.g. celery and hook it up with django. In this, you can start an async task in celery and return a response to the user immediately. You can also choose to display progress of the running celery task to the user (if required).
I'm having some trouble understanding and implementing the Google Directory API's users watch function and push notification system (https://developers.google.com/admin-sdk/reports/v1/guides/push#creating-notification-channels) in my Python GAE app. What I'm trying to achieve is that any user (admin) who uses my app would be able to watch user changes within his own domain.
I've verified the domain I want to use for notifications and implemented the watch request as follows:
directoryauthdecorator = OAuth2Decorator(
approval_prompt='force',
client_id='my_client_id',
client_secret='my_client_secret',
callback_path='/oauth2callback',
scope=['https://www.googleapis.com/auth/admin.directory.user'])
class PushNotifications(webapp.RequestHandler):
#directoryauthdecorator.oauth_required
def get(self):
auth_http = directoryauthdecorator.http()
service = build("admin", "directory_v1", http=auth_http)
uu_id=str(uuid.uuid4())
param={}
param['customer']='my_customer'
param['event']='add'
param['body']={'type':'web_hook','id':uu_id,'address':'https://my-domain.com/pushNotifications'}
watchUsers = service.users().watch(**param).execute()
application = webapp.WSGIApplication(
[
('/pushNotifications',PushNotifications),
(directoryauthdecorator.callback_path, directoryauthdecorator.callback_handler())],
debug=True)
Now, the receiving part is what I don't understand. When I add a user on my domain and check the app's request logs I see some activity, but there's no usable data. How should I approach this part?
Any help would be appreciated. Thanks.
The problem
It seems like there's been some confusion in implementing the handler. Your handler actually sets up the notifications channel by sending a POST request to the Reports API endpoint. As the docs say:
To set up a notification channel for messages about changes to a particular resource, send a POST request to the watch method for the resource.
source
You should only need to send this request one time to set up the channel, and the "address" parameter should be the URL on your app that will receive the notifications.
Also, it's not clear what is happening with the following code:
param={}
param['customer']='my_customer'
param['event']='add'
Are you just breaking the code in order to post it here? Or is it actually written that way in the file? You should actually preserve, as much as possible, the code that your app is running so that we can reason about it.
The solution
It seems from the docs you linked - in the "Receiving Notifications" section, that you should have code inside the "address" specified to receive notifications that will inspect the POST request body and headers on the notification push request, and then do something with that data (like store it in BigQuery or send an email to the admin, etc.)
Managed to figure it out. In the App Engine logs I noticed that each time I make a change, which is being 'watched', on my domain I get a POST request from Google's API, but with a 302 code. I discovered that this was due to the fact I had login: required configured in my app.yaml for the script, which was handling the requests and the POST request was being redirected to the login page, instead of the processing script.
I am researching what it would take to make a web app that would interact with e-mails directly. Like you would send to something#myapp.com and the app would tear it apart and determine who it's from, if they are in the DB, what is the subject line, etc.
I am working with/most familiar with python and flask.
Could anyone get me started in the right direction of how to get an e-mail to interface with my flask app code?
There are several approach you can take:
write some code which uses IMAP or POP to retrieve emails and process them. Either run this from a crontab (or something similar) or add it to your flask app and trigger it in there, either through a crontab that requests a magic URL or setting up a custom timer thread.
configure your MTA to deliver email for something#myapp.com by feeding it to a program you write (for example in Exim you could use a pipe transport) . In that program you can either process it directly, or do something like POSTing it to your flask app.
I've done something along these lines recently, with a simple bookmarking web-app. I have the usual bookmarklet way of bookmarking something to it, but I also wanted to be able to e-mail links to it from apps like Reeder on my iPhone and whatever. You can see what I ended up with on GitHub: subMarks.
I use Google Apps for your Domain for my email, so I created a special address for my app to look at - I really didn't want to try building/configuring my own e-mail server.
the mail_daemon.py file from above is run as a cron job every 5 minutes. It connects to the email server using the poplib Python package, processes the emails that are there and then disconnects (one part I feel compelled to point out is that I check that the emails are from me before they are processed :) )
My Flask app then provides the front end to the bookmarks, displaying them from the database.
I decided not to put the email handling code into the actual flask app, because it can be rather slow and would only run when the page was visited, but you could do this if you wanted.
Here's some barebones code to get things going:
import poplib
from email import parser
from email.header import decode_header
import os
import sys
pop_conn = poplib.POP3_SSL('pop.example.com')
pop_conn.user('my-app#example.com')
pop_conn.pass_('password')
#Get messages from server:
messages = [pop_conn.retr(i) for i in range(1, len(pop_conn.list()[1]) + 1)]
# Concat message pieces:
messages = ["\n".join(mssg[1]) for mssg in messages]
#Parse message into an email object:
messages = [parser.Parser().parsestr(mssg) for mssg in messages]
for message in messages:
# check message is from a safe recipient
if 'me#example.com' in message['from']:
# Get the message body text
if message['Content-Type'][:4] == 'text':
text = message.get_payload() #plain text messages only have one payload
else:
text = message.get_payload()[0].get_payload() #HTML messages have more payloads
# decode the subject (odd symbols cause it to be encoded sometimes)
subject = decode_header(message['subject'])[0]
if subject[1]:
bookmark_title = subject[0].decode(subject[1]).encode('ascii', 'ignore') # icky
else:
bookmark_title = subject[0]
# in my system, you can use google's address+tag#gmail.com feature to specifiy
# where something goes, a useful feature.
project = message['to'].split('#')[0].split('+')
### Do something here with the message ###
pop_conn.quit()
My script grabs the content of an rss page gets the urls in that page saves them to a list then it grabs the content of each url and it emails the contents of the page to me. Everything is working very well accept I can't send every link in the list. Typically about 22 links in the list. I don't want to combine the contents of multiple links into one email. If I don't add a timeout I get an over quota error like this
<class 'google.appengine.runtime.apiproxy_errors.OverQuotaError'>: The API call mail.Send() required more quota than is available.
After I added "time.sleep(9)" to slow it down it gives me this error.
<class 'google.appengine.runtime.DeadlineExceededError'>:
Traceback (most recent call last):
Here is my code.. Any thoughts?
size = len(my_tabletest)
a=2
while a < size:
url = my_tabletest[a].split('html</link>')[0] + "print"
url_hhhhhh = urlfetch.fetch(url)
my_story = url_hhhhhh.content
my_story = my_story.split('<div class="printstory">')[1]
my_story_subject = my_story.split('<h1>')[1]
my_story_subject = my_story_subject.split('</h1>')[0]
my_story = ''.join(BeautifulSoup(my_story).findAll(text=True))
message = mail.EmailMessage(sender="me<me#someplace.com>",
subject=my_story_subject)
message.to = "Jim <me#someplace.com>"
message.body = my_story
message.html = my_story_html
message.send()
time.sleep(9)
a=a+1
Welcome to Stack Overflow!
The task queue is built to solve this problem. You can leverage it with minimal change to your existing code using the deferred library:
Instead of calling message.send(), do something like this:
def send_email(message):
message.send()
deferred.defer(send_email, message)
This will create a batch of ad-hoc tasks that send your emails in the background, after your main request handler has returned. Some of these tasks will probably fail on the first try as your app hits short term quota limits for outbound mail. That's OK; failed tasks will back off and retry automatically until they succeed.
Edit: Oh, and take the sleep out of your code. =)
Edit #2: You can speed things up further by moving the urlfetch into the task, so each task fetches one URL and then sends one email. Fetching 22 URLs in one request handler could be enough to cause timeouts, independent of sending mail.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
I need to test that my Django application sends e-mails with correct content. I don't want to rely on external systems (like an ad-hoc gmail account), since I'm not testing the actual e-mail service...
I would like to, maybe, store the emails locally, within a folder as they are sent.
Any tip on how to achieve it?
Django test framework has some built in helpers to aid you with testing e-mail service.
Example from docs (short version):
from django.core import mail
from django.test import TestCase
class EmailTest(TestCase):
def test_send_email(self):
mail.send_mail('Subject here', 'Here is the message.',
'from#example.com', ['to#example.com'],
fail_silently=False)
self.assertEqual(len(mail.outbox), 1)
self.assertEqual(mail.outbox[0].subject, 'Subject here')
You can use a file backend for sending emails which is a very handy solution for development and testing; emails are not sent but stored in a folder you can specify!
If you are into unit-testing the best solution is to use the In-memory backend provided by django.
EMAIL_BACKEND = 'django.core.mail.backends.locmem.EmailBackend'
Take the case of use it as a py.test fixture
#pytest.fixture(autouse=True)
def email_backend_setup(self, settings):
settings.EMAIL_BACKEND = 'django.core.mail.backends.locmem.EmailBackend'
In each test, the mail.outbox is reset with the server, so there are no side effects between tests.
from django.core import mail
def test_send(self):
mail.send_mail('subject', 'body.', 'from#example.com', ['to#example.com'])
assert len(mail.outbox) == 1
def test_send_again(self):
mail.send_mail('subject', 'body.', 'from#example.com', ['to#example.com'])
assert len(mail.outbox) == 1
Use MailHog
Inspired by MailCatcher, easier to install.
Built with Go - MailHog runs without installation on multiple platforms.
Also, it has a component called Jim, the MailHog Chaos Monkey, which enables you to test sending emails with various problems happening:
What can Jim do?
Reject connections
Rate limit connections
Reject authentication
Reject senders
Reject recipients
Read more about it here.
(Unlike original mailcatcher, which failed on me when sending emails with emoji, encoded in UTF-8 and it WASN'T really fixed in the current release, MailHog just works.)
For any project that doesn't require sending attachments, I use django-mailer, which has the benefit of all outbound emails ending up in a queue until I trigger their sending, and even after they've been sent, they are then logged - all of which is visible in the Admin, making it easy to quickly check what you emailing code is trying to fire off into the intertubes.
Django also has an in-memory email backend. More details in the docs under In-memory backend. This is present in Django 1.6 not sure if it's present in anything earlier.
Patching SMTPLib for testing purposes can help test sending mails without sending them.
http://www.psychicorigami.com/2007/09/20/monkey-patching-pythons-smtp-lib-for-unit-testing/
Why not start your own really simple SMTP Server by inherit from smtpd.SMTPServer and threading.Thread:
class TestingSMTPServer(smtpd.SMTPServer, threading.Thread):
def __init__(self, port=25):
smtpd.SMTPServer.__init__(
self,
('localhost', port),
('localhost', port),
decode_data=False
)
threading.Thread.__init__(self)
def process_message(self, peer, mailfrom, rcpttos, data, **kwargs):
self.received_peer = peer
self.received_mailfrom = mailfrom
self.received_rcpttos = rcpttos
self.received_data = data
def run(self):
asyncore.loop()
process_message is called whenever your SMTP Server receive a mail request, you can do whatever you want there.
In the testing code, do something like this:
smtp_server = TestingSMTPServer()
smtp_server.start()
do_thing_that_would_send_a_mail()
smtp_server.close()
self.assertIn(b'hello', smtp_server.received_data)
Just remember to close() the asyncore.dispatcher by calling smtp_server.close() to end the asyncore loop(stop the server from listening).
Tying a few of the pieces here together, here's a straightforward setup based on filebased.EmailBackend. This renders a list view linking to the individual log files, which have conveniently timestamped filenames. Clicking a link in the list displays that message in the browser (raw):
Settings
EMAIL_BACKEND = "django.core.mail.backends.filebased.EmailBackend"
EMAIL_FILE_PATH = f"{MEDIA_ROOT}/email_out"
View
import os
from django.conf import settings
from django.shortcuts import render
def mailcheck(request):
path = f"{settings.MEDIA_ROOT}/email_out"
mail_list = os.listdir(path)
return render(request, "mailcheck.html", context={"mail_list": mail_list})
Template
{% if mail_list %}
<ul>
{% for msg in mail_list %}
<li>
{{ msg }}
</li>
{% endfor %}
</ul>
{% else %}
No messages found.
{% endif %}
urls
path("mailcheck/", view=mailcheck, name="mailcheck"),
If you have a TomCat server available, or other servlet engine, then a nice approach is "Post Hoc" which is a small server that looks to the application exactly like a SMTP server, but it includes a user interface that allows you to view and inspect the email messages that were sent. It is open source and freely available.
Find it at: Post Hoc GitHub Site
See the blog post: PostHoc: Testing Apps that Send Email