Develop Reference

How to loop GETs until a certain response is received

I'm looking for some advice, or a relevant tutorial regarding the following:
My task is to set up a flask route that POSTs to API endpoint X, receives a new endpoint Y in X's response, then GETs from endpoint Y repeatedly until it receives a certain status message in the body of Y's response, and then returns Y's response.
The code below (irrelevant data redacted) accomplishes that goal in, I think, a very stupid way. It returns the appropriate data occasionally, but not reliably. (It times out 60% of the time.) When I console log very thoroughly, it seems as though I have bogged down my server with multiple while loops running constantly, interfering with each other.
I'll also receive this error occasionally:
SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /book
import sys, requests, time, json
from flask import Flask, request
# create the Flask app
app = Flask(__name__)
# main booking route
#app.route('/book', methods=['POST']) #GET requests will be blocked
def book():
# defining the api-endpoints
PRICING_ENDPOINT = ...
# data to be sent to api
data = {...}
# sending post request and saving response as response object
try:
r_pricing = requests.post(url = PRICING_ENDPOINT, data = data)
except requests.exceptions.RequestException as e:
return e
sys.exit(1)
# extracting response text
POLL_ENDPOINT = r_pricing.headers['location']
# setting data for poll
data_for_poll = {...}
r_poll = requests.get(POLL_ENDPOINT, data = data_for_poll)
# poll loop, looking for 'UpdatesComplete'
j = 1
poll_json = r_poll.json()
update_status = poll_json['Status']
while update_status == 'UpdatesPending':
time.sleep(2)
j = float(j) + float(1)
r_poll = requests.get(POLL_ENDPOINT, data = data_for_poll)
poll_json = r_poll.json()
update_status = poll_json['Status']
return r_poll.text

This is more of an architectural issue more than a Flask issue. Long-running tasks in Flask views are always a poor design choice. In this case, the route's response is dependent on two endpoints of another server. In effect, apart from carrying the responsibility of your app, you are also carrying the responsibility of another server.
Since the application's design seems to be a proxy for another service, I would recommend creating the proxy in the right way. Just like book() offers the proxy for PRICING_ENDPOINT POST request, create another route for POLL_ENDPOINT GET request and move the polling logic to the client code (JS).
Update:
If you cannot for some reason trust the client (browser -> JS) with the POLL_ENDPOINT information in a hidden proxy like situation, then maybe move the polling to a task runner like Celery or Python RQ. Although, it will introduce additional components to your application, it would be the right way to go.

Probably you get that error because of the HTTP connection time out with your API server that is looping. There are some standards for HTTP time connection and loop took more time that is allowed for the connection. The first (straight) solution is to "play" with Apache configs and increase the HTTP connection time for your wsgi. You can also make a socket connection and in it check the update status and close it while the goal was achieved. Or you can move your logic to the client side.

pika for rabbitMQ crashing while using flask server

So we have a single thread flask server running where we receive requests from a python app client. In this flask server we use rabbitMQ with pika library to distribute messages to other clients.
What is happening is that in the get function the program is crashing with the error:
pika.exceptions.ConnectionClosed: (505, 'UNEXPECTED_FRAME - expected
content header for class 60, got non content header frame instead')
I've searched a lot of topics about this in stack overflow and others but they all address problems with multi threading which is not the case. Flask should only serve with one thread unless it is called in app.run(threaded=yes).
The program normally crashes when multiple messages are sent in a short interval (e.g. 5 per second) and it's also important to note that messages are being received every second with a request to this function:
#app.route('/api/users/getMessages', methods=['POST'])
def get_Messages():
data = json.loads(request.data)
token = data['token']
payload = jwt.decode(token, 'SECRET', algorithms=['HS256'])
istid = payload['istid']
print('istid: '+istid)
messages = []
queue = channel.queue_declare(queue=istid)
for i in range(queue.method.message_count):
method_frame, header_frame, body = channel.basic_get(queue=istid, no_ack=True)
if method_frame:
#print(method_frame, header_frame, body)
messages.append(body)
else:
print('No message returned')
res = {'messages':messages, 'error':0}
return jsonify(res)
In this code it crashes normally in the line:
queue = channel.queue_declare(queue=istid)
But we also tried to change the code to use a while instead of a for where it ends when the body is None and it crashes in the line:
method_frame, header_frame, body = channel.basic_get(queue=istid, no_ack=True)
in that case.
Also important, the crashes are random and it can work a few times and then randomly crashes after a get request while messages are being sent. If anyone knows anything related to this we would appreciate any help.
Another note, we thought about using basic_consume with callback instead of basic_get but we didn't find a way in which this would work since we have to send the messages back and have several user making requests to this same function.
EDIT #1:
In the rabbitMQ docs rabbitmq if you search for the function "def basic_get" you will notice there are some TODO comments and also a reference to this
Due to implementation details, this cannot be called a second time
until the callback is executed.
So I suspected that this could be what was happening but even if it is I don't know how could it be solved.

For anyone interested in the solution, as it is in the other comments, the program was not thread safe since flask as of version 1.0 uses threaded = True as default.
The solution is either:
1) running flask with app.run(threaded = False)
2) Making the program thread safe by implementing locks whenever accessing the channel /connection with pika.

Python Twilio making a call with Client

I am trying to make a call using twilio and python with the code below:
account_sid = "***"
auth_token = "***"
client = Client(account_sid, auth_token)
call = client.calls.create(to=phone_number, from_="+***", record=True, url="https://handler.twilio.com/twiml/***")
print call.sid
Here is my xml on that url:
<Response>
<Say>Hi, Thanks for accepting our call!</Say>
</Response>
The call connects, but after the xml triggers, the call ends.
Can someone point me out what I am doing wrong?
I can successfully make a call by doing the approach below, but I need the callsid right after dial for storing the callsid in the database to retrieve the recording later:
resp = VoiceResponse()
dial = Dial(caller_id='+1***', record="record-from-ringing")
dial.number(phone_number, url="https://handler.twilio.com/twiml/***")
resp.append(dial)
return HttpResponse(resp, mimetype='text/xml')
The url above is the same as the first example, but after playing the SAY tag, the call connects. Doing this approach doesn't allow me to get the callsid.
Any ideas?

The first call example ends because you run out of TwiML. You can place your in that TwiML to then connect the outbound-api call to another party.
For the second example using a instead of a REST API to the Calls resource,
You can use the recordingStatusCallbackEvent attribute to be informed of those details, once the recording is available.
https://www.twilio.com/docs/voice/twiml/dial#recordingstatuscallbackevent

django send_mail() function taking several minutes

I'm trying to send emails in a function within my views.py file. I've set up the email in my settings file in the same manner as here.
Python Django Gmail SMTP setup
Email sending does work but it takes several minutes to occur which my users have been complaining about. I am receiving a gethostbyaddress error in my var/log/mail.log file which I'll post here. I used to get nginx timeout errors but put "proxy_read_timeout 150;" into my /etc/nginx/sites-enabled/django file.
http://www.nginxtips.com/upstream-timed-out-110-connection-timed-out-while-reading-response-header-from-upstream/
This solved the timeout errors when interacting with the website but the emails still take several minutes to load. I'm using a digitalocean django droplet and this slow speed has occured on all my droplets.
Here's my view function
#login_required
def AnnouncementPostView(request, leaguepk):
league = League.objects.get(pk=leaguepk)
lblog = league.blog
if request.method == 'POST':
form = AnnouncementPostForm(request.POST)
if form.is_valid():
posttext = request.POST['text']
newAnnouncement = Announcement(text=posttext, poster=request.user)
newAnnouncement.save()
lblog.announce.add(newAnnouncement)
titleText = "%s Announcement" % (league.name)
send_mail(titleText, posttext, settings.EMAIL_HOST_USER, ['mytestemail#gmail.com'], fail_silently=False)
return HttpResponseRedirect(reverse('league-view', args=[league.pk]))
else:
form = AnnouncementPostForm()
return render(request, 'simposting/announcementpost.html', {'form': form, 'league': league})
This has worked, the announcement is posted to the desired page and is even emailed, it's just a time problem, people have come to expect nearly instant emailing processes which makes the 2-3 minutes unacceptable, especially when signing up also causes the 2-3 minute wait.
One issue may be the fact that while trying to solve this issue with the DigitalOcean support team I changed my droplet name and the hostname to be the domain that I set up.
My current hostname and droplet name is mydomain.com. I have it setup that way in my /etc/hostname file. My /etc/hosts file looks like this
127.0.0.1 localhost.localdomain localhost mydomain.com
127.0.1.1 mydomain.com
My var/log/mail.log file responds with this whenever I try to send mail
Oct 6 16:13:24 "oldDropletName" sm-mta[13660]: gethostbyaddr(10.xxx.xx.x) failed: 1
Oct 6 16:13:24 "oldDropletName" sm-mta[13662]: starting daemon (8.14.4): SMTP+queueing#00:10:00
I hope this is enough information to help, it's been troubling for several weeks and usually I can either solve my problems by looking up stuff here or working with the support team but it's got us stumped. Thank you for taking the time to help!

Sending an email is a network bound task and you don't know how long it will take to finish exactly like in your case. Although there might be a latency in your network but it's better to do such task in an async fashion so your main thread is free.
I am using the following code in one my projects.
utils.py
import threading
from django.core.mail import EmailMessage
class EmailThread(threading.Thread):
def __init__(self, subject, html_content, recipient_list, sender):
self.subject = subject
self.recipient_list = recipient_list
self.html_content = html_content
self.sender = sender
threading.Thread.__init__(self)
def run(self):
msg = EmailMessage(self.subject, self.html_content, self.sender, self.recipient_list)
msg.content_subtype = 'html'
msg.send()
def send_html_mail(subject, html_content, recipient_list, sender):
EmailThread(subject, html_content, recipient_list, sender).start()
just call send_html_mail from your view.

I am not particularly familiar with sendmail (I use postfix) but I would suspect this is almost certainly related to something with sendmail and probably not Django. The second log entry has "SMTP+queueing#00:10:00". and this link would indicate that sendmail takes a flag on startup to determine how often to process the mail queue. You may want to look around your init or wherever your startup scripts are and see how sendmail is configured. Also, if you are using Gmail you really can't control any delays on their end, so along with determining the configuration of your mail server, you'll need to check logs for when actions are actually occurring such as the mail being queued/sent. Is the time that line shows in your log from when the view was executed? If so, it is in the hands of sendmail.

What is the first step to getting in e-mail into my python / flask app code?

I am researching what it would take to make a web app that would interact with e-mails directly. Like you would send to something#myapp.com and the app would tear it apart and determine who it's from, if they are in the DB, what is the subject line, etc.
I am working with/most familiar with python and flask.
Could anyone get me started in the right direction of how to get an e-mail to interface with my flask app code?

There are several approach you can take:
write some code which uses IMAP or POP to retrieve emails and process them. Either run this from a crontab (or something similar) or add it to your flask app and trigger it in there, either through a crontab that requests a magic URL or setting up a custom timer thread.
configure your MTA to deliver email for something#myapp.com by feeding it to a program you write (for example in Exim you could use a pipe transport) . In that program you can either process it directly, or do something like POSTing it to your flask app.

I've done something along these lines recently, with a simple bookmarking web-app. I have the usual bookmarklet way of bookmarking something to it, but I also wanted to be able to e-mail links to it from apps like Reeder on my iPhone and whatever. You can see what I ended up with on GitHub: subMarks.
I use Google Apps for your Domain for my email, so I created a special address for my app to look at - I really didn't want to try building/configuring my own e-mail server.
the mail_daemon.py file from above is run as a cron job every 5 minutes. It connects to the email server using the poplib Python package, processes the emails that are there and then disconnects (one part I feel compelled to point out is that I check that the emails are from me before they are processed :) )
My Flask app then provides the front end to the bookmarks, displaying them from the database.
I decided not to put the email handling code into the actual flask app, because it can be rather slow and would only run when the page was visited, but you could do this if you wanted.
Here's some barebones code to get things going:
import poplib
from email import parser
from email.header import decode_header
import os
import sys
pop_conn = poplib.POP3_SSL('pop.example.com')
pop_conn.user('my-app#example.com')
pop_conn.pass_('password')
#Get messages from server:
messages = [pop_conn.retr(i) for i in range(1, len(pop_conn.list()[1]) + 1)]
# Concat message pieces:
messages = ["\n".join(mssg[1]) for mssg in messages]
#Parse message into an email object:
messages = [parser.Parser().parsestr(mssg) for mssg in messages]
for message in messages:
# check message is from a safe recipient
if 'me#example.com' in message['from']:
# Get the message body text
if message['Content-Type'][:4] == 'text':
text = message.get_payload() #plain text messages only have one payload
else:
text = message.get_payload()[0].get_payload() #HTML messages have more payloads
# decode the subject (odd symbols cause it to be encoded sometimes)
subject = decode_header(message['subject'])[0]
if subject[1]:
bookmark_title = subject[0].decode(subject[1]).encode('ascii', 'ignore') # icky
else:
bookmark_title = subject[0]
# in my system, you can use google's address+tag#gmail.com feature to specifiy
# where something goes, a useful feature.
project = message['to'].split('#')[0].split('+')
### Do something here with the message ###
pop_conn.quit()