Develop Reference

Creating a web service best practices

I've published an iOS app using a local database stored directly on the user's device. What I'd like to do for my next app is have a central database on a remote server where users can asynchronously send/receive data. I'm relatively new to web service programming and I'm not sure where to start. I've purchased a server space on a web hosting site and have a MySql database/phpMyAdmin configuration on the server. I'm not sure how to handle the server side code. I know well enough that a database should never be exposed publicly over the internet for obvious security reasons. Therefore I need some kind of web service where my iOS/Android apps can query the service, the web service fetches data and sends it back to the clients in a XML or JSON format. I'd like to write the web service layer in python. I've done a little research and django seems to be recommended by some for these kinds of things. So, my questions are:
What are good resources for making a web service in python on a remote server.
What are the "best practices" for creating/debugging/testing the server side code. Should I try and make a local MySql database and write the server code locally and test and then push it to my remote server when it is finished?
My ultimate starting goal is some kind of proof of concept hello world app. Where from my iOS and android devices I can query the remote database going through the service layer and getting data back or inserting data.
Any tips or advise would be appreciated. I'm a noob in this area but ready to learn.

Too many general questions in one, so I'll just refer you.
I'd start with an awesome Designing Poetic APIs PyCon US 2014 talk.
Since you want to follow the REST principles with Django, take a look at the list of django packages available for writing RESTful apps: Applications that help you build a REST API.
Basically, the major "players" are:
django-tastypie
django-rest-framework
Also see:
What are the differences between django-tastypie and djangorestframework?
Choosing an API framework for Django
Hope that helps.

send google hangout notification using python

Is there an API which allows me to send a notification to Google Hangout? Or is there even a python module which encapsulates the Hangout API?
I would like to send system notification (e.g. hard disk failure reports) to a certain hangout account. Any ideas, suggestions?

Hangouts does not currently have a public API.
That said, messages delivered to the Google Talk XMPP server (talk.google.com:5222) are still being delivered to users via Hangouts. This support is only extended to one-on-one conversations, so the notification can't be delivered to a group of users. The messages will need to be supplied through an authenticated Google account in order to be delivered.

There is pre-alpha library for sending hangouts messages on python:
https://pypi.python.org/pypi/hangups/0.1
The API have been reverse engineered and is not published (as someone posted in comments). Thus it might change a Google's will.
Also, messages sent using XMPP ceased to be delivered to Hangouts users. I guess this is another cut (of the thousand scheduled).

I send alarms and other notifications with a python script (failures on database server, partitions without free space, etc), using hangouts. It's easy. Look at http://www.administracion-linux.com/2014/07/enviar-mensajes-por-hangout-desde.html to send hangouts.

Is there a toolkit that will provide a web-based API (xmlrpc, REST, whatever) to remote IMAP servers?

I'm working on a Google App Engine project that needs to access IMAP. Context.IO isn't quite powerful enough for my purposes, but I'd like something in the same spirit: I want to log into, access, and manipulate hundreds of IMAP mailboxes from within Google App Engine, using either a third-party service or an application server I put on a dedicated hosting server.
As you might imagine, this is mostly to work around the opening sockets limitation in GAE.
Any recommendations?

I don't know of any pre-made solution, but rolling your own shouldn't be very difficult or take too long. You can build on IMAPClient and SimpleXMLRPCServer on the server and use xmlrpclib on the client.
You would need to think about a way to retain state between calls though, since XmlRPC is a connectionless protocol (as most other RPC mechanisms are as well) and implement some form of service authentication. I have written a class inherited from SimpleXMLRPCServer, which supports SSL connections and HTTP Basic Auth (xmlrpclib already has support for both items). If you're interested in the code, give me a shout.

Have a look at Mailgun; it offers a robust API and supports IMAP V4.*
* IMAP mailboxes are on Mailgun

Record audio in Google App Engine using rtmplite?

I am in the process of building a Google App Engine application which requires audio to be recorded and saved in our database. I have found no alternative to using some form of RTMP server for recording audio through flash, so [rtmplite] (http://code.google.com/p/rtmplite/) came into our horizon.
Since I have no experience with rtmplite, is it the right choice for our project? Or is there any other Python-based RTMP solution that allows audio recording? Any flash client you can recommend?
Thanks!

Google App Engine is tricky for RTMP because it does not support sockets. You would have to use something like RTMPT which is tunneled over HTTP, however, this tunneling incurs latency so if you are looking to do anything realtime this could become an issue.
Currently rtmplite does not support RTMPT so this would not be possible at the moment. I am involved in a project, RTMPy (http://rtmpy.org), that is planning support for RTMPT and AppEngine. Unfortunately AppEngine support is probably a few months out.

Try appengine backends, they currently don't whitelist a lot of things required for such streaming. But they might soon do so. Once they enable sockets, then rtmplite or rtmpy could easily be ported to run there. Backends already support unlimited request length which is required for streaming.

Feedback on availability with Google App Engine

We've had some good experiences building an app on Google App Engine, this first app's target audience are Google Apps users, so no issues there in terms of it being hosted on Google infrastructure.
We like it so much that we would like to investigate using it for another app, however this next project is for a client who is not really that interested in what technology it sits on, they just want it to work, and work all of the time.
In this scenario, given that we have the technology applicability and capability side covered, are there any concerns that this stuff is still relatively new and that we may not be as much "in control" as if we had it done with traditional hosting?

You are correct: you are not in as much control vs. traditional hosting. However, hopefully the gains outweight the negatives. App Engine is extremely scalable -- it runs on the same hardware that runs Google itself. How often have you visited http://google.com and had that page or a search result fail?
Although you are letting Google run your code, the code is still your's to do as you please. With new projects like django-nonrel, you can create and run native Django apps directly on top of App Engine, and if it doesn't meet your needs down the line, it's fairly easy to take that app to an ISP that hosts Django apps (and there are plenty of those). More on this project below.
You don't have to worry about hardware, operating systems, coming up with a machine image, databases, web servers, front-end load balancers, CDNs/edge caching, software/package upgrades, license fees, etc. All these things are tangential to the web or other application that you have or will create to solve a particular problem. All this additional infrastructure is required whether you like it or not; but with App Engine, you only need to think about your app/solution and none of this extra stuff.
Obviously another thing you lose is some of your execution environment. To ensure that you're playing nicely with your cloud neighbors (resource hogging, security issues, etc.), you must execute in a sandbox, meaning your app cannot create local files, open network sockets, etc. However, App Engine provides a rich set of APIs and product features so that you at least can create meaningful apps:
scalable distributed object datastore (see below)
Memcache
URLFetch
images service (resize, crop, etc.)
users service/authentication task queues for background processing
Django web templating
blobstore for large files
denial-of-service blacklisting
transational tasks
datastore cursors
sending (and/or receiving) of email
sending (and/or receiving) of chat/IM/instant messages via XMPP
You also have a full dashboarded administration console which will let you monitor your app's usage, your billing settings and history, a full dump of your quota usage, and even your application logs which you can view or download.
To address the "main sore points" from #Anurag:
1a. the free quotas are fairly generous... enough to power a website that gets 5MM views/month. also, if you trust Google to give them your credit card, they will bump up the free quota levels even higher. look at their quota page and refer to the numbers in both the "Free Default Quota" and "Billing Enabled Default Quota" columns... here are some examples: a) # of Requests: 1.3MM default, 43MM w/billing enabled (wBE), b) Datastore API calls: 10MM default, 140MM wBE, c) URL Fetches: 657K default, 46MM wBE
1b. 30s max for requests: this is more security for you, because your app is now in a playground with others. Google has to ensure that all cloud neighbors play nicely with each other and not hog the CPU. However, the App Engine team is working on a way to allow for longer running background tasks... there's no timetable yet, but it is on the public roadmap.
1c. writing a chat server on App Engine is not only possible, but it is quite simple. here's one created using App Engine's XMPP API -- it's pretty dumb and just echoes back to the sender what they transmitted to us (be aware that you must have already invited the user to chat):
from google.appengine.api import xmpp
from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app
class XMPPHandler(webapp.RequestHandler):
def post(self):
msg = xmpp.Message(self.request.POST)
msg.reply("I got your msg: '%s'" % msg.body)
application = webapp.WSGIApplication([
('/_ah/xmpp/message/chat/', XMPPHandler),
], debug=True)
def main():
run_wsgi_app(application)
if __name__ == '__main__':
main()
1d. another item on the the public roadmap is future "[support] for Browser Push (Comet) communication", so that's coming too.
2a. "not SQL" is one of Google App Engine's greatest strengths! relational databases don't scale and must be sharded at some point to keep an RDBMS from falling over. it is true however, that it is slightly more difficult to port because it is not traditional! Based on Google Bigtable, you can think of the App Engine datastore as a scalable distributed object database. App Engine lets you query the datastore using a Query object model, or if you insist, they also provide a SQL-like GqlQuery interface.
2b. with new avantgarde projects like django-nonrel, if you create a Django app and use its ORM, you can take a pure Django app and run it directly on top of App Engine. likewise, you can it off of App Engine and move it directly to more traditional ISP vendor that hosts Django applications. the queries stay exactly the same, and you don't have to care if it does SQL or not.
3a. long-running processes are already addressed in 1b above. Google is aware of this need and are working on it.
3b. the TaskQueue API supports 100k calls, but that's bumped to 1MM wBE... and this is on a daily basis.
3c. Google strongly encourages breaking up tasks into multiple subtasks. low latency apps are seen not to "hog the system" and are given better treatment than those which are slow and consume more resources from their cloud neighbors.

Yes, you would not be in as much control as with traditional hosting. Main sore points of GAE are
Quotas etc, 30 sec max for a request, so comet/reverse ajax etc out of window or very difficult. Try writing a chat server on google app engine.
Not Sql database, so difficult to port to other server if need be and sometime limitation with google database e.g. try sorting a query which has comparison on different column other than the sorted one.
Long running process, there is a Task api but that doesn't suffice if you want to do long running background processing, otherwise you will have to break your task in subtasks, so things get complicated and there are even quotas on how many tasks per sec you can run.
GAE is good if you app can be modeled as request-response registry, with little background processing.
See this too
Feedback on using Google App Engine?