Develop Reference

Converting proto buffer to ProtoRPC

In a Python script, mylibrary.py, I use Protocol Buffers to model data using the following approach:
Defining message formats in a .proto file.
Use the protocol buffer compiler.
Use the Python protocol buffer API to write and read messages in the .py module.
I want to implement Cloud Endpoints Framework on App Engine that imports and uses the aforementioned Python script, however Cloud Endpoints uses ProtoRPC, not 'standard' Protocol Buffers.
My App Engine Python module, main.py, imports from protorpc rather than using the 'offline' protoc compiler to generate serialization and deserialization code:
from protorpc import messages
from protorpc import remote
Messages are not defined​ using .proto files. Instead, classes are defined, inheriting from protorpc.messages.Message:
class MyMessageDefinition(messages.Message)
Can Proto Buffers be converted to Proto RPC equivalents? I don't really want to change mylibrary.py to use ProtoRPC, since it's less generic than Protocol Buffers.

After eight months and lots experimentation, I'll add my opinion. I hope it saves someone time.
Choose Your Framework First
There are different Cloud Endpoint offerings from Google Cloud. All can be used for JSON/REST APIs. This wasn't immediately clear to me. Cloud Endpoints is a very high-level phrase covering development, deployment and management of APIs on multiple Google Cloud backends.
The point here is that after deciding to use Cloud Endpoints, you must still decide on backend technologies to serve your API. The documentation feels a little hidden away, but I strongly recommend starting with the Google Cloud Endpoints doc.
You can choose between:
OpenAPI Specification
Endpoints Frameworks
gRPC
Choose Your Implementation Second
Within each API Framework there’s a choice of Cloud implementations upon which your API (service) can run:
OpenAPI Specification
- for JSON/REST APIs implemented on:
Google App Engine flexible environment
Google Compute Engine
Google Container Engine
Kubernetes
Endpoints Frameworks
- for JSON/REST APIs implemented on:
Google App Engine standard environment with Java
Google App Engine standard environment with Python
gRPC
- for gRPC APIs implemented on:
Google Compute Engine
Google Container Engine
Kubernetes
When posting question here, I was using Endpoints Frameworks running on Google App Engine standard environment with Python. I then migrated my API (service) to gRPC on Google Compute Engine.
The observant among you may notice both the OpenAPI Specification and Endpoints Frameworks can be used for JSON/REST APIs, while gRPC only exposes a gRPC API. So how did I port my REST API from Endpoints Frameworks to gRPC? The answer is Transcoding HTTP/JSON to gRPC (which I learnt along the way, and was not immediately clear to me). So, don't rule out gRPC just because you want REST/HTTP.
The Answer
So how does this related to my original question?
That I was trying to convert between .proto files and gRPC annotations at all, meant I had taken a wrong-turning along the way.
If you want to write an application using plain .proto files, then choose gRPC on Compute Engine. If you need this to be a REST API, this can be done, but you'll need to add an ESP into your backend configuration. It's pretty much an NGINX sever setup as a reverse proxy. The only downside here is you'll need some Docker knowledge to ensure the ESP (proxy) and your gRPC server can communicate (Docker networking).
If your code is already on an App Engine, and you want to expose it as a REST API with minimum effort and still get good API management features, choose Endpoints Frameworks. Warning: I moved away from this because it was prohibitively expensive (I was getting billed in the region of $100 USD monthly).
If you want to avoid .protos altogether, then go with OpenAPI Specification.
Lastly, if you want to offer programmatic integration, client libraries, or you want to offer a microservice, then really do consider gRPC. It's easy to remove the ESP (proxy) and run a gRPC server on nearly any machine (as long as the Protocol Buffer Runtime is installed.
Ultimately I settled on gRPC on Compute Engine with Docker. I also have an ESP to provide a HTTP transcoding to gRPC and vice-versa. I like this approach for a few reasons:
You learn a lot: Docker, Docker Networking, NGINX configuration, Protocol Buffers, ESP (Cloud Proxy), gRPC servers.
The service (core business) logic can be written with plain-old gRPC. This allows the service to be run on any machine without a web server. Your business logic, is the server :)
Protocol Buffers / gRPC are excellent for isolating business logic as a service...or microservice. They're also good for providing well-defined interfaces and libraries.
Avoid These Mistakes
Implementing the first framework / architecture you find. If I could start again, I would not choose Endpoints Frameworks. It's expensive, and uses annotations rather than .proto files, which, IMO, makes the code harder to port.
Read Always Free Usage Limits before deciding upon a framework and implementation. Endpoints Frameworks uses backend App Engine instances - which have almost no free quota. Confusing, frontend App Engine instances have a very generous free quota.
Consider Local Development. Cloud Endpoints local development servers are not officially supported (at least they weren't at the time of my question). Conversely there's a whole page on Running a Local Extensible Service Proxy.

I found the project called pyprotobuf (http://pyprotobuf.readthedocs.io) that can generate a module with protorpc classes starting from the proto file.
According to the documentation (http://pyprotobuf.readthedocs.io/topics/languages/protorpc.html) you need to execute:
protopy --format python example.proto

Amazon AWS web framework for Python

I dealt with GAE before and I like simplicity of its webapp. Now I am trying to learn how to work with Amazon EC2. My question - where do I start to make a simple web application that I will be able to access form browser? I suppose I should use WSGI for this purpose. I don't want to use Django as I want to keep the application small and lightweight without unnecessary features. What can you recommend? Thanks.

AWS and GAE are very different. GAE very strictly defines what you can and can't do in terms of development environment. AWS gives you a server to do whatever you want with.
GAE is good when you don't want to have to figure out how all the pieces fit together to scale well. AWS is good when you need flexibility to do whatever you want in your environment.
So to answer your question -- you can use any framework / environment you like.
Personally, I like Django/Pinax for anything requiring a user system. You want a lighter weight system, I've heard good things about Pylons.
Here's a listing of a few others:
http://wiki.python.org/moin/WebFrameworks
Since we're talking AWS, it doesn't have to be python. Ruby on Rails is great.

As already mentioned, you have a lot more flexibility with Amazon than with GAE. If you want to stick with Python and would like to be able to access your app through your browser, you might consider web2py, which enables you to edit and manage your apps through a web-based IDE and admin interface (see demo).
web2py was designed for ease of use and developer productivity, so it's very easy to learn and get going quickly, and you can get a lot done with very little code. Although it's easy to do simple (and even not-so-simple) things quickly and easily, if your app gets more complex, web2py can handle that too, as it is a well-integrated full-stack framework with lots of built-in functionality, including a database abstraction layer, form handling and validation, access control, web services, and easy Ajax integration.
web2py runs on GAE out of the box, and web2py apps are portable from GAE to other platforms without requiring any code changes. However, if you're looking for a simple, scalable cloud hosting option with more flexibility than GAE, you might take a look at the new DotCloud (still in beta), which actually runs on EC2. There's a demo web2py app running there now, and a tutorial explaining the simple deployment process.
If you have any questions about web2py, you'll get lots of help from the friendly and responsive mailing list. I know some of the users have hosted web2py apps on EC2. For example, this demo Q&A site powered by web2py is hosted on EC2.

How can I run a GAE application on a private server?

I want to develop a GAE application using python, but I fear that Google will be the only company able to host the code. Is it possible to run a GAE app on a private server or other host?
(Note that a previous version of the question incorrectly referred to GWT).

Assuming that by GWT you mean GAE (GWT is for Java and anybody can serve it), appscale is probably the best way to host GAE applications anywhere you'd like (including on Amazon EC2 and in your own data center). Anybody can also start a business providing GAE service with AppScale (on Amazon, their own data center, or whatever), which might be attractive for smaller apps (that don't warrant many EC2 or dedicated servers). Anyway, thanks to AppScale and similar efforts, you definitely need not fear "that google will be the only host to host the code".

You're mixing GWT (a Java to JavaScript compiler) with GAE (the Google server API).
GWT can be served by anybody, after compilation it's just a bunch of .js files; a GAE web app can be served only on Google's servers.
The API is public, and the developer's SDK does work and is OSS; but i don't think it would be a desirable platform for a real service provider. OTOH, according to the Google Code GAE SDK project it's the same infrastructure they use; but it's hard to beleive the backends used to run without GoogleFS, BigTable, MapReduce, etc. could be as scalable as theirs...

For Python support, what company would be best to get hosting from?

I want to be able to run WSGI apps but my current hosting restricts it. Does anybody know a company that can accommodate my requirements?

My automatic response would be WebFaction.
I haven't personally hosted with them, but they are primarily Python-oriented (founded by the guy who wrote CherryPy, for example, and as far as I know they were the first to roll out Python 3.0 support).

I am a big fan of Slicehost -- you get root access to a virtual server that takes about 2 minutes to install from stock OS images. The 256m slice, which has been enough for me, is US$20/mo -- it is cheaper than keeping an old box plugged in, and easy to back up. Very easy to recommend.

Plug plug for PythonAnywhere, our own modest offering in this space.
We offer free hosting for basic web apps, with 1-click config for popular frameworks like Django, Flask, Web2py etc. MySql is included, and you also get full suite of browser-based development tools like an editor and a console...

I have been using WebFaction for years and very happy with the service. They are not only python oriented. You should be able to run anything within the limitations of shared hosting (unless of course you have a dedicated server).
They are probably not the cheapest hosting service though. I don't know the prices. But I can still remember very well my previous hosting provider was unreachable for a week (not their servers, I mean the people).

I've been pretty happy with Dreamhost, and of course Google AppEngine.

Google App engine and OpenHosting.com
Have virtual server by OpenHosting, they are ultra fast with support and have very high uptime.

Check out http://pythonplugged.com/
They are trying to collect information on Python hosting providers using variuos technologies (CGI, FCGI, mod_python, mod_wsgi, etc)

I advise you to have a look at http://www.python-cloud.com
This PaaS platform can automatically scale up and down your application regarding your traffic. You can also finely customize if you want vertical, horizontal or both types of scalability. The consequence of this scaling is that you pay as you go : you only pay for your real consumption and not the potential one.
Deployment via git.
Non AWS, hosted in tier-4+ datacenters.
Free trial ;)

I use AWS micro server, 1 year free and after that you can get a 3 year reserved which works out to about $75/yr :) The micro server has only 20MB/sec throughput, ~600MB of ram, and a slower CPU. I run a few Mezzanine sites on mine and it seems fine.

Python Webframework Confusion

Could someone please explain to me how the current python webframworks fit together?
The three I've heard of are CherryPy, TurboGears and Pylons. However I'm confused because TurboGears seems to use CherryPy as the 'Controller' (although isn't CherryPy a framework in in it's own right?), and TurbGears 2 is going to be built on top of Pylons (which I thought did the same thing?).

There are more to it ofcourse.
Here's a comprehensive list and details!
Web Frameworks for Python
Extract from above link:
Popular Full-Stack Frameworks
A web application may use a
combination of a base HTTP application
server, a storage mechanism such as a
database, a template engine, a request
dispatcher, an authentication module
and an AJAX toolkit. These can be
individual components or be provided
together in a high-level framework.
These are the most popular high-level
frameworks. Many of them include
components listed on the WebComponents
page.
Django (1.0 Released 2008-09-03) a
high-level Python Web framework that
encourages rapid development and
clean, pragmatic design
Pylons (0.9.6.2 Released 2008-05-28) a
lightweight Web framework emphasizing
flexibility and rapid development. It
combines the very best ideas from the
worlds of Ruby, Python and Perl,
providing a structured but extremely
flexible Python Web framework. It's
also one of the first projects to
leverage the emerging WSGI standard,
which allows extensive re-use and
flexibility but only if you need it.
Out of the box, Pylons aims to make
Web development fast, flexible and
easy. Pylons is built on top of Paste
(see below).
TurboGears (1.0.4.4 Released
2008-03-07) the rapid Web development
megaframework you've been looking for.
Combines CherryPy, Kid, SQLObject and
MochiKit. After reviewing the website
check out: QuickStart Manual
web2py (currently version 1.43)
Everything in one package with no
dependencies. Development, deployment,
debugging, testing, database
administration and maintenance of
applications can be done via the
provided web interface. web2py has no
configuration files, requires no
installation, can run off a USB drive.
web2py uses Python for the Model, the
Views and the Controllers, has a
built-in ticketing system to manage
errors, an internationalization
engine, works with MySQL, PostgreSQL,
SQLite , Oracle, MSSQL and the Google
App Engine via an ORM abstraction
layer. web2py includes libraries to
handle HTML/XML, RSS, ATOM, CSV, RTF,
JSON, AJAX, XMLRPC, WIKI markup.
Production ready, capable of
upload/download of very large files,
and always backward compatible.
Grok (0.13 Released 2008-06-23) is
built on the existing Zope 3
libraries, but aims to provide an
easier learning curve and a more agile
development experience. It does this
by placing an emphasis on convention
over configuration and DRY (Don't
Repeat Yourself).
Zope (2.10.4 Released 2007-07-04,
3.3.1 Released 2007-01-14, Zope 3.4.0c1 Released 2008-01-31) Being the grandaddy of Python web frameworks,
Zope has grown into a family of
frameworks over the years. Zope 1 was
released in 1999. Zope 2 is both a web
framework and a general purpose
application server, today it is
primarily used by
ContentManagementSystems. Zope 3 is
both a standalone framework and a
collection of related libraries, which
are also included with newer releases
of Zope 2. All of the Zope frameworks
include the ZODB, an object database
for Python.

Give a try to web2py. It is point and click (you do not even need to install Python, comes with it). You develop using a web interface and you can deploy almost everywhere, including iPhone, Oracle and Google App Engine. Before trying web2py, try install some of the other Python frameworks, you will appreciate it even more.

CherryPy is not a full-stack web framework (like Django for example), in fact it isn't a web framework but a HTTP framework. Writing a web application using CherryPy is much like writing a regular object-oriented application in Python.
Also, CherryPy has it's own production-ready WSGI web server, which can be also used for applications written using other frameworks, hence the confusion surrounding CherryPy as a framework.

If you are looking for a start-to-finish solution then it's worth mentioning that the leader of the pack in that space is Django

Have you tried FastAPI.
It's a is a modern, fast (high-performance), web framework for building APIs with Python 3.6+ based on standard Python type hints.
Pretty easy to learn, quite powerful and lightweight.
https://fastapi.tiangolo.com/
I have been using for my application and it seems to be similar to Flask, but quite robust.
from fastapi import FastAPI
app = FastAPI()
#app.get("/")
def read_root():
return {"Hello": "World"}
Some Video Tutorials can be found here
You can also find a lot of help materials from here