I want to develop a GAE application using python, but I fear that Google will be the only company able to host the code. Is it possible to run a GAE app on a private server or other host?
(Note that a previous version of the question incorrectly referred to GWT).
Assuming that by GWT you mean GAE (GWT is for Java and anybody can serve it), appscale is probably the best way to host GAE applications anywhere you'd like (including on Amazon EC2 and in your own data center). Anybody can also start a business providing GAE service with AppScale (on Amazon, their own data center, or whatever), which might be attractive for smaller apps (that don't warrant many EC2 or dedicated servers). Anyway, thanks to AppScale and similar efforts, you definitely need not fear "that google will be the only host to host the code".
You're mixing GWT (a Java to JavaScript compiler) with GAE (the Google server API).
GWT can be served by anybody, after compilation it's just a bunch of .js files; a GAE web app can be served only on Google's servers.
The API is public, and the developer's SDK does work and is OSS; but i don't think it would be a desirable platform for a real service provider. OTOH, according to the Google Code GAE SDK project it's the same infrastructure they use; but it's hard to beleive the backends used to run without GoogleFS, BigTable, MapReduce, etc. could be as scalable as theirs...
Related
I'm getting my feet wet with GCP and GAE, also nodejs and python and networking (I know).
[+] What I have:
Basically I have some nodejs code that takes in some input and is supposed to then send that input to some python code that will do more stuff to it. My first idea was to deploy the nodejs code via GAE, then host the python code in a python server, then make post requests from the nodejs front-end to the python server backend.
[+] What I would like to be able to do:
just deploy both my nodejs code and my python code in the same project and instance of GAE so that the nodejs is the frontend that people see but so that the python server is also running in the same environment and can just communicate with the nodejs without sending anything online.
[+] What I have read
https://www.netguru.co/blog/use-node-js-backend
Google App Engine - Front and Backend Web Development
and countless other google searches for this type of setup but to no avail.
If anyone can point me in the right direction I would really appreciate it.
You can't have both python and nodejs running in the same instance, but they can run as separate services, each with their own instance(s) inside the same GAE app/project. See Service isolation and maybe Deploying different languages services to the same Application [Google App Engine]
Using post requests can work pretty well, but will likely take some effort to ensure no outside access.
Since you intend to use as frontend the nodejs service you're limited to using only the flexible environment for it, which limits the inter-service communication options - you can't use push queues (properly supported only in the standard environment) which IMHO would be a better/more secure solution than post requests.
Another secure communication option would be for the nodejs service to place the data into the datastore and have the python service pick it up from there - the datastore is shared by all instances/versions/services inside the same GAE app. Also more loosely coupled IMHO - each service can function (at least for a while) without the other being alive (not possible if using the post requests).
Maybe of interest: How to tell if a Google App Engine documentation page applies to the standard or the flexible environment
UPDATE:
Node.JS is currently available in the standard environment as well, so you can use those features, see:
Now, you can deploy your Node.js app to App Engine standard environment
Google App Engine Node.js Standard Environment Documentation
In a Python script, mylibrary.py, I use Protocol Buffers to model data using the following approach:
Defining message formats in a .proto file.
Use the protocol buffer compiler.
Use the Python protocol buffer API to write and read messages in the .py module.
I want to implement Cloud Endpoints Framework on App Engine that imports and uses the aforementioned Python script, however Cloud Endpoints uses ProtoRPC, not 'standard' Protocol Buffers.
My App Engine Python module, main.py, imports from protorpc rather than using the 'offline' protoc compiler to generate serialization and deserialization code:
from protorpc import messages
from protorpc import remote
Messages are not defined using .proto files. Instead, classes are defined, inheriting from protorpc.messages.Message:
class MyMessageDefinition(messages.Message)
Can Proto Buffers be converted to Proto RPC equivalents? I don't really want to change mylibrary.py to use ProtoRPC, since it's less generic than Protocol Buffers.
After eight months and lots experimentation, I'll add my opinion. I hope it saves someone time.
Choose Your Framework First
There are different Cloud Endpoint offerings from Google Cloud. All can be used for JSON/REST APIs. This wasn't immediately clear to me. Cloud Endpoints is a very high-level phrase covering development, deployment and management of APIs on multiple Google Cloud backends.
The point here is that after deciding to use Cloud Endpoints, you must still decide on backend technologies to serve your API. The documentation feels a little hidden away, but I strongly recommend starting with the Google Cloud Endpoints doc.
You can choose between:
OpenAPI Specification
Endpoints Frameworks
gRPC
Choose Your Implementation Second
Within each API Framework there’s a choice of Cloud implementations upon which your API (service) can run:
OpenAPI Specification
- for JSON/REST APIs implemented on:
Google App Engine flexible environment
Google Compute Engine
Google Container Engine
Kubernetes
Endpoints Frameworks
- for JSON/REST APIs implemented on:
Google App Engine standard environment with Java
Google App Engine standard environment with Python
gRPC
- for gRPC APIs implemented on:
Google Compute Engine
Google Container Engine
Kubernetes
When posting question here, I was using Endpoints Frameworks running on Google App Engine standard environment with Python. I then migrated my API (service) to gRPC on Google Compute Engine.
The observant among you may notice both the OpenAPI Specification and Endpoints Frameworks can be used for JSON/REST APIs, while gRPC only exposes a gRPC API. So how did I port my REST API from Endpoints Frameworks to gRPC? The answer is Transcoding HTTP/JSON to gRPC (which I learnt along the way, and was not immediately clear to me). So, don't rule out gRPC just because you want REST/HTTP.
The Answer
So how does this related to my original question?
That I was trying to convert between .proto files and gRPC annotations at all, meant I had taken a wrong-turning along the way.
If you want to write an application using plain .proto files, then choose gRPC on Compute Engine. If you need this to be a REST API, this can be done, but you'll need to add an ESP into your backend configuration. It's pretty much an NGINX sever setup as a reverse proxy. The only downside here is you'll need some Docker knowledge to ensure the ESP (proxy) and your gRPC server can communicate (Docker networking).
If your code is already on an App Engine, and you want to expose it as a REST API with minimum effort and still get good API management features, choose Endpoints Frameworks. Warning: I moved away from this because it was prohibitively expensive (I was getting billed in the region of $100 USD monthly).
If you want to avoid .protos altogether, then go with OpenAPI Specification.
Lastly, if you want to offer programmatic integration, client libraries, or you want to offer a microservice, then really do consider gRPC. It's easy to remove the ESP (proxy) and run a gRPC server on nearly any machine (as long as the Protocol Buffer Runtime is installed.
Ultimately I settled on gRPC on Compute Engine with Docker. I also have an ESP to provide a HTTP transcoding to gRPC and vice-versa. I like this approach for a few reasons:
You learn a lot: Docker, Docker Networking, NGINX configuration, Protocol Buffers, ESP (Cloud Proxy), gRPC servers.
The service (core business) logic can be written with plain-old gRPC. This allows the service to be run on any machine without a web server. Your business logic, is the server :)
Protocol Buffers / gRPC are excellent for isolating business logic as a service...or microservice. They're also good for providing well-defined interfaces and libraries.
Avoid These Mistakes
Implementing the first framework / architecture you find. If I could start again, I would not choose Endpoints Frameworks. It's expensive, and uses annotations rather than .proto files, which, IMO, makes the code harder to port.
Read Always Free Usage Limits before deciding upon a framework and implementation. Endpoints Frameworks uses backend App Engine instances - which have almost no free quota. Confusing, frontend App Engine instances have a very generous free quota.
Consider Local Development. Cloud Endpoints local development servers are not officially supported (at least they weren't at the time of my question). Conversely there's a whole page on Running a Local Extensible Service Proxy.
I found the project called pyprotobuf (http://pyprotobuf.readthedocs.io) that can generate a module with protorpc classes starting from the proto file.
According to the documentation (http://pyprotobuf.readthedocs.io/topics/languages/protorpc.html) you need to execute:
protopy --format python example.proto
I'm developing an aplication built on the python Bottle framework. It's amazing and i want to host it somewhere, but of course i need a service where i can run the server.
What are the best options available (on costs too)?
Thanks!
I work for PythonAnywhere, a Python hosting company, so obviously I think we have a great Bottle hosting solution -- check it out :-)
Some of our competitors -- ones where we feel proud if a customer leaves them to move to us, or where we don't feel too depressed if they leave us to go to them, are Webfaction, Heroku, and (if you don't mind doing OS maintenance yourself and setting up your own front-end webserver) Digital Ocean and Amazon Web Services.
There are definitely other good hosts, but those are the ones that spring to mind.
The easiest way to host is use Ngrok ,its an open source tool where use get a temporary link where you can access your local host server on any machine through that link .you can download it from here https://ngrok.com/
I've done a lot of research for Google App engine. I ended up with webapp2. Now we realized that AWS has many advantages over GAE for our project.
I feel a little bit overwhelmed because there is so much information about AWS.
I ended up with the nosql dynamoDB solution.
Now I really love lightweight frameworks like webapp2 , its really simple and easy to use. It also don't hide post and get from you.
Now does it makes sense to use webapp2 for AWS ? Mabye there other frameworks that would fit better for our project.
(We want to go in a filehosting direction)
I mean there are literally 50 frameworks for python. And I didn't find a chart with pro/cons.
The last frameworks that I investigated are : web2py , pylon(now pyramids?) and tornado.
To be honest I am really confused.
So if I want to have a lightweight framework for AWS would you recommend me to stay with webapp2?
Resources:
Amazon AWS web framework for Python
Edit #1
Now it's a decision between tornado and webapp2. considering that I want to use dynamoDB.
Tornado is a fast non-blocking web server itself. It's easy to write simple apps in a few minutes, but it has nothing AFAIK like templates, views, etcetera. If you want to serve files it would be very quick to connect a tornado server together with boto (https://github.com/boto/boto) and use dynamoDB or S3.
On the other hand, pylons (now pyrmaid) and we2py are full web application frameworks with no web server. In fact, they come with a lightweight development server (at least pylons), but if you want to put up a site in AWS you should use something like nginx (http://nginx.org/), apache or some other wsgi server (http://wsgi.readthedocs.org/).
If you want to go for a lightweight framework and use python, I'd go for web2py, easy to configure and build apps. (A curious note: it used to be a single .py file) you can try it online: http://www.web2py.com/demo_admin/default/site
Regarding webapp2 I've never used it, but I've heard it's similar to Appengine web framework. So if you're comfortable with it stay there.
Either direction, boto is the python interface for AWS so if you choose python, you'll have to check it out. It is actively mantained.
I wrote an application which makes use of the websocket implementation of Tornado and I am trying to find a host for it. As far as I can tell by reading google search results, google appengine does not support websockets at the moment. I'm not sure about heroku since I couldnt find any information. So my question is, if anybody knows a hoster where I could host my application?
The easiest thing might be to use a general IaaS (Infrastructure as a Service) cloud provider such as Amazon EC2 or Rackspace. Or even just use a regular hosted server like with Dreamhost. With those you get direct access to the OS and network configuration and you can do whatever you want. The downside is those will be more expensive and you for EC2 and Rackspace you will need to manage the host itself.
Update: you can now use Websockets on Heroku. Here is some information about using Websockets with Python on Heroku.