For my little framework Pyxer I would like to to be able to use the Google AppEngine datastores also outside of AppEngine projects, because I'm now used to this ORM pattern and for little quick hacks this is nice. I can not use Google AppEngine for all of my projects because of its's limitations in file size and number of files.
A great alternative would also be, if there was a project that provides an ORM with the same naming as the AppEngine datastore. I also like the GQL approach very much, since this is a nice combination of ORM and SQL patterns.
Any ideas where or how I might find such a solution? Thanks.
Nick Johnson, from the app engine team himself, has a blog posting listing some of the alternatives, including his BDBdatastore.
However, that assumes you want to use exactly the same ORM that you use now in app engine. There are tons of ORM options in general out there, though I am not familiar with the state of the art in Python. This question does seem to address the issue though.
You might also want to look at AppScale, which is "a platform that allows users to deploy and host their own Google App Engine applications".
It's probably overkill for your purposes, but definitely something to look into.
There is also the Remote API which the bulkloader tool uses to upload or download data into/from the Datastore.
Maybe it could be used to have applications which are not hosted on AppEngine to still use the Datastore there.
Related
Before you close the topic, yes it's been asked before, but the last time was early 2010.
Are there any up-to-date efforts to use Django on GAE? Django-norel seems a little dated, along with its effort to get JOINs (and hence Many-to-Many, which I need). I haven't gone too far with Django, so if I'd save myself a headache by changing to a different framework that still has ORM, I'll accept that as a good answer too.
In the long run, I'm trying to run something with the ORM capabilities of Django, and the template capabilities of Django, on Google App Engine, so I'll take whatever solution meets my needs.
(For future readers, I ended up using Flask, and the Google App Engine's inbuilt datastore models.)
Django-nonrel is not dated. Yes, former developers left the project, but it has been picked up and developed on github: https://github.com/django-nonrel
Basically you have 2 choices with running Django on GAE:
1. Use Django-nonrel and Google AppEngine's storage. Your data will be stored in a NoSQL database, which has its limits and advantages, but you can still mostly use Django's ORM classes.
2. Use regular Django and Google Cloud SQL. Cloud SQL is basically MySQL in the cloud: it should be very familiar for most developers
Google Cloud SQL is currently in preview. You can use it for free, but your database size is limited to 10Gb. The FAQ says:
We will not be billing for this service in 2011. We will give you at least 30 days’ advance notice before we begin billing in the future.
The pricing of AppEngine's Datastore is described here
Yes, there's support for Django on Google App Engine currently. Check https://developers.google.com/cloud-sql/docs/django
Today, you need to make a request for a limited preview: https://developers.google.com/cloud-sql/docs/before_you_begin
They took less than a week to answer my request.
There are plenty of well known sites that use GAE
Example of large sites running on Google App Engine
But I wonder if there are examples of python open-source projects built on the GAE that I can simply download and play with.
I find it easier to learn by doing rather than reading throught the docs. Having a working model that I can launch locally would be more useful than all the reference material in the world because it would show me how to structure, configure + some best practices.
If I could have my wish-list, I'd look for an example that:
Actually does something - not just static pages
Makes use of a number of different entities, preferably with some relationships between them
Some templates
Possibly a RESTFUL API
Using the builtin webapp framework (not django etc)
Thanks!
Take a look at Google App Engine samples repository.
I would like to ask about some sources for developing applications with Python and Google App Engine.
For example, some controls to generate automatically pages with the insert/update/delete of a database table, or any other useful resources are welcome.
Thank you!
The Python community tends to look askance at code generation; so, #Hoang, if you think code generation is THE way to go, I suggest you try just about any other language BUT Python.
#Dominic has already suggested some excellent resources, I could point you to more (App Engine Fan, App Engine Utilities, etc, etc) but they're all based on the Pythonic mindset: understand what you need and what you could be doing, wrap as much of it as feasible into reusable components, reuse those components from your own sources.
You want magic, wizards and code generation that basically excused you (in theory) from STUDYING and UNDERSTANDING: give up on Python, it's SO not the language for that,
The google app engine "Getting Started" tutorial is very good. The django documentation is also really detailed.
Take a look at GoogleIO on youtube and watch some of the tutorials.
App Engine Documentation
http://code.google.com/appengine/docs/
App Engine Google Group
http://groups.google.com/group/google-appengine
Google I/O conference videos
http://code.google.com/events/io/
App Engine Cookbook
http://appengine-cookbook.appspot.com/
and, of course, stackoverflow
I'm developing a web application and considering Django, Google App Engine, and several other options. I wondered what kind of "penalty" I will incur if I develop a complete Django application assuming it runs on a dedicated server, and then later want to migrate it to Google App Engine.
I have a basic understanding of Google's data store, so please assume I will choose a column based database for my "stand-alone" Django application rather than a relational database, so that the schema could remain mostly the same and will not be a major factor.
Also, please assume my application does not maintain a huge amount of data, so that migration of tens of gigabytes is not required. I'm mainly interested in the effects on the code and software architecture.
Thanks
Most (all?) of Django is available in GAE, so your main task is to avoid basing your designs around a reliance on anything from Django or the Python standard libraries which is not available on GAE.
You've identified the glaring difference, which is the database, so I'll assume you're on top of that. Another difference is the tie-in to Google Accounts and hence that if you want, you can do a fair amount of access control through the app.yaml file rather than in code. You don't have to use any of that, though, so if you don't envisage switching to Google Accounts when you switch to GAE, no problem.
I think the differences in the standard libraries can mostly be deduced from the fact that GAE has no I/O and no C-accelerated libraries unless explicitly stated, and my experience so far is that things I've expected to be there, have been there. I don't know Django and haven't used it on GAE (apart from templates), so I can't comment on that.
Personally I probably wouldn't target LAMP (where P = Django) with the intention of migrating to GAE later. I'd develop for both together, and try to ensure if possible that the differences are kept to the very top (configuration) and the very bottom (data model). The GAE version doesn't necessarily have to be perfect, as long as you know how to make it perfect should you need it.
It's not guaranteed that this is faster than writing and then porting, but my guess is it normally will be. The easiest way to spot any differences is to run the code, rather than relying on not missing anything in the GAE docs, so you'll likely save some mistakes that need to be unpicked. The Python SDK is a fairly good approximation to the real App Engine, so all or most of your tests can be run locally most of the time.
Of course if you eventually decide not to port then you've done unnecessary work, so you have to think about the probability of that happening, and whether you'd consider the GAE development to be a waste of your time if it's not needed.
Basically, you will change the data model base class and some APIs if you use them (PIL, urllib2, etc).
If your goal is app-engine, I would use the app engine helper http://code.google.com/appengine/articles/appengine_helper_for_django.html. It can run it on your server with a file based DB and then push it to app-engine with no changes.
It sounds like you have awareness of the major limitation in building/migrating your app -- that AppEngine doesn't support Django's ORM.
Keep in mind that this doesn't just affect the code you write yourself -- it also limits your ability to use a lot of existing Django code. That includes other applications (such as the built-in admin and auth apps) and ORM-based features such as generic views.
There are a few things that you can't do on the App Engine that you can do on your own server like uploading of files. On the App Engine you kinda have to upload it and store the datastore which can cause a few problems.
Other than that it should be fine from the Presentation part. There are a number of other little things that are better on your own dedicated server but I think eventually a lot of those things will be in the App Engine
I am thinking about using Google App Engine.It is going to be a huge website. In that case, what is your piece of advice using Google App Engine. I heard GAE has restrictions like we cannot store images or files more than 1MB limit(they are going to change this from what I read in the GAE roadmap),query is limited to 1000 results, and I am also going to se web2py with GAE. So I would like to know your comments.
Thanks
Having developed a smallish site with GAE, I have some thoughts
If you mean "huge" like "the next YouTube", then GAE might be a great fit, because of the previously mentioned scaling.
If you mean "huge" like "massively complex, with a whole slew of screens, models, and features", then GAE might not be a good fit. Things like unit testing are hard on GAE, and there's not a built-in structure for your app that you'd get with something like (famously) (Ruby on) Rails, or (Python powered) Turbogears.
ie: there is no staging environment: just your development copy of the system and production. This may or may not be a bad thing, depending on your situation.
Additionally, it depends on the other Python modules you intend to pull in: some Python modules just don't run on GAE (because you can't talk to hardware, or because there are just too many files in the package).
Hope this helps
using web2py on Google App Engine is a great strategy. It lets you get up and running fast, and if you do outgrow the restrictions of GAE then you can move your web2py application elsewhere.
However, keeping this portability means you should stay away from the advanced parts of GAE (Task Queues, Transactions, ListProperty, etc).
The AppEngine uses BigTable as it's datastore backend. Don't try to write a traditional relational-database driven application. BigTable is much more well suited for use as a highly-scalable key-value store. Avoid joins if at all possible.
I wouldn't worry about any of this. After having played with Google App Engine for a while now, I've found that it scales quite well for large data sets. If your data elements are large (i.e. photos), then you'll need to integrate with another service to handle them, but that's probably going to be true no matter what with data of that size. Also, I've found BigTable relatively easy to work with having come from a background entirely in relational databases. Finally, Django is a somewhat hidden, but awesome, "feature" of Google App Engine. If you've never used it, it's a really nice, elegant web framework that makes a lot of common tasks trivial (forms come to mind here).
Google has just released version 1.3.0 of the SDK with support with a new Blobstore API for storage of files up to 50MB. See the post "App Engine SDK 1.3.0 Released Including Support for Larger User Uploads".
What about Google Wave? It's being built on appengine, and once live, real-time translatable chat reaches the corporate sector... I could see it hitting top 1000th... But then again, that's an internal project that gets to do special stuff other appengine apps can't.... Like hanging threads; I think... And whatever else Wave has under the hood...
If you are planning on a 'huge' website, then don't use App Engine. Simple as that. The App Engine is not built to deliver the next top 1000th website.
Allow me to also ask what do you mean by 'huge', how many simultaneous users? Queries per second? DB load?