I'd like to use the Python version of App Engine but rather than write my code specifically for the Google Data Store, I'd like to create my models with a generic Python ORM that could be attached to Big Table, or, if I prefer, a regular database at some later time. Is there any Python ORM such as SQLAlchemy that would allow this?
Technically this wouldn't be called an ORM (Object Relational Mapper) but a DAL (Database Abstraction Layer). The ORM part is not really interesting for AppEngine as the API already takes care of the object mapping and does some simple relational mapping (see RelationProperty).
Also realize that a DAL will never let you switch between AppEngine's datastore and a "normal" sql database like mysql because they work very differently. It might let you switch between different key value stores, like reddis, mongo or tokyo cabinet. But as they all have such very different characteristics I would really think twice before using one.
Lastly, the DAL traditionally sits on top of the DB interface, but with AppEngine's api you can implement your own "stubs" that basically let you use other storage backends on their api. The people at Mongo wrote one for MongoDB which is very nice. And the dev_appserver comes with a filesystem-based one.
And now to the answer: yes there is one! It's part of web.py. I haven't really tried if for the reasons above, so I can't really say if it's good.
PS. I know Ruby has a nice DAL project for keyvalue stores in the works too, but I can't find it now... Maybe nice to port to Python at some point.
Nowadays they do since Google has launched Cloud SQL
Related
I was curious to know if, how Spring boot has JPA repository to communicate with databases, Python also has anything similar?
If not, what would be a good way to structure my app such that a service layer would community with a dao layer? An example/demonstration would be great!
You can use python framework such as Django, Flask etc.
ORM provided by these frameworks are quite good and easier to write complex SQL queries.
For instance,
If I have to execute this SQL query to filter all users with name has "stack" keyword, then SQL query & django query will be like -
SELECT * from users where name LIKE '%stack%'; // SQL
Users.objects.filter(name__contains="stack") // Django
The best part about these ORMs is that they allow all of the functionality provided by SQL queries and JPA repository.
You can refer to Django/Flask docs to more info.
We have inherited a DynamoDB database and have been told that a custom script is required if we want to edit/delete multiple items. ie you can't just run an easy mysql query to do this like in a relational database.
What is the most common way this is done? e.g. python script, lambda, etc.
thanks
You can connect to DynamoDB programmatically through a variety of SDKs (Javascript, .Net, Java, Go, C++, Ruby, Python, etc),in the AWS web UI console, and through the AWS command line interface (among others). I don't think it's terribly different than other databases in that regard.
If you are just getting started, I'd start by writing a script in your preferred programming languages with one of the many SDKs available. The NoSQL Workbench (an app available from Amazon directly) has a useful Operation Builder that will not only help with query syntax, but will even create a script to execute the operation in Java, Javascript and Python.
The DynamoDB API references the operations you can perform on DynamoDB (CRUD operations, transactions, etc). Each API method has a section at the bottom of the page with links to examples of that method being called in each of the supported SDKs. To mutate multiple items at once, you may want to check out the Batch operations (BatchWriteItem, BatchGetItem).
I'm not entirely sure I understand the advice you were given. DynamoDB is a noSQL database, so you cannot access it via SQL queries. NoSQL databases can come with a steep learning curve. I wouldn't say it's any harder than working with SQL databases, just different.
Familiarize yourself with DynamoDB data modeling, it is very different than data modeling in SQL databases. When learning DynamoDB, try hard to forget everything you know about working with SQL databases. Trying to find the SQL equivalent for something in DynamoDB consistently gives people a hard time.
I would highly recommend The Dynamo Guide to start the learning process. Again, it's not hard, just different than SQL databases. The AWS docs can be a bit terse, so I found resources like The DynamoDB Guide to be a lifesaver!
I'm comfortable with sqlite but when I use two different python program to access the same database. It throws an error like table is locked.
What are the different portable database to use with python?
SQLite is great but is designed as a small, fast, single user database. it isn't designed for the use you describe.
You can just pretty much any database with Python. The Python database API provides a straightforward way to interface with most relational databases, including SQLite.
However, to interact with a database in a more natural Python style, I've enjoyed using SQLAlchemy. It took a bit to work through the tutorial but it's great.
My personal preferred database is Postgres, but there are many other choices.
This is what I have :-
Ubuntu 11.10.
Django 1.3
Python 2.7
What I want to do is build an app that is similar to top-coder and I have the skeletal version of the app sketched out. The basic requirements would be:-
1. Saving the code.
2. Saving the user name and ranks.(User-profile)
3. Should allow a teacher to create multiple choice questions too.( Similar to Google docs).
I have basic knowledge of Django and have built couple of (basic) apps before. Rather than building an online tool, is it possible to build something very similar to conf2py that sits on top of web2py, in Django.
Lets call this small project examPy( I know, very original), is it possible to build an app that acts more a plug-in to Django or is my concept of Django absolutely wrong?
The primary question being:
As I want to learn a new DB and have worked on postgres in Django, should I chose CouchDB or MongoDB for Django?
Answers can be explanations or links to certain documentations or blogs that can tell me the pros and cons.
General Differences
Comparing MongoDB and CouchDB
MongoDB VS CouchDB
Battle of the NoSQL Stars
Django Specific
MongoDB for Django-nonrel
Django-MongoDB Engine
MongoDB Hearts Django?
An Introduction to Using CouchDB with Django
Using CouchDB with Django
All my research points me towards the idea that Mongo and Couch are similar enough that your choice would probably boil down to personal (subjective) preference even over the use-case. Personally, I've developed a CouchDB fetish and am looking for a reason to use it.
The key factor influencing your decision should probably be which noSQL solution has the most mature ORM framework for Django?
I've used mongo-engine with Django but you need to create a file specifically for Mongo documents eg. Mongo_models.py. In that file you define your Mongo documents. You then create forms to match each Mongo document. Each form has a save method which inserts or updates whats stored in Mongo. Django forms are designed to plug into any data back end ( with a bit of craft ).
If you go this route you can dodge Django non-rel which is still not part of Django 1.4. In addition I believe django-nonrel is on hiatus right now.
I've used both CouchDB and Mongo extensively. CouchDB has a lovely interface. My colleague is working on something similar for Mongo. Mongo's map and reduce are far faster than CouchDB. Mongo is more responsive loading and retrieving data. The python libraries for Mongo are easier to get working with ( both pymongo and mongo-engine are excellent )
Be sure you read the Mongo production recommendations! Do not run one instance on the same node as Django or prepare to be savagely burned when traffic peaks. Mondo works great with Memcache/Redis where one can store reduced data for rapid lookups.
BEWARE: If you have very well defined and structured data that can be described in documents or models then don't use Mongo. Its not designed for that and something like PostGreSQL will work much better.
I use PostGreSQL for relational or well structured data because its good for that. Small memory footprint and good response.
I use Redis to cache or operate in memory queues/lists because its very good for that. great performance providing you have the memory to cope with it.
I use Mongo to store large JSON documents and to perform Map and reduce on them ( if needed ) because its very good for that. Be sure to use indexing on certain columns if you can to speed up lookups.
Don't use a circle to fill a square hole. It won't fill it.
I've used CouchDB with Django for a production application. Couch is fine and has some great ideas, but I'm moving that app over to MongoDB. Why? There is support for Mongo in the Django community. Django-nonrel has a MongoDB backend. Using Django-toolbox I can embed models in models and have some basic admin support.
If I remember correctly, Django-nonrel will eventually be rolled into Django core. In five years time I see much more support for Mongo in Django than Couch. Of course that can change, but I see Mongo a better fit.
which is the best back end for python applications and what is the advantage of using sqlite ,how it can be connected to python applications
What do you mean with back end? Python apps connect to SQLite just like any other database, you just have to import the correct module and check how to use it.
The advantages of using SQLite are:
You don't need to setup a database server, it's just a file
No configurations needed
Cross platform
Mainly, desktops applications are the ones that take real advantage of this. For web apps, SQLite is not recommended, since the file containing the data, is easily readable (lacks any kind of encryption), and when the web server lacks special configuration, the file is downloadable by anyone.
Django, Twisted, and CherryPy are popular Python "Back-Ends" as far as web applications go, with Twisted likely being the most flexible as far as networking is concerned.
SQLite can, as has been previously posted, be directly interfaced with using SQL commands as it has native bindings for Python, or it can be accessed with an Object Relational Manager such as SQLObject (another Python library).
As far as performance is concered, SQLite is fairly scalable and should be able to handle most use cases that don't require a seperate database server (nothing enterprise level). An additional benefit of SQLite is that the database is self-contained in a single file allowing for easy backup while remained a common enough format that multiple applications can access the data. A word of advice on using SQLite with Python, however, is that you may run into issues with threading (in the past most of the bindings for SQLite were not thread-safe, although this may have changed over time).
The language you are using at the application layer has little to do with your database choice underneath. You need to examine the advantages of other DB packages to get an idea of what you want.
Here are some popular database packages for cheap or free:
ms sql server express, pg/sql, mysql
If you mean "what is the best database?" then there's simply no way to answer this question. If you just want a small database that won't be used by more than a handful of people at a time, SQLite is what you're looking for. If you're running a database for a giant corporation serving thousands, you're probably looking for Oracle. In between those, you have MySQL, PostgreSQL, SQL Server, db2, and probably more.
If you're familiar with one of those, that may be the best to go with from a practical standpoint. If you're doing a typical webapp, my advice would be to go with MySQL or PostgreSQL as they're free and well supported by just about any ORM you could think of (my personal preference is towards PostgreSQL, but I'm not experienced enough with either of these to make a good argument one way or another). If you do go with one of those two, my recommendation is to use storm as the ORM.
(And yes, there are free versions of SQL Server and Oracle. You won't have as many choices as far as ORMs go though)