I'm working on a new project written in Python with Flask & SQLAlchemy. The latter seems like a decent ORM but doesn't seem to consider caching as its responsibility.
It provides some examples on how to leverage dogpile.cache # http://docs.sqlalchemy.org/en/latest/orm/examples.html#module-examples.dogpile_caching but this seems to offer no more than basic query results caching which is really only valuable for read-only models.
What I am looking for is a real Level 2 Object Cache integrated into the ORM which store each object via its PK and ensures identity in a concurrently shared cache. I have experience with multiple caching libraries working with Java ORMs but I'm not able to find anything for SQLAlchemy. Does no such thing exist for Python yet?
Related
We have inherited a DynamoDB database and have been told that a custom script is required if we want to edit/delete multiple items. ie you can't just run an easy mysql query to do this like in a relational database.
What is the most common way this is done? e.g. python script, lambda, etc.
thanks
You can connect to DynamoDB programmatically through a variety of SDKs (Javascript, .Net, Java, Go, C++, Ruby, Python, etc),in the AWS web UI console, and through the AWS command line interface (among others). I don't think it's terribly different than other databases in that regard.
If you are just getting started, I'd start by writing a script in your preferred programming languages with one of the many SDKs available. The NoSQL Workbench (an app available from Amazon directly) has a useful Operation Builder that will not only help with query syntax, but will even create a script to execute the operation in Java, Javascript and Python.
The DynamoDB API references the operations you can perform on DynamoDB (CRUD operations, transactions, etc). Each API method has a section at the bottom of the page with links to examples of that method being called in each of the supported SDKs. To mutate multiple items at once, you may want to check out the Batch operations (BatchWriteItem, BatchGetItem).
I'm not entirely sure I understand the advice you were given. DynamoDB is a noSQL database, so you cannot access it via SQL queries. NoSQL databases can come with a steep learning curve. I wouldn't say it's any harder than working with SQL databases, just different.
Familiarize yourself with DynamoDB data modeling, it is very different than data modeling in SQL databases. When learning DynamoDB, try hard to forget everything you know about working with SQL databases. Trying to find the SQL equivalent for something in DynamoDB consistently gives people a hard time.
I would highly recommend The Dynamo Guide to start the learning process. Again, it's not hard, just different than SQL databases. The AWS docs can be a bit terse, so I found resources like The DynamoDB Guide to be a lifesaver!
At my work, we are using different SQL and NOSQL database types (i.e postgres, mysql, sqlite, mongodb, arangodb).
As we are creating lots of packages communicating with a database to retrieve some data. Recreating the CRUD functionality (+ methods to create schema => create table/collection/db) for each package make it harder to maintain the whole pipeline while scaling up.
In order to solve this situation more easily, we decided to create a database plugins API that allows us to abstract CRUD methods so you don't care about the backend database being used anymore on the backend side. All packages that need to communicate with a database will just wrap the plugins CRUD methods and add their own package specific behaviour.
I'm pretty sure we are not the only having this kind of situation out there, so I 've been looking on the web to try to find if such a thing has already been done without finding anything interesting.
Could anyone refer me to anything that may do/try to do it?
Is there anything that already does it using Python?
Thanks
I am writing a micro-service that will have to share database owned by a different micro-service.
I understand that from a micro-services architecture perspective, this is not a good design. Hence, I decided to separate out the database access as another micro-service, whose only task it to manage access to db.
I need pointers on how I may write such an app using Python which exposes API for read/write to a database?
I realize this is not a design perspective answer.
Did you have a chance to take a look at sandman, a Python library that can generate a REST API over a database?
This is what I have :-
Ubuntu 11.10.
Django 1.3
Python 2.7
What I want to do is build an app that is similar to top-coder and I have the skeletal version of the app sketched out. The basic requirements would be:-
1. Saving the code.
2. Saving the user name and ranks.(User-profile)
3. Should allow a teacher to create multiple choice questions too.( Similar to Google docs).
I have basic knowledge of Django and have built couple of (basic) apps before. Rather than building an online tool, is it possible to build something very similar to conf2py that sits on top of web2py, in Django.
Lets call this small project examPy( I know, very original), is it possible to build an app that acts more a plug-in to Django or is my concept of Django absolutely wrong?
The primary question being:
As I want to learn a new DB and have worked on postgres in Django, should I chose CouchDB or MongoDB for Django?
Answers can be explanations or links to certain documentations or blogs that can tell me the pros and cons.
General Differences
Comparing MongoDB and CouchDB
MongoDB VS CouchDB
Battle of the NoSQL Stars
Django Specific
MongoDB for Django-nonrel
Django-MongoDB Engine
MongoDB Hearts Django?
An Introduction to Using CouchDB with Django
Using CouchDB with Django
All my research points me towards the idea that Mongo and Couch are similar enough that your choice would probably boil down to personal (subjective) preference even over the use-case. Personally, I've developed a CouchDB fetish and am looking for a reason to use it.
The key factor influencing your decision should probably be which noSQL solution has the most mature ORM framework for Django?
I've used mongo-engine with Django but you need to create a file specifically for Mongo documents eg. Mongo_models.py. In that file you define your Mongo documents. You then create forms to match each Mongo document. Each form has a save method which inserts or updates whats stored in Mongo. Django forms are designed to plug into any data back end ( with a bit of craft ).
If you go this route you can dodge Django non-rel which is still not part of Django 1.4. In addition I believe django-nonrel is on hiatus right now.
I've used both CouchDB and Mongo extensively. CouchDB has a lovely interface. My colleague is working on something similar for Mongo. Mongo's map and reduce are far faster than CouchDB. Mongo is more responsive loading and retrieving data. The python libraries for Mongo are easier to get working with ( both pymongo and mongo-engine are excellent )
Be sure you read the Mongo production recommendations! Do not run one instance on the same node as Django or prepare to be savagely burned when traffic peaks. Mondo works great with Memcache/Redis where one can store reduced data for rapid lookups.
BEWARE: If you have very well defined and structured data that can be described in documents or models then don't use Mongo. Its not designed for that and something like PostGreSQL will work much better.
I use PostGreSQL for relational or well structured data because its good for that. Small memory footprint and good response.
I use Redis to cache or operate in memory queues/lists because its very good for that. great performance providing you have the memory to cope with it.
I use Mongo to store large JSON documents and to perform Map and reduce on them ( if needed ) because its very good for that. Be sure to use indexing on certain columns if you can to speed up lookups.
Don't use a circle to fill a square hole. It won't fill it.
I've used CouchDB with Django for a production application. Couch is fine and has some great ideas, but I'm moving that app over to MongoDB. Why? There is support for Mongo in the Django community. Django-nonrel has a MongoDB backend. Using Django-toolbox I can embed models in models and have some basic admin support.
If I remember correctly, Django-nonrel will eventually be rolled into Django core. In five years time I see much more support for Mongo in Django than Couch. Of course that can change, but I see Mongo a better fit.
I'd like to use the Python version of App Engine but rather than write my code specifically for the Google Data Store, I'd like to create my models with a generic Python ORM that could be attached to Big Table, or, if I prefer, a regular database at some later time. Is there any Python ORM such as SQLAlchemy that would allow this?
Technically this wouldn't be called an ORM (Object Relational Mapper) but a DAL (Database Abstraction Layer). The ORM part is not really interesting for AppEngine as the API already takes care of the object mapping and does some simple relational mapping (see RelationProperty).
Also realize that a DAL will never let you switch between AppEngine's datastore and a "normal" sql database like mysql because they work very differently. It might let you switch between different key value stores, like reddis, mongo or tokyo cabinet. But as they all have such very different characteristics I would really think twice before using one.
Lastly, the DAL traditionally sits on top of the DB interface, but with AppEngine's api you can implement your own "stubs" that basically let you use other storage backends on their api. The people at Mongo wrote one for MongoDB which is very nice. And the dev_appserver comes with a filesystem-based one.
And now to the answer: yes there is one! It's part of web.py. I haven't really tried if for the reasons above, so I can't really say if it's good.
PS. I know Ruby has a nice DAL project for keyvalue stores in the works too, but I can't find it now... Maybe nice to port to Python at some point.
Nowadays they do since Google has launched Cloud SQL