For working with data from a database inside of Python programmes, we generally use Object Relational Mappers, to translate database entries into python objects we can work with, with sqlAlchemy and Django Models probably being the most common and advanced ORMs.
Are there ORMs that do not connect to a database but to a third party (JSON) REST API instead? I would like to have a framework which lets me deal with Python objects to perform CRUD operations on the API. This should have all the well-established standard functionalities of an ORM, including Unit of Work and Lazy Loading. Actually, I would want my python code to be agnostic about whether the model is stored in a database or being fetched from a third party API.
It is hard for me to imagine that such a thing does not yet exist. But I am not able to find it. Maybe I am not knowing the right words to search for it?
ORMs Frameworks are frameworks that connect to databases. From your description you are talking about a DAO pattern, not about a Framework. This is a common programming pattern in other languages such as Java.
The right words or searches would be:
Search for the DAO pattern, what to expect from it and how to code it.
Check a couple of links on examples of DAO patterns in python such as this one, or this other one
Analyze your specific problem. You might not need all the code other solutions offer you. And you might be better off coding yourself your own class adjusted to your needs.
Remember KISS and DRY.
PS: Different languages use different paradigms, it is a common error to try to extrapolate patterns and coding uses from one language to another. So something that is solved in e.g. Java in a way, might not be the best option for Python. Keep that in mind too.
Related
I'm new to python and I'm comming from the c# world.
Over there it seemed like the repository pattern was the way to go, but I am having trouble finding any tutorials of how to best do this on Python.
edit I understand that it can be implemented, I'm just wondering if there is any reason why I am finding close to nothing for how to go about doing this.
Thanks!
I wasn't immediately familiar with the "repository pattern", so I looked it up. It appears to be the idea of putting a more general API, like a dictionary-like key/value lookup, in front of a database or other data store. It seems that the idea is to add an additional layer of abstraction that can allow multiple types of data sources (like both a relational database and a CVS file) to be accessed transparently via a common API.
Given this definition, I can think of no reason why this design pattern wouldn't be equally applicable to a problem addressed with Python vs any other programming language.
I am making a database with data in it. That database has two customers: 1) a .NET webserver that makes the data visible to users somehow someway. 2) a python dataminer that creates the data and populates the tables.
I have several options. I can use the .NET Entity Framework to create the database, then reverse engineer it on the python side. I can vice versa that. I can just write raw SQL statements in one or the other systems, or both. What are possible pitfalls of doing this one way or the other? I'm worried, for example, that if I use the python ORM to create the tables, then I'm going to have a hard time in the .NET space...
I love questions like that.
Here is what you have to consider, your web site has to be fast, and the bottleneck of most web sites is a database. The answer to your question would be - make it easy for .NET to work with SQL. That will require little more work with python, like specifying names of the table, maybe row names. I think Django and SQLAlchemy are both good for that.
Another solution could be to have a bridge between database with gathered data and database to display data. On a background you can have a task/job to migrate collected data to your main database. That is also an option and will make your job easier, at least all database-specific and strange code will go to the third component.
I've been working with .NET for quite a long time before I switched to python, and what you should know is that whatever strategy you chose it will be possible to work with data in both languages and ORMs. Do the hardest part of the job in the language your know better. If you are a Python developer - pick python to mess with the right names of tables and rows.
i'm working on a project (written in Django) which has only a few entities, but many rows for each entity.
In my application i have several static "reports", directly written in plain SQL. The users can also search the database via a generic filter form. Since the target audience is really tech-savvy and at some point the filter doesn't fit their needs, i think about creating a query language for my database like YQL or Jira's advanced search.
I found http://sourceforge.net/projects/littletable/ and http://www.quicksort.co.uk/DeeDoc.html, but it seems that they only operate on in-memory objects. Since the database can be too large for holding it in-memory, i would prefer that the query is translated in SQL (or better a Django query) before doing the actual work.
Are there any library or best practices on how to do this?
Writing such a DSL is actually surprisingly easy with PLY, and what ho—there's already an example available for doing just what you want, in Django. You see, Django has this fancy thing called a Q object which make the Django querying side of things fairly easy.
At DjangoCon EU 2012, Matthieu Amiguet gave a session entitled Implementing Domain-specific Languages in Django Applications in which he went through the process, right down to implementing such a DSL as you desire. His slides, which include all you need, are available on his website. The final code (linked to from the last slide, anyway) is available at http://www.matthieuamiguet.ch/media/misc/djangocon2012/resources/compiler.html.
Reinout van Rees also produced some good comments on that session. (He normally does!) These cover a little of the missing context.
You see in there something very similar to YQL and JQL in the examples given:
groups__name="XXX" AND NOT groups__name="YYY"
(modified > 1/4/2011 OR NOT state__name="OK") AND groups__name="XXX"
It can also be tweaked very easily; for example, you might want to use groups.name rather than groups__name (I would). This modification could be made fairly trivially (allow . in the FIELD token, by modifying t_FIELD, and then replacing . with __ before constructing the Q object in p_expression_ID).
So, that satisfies simple querying; it also gives you a good starting point should you wish to make a more complex DSL.
I've faced exactly this problem - a large database which needs searching. I made some static reports and several fancy filters using django (very easy with django) just like you have.
However the power users were clamouring for more. I decided that there already was a DSL that they all knew - SQL. The question was how to make it secure enough.
So I used django permissions to give the power users permission to make SQL queries in a new table. I then made a view for the not-quite-so-power users to use these queries. I made them take optional parameters. The queries were run using Python's lower level DB-API which django is using under the hood for its ORM anyway.
The real trick was opening a read only database connection to run these queries just to make sure that no updates were ever run. I made a read only connection by creating a different user in the database with lower permissions and opening a specific connection for that in the view.
TL;DR - SQL is the way to go!
Depending on the form of your data, the types of queries your users need to use, and the frequency that your data is updated, an alternative to the pure SQL solution suggested by Nick Craig-Wood is to index your data in Solr and then run queries against it.
Solr is an added layer of complexity (configuration, data synchronization) but it is super-fast, can handle large datasets, and provides a (relatively) intuitive query language.
You could write your own SQL-ish language using pyparsing, actually. There is even pretty verbose example you could extend.
Is it even possible to create an abstraction layer that can accommodate relational and non-relational databases? The purpose of this layer is to minimize repetition and allows a web application to use any kind of database by just changing/modifying the code in one place (ie, the abstraction layer). The part that sits on top of the abstraction layer must not need to worry whether the underlying database is relational (SQL) or non-relational (NoSQL) or whatever new kind of database that may come out later in the future.
There's a Summer of Code project going on right now to add non-relational support to Django's ORM. It seems to be going well and chances are good that it will be merged into core in time for Django 1.3.
You could use stock Django and Django-nonrel ( http://www.allbuttonspressed.com/projects/django-nonrel ) together to get a quite unified experience. Some limits apply, read docs carefully though, remembering Spolsky's "All abstractions are leaky".
Yo may also check web2py, they support relational databases and GAE on the core.
Regarding App Engine, all existing attempts limit you in some way (web2py doesn't support transactions or namespaces and probably many other stuff, for example). If you plan to work with GAE, use what GAE provides and forget looking for a SQL-NoSQL holy grail. Existing solutions are inevitably limited and affect performance negatively.
Thank you for all the answers. To summarize the answers, currently only web2py and Django supports this kind of abstraction.
It is not about a SQL-NoSQL holy grail, using abstraction can make the apps more flexible. Lets assume that you started a project using NoSQL, and then later on you need to switch over to SQL. It is desirable that you only make changes to the codes in a few spots instead of all over the place. For some cases, it does not really matter whether you store the data in a relational or non-relational db. For example, storing user profiles, text content for dynamic page, or blog entries.
I know there must be a trade off by using the abstraction, but my question is more about the existing solution or technical insight, instead of the consequences.
I want to develop a desktop application using python with basic crud operation. Is there any library in python that can generate a code for CRUD functionality and user interface given a database table.
Hopefully, this won't be the best option you end up with, but, in the tradition of using web-interfaces for desktop applications, you could always try django. I would particularLY take a look at the inspectdb command, which will generate the ORM code for you.
The advantage is that it won't require that much code to get off the ground, and if you just want to use it from the desktop, you don't need a webserver; you can use the provided test server. The bundled admin site is easy to get off the ground, and flexible up to a point; past which people seem to invest a lot of time battling it (probably a testimony to how helpful it is at first).
There are many disadvantages, not the least of which is the possibility of having to use html/javascript/css when you want to start customizing a lot.
If it were me, I would consider borrowing django's ORM, but then again, I'm already familiar with it.
Having said that, I like working with it, it's usable outside the framework, and it will give you mysql, postgres, or sqlite support. You could also hook up the django admin site to your models and have a web-based editor.
There are surely other ORMs and code generators out there too (I hope some python gurus will point some out, I'm kind of curious).
If you want something really small and simple, I like the Autumn ORM.
If you use the Django ORM, you can use the automatically-generated Django admin interface, which is really nice. It's basically a web-based GUI for browsing and editing records in your database.
If you think you will need advanced SQL features, SQLAlchemy is a good way to go. I suspect for a desktop application, Django or Autumn would be better.
There are other Python ORMs, such as Storm. Do a Google search on "python ORM". See also the discussion on this web site: What are some good Python ORM solutions?