The following query:
SELECT L1.task_id FROM task_log L1
LEFT JOIN task_log L2 ON L1.started BETWEEN L2.started AND L2.ended
WHERE L2.task_id IS NULL;
Simply reads all task logs that were started when no other tasks were running. An abstraction in PHP/ZF2 looks like this:
$db->select(array('L1' => 'task_log'))
->columns('task_id')
->join(array('L2' => 'task_log'), "L1.started BETWEEN L2.started AND L2.ended", array(), 'left');
I was looking for something similar in Django, or at least python. As per this discussion, it is (understandably) out of question to build this in django models. Are there any other alternatives in python/django?
There seem to be these options:
(A) RawSQL with string templates and maybe some generic model field reflection code to make it a bit more generic.
(B) Django's Expression API which might allow you to re-use some of the Django ORM stuff and only extend it in order to make SELF JOINs work. Though I'm wondering why there wouldn't be some Django module already. Maybe it's too rare of a use case.
Looking up i found this: http://docs.sqlalchemy.org/en/latest/core/tutorial.html which is exactly what i needed.
Related
I'm building a project using flask and flask-SQLAlchemy (with sqlite3 and with python 3.8). I have models.py which holds all the tables of the db, and I want to have static method which deletes rows in Articles table depends on one of their attribute.
I thought about writing this funciton in the models class and wanted to ask if that's fine.
The function looks like:
def update_articles():
all_articles = Articles.query.all()
for article in all_articles:
if needs_to_be_deleted(article):
db.session.delete(article)
db.session.commit()
Is that funciton fine (don't mind the needs_to_be_deleted(article) thing). Is the way I delete the article good? and is the place for this function can be at the models.py file?
As per my experience its all good here. As a general rule you should design your application as:
Models should do the most powerful stuff
Views should carry out only logical work
Templates should be dumb. They should just render things.
Now by saying the above i mean to say all the stuff relating to the operations with database should be in the models. Just like you are doing. Most of the time all the transactions with your database are simple ones and can be written with a couple of lines with ORM. But sometimes if you feel that something needs to be done over and over again or you feel like having a method for doing something big. Then its better to have that in your models only. The method you mentioned should be a part of your Article model. Other things that can be there in your model could be operations like sending emails to users or some other routine tasks.
As far as your views are concerned. They should carry out all the logical stuff. Your business logic is basically implemented by the views.
Lastly templates should do the work of rendering things whatever is fed to them by the views.
Obviously there are no such hard and fast rule. There can be exceptions to the above. Or someone could find any other way of doing things better rather than that i mentioned. For that you need to use your own conscience and no one can teach you that. It comes with experience.
For your reference please go through the following articles:
https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world
https://flask.palletsprojects.com/en/1.1.x/tutorial/
Also you might be using this but in case your aren't:
https://flask-marshmallow.readthedocs.io/en/latest/
Just one small suggestion too. though i am not clear about your requirements. But it would be better if you could commit after the for loop i.e at the very end of the method. A request should generally have only one commit.
Django 1.11 introduced cool feature to combine querysets.
You can make something like:
combined_qs = qs1.union(qs2, qs3)
And voila! combined_qs has all querysets together keeping their order (btw, am I right here?). I need this, as qs1 should be the first, qs2 the second and so on.
But the challenging part is those querysets can't be ordered by any field. As I get it, it happens due MySql ignores any ORDER BY statements in UNION, and it can be done only via subquery. And I don't get it how it can be done via Django ORM without too much raw SQL magic.
Any ideas on how those querysets can be unioned with respect to their order_by() statements?
Upd.
Seems like I saw what I wanted to see and not what it is. Perhaps, I still should stay with django-querysetsequence. Which is a good lib, but I had a hope we have finally got the native method.
i'm working on a project (written in Django) which has only a few entities, but many rows for each entity.
In my application i have several static "reports", directly written in plain SQL. The users can also search the database via a generic filter form. Since the target audience is really tech-savvy and at some point the filter doesn't fit their needs, i think about creating a query language for my database like YQL or Jira's advanced search.
I found http://sourceforge.net/projects/littletable/ and http://www.quicksort.co.uk/DeeDoc.html, but it seems that they only operate on in-memory objects. Since the database can be too large for holding it in-memory, i would prefer that the query is translated in SQL (or better a Django query) before doing the actual work.
Are there any library or best practices on how to do this?
Writing such a DSL is actually surprisingly easy with PLY, and what ho—there's already an example available for doing just what you want, in Django. You see, Django has this fancy thing called a Q object which make the Django querying side of things fairly easy.
At DjangoCon EU 2012, Matthieu Amiguet gave a session entitled Implementing Domain-specific Languages in Django Applications in which he went through the process, right down to implementing such a DSL as you desire. His slides, which include all you need, are available on his website. The final code (linked to from the last slide, anyway) is available at http://www.matthieuamiguet.ch/media/misc/djangocon2012/resources/compiler.html.
Reinout van Rees also produced some good comments on that session. (He normally does!) These cover a little of the missing context.
You see in there something very similar to YQL and JQL in the examples given:
groups__name="XXX" AND NOT groups__name="YYY"
(modified > 1/4/2011 OR NOT state__name="OK") AND groups__name="XXX"
It can also be tweaked very easily; for example, you might want to use groups.name rather than groups__name (I would). This modification could be made fairly trivially (allow . in the FIELD token, by modifying t_FIELD, and then replacing . with __ before constructing the Q object in p_expression_ID).
So, that satisfies simple querying; it also gives you a good starting point should you wish to make a more complex DSL.
I've faced exactly this problem - a large database which needs searching. I made some static reports and several fancy filters using django (very easy with django) just like you have.
However the power users were clamouring for more. I decided that there already was a DSL that they all knew - SQL. The question was how to make it secure enough.
So I used django permissions to give the power users permission to make SQL queries in a new table. I then made a view for the not-quite-so-power users to use these queries. I made them take optional parameters. The queries were run using Python's lower level DB-API which django is using under the hood for its ORM anyway.
The real trick was opening a read only database connection to run these queries just to make sure that no updates were ever run. I made a read only connection by creating a different user in the database with lower permissions and opening a specific connection for that in the view.
TL;DR - SQL is the way to go!
Depending on the form of your data, the types of queries your users need to use, and the frequency that your data is updated, an alternative to the pure SQL solution suggested by Nick Craig-Wood is to index your data in Solr and then run queries against it.
Solr is an added layer of complexity (configuration, data synchronization) but it is super-fast, can handle large datasets, and provides a (relatively) intuitive query language.
You could write your own SQL-ish language using pyparsing, actually. There is even pretty verbose example you could extend.
I am in need of a lightweight way to store dictionaries of data into a database. What I need is something that:
Creates a database table from a simple type description (int, float, datetime etc)
Takes a dictionary object and inserts it into the database (including handling datetime objects!)
If possible: Can handle basic references, so the dictionary can reference other tables
I would prefer something that doesn't do a lot of magic. I just need an easy way to setup and get data into an SQL database.
What would you suggest? There seems to be a lot of ORM software around, but I find it hard to evaluate them.
SQLAlchemy's SQL expression layer can easily cover the first two requirements. If you also want reference handling then you'll need to use the ORM, but this might fail your lightweight requirement depending on your definition of lightweight.
SQLAlchemy offers an ORM much like django, but does not require that you work within a web framework.
From it's description, perhaps Axiom is a pythonic tool for this .
Seeing as you have mentioned sql, python and orm in your tags, are you looking for Django? Of all the web frameworks I've tried, I like this one the best. You'd be looking at models, specifically. This could be too fancy for your needs, perhaps, but that shouldn't stop you looking at the code of Django itself and learning from it.
Rather than use an ORM, I am considering the following approach in Python and MySQL with no ORM (SQLObject/SQLAlchemy). I would like to get some feedback on whether this seems likely to have any negative long-term consequences since in the short-term view it seems fine from what I can tell.
Rather than translate a row from the database into an object:
each table is represented by a class
a row is retrieved as a dict
an object representing a cursor provides access to a table like so:
cursor.mytable.get_by_ids(low, high)
removing means setting the time_of_removal to the current time
So essentially this does away with the need for an ORM since each table has a class to represent it and within that class, a separate dict represents each row.
Type mapping is trivial because each dict (row) being a first class object in python/blub allows you to know the class of the object and, besides, the low-level database library in Python handles the conversion of types at the field level into their appropriate application-level types.
If you see any potential problems with going down this road, please let me know. Thanks.
That doesn't do away with the need for an ORM. That is an ORM. In which case, why reinvent the wheel?
Is there a compelling reason you're trying to avoid using an established ORM?
You will still be using SQLAlchemy. ResultProxy is actually a dictionary once you go for .fetchmany() or similar.
Use SQLAlchemy as a tool that makes managing connections easier, as well as executing statements. Documentation is pretty much separated in sections, so you will be reading just the part that you need.
web.py has in a decent db abstraction too (not an ORM).
Queries are written in SQL (not specific to any rdbms), but your code remains compatible with any of the supported dbs (sqlite, mysql, postresql, and others).
from http://webpy.org/cookbook/select:
myvar = dict(name="Bob")
results = db.select('mytable', myvar, where="name = $name")