So, I'm looking into the Django and Neo4j integration, but there's not much about it out there ... What I want to know is if I have a model like the one here:
If I want to add a new property to a model, it would be as simple as doing:
node.setProperty( "newProperty", "something" );
??
Moreover, all the queries in django would work ? How is the traversing made ?
I would appreciate any response
:D
Thanks.
We've been working on updating the Neo4j/Django integration to work with neo4j-rest-client - the fruits of our labor are on GitHub, with some quick comments on my blog.
There are some pros and cons to our integration. The most obviously impacting is our use of a REST client- you get to use a remote database, while losing quite a bit, performance-wise. OTOH, the integration works alongside a relational database, so you can still use django.contrib stuff that relies on the original ORM, and it handles indexing and query sets pretty nicely.
To do what you want above using neo4django, you'd simply get the neo4j-rest-client based node from the model instance, and have at it.
model_instance.node['newProperty'] = 'something'
We're still cranking on making the integration more dynamic, supporting traversals/etc in a Pythonic way, and (currently most important) improving the performance. If you're interested, I'd love feedback.
Have you seen Tobias's blog post about Django integration? It's kinda old now, but still relevant. Plus there are examples you can check out too.
Related
i'm working on a project (written in Django) which has only a few entities, but many rows for each entity.
In my application i have several static "reports", directly written in plain SQL. The users can also search the database via a generic filter form. Since the target audience is really tech-savvy and at some point the filter doesn't fit their needs, i think about creating a query language for my database like YQL or Jira's advanced search.
I found http://sourceforge.net/projects/littletable/ and http://www.quicksort.co.uk/DeeDoc.html, but it seems that they only operate on in-memory objects. Since the database can be too large for holding it in-memory, i would prefer that the query is translated in SQL (or better a Django query) before doing the actual work.
Are there any library or best practices on how to do this?
Writing such a DSL is actually surprisingly easy with PLY, and what ho—there's already an example available for doing just what you want, in Django. You see, Django has this fancy thing called a Q object which make the Django querying side of things fairly easy.
At DjangoCon EU 2012, Matthieu Amiguet gave a session entitled Implementing Domain-specific Languages in Django Applications in which he went through the process, right down to implementing such a DSL as you desire. His slides, which include all you need, are available on his website. The final code (linked to from the last slide, anyway) is available at http://www.matthieuamiguet.ch/media/misc/djangocon2012/resources/compiler.html.
Reinout van Rees also produced some good comments on that session. (He normally does!) These cover a little of the missing context.
You see in there something very similar to YQL and JQL in the examples given:
groups__name="XXX" AND NOT groups__name="YYY"
(modified > 1/4/2011 OR NOT state__name="OK") AND groups__name="XXX"
It can also be tweaked very easily; for example, you might want to use groups.name rather than groups__name (I would). This modification could be made fairly trivially (allow . in the FIELD token, by modifying t_FIELD, and then replacing . with __ before constructing the Q object in p_expression_ID).
So, that satisfies simple querying; it also gives you a good starting point should you wish to make a more complex DSL.
I've faced exactly this problem - a large database which needs searching. I made some static reports and several fancy filters using django (very easy with django) just like you have.
However the power users were clamouring for more. I decided that there already was a DSL that they all knew - SQL. The question was how to make it secure enough.
So I used django permissions to give the power users permission to make SQL queries in a new table. I then made a view for the not-quite-so-power users to use these queries. I made them take optional parameters. The queries were run using Python's lower level DB-API which django is using under the hood for its ORM anyway.
The real trick was opening a read only database connection to run these queries just to make sure that no updates were ever run. I made a read only connection by creating a different user in the database with lower permissions and opening a specific connection for that in the view.
TL;DR - SQL is the way to go!
Depending on the form of your data, the types of queries your users need to use, and the frequency that your data is updated, an alternative to the pure SQL solution suggested by Nick Craig-Wood is to index your data in Solr and then run queries against it.
Solr is an added layer of complexity (configuration, data synchronization) but it is super-fast, can handle large datasets, and provides a (relatively) intuitive query language.
You could write your own SQL-ish language using pyparsing, actually. There is even pretty verbose example you could extend.
I want to try Mongodb w/ mongoengine. I'm new to Django and databases and I'm having a fit with Foreign Keys, Joins, Circular Imports (you name it). I know I could eventually work through these issues but Mongo just seems like a simpler solution for what I am doing. My question is I'm using a lot of pluggable apps (Imagekit, Haystack, Registration, etc) and wanted to know if these apps will continue to work if I make the switch. Are there any known headaches that I will encounter, if so I might just keep banging my head with MySQL.
There's no reason why you can't use one of the standard RDBMSs for all the standard Django apps, and then Mongo for your app. You'll just have to replace all the standard ways of processing things from the Django ORM with doing it the Mongo way.
So you can keep urls.py and its neat pattern matching, views will still get parameters, and templates can still take objects.
You'll lose querysets because I suspect they are too closely tied to the RDBMS models - but they are just lazily evaluated lists really. Just ignore the Django docs on writing models.py and code up your database business logic in a Mongo paradigm.
Oh, and you won't have the Django Admin interface for easy access to your data.
You might want to check out django-nonrel, which is a young but promising attempt at a NoSQL backend for Django. Documentation is lacking at the moment, but it works great if you just work it out.
I've used mongoengine with django but you need to create a file like mongo_models.py for example. In that file you define your Mongo documents. You then create forms to match each Mongo document. Each form has a save method which inserts or updates whats stored in Mongo. Django forms are designed to plug into any data back end ( with a bit of craft )
BEWARE: If you have very well defined and structured data that can be described in documents or models then don't use Mongo. Its not designed for that and something like PostGreSQL will work much better.
I use PostGreSQL for relational or well structured data because its good for that. Small memory footprint and good response.
I use Redis to cache or operate in memory queues/lists because its very good for that. great performance providing you have the memory to cope with it.
I use Mongo to store large JSON documents and to perform Map and reduce on them ( if needed ) because its very good for that. Be sure to use indexing on certain columns if you can to speed up lookups.
Don't circle to fill a square hole. It won't fill it.
I've seen too many posts where someone wanted to swap a relational DB for Mongo because Mongo is a buzz word. Don't get me wrong, Mongo is really great... when you use it appropriately. I love using Mongo appropriately
Upfront, it won't work for any existing Django app that ships it's models. There's no backend for storing Django's Model data in mongodb or other NoSQL storages at the moment and, database backends aside, models themselves are somewhat of a moot point, because once you get in to using someones app (django.contrib apps included) that ships model-template-view triads, whenever you require a slightly different model for your purposes you either have to edit the application code (plain wrong), dynamically edit the contents of imported Python modules at runtime (magical), fork the application source altogether (cumbersome) or provide additional settings (good, but it's a rare encounter, with django.contrib.auth probably being the only widely known example of an application that allows you to dynamically specify which model it will use, as is the case with user profile models through the AUTH_PROFILE_MODULE setting).
This might sound bad, but what it really means is that you'll have to deploy SQL and NoSQL databases in parallel and go from an app-to-app basis--like Spacedman suggested--and if mongodb is the best fit for a certain app, hell, just roll your own custom app.
There's a lot of fine Djangonauts with NoSQL storages on their minds. If you followed the streams from the past Djangocon presentations, every year there's been important discussions about how Django should leverage NoSQL storages. I'm pretty sure, in this year or the next, someone will refactor the apps and models API to pave the path to a clean design that can finally unify all the different flavors of NoSQL storages as part of the Django core.
I have recently tried this (although without Mongoengine). There are a huge number of pitfalls, IMHO:
No admin interface.
No Auth django.contrib.auth relies on the DB interface.
Many things rely on django.contrib.auth.User. For example, the RequestContext class. This is a huge hindrance.
No Registration (Relies on the DB interface and django.contrib.auth)
Basically, search through the django interface for references to django.contrib.auth and you'll see how many things will be broken.
That said, it's possible that MongoEngine provides some support to replace/augment django.contrib.auth with something better, but there are so many things that depend on it that it's hard to say how you'd monkey patch something that much.
Primary pitfall (for me): no JOINs!
I want to develop a desktop application using python with basic crud operation. Is there any library in python that can generate a code for CRUD functionality and user interface given a database table.
Hopefully, this won't be the best option you end up with, but, in the tradition of using web-interfaces for desktop applications, you could always try django. I would particularLY take a look at the inspectdb command, which will generate the ORM code for you.
The advantage is that it won't require that much code to get off the ground, and if you just want to use it from the desktop, you don't need a webserver; you can use the provided test server. The bundled admin site is easy to get off the ground, and flexible up to a point; past which people seem to invest a lot of time battling it (probably a testimony to how helpful it is at first).
There are many disadvantages, not the least of which is the possibility of having to use html/javascript/css when you want to start customizing a lot.
If it were me, I would consider borrowing django's ORM, but then again, I'm already familiar with it.
Having said that, I like working with it, it's usable outside the framework, and it will give you mysql, postgres, or sqlite support. You could also hook up the django admin site to your models and have a web-based editor.
There are surely other ORMs and code generators out there too (I hope some python gurus will point some out, I'm kind of curious).
If you want something really small and simple, I like the Autumn ORM.
If you use the Django ORM, you can use the automatically-generated Django admin interface, which is really nice. It's basically a web-based GUI for browsing and editing records in your database.
If you think you will need advanced SQL features, SQLAlchemy is a good way to go. I suspect for a desktop application, Django or Autumn would be better.
There are other Python ORMs, such as Storm. Do a Google search on "python ORM". See also the discussion on this web site: What are some good Python ORM solutions?
If I change a field in a Django model, how can I synchronize it with the database tables? Do I need to do it manually on the database or is there a tool that does helps with the process?
Alas, Django does not support any easy solution to this.
The only thing django will do for you, is restart your database with new tables that match your new models:
$ #DON'T DO THIS UNLESS YOU CAN AFFORD TO LOSE ALL YOUR DATA!
$ python PROJECT_DIR/manage.py syncdb
the next option is to use the various sql* options to manage.py to see what django would do to match the current models to the database, then issue your own ALTER TABLE commands to make everything work right. Of course this is error prone and difficult.
The real solution is to use a database migration tool, such as south to generate migration code.
Here is a similar question with discussion about various database migration options for django.
Can't seem to be able to add a comment to the marked answer, probably because I haven't got enough rep (be nice if SO told me so though).
Anyway, just wanted to add that in the answered post, I believe it is wrong about syncdb - syncdb does not touch tables once they have been created and have data in them. You should not lose data by calling it (otherwise, how could you add new tables for new apps?)
I believe the poster was referring to the reset command instead, which does result in data loss - it will drop the table and recreate it and hence it'll have all the latest model changes.
Django Evolution can help, but the best option really is to plan out your schema in advance, or to make simple modifications manually. Or, to be willing to toast your test data by dropping tables and re-syncing.
Django does not provide for this out of the box.
Here's some information from the Django Book on doing it by hand (see Making Changes to a Database Schema). This works for straightforward, simple changes.
Longer-term, you'll probably want to use a migration tool. There are three major options:
django-evolution
Dmigrations (written by Simon Willison, one of the creators of Django) (works only with MySQL)
South
EDIT: Looking through the question linked by TokenMacGuy, I'll add two more to the list for the sake of completeness:
Migratory
simplemigrations
Just to throw in an extra opinion - dmigrations is pretty nice and clear to use, but I'd say South is your best bet. Again, it's easy to get into, but it's more powerful and also has support for more database backends than just MySQL. It even handles MSSQL, if that's your thing
I understand that there are big differences in data-store, but surely since django is bundled and it abstracts data-store away from Satchmo, something can be done?
Truth is that I am not a Python guy, been mostly Java/PHP thus far, but I am willing to learn.
Plus, if this is not possible today, lets band together and form a new Open Source project to "extend" satchmo or perhaps branch it, for compatibility?
Possible if:
Someone writes a generic ORM to Bigtable mapper. Most probably, Appengine Patch Guys
Someone rewrites the views and models of Satchmo to remove existing ORM queries and use the minimal functionality of the ORM provided by the patch project, should be either you or the Satchmo guys.
Someone hacks around a lot using the django helper project, can only be helper project guys.
You can't. There are alot of dependencies in Satchmo that you aren't allowed to install on AppEngine.
See this thread as well: http://groups.google.com/group/satchmo-users/browse_thread/thread/509265ccd5f5fc1e?pli=1
Nothing is impossible - this will just require lots of effort - if there will be somebody wishing to do so - why not? But it might be easier (cheaper) to get Django friendly hosting instead of spending hours on hacking the code.