I am over accustomed to Django ORM and feel handicapped when trying to build a standalone python-twisted application which needs database integration.
SQLAlchemy looks promising - true. But I am trying to tinker with twisted as well and am unable to find anything on the lines of a good async python orm.
what I have found (https://stackoverflow.com/a/1705987/338691) would force me to write raw sql queries - doesn't feel quite right after my elongated stint with django.
So how does one play with database schema in a twisted application?
There is also http://findingscience.com/twistar/ which unfortunately follows the Active Record pattern and last time I checked, the author feels that migrations are out of scope of the project. So you would end up writing migrations manually anyway (maybe there could be some adapter for alembic for that, that would be cool).
Also I remember seeing github repo where the author tries to make twisted play nicely with sqlalchemy (without deferToThread) but I haven't followed to see if it was a success and can't find the URL. (also Twisted + SQLAlchemy and the best way to do it)
And lastly, recent versions of psycopg supports setting an async callback. Maybe that could be leveraged to something (integration with SQLAlchemy? or something).
UPDATE: also recently appeared this interesting project - alchimia
Related
I'm using a Nodejs server for a WebApp and Mongoose is acting as the ORM.
I've got some hooks that fire when data is inserted into a certain collection.
I want those hooks to fire when a python script inserts into the mongoDB instance. So if I have a pre save hook, it would modify the python scripts insert according to that hook.
Is this possible? If so, How do I do it?
If not, please feel free to explain to me why this is impossible and/or why I'm stupid.
EDIT: I came back to this question some months later and cringed just at how green I was when I asked it. All I really needed done was to create an API endpoint/flag on the NodeJS server that is specifically for automated tasks like the python script to send data to, and have mongoose in NodeJS land structure.
It is impossible because python and nodejs are 2 different runtimes - separate isolated processes which don't have access to each other memories.
Mongoose is a nodejs ORM - a library that maps Javascript objects to Mongodb documents and handles queries to the database.
All mongoose hooks belong to javascript space. They are executed on javascript objects before Mongoose sends any request to mongo. 2 outcomes from there: no other process can mess up with these hooks, not even another nodejs, and once the query reaches mongodb it's final, no more hooks, no more modifications.
One said a picture worth 100 words:
Neither python nor mongo are aware about mongoose hooks. All queries to mongo are initiated on the client side - a script sends a request to modify state of the database or to query state of the database.
The only way to trigger a javascript code execution from an update on mongodb side is to use change streams
Change streams are not mongoose hooks but can be used to hook into the updates on mongo side. It's a bit more advanced use of the database. It comes with additional requirements for mongo set up, size of the oplog, availability of the changestream clients, error handling etc.
You can learn more about change streams here https://docs.mongodb.com/manual/changeStreams/ I would strongly recommend to seek professional advice to architect such set up to avoid frustration and unexpected behaviour.
Mongo itself does not support hooks as a feature, mongoose gives you out of the box hooks you can use as you've mentioned. So what can you do to make it work in python?
Use an existing framework like python's eve, eve gives you database hooks, much like mongoose does. Now eve is a REST api framework which from your description doesn't sound like what you're looking for. Unfortunately I do not know of any package that's a perfect fit to your needs (if you do find one it would be great if you share a link in your question).
Build your own custom wrapper like this one. You can just built a custom wrapper class real quick and implement your own logic very easily.
At my work, we are using different SQL and NOSQL database types (i.e postgres, mysql, sqlite, mongodb, arangodb).
As we are creating lots of packages communicating with a database to retrieve some data. Recreating the CRUD functionality (+ methods to create schema => create table/collection/db) for each package make it harder to maintain the whole pipeline while scaling up.
In order to solve this situation more easily, we decided to create a database plugins API that allows us to abstract CRUD methods so you don't care about the backend database being used anymore on the backend side. All packages that need to communicate with a database will just wrap the plugins CRUD methods and add their own package specific behaviour.
I'm pretty sure we are not the only having this kind of situation out there, so I 've been looking on the web to try to find if such a thing has already been done without finding anything interesting.
Could anyone refer me to anything that may do/try to do it?
Is there anything that already does it using Python?
Thanks
I am using MongoDB as my primary(and only till now) database and because of google and the links it provided me i am confused between Django or Pyramid.
I am comfortable with python but never done web development in python(i have done in PHP). Now because i will be using Mongo so i wont use Django ORM will that take away the ease of development people associate with Django?
I am new to Django(just a few hours) so i am not sure what parts of the framework the ORM affects.
Or should i go with the django fork django-norel with django mongodb engine (they are not actively maintained though) or should i use Pyramid because i plan to use jinja2 as my template layer so that makes 2 parts of django useless to me.
After removing these batteries from django does it still remain true that it a framework for people with deadlines?
Advice?
A year or two ago I was also deciding between django and pyramid w/ mongodb to build a high performance web application. I ultimately chose Pyramid :
Pros:
Pyramid is very light weight for a full stack framework. There is a minmal amount of 'magic' going on under the hood. I was able to wrap my head around all the pieces.
It is not as opinionated as Django. You can plug and play whatever templating engines or databases you need relatively easily.
I was impressed by the performance benchmarks between Pyramid over other full stack frameworks.
The Pylons/Pyramid culture of 100% code coverage and "only pay for what you eat" was appealing to my own development style.
Cons:
The community is growing, but still not as mature as Django
There's a lot of documentation, but it's not as robust as some of the Django docs out there
The lack of 'magic' and beginner friendly documentation make for a steeper learning curve.
As for using an ORM with MongoDB; I recommend you try building without one at first. After trying a few mongodb ORMs, I've ended up going back to plain pymongo with Colander or Validictory for validation. Pymongo is already very ORM-like, fast, and flexible. I felt the extra layers of abstraction took away from that flexibility, and caused me to write more code than was needed.
I'm going to suggest an alternative that has not been mentioned: Flask. Flask has a really great (albeit smaller than Django) community and there are a lot of extensions available for common web-app extensions, in the extensions directory.
There are several MongoDB extensions for Flask, which help to integrate MongoDB into the framework, but I would also highly recommend the mongoengine ORM. One of the people working on mongoengine has release an extension for Flask integration, Flask-MongoEngine.
Try it all (Django, Pyramid, Flask), read the docs, and you done.
If you will select Django, you will not use some parts of it (orm, formlib), and that is all.
ALso see this video:
https://www.10gen.com/presentations/mongosf-2011/mongodb-with-python-pylons-pyramid
Is there any simple tutorial for learning Neo4J. I went through its official manual and found very confusing for me since i am from mysql background. I just wanted to learn Neo4J but i am really finding a hard time. Also there is not much books available for Neo4J
Please help me with this
I just updated neo4django's documentation, and like to think it reads like a tutorial. I'd appreciate feedback!
Look at the Bulbs Quickstart tutorial (http://bulbflow.com/quickstart/). Bulbs works with any Python Web framework, including Django, Flask, and Pyramid.
There is a nice integration for Neo4j+Django: https://github.com/scholrly/neo4django
and I notice a tutorial on github: https://github.com/johanlundberg/neo4j-django-tutorial
There are a few books in the works, covering general concepts about working with a graph database and Neo4j specifics. But they haven't yet hit the shelves.
For now, the Neo4j manual is the best material available for learning.
-Andreas
neo4django is super outdated. Check the updated options at:
https://neo4j.com/developer/python/
About the tutorial you can find many free resources at the website. You can also start with Cypher, the query language:
https://neo4j.com/developer/cypher/
FYI: neomodel is also beginning to be outdated. It's not compatible with the latest neo-driver version for instance.
It lacks updates, and apart from that, we had some serious threading issues with it - seems like it is not thread safe/not synchronous, or not doing the whole async/sync thing correctly. (since Neo4J is async, and Django is not (yet))
The idea behind these libs was to have some kind of abstraction layer, like an ORM, but what's the point if it is not any more convenient than Cypher, nor can it do everything Cypher can, so you start writing queries in Cypher anyhow. Having something graph database agnostic is also fake, since it only serves Neo4J.
We already rewrote all queries in Cypher, now I just have to find a way to do the data modelling natively.
I want to be able to instrument Python applications so that I know:
Page generation time.
Percentage of time spent in external requests (mysql, api calls).
Number of mysql queries, what the MySQL queries were.
I want this data from production (not offline profiling) - because the time spent in various places will be different under load.
In PHP I can do this with XHProf or instrumentation-for-php. In Ruby on Rails/.NET/Java, I can do this with New Relic.
Is there such a package recommended for Python or django?
Yes, it's perfectly possible. E.g. use some magic switch in URL, like "?profile-me" which triggers profiling in Django middleware.
There are a number of snippets on the Internet, like this one: http://djangosnippets.org/snippets/70/ or modules like this one: http://code.google.com/p/django-profiling/ - but I haven't used any of them so I cannot recommend anything.
Anyway, the approach they take is similar to what I do - i.e. use Python Hotshot profiler module in a middleware that wraps your view. For the MySQL part, you can just use connection.queries form Django.
The nice thing about Hotshot is that its output can be browsed using Kcachegrind like here: http://www.rkblog.rk.edu.pl/w/p/django-profiling-hotshot-and-kcachegrind/
New Relic now had a package for Python, including Django through mod_wsgi.
https://support.newrelic.com/help/kb/python
django-prometheus is a good choice for handling production workloads, especially in a container environment like Kubernetes. Out of the box, it has middleware for tracking request latencies and counts (by view method), as well as Database and cache access times. It wouldn't be a good solution for tracking which queries are actually executing, but that's where a logging solution like ELK would come into play. If it helps, I've written a post which walks through how to add custom metrics to a Django application.