I'm running into an architectural issue in regards to how to organize a certain project.
The project is to create a package of models and relative database connections to be used in multiple different web applications.
I wish to create a model hierarchy with the following models:
User
Application
Log
There are more models but these three exemplify the basic underlying problem.
The User model has a many to many relationship with Application.
And an Application has a one to many relationship with Log.
User * - * Application
Application 1 - * Log
The problem i am running into is that i want User and Application to exist in a Postgres database and the logs to exist in a Mongo datastore.
The main reason i have for putting Logs into a different data store is that i don't need to do any updates, just inserts for said model. I will not need immediate consistency with the inserts. And i want to utilize the aggregation framework and map reductions on a weekly basis. Also the amount of documents of logs i will have will trump the amount of users/applications i will have. On a scale of millions to 1. The server i will be using this model mostly in will be communicating over websockets so the amount of traffic and inserts will be high and i don't really need postgres for it. (But please feel free to (nicely) steer me in a different direction if you feel so obliged).
Now i am using python with SQLAlchemy for the modeling, engine, and the migrations.
And while i haven't started yet i plan on using MongoKit for the mongo model(s).
So far i have the SQLAlchemy engine setup with the models and migrations for the User and Application models.
How do i setup the entire package so that when i include this repo as a dependency for a Flask application i can utilize both engines and connection pools?
Because i can already see that having a property on the Application for retrieving the latest logs may become an issue.
Related
I'm a noobie to Django and I'm almost at the stage of deploying a web server.
I was just having some doubts with Django's database. Currently I'm using the default sqlite3 database to store all the user models as well as the info models. I'm thinking of using AWS to deploy my web sever.
So when I get to that stage, should I continue with sqlite or should I switch to AWS's database or something like Firebase. If I continue with sqlite, where and how exactly will the information be stored? And what if I switch to something like PostgreSQL, where will the information be stored and will it be secure/fast (even if I manage to get thousands of users)?
Thanks so much, this question might be really basic but I'm super confused.
sqlite is a flat file database, it uses an exposed file in your project to save your data, this is fine in local environment, but when deploying you need to consider that the server and the database are in the same machine and using the same disk. that means if you accidentally remove the machine -and its disk space- used to serve the application, then the database itself will be deleted with all records.
Plus you will face problems if you tried to scale your servers, that is every server will have his own copy of the database and syncing all those files will be huge headache.
If your data is not that important then you can keep using sqlite, but if you are expecting high traffic and complex db structure, then I would recommend you consider a db engine like Mysql or maybe look up the databases offered by amazon here:
https://aws.amazon.com/products/databases/
For django, you will need to change the adapter when using a different db like mysql, sqlite or anything else.
https://docs.djangoproject.com/en/3.0/ref/databases/
I am developing a Cloud based data analysis tool, and I am using Django(1.10) for that.
I have to add columns to the existing tables, create new tables, change data-type of columns(part of data-cleaning activity) at the run time and can't figure out a way to update/reflect those changes, in run time, in the Django model, because those changes will be required in further analysis process.
I have looked into 'inspectdb' and 'syncdb', but all of these options would require taking the portal offline and then making those changes, which I don't want.
Please can you suggest a solution or a work-around of how to achieve this.
Also, is there a way in which I can select what database I want to work from the list of databases on my MySQL server, after running Django.
Django's ORM might not be the right tool for you if you need to change your schema (or db) online - the schema is defined in python modules and loaded once when Django's web server starts.
You can still use Django's templates, forms and other libraries and write your own custom DB access layer that manipulates a DB dynamically using python.
I have a live Django 1.6 project that makes use of a postgresql database. I've currently been developing an API for the project to support a mobile app - making use of South migrations as I go along.
As a result of the of this: I have a new database structure and code-base that is quite far removed from my production database although many of the fields in the production database are still present in the new one.
So as it sits - I have the mobile app running off the new API and the website running off the old database.
How can I go about synchronizing data between the two databases while the front-end for the new website is being developed?
i.e. When a user makes use of the app and then later goes to the website, I'd like their data to be available and vice-versa.
Solutions I've tried:
I cannot use the API to update the new database because it would trigger activation emails for users that have already been activated.
I've tried to use peewee to create a synchronization script where each field in one database is mapped to a field in the other database. This has been effective for tables where the schema are similar but I've had trouble when it comes to keeping foreign key relationships in tact.
Is there software that provides multi-DB multi-tenant support for Django and works with MongoDB?
I think I only need multi-tenancy at the database level and maybe at the schema level but not at the application level.
I have a pretty complicated user model. Some users can view certain data inputted by other users. Users usually belong to organizations. Organizations can be nested hierarchically, and there can be similarities in how the application is configured for users within the organization (e.g., all users within an organization will fill out the same form, unless that's overriden for an individual user). Sometimes certain data that users submit can be viewed by users outside of their organization and even outside of the hierarchy that their organization's within. Organizations using the app can be competitors, and the data we're dealing with is sensitive, so it needs to be very secure. It also needs to be developed very quickly.
I'm thinking of giving each user their own DB, and then either having shared DBs or one shared DB with multiple schemas in order to store configurations that are shared across users within organizations.
Multi tenancy on MongoDB is perfectly viable, we are using it in production at onliquid.com.
I don't know of any lib, plugin or specific software that does it for you but it is doable with not that much effort. If you want to dive into it I would advise to pay special attention to how the driver you are using behaves when selecting the database to read and write to and start working on that. Also take a look at MongoDB configuration options like smallfiles and directoryperdb which allow you to better manage the differences and avoid some problems.
I wrote a blog post some time ago about this for Ruby on Rails using Mongoid, most of the details are applicable to all web frameworks and specific to the inner workings of MongoDB.
This is what I have :-
Ubuntu 11.10.
Django 1.3
Python 2.7
What I want to do is build an app that is similar to top-coder and I have the skeletal version of the app sketched out. The basic requirements would be:-
1. Saving the code.
2. Saving the user name and ranks.(User-profile)
3. Should allow a teacher to create multiple choice questions too.( Similar to Google docs).
I have basic knowledge of Django and have built couple of (basic) apps before. Rather than building an online tool, is it possible to build something very similar to conf2py that sits on top of web2py, in Django.
Lets call this small project examPy( I know, very original), is it possible to build an app that acts more a plug-in to Django or is my concept of Django absolutely wrong?
The primary question being:
As I want to learn a new DB and have worked on postgres in Django, should I chose CouchDB or MongoDB for Django?
Answers can be explanations or links to certain documentations or blogs that can tell me the pros and cons.
General Differences
Comparing MongoDB and CouchDB
MongoDB VS CouchDB
Battle of the NoSQL Stars
Django Specific
MongoDB for Django-nonrel
Django-MongoDB Engine
MongoDB Hearts Django?
An Introduction to Using CouchDB with Django
Using CouchDB with Django
All my research points me towards the idea that Mongo and Couch are similar enough that your choice would probably boil down to personal (subjective) preference even over the use-case. Personally, I've developed a CouchDB fetish and am looking for a reason to use it.
The key factor influencing your decision should probably be which noSQL solution has the most mature ORM framework for Django?
I've used mongo-engine with Django but you need to create a file specifically for Mongo documents eg. Mongo_models.py. In that file you define your Mongo documents. You then create forms to match each Mongo document. Each form has a save method which inserts or updates whats stored in Mongo. Django forms are designed to plug into any data back end ( with a bit of craft ).
If you go this route you can dodge Django non-rel which is still not part of Django 1.4. In addition I believe django-nonrel is on hiatus right now.
I've used both CouchDB and Mongo extensively. CouchDB has a lovely interface. My colleague is working on something similar for Mongo. Mongo's map and reduce are far faster than CouchDB. Mongo is more responsive loading and retrieving data. The python libraries for Mongo are easier to get working with ( both pymongo and mongo-engine are excellent )
Be sure you read the Mongo production recommendations! Do not run one instance on the same node as Django or prepare to be savagely burned when traffic peaks. Mondo works great with Memcache/Redis where one can store reduced data for rapid lookups.
BEWARE: If you have very well defined and structured data that can be described in documents or models then don't use Mongo. Its not designed for that and something like PostGreSQL will work much better.
I use PostGreSQL for relational or well structured data because its good for that. Small memory footprint and good response.
I use Redis to cache or operate in memory queues/lists because its very good for that. great performance providing you have the memory to cope with it.
I use Mongo to store large JSON documents and to perform Map and reduce on them ( if needed ) because its very good for that. Be sure to use indexing on certain columns if you can to speed up lookups.
Don't use a circle to fill a square hole. It won't fill it.
I've used CouchDB with Django for a production application. Couch is fine and has some great ideas, but I'm moving that app over to MongoDB. Why? There is support for Mongo in the Django community. Django-nonrel has a MongoDB backend. Using Django-toolbox I can embed models in models and have some basic admin support.
If I remember correctly, Django-nonrel will eventually be rolled into Django core. In five years time I see much more support for Mongo in Django than Couch. Of course that can change, but I see Mongo a better fit.