Is there software that provides multi-DB multi-tenant support for Django and works with MongoDB?
I think I only need multi-tenancy at the database level and maybe at the schema level but not at the application level.
I have a pretty complicated user model. Some users can view certain data inputted by other users. Users usually belong to organizations. Organizations can be nested hierarchically, and there can be similarities in how the application is configured for users within the organization (e.g., all users within an organization will fill out the same form, unless that's overriden for an individual user). Sometimes certain data that users submit can be viewed by users outside of their organization and even outside of the hierarchy that their organization's within. Organizations using the app can be competitors, and the data we're dealing with is sensitive, so it needs to be very secure. It also needs to be developed very quickly.
I'm thinking of giving each user their own DB, and then either having shared DBs or one shared DB with multiple schemas in order to store configurations that are shared across users within organizations.
Multi tenancy on MongoDB is perfectly viable, we are using it in production at onliquid.com.
I don't know of any lib, plugin or specific software that does it for you but it is doable with not that much effort. If you want to dive into it I would advise to pay special attention to how the driver you are using behaves when selecting the database to read and write to and start working on that. Also take a look at MongoDB configuration options like smallfiles and directoryperdb which allow you to better manage the differences and avoid some problems.
I wrote a blog post some time ago about this for Ruby on Rails using Mongoid, most of the details are applicable to all web frameworks and specific to the inner workings of MongoDB.
Related
Currently in the company I have developed about six applications with Django, these applications are managed in different databases since they are focused on different departments. They are asking me that access to these applications be done through a single credential, that is, if a user must handle two applications, they should not have two different access credentials, but rather they can access both applications with a single credential.
I have consulted various sources, one person recommended me to develop an application that is only responsible for access, but I am a bit disoriented with this problem.
I appreciate the help!
I recently started developing a Desktop python application and I would like to know how more expert people would handle this issue.
I used to develop (about 5-10 years ago) web applications in the past using PHP + MySQL and there, since the code/program is located on the server where the user doesn't have access (except the web page), I could simply store the user/group permissions in the database in a table say users, users_groups, users_permissions, and so on. I would then check at every page load if the user had the right to access that page / update that record in the database.
With a desktop application where the user has access to the executable (which can relatively easy be decompiled to source code, being written in Python) the approach will likely be quite different.
Since MySQL has forked into MariaDB and is not so actively developed anymore, PostgreSQL looked promising to start. I thought about creating different users on PostgreSQL level and letting PostgreSQL handle the permissions (instad of my application handling them directly).
However, this only allows tuning of the permissions down to the table level. A user will be allowed to create/delete/update records in a table, however no further control is available. AFAIK you cannot tell "let this user only update his own records" or "this user can only delete the records from this group", or "users from group X can only update their own records while users from group Y can update everybody's records".
My understanding as how to handle this kind of issue would be to put some kind of middleware application between the user and the database, located on the server, such as:
Desktop application <-----> Server-side application permissions handler <-----> Database
Where server-side permission handler could be as simple as adding a "WHERE user=..." to each query as well as much more advanced stuff (first check user permissions stored in the database, based on that decide if letting user execute the query or reject it). I think this is a common problem for all desktop applications and would therefore expect that such a server-side application already exist. Am I missing something obvious or maybe PostgreSQL allows for more detailed fine-tuning?
Thank you for all your help ;)
Your intuition is right. It is never a good idea having a client access directly a database. Take a look a Django https://www.djangoproject.com and https://www.django-rest-framework.org
This would be the the basis for your server side. You would handle here business logic, authentication, authorization. The client should basically present the data within the UI and delegate all the decision making to the server.
Here you can find a step by step tutorial about how to implement a REST api with user authentication in Django. https://wsvincent.com/django-rest-framework-authentication-tutorial/
I have earlier worked on Java+Spring to create a web-app.
I have to build a new web-app now.
It will have one centralized db.
There will be two different type of instance of web-app.
Web-App 1:
a) It would have nothing to UI render, no html,js etc.
b) All it need to give is some set of rest API which will
b.1) create some new entries in DB
b.2) modify some entries in DB
b.3) retrieve some of DB records in JSON format.
some frontend code ( doesn't belong to this app) will periodically fetch
this details.
c) it will be used by max by 100,000 people but at a given point of time,
we can expect about 1000 users logged in and doing whats being said in b)
Web-App2 :
a) It will have some dashboards
b) 90% of DB operations would be read operations
c) 10% of DB operations would be write/modify
d) There will be about 1000s of user of this system and at any given point of time
hardly 50-1000 people will be accessing it.
I am thinking of following.
Have Web-App 1 created in python+Django and Web-App 2 created in RoR.
I am planning to use to Dynamo DB and memcache.
Why two different frameworks?
1) So that I get to learn both of them
2) There have been concern about scalability in RoR (and I also know people claim its not there), Web-app 1 may have scaling needs in future.
My questions is Do you see any problem with this combination?
for example active records would want you to use specific namings format for your data base tables? Are there any other concerns similar to this?
Anyone else who have used similar technology stack?
both frameworks are full stack framework and and provide MVC, templating, unit testing, security, db migration, caching, security, ORMs.
For my startup, we also needed to put out a full fleshed website along with an API. We are also using DynamoDB for storing most of the data and are only using MySQL for session info.
I opted to use Ruby on Rails for the Webapp and Sinatra for the API. If you're criteria is simply learning as many new things as possible, then it would make sense to opt for relatively different stacks (django/python and RoR). In our case, we went with sinatra because it's essentially a very lightweight wrapper around Rack and perfect for an API which essentially receives requests, calls one or more services or does some processing and hands out a formatted response. While I don't see any problem with using python/django instead of sinatra, in our case the benefit was having to spend less time working with a different language.
Also, scalability in rails is a bit of an iffy subject. In the end, it's about how you use it. We've had no issues scaling rails with unicorn and nginx. Our business logic is all in the API service and the rails server as well uses the API for most of the work. This means we don't use active record on rails and the website is just another consumer for our API which does all the heavy lifting whether the request comes from an app or the website. Using MySQL for the session store ensures we can route requests to any of the application servers without having to worry about always routing requests from the same client to the same server every time. This allows us to ramp up and down easily only considering the amount of traffic we're getting.
At the time we started working on this, there wasn't an ORM for dynamo db which looked and felt just like active record, so we ended up writing a few high level classes of our own to handle storage and retrieval of models on DynamoDb. Considering DynamoDB is not tailored for scans or joins, this didn't take a lot of effort since we were almost always doing lookups based on keys and ranges. This meant we didn't really need a replacement for active record since the real strength of active record is being able to intuitively do joins, etc. by convention.
DynamoDB does have it's limitations though and you might find yourself in situations where you will need to scan a large number of records. In our case, we also use CloudSearch to index some important info and use it as a fallback for cases when we need to do text based searches which need to scan all our data.
Can you advice me with some articles/applications that allows you create SaaS(Software as a Service) application with Python and Django.
For the moment the general topics I do not understand are:
Do you have one working application for all clients or one app per client
How do you manage database access, permissions or different DB for each client
Are there any tools that allows you to convert one app to SaaS
one project, this will make maintenance easier. I handle host resolution with middleware in django-ikari.
you don't. see #1
I use the following :
django-ikari : anchored (sub)domains
django-guardian : per object permissions
django-tastypie : easy RESTful api
django-userprofiles : better than django-registration
django-billing : plan based subscription controls
django-pricing : plan based subscription definition
While not necessary, the following will help in the long run:
django-hunger : private beta signups
django-waffle : feature flip
django-classy-tags : nice, easy and neat templatetag creation
django-merchant : abstracted payment gateway framework
django-mockups : fast testing with models
django-merlin : better multi-step forms (wizards)
Finally, nice to have
django-activity-stream
A very basic, elementary example of how you would go about it.
Suppose you have a simple app designed to solve a particular business case. For example, you created an app to handle room reservations at your office.
To "convert" this app into a service you have to configure it such that most of the user-specific parts of the application are parametric (they can be "templatized" - for lack of better word).
This is how the front end would be converted. You might create variables to hold the logo, headline, teaser, color scheme for the app; allowing each user to customize their instance.
So far, your app is able to customize itself on the front end. It is still using the same database that was designed in phase one.
Now comes the matter of showing only those fields that are relevant to a particular user. This would be parameterizing the database. So you might add a column that identifies each row as belonging to a particular user; then create views or stored procedures that filter records based on the logged in user.
Now the application is able to be "rented" out; since you are able to customize the instance based on the user.
It then just gets bigger from here - depending on the scale, type and intended customization of your application. You might decide that your application performs better when each user has their own dedicated database instead of the stored procedure + view combo.
You may decide that for some user types (or "packages"), you need a dedicated instance of your application running. So for "premium" or "ultra" users you want to have their own dedicated system running.
If your application requires lots of storage - you might decide to charge separately for storage.
The bottom line is it has nothing to do with the language used. Its more an architecture and design problem.
Software as a Service is just a marketing word, it's technically no different from a server that is accessible over the internet. So question 3 makes no sense. That leaves us with question 1 and 2:
What do you mean with 'app' in this context? Your web application (built with Python and Django) can have multiple Django apps (components that make up the web application) but I think that's not what you mean. You can build your website in Python/Django and have various customization options depending on which user (client) is logged in. For example, a premium client can have several advanced options enabled but it's still part of the same codebase. It's just that some options (buttons/controls, etc) are not shown for certain clients
Django has plenty of tools for user management, permissions and groups. You can give each user (each client) different permissions and these permissions determine what they can do. Database access should be managed by your web application. For example, the code determines what information needs to be displayed on the webpage (depending on which client is logged in) and that code retrieves the information from the database. Depending on the scale that you're aiming for, you can also specify which database should be used to retrieve the information from.
I have a blog post describing my proposal of how to go about making a multi tenant SAAS web application using Django. Multi-tenancy here means that when user registers, they have their sub-domain. To recap:
All tenants share one database, but each has their own schemas. Imagine you have website abc.com and someone registered a xyz tenant so that they access their page through xyz.abc.com, then for a tenant xyz you have a separate schema containing all the tables thus encapsulating data related only to xyz tenant. There are other ways, like having one database and one schema for all, or having even separate databases. But schemas approach is the best trade-off. The django-tenants library's documentation contains more detailed info if you are interested
Use django-tenants library to abstract away work with tenants. When someone accesses xyz.abc.com, you need to know that xyz is the tenant and that you should use xyz schema. django-tenants library does this for you so on each request you can obtain the tenant object by simply doing current_tenant = request.tenant
You need to differentiate between shared tables and tenant-specific tables. For example, having table with list of orders is tenant-specific. Every tenant might have their own database containing all their orders. This table should be inside xyz schema. At the same time, you will have some core Django tables, like user. The data can be shared, for example, to disallow two users registering with the same email.
You need to configure your DNS to catch a wildcard expression *.abc.com, for which you can add an A record inside your CPanel with *.abc.com linking to the IP of your server
I am making a simple sqlite database for storing some non-sensitive client information. I am very familiar with python+sqlite and would prefer to stick with this combo on this project. I would like to create an simple GUI interface for data entry and searching of the database... something very similar to what MS Access provides. I want my wife to be able to enter/search data easily, so PHPmyadmin style things are out of the question.
I understand I could just give in and get MS Access, but if reasonbly possible would rather just write the code myself so it will run on my computers (*nix) and is flexible (so I can later integrate it with a web application and our smart phones.)
Can you developers recommend any interfaces/packages/etc (preferably pythonic) that can accomplish this with reasonable ease?
Thanks!
Since you're interested in future integration with a web application, you might consider using a Python web framework and running the app locally on your machine, using your web browser as the interface. In that case, one easy option would be web2py. Just download, unzip, and run, and you can use the web-based IDE (demo) to create a simple CRUD app very quickly (if you really want to keep it simple, you can even use the "New application wizard" (demo) to build the app). It includes its own server, so you can run your app locally, just like a desktop app.
You can use the web2py DAL (database abstraction layer) to define and create your SQLite database and tables (without writing any SQL). For example:
db = DAL('sqlite://storage.db')
db.define_table('customer',
Field('name', requires=IS_NOT_IN_DB(db, 'customer.name')),
Field('address'),
Field('email', requires=IS_EMAIL()))
The above code will create a SQLite database called storage.db and create a table called 'customer'. It also specifies form validators for the 'name' and 'email' fields, so whenever those fields are filled via a form, the entries will be validated ('name' cannot already be in the DB, and 'email' must be a valid email address format) -- if validation fails, the form will display appropriate error messages (which can be customized).
The DAL will also handle schema migrations automatically, so if you change your table definitions, the database schema will be updated (if necessary, you can turn off migrations completely or on a per table basis).
Once you have your data models defined, you can use web2py's CRUD system to handle all the data entry and searching. Just include these two lines (actually, they're already included in the 'welcome' scaffolding application):
from gluon.tools import Crud
crud = Crud(db)
And in a controller, define the following action:
def data():
return dict(form=crud())
That will expose a set of pre-defined URLs that will enable you to create, list, search, view, update, and delete records in any table.
Of course, if you don't like some of the default behavior, there are lots of ways to customize the CRUD forms/displays, or you can use some of web2py's other forms functionality to build a completely custom interface. And web2py is a full-stack framework, so it will be easy to add functionality to your app as your needs expand (e.g., access control, notifications, etc.).
Note, web2py requires no installation or configuration and has no dependencies, so it's very easy to distribute your app to other machines -- just zip up the entire web2py folder (which will include your app folder) and unzip it on another machine. It will run on *nix, Mac, and Windows (on Windows, you will either need to install Python or download the web2py Windows binary instead of the source version -- the Windows binary includes its own Python interpreter).
If you have any questions, there's a very helpful and responsive mailing list. You might also get some ideas from some existing web2py applications.
I normally use GTK+ that has well documented Python bindings.
The biggest advantage is that you can use a fairly intuitive GUI editor (Glade) and automagically link callbacks to events (to be honest most other major graphical toolkits have this possibility too, like for example QT, but my perception is that GTK+ enjoys wider adoption in the Python community). EDIT: additionally GTK is used by Gnome and many other desktop environments (KDE uses QT though).
That said, if all you need is truly just data insertion from a trusted person, you could use something already done like SQLite manager (it's a FireFox plugin).
Radically alternative solution: use django and you can literally pass from reading the tutorial to have your app up and running in a matter of hours, inclusive of user authentications, back-end administrative interface etc. (your project is exactly what I did with it to allow my wife to insert expenses in our family budget).
Django is written in Python and you can use SQLite as a back-end.