At my work, we are using different SQL and NOSQL database types (i.e postgres, mysql, sqlite, mongodb, arangodb).
As we are creating lots of packages communicating with a database to retrieve some data. Recreating the CRUD functionality (+ methods to create schema => create table/collection/db) for each package make it harder to maintain the whole pipeline while scaling up.
In order to solve this situation more easily, we decided to create a database plugins API that allows us to abstract CRUD methods so you don't care about the backend database being used anymore on the backend side. All packages that need to communicate with a database will just wrap the plugins CRUD methods and add their own package specific behaviour.
I'm pretty sure we are not the only having this kind of situation out there, so I 've been looking on the web to try to find if such a thing has already been done without finding anything interesting.
Could anyone refer me to anything that may do/try to do it?
Is there anything that already does it using Python?
Thanks
Related
I am developing a Cloud based data analysis tool, and I am using Django(1.10) for that.
I have to add columns to the existing tables, create new tables, change data-type of columns(part of data-cleaning activity) at the run time and can't figure out a way to update/reflect those changes, in run time, in the Django model, because those changes will be required in further analysis process.
I have looked into 'inspectdb' and 'syncdb', but all of these options would require taking the portal offline and then making those changes, which I don't want.
Please can you suggest a solution or a work-around of how to achieve this.
Also, is there a way in which I can select what database I want to work from the list of databases on my MySQL server, after running Django.
Django's ORM might not be the right tool for you if you need to change your schema (or db) online - the schema is defined in python modules and loaded once when Django's web server starts.
You can still use Django's templates, forms and other libraries and write your own custom DB access layer that manipulates a DB dynamically using python.
I'm working on a new project written in Python with Flask & SQLAlchemy. The latter seems like a decent ORM but doesn't seem to consider caching as its responsibility.
It provides some examples on how to leverage dogpile.cache # http://docs.sqlalchemy.org/en/latest/orm/examples.html#module-examples.dogpile_caching but this seems to offer no more than basic query results caching which is really only valuable for read-only models.
What I am looking for is a real Level 2 Object Cache integrated into the ORM which store each object via its PK and ensures identity in a concurrently shared cache. I have experience with multiple caching libraries working with Java ORMs but I'm not able to find anything for SQLAlchemy. Does no such thing exist for Python yet?
I'm building a webapp and needed to choose between Django and Pyramid. I decided to go with Pyramid.
I understand Pyramid comes with its own authentication/authorization framework which looks nice. But I haven't seen anywhere in Pyramid where users/groups/permissions are defined. In Django these things come for free.
I'm using SQLAlchemy and was wondering if there are similar users/groups/permissions already built that I can import. I'd rather not define these objects/mappings and hash/salting of passwords myself.
Django's definitions of these things are pretty much all I need.
Can anyone point my to something I can use? Or do I need to roll my own?
Pyramid has a much more flexible authentication system. And yes, if you want something simple like Django's user/group/permission concept, then flexible might be scary.
Pyramid does not have a "User" object, as it makes no assumptions about how you store your data or what ORM you use, therefore there isn't something for you like contrib.auth. You will need to hash/salt the passwords yourself using a library such as cryptacular or passlib, both found on PYPI.
As far as wanting user/group/permissions within Pyramid's system, this is achievable pretty simply by defining a RootFactory that has an __acl__ that maps groups to permissions. Permissions are assigned to views, thus are pretty static usually. If you'd like the groups (what Pyramid calls "principals") to be dynamic that is also achievable.
I'd suggest looking at the Pyramid wiki2 tutorial, as well as the shootout demo.
There are also a couple third-party packages for assisting with authorization within Pyramid if you plan to be using SQLAlchemy. apex is a more full stack solution, and ziggurat_foundations is a lower-level layer above SQLAlchemy to help you set up users and groups for your application.
Your question is fairly high level and authorization is a "hard problem", so I'll stop here and avoid regurgitating the tutorials and resources that already exist from the Pyramid tutorials to several third-party examples. If you have any specific questions please feel free to ask those in another question.
which is the best back end for python applications and what is the advantage of using sqlite ,how it can be connected to python applications
What do you mean with back end? Python apps connect to SQLite just like any other database, you just have to import the correct module and check how to use it.
The advantages of using SQLite are:
You don't need to setup a database server, it's just a file
No configurations needed
Cross platform
Mainly, desktops applications are the ones that take real advantage of this. For web apps, SQLite is not recommended, since the file containing the data, is easily readable (lacks any kind of encryption), and when the web server lacks special configuration, the file is downloadable by anyone.
Django, Twisted, and CherryPy are popular Python "Back-Ends" as far as web applications go, with Twisted likely being the most flexible as far as networking is concerned.
SQLite can, as has been previously posted, be directly interfaced with using SQL commands as it has native bindings for Python, or it can be accessed with an Object Relational Manager such as SQLObject (another Python library).
As far as performance is concered, SQLite is fairly scalable and should be able to handle most use cases that don't require a seperate database server (nothing enterprise level). An additional benefit of SQLite is that the database is self-contained in a single file allowing for easy backup while remained a common enough format that multiple applications can access the data. A word of advice on using SQLite with Python, however, is that you may run into issues with threading (in the past most of the bindings for SQLite were not thread-safe, although this may have changed over time).
The language you are using at the application layer has little to do with your database choice underneath. You need to examine the advantages of other DB packages to get an idea of what you want.
Here are some popular database packages for cheap or free:
ms sql server express, pg/sql, mysql
If you mean "what is the best database?" then there's simply no way to answer this question. If you just want a small database that won't be used by more than a handful of people at a time, SQLite is what you're looking for. If you're running a database for a giant corporation serving thousands, you're probably looking for Oracle. In between those, you have MySQL, PostgreSQL, SQL Server, db2, and probably more.
If you're familiar with one of those, that may be the best to go with from a practical standpoint. If you're doing a typical webapp, my advice would be to go with MySQL or PostgreSQL as they're free and well supported by just about any ORM you could think of (my personal preference is towards PostgreSQL, but I'm not experienced enough with either of these to make a good argument one way or another). If you do go with one of those two, my recommendation is to use storm as the ORM.
(And yes, there are free versions of SQL Server and Oracle. You won't have as many choices as far as ORMs go though)
I'd like to use the Python version of App Engine but rather than write my code specifically for the Google Data Store, I'd like to create my models with a generic Python ORM that could be attached to Big Table, or, if I prefer, a regular database at some later time. Is there any Python ORM such as SQLAlchemy that would allow this?
Technically this wouldn't be called an ORM (Object Relational Mapper) but a DAL (Database Abstraction Layer). The ORM part is not really interesting for AppEngine as the API already takes care of the object mapping and does some simple relational mapping (see RelationProperty).
Also realize that a DAL will never let you switch between AppEngine's datastore and a "normal" sql database like mysql because they work very differently. It might let you switch between different key value stores, like reddis, mongo or tokyo cabinet. But as they all have such very different characteristics I would really think twice before using one.
Lastly, the DAL traditionally sits on top of the DB interface, but with AppEngine's api you can implement your own "stubs" that basically let you use other storage backends on their api. The people at Mongo wrote one for MongoDB which is very nice. And the dev_appserver comes with a filesystem-based one.
And now to the answer: yes there is one! It's part of web.py. I haven't really tried if for the reasons above, so I can't really say if it's good.
PS. I know Ruby has a nice DAL project for keyvalue stores in the works too, but I can't find it now... Maybe nice to port to Python at some point.
Nowadays they do since Google has launched Cloud SQL