Is there any library or tool to perform database migrations in python without SQLAlchemy? I am using a PostgreSQL database and the queries are raw SQL. I use psycopg2 for the database connection.
In short, "yes."
There are numerous possibilities. The one you choose will depend on doing some homework to determine which one best suits your needs.
https://github.com/amacneil/dbmate
https://pypi.org/project/yoyo-migrations/
https://pypi.org/project/alembic/
Suggest when asking questions that can be answered with a google search, to be more specific about your situation, to help answerers to understand why Google couldn't answer your question in the first page of results.
If you can bring yourself to do your db migrations outside of the CLR, I'd strongly recommend considering FlyWay. It has been around forever, it's supported by RedGate, and the community version is free and open source. It has built-in support for all the major db platforms and installs like any other utility you'd use. For a Python app you'd do all of your db migrations via shell scripts or process calls instead of directly in the code.
My team settled on this tech because:
Yoyo is pretty immature and buggy for us
We don't use SQLAlchemy's OR framework
Alembic's syntax and framework seems pretty heavyweight for us
Under FlyWay's model we just write SQL scripts and check them into a directory. It's really lightweight.
It's been proven to work nicely in clustered environments
Give it a look.
Related
I am leaning Python programming language. I have no problems with Python. I read Python official docs, and I can write small programs in Python. I want to familiarize myself with mysql database because it is useful in learning software development concepts. I've installed mysql database and Django on my computer. I have Ubuntu 14.04 and python 3.4 installed. I've configured Django settings to use mysql database. I tested Django connection to mysql db and all things work properly.
I am a complete newbie with web development. I didn't create my own website and I didn't start developing any web application.
My purpose currently is to master creation of mysql database and tables, making changes/migrations/queries, using Django models and Python.
Is it reasonable/possible to use Django ORM for work with mysql database without simultaneous development of a web application/local application? As I've said, I don't have my own website. I want just to try using mysql and Django together on my computer in order to get deeper knowledge as to Django and mysql in this respect.
My final purpose is development in Python, including work with mysql database.
Mysql without Python and Django is of no use for me.
If you want to learn and use MySQL, then start without anything above it - no Django, no ORM, not even a Python script. Learn to configure your mysql server (the server process I mean - doesn't have to be on a distinct computer), to work with the command-line mysql client (database creation, tables creations / modifications, adding/updating/deleting rows, and, most important, doing simple and complex queries).
While you're at it, learn about proper relational data modeling (normalisations etc) so you fully understand how to design your schemas.
Once you're confortable with this, spend some time (should be quite fast at this point) learning to do the same things from python scripts with your python's version mysql connector.
Then if you want to learn web development and Django, well go for it. The ORM is quite easy to use when you already have a good understanding of what happens underneath, so by that time you shouldn't have much problems with this part. What you'll still have to learn are the HTTP protocol (trying to do web programming without understanding the HTTP protocol is like trying to win a car race without knowing how to drive a car - it's not technically impossible but it might end up being very painful experience), then front-end stuff (html/css/javascript) and finally the views / templates parts of Django (which should be easy once you know HTTP and html).
You can of course jump right in into Django, but you will probably fight with way too many concepts at once and end up spending twice more time to figure out how everything works and why it works that way.
I'm currently exploring using python to develop my server-side implementation. I've decided to use SQLAlchemy for database stuff.
What I'm not currently to sure about is how it should be set up so that more than one developer can work on the project. For the code it is not a problem but how do I handle the database modifications? How do the users sync databases and how should potential data be set up? Should/can each developer use their own sqlite db for development?
For production postgresql will be used but the developers must be able to work offline.
You can use Alembic to manage your DB structure. Keep your migrations under version control in order to ensure that it's easy for developers to keep in sync with changes.
As regards data itself, you can either work with a set of fixtures, or you could write some Fabric commands to import/export Postgres dumps, which you could also keep under version control if necessary – depending on data size, you may not want to keep a large uncompressed SQL dump under version control (though it will compress well using git or hg), however. This aspect of your question depends on the size of your development team, and the size of your database.
Make sure you have a python programs or programs to fill databases with test data from scratch. It allows each developer to work from different starting points, but also test with the same environment.
I need to write a very light database (sqlite is fine) app that will initially be run locally on a clients windows PC but could, should it ever be necessary, be upgraded to work over the public interwebs without a complete rewrite.
My end user is not very technically inclined and I'd like to keep things as simple as posible. To that end I really want to avoid having to install a local webserver, however "easy" that may seem to you or I. Django specifically warns not to use it's inbuilt webserver in production so my two options seem to be...
a) Use django's built in server anyway while the app is running locally on windows and, if it ever needs to be upgraded to work over the net just stick it behind apache on a linux box somewhere in the cloud.
b) Use a framework that has a more robust built in web server from the start.
My understanding is that the only two disadvantages of django's built in server are a lack of security testing (moot if running only locally) and it's single threaded nature (not likely to be a big deal either for a low/zero concurrency single user app running locally). Am I way off base?
If so, then can I get some other "full stack" framework recommendations please - I strongly prefer python but I'm open to PHP and ruby based solutions too if there's no clear python winner. I'm probably going to have to support this app for a decade or more so I'd rather not use anything too new or esoteric unless it's from developers with some serious pedigree.
Thanks for your advice :)
Roger
I find Django's admin very easy to use for non-technical clients. In fact, that is the major consideration for my using Django as of late. Once set up properly, non-technical people can very easily update information, which can reflected on the front end immediately.
The client feels empowered.
Use Django. It's very simple for you to get started. Also, they have the best documentation. Follow the step by step app creating tutorial. Django supports all the databases that exist. Also, the built in server is very simple to use for the development and production server. I would highly recommend Django.
I am looking for a pure-python SQL library that would give access to both MySQL and PostgreSQL.
The only requirement is to run on Python 2.5+ and be pure-python, so it can be included with the script and still run on most platforms (no-install).
In fact I am looking for a simple solution that would allow me to write SQL and export the results as CSV files.
Two part answer:
A) This is absolutely possible.
B) Depending on your exact concerns, a pure-python may or may not be a good approach to your problem.
Explained:
The SqlAlchemy library comes with two components : the more popular "ORM" , and the "Core" which it sits on top of. Either one will let you write your SQL commands in the SqlAlchemy format (which is just Python); SqlAlchemy will then compile the statements to Mysql or PostgreSQL and connect to the appropriate database.
SqlAlchemy is a great library, and I recommend it for just about everything. While you do have to write your statements in their format, it's easy to pick up -- and you can switch to virtually any underlying database library you want... at any time. It's the perfect platform to use in any database project , whether or not you need to support multiple backends.
SqlAlchemy talks to the database via the standard DBAPI drivers, and does support multiple options for pure python, notably the pymysql and pypostgresql drivers ( http://docs.sqlalchemy.org/en/latest/core/engines.html#supported-dbapis )
As for writing csv, the standard library has you covered.
import csv
So the caveat?
The following may or may not apply to your situation:
Most higher level db modules in the python universe are still recommending mysql-python and psycopg - both of which are not pure-python and compile/link/configure against the installed database. This largely seems to be from a mix of API / integration concerns and the speed of the various pure-python packages compared to c-extensions when run under CPython.
There are pure-python drivers like I recommended, but most reviewers state that they're largely experimental. The pymysql authors claim stability and production readiness, some developers who blog have challenged that. As for what "stabile" or "experimental" means, that varies with the projects. Some have a changing API, others are incomplete, some are buggy.
You'd need to ensure that you can find pure-python drivers for each system that support the exact operations you need. This could be simple, or this could be messy. Whether you use SqlAlchemy or something else, you'll still face this concern when selecting a DBAPI driver.
The PyPy project ( pure-python python interpreter ) has a wiki listing the compatibility of various packages : https://bitbucket.org/pypy/compatibility/wiki/Home I would defer to them for specific driver suggestions. If PyPy is your intended platform, SqlAlchemy runs perfectly on it as well.
Are you looking for an ORM, or a single library that would allow you to write SQL statements directly and convert where there are differences?
I'm not sure whether psycopg2 is pure-python or not, but it certainly has bindings and works on all platforms. You'd still have to install at least psycopg2 to communicate with the PostgreSQL database as Python (as far as I know) doesn't ship with it natively.
From there, any additional ORM library you want would also need to be installed, but most are pure-python on-top of whatever backend they use.
Storm, Django, SQLAlchemy all have abstracted layers on top of their database layer - based on your description, Django is probably too large a framework for your needs (was for mine) but is a popular one, SQLAlchemy is a tried and true system - tho a bit clunky particularly if you have to deal with inheritance (in my opinion). I have heard that Storm is good, tho I haven't tested too much with it so I can't fully say.
If you are looking to mix and match (some tables in MySQL and some tables in PostgreSQL) vs. a single database that could be either MySQL or PostgreSQL, I've been working on an ORM called ORB that focuses more on object-oriented design and allows for multiple databases and relationships between databases. Right now it only supports PostgreSQL and Mongo, just cause I haven't needed MySQL, but I'd be up for writing that backend. The code for that can be found at http://docs.projexsoftware.com/api/orb
Use SQL-Alchemy. It will work with most database types, and certainly does work with postgres and MySQL.
i am new in python.before this i done some work in php with codeigniter framework.
now i want to develop website using python and sqlite3.
so which framework will you suggest me.
i also want to know IDE for sqlite.
Do you mean something similar to sqliteman or dbeaver? These are not not IDEs for SQLite, but instead, are GUI interfaces. There are more here: http://www.google.com/search?&q=sqlite+gui
3rd link down is another stack overflow question: What are good open source GUI SQLite database managers?
For Python web frameworks, a very popular full-stack framework is the Django Project
Check out web2py. It's a feature-rich full-stack framework, but it's also very easy to set up, learn, and use. It requires no installation or configuration, has no dependencies, and includes its own web server and web-based IDE/administrative interface (see demo). If you download the binary version for Windows or Mac, it even includes its own Python interpreter (so you don't need to have Python installed).
The database abstraction layer (DAL) works with 10 different database backends, including SQLite. It doesn't include its own SQLite IDE, though it does include a database administrative interface called appadmin, which allows you to view all database tables; view, insert, update, and delete records within each table; and import/export tables to CSV. You can also import and export the entire database to CSV. (Note, these features work for all database backends, not just SQLite.)
You will probably also find that you don't really need an IDE for SQLite when using web2py because the DAL takes care of everything for you. You specify all of your data models using the DAL syntax, and then the DAL automatically creates the SQLite database, creates all tables, and handles migrations when you changes your models.
If you have any questions, there's a friendly and responsive mailing list that will provide lots of help.