Develop Reference

How do i create a MYSQL database locally using Database folder downloaded from github

I know the above question is incomplete. Let me explain this in brief. I downloaded the repository from this github link https://github.com/datacharmer/test_db and as per instructions on readme file i tried to create a database locally but i got syntax error when i ran a command mysql < employees.sql . I tried from both windows cli and MYSQL clc. Can someone help me to create a mysql database using above github data.
Thanks in advance

The SQL queries in that repository are quite destructive, many starting with a DROP DATABASE, which can bite you if you're not paying attention.
Do yourself a favour and create the databases manually. Look at the .sql files and you will see the CREATE DATABASE and CREATE TABLE statements. Run them one by one. This will help you become accustomed to how to create databases and tables in MySQL.
At the bottom of the .sql files, you'll see lines that look like this:
SELECT 'LOADING departments' as 'INFO';
source load_departments.dump ;
What you are interested in is the file name that comes after source. Open those files and you will see the INSERT statements that populate the tables you created in the previous step. This will help you become accustomed to inserting records into your tables.
Once this is done, you will have a new database with tables and data for you to work with. Do not trust just any SQL or bash script with the contents of your database. Ever.

Export Django Database into YAML file

I know you can easily import a YAML file into a Django database (in order to populate the database before starting the project for instance) but how can I do the opposite (ie save the complete database into a single .yaml file).
I read there is a way to export one single table into a file:
YAMLSerializer = serializers.get_serializer("yaml")
yaml_serializer = YAMLSerializer()
with open("file.yaml", "w") as out:
yaml_serializer.serialize(SomeModel.objects.all(), stream=out)
but I need to do it on the complete database (which has many tables with complex relations between each ones).
I could write a script to do that for me, but I don't want to redo something which has probably been done already, and I wouldn't know how to do it the better way so that Django has no difficulties to read it after.
So far, I've been working on a SQLITE3 database engine.
Any ideas?

You need the dumpdata management command.
pip install pyyaml
python manage.py dumpdata --format=yaml > /path/to/dump_file.yaml

How to automatically move specific MySQL tables from one machine to another?

I have a MySQL database with tables in the form of "shard_0", "shard_1", "shard_2", etc.
These are virtual shards. Now I want to add another DB server and move the even-numbered shards ("shard_0", "shard_2", "shard_4", ...) to the new machine.
What is the best way to do that? There are many tables so ideally I wouldn't have to type out each table name individually but do something automatically. Perhaps something like:
# pseudo code
for i in range(n):
tablename = "shard_"+str(2*i)
# Move tablename to new machine
Thanks

I'd create a single (or perhaps multiple) mysqldump invocations, like so
print "mysqldump database",
for i in range(n):
print "shard_"+str(2*i),
Run this command in a shell, and move the dump file to the new machine, then run it there through mysql.
Then generate and run the "drop table" statements for the tables you have moved.

I'm not sure I see the problem, but if I got it right, you can use Python to generate the export SQL script, and the import one for the other machine.
That'll save you the trouble of doing it manually. As for your code snippet, I think the best way to go about migrating a database from a server to another one is using the engine's own capabilities.

how to generate various database dumps

I have a CSV file and want to generate dumps of the data for sqlite, mysql, postgres, oracle, and mssql.
Is there a common API (ideally Python based) to do this?
I could use an ORM to insert the data into each database and then export dumps, however that would require installing each database. It also seems a waste of resources - these CSV files are BIG.
I am wary of trying to craft the SQL myself because of the variations with each database. Ideally someone has already done this hard work, but I haven't found it yet.

SQLAlchemy is a database library that (as well as ORM functionality) supports SQL generation in the dialects of the all the different databases you mention (and more).
In normal use, you could create a SQL expression / instruction (using a schema.Table object), create a database engine, and then bind the instruction to the engine, to generate the SQL.
However, the engine is not strictly necessary; the dialects each have a compiler that can generate the SQL without a connection; the only caveat being that you need to stop it from generating bind parameters as it does by default:
from sqlalchemy.sql import expression, compiler
from sqlalchemy import schema, types
import csv
# example for mssql
from sqlalchemy.dialects.mssql import base
dialect = base.dialect()
compiler_cls = dialect.statement_compiler
class NonBindingSQLCompiler(compiler_cls):
def _create_crud_bind_param(self, col, value, required=False):
# Don't do what we're called; return a literal value rather than binding
return self.render_literal_value(value, col.type)
recipe_table = schema.Table("recipe", schema.MetaData(), schema.Column("name", types.String(50), primary_key=True), schema.Column("culture", types.String(50)))
for row in [{"name": "fudge", "culture": "america"}]: # csv.DictReader(open("x.csv", "r")):
insert = expression.insert(recipe_table, row, inline=True)
c = NonBindingSQLCompiler(dialect, insert)
c.compile()
sql = str(c)
print sql
The above example actually works; it assumes you know the target database table schema; it should be easily adaptable to import from a CSV and generate for multiple target database dialects.

I am no database wizard, but AFAIK in Python there's not a common API that would do out-of-the-box what you ask for. There is PEP 249 that defines an API that should be used by modules accessing DB's and that AFAIK is used at least by the MySQL and Postgre python modules (here and here) and that perhaps could be a starting point.
The road I would attempt to follow myself - however - would be another one:
Import the CVS nto MySQL (this is just because MySQL is the one I know best and there are tons of material on the net, as for example this very easy recipe, but you could do the same procedure starting from another database).
Generate the MySQL dump.
Process the MySQL dump file in order to modify it to meet SQLite (and others) syntax.
The scripts for processing the dump file could be very compact, although they might somehow be tricky if you use regex for parsing the lines. Here's an example script MySQL → SQLite that I simply pasted from this page:
#!/bin/sh
mysqldump --compact --compatible=ansi --default-character-set=binary mydbname |
grep -v ' KEY "' |
grep -v ' UNIQUE KEY "' |
perl -e 'local $/;$_=<>;s/,\n\)/\n\)/gs;print "begin;\n";print;print "commit;\n"' |
perl -pe '
if (/^(INSERT.+?)\(/) {
$a=$1;
s/\\'\''/'\'\''/g;
s/\\n/\n/g;
s/\),\(/\);\n$a\(/g;
}
' |
sqlite3 output.db
You could write your script in python (in which case you should have a look to re.compile for performance).
The rationale behind my choice would be:
I get the heavy-lifting [importing and therefore data consistency checks + generating starting SQL file] done for me by mysql
I only have to have one database installed.
I have full control on what is happening and the possibility to fine-tune the process.
I can structure my script in such a way that it will be very easy to extend it for other databases (basically I would structure it like a parser that recognises individual fields + a set of grammars - one for each database - that I can select via command-line option)
There is much more documentation on the differences between SQL flavours than on single DB import/export libraries.
EDIT: A template-based approach
If for any reason you don't feel confident enough to write the SQL yourself, you could use a sort of template-based script. Here's how I would do it:
Import and generate a dump of the table in all the 4 DB you are planning to use.
For each DB save the initial part of the dump (with the schema declaration and all the rest) and a single insert instruction.
Write a python script that - for each DB export - will output the "header" of the dump plus the same "saved line" into which you will programmatically replace the values for each line in your CVS file.
The obvious drawback of this approach is that your "template" will only work for one table. The strongest point of it is that writing such script would be extremely easy and quick.
HTH at least a bit!

You could do this - Create SQL tables from CSV files
or Generate Insert Statements from CSV file
or try this Generate .sql from .csv python
Of course you might need to tweak the scripts mentioned to suite your needs.

Copy whole SQL Server database into JSON from Python

I facing an atypical conversion problem. About a decade ago I coded up a large site in ASP. Over the years this turned into ASP.NET but kept the same database.
I've just re-done the site in Django and I've copied all the core data but before I cancel my account with the host, I need to make sure I've got a long-term backup of the data so if it turns out I'm missing something, I can copy it from a local copy.
To complicate matters, I no longer have Windows. I moved to Ubuntu on all my machines some time back. I could ask the host to send me a backup but having no access to a machine with MSSQL, I wouldn't be able to use that if I needed to.
So I'm looking for something that does:
db = {}
for table in database:
db[table.name] = [row for row in table]
And then I could serialize db off somewhere for later consumption... But how do I do the table iteration? Is there an easier way to do all of this? Can MSSQL do a cross-platform SQLDump (inc data)?
For previous MSSQL I've used pymssql but I don't know how to iterate the tables and copy rows (ideally with column headers so I can tell what the data is). I'm not looking for much code but I need a poke in the right direction.

Have a look at the sysobjects and syscolumns tables. Also try:
SELECT * FROM sysobjects WHERE name LIKE 'sys%'
to find any other metatables of interest. See here for more info on these tables and the newer SQL2005 counterparts.

I've liked the ADOdb python module when I've needed to connect to sql server from python. Here is a link to a simple tutorial/example: http://phplens.com/lens/adodb/adodb-py-docs.htm#tutorial

I know you said JSON, but it's very simple to generate a SQL script to do an entire dump in XML:
SELECT REPLACE(REPLACE('SELECT * FROM {TABLE_SCHEMA}.{TABLE_NAME} FOR XML RAW', '{TABLE_SCHEMA}',
QUOTENAME(TABLE_SCHEMA)), '{TABLE_NAME}', QUOTENAME(TABLE_NAME))
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_TYPE = 'BASE TABLE'
ORDER BY TABLE_SCHEMA
,TABLE_NAME

As an aside to your coding approach - I'd say :
set up a virtual machine with an eval on windows
put sql server eval on it
restore your data
check it manually or automatically using the excellent db scripting tools from red-gate to script the data and the schema
if fine then you have (a) a good backup and (b) a scripted output.