I facing an atypical conversion problem. About a decade ago I coded up a large site in ASP. Over the years this turned into ASP.NET but kept the same database.
I've just re-done the site in Django and I've copied all the core data but before I cancel my account with the host, I need to make sure I've got a long-term backup of the data so if it turns out I'm missing something, I can copy it from a local copy.
To complicate matters, I no longer have Windows. I moved to Ubuntu on all my machines some time back. I could ask the host to send me a backup but having no access to a machine with MSSQL, I wouldn't be able to use that if I needed to.
So I'm looking for something that does:
db = {}
for table in database:
db[table.name] = [row for row in table]
And then I could serialize db off somewhere for later consumption... But how do I do the table iteration? Is there an easier way to do all of this? Can MSSQL do a cross-platform SQLDump (inc data)?
For previous MSSQL I've used pymssql but I don't know how to iterate the tables and copy rows (ideally with column headers so I can tell what the data is). I'm not looking for much code but I need a poke in the right direction.
Have a look at the sysobjects and syscolumns tables. Also try:
SELECT * FROM sysobjects WHERE name LIKE 'sys%'
to find any other metatables of interest. See here for more info on these tables and the newer SQL2005 counterparts.
I've liked the ADOdb python module when I've needed to connect to sql server from python. Here is a link to a simple tutorial/example: http://phplens.com/lens/adodb/adodb-py-docs.htm#tutorial
I know you said JSON, but it's very simple to generate a SQL script to do an entire dump in XML:
SELECT REPLACE(REPLACE('SELECT * FROM {TABLE_SCHEMA}.{TABLE_NAME} FOR XML RAW', '{TABLE_SCHEMA}',
QUOTENAME(TABLE_SCHEMA)), '{TABLE_NAME}', QUOTENAME(TABLE_NAME))
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_TYPE = 'BASE TABLE'
ORDER BY TABLE_SCHEMA
,TABLE_NAME
As an aside to your coding approach - I'd say :
set up a virtual machine with an eval on windows
put sql server eval on it
restore your data
check it manually or automatically using the excellent db scripting tools from red-gate to script the data and the schema
if fine then you have (a) a good backup and (b) a scripted output.
Related
I am creating a simple python program that needs to search a somewhat large database ( ~40 tables, 6 Million or so rows all together ).
Currently, I use MySQLdb to query my local MySQL database then I have some other python function that work with the data and returns some statistics and other stuff. I would like to share this with others that do not want to construct their own database. At this point the database is used for queries only.
How best can I share the database and python program as a "package". Do I have to give up on the SQL method and switch to some sort of text file database or is there an easier way... sqlite maybe?
If the answer is sqlite how do I go about exporting my current SQL database to the sqlite database? Is there any gotchas I should know about?
Currently I use simple SELECT quarries with a few WHERE statements to locate the data I need. I am afraid that if I switched to text based database I would end up having to write a large amount of code to make these queries.
Thank you in advance for any suggestions.
EDIT
So I wrote my little python program with an sqlite3 database and it works perfectly.
I ended up using using a shell script called mysql2sqlite.sh found here to convert my MySQL database to sqlite. It worked flawlessly.
I only had to change 2 lines of python code. Awesome.
My little program runs in osx, windows and linux (ubuntu and redhat) without any changes or hassle. Thanks for the advise!
Converting your database could be as easy as an sql-dump and then an import, depending on the complexity of your db. See this post for strategies and alternatives.
I have a requirement where I need to insert the postgres data into mysql. Suppose I have user table in postgres. I have user table also in mysql. I tried to do something like this:
gts = 'cd '+js_browse[0].js_path #gts prints correct folder name/usr/local/myfolder_name
os.system(gts)
gts_home = 'export GTS_HOME='+js_browse[0].js_path
os.system(gts_home)
tt=gts+'&& sh bin/admin.sh User --input-dir /tmp/import'
#inside temp/import i import store my postgres user table data
#bin is the folder inside myfolder_name
In mysql if I use the command it works perfectly fine:
cd /usr/local/myfolder_name
bin/admin.sh User -account=1 user=hamid -create'
I am unable to store data inside mysql this way. Any help shall be appreciated.
You don't really give us much information. And why would go from postgres to mysql?
But you can use one of these tools - I have seen people talk good about them
pg2mysql or pgs2sql
Hope it works out.
PostgreSQL provides possibility to dump data into the CSV format using COPY command.
The easiest path for you will be to spend time once to copy schema objects from PostgreSQL to MySQL, you can use pg_dump -s for this on the PostgreSQL side. IMHO, it will be the biggest challenge to properly move schemas.
And then you should import CSV-formatted data dumps into the MySQL, check this for reference. Scrolling down to the comments you'll find recipes for Windows also. Something like this should do the trick (adjust parameters accordingly):
LOAD DATA LOCAL INFILE C:\test.csv
INTO TABLE tbl_temp_data
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n';
We're currently working on a python project that basically reads and writes M2M data into/from a SQLite database. This database consists of multiple tables, one of them storing current values coming from the cloud. This last table is worrying me a bit since it's being written very often and the application runs on a flash drive.
I've read that virtual tables could be the solution. I've thought in converting the critical table into a virtual one and then link its contents to a real file (XML or JSON) stored in RAM (/tmp for example in Debian). I've been reading this article:
http://drdobbs.com/database/202802959?pgno=1
that explains more or less how to do what I want. It's quite complex and I think that this is not very doable using Python. Maybe we need to develop our own sqlite extension, I don't know...
Any idea about how to "place" our conflicting table in RAM whilst the rest of the database stays in FLASH? Any better/simpler approach about how take the virtual table way under Python?
A very simple, SQL-only solution to create a in-memory table is using SQLite's ATTACH command with the special ":memory:" pseudo-filename:
ATTACH DATABASE ":memory:" AS memdb;
CREATE TABLE memdb.my_table (...);
Since the whole database "memdb" is kept in RAM, the data will be lost once you close the database connection, so you will have to take care of persistence by yourself.
One way to do it could be:
Open your main SQLite database file
Attach a in-memory secondary database
Duplicate your performance-critical table in the in-memory database
Run all queries on the duplicate table
Once done, write the in-memory table back to the original table (BEGIN; DELETE FROM real_table; INSERT INTO real_table SELECT * FROM memory_table;)
But the best advice I can give you: Make sure that you really have a performance problem, the simple solution could just as well be fast enough!
Use an in-memory data structure server. Redis is a sexy option, and you can easily implement a table using lists. Also, it comes with a decent python driver.
Looks like there are several options:
mysqldump + rsync - can this be done for only specific data from an existing table, and not a whole table?
An insert to a federated table - this seems pretty untested and unknown at this point...
A Python script (pull into memory from A, then insert into B) - this would probably be pretty slow...
What kind of data warrants what method?
You also have another option -- mysql replication!
Can you not extract that single table into its own database and replicate just that to the second server (or as many servers as you like)?
I come from a PHP background where MySQL easily works from PHP, and don't the process for getting MySQL to work from Python. From all the research i did and reading of similar yet non-exact questions, it seems to me that there are different ways to achieve this, which makes it even harder for me to wrap my head around. So far I have MySQL-python-1.2.3 installed for python 2.7.1 in Windows XP 32Bit. Can anyone give me an overview of what is necessary to get MySQL working from Python in Windows, or even what is next after my steps all the way to fetching a table row? Thanks in advance.
UPDATE:
#Mahmoud, using your suggestion i have triggered the following:
If you just want to use the DBAPI, then here's a simple snippet explaining how to issue a SELECT query.
import MySQLdb
db = MySQLdb.connect(host="host", user="username", passwd="your-pass", db="the-db-name")
To perform a query, you first need a cursor, and then you can execute queries on it:
cursor = db.cursor()
max_age = 42
cursor.execute("""SELECT name FROM employees WHERE age < %s""", (max_age,))
print cursor.fetchone()
However, you most likely want to use an ORM, I recommend SQLAlchemy. It essentially trivializes database interaction by providing a super-powerful abstraction layer.