I am aware a similiar question exists. It has not been marked as answered yet and I tried all suggestions so far. I am also not native speaker, please excuse spelling mistakes.
I have written a small class in python to interact with a SQL Database.
Now I want to be able to connect to either SQL or MYSQL Database with the same functionalities.
It would be perfect for me to just change the connection type in the instance initiation to keep my class maintainable. Else I would need to create a seconde class using for example the mysql.connector, which would result in two classes with nearly the same structure and content.
This is how I tried to use pyodbc so far:
conn = pyodbc.connect('Driver={SQL Server};'
'Server=xyzhbv;'
'Database=Test;'
'ENCRYPT=yes;'
'UID=root;'
'PWD=12345;')
Please note that I changed all credentials.
What do I need to change to use pyodbc for MySQL?
Is that even possible?
Or
Can I use both libaries within one class without confusing? (they share function names)
Many Thanks for any help.
Have a great day.
I am trying to fetch a clob data from Oracle server and the connection is made through ssh tunnel.
When I tried to run the following code:
(id,clob) = cursor.fetchone()
print('one fetched')
clob_data = clob.read()
print(clob_data)
the execution freezes
Can someone help me with what's wrong here because I have referred to cx_oracle docs and the example code is just the same.
It is possible that there is a round trip taking place that is not being handled properly by the cx_Oracle driver. Please create an issue here (https://github.com/oracle/python-cx_Oracle/issues) with a few more details such as platform, Python version, Oracle database/client version, etc.
You can probably work around the issue, however, by simply returning the CLOBs as strings as can be seen in this sample: https://github.com/oracle/python-cx_Oracle/blob/master/samples/ReturnLobsAsStrings.py.
I am creating a simple python program that needs to search a somewhat large database ( ~40 tables, 6 Million or so rows all together ).
Currently, I use MySQLdb to query my local MySQL database then I have some other python function that work with the data and returns some statistics and other stuff. I would like to share this with others that do not want to construct their own database. At this point the database is used for queries only.
How best can I share the database and python program as a "package". Do I have to give up on the SQL method and switch to some sort of text file database or is there an easier way... sqlite maybe?
If the answer is sqlite how do I go about exporting my current SQL database to the sqlite database? Is there any gotchas I should know about?
Currently I use simple SELECT quarries with a few WHERE statements to locate the data I need. I am afraid that if I switched to text based database I would end up having to write a large amount of code to make these queries.
Thank you in advance for any suggestions.
EDIT
So I wrote my little python program with an sqlite3 database and it works perfectly.
I ended up using using a shell script called mysql2sqlite.sh found here to convert my MySQL database to sqlite. It worked flawlessly.
I only had to change 2 lines of python code. Awesome.
My little program runs in osx, windows and linux (ubuntu and redhat) without any changes or hassle. Thanks for the advise!
Converting your database could be as easy as an sql-dump and then an import, depending on the complexity of your db. See this post for strategies and alternatives.
I facing an atypical conversion problem. About a decade ago I coded up a large site in ASP. Over the years this turned into ASP.NET but kept the same database.
I've just re-done the site in Django and I've copied all the core data but before I cancel my account with the host, I need to make sure I've got a long-term backup of the data so if it turns out I'm missing something, I can copy it from a local copy.
To complicate matters, I no longer have Windows. I moved to Ubuntu on all my machines some time back. I could ask the host to send me a backup but having no access to a machine with MSSQL, I wouldn't be able to use that if I needed to.
So I'm looking for something that does:
db = {}
for table in database:
db[table.name] = [row for row in table]
And then I could serialize db off somewhere for later consumption... But how do I do the table iteration? Is there an easier way to do all of this? Can MSSQL do a cross-platform SQLDump (inc data)?
For previous MSSQL I've used pymssql but I don't know how to iterate the tables and copy rows (ideally with column headers so I can tell what the data is). I'm not looking for much code but I need a poke in the right direction.
Have a look at the sysobjects and syscolumns tables. Also try:
SELECT * FROM sysobjects WHERE name LIKE 'sys%'
to find any other metatables of interest. See here for more info on these tables and the newer SQL2005 counterparts.
I've liked the ADOdb python module when I've needed to connect to sql server from python. Here is a link to a simple tutorial/example: http://phplens.com/lens/adodb/adodb-py-docs.htm#tutorial
I know you said JSON, but it's very simple to generate a SQL script to do an entire dump in XML:
SELECT REPLACE(REPLACE('SELECT * FROM {TABLE_SCHEMA}.{TABLE_NAME} FOR XML RAW', '{TABLE_SCHEMA}',
QUOTENAME(TABLE_SCHEMA)), '{TABLE_NAME}', QUOTENAME(TABLE_NAME))
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_TYPE = 'BASE TABLE'
ORDER BY TABLE_SCHEMA
,TABLE_NAME
As an aside to your coding approach - I'd say :
set up a virtual machine with an eval on windows
put sql server eval on it
restore your data
check it manually or automatically using the excellent db scripting tools from red-gate to script the data and the schema
if fine then you have (a) a good backup and (b) a scripted output.
I've got mssql 2005 running on my personal computer with a database I'd like to run some python scripts on. I'm looking for a way to do some really simple access on the data. I'd like to run some select statements, process the data and maybe have python save a text file with the results.
Unfortunately, even though I know a bit about python and a bit about databases, it's very difficult for me to tell, just from reading, if a library does what I want. Ideally, I'd like something that works for other versions of mssql, is free of charge and licensed to allow commercial use, is simple to use, and possibly works with ironpython.
Everyone else seems to have the cPython -> SQL Server side covered. If you want to use IronPython, you can use the standard ADO.NET API to talk to the database:
import clr
clr.AddReference('System.Data')
from System.Data.SqlClient import SqlConnection, SqlParameter
conn_string = 'data source=<machine>; initial catalog=<database>; trusted_connection=True'
connection = SqlConnection(conn_string)
connection.Open()
command = connection.CreateCommand()
command.CommandText = 'select id, name from people where group_id = #group_id'
command.Parameters.Add(SqlParameter('group_id', 23))
reader = command.ExecuteReader()
while reader.Read():
print reader['id'], reader['name']
connection.Close()
If you've already got IronPython, you don't need to install anything else.
Lots of docs available here and here.
I use SQL Alchemy with cPython (I don't know if it'll work with IronPython though). It'll be pretty familiar to you if you've used Hibernate/nHibernate. If that's a bit too verbose for you, you can use Elixir, which is a thin layer on top of SQL Alchemy. To use either one of those, you'll need pyodbc, but that's a pretty simple install.
Of course, if you want to write straight SQL and not use an ORM, you just need pyodbc.
pyodbc comes with Activestate Python, which can be downloaded from here. A minimal odbc script to connect to a SQL Server 2005 database looks like this:
import odbc
CONNECTION_STRING="""
Driver={SQL Native Client};
Server=[Insert Server Name Here];
Database=[Insert DB Here];
Trusted_Connection=yes;
"""
db = odbc.odbc(CONNECTION_STRING)
c = db.cursor()
c.execute ('select foo from bar')
rs = c.fetchall()
for r in rs:
print r[0]
I also successfully use pymssql with CPython. (With and without SQLAlchemy).
http://adodbapi.sourceforge.net/ can be used with either CPython or IronPython. I have been very pleased with it.
PyPyODBC (http://code.google.com/p/pypyodbc) works under PyPy, Ironpython and CPython.
This article shows a Hello World sample of accessing mssql in Python.
PyPyODBC has almostly same usage as pyodbc, as it can been seen as a re-implemenation of the pyodbc moudle. Because it's written in pure Python, it can also run on IronPython and PyPy.
Actually, when switch to pypyodbc in your existing script, you can do this:
#import pyodbc <-- Comment out the original pyodbc importing line
import pypyodbc as pyodbc # Let pypyodbc "pretend" the pyodbc
pyodbc.connect(...) # pypyodbc has 99% same APIs as pyodbc
...
I've used pymssql with standard python and liked it. Probably easier than the alternatives mentioned if you're just looking for basic database access.
Sample code.
If you are want the quick and dirty way with CPython (also works for 3.X python):
Install PYWIN32 after you install python http://sourceforge.net/projects/pywin32/files/pywin32/
Import the following library:
import odbc
I created the following method for getting the SQL Server odbc driver (it is slightly different in naming depending on your version of Windows, so this will get it regardless):
def getSQLServerDriver():
key = winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE, r"SOFTWARE\ODBC\ODBCINST.INI")
sqlServerRegExp = re.compile('sql.*server', re.I | re.S)
try:
for i in range(0, 2048):
folder = winreg.EnumKey(key, i)
if sqlServerRegExp.match(folder):
return folder.strip()
except WindowsError:
pass
Note: if you use the above function, you'll need to also import these two libraries: winreg and re
Then you use the odbc API 1 information as defined here: http://www.python.org/dev/peps/pep-0248/
Your connection interface string should look something like this (assuming you are using my above method for getting the ODBC driver name, and it is a trusted connection):
dbString = "Driver={SQLDriver};Server=[SQL Server];Database=[Database Name];Trusted_Connection=yes;".replace('{SQLDriver}', '{' + getSQLServerDriver() + '}')
This method has many down sides. It is clumsy because of only supporting ODBC API 1, and there are a couple minor bugs in either the API or the ODBC driver that I've run across, but it does get the job done in all versions of CPython in Windows.