What are the most popular open source ODBC interfaces for python? - python

I was wondering what are the most popular open source obdc/database connection libraries are.
I've heard of pyodbc, but I wasn't sure how widely used it was.
Thanks

If you use Windows, then in popular Active State distribution you will find odbc module. I think it is part of pywin32 package. Of course pyodbc will be better if you do not use MS Windows. All you have to do is:
import odbc
connection = odbc.odbc('dsnname/user/passwd')
While pydobc connect string looks different you can made your program work with both libraries:
if '/' in connect_string:
import odbc
# dsnname/user/password
_CONN = odbc.odbc(connect_string)
elif connect_string.startswith('Driver='):
import pyodbc
# Driver={PostgreSQL};Server=db-test;Port=5435;Database=dbname;Uid=user;Pwd=password;
_CONN = pyodbc.connect(connect_string)

I would also like to point out turbodbc. It is open source, actively maintained, compatible with Python 2 and 3, and available for Linux, OSX, and Windows. You might want to give it a shot since it often improves on pyodbc's performance due to the use of bulk operations. It also offers built-in NumPy support, if you are into that kind of thing. Check out the official documentation, in particular the getting started guide.

http://wiki.python.org/moin/ODBC details some of the ODBC solutions for Python. pyodbc seems to be your best option for a few reasons:
It appears to be the only open source option, having an MIT style license
It's cross-platform
Sadly, I have no way of providing you much further information because Google Code is down, and that's where pyodbc is hosted (along with many others). It seems to be the closest to a standard solution, however.
Update
Google Code is back up and it seems to me from the project site that pyodbc is very actively contributed to and maintained, and it seems to be keeping up with Python, as a 3.x port is coming soon.

Related

is PyMySQL well supported? and is it slower than than MySQLdb?

I'm building a python3 app which requires high-speed connections. Being a pure python library, will PyMySQL be significantly slower than the C based MySQLdb in connecting and executive queries?
Is PyMySQL well supported for future versions of python?
If not, can anyone suggest a reliable alternative?
PyMySQL is implemented in pure Python while MySQLdb is C extension. PyMySQL is easy to install (and some times the only way) in some system while MySQLdb sometimes give problems.
But both are just MySQL python connectors.
Why PyMySQL? MySQL is an immensely popular RDBMS, and you usually need
to talk to it when writing a web application in Python. The defacto
standard, MySQLdb, is a C extension module that has a reputation of
being difficult to compile, especially if you're on a Mac (like I am).
Additionally, end-users need to wait for new binaries to be compiled
for each new release of Python, and MySQLdb will never run on Jython,
IronPython, or PyPy (without something like cpyext or IronClad). We
also maintain 100% compatibility between Python 2 and Python 3, so all
advancements made on the 2.x trunk will be immediately available on
Python 3.
We are developing a drop-in replacement for MySQLdb that "just works"
without the hassle of compiling and installing C extensions and
without worrying what platform you're on.
Source https://code.google.com/p/pymysql/wiki/Goals
Many projects use PyMySQL because it supports Python 3 (3.3 and up) as well as Python 2. At the time of writing it has over 650 forks on Github and over 60 contributors. There are regular updates and patches to it every couple of months or so and it's popular in the Python community for it's relatively easy installation and Python 3 support. So all that said, yes, I would say it's well supported.
Now for speed: since it's written pure Python whereas MySQLdb is C, you can expect PyMySQL to be slower. For an in depth analysis I suggest a look at the openstack link at the bottom of this post. Based on their testing, PyMySQL was roughly 10x slower than MySQLdb. However, with DB calls you are mostly IO bound anyways so the advantage of PyMySQL supporting gevent or something similar for asynchronous IO may make up for the speed depending on your application.
Sources:
https://github.com/PyMySQL/PyMySQL
https://wiki.openstack.org/wiki/PyMySQL_evaluation

How to distribute a Python program with embedded Firebird SQL for Linux and Windows

Summary:
What is the best (easiest, most flexible, simplest) way to redistribute a Firebird SQL database with Python code in a way that end users can use it without going to the trouble of installing and maintaining Firebird?
Background (somewhat long-winded):
I've been trying to write a program that sifts through stock fundamentals and appraises different companies' stock based on those fundamentals and randomized weights. I noticed that after some length of time, the program seemed to just stall. I did use multi-threading here and there, and I considered dead/livelocks, but apart from just combing through the code and seeing if it makes sense, I can't debug that. I noticed I was also eating up a lot of RAM, since all this data was held in big Python dicts in memory. So I figured putting it in SQL databases would fix that.
After a few weeks, I got the code working again with SQLAlchemy and SQLite. Now the problem is that the appraisal function takes ten minutes (!) per stock. Multiplied by a total of twelve "genomes" competing initially, this would add up to around 200 hours. I started thinking maybe this had to do with SQLite's concurrency locks, or something along those lines, so I started trying to use Firebird, since it is the only other one I know that stores a database in a file.
Question elaboration:
Ideally, I would be able to just throw my code on a disk or a server, take it to another computer with Python on it, and run everything out of the box. That's doable with SQLite. Is it possible with Firebird? I know that there is a separate embedded package for Windows, but that Linux only has the libfbembed library that ships with the Classic server. The docs said that Linux always requires some version of the Firebird server proper to be installed.
Will end users need to do any database administration to make that work; that is, will they need to set up users and such manually, as if I had just given them an fdb file and told them to figure out the rest? Or is it enough to install the basic packages for Firebird? Will I ever be able to get something close to the simplicity of SQLite in redistributing Firebird databases? Is there any special syntax I need to pass to SQLalchemy/FDB/Kinterbasedb to use an embedded server? (I could not find anything about this on either SQLalchemy or FDB's websites). Can my program run seamlessly on Linux and Windows, or will there need to be slightly different setups for each case?
Thanks in advance, anyone who can answer some of these questions.
Well, i only can give partial answers. But i think that'll be enough to start with.
Let's start with the Firebird embedded thing: As you have written - using linux as os you have to provide a full install. There is no other way.
HINT: Use the native tgz provided from firebird, not any package delivered from distribution - to avoid dependency hell.
Installing Firebird on Windows : The Windows Firebird Installer is mostly a 'click-through' thing. Luckily you can customize the installer: Install Firebird and look into doc\scripted-install.txt.
HINT: on Win7/8 don't install into %PROGRAM FILES% or %PROGRAM FILES (x86)%
Talking to firebird:
AFAIK you have two options, but for both i don't know if and how they will work with SQLAlchemy:
The fdb module which comes from firebird. When installing the fdb package make shure the appropriate fbclient.dll is in the search path.
the pyfirebirdsql module: https://github.com/nakagami/pyfirebirdsql/ which needs no dll or that. Partial drawback - it is not as fast as the fdb module as there is no real database engine. Personally i only use it for short lookups.
Using the fdb module you also can talk to the firebird services api - from creating over dropping databases to querying header statistics, ending up with backup/restore actions.
That should at least answer the question if the end user needs to perform any database administration.

Which Twitter wrapper libs support Python 3.x?

I'd like to add support for Twitter in a project built with Python 3.2, but I can't figure out which libraries support it.
I've done a bit of googling and I cannot come up with a clear answer; the closest thing I've found is Twython, mentioning in their readme that it's uber-experimental and you need a hacked version of python-oauth2, so it's basically hacks all the way down; surely there is something better out there?
If you didn't already know them, you may want to take a look at the Python Twitter Tools that turned out from my bit of googling: they are listed among Python 3 packages on PyPI (under the package name twitter), and seem to be actively developed on GitHub.
Twython library now (as time of answering th question) fully support Python 3.x, not to mention that it's easy to use and implement, let alone their extensive documentation
As their official documentation says:
Seamless Python 3 support!
OAuth 2 Application Only (read-only) Support
Support for Twitter’s Streaming API
I've been using it for some time and it's working great for me, so consider giving it another chance.
https://github.com/geduldig/TwitterAPI
This one supports Python 3.x.
Features include:
REST and Streaming API
Premium Search API
OAuth1 and OAuth2
Firewall proxy authentication

What is PyMySQL and how does it differ from MySQLdb? Can it affect Django deployment?

I just solved some problems in my Django 1.3 app by using PyMySQL instead of MySQLdb. I followed this tutorial on how to make the switch: http://web-eng-help.blogspot.com/2010/09/install-mysql-5-for-python-26-and.html
Now I want to know what PyMySQL actually is and how it is different from MySQLdb.
I am using it on localhost and will then upload it to some hosting.
Is it fine to use PyMySQL on localhost and on hosting whatever they provide? Since I have changed "MySQLdb" in base.py and introspection.py to "PyMySQL", will I need to upload it to the server after changing these files? Or as it is Django's files, since Django will be uploaded there already, does it not matter much?
PyMySQL and MySQLdb provide the same functionality - they are both database connectors. The difference is in the implementation where MySQLdb is a C extension and PyMySQL is pure Python.
There are a few reasons to try PyMySQL:
it might be easier to get running on some systems
it works with PyPy
it can be "greened" and works with gevent
The proper way to use it with Django is to import it and tell it to impersonate MySQLdb in your top-level file, usually manage.py. Put the following code at the very top of your manage.py (or whatever file you call when starting your server):
try:
import pymysql
pymysql.install_as_MySQLdb()
except ImportError:
pass
PyMySQL and MySQLdb are both database connectors for Python, libraries to enable Python programs to talk to a MySQL server.
You would normally never upload core Django files when deploying an app. If Django is working fine on your deployment server, you definitely don't need to change anything there. The DB driver is a step or two below the ORM even, and certainly none of the code you have written depends on which of these is in use.
Your first point:
According to pymysql wiki page:
MySQLdb, is a C extension module that has a reputation of being
difficult to compile, especially if you're on a Mac. Additionally,
end-users need to wait for new binaries to be compiled for each new
release of Python, and MySQLdb will never run on Jython, IronPython,
or PyPy (without something like cpyext or IronClad). We also maintain
100% compatibility between Python 2 and Python 3, so all advancements
made on the 2.x trunk will be immediately available on Python 3.
Your second point:
If django is working fine on your localhost, then it should be fine on
your development.
As per my experience I had difficulty in installing "MySQL-python" - (MySQLdb).
It made me search for more alternatives so I found pymysql, and it also got installed easily and worked in first go like a charm.
So I would suggest using pymysql only instead of MySQLdb.
I am starting with these topic, I only want to say an observation: PyMSQL has CPYTHON as a requirement(is optionall perhaps for performance https://pypi.org/project/PyMySQL/#requirements) to install, and Cpyhton depend on 'C', I tested Cpython and I had trouble when installed for the version of C too... then both implementation depend on 'C' [if you want performance], is the same thing for me...if the performance is not problem, perhaps PyMySQL is good without Cpython, free of 'C'. Perhaps we can start with PyMySQL alone with Python and switch Python to Cpython to gain in performance that could be a good thing we have all the things running and after that we can try to switch to Cpython,,,
We need to know the funcionality difference. Why in Django PyMySQL run better for you or give to you solutions for some problems, what problems? That is the important thing for change perhaps. MySQLdb perhaps depend directly from 'C' but PyMSQL indirectly thru Cpython(in case of similar performance), I prefer a direct dependence without limitations of jumps...Greetings.

Has anyone here tried using the iSeries Python port?

I found http://www.iseriespython.com/, which is a version of Python for the iSeries apparently including some system specific data access classes. I am keen to try this out, but will have to get approval at work to do so. My questions are:
Does the port work well, or are there limits to what the interpreter can handle compared with standard Python implementations?
Does the iSeries database access layer work well, creating usable objects from table definitions?
From what I have seen so far, it works pretty well. Note that I'm using iSeries Python 2.3.3. The fact that strings are natively EBCDIC can be a problem; it's definitely one of the reasons many third-party packages won't work as-is, even if they are pure Python. (In some cases they can be tweaked and massaged into working with judicious use of encoding and decoding.) Supposedly 2.5 uses ASCII natively, which would in principle improve compatibility, but I have no way to test this because I'm on a too-old version of OS/400.
Partly because of EBCDIC and partly because OS/400 and the QSYS file system are neither Unix-like nor Windows-like, there are some pieces of the standard library that are not implemented or are imperfectly implemented. How badly this would affect you depends on what you're trying to do.
On the plus side, the iSeries-specific features work quite well. It's very easy to work with physical files as well as stream files. Calling CL or RPG programs from Python is fairly painless. On balance, I find iSeries Python to be highly usable and very worthwhile.
Update (2012): A lot of work has gone into iSeries Python since this question was asked. Version 2.7 is now available, meaning it's up-to-date as far as 2.x versions go. A few participants of the forum are reasonably active and provide amazing support. One of them has gotten Django working on the i. As expected, the move to native ASCII strings solves a lot of the EBCDIC problems and greatly increases compatibility with third-party packages. I enthusiastically recommend iSeries Python 2.7 for anyone on V5R3 or later. (I still strongly recommend iSeries Python 2.3.3 for those who are on earlier versions of the operating system.)
Update (2021): Unfortunately, iSeriesPython is no longer maintained, and the old website and forum are gone. You can still get the software from its SourceForge repository, and it is still an amazingly useful and worthwhile asset for those who are stuck on old (pre-7.2) versions of the operating system. For those who are on 7.2 or newer, there is a Python for PASE from IBM, which should be considered the preferred way to run Python on the midrange platform. This version of Python is part of a growing ecosystem of open source software on IBM i.
It sounds like it is would work as expected. Support for other libraries might be pretty limited, though.
Timothy Prickett talks about some Python ports for the iSeries in this article:
http://www.itjungle.com/tfh/tfh041706-story02.html
Also, some discussion popped up in the Python mailing archives:
http://mail.python.org/pipermail/python-list/2004-January/245276.html
iSeriesPython is working very well.
We are usning it since 2005 (or earlier) in our Development and Production Environments as an utility language, for generating of COBOL source code, generating of PCML interfaces, sending SMS, validating/correcting some data ... etc.
With iSeriesPython you can access the iSeries database at 2 ways: using File400 and/or db2 module. You can execute OS/400 commands and you can work with both QSYS.LIB members and IFS stream files.
IMHO, iSeries Python is very powerful tool, more better than REXX included with iSeries.
Try it!
I got permission to install iSeries Python on a box about 3 years ago. I found that it worked pretty much as advertised. I contacted the developer and he was very good about answering questions. However, before I could think about using it in production, I had to approach the developer regarding a support contract. That really isn't his gig, so he said no and we scrapped the idea. The main limitation I found is that it is several releases behind Python on other platforms.
I have also had very good experience with Jython on the iSeries. Java is completely supported on the iSeries. Theoretically, everything you can do in RPG on the iSeries, you can do in Java, which means you can do it in Jython. I was sending email from an AS/400 (old name for iSeries) via JPython (old name for Jython) and smtplib.py in 1999 or 2000.
Another place to look is on the mailing list MIDRANGE-L or search the archives for the list at midrange.com. I know they have talked about this a while back.

Categories

Resources