When I access an Sqlite database using Python 2.5, it takes too long, but if I access the same database by other ways (including Python 3.2) it takes much shorter. What's going on? (I need Python 2.5 and can't switch to Python 3.2)
It should go without saying that if you use an old version of a program, you don't get the performance improvements, bug fixes, and feature additions that were added later. Python 2.5 is slower, buggier, and less powerful than 2.7 (or 3.2) in many ways, and for the most part, the only solution is "Stop using 2.5", unless you want to track down the specific improvement in the changelogs and backport it to the 2.5 codebase.
But in this case, it's a lot easier, because sqlite3 is developed independently of Python (in fact, before 2.5, it didn't even come built in), as pysqlite. Here's some version history (as seen in the What's New documentation for 2.6 and 2.7):
Python 2.5: pysqlite 2.3.2
Python 2.6: pysqlite 2.4.1
Python 2.7: pysqlite 2.6.0
The latest version is 2.6.3. You can install it by using any of the usual means (pip, easy_install, downloading it from the website and following the instructions, downloading the Windows binary installers from the website and running them, etc.).
In fact, if you're building a package with setuptools/distribute, you may just be able to put in a requirement for >= 2.6.0, and pip install mypackage will automatically get the new version for 2.5 (and 2.6 users).
It's also possible that your problem is with the underlying C sqlite3 library, not the pysqlite wrapper. If you install sqlite3 and then build pysqlite from source, you can solve that too.
If you're not sure which version you have, you can check at runtime, because the module has a human-readable version attribute (and a version_info tuple just like the one in sys, so you can check version_info >= (2, 6)), and likewise sqlite_version and sqlite_version_info for the underlying C library.
In addition to abarnert's excellent answer, look at how often you are committing. Comitting is slow in SQLite and generally performance benefits if it is put off. I encountered this myself during some other testing I did and the performance difference of reducing the number of commits is enormous.
Related
I want to develop and test my project on the up-to-date version of Python 2.7 (say 2.7.18), but I want my project to be still fully usable on earlier versions of 2.7 (say 2.7.7). Setting up many variants of 2.7 locally or/and on CI for testing can be redundant.
So there are the following questions about compatibility of 2.7.X.
Can there be any changes in syntax which make code not working?
Can there be any changes in available standard imports, for example, can some imports from __future__ be unavailable in earlier versions?
Since I have to distribute compiled Python files (.pyc, compiled via py_compile module), I'm also wondering if there can be any changes in Python bytecode which block code execution in earlier versions.
I guess if all the answers are "no", I can develop and test my project only on a single 2.7 version without worries.
I've tried to search it but there is no success. Please share your experience and/or links.
UPD 1: I should have clearly said from the beginning that it's not my desire to use 2.7, it's a requirement from the environment.
At least Python 2.7.9 introduced massive changes to the 'ssl' module, so trying to use code using SSL for 2.7.18 on Python older than 2.7.9 will fail. So a clear "yes" to number 2.
In general compatbility for most projects works the other way round, use the oldest version you need to support and work upwards from old to new, not downwards from newer to older. I do not know of any software project that makes the guarantees in the other direction.
Note that Python 2.7 dropped out of support with 2.7.18, so unless you use a compatible version like PyPy (https://www.pypy.org/) your freshly developed project will run on outdated Python versions from the start.
If you want to provide a shrink wrapped product, maybe have a look at the usual solution for this like pyinstaller (https://www.pyinstaller.org/) or freeze (https://wiki.python.org/moin/Freeze)
The #3 may work, if you study the list of bytecode opcodes which do not change that much over time (https://github.com/python/cpython/commits/2.7/Include/opcode.h) but no idea if the on-disk format changed.
If I'm writing a python package in python 3.6, then how do I ensure my code can be downloaded and ran in other python 3 environments, like python 3.5?
What makes some python packages (e.g. Tensorflow) compatible with all python 3 minor versions, while other python packages (e.g. OpenAI Gym) only compatible with Python 3.5?
Finally: if my goal is to write code that is compatible for Python 3.5 and 3.6, then would it be better to just use a python 3.5 environment?
The glib-but-true answer:
Test your 3.6 code with those other versions of Python.
If you want to see if something runs correctly in environment X, there's no substitute for actually running it in environment X.
Tox
is a Python testing framework designed to do exactly this on your workstation.
Something like this is often part of a larger
continuous integration
framework or service, which might be hosted on a remote server.
Writing your code in the earliest syntax you need to support is always a good idea, but it's not enough.
You still need to test later versions, because functions or classes can change, and even bug-fixes can break your code if you were unwittingly depending on their behavior.
As for why some packages don't work under a specific minor version, the most likely reason is that they use a Python language feature that was introduced later.
(Possibly it's one of their dependencies that requires the language feature.)
This can include language syntax changes like
Python 3.5's
# matrix-multiplication operator,
all the way down to seeming trivia like
Python 3.1's
printing commas as thousands-separators,
which is still enough to raise an exception.
It's actually a bit more complicated than just
"supports version x or greater",
because there are some gaps in Python's history.
The most likely headache is the u'' Unicode literal syntax from Python 2.
It was removed in Python 3.0...
and then
restored in Python 3.3,
after its absence caused more grief than expected.
That one change means any script with a u'Unicode literal' could work under Python 2.7 and 3.3+ while not working under Python 3.0, 3.1, or 3.2.
The Python documentation is very good about keeping track of when a feature was introduced.
For instance, the first thing the
typing module
tells you is:
26.1. typing - Support for type hints
New in version 3.5.
A Ctrl-F style search for "new in" or "changed in" will generally turn up all the most important changes.
After going through hell trying to install the latest version of postgresql and psycopg2 today I'm going for a complete reinstall of Leopard.
I've been sticking with macpython 2.5 for the past year but now I'm considering macports even 2.6
For me it's most important for Twisted, PIL and psycopg2 to be working without a problem.
Can anyone give some guidelines for what version I should choose, based on experience?
Edit:
Ok I've decided to go without reinstalling the os. Hacked around to clean up the bad PostgresPlus installation and installed another one. The official python 2.6.1 package works great, no problem installing it alongside 2.5.2. Psycopg2 works. But as expected PIL wont compile.
I guess I'll be switching between the 2.5 from macports and the official 2.6 for different tasks, since I know the macports python has it's issues with some packages.
Another Edit:
I've now compiled PIL. Had to hide the whole macports directory and half the xcode libraries, so it would find the right ones. It wouldn't accept the paths I was feeding it. PIL is notorious for this on leopard.
You can install them side-by-side. If you've encounter problems just set python 2.5 as the standard python and use e.g. python26 for a newer version.
Read this
http://farmdev.com/thoughts/66/python-3-0-on-mac-os-x-alongside-2-6-2-5-etc-/
I still use macports python25, because so many other packages depend on it, and have not updated to use python26.
$ port dependents python25
gnome-doc-utils depends on python25
mod_python25 depends on python25
postgresql83 depends on python25
gtk-doc depends on python25
at-spi depends on python25
gnome-desktop depends on python25
mercurial depends on python25
And that's excluding the py25-* packages I have installed.
I wrote something today on this very subject, my recommendation? Run multiple version, and slap virtualenv down to compartmentalize things.
http://jessenoller.com/2009/03/16/so-you-want-to-use-python-on-the-mac/
I also wouldn't both with macports. I don't see a need for it.
I've updated my macbook running leopard to python 2.6 and haven't had any problems with psycopg2. For that matter, I haven't had any compatibility issues anywhere with 2.6, but obviously switching to python3k isn't exactly recommended if you're concerned about backwards compatibility.
I would stick with the MacPython version 2.5.x (I believe 2.5.4 currently). Here's my rationale:
Snow Leopard may still be on the 2.5 series, so you might as well be consistent with the future OS (i.e. no point in going too far ahead).
For most production apps, nobody is going to want to use 2.6 for another year.
No frameworks/programs are going to leave 2.5 behind for at least 2 years.
In other words, my approach is that the only reason to do 2.6 is for fun. If you're looking to have fun, just go for 3.0.
I use both Twisted and Psycopg2 extensively on OSX, and both work fine with Python 2.6. Neither has been ported to Python 3.0, as far as I know.
Several of Python 3.0's features have been back-ported to 2.6, so you gain quite a bit by moving from 2.5 to 2.6. But I wouldn't switch to 3.0 until all of your thirdparty libraries support it; and this may not happen for some time.
I had some trouble installing PIL. I compiled it and it worked with the modification explained on this post http://passingcuriosity.com/2009/installing-pil-on-mac-os-x-leopard/
After that it worked fine.
I am using Python 2.5.1. It's working great for me for general scripting and some CherryPy web projects.
If your using Macports, I recommend downloading the python_select package, which facilitates easy switching between different versions including the built in apple versions. Makes life a lot easier.
In our group we primarily do search engine architecture and content integration work and most of that code base is in Python. All our build tools and Python module dependencies are in source control so they can be checked out and the environment loaded for use regardless of os/platform, kinda similar to the approach virtualenv uses.
For years we've maintained a code base compatible with Python 2.3 because one of the commercial products we use depends on Python 2.3. Over the years this has caused more and more issues as newer tools and libraries need newer versions of Python since 2.3 came out in ~2004.
We've recently decoupled our build environment from dependencies on the commercial product's environment and can use any version of Python (or Java) we want. Its been about a month or so since we standardized on Python 2.6 as the newest version of Python that is backwards compatible with previous versions.
Python 3.0 is not an option (for now) since we'd have to migrate too much of our code base to make our build and integration tools to work correctly again.
We like many of the new features of Python 2.6, especially the improved modules and things like class decorators, but many modules we depend on cause the Python 2.6 interpreter to spout various depreciation warnings. Another tool we're interested in for managing EC2 cloud cluster nodes, Supervisor doesn't even work correctly with Python 2.6.
Now I am wondering if we should standardize on Python 2.5 for now instead of using Python 2.6 in development of production environment tools. Most of the tools we want/need seem to work correctly with Python 2.5. We're trying to sort this out now before there are many dependencies on Python 2.6 features or modules.
Many Thanks!
-Michael
I wouldn't abandon 2.6 just because of deprecation warnings; those will disappear over time. (You can use the -W ignore option to the Python interpreter to prevent them from being printed out, at least) But if modules you need to use actually don't work with Python 2.6, that would be a legitimate reason to stay with 2.5. Python 2.5 is in wide use now and probably will be for a long time to come (consider how long 2.3 has lasted!), so even if you go with 2.5, you won't be forced to upgrade for a while.
I use Python 2.5 for all my development work, but only because it's the version that happens to be available in Gentoo (Linux)'s package repository. When the Gentoo maintainers declare Python 2.6 "stable"*, I'll switch to that. Of course, this reasoning wouldn't necessarily apply to you.
* Python 2.6 actually is stable, the reason it's not declared as such in Gentoo is that Gentoo relies on other programs which themselves depend on Python and are not yet upgraded to work with 2.6. Again, this reasoning probably doesn't apply to you.
My company is standardized in 2.5. Like you we can't make the switch to 3.0 for a million reasons, but I very much wish we could move up to 2.6.
Doing coding day to day I'll be looking through the documentation and I'll find exactly the module or function that I want, but then it'll have the little annotation: New in Version 2.6
I would say go with the newest version, and if you have depreciation warnings pop up (there will probably be very few) then just go in a find a better way to do it. Overall your code will be better with 2.6.
To me the most important to stick with python 2.5+ is because it officially supports ctypes, which changed many plugin systems.
Although you can find ctypes to work with 2.3/2.4, they are not officially bundled.
So my suggestion would be 2.5.
We're sticking with 2.5.2 for now. Our tech stack centers on Django (but we have a dozen other bits and bobs.) So we stay close to what they do.
We had to go back to docutils to 0.4 so it would work with epydoc 3.0.1. So far, this hasn't been a big issue, but it may -- at some point -- cause us to rethink our use of epydoc.
The 2.6 upgrade is part of our development plan. We have budget, but not fixed schedule right now.
The 3.0 upgrade, similarly, is something I remind folks of. We have to budget for it. We won't do it this year unless Django leaps to 3.0. We might do it next year.
I think the best solution is to give up the hope on total uniformity, although having a common environment is something to strive for. You will always will be confronted with version problems, for example when upgrading to the next best interpreter version.
So instead of dealing with it on a per issue base you could solve this problem by taking a good look on your release management.
Instead of releasing source, go for platform depending binaries (besides the source distribution).
So what you do is that you define a number of supported CPU's, for example:
x86-32, x86-64, sparc
Then which operating systems:
Linux, Windows, Solaris, FreeBSD
For each OS you support a number of their major versions.
Next step is that you provide binaries for all of them.
Yes indeed this will require quite some infrastructure investment and setting up of automatic building from your repositories (you do have them?).
The advantages is that your users only have 'one' thing to install and you can easily switch versions or even mixing versions. Actually you can even use different programming languages with this approach without affecting your release management too much.
I've written a web-app in python using SQLite and it runs fine on my server at home (with apache and python 2.5.2). I'm now trying to upload it to my web host and there servers use python 2.2.3 without SQLite.
Anyone know of a way to use SQLite in python 2.2.3 e.g. a module that I can upload and import? I've tried butchering the module from newer versions of python, but they don't seem to be compatible.
Thanks,
Mike
There is no out-of-the-box solution; you either have to backport the SQLlite module from Python 2.5 to Python 2.2 or ask your web hoster to upgrade to the latest Python version.
Python 2.2 is really ancient! At least for security reasons, they should upgrade (no more security fixes for 2.2 since May 30, 2003!).
Note that you can install several versions of Python in parallel. Just make sure you use "/usr/bin/python25" instead of "/usr/bin/python" in your scripts. To make sure all the old stuff is still working, after installing Python 2.5, you just have to fix the two symbolic links "/usr/bin/python" and "/usr/lib/python" which should now point to 2.5. Bend them back to 2.2 and you're good.
Look here: http://oss.itsystementwicklung.de/download/pysqlite/
From the release notes (http://oss.itsystementwicklung.de/trac/pysqlite/browser/doc/install-source.txt)
Python:
Python 2.3 or later
You may not be able to do what you're trying to do.
If you have shell access to your web server, you can probably build you're own version of Python and SQLite. This will let you use the latest version. Download the source code, then when you configure it, do something like "./configure --prefix=$HOME/packages".
Next, fiddle around with your .profile, or .bashrc or whatever it is to make sure $HOME/packages/bin comes first in your path. This will cause your private Python to override the one installed by your web server.
This page might give you a little more information for how to do this on a server like Dreamhost: http://wiki.dreamhost.com/Python
In case anyone comes across this question, the reason why neither pysqlite nor APSW are available for Python 2.2 is because Python 2.3 added the simplified GIL API. Prior to Python 2.3 it required a lot of code to keep track of the GIL. (The GIL is the lock used by Python to ensure correct behaviour while multi-threading.)
Doing a backport to 2.2 would require ripping out all the threading code. Trying to make it also be thread safe under 2.2 would be a nightmare. There was a reason they introduced the simplified GIL API!
I am still astonished at just how popular older Python versions are. APSW for Python 2.3 is still regularly downloaded.