I have been trying to install a python module called OBITools (https://pythonhosted.org/OBITools/welcome.html, https://anaconda.org/bioconda/obitools) that was designed to python 2.7. Apart from the installation strategy, it will invariably call a script called 'get-obitools.py' (https://pythonhosted.org/OBITools/_downloads/get-obitools.py). However, the 'get-obitools.py' script, even when called from python 2.7 will try to install up-to-date dependencies. By doing so, the installation will be trapped in an infinite loop because it will fail to install certain dependencies, usually due to python syntax change in the most recent python versions, i.e.:
print(error, file=sys.stderr)
^
SyntaxError: invalid syntax
I have to say that I am not a heavy python user. Thus, I might be missing something basic here. However, it is really strange to me that basic syntax changed from python2 to python3. It seems logic that such huge change would break old modules at some point. In fact, it may be the case for OBITools as show above.
In summary, is it possible to force OBITools installation to search for old dependencies (i.e. versions that worked at python 2.7) while installing?
There is a very simple and illogical hack to fix this issue.
you can simply remove argument name from source code of get-obitools.py.
For Example
your code will become
print(error, sys.stderr)
And this is not a huge change, It's basic change in python 2.x print was not a function but in python 3.x it's a function
Related
I installed python using the command:
brew install python3
Now when I use 'which python', path is '/usr/bin/python' and when I use 'which python3' path is 'usr/local/bin/python3'
Shouldn't it fetch the same path ?
As python3, which is a version, still falls under python ?
I just am not clear as to why different paths are thrown. Please Explain!
Python is a program, that will take inputs, and interpret them. How will it interpret them ? Following a set of rules, written in a lot of files. Where are these files written ? Somewhere the program knows. And by default, the 2 paths you have are the paths it knows where to go.
Python2 and Python3 have different files, because even if they have the same base, they are not same and behave differently. Hence the 2 differents paths.
Though, through the years, people have come up with solution to avoid these "python version collisions" on computers : it is called a virtualenv.
Virtualenv is basically a script that will contain a whole new python (at the version you wish you install it), and, when you "activate" it, you will be able to use python, and have the version you wish to develop with. Doing this, everyone is able to only use python and still use different versions depending of the program you with to use.
Example : You have python on your system which writes Python 2.7.12 when you do python --version. If you instantiate your virtualenv (see the doc), and then use python --version again, you might see something else along the lines of Python 3.6.8. Your former computer, or other scripts, can still use the previous python version, and you new script can use the new one, without any conflicts.
I would like to upgrade my Python version from 2.6.6 to the newest version of 3.6.5 on a server. However, there is a program that has been running for multiple days and will be running for a week or more.
Will my program continue to run until it finishes if I upgrade my Python version?
P.S.: I ran my Python program with the screen command so I wouldn't have to stop it to be able to upgrade my Python version.
Yes, your Python 2 script should continue to run while you install/upgrade Python 3, since Python 2 and 3 are different programs.
However, the installation will be be platform specific and there might be common dependencies, so I would test this on a different machine (e.g. local virtual machine) first.
Several aspects are to be considered here:
In general, a Python program that is running does not access the installed binaries of the interpreter anymore. Also all the modules already loaded will not be read again from disk during the execution time of this program. (In this aspect it differs from e. g. a Bash script which reads its script source while executing.) So you probably are safe in even removing the complete Python installation; of course that's not what you plan on and I wouldn't recommend it for other reasons, but even that should not influence your running Python program. If, however, your running Python program tries to import a module for the first time after the removal or upgrade of your Python installation, this might fail due to incompatibilities between the versions. Most programs do their imports only at the beginning of their runtime, though.
Python 2 and 3 are different languages. Scripts for the one need to be at least adjusted to run with the other. Adjustments aren't too hard to do, though. Most distributions have both installed side-by-side. Most of the time, Python2 is still standard (and accessed by typing python). python2 and python3 are then to make it explicit. This means that you probably should not remove Python 2 from your system, even if you plan on installing Python 3. There probably are scripts relying on the installation of Python 2.
There's a lot going on here that you've not explained/described, so it's hard to give you a definitive answer (such as the operating system you're using). This is also not exactly a programming question, so you might find that the question gets voted to close. That said...
It possible to run multiple versions of Python side by side. Because 2.x and 3.x are very different, it might be much better to have both versions installed. See this answer for some details on doing that:
Official multiple Python versions on the same machine?
It's likely that your code will need some modifications to run on Python 3, so you're going to have to redeploy a new version of your code at some point anyway.
Background
I write small python packages for a system that uses modules (https://luarocks.org/) to manage packages. For those of you who don't know it, you can run module load x and a small script is run that modifies various environmental variables to make software 'x' work, you can then undo this with module unload x.
This method of software management is nearly ubiquitous in scientific computing and has a lot of value in that arena: you can run ancient unmaintained software alongside packages that that software would interfere with, you can run multiple versions of software, which allows you to reproduce your data exactly (you can go back to old versions), and you can run frankly poorly written non updated software with outdated dependencies.
These features are great, but they create an issue with the python 2/3 split:
What if you want to write a package that works with both python 2 and 3 and use it alongside software that requires either python 2 or 3?
The way you make old python2 dependent software work on these large systems is that you make a python/2.7.x module and a python/3.5 module. When you want to run a script that uses python 2, you load that module, etc.
However, I want to write a single python package that can work in either environment, because I want that software to be active regardless of which python interpreter is being used.
This is fundamentally extremely easy: just use a #!/usr/bin/env python shebang line, done. That works. I write all my software to work with either, so no problem.
Question
The issue is: I want to use setuptools to distribute my package to other scientists in the same situation, and setup tools mangles the shebang line.
I don't want to get into a debate about whether mangling the shebang line is a good idea or not, I am sure it is since it has existed for years now in the same state. I honestly don't care, it doesn't work for me. The default setuptools install causes the software not to run because when a python interpreter's module is not loaded, that python interpreter does not function, the PYTHONPATH is totally wrong for it.
If all of my users had root access, I could use the data_files option to just copy the scripts to /usr/bin, but this is a bad idea for compatibility, and my users don't have root access anyway so it is a moot point.
Things I tried so far:
I tried setting the sys.executable to /usr/bin/env python in the setup.py file, but that doesn't work, because then the shebang is: #!"/usr/bin/env python", which obviously doesn't work.
I tried the Don't touch my shebang class idea in this question: Don't touch my shebang! (it is the bottom answer with 0 votes). That didn't work either, probably because it is written for distutils and not setuptools. Plus that question is 6 years old.
I also looked at these questions:
Setuptools entry_points/console_scripts have specific Python version in shebang
Changing console_script entry point interpreter for packaging
The methods described there do not work, the shebang line is still altered.
Creating a setup.cfg file with the contents::
[build]
executable = /usr/bin/env python
also does not change the shebang line mangling behavior.
There is an open issue on the setuptools github page that discusses something similar:
https://github.com/pypa/setuptools/issues/494
So I assume this isn't possible to do natively, but I wonder if there is a workaround?
Finally, I don't like any solution that involves asking the user to modify their install flags, e.g. with -e.
Is there anyway to modify this behavior, or is there another distribution system I can use instead? Or is this too much of an edge case and I just need to write some kind of custom installation script?
Thanks all.
Update
I think I was not clear enough in my original question, what I want the user to be able to do is:
Install the package in both python2 and python3 (the modules will go into lib/pythonx/site-lib.
Be able to run the scripts irrespective of which python environment is active.
If there is a way to accomplish this without preventing shebang munging, that would be great.
All my code is already compatible with python 2.7 and python 3.3+ out of the box, the main thing is just making the scripts run irrespective of active python environment.
I accidentally stumbled onto a workaround while trying to write a custom install script.
import os
from setuptools import setup
from setuptools.command.install import install
here = os.path.abspath(os.path.dirname(__file__))
# Generate a list of python scripts
scpts = []
scpt_dir = os.listdir(os.path.join(here, 'bin'))
for scpt in scpt_dir:
scpts.append(os.path.join(here, 'bin', scpt))
class ScriptInstaller(install):
"""Install scripts directly."""
def run(self):
"""Wrapper for parent run."""
super(ScriptInstaller, self).run()
setup(
cmdclass={'install': ScriptInstaller},
scripts=scpts,
...
)
This code doesn't do exactly what I wanted (alter just the shebang line), it actually just copies the whole script to ~/.local/bin, instead of wrapping it in::
__import__('pkg_resources').run_script()
Additionally, and more concerningly, this method makes setuptools create a root module directory plus an egg-info directory like this::
.local/lib/python3.5/site-packages/cluster
.local/lib/python3.5/site-packages/python_cluster-0.6.1-py3.5.egg-info
Instead of a single egg, which is the usual behavior::
.local/lib/python3.5/site-packages/python_cluster-0.6.1-py3.5.egg
As far as I am aware this is the behavior of the old distutils, which makes me worry that this install would fail on some systems or have other unexpected behavior (although please correct me if I am wrong, I really am no expert on this).
However, given that my code is going to be used almost entirely on linux and OS X, this isn't the end of the world. I am more concerned that this behavior will just disappear sometime very soon.
I posted a comment on an open feature request on the setuptools github page:
https://github.com/pypa/setuptools/issues/494
The ideal solution would be if I could add an executable=/usr/bin/env python statement to setup.cfg, hopefully that is reimplemented soon.
This workaround will work for me for now though. Thanks all.
I'm trying to run rst2odt.py and rst2wordml.py with Matti Pastelli's example on a Windows XP 32-bit computer. The R step works well, then I get the files ascii-example.rst, ascii-example.rst and ascii-example.pdf.
With Portable Python 2.7.3.1 I get the following errors:
With Portable Python 3.2.1.1 I get the following errors:
I am almost ignorant about the Python language hence I totally don't know what to do.
Not really an answer, but a couple of observations to look at
It's difficult to tell but two things stand out - the error you're receiving for the 2.7.x version is that print was a statement in the 2.x and was changed to a function in the 3.x. Hence, it doesn't accept parameters ergo the error.
In 2.x file and open were effectively the same thing, and in the 3.x series file was removed, which is another error.
As to your "image size not fully specified and PIL not installed" - you'll either need to look at the help for rst2ody to see if it can be specified, or install PIL (which is straightforward enough using pip or easy_install).
I'm afraid I don't have Portable Python, and although I could suggest some kludges, I'm hoping this at least gives you some pointers...
Is there a standard way to make sure a python script will be interpreted by python2 and not python3? On my distro, I can use #!/usr/bin/env python2 as the shebang, but it seems not all distros ship "python2". I could explicitly call a specific version (eg. 2.6) of python, but that would rule out people who don't have that version.
It seems to me that this is going to be increasingly a problem when distros will start putting python3 as the default python interpreter.
http://docs.python.org/library/sys.html#sys.version_info
using the sys module you can determine the version of python that is running and raise an exception or exit or whatever you like.
UPDATE:
You could use this to call the appropriate interpreter. For example, set up a small script that does the checking for you, and use it in the shbang. It would check the python version running, and if not what you want, looks for one you want. Then it would run the script in that version of python (or fail if nothing good was found).
This is a bit of a messy issue during what will be a very long transition time period. Unfortunately, there is no fool-proof, cross-platform way to guarantee which Python version is being invoked, other than to have the Python script itself check once started. Many, if not most, distributions that ship Python 3 are ensuring the generic python command is aliased by default to the most recent Python 2 version while python3 is aliased to the most recent Python 3. Those distributions that don't should be encouraged to do so. But there is no guarantee that a user won't override that. I think the best practice available for the foreseeable future is to for packagers, distributors, and users to assume python refers to Python 2 and, where necessary, build a run-time check into the script.
Using sys.version_info you can do a simple value test against it. For example if you only want to support version 2.6 or lower:
import sys
if sys.version_info > (2,6):
sys.exit("Sorry, only we only support up to Python 2.6!")
Not quite the same situation, but the company I work for has an app that can run Python scripts (among its many features). After numerous support issues involving Python installations on various platforms, we decided to just install our own Python interpreter with the app. That way we know exactly where it is installed and what version it is. This approach may be too heavyweight for your needs (the Python package is only about 10% of our app's bits) but it definitely works.
Depends on how you're distributing it, I guess.
If you're using a normal setup.py file to manage your distribution, have it bomb out if the user is trying to install it in Python 3.
Once it's installed, the shebang of the console script created by (say) setuptools will likely be linked to the specific interpreter used to install it.
If you're doing something weird for your installation, you can in whatever installation script you're using look for python interpreters and store a choice. You might first check whether whatever is called "python" is a 2.x. If not, check for "python2.7", "python2.6", etc to see what's available.
As I understand different distros will be in different locations in your drive. Here are some suggestions that come to mind -
You could use UNIX alias to create shortcuts pointing to the different distros. Eg: alias py2="/usr/bin/python2.X". So when you run your script you could use py2 xx.py
Or other way could be to modify your PYTHON_PATH environment variable.
Or if I am not wrong there is a provision in sys module to get the current python version number. You could get that & deal appropriately.
This should do it...
You can use the autotools to pick a Python 2 interpreter. Here is how to do that. Guaranteeing a correct shebang may be tricky to do elegantly; here is one way to do that. It may be easier to simply have a light Bash wrapper script, wrapper.sh.in that looks something like:
#!/bin/bash
PYTHON2="#PYTHON#" #That first link enables this autotool variable
"$PYTHON2" "$#" #Call the desired Python 2 script with its arguments
Call wrapper.sh (after a ./configure) like:
./wrapper.sh my_python2_script.py --an_option an_argument
I believe this will do what you want, namely test for a non-specific version of Python less than 3.x (as long as it doesn't contain a from __future__ import print_function statement).
try:
py3 = eval('print')
except SyntaxError:
py3 = False
if py3: exit('requires Python 2')
...
It works by testing to see if print is a built-in function as opposed to a statement, as it is in Python3. When it's not a function, the eval() function will raise an exception, meaning the code is running on a pre-Python 3.0 interpreter with the caveat mentioned above.