I installed Python 3.5 from source and broke a number of modules on Python 3.4, which unfortunately, was an essential part of Ubuntu. I've been trying to fix the system, now I'm almost there, with (hopefully) the last problem: My Python 3.4 only recognize C modules with name *.cpython-34m.so, while all packages from Ubuntu repository are named *.cpython-34m-x86_64-linux-gnu.so. It seems that the cpython-34m stuff is the full name of Python, so I need to change it in accord with Ubuntu's expectation. How can I achieve this?
What you are trying makes no sense. The name cannot be changed, for a good reason. The reason the names are different is to prevent incompatible versions from mixing up each other. You can compile a different version with different options and then the name will be different, too.
The essence of the package management is to satisfy all dependencies through the repositories. Theoretically there should be no need to install anything from the sources.
If you need a different Python version for some specifical reasons, virtual environments would be the thing to look for:
http://docs.python-guide.org/en/latest/dev/virtualenvs/
I reinstalled Python 3.4 via Ubuntu package system, and suddenly everything worked fine. I still have no clue how Ubuntu customize its own Python, since Python's configure command has no related option at all. Anyway, it works, so why bother :P. Finally, thank you for helping me with this problem.
Related
up until recently I have only worked with one version of Python and used virtual environments every now and then. Now, I am working with some libraries that require older version of Python. So, I am very confused. Could anyone please clear up some of my confusion?
How do I install multiple Python versions?
I initially had Python version 3.8.x but upgraded to 3.10.x last month. There is currently only that one version on my PC now.
I wanted to install one of the Python 3.8.x version and went to https://www.python.org/downloads/. It lists a lot of versions and subversions like 3.6, 3.7, 3.8 etc. etc. with 3.8.1, 3.8.2 till 3.8.13. Which one should I pick?
I actually went ahead with 3.8.12 and downloaded the Tarball on the page: https://www.python.org/downloads/release/python-3812/
I extracted the tarball (23.6MB) and it created a folder with a setup.py file.
Is Python 3.8.12 now installed? Clicking on the setup.py file simply flashes the terminal for a second.
I have a few more questions. Hopefully, they won't get me downvoted. I am just confused and couldn't find proper answers for them.
Why does Python have such heavy dependency on the exact versions of libraries and packages etc?
For example, this question
How can I run Mozilla TTS/Coqui TTS training with CUDA on a Windows system?. This seems very beginner unfriendly. Slightly mismatched package
version can prevent any program from running.
Do virtual environments copy all the files from the main Python installation to create a virtual environment and then install specific packages inside it? Isn't that a lot of wasted resources in duplication because almost all projects require there own virtual environment.
Your questions depend a bit on "all the other software". For example, as #leiyang indicated, the answer will be different if you use conda vs just pip on vanilla CPython (the standard Windows Python).
I'm also going to assume you're actually on Windows, because on Linux I would recommend looking at pyenv. There is a pyenv-win, which may be worth looking into, but I don't use it myself because it doesn't play as nice if you also want (mini)conda environments.
1. (a) How do I install multiple Python versions?
Simply download the various installers and install them in sensible locations. E.g. "C:\Program Files\Python39" for Python 3.9, or some other location where you're allowed to install software.
Don't have Python add itself to the PATH though, since that'll only find the last version to do so and can really confuse things.
Also, you probably want to use virtual environments consistently, as this ties a specific project very clearly to a specific Python version, avoiding future confusion or problems.
1. (b) "3.8.1, 3.8.2 till 3.8.13" which should I pick?
Always pick the latest 3.x.y, so if there's a 3.8.13 for Windows, but no 3.8.14, pick that. Check if the version is actually available for your operating system, sometimes there are later versions for one OS, but not for another.
The reason is that between a verion like 3.6 and 3.7, there may be major changes that change how Python works. Generally, there will be backwards compatibility, but some changes may break how some of your packages work. However, when going up a minor version, there won't be any such breaking changes, just fixes and additions that don't get in the way of what was already there. A change from 2.x to 3.x only happens if the language itself goes through a major change, and rarely happens (and perhaps never will again, depending on who you ask).
An exception to the "no minor version change problems" is of course if you run some script that very specifically relies on something that was broken in 3.8.6, but no fixed in 3.8.7+ (as an example). However, that's very bad coding, to rely on what's broken and not fixing it later, so only go along with that if you have no other recourse. Otherwise, just the latest minor version of any version you're after.
Also: make sure you pick the correct architecture. If there's no specific requirement, just pick 64-bit, but if your script needs to interact with other installed software at the binary level, it may require you to install 32-bit Python (and 32-bit packages as well). If you have no such requirement, 64-bit allows more memory access and has some other benefits on modern computers.
2. Why does Python have such heavy dependency on the exact versions of libraries and packages etc?
It's not just Python, this is true for many languages. It's just more visible to the end user for Python, because you run it as an interpreted language. It's only compiled at the very last moment, on the computer it's running on.
This has the advantage that the code can run on a variety of computers and operating systems, but the downside that you need the right environment where you're running it. For people who code in languages like C++, they have to deal with this problem when they're coding, but target a much smaller number of environments (although there's still runtimes to contend with, and DirectX versions, etc.). Other languages just roll everything up into the program that's being distributed, while a Python script by itself can be tiny. It's a design choice.
There are a lot of tools to help you automate the process though and well-written packages will make the process quite painless. If you feel Python is very shakey when it comes to this, that's probable to blame on the packages or scripts you're using, not really the language. The only fault of the language is that it makes it very easy for developers to make such a mess for you and make your life hard with getting specific requirements.
Look for alternatives, but if you can't avoid using a specific script or package, once you figure out how to install or use it, document it or better yet, automate it so you don't have to think about it again.
3. Do virtual environments copy all the files from the main Python installation to create a virtual environment and then install specific packages inside it? Isn't that a lot of wasted resources in duplication because almost all projects require there own virtual environment.
Not all of them, but quite a few of them. However, you still need the original installation to be present on the system. Also, you can't pick up a virtual environment and put it somewhere else, not even on the same PC without some careful changes (often better to just recreate it).
You're right that this is a bit wasteful - but this is a difficult choice.
Either Python would be even more complicated, having to manage many different version of packages in a single environment (Java developers will be able to tell you war stories about this, with their dependency management - or wax lyrically about it, once they get it themselves).
Or you get what we have: a bit wasteful, but in the end diskspace is a lot cheaper than your time. And unlike your time, diskspace is almost infinitely expandable.
You can share virtual environments between very similar projects though, but especially if you get your code from someone else, it's best to not have to worry and just give up a few dozen MB for the project. On the upside: you can just delete a virtual environment directory and that pretty much gets rid of the whole things. Some applications like PyCharm may remember that it was once there, but other than that, that's the virtual environment gone.
Just install them. You can have any number of Python installations side by side. Unless you need to have 2 different minor versions, for example 3.10.1 and 3.10.2, there is no need to do anything special. (And if you do need that then you don't need any advice.) Just set up separate shortcuts for each one.
Remember you have to install any 3rd-party libraries you need in each version. To do this, navigate to the Scripts folder in the version you want to do the install in, and run pip from that folder.
Python's 3rd-party libraries are open-source and come from projects that have release schedules that don't necessarily coincide with Python's. So they will not always have a version available that coincides with the latest version of Python.
Often you can get around this by downloading unofficial binaries from Christoph Gohlke's site. Google Python Gohlke.
Install Python using the windows executable installers from python.org. If the version is 3.x.y, use the highest y that has a windows executable installer. Unless your machine is very old, use the 64-bit versions. Do not have them add python to your PATH environment variable, but in only one of the installs have it install the python launcher py. That will help you in using multiple versions. See e.g. here.
Python itself does not. But some modules/libraries do. Especially those that are not purely written in Python but contain extensions written in C(++). The reason for this is that compiling programs on ms-windows can be a real PITA. Unlike UNIX-like operating systems with Linux, ms-windows doesn't come with development tools as standard. Nor does it have decent package management. Since the official Python installers are built with microsoft tools, you need to use those with C(++) extensions as well. Before 2015, you even had to use exactly the same version of the compiler that Python was built with. That is still a good idea, but no longer strictly necessary. So it is a signigicant amount of work for developers to release binary packages for each supported Python version. It is much easier for them to say "requires Python 3.x".
I just dealt with the worse bug in my entire 3 years of computer programming! It turns out that because I wanted to work with the natural language toolkit I had to install python 3.5 even though I'm using python 3.6. So I downloaded 3.5 and now my terminal is using python 3.5 by default and I can't get it back to 3.6. Because I was using python 3.5 which does not automatically order dictionaries it was throwing my program off because it relies on ordered dictionaries. It took me 4 hours to figure that out.
You want to use virtualenv and/or virtualenvwrapper. This is a utility that allows you to use multiple different environments, with different Python versions, different pip packages installed, etc.
To find the 3.5 version, run which python in your terminal to find the path to the python executable; then look at your PATH environment, and see where the location of that Python is on your PATH. Then you need to find out where that path is getting added; this will depend on your OS/Shell.
Tough times for sure, sorry to hear that.
I use pyenv to manage the different python versions on my system. This allows you to create virtual environments using whichever version you want.
EDIT to address comments.
I totally understand that setting up virtualenv or something like pyenv is not simple. However, it is unfortunately the easiest way to deal with (and avoid) situations like this. There are two essential concepts that are important here:
1) Isolation - Virtualenv takes care of this. When you install dependencies in a virtual environment, they will not affect your other environments or system python installation.
2) Multiple Python Versions - In your case, you needed to use a module that did not support 3.6. Instead of creating a virtual environment using python 3.5, you accidentally messed up your system installation of 3.6. Recovering from these types of misconfigurations can be difficult, and it is often easier to simply prevent it in the first place.
Again, I completely understand that this might be complicated, I remember thinking the same thing, but it is less complicated than troubleshooting the misconfigurations that can occur without this tooling.
It seems that when I install Anaconda, I can't neither normal python or the python provided with Anaconda, even though Anaconda already in path.
I do realize that Anaconda also come with a python, but it come both with 2.7 and 3.2, but it's kinda scary due to the path conflict that I had earlier. It may ended like this, fortunately it goes normal when I uninstalled it
After uninstalling, my plan is to only take the installed library then uninstall Anaconda:
Reinstall Anaconda
Copy the library (scipy,numpy,etc)
Paste it to normal Python2
Uninstall Anaconda and its family
Hapilly ever after
But this seems not foolproof, is there any better way?
Note: As I mentioned, I know Anaconda have python avaliable too, so my other alternative is to uninstall normal python and just use Anaconda. But again, when I saw they provided two version, I decided to take the way mentioned above.
Python is getting more complex and installing libraries in a way that they work is becoming more brittle. You can install pip which will try to download the source code for libraries and compile them for your OS (which might or might not need a C compiler locally installed and working).
Anaconda tries to solve this hazzle by providing a set of working, well maintained libraries which you can install easily using the conda tool. When I installed the product last time, it didn't try to install both Python 2 and 3 - you have to select either. It also asks whether it should add itself to your path; you can say "no".
But you have to chose between: "I know exactly what I'm doing" - then you're on your own. Or you can say "I don't know enough" and trust some unknown expert to get it right most of the time.
You copy&paste approach might work since I haven't seen a Python library where absolute paths were compiled in. On the other hand, some of those libraries have hundreds of thousands of lines of code. It's hard to say which one of them will break when you start moving things around.
My program requires specific versions of several python packages. I don't want to have to require the user to specifically install the specific version, so I feel that the best solution is to simply install the package within the source repository, and to distribute it along with my package.
What is the simplest way to do this?
(Please be detailed - I'm familiar with pip and easy_install, but they don't seem to do this, at least not by default).
Go for virtualenv. Life will be much easier. MUCH easier. Basically, it allows you to create specific python environments as needed.
There are indeed two ways to get this done.
I usually use buildout (see a post by Jacob from Django: http://jacobian.org/writing/django-apps-with-buildout/) - and have everything from django up installed locally at the project's environment, with pydev and django support. It's very easy since I have projects that use latest versions of open source software and others that use specific versions of the same packages.
Another alternative is, as Charlie says, the virtualenv,which is designed to do just that. Many people recommend it, I've never used it myself as I'm happy with buildout.
I bought a low-end MacBook about a month ago and am finally getting around to configuring it for Python. I've done most of my Python work in Windows up until now, and am finding the choices for OS X a little daunting. It looks like there are at least five options to use for Python development:
"Stock" Apple Python
MacPython
Fink
MacPorts
roll-your-own-from-source
I'm still primarily developing for 2.5, so the stock Python is fine from a functionality standpoint. What I want to know is: why should I choose one over the other?
Update:
To clarify, I am looking for a discussion of the various options, not links to the documentation. I've marked this as a Community Wiki question, as I don't feel there is a "correct" answer. Thanks to everyone who has already commented for their insight.
One advantage I see in using the "stock" Python that's included with Mac OS X is that it makes deployment to other Macs a piece of cake. I don't know what your deployment scenario is, but for me this is important. My code has to run on any number of Macs at work, and I try to minimize the amount of work it takes to run my code on all of those systems.
I would highly recommend using MacPorts with Porticus for managing your Python installation. It takes a while to build everything, but the advantage is that whatever you build yourself will be built against the same libraries, so you won't have to futz around with statically linked shared objects, etc. if you want your Python stuff to work with Apache, PostgreSQL, etc.
If you choose to go this way, remember to install the python_select port and use it to make your system use the Python installed from MacPorts.
As an added bonus, MacPorts has packages for most main-stream Python eggs, so if you should be able to have MacPorts keep you up-to-date with the latest versions of all that stuff :)
Here's some helpful info to get you started. http://www.python.org/download/mac/
Depends what you are using python for. If you are using MacOS funitionality and things like PyObjC you are probably best of with MacPython or the python provided by Apple.
I use Python on my Mac mostly for development of server side applications which later will run on FreeBSD & Linux boxes. For that I have used fink python for a few years and ever since MacPorts python. With mac ports it is simple to add required c modules (like database driver etc). It's also easy to keep two python Versions (2.5 & 2.6 in my case) around.
I used "compile your own" python to test pre-3.0 python but generally I find managing dependencies to c modules painfull if done by hand.
Thanks to easy_install installing pure python modules is fast and easy for all the options mentioned above.
I was never very much an IDE person. For development I use command line subversion installed by MacPorts, Textmate and occasionaly Expandrive do directly access files on servers. I personally are very dependent on Bicyclerepairman for Textmade to handle my refactoring needs.
Others seem to be very happy with Eclipse & Pydev.
How about EPD from Enthought? Yes, it's large but it is a framework build and includes things like wxPython, vtk, numpy, scipy, and ipython built-in.
I recommend using Python Virtual environments, especially if you use a Timecapsule because Timecapsule will back everything up, except modules you added to Python!
Based on the number of bugs and omissions people have been encountering in Leopard python (just here on SO!), I couldn't recommend that version. e.g., see:
Why do I get wrong results for hmac in Python but not Perl?
Problems on select module on Python 2.5
I would choose MacPorts.
It does not eliminate your existing python supplied by Apple since it installs by default in /opt/local/bin (plays nice with it) and plus it is easy to download and install additional python modules (even binary modules that you need to compile!). I use Porticus GUI to maintain my MacPorts installed list of packages, including python.
In my windows environment I use Eclipse and PyDev, which works quite well together, even if it's a bit sparse. Apparently the exact same environment is available for the Mac as well, so I suggest downloading Eclipse and using the internal update software function to update PyDev with the URL http://pydev.sourceforge.net/updates/. To look further into PyDev, look here.
Apple's supplied python is quite old – my tiger install has 2.3.5. This may not be a problem for you, but you would be missing out on a lot. Also, there is a risk that Apple will update it. I'm not sure if moving from 2.3.5 to (say) 2.4 would cause code to break, but I guess it's possible. This happened to perl people recently: http://developers.slashdot.org/article.pl?sid=09/02/18/1435227
Macpython is a framework build (as is Apple's, I believe). To be honest, I'm not sure exactly what that means, but it's a prerequisite for some modules, in particular wxPython. If you get python from macports or fink, you will not be able to run wxPython (unless you run it through X11).
And guess what was forgotten by every answer here ... ActivePython.
No compilation required, even for third-party modules such as numpy, lxml, pyqt and thousands of others.
I recommend python (any python?) plus the ipython shell. My most recent experience with MacPython was MacPython 2.5, and I found IDLE frustrating to use as an editor. It's not very featureful, and its' very slow to scroll large quantities of output.