I am looking to install Python 3.10.9 and some packages behind a firewall on RHEL 7 machine.
I do not have administrative rights, nor access to elevated privileges. Some core requirements are missing to build items from source. I've been unable to build with OpenSSL, and thus can't pip install packages.
I prefer to stick to packaged solutions as much as possible, so I've pursued a path that allows installing dependencies with yum.
Relevant related questions
Preparing _tkinter and sqlite3 for Python installation (no admin rights)
Relevant lessons: How to install packages in Linux (CentOS) without root user with automatic dependency handling?
# Install prerequisites
mkdir ~/rpm
mkdir ~/rhel
yumdownloader --destdir ~/rpm --resolve zlib-devel bzip2 bzip2-devel readline-devel sqlite sqlite-devel openssl-devel tk-devel libffi-devel xz-devel libffi
cd ~/rhel && for i in ~/rpm/*.rpm; do rpm2cpio $i | cpio -idv; done
# Modify ~/.bashrc with necessary flags and includes. See .bashrc below
source ~/.bashrc
#Download pyenv and install
curl https://pyenv.run | bash
pyenv install 3.10.9
The part I seem to be getting stuck on, is correctly referencing the various libraries that are installed.
Rather than build dependencies from source, utilized yum to install:
Current ~/.bashrc
# For building custom YUM locally
export PATH="$HOME/rhel/usr/sbin:$HOME/rhel/usr/bin:$HOME/rhel/bin:$PATH"
export MANPATH="$HOME/rhel/usr/share/man:$MANPATH"
L='/lib:/lib64:/usr/lib:/usr/lib64'
export LD_LIBRARY_PATH="$HOME/rhel/usr/lib:$HOME/rhel/usr/lib64:$L"
# PyEnv install https://www.mpietruszka.com/install-pyenv-ce-rhel8.html
export PYENV_ROOT="$HOME/.pyenv"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
#export PKG_CONFIG_PATH=$HOME/rhel/usr/lib/pkgconfig:$PKG_CONFIG_PATH
#LDFLAGS
export LDFLAGS="-L$HOME/rhel/usr/lib -L$HOME/rhel/usr/lib64 -L/usr/lib -L/usr/lib64 -L/lib -L/lib64"
#CPPFLAGS
export CPPFLAGS="-I$HOME/rhel/usr/lib -I$HOME/rhel/usr/lib64 -I/usr/lib -I/usr/lib64 -I/lib -I/lib64"
#LD_Library
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$HOME/rhel/usr/include"
The errors I'm currently getting
pyenv install 3.10.9
Downloading Python-3.10.9.tar.xz...
-> https://www.python.org/ftp/python/3.10.9/Python-3.10.9.tar.xz
Installing Python-3.10.9...
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "~/.pyenv/versions/3.10.9/lib/python3.10/ctypes/__init__.py", line 8, in <module>
from _ctypes import Union, Structure, Array
ModuleNotFoundError: No module named '_ctypes'
WARNING: The Python ctypes extension was not compiled. Missing the libffi lib?
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'readline'
WARNING: The Python readline extension was not compiled. Missing the GNU readline lib?
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "~/.pyenv/versions/3.10.9/lib/python3.10/ssl.py", line 99, in <module>
import _ssl # if we can't import it, let the error propagate
ModuleNotFoundError: No module named '_ssl'
ERROR: The Python ssl extension was not compiled. Missing the OpenSSL lib?
Please consult to the Wiki page to fix the problem.
https://github.com/pyenv/pyenv/wiki/Common-build-problems
BUILD FAILED (RedHatEnterpriseServer 7.7 using python-build 20180424)
Inspect or clean up the working tree at /tmp/python-build.20230208180402.153250
Results logged to /tmp/python-build.20230208180402.153250.log
Last 10 log lines:
$ensurepip --root=/ ; \
fi
Looking in links: /tmp/tmp540mosi3
Processing /tmp/tmp540mosi3/setuptools-65.5.0-py3-none-any.whl
Processing /tmp/tmp540mosi3/pip-22.3.1-py3-none-any.whl
Installing collected packages: setuptools, pip
WARNING: The scripts pip3 and pip3.10 are installed in '~/.pyenv/versions/3.10.9/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pip-22.3.1 setuptools-65.5.0
make: warning: Clock skew detected. Your build may be incomplete.
I'd appreciate some criticism on correctly calling these libraries...
from here: https://stackoverflow.com/questaions/34507045/how-to-install-man-pages-for-c11 there is mentioned,
cppman is no longer supported under Ubuntu/apt
and therefor no libstdc++6-<version>-doc could be install via apt anymore.
and the only way to install the man page is via pip3. So I have tried to install
sudo apt-get install python-pip and sudo apt-get install python3-pip. Now, having python3, I can install it via pip3:
pip3 install cppman, everything seems correct, before I tried to cache the man pages: cppman -c, which giver error:
Traceback (most recent call last):
File "/usr/local/bin/cppman", line 40, in <module>
from cppman.main import Cppman
File "/usr/local/lib/python2.7/dist-packages/cppman/main.py", line 26, in <module>
import html
ImportError: No module named html
As could be seen, it uses Python2.7 instead of Python3, that is strange.
From github, I have found similar problem here: https://github.com/aitjcize/cppman/issues/80, where they suggest to do
pip uninstall mancpp
pip3 install mancpp
Which I did, but when installing mancpp again from pip3, then another server error:
Collecting mancpp
Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://pypi.org/simple/mancpp/
So what now? If you read till here, than you can see there are many errors, but cannot find solution anywhere in stack sites. I only want to have cpp manuals for searching for functions and classes (like std) and mainly for glibc++. As I could do simply with c functions. Why is it so hard? Anyway, how to install the mancpp via pip3?
uname -a:
Linux 4.19.0-9-amd64 #1 SMP Debian 4.19.118-2+deb10u1 (2020-06-07) x86_64 GNU/Linux
The script uses #!/usr/bin/env python shebang and on your system python is most probably 2.7. Just edit the script:
sudo vim /usr/local/bin/cppman
Append 3 at the end: #!/usr/bin/env python3 and try again.
I am running Ubuntu 12.10 that has python 2.7 as its default. I have installed python 3.3 in /opt/python3.3 (using ./configure --prefix=/opt/python3.3) and have created a symlink /usr/bin/python33 that points to the executable in /opt. There is already a symlink /usr/bin/python3 that points to the python 3.2 (actually it points to python3.2mu: if you can explain as a side note as to why this 'mu' is included :) ) installation that came with the ubuntu installation.
I installed distribute tools using python3 without any problems. But when I tried the same thing with python33 it gave me the following error:
ankur#junk-mechanism:~$ sudo python33 distribute_setup.py
Extracting in /tmp/tmp685lyf
lzma module is not available
not a bzip2 file
gzip module is not available
bad checksum
Traceback (most recent call last):
File "distribute_setup.py", line 550, in <module>
sys.exit(main())
File "distribute_setup.py", line 547, in main
return _install(tarball, _build_install_args(options))
File "distribute_setup.py", line 78, in _install
tar = tarfile.open(tarball)
File "/opt/python3.3/lib/python3.3/tarfile.py", line 1578, in open
raise ReadError("file could not be opened successfully" + str(name) + repr(fileobj))
tarfile.ReadError: file could not be opened successfully/home/ankur/distribute-0.6.35.tar.gzNone
So in tarfile.py, the classmethod Tarfile.open calls the class method Tarfile.gzopen where the module gzip is imported:
try:
import gzip
gzip.GzipFile
except (ImportError, AttributeError):
raise CompressionError("gzip module is not available")
The error raised here is ImportError because gzip.py has an import zlib which it cant find. But even the other installations of python (2.7 and 3.2) dont contain zlib in the respective standard lib /usr/lib/pythonx.x/lib/. So what is the difference?
It sounds like you didn't build everything because of missing dependencies. You can try getting the missing dependencies and rebuilding. To get the missing dependencies, I'd just work from what Python 3.2 requires and do:
sudo apt-get build-dep python3
Then you can reconfigure and rebuild Python 3.3 to get all the required modules.
Another option is to use the dead snakes ppa:
sudo add-apt-repository ppa:fkrull/deadsnakes
sudo apt-get update
sudo apt-get install python3.3
I usually install the -dev package too so that I can install packages that have C extension modules:
sudo apt-get install python3.3-dev
Scripts like that:
from PIL import Image, ImageOps
img = Image.open('1.JPG')
thumb = ImageOps.fit(img, (200,200) , Image.ANTIALIAS, (0.5, 0.5))
cause this IOError:
Traceback (most recent call last):
(...)
File "/Library/Python/2.7/site-packages/PIL/Image.py", line 385, in _getdecoder
raise IOError("decoder %s not available" % decoder_name)
IOError: decoder jpeg not available
How do I get jpeg support for my PIL? That issue seems to be well known but the existing threads don't solve it for me. I use brew and pip for my python packages and have already tried the following:
brew install jpeg
causes Error: jpeg-8d already installed
brew install libjpeg causes Error: jpeg-8d already installed
sudo pip install Pillow - installation works but does not change anything
How do I get jpeg support for my PIL installation? Any ideas?
I don't understand why, but reinstalling PIL fixed the issue:
sudo pip uninstall pil
sudo pip install pil
I ran into a similar problem on Ubuntu 12.04 64-bit and solved it by symlinking the libraries PIL was looking for into /usr/lib (where it was actually looking for them):
# symlink image libraries so PIL can find them
ln -s /usr/lib/x86_64-linux-gnu/libjpeg.so /usr/lib
ln -s /usr/lib/x86_64-linux-gnu/libfreetype.so /usr/lib
ln -s /usr/lib/x86_64-linux-gnu/libz.so /usr/lib
# etc for any other libraries
For non OSX people landing here, I found linking to the 64 bit libraries on my 64 bit Ubuntu system to fix the compile issue.
I have read the other posts about the notorious _imaging C module error when installing PIL on Mac OS X and none of the solutions provided anywhere, including the PIL FAQ, have proven helpful.
I have the newest versions of libjpeg and zlib freshly installed from source. I have edited the Makefiles in each of these to include the option -arch i386 in the LD_FLAGS variable for 32-bit builds. PIL installs with no problems of any kind and the install summary printed to the terminal says that JPEG, TIFF, and PNG support are all OK. After that I try the self test:
new-host:Imaging-1.1.7 ely$ python selftest.py
*** The _imaging C module is not installed
This is commonly seen for a variety of reasons. Probing deeper, here I try to import _imaging directly in python.
new-host:Imaging-1.1.7 ely$ python
ActivePython 2.7.1.4 (ActiveState Software Inc.) based on
Python 2.7.1 (r271:86832, Feb 7 2011, 11:33:10)
[GCC 4.2.1 (Apple Inc. build 5664)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import PIL
>>> import _imaging
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PIL/_imaging.so, 2): Symbol not found: _jpeg_resync_to_restart
Referenced from: /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PIL/_imaging.so
Expected in: dynamic lookup
Here again, the 'Symbol not found: _jpeg_resync_to_restart' is notorious and common, and many people have suggested this has to do with incorrect path to libjpeg. I've checked again and again and I only have libjpeg (as well as zlib, etc.) installed in my home directory, all in separate folders, from source, and all of this is marked correctly in the file setup.py.
So, PIL should be getting its jpeg support (and zlib, etc.) from the right places. All the dependencies are installed. I can see _imaging.so in my sys.path, yet I still get this _jpeg_resync_to_restart error.
Are there any ideas that don't link to alternate posts? I have spent ~7 hours reading and trying possible solutions from posts in every forum I can find.
Since you have been trying this a few times, I recommend running a few commands to clean out the old items first and start from the beginning.
I used jpeg v8c and Imaging 1.1.6 on Mac OS X, 10.6 and 10.7
get v8c of jpeg
cd into jpeg directory.
sudo make clean
CC="gcc -arch i386" ./configure --enable-shared --enable-static
make
sudo make install
get imaging i am using 1.1.6
untar
cd into imaging
sudo rm -rf build
vi setup.py
JPEG_ROOT = libinclude(“/usr/local/lib”)
sudo python setup.py install
And it’s that simple
Run your python interpreter,
import PIL
import _imaging
import Image
if all is well, then your all set.
Here is the full article on my blog Python 2.7, OSX Lion, PIL and Imaging
I have faced the same problem this evening on my mac running Mac OS X v10.7.5, Python v2.7.1 with PIL Imaging 1.1.7 and jpeg 8d.
Summarizing - the key to success in my case was:
export ARCHFLAGS="-arch x86_64"
Nothing from the other answers have solved the issue, constantly I've been getting:
The _imaging C module is not installed caused by:
Symbol not found: _jpeg_resync_to_restart.
Digging here and there finally I've found the solution, at least works for me:
ensure you don't have any obvious libjpeg libs on the system: find / -iname "libjpe*" will tell you that. I have temporarily changed places where they exist so no one could find them (especially /sw was interacting with PIL in my case):
mv opt opt-OFF
mv sw sw-OFF
These locations have been created by mac ports and fink - be warned that it is quite possible that FREETYPE2 might not be found by PIL after this move - if you need it, then just rename libjpeg parts in those locations.
in PIL Imaging src directory edit setup.py as described above to set JPEG_ROOT to /usr/local, try to run:
rm -rf build ; python setup.py build 2> /tmp/err > /tmp/log
Check in /tmp/log - you should not see JPEG support available. You can now go to the next step.
go to jpeg-8d src directory and run:
export ARCHFLAGS="-arch x86_64"
sudo make clean; CC="gcc -arch x86_64" ./configure --enable-shared --enable-static
sudo make install
go back to PIL Imaging, run these commands:
export ARCHFLAGS="-arch x86_64"
sudo rm -rf build ; python setup.py build 2> /tmp/err > /tmp/log
Check in /tmp/log - you should now see "JPEG support available", check /tmp/err - search for "jpeg" - if you see something like this:
ld: warning: ignoring file /usr/local/lib/libjpeg.dylib, file was built for unsupported file format which is not the architecture being linked (i386) - then the arch flags (both set by ARCHFLAGS and -arch) failed to trigger - investigate that case, it is crucial. If you don't see this, then you're lucky and you can invoke the installation:
export ARCHFLAGS="-arch x86_64"
sudo rm -rf build ; sudo python setup.py install
Check if your PIL works:
python selftest.py
or
echo "import _imaging" | python && echo "Works"
I hope this helps.
The problem is that the _imaging module is linked to libjpeg dynamically, not statically. The libjpeg code is not included in the _imaging module directly. This means that your platform's dynamic linker has to be able to find libjpeg in order to load and link it. My MacOS knowledge here is fuzzy, but as I recall its dynamic linker is called dyld, and its manpage may provide more information on the options you have.
Normally, the platform's dynamic linker won't be looking in your homedirectory for libraries -- but you can tell it to, for example by setting the DYLD_LIBRARY_PATH environment variable, or editing the system-wide configuration (if there is one.) Setting that environment variable usually has to be done before you start Python, though, so it may not be suitable. You may want to try to embed the runtime search path into the _imaging extension module, which is usually done by passing -rpath to the linker -- but I don't know if MacOS's linker offers that capability. Finally, you can just build libjpeg as a static library, instead of a shared one, and have the _imaging module link against that. That would avoid the whole shared library situation for libjpeg. For libjpeg, this is probably done by running its configure script with --enable-static --disable-shared.
I tried many most all of these suggestions (as well as two other suggestions on referenced blogs) on an old 10.6 Mac install. None of them worked as-is, however, reading behind the lines I was able to fix my problem. I added to the PIL setup.py in find_include_file() right before the return 1 line "print os.path.join(directory, include)". This allowed me to track down which libjpeg PIL is building against. Then I would build PIL, find the libjpeg it referenced (various copies in /sw, /opt/, /usr/local/lib, ...), and delete that libjpeg (both header and lib files).
Finally with a clean system I built and installed libjpeg source tarball that I'd downloaded myself, followed by building and installing PIL from source. This worked. As a fallback you could always disable libjpeg by removing the above files as described or else always returning zero from the described above function in setup.py.
Works well for me on Mountain Lion 10.8.2:
Step One. Removing all jpeg packages. For MacPorts:
sudo port -f uninstall jpeg or sudo port -f uninstall jpeg #version_here
We need to remove all jpeg versions!
Step Two. Remove PIL: pip uninstall PIL
Step Three. Install jpeg package again. For MacPorts: sudo port install jpeg
Step Four. Install PIL again: pip install PIL
>>> import PIL
>>> import _imaging
No errors!
How to remove ALL jpeg packages?
$ port installed | grep -i jpeg
jpeg #8c_0
jpeg #9a_0 (active)
$ sudo port -f uninstall jpeg #8c_0
$ sudo port -f uninstall jpeg #9a_0
Don't care about dependies. Beacause we need to install jpeg package again:
$ sudo port install jpeg
I ran into all the errors you guys have mentioned. I broke down and just used virtualenv and installed Pillow instead. it worked:
sudo pip install virtualenv
virtualenv python_script && cd !$
. /activate/bin
pip install Pillow
I had the same problem, with Python 2.7 and OSX Lion, and basically followed #ApPeL process and re-installed libjpeg and PIL. libjpeg seemed to be installed correctly, and PIL seemed to find it correctly, but running python -v and then import _imaging gave always this error:
ImportError:
dlopen(/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PIL/_imaging.so,
2): Symbol not found: _jpeg_resync_to_restart
After installing and re-installing libjpeg (version 8d) and PIL (version 1.1.7) twenty times with slightly different options, what did the trick was making sure I removed all libjpeg files under /usr/local/include (headers), as well as the files under /user/local/lib.
I didn't need to install PIL from source, I used pip install pil
This worked for me-
http://www.thetoryparty.com/2010/08/31/pil-on-snow-leopard-_jpeg_resync_to_restart-error/
I have been attempting to install PIL (OS X 10.7.5, Python 2.7.3) for about 5 hours. I too have been bogged down with the 'Symbol not found: _jpeg_resync_to_restart" error and have tried many of the proposed solutions to no avail, including reinstalling all of its dependencies. Finally, I discovered a double-clickable installation of Pillow.
Thank you Rudix! Now "import PIL" and "import _imaging" work!
p.s. I had installed libjpeg via i didn't specifically delete this install, so I'm not sure if this was part of the final solution or not.