Develop Reference

Can't install dbt-snowflake; "No module named 'cmake'"

(Context: Barely a novice at Terminal, know some SQL, don't know Python)
I'm taking a course on dbt, have my own Snowflake set up, and I'm trying to install dbt-snowflake on Mac. Attempting to run:
pip install dbt-snowflake
This fails, and it returns:
`
Traceback (most recent call last):
File "/Users/xxxx/course/venv/bin/cmake", line 5, in <module>
from cmake import cmake
ModuleNotFoundError: No module named 'cmake'
error: command '/Users/xxxx/course/venv/bin/cmake' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for pyarrow
Failed to build pyarrow
ERROR: Could not build wheels for pyarrow, which is required to install pyproject.toml-based projects
`
I once successfully changed Python version in virtual environment to 3.9 to see whether that was the problem (but I can't replicate how I changed it), that didn't solve the problem, and currently it's back to 3.11.
Have installed multiple versions of snowflake (3.7, 3.9, 3.11). I don't really understand how this works in context of everything.
I have tried deleting and recreating virtual environments (under same name, venv)
uninstalled and reinstalled cmake (pip install --upgrade cmake); this is version 3.25.0
I don't understand how I can have cmake installed and be told there is no module named 'cmake'.
Is there a particular version of Python I should be running, and if so, how do I reliably change that in virtual environments and different directories?
EDIT TO ADDRESS A COMMENT: I installed Python versions via terminal brew install python#3.9

Failed building wheel for pyarrow in pypy3 interpreter

I am installing streamlit with pypy3 as interpreter in pycharm and stuck at this ERROR: Failed building wheel for pyarrow
I tried every solutions found on the web related with pyarrow, but seems like all solutions posted are for python as interpreter and not for pypy. If you guys have any solution, please let me know.
Command : pip install streamlit
Operating system : ubuntu 22.04
pypy3 version : 7.3.9
Traceback (most recent call last):
File "/home/lungsang/Desktop/streamlit/.env/bin/cmake", line 5, in <module>
from cmake import cmake
ModuleNotFoundError: No module named 'cmake'
error: command '/home/lungsang/Desktop/streamlit/.env/bin/cmake' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for pyarrow
Failed to build pyarrow
ERROR: Could not build wheels for pyarrow, which is required to install pyproject.toml-based projects
I have installed cmake before running the command.

You need to pypy3 -m pip install cmake. With that, I am not sure this will work. Even if it does, pyarrow is a c++-based project, so the PyPy python JIT will have a hard time finding opportunities to speed up python code. Additionally, the need to emulate the CPython interactions between python and C mean the code may actually run slower under PyPy than under CPython, if you can get it to run at all.

Seems like currently pyarrow can't be use with pypy.
https://github.com/apache/arrow/issues/2089

Cannot install nlopt python module

I am trying to install nlopt onto macOS 10.15.5. I downloaded the nlopt-2.6.2.tar.gz file from the NLopt docs and ran the following from the nlopt-2.6.2 directory:
mkdir build
cd build
cmake -DNLOPT_OCTAVE=Off -DNLOPT_MATLAB=Off -DNLOPT_GUILE=Off ..
make
sudo make install
I got the following output: cmake.txt.
The header file (nlopt.h) installs correctly to /usr/local/include and the dynamic library (libnlopt.dylib) installs correctly to /usr/local/lib/, but neither the dist-info file nor the nlopt module itself installs.
I have also tried installing via pip, brew, and conda, none of which have worked. I have also tried cloning from this Github, which didn't work either.
I appreciate any help with this, because I am completely lost. I am relatively new to this kind of stuff, and I couldn't find any good answers online.

The official docs are somewhat laconic about the exact steps required for building nlopt with Python bindings. First of all, you'll need SWIG installed:
$ brew install swig
Then, you'll need numpy to be available for the target Python interpreter. It is already preinstalled for the system Python, otherwise install it via Homebrew or pip, depending on your Python installation.
Now run cmake:
$ cmake -DNLOPT_GUILE=OFF -DNLOPT_MATLAB=OFF -DNLOPT_OCTAVE=OFF -DNLOPT_TESTS=OFF
This will build bindings against the default Python 2.7 installation preinstalled on MacOS. If you need to build against custom Python installation (e.g. when you've installed Python 3 via Homebrew or PKG installer from https://www.python.org/downloads), pass it via the PYTHON_EXECUTABLE arg:
$ cmake -DNLOPT_GUILE=OFF -DNLOPT_MATLAB=OFF -DNLOPT_OCTAVE=OFF -DNLOPT_TESTS=OFF -DPYTHON_EXECUTABLE=/usr/local/bin/python3
Inspect the log now - Python, SWIG and numpy headers should be successfully located. Example output snippet (you may have different paths/versions printed):
-- Found PythonInterp: /usr/local/bin/python3.8 (found version "3.8.3")
-- Found PythonLibs: /Library/Frameworks/Python.framework/Versions/3.8/lib/libpython3.8.dylib (found suitable exact version "3.8.3")
-- Found NumPy: /Users/hoefling/Library/Python/3.8/lib/python/site-packages/numpy/core/include (found version "1.19")
-- Found SWIG: /usr/local/bin/swig (found version "4.0.2")
If any of those conditions is not satisfied (e.g. you see Could NOT find NumPy, Could NOT find PythonLibs or Could NOT find SWIG), then stop and make sure the configuration succeeds before proceeding next.
Now compile:
$ make
...
Scanning dependencies of target nlopt_python_swig_compilation
[ 96%] Swig compile nlopt.i for python
[ 96%] Built target nlopt_python_swig_compilation
Scanning dependencies of target nlopt_python
[ 98%] Building CXX object src/swig/CMakeFiles/nlopt_python.dir/CMakeFiles/nlopt_python.dir/nloptPYTHON_wrap.cxx.o
[100%] Linking CXX shared module _nlopt.so
[100%] Built target nlopt_python
Install:
$ make install
...
-- Installing: /usr/local/lib/python3.8/site-packages/nlopt.py
-- Installing: /usr/local/lib/python3.8/site-packages/_nlopt.so
Test the Python bindings are importable:
$ python -c "import nlopt; print(nlopt.__version__)"
2.6.2

gmpy2 installs but can't find libmpc.so.3

I want to use gmpy2 with python 2.7 but when I try to import it I get:
>>> import gmpy2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: libmpc.so.3: cannot open shared object file: No such file or directory
I installed gmpy2 using pip: pip install -user gmpy2 and the install looks ok apart from saying
Could not find .egg-info directory in install record for gmpy2
but after that it says that the install was a success.
I have installed MPC (1.0.3), GMP (6.1.1) and MPFR (3.1.4) and they all work, by which I mean I can call gcc foo.c -lmpc and gcc bar.c -lmpfr and the code compiles and works as expected. I've also got gmpy working using pip install. I think the problem will be to do with them not being installed in the default directories as I don't have sudo rights.
The directory where libmpc.so.3 is located is in the gcc call that pip spits out, I've also set CPATH and CPPFLAGS to look in my_prefix/include and LDFLAGS to look my_prefix/lib.
I don't really want to use the functionality from MPC so if there's a simple option to not install that part of gmpy2 I'd be happy with that.
I'm really confused, I've had it that pip fails to build a library and I've gone away and installed dependencies but normally once a library is passed pip it works.

I maintain gmpy2 and there are a couple of command line options that can be passed to setup.py that may help. I can't test the pip syntax right now but here are some options:
--shared=/path/to/gmp,mpfr,mpc will configure gmpy2 to load the libraries from the specified directory.
--static or --static=/path/to/gmp,mpfr,mpc will create a statically linked version of gmpy2 if the proper libraries can be found.
You can also try a build using setup.py directly. It may produce better error messages. Again, untested command:
python setup.py build_ext --static=/path/to/gmp,mpfr,mpc should compile a standalone, staticly linked gmpy2.so which will need to moved to the appropriate location.
Update
I've been able to test the options to pip.
If you are trying to use versions of GMP, MPFR, and MPC that are not those provided by the Linux distribution, you will need to specify the location of the new files to the underlying setup.py that is called by pip. For example, I have updated versions installed locally in /home/case/local. The following command will configure gmpy2 to use those versions:
pip install --install-option="--shared=/home/case/local" --user gmpy2
To compile a statically linked version (for example, to simplify distribution to other systems in cluster), you should use the following:
pip install --install-option="--static=/home/case/local" --user gmpy2
setup.py will use the specified base directory to configure the correct INCLUDE path (/home/case/local/include) and runtime library path (/home/case/local/lib).

Try to do the following as it might me fixed in an older version:
pip install --upgrade setuptools pip
pip uninstall gmpy2
pip install gmpy2

How to deal with Linux/Python dependencies?

Due to lack of support for some libraries I want to use, I moved some Python development from Windows to Linux development. I've spent most of the day messing about getting nowhere with dependencies.
The question
Whenever I pick up Linux, I usually run into some kind of dependency issue, usually with development libraries, whether they're installed via apt-get, easy_install or pip. I can waste days on what should be simple tasks, spending longer on getting libraries to work than writing code. Where can I learn about strategy for dealing with these kind of issues rather than aimlessly googling for someone who's come across the same problem before?
An example
Just one example: I wanted to generate some QR codes. So, I thought I'd use github.com/bitly/pyqrencode which is based on pyqrcode.sourceforge.net but supposedly without the Java dependencies. There are others (pyqrnative, github.com/Arachnid/pyqrencode) but that one seemed like the best bet for my needs.
So, I found the package on pypi and thought using that would make life easier:
(I've perhaps made life more difficult for myself by using virtualenv to keep things neat and tidy.)
(myenv3)mat#ubuntu:~/myenv3$ bin/pip install pyqrencode
Downloading/unpacking pyqrencode
Downloading pyqrencode-0.2.tar.gz
Running setup.py egg_info for package pyqrencode
Installing collected packages: pyqrencode
Running setup.py install for pyqrencode
building 'qrencode' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c qrencode.c -o build/temp.linux-i686-2.7/qrencode.o
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions build/temp.linux-i686-2.7/qrencode.o -lqrencode -o build/lib.linux-i686-2.7/qrencode.so
Successfully installed pyqrencode
Cleaning up...
(I guess I probably sudo apt-get install libqrencode-dev at some point prior to that too.)
So then I tried to run the test script:
(myenv3)mat#ubuntu:~/myenv3$ python test_qr.py
Traceback (most recent call last):
File "test_qr.py", line 1, in <module>
from qrencode import Encoder
File "qrencode.pyx", line 1, in init qrencode (qrencode.c:1520)
ImportError: No module named ImageOps
:(
Well, investigations revealed that ImageOps appears to be part of PIL...
(myenv3)mat#ubuntu:~/myenv3$ pip install pil
Downloading/unpacking pil
Downloading PIL-1.1.7.tar.gz (506Kb): 122Kb downloaded
Operation cancelled by user
Storing complete log in /home/mat/.pip/pip.log
(myenv3)mat#ubuntu:~/myenv3$ bin/pip install pil
Downloading/unpacking pil
Downloading PIL-1.1.7.tar.gz (506Kb): 506Kb downloaded
Running setup.py egg_info for package pil
WARNING: '' not a valid package name; please use only.-separated package names in setup.py
Installing collected packages: pil
Running setup.py install for pil
WARNING: '' not a valid package name; please use only.-separated package names in setup.py
building '_imaging' extension
gcc ...
building '_imagingmath' extension
gcc ...
--------------------------------------------------------------------
PIL 1.1.7 SETUP SUMMARY
--------------------------------------------------------------------
version 1.1.7
platform linux2 2.7.1+ (r271:86832, Apr 11 2011, 18:05:24)
[GCC 4.5.2]
--------------------------------------------------------------------
*** TKINTER support not available
*** JPEG support not available
*** ZLIB (PNG/ZIP) support not available
*** FREETYPE2 support not available
*** LITTLECMS support not available
--------------------------------------------------------------------
To add a missing option, make sure you have the required
library, and set the corresponding ROOT variable in the
setup.py script.
To check the build, run the selftest.py script.
...
Successfully installed pil
Cleaning up...
Hmm, PIL's installed but hasn't picked up the libraries I installed with sudo apt-get install libjpeg62 libjpeg62-dev libpng12-dev zlib1g zlib1g-dev earlier. I'm not sure how to tell pip to feed the library locations to setup.py. Googling suggests a variety of ideas which I've tried, but none of them seem to help much other than to send me round in circles.
Ubuntu 11.04: Installing PIL into a virtualenv with PIP suggests using the pillow package instead, so let's try that:
(myenv3)mat#ubuntu:~/myenv3$ pip install pillow
Downloading/unpacking pillow
Downloading Pillow-1.7.5.zip (637Kb): 637Kb downloaded
Running setup.py egg_info for package pillow
...
Installing collected packages: pillow
Running setup.py install for pillow
building '_imaging' extension
gcc ...
--------------------------------------------------------------------
SETUP SUMMARY (Pillow 1.7.5 / PIL 1.1.7)
--------------------------------------------------------------------
version 1.7.5
platform linux2 2.7.1+ (r271:86832, Apr 11 2011, 18:05:24)
[GCC 4.5.2]
--------------------------------------------------------------------
*** TKINTER support not available
--- JPEG support available
--- ZLIB (PNG/ZIP) support available
--- FREETYPE2 support available
*** LITTLECMS support not available
--------------------------------------------------------------------
To add a missing option, make sure you have the required
library, and set the corresponding ROOT variable in the
setup.py script.
To check the build, run the selftest.py script.
...
Successfully installed pillow
Cleaning up...
Well, we seem to have the JPEG and PNG support this time, yay!
(myenv3)mat#ubuntu:~/myenv3$ python test_qr.py
Traceback (most recent call last):
File "test_qr.py", line 1, in <module>
from qrencode import Encoder
File "qrencode.pyx", line 1, in init qrencode (qrencode.c:1520)
ImportError: No module named ImageOps
Still no ImageOps though. Now I'm stumped, is ImageOps missing from pillow, or is it a different problem that was also there with pil.

I see two separate problems here:
Keeping track of all the python modules you need for your project.
Keeping track of all the dynamic libraries you need for the python modules in your project.
For the first problem, I have found that buildout is good help, althought it takes a litle while to grasp.
In your case, I would start by creating a directory for my new project. I would then go into that directory and download bootstrap.py
wget http://python-distribute.org/bootstrap.py
I would then create a buildout.cfg file:
[buildout]
parts = qrproject
python
eggs = pyqrencode
[qrproject]
recipe = z3c.recipe.scripts
eggs = ${buildout:eggs}
entry-points= qrproject=qrprojectmodule:run
extra-paths = ${buildout:directory}
# This is a simple way of creating an interpreter that will have
# access to all the eggs / modules that this project uses.
[python]
recipe = z3c.recipe.scripts
interpreter = python
eggs = ${buildout:eggs}
extra-paths = ${buildout:directory}
In this buildout.cfg I'm referencing the module qrprojectmodule (in entry-points under [qrproject]. This will create a bin/qrproject that runs the function run in the module qrprojectmodule. So I will also create the file qrprojectmodule.py
import qrencode
def run():
print "Entry point for qrproject. Happily imports qrencode module"
Now it's time to run bootstrap.py with the python binary you want to use:
python bootstrap.py
Then run the generated bin/buildout
bin/buildout
This will create two additional binaries in the bin/ directory - bin/qrproject and bin/python. The former is your project's main binary. It will be created automatically each time you run buildout and will have all the modules and eggs you want loaded. The second is simply a convenient way to get a python prompt where all your modules and eggs are loaded, for easy debugging. The fine thing here is that bin/buildout will automatically install any python eggs that the eggs (in your case pyqrencode) have specified as dependencies.
Actually, you will probably get a compilation error in the step where you run bin/buildout. This is because you need to address problem 2: All dynamic libraries being available on your system. On Linux, it's usually best to get help from your packaging system. I'm going to assume you're using a Debian derivate such as Ubuntu here.
The pyqrencode web site specifies that you need the libqrencode library for pyqrencode to work. So I used my package manager to search for that:
$ apt-cache search libqrencode
libqrencode-dev - QR Code encoding library -- development
libqrencode3 - QR Code encoding library
qrencode - QR Code encoder into PNG image
In this case, I want the -dev package, as that installs linkable libraries and header files required to compile python C-modules. Also, the dependency system in the package manager will make sure that if I install libqrencode-dev, I will also get libqrencode3, as that is required at runtime, i.e. after compilation of the module.
So, I install the package:
sudo apt-get install libqrencode-dev
Once that has completed, rerun bin/buildout and the pyqrencode module will (hopefully) compile and install successfully. Now try to run bin/qrproject
$ bin/qrproject
Entry point for qrproject. Happily imports qrencode module
Success! :-)
So, in summary:
Use buildout to automatically download and install all the python modules/eggs you need for your project.
Use your system's package manager to install any dynamic (C) libraries required by the python modules you use.
Be aware that in many cases there are already packaged versions of your python modules available in the package system. For example, pil is available by installing the python-imaging package on Ubuntu. In this case, you don't need to install it via buildout, and you don't need to worry about libraries being available - the package manager will install all dependencies required for the module to run. Doing it via buildout can however make it easier to distribute your project and make it run on other systems.

Your story reminds me of my early experiences with Linux, and why I love APT.
There is no universal solution to your general problem; the best you can do is to take advantage of the work or others. The Debian packagers do a great job of flagging the dependencies of packages, so apt-get will pull in what you need. So, my strategy is simply to avoid building and installing stuff on my own, and use apt-get wherever possible.
Note that Ubuntu is based on Debian and thus gains the benefit of the work of the Debian packagers. I haven't used Fedora but I hear that the packages are not as well-organized as the ones from Debian.