Develop Reference

Error installing python 3.7.6 using pyenv on new macbook pro M1 in OS 12.3+

I am struggling to install python version 3.7.6 using pyenv on my new macbook pro M1 running on mac os 12.3.1.
My configuration
$ clang -v
Apple clang version 13.1.6 (clang-1316.0.21.2)
Target: arm64-apple-darwin21.4.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
$ pyenv install 3.7.6
python-build: use openssl#1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.7.6.tar.xz...
-> https://www.python.org/ftp/python/3.7.6/Python-3.7.6.tar.xz
Installing Python-3.7.6...
python-build: use tcl-tk from homebrew
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
BUILD FAILED (OS X 12.3.1 using python-build 2.2.5-10-g58427b9a)
Inspect or clean up the working tree at /var/folders/4t/1qfwng092qz2qxwxm6ss2f1c0000gp/T/python-build.20220405170233.32567
Results logged to /var/folders/4t/1qfwng092qz2qxwxm6ss2f1c0000gp/T/python-build.20220405170233.32567.log
Last 10 log lines:
checking for --with-cxx-main=<compiler>... no
checking for clang++... no
configure:
By default, distutils will build C++ extension modules with "clang++".
If this is not intended, then set CXX on the configure command line.
checking for the platform triplet based on compiler characteristics... darwin
configure: error: internal configure error for the platform triplet, please file a bug report

Finally this patch works in installing 3.7.6 on macbook m1 using pyenv.
To install python 3.7.6 version in mac os 12+ , M1 chip, apple clang version 13+ using pyenv, create a file anywhere in your local and call it python-3.7.6-m1.patch and copy the contents(below) to that file and save it.
diff --git a/configure b/configure
index b769d59629..8b018b6fe8 100755
--- a/configure
+++ b/configure
## -3370,7 +3370,7 ## $as_echo "#define _BSD_SOURCE 1" >>confdefs.h
# has no effect, don't bother defining them
Darwin/[6789].*)
define_xopen_source=no;;
- Darwin/1[0-9].*)
+ Darwin/[12][0-9].*)
define_xopen_source=no;;
# On AIX 4 and 5.1, mbstate_t is defined only when _XOPEN_SOURCE == 500 but
# used in wcsnrtombs() and mbsnrtowcs() even if _XOPEN_SOURCE is not defined
## -5179,8 +5179,6 ## $as_echo "$as_me:
fi
-MULTIARCH=$($CC --print-multiarch 2>/dev/null)
-
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for the platform triplet based on compiler characteristics" >&5
$as_echo_n "checking for the platform triplet based on compiler characteristics... " >&6; }
## -5338,6 +5336,11 ## $as_echo "none" >&6; }
fi
rm -f conftest.c conftest.out
+if test x$PLATFORM_TRIPLET != xdarwin; then
+ MULTIARCH=$($CC --print-multiarch 2>/dev/null)
+fi
+
+
if test x$PLATFORM_TRIPLET != x && test x$MULTIARCH != x; then
if test x$PLATFORM_TRIPLET != x$MULTIARCH; then
as_fn_error $? "internal configure error for the platform triplet, please file a bug report" "$LINENO" 5
## -9247,6 +9250,9 ## fi
ppc)
MACOSX_DEFAULT_ARCH="ppc64"
;;
+ arm64)
+ MACOSX_DEFAULT_ARCH="arm64"
+ ;;
*)
as_fn_error $? "Unexpected output of 'arch' on OSX" "$LINENO" 5
;;
diff --git a/configure.ac b/configure.ac
index 49acff3136..2f66184b26 100644
--- a/configure.ac
+++ b/configure.ac
## -490,7 +490,7 ## case $ac_sys_system/$ac_sys_release in
# has no effect, don't bother defining them
Darwin/#<:#6789#:>#.*)
define_xopen_source=no;;
- Darwin/1#<:#0-9#:>#.*)
+ Darwin/#<:#[12]#:>##<:#0-9#:>#.*)
define_xopen_source=no;;
# On AIX 4 and 5.1, mbstate_t is defined only when _XOPEN_SOURCE == 500 but
# used in wcsnrtombs() and mbsnrtowcs() even if _XOPEN_SOURCE is not defined
## -724,8 +724,7 ## then
fi
-MULTIARCH=$($CC --print-multiarch 2>/dev/null)
-AC_SUBST(MULTIARCH)
+
AC_MSG_CHECKING([for the platform triplet based on compiler characteristics])
cat >> conftest.c <<EOF
## -880,6 +879,11 ## else
fi
rm -f conftest.c conftest.out
+if test x$PLATFORM_TRIPLET != xdarwin; then
+ MULTIARCH=$($CC --print-multiarch 2>/dev/null)
+fi
+AC_SUBST(MULTIARCH)
+
if test x$PLATFORM_TRIPLET != x && test x$MULTIARCH != x; then
if test x$PLATFORM_TRIPLET != x$MULTIARCH; then
AC_MSG_ERROR([internal configure error for the platform triplet, please file a bug report])
NOW we can Install python 3.7.6 using pyenv as follows (need to be in the same directory as the patch file that we just created):
pyenv install --patch 3.7.6 < python-3.7.6-m1.patch
To install other python version on mac os 12+ , M1 chip, apple clang version 13+ using pyenv (not tested but should work)
Shallow clone the branch of python version you are interested in installing. go to https://github.com/python/cpython and find the versions available for cloning under "tags" dropdown
git clone https://github.com/python/cpython --branch v3.x.x --single-branch
cd cpython
Now make changes to the two files in it (configure.ac and configure). the git diff should look like the one shown above. The line numbers will be different based on which version of python you are installing, this git diff file is for 3.7.6 and can't be directly used for other versions. for other versions of python, search for the exact line of code being edited/deleted in the exact file as shown in the above git diff and make the changes accordingly. then save the git diff in a new file as follows.
git diff > python-3.x.x-m1.patch
Now we can install that version using:
pyenv install --patch 3.x.x < python-3.x.x-m1.patch

If you are not bound to a particular patch (i.e. z in x.y.z), this excerpt from the following link could help you:
I can confirm yesterday's releases 3.7.13, 3.8.13, 3.9.11 and 3.10.3 all work fine on my Intel mac. The GCC solution was not a viable one for me as it caused issues with pip modules that were built with clang. I don't think I need 3.6 or older branches at the moment but I suspect they might need manual patches via pyenv since they're EOL upstream.
I personally needed the 3.8.x version and succeeded with pyenv install 3.8.13.

ansible commands run only with absolute path

On Ubuntu 20.04.2 LTS there is ansible engine installed with pip3 command:
mariusz#g3:~$ pip3 show ansible
Name: ansible
Version: 4.1.0
However running ansible commands ends with below error:
mariusz#g3:~$ ansible
python3: can't open file '/usr/bin/ansible': [Errno 2] No such file or directory
The PATH variable is set correctly:
mariusz#g3:~$ which ansible
/home/mariusz/.local/bin/ansible
And I can run ansible command with absolute path:
mariusz#g3:~$ /home/mariusz/.local/bin/ansible --version
ansible [core 2.11.1]
config file = /home/mariusz/.ansible.cfg
configured module search path = ['/home/mariusz/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /home/mariusz/.local/lib/python3.8/site-packages/ansible
ansible collection location = /home/mariusz/.ansible/collections:/usr/share/ansible/collections
executable location = /home/mariusz/.local/bin/ansible
python version = 3.8.5 (default, May 27 2021, 13:30:53) [GCC 9.3.0]
jinja version = 2.11.3
libyaml = True
Any ideas how to solve it without root privileges i.e. creating /usr/bin/ansible symlink?

The entries in the $PATH variable are tried in order, and thus you'd want to relocate your $HOME/.local/bin to the beginning of the list in order for it to win out over the /usr/bin entry that's there now
You can do this in an interactive shell to confirm or deny the theory, and then put at the end of your ~/.bashrc to make it permanent
PATH=$HOME/.local/bin:$PATH

It seems that ansible package, which was installed before, left bash aliases file that was not removed during package uninstall.
$ cat ~/.bash_aliases
alias ansible='python3 /usr/bin/ansible'
alias ansible-doc='python3 /usr/bin/ansible-doc'
alias ansible-galaxy='python3 /usr/bin/ansible-galaxy'
alias ansible-inventory='python3 /usr/bin/ansible-inventory'
alias ansible-playbook='python3 /usr/bin/ansible-playbook'
alias ansible-vault='python3 /usr/bin/ansible-vault'

Use facts gathered by ansible programmatically

I'd like to write a Python program that uses the facts that Ansible gives me with ansible HOST -m setup.
When I call this, I get a response which makes it only almost pure JSON:
$ ansible localhost -m setup
localhost | success >> {
// actual data
}
Is there some way to get this JSON response directly without parsing the shell output (which might not be too stable)? Could I even use Ansible directly in a Python 3 program?

version stable-2.2, stable-2.3, and 2.4+
The latest ansible releases for 2.2, 2.3, and 2.4 all support ANSIBLE_STDOUT_CALLBACK variable. To use it, you need to add an ansible.cfg file that looks like:
[defaults]
bin_ansible_callbacks = True
callback_plugins = ~/.ansible/callback_plugins
You can place it wherever you're using ansible. Then, you need to create the callback_plugins directory, if you haven't already. Finally, you need to add a custom json parser to the directory. I copied the json parser that is bundled with ansible to the callback_plugins directory, then edited a single line in it to make it work.
I found the json.py file by first executing ansible --version
$ ansible --version
ansible 2.4.0.0
config file = /Users/artburkart/Code/ansible.cfg
configured module search path = [u'/Users/artburkart/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python2.7/site-packages/ansible
executable location = /usr/local/bin/ansible
python version = 2.7.13 (default, Jul 18 2017, 09:17:00) [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)]
Then using the "ansible python module location" to find the json.py.
cp /usr/local/lib/python2.7/site-packages/ansible/plugins/callback/json.py ~/.ansible/callback_plugins/
Finally I edited the v2_runner_on_ok function in the json.py file to look like this (courtesy of armab on GitHub):
def v2_runner_on_ok(self, result, **kwargs):
host = result._host
self.results[-1]['tasks'][-1]['hosts'][host.name] = result._result
print(json.dumps({host.name: result._result}, indent=4))
Once that was all set up, the command is very simple:
ANSIBLE_STDOUT_CALLBACK=json ansible all -i localhost, -c local -m setup | jq
If you always want to parse JSON output, you can add the following line to the end of the ansible.cfg file I described above.
stdout_callback = json
That way, you don't need to include the environment variable anymore.
versions <= latest 2.2 stable
When querying against instances, I use the following command:
ansible all --inventory 127.0.0.1, --connection local --module-name setup | sed '1 s/^.*|.*=>.*$/{/g'
If you pipe the output into jq, as leucos suggested, it happily parses the semi-valid JSON. For example:
ansible all -i hosts -m setup | sed '1 s/^.*|.*=>.*$/{/g' | jq -r '.ansible_facts.ansible_distribution'
CentOS
Ubuntu

If Python2 is OK for you, you can use the Ansible API directly. You can find detailled instructions here: http://docs.ansible.com/developing_api.html
It's really easy.
And alternate, shell centric way is to use jq. There is a quick intro here: http://xmodulo.com/how-to-parse-json-string-via-command-line-on-linux.html

Why does cqlsh fail with LookupError: unknown encoding?

I've just installed Cassandra using brew on Mac OS X 10.9.4:
➜ ~ brew info cassandra
cassandra: stable 2.1.0
http://cassandra.apache.org
/usr/local/Cellar/cassandra/2.0.9 (3466 files, 79M) *
Built from source
From: https://github.com/Homebrew/homebrew/blob/master/Library/Formula/cassandra.rb
==> Caveats
If you plan to use the CQL shell (cqlsh), you will need the Python CQL library
installed. Since Homebrew prefers using pip for Python packages, you can
install that using:
pip install cql
To have launchd start cassandra at login:
ln -sfv /usr/local/opt/cassandra/*.plist ~/Library/LaunchAgents
Then to load cassandra now:
launchctl load ~/Library/LaunchAgents/homebrew.mxcl.cassandra.plist
➜ ~ uname -a
Darwin xxx 13.3.0 Darwin Kernel Version 13.3.0: Tue Jun 3 21:27:35 PDT 2014; root:xnu-2422.110.17~1/RELEASE_X86_64 x86_64
As advised in the above info message, to install cql I executed sudo easy_install pip followed by pip install cql.
With the software installed, upon executing cqlsh I'm facing the error:
➜ ~ cqlsh
Traceback (most recent call last):
File "/usr/local/bin/cqlsh", line 2084, in <module>
main(*read_options(sys.argv[1:], os.environ))
File "/usr/local/bin/cqlsh", line 2067, in main
single_statement=options.execute)
File "/usr/local/bin/cqlsh", line 509, in __init__
self.output_codec = codecs.lookup(encoding)
LookupError: unknown encoding:
How can I fix it?

After some googling and debugging, it turned out that the call to locale.getpreferredencoding() expects proper LC_ALL as described in 22.2. locale — Internationalization services:
To maintain compatibility with other platforms, not only the LANG
variable is tested, but a list of variables given as envvars
parameter. The first found to be defined will be used. envvars
defaults to the search path used in GNU gettext; it must always
contain the variable name LANG. The GNU gettext search path contains
'LANGUAGE', 'LC_ALL', 'LC_CTYPE', and 'LANG', in that order.
On my system LC_ALL was set to pl_PL:
➜ ~ echo $LC_ALL
pl_PL
After a change to LC_ALL to pl or pl_pl.utf-8 the Cassandra shell cqlsh started fine:
➜ ~ export LC_ALL=pl_pl.utf-8
➜ ~ cqlsh
Connected to Test Cluster at localhost:9160.
[cqlsh 4.1.1 | Cassandra 2.0.9 | CQL spec 3.1.1 | Thrift protocol 19.39.0]
Use HELP for help.
cqlsh>
See the thread Problem start cqlsh on OSX - Lion for a sample Python application to check your locale:
python -c 'import locale, codecs; encoding = locale.getpreferredencoding(); print encoding; print codecs.lookup(encoding)'
Once it's worked fine, the issue can be considered solved.

How to use gdb python debugging extension inside virtualenv

I'm running ubuntu, and installed the python-dbg package. When trying to use the installed version directly everything works great:
$ gdb python2.7-dbg
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
---x snipped x---
Reading symbols from /usr/bin/python2.7-dbg...done.
(gdb) r
Starting program: /usr/bin/python2.7-dbg
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Python 2.7.3 (default, Feb 27 2014, 19:39:25)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
Program received signal SIGINT, Interrupt.
0x00007ffff6997743 in __select_nocancel () at ../sysdeps/unix/syscall-template.S:82
82 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) py-bt (<--- works, just has nothing to do)
(gdb)
So, I've been building a virtualenv using the package's binary python2.7-dbg (since some libraries need recompiling), using this command line:
~$ virtualenv ved -p /usr/bin/python2.7-dbg
Its all working fine, but when I'm using gdb inside the virtualenv, atleast the python pretty printers stop working:
~$ . ved/bin/activate
(ved)~$ gdb python
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
---x snipped x---
Reading symbols from /home/itai/ved/bin/python...done.
(gdb) r
Starting program: /home/itai/ved/bin/python
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Python 2.7.3 (default, Feb 27 2014, 19:39:25)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
Program received signal SIGINT, Interrupt.
0x00007ffff6997743 in __select_nocancel () at ../sysdeps/unix/syscall-template.S:82
82 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) py-bt
Undefined command: "py-bt". Try "help". (<---- PROBLEM)
(gdb)
Am I missing something within my virtualenv?

I've solved the problem by using strace on gdb, grepping the "open" syscalls.
It seems that gdb makes a search for python-gdb.py in several paths it guesses (according to the python binary), and whenever the file is not found it just fails silently.
Eventually the way to solve the problem is to link /usr/lib/debug/usr/bin/python2.7-gdb.py into the env's bin directory. The name of the link should be <python binary name>-gdb.py, being in my case python2.7-dbg-gdb.py (...).
After that, everything seems to work.

#itai's answer only partially worked for me on Ubuntu Trusty (14.04). I found a couple of other things worked better:
sudo apt-get install python2.7-dbg
then, in the virtualenv:
. bin/activate
mkdir bin/.debug
ln -s /usr/lib/debug/usr/bin/python2.7-gdb.py bin/.debug/python-gdb.py
ln -s /usr/lib/debug/usr/bin/python2.7 bin/.debug/
gdb --args bin/python2.7 ...
This helped gdb find the python debugging symbols as well as the py-bt etc commands.

In Debian 11 with Python 3.7, gdb debugging works out of the box inside virtualenv.
Make sure that you created the venv with the same Python3 version than the python3-dbg package version installed.
In case symbols don't load correctly, rebuild the venv from scratch and try again.

On Ubuntu 12.04, #craigds's answer was very helpful but didn't get me quite all the way there: I was still running into:
IOError: invalid Python installation: unable to open /path/to/venv/lib/python2.7/config_d/Makefile (No such file or directory)
Fixed that, then I ran into:
IOError: invalid Python installation: unable to open /path/to/venv/local/include/python2.7_d/pyconfig.h (No such file or directory)
So the full steps for me to fix up my virtualenv were:
source /path/to/venv/bin/activate
mkdir /path/to/venv/bin/.debug
ln -s /usr/lib/debug/usr/bin/python2.7-gdb.py /path/to/venv/bin/.debug/python-gdb.py
ln -s /usr/lib/debug/usr/bin/python2.7 /path/to/venv/bin/.debug/
ln -s /usr/lib/python2.7/config_d/ /path/to/venv/lib/python2.7/config_d
ln -s /usr/include/python2.7_d/ /path/to/venv/local/include/python2.7_d
ln -s /usr/lib/debug/usr/bin/python2.7-gdb.py /path/to/venv/bin/python-gdb.py