Develop Reference

UsePythonVersion#0 sets wrong Python versions in Azure Pipelines

I added Python 3.9 to self hosted Agent toolcache successfully as shown in docs (https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/use-python-version-v0?view=azure-pipelines&viewFallbackFrom=azure-devops)
I am running the following pipeline:
trigger: none
pool:
name: self-hosted-agent
demands:
- Agent.Name -equals agentname
- agent.name -equals agentname
- Agent.OS -equals Linux
steps:
- task: UsePythonVersion#0
inputs:
versionSpec: '3.9'
addToPath: true
- script: |
python --version
python3 --version
python print('Hello!')
- task: Bash#3
displayName: Run Python Code
inputs:
targetType: inline
script: |
python --version
python print('Hello!')
I can see in the logs that UsePythonVersion#0 can find this additionally installed python 3.9.1 version:
Starting: UsePythonVersion
==============================================================================
Task : Use Python version
Description : Use the specified version of Python from the tool cache, optionally adding it to the PATH
Version : 0.214.0
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/tool/use-python-version
==============================================================================
Found tool in cache: Python 3.9.1 x64
Prepending PATH environment variable with directory: /home/.../myagent/_work/_tool/Python/3.9.1/x64
Prepending PATH environment variable with directory: /home/.../myagent/_work/_tool/Python/3.9.1/x64/bin
Finishing: UsePythonVersion
But when I check python version in script:
Starting: CmdLine
==============================================================================
Task : Command line
Description : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
Version : 2.212.0
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
==============================================================================
Generating script.
========================== Starting Command Output ===========================
/usr/bin/bash --noprofile --norc /home/.../myagent/_work/_temp/0225cb7f-9153-4b7c-ad42-321b8b9bfa97.sh
Python 2.7.16
Python 3.7.3
I get wrong versions (probably the ones that are installed globally in the Agent Pool machine)
But as I understand UsePythonVersion#0 task should change the version of python to the one defined in the task?

command not found when using makefile to run batch file on Windows 10

When I try to run a makefile (type "make test" in terminal) which runs a batch file I get the equivalent error:
cd directory_path && test.bat
/bin/sh: test.bat: command not found
make: *** [makefile:58: test] Error 127
The makefile is:
.PHONY: test
test:
cd directory_path && test.bat
python path/test.py
(I changed names of the batch file, the directory path, etc. to try and make things more generic)
I can manually type "cd directory_path" and then "test.bat" and those both work. This makefile works on other systems. It only doesn't work on mine. I think it is an issue with how I installed Cygwin and how I run "make.exe".
I can use make to compile C code, but I also get an error trying to use make to run python scripts. The makefile from before also has a command for python path/test.py. This also isn't working on my system. If I delete the batch file line, but keep the python command it throws the error:
python path/test.py
make: python: No such file or directory
make: *** [makefile:59: test] Error 127.
I don't understand why it throws the error of no such file or directory since the path leads to the file, and if i put the python script in the same working directory as the makefile then it still can't find it.
Any ideas/solutions on these problems? Thank you!
EDIT:
Doug Henderson:
I started a cmd prompt at the place with the make file and entered all of these commands in
uname -a
CYGWIN_NT-10.0 james-mobl2 3.1.6(0.340/5/3) 2020-07-09 08:20 x86_64 Cygwin
which make
/usr/bin/make
make -v
GNU Make 4.3
Built for x86_64-pc-cygwin
Copyright (C) 1988-2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
which python
which: no python in
(/cygdrive/c/Program Files/AdoptOpenJDK/jdk-8.0.252.09-hotspot/bin:
/cygdrive/c/windows/system32:
/cygdrive/c/windows:
/cygdrive/c/windows/System32/Wbem:
/cygdrive/c/windows/System32/WindowsPowerShell/v1.0:
/cygdrive/c/windows/System32/OpenSSH:
/cygdrive/c/Program Files/Git/cmd:
/cygdrive/c/Users/james/AppData/Local/Microsoft/WindowsApps/python:
/cygdrive/c/Program Files/PuTTY:
/usr/bin:
/cygdrive/c/Program Files/AdoptOpenJDK/jdk-8.0.252.09 hotspot/bin:
/cygdrive/c/Users/james/AppData/Local/Microsoft/WindowsApps)
(I cleaned up the output a little to make it more legible). This and which test.bat are the only commands that failed.
python -V
Python 3.8.5
The change directory changed correctly
which test.bat
which: no test.bat in
(/cygdrive/c/Program Files/AdoptOpenJDK/jdk-8.0.252.09-hotspot/bin:
/cygdrive/c/windows/system32:
/cygdrive/c/windows:
/cygdrive/c/windows/System32/Wbem:
/cygdrive/c/windows/System32/WindowsPowerShell/v1.0:
/cygdrive/c/windows/System32/OpenSSH:
/cygdrive/c/Program Files/Git/cmd:
/cygdrive/c/Users/james/AppData/Local/Microsoft/WindowsApps/python:
/cygdrive/c/Program Files/PuTTY:
/usr/bin:
/cygdrive/c/Program Files/AdoptOpenJDK/jdk-8.0.252.09 hotspot/bin:
/cygdrive/c/Users/james/AppData/Local/Microsoft/WindowsApps)
I added cmd /c test.bat to the makefile and it worked, but I've also had the batch file run on different systems without that addition. I also have a lot batch files in the actual makefile so this solution works but isn't ideal. It also didn't fix the python script not running.
I didn't initially have cmd.exe in my path, but I added it before entering those commands.
I'm not sure what you mean by starting my PATH in bash with /usr/bin;/bin;THE_REST
MadScientist:
I was manually entering it into a Windows Terminal. Is it possible to have Cygwin invoke a Windows command.com shell?
What I did was go to directory that contained the makefile. Open a windows terminal by typing 'cmd' in the address bar. I would then type 'make test'. That obviously led to the issues. I then went through and manually entered the commands I thought the makefile was supposed to invoke. So I entered 'cd directory_path && test.bat'. This ran.
Matzeri:
I don't really know. I'm completely new to POSIX, etc. I'm guessing I'm using Windows style, and judging from other responses, I'm trying to use Cygwin to do a non-POSIX style.
Thank you, everyone for responses!

Make sure cygwin is installed with basic packages --> Some basic packages not installed with cygwin try below link to install some basic package, uninstall the cygwin and install it again with following basic package :
https://wiki.usask.ca/display/MESH/Running+Python+from+the+Cygwin+Terminal#RunningPythonfromtheCygwinTerminal-Cygwin
make sure cygwin's bin folder is added to your environment variable (e.g., C:\cygwin64\bin)

If cmd /c test.bat works then . is probably not in your PATH. Always do ./test.bat when you want to run something in the current directory. Never count on the PATH unless you have set it yourself. Also make sure the file is executable. chmod +x test.bat

Nixos: How do I get get a python with debug info included with packages?

I'm trying to do cython debugging on NixOS. I can easily install cython in a nix-shell (chosen for simplicity of example), like this:
$ nix-shell -p 'python27.withPackages( p: [ p.cython ])'
$ cat /nix/store/s0w3phb2saixi0a9bzk8pjbczjaz8d7r-python-2.7.14-env/bin/python
#! /nix/store/jgw8hxx7wzkyhb2dr9hwsd9h2caaasdc-bash-4.4-p12/bin/bash -e
export PYTHONHOME="/nix/store/s0w3phb2saixi0a9bzk8pjbczjaz8d7r-python-2.7.14-env"
export PYTHONNOUSERSITE="true"
exec "/nix/store/i3bx1iw2d0i3vh9sa1nf92ynlrw324w8-python-2.7.14/bin/python" "${extraFlagsArray[#]}" "$#"
Then we do a normal nix-shell of just python and see what version of python we get.
[henry#bollum:~/Projects/eyeserver/nixshell]$ nix-shell -p 'python27'
$ which python
/nix/store/i3bx1iw2d0i3vh9sa1nf92ynlrw324w8-python-2.7.14/bin/python
# ^^^^^^^
# non-debug
All is well -- we get a non-debug python in both cases. And if we gdb it, we get (no debugging symbols found). See the last line.
$ gdb
/nix/store/i3bx1iw2d0i3vh9sa1nf92ynlrw324w8-python-2.7.14/bin/python
GNU gdb (GDB) 8.0.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /nix/store/i3bx1iw2d0i3vh9sa1nf92ynlrw324w8-python-2.7.14/bin/python...(no debugging symbols found)...done.
When we use enableDebugging, on python by itself, we get different results.
$ nix-shell -p 'enableDebugging python27'
$ which python
/nix/store/a4wd8mcpqr54hmw0x95fw8fhvk8avh5a-python-2.7.14/bin/python
# ^^^^^^
# debug
$ gdb
/nix/store/a4wd8mcpqr54hmw0x95fw8fhvk8avh5a-python-2.7.14/bin/python
GNU gdb (GDB) 8.0.1
...
Reading symbols from /nix/store/a4wd8mcpqr54hmw0x95fw8fhvk8avh5a-python-2.7.14/bin/python...done.
The problems come when we try to do this with cython (or any other package) included.
$ nix-shell -p '(enableDebugging python27).withPackages( p: [ p.cython ])'
$ cat `which python`
#! /nix/store/jgw8hxx7wzkyhb2dr9hwsd9h2caaasdc-bash-4.4-p12/bin/bash -e
export PYTHONHOME="/nix/store/s0w3phb2saixi0a9bzk8pjbczjaz8d7r-python-2.7.14-env"
export PYTHONNOUSERSITE="true"
exec "/nix/store/i3bx1iw2d0i3vh9sa1nf92ynlrw324w8-python-2.7.14/bin/python" "${extraFlagsArray[#]}" "$#"
# ^^^^^^^
# non-debug
$ gdb /nix/store/i3bx1iw2d0i3vh9sa1nf92ynlrw324w8-python-2.7.14/bin/python
GNU gdb (GDB) 8.0.1
...
Reading symbols from /nix/store/i3bx1iw2d0i3vh9sa1nf92ynlrw324w8-python-2.7.14/bin/python...(no debugging symbols found)...done.
(gdb)
quit
The version of python in the environment is now non-debug, and attempting to debug it gets the dreaded (no debugging symbols found). This makes gdb much less useful for debugging cython programs.

Solution
with import <nixpkgs> {};
let
self = enableDebugging python;
in [ gdb ((python.override{inherit self;}).withPackages(ps: with ps; [ cython ])) ]
The executable referred to in the wrapper now has debug symbols.
Background
If we look in pkgs/all-packages.nix we see the implementation of the enableDebugging function:
enableDebugging = pkg: pkg.override { stdenv = stdenvAdapters.keepDebugInfo pkg.stdenv; };
It overrides a single derivation to use a different stdenv. In your case, you would like to override the Python interpreter, which is a dependency of the derivation obtained with python.withPackages.
Your attempt with enableDebugging python was in the right direction, however, python.withPackages uses a reference to python that needs to be updated as well.

#fridh answer requires an extra little code to make this usable.
Basically you need a shell.nix with this:
with import <nixpkgs> {};
let
python = enableDebugging pkgs.python;
in
stdenv.mkDerivation {
name = "test";
buildInputs = [ python ];
}
Then when you enter the nix-shell, you can run gdb -p <pid of python process>.
Note that this only debugs the Python interpreter itself. This does not enable the python extensions to gdb allowing you to debug application level python code.
The enableDebugging also does not propagate to dependencies. Each dependency will require its own enableDebugging if you want those debug symbols to be enabled. Should there be a function to enable debugging recursively?

Why does cqlsh fail with LookupError: unknown encoding?

I've just installed Cassandra using brew on Mac OS X 10.9.4:
➜ ~ brew info cassandra
cassandra: stable 2.1.0
http://cassandra.apache.org
/usr/local/Cellar/cassandra/2.0.9 (3466 files, 79M) *
Built from source
From: https://github.com/Homebrew/homebrew/blob/master/Library/Formula/cassandra.rb
==> Caveats
If you plan to use the CQL shell (cqlsh), you will need the Python CQL library
installed. Since Homebrew prefers using pip for Python packages, you can
install that using:
pip install cql
To have launchd start cassandra at login:
ln -sfv /usr/local/opt/cassandra/*.plist ~/Library/LaunchAgents
Then to load cassandra now:
launchctl load ~/Library/LaunchAgents/homebrew.mxcl.cassandra.plist
➜ ~ uname -a
Darwin xxx 13.3.0 Darwin Kernel Version 13.3.0: Tue Jun 3 21:27:35 PDT 2014; root:xnu-2422.110.17~1/RELEASE_X86_64 x86_64
As advised in the above info message, to install cql I executed sudo easy_install pip followed by pip install cql.
With the software installed, upon executing cqlsh I'm facing the error:
➜ ~ cqlsh
Traceback (most recent call last):
File "/usr/local/bin/cqlsh", line 2084, in <module>
main(*read_options(sys.argv[1:], os.environ))
File "/usr/local/bin/cqlsh", line 2067, in main
single_statement=options.execute)
File "/usr/local/bin/cqlsh", line 509, in __init__
self.output_codec = codecs.lookup(encoding)
LookupError: unknown encoding:
How can I fix it?

After some googling and debugging, it turned out that the call to locale.getpreferredencoding() expects proper LC_ALL as described in 22.2. locale — Internationalization services:
To maintain compatibility with other platforms, not only the LANG
variable is tested, but a list of variables given as envvars
parameter. The first found to be defined will be used. envvars
defaults to the search path used in GNU gettext; it must always
contain the variable name LANG. The GNU gettext search path contains
'LANGUAGE', 'LC_ALL', 'LC_CTYPE', and 'LANG', in that order.
On my system LC_ALL was set to pl_PL:
➜ ~ echo $LC_ALL
pl_PL
After a change to LC_ALL to pl or pl_pl.utf-8 the Cassandra shell cqlsh started fine:
➜ ~ export LC_ALL=pl_pl.utf-8
➜ ~ cqlsh
Connected to Test Cluster at localhost:9160.
[cqlsh 4.1.1 | Cassandra 2.0.9 | CQL spec 3.1.1 | Thrift protocol 19.39.0]
Use HELP for help.
cqlsh>
See the thread Problem start cqlsh on OSX - Lion for a sample Python application to check your locale:
python -c 'import locale, codecs; encoding = locale.getpreferredencoding(); print encoding; print codecs.lookup(encoding)'
Once it's worked fine, the issue can be considered solved.

How to use gdb python debugging extension inside virtualenv

I'm running ubuntu, and installed the python-dbg package. When trying to use the installed version directly everything works great:
$ gdb python2.7-dbg
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
---x snipped x---
Reading symbols from /usr/bin/python2.7-dbg...done.
(gdb) r
Starting program: /usr/bin/python2.7-dbg
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Python 2.7.3 (default, Feb 27 2014, 19:39:25)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
Program received signal SIGINT, Interrupt.
0x00007ffff6997743 in __select_nocancel () at ../sysdeps/unix/syscall-template.S:82
82 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) py-bt (<--- works, just has nothing to do)
(gdb)
So, I've been building a virtualenv using the package's binary python2.7-dbg (since some libraries need recompiling), using this command line:
~$ virtualenv ved -p /usr/bin/python2.7-dbg
Its all working fine, but when I'm using gdb inside the virtualenv, atleast the python pretty printers stop working:
~$ . ved/bin/activate
(ved)~$ gdb python
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
---x snipped x---
Reading symbols from /home/itai/ved/bin/python...done.
(gdb) r
Starting program: /home/itai/ved/bin/python
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Python 2.7.3 (default, Feb 27 2014, 19:39:25)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
Program received signal SIGINT, Interrupt.
0x00007ffff6997743 in __select_nocancel () at ../sysdeps/unix/syscall-template.S:82
82 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) py-bt
Undefined command: "py-bt". Try "help". (<---- PROBLEM)
(gdb)
Am I missing something within my virtualenv?

I've solved the problem by using strace on gdb, grepping the "open" syscalls.
It seems that gdb makes a search for python-gdb.py in several paths it guesses (according to the python binary), and whenever the file is not found it just fails silently.
Eventually the way to solve the problem is to link /usr/lib/debug/usr/bin/python2.7-gdb.py into the env's bin directory. The name of the link should be <python binary name>-gdb.py, being in my case python2.7-dbg-gdb.py (...).
After that, everything seems to work.

#itai's answer only partially worked for me on Ubuntu Trusty (14.04). I found a couple of other things worked better:
sudo apt-get install python2.7-dbg
then, in the virtualenv:
. bin/activate
mkdir bin/.debug
ln -s /usr/lib/debug/usr/bin/python2.7-gdb.py bin/.debug/python-gdb.py
ln -s /usr/lib/debug/usr/bin/python2.7 bin/.debug/
gdb --args bin/python2.7 ...
This helped gdb find the python debugging symbols as well as the py-bt etc commands.

In Debian 11 with Python 3.7, gdb debugging works out of the box inside virtualenv.
Make sure that you created the venv with the same Python3 version than the python3-dbg package version installed.
In case symbols don't load correctly, rebuild the venv from scratch and try again.

On Ubuntu 12.04, #craigds's answer was very helpful but didn't get me quite all the way there: I was still running into:
IOError: invalid Python installation: unable to open /path/to/venv/lib/python2.7/config_d/Makefile (No such file or directory)
Fixed that, then I ran into:
IOError: invalid Python installation: unable to open /path/to/venv/local/include/python2.7_d/pyconfig.h (No such file or directory)
So the full steps for me to fix up my virtualenv were:
source /path/to/venv/bin/activate
mkdir /path/to/venv/bin/.debug
ln -s /usr/lib/debug/usr/bin/python2.7-gdb.py /path/to/venv/bin/.debug/python-gdb.py
ln -s /usr/lib/debug/usr/bin/python2.7 /path/to/venv/bin/.debug/
ln -s /usr/lib/python2.7/config_d/ /path/to/venv/lib/python2.7/config_d
ln -s /usr/include/python2.7_d/ /path/to/venv/local/include/python2.7_d
ln -s /usr/lib/debug/usr/bin/python2.7-gdb.py /path/to/venv/bin/python-gdb.py