Develop Reference

How do I get error messages from nosetests

The nosetest command is failing with no messages. If I cd to my home directory I get the message I would expect:
(base) raysalemi#RayProMac ~ % nosetests
----------------------------------------------------------------------
Ran 0 tests in 0.003s
OK
But if I cd to my tests directory I get this:
/Users/raysalemi/repos/pyuvm/tests/nosetests
(base) raysalemi#RayProMac nosetests % ls
__pycache__ pyuvm_unittest.py test_05_base_classes.py test_06_reporting_classes.py
(base) raysalemi#RayProMac nosetests % nosetests
(base) raysalemi#RayProMac nosetests % echo $?
1
This has been running for months so I'm not certain of the change, but I can't get an error message to check, only the exit status.
Suggestions?

The solution was to CD to my test directory and run python with the unittest module on the same command line:
(base) raysalemi#RayProMac nosetests % python -m unittest
No module named 'copyz'
No module named 'copyz'
EE
I had accidentally stuck a z into my code and the import was failing. The only way to get that message was to use unittest directly.

pipe: Bad file descriptor when running make command

Anyone can help me with this issue?
I am trying to run an app, whenever I run make in the terminal I got that error
process_begin: CreateProcess(NULL, python3.8 -c "import sys; print(sys.version_info[:2] >= (3, 8))", ...) failed.enter code here
Makefile:24: pipe: Bad file descriptor
SETUP Creating virtualenv
FATAL Python not found (python3.8)
make: *** [Makefile:21: .venv] Error 1
I've created a venv before running and installed python8.3, pip as well.
Makefile
PYTHON ?= python3.8
FLASK_HOST ?= 127.0.0.1
FLASK_PORT ?= 5000
VENV ?= .venv
SHELL := /bin/bash
PIP := ${VENV}/Scripts/pip
FLASK := ${VENV}/Scripts/flask
CONFIG := licmon/licmon.cfg
SERVERS := licmon/servers.cfg
.PHONY: all
all: ${VENV} config
#printf "\033[38;5;154mSETUP\033[0m \033[38;5;105mInstalling licmon python package\033[0m\n"
#${PIP} install -q -e '.[dev]'
${VENV}:
#printf "\033[38;5;154mSETUP\033[0m \033[38;5;105mCreating virtualenv\033[0m\n"
ifeq (, $(shell which ${PYTHON} 2> /dev/null))
#printf "\033[38;5;220mFATAL\033[0m \033[38;5;196mPython not found (${PYTHON})\033[0m\n"
#exit 1
endif
ifneq (True, $(shell ${PYTHON} -c 'import sys; print(sys.version_info[:2] >= (3, 8))'))
#printf "\033[38;5;220mFATAL\033[0m \033[38;5;196mYou need at least Python 3.8\033[0m\n"
#exit 1
endif
#${PYTHON} -m venv --prompt licmon .venv
#${PIP} install -q -U pip setuptools
${CONFIG}: | ${CONFIG}.example
#printf "\033[38;5;154mSETUP\033[0m \033[38;5;105mCreating config [\033[38;5;147m${CONFIG}\033[38;5;105m]\033[0m\n"
#cp ${CONFIG}.example ${CONFIG}
#printf "\033[38;5;154mSETUP\033[0m \033[38;5;105mCreating config [\033[38;5;147m${SERVERS}\033[38;5;105m]\033[0m\n"
#cp ${SERVERS}.example ${SERVERS}
#sed -i.bak "s/^SECRET_KEY = None/SECRET_KEY = '$$(LC_ALL=C tr -dc A-Za-z0-9 < /dev/urandom | head -c 32)'/" ${CONFIG}
#sed -i.bak "s/^SKIP_LOGIN = False/SKIP_LOGIN = True/" ${CONFIG}
#sed -i.bak "s/^EMAIL_BACKEND = '[^']\+'/EMAIL_BACKEND = 'licmon.vendor.django_mail.backends.console.EmailBackend'/" ${CONFIG}
#rm -f ${CONFIG}.bak
#printf " \033[38;5;82mDon't forget to update the config files if needed!\033[0m\n"
.PHONY: flask-server
flask-server:
#printf " \033[38;5;154mRUN\033[0m \033[38;5;75mRunning Flask dev server [\033[38;5;81m${FLASK_HOST}\033[38;5;75m:\033[38;5;81m${FLASK_PORT}\033[38;5;75m]\033[0m\n"
#${FLASK} run -h ${FLASK_HOST} -p ${FLASK_PORT} --extra-files $(abspath licmon/licmon.cfg):$(abspath licmon/servers.cfg)
.PHONY: build
build:
#printf " \033[38;5;154mBUILD\033[0m \033[38;5;176mBuilding production package\033[0m\n"
#rm -rf build
#source ${VENV}/bin/activate
#${PIP} list
#python setup.py bdist_wheel -q
.PHONY: config
config: ${CONFIG}
.PHONY: clean
clean:
#printf "\033[38;5;154mCLEAN\033[0m \033[38;5;202mDeleting all generated files...\033[0m\n"
#rm -rf .venv newdle.egg-info pip-wheel-metadata dist build
#find newdle/ -name __pycache__ -exec rm -rf {} +

Never use # to suppress echoing makefile commands until you are sure that the makefile is working correctly. And even then, consider not doing so.
Also, I urge you to avoid putting ANSI escape sequences into your makefile. They are obnoxious to read, and they will not be effective in all contexts. Absolutely omit them from any makefile you present here, as they are not part of any minimal reproducible example unless it is asking specifically about them.
Anyway, this problem ...
process_begin: CreateProcess(NULL, python3.8 -c "import sys; print(sys.version_info[:2] >= (3, 8))", ...) failed.enter code here
Makefile:24: pipe: Bad file descriptor
... appears to be a consequence of this problem ...
FATAL Python not found (python3.8)
. The messages are ordered that way in the output because that is the order in which they are generated, reflecting a deep flaw in your approach.
GNU make directives, such as ifeq, are evaluated when the makefile is parsed, before any recipes are run. Thus, this code ...
ifeq (, $(shell which ${PYTHON} 2> /dev/null))
#printf "\033[38;5;220mFATAL\033[0m \033[38;5;196mPython not found (${PYTHON})\033[0m\n"
#exit 1
endif
... does not halt the build immediately if the requested Python interpreter is not found. The exit command is part of a recipe, which isn't (yet) being run. As a consequence, make proceeds to evaluate the next directive either way:
ifneq (True, $(shell ${PYTHON} -c 'import sys; print(sys.version_info[:2] >= (3, 8))'))
. If indeed ${PYTHON} was not found, then the shell command being executed fails with the error reported, but, occurring as that does inside an execution of the $(shell) function, that does not terminate the build. Make decides to build the ${VENV} target, and when executing the recipe for that target, it finally gets around to printing the "FATAL" message and executing the #exit 1 guarded by the above ifeq. That causes make to stop the build and report that the recipe for .venv (== ${VENV}) had an error.
I've created a venv before running and installed python8.3, pip as well.
Notwithstanding what software may or may not have been installed, whatever shell make uses to execute the $(shell) function (which it will also use to execute recipes) does not have a command python3.8 in its executable search path. There is not enough information provided to troubleshoot that further.
Also, make does not see a subdirectoy .venv in its working directory, for it would not otherwise be executing the recipe for the .venv target. If that surprises you, then possibly it results from running make from the wrong working directory. Not much more can be said about that, either.
Overall, I think you're trying too hard, at least on the make side. I would just let the execution of ${PYTHON} in a recipe fail if need be, having not suppressed command echoing. The makefile would be easier to write, and the diagnostic output easier to interpret.

Run Python scripts without explicitly invoking `python`

I am trying to run my python script without writing python func.py.
I've added to my script file #!/usr/bin/python2.7
did chmod +x func.py
when I try to run: ./func.py -p show -c all
the script works fine but if I try to take off the "flags" (-p , -c) or the "./" or ".py" it won't work.
taking flags off returns:
[root#pg66 tmp]# ./func.py show all
usage: func.py [-h] [-p PROCESS] [-c CLUSTER] [-t TYPE]
func.py: error: unrecognized arguments: show all
taking "./" & ".py" off returns:
[root#pg66 tmp]# func.py -p show -c all
-bash: func.py: command not found
edit: i have 3 flags -p -c -t , dont know where the -h came from. -t can be null so when i write -p show -c all it works.

you can simply use the chmod +* x in the location with the python file. This makes it easier for linux/unix After that you can run it by ./main.py Assuming the file is named as main.py. You can also remove the .py extension then give the permission by chmod +* x and after that then just execute by ./main. Another way to do this is simply going to the execution script of your terminal then add an alias for directly running the script just by typing the alias name

How can a Colab notebook detect an error in a bash script/command

If an error arises in a %%bash command, I'd like to stop execution in Colab.
The following approach uses flag files, which feels to me like a kludge. Is there a better way?
import os
flagfile_path = '/content/job_completed.flag'
! rm $flagfile_path
%%bash -e -s $flagfile_path
FLAGFILE_PATH=$1
function mk_flag_file(){
touch $FLAGFILE_PATH
}
echo Cmd1
ls .
echo Cmd2
ls x
echo Cmd3
ls /content
mk_flag_file
class BashScriptError(Exception):
pass
if not os.path.exists(flagfile_path):
raise BashScriptError

Strange behaviour with double pipe operator and python virtualenv

I have a bash build script which I run sourced so I can activate a Python virtual environment. I firstly run the unit-tests with python3.7 -m unittest. However, if these fail I don't want to run the main program. So, I need to deactivate the virtual environment (so the terminal is back to its original state) and then return 1 to exit the build script.
So my script looks like this
# activate virtual env ...
python3.7 -m unittest || deactivate; return 1;
python3.7 app.py
deactivate
When the unit tests fail, python3.7 -m unittest returns 1 and the virtual environment deactivates as expected.
When the unit tests run successfully, python3.7 -m unittest returns 0, however strangely the right side of the pipe seems to sort of run. I haven't figured out if it's an oddity with bash or with deactivate but here are some samples of the behaviour:
(exit 0) || deactivate; echo "Tests failed"; return 1; (Output: "Tests failed", deactivate not run)
(exit 0) || echo "Deactivating"; deactivate; echo "Tests failed"; return 1; (Output: "Tests failed", deactivate ran)
(exit 0) || echo "Tests failed"; return 1; (Output: Nothing, deactivate not run)
The last case of those three makes sense and follows expected behaviour, the other two do not.

This is related to Bash Pitfall 22, but not exactly the same. The important point is how the statements are grouped:
cmd1 || cmd2; cmd3
will run cmd1, and if the exit status is non-zero, cmd2; then, no matter what, cmd3.
The intention is rather this:
cmd1 || { cmd2; cmd3; }
If cmd1 fails, run cmd2 and cmd3.