Develop Reference

Error response from daemon: open \\.\pipe\docker_engine_linux

I installed the latest version of docker, installed WSL 2 according by the manual. And installed the container with the command docker-compose up. I need to run the tests by command tests/run_tests.sh. But after launching, after a few seconds, the window with the test closes, my container disappears in the docker, and when I try to write the command docker-compose up again, I get an error Error response from daemon: open \\.\pipe\docker_engine_linux: The system cannot find the file specified.
run_tests:
#!/usr/bin/env sh
# To run locally, execute the command NOT in container:
# bash tests/run_tests.sh
set -x
if [ -z "$API_ENV" ]; then
API_ENV=test
fi
if [ "$API_ENV" = "bitbucket_test" ]; then
COMPOSE_FILE="-f docker-compose.test.yml"
fi
docker-compose build connectors
API_ENV=$API_ENV docker-compose ${COMPOSE_FILE} up -d --force-recreate
connectors_container=$(docker ps -f name=connectors -q | tail -n1)
if [ "$API_ENV" = "bitbucket_test" ]; then
mkdir -p artifacts && docker logs --follow ${connectors_container} > ./artifacts/docker_connectors_logs.txt 2>&1 &
pytest_n_processes=100
else
pytest_n_processes=25
fi
# Timeout for the tests. In bitbucket we want to stop the tests a bit before the max time, so that
# artifacts are created and logs can be inspected
timeout_cmd="timeout 3.5m"
if [ "$API_ENV" = "bitbucket_test" ] || [ "$API_ENV" = "test" ]; then
export PYTEST_SENTRY_DSN='http://d07ba0bfff4b41888e311f8398321d14#sentry.windsor.ai/4'
export PYTEST_SENTRY_ALWAYS_REPORT=1
fi
git fetch origin "+refs/heads/master:refs/remotes/origin/master"
# Lint all the files that are modified in this branch
$(dirname "$0")/run_linters.sh &
linting_pid=$!
# bitbucket pipelines have 8 workers, use 6 for tests
#
# WARNING: Tests require gunicorn and is enabled when containers are started with: API_ENV=test docker-compose up -d --force-recreate
# Tests are run in parallel and the cache-locking in threaded flask doesnt work in this case
${timeout_cmd} docker exec ${connectors_container} bash -c \
"PYTEST_SENTRY_DSN=$PYTEST_SENTRY_DSN \
PYTEST_SENTRY_ALWAYS_REPORT=$PYTEST_SENTRY_ALWAYS_REPORT \
pytest \
--cov=connectors --cov=api --cov=base \
--cov-branch --cov-report term-missing --cov-fail-under=71.60 \
--timeout 60 \
-v \
--durations=50 \
-n $pytest_n_processes \
tests || ( \
code=$? `# store the exit code to exit with it` \
&& echo 'TESTS FAILED' \
&& mkdir -p ./artifacts \
&& docker logs ${connectors_container} > ./artifacts/docker_connectors_failure_logs.txt 2>&1 `# Ensure that the logs are complete` \
) "&
# Get the tests pid
tests_pid=$!
# wait for linting to finish
wait $linting_pid
linting_code=$?
echo "Linting code: ${linting_code}"
if [ $linting_code -ne 0 ]; then
echo 'Linting failed'
# kill running jobs on exit in local ubuntu. Some tests were left running by only killing the test_pid.
kill "$(jobs -p)"
# kills the test process explicitly in gitlab pipelines. Was needed because jobs returns empty in gitlab pipelines.
kill $tests_pid
exit 1
fi
# wait for tests to finish
wait $tests_pid
testing_code=$?
echo "Testing code: ${testing_code}"
if [ $testing_code -ne 0 ]; then
echo 'Tests failed'
exit 1
else
echo 'Tests and linting passed'
exit 0
fi

How do i halt execution of bash script when a python script fails?

So I have the following in my shell script:
python get_link.py $password | wget --content-disposition -i-
mkdir web_folder
mv *.zip web_folder
So the first line is executing a python script i wrote which prints out a website link and wget immediately retrieves the link returned by the python script and downloads a zip file.
The second line makes a new folder called "web_folder" and the third line is moving the zip file that was downloaded by wget into the "web_folder"
The problem I'm facing is that, if the python script fails due to error such as $password is having the wrong password, the rest of the shell script command is still executing. In my case, the following is printed:
mv: cannot stat ‘*.zip’: No such file or directory
The mkdir and the mv command somewhat still executes even if the python script fails. How do i ensure that the script comes to a complete halt when the python script fails?

If you are using bash, look into PIPESTATUS variable.
${PIPESTATUS[0]} will have the return code from the first pipe.
#!/bin/bash
python get_link.py $password | wget --content-disposition -i-
if [ ${PIPESTATUS[0]} -eq 0 ]
then
echo "python get_link.py successful code here"
else
echo "python get_link.py failed code here"
fi

A compact solution, chain everything with and's:
(python get_link.py $password | wget --content-disposition -i-) && (mkdir web_folder) && (mv *.zip web_folder)
Less compact solution:
python get_link.py $password | wget --content-disposition -i-
if [ $? -eq 0 ]; then
mkdir web_folder
mv *.zip web_folder
fi

Python subprocess.run exit codes not working with bash script

I use Python to call a Bash script. I use the run() function for that which was introduced in Python 3.5.
I want to use the returncode for something, so I use this:
result = subprocess.run(["./app/first_deployment.sh", arg], stdout=subprocess.PIPE,)
if result.returncode == 0:
# do something
My Bash file:
# First condition
if grep -q 'string' file.txt
then
# Second condition
if grep -q 'anotherstring' file.txt
then
echo "Success"
exit 0
else
echo "Fail message 2"
exit 1
fi
else
echo "Fail message 1"
exit 1
fi
So it seems to work, the correct messages I do see in the logs. However result.returncode ALWAYS is code 0, which means succesfull. Why is that and how can I make sure it works?
Update (full script):
#!/bin/bash
basedir="/home/dpa/clients"
user=$1
archive_url=$2
repo_name=$3
port=$4
deployment_tag=$5
mkdir $basedir/$user
mkdir $basedir/$user/$repo_name
curl -o $basedir/$user/$repo_name/$deployment_tag.tar.gz $archive_url
mkdir $basedir/$user/$repo_name/$deployment_tag
tar -xvf $basedir/$user/$repo_name/$deployment_tag.tar.gz -C $basedir/$user/$repo_name/$deployment_tag --strip-components 1
rm -rf $basedir/$user/$repo_name/$deployment_tag.tar.gz
# Check if a production.yml file exists in the new directory
if [ -f "$basedir/$user/$repo_name/$deployment_tag/production.yml" ]
then
# Check for the websecure endpoint
if grep -q 'traefik.http.routers.$$$UNIQUE_DEPLOYMENT_TAG-secure.entrypoints=websecure' $basedir/$user/$repo_name/$deployment_tag/production.yml
then
# Check for the host rule
if grep -q 'traefik.http.routers.$$$UNIQUE_DEPLOYMENT_TAG-secure.rule=Host' $basedir/$user/$repo_name/$deployment_tag/production.yml
then
# Check if the proxy network exists
if grep -q 'network=proxy' $basedir/$user/$repo_name/$deployment_tag/production.yml
then
sed -i "s/\$\$\$PORT/${port}/g" $basedir/$user/$repo_name/$deployment_tag/production.yml
sed -i "s/\$\$\$UNIQUE_DEPLOYMENT_TAG/${deployment_tag}/g" $basedir/$user/$repo_name/$deployment_tag/production.yml
# docker-compose -f $basedir/$user/$repo_name/$deployment_tag/production.yml build
# docker-compose -f $basedir/$user/$repo_name/$deployment_tag/production.yml up -d
echo "Deployment succesfull! Your app is online :)"
exit 0
else
echo "Proxy network rule not found in yml config."
exit 1
fi
else
echo "Traefik host rule not found in yml config."
exit 1
fi
else
echo "Traefik websecure endpoint not found in yml config."
set -x
exit 1
fi
else
echo "No production.yml could be found. Please follow the docs and include the correct YAML file."
exit 1
fi
UPDATE2 (output):
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1845 100 1845 0 0 7561 0 --:--:-- --:--:-- --:--:-- 7530
+ mkdir /home/dpa/clients/foo/testapi1/testapi1-305983855
+ tar -xvf /home/dpa/clients/foo/testapi1/testapi1-305983855.tar.gz -C /home/dpa/clients/foo/testapi1/testapi1-305983855 --strip-components 1
+ rm -rf /home/dpa/clients/foo/testapi1/testapi1-305983855.tar.gz
+ '[' -f /home/dpa/clients/foo/testapi1/testapi1-305983855/production.yml ']'
+ grep -q 'traefik.http.routers.$$$UNIQUE_DEPLOYMENT_TAG-secure.entrypoints=websecure' /home/dpa/clients/foo/testapi1/testapi1-305983855/production.yml
+ echo 'Traefik websecure endpoint not found in yml config.'
+ exit 1
CompletedProcess(args=['./app/first_deployment.sh', 'foo', 'https://codeload.github.com/foo/testapi1/legacy.tar.gz/master?token=changedthis', 'testapi1', '7039', 'testapi1-305983855'], returncode=0, stdout=b'foo-testapi1-b1c9fd5be165b850d2b94cde30affa622b5c3621/.github/\nfoo-testapi1-b1c9fd5be165b850d2b94cde30affa622b5c3621/.github/dependabot.yml\nfoo-testapi1-b1c9fd5be165b850d2b94cde30affa622b5c3621/.github/workflows/\nfoo-testapi1-b1c9fd5be165b850d2b94cde30affa622b5c3621/.github/workflows/ci.yml\nfoo-testapi1-b1c9fd5be165b850d2b94cde30affa622b5c3621/.gitignore\nfoo-testapi1-b1c9fd5be165b850d2b94cde30affa622b5c3621/.vscode/\nfoo-testapi1-b1c9fd5be165b850d2b94cde30affa622b5c3621/.vscode/settings.json\nfoo-testapi1-b1c9fd5be165b850d2b94cde30affa622b5c3621/docker-compose.yml\nfoo-testapi1-b1c9fd5be165b850d2b94cde30affa622b5c3621/local.yml\nfoo-testapi1-b1c9fd5be165b850d2b94cde30affa622b5c3621/production.yml\nfoo-testapi1-b1c9fd5be165b850d2b94cde30affa622b5c3621/testapi1/\nfoo-testapi1-b1c9fd5be165b850d2b94cde30affa622b5c3621/testapi1/.dockerignore\nfoo-testapi1-b1c9fd5be165b850d2b94cde30affa622b5c3621/testapi1/Dockerfile\nfoo-testapi1-b1c9fd5be165b850d2b94cde30affa622b5c3621/testapi1/app/\nfoo-testapi1-b1c9fd5be165b850d2b94cde30affa622b5c3621/testapi1/app/__init__.py\nfoo-testapi1-b1c9fd5be165b850d2b94cde30affa622b5c3621/testapi1/app/main.py\nfoo-testapi1-b1c9fd5be165b850d2b94cde30affa622b5c3621/testapi1/requirements.txt\nTraefik websecure endpoint not found in yml config.\n')
Update 3:
So after following the advice of simplying the script, I tried the bash script file with just the config below. And this gave me.. exit code 1! As expected. Great that seems to work.
#!/bin/bash
basedir="/home/dpa/clients"
user=$1
archive_url=$2
repo_name=$3
port=$4
deployment_tag=$5
# # Check if a production.yml file exists in the new directory
if [ -f "$basedir/$user/$repo_name/$deployment_tag/production.yml" ]
then
echo "Test complete"
exit 0
else
echo "No production.yml could be found. Please follow the docs and include the correct YAML file."
exit 1
fi
Now when I added just the line mkdir before that. It ended up in a exit code 0. Which is weird because it should give me an exit code 1, since the directory does not exist. So the following code:
#!/bin/bash
basedir="/home/dpa/clients"
user=$1
archive_url=$2
repo_name=$3
port=$4
deployment_tag=$5
mkdir "$basedir/$user"
# # Check if a production.yml file exists in the new directory
if [ -f "$basedir/$user/$repo_name/$deployment_tag/production.yml" ]
then
echo "Test complete"
exit 0
else
echo "No production.yml could be found. Please follow the docs and include the correct YAML file."
exit 1
fi
I tried this with commands like cd or ls as well, all ending up in the same result, an exit code 0. So for some reason whenever a shell command is ran succesfully, it results in exit code 0 in the Python function. Because a file check only did work. So it must be a Python related problem..

Execute a shell command in Python

I'm trying to execute the following shell command in Python:
echo abc && echo -ne \\x00 > file && echo efg
I tried doing it by calling os.system but it seems to ignore the -ne flag of echo.
What can I do to run it properly?
import os
os.system('echo abc && echo -ne \\x00 > file && echo efg')

How do i run nikto.pl file from wapiti.py file?

I am working on a project which involves wapiti and nikto web tools. i have managed to produce one report for both these tool with this command
python wapiti.py www.kca.ac.ke ;perl nikto.pl -h www.kca.ac.ke -Display V -F htm -output /root/.wapiti/generated_report/index.html.
But i would like to run a command like
python wapiti.py www.kca.ac.ke
and get both the wapiti and nikto web scan report. How do i achieve this guys?

A shell script would work. Save the following as 'run_wapiti_and_nikto_scans', then run it as:
bash run_wapiti_and_nikto_scans www.my.site.com
Here is the script:
#!/bin/bash
SITE=$1
if [ -n "$SITE" ]; then # -n tests to see if the argument is non empty
echo "Looking to scan $SITE"
echo "Running 'python wapiti.py $SITE'"
python wapiti.py $SITE || echo "Failed to run wapiti!" && exit 1;
echo "Running 'perl nikto.pl -h $SITE -Display V -F htm -output /root/.wapiti/generated_report/index.html'"
perl nikto.pl -h $SITE -Display V -F htm -output /root/.wapiti/generated_report/index.html || echo "Failed to run nikto!" && exit 1;
echo "Done!"
exit 0; # Success
fi
echo "usage: run_wapiti_and_nikto_scans www.my.site.com";
exit 1; # Failure