Develop Reference

Failed to execute script docker-compose error

I am having trouble getting my docker-compose.yml file to run, I am running the command docker-compose -f docker-compose.yml and am getting the error "TypeError: You must specify a directory to build in path
[7264] Failed to execute script docker-compose"
The file I have this in is called Labsetup and inside the file I have
docker-compose.yml,dockerClient.yml,dokcerServer.yml,and
A file called Volumes that have Server.py and Client.py
To this point I have edit my docker-compose.yml file and tried running "docker-compose -f docker-compose.yml up" to run the yml below is my docker-compose.yml
version: "3"
services:
# new code to execute python scripts in /volumes
server:
image: 85345aa61801 # from DockerServer.yml
container_name: server
build: DockerServer.yml
command: python3 ./volumes/Server.py
ports:
- 8080:8080
client:
image: 700c22b0d9d6 # from DockerClient.yml
container_name: client
build: DockerClient.yml
command: python3 ./volumes/Client.py
network_mode: host
depends_on:
- server
# below is original code for containers
malicious-router:
image: handsonsecurity/seed-ubuntu:large
container_name: malicious-router-10.9.0.111
tty: true
cap_add:
- ALL
sysctls:
- net.ipv4.ip_forward=1
- net.ipv4.conf.all.send_redirects=0
- net.ipv4.conf.default.send_redirects=0
- net.ipv4.conf.eth0.send_redirects=0
privileged: true
volumes:
- ./volumes:/volumes
networks:
net-10.9.0.0:
ipv4_address: 10.9.0.111
command: bash -c "
ip route add 192.168.60.0/24 via 10.9.0.11 &&
tail -f /dev/null "
HostB1:
image: handsonsecurity/seed-ubuntu:large
container_name: host-192.168.60.5
tty: true
cap_add:
- ALL
volumes:
- ./volumes:/volumes
networks:
net-192.168.60.0:
ipv4_address: 192.168.60.5
command: bash -c "python3 Server.py"
HostB2:
image: handsonsecurity/seed-ubuntu:large
container_name: host-192.168.60.6
tty: true
cap_add:
- ALL
volumes:
- ./volumes:/volumes
networks:
net-192.168.60.0:
ipv4_address: 192.168.60.6
command: bash -c "python3 Client.py"
HostB3:
image: handsonsecurity/seed-ubuntu:large
container_name: host-192.168.60.7
tty: true
cap_add:
- ALL
volumes:
- ./volumes:/volumes
networks:
net-192.168.60.0:
ipv4_address: 192.168.60.7
command: bash -c "python3 Client.py 8080"
Router:
image: handsonsecurity/seed-ubuntu:large
container_name: router
tty: true
cap_add:
- ALL
sysctls:
- net.ipv4.ip_forward=1
volumes:
- ./volumes:/volumes
networks:
net-10.9.0.0:
ipv4_address: 10.9.0.11
net-192.168.60.0:
ipv4_address: 192.168.60.11
command: bash -c "
ip route del default &&
ip route add default via 10.9.0.1 &&
tail -f /dev/null "
networks:
net-192.168.60.0:
name: net-192.168.60.0
ipam:
config:
- subnet: 192.168.60.0/24
net-10.9.0.0:
name: net-10.9.0.0
ipam:
config:
- subnet: 10.9.0.0/24
My DockerServer.yml
# Dockerfile.
# FROM ubuntu:20.04 # FROM python:latest
# ... most recent "image" created since DockerServer build = f48ea80eae5a <-- needed in FROM
# after running command: docker build -f DockerServer.yml .
# ... it output this upon completion ...
# ---> Running in 533f2cbd3135
# Removing intermediate container 533f2cbd3135 ---> 85345aa61801 Successfully built 85345aa61801
# FROM requires this syntax:
# FROM [--platform=<platorm>] <image> [AS <name>]
FROM python3:3.8.5 85345aa61801 AS serverImg
ADD Server.py /volumes/
WORKDIR /volumes/
My DokcerClient.yml
# Dockerfile.
# FROM ubuntu:20.04 server # python:latest #FROM ubuntu:latest ?
# step1/3 image --> f48ea80eae5a ... Successfully built 700c22b0d9d6
# image needed for 2nd arg
# after running command: docker build -f DockerClient.yml
# ... it output this upon completion ...
# Removing intermediate container 29037f3d23fb ---> 700c22b0d9d6 Successfully built 700c22b0d9d6
# FROM requires this syntax:
# FROM [--platform=<platorm>] <image> [AS <name>]
FROM python3:3.8.5 700c22b0d9d6 AS clientImg
ADD Client.py /volumes/
WORKDIR /volumes/
I am creating a project for a networking class that connects users to chat on a terminal. I have two python codes one being called Server.py that starts the server and tells who is in the chat room and another called Client.py that asks the user to put in a name and they will be able to chat. I am running this all on a virtual box Ubuntu20.04.
Code for Server.py
import socket
import _thread
import threading
from datetime import datetime
class Server():
def __init__(self):
# For remembering users
self.users_table = {}
# Create a TCP/IP socket and bind it the Socket to the port
self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.server_address = ('localhost', 8080)
self.socket.bind(self.server_address)
self.socket.setblocking(1)
self.socket.listen(10)
print('Starting up on {} port {}'.format(*self.server_address))
self._wait_for_new_connections()
def _wait_for_new_connections(self):
while True:
connection, _ = self.socket.accept()
_thread.start_new_thread(self._on_new_client, (connection,))
def _on_new_client(self, connection):
try:
# Declare the client's name
client_name = connection.recv(64).decode('utf-8')
self.users_table[connection] = client_name
print(f'{self._get_current_time()} {client_name} joined the room !!')
while True:
data = connection.recv(64).decode('utf-8')
if data != '':
self.multicast(data, owner=connection)
else:
return
except:
print(f'{self._get_current_time()} {client_name} left the room !!')
self.users_table.pop(connection)
connection.close()
def _get_current_time(self):
return datetime.now().strftime("%H:%M:%S")
def multicast(self, message, owner=None):
for conn in self.users_table:
data = f'{self._get_current_time()} {self.users_table[owner]}: {message}'
conn.sendall(bytes(data, encoding='utf-8'))
if __name__ == "__main__":
Server()
Code for Client.py
import sys
import socket
import threading
class Client():
def __init__(self, client_name):
client_name = input("Choose Your Nickname:")
# Create a TCP/IP socket and connect the socket to the port
self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.server_address = ('localhost', 8080)
self.socket.connect(self.server_address)
self.socket.setblocking(1)
print("{} is connected to the server, you may now chat.".format(client_name))
self.client_name = client_name
send = threading.Thread(target=self._client_send)
send.start()
receive = threading.Thread(target=self._client_receive)
receive.start()
def _client_send(self):
self.socket.send(bytes(self.client_name, encoding='utf-8'))
while True:
try:
c = input()
sys.stdout.write("\x1b[1A\x1b[2K") # Delete previous line
self.socket.send(bytes(c, encoding='utf-8'))
except:
self.socket.close()
return
def _client_receive(self):
while True:
try:
print(self.socket.recv(1024).decode("utf-8"))
except:
self.socket.close()
return
if __name__ == "__main__":
client_name = sys.argv[1]
Client(client_name)
Edit: I found a example online and followed it and changed a few
[11/26/21]seed#VM:~/.../Labsetup$ docker build -f DockerServer.yml .
Sending build context to Docker daemon 33.28kB
Step 1/3 : FROM python:latest
latest: Pulling from library/python
647acf3d48c2: Pull complete
b02967ef0034: Pull complete
e1ad2231829e: Pull complete
5576ce26bf1d: Pull complete
a66b7f31b095: Pull complete
05189b5b2762: Pull complete
af08e8fda0d6: Pull complete
287d56f7527b: Pull complete
dc0580965fb6: Pull complete
Digest: sha256:f44726de10d15558e465238b02966a8f83971fd85a4c4b95c263704e6a6012e9
Status: Downloaded newer image for python:latest
---> f48ea80eae5a
Step 2/3 : ADD Server.py /volumes/
---> df60dab46969
Step 3/3 : WORKDIR /volumes/
---> Running in 533f2cbd3135
Removing intermediate container 533f2cbd3135
---> 85345aa61801
Successfully built 85345aa61801
[11/26/21]seed#VM:~/.../Labsetup$ docker build -f DockerClient.yml .
Sending build context to Docker daemon 33.28kB
Step 1/3 : FROM python:latest
---> f48ea80eae5a
Step 2/3 : ADD Client.py /volumes/
---> 76af351c1c7a
Step 3/3 : WORKDIR /volumes/
---> Running in 29037f3d23fb
Removing intermediate container 29037f3d23fb
---> 700c22b0d9d6
Successfully built 700c22b0d9d6

Port mapping in docker doesn't work for python server

I want to build a server with sagemath, it should take in code, execute it and send back the result. SageMath has a python interface with which I thought this could be achieved. I don't really know python that well but I found a good starting point here https://ask.sagemath.org/question/23431/running-sage-from-other-languages-with-higher-performance/. The problem is that I wanted to run this in a docker container so and just map the port but this doesn't seem to be working.
I changed the python file to adjust it to the newer version:
import socket
import sys
from io import StringIO
from sage.all import *
from sage.calculus.predefined import x
from sage.repl.preparse import preparse
SHUTDOWN = False
HOST = 'localhost'
PORT = 8888
MAX_MSG_LENGTH = 102400
# Create socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
print('Socket created')
# Bind socket to localhost and port
try:
s.bind((HOST, PORT))
except (socket.error , msg):
print('Bind failed. Error Code : ' + str(msg[0]) + ' Message ' + msg[1])
sys.exit()
print('Socket bind complete')
# Start listening on socket
s.listen(10)
print('Socket now listening')
# Loop listener for new connections
while not SHUTDOWN:
# Wait to accept a new client connection
conn, addr = s.accept()
print('Connected with ' + addr[0] + ':' + str(addr[1]))
# Receive message from client
msg = conn.recv(MAX_MSG_LENGTH)
if msg:
if msg == "stop":
SHUTDOWN = True
else:
parsed = preparse(msg)
if parsed.startswith('load') or parsed.startswith('attach'):
os.system('sage "' + os.path.join(os.getcwd(), parsed.split(None, 1)[1]) + '"')
else:
# Redirect stdout to my stdout to capture into a string
sys.stdout = mystdout = io.StringIO()
# Evalutate msg
try:
eval(compile(parsed,'<cmdline>','exec'))
result = mystdout.getvalue() # Get result from mystdout
except Exception as e:
result = "ERROR: " + str(type(e)) + " " + str(e)
# Restore stdout
sys.stdout = sys.__stdout__
# Send response to connected client
if result == "":
conn.sendall("Empty result, did you remember to print?")
else:
conn.sendall(result)
# Close client connection
conn.close()
# Close listener
s.close()
then I created a script to run it
#!/bin/bash
/usr/bin/sage -python /var/app/sage-server.py
afterwards I created the Dockerfile to install SageMath
FROM ubuntu:20.04
COPY src/ /var/app
RUN apt-get update \
&& DEBIAN_FRONTEND="noninteractive" TZ="Europe/London" apt-get -y install tzdata \
&& apt-get install sagemath -y
WORKDIR /var/app
RUN ["chmod", "+x", "/var/app/server.sh"]
CMD ["/var/app/server.sh"]
and finally a docker-compose file to automate the mapping and to include it into the main project
version: '3'
services:
sage-server:
build: .
volumes:
- ./src:/var/app
ports:
- "9999:8888"
All parts execute well and without any error (on an ubuntu). I even see that the port is used when I go into the container but the port 9999 is not used in the host system.
Does any of you have an idea?

As suggested by #David Maze, you need to expose on 0.0.0.0 to be able to be reachable in the container.
HOST = '0.0.0.0'
Then you get some issues:
preparse is expecting a str so you need to decode the bytes received:
msg = conn.recv(MAX_MSG_LENGTH).decode('utf-8')
StringIO was directly imported so no need to prefix with io.:
sys.stdout = mystdout = StringIO()
you need to send back a bytes, but result is an str, so you need to encode:
conn.sendall(result.encode('utf-8'))
The Dockerfile can be improved (https://docs.docker.com/develop/develop-images/dockerfile_best-practices/):
FROM ubuntu:20.04
# First install dependencies so next builds will reuse docker build cache
# Use `--no-install-recommends` to minimize the installed packages
# Clean apt data
RUN apt-get update \
&& DEBIAN_FRONTEND="noninteractive" TZ="Europe/London" apt-get -y install tzdata \
&& apt-get install --no-install-recommends sagemath -y \
&& rm -rf /var/lib/apt/lists/*
# Copy changing data as late as possible to use docker build cache
COPY src/ /var/app
WORKDIR /var/app
# Explicit the listening ports
EXPOSE 8888
# No need for a wrapper script for simple command (and workaround chmod issue)
ENTRYPOINT ["/usr/bin/sage", "-python", "/var/app/sage-server.py"]
As the build-time copy of the source code is overridden by the volume mount at run-time, the permissions have to be right at host side (the build RUN chmod +x has no effect on the run-time mounted files)

Subscriber not getting the message in Redis

I am using docker and redis to learn how multi container in docker work.I am using below simple flask application which has a publish and subscribe code.However I don't see any message received by the subscriber
import redis
from flask import Flask
app = Flask(__name__)
client = redis.Redis(host="redis-server", decode_responses=True)
def event_handler(msg):
print("Handler", msg, flush=True)
#app.route("/")
def index():
print("Request received",flush=True)
print(client.ping(), flush=True)
client.publish("insert", "This is a test 1")
client.publish("insert", "This is a test 2")
client.publish("insert", "This is a test 3")
ps = client.pubsub()
ps.subscribe("insert",)
print(ps.get_message(), flush=True)
print(ps.get_message(), flush=True)
print(ps.get_message(), flush=True)
return "Hello World"
if __name__ == '__main__':
app.run(host="0.0.0.0", port="5000")
and below is my docker-compose
version: "3"
services:
redis-server:
image: "redis"
python-app:
build: .
ports:
- "4001:5000"
and docker file
# Specify the base image
FROM python:alpine
# specify the working directory
WORKDIR /usr/app
# copy the requiremnts file
COPY ./requirements.txt ./
RUN pip install -r requirements.txt
COPY ./ ./
CMD ["python","./app.py"]
And below is the output I am getting
Can someone please help me in identifying what I am doing wrong.Thanks

I was able to fix above issue by creating two separate scripts one for publisher and other for subscriber and starting the subscriber first.

Docker: Python server not listening

I'm trying to write a simple python server that writes a message (from JSON) to a file. When I deploy my docker container, nothing happens. When I stop the container (keyboard interrupt) all console output is written at once an the container shuts down.
My python code:
import socketserver
import json
class PoCServer(socketserver.BaseRequestHandler):
def handle(self):
addr = self.client_address[0]
print("[{}] incoming connection...".format(addr))
buff = bytes()
while True:
rawdata = self.request.recv(256)
if not rawdata: break
buff = buff + rawdata
data = json.loads(buff.decode())
with open("data/" + data["name"] + ".txt", "w") as f:
f.write(data["msg"])
print("[{}] file ".format(addr) + data["name"] + ".txt written...")
server = socketserver.ThreadingTCPServer(("localhost", 10000), PoCServer)
print("[+] server listening...")
server.serve_forever()
My Dockerfile:
FROM python
WORKDIR /app
RUN mkdir /app/data
COPY server.py /app
EXPOSE 10000
CMD ["python", "server.py"]
Thank you!

Since the server listening message is visible after keyboard interrupt this means that code is working normally but the outputs are getting buffered. They are displayed once the program exits.
Running your code with -u flag should help solve this issue. According to python help page:
-u : unbuffered binary stdout and stderr;
which seems to be the problem. So in your docker file replace entry point with CMD ["python", "-u", "server.py"]
Now though, this will print the output without buffering but you should be careful in exposing the right ports and mapping them to ports on local system to actually send/receive response to server.

Starting docker container using python script

I would like to start the docker container from a python script. When i call the docker image through my code , i am unable to start the docker container
import subprocess
import docker
from subprocess import Popen, PIPE
def kill_and_remove(ctr_name):
for action in ('kill', 'rm'):
p = Popen('docker %s %s' % (action, ctr_name), shell=True,
stdout=PIPE, stderr=PIPE)
if p.wait() != 0:
raise RuntimeError(p.stderr.read())
def execute():
ctr_name = 'sml/tools:8' # docker image file name
p = Popen(['docker', 'run', '-v','/lib/modules:/lib/modules',
'--cap-add','NET_ADMIN','--name','o-9000','--restart',
'always', ctr_name ,'startup',' --base-port',
9000,' --orchestrator-integration-license',
' --orchestrator-integration-license','jaVl7qdgLyxo6WRY5ykUTWNRl7Y8IzJxhRjEUpKCC9Q='
,'--orchestrator-integration-mode'],
stdin=PIPE)
out = p.stdin.write('Something')
if p.wait() == -20: # Happens on timeout
kill_and_remove(ctr_name)
return out
following are docker container details for your reference
dev#dev-VirtualBox:sudo docker ps -a
[sudo] password for dev:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
79b3b9d215f3 sml/tools:8 "/home/loadtest/st..." 46 hours ago Up 46 hours pcap_replay_192.168.212.131_9000_delay_dirty_1
Could some one suggest me why i could not start my container through my program

docker-py (https://github.com/docker/docker-py) should be used to control Docker via Python.
This will start an Ubuntu container running sleep infinity.
import docker
client = docker.from_env()
client.containers.run("ubuntu:latest", "sleep infinity", detach=True)
Have a look at https://docker-py.readthedocs.io/en/stable/containers.html for more details (capabilities, volumes, ..).