Develop Reference

python locust pass custom arguments to workers

I need to run locust in distributed mode. I'd like to use custom arguments, and it is needed to add specific value of argument to each worker.
Here is sample python code:
Locust test runner
"""
from locust import HttpUser, task, events, constant_pacing, tag
#events.init_command_line_parser.add_listener
def add_custom_parameters(parser):
"""Set arguments which can be passed also via web ui"""
parser.add_argument(
"--property",
type=str,
env_var="PROPERTY",
default="",
help="set name or id",
)
class AwesomeUser(HttpUser):
"""
One AwesomeUser class to rule them all...
"""
host = "EMPTY"
wait_time = constant_pacing(1)
def on_start(self):
"""
On start procedure.
"""
print(f"HERE: {self.environment.parsed_options.property}")
#task(10)
#tag("test_it")
def test_it(self):
"""
Test if custom parameters can be used in that way.
"""
print(f"property: {self.environment.parsed_options.property}")
if __name__ == "__main__":
AwesomeUser.tasks = [AwesomeUser.test_it]
I'd like to use docker-compose.yaml, there were many attempts, but it looks that I cananot manage it. Sample code that is not working:
version: '3'
services:
master:
build:
context: .
volumes:
- type: bind
source: "./tests"
target: "/home/locust/tests"
ports:
- "8089:8089"
command: -f /home/locust/tests/load_test.py --master -u 3 -r 1
worker:
build:
context: .
volumes:
- type: bind
source: "./tests"
target: "/home/locust/tests"
command: -f /home/locust/tests/load_test.py --worker --master-host master --property "SOSN_1"
worker2:
build:
context: .
volumes:
- type: bind
source: "./tests"
target: "/home/locust/tests"
command: -f /home/locust/tests/load_test.py --worker --master-host master --property "SOSN_2"
worker3:
build:
context: .
volumes:
- type: bind
source: "./tests"
target: "/home/locust/tests"
command: -f /home/locust/tests/load_test.py --worker --master-host master --property "SOSN_3"
There is workaround - run it in each screen, each worker as a master (in that case I'm able to run parallel many locust scripts):
killall screen
source venv/bin/activate
for i in {1..3}; do
sleep 2
echo create worker screen worker_$i
screen -dmS "worker_$i" locust -f tests/load_test.py --property "SOSN_$i" -t 10m --headless
done
However I hope that it can be done via docker-compose. An ideal it will be if I can use command like it: docker-compose up --scale worker=3 and as a result of it it will be run locust in distributed mode with 3 workers, each will use different value of my custom argument.
Is it possible?

docker-compose container port not showing up on localhost

I am trying to run a docker-compose app that has two services. One to build a web server and the other to run the tests on it.
docker-compose.yml
version: "3.7"
services:
web:
build: .
ports:
- "127.0.0.1:5000:5000"
expose:
- 5000
test:
# expose:
# - 5000
depends_on:
- web
build: test_python/.
./Dockerfile
FROM python:buster
RUN curl https://sh.rustup.rs -sSf | sh -s -- -y
# Add .cargo/bin to PATH
ENV PATH="/root/.cargo/bin:${PATH}"
# Check cargo is visible
RUN cargo --help
WORKDIR /code
COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt
EXPOSE 5000
COPY test_python .
CMD [ "python3", "base_routes.py" ]
test_python/Dockerfile
FROM python:buster
RUN pip3 install pytest requests
COPY . .
base_routes.py
from robyn import Robyn, static_file, jsonify
import asyncio
app = Robyn(__file__)
callCount = 0
#app.get("/")
async def h(request):
print(request)
global callCount
callCount += 1
message = "Called " + str(callCount) + " times"
return message
#app.get("/test")
async def test(request):
import os
path = os.path.abspath(os.path.join(os.path.dirname(os.path.realpath(__file__)), "test_python/index.html"))
return static_file(path)
#app.get("/jsonify")
async def json_get(request):
return jsonify({"hello": "world"})
#app.post("/jsonify")
async def json(request):
print(request)
return jsonify({"hello": "world"})
#app.post("/post")
async def postreq(request):
return bytearray(request["body"]).decode("utf-8")
#app.put("/put")
async def putreq(request):
return bytearray(request["body"]).decode("utf-8")
#app.delete("/delete")
async def deletereq(request):
return bytearray(request["body"]).decode("utf-8")
#app.patch("/patch")
async def patchreq(request):
return bytearray(request["body"]).decode("utf-8")
#app.get("/sleep")
async def sleeper():
await asyncio.sleep(5)
return "sleep function"
#app.get("/blocker")
def blocker():
import time
time.sleep(10)
return "blocker function"
if __name__ == "__main__":
app.add_header("server", "robyn")
app.add_directory(route="/test_dir",directory_path="./test_dir/build", index_file="index.html")
app.start(port=5000)
These are the files that I have used in my project. When I try to open 127.0.0.1:5000 from my machine, it shows nothing. However, when I log in the web container and do curl http://localhost:5000/, it gives the right response.
I am unable to figure out how to access it on the host machine?

I had to make the python server listen at '0.0.0.0'.
I added the following line in my codebase
app.start(port=5000, url='0.0.0.0')

celerybeat docker paho mqtt

my use case is, that I read a csv file and send these data via mqtt to a hivemq broker.
When I start the celery worker and the clerybeat locally (change rabbitmq and hivemq to localhost), everything works fine and I get the data in mqttFX Client. It does not work with the Docker-Configuration. The two worker and the celerybeat start correctly and in the method send_mqtt the inner function on_publish is called.
When I start it with the docker-compose file I get no data in my mqttFX client. The worker can connect to RabbitMQ and HiveMQ...I think...
Sum up: Everything works when I start the celeryworkers and the celerybeat in 3 terminals. It just does not work with my Docker-Configuration.
app.py
from celery import Celery
import pandas as pd
import paho.mqtt.client as mqtt
import datetime
import json
celery = Celery('app', broker='rabbitmq')
current_date = datetime.datetime.now().strftime('%Y-%m-%d')
filepath = f'/var/lib/excel/{current_date}.csv'
celery.conf.beat_schedule = {
"readcsv-task": {
"task": "app.read_csv",
"schedule": 10.0,
"args": (filepath,),
}
}
celery.conf.task_routes = ([
('app.read_csv', {'queue': 'csv'}),
('app.send_mqtt', {'queue': 'mqtt'}),
],)
#celery.task()
def send_mqtt(data):
def on_publish(client, userdata, result):
print("data published \n")
pass
topic = 'data/mqtt/'
client = mqtt.Client("ExcelService")
client.on_publish = on_publish
client.connect('hivemq', 1883)
mqtt_data = data.encode('utf-8')
client.publish(topic, mqtt_data)
print(f'mqtt data {mqtt_data} send, type {type(mqtt_data)}')
#celery.task()
def read_csv(filepath):
try:
df = pd.read_csv(filepath, sep=";")
last_row = df.tail(1)
value_id = int(last_row.iloc[0][0])
payload = int(last_row.iloc[0][2])
date = str(last_row.iloc[0][1])
data = {
"id": value_id,
"payload": payload,
"date": date
}
mqtt_data = json.dumps(data)
send_mqtt.delay(mqtt_data)
except IOError:
print(f'No file "{filepath}" found. Restart in 10sec.')
docker-compose.yaml:
version: "3.7"
services:
excelservice:
build:
context: ./app
dockerfile: ./Dockerfile
volumes:
- ../excel/:/var/lib/excel
networks:
- test
restart: always
depends_on:
- hivemq
- rabbitmq
rabbitmq:
image: "rabbitmq:3-management"
hostname: "test-rabbitmq"
restart: always
labels:
NAME: "test-rabbitmq"
networks:
- test
volumes:
- ./rabbitmq/rabbitmq.conf:/etc/rabbitmq/rabbitmq.config
ports:
- "5672:5672"
- "15672:15672"
hivemq:
container_name: hivemq
image: hivemq/hivemq3
restart: always
networks:
- test
ports:
- "1883:1883"
- "8080:8080"
networks:
test: {}
Start-script 4 celery worker and celerybeat:
#!/bin/sh -e
echo "Wait for RabbitMQ to start..."
sleep 15
echo "Start csvworker..." &
celery worker -A app --pool=eventlet -Q csv -n csvworker#%h --loglevel=INFO &
sleep 5
echo "Start mqttworker..." &
celery worker -A app --pool=eventlet -Q mqtt -n mqttworker#%h --loglevel=INFO &
sleep 5
echo "Start celerybeat..." &
celery -A app beat --loglevel=info
Dockerfile 4 app.py
FROM python:3.7-slim
RUN apt-get update && apt-get install -y dos2unix
COPY requirements.txt .
RUN pip install --upgrade pip
RUN pip install --upgrade setuptools
RUN pip install --no-cache-dir -r requirements.txt
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
RUN mkdir /code
WORKDIR /code
COPY . /code
ADD ./prod_start.sh /usr/local/bin/prod_start.sh
RUN dos2unix /usr/local/bin/prod_start.sh && apt-get --purge remove -y dos2unix && rm -rf /var/lib/apt/lists/*
RUN chmod u+x /usr/local/bin/prod_start.sh
ENTRYPOINT /usr/local/bin/prod_start.sh
EDIT EDIT EDIT
I changed the celery.task -> send_mqtt and put the client initalisation out of the method and it works. maybe someone can explain me, why this is working?
new app.py
from celery import Celery
import pandas as pd
import paho.mqtt.client as mqtt
import datetime
import json
# celery = Celery('app', broker='amqp://guest:guest#localhost:5672//')
celery = Celery('app', broker='rabbitmq')
current_date = datetime.datetime.now().strftime('%Y-%m-%d')
filepath = f'/var/lib/excel/{current_date}.csv'
celery.conf.beat_schedule = {
"readcsv-task": {
"task": "app.read_csv",
"schedule": 10.0,
"args": (filepath,),
}
}
celery.conf.task_routes = ([
('app.read_csv', {'queue': 'csv'}),
('app.send_mqtt', {'queue': 'mqtt'}),
],)
def on_publish(client, userdata, result):
print("data published \n")
pass
def on_disconnect(client, userdata, rc):
print("client disconnected ok")
def on_log(client, obj, level, string):
print(string)
client = mqtt.Client("ExcelService")
client.on_publish = on_publish
client.on_log = on_log
client.on_disconnect = on_disconnect
#celery.task()
def send_mqtt(data):
topic = 'data/mqtt/'
client.connect('hivemq', 1883)
mqtt_data = data.encode('UTF-8')
client.publish(topic, mqtt_data)
# client.disconnect()
print(f'mqtt data {mqtt_data} send, type {type(mqtt_data)}')
#celery.task()
def read_csv(filepath):
try:
df = pd.read_csv(filepath, sep=";")
last_row = df.tail(1)
value_id = int(last_row.iloc[0][0])
payload = int(last_row.iloc[0][2])
date = str(last_row.iloc[0][1])
data = {
"id": value_id,
"payload": payload,
"date": date
}
mqtt_data = json.dumps(data)
send_mqtt.delay(mqtt_data)
except IOError:
print(f'No file "{filepath}" found. Restart in 10sec.')

Python UDP server running in Docker container

I would like to test a simple UDP server in Docker before continue with the develop. I've never worked with Docker before so I assume it should be a beginner error
What I have currently?
The server
import threading
import socketserver
class ThreadedUDPRequestHandler(socketserver.BaseRequestHandler):
def handle(self):
data = self.request[0].strip()
current_thread = threading.current_thread()
print("Thread: {} client: {}, wrote: {}".format(current_thread.name, self.client_address, data))
Split = threading.Thread(target=ParseIncomingData,args=(data,))
Split.start()
class ThreadedUDPServer(socketserver.ThreadingMixIn, socketserver.UDPServer):
pass
def publish_messages(data):
"""Publishes multiple messages to a Pub/Sub topic."""
print('Published {} .'.format(data))
def ParseIncomingData(message):
sender = threading.Thread(target=publish_messages, args=(message,))
sender.start()
if __name__ == "__main__":
HOST, PORT = "0.0.0.0", 6071
try:
serverUDP = ThreadedUDPServer((HOST, PORT), ThreadedUDPRequestHandler)
server_thread_UDP = threading.Thread(target=serverUDP.serve_forever)
server_thread_UDP.daemon = True
server_thread_UDP.start()
serverUDP.serve_forever()
except (KeyboardInterrupt, SystemExit):
serverUDP.shutdown()
serverUDP.server_close()
exit()
The dockerfile
# Use an official Python runtime as a base image
FROM python:3.7.2-slim
# Set the working directory to /app
WORKDIR /app
# Copy the current directory contents into the container at /app
ADD . /app
# Make port 6071 available outside this container
EXPOSE 6071/udp
# Run app.py when the container launches
CMD ["python", "app.py"]
How I execute the container?
docker run --rm -p 6071:6071/udp listener
I have tried with several combinations about the usage of the port but I don't see anything when I run it (Windows)
EXTRA:
To test the server I'm using hercules to send the USP data:

It works with this configuration:
Using unbuffered output in dockerfile with
CMD ["python","-u","main.py"]

How can run docker image in kubernetes initiate from another and pass arguments

I am having two dockerized application which needs to run in kubernetes.
Here is the scenario which needs to achieve.
Docker-1 which is flask application.
Docker-2 which is python script will take input from the Docker-1 and execute and need to write some file in a shared volume of the Docker-1 container.
Here is the flask web-app code.
from flask import Flask, request, Response, jsonify
app = Flask(__name__)
#app.route('/')
def root():
return "The API is working fine"
#app.route('/run-docker')
def run_docker_2():
args = "input_combo"
query = <sql query>
<initiate the docker run and pass params>
exit
#No return message need run as async
if __name__ == "__main__":
app.run(debug=True, host='0.0.0.0', port=8080, threaded=True)
Docker file
FROM ubuntu:latest
MAINTAINER Abhilash KK "abhilash.kk#searshc.com"
RUN apt-get update -y
RUN apt-get install -y python-pip python-dev build-essential python-tk
COPY . /app
WORKDIR /app
RUN pip install -r requirements.txt
ENTRYPOINT ["/usr/bin/python"]
CMD ["app.py"]
requirements.txt
flask
Python script for the second docker. start_docker.py
import sys
input_combo = sys.argv[1]
query = sys.argv[2]
def function_to_run(input_combination,query):
#starting the model final creating file
function_to_run(input_combo, query)
Docker file 2
FROM python
COPY . /script
WORKDIR /script
CMD ["python", "start_docker.py"]
Please help me to connect with the docker images. or let me know any other way to achieve this problem. The basic requirement is to add a message to some queue and that queue listens for in time interval and starts the process with FIFO manner.
Any other approach in GCP service to initiate an async job will take input from the client and create a file which is accessible from web-app python.

First, create a Pod running "Docker-1" application. Then Kubernetes python client to spawn a second pod with "Docker-2".
You can share a volume between your pods in order to return the data to Docker1. In my code sample I'm using a host_path volume but you need to ensure that both pods are on the same node. I did add that code for readability.
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: docker1
labels:
app: docker1
spec:
replicas: 1
selector:
matchLabels:
app: docker1
template:
metadata:
labels:
app: docker1
spec:
containers:
- name: docker1
image: abhilash/docker1
ports:
- containerPort: 8080
volumeMounts:
- mountPath: /shared
name: shared-volume
volumes:
- name: shared-volume
hostPath:
path: /shared
The code of run_docker_2 handler:
from kubernetes import client, config
...
args = "input_combo"
config.load_incluster_config()
pod = client.V1Pod()
pod.metadata = client.V1ObjectMeta(name="docker2")
container = client.V1Container(name="docker2")
container.image = "abhilash/docker2"
container.args = [args]
volumeMount = client.V1VolumeMount(name="shared", mount_path="/shared")
container.volume_mounts = [volumeMount]
hostpath = client.V1HostPathVolumeSource(path = "/shared")
volume = client.V1Volume(name="shared")
volume.host_path = hostpath
spec = client.V1PodSpec(containers = [container])
spec.volumes = [volume]
pod.spec = spec
v1.create_namespaced_pod(namespace="default", body=pod)
return "OK"
A handler to read the returned results:
#app.route('/read-results')
def run_read():
with open("/shared/results.data") as file:
return file.read()
Note that it could be useful to add a watcher to wait for the pod to finish the job and then do some cleanup (delete the pod for instance)

From what I can understand you'd want the so called "sidecar pattern", you can run multiple containers in one pod and share a volume, e.g.:
apiVersion: v1
kind: Pod
metadata:
name: www
spec:
containers:
- name: nginx
image: nginx
volumeMounts:
- mountPath: /srv/www
name: www-data
readOnly: true
- name: git-monitor
image: kubernetes/git-monitor
env:
- name: GIT_REPO
value: http://github.com/some/repo.git
volumeMounts:
- mountPath: /data
name: www-data
volumes:
- name: www-data
emptyDir: {}
You could also benefit from getting to know the basics of how Kubernetes work: Kubernetes Basics