Docker-compose is not building one container - python

I have the following docker-compose.yml
version: '2.1'
services:
files:
image: busybox
volumes:
- ./file/input:/file/input
- ./file/output:/file/output
[..]
classificatore:
build: classificatore/.
volumes:
- ./classificatore:/src
volumes_from:
- files
ports:
- 8080:5000
command: ["python", "/src/main.py"]
depends_on:
rabbit:
condition: service_healthy
mpeg-pre-encoder:
build: mpeg-pre-encoder/.
volumes:
- ./mpeg-pre-encoder:/src
- ./gabac_encoder:/src/gabac_encoder
volumes_from:
- files
depends_on:
rabbit:
condition: service_healthy
rabbit:
image: rabbitmq
ports:
- 5672:5672
healthcheck:
test: ["CMD", "rabbitmqctl", "cluster_status"]
interval: 5s
timeout: 2s
retries: 20
and the following dir structure:
classificatore/
Dockerfile
[..]
aggregatore/
Dockerfile
[..]
mpeg-pre-encoder/
Dockerfile
[..]
docker-compose.yml
[..]
the mpeg-pre-encoder Dockerfile
FROM python:3
RUN mkdir /src
ADD requirements.txt /src/.
WORKDIR /src
RUN pip install -r requirements.txt
ADD . /src/.
CMD ["python", "main.py"]
When I run
docker-compose up -d --build
and check for running containers
docker ps -a
I get
b6d7bac3e9b1 encoder_mpeg-pre-encoder "python main.py" 13 seconds ago Exited (1) 9 seconds ago encoder_mpeg-pre-encoder_1
b1182f765113 encoder_classificatore "python /src/main.py" 32 minutes ago Up 32 minutes 0.0.0.0:8080->5000/tcp encoder_classificatore_1
e9d8387430f2 encoder_aggregatore "python /src/main.py" 32 minutes ago Up 32 minutes 0.0.0.0:8000->8000/tcp encoder_aggregatore_1
a69183f42ab6 grafana/grafana:5.1.0 "/run.sh" About an hour ago Up About an hour 0.0.0.0:3000->3000/tcp encoder_grafana_1
ade83e898a98 prom/prometheus "/bin/prometheus --c…" About an hour ago Up 33 minutes 0.0.0.0:9090->9090/tcp encoder_prometheus_1
ff9960a7e924 busybox "sh" About an hour ago Exited (0) 12 seconds ago encoder_files_1
1738eea0645d rabbitmq "docker-entrypoint.s…" 2 weeks ago Up 2 hours (healthy) 4369/tcp, 5671/tcp, 25672/tcp, 0.0.0.0:5672->5672/tcp encoder_rabbit_1
(
The encoder_mpeg-pre-encoder container fails during building process.
Any hints on why I get this error?
EDIT 1: $docker logs b6d7bac3e9b1
2019-06-10 09:25:58,560 In attesa di nuovi job....
2019-06-10 09:25:58,560 RECEIVED: b"{'file_id': '59f311da-f02f-496a-a899-b1afd1768d64', 'parameter_set': {'reads_len': 136, 'myRnameDict': {'chrM': 0}}, 'au': {'id': 0, 'parameter_set_id': 16, 'au_type': 3, 'sequence_id': 0, 'file_name': '/file/output/minisam/59f311da-f02f-496a-a899-b1afd1768d64/CLASS_M chrM 1:7762.sam', 'AU_start_position': 0, 'AU_end_position': 7897}, 'au_count': 1}"
2019-06-10 09:25:58,561 reading the mini sam file
2019-06-10 09:25:58,582 Tempo totale impiegato per la lettura: 0 sec
2019-06-10 09:25:58,582 Tempo totale impiegato per la lettura: 0 sec
Traceback (most recent call last):
File "main.py", line 68, in <module>
rabbit.wait_for_job(decoded_symbols_from_sam_file)
File "/src/my_util.py", line 55, in wait_for_job
self.channel_input.start_consuming()
File "/usr/local/lib/python3.7/site-packages/pika/adapters/blocking_connection.py", line 1822, in start_consuming
self.connection.process_data_events(time_limit=None)
File "/usr/local/lib/python3.7/site-packages/pika/adapters/blocking_connection.py", line 758, in process_data_events
self._dispatch_channel_events()
File "/usr/local/lib/python3.7/site-packages/pika/adapters/blocking_connection.py", line 521, in _dispatch_channel_events
impl_channel._get_cookie()._dispatch_events()
File "/usr/local/lib/python3.7/site-packages/pika/adapters/blocking_connection.py", line 1445, in _dispatch_events
evt.body)
File "/src/my_util.py", line 48, in callback
worker_function(eval(body))
File "<decorator-gen-1>", line 2, in decoded_symbols_from_sam_file
File "/usr/local/lib/python3.7/site-packages/prometheus_client/context_managers.py", line 66, in wrapped
return func(*args, **kwargs)
File "main.py", line 38, in decoded_symbols_from_sam_file
mpegg_fields = convert_sam_to_mpegg(sam_fields, job_data['parameter_set']['myRnameDict'])
File "/src/mpegg_fields.py", line 40, in convert_sam_to_mpegg
mpegg_line[MPEGG_FIELD_ECIGAR_STRING] = calculate_ECIGAR_STRING(sam_line)
File "/src/mpegg_fields.py", line 107, in calculate_ECIGAR_STRING
raise ('Impossible to calculate ecigar string')
TypeError: exceptions must derive from BaseException
EDIT 2: running mpeg-pre-encoder script outside Docker
$python main.py
2019-06-10 12:49:29,541 Address resolution failed: gaierror(-2, 'Name or service not known')
2019-06-10 12:49:29,542 getaddrinfo failed: gaierror(-2, 'Name or service not known').
2019-06-10 12:49:29,542 AMQP connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - gaierror(-2, 'Name or service not known'); first exception - None.
2019-06-10 12:49:29,543 AMQPConnectionWorkflow - reporting failure: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - gaierror(-2, 'Name or service not known'); first exception - None
2019-06-10 12:49:29,543 Connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - gaierror(-2, 'Name or service not known'); first exception - None
2019-06-10 12:49:29,543 Error in _create_connection().
Traceback (most recent call last):
File "/home/martina/PycharmProjects/mpegg_std/venv/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection
raise self._reap_last_connection_workflow_error(error)
File "/home/martina/PycharmProjects/mpegg_std/venv/lib/python3.6/site-packages/pika/adapters/utils/selector_ioloop_adapter.py", line 564, in _resolve
self._flags)
File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
Traceback (most recent call last):
File "main.py", line 18, in <module>
rabbit = Rabbit()
File "/home/martina/PycharmProjects/mpegg_std/venv/lib/python3.6/site-packages/singleton_decorator/decorator.py", line 14, in __call__
self._instance = self.__wrapped__(*args, **kwargs)
File "/home/martina/PycharmProjects/mpegg_std/applicazione/encoder/mpeg-pre-encoder/my_util.py", line 26, in __init__
self.initConnect()
File "/home/martina/PycharmProjects/mpegg_std/applicazione/encoder/mpeg-pre-encoder/my_util.py", line 19, in initConnect
self.connection = pika.BlockingConnection(pika.ConnectionParameters(host=RABBIT_HOST))
File "/home/martina/PycharmProjects/mpegg_std/venv/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 360, in __init__
self._impl = self._create_connection(parameters, _impl_class)
File "/home/martina/PycharmProjects/mpegg_std/venv/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection
raise self._reap_last_connection_workflow_error(error)
File "/home/martina/PycharmProjects/mpegg_std/venv/lib/python3.6/site-packages/pika/adapters/utils/selector_ioloop_adapter.py", line 564, in _resolve
self._flags)
File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
EDIT 3:
I isolated the error coming from the exception, but I get a more serious error. In fact the mpeg-pre-encoder service is fired when the classificatore service sends data to a queue. I think something is broken in rabbitMQ, how can I check if I'm right?
Below the mpeg-pre-encoder service building log:
Building mpeg-pre-encoder
Step 1/7 : FROM python:3
---> a4cc999cf2aa
Step 2/7 : RUN mkdir /src
---> Using cache
---> 31d8a2dc751d
Step 3/7 : ADD requirements.txt /src/.
---> Using cache
---> 9c58880cc0b1
Step 4/7 : WORKDIR /src
---> Using cache
---> 910b4249cc31
Step 5/7 : RUN pip install -r requirements.txt
---> Using cache
---> a8dc0f729257
Step 6/7 : ADD . /src/.
---> Using cache
---> 591968ce0427
Step 7/7 : CMD ["python", "main.py"]
---> Using cache
---> b1500f95a936
Successfully built b1500f95a936
Successfully tagged encoder_mpeg-pre-encoder:latest
It seems building fine but as soon as I run docker ps -a
9e7c4723cd98 encoder_mpeg-pre-encoder "python main.py" 17 seconds ago Exited (1) 14 seconds ago encoder_mpeg-pre-encoder_1
with a different error
Traceback (most recent call last):
File "main.py", line 45, in <module>
rabbit.wait_for_job(decoded_symbols_from_sam_file)
File "/src/my_util.py", line 55, in wait_for_job
self.channel_input.start_consuming()
File "/usr/local/lib/python3.7/site-packages/pika/adapters/blocking_connection.py", line 1822, in start_consuming
self.connection.process_data_events(time_limit=None)
File "/usr/local/lib/python3.7/site-packages/pika/adapters/blocking_connection.py", line 758, in process_data_events
self._dispatch_channel_events()
File "/usr/local/lib/python3.7/site-packages/pika/adapters/blocking_connection.py", line 521, in _dispatch_channel_events
impl_channel._get_cookie()._dispatch_events()
File "/usr/local/lib/python3.7/site-packages/pika/adapters/blocking_connection.py", line 1445, in _dispatch_events
evt.body)
File "/src/my_util.py", line 48, in callback
worker_function(eval(body))
File "<decorator-gen-1>", line 2, in decoded_symbols_from_sam_file
File "/usr/local/lib/python3.7/site-packages/prometheus_client/context_managers.py", line 66, in wrapped
return func(*args, **kwargs)
File "main.py", line 26, in decoded_symbols_from_sam_file
COUNTER_INPUT_FILE_SIZE.inc(path.getsize(file_path))
File "/usr/local/lib/python3.7/genericpath.py", line 50, in getsize
return os.stat(filename).st_size
FileNotFoundError: [Errno 2] No such file or directory: '/file/output/minisam/9fdb727a-5093-4800-a1ba-c807dc48ca99/CLASS_P chrM 1:7757.sam'

You are trying to run python 2 code with python 3 interpreter. They changed how the exceptions are handled between the versions, so now you can’t just raise 'My exception'.
Change the base container to python 2 or rewrite the code, python 2 is nearing the end of life in 2020.

Related

How to start a Ray cluster on one local server using yaml file config without docker

Could any one help me how to start ray on local server by a config file?
My current local server can run Ray successfully when using below command:
ray start --head --node-ip-address 127.0.0.1 --port 6379 --dashboard-host 0.0.0.0 --dashboard-port 8265 --gcs-server-port 8075 --object-manager-port 8076 --node-manager-port 8077 --min-worker-port 10002 --max-worker-port 19999
But now I need to move it to a config file so that other service can control it. I did try to make a cluster.yaml file and start it with command ray up cluster.yaml, but it's facing error.
The content of cluster.yaml file is:
cluster_name: default
max_workers: 1
upscaling_speed: 1.0
idle_timeout_minutes: 5
provider:
​type: local
​head_ip: 0.0.0.0
​worker_ips:
​- 127.0.0.1
auth:
​ssh_user: root
​ssh_private_key: ~/.ssh/id_rsa
file_mounts: {}
cluster_synced_files: []
file_mounts_sync_continuously: False
rsync_exclude:
​- "**/.git"
​- "**/.git/**"
rsync_filter:
​- ".gitignore"
initialization_commands: []
setup_commands: []
head_setup_commands: []
worker_setup_commands: []
head_start_ray_commands:
​- ray stop
​- ray start --head --port 6379 --dashboard-host '0.0.0.0' --dashboard-port 8265 --gcs-server-port 8075 --object-manager-port 8076 --node-manager-port 8077 --min-worker-port 10002 --max-worker-port 19999 --autoscaling-config=~/ray_bootstrap_config.yaml
worker_start_ray_commands:
​- ray stop
​- ray start --address=$RAY_HEAD_IP:6379 --object-manager-port=8076
head_node: {}
worker_nodes: {}
But it's unable to start due to below error:
<1/1> Setting up head node
​Prepared bootstrap config
2022-01-27 05:52:26,910 INFO node_provider.py:103 -- ClusterState: Writing cluster state: ['127.0.0.1', '0.0.0.0']
[1/7] Waiting for SSH to become available
​Running `uptime` as a test.
​Fetched IP: 0.0.0.0
05:52:33 up 3:43, 0 users, load average: 0.10, 0.18, 0.16
Shared connection to 0.0.0.0 closed.
​Success.
[2/7] Processing file mounts
[3/7] No worker file mounts to sync
[4/7] No initialization commands to run.
[5/7] Initalizing command runner
[6/7] No setup commands to run.
[7/7] Starting the Ray runtime
Shared connection to 0.0.0.0 closed.
Local node IP: 192.168.1.18
2022-01-27 05:52:37,541 INFO services.py:1340 -- View the Ray dashboard at http://192.168.1.18:8265
Traceback (most recent call last):
​File "/usr/local/lib/python3.7/dist-packages/ray/node.py", line 240, in __init__
​self.redis_password)
​File "/usr/local/lib/python3.7/dist-packages/ray/_private/services.py", line 324, in wait_for_node
​raise TimeoutError("Timed out while waiting for node to startup.")
TimeoutError: Timed out while waiting for node to startup.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
​File "/usr/local/bin/ray", line 8, in <module>
​sys.exit(main())
​File "/usr/local/lib/python3.7/dist-packages/ray/scripts/scripts.py", line 1989, in main
​return cli()
​File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1128, in __call__
​return self.main(*args, **kwargs)
​File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1053, in main
​rv = self.invoke(ctx)
​File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1659, in invoke
​return _process_result(sub_ctx.command.invoke(sub_ctx))
​File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1395, in invoke
​return ctx.invoke(self.callback, **ctx.params)
​File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 754, in invoke
​return __callback(*args, **kwargs)
​File "/usr/local/lib/python3.7/dist-packages/ray/scripts/scripts.py", line 633, in start
​ray_params, head=True, shutdown_at_exit=block, spawn_reaper=block)
​File "/usr/local/lib/python3.7/dist-packages/ray/node.py", line 243, in __init__
​"The current node has not been updated within 30 "
Exception: The current node has not been updated within 30 seconds, this could happen because of some of the Ray processes failed to startup.
Shared connection to 0.0.0.0 closed.
2022-01-27 05:53:07,718 INFO node_provider.py:103 -- ClusterState: Writing cluster state: ['127.0.0.1', '0.0.0.0']
​New status: update-failed
​!!!
​SSH command failed.
​!!!
​Failed to setup head node.

Timeout in airflow DockerOperator

I am new to Airflow.
I am trying to run a container by an Airflow, but getting a timeout error:
[2021-07-21 07:02:06,176] {docker.py:231} INFO - Starting docker container from image python:3.9.2-slim
[2021-07-21 07:03:06,171] {taskinstance.py:1501} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "/usr/local/lib/python3.6/http/client.py", line 1379, in getresponse
response.begin()
File "/usr/local/lib/python3.6/http/client.py", line 311, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.6/http/client.py", line 272, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/local/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 410, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/packages/six.py", line 735, in reraise
raise value
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
chunked=chunked,
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 428, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 336, in _raise_timeout
self, url, "Read timed out. (read timeout=%s)" % timeout_value
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='host.docker.internal', port=2375): Read timed out. (read timeout=60)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1157, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1331, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1361, in _execute_task
result = task_copy.execute(context=context)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/docker/operators/docker.py", line 319, in execute
return self._run_image()
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/docker/operators/docker.py", line 258, in _run_image
tty=self.tty,
File "/home/airflow/.local/lib/python3.6/site-packages/docker/api/container.py", line 430, in create_container
return self.create_container_from_config(config, name)
File "/home/airflow/.local/lib/python3.6/site-packages/docker/api/container.py", line 440, in create_container_from_config
res = self._post_json(u, data=config, params=params)
File "/home/airflow/.local/lib/python3.6/site-packages/docker/api/client.py", line 296, in _post_json
return self._post(url, data=json.dumps(data2), **kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/docker/utils/decorators.py", line 46, in inner
return f(self, *args, **kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/docker/api/client.py", line 233, in _post
return self.post(url, **self._set_request_timeout(kwargs))
File "/home/airflow/.local/lib/python3.6/site-packages/requests/sessions.py", line 590, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/requests/adapters.py", line 529, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='host.docker.internal', port=2375): Read timed out. (read timeout=60)
[2021-07-21 07:03:06,179] {taskinstance.py:1551} INFO - Marking task as UP_FOR_RETRY. dag_id=etl_in_ch, task_id=etl_in_ch, execution_date=20210721T070203, start_date=20210721T070205, end_date=20210721T070306
[2021-07-21 07:03:06,215] {local_task_job.py:149} INFO - Task exited with return code 1
I have Mac system, and have configured docker socket as shown here: https://github.com/puckel/docker-airflow/issues/543#issuecomment-741842728
My code for Airflow is:
from datetime import datetime, timedelta
from airflow import DAG
from airflow.providers.docker.operators.docker import DockerOperator
from airflow.operators.dummy import DummyOperator
default_args = {
'owner' : 'airflow',
'description' : 'Extract data from different sources into CH and train model with it',
'depend_on_past' : False,
'start_date' : datetime(2021, 7, 19),
'email_on_failure' : False,
'email_on_retry' : False,
'retries' : 1,
'retry_delay' : timedelta(minutes=5)
}
with DAG('etl_in_ch', default_args=default_args, schedule_interval="00 23 * * *", catchup=False) as dag:
start_dag = DummyOperator(
task_id='start_dag'
)
end_dag = DummyOperator(
task_id='end_dag'
)
t1 = DockerOperator(
task_id='etl_in_ch',
image='python:3.9.2-slim',
container_name='etl_in_ch',
api_version='auto',
auto_remove=True,
command="apt-get update && apt-get install -y cron && apt-get install -y libxml2 libxslt-dev wget bzip2 gcc \
&& pip install --no-cache-dir --upgrade pip \
&& pip install --no-cache-dir poetry==1.1.5 \
&& poetry config virtualenvs.create false\
&& poetry install --no-interaction --no-ansi \
&& chmod +x /src/__main__.py \
&& python __main__.py",
docker_url="tcp://host.docker.internal:2375",
network_mode="bridge",
environment={"PYTHONDONTWRITEBYTECODE": 1, "PYTHONUNBUFFERED": 1},
working_dir="/usr/src/copy_data",
mounts=['./CH_ETL/src:/usr/src/copy_data', './pyproject.toml:pyproject.toml'],
xcom_all=True
)
start_dag >> t1
t1 >> end_dag
I saw that I may need to increase a docker timeout, but I do not understand exactly where, and I actually have tried already - on my machine, inside airflow-worker, inside bobrik/socat docker. Did not help.
The comment in Puckel image comment shows kinda complex solution.
If you have local docker that you want to use from within container, I think much better solution will be to switch to the official image of Apache airflow https://airflow.apache.org/docs/docker-stack/index.html and use Docker-in-Docker Solution where you map your docker socket to inside the container.
If you have remote docker engine, you are better of with explicitly specifying remote docker engine URL.
You need to make sure you get the right permissions (see for example https://forums.docker.com/t/docker-daemon-access-within-docker-as-non-root-user-permission-denied-while-trying-to-connect-to-docker-daemon-socket/94181) or run Airflow as root user (you can do it as of 2.0.2 release I think).
Note that there is a bug in recent release of docker provider for Airflow 2.0.0 (https://airflow.apache.org/docs/apache-airflow-providers-docker/stable/index.html) that will generally prevent you from running DockerOperator via either Docker-in-Docker or remote URL. You can either use previous version of the provider or wait for a fix that's coming in 2.1.0 https://github.com/apache/airflow/pull/16932

Flower: Set SSL 'verify_mode'

Using: Flower 0.9.5 (installed Tornado 6.0.4), Celery 4.4.6, Python 3.7
When starting Flower with
celery -A myProj flower
everything works as expected. Flower serves at http://localhost:5555.
When starting Flower with
celery -A myProj flower --keyfile=/home/me/cert/key.pem --certfile=/home/me/cert/cert.pem
it serves at https://localhost:5555 but when trying to access it, Chrome states ERR_CONNECTION_RESET and Flower logs
2020-09-16 17:19:37,421 - tornado.general - ERROR - Uncaught exception, closing connection.
Traceback (most recent call last):
File "/home/me/.env/lib/python3.7/site-packages/tornado/iostream.py", line 711, in _handle_events
self._handle_read()
File "/home/me/.env/lib/python3.7/site-packages/tornado/iostream.py", line 1498, in _handle_read
self._do_ssl_handshake()
File "/home/me/.env/lib/python3.7/site-packages/tornado/iostream.py", line 1458, in _do_ssl_handshake
if not self._verify_cert(self.socket.getpeercert()):
File "/home/me/.env/lib/python3.7/site-packages/tornado/iostream.py", line 1481, in _verify_cert
assert verify_mode in (ssl.CERT_NONE, ssl.CERT_REQUIRED, ssl.CERT_OPTIONAL)
UnboundLocalError: local variable 'verify_mode' referenced before assignment
2020-09-16 17:19:37,423 - asyncio - ERROR - Exception in callback None()
handle: <Handle cancelled>
Traceback (most recent call last):
File "/home/me/python/lib/python3.7/asyncio/events.py", line 88, in _run
self._context.run(self._callback, *self._args)
File "/home/me/.env/lib/python3.7/site-packages/tornado/platform/asyncio.py", line 139, in _handle_events
handler_func(fileobj, events)
File "/home/me/.env/lib/python3.7/site-packages/tornado/iostream.py", line 711, in _handle_events
self._handle_read()
File "/home/me/.env/lib/python3.7/site-packages/tornado/iostream.py", line 1498, in _handle_read
self._do_ssl_handshake()
File "/home/me/.env/lib/python3.7/site-packages/tornado/iostream.py", line 1458, in _do_ssl_handshake
if not self._verify_cert(self.socket.getpeercert()):
File "/home/me/.env/lib/python/site-packages/tornado/iostream.py", line 1481, in _verify_cert
assert verify_mode in (ssl.CERT_NONE, ssl.CERT_REQUIRED, ssl.CERT_OPTIONAL)
UnboundLocalError: local variable 'verify_mode' referenced before assignment
Note: Everything works when running Flower with
celery -B brokerURL flower --keyfile=/home/me/cert/key.pem --certfile=/home/me/cert/cert.pem
In /home/me/.env/lib/python3.7/site-packages/tornado/iostream.py there is:
def _verify_cert(self, peercert: Any) -> bool:
"""Returns ``True`` if peercert is valid according to the configured
validation mode and hostname.
The ssl handshake already tested the certificate for a valid
CA signature; the only thing that remains is to check
the hostname.
"""
if isinstance(self._ssl_options, dict):
verify_mode = self._ssl_options.get("cert_reqs", ssl.CERT_NONE)
elif isinstance(self._ssl_options, ssl.SSLContext):
verify_mode = self._ssl_options.verify_mode
assert verify_mode in (ssl.CERT_NONE, ssl.CERT_REQUIRED, ssl.CERT_OPTIONAL) # LINE 1481
if verify_mode == ssl.CERT_NONE or self._server_hostname is None:
return True
cert = self.socket.getpeercert()
if cert is None and verify_mode == ssl.CERT_REQUIRED:
gen_log.warning("No SSL certificate given")
return False
try:
ssl.match_hostname(peercert, self._server_hostname)
except ssl.CertificateError as e:
gen_log.warning("Invalid SSL certificate: %s" % e)
return False
else:
return True
How can I pass verify_mode = ssl.CERT_REQUIRED to tornado via Flower? Setting it manually inside _verify_cert does work.

airflow task is unable to connect to remote oracle database

airflow task is unable to connect to remote oracle database but i am able to connect to the same remote oracle database using the same code from python shell.
i checked running environment is same for both the shell and airflow.
i am attaching the log error which i am getting.
[2018-07-18 13:12:11,037] {models.py:1428} INFO - Executing <Task(AadharNumberValidation): validating_data> on 2018-07-18 07:42:05.573491
[2018-07-18 13:12:11,037] {base_task_runner.py:115} INFO - Running: ['bash', '-c', 'airflow run data_validation validating_data 2018-07-18T07:42:05.573491 --job_id 206 --raw -sd /Users/b0204890/Desktop/python/airflow_home/dags/data_validaton.py']
[2018-07-18 13:12:11,531] {base_task_runner.py:98} INFO - Subtask: [2018-07-18 13:12:11,531] {__init__.py:45} INFO - Using executor SequentialExecutor
[2018-07-18 13:12:11,588] {base_task_runner.py:98} INFO - Subtask: [2018-07-18 13:12:11,588] {models.py:189} INFO - Filling up the DagBag from /Users/b0204890/Desktop/python/airflow_home/dags/data_validaton.py
[2018-07-18 13:12:11,661] {cli.py:374} INFO - Running on host LTB0204890-Mac.local
[2018-07-18 13:12:11,669] {base_task_runner.py:98} INFO - Subtask: [2018-07-18 13:12:11,669] {validation_operators.py:37} INFO - operator_param yadav: Script one validation
[2018-07-18 13:12:11,678] {models.py:1595} ERROR - DPI-1047: 64-bit Oracle Client library cannot be loaded: "dlopen(libclntsh.dylib, 1): image not found". See https://oracle.github.io/odpi/doc/installation.html#macos for help
Traceback (most recent call last):
File "/Users/b0204890/venv/python3/lib/python3.6/site-packages/airflow/models.py", line 1493, in _run_raw_task
result = task_copy.execute(context=context)
File "/Users/b0204890/Desktop/python//airflow_home/plugins/validation_operators.py", line 38, in execute
cursor = create_connection(user="USERNAME",port="PORT",host="HOST",pwd="password",sid="SID")
File "/Users/b0204890/Desktop/python/airflow_home/utility/validation.py", line 30, in create_connection
connection = cx_Oracle.connect(user, pwd, service)
cx_Oracle.DatabaseError: DPI-1047: 64-bit Oracle Client library cannot be loaded: "dlopen(libclntsh.dylib, 1): image not found". See https://oracle.github.io/odpi/doc/installation.html#macos for help
[2018-07-18 13:12:11,681] {models.py:1616} INFO - Marking task as UP_FOR_RETRY
[2018-07-18 13:12:11,682] {base_task_runner.py:98} INFO - Subtask: [2018-07-18 13:12:11,682] {configuration.py:206} WARNING - section/key [smtp/smtp_user] not found in config
[2018-07-18 13:12:11,684] {models.py:1628} ERROR - Failed to send email to: ['tushar.smartcorp#gmail.com']
[2018-07-18 13:12:11,684] {models.py:1629} ERROR - [Errno 61] Connection refused
Traceback (most recent call last):
File "/Users/b0204890/venv/python3/lib/python3.6/site-packages/airflow/models.py", line 1493, in _run_raw_task
result = task_copy.execute(context=context)
File "/Users/b0204890/Desktop/python//airflow_home/plugins/validation_operators.py", line 38, in execute
cursor = create_connection(user="USERNAME",port="PORT",host="HOST",pwd="PASSWORD",sid="SID")
File "/Users/b0204890/Desktop/python/airflow_home/utility/validation.py", line 30, in create_connection
connection = cx_Oracle.connect(user, pwd, service)
cx_Oracle.DatabaseError: DPI-1047: 64-bit Oracle Client library cannot be loaded: "dlopen(libclntsh.dylib, 1): image not found". See https://oracle.github.io/odpi/doc/installation.html#macos for help
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/b0204890/venv/python3/lib/python3.6/site-packages/airflow/models.py", line 1618, in handle_failure
self.email_alert(error, is_retry=True)
File "/Users/b0204890/venv/python3/lib/python3.6/site-packages/airflow/models.py", line 1779, in email_alert
send_email(task.email, title, body)
File "/Users/b0204890/venv/python3/lib/python3.6/site-packages/airflow/utils/email.py", line 44, in send_email
return backend(to, subject, html_content, files=files, dryrun=dryrun, cc=cc, bcc=bcc, mime_subtype=mime_subtype)
File "/Users/b0204890/venv/python3/lib/python3.6/site-packages/airflow/utils/email.py", line 87, in send_email_smtp
send_MIME_email(SMTP_MAIL_FROM, recipients, msg, dryrun)
File "/Users/b0204890/venv/python3/lib/python3.6/site-packages/airflow/utils/email.py", line 107, in send_MIME_email
s = smtplib.SMTP_SSL(SMTP_HOST, SMTP_PORT) if SMTP_SSL else smtplib.SMTP(SMTP_HOST, SMTP_PORT)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/smtplib.py", line 251, in __init__
(code, msg) = self.connect(host, port)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/smtplib.py", line 336, in connect
self.sock = self._get_socket(host, port, self.timeout)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/smtplib.py", line 307, in _get_socket
self.source_address)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socket.py", line 724, in create_connection
raise err
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socket.py", line 713, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
Without following all of the instructions the connector will not work. Simply doing an installation for the Python package cx-Oracle is not enough. In this case it could also be that the venv of Airflow cannot access all neccessary files or that there is some rights issue.
As stated in the message, you need to follow all steps seen in: https://oracle.github.io/odpi/doc/installation.html#macos

Deploying Flower to Heroku

I'm following the instructions on https://github.com/jorilallo/celery-flower-heroku to deploy Flower celery monitoring app to Heroku.
After configuring and deploying my app I see the following in heroku logs:
Traceback (most recent call last):
File "/app/.heroku/python/bin/flower", line 9, in <module>
load_entry_point('flower==0.7.0', 'console_scripts', 'flower')()
File "/app/.heroku/python/lib/python2.7/site-packages/flower/__main__.py", line 11, in main
flower.execute_from_commandline()
File "/app/.heroku/python/lib/python2.7/site-packages/celery/bin/base.py", line 306, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "/app/.heroku/python/lib/python2.7/site-packages/flower/command.py", line 99, in handle_argv
return self.run_from_argv(prog_name, argv)
File "/app/.heroku/python/lib/python2.7/site-packages/flower/command.py", line 75, in run_from_argv
**app_settings)
File "/app/.heroku/python/lib/python2.7/site-packages/flower/app.py", line 40, in __init__
max_tasks_in_memory=max_tasks)
File "/app/.heroku/python/lib/python2.7/site-packages/flower/events.py", line 60, in __init__
state = shelve.open(self._db)
File "/app/.heroku/python/lib/python2.7/shelve.py", line 239, in open
return DbfilenameShelf(filename, flag, protocol, writeback)
File "/app/.heroku/python/lib/python2.7/shelve.py", line 223, in __init__
Shelf.__init__(self, anydbm.open(filename, flag), protocol, writeback)
File "/app/.heroku/python/lib/python2.7/anydbm.py", line 85, in open
return mod.open(file, flag, mode)
File "/app/.heroku/python/lib/python2.7/dumbdbm.py", line 250, in open
return _Database(file, mode)
File "/app/.heroku/python/lib/python2.7/dumbdbm.py", line 71, in __init__
f = _open(self._datfile, 'w')
IOError: [Errno 2] No such file or directory: 'postgres://USERNAME:PASSWORD#ec2-HOST.compute-1.amazonaws.com:5432/DBNAME.dat'
Notice the .dat appendix there? No idea where it comes from, its not present int my DATABASE_URL env variable.
Furthermore, the error above is with flower 0.7. I also tried installing 0.6, with which I do get further (namely the DB is correctly recognized and connection established), but I then get the following warnings once flower starts:
2014-06-19T15:14:02.464424+00:00 app[web.1]: [E 140619 15:14:02 state:138] Failed to inspect workers: '[Errno 104] Connection reset by peer', trying again in 128 seconds
2014-06-19T15:14:02.464844+00:00 app[web.1]: [E 140619 15:14:02 events:103] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 128 seconds.
Loading flower in my browser does show a few tabs of stuff, but there is no data.
How do I resolve these issues?
Flower doesn't support database persistence. It saves the state to file(s) using shelve module.

Categories

Resources