Develop Reference

Airflow Scheduler fails with error `sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (2006, 'MySQL server has gone away')`

I'm using Airflow 2.5.1 and setting up Airflow scheduler (Using LocalExecutor) + webserver on Deb9 Instance.
MySQL DB is on another instance and I checked using PING and airflow db check that the connection to the MySQL server is successful. I had even run airflow db init from this instance and it was able to create all the tables successfully.
When I start the scheduler using airflow scheduler command, getting the following error:
airflow-scheduler.err log below:
sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (2006, 'MySQL server has gone away')
(Background on this error at: https://sqlalche.me/e/14/e3q8)
[2023-02-09 01:12:34 +0530] [15451] [ERROR] Connection in use: ('0.0.0.0', 8793)
[2023-02-09 01:12:34 +0530] [15451] [ERROR] Retrying in 1 second.
[2023-02-09 01:12:35 +0530] [15451] [ERROR] Connection in use: ('0.0.0.0', 8793)
[2023-02-09 01:12:35 +0530] [15451] [ERROR] Retrying in 1 second.
[2023-02-09 01:12:36 +0530] [15451] [ERROR] Connection in use: ('0.0.0.0', 8793)
[2023-02-09 01:12:36 +0530] [15451] [ERROR] Retrying in 1 second.
[2023-02-09 01:12:37 +0530] [15451] [ERROR] Can't connect to ('0.0.0.0', 8793)
airflow-scheduler.log below:
sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (2006, 'MySQL server has gone away')
(Background on this error at: https://sqlalche.me/e/14/e3q8)
2023-02-09 01:12:33,040 INFO - Shutting down LocalExecutor; waiting for running tasks to finish. Signal again if you don't want to wait.
2023-02-09 01:12:34,066 INFO - Sending Signals.SIGTERM to group 15492. PIDs of all processes in the group: [15492]
2023-02-09 01:12:34,066 INFO - Sending the signal Signals.SIGTERM to group 15492
2023-02-09 01:12:34,158 INFO - Process psutil.Process(pid=15492, status='terminated', exitcode=0, started='01:12:32') (15492) terminated with exit code 0
2023-02-09 01:12:34,159 INFO - Exited execute loop
Any idea why is this happening?
[Update]
Connection in use: ('0.0.0.0', 8793) was because of some processes running from previous run. Have killed those processes. But I'm still getting MySQL server has gone away error.
airflow-scheduler.err log:
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/airflow/airflow_venv/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/airflow_venv/lib/python3.7/site-packages/airflow/__main__.py", line 39, in main
args.func(args)
File "/home/airflow/airflow_venv/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 52, in command
return func(*args, **kwargs)
File "/home/airflow/airflow_venv/lib/python3.7/site-packages/airflow/utils/cli.py", line 108, in wrapper
return f(*args, **kwargs)
File "/home/airflow/airflow_venv/lib/python3.7/site-packages/airflow/cli/commands/scheduler_command.py", line 68, in scheduler
_run_scheduler_job(args=args)
File "/home/airflow/airflow_venv/lib/python3.7/site-packages/airflow/cli/commands/scheduler_command.py", line 43, in _run_scheduler_job
job.run()
File "/home/airflow/airflow_venv/lib/python3.7/site-packages/airflow/jobs/base_job.py", line 258, in run
self._execute()
File "/home/airflow/airflow_venv/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 759, in _execute
self._run_scheduler_loop()
File "/home/airflow/airflow_venv/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 840, in _run_scheduler_loop
self.adopt_or_reset_orphaned_tasks()
File "/home/airflow/airflow_venv/lib/python3.7/site-packages/airflow/utils/session.py", line 75, in wrapper
return func(*args, session=session, **kwargs)
File "/usr/local/lib/python3.7/contextlib.py", line 119, in __exit__
next(self.gen)
File "/home/airflow/airflow_venv/lib/python3.7/site-packages/airflow/utils/session.py", line 36, in create_session
session.commit()
File "/home/airflow/airflow_venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1451, in commit
self._transaction.commit(_to_root=self.future)
File "/home/airflow/airflow_venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 836, in commit
trans.commit()
File "/home/airflow/airflow_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2459, in commit
self._do_commit()
File "/home/airflow/airflow_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2649, in _do_commit
self._connection_commit_impl()
File "/home/airflow/airflow_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2620, in _connection_commit_impl
self.connection._commit_impl()
File "/home/airflow/airflow_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1091, in _commit_impl
self._handle_dbapi_exception(e, None, None, None, None)
File "/home/airflow/airflow_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2125, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/home/airflow/airflow_venv/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
raise exception
File "/home/airflow/airflow_venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1089, in _commit_impl
self.engine.dialect.do_commit(self.connection)
File "/home/airflow/airflow_venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 686, in do_commit
dbapi_connection.commit()
sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (2006, 'MySQL server has gone away')
(Background on this error at: https://sqlalche.me/e/14/e3q8)

When running the LocalExecutor, the scheduler also starts a process to serve log files, by default on port 8793. The error tells you you've already got something running on port 8793, therefore it can't start and returns an error. You've probably already got a scheduler running.
The port is configurable by AIRFLOW__LOGGING__WORKER_LOG_SERVER_PORT. For more information, see https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#worker-log-server-port.

How to start a Ray cluster on one local server using yaml file config without docker

Could any one help me how to start ray on local server by a config file?
My current local server can run Ray successfully when using below command:
ray start --head --node-ip-address 127.0.0.1 --port 6379 --dashboard-host 0.0.0.0 --dashboard-port 8265 --gcs-server-port 8075 --object-manager-port 8076 --node-manager-port 8077 --min-worker-port 10002 --max-worker-port 19999
But now I need to move it to a config file so that other service can control it. I did try to make a cluster.yaml file and start it with command ray up cluster.yaml, but it's facing error.
The content of cluster.yaml file is:
cluster_name: default
max_workers: 1
upscaling_speed: 1.0
idle_timeout_minutes: 5
provider:
​type: local
​head_ip: 0.0.0.0
​worker_ips:
​- 127.0.0.1
auth:
​ssh_user: root
​ssh_private_key: ~/.ssh/id_rsa
file_mounts: {}
cluster_synced_files: []
file_mounts_sync_continuously: False
rsync_exclude:
​- "**/.git"
​- "**/.git/**"
rsync_filter:
​- ".gitignore"
initialization_commands: []
setup_commands: []
head_setup_commands: []
worker_setup_commands: []
head_start_ray_commands:
​- ray stop
​- ray start --head --port 6379 --dashboard-host '0.0.0.0' --dashboard-port 8265 --gcs-server-port 8075 --object-manager-port 8076 --node-manager-port 8077 --min-worker-port 10002 --max-worker-port 19999 --autoscaling-config=~/ray_bootstrap_config.yaml
worker_start_ray_commands:
​- ray stop
​- ray start --address=$RAY_HEAD_IP:6379 --object-manager-port=8076
head_node: {}
worker_nodes: {}
But it's unable to start due to below error:
<1/1> Setting up head node
​Prepared bootstrap config
2022-01-27 05:52:26,910 INFO node_provider.py:103 -- ClusterState: Writing cluster state: ['127.0.0.1', '0.0.0.0']
[1/7] Waiting for SSH to become available
​Running `uptime` as a test.
​Fetched IP: 0.0.0.0
05:52:33 up 3:43, 0 users, load average: 0.10, 0.18, 0.16
Shared connection to 0.0.0.0 closed.
​Success.
[2/7] Processing file mounts
[3/7] No worker file mounts to sync
[4/7] No initialization commands to run.
[5/7] Initalizing command runner
[6/7] No setup commands to run.
[7/7] Starting the Ray runtime
Shared connection to 0.0.0.0 closed.
Local node IP: 192.168.1.18
2022-01-27 05:52:37,541 INFO services.py:1340 -- View the Ray dashboard at http://192.168.1.18:8265
Traceback (most recent call last):
​File "/usr/local/lib/python3.7/dist-packages/ray/node.py", line 240, in __init__
​self.redis_password)
​File "/usr/local/lib/python3.7/dist-packages/ray/_private/services.py", line 324, in wait_for_node
​raise TimeoutError("Timed out while waiting for node to startup.")
TimeoutError: Timed out while waiting for node to startup.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
​File "/usr/local/bin/ray", line 8, in <module>
​sys.exit(main())
​File "/usr/local/lib/python3.7/dist-packages/ray/scripts/scripts.py", line 1989, in main
​return cli()
​File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1128, in __call__
​return self.main(*args, **kwargs)
​File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1053, in main
​rv = self.invoke(ctx)
​File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1659, in invoke
​return _process_result(sub_ctx.command.invoke(sub_ctx))
​File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1395, in invoke
​return ctx.invoke(self.callback, **ctx.params)
​File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 754, in invoke
​return __callback(*args, **kwargs)
​File "/usr/local/lib/python3.7/dist-packages/ray/scripts/scripts.py", line 633, in start
​ray_params, head=True, shutdown_at_exit=block, spawn_reaper=block)
​File "/usr/local/lib/python3.7/dist-packages/ray/node.py", line 243, in __init__
​"The current node has not been updated within 30 "
Exception: The current node has not been updated within 30 seconds, this could happen because of some of the Ray processes failed to startup.
Shared connection to 0.0.0.0 closed.
2022-01-27 05:53:07,718 INFO node_provider.py:103 -- ClusterState: Writing cluster state: ['127.0.0.1', '0.0.0.0']
​New status: update-failed
​!!!
​SSH command failed.
​!!!
​Failed to setup head node.

Gunicorn autorestart causing errors

I am running gunicorn in async mode behind a nginx reverse proxy. Both are in separate docker containers on the same VM in the host network and everything is running fine as long as I don't configure max_requests to autorestart workers after a certain amount of requests. With autorestart configured the reboot of workers is not handled correctly, throwing errors and causing failed responses. I need this settings to fix problems with memory leaks and prevent gunicorn and other application components from crashing.
Gunicorn log:
2020-08-07 06:55:23 [1438] [INFO] Autorestarting worker after current request.
2020-08-07 06:55:23 [1438] [ERROR] Socket error processing request.
Traceback (most recent call last):
File "/opt/mapproxy/lib/python3.5/site-packages/gunicorn/workers/base_async.py", line 65, in handle
util.reraise(*sys.exc_info())
File "/opt/mapproxy/lib/python3.5/site-packages/gunicorn/util.py", line 625, in reraise
raise value
File "/opt/mapproxy/lib/python3.5/site-packages/gunicorn/workers/base_async.py", line 38, in handle
listener_name = listener.getsockname()
OSError: [Errno 9] Bad file descriptor
Gunicorn is running with the following configuration:
bind = '0.0.0.0:8081'
worker_class = 'eventlet'
workers = 8
timeout = 60
no_sendfile = True
max_requests = 1000
max_requests_jitter = 500

gunicorn threads getting killed silently

gunicorn version 19.9.0
Got the following gunicorn config:
accesslog = "access.log"
worker_class = 'sync'
workers = 1
worker_connections = 1000
timeout = 300
graceful_timeout = 300
keepalive = 300
proc_name = 'server'
bind = '0.0.0.0:8080'
name = 'server.py'
preload = True
log_level = "info"
threads = 7
max_requests = 0
backlog = 100
As you can see, the server is configured to run 7 threads.
The server is started with:
gunicorn -c gunicorn_config.py server:app
Here are the number of lines and thread IDs from our log file at the beginning (with the last line being the thread of the main server):
10502 140625414080256
10037 140624842843904
9995 140624859629312
9555 140625430865664
9526 140624851236608
9409 140625405687552
2782 140625422472960
6 140628359804736
So 7 threads are processing the requests. (Already we can see that thread 140625422472960 is processing substantially fewer requests than the other threads.)
But after the lines examined above, thread 140625422472960 just vanishes and the log file only has:
19602 140624859629312
18861 140625405687552
18766 140624851236608
18765 140624842843904
12523 140625414080256
2111 140625430865664
(excluding the main thread here)
From the server logs we could see that the thread received a request and started processing it, but never finished. The client received no response either.
There is no error/warning in the log file, nor in stderr.
And running the app for a little longer, two more threads are gone:
102 140624842843904
102 140624851236608
68 140624859629312
85 140625405687552
How to debug this?

Digging into the stderr logs further, finally found something like this exception stack trace:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 134, in handle
req = six.next(parser)
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/parser.py", line 41, in __next__
self.mesg = self.mesg_class(self.cfg, self.unreader, self.req_count)
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 181, in __init__
super(Request, self).__init__(cfg, unreader)
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 54, in __init__
unused = self.parse(self.unreader)
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 230, in parse
self.headers = self.parse_headers(data[:idx])
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 74, in parse_headers
remote_addr = self.unreader.sock.getpeername()
OSError: [Errno 107] Transport endpoint is not connected
[2018-11-04 17:57:55 +0330] [31] [ERROR] Socket error processing request.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 134, in handle
req = six.next(parser)
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/parser.py", line 41, in __next__
self.mesg = self.mesg_class(self.cfg, self.unreader, self.req_count)
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 181, in __init__
super(Request, self).__init__(cfg, unreader)
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 54, in __init__
unused = self.parse(self.unreader)
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 230, in parse
self.headers = self.parse_headers(data[:idx])
File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 74, in parse_headers
remote_addr = self.unreader.sock.getpeername()
OSError: [Errno 107] Transport endpoint is not connected
This is due to this gunicorn bug.
An interim solution until this bug is fixed is to monkey patch gunicorn as done by asantoni.

Twisted plugin needs to fail fast of port is taken

I have a twistd plugin that listens on a port and does very simple things. The problem is that when I start it, if the post is not available it just sits there with the process running, but doing nothing. I need the process to exit immediately in this case so the larger system can notice and deal with the problem
I have code like this:
def makeService(options):
root = Resource() # Not what I actually have...
factory = server.Site(root)
server_string = b'tcp:{0}:interface={1}'.format(options['port'], options['interface'])
endpoint = endpoints.serverFromString(reactor, server_string)
service = internet.StreamServerEndpointService(endpoint, factory)
return service
This results in:
2016-12-19T11:42:21-0600] [info] [3082] [-] Log opened.
[2016-12-19T11:42:21-0600] [info] [3082] [-] twistd 15.5.0 (/home/matthew/code-venvs/wgcbap/bin/python 2.7.6) starting up.
[2016-12-19T11:42:21-0600] [info] [3082] [-] reactor class: twisted.internet.epollreactor.EPollReactor.
[2016-12-19T11:42:21-0600] [critical] [3082] [-] Unhandled Error
Traceback (most recent call last):
File "/home/matthew/code-venvs/wgcbap/local/lib/python2.7/site-packages/twisted/scripts/_twistd_unix.py", line 394, in startApplication
service.IService(application).privilegedStartService()
File "/home/matthew/code-venvs/wgcbap/local/lib/python2.7/site-packages/twisted/application/service.py", line 278, in privilegedStartService
service.privilegedStartService()
File "/home/matthew/code-venvs/wgcbap/local/lib/python2.7/site-packages/twisted/application/internet.py", line 352, in privilegedStartService
self._waitingForPort = self.endpoint.listen(self.factory)
File "/home/matthew/code-venvs/wgcbap/local/lib/python2.7/site-packages/twisted/internet/endpoints.py", line 457, in listen
interface=self._interface)
--- <exception caught here> ---
File "/home/matthew/code-venvs/wgcbap/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 121, in execute
result = callable(*args, **kw)
File "/home/matthew/code-venvs/wgcbap/local/lib/python2.7/site-packages/twisted/internet/posixbase.py", line 478, in listenTCP
p.startListening()
File "/home/matthew/code-venvs/wgcbap/local/lib/python2.7/site-packages/twisted/internet/tcp.py", line 984, in startListening
raise CannotListenError(self.interface, self.port, le)
twisted.internet.error.CannotListenError: Couldn't listen on 127.0.0.1:9999: [Errno 98] Address already in use.
And it continues to run, doing nothing....
Adding a line service._raiseSynchronously = True just above the return works, but seems to be undocumented and feels dirty.
Is there an approved way to do this?