Develop Reference

How to start a Ray cluster on one local server using yaml file config without docker

Could any one help me how to start ray on local server by a config file?
My current local server can run Ray successfully when using below command:
ray start --head --node-ip-address 127.0.0.1 --port 6379 --dashboard-host 0.0.0.0 --dashboard-port 8265 --gcs-server-port 8075 --object-manager-port 8076 --node-manager-port 8077 --min-worker-port 10002 --max-worker-port 19999
But now I need to move it to a config file so that other service can control it. I did try to make a cluster.yaml file and start it with command ray up cluster.yaml, but it's facing error.
The content of cluster.yaml file is:
cluster_name: default
max_workers: 1
upscaling_speed: 1.0
idle_timeout_minutes: 5
provider:
​type: local
​head_ip: 0.0.0.0
​worker_ips:
​- 127.0.0.1
auth:
​ssh_user: root
​ssh_private_key: ~/.ssh/id_rsa
file_mounts: {}
cluster_synced_files: []
file_mounts_sync_continuously: False
rsync_exclude:
​- "**/.git"
​- "**/.git/**"
rsync_filter:
​- ".gitignore"
initialization_commands: []
setup_commands: []
head_setup_commands: []
worker_setup_commands: []
head_start_ray_commands:
​- ray stop
​- ray start --head --port 6379 --dashboard-host '0.0.0.0' --dashboard-port 8265 --gcs-server-port 8075 --object-manager-port 8076 --node-manager-port 8077 --min-worker-port 10002 --max-worker-port 19999 --autoscaling-config=~/ray_bootstrap_config.yaml
worker_start_ray_commands:
​- ray stop
​- ray start --address=$RAY_HEAD_IP:6379 --object-manager-port=8076
head_node: {}
worker_nodes: {}
But it's unable to start due to below error:
<1/1> Setting up head node
​Prepared bootstrap config
2022-01-27 05:52:26,910 INFO node_provider.py:103 -- ClusterState: Writing cluster state: ['127.0.0.1', '0.0.0.0']
[1/7] Waiting for SSH to become available
​Running `uptime` as a test.
​Fetched IP: 0.0.0.0
05:52:33 up 3:43, 0 users, load average: 0.10, 0.18, 0.16
Shared connection to 0.0.0.0 closed.
​Success.
[2/7] Processing file mounts
[3/7] No worker file mounts to sync
[4/7] No initialization commands to run.
[5/7] Initalizing command runner
[6/7] No setup commands to run.
[7/7] Starting the Ray runtime
Shared connection to 0.0.0.0 closed.
Local node IP: 192.168.1.18
2022-01-27 05:52:37,541 INFO services.py:1340 -- View the Ray dashboard at http://192.168.1.18:8265
Traceback (most recent call last):
​File "/usr/local/lib/python3.7/dist-packages/ray/node.py", line 240, in __init__
​self.redis_password)
​File "/usr/local/lib/python3.7/dist-packages/ray/_private/services.py", line 324, in wait_for_node
​raise TimeoutError("Timed out while waiting for node to startup.")
TimeoutError: Timed out while waiting for node to startup.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
​File "/usr/local/bin/ray", line 8, in <module>
​sys.exit(main())
​File "/usr/local/lib/python3.7/dist-packages/ray/scripts/scripts.py", line 1989, in main
​return cli()
​File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1128, in __call__
​return self.main(*args, **kwargs)
​File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1053, in main
​rv = self.invoke(ctx)
​File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1659, in invoke
​return _process_result(sub_ctx.command.invoke(sub_ctx))
​File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1395, in invoke
​return ctx.invoke(self.callback, **ctx.params)
​File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 754, in invoke
​return __callback(*args, **kwargs)
​File "/usr/local/lib/python3.7/dist-packages/ray/scripts/scripts.py", line 633, in start
​ray_params, head=True, shutdown_at_exit=block, spawn_reaper=block)
​File "/usr/local/lib/python3.7/dist-packages/ray/node.py", line 243, in __init__
​"The current node has not been updated within 30 "
Exception: The current node has not been updated within 30 seconds, this could happen because of some of the Ray processes failed to startup.
Shared connection to 0.0.0.0 closed.
2022-01-27 05:53:07,718 INFO node_provider.py:103 -- ClusterState: Writing cluster state: ['127.0.0.1', '0.0.0.0']
​New status: update-failed
​!!!
​SSH command failed.
​!!!
​Failed to setup head node.

Timeout in airflow DockerOperator

I am new to Airflow.
I am trying to run a container by an Airflow, but getting a timeout error:
[2021-07-21 07:02:06,176] {docker.py:231} INFO - Starting docker container from image python:3.9.2-slim
[2021-07-21 07:03:06,171] {taskinstance.py:1501} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "/usr/local/lib/python3.6/http/client.py", line 1379, in getresponse
response.begin()
File "/usr/local/lib/python3.6/http/client.py", line 311, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.6/http/client.py", line 272, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/local/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 410, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/packages/six.py", line 735, in reraise
raise value
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
chunked=chunked,
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 428, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 336, in _raise_timeout
self, url, "Read timed out. (read timeout=%s)" % timeout_value
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='host.docker.internal', port=2375): Read timed out. (read timeout=60)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1157, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1331, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1361, in _execute_task
result = task_copy.execute(context=context)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/docker/operators/docker.py", line 319, in execute
return self._run_image()
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/docker/operators/docker.py", line 258, in _run_image
tty=self.tty,
File "/home/airflow/.local/lib/python3.6/site-packages/docker/api/container.py", line 430, in create_container
return self.create_container_from_config(config, name)
File "/home/airflow/.local/lib/python3.6/site-packages/docker/api/container.py", line 440, in create_container_from_config
res = self._post_json(u, data=config, params=params)
File "/home/airflow/.local/lib/python3.6/site-packages/docker/api/client.py", line 296, in _post_json
return self._post(url, data=json.dumps(data2), **kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/docker/utils/decorators.py", line 46, in inner
return f(self, *args, **kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/docker/api/client.py", line 233, in _post
return self.post(url, **self._set_request_timeout(kwargs))
File "/home/airflow/.local/lib/python3.6/site-packages/requests/sessions.py", line 590, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/requests/adapters.py", line 529, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='host.docker.internal', port=2375): Read timed out. (read timeout=60)
[2021-07-21 07:03:06,179] {taskinstance.py:1551} INFO - Marking task as UP_FOR_RETRY. dag_id=etl_in_ch, task_id=etl_in_ch, execution_date=20210721T070203, start_date=20210721T070205, end_date=20210721T070306
[2021-07-21 07:03:06,215] {local_task_job.py:149} INFO - Task exited with return code 1
I have Mac system, and have configured docker socket as shown here: https://github.com/puckel/docker-airflow/issues/543#issuecomment-741842728
My code for Airflow is:
from datetime import datetime, timedelta
from airflow import DAG
from airflow.providers.docker.operators.docker import DockerOperator
from airflow.operators.dummy import DummyOperator
default_args = {
'owner' : 'airflow',
'description' : 'Extract data from different sources into CH and train model with it',
'depend_on_past' : False,
'start_date' : datetime(2021, 7, 19),
'email_on_failure' : False,
'email_on_retry' : False,
'retries' : 1,
'retry_delay' : timedelta(minutes=5)
}
with DAG('etl_in_ch', default_args=default_args, schedule_interval="00 23 * * *", catchup=False) as dag:
start_dag = DummyOperator(
task_id='start_dag'
)
end_dag = DummyOperator(
task_id='end_dag'
)
t1 = DockerOperator(
task_id='etl_in_ch',
image='python:3.9.2-slim',
container_name='etl_in_ch',
api_version='auto',
auto_remove=True,
command="apt-get update && apt-get install -y cron && apt-get install -y libxml2 libxslt-dev wget bzip2 gcc \
&& pip install --no-cache-dir --upgrade pip \
&& pip install --no-cache-dir poetry==1.1.5 \
&& poetry config virtualenvs.create false\
&& poetry install --no-interaction --no-ansi \
&& chmod +x /src/__main__.py \
&& python __main__.py",
docker_url="tcp://host.docker.internal:2375",
network_mode="bridge",
environment={"PYTHONDONTWRITEBYTECODE": 1, "PYTHONUNBUFFERED": 1},
working_dir="/usr/src/copy_data",
mounts=['./CH_ETL/src:/usr/src/copy_data', './pyproject.toml:pyproject.toml'],
xcom_all=True
)
start_dag >> t1
t1 >> end_dag
I saw that I may need to increase a docker timeout, but I do not understand exactly where, and I actually have tried already - on my machine, inside airflow-worker, inside bobrik/socat docker. Did not help.

The comment in Puckel image comment shows kinda complex solution.
If you have local docker that you want to use from within container, I think much better solution will be to switch to the official image of Apache airflow https://airflow.apache.org/docs/docker-stack/index.html and use Docker-in-Docker Solution where you map your docker socket to inside the container.
If you have remote docker engine, you are better of with explicitly specifying remote docker engine URL.
You need to make sure you get the right permissions (see for example https://forums.docker.com/t/docker-daemon-access-within-docker-as-non-root-user-permission-denied-while-trying-to-connect-to-docker-daemon-socket/94181) or run Airflow as root user (you can do it as of 2.0.2 release I think).
Note that there is a bug in recent release of docker provider for Airflow 2.0.0 (https://airflow.apache.org/docs/apache-airflow-providers-docker/stable/index.html) that will generally prevent you from running DockerOperator via either Docker-in-Docker or remote URL. You can either use previous version of the provider or wait for a fix that's coming in 2.1.0 https://github.com/apache/airflow/pull/16932

Flower: Set SSL 'verify_mode'

Using: Flower 0.9.5 (installed Tornado 6.0.4), Celery 4.4.6, Python 3.7
When starting Flower with
celery -A myProj flower
everything works as expected. Flower serves at http://localhost:5555.
When starting Flower with
celery -A myProj flower --keyfile=/home/me/cert/key.pem --certfile=/home/me/cert/cert.pem
it serves at https://localhost:5555 but when trying to access it, Chrome states ERR_CONNECTION_RESET and Flower logs
2020-09-16 17:19:37,421 - tornado.general - ERROR - Uncaught exception, closing connection.
Traceback (most recent call last):
File "/home/me/.env/lib/python3.7/site-packages/tornado/iostream.py", line 711, in _handle_events
self._handle_read()
File "/home/me/.env/lib/python3.7/site-packages/tornado/iostream.py", line 1498, in _handle_read
self._do_ssl_handshake()
File "/home/me/.env/lib/python3.7/site-packages/tornado/iostream.py", line 1458, in _do_ssl_handshake
if not self._verify_cert(self.socket.getpeercert()):
File "/home/me/.env/lib/python3.7/site-packages/tornado/iostream.py", line 1481, in _verify_cert
assert verify_mode in (ssl.CERT_NONE, ssl.CERT_REQUIRED, ssl.CERT_OPTIONAL)
UnboundLocalError: local variable 'verify_mode' referenced before assignment
2020-09-16 17:19:37,423 - asyncio - ERROR - Exception in callback None()
handle: <Handle cancelled>
Traceback (most recent call last):
File "/home/me/python/lib/python3.7/asyncio/events.py", line 88, in _run
self._context.run(self._callback, *self._args)
File "/home/me/.env/lib/python3.7/site-packages/tornado/platform/asyncio.py", line 139, in _handle_events
handler_func(fileobj, events)
File "/home/me/.env/lib/python3.7/site-packages/tornado/iostream.py", line 711, in _handle_events
self._handle_read()
File "/home/me/.env/lib/python3.7/site-packages/tornado/iostream.py", line 1498, in _handle_read
self._do_ssl_handshake()
File "/home/me/.env/lib/python3.7/site-packages/tornado/iostream.py", line 1458, in _do_ssl_handshake
if not self._verify_cert(self.socket.getpeercert()):
File "/home/me/.env/lib/python/site-packages/tornado/iostream.py", line 1481, in _verify_cert
assert verify_mode in (ssl.CERT_NONE, ssl.CERT_REQUIRED, ssl.CERT_OPTIONAL)
UnboundLocalError: local variable 'verify_mode' referenced before assignment
Note: Everything works when running Flower with
celery -B brokerURL flower --keyfile=/home/me/cert/key.pem --certfile=/home/me/cert/cert.pem
In /home/me/.env/lib/python3.7/site-packages/tornado/iostream.py there is:
def _verify_cert(self, peercert: Any) -> bool:
"""Returns ``True`` if peercert is valid according to the configured
validation mode and hostname.
The ssl handshake already tested the certificate for a valid
CA signature; the only thing that remains is to check
the hostname.
"""
if isinstance(self._ssl_options, dict):
verify_mode = self._ssl_options.get("cert_reqs", ssl.CERT_NONE)
elif isinstance(self._ssl_options, ssl.SSLContext):
verify_mode = self._ssl_options.verify_mode
assert verify_mode in (ssl.CERT_NONE, ssl.CERT_REQUIRED, ssl.CERT_OPTIONAL) # LINE 1481
if verify_mode == ssl.CERT_NONE or self._server_hostname is None:
return True
cert = self.socket.getpeercert()
if cert is None and verify_mode == ssl.CERT_REQUIRED:
gen_log.warning("No SSL certificate given")
return False
try:
ssl.match_hostname(peercert, self._server_hostname)
except ssl.CertificateError as e:
gen_log.warning("Invalid SSL certificate: %s" % e)
return False
else:
return True
How can I pass verify_mode = ssl.CERT_REQUIRED to tornado via Flower? Setting it manually inside _verify_cert does work.

airflow task is unable to connect to remote oracle database

airflow task is unable to connect to remote oracle database but i am able to connect to the same remote oracle database using the same code from python shell.
i checked running environment is same for both the shell and airflow.
i am attaching the log error which i am getting.
[2018-07-18 13:12:11,037] {models.py:1428} INFO - Executing <Task(AadharNumberValidation): validating_data> on 2018-07-18 07:42:05.573491
[2018-07-18 13:12:11,037] {base_task_runner.py:115} INFO - Running: ['bash', '-c', 'airflow run data_validation validating_data 2018-07-18T07:42:05.573491 --job_id 206 --raw -sd /Users/b0204890/Desktop/python/airflow_home/dags/data_validaton.py']
[2018-07-18 13:12:11,531] {base_task_runner.py:98} INFO - Subtask: [2018-07-18 13:12:11,531] {__init__.py:45} INFO - Using executor SequentialExecutor
[2018-07-18 13:12:11,588] {base_task_runner.py:98} INFO - Subtask: [2018-07-18 13:12:11,588] {models.py:189} INFO - Filling up the DagBag from /Users/b0204890/Desktop/python/airflow_home/dags/data_validaton.py
[2018-07-18 13:12:11,661] {cli.py:374} INFO - Running on host LTB0204890-Mac.local
[2018-07-18 13:12:11,669] {base_task_runner.py:98} INFO - Subtask: [2018-07-18 13:12:11,669] {validation_operators.py:37} INFO - operator_param yadav: Script one validation
[2018-07-18 13:12:11,678] {models.py:1595} ERROR - DPI-1047: 64-bit Oracle Client library cannot be loaded: "dlopen(libclntsh.dylib, 1): image not found". See https://oracle.github.io/odpi/doc/installation.html#macos for help
Traceback (most recent call last):
File "/Users/b0204890/venv/python3/lib/python3.6/site-packages/airflow/models.py", line 1493, in _run_raw_task
result = task_copy.execute(context=context)
File "/Users/b0204890/Desktop/python//airflow_home/plugins/validation_operators.py", line 38, in execute
cursor = create_connection(user="USERNAME",port="PORT",host="HOST",pwd="password",sid="SID")
File "/Users/b0204890/Desktop/python/airflow_home/utility/validation.py", line 30, in create_connection
connection = cx_Oracle.connect(user, pwd, service)
cx_Oracle.DatabaseError: DPI-1047: 64-bit Oracle Client library cannot be loaded: "dlopen(libclntsh.dylib, 1): image not found". See https://oracle.github.io/odpi/doc/installation.html#macos for help
[2018-07-18 13:12:11,681] {models.py:1616} INFO - Marking task as UP_FOR_RETRY
[2018-07-18 13:12:11,682] {base_task_runner.py:98} INFO - Subtask: [2018-07-18 13:12:11,682] {configuration.py:206} WARNING - section/key [smtp/smtp_user] not found in config
[2018-07-18 13:12:11,684] {models.py:1628} ERROR - Failed to send email to: ['tushar.smartcorp#gmail.com']
[2018-07-18 13:12:11,684] {models.py:1629} ERROR - [Errno 61] Connection refused
Traceback (most recent call last):
File "/Users/b0204890/venv/python3/lib/python3.6/site-packages/airflow/models.py", line 1493, in _run_raw_task
result = task_copy.execute(context=context)
File "/Users/b0204890/Desktop/python//airflow_home/plugins/validation_operators.py", line 38, in execute
cursor = create_connection(user="USERNAME",port="PORT",host="HOST",pwd="PASSWORD",sid="SID")
File "/Users/b0204890/Desktop/python/airflow_home/utility/validation.py", line 30, in create_connection
connection = cx_Oracle.connect(user, pwd, service)
cx_Oracle.DatabaseError: DPI-1047: 64-bit Oracle Client library cannot be loaded: "dlopen(libclntsh.dylib, 1): image not found". See https://oracle.github.io/odpi/doc/installation.html#macos for help
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/b0204890/venv/python3/lib/python3.6/site-packages/airflow/models.py", line 1618, in handle_failure
self.email_alert(error, is_retry=True)
File "/Users/b0204890/venv/python3/lib/python3.6/site-packages/airflow/models.py", line 1779, in email_alert
send_email(task.email, title, body)
File "/Users/b0204890/venv/python3/lib/python3.6/site-packages/airflow/utils/email.py", line 44, in send_email
return backend(to, subject, html_content, files=files, dryrun=dryrun, cc=cc, bcc=bcc, mime_subtype=mime_subtype)
File "/Users/b0204890/venv/python3/lib/python3.6/site-packages/airflow/utils/email.py", line 87, in send_email_smtp
send_MIME_email(SMTP_MAIL_FROM, recipients, msg, dryrun)
File "/Users/b0204890/venv/python3/lib/python3.6/site-packages/airflow/utils/email.py", line 107, in send_MIME_email
s = smtplib.SMTP_SSL(SMTP_HOST, SMTP_PORT) if SMTP_SSL else smtplib.SMTP(SMTP_HOST, SMTP_PORT)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/smtplib.py", line 251, in __init__
(code, msg) = self.connect(host, port)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/smtplib.py", line 336, in connect
self.sock = self._get_socket(host, port, self.timeout)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/smtplib.py", line 307, in _get_socket
self.source_address)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socket.py", line 724, in create_connection
raise err
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socket.py", line 713, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

Without following all of the instructions the connector will not work. Simply doing an installation for the Python package cx-Oracle is not enough. In this case it could also be that the venv of Airflow cannot access all neccessary files or that there is some rights issue.
As stated in the message, you need to follow all steps seen in: https://oracle.github.io/odpi/doc/installation.html#macos

Deploying Flower to Heroku

I'm following the instructions on https://github.com/jorilallo/celery-flower-heroku to deploy Flower celery monitoring app to Heroku.
After configuring and deploying my app I see the following in heroku logs:
Traceback (most recent call last):
File "/app/.heroku/python/bin/flower", line 9, in <module>
load_entry_point('flower==0.7.0', 'console_scripts', 'flower')()
File "/app/.heroku/python/lib/python2.7/site-packages/flower/__main__.py", line 11, in main
flower.execute_from_commandline()
File "/app/.heroku/python/lib/python2.7/site-packages/celery/bin/base.py", line 306, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "/app/.heroku/python/lib/python2.7/site-packages/flower/command.py", line 99, in handle_argv
return self.run_from_argv(prog_name, argv)
File "/app/.heroku/python/lib/python2.7/site-packages/flower/command.py", line 75, in run_from_argv
**app_settings)
File "/app/.heroku/python/lib/python2.7/site-packages/flower/app.py", line 40, in __init__
max_tasks_in_memory=max_tasks)
File "/app/.heroku/python/lib/python2.7/site-packages/flower/events.py", line 60, in __init__
state = shelve.open(self._db)
File "/app/.heroku/python/lib/python2.7/shelve.py", line 239, in open
return DbfilenameShelf(filename, flag, protocol, writeback)
File "/app/.heroku/python/lib/python2.7/shelve.py", line 223, in __init__
Shelf.__init__(self, anydbm.open(filename, flag), protocol, writeback)
File "/app/.heroku/python/lib/python2.7/anydbm.py", line 85, in open
return mod.open(file, flag, mode)
File "/app/.heroku/python/lib/python2.7/dumbdbm.py", line 250, in open
return _Database(file, mode)
File "/app/.heroku/python/lib/python2.7/dumbdbm.py", line 71, in __init__
f = _open(self._datfile, 'w')
IOError: [Errno 2] No such file or directory: 'postgres://USERNAME:PASSWORD#ec2-HOST.compute-1.amazonaws.com:5432/DBNAME.dat'
Notice the .dat appendix there? No idea where it comes from, its not present int my DATABASE_URL env variable.
Furthermore, the error above is with flower 0.7. I also tried installing 0.6, with which I do get further (namely the DB is correctly recognized and connection established), but I then get the following warnings once flower starts:
2014-06-19T15:14:02.464424+00:00 app[web.1]: [E 140619 15:14:02 state:138] Failed to inspect workers: '[Errno 104] Connection reset by peer', trying again in 128 seconds
2014-06-19T15:14:02.464844+00:00 app[web.1]: [E 140619 15:14:02 events:103] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 128 seconds.
Loading flower in my browser does show a few tabs of stuff, but there is no data.
How do I resolve these issues?

Flower doesn't support database persistence. It saves the state to file(s) using shelve module.