Develop Reference

Connection problem with PostgreSQL and Airflow (Built via Docker-compose)

I'm building a Airflow server from the official docker image:
curl -LfO "https://airflow.apache.org/docs/apache-airflow/2.3.0/docker-compose.yaml"
It creates an standard user (airflow) with standard password (airflow). I have been trying to connect to my Airflow DAG with a PostgresHook, but it keeps returning this:
[2022-05-18, 01:29:32 UTC] {base.py:68} INFO - Using connection ID 'postgres_db' for task execution.
[2022-05-18, 01:29:32 UTC] {taskinstance.py:1889} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 171, in execute
return_value = self.execute_callable()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 189, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/opt/airflow/dags/etl_hover.py", line 35, in insert_data_to_table
pg_conn = hook.get_conn()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/postgres/hooks/postgres.py", line 113, in get_conn
self.conn = psycopg2.connect(**conn_args)
File "/home/airflow/.local/lib/python3.7/site-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server at "host.docker.internal" (192.168.65.2), port 5432 failed: FATAL: password authentication failed for user "***"
And, this is the connection:
As you can see, It is pretty much a standard connection (I have created succesfully an entry point with the database api_db, by the way. But It keeps telling me something about a wrong password. Any ideas?

sqlite3.OperationalError) no such table on Heroku

I'm working with https://github.com/hack4impact/flask-base (a flask boilerplate) as a basis for a project. The project runs as expected locally on windows.
I don't need to make any db changes on the production version of my code, so for simplicity's sake, I've decided to use sqllite both locally and on heroku. With that in mind I changed the production class in https://github.com/hack4impact/flask-base/blob/master/config.py to :
class ProductionConfig(Config):
# SQLALCHEMY_DATABASE_URI = os.environ.get('DATABASE_URL') or 'sqlite:///' + os.path.join(basedir, 'data.sqlite')
SQLALCHEMY_DATABASE_URI = 'sqlite:///' + os.path.join(basedir, 'data-dev.sqlite')
print('IN PRODUCTION '+ SQLALCHEMY_DATABASE_URI)
SSL_DISABLE = (os.environ.get('SSL_DISABLE') or 'True') == 'True'
When I deploy the code and try to login I see a 500 error. The logs show:
2019-05-15T21:47:28.749283+00:00 app[web.1]: raise value.with_traceback(tb)
2019-05-15T21:47:28.749284+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
2019-05-15T21:47:28.749286+00:00 app[web.1]: context)
2019-05-15T21:47:28.749287+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
2019-05-15T21:47:28.749289+00:00 app[web.1]: cursor.execute(statement, parameters)
2019-05-15T21:47:28.749301+00:00 app[web.1]: sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: users [SQL: 'SELECT users.id AS users_id, users.confirmed AS users_confirmed, users.first_name AS users_first_name, users.last_name AS users_last_name, users.email AS users_email, users.password_hash AS users_password_hash, users.role_id AS users_role_id \nFROM users \nWHERE users.email = ?\n LIMIT ? OFFSET ?'] [parameters: ('me#test.com', 1, 0)]
Disconnected from log stream. There may be events happening that you do not see here! Attempting to reconnect...
2019-05-15T21:47:28.749277+00:00 app[web.1]: exc_info
2019-05-15T21:47:28.749278+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2019-05-15T21:47:28.749280+00:00 app[web.1]: reraise(type(exception), exception, tb=exc_tb, cause=cause)
2019-05-15T21:47:28.749281+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
2019-05-15T21:47:28.749283+00:00 app[web.1]: raise value.with_traceback(tb)
2019-05-15T21:47:28.749284+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
2019-05-15T21:47:28.749286+00:00 app[web.1]: context)
2019-05-15T21:47:28.749287+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
2019-05-15T21:47:28.749289+00:00 app[web.1]: cursor.execute(statement, parameters)
2019-05-15T21:47:28.749301+00:00 app[web.1]: sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: users [SQL: 'SELECT users.id AS users_id, users.confirmed AS users_confirmed, users.first_name AS users_first_name, users.last_name AS users_last_name, users.email AS users_email, users.password_hash AS users_password_hash, users.role_id AS users_role_id \nFROM users \nWHERE users.email = ?\n LIMIT ? OFFSET ?'] [parameters: ('me#test.com', 1, 0)]
So it appears sqlalchemy can't find the table. I'm wondering if the line:
SQLALCHEMY_DATABASE_URI = 'sqlite:///' + os.path.join(basedir, 'data-dev.sqlite')
is then wrong way to access the flatfile db (data-dev.sqlite)

You really shouldn't use SQLite on Heroku. Its filesystem is ephemeral. Any changes you make to files will be lost the next time your dyno restarts. This happens frequently (at least once per day).
The original code did the right thing:
SQLALCHEMY_DATABASE_URI = os.environ.get('DATABASE_URL') or 'sqlite:///' + os.path.join(basedir, 'data.sqlite')
If the DATABASE_URL environment variable is set it will be used to connect to your database. Since psycopg2 is in your requirements.txt file Heroku should automatically provision a PostgreSQL database and set the DATABASE_URL variable to its connection string.
If you really must use SQLite (and I strongly advise against that), treat it as read-only. You'll have to commit your database file on your local development machine and push that commit to Heroku. Even then it might not work properly. Heroku famously generates errors if Ruby users even try to install the sqlite3 gem.

pymysql.err.InternalError: (1049, "Unknown database")

I want to connect MySQL RDS DB using python from raspberrypi.(i want to get seq from MySQL table 'face' using select query.)
and I have an error but i can not fix it.
This is rds mysql connection code:
import rds_config
import pymysql
rds_host = rds_config.rds_host
name = rds_config.rds_user
password = rds_config.rds_pwd
db_name = rds_config.rds_db
conn = pymysql.connect(rds_host, user=name, passwd=password, db=db_name,
connect_timeout=10)
with conn.cursor() as cur:
cur.execute("select seq from face")
conn.commit()
rds_config:
rds_host='rds endpoint'
rds_port=3306
rds_user='user'
rds_pwd='password'
rds_db='db name'
and This is traceback:
Traceback (most recent call last):
File "getRds.py", line 18, in <module>
conn = pymysql.connect(rds_host, user=name, passwd=password, db=db_name, connect_timeout=10)
File "/usr/local/lib/python2.7/dist-packages/pymysql/__init__.py", line 94, in Connect
return Connection(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 327, in __init__
self.connect()
File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 598, in connect
self._request_authentication()
File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 862, in _request_authentication
auth_packet = self._process_auth(plugin_name, auth_packet)
File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 933, in _process_auth
pkt = self._read_packet()
File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 683, in _read_packet
packet.check_error()
File "/usr/local/lib/python2.7/dist-packages/pymysql/protocol.py", line 220, in check_error
err.raise_mysql_exception(self._data)
File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 109, in raise_mysql_exception
raise errorclass(errno, errval)
pymysql.err.InternalError: (1049, u"Unknown database 'bsb-rds'")
i alread added ip address in vpc security group and public access is on.
it was possible to connect through mysql cli or workbench.
can anyone help me?

tl;dr you need to create bsb-rd. Execute this command: create database bsb-rds either with cur.execute() in python or in your favorite sql cli.
If you get this error, good news! You can connect to your RDS instance. This means you have the security group set up right, the host url, username, password and port are all correct! What this error is telling you is that your database bsb-rds does not exist on your RDS instance. If you have just made the RDS instance it is probably because you have not created the database yet. So create it!
Here are two ways to do this
mysql cli
In your terminal run
mysql --host=the_host_address user=your_user_name --password=your_password
Then inside the mysql shell execute
create database bsb-rd;
Now try your code again!
Python with pymysql
import rds_config
conn = pymysql.connect('hostname', user='username', passwd='password', connect_timeout=10)
with conn.cursor() as cur:
cur.execute('create database bsb-rd;')
I came to this page looking for a solution and didn't get it. I eventually found the answer and now share what helped me.

I ran into your exact issue and I believe I have the solution:
Context:
My tech stack looks like the following
Using AWS RDS (MySQL)
Using Flask Development on local host
RDS instance name: "test-rds"
Actual DB Name: test-database
Here is where my issue was and I believe your issue is in the same place:
You are using the AWS RDS NAME in your connection rather than using the true Database name.
Simply changing the DB name in my connection to the true DB name that I had setup via MySQL Workbench fixed the issue.
Other things things to note for readers:
Please ensure the following:
If you are connecting from outside your AWS VPC make sure you have public access enabled. This is a huge "gotcha". (Beware security risks)
Make sure your connection isn't being blocked by a NACL
Make sure your connection is allowed by a Security Group Rule

PyMySQL keeps connection error

I've tried a bunch of thinks including trying to specify the UNIX socket to no avail, I'm not running any queries and I haven't even initialized a cursor but I keep getting this error, what gives?
Python Block:
connection = mysql.connect(user = "root", password = None, port = 8080, host = 'localhost', db ='Articles')
Error:
Traceback (most recent call last):
File "/Users/Adrian/Desktop/Python/webcrawl.py", line 10, in <module>
connection = mysql.connect(user = "root", password = None, port = 8080, host = 'localhost', db ='Articles')
File "/Users/Adrian/anaconda3/lib/python3.6/site-packages/pymysql/__init__.py", line 90, in Connect
return Connection(*args, **kwargs)
File "/Users/Adrian/anaconda3/lib/python3.6/site-packages/pymysql/connections.py", line 699, in __init__
self.connect()
File "/Users/Adrian/anaconda3/lib/python3.6/site-packages/pymysql/connections.py", line 935, in connect
self._get_server_information()
File "/Users/Adrian/anaconda3/lib/python3.6/site-packages/pymysql/connections.py", line 1249, in _get_server_information
packet = self._read_packet()
File "/Users/Adrian/anaconda3/lib/python3.6/site-packages/pymysql/connections.py", line 991, in _read_packet
packet_header = self._read_bytes(4)
File "/Users/Adrian/anaconda3/lib/python3.6/site-packages/pymysql/connections.py", line 1037, in _read_bytes
CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')
Edit: I'm running XAMPP 7.2 on MacOSX with port forwarding enabled over SSH (localhost:8080 -> 80) and the opt/lampp volumes are mounted

It's not advisable to use root without password.
In some databases you are forced to set it if you want to connect as root.
You can set any password: dev.mysql.com/doc/refman/5.7/en/resetting-permissions.html
Also, 8080 might be the port of your web server, not the mySQL one, try with 3306 port which is the default one for mySQL.
You can open my.cnf file located in the /Applications/XAMPP/xamppfiles/etc/ directory to chech in which port is your database listening.
Also check that the mySQL daemon is running. In mac os X go to /Applications/XAMPP/XAMPP Control in Finder and check that Apache and MySQL are running.
If your MySQL server isn't starting, you may need to set the permissions for it using Terminal with this command:
chmod -R 777 /Applications/XAMPP/xamppfiles/var
To check that your Articles database exists:
mysql -u root -p
USE Articles;
If it's not created;
mysql -u root -p
CREATE DATABASE Articles;
And connect this way:
#!/usr/bin/python
import MySQLdb
connection = MySQLdb.connect(host="localhost",
user="root",
passwd="your pwd",
db="Articles")

Peewee - Can't connect to MySQL server on host

I'm developing a Flask based python app using the peewee ORM. I was initially connecting to the database that was being stored locally on my machine and I'm now trying to transition to connecting to the db remotely. I've set up the database in phpmyadmin via my server's cpanel section.
The Issue
I've set up my IP address to be able to remotely access my databases but I am getting the following error when I attempt to connect to the database:
Traceback (most recent call last):
File "app.py", line 294, in <module>
models.initialize()
File "/Users/wyssuser/Desktop/dscraper/models.py", line 145, in initialize
DATABASE.connect()
File "/Library/Python/2.7/site-packages/peewee.py", line 2767, in connect
self.__local.closed = False
File "/Library/Python/2.7/site-packages/peewee.py", line 2688, in __exit__
reraise(new_type, new_type(*exc_value.args), traceback)
File "/Library/Python/2.7/site-packages/peewee.py", line 2766, in connect
**self.connect_kwargs)
File "/Library/Python/2.7/site-packages/peewee.py", line 3209, in _connect
return mysql.connect(db=database, **conn_kwargs)
File "/Library/Python/2.7/site-packages/pymysql/__init__.py", line 88, in Connect
return Connection(*args, **kwargs)
File "/Library/Python/2.7/site-packages/pymysql/connections.py", line 644, in __init__
self._connect()
File "/Library/Python/2.7/site-packages/pymysql/connections.py", line 869, in _connect
raise exc
peewee.OperationalError: (2003, "Can't connect to MySQL server on '142.157.25.22' ([Errno 61] Connection refused)")
This is the portion of my code that references the database connection:
app.py
if __name__ == '__main__':
models.initialize()
app.run(debug=DEBUG, port=PORT, host=HOST)
config.py
DATABASE = {
'db': 'my_dbname',
'host': '142.157.25.22',
'port': 3306,
'user': 'my_username',
'passwd': 'my_pswd',
}
models.py
from peewee import *
import config
DATABASE = MySQLDatabase(config.DATABASE['db'], host=config.DATABASE['host'], port=config.DATABASE['port'], user=config.DATABASE['user'], passwd=config.DATABASE['passwd'])
...all of my models related code
def initialize():
print 'starting db connection'
DATABASE.connect()
print 'connected'
DATABASE.create_tables([Batch, Company, User, Post],safe=True)
DATABASE.close()
I've also tried connecting to 'localhost' as the host but that doesn't seem to work here, is there a different host I should be connecting to?

Solution is bad default port:
the example from peewee doc is
# Connect to a MySQL database on network.
mysql_db = MySQLDatabase('my_app', user='app', password='db_password',
host='10.1.0.8', port=3316)
but defaul port is 3306