I'm building a Airflow server from the official docker image:
curl -LfO "https://airflow.apache.org/docs/apache-airflow/2.3.0/docker-compose.yaml"
It creates an standard user (airflow) with standard password (airflow). I have been trying to connect to my Airflow DAG with a PostgresHook, but it keeps returning this:
[2022-05-18, 01:29:32 UTC] {base.py:68} INFO - Using connection ID 'postgres_db' for task execution.
[2022-05-18, 01:29:32 UTC] {taskinstance.py:1889} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 171, in execute
return_value = self.execute_callable()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 189, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/opt/airflow/dags/etl_hover.py", line 35, in insert_data_to_table
pg_conn = hook.get_conn()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/postgres/hooks/postgres.py", line 113, in get_conn
self.conn = psycopg2.connect(**conn_args)
File "/home/airflow/.local/lib/python3.7/site-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server at "host.docker.internal" (192.168.65.2), port 5432 failed: FATAL: password authentication failed for user "***"
And, this is the connection:
As you can see, It is pretty much a standard connection (I have created succesfully an entry point with the database api_db, by the way. But It keeps telling me something about a wrong password. Any ideas?
Related
I am trying to connect to a database on Cloud SQL, but I keep getting the same error. Not sure what it is, and tried several approaches.
Input:
import pymysql
connection = pymysql.connect(host='127.0.0.1',
user='',
password='XXXX',
db='cmcsql')
output:
C:\Users\Ejer\anaconda3\envs\pythonProject\python.exe C:/Users/Ejer/PycharmProjects/pythonProject/CloudSQL_test.py
Traceback (most recent call last):
File "C:\Users\Ejer\anaconda3\envs\pythonProject\lib\site-packages\pymysql\connections.py", line 569, in connect
sock = socket.create_connection(
File "C:\Users\Ejer\anaconda3\envs\pythonProject\lib\socket.py", line 808, in create_connection
raise err
File "C:\Users\Ejer\anaconda3\envs\pythonProject\lib\socket.py", line 796, in create_connection
sock.connect(sa)
ConnectionRefusedError: [WinError 10061] Der kunne ikke oprettes forbindelse, fordi destinationscomputeren aktivt nægtede det
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/Ejer/PycharmProjects/pythonProject/CloudSQL_test.py", line 2, in <module>
connection = pymysql.connect(host='127.0.0.1',
File "C:\Users\Ejer\anaconda3\envs\pythonProject\lib\site-packages\pymysql\__init__.py", line 94, in Connect
return Connection(*args, **kwargs)
File "C:\Users\Ejer\anaconda3\envs\pythonProject\lib\site-packages\pymysql\connections.py", line 327, in __init__
self.connect()
File "C:\Users\Ejer\anaconda3\envs\pythonProject\lib\site-packages\pymysql\connections.py", line 619, in connect
raise exc
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on '127.0.0.1' ([WinError 10061] Der kunne ikke oprettes forbindelse, fordi destinationscomputeren aktivt nægtede det)")
Process finished with exit code 1
You should not be using 127.0.0.1 to connect to cloud sql instance.
127.0.0.1 is (in most cases) for localhost connection (when you run the db server locally on the same machine as where the client is). Instead you should be using the ip address given to your cloud sql instance. You can find it on the overview tab of the sql instance under Connect to this instance.
You should also create your user on the users tab and then use it in your code.
Don't forget about authentication, check out connections tab and read more about it here
Making sure that you have proper ip, user, existing db and connecting from authorized network should do the trick.
PyMysql connects to port 3306 by default, if your process is running on a different port it won't be able to connect.
Apart from the host, user, password parameters you also have to provide the port on which the sql process is running.
I am new to python and postgresql. This is the first time i was trying to connect python to postgresql.But i am getting this error:
`Traceback (most recent call last):
File "/home/vishal/Courses/database/app.py", line 5, in <module>
user.save_to_db()
File "/home/vishal/Courses/database/user.py", line 13, in save_to_db
connection = psycopg2.connect(user='postgres', password='1234', database='learning', host='localhost')
File "/home/vishal/Courses/database/venv/lib/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL: password authentication failed for user "postgres"
FATAL: password authentication failed for user "postgres"`
Even i tried changing the password and creating new user with different password the result is same.
Where am i going wrong?
airflow task is unable to connect to remote oracle database but i am able to connect to the same remote oracle database using the same code from python shell.
i checked running environment is same for both the shell and airflow.
i am attaching the log error which i am getting.
[2018-07-18 13:12:11,037] {models.py:1428} INFO - Executing <Task(AadharNumberValidation): validating_data> on 2018-07-18 07:42:05.573491
[2018-07-18 13:12:11,037] {base_task_runner.py:115} INFO - Running: ['bash', '-c', 'airflow run data_validation validating_data 2018-07-18T07:42:05.573491 --job_id 206 --raw -sd /Users/b0204890/Desktop/python/airflow_home/dags/data_validaton.py']
[2018-07-18 13:12:11,531] {base_task_runner.py:98} INFO - Subtask: [2018-07-18 13:12:11,531] {__init__.py:45} INFO - Using executor SequentialExecutor
[2018-07-18 13:12:11,588] {base_task_runner.py:98} INFO - Subtask: [2018-07-18 13:12:11,588] {models.py:189} INFO - Filling up the DagBag from /Users/b0204890/Desktop/python/airflow_home/dags/data_validaton.py
[2018-07-18 13:12:11,661] {cli.py:374} INFO - Running on host LTB0204890-Mac.local
[2018-07-18 13:12:11,669] {base_task_runner.py:98} INFO - Subtask: [2018-07-18 13:12:11,669] {validation_operators.py:37} INFO - operator_param yadav: Script one validation
[2018-07-18 13:12:11,678] {models.py:1595} ERROR - DPI-1047: 64-bit Oracle Client library cannot be loaded: "dlopen(libclntsh.dylib, 1): image not found". See https://oracle.github.io/odpi/doc/installation.html#macos for help
Traceback (most recent call last):
File "/Users/b0204890/venv/python3/lib/python3.6/site-packages/airflow/models.py", line 1493, in _run_raw_task
result = task_copy.execute(context=context)
File "/Users/b0204890/Desktop/python//airflow_home/plugins/validation_operators.py", line 38, in execute
cursor = create_connection(user="USERNAME",port="PORT",host="HOST",pwd="password",sid="SID")
File "/Users/b0204890/Desktop/python/airflow_home/utility/validation.py", line 30, in create_connection
connection = cx_Oracle.connect(user, pwd, service)
cx_Oracle.DatabaseError: DPI-1047: 64-bit Oracle Client library cannot be loaded: "dlopen(libclntsh.dylib, 1): image not found". See https://oracle.github.io/odpi/doc/installation.html#macos for help
[2018-07-18 13:12:11,681] {models.py:1616} INFO - Marking task as UP_FOR_RETRY
[2018-07-18 13:12:11,682] {base_task_runner.py:98} INFO - Subtask: [2018-07-18 13:12:11,682] {configuration.py:206} WARNING - section/key [smtp/smtp_user] not found in config
[2018-07-18 13:12:11,684] {models.py:1628} ERROR - Failed to send email to: ['tushar.smartcorp#gmail.com']
[2018-07-18 13:12:11,684] {models.py:1629} ERROR - [Errno 61] Connection refused
Traceback (most recent call last):
File "/Users/b0204890/venv/python3/lib/python3.6/site-packages/airflow/models.py", line 1493, in _run_raw_task
result = task_copy.execute(context=context)
File "/Users/b0204890/Desktop/python//airflow_home/plugins/validation_operators.py", line 38, in execute
cursor = create_connection(user="USERNAME",port="PORT",host="HOST",pwd="PASSWORD",sid="SID")
File "/Users/b0204890/Desktop/python/airflow_home/utility/validation.py", line 30, in create_connection
connection = cx_Oracle.connect(user, pwd, service)
cx_Oracle.DatabaseError: DPI-1047: 64-bit Oracle Client library cannot be loaded: "dlopen(libclntsh.dylib, 1): image not found". See https://oracle.github.io/odpi/doc/installation.html#macos for help
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/b0204890/venv/python3/lib/python3.6/site-packages/airflow/models.py", line 1618, in handle_failure
self.email_alert(error, is_retry=True)
File "/Users/b0204890/venv/python3/lib/python3.6/site-packages/airflow/models.py", line 1779, in email_alert
send_email(task.email, title, body)
File "/Users/b0204890/venv/python3/lib/python3.6/site-packages/airflow/utils/email.py", line 44, in send_email
return backend(to, subject, html_content, files=files, dryrun=dryrun, cc=cc, bcc=bcc, mime_subtype=mime_subtype)
File "/Users/b0204890/venv/python3/lib/python3.6/site-packages/airflow/utils/email.py", line 87, in send_email_smtp
send_MIME_email(SMTP_MAIL_FROM, recipients, msg, dryrun)
File "/Users/b0204890/venv/python3/lib/python3.6/site-packages/airflow/utils/email.py", line 107, in send_MIME_email
s = smtplib.SMTP_SSL(SMTP_HOST, SMTP_PORT) if SMTP_SSL else smtplib.SMTP(SMTP_HOST, SMTP_PORT)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/smtplib.py", line 251, in __init__
(code, msg) = self.connect(host, port)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/smtplib.py", line 336, in connect
self.sock = self._get_socket(host, port, self.timeout)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/smtplib.py", line 307, in _get_socket
self.source_address)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socket.py", line 724, in create_connection
raise err
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socket.py", line 713, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
Without following all of the instructions the connector will not work. Simply doing an installation for the Python package cx-Oracle is not enough. In this case it could also be that the venv of Airflow cannot access all neccessary files or that there is some rights issue.
As stated in the message, you need to follow all steps seen in: https://oracle.github.io/odpi/doc/installation.html#macos
I'm trying to host a local MySQL database using ClearDB on Heroku. The build is successful, but when I try to access the application, the application fails.
Here is my app.py code (it fails on the first line):
db = MySQLdb.connect(host="localhost", user="root", db="database")
cur = db.cursor()
query = """SELECT * from database"""
cur.execute(query)
data = cur.fetchall()
Here is the log output:
2016-06-01T14:48:48.720931+00:00 heroku[web.1]: Starting process with command `python app.py`
2016-06-01T14:48:51.790883+00:00 app[web.1]: File "app.py", line 30, in <module>
2016-06-01T14:48:51.790864+00:00 app[web.1]: Traceback (most recent call last):
2016-06-01T14:48:51.790910+00:00 app[web.1]: db = MySQLdb.connect(host="localhost", user="root", db="database")
2016-06-01T14:48:51.790981+00:00 app[web.1]: _mysql_exceptions.OperationalError: (2002, "Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)")
2016-06-01T14:48:51.790911+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/MySQLdb/__init__.py", line 81, in Connect
2016-06-01T14:48:51.790912+00:00 app[web.1]: return Connection(*args, **kwargs)
2016-06-01T14:48:51.790912+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/MySQLdb/connections.py", line 193, in __init__
2016-06-01T14:48:51.790937+00:00 app[web.1]: super(Connection, self).__init__(*args, **kwargs2)
2016-06-01T14:48:52.491629+00:00 heroku[web.1]: Process exited with status 1
Here is what heroku config prints:
CLEARDB_DATABASE_URL: mysql://actualurl
DATABASE_URL: mysql://actualurl
MYSQL_DB: database
MYSQL_HOST: localhost
MYSQL_USER: root
There's no point in posting your Heroku config, because it's clear that your app isn't using it. You have hardcoded the database host as "localhost", rather than using the DATABASE_URL environment variable.
I've read everything possible and I just can't connect to my mysql server. Here is my code. very simple. (I used xxx to hide all the private info. everything else is the way i got it.
import mysql.connector
conn = mysql.connector.connect(user='xxxx',password='xxxxx',host='xxxx',db='xxx',port=3306)
The login info is definitely correct. I tried it in a mysql client and it connected just fine. The MySQL server is hosted on dreamhost. I set the allowable IPs for the db user to wildcard %.%.%.% so anyone could connect.I don't know what else could possibly be wrong. When I try to connect in python i get:
C:\Python33\python.exe D:/Dropbox/python/v2/test.py
Traceback (most recent call last):
File "D:/Dropbox/python/v2/test.py", line 3, in <module>
conn = mysql.connector.connect(user='xxx',password='xxx',host='xxx',db='xxx',port=3306)
File "C:\Python33\lib\site-packages\mysql\connector\__init__.py", line 101, in connect
return MySQLConnection(*args, **kwargs)
File "C:\Python33\lib\site-packages\mysql\connector\connection.py", line 117, in __init__
self.connect(**kwargs)
File "C:\Python33\lib\site-packages\mysql\connector\connection.py", line 383, in connect
self._open_connection()
File "C:\Python33\lib\site-packages\mysql\connector\connection.py", line 350, in _open_connection
self._ssl)
File "C:\Python33\lib\site-packages\mysql\connector\connection.py", line 176, in _do_auth
raise errors.get_exception(packet)
mysql.connector.errors.ProgrammingError: 1044 (42000): Access denied for user 'xxx'#'%.%.%.%' to database 'xxx'
Any help would be greatly appreciated.
You will have to GRANT permissions to the user.
Something like:
GRANT ALL PRIVILEGES ON *.* TO 'user2'#'localhost' IDENTIFIED BY PASSWORD 'xxxxx'