I am trying to connect to a local instance of MongoDB in Python, using the following connection. The version (commented out line) with connection to cloud works perfectly, however the localhost line with 27017 fails to get the list of attributes from database. What could the problem be?
#mongo_url = "mongodb+srv://admin:mypassword#cluster0.l2vfs.mongodb.net/MYDB?retryWrites=true&w=majority" #WORKS
mongo_url = "mongodb://admin:mypassword#localhost:27017/MYDB?retryWrites=true&w=majority" #FAILS
client = MongoClient(mongo_url)
database = client["MYDB"]
date_price_collection = database["historical_prices"]
def gettickers(attribute_name):
try:
cursor = date_price_collection.find({},{attribute_name: 1})
list_of_tickers = []
for document in cursor:
list_of_tickers.append(document[attribute_name])
return list_of_tickers
except Exception as e:
print("Error: could not retrieve from Mongo")
return ""
print(gettickers("ticker"))
Grateful for any help
Related
I'm writing some Python code to connect to our Snowflake Database and getting an error in my Python (I'm new to Python so i've likely done something wrong here).
I'm getting <class 'NameError'> TestConnection.py 98 inside my script, which has the relevant parts below
def connection(obj):
try:
global cur, engine
cur = obj.cursor_connection().cursor()
engine = obj.engine_connection().engine.connect()
except Exception as e:
print(e)
sql = "select current_warehouse(), current_database(), current_schema();"
try:
print("Cursor connection successful")
except Exception as e:
print(e)
try:
print("Engine connection successful")
except Exception as e:
print(e)
return cur, engine
try:
#Setup Connection with both cursor and the engine
db, schema= login.db, login.schema
obj = connect(db, schema)
cur , engine = connection(obj)
The line I'm getting the error on is the cur,engine = connection(obj) part.
I had a previous error before (UnboundLocalError) but putting global cur, engine inside the connection function fixed that, but getting NameError now.
What am I doing wrong here?
Thanks
try this
import snowflake.connector
ctx = snowflake.connector.connect(
user='<user_name>',
password='<password>',
account='<account_identifier>'
)
cs = ctx.cursor()
Ok, I've fixed this now, it was indentation on the functions and class.
I ended up with
class connect:
def __init__
def cursor_connection
def engine_connection
def connection
and that has it working whereas having def_connection tabbed in was breaking it.
I am developing a web app using Angular 9 for frontend and Flask in python 3.8 for backend, everything was going well till i tried to connect the backend with the data base, where the code apparently its ok, flask start running succesfully but when i tried to use my endpoint for authentication, flask throws the error:
**local variable 'conn' referenced before assignment**
i have been checking some forums and so on but i dont understand what is going on. Thanks in advance for your help.
import pymysql
database_name = "dbx80"
database_user = "user_auth"
database_password = "Jan2019#"
def conection_database(db):
try:
conn = (pymysql.connect(
unix_socket="/cloudsql/servicisEX:us-central1:dbx43",
port = 3306,
user = database_user,
password = database_password,
database = database_name,
charset="utf8"
))
print(f"Connection {database_name} Succesful")
except Exception as e:
print(f"Error connecting to {database_name}")
return conn
Converting comment to an answer.
return conn is called no matter the result of the try except block.
If you put the return statement inside the try block, the error will not occur.
Example:
import pymysql
database_name = "dbx80"
database_user = "user_auth"
database_password = "Jan2019#"
def conection_database(db):
try:
conn = (pymysql.connect(
unix_socket="/cloudsql/servicisEX:us-central1:dbx43",
port = 3306,
user = database_user,
password = database_password,
database = database_name,
charset="utf8"
))
print(f"Connection {database_name} Succesful")
return conn
except Exception as e:
print(f"Error connecting to {database_name}")
I want to get the column names in redshift using python boto3
Creaed Redshift Cluster
Insert Data into it
Configured Secrets Manager
Configure SageMaker Notebook
Open the Jupyter Notebook wrote the below code
import boto3
import time
client = boto3.client('redshift-data')
response = client.execute_statement(ClusterIdentifier = "test", Database= "dev", SecretArn= "{SECRET-ARN}",Sql= "SELECT `COLUMN_NAME` FROM `INFORMATION_SCHEMA`.`COLUMNS` WHERE `TABLE_SCHEMA`='dev' AND `TABLE_NAME`='dojoredshift'")
I got the response but there is no table schema inside it
Below is the code i used to connect I am getting timed out
import psycopg2
HOST = 'xx.xx.xx.xx'
PORT = 5439
USER = 'aswuser'
PASSWORD = 'Password1!'
DATABASE = 'dev'
def db_connection():
conn = psycopg2.connect(host=HOST,port=PORT,user=USER,password=PASSWORD,database=DATABASE)
return conn
How to get the ip address go to https://ipinfo.info/html/ip_checker.php
pass your hostname of redshiftcluster xx.xx.us-east-1.redshift.amazonaws.com or you can see in cluster page itself
I got the error while running above code
OperationalError: could not connect to server: Connection timed out
Is the server running on host "x.xx.xx..xx" and accepting
TCP/IP connections on port 5439?
I fixed with the code, and add the above the rules
import boto3
import psycopg2
# Credentials can be set using different methodologies. For this test,
# I ran from my local machine which I used cli command "aws configure"
# to set my Access key and secret access key
client = boto3.client(service_name='redshift',
region_name='us-east-1')
#
#Using boto3 to get the Database password instead of hardcoding it in the code
#
cluster_creds = client.get_cluster_credentials(
DbUser='awsuser',
DbName='dev',
ClusterIdentifier='redshift-cluster-1',
AutoCreate=False)
try:
# Database connection below that uses the DbPassword that boto3 returned
conn = psycopg2.connect(
host = 'redshift-cluster-1.cvlywrhztirh.us-east-1.redshift.amazonaws.com',
port = '5439',
user = cluster_creds['DbUser'],
password = cluster_creds['DbPassword'],
database = 'dev'
)
# Verifies that the connection worked
cursor = conn.cursor()
cursor.execute("SELECT VERSION()")
results = cursor.fetchone()
ver = results[0]
if (ver is None):
print("Could not find version")
else:
print("The version is " + ver)
except:
logger.exception('Failed to open database connection.')
print("Failed")
I am currently using AWS Lambda (Python 3.6) to talk to a MySQL database. I also have Slack commands triggering the queries to the database. On occasion, I have noticed that I can change things directly through MySQL Workbench and then trigger a query through Slack which returns old values. I currently connect to MySQL outside of the python handler like this:
BOT_TOKEN = os.environ["BOT_TOKEN"]
ASSET_TABLE = os.environ["ASSET_TABLE"]
REGION_NAME = os.getenv('REGION_NAME', 'us-east-2')
DB_NAME = os.environ["DB_NAME"]
DB_PASSWORD = os.environ["DB_PASSWORD"]
DB_DATABASE = os.environ["DB_DATABASE"]
RDS_HOST = os.environ["RDS_HOST"]
port = os.environ["port"]
try:
conn = pymysql.connect(RDS_HOST, user=DB_NAME, passwd=DB_PASSWORD, db=DB_DATABASE, connect_timeout=5, cursorclass=pymysql.cursors.DictCursor)
cursor = conn.cursor()
except:
sys.exit()
The MySQL connection is done outside of any definition at the very top of my program. When Slack sends a command, I call another definition that then queries MySQL. This works okay sometimes, but other times can send my old data that has not updated. The whole layout is like this:
imports
SQL connections
SQL query definitions
handler definition
I tried moving the MySQL connection portion inside of the handler, but then the SQL query definitions do not recognize my cursor (out of scope, I guess).
So my question is, how do I handle this MySQL connection? Is it best to keep the MySQL connection outside of any definitions? Should I open and close the connection each time? Why is my data stale? Will Lambda ALWAYS run the entire routine or can it try to split the load between servers (I swear I read somewhere that I cannot rely on Lambda to always read my entire routine; sometimes it just reads the handler)?
I'm pretty new to all this, so any suggestions are much appreciated. Thanks!
Rest of the code if it helps:
################################################################################################################################################################################################################
# Slack Lambda handler.
################################################################################################################################################################################################################
################################################################################################################################################################################################################
# IMPORTS
###############
import sys
import os
import pymysql
import urllib
import math
################################################################################################################################################################################################################
################################################################################################################################################################################################################
# Grab data from AWS environment.
###############
BOT_TOKEN = os.environ["BOT_TOKEN"]
ASSET_TABLE = os.environ["ASSET_TABLE"]
REGION_NAME = os.getenv('REGION_NAME', 'us-east-2')
DB_NAME = os.environ["DB_NAME"]
DB_PASSWORD = os.environ["DB_PASSWORD"]
DB_DATABASE = os.environ["DB_DATABASE"]
RDS_HOST = os.environ["RDS_HOST"]
port = os.environ["port"]
################################################################################################################################################################################################################
################################################################################################################################################################################################################
# Attempt SQL connection.
###############
try:
conn = pymysql.connect(RDS_HOST, user=DB_NAME, passwd=DB_PASSWORD, db=DB_DATABASE, connect_timeout=5, cursorclass=pymysql.cursors.DictCursor)
cursor = conn.cursor()
except:
sys.exit()
################################################################################################################################################################################################################
# Define the URL of the targeted Slack API resource.
SLACK_URL = "https://slack.com/api/chat.postMessage"
################################################################################################################################################################################################################
# Function Definitions.
###############
def get_userExistance(user):
statement = f"SELECT 1 FROM slackDB.users WHERE userID LIKE '%{user}%' LIMIT 1"
cursor.execute(statement, args=None)
userExists = cursor.fetchone()
return userExists
def set_User(user):
statement = f"INSERT INTO `slackDB`.`users` (`userID`) VALUES ('{user}');"
cursor.execute(statement, args=None)
conn.commit()
return
################################################################################################################################################################################################################
################################################################################################################################################################################################################
# Slack command interactions.
###############
def lambda_handler(data, context):
# Slack challenge answer.
if "challenge" in data:
return data["challenge"]
# Grab the Slack channel data.
slack_event = data['event']
slack_userID = slack_event['user']
slack_text = slack_event['text']
channel_id = slack_event['channel']
slack_reply = ""
# Check sql connection.
try:
conn = pymysql.connect(RDS_HOST, user=DB_NAME, passwd=DB_PASSWORD, db=DB_DATABASE, connect_timeout=5, cursorclass=pymysql.cursors.DictCursor)
cursor = conn.cursor()
except pymysql.OperationalError:
connected = 0
else:
connected = 1
# Ignore bot messages.
if "bot_id" in slack_event:
slack_reply = ""
else:
# Start data sift.
if slack_text.startswith("!addme"):
if get_userExistance(slack_userID):
slack_reply = f"User {slack_userID} already exists"
else:
slack_reply = f"Adding user {slack_userID}"
set_user(slack_userID)
# We need to send back three pieces of information:
data = urllib.parse.urlencode(
(
("token", BOT_TOKEN),
("channel", channel_id),
("text", slack_reply)
)
)
data = data.encode("ascii")
# Construct the HTTP request that will be sent to the Slack API.
request = urllib.request.Request(
SLACK_URL,
data=data,
method="POST"
)
# Add a header mentioning that the text is URL-encoded.
request.add_header(
"Content-Type",
"application/x-www-form-urlencoded"
)
# Fire off the request!
urllib.request.urlopen(request).read()
# Everything went fine.
return "200 OK"
################################################################################################################################################################################################################
All of the code outside the lambda handler is only run once per container. All code inside the handler is run every time the lambda is invoked.
A lambda container lasts for between 10 and 30 minutes depending on usage. A new lambda invocation may or may not run on an already running container.
It's possible you are invoking a lambda in a container that is over 5 minutes old where your connection has timed out.
BRIEF DESCRIPTION OF THE PROBLEM: Psycopg2 won't connect to a test DB within docker from unittest, but connects fine from console.
ERROR MESSAGE:
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally before or while processing the request.
DETAILS:
I'm trying to set up a testing database in docker, that will be created and populated before testing and then removed after.
Here's the fuction to set up database:
def set_up_empty_test_db():
client = docker.from_env()
try:
testdb = client.containers.get("TestDB")
testdb.stop()
testdb.remove()
testdb = client.containers.run(
"postgres",
ports={5432: 5433},
detach=True,
name="TestDB",
environment=["POSTGRES_PASSWORD=yourPassword"],
)
except NotFound:
testdb = client.containers.run(
"postgres",
ports={5432: 5433},
detach=True,
name="TestDB",
environment=["POSTGRES_PASSWORD=yourPassword"],
)
while testdb.status != "running":
testdb = client.containers.get("TestDB")
return
If I launch this function from console it works without an error and I can see TestDB container running. I can successfully initiate connection:
conn = psycopg2.connect("host='localhost' user='postgres' password='yourPassword' port='5433'")
But it doesn't work when unit testing. Here's the testing code:
class TestCreateCity(unittest.TestCase):
def setUp(self):
set_up_empty_test_db()
con = psycopg2.connect("host='localhost' user='postgres' password='yourPassword' port='5433'")
self.assertIsNone(con.isolation_level)
cur = con.cursor()
sql_file = open(os.path.join(ROOT_DIR + "/ddl/creates/schema_y.sql"), "r")
cur.execute(sql_file.readline())
con.commit()
con.close()
self.session = Session(creator=con)
def test_create_city(self):
cs.create_city("Test_CITY", "US")
q = self.session.query(City).filter(City.city_name == "Test_CITY").one()
self.assertIs(q)
self.assertEqual(q.city_name, "Test_CITY")
self.assertEqual(q.country_code, "US")
It breaks when trying to initiate connection. Please advise.
I know this is an old question, but I needed to do the same thing today. You try and connect to the postgres server too quickly after starting it - that's why it works in the console.
All you need to do is replace:
set_up_empty_test_db()
con = psycopg2.connect("host='localhost' user='postgres' password='yourPassword' port='5433'")
with:
set_up_empty_test_db()
con = None
while con == None:
try:
con = psycopg2.connect("host='localhost' user='postgres' password='yourPassword' port='5433'")
except psycopg2.OperationalError:
time.sleep(0.5);
Hope this helps someone else. Cheers!