I have Python script that I am trying to use to execute this function below on my SQL Server
DBCC CHECKIDENT('TableName', RESEED, 0)
My script looks like this:
qry = '''DBCC CHECKIDENT('TableName', RESEED, 0)'''
def mssql_cmd(qry, env):
# Import Dependencies
import pyodbc
import sqlalchemy as sa
import urllib
import pandas as pd
import sqlalchemy
import json
try:
# Read config json file into config dict
with open("../parameters/config.json") as cf:
config = json.load(cf)
# Try to establish the connection to MSSQL
params = urllib.parse.quote_plus(f'DRIVER={config[env][0]["driver"]};'
f'Server={config[env][0]["server"]};'
f'Database={config[env][0]["database"]};'
f'User={config[env][0]["user"]};'
f'Password={config[env][0]["password"]};'
f'Trusted_connection={config[env][0]["Trusted_connection"]};')
# Establish the engine
engine = sa.create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
db = engine.connect()
print("Connection to Data Warehouse -- SUCCESSFUL")
if db.connect():
try:
db.execute(qry)
db.close()
engine.dispose()
except Exception as e:
print(e)
except Exception as e:
print(e)
I don't get any errors the scrips executes but it doesn't reset my autogen Id on the table.
If I replace the line
db.execute(qry)
with
data = pd.read_sql(sql_qry, db)
then I am able to extract the data.
So the script works if I run the query however I can't make it to run the function to reset my auto gen id.
Does anyone have any clue as to what I am doing wrong here?
I have a python application that is reading from mysql/mariadb, uses that to fetch data from an api and then inserts results into another table.
I had setup a module with a function to connect to the database and return the connection object that is passed to other functions/modules. However, I believe this might not be a correct approach. The idea was to have a small module that I could just call whenever I needed to connect to the db.
Also note, that I am using the same connection object during loops (and within the loop passing to the db_update module) and call close() when all is done.
I am also getting some warnings from the db sometimes, those mostly happen at the point where I call db_conn.close(), so I guess I am not handling the connection or session/engine correctly. Also, the connection id's in the log warning keep increasing, so that is another hint, that I am doing it wrong.
[Warning] Aborted connection 351 to db: 'some_db' user: 'some_user' host: '172.28.0.3' (Got an error reading communication packets)
Here is some pseudo code that represents the structure I currently have:
################
## db_connect.py
################
# imports ...
from sqlalchemy import create_engine
def db_connect():
# get env ...
db_string = f"mysql+pymysql://{db_user}:{db_pass}#{db_host}:{db_port}/{db_name}"
try:
engine = create_engine(db_string)
except Exception as e:
return None
db_conn = engine.connect()
return db_conn
################
## db_update.py
################
# imports ...
def db_insert(db_conn, api_result):
# ...
ins_qry = "INSERT INTO target_table (attr_a, attr_b) VALUES (:a, :b);"
ins_qry = text(ins_qry)
ins_qry = ins_qry.bindparams(a = value_a, b = value_b)
try:
db_conn.execute(ins_qry)
except Exception as e:
print(e)
return None
return True
################
## main.py
################
from sqlalchemy import text
from db_connect import db_connect
from db_update import db_insert
def run():
try:
db_conn = db_connect()
if not db_conn:
return False
except Exception as e:
print(e)
qry = "SELECT *
FROM some_table
WHERE some_attr IN (:some_value);"
qry = text(qry)
search_run_qry = qry.bindparams(
some_value = 'abc'
)
result_list = db_conn.execute(qry).fetchall()
for result_item in result_list:
## do stuff like fetching data from api for every record in the query result
api_result = get_api_data(...)
## insert into db:
db_ins_status = db_insert(db_conn, api_result)
## ...
db_conn.close
run()
EDIT: Another question:
a) Is it ok in a loop, that does an update on every iteration to use the same connection, or would it be wiser to instead pass the engine to the run() function and call db_conn = engine.connect() and db_conn.close() just before and after each update?
b) I am thinking about using ThreadPoolExecutor instead of the loop for the API calls. Would this have implications on how to use the connection, i.e. can I use the same connection for multiple threads that are doing updates to the same table?
Note: I am not using the ORM feature mostly because I have a strong DWH/SQL background (though not so much as DBA) and I am used to writing even complex sql queries. I am thinking about switching to just using PyMySQL connector for that reason.
Thanks in advance!
Yes you can return/pass connection object as parameter but what is the aim of db_connect method, except testing connection ? As I see there is no aim of this db_connect method therefore I would recommend you to do this as I done it before.
I would like to share a code snippet from one of my project.
def create_record(sql_query: str, data: tuple):
try:
connection = mysql_obj.connect()
db_cursor = connection.cursor()
db_cursor.execute(sql_query, data)
connection.commit()
return db_cursor, connection
except Exception as error:
print(f'Connection failed error message: {error}')
and then using this one as for another my need
db_cursor, connection, query_data = fetch_data(sql_query, query_data)
and after all my needs close the connection with this method and method call.
def close_connection(connection, db_cursor):
"""
This method used to close SQL server connection
"""
db_cursor.close()
connection.close()
and calling method
close_connection(connection, db_cursor)
I am not sure can I share my github my check this link please. Under model.py you can see database methods and to see how calling them check it main.py
Best,
Hasan.
my goal is to get the updates of an rtd server in python
I've following call in excel which is working:
=RTD("xrtd.xrtd";;"EUCA")
For python I've found following client library: https://github.com/brotchie/pyrtd/blob/master/rtd/client.py
I tried to get a simple example where I can connect to the server
import sys
sys.path.append(".")
from client import RTDClient
name = "xrtd.xrtd"
try:
client = RTDClient(name)
client.connect(False)
client.register_topic('EUCA')
except Exception as identifier:
print(str(name) + " error : " + str(identifier))
My first problem was that I've used 64bit python, but after I solved this I receive following exception from the connect():
xrtd.xrtd error : This COM object can not automate the makepy process
please run makepy manually for this object
I've no idea what I've to do now. I've python experience but no experience with COM Objects
Try this
import pythoncom
from rtd import RTDClient
if __name__ == '__main__':
time = RTDClient('xrtd.xrtd')
time.connect()
time.register_topic('EUCA')
while 1:
pythoncom.PumpWaitingMessages()
if time.update():
print time.get('EUCA')
I wrote a bot for TeamSpeak 3 that runs over ServerQuery (a telnet interface).
But the bot keeps responding later and later, in the beginning it takes like 0.1 sec, after like 1 minute the bot takes about 10 seconds to respond, and using commands makes it even faster.
Any idea why?
So basically the telnet interface sends data from the TS3 Server to my python script, the ts3 module recieves and processes the data, then the script will make a decision of what the action will be.
As modules I am using MySQLdb and ts3(https://github.com/benediktschmitt/py-ts3)
My sourcecode is here: https://pastebin.com/cJuyB9ZH
Another script, which just takes all clients and pushes them into a database every 5 min, runs multiple days without any issues.
I checked the code multiple times now and even deleted variables right after they have been used, but it still has the same issue.
My guess would be that is sortof clogges the RAM, so I looked through the code multiple times, but couldn't find out why or where.
Sidenote: I know I sometimes call a commit() when its totally not necessary, but I don't know if that might cause problems, but I dont see how.
Short(er) version of my code:
import ts3
import MySQLdb
# Some other imports like time and threading and such
## Connect to TS3
tsConn = ts3.query.TS3Connection(tsAddr, tsPort)
try:
tsConn.login(client_login_name=tsUser, client_login_password=tsPass)
tsConn.use(sid=tsSID, virtual=True)
print(" ==>> CONNECTED TO TS3 SERVER: " + tsAddr)
except ts3.query.TS3QueryError as e:
print("Login to TS Server failed! Aborting...")
exit(1)
## Connect to mySQL
try:
qConn = MySQLdb.connect(host=qHost, user=qUser, passwd=qPass, db=qDB)
qServer = qConn.cursor()
print(" ==>> CONNECTED TO mySQL SERVER: " + qHost)
except OperationalError:
print("Cannot connect to mySQL Database! Aborting...")
exit(1)
running = True
while running:
tsConn.send_keepalive()
qServer.execute("SELECT 1") # keepalive
try:
e = tsConn.wait_for_event(timeout=1)
except TS3TimeoutError:
pass
else:
try:
# <some command processing here>
except KeyError:
try:
if event[0]["reasonid"] == "0":
tsConn.sendtextmessage(targetmode=1, target=event[0]["clid"], msg=greetingmsg.format(event[0]["client_nickname"]))
except:
pass
how can I do it?
I thought, I can read something from database, but it looks too much, is there something like?:
settings.DATABASES['default'].check_connection()
All you need to do is start a application and if its not connected it will fail. Other way you can try is on shell try following -
from django.db import connections
from django.db.utils import OperationalError
db_conn = connections['default']
try:
c = db_conn.cursor()
except OperationalError:
connected = False
else:
connected = True
Run the shell
python manage.py shell
Execute this script
import django
print(django.db.connection.ensure_connection())
If it print None means everything is okay, otherwise it will throw an error if something wrong happens on your db connection
It's an old question but it needs an updated answer
python manage.py check --database default
If you're not using default or if you want to test other databases listed in your settings just name it.
It is available since version 3.1 +
Check the documentation
I use the following Django management command called wait_for_db:
import time
from django.db import connection
from django.db.utils import OperationalError
from django.core.management.base import BaseCommand
class Command(BaseCommand):
"""Django command that waits for database to be available"""
def handle(self, *args, **options):
"""Handle the command"""
self.stdout.write('Waiting for database...')
db_conn = None
while not db_conn:
try:
connection.ensure_connection()
db_conn = True
except OperationalError:
self.stdout.write('Database unavailable, waiting 1 second...')
time.sleep(1)
self.stdout.write(self.style.SUCCESS('Database available!'))
Assuming you needed this because of docker, BUT is not limitted to docker, remember this is at the end of the day Bash, and thus works everywhere *NIX.
You will first need to be using django-environ, since it will make this a whole lot easier.
The DATABASE_URL environment variable will be used inside your Django app, and here. Your settings would look like this:
import environ
env = environ.Env()
...
DATABASES = {
'default': env.db('DATABASE_URL'),
'other': env.db('DATABASE_OTHER_URL') # for illustration purposes
}
...
Your environment variables should look something like this: (more info here)
# This works with ALL the databases django supports ie (mysql/mssql/sqlite/...)
DATABASE_URL=postgres://user:pass#name_of_box:5432/database_name
DATABASE_OTHER_URL=oracle://user:pass#/(description=(address=(host=name_of_box)(protocol=tcp)(port=1521))(connect_data=(SERVICE_NAME=EX)))
Inside your entrypoint.sh do something like this:
function database_ready() {
# You need to pass a single argument called "evironment_dsn"
python << EOF
import sys
import environ
from django.db.utils import ConnectionHandler, OperationalError
env = environ.Env()
try:
ConnectionHandler(databases={'default': env.db('$1')})['default'].ensure_connection()
except (OperationalError, DatabaseError):
sys.exit(-1)
sys.exit(0)
EOF
}
Then, lets say you want to wait for your main db [the postgres in this case], you add this inside the same entrypoint.sh, under the database_ready function.
until database_ready DATABASE_URL; do
>&2 echo "Main DB is unavailable - sleeping"
sleep 1
done
This will only continue, IF postgres is up and running. What about oracle? Same thing, under the code above, we add:
until database_ready DATABASE_OTHER_URL; do
>&2 echo "Secondary DB is unavailable - sleeping"
sleep 1
done
Doing it this way will give you a couple of advantages:
you don't need to worry about other dependencies such as binaries and the likes.
you can switch databases and not have to worry about this breaking. (code is 100% database agnostic)
Create a file your_app_name/management/commands/waitdb.py and paste the bellow.
import time
from django.core.management.base import BaseCommand
from django.db import connection
from django.db.utils import OperationalError
from django.utils.translation import ngettext
class Command(BaseCommand):
help = 'Checks database connection'
def add_arguments(self, parser):
parser.add_argument(
'--seconds',
nargs='?',
type=int,
help='Number of seconds to wait before retrying',
default=1,
)
parser.add_argument(
'--retries',
nargs='?',
type=int,
help='Number of retries before exiting',
default=3,
)
def handle(self, *args, **options):
wait, retries = options['seconds'], options['retries']
current_retries = 0
while current_retries < retries:
current_retries += 1
try:
connection.ensure_connection()
break
except OperationalError:
plural_time = ngettext('second', 'seconds', wait)
self.stdout.write(
self.style.WARNING(
f'Database unavailable, retrying after {wait} {plural_time}!'
)
)
time.sleep(wait)
python manage.py waitdb --seconds 5 --retries 2
python manage.py waitdb # defaults to 1 seconds & 3 retries
I had a more complicated case where I am using mongodb behind djongo module, and RDS mysql. So not only is it multiple databases, but djongo throws an SQLDecode error instead. I also had to execute and fetch to get this working:
from django.conf import settings
if settings.DEBUG:
# Quick database check here
from django.db import connections
from django.db.utils import OperationalError
dbs = settings.DATABASES.keys()
for db in dbs:
db_conn = connections[db] # i.e. default
try:
c = db_conn.cursor()
c.execute("""SELECT "non_existent_table"."id" FROM "non_existent_table" LIMIT 1""")
c.fetchone()
print("Database '{}' connection ok.".format(db)) # This case is for djongo decoding sql ok
except OperationalError as e:
if 'no such table' in str(e):
print("Database '{}' connection ok.".format(db)) # This is ok, db is present
else:
raise # Another type of op error
except Exception: # djongo sql decode error
print("ERROR: Database {} looks to be down.".format(db))
raise
I load this in my app __init__.py, as I want it to run on startup only once and only if DEBUG is enabled. Hope it helps!
It seems Javier's answer is no longer working. He's one I put together to perform the task of checking database availability in a Docker entrypoint, assuming you have the psycopg2 library available (you're running a Django application, for instance):
function database_ready() {
python << EOF
import psycopg2
try:
db = psycopg2.connect(host="$1", port="$2", dbname="$3", user="$4", password="$5")
except:
exit(1)
exit(0)
EOF
}
until database_ready $DATABASE_HOST $DATABASE_PORT $DATABASE_NAME $DATABASE_USER $DATABASE_PASSWORD; do
>&2 echo "Database is unavailable at $DATABASE_HOST:$DATABASE_PORT/$DATABASE_NAME - sleeping..."
sleep 1
done
echo "Database is ready - $DATABASE_HOST:$DATABASE_PORT/$DATABASE_NAME"```