I am writing a code in python which will run sql select query and return the result. How do I display the output from python script in Splunk?
Currently I just have a python script running sql query and have tried importing import splunklib.client as clientwhich fails as [pylint] E0401:Unable to import 'splunklib.client'
Tried this:
import mysql.connector
import splunklib.client as client
#splunk credentials
HOST = "localhost"
PORT = 8089
USERNAME = "admin"
PASSWORD = "yourpassword"
# Connect to splunk and log in
service = client.connect(
host=HOST,
port=PORT,
username=USERNAME,
password=PASSWORD)
But it gives an error while importing library
I expect the output of python script which will be something like :
STAGED = 1
FAILED = 2
VALIDATED =1
to be displayed in SPLUNK using python script itself.
You have to pass this command in commands.conf file. In stanza give command name and key value pair pass filename = python filename
[commandname]
filename = python.py
You can more knowledge on docs.splunk about commands.conf file
I am new to using python to connect to a mysql DB and I am getting the following error:
OperationalError: (pymysql.err.OperationalError) (1045, u"Access denied for user 'xxxxxxadmin'#'xx.xx.xx.xx' (using password: YES)") (Background on this error at: http://sqlalche.me/e/e3q8)
xx.xxx.216.44 - - [02/Apr/2018 17:27:49] "GET /testconnect HTTP/1.1" 500 -
This is most of the connect script in my python file:
#!/usr/bin/python3
from flask import Flask, request
from flask_restful import Resource, Api
from sqlalchemy import create_engine
from json import dumps
from flask.ext.jsonpify import jsonify
db_connect = create_engine("mysql+pymysql://xxxxxxxadmin:password#,mymaindb.xxxxxxxx.us-east-2.rds.amazonaws.com:3306/myDBname")
app = Flask(__name__)
api = Api(app)
class TestConnect(Resource):
def get(self):
conn = db_connect.connect() # connect to database
query = conn.execute("select * from Players") # This line performs query and returns json result
return {'employees': [i[0] for i in query.cursor.fetchall()]} # Fetches first column that is Employee ID
api.add_resource(TestConnect, '/testconnect') # Route_1
if __name__ == '__main__':
app.run(host='0.0.0.0', debug = False)
Other background:
But when I try to connect to the same mysql database using the exact same credentials via the command line on the server running the python script I am able to get in.
Not sure how to test more to get a better error result that will help me figure this issue out.
UPDATE
So I was able to connect to my DB via mysql workbench with the connection strings and information I have in the python script. Does this mean my python script is doing something wrong?
Why not use:
mysql+pymysql://xxxxxxxadmin:password#mymaindb.xxxxxxxx.us-east-2.rds.amazonaws.com:3306/myDBname
instead of
mysql+pymysql://xxxxxxxadmin:password#**,**mymaindb.xxxxxxxx.us-east-2.rds.amazonaws.com:3306/myDBname
Not sure why you're connection string has a comma. Might just be a typo?
On that note, I usually build the connection URL before passing it to create_engine just to make it easier to manage in the future incase I have to pull the actual values from the environmental variables:
HOST = "mymaindb.xxxxxxxx.us-east-2.rds.amazonaws.com"
PORT = 3306
USERNAME = "xxxxxxxadmin"
PASSWORD = "password"
DBNAME = "myDBname"
CONNECTION_URL = 'mysql+pymysql://%s:%s#%s:%s/%s' % (
USERNAME,
PASSWORD,
HOST,
PORT,
DBNAME
)
I have trouble to connect mysql server via Flask.
I have 4 files: server.py ,testDB.py,testDB2.py and init.py
First , if init.py import testDB.py and I run python server.py, it will print
changes in the database
* Running on http://0.0.0.0:8000/ (Press CTRL+C to quit)
My user_info table will have user1.
However ,if init.py import testDB2.py and I run python server.py,it just print
* Running on http://0.0.0.0:8000/ (Press CTRL+C to quit)
My user_info table will not appear user2.
How do I solve this problem ?
The difference between testDb.py and testDB2.py is I defined a function in testDB2.py
init.py
from flask import Flask
app = Flask(__name__)
import testDB
server.py
from Sw import app
if __name__ == '__main__':
port = 8000
host = '0.0.0.0'
app.run(host = host, port = port)
testDB.py
import MySQLdb
db = MySQLdb.connect(host="127.0.0.1",user="root",passwd="1234",db="testdb")
cursor = db.cursor()
sql="""INSERT INTO user_info (user_id, user_email, user_password) VALUES ('user1','00000','000000')"""
try:
# Execute the SQL command
cursor.execute(sql)
# Commit your changes in the database
print "changes in the database"
db.commit()
except:
# Rollback in case there is any error
print "there is any error"
db.rollback()
db.close()
testDB2.py
import MySQLdb
def testDB():
db = MySQLdb.connect(host="127.0.0.1",user="root",passwd="1234",db="testdb")
cursor = db.cursor()
sql="""INSERT INTO user_info (user_id, user_email, user_password) VALUES ('user2','00000','000000')"""
try:
# Execute the SQL command
cursor.execute(sql)
# Commit your changes in the database
print "changes in the database"
db.commit()
except:
# Rollback in case there is any error
print "there is any error"
db.rollback()
db.close()
Just as #dirn said in the comment, the reason the database isn't updated in the second case is because you've defined a function, but then never made use of it. Just as any other function in Python, it waits for another line of code to call it into action. When you import it into your init.py file, you have two ways of running it. You can modify init.py like this:
from flask import Flask
app = Flask(__name__)
import testDB2
testDB2.testDB()
Which then runs your function from the init.py file, or you can modify testDB2.py and run the function from there, just by adding testDB() to the end of that file (outside the function, of course).
How to connect to a mongodb host remotely by specifying Username, Password, Hostname and also how to get db.serverStatus() output through pymongo ???
"I have commented the bind_ip in **mongod.conf* file, so that it allows remote connection"
import pymongo
from pymongo import MongoClient
connection=MongoClient(???)
Following is a sample code:
import pymongo
MONGO_HOST = ''
MONGO_PORT = <PORT>
MONGO_DB=''
MONGO_USER=''
MONGO_PASS=''
def get_mongo_db():
con=pymongo.Connection(MONGO_HOST,MONGO_PORT)
db=con[MONGO_DB]
try:
db.authenticate(MONGO_USER,MONGO_PASS)
except:
return None
return db
Attention, if your mongo doesn't open auth (--auth), you needn't to auth, but it's recommended to open auth for security.
then, you can use db for more ops, as you said, db.serverStatus() (I haven't tried, maybe a little different)
how can I do it?
I thought, I can read something from database, but it looks too much, is there something like?:
settings.DATABASES['default'].check_connection()
All you need to do is start a application and if its not connected it will fail. Other way you can try is on shell try following -
from django.db import connections
from django.db.utils import OperationalError
db_conn = connections['default']
try:
c = db_conn.cursor()
except OperationalError:
connected = False
else:
connected = True
Run the shell
python manage.py shell
Execute this script
import django
print(django.db.connection.ensure_connection())
If it print None means everything is okay, otherwise it will throw an error if something wrong happens on your db connection
It's an old question but it needs an updated answer
python manage.py check --database default
If you're not using default or if you want to test other databases listed in your settings just name it.
It is available since version 3.1 +
Check the documentation
I use the following Django management command called wait_for_db:
import time
from django.db import connection
from django.db.utils import OperationalError
from django.core.management.base import BaseCommand
class Command(BaseCommand):
"""Django command that waits for database to be available"""
def handle(self, *args, **options):
"""Handle the command"""
self.stdout.write('Waiting for database...')
db_conn = None
while not db_conn:
try:
connection.ensure_connection()
db_conn = True
except OperationalError:
self.stdout.write('Database unavailable, waiting 1 second...')
time.sleep(1)
self.stdout.write(self.style.SUCCESS('Database available!'))
Assuming you needed this because of docker, BUT is not limitted to docker, remember this is at the end of the day Bash, and thus works everywhere *NIX.
You will first need to be using django-environ, since it will make this a whole lot easier.
The DATABASE_URL environment variable will be used inside your Django app, and here. Your settings would look like this:
import environ
env = environ.Env()
...
DATABASES = {
'default': env.db('DATABASE_URL'),
'other': env.db('DATABASE_OTHER_URL') # for illustration purposes
}
...
Your environment variables should look something like this: (more info here)
# This works with ALL the databases django supports ie (mysql/mssql/sqlite/...)
DATABASE_URL=postgres://user:pass#name_of_box:5432/database_name
DATABASE_OTHER_URL=oracle://user:pass#/(description=(address=(host=name_of_box)(protocol=tcp)(port=1521))(connect_data=(SERVICE_NAME=EX)))
Inside your entrypoint.sh do something like this:
function database_ready() {
# You need to pass a single argument called "evironment_dsn"
python << EOF
import sys
import environ
from django.db.utils import ConnectionHandler, OperationalError
env = environ.Env()
try:
ConnectionHandler(databases={'default': env.db('$1')})['default'].ensure_connection()
except (OperationalError, DatabaseError):
sys.exit(-1)
sys.exit(0)
EOF
}
Then, lets say you want to wait for your main db [the postgres in this case], you add this inside the same entrypoint.sh, under the database_ready function.
until database_ready DATABASE_URL; do
>&2 echo "Main DB is unavailable - sleeping"
sleep 1
done
This will only continue, IF postgres is up and running. What about oracle? Same thing, under the code above, we add:
until database_ready DATABASE_OTHER_URL; do
>&2 echo "Secondary DB is unavailable - sleeping"
sleep 1
done
Doing it this way will give you a couple of advantages:
you don't need to worry about other dependencies such as binaries and the likes.
you can switch databases and not have to worry about this breaking. (code is 100% database agnostic)
Create a file your_app_name/management/commands/waitdb.py and paste the bellow.
import time
from django.core.management.base import BaseCommand
from django.db import connection
from django.db.utils import OperationalError
from django.utils.translation import ngettext
class Command(BaseCommand):
help = 'Checks database connection'
def add_arguments(self, parser):
parser.add_argument(
'--seconds',
nargs='?',
type=int,
help='Number of seconds to wait before retrying',
default=1,
)
parser.add_argument(
'--retries',
nargs='?',
type=int,
help='Number of retries before exiting',
default=3,
)
def handle(self, *args, **options):
wait, retries = options['seconds'], options['retries']
current_retries = 0
while current_retries < retries:
current_retries += 1
try:
connection.ensure_connection()
break
except OperationalError:
plural_time = ngettext('second', 'seconds', wait)
self.stdout.write(
self.style.WARNING(
f'Database unavailable, retrying after {wait} {plural_time}!'
)
)
time.sleep(wait)
python manage.py waitdb --seconds 5 --retries 2
python manage.py waitdb # defaults to 1 seconds & 3 retries
I had a more complicated case where I am using mongodb behind djongo module, and RDS mysql. So not only is it multiple databases, but djongo throws an SQLDecode error instead. I also had to execute and fetch to get this working:
from django.conf import settings
if settings.DEBUG:
# Quick database check here
from django.db import connections
from django.db.utils import OperationalError
dbs = settings.DATABASES.keys()
for db in dbs:
db_conn = connections[db] # i.e. default
try:
c = db_conn.cursor()
c.execute("""SELECT "non_existent_table"."id" FROM "non_existent_table" LIMIT 1""")
c.fetchone()
print("Database '{}' connection ok.".format(db)) # This case is for djongo decoding sql ok
except OperationalError as e:
if 'no such table' in str(e):
print("Database '{}' connection ok.".format(db)) # This is ok, db is present
else:
raise # Another type of op error
except Exception: # djongo sql decode error
print("ERROR: Database {} looks to be down.".format(db))
raise
I load this in my app __init__.py, as I want it to run on startup only once and only if DEBUG is enabled. Hope it helps!
It seems Javier's answer is no longer working. He's one I put together to perform the task of checking database availability in a Docker entrypoint, assuming you have the psycopg2 library available (you're running a Django application, for instance):
function database_ready() {
python << EOF
import psycopg2
try:
db = psycopg2.connect(host="$1", port="$2", dbname="$3", user="$4", password="$5")
except:
exit(1)
exit(0)
EOF
}
until database_ready $DATABASE_HOST $DATABASE_PORT $DATABASE_NAME $DATABASE_USER $DATABASE_PASSWORD; do
>&2 echo "Database is unavailable at $DATABASE_HOST:$DATABASE_PORT/$DATABASE_NAME - sleeping..."
sleep 1
done
echo "Database is ready - $DATABASE_HOST:$DATABASE_PORT/$DATABASE_NAME"```