Develop Reference

Speedtest-cli python script, inserting sql with wrong values

Relatively new to python scripts, so bare with.
I have used speedtest-cli before. I have edited the script so it will insert the values into a sql table as below, however having an issue with one of the inserts. It will insert ping, and download ok, however, the upload is always 2.74 or 2.75 for example, but ONLY when run from a crontab.. very weird.
If I run the python script from cli it will insert values fine.
This is my query, and the values ping, download and upload are coming from the speedtest-cli script.
Here is the full script
import re
import subprocess
import time
import mysql.connector
from mysql.connector import Error
from mysql.connector import errorcode
print "----------------------------------"
print 'Started: {} {}'.format(time.strftime('%d/%m/%y %H:%M:%S'), "")
response = subprocess.Popen('speedtest-cli --simple', shell=True, stdout=subprocess.PIPE).stdout.read()
ping = re.findall('Ping:\s(.*?)\s', response, re.MULTILINE)
download = re.findall('Download:\s(.*?)\s', response, re.MULTILINE)
upload = re.findall('Upload:\s(.*?)\s', response, re.MULTILINE)
ping[0] = ping[0].replace(',', '.')
download[0] = download[0].replace(',', '.')
upload[0] = upload[0].replace(',', '.')
try:
if os.stat('/var/www/html/speed/log.txt').st_size == 0:
print 'Date,Time,Ping (ms),Download (Mbit/s),Upload (Mbit/s)'
except:
pass
print 'PING: {}, DOWN: {}, UP: {}'.format(ping[0], download[0], upload[0])
try:
connection = mysql.connector.connect(host='localhost',
database='dev',
user='dev',
password='dev1')
sql_insert_query = ("""INSERT INTO speedtest(ping, download, upload) VALUES (%s,%s,%s)""", (ping[0], download[0], upload[0]))
cursor = connection.cursor()
result = cursor.execute(*sql_insert_query)
connection.commit()
print ("Insert success into speedtest tbl")
except mysql.connector.Error as error :
connection.rollback() #rollback if any exception occured
print("Failed inserting record into speedtest table {}".format(error))
finally:
#closing database connection.
if(connection.is_connected()):
cursor.close()
connection.close()
print("MySQL conn closed")
print 'Finished: {} {}'.format(time.strftime('%d/%m/%y %H:%M:%S'), "")
Manual script runs ok, just from crontab I get unexpected values. Not sure how to solve.

Connecting to Postgres Database via Python

I am trying to connect to my database via Python 2.7 with this code:
import csv
import psycopg2
try:
conn = psycopg2.connect("dbname='student', user='postgres',password='password', host='localhost'")
cursor = conn_cursor()
reader = csv.reader(open('last_file.csv', 'rb'))
print "connected"
except:
print "not Connected"
It did work last week and we don't think we've changed anything, but now it won't connect.
We've tried using it with the database open and closed, nothing worked.
The database does exist in Postgres.

import psycopg2
try:
conn = psycopg2.connect("dbname='database_name' user='postgres_user_name' host='localhost' password='user_passwd'")
except:
print "I am unable to connect to the database"
cur = conn.cursor()
cur.execute("""SELECT * from table_name""")
rows = cur.fetchall()
print "\nShow me the data:\n"
for row in rows:
print " ", row[0]
print " ", row[1]

Exception part add like this to see what is error
except Exception as ex:
print "not Connected"
print "Error: "+ str(ex)

Try this:
import csv
import psycopg2
try:
conn = psycopg2.connect("dbname='student', user='postgres',password='password', host='localhost'")
except:
print "I am unable to connect to the database."
cursor = conn.cursor()
try:
reader = csv.reader(open('last_file.csv', 'rb'))
print "connected"
except:
print "not Connected"

Seems like there are something wrong with your postgres.
Try and see postgres log.
Location of postgres log by default :
tail -f /var/log/postgresql/<>/main/postgresql.log
something like this.
Also don't forget to check firewall. Maybe someone disable it by accident.

Also try for pip install PyGreSQL package. Since psycopg2 (some of versions) is under GPL license. It could be tricky for open source license. Just for your information.

Lambda function unable to connect to Redshift : Temporary failure in name resolution

I have an AWS Lambda function which is connected to a Jira webhook. Anytime an issue is created/changed/deleted, it sends a request to my API Gateway, which calls my function. I'm receiving the following traceback when I try to connect to Redshift inside the function:
Traceback (most recent call last):
File "/var/task/jira_webhook_proc.py", line 171, in lambda_handler
update_item(request, issue, issue_key)
File "/var/task/jira_webhook_proc.py", line 193, in update_item
delete_item(delete_key)
File "/var/task/jira_webhook_proc.py", line 277, in delete_item
port=REDSHIFT_PORT) as conn:
File "/var/task/psycopg2/__init__.py", line 164, in connect
conn = _connect(dsn, connection_factory=connection_factory, async=async)
OperationalError: could not translate host name "***" to address: Temporary failure in name resolution
I'm using a pre-compiled psycopg2 library that can be found here https://github.com/jkehler/awslambda-psycopg2. From googling it seems this might be a PostgreSQL error where I don't have a config file set right to listen to all ports. I'm not sure how I would change that however.

I use this code added below to connect redshift from lambda function. Using psycopg2
conn_string = "dbname='your_db_name' port='5439' user='redshift_user' password='^%+^+&7!+' host='xxxxxxx.yyyyyyy.eu-west-1.redshift.amazonaws.com'"
conn = psycopg.connect(conn_string)
cursor = conn.cursor()
cursor.execute("COPY data_table ........ ")
conn.commit()
cursor.close()
conn.close()

It might help someone as it worked perfectly in my case.
import psycopg2
import od_red_load_cred as CONSTANTS
def main():
# Amazon Redshift connect string
try:
conn_string = "dbname={} port={} user={} password={} host={}"\
.format(CONSTANTS.DBNAME, CONSTANTS.PORT, CONSTANTS.USER,
CONSTANTS.PASSWORD, CONSTANTS.HOST)
print("Connect string Successful!")
except:
print("Unable to create the connect string")
# query generate
try:
sql="""copy {}.{} from '{}'\
credentials 'aws_access_key_id={};aws_secret_access_key={}' \
DELIMITER ',' ACCEPTINVCHARS EMPTYASNULL ESCAPE COMPUPDATE OFF;commit;"""\
.format(CONSTANTS.SCHEMA, CONSTANTS.TABLE_NAME_BV, CONSTANTS.S3_PATH_BV,
CONSTANTS.ACCESS_KEY_ID, CONSTANTS.SECRET_ACCESS_KEY)
print("sql ==>", sql)
print("Query creation Successful!")
print(" ")
except:
print("Unable to create the query")
# get connection
try:
con = psycopg2.connect(conn_string);
print("con ==>", con)
print("Connection Successful!")
except Exception as e:
print("Unable to connect to Redshift")
print(e)
exit(-1)
# get cursor connection
try:
cur = con.cursor()
except Exception as e:
print("Unable to get the cursor from connection of Redshift")
print(e)
exit(-1)
# execute cursor connection
try:
cur.execute(sql)
print("Query executed successfully")
except Exception as e:
print("Failed to execute the query")
print(e)
exit(-1)
# close connection
try:
con.close()
print("Connection closed Successfully!")
except Exception as e:
print("Unable to close connection to Redshift")
print(e)
print(" ")
# Main method
if __name__ == '__main__':
print("start: __main__ ")
main()
print("end: __main__ ")

Python don't register in MySQL server

there’s something wrong in my python script: when I try to put some data in my database and print it, it looks like it’s working, but when I rerun the code, or if I check the phpmyadmin, there’s no data saved in the db. Does anyone have some idea on how to solve this problem?
import mysql.connector
from mysql.connector import errorcode
def connect():
""" Connect to MySQL database """
try:
conn = mysql.connector.connect(host='localhost',
database='Temperature',
user='Temperature',
password='mypass')
if conn.is_connected():
print('Connected to MySQL database')
cur = conn.cursor()
query = "INSERT INTO Temp(temp, humi) " \
"VALUES(315, 55)"
try:
cur.execute(query)
except MySQLdb.ProgrammingError as e:
print(e)
query = "SELECT * FROM Temp"
try:
cur.execute(query)
for reading in cur.fetchall():
print (str(reading[0])+" "+str(reading[1]))
except MySQLdb.ProgrammingError as e:
print(e)
except Error as e:
print(e)
finally:
conn.close()
if __name__ == '__main__':
connect()

You will need to add conn.commit() before conn.close(). That should solve the problem.

Closing a cx_Oracle Connection While Allowing for a Down Database

The following cx_Oracle code works fine when the database is up:
#!C:\Python27
import cx_Oracle
try:
conn = cx_Oracle.connect("scott/tiger#oracle")
try:
curs = conn.cursor()
curs.execute("SELECT dummy FROM sys.dual")
print curs.fetchone()[0]
finally:
curs.close()
finally:
conn.close()
But if the database happens to be down when I run this script, a NameError is raised:
Traceback (most recent call last):
File "C:\Users\ArtMetzer\Documents\Code\Python\db_conn_test.py", line 14, in <module>
conn.close()
NameError: name 'conn' is not defined
This makes sense to me: cx_Oracle wasn't able to instantiate a connection, so the variable conn never got set, and hence has no close() method.
In Python, what's the best way to ensure your database connection closes, while still gracefully handling the condition of a down database?
Doing something like the following seems like a massive kludge to me:
finally:
try:
conn.close()
except NameError:
pass

You can try initializing conn to something like None before-hand and testing that in the finally block. This works because the only place the connection is set to something else is when it is opened. So opened implies non-None and None implies not-opened:
#!C:\Python27
import cx_Oracle
conn = None
try:
conn = cx_Oracle.connect("scott/tiger#oracle")
try:
curs = conn.cursor()
curs.execute("SELECT dummy FROM sys.dual")
print curs.fetchone()[0]
finally:
curs.close()
finally:
if conn is not None:
conn.close()

According to the docs, you don't really need to bother closing the connection:
https://cx-oracle.readthedocs.io/en/latest/user_guide/connection_handling.html#closing-connections
The important line being:
Alternatively, you may prefer to let connections be automatically cleaned up when references to them go out of scope. This lets cx_Oracle close dependent resources in the correct order.

(not exactly an answer, but comments don't have nice formatting)
Try this:
#!C:\Python27
import cx_Oracle
try:
conn = cx_Oracle.connect("scott/tiger#oracle")
try:
curs = conn.cursor()
curs.execute("SELECT dummy FROM sys.dual")
print curs.fetchone()[0]
finally:
curs.close()
conn.close()
except Exception as e:
print e
Not ideal, but should work better. I'm also wondering why so much nesting. Why not do this:
#!C:\Python27
import cx_Oracle
try:
conn = cx_Oracle.connect("scott/tiger#oracle")
curs = conn.cursor()
curs.execute("SELECT dummy FROM sys.dual")
print curs.fetchone()[0]
curs.close()
conn.close()
except Exception as e:
print e
BTW, I have this assumption that the connection and the cursor will close automatically on exit, removing the need to close them explicitly.