Develop Reference

Download files from bytea column in postgresql with python

Hi,
Can I download a file from my database from a column type bytea in python?
I'm trying to do with the psycopg2, I upload a .txt file but when I tried to retrieved it to my local machine, it just save a .txt file with non-readable data, the txt file starts like this "U0ZUUCBhZGRy...." so looks like bytes info, the same as the DB saves.
An screenshot of the DB in dbeaver example_dbeaver_column
This is the code I used.
import psycopg2
connection = psycopg2.connect(dbname=dbname,
host=host,
port=port,
user=user,
password=password)
# get cursor
cursor = connection.cursor()
query = "select c.file from my_table t where t.file_name = 'credentials.txt'"
cursor.execute(query)
data = cursor.fetchall()
file_binary=data[0][0].tobytes()
with open('my_text.txt','wb') as file:
file.write(file_binary)
Any ideas how I can solve this problem?
Thanks for your help

I found it!
In postgre it's encoded in base64 so I need to decoded with base64 library.
import base64
import psycopg2
connection = psycopg2.connect(dbname=dbname,
host=host,
port=port,
user=user,
password=password)
# get cursor
cursor = connection.cursor()
query = "select c.file from my_table t where t.file_name = 'credentials.txt'"
cursor.execute(query)
data = cursor.fetchall()
file_binary=data[0][0].tobytes()
with open('my_text.txt','wb') as file:
file.write(base64.b64decode(file_binary))

Unable to copy data into AWS RedShift

I tried a lot however I am unable to copy data available as json file in S3 bucket(I have read only access to the bucket) to Redshift table using python boto3. Below is the python code which I am using to copy the data. Using the same code I was able to create the tables in which I am trying to copy.
import configparser
import psycopg2
from sql_queries import create_table_queries, drop_table_queries
def drop_tables(cur, conn):
for query in drop_table_queries:
cur.execute(query)
conn.commit()
def create_tables(cur, conn):
for query in create_table_queries:
cur.execute(query)
conn.commit()
def main():
try:
config = configparser.ConfigParser()
config.read('dwh.cfg')
# conn = psycopg2.connect("host={} dbname={} user={} password={} port={}".format(*config['CLUSTER'].values()))
conn = psycopg2.connect(
host=config.get('CLUSTER', 'HOST'),
database=config.get('CLUSTER', 'DB_NAME'),
user=config.get('CLUSTER', 'DB_USER'),
password=config.get('CLUSTER', 'DB_PASSWORD'),
port=config.get('CLUSTER', 'DB_PORT')
)
cur = conn.cursor()
#drop_tables(cur, conn)
#create_tables(cur, conn)
qry = """copy DWH_STAGE_SONGS_TBL
from 's3://udacity-dend/song-data/A/A/A/TRAAACN128F9355673.json'
iam_role 'arn:aws:iam::xxxxxxx:role/MyRedShiftRole'
format as json 'auto';"""
print(qry)
cur.execute(qry)
# execute a statement
# print('PostgreSQL database version:')
# cur.execute('SELECT version()')
#
# # display the PostgreSQL database server version
# db_version = cur.fetchone()
# print(db_version)
print("Executed successfully")
cur.close()
conn.close()
# close the communication with the PostgreSQL
except Exception as error:
print("Error while processing")
print(error)
if __name__ == "__main__":
main()
I don't see any error in the Pycharm console but I see Aborted status in the redshift query console. I don't see any reason why it has been aborted(or I don't know where to look for that)
Other thing that I have noticed is when I run the copy statement in Redshift query editor , it runs fine and data gets moved into the table. I tried to delete and recreate the cluster but no luck. I am not able to figure what I am doing wrong. Thank you

Quick read - it looks like you haven't committed the transaction and the COPY is rolled back when the connection closes. You need to either change the connection configuration to be in "autocommit" or add an explicit "commit()".

Python - Data for mysql with dataframe

I'm new to Python and I wanted to ask you for help.
I want to put the data of a view in SQL Server into a table of my database in MySQL, when I try to give the following error:
Execution failed on sql: SELECT name FROM sqlite_master WHERE
type='table' AND name=?; not all arguments converted during string
formatting unable to rollback
Using Python version 3.7
Below is the code I use:
import pymysql.cursors
import pyodbc
import pandas as pd
# SQL Server Connection
connection = pyodbc.connect("DSN=SQLServer") #autocommit=True
try:
with connection.cursor() as cursor:
result = "SELECT * FROM dw.dbo.vW_sale"
df = pd.read_sql_query("SELECT * FROM dw.dbo.vW_sale", connection)
cursor.execute(result)
table = cursor.fetchall()
print(table)
finally:
connection.close()
# MySQL connection
cnx = pymysql.connect(host='test',
user='test',
password='test',
db='dw')
try:
with cnx.cursor() as cursor:
mysql = "select *from ft_sale_test"
cursor.execute(mysql)
result = cursor.fetchall()
#print(result)
finally:
cnx.close()
# using if_exists to handle the table that already exists
The error happens right here
df.to_sql(con=cnx, name= 'ft_sale_test', if_exists= 'replace')

Python to SQL Server Insert

I'm trying to follow the method for inserting a Panda data frame into SQL Server that is mentioned here as it appears to be the fastest way to import lots of rows.
However I am struggling with figuring out the connection parameter.
I am not using DSN , I have a server name, a database name, and using trusted connection (i.e. windows login).
import sqlalchemy
import urllib
server = 'MYServer'
db = 'MyDB'
cxn_str = "DRIVER={SQL Server Native Client 11.0};SERVER=" + server +",1433;DATABASE="+db+";Trusted_Connection='Yes'"
#cxn_str = "Trusted_Connection='Yes',Driver='{ODBC Driver 13 for SQL Server}',Server="+server+",Database="+db
params = urllib.parse.quote_plus(cxn_str)
engine = sqlalchemy.create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
conn = engine.connect().connection
cursor = conn.cursor()
I'm just not sure what the correct way to specify my connection string is. Any suggestions?

I have been working with pandas and SQL server for a while and the fastest way I found to insert a lot of data in a table was in this way:
You can create a temporary CSV using:
df.to_csv('new_file_name.csv', sep=',', encoding='utf-8')
Then use pyobdc and BULK INSERT Transact-SQL:
import pyodbc
conn = pyodbc.connect(DRIVER='{SQL Server}', Server='server_name', Database='Database_name', trusted_connection='yes')
cur = conn.cursor()
cur.execute("""BULK INSERT table_name
FROM 'C:\\Users\\folders path\\new_file_name.csv'
WITH
(
CODEPAGE = 'ACP',
FIRSTROW = 2,
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)""")
conn.commit()
cur.close()
conn.close()
Then you can delete the file:
import os
os.remove('new_file_name.csv')
It was a second to charge a lot of data at once into SQL Server. I hope this gives you an idea.
Note: don't forget to have a field for the index. It was my mistake when I started to use this lol.

Connection string parameter values should not be enclosed in quotes so you should use Trusted_Connection=Yes instead of Trusted_Connection='Yes'.

MYSql CGI Script

Below Code is my CGI Script, where am trying to do a insert Command.
#! C:\Python27\python.exe -u
import cgi
import MySQLdb
import xml.sax.saxutils
query = cgi.parse()
db = MySQLdb.connect(
host = "127.0.0.1",
user = "root",
passwd = "mysql123",
db = "welcome")
print 'Content-type: text/plain\n\n<?xml version="1.0" encoding="utf-8"?>\n<result>'
try:
c = db.cursor()
c.execute("insert into welcome.registrations values ('test','test',now())")
print '\t<update>true</update>'
except:
print '\t<update>false</update>'
print "</result>"
when i run the go to the url - .com/cgi-bin/reg.cgi, am not finding any insert operation done in mysql DB

You need to do db.commit() after c.execute().
Or you could do:
with db:
c = db.cursor()
c.execute("insert into welcome.registrations values ('test','test',now())")

make sure you do a db.commit() after every insert (or in the end, any will work ):
try:
c = db.cursor()
c.execute("insert into welcome.registrations values ('test','test',now())")
db.commit()
db.close() # Make sure you close the connection else multiple connections will result in hanging of mysql server.