Override generated insert statement in JdbcIO with Apache Beam - python

I wanted to write records in Postgres with a column type set as json. In Python version of JdbcIO, WriteToJdbc has a "statement" parameter which is suppose to override the generated insert statement. It seems that it is not working at all
| 'Write to jdbc' >> WriteToJdbc(
table_name="teacher",
driver_class_name='org.postgresql.Driver',
jdbc_url='jdbc:{}://{}:{}/{}'.format("postgresql", "your ip address", "5432", "postgres"),
username="postgres",
password="password"
statement = 'insert into researcher(id, first_name, last_name, total, payload) values (?, ?, ?, ?, cast(? as json))'
)
Am I missing something here ?

The Python version of JdbcIO is still experimental and not fully supported. There's an open Beam JIRA ticket for this feature: https://issues.apache.org/jira/browse/BEAM-10750

Related

Insert datetime entry into table with TIMESTAMP data type

I am trying to create a system (with a discord bot, but that's not relevant to this) where it lists infractions of a user, like when it happened, where, why, etc. and I want a "date" datatype that logs the timestamp that it happened.
I tried having the DATE datatype to be "timestamp" (as well as "datetime", but the same error happens)
conn1 = apsw.Connection('./dbs/warns.db')
warns = conn1.cursor()
warns.execute(
"""
CREATE TABLE IF NOT EXISTS warns
(id INTEGER PRIMARY KEY AUTOINCREMENT,
date timestamp,
server string,
user string,
author string,
reason string)
"""
)
def add_warn(guild: str, user: str, author: str, reason):
now = datetime.datetime.utcnow()
with conn1:
warns.execute("INSERT INTO warns (date, server, user, author, reason) VALUES (?, ?, ?, ?, ?)", (now, guild, user, author, reason))
I end up getting a TypeError: Bad binding argument type supplied - argument #1: type datetime.datetime error
From the syntax of the create table statement (AUTOINCREMENT without an underscore) and the apsw tag, I suspect that you are using a SQLite database.
If you are looking to insert the current timestamp to a timestamp column, my first suggestion is to do it directly in SQL, instead of using a variable generated in python. In sqlite, CURRENT_TIMESTAP gives you the current date/time as a timestamp:
warns.execute(
"INSERT INTO warns (wdate, server, user, author, reason) VALUES (CURRENT_TIMESTAMP, ?, ?, ?, ?)",
(guild, user, author, reason)
)
Another option, that would furthermore simplify your code, is to set a default for the timestamp column when creating the table. Then, you can just ignore this column when inserting, and rest assured that the correct value will be assigned:
warns.execute(
"""
CREATE TABLE IF NOT EXISTS warns (
id INTEGER PRIMARY KEY AUTOINCREMENT,
wdate timestamp DEFAULT CURRENT_TIMESTAMP,
server string,
user string,
author string,
reason string
)
"""
)
def add_warn(guild: str, user: str, author: str, reason):
with conn1:
warns.execute(
"INSERT INTO warns (server, user, author, reason) VALUES (?, ?, ?, ?)",
(now, guild, user, author, reason)
)
Note: date is not a sensible column name, since it clashes with a datatype name. I renamed it wdate in all the above code.

Facing issues in Python to MYSQL insertion

I've tried to use couple of methods to insert data into mysql database but getting error in all:
In the first method:
sql = ('''Insert into lgemployees (EmpID,Name,Gender,DOB,Address,PhoneNumber,Email)
VALUES (%d,$s,$s,$s,$s,$d,$s)''', (eid, name, gen, dob, add, mob, email))
mycursor.execute(sql)
mycursor.commit()
Error in this approach:
'tuple' object has no attribute 'encode'
2nd method:
sql = "Insert into lgemployees (EmpID,Name,Gender,DOB,Address,PhoneNumber,Email) VALUES(?,?,?,?,?,?,?,)"
val = (eid, name, gen, dob, add, mob, email)
mycursor.execute(sql, val)
mycursor.commit()
Error in this approach :
"Not all parameters were used in the SQL statement")
mysql.connector.errors.ProgrammingError: Not all parameters were used in the SQL statement
I've troubleshooted a lot from my end but no luck. Can any one please help as where am I wrong or what else can be a good option to insert data into mysql from python.
I dont know where you error is at, but ive tested with this code and it works.
insert_tuple = (eid, name, gen, dob, add, mob, email)
sql = """INSERT INTO lgemployees (`EmpID `,
`Name`,`Gender`, `DOB`, `Address`, `PhoneNumber`, `Email`)
VALUES (%s,%s,%s,%s,%s,%s,%s)"""
mycursor = mySQLconnection.cursor()
mycursor.execute(sql, insert_tuple)
mySQLconnection.commit()
mycursor.close()
your code throws this because one of the parameters are empty or are in a format it cant read.
"Not all parameters were used in the SQL statement")
mysql.connector.errors.ProgrammingError: Not all parameters were used in the SQL statement

How can I store a string with mixed quotes in a Sqlite database using python?

I have a string with mixed quotes that is " and '. I want to store the string in a Text field in a sqlite3 database using python.
Here is the query I'm using and I have a function that executes these queries.
"""INSERT INTO SNIPPETS (CONTENT, LANGUAGE, TITLE, BACKGROUND)
VALUES("{0}" ,"{1}","{2}", "{3}")
""".format(content, language, title, background)
Something like:
with self.connection as conn:
cursor = conn.cursor()
try:
result = cursor.execute(statement)
if(insert_operation):
return cursor.lastrowid
return result.fetchall()
You don't have to make it too complicated. Python sqlite3 will take care of the quoting for you.
statement = 'INSERT INTO SNIPPETS (CONTENT, LANGUAGE, TITLE, BACKGROUND) VALUES (?, ?, ?, ?)'
cursor.execute(statement, (content, language, title, background))

Insert query with variables postgresql python

I'm trying to insert several variables in a insert query on postgres using python. I can't wrap my head around how to use the string formatting.
For example, this works fine:
cursor.execute('''CREATE TABLE %s
(id SERIAL PRIMARY KEY,
sender varchar(255) not null,
receiver varchar(255) not null,
message varchar(255))''' %username)
as does this:
cursor.execute('''INSERT INTO test (sender, receiver, message)
VALUES(%s,%s,%s)''', (sender, receiver,message))
My problem is that I want to have the table name as a variable too. I have tried:
cursor.execute('''INSERT INTO %s (sender, receiver, message)
VALUES(%s,%s,%s)''' %username, (sender, receiver, message))
I get the following error:
TypeError: not enough arguments for format string
I get that I have to change the parentheses somehow, but I don't know how.
Thanks in advance.
EDIT:
Choose a different approach from this psycopg2 which worked perfectly.
You are passing the arguments in a wrong way. The arguments passed are causing you the trouble. Use format function instead of % as it is more sophisticated and readable.
"INSERT INTO {} (sender, receiver, message) VALUES({},{},{})".format("some", "world", "world","hello")
The output of the above:
'INSERT INTO some (sender, receiver, message) VALUES(world,world,hello)'
Use the high level sql module to avoid likely mistakes:
from psycopg2 import sql
query = sql.SQL('''
insert into {} (sender, receiver, message)
values (%s, %s, %s)
''').format(sql.Identifier(username))
cursor.execute (query, (sender, receiver, message))

Psycopg2 issue inserting values into an existing table in the database

I am having a hard time understanding why psycopg2 has a problem with the word 'user'. I am trying to insert values into a table called user with the columns user_id, name, password. I am getting a programmingError: syntax error at or near "user". open_cursor() is a function used to open a cursor for database operations.
Here is my code:
query = """INSERT INTO user (name, password) VALUES (%s, %s);"""
data = ('psycouser', 'sha1$ba316b$52dd71da1e331247f0a7ab869e1b072210add9c1')
with open_cursor() as cursor:
cursor.execute(query, data)
print "Done."
because user is a part of sql language.
try taking it in dbl quotes:
query = 'INSERT INTO "user" (name, password) VALUES (%s, %s);'

Categories

Resources