I wanted to write records in Postgres with a column type set as json. In Python version of JdbcIO, WriteToJdbc has a "statement" parameter which is suppose to override the generated insert statement. It seems that it is not working at all
| 'Write to jdbc' >> WriteToJdbc(
table_name="teacher",
driver_class_name='org.postgresql.Driver',
jdbc_url='jdbc:{}://{}:{}/{}'.format("postgresql", "your ip address", "5432", "postgres"),
username="postgres",
password="password"
statement = 'insert into researcher(id, first_name, last_name, total, payload) values (?, ?, ?, ?, cast(? as json))'
)
Am I missing something here ?
The Python version of JdbcIO is still experimental and not fully supported. There's an open Beam JIRA ticket for this feature: https://issues.apache.org/jira/browse/BEAM-10750
Related
I am trying to create a system (with a discord bot, but that's not relevant to this) where it lists infractions of a user, like when it happened, where, why, etc. and I want a "date" datatype that logs the timestamp that it happened.
I tried having the DATE datatype to be "timestamp" (as well as "datetime", but the same error happens)
conn1 = apsw.Connection('./dbs/warns.db')
warns = conn1.cursor()
warns.execute(
"""
CREATE TABLE IF NOT EXISTS warns
(id INTEGER PRIMARY KEY AUTOINCREMENT,
date timestamp,
server string,
user string,
author string,
reason string)
"""
)
def add_warn(guild: str, user: str, author: str, reason):
now = datetime.datetime.utcnow()
with conn1:
warns.execute("INSERT INTO warns (date, server, user, author, reason) VALUES (?, ?, ?, ?, ?)", (now, guild, user, author, reason))
I end up getting a TypeError: Bad binding argument type supplied - argument #1: type datetime.datetime error
From the syntax of the create table statement (AUTOINCREMENT without an underscore) and the apsw tag, I suspect that you are using a SQLite database.
If you are looking to insert the current timestamp to a timestamp column, my first suggestion is to do it directly in SQL, instead of using a variable generated in python. In sqlite, CURRENT_TIMESTAP gives you the current date/time as a timestamp:
warns.execute(
"INSERT INTO warns (wdate, server, user, author, reason) VALUES (CURRENT_TIMESTAMP, ?, ?, ?, ?)",
(guild, user, author, reason)
)
Another option, that would furthermore simplify your code, is to set a default for the timestamp column when creating the table. Then, you can just ignore this column when inserting, and rest assured that the correct value will be assigned:
warns.execute(
"""
CREATE TABLE IF NOT EXISTS warns (
id INTEGER PRIMARY KEY AUTOINCREMENT,
wdate timestamp DEFAULT CURRENT_TIMESTAMP,
server string,
user string,
author string,
reason string
)
"""
)
def add_warn(guild: str, user: str, author: str, reason):
with conn1:
warns.execute(
"INSERT INTO warns (server, user, author, reason) VALUES (?, ?, ?, ?)",
(now, guild, user, author, reason)
)
Note: date is not a sensible column name, since it clashes with a datatype name. I renamed it wdate in all the above code.
I've tried to use couple of methods to insert data into mysql database but getting error in all:
In the first method:
sql = ('''Insert into lgemployees (EmpID,Name,Gender,DOB,Address,PhoneNumber,Email)
VALUES (%d,$s,$s,$s,$s,$d,$s)''', (eid, name, gen, dob, add, mob, email))
mycursor.execute(sql)
mycursor.commit()
Error in this approach:
'tuple' object has no attribute 'encode'
2nd method:
sql = "Insert into lgemployees (EmpID,Name,Gender,DOB,Address,PhoneNumber,Email) VALUES(?,?,?,?,?,?,?,)"
val = (eid, name, gen, dob, add, mob, email)
mycursor.execute(sql, val)
mycursor.commit()
Error in this approach :
"Not all parameters were used in the SQL statement")
mysql.connector.errors.ProgrammingError: Not all parameters were used in the SQL statement
I've troubleshooted a lot from my end but no luck. Can any one please help as where am I wrong or what else can be a good option to insert data into mysql from python.
I dont know where you error is at, but ive tested with this code and it works.
insert_tuple = (eid, name, gen, dob, add, mob, email)
sql = """INSERT INTO lgemployees (`EmpID `,
`Name`,`Gender`, `DOB`, `Address`, `PhoneNumber`, `Email`)
VALUES (%s,%s,%s,%s,%s,%s,%s)"""
mycursor = mySQLconnection.cursor()
mycursor.execute(sql, insert_tuple)
mySQLconnection.commit()
mycursor.close()
your code throws this because one of the parameters are empty or are in a format it cant read.
"Not all parameters were used in the SQL statement")
mysql.connector.errors.ProgrammingError: Not all parameters were used in the SQL statement
I have a string with mixed quotes that is " and '. I want to store the string in a Text field in a sqlite3 database using python.
Here is the query I'm using and I have a function that executes these queries.
"""INSERT INTO SNIPPETS (CONTENT, LANGUAGE, TITLE, BACKGROUND)
VALUES("{0}" ,"{1}","{2}", "{3}")
""".format(content, language, title, background)
Something like:
with self.connection as conn:
cursor = conn.cursor()
try:
result = cursor.execute(statement)
if(insert_operation):
return cursor.lastrowid
return result.fetchall()
You don't have to make it too complicated. Python sqlite3 will take care of the quoting for you.
statement = 'INSERT INTO SNIPPETS (CONTENT, LANGUAGE, TITLE, BACKGROUND) VALUES (?, ?, ?, ?)'
cursor.execute(statement, (content, language, title, background))
I'm trying to insert several variables in a insert query on postgres using python. I can't wrap my head around how to use the string formatting.
For example, this works fine:
cursor.execute('''CREATE TABLE %s
(id SERIAL PRIMARY KEY,
sender varchar(255) not null,
receiver varchar(255) not null,
message varchar(255))''' %username)
as does this:
cursor.execute('''INSERT INTO test (sender, receiver, message)
VALUES(%s,%s,%s)''', (sender, receiver,message))
My problem is that I want to have the table name as a variable too. I have tried:
cursor.execute('''INSERT INTO %s (sender, receiver, message)
VALUES(%s,%s,%s)''' %username, (sender, receiver, message))
I get the following error:
TypeError: not enough arguments for format string
I get that I have to change the parentheses somehow, but I don't know how.
Thanks in advance.
EDIT:
Choose a different approach from this psycopg2 which worked perfectly.
You are passing the arguments in a wrong way. The arguments passed are causing you the trouble. Use format function instead of % as it is more sophisticated and readable.
"INSERT INTO {} (sender, receiver, message) VALUES({},{},{})".format("some", "world", "world","hello")
The output of the above:
'INSERT INTO some (sender, receiver, message) VALUES(world,world,hello)'
Use the high level sql module to avoid likely mistakes:
from psycopg2 import sql
query = sql.SQL('''
insert into {} (sender, receiver, message)
values (%s, %s, %s)
''').format(sql.Identifier(username))
cursor.execute (query, (sender, receiver, message))
I am having a hard time understanding why psycopg2 has a problem with the word 'user'. I am trying to insert values into a table called user with the columns user_id, name, password. I am getting a programmingError: syntax error at or near "user". open_cursor() is a function used to open a cursor for database operations.
Here is my code:
query = """INSERT INTO user (name, password) VALUES (%s, %s);"""
data = ('psycouser', 'sha1$ba316b$52dd71da1e331247f0a7ab869e1b072210add9c1')
with open_cursor() as cursor:
cursor.execute(query, data)
print "Done."
because user is a part of sql language.
try taking it in dbl quotes:
query = 'INSERT INTO "user" (name, password) VALUES (%s, %s);'