Develop Reference

TypeError: not all arguments converted during string formatting in vertica_python

I'm trying to insert some values into my vertica database using the vertica_python module:
data = {'SalesNo': ['12345', '678910'],
'ProductID': ['12345_2021-10-21_08:51:22', '678910_2021-10-21_10:27:03'],
'StoreID': ['6d522936e240cd64e1cf9176c5bfdff3bfe8146a345ff2', 'a7274d507d443c752be66b2851415138d75bd913d4949e'],
'PurchaseTime': ['2021-10-21 08:51:22.846000', '2021-10-21 10:44:06.218000'],
'Date': ['2021-10-21', '2021-10-21'],
'StoreNumber': ['0', '1'],
'PurchaseValue': ['348.0', '4893.23']
}
dataset = pd.DataFrame(data)
column = dataset.columns
n = len(column)
SQL_insert = f"INSERT INTO table_name ({','.join(list(column))}) VALUES ({' ?,'*(n-1)} ?);"
valueTuplelst = []
for i,row in dataset.iterrows():
valuelist = list(map(lambda x: str(x), row))
valuelist = [None if element == "None" else element for element in valuelist]
valuetup = tuple(valuelist)
valueTuplelst.append(valuetup)
connection.cursor().executemany(SQL_insert, valueTuplelst)
The equivalent SQL statement is
INSERT INTO table_name (SalesNo,ProductID,StoreID,PurchaseTime,Date,StoreNumber,PurchaseValue) VALUES ('12345', '12345_2021-10-21_08:51:22', '6d522936e240cd64e1cf9176c5bfdff3bfe8146a345ff2', '2021-10-21 08:51:22.846000', '2021-10-21', '0', '348.0'), ('678910', '678910_2021-10-21_10:27:03', 'a7274d507d443c752be66b2851415138d75bd913d4949e', '2021-10-21 10:44:06.218000', '2021-10-21', '1', '4893.23')
which works perfectly in SQL when I execute it.
However I get the error
File "C:\tools\Anaconda3\lib\site-packages\vertica_python\vertica\cursor.py", line 576, in format_operation_with_parameters
operation = operation % tuple(tlist)
TypeError: not all arguments converted during string formatting
I can't seem to figure out why I get this error as I convert all of my data to string format. Any idea where I'm going wrong?

My guess is that you should use %s and not ? as placeholder in your string:
SQL_insert = f"INSERT INTO table_name ({','.join(list(column))}) VALUES ({' %s,'*(n-1)} %s);"
Then the output string will be 'INSERT INTO table_name (SalesNo,ProductID,StoreID,PurchaseTime,Date,StoreNumber,PurchaseValue) VALUES ( %s, %s, %s, %s, %s, %s, %s);', which is compatible with operation = operation % tuple(tlist) replacement

Psycopg2 - Inserting complex query with strings + numbers? [duplicate]

I have a tuple as below
data = ({'weather station name': 'Substation', 'wind': '6 km/h', 'barometer': '1010.3hPa', 'humidity': '42%', 'temperature': '34.5 C', 'place_id': '001D0A00B36E', 'date': '2016-05-10 09:48:58'})
I am trying to push the values from the above tuple to the postgres table using the code below:
try:
con = psycopg2.connect("dbname='WeatherForecast' user='postgres' host='localhost' password='****'")
cur = con.cursor()
cur.executemany("""INSERT INTO weather_data(temperature,humidity,wind,barometer,updated_on,place_id) VALUES (%(temperature)f, %(humidity)f, %(wind)f, %(barometer)f, %(date)s, %(place_id)d)""", final_weather_data)
ver = cur.fetchone()
print(ver)
except psycopg2.DatabaseError as e:
print('Error {}'.format(e))
sys.exit(1)
finally:
if con:
con.close()
Where datatype of each field in the DB is as below:
id serial NOT NULL,
temperature double precision NOT NULL,
humidity double precision NOT NULL,
wind double precision NOT NULL,
barometer double precision NOT NULL,
updated_on timestamp with time zone NOT NULL,
place_id integer NOT NULL,
When i run the code to push the data into postgres table using psycopg2, it is raising an error "ValueError: unsupported format character 'f'"
I hope the issue is in formatting. Am using Python3.4

Have a look at the documentation:
The variables placeholder must always be a %s, even if a different placeholder (such as a %d for integers or %f for floats) may look more appropriate:
>>> cur.execute("INSERT INTO numbers VALUES (%d)", (42,)) # WRONG
>>> cur.execute("INSERT INTO numbers VALUES (%s)", (42,)) # correct
While, your SQL query contains all type of placeholders:
"""INSERT INTO weather_data(temperature,humidity,wind,barometer,updated_on,place_id)
VALUES (%(temperature)f, %(humidity)f, %(wind)f, %(barometer)f, %(date)s, %(place_id)d)"""

confused about dictionary created variables

# -*- coding: utf-8 -*-
states = {
'oregon': 'OR',
'florida': 'FL',
'california': 'CA',
'new york': 'NY',
'michigan': 'MI'
}
cities = {
'CA': 'san francisco',
'MI': 'detroit',
'FL': 'jacksonville'
}
cities['NY'] = 'new york'
cities['OR'] = 'portland'
for state, abbrev in states.items(): # add two variables
print "%s is abbreviated %s" % (state, abbrev)
print '-' * 10
for abbrev, city in cities.items():
print "%s has the city %s" % (abbrev, city)
print '-' * 10
for state, abbrev in states.items(): # this is what i'm confusing about
print "%s state is abbreviated %s and has city %s" % (state, abbrev, cities[abbrev])
I just want to know on the questionable line, there are only two variables inputed(state & abbrev), why there could be three variables quoted(state & abbrev & cities[abbrev])?
My guess is that 'abbrev' is being used twice, once in states dict and once in cities dict. So cities[abbrev] means to return the second value of each paired things? Can someone please confirm if my guess is correct?
If that's the case, why do I get a keyerror when I change cities[abbrev] into cities[state]? The error code is: KeyError: 'california'. It should return the first value of each pair.
I am confused on how this is working, can you please help me find a way out?

In your case, states.items() iterates of the key-value pairs, e.g. ('oregon', 'OR'). state will be 'oregon' and abbrev will be 'OR'. What cities[abbrev] does is finding the value of 'OR' in the dictionary cities. In the case of 'OR' that is 'portland'.
If you tried a value that is not in the keys of the dictionary, e.g. banana, then Python would throw a KeyError because the value banana is not a key in that dictionary.
To ensure that a key is present in a dictionary, you can use the in operator.
for state, abbrev in states.items():
# if the value of abbrev (e.g. 'OR') is a value in the cities dictionary
# we know that there is a city in this state. Print it.
if abbrev in cities:
print "%s state is abbreviated %s and has city %s" % (state, abbrev, cities[abbrev])
else:
print "%s is abbreviated %s" % (state, abbrev)

How to fix multiple error inserting in Python?

I use the following query:
cur.executemany(sql, data)
conn.commit()
Where data is:
[('ON-LOOK', 'ONLOOK KOREA', 'master#on-look.co.kr', 'Entertainment', '1.287', 'USK: All ages', 'https://www.google.com/url', 'https://ama.com', 'December 13, 2016', '//lh3.googleusercontent.com/bZrL9zRTXujVSFsNS_vD8AomApZeP2xgI1vGTsOz_zvoVKOz5T9pk7nFB3ONXXf04rqj=w300', '100 - 500', '4.7', '6', '4.3 and up')]
And sql is:
sql = """insert into apps_global (app_name, app_developer, app_email, app_category, app_version, app_age, app_website, app_url, app_developer_url, app_updated, app_image, app_installations, app_score, app_score_count, app_android \
) values ('%s', '%s', '%s', '%s', '%s', '%s', '%s', '%s', '%s', '%s', '%s', '%s', '%s', '%s', '%s')"""
When I try to execute query I get error:
ProgrammingError(1064, "1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'ON-LOOK'', ''ONLOOK KOREA'', ''master#on-look.co.kr'', ''Entertainment'', ''1.28' at line 1", u'42000'))
How to fix it?

MySQL INSERT fails in Python but works fine in MySQL Workbench

Here is a query I have that runs fine in MySQL workbench with the included sample values and works fine if I manually plug in the values in the code, but fails when I use the values as parameters. Any ideas?
Python Code:
print player
cur.execute("""
INSERT INTO scoredata
(gameid, playerid, starter, pos, min, fgm, fga, tpm, tpa, ftm, fta, oreb, reb, ast, stl, blk, tos, pf, pts)
VALUES
(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s);
"""), (player[0],
int(player[20]),
int(player[19]),
player[3],
int(player[4]),
int(player[5]),
int(player[6]),
int(player[7]),
int(player[8]),
int(player[9]),
int(player[10]),
int(player[11]),
int(player[12]),
int(player[13]),
int(player[14]),
int(player[15]),
int(player[16]),
int(player[17]),
int(player[18]) )
db.commit()
Error message:
['330060130', 103, 'Roy Devyn Marble', 'G-F', '28', '4', '9', '3', '6', '3', '3', '0', '2', '1', '0', '0', '0', '1', '14', 1, 1391]
Traceback (most recent call last):
File "C:\Users\jcaine\workspace\BasketballStats\src\BasketballStats\basketballstats.py", line 350, in <module>
insert_game_data('20130106', '20130106')
File "C:\Users\jcaine\workspace\BasketballStats\src\BasketballStats\basketballstats.py", line 284, in insert_game_data
"""), (player[0], int(player[20]), int(player[19]), player[3], int(player[4]), int(player[5]), int(player[6]), int(player[7]), int(player[8]), int(player[9]), int(player[10]), int(player[11]), int(player[12]), int(player[13]), int(player[14]), int(player[15]), int(player[16]), int(player[17]), int(player[18]) )
File "c:\users\jcaine\appdata\local\temp\easy_install-7_fysp\MySQL_python-1.2.3-py2.7-win32.egg.tmp\MySQLdb\cursors.py", line 174, in execute
File "c:\users\jcaine\appdata\local\temp\easy_install-7_fysp\MySQL_python-1.2.3-py2.7-win32.egg.tmp\MySQLdb\connections.py", line 36, in defaulterrorhandler
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)' at line 4")
MySQL scoredata Table Columns:
gameid varchar
playerid int
starter int
pos varchar
min int
fgm int
fga int
tpm int
tpa int
ftm int
fta int
oreb int
reb int
ast int
stl int
blk int
tos int
pf int
pts int
MySQL Code that runs fine in Workbench:
INSERT INTO scoredata (gameid, playerid, starter, pos, min, fgm, fga, tpm,
tpa, ftm, fta, oreb, reb, ast, stl, blk, tos, pf, pts)
VALUES ('3300601300', 1391, 1, 'G-F', 28, 4, 9, 3, 6, 3, 3, 0, 2, 1, 0, 0, 0, 1, 14)

You're not passing data to the execute call. Note the closing brace in your example.
cur.execute("""
INSERT INTO scoredata
(gameid, playerid, starter, pos, min, fgm, fga, tpm, tpa, ftm, fta, oreb, reb, ast, stl, blk, tos, pf, pts)
VALUES
(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s);
""")//*Remove me*
, (player[0],
int(player[20]),
int(player[19]),
player[3],
int(player[4]),
int(player[5]),
int(player[6]),
int(player[7]),
int(player[8]),
int(player[9]),
int(player[10]),
int(player[11]),
int(player[12]),
int(player[13]),
int(player[14]),
int(player[15]),
int(player[16]),
int(player[17]),
int(player[18]) )
db.commit()