Develop Reference

PyMySql Column Truncated and Duplicate Index Error

Here is my table creation code:
CREATE TABLE `crypto_historical_price2` (
`Ticker` varchar(255) COLLATE latin1_bin NOT NULL,
`Timestamp` varchar(255) COLLATE latin1_bin NOT NULL,
`PerpetualPrice` double DEFAULT NULL,
`SpotPrice` double DEFAULT NULL,
`Source` varchar(255) COLLATE latin1_bin NOT NULL,
PRIMARY KEY (`Ticker`,`Timestamp`,`Source`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_bin
I'm updating stuff in batch with sql statements like the following
sql = "INSERT INTO crypto."+TABLE+"(Ticker,Timestamp,PerpetualPrice,SpotPrice,Source) VALUES %s;" % batchdata
where batchdata is just a string of data in "('SOL', '2022-11-03 02:01:00', '31.2725', '31.2875', 'FTX'),('SOL', '2022-11-03 02:02:00', '31.3075', '31.305', 'FTX')
Now my script runs for a bit of time successfully inserting data in to the table but then it barfs with the following errors:
error 1265 data truncated for column PerpetualPrice
and
Duplicate entry 'SOL-2022-11-02 11:00:00-FTX' for key 'primary'
I've tried to solve the second error with
sql = "INSERT INTO crypto.crypto_historical_price2(Ticker,Timestamp,PerpetualPrice,SpotPrice,Source) VALUES %s ON DUPLICATE KEY UPDATE Ticker = VALUES(Ticker), Timestamp = VALUES(Timestamp), PerpetualPrice = VALUES(PerpetualPrice), SpotPrice = VALUES(SpotPrice), Source = VALUES(Source);" % batchdata
and
sql = "INSERT INTO crypto.crypto_historical_price2(Ticker,Timestamp,PerpetualPrice,SpotPrice,Source) VALUES %s ON DUPLICATE KEY UPDATE Ticker = VALUES(Ticker),Timestamp = VALUES(Timestamp),Source = VALUES(Source);" % batchdata
The above 2 attempted remedy runs and doesn't throw an duplicate entry error but it doesn't update the table at all.
If I pause my script a couple of minutes and re-run, the error duplicate error goes away and it updates which even confuses me EVEN more lol.
Any ideas?

Error Inserting records into mariadb table using Python : 1366 (22007): Incorrect integer value: '%s' for column

I am inserting records in maria db table from a file using python. Population column in the file is empty. I want it to go as empty value in table as well. Population column in table is set as integer and can accept null value. I am trying the below code -
Table Definition -
CREATE TABLE local_db.table_x (
Unique_code varchar(50) NOT NULL,
city varchar(200) DEFAULT NULL,
state varchar(50) DEFAULT NULL,
population bigint(20) DEFAULT NULL,
Govt varchar(50) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
input_file = "input_file"
csv_data = csv.reader(open(input_file))
try:
connection = mysql.connector.connect(host='localhost',
database='local_db',
user='root',
password='root',
port = '3306')
cursor = connection.cursor()
for row in csv_data:
cursor.execute("""
INSERT INTO table_x(Unique_code,city,state,population,Govt)
VALUES("%s", "%s", "%s","%s", "%s")
ON DUPLICATE KEY UPDATE city = VALUES(city),state = VALUES(state), \
population = VALUES(population),Govt = VALUES(Govt)""")
connection.commit()
print(cursor.rowcount, "Record inserted successfully into table_x")
cursor.close()
except mysql.connector.Error as error:
print("Failed to insert record into table_x table {}".format(error))
finally:
if (connection.is_connected()):
connection.close()
print("MySQL connection is closed")
But I am getting below error -
Failed to insert record into table_x table 1366 (22007): Incorrect integer value: '%s' for column local_db.table_x.population at row 1
MySQL connection is closed
In other thread it was suggested to change
SET sql_mode = ""
But its not an option for me since I would be running it on organization server which I can not change for this only. Please suggest what code changes I can do here to handle this situation.

The population column is a bigint(20) DEFAULT NULL & hence you cannot provide the value "%s" format it should be a integer which is being passed.
Also on values front.
It can have NULL
Any integer value.
So incase you want it to be null & not empty string("") you can skip to insert value to population column all together by removing it from the insert command
INSERT INTO table_x(Unique_code,city,state,Govt)
VALUES("%s", "%s", "%s","%s")
ON DUPLICATE KEY UPDATE city = VALUES(city),state = VALUES(state), Govt = VALUES(Govt)""")

Assuming that each row of your CSV has 5 values, corresponding to the code, city, state, population, and government, in that order, you should be using this syntax for the insert query:
for row in csv_data:
params = row.split(",")
sql = """
INSERT INTO table_x (Unique_code, city, state, population, Govt)
VALUES(?, ?, ?, ?, ?)
ON DUPLICATE KEY UPDATE city = VALUES(city), state = VALUES(state),
population = VALUES(population),Govt = VALUES(Govt)"""
cursor.execute(sql, params)
The first parameter to execute() should be the SQL prepared statement, and the second parameter should be a tuple. In this case, it should be a tuple with 5 values, corresponding to the five ? placeholders in the VALUES clause of your query. An example of a valid tuple here might be:
params = ('ABC123', 'New York', 'New York', 15000000, 'USA',)
Then call via:
cursor.execute(sql, params)

Mysql insert lock wait timeout exceeded - auto increment

I'm having an issue with my application causing MySQL table to be locked due to inserts which take a long time, after reviewing online articles, it seems like it's related to auto increment, info below -
Python that inserts data (row at a time unfortunately as I need the auto incremented id for reference in future inserts) -
for i, flightobj in stats[ucid]['flight'].items():
flight_fk = None
# Insert flights
try:
with mysqlconnection.cursor() as cursor:
sql = "insert into cb_flights(ucid,takeoff_time,end_time,end_event,side,kills,type,map_fk,era_fk) values(%s,%s,%s,%s,%s,%s,%s,%s,%s);"
cursor.execute(sql, (
ucid, flightobj['start_time'], flightobj['end_time'], flightobj['end_event'], flightobj['side'],
flightobj['killnum'], flightobj['type'], map_fk, era_fk))
mysqlconnection.commit()
if cursor.lastrowid:
flight_fk = cursor.lastrowid
else:
flight_fk = 0
except pymysql.err.ProgrammingError as e:
logging.exception("Error: {}".format(e))
except pymysql.err.IntegrityError as e:
logging.exception("Error: {}".format(e))
except TypeError as e:
logging.exception("Error: {}".format(e))
except:
logging.exception("Unexpected error:", sys.exc_info()[0])
The above runs every 2 minutes on the same data and is supposed to insert only non duplicates as the MySQL would deny duplicates due to the unique ucid_takeofftime index.
MYSQL info, cb_flights table -
`pk` int(11) NOT NULL AUTO_INCREMENT,
`ucid` varchar(50) NOT NULL,
`takeoff_time` datetime DEFAULT NULL,
`end_time` datetime DEFAULT NULL,
`end_event` varchar(45) DEFAULT NULL,
`side` varchar(45) DEFAULT NULL,
`kills` int(11) DEFAULT NULL,
`type` varchar(45) DEFAULT NULL,
`map_fk` int(11) DEFAULT NULL,
`era_fk` int(11) DEFAULT NULL,
`round_fk` int(11) DEFAULT NULL,
PRIMARY KEY (`pk`),
UNIQUE KEY `ucid_takeofftime` (`ucid`,`takeoff_time`),
KEY `ucid_idx` (`ucid`) /*!80000 INVISIBLE */,
KEY `end_event` (`end_event`) /*!80000 INVISIBLE */,
KEY `side` (`side`)
) ENGINE=InnoDB AUTO_INCREMENT=76023132 DEFAULT CHARSET=utf8;
Now inserts into the table from the Python code, can take sometimes over 60 seconds.
I beleive it might be related to the auto increment that is creating the lock on the table, if so, I'm looking for a workaround.
innodb info -
innodb_autoinc_lock_mode 2
innodb_lock_wait_timeout 50
buffer is used up to 70% more or less.
Appreciate any assistance with this, either from application side or MySQL side.
EDIT
Adding the create statement for the cb_kills table which is also used with inserts but without an issue as far as I can see, this is in response to the comment on the 1st answer.
CREATE TABLE `cb_kills` (
`pk` int(11) NOT NULL AUTO_INCREMENT,
`time` datetime DEFAULT NULL,
`killer_ucid` varchar(50) NOT NULL,
`killer_side` varchar(10) DEFAULT NULL,
`killer_unit` varchar(45) DEFAULT NULL,
`victim_ucid` varchar(50) DEFAULT NULL,
`victim_side` varchar(10) DEFAULT NULL,
`victim_unit` varchar(45) DEFAULT NULL,
`weapon` varchar(45) DEFAULT NULL,
`flight_fk` int(11) NOT NULL,
`kill_id` int(11) NOT NULL,
PRIMARY KEY (`pk`),
UNIQUE KEY `ucid_killid_flightfk_uniq` (`killer_ucid`,`flight_fk`,`kill_id`),
KEY `flight_kills_fk_idx` (`flight_fk`),
KEY `killer_ucid_fk_idx` (`killer_ucid`),
KEY `victim_ucid_fk_idx` (`victim_ucid`),
KEY `time_ucid_killid_uniq` (`time`,`killer_ucid`,`kill_id`),
CONSTRAINT `flight_kills_fk` FOREIGN KEY (`flight_fk`) REFERENCES `cb_flights` (`pk`)
) ENGINE=InnoDB AUTO_INCREMENT=52698582 DEFAULT CHARSET=utf8;

You can check if autocommit is set to 1, this forces to commit every row and disabling it makes it somewhat faster
Instead of committing every insert try to bulk insert.
For that you should check
https://dev.mysql.com/doc/refman/8.0/en/optimizing-innodb-bulk-data-loading.html
and do something like
data = [
('city 1', 'MAC', 'district 1', 16822),
('city 2', 'PSE', 'district 2', 15642),
('city 3', 'ZWE', 'district 3', 11642),
('city 4', 'USA', 'district 4', 14612),
('city 5', 'USA', 'district 5', 17672),
]
sql = "insert into city(name, countrycode, district, population)
VALUES(%s, %s, %s, %s)"
number_of_rows = cursor.executemany(sql, data)
db.commit()

I want to put in here some of the ways I worked on finding a solution to this problem. I'm not an expert in MySQL but I think these steps can help anyone looking to find out why he has lock wait timeouts.
So the troubleshooting steps I took are as follows -
1- Check if I can find in the MySQL slow log the relevant query that is locking my table. Usually it's possible to find queries that run a long time and also locks with the info below and the query right after it
# Time: 2020-01-28T17:31:48.634308Z
# User#Host: # localhost [::1] Id: 980397
# Query_time: 250.474040 Lock_time: 0.000000 Rows_sent: 10 Rows_examined: 195738
2- The above should give some clue on what's going on in the server and what might be waiting for a long time. Next I ran the following 3 queries to identify what is in use:
check process list on which process are running -
show full processlist;
check tables in use currently -
show open tables where in_use>0;
check running transactions -
SELECT * FROM `information_schema`.`innodb_trx` ORDER BY `trx_started`;
3- The above 2 steps should give enough information on which query is locking the tables. in my case here I had a SP that ran an insert into <different table> select from <my locked table>, while it was inserting to a totally different table, this query was locking my table due to the select operation that took a long time.
To work around it, I changed the SP to work with temporary tables and now although the query is still not completely optimized, there are no locks on my table.
Adding here how I run the SP on temporary tables for async aggregated updates.
CREATE DEFINER=`username`#`%` PROCEDURE `procedureName`()
BEGIN
drop temporary table if exists scheme.temp1;
drop temporary table if exists scheme.temp2;
drop temporary table if exists scheme.temp3;
create temporary table scheme.temp1 AS select * from scheme.live1;
create temporary table scheme.temp2 AS select * from scheme.live2;
create temporary table scheme.temp3 AS select * from scheme.live3;
create temporary table scheme.emptytemp (
`cName1` int(11) NOT NULL,
`cName2` varchar(45) NOT NULL,
`cName3` int(11) NOT NULL,
`cName4` datetime NOT NULL,
`cName5` datetime NOT NULL,
KEY `cName1` (`cName1`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
INSERT into scheme.emptytemp
select t1.x,t2.y,t3.z
from scheme.temp1 t1
JOIN scheme.temp2 t2
ON t1.x = t2.x
JOIN scheme.temp3 t3
ON t2.y = t3.y
truncate table scheme.liveTable;
INSERT into scheme.liveTable
select * from scheme.emptytemp;
END
Hope this helps anyone that encounters this issue

executemany() not inserting all records to database

I'm trying to use execute many to insert four (4) records at a time to a MySQL Database table, using MySQL Connector driver.
The problem is executemany() is consistently "skipping" the first 3 records without throwing an error and I can't understand what's wrong.
I'm trying to insert the values in a list:
my_records = [(334, 20533, 387.5, 'Label1'), (335, 20534, 387.5, 'Label2'), (336, 108659, 387.5, 'Label3'), (337, 108660, 387.5, 'Label4')]
And then here's my code:
try:
mydb = mysql.connector.connect(
host=os.getenv('DBM_HOST', 'x.x.x.x'),
user=os.getenv('DBM_USER', 'username'),
passwd=os.getenv('DBM_PASSWORD', 'password'),
database=os.getenv('DBM_NAME', 'my_database')
)
mycursor = mydb.cursor()
sql = """
INSERT INTO my_table (
batch_id, user_id, assessment, label
) VALUES (
%s, %s, %s, %s
)
"""
mycursor.executemany(sql, my_records)
mydb.commit()
mycursor.close()
mydb.close()
return True
except Exception as exception:
print(exception)
return None
Does anyone know what's happening? Or why the first 3 records are not inserting?
Here's the table structure:
Field Type Null Key Default Extra
id int(10) unsigned NO PRI auto_increment
batch_id int(10) unsigned NO MUL
user_id int(10) unsigned NO
assessment decimal(10,2) unsigned NO
label varchar(250) YES

How to get columns name in mysqldb with a Python 2.7?

If I am using select * from query it is working well, but when I am trying to query the columns name too, it isnt working (maybe because I have got a column called "FROM" but that's why i used 'FROM!?)
Here my code:
connection = MySQLdb.connect(host='localhost',
user='admin',
passwd='',
db='database1',
use_unicode=True,
charset="utf8")
cursor = connection.cursor()
query = """ select ACTUAL_TIME, 'FROM, ID
union all
select ACTUAL_TIME, FROM , ID
from TEST
into outfile '/tmp/test.csv'
fields terminated by ';'
enclosed by '"'
lines terminated by '\n';
"""
cursor.execute(query)
connection.commit()
cursor.close()
I get this error message:
raise errorvalue
_mysql_exceptions.OperationalError: (1054, "Unknown column 'ACTUAL_TIME' in 'field list'")
EDIT: SHOW CREATE TABLE TEST;
| TEST | CREATE TABLE `TEST` (
`ACTUAL_TIME` varchar(100) DEFAULT NULL,
`FROM` varchar(100) DEFAULT NULL,
`STATUS` varchar(100) DEFAULT NULL,
`ID` int(10) NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`ID`)
) ENGINE=InnoDB AUTO_INCREMENT=76287 DEFAULT CHARSET=utf8 |

try this :
connection = MySQLdb.connect(host='localhost',
user='admin',
passwd='',
db='database1',
use_unicode=True,
charset="utf8")
cursor = connection.cursor()
query = """ select 'ACTUAL_TIME', 'FROM', 'ID' -- add single quotes
union all
select `ACTUAL_TIME`, `FROM`, `ID` -- add here backtick in column names
from TEST
into outfile '/tmp/test.csv'
fields terminated by ';'
enclosed by '"'
lines terminated by '\n';
"""
cursor.execute(query)
connection.commit()
cursor.close()
or else you can use this to get column names "SHOW columns"
or :
cursor.execute("SELECT * FROM table_name LIMIT 0")
print cursor.description

columns = cursor.description
result = [{columns[index][0]:column for index, column in enumerate(value)} for value in cursor.fetchall()]
print(result)