Develop Reference

Python Mysql call fails silently (MariaDB)

Trying to figure out what is happening during Mysql Insert statement.
I have a utility I am writing and am attempting to open my database (Django database in Mysql) and add a record from another database into it. I think I am including all the fields it needs, and If I actually cut and paste the Insert statement that gets generated - it works.
It however does not work programmatically. It doesn't seem to generate an error. The last insert row seems to indicate success, but the actual record never gets there (I think it is rolling it back for some reason). Just don't see what the problem is. I successfully use the same cursor to check something else in the same database right before this, so the cursor should be good.
Below is the insert code in python.
create_string = """Insert INTO trackx_site_program
SET air_date = '%s',
air_time = '%s',
service = '%s',
block_time = '%s',
block_time_delta = %d,
running_time = '%s',
running_time_delta = %d,
remaining_time = '%s',
remaining_time_delta = %d,
title = '%s',
locked_flag = %d,
deleted_flag = %d,
library = '%s',
mc = '%s',
producer = '%s',
editor = '%s',
remarks = '%s',
audit_time = '%s',
audit_user = 'todd' """ % (
air_date, air_time, service_name, block_time, block_time_delta,
running_time, running_time_delta, remaining_time, remaining_time_delta,
title, locked_flag, deleted_flag, library, mc, producer, editor,
remarks, audit_time)
print(" Create String = \n %s" % create_string)
num_rows = new_trackx_cursor.execute(create_string)
print ("Num_rows from execute = %s" % num_rows)
new_program_id = new_trackx_cursor.lastrowid
print("Last Row ID Inserted was %s " % new_program_id)
new_trackx_cursor.close()
sys.exit("Exiting after insert")
An example of the string is below:
Create String =
Insert INTO trackx_site_program
SET air_date = '2001-06-13',
air_time = '18:00:00',
service = 'TheService',
block_time = '0:57:00',
block_time_delta = 3420000000,
running_time = '00:00:00',
running_time_delta = 0,
remaining_time = '0:57:00',
remaining_time_delta = 3420000000,
title = 'My061301',
locked_flag = 1,
deleted_flag = 0,
library = 'K061301-PM',
mc = 'ToddS',
producer = 'TheProducer',
editor = 'theEditor',
remarks = 'REGULAR PROGRAM',
audit_time = '2001-06-13 10:55:16',
audit_user = 'toadyb'
Num_rows from execute = 1
Last Row ID Inserted was 22
Exiting after insert
The Actual Database Table looks like this:
desc tracks_site_program;
+----------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| air_date | date | NO | | NULL | |
| air_time | time(6) | NO | | NULL | |
| service | varchar(10) | NO | | NULL | |
| block_time | time(6) | NO | | NULL | |
| block_time_delta | bigint(20) | NO | | NULL | |
| running_time | time(6) | NO | | NULL | |
| running_time_delta | bigint(20) | NO | | NULL | |
| remaining_time | time(6) | NO | | NULL | |
| remaining_time_delta | bigint(20) | NO | | NULL | |
| title | varchar(190) | NO | | NULL | |
| locked_flag | tinyint(1) | NO | | NULL | |
| locked_expiration | datetime(6) | YES | | NULL | |
| deleted_flag | tinyint(1) | NO | | NULL | |
| library | varchar(190) | YES | | NULL | |
| mc | varchar(64) | NO | | NULL | |
| producer | varchar(64) | NO | | NULL | |
| editor | varchar(64) | NO | | NULL | |
| remarks | longtext | YES | | NULL | |
| audit_time | datetime(6) | NO | | NULL | |
| audit_user | varchar(32) | YES | | NULL | |
+----------------------+--------------+------+-----+---------+----------------+
21 rows in set (0.00 sec)
As you can see I print out the actual string, and if I cut and paste into a mysql session, it works just fine.
Anybody know what gives here? Is there a step I am missing?
Thanks

Not quite sure why, but my connections in Python were all preceded with autocommit=0 (False), so my transaction were automatically rolling back. I did a commit on the database connection handle after the insert, and the transaction completed successfully.

I am trying to import data from excel into MySQL using Python

I have created a small excel file to try this process
+----------+-------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra |
+----------+-------------+------+-----+---------+-------+ | S no | int(11) | NO | PI | NULL | | | Order | int(11)
| YES | | NULL | | | Price | int(11) | YES |
| NULL | | | comments | var char(45) | YES | | NULL
| |
+----------+-------------+------+-----+---------+-------+ 4 rows in set (1.21 sec)
This is the code I am using in python for the same
import xlrd import pymysql book =
xlrd.open_workbook(r'C:\Users\hp\Desktop\excel-python-mysql.xlsx')
sheet = book.sheet_by_index(0)
# Connect to the database
connection = pymysql.connect(host='localhost',
user='root',
password='*****',
db='trial',
charset='utf8mb4',
cursorclass=pymysql.cursors.DictCursor)
cursor = connection.cursor()
query = """INSERT INTO demo(Sno,Order,Price,Comments)VALUES(%d,%d,%d,%s)"""
for r in range(1,sheet.nrows):
Sno = sheet.cell(r,0).value
Order = sheet.cell(r,1).value
Price = sheet.cell(r,2).value
Comments = sheet.cell(r,3).value
values = (Sno,Order,Price,Comments)
cursor.execute(query,values)
cursor.close()
connection.commit()
connection.close()
but I am facing error stating "%d format: a number is required, not str" I want to move the data to MySQL for further use it at Metabase.

Consume data from KAFKA topic and extract fields from it and store in MySQL using python

I want to consume data from a Kafka topic with the following command as follows:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myTestTopic --from-beginning
Then this will output the following (just pasting top 2 lines output, but it will be many lines...):
&time=1561768216000&gameCategory=PINPOINT&category=ONE&uniqueId=2518Z-0892A-0030O-16H70&transactionType=CRD&familyId=000-222-115-11119&realTs=1561768319000&sortId=1&msg=SET-UP+PRAYER+%26+intercession+begins+in+just+30+minutes.&remoteIpAddress=127.0.0.1&userAgent=HTTP&
&uniqueId=872541806296826880&time=1571988786000&gameCategory=NOTIFY&category=TWO&transactionType=CRD&familyId=401-222-115-89387&sortId=1&realTs=1571988989000&msg=This-is+a+reminder.&remoteIpAddress=127.0.0.1&userAgent=HTTPS&
I want to consume the following from the output:
realTs
familyId
msg
uniqueId
and you can see that each element is seperated by an ampersand ('&'). They are not always in same index/place so I'm not sure if I need a regex? Eventually when I do the query on a local running MySQL, i'd see this:
describe testTable;
+----------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------+--------------+------+-----+---------+-------+
| realTs | bigint(20) | YES | | NULL | |
| familyId | varchar(255) | YES | | NULL | |
| msg | text | YES | | NULL | |
| uniqueId | varchar(255) | YES | | NULL | |
+----------+--------------+------+-----+---------+-------+
4 rows in set (0.00 sec)
SELECT * FROM testTable;
+---------------+-------------------+-----------------------------------------------------------+-------------------------+
| realTs | familyId | msg | uniqueId |
+---------------+-------------------+-----------------------------------------------------------+-------------------------+
| 1561768319000 | 000-222-115-11119 | SET-UP+PRAYER+%26+intercession+begins+in+just+30+minutes. | 2518Z-0892A-0030O-16H70 |
| 1571988989000 | 401-222-115-89387 | This-is+a+reminder. | 872541806296826880 |
+---------------+-------------------+-----------------------------------------------------------+-------------------------+
What do I have so far?
I have a mysql-connector with python where I can connect to a local mysql etc, but i'm struggling with parsing this and inserting it...

With Python, you can use urllib.parse.parse_qs to retrieve URL query string components in a Python dictionary that you can iterate later to insert data in your MySQL database.
For instance:
from urllib.parse import parse_qs
line = "&time=1561768216000&gameCategory=PINPOINT&category=ONE&uniqueId=2518Z-0892A-0030O-16H70&transactionType=CRD&familyId=000-222-115-11119&realTs=1561768319000&sortId=1&msg=SET-UP+PRAYER+%26+intercession+begins+in+just+30+minutes.&remoteIpAddress=127.0.0.1&userAgent=HTTP&uniqueId=872541806296826880&time=1571988786000&gameCategory=NOTIFY&category=TWO&transactionType=CRD&familyId=401-222-115-89387&sortId=1&realTs=1571988989000&msg=This-is+a+reminder.&remoteIpAddress=127.0.0.1&userAgent=HTTPS&"
o = parse_qs(line)
print(o)
Result:
{'time': ['1561768216000', '1571988786000'], 'gameCategory': ['PINPOINT', 'NOTIFY'], 'category': ['ONE', 'TWO'], 'uniqueId': ['2518Z-0892A-0030O-16H70', '872541806296826880'], 'transactionType': ['CRD', 'CRD'], 'familyId': ['000-222-115-11119', '401-222-115-89387'], 'realTs': ['1561768319000', '1571988989000'], 'sortId': ['1', '1'], 'msg': ['SET-UP PRAYER & intercession begins in just 30 minutes.', 'This-is a reminder.'], 'remoteIpAddress': ['127.0.0.1', '127.0.0.1'], 'userAgent': ['HTTP', 'HTTPS']}

Python MySQLdb: combine 2 tables,select data as dict. I am confused about the dict's key

I have two tables:user and post
and the structures of them are:
post:
+---------+----------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------+----------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| name | char(30) | YES | | NULL | |
| user_id | int(11) | YES | | NULL | |
+---------+----------+------+-----+---------+----------------+
user:
+---------+----------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------+----------+------+-----+---------+----------------+
| user_id | int(11) | NO | PRI | NULL | auto_increment |
| name | char(30) | YES | | NULL | |
| email | char(30) | YES | | NULL | |
+---------+----------+------+-----+---------+----------------+
i get this:(keys of data dict)
['post.user_id', 'user_id', 'name', 'email', 'post.name', 'id']
my python code is:
import MySQLdb
import MySQLdb.cursors
con = MySQLdb.connect(user = "root", passwd = "123456", db = "mydb", cursorclass=MySQLdb.cursors.DictCursor)
cur = con.cursor()
cur.execute("select * from user, post where user.user_id = post.user_id")
print cur.fetchone().keys()
but,why the keys of data dict is that? thanks. My English is not so well,excuse me

When you select *, you ask for all columns in both user and post. Since user and post have columns with overlapping names, the tablename is added before a few of them, to create unique keys.
I'm not sure what you were expecting, but you can explicitly control the keys you get by giving the columns aliases:
"select user.user_id as user_id, post.name as post_name, user.name as user_name ..."

Having trouble insert data to table with default values

I have a table that has 4 attributes.
+--------------+--------------+------+-----+-------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+--------------+------+-----+-------------------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| url | varchar(255) | YES | | NULL | |
| mail | varchar(255) | YES | | NULL | |
| date_entered | timestamp | NO | | CURRENT_TIMESTAMP | |
| active | tinyint(1) | NO | | 1 | |
+--------------+--------------+------+-----+-------------------+----------------+
Now i want to insert only the data_entered and other attributes to get default values.
I'm doing it for the id field which I need to be exact to another id I insereted to different table.
this is the code:
tx.execute(\
"insert into company_career (date_entered) "
"values (%s)",
(time.time())
)
This is the error:
query = query % db.literal(args)
exceptions.TypeError: not all arguments converted during string formatting
How to fix it?

Try this:
tx.execute(\
"insert into company_career (date_entered) "
"values (FROM_UNIXTIME('%s'))",
(time.time())
)

when you are providing values into a insert query dont forget to enclode it in commas
"insert into company_career (date_entered) values ('%s')"

I did manage to insert with: insert into company(url,mail,date_entered) values("","","now");