I am trying to update my mysql database field with a concatenation. I have to read my file line by line, and i need to append the existing string with the loaded line. I have to do it like this because my goal is to insert a 3gb long whitespace separated text file into one longtext field, and mysql only capable of handling 1gb text to insert.
The problem with my code is that if i add the field name to the concat function like seq=concat(seq, %s) I get a SQL syntax error, but when I add the field name as a variable, python acts like it's a string.
So short story long with this input file:
aaa
bbb
ccc
I want to have an updated mysql field like this:
aaabbbccc
But I get this: seqccc
Any idea how should i work with the fieldname to get this work?
import mysql.connector
connection = mysql.connector.connect(host='localhost',
database='sys',
user='Pannka',
password='???')
cursor = connection.cursor()
with open('test.txt', 'r') as f:
for line in f:
sql = "update linedna set seq=concat(%s, %s) where id=1"
val=('seq', line.rstrip())
print(line.rstrip())
cursor.execute(sql, val)
connection.commit()
cursor.close()
connection.close()
f.close()
print(0)
I think that you want:
sql = "update linedna set seq = concat(seq, %s) where id=1"
val=(line.rstrip())
cursor.execute(sql, val)
connection.commit()
This will append each new line at the end of the already existing database value in column seq.
Related
I want to use sqlite3 to deal with data in Ubuntu with python. But I always failed and get errors. Codes related to database are as follows:
sqlite = "%s.db" % name
#connnect to the database
conn = sqlite3.connect(sqlite)
print "Opened database successfully"
c = conn.cursor()
#set default separator to "\t" in database
c.execute(".separator "\t"")
print "Set separator of database successfully"
#create table data_node
c.execute('''create table data_node(Time int,Node Text,CurSize int,SizeVar int,VarRate real,Evil int);''')
print "Table data_node created successfully"
node_info = "%s%s.txt" % (name,'-PIT-node')
c.execute(".import %\"s\" data_node") % node_info
print "Import to data_node successfully"
#create table data_face
data_info = "%s%s.txt" % (name,'-PIT-face')
c.execute('''create table data_face(Time int,Node Text,TotalEntry real,FaceId int,FaceEntry real,Evil int);''')
c.execute(".import \"%s\" data_face") % face_info
#get the final table : PIT_node
c.execute('''create table node_temp as select FIRST.Time,FIRST.Node,ROUND(FIRST.PacketsRaw/SECOND.PacketsRaw,4) as SatisRatio from tracer_temp FIRST,tracer_temp SECOND WHERE FIRST.Time=SECOND.Time AND FIRST.Node=SECOND.Node AND FIRST.Type='InData' AND SECOND.Type='OutInterests';''')
c.execute('''create table PIT_node as select A.Time,A.Node,B.SatisRatio,A.CurSize,A.SizeVar,A.VarRate,A.Evil from data_node A,node_temp B WHERE A.Time=B.Time AND A.Node=B.Node;''')
#get the final table : PIT_face
c.execute('''create table face_temp as select FIRST.Time,FIRST.Node,FIRST.FaceId,ROUND(FIRST.PacketsRaw/SECOND.PacketsRaw,4) as SatisRatio,SECOND.Packets from data_tracer FIRST,data_tracer SECOND WHERE FIRST.Time=SECOND.Time AND FIRST.Node=SECOND.Node AND FIRST.FaceId=SECOND.FaceId AND FIRST.Type='OutData' AND SECOND.Type='InInterests';''')
c.execute('''create table PIT_face as select A.Time,A.Node,A.FaceId,B.SatisRatio,B.Packets,ROUND(A.FaceEntry/A.TotalEntry,4),A.Evil from data_face as A,face_temp as B WHERE A.Time=B.Time AND A.Node=B.Node AND A.FaceId = B.FaceId;''')
conn.commit()
conn.close()
These sql-commands are right. When I run the code, it always shows sqlite3.OperationalError: near ".": syntax error. So how to change my code and are there other errors in other commands such as create table?
You have many problems in your code as posted, but the one you're asking about is:
c.execute(".separator "\t"")
This isn't valid Python syntax. But, even if you fix that, it's not valid SQL.
The "dot-commands" are special commands to the sqlite3 command line shell. It intercepts them and uses them to configure itself. They mean nothing to the actual database, and cannot be used from Python.
And most of them don't make any sense outside that shell anyway. For example, you're trying to set the column separator here. But the database doesn't return strings, it returns row objects—similar to lists. There is nowhere for a separator to be used. If you want to print the rows out with tab separators, you have to do that in your own print statements.
So, the simple fix is to remove all of those dot-commands.
However, there is a problem—at least one of those dot-commands actually does something:
c.execute(".import %\"s\" data_node") % node_info
You will have to replace that will valid calls to the library that do the same thing as the .import dot-command. Read what it does, and it should be easy to understand. (You basically want to open the file, parse the columns for each row, and do an executemany on an INSERT with the rows.)
I am attempting to write a simple python script to import from a text file to a mysql database, and encounter a perplexing error
Windows 10, Mysql 5.7.18, Python 3.6, pymysql
The contents of the text file:
nickname|fullname|cell|email|updatedt
andrew|Andrew Jones|+12395551172|arj#domain.com|2017-05-04 13:26:10
laurelai|Laurelai Smith||lsmith#domain.net|2017-05-04 13:27:47
I read in the data to construct a sql string:
insert into contacts (nickname,fullname,cell,email,updatedt) values(%,%,%,%,%)
The field values to be inserted are read in as follows:
['andrew', 'Andrew Jones', '+12395551172', 'arj#domain.com', '2017-05-04 13:26:10']
This is of course a Python list object. I have tried converting it to a tuple, but with same result
The routine to insert the row into the table is as follows:
def insert(sql, values):
#insert a single row of data from the input file
connection = getconn()
with connection.cursor() as cursor:
try:
cursor.execute(sql, values)
#except ValueError:
#print('Value error from pymysql')
finally:
cursor.close()
The following ValueError is returned:
ValueError: unsupported format character ',' (0x2c) at index 69
if, however, I extract the data values and insert them into the sqlstring by concatenation, I get:
insert into contacts (nickname,fullname,cell,email,updatedt) values('laurelai','Laurelai Smith','','lsmith#domain.net','2017-05-04 13:27:47')
This inserts the rows without error
What causes the ValueError?
Change your insert query to %s instead of %:
insert into contacts (nickname,fullname,cell,email,updatedt) values(%s,%s,%s,%s,%s)
Refer to doc.
I have the following python code, it reads through a text file line by line and takes characters x to y of each line as the variable "Contract".
import os
import pyodbc
cnxn = pyodbc.connect(r'DRIVER={SQL Server};CENSORED;Trusted_Connection=yes;')
cursor = cnxn.cursor()
claimsfile = open('claims.txt','r')
for line in claimsfile:
#ldata = claimsfile.readline()
contract = line[18:26]
print(contract)
cursor.execute("USE calms SELECT XREF_PLAN_CODE FROM calms_schema.APP_QUOTE WHERE APPLICATION_ID = "+str(contract))
print(cursor.fetchall())
When including the line cursor.fetchall(), the following error is returned:
Programming Error: Previous SQL was not a query.
The query runs in SSMS and replace str(contract) with the actual value of the variable results will be returned as expected.
Based on the data, the query will return one value as a result formatted as NVARCHAR(4).
Most other examples have variables declared prior to the loop and the proposed solution is to set NO COUNT on, this does not apply to my problem so I am slightly lost.
P.S. I have also put the query in its own standalone file without the loop to iterate through the file in case this was causing the problem without success.
In your SQL query, you are actually making two commands: USE and SELECT and the cursor is not set up with multiple statements. Plus, with database connections, you should be selecting the database schema in the connection string (i.e., DATABASE argument), so TSQL's USE is not needed.
Consider the following adjustment with parameterization where APPLICATION_ID is assumed to be integer type. Add credentials as needed:
constr = 'DRIVER={SQL Server};SERVER=CENSORED;Trusted_Connection=yes;' \
'DATABASE=calms;UID=username;PWD=password'
cnxn = pyodbc.connect(constr)
cur = cnxn.cursor()
with open('claims.txt','r') as f:
for line in f:
contract = line[18:26]
print(contract)
# EXECUTE QUERY
cur.execute("SELECT XREF_PLAN_CODE FROM APP_QUOTE WHERE APPLICATION_ID = ?",
[int(contract)])
# FETCH ROWS ITERATIVELY
for row in cur.fetchall():
print(row)
cur.close()
cnxn.close()
I'm using the dbfpy module to read data from DBF files with the intention of writing that same data to equivalent MySQL tables. Here's a rough version of my code:
###################
# BEGIN CONFIG
###################
import_root = '/Users/shawn/Dropbox/ITistic Inc/Digital Aspire/Clients/MVP/Automated Sales Report Project/pdq dbf export 1.30.2013'
concept_id = 1
###################
# END CONFIG
###################
import os
import datetime
from dbfpy import dbf
import MySQLdb
# Connect to MySQL
db = MySQLdb.connect('localhost', 'dbposireporting', 'posi', 'dbposireporting')
cur = db.cursor()
discount = dbf.Dbf(os.path.join(path, 'DISCOUNT.DBF'))
for rec in discount:
print rec['date']
print
# LINE BELOW NOT WORKING:
cur.execute("INSERT INTO discount VALUES (%s, %s, %s, %s, %s, %s)", rec)
discount.close()
db.close()
The MySQL table I'm trying to insert into contains one additional field which I need to populate with the concept_id value set at the top of the script. That value is not part of the DBF records (rec). What's the best way for me to insert this data?
I would think you could replace rec in the cur.execute() line with
tuple(rec) + (concept_id,)
Oh, and don't forget to add one more %s for it.
If any of the fields are strings, you'll need to wrap them in quotes for the SQL to be valid.
I have created a database with MySQLdb.
In database I have a table with name student with columns:
id(is int),
id_user(is int),
f_name(is str),
l_name(is str)
I want to update a row.
My code is below:
db=mdb.connect(host="localhost", use_unicode="True", charset="utf8",
user="", passwd="", db="test")
# prepare a cursor object using cursor() method
cursor = db.cursor()
sql="""SELECT id_user FROM student"""
try:
# Execute the SQL command
cursor.execute(sql)
# Commit your changes in the database
db.commit()
except:
# Rollback in case there is any error
db.rollback()
rows = cursor.fetchall()
the=int(7)
se=str('ok')
for row in rows:
r=int(row[0])
if r==the:
sql2 = """UPDATE student
SET f_name=%s
WHERE id_user = %s"""% (se,the)
# Execute the SQL command
cursor.execute(sql2)
# Commit your changes in the database
db.commit()
db.rollback()
# disconnect from server
db.close()
When I run it I take the error there is column with name ok why?
Can anyone help me find what I am doing wrong please?
str doesn't wrap its argument in quotation marks, so your statement is this:
UPDATE student SET f_name=ok WHERE id_user = 7
when it needs to be this:
UPDATE student SET f_name='ok' WHERE id_user = 7
So, either change this line:
SET f_name=%s
to this:
SET f_name='%s'
or else change this line:
se=str('ok')
to this:
se="'" + str('ok') + "'"
Though I recommend reading about SQL injection, which will become a concern as soon as you start using user-supplied data instead of hard-coded values.
You should run the query like this:
sql2 = """UPDATE student
SET f_name = %s
WHERE id_user = %s"""
cursor.execute(sql2, (se, the))
Don't use string interpolation, let the database driver handle passing the parameters for you. Otherwise you have to deal with syntax errors like this, or worse, SQL injection.
More details here.
You should always enclose your data with quotes.
Instead of %s solely use '%s' the only types you dont need it are numeric ones, but even there i would enclose %d with '%d' cos it is more save.
And you should use at least db.escape_string(your_data) before inserting or updating same values into your database.
Or have a look at the pdo-using style of mysqldb:
http://mysql-python.sourceforge.net/MySQLdb.html#some-examples
c=db.cursor()
max_price=5
c.execute("""SELECT spam, eggs, sausage FROM breakfast
WHERE price < %s""", (max_price,))