Develop Reference

SQL concatenating with python

I am trying to update my mysql database field with a concatenation. I have to read my file line by line, and i need to append the existing string with the loaded line. I have to do it like this because my goal is to insert a 3gb long whitespace separated text file into one longtext field, and mysql only capable of handling 1gb text to insert.
The problem with my code is that if i add the field name to the concat function like seq=concat(seq, %s) I get a SQL syntax error, but when I add the field name as a variable, python acts like it's a string.
So short story long with this input file:
aaa
bbb
ccc
I want to have an updated mysql field like this:
aaabbbccc
But I get this: seqccc
Any idea how should i work with the fieldname to get this work?
import mysql.connector
connection = mysql.connector.connect(host='localhost',
database='sys',
user='Pannka',
password='???')
cursor = connection.cursor()
with open('test.txt', 'r') as f:
for line in f:
sql = "update linedna set seq=concat(%s, %s) where id=1"
val=('seq', line.rstrip())
print(line.rstrip())
cursor.execute(sql, val)
connection.commit()
cursor.close()
connection.close()
f.close()
print(0)

I think that you want:
sql = "update linedna set seq = concat(seq, %s) where id=1"
val=(line.rstrip())
cursor.execute(sql, val)
connection.commit()
This will append each new line at the end of the already existing database value in column seq.

INSERT WHERE NOT EXISTS clause using python variables

I am attempting to write an SQL query to insert rows into a table when the script is run. I have been able to get it working whilst duplicating data, but am unable to set up an insert that can determine whether or not the record is unique.
I have tried following these threads:
Most efficient way to do a SQL 'INSERT IF NOT EXISTS'
Python MySQLdb / MySQL INSERT IGNORE & Checking if Ignored
Currently, I can get the system to add all records, but due to the frequency the script is ran at the database is being filled with the same records. This is my current code:
for post in r.get_subreddit("listentothis").get_new(limit=5):
if 'youtube.com' in post.url or 'youtu.be' in post.url:
print(str(post.title))
print(str(post.url))
sql = """INSERT INTO listentothis2(title,url) VALUES ('%s', '%s') WHERE NOT EXISTS (SELECT 1 FROM listentothis2 WHERE url = '%s') """ % (post.title, post.url, post.url)
try:
cursor.execute(sql)
db.commit()
except:
db.rollback()
I have also tried:
sql = """INSERT IGNORE INTO listentothis2(title,url) VALUES ('%s', '%s') """ % (post.title, post.url)
However, this adds the 3 most recent records whether or not they are commited to the database already. Any help is greatly appreciated!

how to populate database using python faker

I am trying to populate a table using python faker and I am getting this error . Here is my code
import psycopg2
from faker import Faker
fake = Faker()
conn = psycopg2.connect(database="testdb", user="****", password="****", host="127.0.0.1", port="5432")
print "Opened database successfully"
cur = conn.cursor()
for i in range (10):
Id =fake.random_digit_not_null()
name = fake.name()
age=fake.random_number(digits=None)
adress =fake.address()
salary = fake.random_int(min=0, max=9999)
cur.execute("INSERT INTO COMPANY (ID,NAME,AGE,ADDRESS,SALARY) \
VALUES (Id,name,age,adress,salary)");
conn.commit()
print "Records created successfully";
conn.close()
here is the error
Traceback (most recent call last):
File "fakegenerator.py", line 16, in <module>
VALUES (Id,name,age,adress,salary)");
psycopg2.ProgrammingError: column "id" does not exist
LINE 1: ...OMPANY (ID,NAME,AGE,ADDRESS,SALARY) VALUES (Id,name,ag...
^
HINT: There is a column named "id" in table "company", but it cannot be referenced from this part of the query.

You're not filling in the values into your query, instead you're sending the string as-is to the database. This would actually fill your query with values:
cur.execute("INSERT INTO COMPANY (ID,NAME,AGE,ADDRESS,SALARY) VALUES (%s, %s, %s, %s, %s)", (Id, name, age, adress, salary));
This wraps the variables filled with the values you want to insert into a tuple and let's psycopg2 handle quoting your strings correctly which is less work for you and keeps you safe from SQL injection, should you use your code as a base for productive code. This is documented in the module's documentation.

the sql request in cur.execute seems to be a problem try this
cur.execute("INSERT INTO COMPANY (ID,NAME,AGE,ADDRESS,SALARY) \
VALUES ({},{},{},{},{})".format(Id,name,age,adress,salary));

How do I insert data into table?

I have created table using this create command as:
CREATE TABLE test_table(id INT PRIMARY KEY,name
VARCHAR(50),price INT)
i want to insert into this table wherein values are stored already in variable
bookdb=# name = 'algorithms'
bookdb-# price = 500
bookdb-# INSERT INTO test_table VALUES(1,'name',price);
I get the following error:
ERROR: syntax error at or near "name"
LINE 1: name = 'algorithms'
Can anyone point out the mistake and propose solution for the above?
Thanks in advance
Edit:
import psycopg2
import file_content
try:
conn = psycopg2.connect(database='bookdb',user='v22')
cur = conn.cursor()
cur.execute("DROP TABLE IF EXISTS book_details")
cur.execute("CREATE TABLE book_details(id INT PRIMARY KEY,name VARCHAR(50),price INT)")
cur.execute("INSERT INTO book_details VALUES(1,'name',price)")
conn.commit()
except:
print "unable to connect to db"
I have used the above code to insert values into table,variables name and price containing the values to be inserted into table are available in file_content python file and i have imported that file.The normal INSERT statement takes values manually but i want my code to take values which are stored in variables.

SQL does not support the concept of variables.
To use variables, you must use a programming language, such as Java, C, Xojo. One such language is PL/pgSQL, which you can think of as a superset of SQL. PL/PgSQL is often bundled as a part of Postgres installers, but not always.
I suggest you read some basic tutorials on SQL.
See this similar question: How do you use script variables in PostgreSQL?

don't have postgres installed here, but you can try this
import psycopg2
import file_content
try:
conn = psycopg2.connect(database='bookdb',user='v22')
cur = conn.cursor()
cur.execute("DROP TABLE IF EXISTS book_details")
cur.execute("CREATE TABLE book_details(id INT PRIMARY KEY,name VARCHAR(50),price INT)")
cur.execute("INSERT INTO book_details VALUES(1, '%s', %s)" % (name, price))
conn.commit()
except:
print "unable to connect to db"

If you are using PSQL console:
\set name 'algo'
\set price 10
insert into test_table values (1,':name',:price)
\g

Error message in python-mysql cursor: 1054 unknown column "x" in 'field list'

This is my first post! I also just started programming, so please bear with me!
I am trying to load a bunch of .csv files into a database, in order to later perform various reports on the data. I started off by creating a few tables in mysql with matching field names and data types to what will be loaded into the tables. I am manipulating the filename (in order to parse out the date to use as a field in my table) and cleaning up the data with python.
So my problem right now (haha...) is that I get this error message when I attempt the 'Insert Into' query to mysql.
Traceback (most recent call last):
File "C:\Program Files\Python\load_domains2.py", line 80, in <module>
cur.execute(sql)
File "C:\Program Files\Python\lib\site-packages\MySQLdb\cursors.py", line 166, in execute
self.errorhandler(self, exc, value)
File "C:\Program Files\Python\lib\site-packages\MySQLdb\connections.py", line 35, in defaulterrorhandler
raise errorclass, errorvalue
OperationalError: (1054, "Unknown column 'a1200e.com' in 'field list'")
'a1200e.com' refers to a specific domain name I'm inserting into that column. My query is as follows:
sql="""INSERT INTO temporary_load
(domain_name, session_count, search_count, click_count,
revenue, revenue_per_min, cost_per_click, traffic_date)
VALUES (%s, %d, %d, %d, %d, %d, %d, %s)""" %(cell[0],
int(cell[1]),
int(cell[2].replace (",","")),
int(cell[3].replace(",","")),
float(cell[4].replace("$","")),
float(cell[5].replace("$","")),
float(cell[6].replace("$","")),
parsed_date)
cur.execute(sql)
I am very new at all this, so I'm sure my code isn't at all efficient, but I just wanted to lay everything out so it's clear to me. What I don't understand is that I have ensured my table has correctly defined data types (corresponding to those in my query). Is there something I'm missing? I've been trying to work this out for a while, and don't know what could be wrong :/
Thanks so much!!!
Val

Thomas is, as usual, absolutely correct: feel free to let MySQLdb handle the quoting issues.
In addition to that recommendation:
The csv module is your friend.
MySQLdb uses the "format" parameter style as detailed in PEP 249.
What does that mean for you?
All parameters, whatever type, should be passed to MySQLdb as strings (like this %s). MySQLdb will make sure that the values are properly converted to SQL literals.
By the way, MySQLdb has some good documentation.
Feel free to include more detail about your source data. That may make diagnosing the problem easier.
Here's one way to insert values to a MySQL database from a .csv file:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import csv
import MySQLdb
import os
def main():
db = MySQLdb.connect(db="mydb",passwd="mypasswd",) # connection string
filename = 'data.csv'
f = open(filename, "rb") # open your csv file
reader = csv.reader(f)
# assuming the first line of your csv file has column names
col_names = reader.next() # first line of .csv file
reader = csv.DictReader(f, col_names) # apply column names to row values
to_db = [] # this list holds values you will insert to db
for row in reader: # loop over remaining lines in .csv file
to_db.append((row['col1'],row['col2']))
# or if you prefer one-liners
#to_db = [(row['col1'],row['col2']) for row in reader]
f.close() # we're done with the file now
cursor = db.cursor()
cursor.executemany('''INSERT INTO mytable (col1,col2)
VALUES (%s, %s)''', to_db) # note the two arguments
cursor.close()
db.close()
if __name__ == "__main__":
main()

You should be using DB-API quoting instead of including the data in the SQL query directly:
sql = """INSERT INTO temporary_load
(domain_name, session_count, search_count, click_count,
revenue, revenue_per_min, cost_per_click, traffic_date)
VALUES (%s, %d, %d, %d, %d, %d, %d, %s)"""
args = (cell[0],
int(cell[1]),
int(cell[2].replace (",","")),
int(cell[3].replace(",","")),
float(cell[4].replace("$","")),
float(cell[5].replace("$","")),
float(cell[6].replace("$","")),
parsed_date)
cur.execute(sql, args)
This makes the DB-API module quote the values appropriately, and resolves a whole host of issues that you might get when doing it by hand (and usually incorrectly.)