Develop Reference

Python can't insert multiple rows at once into a MySQL table

I am learning to use the MySQL Connector/Python. When I insert multiple rows together into the table using the code below, it shows an error. But it works when I insert only one row at at time.
Here's the python code I have written:
import mysql.connector as sql
mydb = sql.connect(host = "localhost", user = "root", passwd = "your_password", database = "testdb")
mycursor = mydb.cursor()
str = "insert into test (Name, Address) values (%s, %s)"
val = [
('Foster', 'Moonland'),
('Toblerone', 'Marstown'),
('Hershey', 'Neptunestadt'),
('Cadbury', 'Jupiter DC')
]
mycursor.execute(str, val)
mydb.commit()
This the error message:
Traceback (most recent call last):
File "C:\Users\shoun\AppData\Local\Programs\Python\Python37\lib\site-packages\mysql\connector\conversion.py", line 183, in to_mysql
return getattr(self, "_{0}_to_mysql".format(type_name))(value)
AttributeError: 'MySQLConverter' object has no attribute '_tuple_to_mysql'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\shoun\AppData\Local\Programs\Python\Python37\lib\site-packages\mysql\connector\cursor.py", line 432, in _process_params
res = [to_mysql(i) for i in res]
File "C:\Users\shoun\AppData\Local\Programs\Python\Python37\lib\site-packages\mysql\connector\cursor.py", line 432, in <listcomp>
res = [to_mysql(i) for i in res]
File "C:\Users\shoun\AppData\Local\Programs\Python\Python37\lib\site-packages\mysql\connector\conversion.py", line 186, in to_mysql
"MySQL type".format(type_name))
TypeError: Python 'tuple' cannot be converted to a MySQL type
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "d:\coding\python\mysql_connector\connectSQL.py", line 24, in <module>
mycursor.execute(str, val)
File "C:\Users\shoun\AppData\Local\Programs\Python\Python37\lib\site-packages\mysql\connector\cursor.py", line 557, in execute
psub = _ParamSubstitutor(self._process_params(params))
File "C:\Users\shoun\AppData\Local\Programs\Python\Python37\lib\site-packages\mysql\connector\cursor.py", line 437, in _process_params
"Failed processing format-parameters; %s" % err)
mysql.connector.errors.ProgrammingError: Failed processing format-parameters; Python 'tuple' cannot be converted to a MySQL type
But if val is a single tuple, the code executes without producing any error.

How to import CSV files to SQLite3 with python

This is my code.
import sqlite3
import pandas
db = sqlite3.connect('testdb.db')
df = pandas.read_csv('testcsv.csv')
df.to_sql('testTable', 'db', if_exists='append', index=False)
I got the last two lines of code from another article on stackoverflow, but it doesn't work for me. This is the error I get, even after I installed sqlalchemy, because it complained that it wasn't installed.
Traceback (most recent call last):
File "C:/Users/pitye/PycharmProjects/gradeCalcV2/venv/sqlite.py", line 7, in <module>
df.to_sql('testTable', 'db', if_exists='append', index=False)
File "C:\Users\pitye\PycharmProjects\gradeCalcV2\venv\lib\site-packages\pandas\core\generic.py", line 2663, in to_sql
method=method,
File "C:\Users\pitye\PycharmProjects\gradeCalcV2\venv\lib\site-packages\pandas\io\sql.py", line 503, in to_sql
pandas_sql = pandasSQL_builder(con, schema=schema)
File "C:\Users\pitye\PycharmProjects\gradeCalcV2\venv\lib\site-packages\pandas\io\sql.py", line 577, in pandasSQL_builder
con = _engine_builder(con)
File "C:\Users\pitye\PycharmProjects\gradeCalcV2\venv\lib\site-packages\pandas\io\sql.py", line 564, in _engine_builder
con = sqlalchemy.create_engine(con)
File "C:\Users\pitye\PycharmProjects\gradeCalcV2\venv\lib\site-packages\sqlalchemy\engine\__init__.py", line 479, in create_engine
return strategy.create(*args, **kwargs)
File "C:\Users\pitye\PycharmProjects\gradeCalcV2\venv\lib\site-packages\sqlalchemy\engine\strategies.py", line 54, in create
u = url.make_url(name_or_url)
File "C:\Users\pitye\PycharmProjects\gradeCalcV2\venv\lib\site-packages\sqlalchemy\engine\url.py", line 229, in make_url
return _parse_rfc1738_args(name_or_url)
File "C:\Users\pitye\PycharmProjects\gradeCalcV2\venv\lib\site-packages\sqlalchemy\engine\url.py", line 291, in _parse_rfc1738_args
"Could not parse rfc1738 URL from string '%s'" % name
sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string 'db'
I just want to create a table from a CSV file in SQLite. Is this even the right way of doing it, or am I waaay off?

I think you just have to replace
df.to_sql('testTable', 'db', if_exists='append', index=False)
With
df.to_sql('testTable', db, if_exists='append', index=False)

Cannot access an sqlite3 database file with python

I'm trying to take recipe hyperlinks from a pre-existing file and add them to a table in a database. I have already created the database and given the file name to set up a connection and insert the data into the table but whenever I do so I get the error sqlite3.OperationalError: unable to open database file.
Here's my code:
import bs4, os, requests, time
import sqlite3
from flask import current_app, g
def create_connection(db_file):
db = sqlite3.connect(db_file)
c = db.cursor()
return db
def get_html():
'''
Get the BBC Food sitemap and save it to a local file.
'''
page = None
db = create_connection("pantry/instance/flaskr.sqlite")
for attempt in range(1, 4):
print("line 40")
page = requests.get('http://www.bbc.co.uk/food/sitemap.xml')
try:
page.raise_for_status()
break
except requests.RequestException:
time.sleep(attempt * 10)
if not page:
raise Exception('Failed to get sitemap.xml')
sitemap = bs4.BeautifulSoup(page.text, 'html.parser')
# Write the recipe urls to a text file
print("line 53")
for line in sitemap.find_all('loc'):
for string in line.stripped_strings:
if string.startswith('https://www.bbc.co.uk/food/recipes/'):
print("line 57")
recipeUrl = string
if (
db.execute("SELECT recipeID FROM recipe WHERE weblink = ?", (recipeUrl,)).fetchone()
is not None
):
error = "recipe weblink {0} is already inputted.".format(recipeUrl)
if error is None:
db.execute(
'INSERT INTO recipe (weblink) VALUES (?)',
recipeUrl
)
db.commit()
db.close()
And the error message:
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "C:\Program Files\JetBrains\PyCharm 2019.2.2\helpers\pydev\_pydev_bundle\pydev_umd.py", line 197, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File "C:\Program Files\JetBrains\PyCharm 2019.2.2\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "C:/Users/Eva Morris/PycharmProjects/pantry/flaskr/BBCscraper/scraperecipes.py", line 161, in <module>
get_html()
File "C:/Users/Eva Morris/PycharmProjects/pantry/flaskr/BBCscraper/scraperecipes.py", line 34, in get_html
db = create_connection("pantry/instance/flaskr.sqlite")
File "C:/Users/Eva Morris/PycharmProjects/pantry/flaskr/BBCscraper/scraperecipes.py", line 23, in create_connection
db = sqlite3.connect(db_file)
sqlite3.OperationalError: unable to open database file

Based on the consoles error report, your .py file is located in: C:/Users/Eva Morris/PycharmProjects/pantry/flaskr/BBCscraper/scraperecipes.py".
In your code you set your databases directory using the relative path as:pantry/instance/flaskr.sqlite.
This means python is looking for the directory: C:/Users/Eva Morris/PycharmProjects/pantry/flaskr/BBCscraper/pantry/instance/ to create/link your database file in.
If that is not a preexisting directory, sqlite3.connect will not be able to create your database flaskr.sqlite.
You may need to update the code db = create_connection("pantry/instance/flaskr.sqlite") to something like db = create_connection("flaskr.sqlite") which should work without any problem since it will just create the database in your working directory.

getting error with execute many in python when i tried to connect to DB

I am trying to use functions with mysql and python and I am getting errors:
I am reading a file cnn.cvs and I want to insert in a table noticias but I have errors in the code. Here I share the code:
import csv
import MySQLdb
mydb = MySQLdb.connect(host='localhost',
user='root',
passwd='password',
db='cnn')
cursor = mydb.cursor()
f = open('cnn.csv', 'r')
csv_data = csv.reader(f)
for row in csv_data:
cursor.execute('INSERT INTO noticias(title, \
link, pubDate )' \
'VALUES("%s", "%s", "%s")',
row)
#close the connection to the database.
mydb.commit()
cursor.close()
print ("Done")
and when i execute this is the result:
C:\Users\SoriyAntony\AppData\Local\Programs\Python\Python36-32\python.exe
"C:\Program Files\JetBrains\PyCharm Community Edition
2017.1.4\helpers\pydev\pydevd.py" --multiproc --qt-support --client 127.0.0.1 --port 59726 --file C:/Users/SoriyAntony/PycharmProjects/cnnbd/cnnbd
pydev debugger: process 10368 is connecting
Connected to pydev debugger (build 171.4694.38)
Traceback (most recent call last):
File "C:\Users\SoriyAntony\AppData\Local\Programs\Python\Python36-
32\lib\site-packages\MySQLdb\cursors.py", line 238, in execute
query = query % args
TypeError: not enough arguments for format string
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files\JetBrains\PyCharm Community Edition
2017.1.4\helpers\pydev\pydevd.py", line 1591, in <module>
globals = debugger.run(setup['file'], None, None, is_module)
File "C:\Program Files\JetBrains\PyCharm Community Edition
2017.1.4\helpers\pydev\pydevd.py", line 1018, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "C:\Program Files\JetBrains\PyCharm Community Edition
2017.1.4\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "C:/Users/SoriyAntony/PycharmProjects/cnnbd/cnnbd", line 17, in
<module>
row)
File "C:\Users\SoriyAntony\AppData\Local\Programs\Python\Python36-
32\lib\site-packages\MySQLdb\cursors.py", line 240, in execute
self.errorhandler(self, ProgrammingError, str(m))
File "C:\Users\SoriyAntony\AppData\Local\Programs\Python\Python36-
32\lib\site-
packages\MySQLdb\connections.py", line 52, in defaulterrorhandler
raise errorclass(errorvalue)
_mysql_exceptions.ProgrammingError: not enough arguments for format string
Process finished with exit code 1
I am using python 3.6, any idea of the ?
UPDATE:
This issue is resolved:
I use
cursor.executemany()
for this and its work.

The problem is that each row read from the csv file is returned as a list of strings for csv.reader, you have to convert the list row to tuple, change your code to:
cursor.execute('INSERT INTO noticias(title, \
link, pubDate )' \
'VALUES("%s", "%s", "%s")',
tuple(row))
Update:
Make sure there is table noticias in your mysql db, for example:
create table noticias (title Varchar(255),link varchar(255),pubDate varchar(255))
your cnn.csv file contains empty line, when read it, there will be empty list, you have to drop it, change your for loop like this:
for row in csv_data:
if len(row)!=0:
cursor.execute('INSERT INTO noticias(title, link, pubDate ) VALUES("%s", "%s", "%s")',tuple(row))

pymssql.operational Error

I am using python and pymssql for SQL connection.
Basically i am fetching some data from 'table1' and doing some
Mathematical calculation and inserting into 'table2'
I have tested my python script with limited number of
records from 'table1' and it was working fine.
Then I have started my script with real time data,ie really huge
amount of data(122457 records)
Then I have faced given below issue
Traceback (most recent call last):
File "trakryt_main_mod.py", line 312, in <module>
create_visitwalkby_table()
File "trakryt_main_mod.py", line 126, in create_visitwalkby_table
scenarios.one_beacon(cursor,orgid)
File "C:\Nikhil\trak_reporting\python\scenarios.py", line 260, in one_beacon
close_visit()
File "C:\Nikhil\trak_reporting\python\scenarios.py", line 207, in close_visit
conn = pymssql.connect(server,user,password,db,port=port)
File "pymssql.pyx", line 645, in pymssql.connect (pymssql.c:10301)
File "pymssql.pyx", line 247, in pymssql.Connection.__init__ (pymssql.c:3771)
pymssql.OperationalError: Cannot start transaction: (20004, 'DB-Lib error messag
e 20004, severity 9:\nRead from the server failed\nNet-Lib error during Unknown
error (10054)\n')