it supposed to collect data from website and save it to database, it can collect data but cannot save it, I have tested connection between mysql is work fine, but it cannot be save. someone please help...
from bs4 import BeautifulSoup
from urllib import request
from urllib import parse
import MySQLdb
db_connection = MySQLdb.connect(host='localhost', db='database', user='root', passwd='123321')
cursor = db_connection.cursor()
url = "https://srh.bankofchina.com/search/whpj/search_cn.jsp"
Form_Data = {}
Form_Data['erectDate'] = ''
Form_Data['nothing'] = ''
Form_Data['pjname'] = '欧元'
data = parse.urlencode(Form_Data).encode('utf-8')
html = request.urlopen(url,data).read()
soup = BeautifulSoup(html,'html.parser')
div = soup.find('div', attrs = {'class':'BOC_main publish'})
table = div.find('table')
tr = table.find_all('tr')
td = tr[1].find_all('td')
c_name = td[0].text.strip()
c_updated = td[3].text.strip()
print(td[0].get_text(),td[3].get_text(),td[6].get_text())
sql = ("INSERT INTO currency_rate(c_name,c_updated)" "VALUES (%s,%s)" %(c_name,c_updated))
try:
cursor.execute(sql)
db_connection.commit()
except:
db_connection.rollback()
db_connection.close()
that is error message:
Traceback (most recent call last):
File "D:\Python\Python38\test1.py", line 33, in
cursor.execute(sql)
File "D:\Python\Python38\lib\site-packages\MySQLdb\cursors.py", line 191, in execute
query = query.encode(db.encoding)
File "D:\Python\Python38\lib\encodings\cp1252.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError: 'charmap' codec can't encode characters in position 52-53: character maps to
Looks like you have a unicode decode/encode error. Have a look at the docs on the subject.
From what I can see on the traceback, it seems that your database uses an encoding that do not support some of ythe characters you are trying to save.
Try change the encoding of your database (also called collation in MySql world) or encode your strings into something that your db can accept before executing the query.
p.s. : it also seem that your sql is a little malformed, looks like you are missing a couple apexes:
try modify this line:
sql = ("INSERT INTO currency_rate(c_name,c_updated)" "VALUES (%s,%s)" %(c_name,c_updated))
into:
sql = ("INSERT INTO currency_rate(c_name,c_updated)" "VALUES ('%s','%s')" %(c_name,c_updated))
Related
I have read many answers to this same question, but haven't seen anything that works for me.
I have a very basic app just to test things (in app.py and database.py). I'm using ElephantSQL, psycopg2-binary and have checked - the table on ElephantSQL is made from this code, but cannot insert from here. I can obviously insert from ElephantSQL.
I am just testing with the headline string, but headline will ultimately be the result of a web scrape using bs4.
This is my first time using a database with Python, but I'm absolutely blocked on what should be quite simple code. Please help.
This is database.py:
from psycopg2.extras import execute_values
CREATE_HEADLINES = """CREATE TABLE IF NOT EXISTS headlines
(headline TEXT);"""
INSERT_HEADLINE = "INSERT INTO headlines (headline) VALUES %s;"
SELECT_ALL_HEADLINES = "SELECT * FROM headlines;"
def create_tables(connection):
with connection:
with connection.cursor() as cursor:
cursor.execute(CREATE_HEADLINES)
def create_headlines(connection):
with connection:
with connection.cursor() as cursor:
cursor.execute(CREATE_HEADLINES)
def add_headline(connection, headline):
with connection:
with connection.cursor() as cursor:
cursor.execute(INSERT_HEADLINE, (headline))
def get_headlines(connection):
with connection:
with connection.cursor() as cursor:
cursor.execute(SELECT_ALL_HEADLINES)
return cursor.fetchall()
this is the main part of my app.py (I've hidden sensitive stuff):
import psycopg2
from bs4 import BeautifulSoup
import database
connection = psycopg2.connect(db_url)
headline = 'Hello, world!'
print(headline)
print(type(headline))
database.create_tables(connection)
database.create_headlines(connection)
database.add_headline(connection, headline)
Returns this error:
Hello, world!
<class 'str'>
Traceback (most recent call last):
File "/Users/jamesdanielmalvern/femicide/scraper/app.py", line 33, in <module>
database.add_headline(connection, headline)
File "/Users/jamesdanielmalvern/femicide/scraper/database.py", line 21, in add_headline
cursor.execute(INSERT_HEADLINE, (headline))
TypeError: not all arguments converted during string formatting
An INSERT needs around the values parenthesis, that is also true for placejholders
INSERT_HEADLINE = "INSERT INTO headlines (headline) VALUES (%s);
And You need also two elents in a list when you insert, if you have only 1 you need to add and empty one
cursor.execute(INSERT_HEADLINE, (headline,))
I am getting
File "getweather.py", line 15 SyntaxError: unexpected EOF while
parsing
I retyped it into a new completely new file. But am still seeing the same issue. Can anyone see what could be causing it?
from urlib.request import urlopen
import json
import psycopg2
import psycopg2.extras
try:
db = psycopg2.connect("dbname=myapp user=postgres password=postgres")
cursor = db.cursor()
query = "select city from myapp"
cursor.execute(query)
data = cursor.fetchall()
for row in data:
print (row)
db.close()
You can't have a try clause without an accompanying except; that's the cause of your error.
See the docs on Handling Exceptions.
I have a MySQL table, with XML content stored in a longtext field, encoded as utf8mb4_general_ci
Database Table
I want to use a Python script to read in the XML data from the transcript field, modify an element, and then write the value back to the database.
When I try to get the XML content into an Element using ElementTree.tostring I get the following encoding error:
Traceback (most recent call last):
File "ImageProcessing.py", line 33,
in <module> root = etree.fromstring(row[1])
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1300,
in XML parser.feed(text)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etre e/ElementTree.py", line 1640,
in feed self._parser.Parse(data, 0)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2014' in position 9568: ordinal not in range(128)
Code:
import datetime
import mysql.connector
import xml.etree.ElementTree as etree
# Creates the config parameters, connects
# to the database and creates a cursor
config = {
'user': 'username',
'password': 'password',
'host': '127.0.0.1',
'database': 'dbname',
'raise_on_warnings': True,
'use_unicode': True,
'charset': 'utf8',
}
cnx = mysql.connector.connect(**config)
cursor = cnx.cursor()
# Structures the SQL query
query = ("SELECT * FROM transcription")
# Executes the query and fetches the first row
cursor.execute(query)
row = cursor.fetchone()
while row is not None:
print(row[0])
#Some of the things I have tried to resolve the encoding issue
#parser = etree.XMLParser(encoding="utf-8")
#root = etree.fromstring(row[1], parser=parser)
#row[1].encode('ascii', 'ignore')
#Line where the encoding error is being thrown
root = etree.fromstring(row[1])
for img in root.iter('img'):
refno = img.text
img.attrib['href']='http://www.link.com/images.jsp?doc=' + refno
print img.tag, img.attrib, img.text
row = cursor.fetchone()
cursor.close()
cnx.close()
You've got everything well setup and your database connection is returning Unicodes, which is a good thing.
Unfortunately, ElementTree's fromstring() requires a byte str not a Unicode. This is so ElementTree can decode it using the encoding defined in the XML header.
You need to use this instead:
utf_8_xml = row[1].encode("utf-8")
root = etree.fromstring(utf_8_xml)
i neeed a python script to generate a csv file from my database XXXX. i wrote thise script but i have something wrong :
import mysql.connector
import csv
filename=open('test.csv','wb')
c=csv.writer(filename)
cnx = mysql.connector.connect(user='XXXXXXX', password='XXXXX',
host='localhost',
database='XXXXX')
cursor = cnx.cursor()
query = ("SELECT `Id_Vendeur`, `Nom`, `Prenom`, `email`, `Num_magasin`, `Nom_de_magasin`, `Identifiant_Filiale`, `Groupe_DV`, `drt_Cartes`.`gain` as 'gain', `Date_Distribution`, `Status_Grattage`, `Date_Grattage` FROM `drt_Cartes_Distribuer`,`drt_Agent`,`drt_Magasin`,`drt_Cartes` where `drt_Cartes_Distribuer`.`Id_Vendeur` = `drt_Agent`.`id_agent` AND `Num_magasin` = `drt_Magasin`.`Numero_de_magasin` AND `drt_Cartes_Distribuer`.`Id_Carte` = `drt_Cartes`.`num_carte`")
cursor.execute(query)
for Id_Vendeur, Nom, Prenom, email, Num_magasin, Nom_de_magasin, Identifiant_Filiale, Groupe_DV, gain, Date_Distribution, Status_Grattage, Date_Grattage in cursor:
c.writerow([Id_Vendeur, Nom, Prenom, email, Num_magasin, Nom_de_magasin, Identifiant_Filiale, Groupe_DV, gain, Date_Distribution, Status_Grattage, Date_Grattage] )
cursor.close()
filename.close()
cnx.close()
when i executing the command on phpmyadmin its look working very well but from my shell i got thise message :
# python test.py
Traceback (most recent call last):
File "test.py", line 18, in <module>
c.writerow([Id_Vendeur, Nom, Prenom, email, Num_magasin, Nom_de_magasin, Identifiant_Filiale, Groupe_DV, gain, Date_Distribution, Status_Grattage, Date_Grattage] )
UnicodeEncodeError: 'ascii' codec can't encode character u'\xeb' in position 5: ordinal not in range(128)
It looks you are using csv for Python 2.7. Quoting docs:
Note This version of the csv module doesn’t support Unicode input. Also, there are currently some issues regarding ASCII NUL characters. Accordingly, all input should be UTF-8 or printable ASCII to be safe; see the examples in section Examples.
Options, choice one of them:
Follow doc link, go to samples section, and modify your code accordantly.
Use a csv packet with unicode supprt like https://pypi.python.org/pypi/unicodecsv
Your data from the database are not only ascii characteres. I suggest you use the 'unicodecvs' python module as suggested in the answer to this question: How to write UTF-8 in a CSV file
i need help with variables in python, where they will be taken during run time, if i give the actual table name in the select query it works but when i am trying variable it fails.
import cx_Oracle
import csv
import sys
tablename = sys.argv[1] # pasing it as input at runtime
host = 'server.com'
port = 7111
SERVICE_NAME = 'gtpts'
login = 'USER'
passwrd = 'password'
SID = 'server.com'
dsn = cx_Oracle.makedsn(host, port, SID).replace('SID','SERVICE_NAME')
con = cx_Oracle.connect(login, passwrd, dsn)
cur = con.cursor()
cur.execute('select * from table') # direct table name works but variable doesnot
row = cur.fetchall()
Getting error as below:
[20020663]:20020663> python oracleconnect.py employee
Traceback (most recent call last):
File "oracleconnect.py", line 16, in <module>
cur.execute('select * from tablename')
cx_Oracle.DatabaseError: ORA-00911: invalid character
I could do the same by using $variable in perl do we have anything in python for same.
This uses the Oracle Bind Variable syntax, e.g.:
my_tablename = "SomeTableName"
cur = connection.cursor()
sql = "SELECT * FROM :tablename"
param = {"tablename": my_tablename}
cur.execute(sql, param)
row = cur.fetchall()
Note that the parameters here are passed as a dict and Oracle DB API will handle the escaping and quoting of variables, such as your table name. This method, as you can read about in the above link, is also much faster than doing the "hard parse" of the string.
Be careful not to use the string formatting operator %, like so:
cur.execute('SELECT * FROM %s' % my_tablename)
because it does not do any escaping or quoting, and it's prone to things like SQL injection, (not to mention it's less performant).
Thanks for this post. this has helped me to develop parameter driven connection strings.
following parameters are passed when the python script is run, after that python logs me into the database and I can perform my inserts.
import cx_Oracle
import csv
import sys
host_args = sys.argv[1]
port_args = sys.argv[2]
SID_args = sys.argv[3]
login_args = sys.argv[4]
passwrd_args = sys.argv[5]
data_dir = sys.argv[6]
dsn = cx_Oracle.makedsn(host_args, port_args, SID_args).replace('SID','SERVICE_NAME')
con = cx_Oracle.connect(login_args, passwrd_args, dsn)
The following command, I type on the command line to run this python script.
python /dera/ordsDataBranch/nsar/Load_NSARs_to_Oracle.py "myservername" "1521" "devrf1" "hr" "password2016" "/dera/ordsDataBranch/nsar/data"