I am following the link http://code.google.com/p/django-multilingual-model/ Basically i am trying to insert hindi characters into mysql db
I have created the files in the test directory as in the above said link and using django version 1.1 an dpython version is 2.4.3
I geta error message as the following.
from testlanguages import Language,BookTranslation,Book
>>> lang_pl = Language(code="pl", name="Polish")
>>> lang_pl.save()
>>> book_pl.language = lang_pl
>>> book_pl.save()
>>> lang_hi = Language(code="hi", name="Hindi")
>>> lang_hi.save()
>>> book_hi = BookTranslation()
>>> book_hi.title = "का सफल प्रक्षेपण किया है. इस राकेट ने पाँच" Sum characters as in bbc.co.uk/hindi
>>> book_hi.description = "का सफल प्रक्षेपण किया है. इस राकेट ने पाँच" Sum characters as in bbc.co.uk/hindi
>>> book_hi.model = book
>>> book_hi.language = lang_hi
>>> book_hi.save()
Traceback (most recent call last):
File "<console>", line 1, in ?
File "/opt/project/django/django/db/models/base.py", line 410, in save
self.save_base(force_insert=force_insert, force_update=force_update)
File "/opt/project/django/django/db/models/base.py", line 495, in save_base
result = manager._insert(values, return_id=update_pk)
File "/opt/project/django/django/db/models/manager.py", line 177, in _insert
return insert_query(self.model, values, **kwargs)
File "/opt/project/django/django/db/models/query.py", line 1087, in insert_query
return query.execute_sql(return_id)
File "/opt/project/django/django/db/models/sql/subqueries.py", line 320, in execute_sql
cursor = super(InsertQuery, self).execute_sql(None)
File "/opt/project/django/django/db/models/sql/query.py", line 2369, in execute_sql
cursor.execute(sql, params)
File "/opt/project/django/django/db/backends/util.py", line 19, in execute
return self.cursor.execute(sql, params)
File "/opt/project/django/django/db/backends/mysql/base.py", line 84, in execute
return self.cursor.execute(query, args)
File "/usr/lib/python2.4/site-packages/MySQLdb/cursors.py", line 165, in execute
self._warning_check()
File "/usr/lib/python2.4/site-packages/MySQLdb/cursors.py", line 80, in _warning_check
warn(w[-1], self.Warning, 3)
File "/usr/lib/python2.4/warnings.py", line 61, in warn
warn_explicit(message, category, filename, lineno, module, registry)
File "/usr/lib/python2.4/warnings.py", line 96, in warn_explicit
raise message
Warning: Incorrect string value: '\xE0\xA4\x95\xE0\xA4\xBE...' for column 'title' at row 1
How to resolve this.
Is there any easy method to insert the special characters into DB
Stop using bytestrings. Start using unicode.
book_hi.title = u"का सफल प्रक्षेपण किया है. इस राकेट ने पाँच"
conn = MySQLdb.connect( host = "localhost",
user = "root",
passwd = "",
db = "test");
conn.set_character_set('utf8')
this code will solve you problem
Note *. conn.set_character_set('utf8') is notable part
This is most likely a MySQL problem, which doesn't always support Unicode characters by default.
You should probably declare the DEFAULT CHARSET to utf8 and DEFAULT COLLATION to utf8_general_ci on your database (use the ALTER DATABASE or modify the CREATE DATABASE statements, as explained here).
Related
I am using pyfirebirdsql version 0.8.5 to connect to Interbase database.
import firebirdsql
conn = firebirdsql.connect(
host='192.168.133.121',
database='E:\\test\test.gdb',
port=3050,
user='sysdba',
password='masterkey'
#charset="WIN1251"
#charset="UTF8"
)
cur = conn.cursor()
cur.execute("select column1 from table1")
for c in cur.fetchall():
print(c)
conn.close()
When I try to run the script I get the following:
Traceback (most recent call last):
File "123.py", line 15, in <module>
for c in cur.fetchall():
File "/usr/lib/python2.7/site-packages/firebirdsql/fbcore.py", line 291, in fetchall
return [tuple(r) for r in self._fetch_records]
File "/usr/lib/python2.7/site-packages/firebirdsql/fbcore.py", line 225, in _fetch_generator
stmt_handle, self._xsqlda)
File "/usr/lib/python2.7/site-packages/firebirdsql/wireprotocol.py", line 728, in _op_fetch_response
return self._parse_op_response() # error occured
File "/usr/lib/python2.7/site-packages/firebirdsql/wireprotocol.py", line 210, in _parse_op_response
raise OperationalError(message, gds_codes, sql_code)
firebirdsql.OperationalError: arithmetic exception, numeric overflow, or string truncation
Cannot transliterate character between character sets
Adding charset both "WIN1251" and "UTF8" doesn't solve the issue. Do you have any ideas?
I have to read from a OrientDB. To test that everything works I tried to read from the Database with the SELECT Statement.
like this:
import pyorient
client = pyorient.OrientDB("adress", 2424)
session_id = client.connect("root", "password")
client.db_open("table","root","password")
print str(client.db_size())
client.query("SELECT * FROM L1_Req",1)
The Connection works fine and also the print str(client.db_size()) line.
But at client.query("SELECT * FROM L1_Req",1) it returns the following Error Message:
Traceback (most recent call last):
File "testpy.py", line 9, in <module>
client.query("SELECT * FROM L1_Req",1)
File "C:\app\tools\python27\lib\site-packages\pyorient\orient.py", line 470, i
n query
.prepare(( QUERY_SYNC, ) + args).send().fetch_response()
File "C:\app\tools\python27\lib\site-packages\pyorient\messages\commands.py",
line 144, in fetch_response
super( CommandMessage, self ).fetch_response()
File "C:\app\tools\python27\lib\site-packages\pyorient\messages\base.py", line
265, in fetch_response
self._decode_all()
File "C:\app\tools\python27\lib\site-packages\pyorient\messages\base.py", line
249, in _decode_all
self._decode_header()
File "C:\app\tools\python27\lib\site-packages\pyorient\messages\base.py", line
176, in _decode_header
serialized_exception = self._decode_field( FIELD_STRING )
File "C:\app\tools\python27\lib\site-packages\pyorient\messages\base.py", line
366, in _decode_field
_decoded_string = self._orientSocket.read( _len )
File "C:\app\tools\python27\lib\site-packages\pyorient\orient.py", line 164, i
n read
buf = bytearray(_len_to_read)
MemoryError
I also tried some ohter SQL Statements like:
client.query("SELECT subSystem FROM L1_Req",1)
I cant't get why this happends. Can you guys help me ?
import dataset
db = dataset.connect(....)
table = db[...]
When I try to insert some value into Mysql table, this error occurred.
Sample Value I am inserting to the table:
print("Buy", ticker, price, date, OType, OSize)
Buy AAPL 93.4357142857 2016-05-12 Market 200
data = dict(Order_Side='Buy',Ticker=ticker, Price=price,
Order_Date= date, Order_Type = OType, Volume = OSize )
table.insert(data)
error message:
Traceback (most recent call last):
File "<ipython-input-3-b7ab0c98f72f>", line 1, in <module>
runfile('C:/Users/SwigelUser/Documents/GitHub/Trading/Strategies/strat_MA_ExecTest.py', wdir='C:/Users/SwigelUser/Documents/GitHub/Trading/Strategies')
File "C:\Users\SwigelUser\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 880, in runfile
execfile(filename, namespace)
File "C:\Users\SwigelUser\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "C:/Users/SwigelUser/Documents/GitHub/Trading/Strategies/strat_MA_ExecTest.py", line 69, in <module>
MA_Stra(start_length=7,end_length=10,start_date=date(2016,5,12),end_date=date(2016,6,18))
File "C:/Users/SwigelUser/Documents/GitHub/Trading/Strategies/strat_MA_ExecTest.py", line 66, in MA_Stra
table.insert(data1)
File "C:\Users\SwigelUser\Anaconda3\lib\site-packages\dataset\persistence\table.py", line 87, in insert
res = self.database.executable.execute(self.table.insert(row))
File "C:\Users\SwigelUser\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 945, in execute
return meth(self, multiparams, params)
File "C:\Users\SwigelUser\Anaconda3\lib\site-packages\sqlalchemy\sql\elements.py", line 263, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "C:\Users\SwigelUser\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1053, in _execute_clauseelement
compiled_sql, distilled_params
File "C:\Users\SwigelUser\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1189, in _execute_context
context)
File "C:\Users\SwigelUser\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1405, in _handle_dbapi_exception
util.reraise(*exc_info)
File "C:\Users\SwigelUser\Anaconda3\lib\site-packages\sqlalchemy\util\compat.py", line 187, in reraise
raise value
File "C:\Users\SwigelUser\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1182, in _execute_context
context)
File "C:\Users\SwigelUser\Anaconda3\lib\site-packages\sqlalchemy\engine\default.py", line 470, in do_execute
cursor.execute(statement, parameters)
File "C:\Users\SwigelUser\Anaconda3\lib\site-packages\pymysql\cursors.py", line 164, in execute
query = self.mogrify(query, args)
File "C:\Users\SwigelUser\Anaconda3\lib\site-packages\pymysql\cursors.py", line 143, in mogrify
query = query % self._escape_args(args, conn)
File "C:\Users\SwigelUser\Anaconda3\lib\site-packages\pymysql\cursors.py", line 123, in _escape_args
return dict((key, conn.literal(val)) for (key, val) in args.items())
File "C:\Users\SwigelUser\Anaconda3\lib\site-packages\pymysql\cursors.py", line 123, in <genexpr>
return dict((key, conn.literal(val)) for (key, val) in args.items())
File "C:\Users\SwigelUser\Anaconda3\lib\site-packages\pymysql\connections.py", line 821, in literal
return self.escape(obj, self.encoders)
File "C:\Users\SwigelUser\Anaconda3\lib\site-packages\pymysql\connections.py", line 814, in escape
return escape_item(obj, self.charset, mapping=mapping)
File "C:\Users\SwigelUser\Anaconda3\lib\site-packages\pymysql\converters.py", line 27, in escape_item
val = encoder(val, mapping)
File "C:\Users\SwigelUser\Anaconda3\lib\site-packages\pymysql\converters.py", line 110, in escape_unicode
return u"'%s'" % _escape_unicode(value)
File "C:\Users\SwigelUser\Anaconda3\lib\site-packages\pymysql\converters.py", line 73, in _escape_unicode
return value.translate(_escape_table)
AttributeError: 'numpy.float64' object has no attribute 'translate'
What have caused this error? How could I solve this.
It seems like I entered too much code here,thus I will have to type lots of none sense in order to submit.
my price and date variable comes from a dataframe:
for i in range(len(All_Tickers)-1-len(Current_Date),len(All_Tickers)-1):
price = All_Tickers[str(length) + 'day_MA'][i]
date = All_Tickers['Date'][i+1]
According to the answer below, the problem should be the type of price is np.float64, how can I convert this variable to type float?
Your library tries to format the provided arguments to a format MySQL will understand. To do so, it checks the type of each argument, to determine how the input should be formatted.
However, since your lib doesn't knows numpy.float64, it fallbacks to a default encoder, which happens to be one for strings (unicode). Here is the relevent piece of code.
def escape_item(val, charset, mapping=None):
if mapping is None:
mapping = encoders
encoder = mapping.get(type(val))
# Fallback to default when no encoder found
if not encoder:
try:
encoder = mapping[text_type]
except KeyError:
raise TypeError("no default type converter defined")
if encoder in (escape_dict, escape_sequence):
val = encoder(val, charset, mapping)
else:
val = encoder(val, mapping)
return val
This encoder, assuming the input is indeed a string, tries to call the translate() method on this string. But, since this method isn't defined for float64, you get this error.
You should try to convert your float64 to a regular float.
Or, you can create your own encoder, and add it in the encoders dict used as the default mapping of python types to encoder. If you're going to use a lot this lib with float64, it may be worth doing.
Add the command below to anywhere before making pymysql connection. It adds new encoder of numpy.float64.
pymysql.converters.encoders[np.float64] = pymysql.converters.escape_float
pymysql.converters.conversions = pymysql.converters.encoders.copy()
pymysql.converters.conversions.update(pymysql.converters.decoders)
You can solve this problem by just adding a float method for every np.float variable like below:
variable = float(variable)
Another option, that could help it is:
from sqlalchemy import event
import numpy as np
import sqlalchemy
engine = sqlalchemy.create_engine(...)
def add_own_encoders(conn, cursor, query, *args):
cursor.connection.encoders[np.float64] = lambda value, encoders: float(value)
event.listen(engine, "before_cursor_execute", add_own_encoders)
I am trying to execute a query and return the results to Excel. The query takes in a string of years as input parameters. I am calling it in Python like this:
def flatten(l):
for el in l:
try:
yield from flatten(el)
except TypeError:
yield el
my_list = (previous_year_1,previous_year,current_year)
sql = 'select year,sum(sales)/case when sum(t_count)=0 then 1 else sum(t_count) as tx_sales from t_sales where year in ({1})'+ 'group by year' + 'order by year'
sql = sql.format ('?',','.join('?' * len(my_list)))
params = tuple(flatten(member_list))
ind_data = pd.read_sql(sql,engine,params)
The query itself, after fixing the end clause, works perfectly when run through SSMS. Just not through the Python code. The error I'm getting is:
Traceback (most recent call last):
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1139, in _execute_context
context)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\default.py", line 450, in do_execute
cursor.execute(statement, parameters)
pyodbc.Error: ('07002', '[07002] [Microsoft][SQL Server Native Client 11.0]COUNT field incorrect or syntax error (0) (SQLExecDirectW)')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "c:\pbp_proj\pbp_proj.py", line 61, in pull_metrics
ind_data = pd.read_sql_query(sql, engine, params)
File "C:\Anaconda3\lib\site-packages\pandas\io\sql.py", line 411, in read_sql_query
parse_dates=parse_dates, chunksize=chunksize)
File "C:\Anaconda3\lib\site-packages\pandas\io\sql.py", line 1128, in read_query
result = self.execute(*args)
File "C:\Anaconda3\lib\site-packages\pandas\io\sql.py", line 1022, in execute
return self.engine.execute(*args, **kwargs)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1989, in execute
return connection.execute(statement, *multiparams, **params)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 906, in execute
return self._execute_text(object, multiparams, params)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1054, in _execute_text
statement, parameters
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1146, in _execute_context
context)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1341, in _handle_dbapi_exception
exc_info
File "C:\Anaconda3\lib\site-packages\sqlalchemy\util\compat.py", line 188, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=exc_value)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\util\compat.py", line 181, in reraise
raise value.with_traceback(tb)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1139, in _execute_context
context)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\default.py", line 450, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.DBAPIError: (pyodbc.Error) ('07002', '[07002] [Microsoft][SQL Server Native Client 11.0]COUNT field incorrect or syntax error (0) (SQLExecDirectW)')
How can I resolve this error?
As #MYGz has already mentioned there is a missing space before order by.
Beside that there is a missing space before group by and the most important one - your CASE ... statement should be "closed" with END.
That said try the following SQL:
sql = 'select year,sum(sales)/(case when sum(t_count)=0 then 1 else sum(t_count) end)' \
+' as tx_sales from t_sales where year in ({1})'+' group by year order by year'
You can use your SQL pattern directly using .format() - there is no need to overwrite it:
params = tuple(flatten(member_list))
ind_data = pd.read_sql(sql.format('?',','.join('?' * len(params))), engine, params)
You have missed a space in your sql string between year and order by.
Try this:
sql = 'select year,sum(sales)/case when sum(t_count)=0 then 1 else sum(t_count) as tx_sales from t_sales where year in ({1}) '+ 'group by year ' + 'order by year '
Resolved this. A bit of hack, but works. I first changed it to using pyodbc instead of sqlalchemy.
so my query string became:
sql = 'select year,sum(sales)/case when sum(t_count)=0 then 1 else sum(t_count) end as tx_sales from t_sales where year in (?,?,?) '+ ' group by year' + ' order by year'
ind_data = pd.read_sql(sql, conn, params=member_list)
summary = ind_data.transpose()
I then had to add another AND clause with another parameter. For this I created:
cur_params = (member_list)
cur_params.append(var_premium)
then passsed cur_params to ind_data.
ind_data = pd.read_sql(sql, conn, params=cur_params)
both sets return data correctly now.
Thank you all for reading my post and for all the suggestions.
Trying to bulk insert a CSV file using pymssql here's the code:
conn = pymssql.connect(host='server', user='user', password='secret', database='My_Dev')
cur = conn.cursor()
load = 'BULK INSERT TempStaging FROM \'/home/dross/python/scripts/var/csv/' + f + '.csv\' WITH (FIRSTROW = 1,FIELDTERMINATOR = ',',ROWTERMINATOR = \'\\n\') GO")'
cur.execute(load)
When executing get following error:
Traceback (most recent call last):
File "./uploadResults.py", line 46, in <module>
cur.execute(sweepload)
File "pymssql.pyx", line 447, in pymssql.Cursor.execute (pymssql.c:7092)
File "_mssql.pyx", line 1009, in _mssql.MSSQLConnection.execute_query (_mssql.c:11585)
File "_mssql.pyx", line 1040, in _mssql.MSSQLConnection.execute_query (_mssql.c:11459)
File "_mssql.pyx", line 1160, in _mssql.MSSQLConnection.format_and_run_query (_mssql.c:12652)
File "_mssql.pyx", line 203, in _mssql.ensure_bytes (_mssql.c:2733)
AttributeError: 'tuple' object has no attribute 'encode'
Line 46 is cur.execute line
Note that .format() could allow sql injection, but if you control the filename then it's not that bad (not sure if a parameter would work here).
Also, you should use triple-quoted strings when dealing with SQL, your life will be so much better. Like this:
load = '''BULK INSERT TempStaging FROM /home/dross/python/scripts/var/csv/{}.csv WITH (FIRSTROW=1, FIELDTERMINATOR=',', ROWTERMINATOR='\n')'''.format(filename)
Being triple quoted, you can also break it up to make it easier to read:
load = '''
BULK INSERT TempStaging
FROM /home/dross/python/scripts/var/csv/{}.csv
WITH (
FIRSTROW=1
, FIELDTERMINATOR=','
, ROWTERMINATOR='\n'
)
'''.format(filename)
You should defined the string as below:
load = "BULK INSERT TempStaging FROM /home/dross/python/scripts/var/csv/" + f + ".csv WITH ( FIRSTROW=1 , FIELDTERMINATOR=',' , ROWTERMINATOR='\\n')"