Develop Reference

PyOrient SQL Query MemoryError

I have to read from a OrientDB. To test that everything works I tried to read from the Database with the SELECT Statement.
like this:
import pyorient
client = pyorient.OrientDB("adress", 2424)
session_id = client.connect("root", "password")
client.db_open("table","root","password")
print str(client.db_size())
client.query("SELECT * FROM L1_Req",1)
The Connection works fine and also the print str(client.db_size()) line.
But at client.query("SELECT * FROM L1_Req",1) it returns the following Error Message:
Traceback (most recent call last):
File "testpy.py", line 9, in <module>
client.query("SELECT * FROM L1_Req",1)
File "C:\app\tools\python27\lib\site-packages\pyorient\orient.py", line 470, i
n query
.prepare(( QUERY_SYNC, ) + args).send().fetch_response()
File "C:\app\tools\python27\lib\site-packages\pyorient\messages\commands.py",
line 144, in fetch_response
super( CommandMessage, self ).fetch_response()
File "C:\app\tools\python27\lib\site-packages\pyorient\messages\base.py", line
265, in fetch_response
self._decode_all()
File "C:\app\tools\python27\lib\site-packages\pyorient\messages\base.py", line
249, in _decode_all
self._decode_header()
File "C:\app\tools\python27\lib\site-packages\pyorient\messages\base.py", line
176, in _decode_header
serialized_exception = self._decode_field( FIELD_STRING )
File "C:\app\tools\python27\lib\site-packages\pyorient\messages\base.py", line
366, in _decode_field
_decoded_string = self._orientSocket.read( _len )
File "C:\app\tools\python27\lib\site-packages\pyorient\orient.py", line 164, i
n read
buf = bytearray(_len_to_read)
MemoryError
I also tried some ohter SQL Statements like:
client.query("SELECT subSystem FROM L1_Req",1)
I cant't get why this happends. Can you guys help me ?

Pythons Bottle with SQLITE3

I have been trying to practise Bottle Py. There is a tutorial about making an APP: TODO.
It works fine. But If task id exceeds 1 character that means 10 instead of 1,2,3,4,5,6,7,8,9
It shows error like below.
ProgrammingError('Incorrect number of bindings supplied. The current
statement uses 1, and there are 2 supplied.',)
Code is:
#route('/edit/<no:int>', method='GET')
def edit_item(no):
if request.GET.save:
edit = request.GET.task.strip()
status = request.GET.status.strip()
if status == 'open':
status = 1
else:
status = 0
conn = sqlite3.connect('todo.db')
c = conn.cursor()
c.execute("UPDATE todo SET task = ?, status = ? WHERE id LIKE ?", (edit, status, no))
conn.commit()
return '<p>The item number %s was successfully updated</p>' % no
else:
conn = sqlite3.connect('todo.db')
c = conn.cursor()
c.execute("SELECT task FROM todo WHERE id LIKE ?", (str(no)))
cur_data = c.fetchone()
return template('edit_task', old=cur_data, no=no)
Tracebacks:
1.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/bottle.py", line 862, in _handle
return route.call(**args)
File "/usr/lib/python2.7/dist-packages/bottle.py", line 1737, in wrapper
rv = callback(*a, **ka)
File "todo.py", line 67, in edit_item
c.execute('SELECT task FROM todo WHERE id LIKE ?', no)
ValueError: parameters are of unsupported type
2.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/bottle.py", line 862, in _handle
return route.call(**args)
File "/usr/lib/python2.7/dist-packages/bottle.py", line 1737, in wrapper
rv = callback(*a, **ka)
File "todo.py", line 67, in edit_item
c.execute('SELECT task FROM todo WHERE id LIKE ?', (no))
ValueError: parameters are of unsupported type
What to do?

This might happen because the execute function will unpack your second parameter when you do (str(no)) the outer () will not convert your tuple, you need to do (str(no),) if you have only one element in the tuple.
For instance, since it recognized as string, it will unpack "10" it into ("1", "0")

pymssql bulk insert error (python)

Trying to bulk insert a CSV file using pymssql here's the code:
conn = pymssql.connect(host='server', user='user', password='secret', database='My_Dev')
cur = conn.cursor()
load = 'BULK INSERT TempStaging FROM \'/home/dross/python/scripts/var/csv/' + f + '.csv\' WITH (FIRSTROW = 1,FIELDTERMINATOR = ',',ROWTERMINATOR = \'\\n\') GO")'
cur.execute(load)
When executing get following error:
Traceback (most recent call last):
File "./uploadResults.py", line 46, in <module>
cur.execute(sweepload)
File "pymssql.pyx", line 447, in pymssql.Cursor.execute (pymssql.c:7092)
File "_mssql.pyx", line 1009, in _mssql.MSSQLConnection.execute_query (_mssql.c:11585)
File "_mssql.pyx", line 1040, in _mssql.MSSQLConnection.execute_query (_mssql.c:11459)
File "_mssql.pyx", line 1160, in _mssql.MSSQLConnection.format_and_run_query (_mssql.c:12652)
File "_mssql.pyx", line 203, in _mssql.ensure_bytes (_mssql.c:2733)
AttributeError: 'tuple' object has no attribute 'encode'
Line 46 is cur.execute line

Note that .format() could allow sql injection, but if you control the filename then it's not that bad (not sure if a parameter would work here).
Also, you should use triple-quoted strings when dealing with SQL, your life will be so much better. Like this:
load = '''BULK INSERT TempStaging FROM /home/dross/python/scripts/var/csv/{}.csv WITH (FIRSTROW=1, FIELDTERMINATOR=',', ROWTERMINATOR='\n')'''.format(filename)
Being triple quoted, you can also break it up to make it easier to read:
load = '''
BULK INSERT TempStaging
FROM /home/dross/python/scripts/var/csv/{}.csv
WITH (
FIRSTROW=1
, FIELDTERMINATOR=','
, ROWTERMINATOR='\n'
)
'''.format(filename)

You should defined the string as below:
load = "BULK INSERT TempStaging FROM /home/dross/python/scripts/var/csv/" + f + ".csv WITH ( FIRSTROW=1 , FIELDTERMINATOR=',' , ROWTERMINATOR='\\n')"

pymysql encoding error while inserting binary data into longblob column

I'm trying to insert contents of a binary file into a longblob column:
Python code:
conn = pymysql.connect(...)
cursor = conn.cursor()
with open('test.bz2', 'rb') as fp:
data = fp.read()
cursor.execute('insert into test_t (test) values (%s)', [data])
Error stack trace:
Traceback (most recent call last):
File "./doit2", line 9, in <module>
cursor.execute('insert into test_t (test) values (%s)', [data])
File "/u02/srm_tp/local/lib/python3.4/site-packages/pymysql/cursors.py", line 127, in execute
result = self._query(query)
File "/u02/srm_tp/local/lib/python3.4/site-packages/pymysql/cursors.py", line 275, in _query
conn.query(q)
File "/u02/srm_tp/local/lib/python3.4/site-packages/pymysql/connections.py", line 763, in query
sql = sql.encode(self.encoding)
UnicodeEncodeError: 'latin-1' codec can't encode character '\udcae' in position 45: ordinal not in range(256)
Create table script:
mysql> show create table test_t;
+--------+--------------------------------------------------------------------------+
| Table | Create Table |
+--------+--------------------------------------------------------------------------+
| test_t | CREATE TABLE `test_t` (
`test` longblob
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci |
+--------+--------------------------------------------------------------------------+
Default Encoding:
=->python3 -c 'import sys; print(sys.getdefaultencoding())'
utf-8
Adding "charset='utf8', use_unicode=True" to connect call, changes the error to:
Traceback (most recent call last):
File "./doit2", line 13, in <module>
cursor.execute('insert into test_t (test) values (%s)', [data])
File "/u02/srm_tp/local/lib/python3.4/site-packages/pymysql/cursors.py", line 127, in execute
result = self._query(query)
File "/u02/srm_tp/local/lib/python3.4/site-packages/pymysql/cursors.py", line 275, in _query
conn.query(q)
File "/u02/srm_tp/local/lib/python3.4/site-packages/pymysql/connections.py", line 763, in query
sql = sql.encode(self.encoding)
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcae' in position 45: surrogates not allowed

This should do the trick:
conn = pymysql.connect(...)
conn.set_character_set('utf8')
cursor = conn.cursor()
cursor.execute('SET NAMES utf8;')
cursor.execute('SET CHARACTER SET utf8;')
cursor.execute('SET character_set_connection=utf8;')
with open('test.bz2', 'rb') as fp:
data = fp.read()
cursor.execute('insert into test_t (test) values (%s)', [data])

Looks like it was a pymysql bug. I upgraded from 0.6.4 to 0.6.6 (latest as of now) and the issue is no longer there.

Inserting language characters in Django

I am following the link http://code.google.com/p/django-multilingual-model/ Basically i am trying to insert hindi characters into mysql db
I have created the files in the test directory as in the above said link and using django version 1.1 an dpython version is 2.4.3
I geta error message as the following.
from testlanguages import Language,BookTranslation,Book
>>> lang_pl = Language(code="pl", name="Polish")
>>> lang_pl.save()
>>> book_pl.language = lang_pl
>>> book_pl.save()
>>> lang_hi = Language(code="hi", name="Hindi")
>>> lang_hi.save()
>>> book_hi = BookTranslation()
>>> book_hi.title = "का सफल प्रक्षेपण किया है. इस राकेट ने पाँच" Sum characters as in bbc.co.uk/hindi
>>> book_hi.description = "का सफल प्रक्षेपण किया है. इस राकेट ने पाँच" Sum characters as in bbc.co.uk/hindi
>>> book_hi.model = book
>>> book_hi.language = lang_hi
>>> book_hi.save()
Traceback (most recent call last):
File "<console>", line 1, in ?
File "/opt/project/django/django/db/models/base.py", line 410, in save
self.save_base(force_insert=force_insert, force_update=force_update)
File "/opt/project/django/django/db/models/base.py", line 495, in save_base
result = manager._insert(values, return_id=update_pk)
File "/opt/project/django/django/db/models/manager.py", line 177, in _insert
return insert_query(self.model, values, **kwargs)
File "/opt/project/django/django/db/models/query.py", line 1087, in insert_query
return query.execute_sql(return_id)
File "/opt/project/django/django/db/models/sql/subqueries.py", line 320, in execute_sql
cursor = super(InsertQuery, self).execute_sql(None)
File "/opt/project/django/django/db/models/sql/query.py", line 2369, in execute_sql
cursor.execute(sql, params)
File "/opt/project/django/django/db/backends/util.py", line 19, in execute
return self.cursor.execute(sql, params)
File "/opt/project/django/django/db/backends/mysql/base.py", line 84, in execute
return self.cursor.execute(query, args)
File "/usr/lib/python2.4/site-packages/MySQLdb/cursors.py", line 165, in execute
self._warning_check()
File "/usr/lib/python2.4/site-packages/MySQLdb/cursors.py", line 80, in _warning_check
warn(w[-1], self.Warning, 3)
File "/usr/lib/python2.4/warnings.py", line 61, in warn
warn_explicit(message, category, filename, lineno, module, registry)
File "/usr/lib/python2.4/warnings.py", line 96, in warn_explicit
raise message
Warning: Incorrect string value: '\xE0\xA4\x95\xE0\xA4\xBE...' for column 'title' at row 1
How to resolve this.
Is there any easy method to insert the special characters into DB

Stop using bytestrings. Start using unicode.
book_hi.title = u"का सफल प्रक्षेपण किया है. इस राकेट ने पाँच"

conn = MySQLdb.connect( host = "localhost",
user = "root",
passwd = "",
db = "test");
conn.set_character_set('utf8')
this code will solve you problem
Note *. conn.set_character_set('utf8') is notable part

This is most likely a MySQL problem, which doesn't always support Unicode characters by default.
You should probably declare the DEFAULT CHARSET to utf8 and DEFAULT COLLATION to utf8_general_ci on your database (use the ALTER DATABASE or modify the CREATE DATABASE statements, as explained here).