Develop Reference

PyOrient SQL Query MemoryError

I have to read from a OrientDB. To test that everything works I tried to read from the Database with the SELECT Statement.
like this:
import pyorient
client = pyorient.OrientDB("adress", 2424)
session_id = client.connect("root", "password")
client.db_open("table","root","password")
print str(client.db_size())
client.query("SELECT * FROM L1_Req",1)
The Connection works fine and also the print str(client.db_size()) line.
But at client.query("SELECT * FROM L1_Req",1) it returns the following Error Message:
Traceback (most recent call last):
File "testpy.py", line 9, in <module>
client.query("SELECT * FROM L1_Req",1)
File "C:\app\tools\python27\lib\site-packages\pyorient\orient.py", line 470, i
n query
.prepare(( QUERY_SYNC, ) + args).send().fetch_response()
File "C:\app\tools\python27\lib\site-packages\pyorient\messages\commands.py",
line 144, in fetch_response
super( CommandMessage, self ).fetch_response()
File "C:\app\tools\python27\lib\site-packages\pyorient\messages\base.py", line
265, in fetch_response
self._decode_all()
File "C:\app\tools\python27\lib\site-packages\pyorient\messages\base.py", line
249, in _decode_all
self._decode_header()
File "C:\app\tools\python27\lib\site-packages\pyorient\messages\base.py", line
176, in _decode_header
serialized_exception = self._decode_field( FIELD_STRING )
File "C:\app\tools\python27\lib\site-packages\pyorient\messages\base.py", line
366, in _decode_field
_decoded_string = self._orientSocket.read( _len )
File "C:\app\tools\python27\lib\site-packages\pyorient\orient.py", line 164, i
n read
buf = bytearray(_len_to_read)
MemoryError
I also tried some ohter SQL Statements like:
client.query("SELECT subSystem FROM L1_Req",1)
I cant't get why this happends. Can you guys help me ?

KeyError while executing SQL with parameters

I am getting KeyError while running below code. I am trying to pass parameters using separate parameters variable.
Code:
import teradata
host,username,password = 'hostname','uname', 'pwd'
udaExec = teradata.UdaExec (appName="APtest", version="1.0", logConsole=False)
connect = udaExec.connect(method="odbc",system=host, username=username, password=password, dsn="dsnname")
val1='NULL'
val2='NULL'
parameters={'param1':val1, 'param2': val2}
qry="""
SELECT number
FROM table
WHERE number = %(param1)s
AND col=%(param2)s
"""
connect.execute(qry, parameters)
Error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/tmp/lib/python2.7/site-packages/teradata/udaexec.py", line 675, in execute
self.internalCursor.execute(query, params, **kwargs)
File "/tmp/lib/python2.7/site-packages/teradata/udaexec.py", line 745, in execute
self._execute(self.cursor.execute, query, params, **kwargs)
File "/tmp/lib/python2.7/site-packages/teradata/udaexec.py", line 787, in _execute
logParamCharLimit)
File "/tmp/lib/python2.7/site-packages/teradata/udaexec.py", line 875, in _getParamsString
if isinstance(params[0], (list, tuple)):
KeyError: 0
If i write the query in below manner then it works but i have very long list of parameters therefore need it in separate parameter variable.
This works:
qry="""
SELECT number
FROM table
WHERE number = '%s'
AND col='%s'
""" % (val1, val2)

Apparently, teradata does not support dictionaries for parameters. Use a list instead.
parameters = [val1, val2]
qry="""
SELECT number
FROM table
WHERE number = %s
AND col=%s
"""
connect.execute(qry, parameters)

Pythons Bottle with SQLITE3

I have been trying to practise Bottle Py. There is a tutorial about making an APP: TODO.
It works fine. But If task id exceeds 1 character that means 10 instead of 1,2,3,4,5,6,7,8,9
It shows error like below.
ProgrammingError('Incorrect number of bindings supplied. The current
statement uses 1, and there are 2 supplied.',)
Code is:
#route('/edit/<no:int>', method='GET')
def edit_item(no):
if request.GET.save:
edit = request.GET.task.strip()
status = request.GET.status.strip()
if status == 'open':
status = 1
else:
status = 0
conn = sqlite3.connect('todo.db')
c = conn.cursor()
c.execute("UPDATE todo SET task = ?, status = ? WHERE id LIKE ?", (edit, status, no))
conn.commit()
return '<p>The item number %s was successfully updated</p>' % no
else:
conn = sqlite3.connect('todo.db')
c = conn.cursor()
c.execute("SELECT task FROM todo WHERE id LIKE ?", (str(no)))
cur_data = c.fetchone()
return template('edit_task', old=cur_data, no=no)
Tracebacks:
1.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/bottle.py", line 862, in _handle
return route.call(**args)
File "/usr/lib/python2.7/dist-packages/bottle.py", line 1737, in wrapper
rv = callback(*a, **ka)
File "todo.py", line 67, in edit_item
c.execute('SELECT task FROM todo WHERE id LIKE ?', no)
ValueError: parameters are of unsupported type
2.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/bottle.py", line 862, in _handle
return route.call(**args)
File "/usr/lib/python2.7/dist-packages/bottle.py", line 1737, in wrapper
rv = callback(*a, **ka)
File "todo.py", line 67, in edit_item
c.execute('SELECT task FROM todo WHERE id LIKE ?', (no))
ValueError: parameters are of unsupported type
What to do?

This might happen because the execute function will unpack your second parameter when you do (str(no)) the outer () will not convert your tuple, you need to do (str(no),) if you have only one element in the tuple.
For instance, since it recognized as string, it will unpack "10" it into ("1", "0")

Pandas.read_sql failing with DBAPIError 07002: COUNT field incorrect or syntax error

I am trying to execute a query and return the results to Excel. The query takes in a string of years as input parameters. I am calling it in Python like this:
def flatten(l):
for el in l:
try:
yield from flatten(el)
except TypeError:
yield el
my_list = (previous_year_1,previous_year,current_year)
sql = 'select year,sum(sales)/case when sum(t_count)=0 then 1 else sum(t_count) as tx_sales from t_sales where year in ({1})'+ 'group by year' + 'order by year'
sql = sql.format ('?',','.join('?' * len(my_list)))
params = tuple(flatten(member_list))
ind_data = pd.read_sql(sql,engine,params)
The query itself, after fixing the end clause, works perfectly when run through SSMS. Just not through the Python code. The error I'm getting is:
Traceback (most recent call last):
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1139, in _execute_context
context)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\default.py", line 450, in do_execute
cursor.execute(statement, parameters)
pyodbc.Error: ('07002', '[07002] [Microsoft][SQL Server Native Client 11.0]COUNT field incorrect or syntax error (0) (SQLExecDirectW)')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "c:\pbp_proj\pbp_proj.py", line 61, in pull_metrics
ind_data = pd.read_sql_query(sql, engine, params)
File "C:\Anaconda3\lib\site-packages\pandas\io\sql.py", line 411, in read_sql_query
parse_dates=parse_dates, chunksize=chunksize)
File "C:\Anaconda3\lib\site-packages\pandas\io\sql.py", line 1128, in read_query
result = self.execute(*args)
File "C:\Anaconda3\lib\site-packages\pandas\io\sql.py", line 1022, in execute
return self.engine.execute(*args, **kwargs)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1989, in execute
return connection.execute(statement, *multiparams, **params)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 906, in execute
return self._execute_text(object, multiparams, params)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1054, in _execute_text
statement, parameters
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1146, in _execute_context
context)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1341, in _handle_dbapi_exception
exc_info
File "C:\Anaconda3\lib\site-packages\sqlalchemy\util\compat.py", line 188, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=exc_value)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\util\compat.py", line 181, in reraise
raise value.with_traceback(tb)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1139, in _execute_context
context)
File "C:\Anaconda3\lib\site-packages\sqlalchemy\engine\default.py", line 450, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.DBAPIError: (pyodbc.Error) ('07002', '[07002] [Microsoft][SQL Server Native Client 11.0]COUNT field incorrect or syntax error (0) (SQLExecDirectW)')
How can I resolve this error?

As #MYGz has already mentioned there is a missing space before order by.
Beside that there is a missing space before group by and the most important one - your CASE ... statement should be "closed" with END.
That said try the following SQL:
sql = 'select year,sum(sales)/(case when sum(t_count)=0 then 1 else sum(t_count) end)' \
+' as tx_sales from t_sales where year in ({1})'+' group by year order by year'
You can use your SQL pattern directly using .format() - there is no need to overwrite it:
params = tuple(flatten(member_list))
ind_data = pd.read_sql(sql.format('?',','.join('?' * len(params))), engine, params)

You have missed a space in your sql string between year and order by.
Try this:
sql = 'select year,sum(sales)/case when sum(t_count)=0 then 1 else sum(t_count) as tx_sales from t_sales where year in ({1}) '+ 'group by year ' + 'order by year '

Resolved this. A bit of hack, but works. I first changed it to using pyodbc instead of sqlalchemy.
so my query string became:
sql = 'select year,sum(sales)/case when sum(t_count)=0 then 1 else sum(t_count) end as tx_sales from t_sales where year in (?,?,?) '+ ' group by year' + ' order by year'
ind_data = pd.read_sql(sql, conn, params=member_list)
summary = ind_data.transpose()
I then had to add another AND clause with another parameter. For this I created:
cur_params = (member_list)
cur_params.append(var_premium)
then passsed cur_params to ind_data.
ind_data = pd.read_sql(sql, conn, params=cur_params)
both sets return data correctly now.
Thank you all for reading my post and for all the suggestions.

pymssql bulk insert error (python)

Trying to bulk insert a CSV file using pymssql here's the code:
conn = pymssql.connect(host='server', user='user', password='secret', database='My_Dev')
cur = conn.cursor()
load = 'BULK INSERT TempStaging FROM \'/home/dross/python/scripts/var/csv/' + f + '.csv\' WITH (FIRSTROW = 1,FIELDTERMINATOR = ',',ROWTERMINATOR = \'\\n\') GO")'
cur.execute(load)
When executing get following error:
Traceback (most recent call last):
File "./uploadResults.py", line 46, in <module>
cur.execute(sweepload)
File "pymssql.pyx", line 447, in pymssql.Cursor.execute (pymssql.c:7092)
File "_mssql.pyx", line 1009, in _mssql.MSSQLConnection.execute_query (_mssql.c:11585)
File "_mssql.pyx", line 1040, in _mssql.MSSQLConnection.execute_query (_mssql.c:11459)
File "_mssql.pyx", line 1160, in _mssql.MSSQLConnection.format_and_run_query (_mssql.c:12652)
File "_mssql.pyx", line 203, in _mssql.ensure_bytes (_mssql.c:2733)
AttributeError: 'tuple' object has no attribute 'encode'
Line 46 is cur.execute line

Note that .format() could allow sql injection, but if you control the filename then it's not that bad (not sure if a parameter would work here).
Also, you should use triple-quoted strings when dealing with SQL, your life will be so much better. Like this:
load = '''BULK INSERT TempStaging FROM /home/dross/python/scripts/var/csv/{}.csv WITH (FIRSTROW=1, FIELDTERMINATOR=',', ROWTERMINATOR='\n')'''.format(filename)
Being triple quoted, you can also break it up to make it easier to read:
load = '''
BULK INSERT TempStaging
FROM /home/dross/python/scripts/var/csv/{}.csv
WITH (
FIRSTROW=1
, FIELDTERMINATOR=','
, ROWTERMINATOR='\n'
)
'''.format(filename)

You should defined the string as below:
load = "BULK INSERT TempStaging FROM /home/dross/python/scripts/var/csv/" + f + ".csv WITH ( FIRSTROW=1 , FIELDTERMINATOR=',' , ROWTERMINATOR='\\n')"