Develop Reference

How can I avoid SQL issue?

When I run my query I got this issue --- pyodbc.ProgrammingError: ('42000', '[42000] [IBM][System i Access ODBC Driver][DB2 for i5/OS]SQL0010 - String constant beginning " " not delimited. (-10) (SQLExecDirectW)')
Can someone help me please!
cursor = db2_conn.cursor()
cursor.execute
("select * from qs36f.DSTHSTP join qs36f.calendar on date_ccyymmd = dhindt join qs36f.itemp i on DHITM#=i.ZIITM# join qs36f.subcl2p on i.zisbcl = s2sbcl and i.ziclss = s2clss join qs36f.subclsp on sbclss=s2clss and SBSBCL=s2sbcl where date_iso between(current date - 10 day) and current date and DHCUS# in (" + open_stores + ") and dhqtss >= 1 and SBDEPT='MT' AND dhclss = " + class_nbr + " and ((dhqtss*dhrt5s)*DHPACK) > " + start_range + "")
I tried run this query and get some data from db for testing. With this query I should get store number, item number and price greater than 250.

This is a classic SQL injection scenario caused by concatenating unchecked input to generate queries. String concatenation or interpolation are the root cause for this problem and can't be solved with any amount of quoting or escaping.
The real and easy solution is to use parameterized queries. Another significant improvement is creating the query string separately from the call. Putting everything in a single line is neither cleaner, faster nor easier.
sql="""select *
from qs36f.DSTHSTP
join qs36f.calendar on date_ccyymmd = dhindt
join qs36f.itemp i on DHITM#=i.ZIITM#
join qs36f.subcl2p on i.zisbcl = s2sbcl and i.ziclss = s2clss
join qs36f.subclsp on sbclss=s2clss and SBSBCL=s2sbcl
where date_iso between(current date - 10 day) and current date
and DHCUS# in (?,?,?)
and dhqtss >= 1
and SBDEPT='MT'
AND dhclss = ?
and ((dhqtss*dhrt5s)*DHPACK) > ?
"""
rows=cursor.execute(sql,shop1,shop2,shop3,class_nbr,start_range)
? specifies query parameters by position. Parameter values never become part of the query. They are sent to the server as strongly typed values (integers, dates, flaots) outside the query itself. The database compiles the query into a parameterized execution plan and then executes that with the parameter values.
This has several benefits :
Obviously, SQL injections are eliminated as the values never become part of the string.
Formatting errors are eliminated because numbers and dates are passed as binary values. You no longer have to worry about 34,56, MM/DD/YYYY or DD/MM/YYYY. You can compare date fields directly against date values, numeric fields with numbers.
The database can reuse already compiled execution plans for the same query string. That offers significant benefits for big, busy systems.

Comparing two tables with SQLAlchemy

I am having a great deal of trouble with this problem. I am trying to compare two different tables from two different databases to see what tuples have been added, what tuples have been deleted, and what tuples have been updated. I do that with the following code:
from sqlalchemy import *
# query the databases to get all tuples from the relations
# save each relation to a list in order to be able to iterate over their tuples multiple times
# iterate through the lists, hash each tuple with k, v being primary key, tuple
# iterate through the "after" relation. for each tuple in the new relation, hash its key in the "before" relation.
# If it's found and the tuple is different, consider that an update, else, do nothing.
# If it is not found, consider that an insert
# iterate through the "before" relation. for each tuple in the "before" relation, hash by the primary key
# if the tuple is found in the "after" relation, do nothing
# if not, consider that a delete.
dev_engine = create_engine('mysql://...')
prod_engine = create_engine('mysql://...')
def transactions(exchange):
dev_connect = dev_engine.connect()
prod_connect = prod_engine.connect()
get_dev_instrument = "select * from " + exchange + "_instrument;"
instruments = dev_engine.execute(get_dev_instrument)
instruments_list = [r for r in instruments]
print 'made instruments_list'
get_prod_instrument = "select * from " + exchange + "_instrument;"
instruments_after = prod_engine.execute(get_prod_instrument)
instruments_after_list = [r2 for r2 in instruments_after]
print 'made instruments after_list'
before_map = {}
after_map = {}
for row in instruments:
before_map[row['instrument_id']] = row
for y in instruments_after:
after_map[y['instrument_id']] = y
print 'formed maps'
update_count = insert_count = delete_count = 0
change_list = []
for prod_row in instruments_after_list:
result = list(prod_row)
try:
row = before_map[prod_row['instrument_id']]
if not row == prod_row:
update_count += 1
for i in range(len(row)):
if not row[i] == prod_row[i]:
result[i] = str(row[i]) + '--->' + str(prod_row[i])
result.append("updated")
change_list.append(result)
except KeyError:
insert_count += 1
result.append("inserted")
change_list.append(result)
for before_row in instruments_list:
result = before_row
try:
after_row = after_map[before_row['instrument_id']]
except KeyError:
delete_count += 1
result.append("deleted")
change_list.append(result)
for el in change_list:
print el
print "Insert: " + str(insert_count)
print "Update: " + str(update_count)
print "Delete: " + str(delete_count)
dev_connect.close()
prod_connect.close()
def main():
transactions("...")
main()
instruments is the "before" table and instruments_after is the "after" table, so I want to see the changes that occurred to change instruments to instruments_after.
The above code works well, but fails when instruments or instruments_after is very large. I have a table that is over 4 million lines long and simply trying to load that into memory causes Python to exit. I have tried overcoming this issue by using LIMIT, OFFSET in my queries to append to the instruments_lists in pieces, but Python still exits because two lists of that size simply take up too much space. My last option is to choose a batch from one relation, and iterate through batches of the second relation and make comparisons, but that is extremely error prone. Is there another way to circumvent this problem? I have considered allocating more memory to my VM but I feel that the space complexity of my code is the issue and that is what should be fixed first.

Listing the records on a SQL lite database with Genie

Now that I have created the SQL database (1, 2), I want to do something with it. I have started a small function to handle the printing of the dataset contents, however I am unable to figure out how to:
adjust space between strings on a print command, in order to it become properly justified? One can do it in python using ljust(), but how to make something similar with Genie?
iterate across all entries on the dataset? As far as I understand there is no equivalent of a cursor in Genie (maybe I am wrong here, though).
connect to the created database? How to open it?
Extra point: how to create a stub with genie? I wanted to create empty functions and classes,to give an empty structure to the code, however there is no pass command in genie and it seems that the compiler do not accept empty if statements or functions.
This is the python code I am trying to mimic:
def PrintAllRecipes(self):
print '%s %s %s %s' % ('Item'.ljust(5),'Name'.ljust(30),'Serves'.ljust(20),'Source'.ljust(30))
print '--------------------------------------------------------------------------------------'
sql = 'SELECT * FROM Recipes'
cntr = 0
for x in cursor.execute(sql):
cntr += 1
print '%s %s %s %s' % (str(x[0]).rjust(5),x[1].ljust(30),x[2].ljust(20),x[3].ljust(30))
print '--------------------------------------------------------------------------------------'
self.totalcount = cntr
This is how far I have got:
[indent=4]
class Cookbook
def PrintAllRecipes()
print "%s %s %s %s" % ("Item".ljust(5),"Name".ljust(30),"Serves".ljust(20),"Source".ljust(30))
print "--------------------------------------------------------------------------------------"
var sql = "SELECT * FROM Recipes"
var cntr = 0
for x in cursor.execute(sql)
cntr += 1
print "%s %s %s %s" % (str(x[0]).rjust(5),x[1].ljust(30),x[2].ljust(20),x[3].ljust(30))
print "--------------------------------------------------------------------------------------"
self.totalcount = cntr
def raw_input (query : string? = null) : string?
if (query != null)
stdout.printf ("%s", query)
return stdin.read_line ()
init
//def Menu()
//cbk = Cookbook()
//var loop = True
print "==================================================="
print " RECIPE DATABASE "
print " 1 - Show All Recipes"
print " 2 - Search for a recipe"
print " 3 - Show a Recipe"
print " 4 - Delete a recipe"
print " 5 - Add a recipe"
print " 6 - Print a recipe"
print " 0 - Exit"
print "==================================================="
response:string = raw_input("Enter a selection -> ")

Genie `print` Formatting
The Genie print command uses printf formatting - see https://en.wikipedia.org/wiki/Printf_format_string
In your example the first line would be:
print "%-5s%-30s%-20s%-30s", "Item", "Name", "Serves", "Source"
The minus sign serves to left justify, then the number is the width.
Opening an SQLite Database
To open an SQLite database you use:
Database.open( "database_name.sqlite", out database )
Just as you do to create the database file. SQLite will create the database file if it does not exist. So open and create are the same command.
Genie `pass` Statement
A stub if statement:
if true
pass
A stub function:
def stub_example()
pass
The pass statement could also be extended to classes and namespaces, but this has not been implemented. So if you get to the stage of hacking on the Genie parser it could be a useful task. For now you have to add a dummy value/function to the class:
class placeholder
dummy:string = ""
Avoid `null`, Use Sensible Defaults
Assigning null to an identifier means it does not have a value, the identifier is a null pointer. Trying to access a null pointer, such as in arithmetic expressions, will cause the program to crash. This lead C.A.R.Hoare to call his invention of null his "billion-dollar mistake" - see https://en.wikipedia.org/wiki/Null_pointer#History for the quote.
C programmers tend to use null a lot so Genie maintains null for compatibility with C interfaces.
It is better, however, to use a sensible default value. For your raw_input function if no prompt is passed then an empty string is fine or a default prompt such "> ". This also avoids your null check, which is another disadvantage of using null. You have to constantly check an identifier is not null. stdin.readline also waits until a string is entered. Even if it is an empty string when the user just presses enter, so it never returns null. Your function could be re-written as:
def raw_input (query:string = ""):string
stdout.printf ("%s", query)
return stdin.read_line ()
Getting the Results of Select Query
Valadoc has examples for using either exec() or a prepared statement. Although they are in Vala.
For exec() you pass in a callback function that gets called for each row in the result. The callback function will be passed the number of columns in the result, an array of values as text and an array of column names. See http://valadoc.org/#!api=sqlite3/Sqlite.Database.exec
For a prepared statement the step() function returns Sqlite.ROWS until there are no more rows in the result. So you loop over that and read the columns from the prepared statement. See http://valadoc.org/#!api=sqlite3/Sqlite.Statement
P.S. For your menu you could have used a verbatim string

cx_Oracle query returns zero rows

Why does the below code not work ? It returns zero rows even though I have many multiple matching the search criteria.
A simple query of the form select * from Table_1 works fine and returns positive number of rows
import cx_Oracle
def function_A (data):
connection = cx_Oracle.connect('omitted details here')
for index in range(len(data)):
# connection is already open
cursor = connection.cursor()
query = "select * from Table_1 where column_1=:column1 and column_2=:column2 and column_3=:column3 and column_4=:column4"
bindVars={'column1':data[index][3], 'column2':data[index][4], 'column4':data[index][5], 'column5':data[index][6]}
cursor.execute(query, bindVars)
cursor.arraysize = 256
rowCount = 0
resultSet = cursor.fetchall()
if (resultSet != None):
logger.debug("Obtained a resultSet with length = %s", len(resultSet))
for index in range(len(resultSet)):
logger.debug("Fetched one row from cursor, incrementing counter !!")
rowCount = rowCount + 1
logger.debug("Fetched one row from cursor, incremented counter !!")
logger.debug("Successfully executed the select statement for table Table_1; that returned %s rows !!", rowCount)
logger.debug("Successfully executed the select statement for table Table_1; that returned %s rows !!", cursor.rowcount)
Please ignore minor formatting issues, code runs just does not give me a positive number of rows.
Code is being run on IBM AIX with python2.6 and a compatible version of cx_Oracle.

Oracle CX's cursor object has a read-only rowcount property. Rowcount is returning how many rows are returned with fetch* methods.
Say the query yields 5 rows, then the interaction is like this
execute rowcount = 0
fetchone rowcount = 1
fetchone rowcount = 2
fetchall rowcount = 5
Thay way you do not need to manually track it. Your query issues will have to be resolved first offcourse :)

Your query returns 0 rows because there are 0 rows that match your query. Either remove a predicate from your WHERE clause or change the value you pass into one.
It's worth noting that you're not binding anything to column3 in your bindVars variable. I'm also not entirely certain why you're iterating, cursor.rowcount, as you have it gives you the number of rows that have been fetched by the cursor.
Generally, if you think a SELECT statement is not returning the correct result then take it our of code and run it directly against the database. Bind all variables first so you can see exactly what you're actually running.

banging my head against monitor on this one... you have to do something like below to check, as the cursor value changes once you operate on it:
result_set = DB_connector.execute(sql)
result_list = result_set.fetchall() # assign the return row to a list
if result_set.rowcount == 0:
print('DB check failed, no row returned')
sql_result_set = None
else:
for row in result_list: # use this instead result_set
print('row fetched: ' + row[0])
sql_result_set.append(row[0])
print('DB test Passed')

Binary search of unaccesible data field in ldap from python

I'm interested in reproducing a particular python script.
I have a friend who was accessing an ldap database, without authentication. There was a particular field of interest, we'll call it nin (an integer) for reference, and this field wasn't accessible without proper authentication. However, my friend managed to access this field through some sort of binary search (rather than just looping through integers) on the data; he would check the first digit, check if it was greater or less than the starting value, he would augment that until it returned a true value indicating existence, adding digits and continuing checking until he found the exact value of the integer nin.
Any ideas on how he went about this? I've access to a similarly set up database.

Your best bet would be to get authorization to access that field. You are circumventing the security of the database otherwise.

Figured it out. I just needed to filter on (&(cn=My name)(nin=guess*) and I managed to filter until it returns the correct result.
Code follows in case anyone else needs to find a field they aren't supposed to access, but can check results for and know the name of.
def lookup(self, username="", guess=0,verbose=0):
guin = guess
result_set = []
varsearch = "(&(name=" + str(username) + ")(" + "nin" + "=" + str(guin) + "*))"
result_id = self.l.search("", ldap.SCOPE_SUBTREE, varsearch, ["nin"])
while True:
try:
result_type, result_data = self.l.result(result_id, 0, 5.0)
if (result_data == []):
break
else:
if result_type == ldap.RES_SEARCH_ENTRY:
result_set.append(result_data)
except ldap.TIMEOUT:
return {"name": username}
if len(result_set) == 0:
return self.lookup(username, guin + 1,verbose)
else:
if guess < 1000000:
return self.lookup(username, guess * 10,verbose)
else:
if verbose==1:
print "Bingo!",
return str(guess)