Case Insensitive search with POstgres Similar To - python

I have such query in my Django code
cond = self.text_query(field.id, f'''
EXISTS(
SELECT *
FROM {self.fieldvalue_db_view}
WHERE entry_id = {self.db_table}.id AND
{self.fieldvalue_db_view}.field_id = {field.id} AND
{self.fieldvalue_db_view}.{text_search_column} SIMILAR TO %(pattern)s
)
''', dict(pattern='%' + val + '%'))
When
val = '%%john smith%%' or val = '%%john%smith%%'
it did not return a result like 'John smith', but if
val = '%%john%smith%|%smith%john%%'
it returns results with 'John smith'.
How this case sensitiveness can be solved?
Thanks,


you can use ilike:
EXISTS(
SELECT
1
FROM
{ self.fieldvalue_db_view }
WHERE
entry_id = { self.db_table }.id
AND { self.fieldvalue_db_view }.field_id = { field.id }
AND { self.fieldvalue_db_view }.{ text_search_column } ILIKE %(pattern) s
)
also you don't need to return all columns when you checking existence of a row , instead pass a constant which is cheaper

Related

error with dynamo occurred (ValidationException) when calling the Query operation: Invalid KeyConditionExpression:

I am a bit new to dynamodb
See error I get when trying to get the max id of my dynamodb table in python lambda function using instructions in below StackOverflow post in below link
Dynamodb max value
An error occurred (ValidationException) when calling the Query operation: Invalid KeyConditionExpression: The expression can not be empty;\"}"
see my lambda function code below
import json
import boto3
TABLE_NAME = 'user-profiles'
dynamo_DB = boto3.resource('dynamodb')
def lambda_handler(event, context):
user_id = event['user_id']
email = event['email']
bvn = event['bvn']
password = event['password']
phone = event['phone']
gender = event['gender']
output = ''
if len(user_id) > 1 and len(password) > 5:
try:
table = dynamo_DB.Table(TABLE_NAME)
values = list(table.query(
KeyConditionExpression='',
ScanIndexForward=False,
Limit=1
)
)
max_id = values[0]['id']
new_id = max_id + 1
Item = {
'id': str(new_id),
'profile-id': str(new_id),
'user_id': user_id,
'email': email,
'bvn': bvn,
'password': password,
'phone': phone,
'gender': gender
}
table.put_item(Item=Item)
output += 'Data Inserted To Dynamodb Successfully'
except Exception as e:
output += 'error with dynamo registration ' + str(e)
# print(output)
else:
output += 'invalid user or password entered, this is ' \
'what i received:\nusername: ' \
+ str(user_id) + '\npassword: ' + str(password)
return {
"statusCode": 200,
"body": json.dumps({
"message": output,
}),
}
# print(output)

You cannot query with empty KeyConditionExpression, if you need to read all records from the table you need to use scan. But you cannot use ScanIndexForward there to order records forward.
Seems like you're trying to implement primary key incrementation. I want to warn you, your solution is not really awesome, because you easily can hit a race condition.
What I would suggest:
I guess you are using id as a primary key (aka partition key). it's okay. what I would do is upsert an extra record in the table, with say increment value:
increment = table.update_item(
Key={'id': 'increment'},
UpdateExpression='ADD #increment :increment',
ExpressionAttributeNames={'#increment': 'increment'},
ExpressionAttributeValues={':increment': 1},
ReturnValues='UPDATED_NEW',
)
new_id = increment['Attributes']['increment']
This query will update the existing record with id: 'increment' and store a new incremented number in the record, if it is the very first query the record will be created with increment: 1 and subsequent calls will increment it. ReturnValues means the query will return the result after the update and you will get a new id.
put the code in place instead of where you query the last record
so your code would look like:
import json
import boto3
TABLE_NAME = 'user-profiles'
dynamo_DB = boto3.resource('dynamodb')
def lambda_handler(event, context):
user_id = event['user_id']
email = event['email']
bvn = event['bvn']
password = event['password']
phone = event['phone']
gender = event['gender']
output = ''
if len(user_id) > 1 and len(password) > 5:
try:
table = dynamo_DB.Table(TABLE_NAME)
increment = table.update_item(
Key={'id': 'increment'},
UpdateExpression='ADD #increment :increment',
ExpressionAttributeNames={'#increment': 'increment'},
ExpressionAttributeValues={':increment': 1},
ReturnValues='UPDATED_NEW',
)
new_id = increment['Attributes']['increment']
Item = {
'id': str(new_id),
'profile-id': str(new_id),
'user_id': user_id,
'email': email,
'bvn': bvn,
'password': password,
'phone': phone,
'gender': gender
}
table.put_item(Item=Item)
output += 'Data Inserted To Dynamodb Successfully'
except Exception as e:
output += 'error with dynamo registration ' + str(e)
# print(output)
else:
output += 'invalid user or password entered, this is ' \
'what i received:\nusername: ' \
+ str(user_id) + '\npassword: ' + str(password)
return {
"statusCode": 200,
"body": json.dumps({
"message": output,
}),
}
# print(output)
and you're good.
Extra thoughts:
And to be 100% sure that there is no race condition on incrementation, you can implement a locking mechanism this way: Before incrementing, put an extra record with id value lock and lock attribute with any value, and use ConditionExpression='attribute_not_exists(lock)'. Then make an increment and then release the lock by removing the record lock. So while the record is there the second attempt to 'make a lock' would break by the condition that attribute lock exists and throw error ConditionalCheckFailedException (you can catch the error and show to a user that the record is locked or whatever.)
Here is an example in JavaScript sorry:
module.exports.DynamoDbClient = class DynamoDbClient {
constructor(tableName) {
this.dynamoDb = new DynamoDB.DocumentClient();
this.tableName = tableName;
}
async increment() {
await this.lock();
const {Attributes: {increment}} = await this.dynamoDb.update({
TableName: this.tableName,
Key: {id: 'increment'},
UpdateExpression: 'ADD #increment :increment',
ExpressionAttributeNames: {'#increment': 'increment'},
ExpressionAttributeValues: {':increment': 1},
ReturnValues: 'UPDATED_NEW',
}).promise();
await this.unlock();
return increment;
}
async lock(key) {
try {
await this.dynamoDb.put({
TableName: this.tableName,
Item: {id: 'lock', _lock: true},
ConditionExpression: 'attribute_not_exists(#lock)',
ExpressionAttributeNames: {'#lock': '_lock'},
}).promise();
} catch (error) {
if (error.code === 'ConditionalCheckFailedException') {
throw new LockError(`Key is locked.`);
}
throw error;
}
}
unlock() {
return this.delete({id: 'lock'});
}
async delete(key) {
await this.dynamoDb.delete({
TableName: this.tableName,
Key: key,
}).promise();
}
}
// usage
const client = new DynamoDbClient('table');
const newId = await client.increment();
...

when passed a dictionary into jinja2 template single apostrophe(') is converted into "'"

JavaScript is throwing an error 'Uncaught Syntax Error: Unexpected token '&''
when debugged in Views.py I got he data with proper Apostrophes.
def newEntry(request):
assert isinstance(request, HttpRequest)
i = 1
for x in lines:
for line in x:
cursor.execute("select distinct regionn FROM [XYZ].[dbo].[Errors] where [Linne] like '%" +line+ "%'")
region[i] = cursor.fetchall()
i = i+1
return render(
request,
'app/newEntry.html',
{
'title': 'New Entry',
'year':datetime.now().year,
'lines': lines,
'regions': region,
}
)
and here is my JS code
var Regions= {{regions}}
function changecat(value) {
if (value.length == 0) document.getElementById("category").innerHTML = "<option>default option here</option>";
else {
var catOptions = "";
for (categoryId in Regions[value]) {
catOptions += "<option>" + categoryId+ "</option>";
}
document.getElementById("category").innerHTML = catOptions;
}
}
Thanks in advance, if this is not a best practice to carry data, suggest me some best process which fills my requirement

Selecting from DB where some data points can be NULL (psycopg2)

I have the following query, where some of the values I am trying to select with can be empty and therefore default to None.
So I've come up with something like this:
db.cursor.execute(
'''
SELECT
s.prod_tires_size_id as size_id
,s.rim_mm
,s.rim_inch
,s.width_mm
,s.width_inch
,s.aspect_ratio
,s.diameter_inch
FROM product_tires.sizes s
WHERE
s.rim_mm %(rim_mm)s
AND s.rim_inch %(rim_inch)s
AND s.width_mm %(width_mm)s
AND s.width_inch %(width_inch)s
AND s.aspect_ratio %(aspect_ratio)s
AND s.diameter_inch %(diameter_inch)s
''', {
'rim_mm': data['RIM_MM'] or None,
'rim_inch': data['RIM_INCH'] or None,
'width_mm': data['WIDTH_MM'] or None,
'width_inch': data['WIDTH_INCH'] or None,
'aspect_ratio': data['ASPECT_RATIO'] or None,
'diameter_inch': data['OVL_DIAMETER'] or None,
}
)
However, = NULL does not work.
If I use IS, then it will not match the values I am proving.
How can I solve this problem?

Generate the query using python:
k = {'rim_mm': 'RIM_MM',
'rim_inch': 'RIM_INCH',
'width_mm': 'WIDTH_MM',
'width_inch': 'WIDTH_INCH',
'aspect_ratio': 'ASPECT_RATIO',
'diameter_inch': 'OVL_DIAMETER',
}
where = []
for column, key in k.items():
if data[key]:
where.append("%s=%%(%s)s" % (column,key)")
else:
where.append("%s IS NULL" % column)
sql = "your select where " + " AND ".join(where)
cursor.execute( sql, data )

How to ignore "IndexError: list index out of range" on SQL Execute Insert statement

I am working with Python 2.7 to extract data from a JSON API and push it into a SQL-Server table.
I am having trouble with inserting data into the database where some of the entries returned from the JSON response are missing a section of the dictionary. IE, "CustomFields": 90% of the entries have information, however 10% don't therefore I get an index error
eg
"CustomFields":[
],
vs
"CustomFields":[
{
"Type":"foo",
"Name":"foo",
"Value":"foo"
},
{
"Type":"foo",
"Name":"foo",
"Value":"foo"
},
{
"Type":"foo",
"Name":"foo",
"Value":"foo"
},
What would I change so that if I get a missing index, replace those with 'NULL' entries into the database.
response = '*API URL*'
json_response = json.loads(urllib2.urlopen(response).read())
conn = pypyodbc.connect(r'Driver={SQL Server};Server=*Address*;Database=*DataBase*;Trusted_Connection=yes;')
conn.autocommit = False
c = conn.cursor()
c.executemany("INSERT INTO phil_targetproccess (ResourceType, Id, Name, StartDate, EndDate, TimeSpent, CreateDate, ModifyDate, LastStateChangeDate, ProjectName, EntityStateName, RequestTypeName, AssignedTeamMember#1, Area, SubArea, BusinessTeam) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)" ,
[(x['ResourceType'],
x['Id'],
x['Name'],
(parse_date(x['StartDate'])),
(parse_date(x['EndDate'])),
x['TimeSpent'],
(parse_date(x['CreateDate'])),
(parse_date(x['ModifyDate'])),
(parse_date(x['LastStateChangeDate'])),
x['Project']['Name'],
x['EntityState']['Name'],
x['RequestType']['Name'],
y['GeneralUser']['FirstName']+' '+y['GeneralUser']['LastName'],
x['CustomFields'][0]['Value'],
x['CustomFields'][1]['Value'],
x['CustomFields'][2]['Value'])
for x in json_response['Items']
for y in x['Assignments']['Items']])
Many thanks.

I think your issue is here
x['CustomFields'][0]['Value'],
x['CustomFields'][1]['Value'],
x['CustomFields'][2]['Value']
When CustomFields has no elements
Try
x['CustomFields'][0]['Value'] if len(x['CustomFields']) > 0 else '',
x['CustomFields'][1]['Value'] if len(x['CustomFields']) > 1 else '',
x['CustomFields'][2]['Value'] if len(x['CustomFields']) > 2 else '',

You can use get method to check whether that value in CustomFields
is available if so check the length of the list and then get the value of the dictionary in that list using the same get method.
For example:
customfield_value = (x['CustomFields'][0]).get("Value",None) if len(x['CustomFields'])>0 else None
This will return None if the value is not present in the index 0. you can follow the same for getting values from other 2 indices. If you didn't understand please comment it 'll explain further.

Final Script. Thanks for the help!
c.executemany("INSERT INTO phil_targetproccess (ResourceType, Id, Name, StartDate, EndDate, TimeSpent, CreateDate, "
"ModifyDate, LastStateChangeDate, ProjectName, EntityStateName, RequestTypeName, AssignedTeamMember1, "
"AssignedTeamMember2, AssignedTeamMember3, AssignedTeamMember4, Area, SubArea, BusinessTeam) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
[(x['ResourceType'],
x['Id'],
x['Name'],
(parse_date(x['StartDate'])),
(parse_date(x['EndDate'])),
x['TimeSpent'],
(parse_date(x['CreateDate'])),
(parse_date(x['ModifyDate'])),
(parse_date(x['LastStateChangeDate'])),
x['Project']['Name'],
x['EntityState']['Name'],
x['RequestType']['Name'],
x['Assignments']['Items'][0]['GeneralUser']['FirstName'] + ' ' + x['Assignments']['Items'][0]['GeneralUser']['LastName'] if len(x['Assignments']['Items']) > 0 else None,
x['Assignments']['Items'][1]['GeneralUser']['FirstName'] + ' ' + x['Assignments']['Items'][1]['GeneralUser']['LastName'] if len(x['Assignments']['Items']) > 1 else None,
x['Assignments']['Items'][2]['GeneralUser']['FirstName'] + ' ' + x['Assignments']['Items'][2]['GeneralUser']['LastName'] if len(x['Assignments']['Items']) > 2 else None,
x['Assignments']['Items'][3]['GeneralUser']['FirstName'] + ' ' + x['Assignments']['Items'][3]['GeneralUser']['LastName'] if len(x['Assignments']['Items']) > 3 else None,
x['CustomFields'][0]['Value'] if len(x['CustomFields']) > 0 else '',
x['CustomFields'][1]['Value'] if len(x['CustomFields']) > 1 else '',
x['CustomFields'][2]['Value'] if len(x['CustomFields']) > 2 else '')
for x in json_response['Items']])

Python utility for parsing blocks?

I have a file that starts something like:
databaseCons = {
main = {
database = "readable_name",
hostname = "hostdb1.serv.com",
instances= {
slaves = {
conns = "8"
}
}
maxconns = "5",
user = "user",
pass = "pass"
}
}
So, what I'd like to do is parse this out into a dict of sub-dicts, something like:
{'main': {'database': 'readable_name', 'hostname': 'hostdb1.serv.com', 'maxconns': '5', 'instances': {'slave': {'maxCount': '8'}}, 'user': 'user', 'pass': 'pass'}}
I think the above makes sense... but please feel free to edit this if it doesn't. Basically I want the equivalent of:
conns = '8'
slave = dict()
slave['maxCount'] = conns
instances = dict()
instances['slave'] = slave
database = 'readable_name'
hostname = 'hostdb1.serv.com'
maxconns = '5'
user = 'user'
pas = 'pass'
main = dict()
main['database'] = database
main['hostname'] = hostname
main['instances'] = instances
main['maxconns'] = maxconns
main['user'] = user
main['pass'] = pas
databaseCons = dict()
databaseCons['main'] = main
Are there any modules out there that can handle this sort of parsing? Even what I've suggested above looks messy.. there's got to be a better way I'd imagine.

Here is a pyparsing parser for your config file:
from pyparsing import *
def to_dict(t):
return {k:v for k,v in t}
series = Forward()
struct = Suppress('{') + series + Suppress('}')
value = quotedString.setParseAction(removeQuotes) | struct
token = Word(alphanums)
assignment = Group(token + Suppress('=') + value + Suppress(Optional(",")))
series << ZeroOrMore(assignment).setParseAction(to_dict)
language = series + stringEnd
def config_file_to_dict(filename):
return language.parseFile(filename)[0]
if __name__=="__main__":
from pprint import pprint
pprint(config_file_to_dict('config.txt'))

Categories

Resources