I need help on a little problem that I have with my AWS Lambda function. This function queries my AWS Athena database.
The code looks like this :
import json
import boto3
import time
def lambda_handler(event, context):
client = boto3.client('athena')
QueryResponse = client.start_query_execution(
QueryString = "MY QUERY;",
QueryExecutionContext = {
'Database' : 'myDatabase'
},
ResultConfiguration = {
'OutputLocation' : 's3://mys3Bucket'
}
)
#Oberserve results :
queryId = QueryResponse['QueryExecutionId']
The code works great, but I am having some troubles with the "WHERE" part of my sql query (that is a long one)
Here is the part of my Query :
WHERE x.id_date > cast(date_format(date_trunc('day', current_timestamp -
interval '3' day), '%Y%m%d') as integer)
and x.id_date <= cast(date_format(current_timestamp, '%Y%m%d') as integer)
and c.label = 'NAME'
My query is written on a single line to fit the Python code replacing "MY QUERY".
Le problem is :
I need to replace the 'NAME' part by a variable (string) that will be given to my Lambda. I tried to use %s to replace by the given variable, but as there is '%Y%m%d' in my query, the code is waiting for string to replace these part too, but it is just made to format the date as I want to. I tried to replace NAME by a string and it works perfectly so I know my query is not the problem. I tried to put 'c.label = '%s' in first to see if it the % method would simply replace the first %s and let the other ones do their job but it didn't work.
So my question is : How can I replace 'NAME' by a str variable ?can I do this keeping my query on a single line ? (if yes, how ?) or at least how can I divide my query in different lines I could interact with ?
Thanks for your help.
As said in comment, the solution was to use :
MyString = 'my string to replace in query'
QueryString = f"SELECT * FROM {MyString};"
Related
I have this method that creates the query and passes two string parameters. But when I test this it has escape characters '' before the single quote '''.
The query can only accept native queries in string form
I also tried string.replace method but doesnt work
replace('\\', '')
Here is the code
def update_query(self, status, row_id):
return '''UPDATE TABLE SET STATUS = {0} WHERE ID = {1}'''.format(status, row_id)
Here is the sample output:
'UPDATE TABLE SET STATUS = 'Success' WHERE ID = 1'
Thank you
You can also use f-string for formatting your string
def update_query(self,status, row_id):
return f"UPDATE TABLE SET STATUS = '{status}' WHERE ID = {row_id}"
>>> update_query("Success",1)
"UPDATE TABLE SET STATUS = 'Success' WHERE ID = 1"
I think you absolutely should be using prepared statements here, which the other answers don't seem to be recommending (for whatever reason). Try using something along these lines:
sql = "UPDATE TABLE SET STATUS = :status WHERE ID = :id"
cursor.prepare(sql)
cursor.execute(None, {'status':status, 'id':row_id})
One advantage of using prepared statements here is that it frees the user from having to worry about how to properly escape the literal placeholders in the query. Instead, we only need to bind a variable with the correct type to the statement, and Oracle will handle the rest.
you need to add \ in the code
def update_query(self, status, row_id):
return '''UPDATE TABLE SET STATUS = \'{0}\' WHERE ID = {1}'''.format(status, row_id)
I am using Sqlalchemy 1.3 to connect to a PostgreSQL 9.6 database (through Psycopg).
I have a very, very raw Sql string formatted using Psycopg2 syntax which I can not modify because of some legacy issues:
statement_str = SELECT * FROM users WHERE user_id=%(user_id)s
Notice the %(user_id)s
I can happily execute that using a sqlalchemy connection just by doing:
connection = sqlalch_engine.connect()
rows = conn.execute(statement_str, user_id=self.user_id)
And it works fine. I get my user and all is nice and good.
Now, for debugging purposes I'd like to get the actual query with the %(user_id)s argument expanded to the actual value. For instance: If user_id = "foo", then get SELECT * FROM users WHERE user_id = 'foo'
I've seen tons of examples using sqlalchemy.text(...) to produce a statement and then get a compiled version. I have that thanks to other answers like this one or this one been able to produce a decent str when I have an SqlAlchemy query.
However, in this particular case, since I'm using a more cursor-specific syntax %(user_id) I can't do that. If I try:
text(statement_str).bindparams(user_id="foo")
I get:
This text() construct doesn't define a bound parameter named 'user_id'
So I guess what I'm looking for would be something like
conn.compile(statement_str, user_id=self.user_id)
But I haven't been able to get that.
Not sure if this what you want but here goes.
Assuming statement_str is actually a string:
import sqlalchemy as sa
statement_str = "SELECT * FROM users WHERE user_id=%(user_id)s"
params = {'user_id': 'foo'}
query_text = sa.text(statement_str % params)
# str(query_text) should print "select * from users where user_id=foo"
Ok I think I got it.
The combination of SqlAlchemy's raw_connection + Psycopg's mogrify seems to be the answer.
conn = sqlalch_engine.raw_connection()
try:
cursor = conn.cursor()
s_str = cursor.mogrify(statement_str, {'user_id': self.user_id})
s_str = s_str.decode("utf-8") # mogrify returns bytes
# Some cleanup for niceness:
s_str = s_str.replace('\n', ' ')
s_str = re.sub(r'\s{2,}', ' ', s_str)
finally:
conn.close()
I hope someone else finds this helpful
Our environment uses Python 2.7 along with BigQuery library 0.27.0.
The query that is to be submitted is part of a JSON string which is loaded by
json.loads(json_blob)
then the value for query is extracted from a key:
query_str = json_blob["sql_command"]
Printing the query_str gives the following value:
('query_str: ', '" select distinct id from my_table where step_count > 3 and lower(name) = \'test\') "')
When the script submits the query for execution as following:
job = self.bq_client.run_async_query(job_id, query_str, udf_resources=udf_obj, query_parameters=query_params)
BigQuery job comes back with an error, and when I lookup the job information using the job_id on https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/get I see that the query that is actually executed is as following:
select distinct id from my_table where step_count \u003e 3 and lower(name) = 'test')
I have read on encoding/decoding and tried them, and doesn't make a diference.
Is there a way that I can convert that query_str retrieved from the json_blob (query_str = json_blob["sql_command"]) to a true string? We know that when we define such query as a string (hard-coded in the script rather than retrieved from a key in a JSON blob) the query gets executed successfully, ex.
query_str = """select distinct id from my_table where step_count > 3 and lower(name) = 'test')"""
Any suggestions is greatly appreciated.
I'm not able to reproduce this using Python 2.7:
$ python
>>> from google.cloud import bigquery
>>> import json
>>> client = bigquery.Client('<project name redacted>')
>>> json_blob = json.loads('{"sql_command":"select distinct id from my_table where step_count > 3 and lower(name) = \'test\'"}')
>>> query_str = json_blob["sql_command"]
>>> query_job = client.query(query_str)
>>> rows = query_job.result()
I get an error that my_table can't be resolved (as expected), but no syntax error. Syntax validation happens prior to table resolution. Checking the job information, I see:
$ bq --format=prettyjson show -j <job id>
{
"configuration": {
"query": {
"priority": "INTERACTIVE",
"query": "select distinct id from my_table where step_count > 3 and lower(name) = 'test'",
"useLegacySql": false
}
},
so there was no problem with passing the query to BigQuery. Some suggestions:
Check the code points in the string; maybe it doesn't have the content that you think it does:
print [ord(c) for c in query_str]
The code point for the greater than sign is 62, for example, so you should see it in the output.
Pick a different serialization format rather than JSON and see if that affects the result. Maybe something else in your process is performing escaping without you realizing it, and you can identify the source of the problem by using e.g. protocol buffers instead and seeing if that makes a difference.
Seeing some strange behavior in Pymongo $in query. Looking for records that meet the following query:
speciesCollection.find({"SPCOMNAME":{"$in":['paddlefish','lake sturgeon']}})
The query returns no records.
If I change it to find_one the it works returning the last value for Lake Sturgeon. The field is a text filed with one vaule. So I am looking for records that match paddlefish or Lake Sturgeon.
It works fine in Mongo Shell like this:
speciesCollection.find({SPCOMNAME:{$in: ['paddlefish','lake strugeon']}},{_id:0})
Here is the result from shell
{ "SPECIES_ID" : 1, "SPECIES_AB" : "LKS", "SPCOMNAME" : "lake sturgeon", "SP_SCINAME" : "Acipenser fulvescens
{ "SPECIES_ID" : 101, "SPECIES_AB" : "PAH", "SPCOMNAME" : "paddlefish", "SP_SCINAME" : "Polyodon spathula" }
Am I missing something here?
I think you have a typo or some other error in your program as I just did a test with your sample data and query and it works - see the GIF
Below is my test code which connects to the database called so and the collection speciesCollection, maybe you find the error in yours with it
import pymongo
client = pymongo.MongoClient('dockerhostlinux1', 30000)
db = client.so
coll = db.speciesCollection
result = coll.find({"SPCOMNAME":{"$in":['paddlefish','lake sturgeon']}})
for doc in result:
print(doc)
I am using Lambda (Python) to query my DynamoDB database. I am using the boto3 library, and I was able to make an "equivalent" query:
This script works:
import boto3
from boto3.dynamodb.conditions import Key, Attr
import json
def create_list(event, context):
resource = boto3.resource('dynamodb')
table = resource.Table('Table_Name')
response = table.query(
TableName='Table_Name',
IndexName='Custom-Index-Name',
KeyConditionExpression=Key('Number_Attribute').eq(0)
)
return response
However, when I change the query expression to this:
KeyConditionExpression=Key('Number_Attribute').gt(0)
I get the error:
"errorType": "ClientError",
"errorMessage": "An error occurred (ValidationException) when calling the Query operation: Query key condition not supported"
According to this [1] resource, "gt" is a method of Key(). Does anyone know if this library has been updated, or what other methods are available other than "eq"?
[1] http://boto3.readthedocs.io/en/latest/reference/customizations/dynamodb.html#ref-dynamodb-conditions
---------EDIT----------
I also just tried the old method using:
response = client.query(
TableName = 'Table_Name',
IndexName='Custom_Index',
KeyConditions = {
'Custom_Number_Attribute':{
'ComparisonOperator':'EQ',
'AttributeValueList': [{'N': '0'}]
}
}
)
This worked, but when I try:
response = client.query(
TableName = 'Table_Name',
IndexName='Custom_Index',
KeyConditions = {
'Custom_Number_Attribute':{
'ComparisonOperator':'GT',
'AttributeValueList': [{'N': '0'}]
}
}
)
...it does not work.
Why would EQ be the only method working in these cases? I'm not sure what I'm missing in the documentation.
From what I think:
Your Partition Key is Number_Attribute, and so you cannot do a gt when doing a query (you can do an eq and that is it.)
You can do a gt or between for your Sort Key when doing a query. It is also called Range key, and because it "smartly" puts the items next to each other, it offers the possibility of doing gt and between efficiently in a query
Now, if you want to do a between to your partition Key, then you will have to use scan like the below:
Key('Number_Attribute').gt(0)
response = table.scan(
FilterExpression=fe
)
Keep in mind of the following concerning scan:
The scan method reads every item in the entire table, and returns all of the data in the table. You can provide an optional filter_expression, so that only the items matching your criteria are returned. However, note that the filter is only applied after the entire table has been scanned.
So in other words, it's a bit of a costly operation comparing to query. You can see an example in the documentation here.
Hope that helps!