Hello guys this will be my first post here as I am learning how to code. When I try to update my table in Dynamodb using a lambda function I get the following error message. "The provided key element does not match the schema" my table name is correct and I am able to connect to it. My primary key is just a hash key which is id. its value is 1 so I do not see why it is giving me this error here.
import json
import boto3
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('Visitors')
def lambda_handler (event, context):
response = table.update_item(
Key={
"id": {"N":"1"}
},
ExpressionAttributeNames = {
"#c": "Counters"
},
UpdateExpression= "set #c = :val",
ExpressionAttributeValues={
":val": {"N":"1"}
}
)
Since you are using the table resource, you should refer to this documentation. For example, the Key parameter should have the following syntax:
Key={
'string': 'string'|123|Binary(b'bytes')|True|None|set(['string'])|set([123])|set([Binary(b'bytes')])|[]|{}
}
This means that the DynamoDB data type is inferred from the Python data type. So instead of {"N":"1"}, you can use 1 directly. Here is a corrected version of your code snippet:
import json
import boto3
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('Visitors')
def lambda_handler (event, context):
response = table.update_item(
Key={
"id": 1
},
ExpressionAttributeNames = {
"#c": "Counters"
},
UpdateExpression= "set #c = :val",
ExpressionAttributeValues={
":val": 1
}
)
Related
I have an object in DynamoDB:
{ 'UserID' : 'Hank', ConnectionList : {'con1', 'con2'} }
By using boto3 in lambda functions, I would like to add 'con3' to the String Set.
So far, I have been trying with the following code without success:
ddbClient = boto3.resource('dynamodb')
table = ddbClient.Table("UserInfo")
table.update_item(
Key={
"UserId" : 'Hank'
},
UpdateExpression =
"SET ConnectionList = list_append(ConnectionList, :i)",
ExpressionAttributeValues = {
":i": { "S": "Something" }
},
ReturnValues="ALL_NEW"
)
However, no matter the way I try to put the information inside the String Set, it always runs error.
Since you're using the resource API, you have to use the Python data type set in your statement:
table.update_item(
Key={
"UserId" : 'Hank'
},
UpdateExpression =
"ADD ConnectionList :i",
ExpressionAttributeValues = {
":i": {"Something"}, # needs to be a set type
},
ReturnValues="ALL_NEW"
)
I have a table that contains an item with the following attributes:
{
"country": "USA",
"names": [
"josh",
"freddy"
],
"phoneNumber": "123",
"userID": 0
}
I'm trying to query an item in a DynameDB by looking for a name using python. So I would write in my code that the item I need has "freddy" in the field "names".
I saw many forums mentioning "contains" but none that show an example...
My current code is the following:
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('users_table')
data = table.query(
FilterExpression: 'names = :name',
ExpressionAttributeValues: {
":name": "freddy"
}
)
I obviously cannot use that because "names" is a list and not a string field.
How can I look for "freddy" in names?
Since names field isn't part of the primary key, so you can't use query. The only way to look for an item by names is to use scan.
import boto3
from boto3.dynamodb.conditions import Key, Attr
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('users_table')
data = table.scan(
FilterExpression=Attr('names').contains('freddy')
)
It is my first encounter with DynamoDB and I have been given a JSON File that looks like:
{
"metadata":{
"schemaVersion":"1.0",
"importType":"LEX",
"importFormat":"JSON"
},
"resource":{
"description":"First Names",
"name":"ASDUKfirstNames",
"version":"1",
"enumerationValues":[
{
"value":"Zeshan"
},
{
"value":"Zoe"
},
{
"value":"Zul"
}
],
"valueSelectionStrategy":"ORIGINAL_VALUE"
}
}
and I want to import the data where value = FirstName in the DynamoDB Table that I have created named customerDetails that contains items CustomerID, FirstName and LastName.
Is there a way to utilize the boto3 put-item function to loop over the contents of the JSON file replacing value with FirstName?
You should use Python to do the data transformation. You can find the boto3 DDB docs here.
import boto3
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('customerDetails')
json_data = { ... load the data into a dict here ... }
for enumeration_value in json_data['resouce']['enumerationValues']:
ddb_item = {
"CustomerID": 123,
"FirstName": enumeration_value['value']]
}
table.put_item(Item=ddb_item)
I would like to write the following data group to Dynamodb.
There are about 100 data. Since images are not necessarily required, there is a mixture with and without the image_url element.
(questionsList.json)
{
"q_id" : "001",
"q_body" : "Where is the capital of the United States?",
"q_answer" : "Washington, D.C.",
"image_url" : "/Washington.jpg",
"keywords" : [
"UnitedStates",
"Washington"
]
},
{
"q_id" : "002",
"q_body" : "Where is the capital city of the UK?",
"q_answer" : "London",
"image_url" : "",
"keywords" : [
"UK",
"London"
]
},
Since it is the writing test phase, Dynamodb to write to is prepared in localhost:8000 using the serverless-dynamodb-local plugin of the serverless framework, not the production environment.
In order to write the above JSON data to this Dynamodb, I wrote the following code in Boto 3 (AWS SDK for Python).
from __future__ import print_function
import boto3
import codecs
import json
dynamodb = boto3.resource('dynamodb', region_name='us-east-1', endpoint_url="http://localhost:8000")
table = dynamodb.Table('questionListTable')
with open("questionList.json", "r", encoding='utf-8') as json_file:
items = json.load(json_file)
for item in items:
q_id = item['q_id']
q_body = item['q_body']
q_answer = item['q_answer']
image_url = item['image_url']
keywords = item['keywords']
print("Adding detail:", q_id, q_body)
table.put_item(
Item={
'q_id': q_id,
'q_body': q_body,
'q_answer': q_answer,
'image_url': image_url,
'keywords': keywords,
}
)
When this code is executed, the following error occurs in the null character part.
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the PutItem operation: One or more parameter values were invalid: An AttributeValue may not contain an empty string
Apparently it seems to be caused by JSON 's null character.
If you exclude the image_url containing the null character from the target of writing as below, the writing is completed without any problem.
from __future__ import print_function
import boto3
import codecs
import json
dynamodb = boto3.resource('dynamodb', region_name='us-east-1', endpoint_url="http://localhost:8000")
table = dynamodb.Table('questionListTable')
with open("questionList.json", "r", encoding='utf-8') as json_file:
items = json.load(json_file)
for item in items:
q_id = item['q_id']
q_body = item['q_body']
q_answer = item['q_answer']
#image_url = item['image_url']
keywords = item['keywords']
print("Adding detail:", q_id, q_body)
table.put_item(
Item={
'q_id': q_id,
'q_body': q_body,
'q_answer': q_answer,
#'image_url': image_url,
'keywords': keywords,
}
)
Since DynamoDB is NoSQL, there may be other methods that make good use of the characteristics, but how to correct the code to write the above data ignoring empty characters? I would like to say "if image_url exists, write it if it does not, ignore it."
Thank you.
I solved my problem. You can set null as follows.
from __future__ import print_function
import boto3
import codecs
import json
dynamodb = boto3.resource('dynamodb', region_name='ap-northeast-1', endpoint_url="http://localhost:8000")
table = dynamodb.Table('questionListTable')
with open("questionList.json", "r", encoding='utf-8_sig') as json_file:
items = json.load(json_file)
for item in items:
q_id = item['q_id']
q_body = item['q_body']
q_answer = item['q_answer']
image_url = item['image_url'] if item['image_url'] else None
keywords = item['keywords'] if item['keywords'] else None
print("Adding detail:", q_id, q_body)
table.put_item(
Item={
'q_id': q_id,
'q_body': q_body,
'q_answer': q_answer,
'image_url': image_url,
'keywords': keywords,
}
)
In order to check the situation of Dynamodb, use the offline plugin of the serverless framework to run the API Gateway in the local environment. When I actually called the API using Postman, Null was properly inserted in the value.
{
"q_id" : "001",
"q_body" : "Where is the capital of the United States?",
"q_answer" : "Washington, D.C.",
"image_url" : "/Washington.jpg",
"keywords" : [
"UnitedStates",
"Washington"
]
},
{
"q_id" : "002",
"q_body" : "Where is the capital city of the UK?",
"q_answer" : "London",
"image_url" : "null",
"keywords" : [
"UK",
"London"
]
},
I'm a new user in boto3 and i'm using DynamoDB.
I went through over the DynamoDB api and I couldn't find any method which tell me if a table is already exists.
What is the best approach dealing this issue?
Should I try to create a new table and wrap it using try catch ?
From reading the documentation, I can see that there are three methods by which you can check if a table exists.
The CreateTable API throws an error ResourceInUseException if the table already exists. Wrap the create_table method with try except to catch this
You can use the ListTables API to get the list of table names associated with the current account and endpoint. Check if the table name is present in the list of table names you get in the response.
The DescribeTable API will throw an error ResourceNotFoundException if the table name you request doesn't exist.
To me, the first option sounds better if you just want to create a table.
Edit:
I see that some people are finding it difficult to catch the exceptions. I will put some code below for you to know how to handle exceptions in boto3.
Example 1
import boto3
dynamodb_client = boto3.client('dynamodb')
try:
response = dynamodb_client.create_table(
AttributeDefinitions=[
{
'AttributeName': 'Artist',
'AttributeType': 'S',
},
{
'AttributeName': 'SongTitle',
'AttributeType': 'S',
},
],
KeySchema=[
{
'AttributeName': 'Artist',
'KeyType': 'HASH',
},
{
'AttributeName': 'SongTitle',
'KeyType': 'RANGE',
},
],
ProvisionedThroughput={
'ReadCapacityUnits': 5,
'WriteCapacityUnits': 5,
},
TableName='test',
)
except dynamodb_client.exceptions.ResourceInUseException:
# do something here as you require
pass
Example 2
import boto3
dynamodb_client = boto3.client('dynamodb')
table_name = 'test'
existing_tables = dynamodb_client.list_tables()['TableNames']
if table_name not in existing_tables:
response = dynamodb_client.create_table(
AttributeDefinitions=[
{
'AttributeName': 'Artist',
'AttributeType': 'S',
},
{
'AttributeName': 'SongTitle',
'AttributeType': 'S',
},
],
KeySchema=[
{
'AttributeName': 'Artist',
'KeyType': 'HASH',
},
{
'AttributeName': 'SongTitle',
'KeyType': 'RANGE',
},
],
ProvisionedThroughput={
'ReadCapacityUnits': 5,
'WriteCapacityUnits': 5,
},
TableName=table_name,
)
Example 3
import boto3
dynamodb_client = boto3.client('dynamodb')
try:
response = dynamodb_client.describe_table(TableName='test')
except dynamodb_client.exceptions.ResourceNotFoundException:
# do something here as you require
pass
import boto3
from botocore.exceptions import ClientError
TABLE_NAME = "myTableName"
dynamodb = boto3.resource('dynamodb', endpoint_url="https://dynamodb.us-east-1.amazonaws.com")
table = dynamodb.Table(TABLE_NAME)
try:
response = client.describe_table(TableName=TABLE_NAME)
except ClientError as ce:
if ce.response['Error']['Code'] == 'ResourceNotFoundException':
print "Table " + TABLE_NAME + " does not exist. Create the table first and try again."
else:
print "Unknown exception occurred while querying for the " + TABLE_NAME + " table. Printing full error:"
pprint.pprint(ce.response)
Alternate approach if you do not want to use boto3.client but only boto3.resource:
import boto3
database = boto3.resource('dynamodb', endpoint_url="http://localhost:8000")
table_name = 'MyTable'
table_names = [table.name for table in database.tables.all()]
if table_name in table_names:
print('table', table_name, 'exists')
You can use describe table API to determine whether the table exists.
Sample code:
from __future__ import print_function # Python 2/3 compatibility
import os
os.environ["TZ"] = "UTC"
import boto3
client = boto3.client('dynamodb', region_name='us-west-2', endpoint_url="http://localhost:8000")
response = client.describe_table(
TableName='Movies'
)
print(response)
If table exists:-
You will get the response
If table doesn't exists:-
You will get ResourceNotFoundException
botocore.errorfactory.ResourceNotFoundException: An error occurred (ResourceNotF
oundException) when calling the DescribeTable operation: Cannot do operations on
a non-existent table
Another way:-
Alternatively, you could use table.wait_until_exists(). from the docs:
Waits until this Table is exists. This method calls
DynamoDB.Waiter.table_exists.wait() which polls.
DynamoDB.Client.describe_table() every 20 seconds until a successful
state is reached. An error is returned after 25 failed checks.
See also: AWS API Documentation
Request Syntax
table.wait_until_exists()
Returns
None
You can use .table_status attr of any boto3 Table instance object. It returns it's status if exists (CREATING, UPDATING, DELETING, ACTIVE) or throws exception botocore.exceptions.ClientError: Requested resource not found: Table: <YOUR_TABLE_NAME> not found. You can wrap those conditions into try / except to have full info on the current table state.
import boto3
from botocore.exceptions import ClientError
dynamodb = boto3.resource('dynamodb', region_name='us-west-2')
table = dynamodb.Table('your_table_name_str')
try:
is_table_existing = table.table_status in ("CREATING", "UPDATING",
"DELETING", "ACTIVE")
except ClientError:
is_table_existing = False
print "Table %s doesn't exist." % table.name
Note that it kind of depends on if you are using client or resource. If you use boto3.client(), you can use the 3 methods the accepted answer suggested. If you are using boto3.resource(), you can only use dynamodb_resource.create_table() and check exceptions.
try:
table = dynamodb_resource.create_table(
...
)
table.meta.client.get_waiter('table_exists').wait(TableName=your_table_name)
except ResourceInUseException:
# do sth here
I know this will have a slight risk if there are more than 10 tables between table.split(0, -1) and table. However, it does save throwing exceptions and the like.
Alas the documentation https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_ListTables.html (examples) imply the the first table in the returned list will be the search item, it is not the case..
class Dynamo {
private m_db!: DynamoDB;
private async ensure_table(name: string) {
const search = await this.db().listTables({
ExclusiveStartTableName: name.slice(0, -1),
Limit: 10 });
const exists = search.TableNames?.includes(name);
exists || await this.create_table(name);
}
private async create_table(name: string) {
// create the table here
}
private db(): DynamoDB {
return this.m_db || (this.m_db = this.create_db());
}
private create_db = (): DynamoDB => {
return new DynamoDB({apiVersion: "2012-08-10"}); }
}
}
You can use the convenient resource API while still handling and catching the exceptions from the client API level, because you can access the client from the resource! This makes the method of checking if table exists the most elegant I have found:
resource = boto3.resource('dynamodb', region_name='eu-north-1')
def table_exists(table_name: str) -> bool:
try:
resource.Table(table_name).table_status
except resource.meta.client.exceptions.ResourceNotFoundException:
return False
return True
print(table_exists('dummy_table'))