I have the next code:
users_table = Table(users_table_name, connection=Core.aws_dynamodb_connection)
users_table.put_item(data={
"login": login,
"password": hashlib.sha256(password.encode("utf-8")).hexdigest(),
"profile": profile,
"registration_date": datetime.now() # PROBLEM IS HERE
})
But when I run it, it fails with error:
TypeError: Unsupported type "< type 'datetime.datetime' >" for value "2015-01-12 05:02:57.053131"
I've tried a lot of ways, but it seems that it isn't possible to save datetime to DynamoDB. Btw it works fine in MongoDB.
Is there any solution?
Okay, I see that DynamoDB does not support any date types. So the only solution is to use unix-like time as integer, or save date as string.
According to alejandro-franco response .isoformat() make the trick.
Just tested and this a working example:
CustomerPreferenceTable.put_item(
Item={
"id": str(uuid4()),
"validAfter": datetime.utcnow().isoformat(),
"validBefore": (datetime.utcnow() + timedelta(days=365)).isoformat(),
"tags": ["potato", "eggplant"]
}
)
According to the documentation:
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/JavaSDKHighLevel.html
Date S (string type). The Date values are stored as ISO-8601 formatted
strings.
old post but maybe still interesting ..
What you can do and how it worked for me:
import datetime
from datetime import datetime
...
now = datetime.now()
x = now.strftime("%m/%d/%Y, %H:%M:%S")
table.put_item(
Item={
'Index': Index,
'Stamp': x,
}
)
And with adaption to the code above:
import datetime
from datetime import datetime
...
now = datetime.now()
x = now.strftime("%m/%d/%Y, %H:%M:%S")
users_table = Table(users_table_name, connection=Core.aws_dynamodb_connection)
users_table.put_item(data={
"login": login,
"password": hashlib.sha256(password.encode("utf-8")).hexdigest(),
"profile": profile,
"registration_date": x,
})
My Output
Reading the documentation lately, I found the right link to the documentation here. The recommended way to store date and time data in DynamoDB is using ISO 8601 strings. so the data type is just string.
I'm not sure why datetime isn't supported in DynamoDB, or in fact I have no experience in it neither.
But if you're so insisted in not converting the datetime to string like people suggested, you can convert the datetime to timestamp, and so you can compare with it.
updated
And you may want to read this SO Question, seemed like numeric comparison is the preferred way.
If you want to use date to find users, you can simply invoke date() function. Like this:
...
users_table = Table(users_table_name, connection=Core.aws_dynamodb_connection)
current = datetime.now()
users_table.put_item(data={
"login": login,
"password": hashlib.sha256(password.encode("utf-8")).hexdigest(),
"profile": profile,
# here use a different name for the entry
"registration_time": current
"registration_date": current.date()
})
...
These are all the supported types for attribute values in DynamoDB as listed in their AWS Docs.
B A Binary data type.
Type: Blob
Required: No
BOOL A Boolean data type.
Type: Boolean
Required: No
BS A Binary Set data type.
Type: array of Blobs
Required: No
L A List of attribute values.
Type: array of AttributeValue objects
Required: No
M A Map of attribute values.
Type: String to AttributeValue object map
Required: No
N A Number data type.
Type: String
Required: No
NS A Number Set data type.
Type: array of Strings
Required: No
NULL A Null data type.
Type: Boolean
Required: No
S A String data type.
Type: String
Required: No
SS A String Set data type.
Type: array of Strings
Required: No
with DDB2, seems like Instant is now supported.
https://github.com/aws/aws-sdk-java-v2/tree/master/services-custom/dynamodb-enhanced#changing-update-behavior-of-attributes
Related
Problem:
Trying to get a record using Django ORM, from a table that contains a JSON field, I'm using the following line:
test_object = House.objects.get(id=301)
Error
TypeError: the JSON object must be str, bytes or bytearray, not dict
Possible issue
Noticed that a previous developer updated the format of the JSON field in the table, seems that JSON had a bad format. Script used to format the JSON column:
for i in data:
jsonstring = json.dumps(i.result)
new_clear = jsonstring.replace("\\", "")
new_clear = jsonstring.replace("NaN", "null")
i.result = json.loads(new_clear)
i.save()
Comments
In pgAdmin the JSON field looks good and it is formatted properly, see a partial copy of the JSON below:
{"owner_id": 45897, "first_name": "John", "last_name": "DNC", "estate_id": 3201, "sale_date": "3/18/19", "property_street": "123 main st", "property_city": "Miami", "property_state": "FL", "property_zipcode": 33125, "Input_First_Name": "John", "Input_Last_Name": "DNC"}
I would like to know how to deal with this JSON field in order to query the object. Any help will be appreciated. Thanks.
Check if there's a custom decoder being used in the field (docs reference).
If the json data is valid in db, try connecting to db in shell using psycopg2.connect(), query, and decode using json.loads().
I'd intended to post this as a comment, but not enough reputation, in case this is of any concern.
I have date in String format and Like to convert in ISOdate format in mongoshell or custom python script to achieve this. I tried all possible commands but couldn't achieve this. I like to know how to convert String date into ISOdate in mongodb.
Assuming
your collection name as: youCollection
field name to be converted from string to date: date
var records = db.youCollection.find();
records.forEach(function(doc) {
var dateStr = doc.date;
if(typeof dateStr === "string")
{
var date = new Date(dateStr);
db.youCollection.update({_id: doc._id}, {$set: {date: date}})
}})
This will convert date of "2018-02-17T05:01:32.028Z" format to ISODate("2018-02-17T05:01:32.028Z")
I found the optimal solution for this later
Get into Database
use yourdatabase_name
Run this afterwards changing according to your requirement
db.YOURCOLLECION.find().forEach(function(element) {
element.DATE = new Date(element.DATE);
db.DATE.save(element);
})
I've just started using mongodb via the pyhton drive (pymongo). When I'm posting new data (which actually comes from a MySQL db) some data types appear incorrectly mapped. For example, some single digit number are inserted as long ints, a timestamp is inserted as a string. Also the date which is stored in MySQL as YY-MM-DD is changed to YY-MM-DD 00:00:00 (i.e a time is added).This seems like a waste of space, is this normal procedure for mongodb or should I somehow change the data types which are incorrectly(?) mapped?
ps I did search through the docs as mongodb but I couldn't find anything to match my query.
post = {
"title": video_title,
"ext_id": video_external_id,
"source": video_source,
"date_added": d1,
"views":{
"views_all": views_all,
"views_year": views_yr,
"views_day": views_day,
"views_week": views_wk,
"views_month": views_mo
},
"video_type": 0,
"hd": video_hd,
"features": featured,
"video_thumbs": video_thumbnails,
"video_main_thumb": video_main_thumbnail,
"length": video_length,
"length_sort": video_length,
"rating": {
"rating_all": rating_all,
"rating_year": rating_yr,
"rating_day": rating_day,
"rating_week": rating_wk,
"rating_month": rating_mo
}
}
posts = db.posts
post_id = video_list.insert(post)
For example, some single digit number are inserted as long ints
PyMongo stores python 2.x long as BSON int64 regardless of value, python int as BSON int32 or BSON int64 depending on the value of the int. See the table here for a mapping of python types to BSON types.
a timestamp is inserted as a string
Assuming the timestamp was passed in ISO-8601 format that's correct. If you want to store the timestamp as a BSON datetime pass a python datetime object instead.
Also the date which is stored in MySQL as YY-MM-DD is changed to YY-MM-DD 00:00:00 (i.e a time is added)
BSON (the storage format used by MongoDB) does not have a date type without a time element. It is up to your application to convert dates to datetimes.
I have a NDB datetime property stored on Google App Engine. I'm trying to query for all records since yesterday by converting the datetime to a date, then filtering the query for anything greater or equal to yesterdays date. However, I am getting the following error:
follower_trans = fol.query(
datetime.datetime.date(fol.followed_date) >= self.yesterday).fetch()
TypeError: descriptor 'date' requires a 'datetime.datetime' object but received a 'DateTimeProperty'
If i use just datetime.date() it sends an error requiring an integer as an argument.
You can't do a conversion as part of the query, so you'll have to create 'yesterday' as a datetime and filter using that.
fol.followed_date is the model's property, not an entity's value. You cannot convert a property to a date, as it has no value. You need to fix the query constructor:
follower_trans = fol.query(fol.followed_date >= self.yesterday).fetch()
(assuming you've created self.yesterday as a datetime object.)
Also, it is proper to use a capital letter for any Class names. Should have been:
class Fol(ndb.Model):
followed_date = ndb.DateTimeProperty(auto_now_add = True)
and
Fol.query(Fol.followed_date >= self.yesterday).fetch()
I've been trying to find a way to create an ISODate object whith pyMongo client, but without any success so far.
I use http://pypi.python.org/pypi/pymongo3 client, which is the only serious one available in Python 3 for now, but the problem doesn't seem to come from this specific pymongo version.
I'd like to know if any of you has found a solution to use this MongoDB object type from a pymongo client... thanks for your help !
You just need to store an instance of datetime.datetime.
Inserting from the python shell:
>>> c.test.test.insert({'date': datetime.datetime.utcnow()})
ObjectId('4e8b388367d5bd2de0000000')
>>> c.test.test.find_one()
{u'date': datetime.datetime(2011, 10, 4, 16, 46, 59, 786000), u'_id': ObjectId('4e8b388367d5bd2de0000000')}
Querying in the mongo shell:
> db.test.findOne()
{
"_id" : ObjectId("4e8b388367d5bd2de0000000"),
"date" : ISODate("2011-10-04T16:46:59.786Z")
}
For those who are wondering how to create ISODate from timestamp:
ts = time.time()
isodate = datetime.datetime.fromtimestamp(ts, None)
This will create datetime object with no timezone. When inserted to MongoDB it will get converted to proper ISODate().
Also, I strongly recommend looking at Python TimeTransitionsImage. Note that tuple here is named tuple (equivalent to struct in C). And also note that tuple fields are not the same as in C counterparts, even though the naming is the same (for instance, tm_wday starts with Monday and not Sunday).
Actually that does not work either. When you try to use either utcfromtimestamp or fromtimestamp, the program errors out saying that it needs a float. Just parse the string into a date time object and use that directly in the Mongodb. filter
from_dt = datetime.strptime('2018-04-01','%Y-%m-%d')
#from_dts = datetime.utcfromtimestamp(from_dt)
to_dt = datetime.strptime('2018-04-30','%Y-%m-%d')
#to_dts = datetime.utcfromtimestamp(to_dt)
filterCondition = {
"LastLogin" : { "$lte" : to_dt},
"LastLogin" : { "$gte" : from_dt}
}
And then
db[(colName)].find({ "<colName>" : filterCondition })
Would work...
result = db.objects.insert_one(
{"last_modified": datetime.datetime.utcnow()})
Here utc stands for Universal Time Coordinates.
For create a document with specific date, for example 03/10/1999, run this:
from datetime import datetime
from pymongo import MongoClient
db = MongoClient().db_name
date = datetime(1999, 03, 10)
db.collection.insert_one({'date': date})