Develop Reference

Converting String to ISOdate in MongoDB using shell or Python Script?

I have date in String format and Like to convert in ISOdate format in mongoshell or custom python script to achieve this. I tried all possible commands but couldn't achieve this. I like to know how to convert String date into ISOdate in mongodb.

Assuming
your collection name as: youCollection
field name to be converted from string to date: date
var records = db.youCollection.find();
records.forEach(function(doc) {
var dateStr = doc.date;
if(typeof dateStr === "string")
{
var date = new Date(dateStr);
db.youCollection.update({_id: doc._id}, {$set: {date: date}})
}})
This will convert date of "2018-02-17T05:01:32.028Z" format to ISODate("2018-02-17T05:01:32.028Z")

I found the optimal solution for this later
Get into Database
use yourdatabase_name
Run this afterwards changing according to your requirement
db.YOURCOLLECION.find().forEach(function(element) {
element.DATE = new Date(element.DATE);
db.DATE.save(element);
})

Using ISO timestamp in Flask-SQLAlchemy

I have a date which I obtain from an API. It is in the form of 2015-01-01T15:04:23Z.
How can I accept this date into a model using Flask-SQLAlchemy?
So far I have tried,
date = db.Column(db.DateTime)
and
date = db.Column(db.DateTime(timezone=True))
which gives me the error
StatementError: (exceptions.TypeError) SQLite DateTime type only accepts Python datetime and date objects as input.
Also, when I retrieve it using a get, I need it to be in the exact same format.
The dateutil python module parses it well but when I retrieve it from the table, I need to get 2015-01-01T15:04:23Z.
>>> from dateutil.parser import parse
>>> parse('2015-01-01T15:04:23Z')
datetime.datetime(2015, 1, 1, 15, 4, 23, tzinfo=tzutc())

You need to convert the string into a Python datetime object, which you can do using time.strptime:
record.date = time.strptime(mytime, "%Y-%m-%dT%H:%M:%SZ")
Then you can safely set it on your instance and commit it.
To get it back in the same format, use time.strftime on the other side:
time.strftime("%Y-%m-%dT%H:%M:%SZ", record.date)
And of course remember to watch out that the order of the arguments is different between strptime and strftime :)
If you need to change/update the format, see time.strftime here : https://docs.python.org/2/library/time.html

Set correct data type with mongodb and python

I've just started using mongodb via the pyhton drive (pymongo). When I'm posting new data (which actually comes from a MySQL db) some data types appear incorrectly mapped. For example, some single digit number are inserted as long ints, a timestamp is inserted as a string. Also the date which is stored in MySQL as YY-MM-DD is changed to YY-MM-DD 00:00:00 (i.e a time is added).This seems like a waste of space, is this normal procedure for mongodb or should I somehow change the data types which are incorrectly(?) mapped?
ps I did search through the docs as mongodb but I couldn't find anything to match my query.
post = {
"title": video_title,
"ext_id": video_external_id,
"source": video_source,
"date_added": d1,
"views":{
"views_all": views_all,
"views_year": views_yr,
"views_day": views_day,
"views_week": views_wk,
"views_month": views_mo
},
"video_type": 0,
"hd": video_hd,
"features": featured,
"video_thumbs": video_thumbnails,
"video_main_thumb": video_main_thumbnail,
"length": video_length,
"length_sort": video_length,
"rating": {
"rating_all": rating_all,
"rating_year": rating_yr,
"rating_day": rating_day,
"rating_week": rating_wk,
"rating_month": rating_mo
}
}
posts = db.posts
post_id = video_list.insert(post)

For example, some single digit number are inserted as long ints
PyMongo stores python 2.x long as BSON int64 regardless of value, python int as BSON int32 or BSON int64 depending on the value of the int. See the table here for a mapping of python types to BSON types.
a timestamp is inserted as a string
Assuming the timestamp was passed in ISO-8601 format that's correct. If you want to store the timestamp as a BSON datetime pass a python datetime object instead.
Also the date which is stored in MySQL as YY-MM-DD is changed to YY-MM-DD 00:00:00 (i.e a time is added)
BSON (the storage format used by MongoDB) does not have a date type without a time element. It is up to your application to convert dates to datetimes.

Is it possible to save datetime to DynamoDB?

I have the next code:
users_table = Table(users_table_name, connection=Core.aws_dynamodb_connection)
users_table.put_item(data={
"login": login,
"password": hashlib.sha256(password.encode("utf-8")).hexdigest(),
"profile": profile,
"registration_date": datetime.now() # PROBLEM IS HERE
})
But when I run it, it fails with error:
TypeError: Unsupported type "< type 'datetime.datetime' >" for value "2015-01-12 05:02:57.053131"
I've tried a lot of ways, but it seems that it isn't possible to save datetime to DynamoDB. Btw it works fine in MongoDB.
Is there any solution?

Okay, I see that DynamoDB does not support any date types. So the only solution is to use unix-like time as integer, or save date as string.

According to alejandro-franco response .isoformat() make the trick.
Just tested and this a working example:
CustomerPreferenceTable.put_item(
Item={
"id": str(uuid4()),
"validAfter": datetime.utcnow().isoformat(),
"validBefore": (datetime.utcnow() + timedelta(days=365)).isoformat(),
"tags": ["potato", "eggplant"]
}
)

According to the documentation:
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/JavaSDKHighLevel.html
Date S (string type). The Date values are stored as ISO-8601 formatted
strings.

old post but maybe still interesting ..
What you can do and how it worked for me:
import datetime
from datetime import datetime
...
now = datetime.now()
x = now.strftime("%m/%d/%Y, %H:%M:%S")
table.put_item(
Item={
'Index': Index,
'Stamp': x,
}
)
And with adaption to the code above:
import datetime
from datetime import datetime
...
now = datetime.now()
x = now.strftime("%m/%d/%Y, %H:%M:%S")
users_table = Table(users_table_name, connection=Core.aws_dynamodb_connection)
users_table.put_item(data={
"login": login,
"password": hashlib.sha256(password.encode("utf-8")).hexdigest(),
"profile": profile,
"registration_date": x,
})
My Output

Reading the documentation lately, I found the right link to the documentation here. The recommended way to store date and time data in DynamoDB is using ISO 8601 strings. so the data type is just string.

I'm not sure why datetime isn't supported in DynamoDB, or in fact I have no experience in it neither.
But if you're so insisted in not converting the datetime to string like people suggested, you can convert the datetime to timestamp, and so you can compare with it.
updated
And you may want to read this SO Question, seemed like numeric comparison is the preferred way.

If you want to use date to find users, you can simply invoke date() function. Like this:
...
users_table = Table(users_table_name, connection=Core.aws_dynamodb_connection)
current = datetime.now()
users_table.put_item(data={
"login": login,
"password": hashlib.sha256(password.encode("utf-8")).hexdigest(),
"profile": profile,
# here use a different name for the entry
"registration_time": current
"registration_date": current.date()
})
...

These are all the supported types for attribute values in DynamoDB as listed in their AWS Docs.
B A Binary data type.
Type: Blob
Required: No
BOOL A Boolean data type.
Type: Boolean
Required: No
BS A Binary Set data type.
Type: array of Blobs
Required: No
L A List of attribute values.
Type: array of AttributeValue objects
Required: No
M A Map of attribute values.
Type: String to AttributeValue object map
Required: No
N A Number data type.
Type: String
Required: No
NS A Number Set data type.
Type: array of Strings
Required: No
NULL A Null data type.
Type: Boolean
Required: No
S A String data type.
Type: String
Required: No
SS A String Set data type.
Type: array of Strings
Required: No

with DDB2, seems like Instant is now supported.
https://github.com/aws/aws-sdk-java-v2/tree/master/services-custom/dynamodb-enhanced#changing-update-behavior-of-attributes

Pymongo find if value has a datatype of NumberLong

I'm using the Pymongo driver and my documents look like this:
{
"_id" : ObjectId("5368a4d583bcaff3629bf412"),
"book_id" : NumberLong(23302213),
"serial_number" : '1122',
}
This works because the serial number is a string:
find_one({"serial_number": "1122"})
However, this doesn't:
find_one({"book_id": "23302213"})
Obviously its because the book_id has a datatype of NumberLong. How can execute the find method based on this datatype?
==================================================
Update:
Still can't get this to work, I can only find string values. Any advise would be much appreciated.

You need to ensure your data types are matching. MongoDB is strict about types. When you execute this:
find_one({"book_id": "23302213"})
you are asking MongoDB for documents with book_id equal to "23302213". As you are not storing the book_id as type string but as type long the query needs to respect that:
find_one({"book_id": long(23302213)})
If, for some reason, you have the ID as string in your app this would also work:
find_one({"book_id": long("23302213")})
Update
Just checked it (MacOS 64bit, MongoDB 2.6, Python 2.7.5, pymongo 2.7) and it works even when providing an integer.
Document in collection (as displayed by Mongo shell):
{ "_id" : ObjectId("536960b9f7e8090e3da4e594"), "n" : NumberLong(222333444) }
Output of python shell:
>>> collection.find_one({"n": 222333444})
{u'_id': ObjectId('536960b9f7e8090e3da4e594'), u'n': 222333444L}
>>> collection.find_one({"n": long(222333444)})
{u'_id': ObjectId('536960b9f7e8090e3da4e594'), u'n': 222333444L}

You can use $type:
http://docs.mongodb.org/manual/reference/operator/query/type/
INT is 16, BIGINT is 18