I have date in String format and Like to convert in ISOdate format in mongoshell or custom python script to achieve this. I tried all possible commands but couldn't achieve this. I like to know how to convert String date into ISOdate in mongodb.
Assuming
your collection name as: youCollection
field name to be converted from string to date: date
var records = db.youCollection.find();
records.forEach(function(doc) {
var dateStr = doc.date;
if(typeof dateStr === "string")
{
var date = new Date(dateStr);
db.youCollection.update({_id: doc._id}, {$set: {date: date}})
}})
This will convert date of "2018-02-17T05:01:32.028Z" format to ISODate("2018-02-17T05:01:32.028Z")
I found the optimal solution for this later
Get into Database
use yourdatabase_name
Run this afterwards changing according to your requirement
db.YOURCOLLECION.find().forEach(function(element) {
element.DATE = new Date(element.DATE);
db.DATE.save(element);
})
Related
I am new to mongodb and pymongo. Have a small clarification in converting the field datatype from string to long. Fieldname = sc_2g
The following is working fine using mongoshell.
db.collection.aggregate({$set: {sc_g: { $toLong: "$sc_g" }}},{$out:"collection"})
but i need equivalent in python. Can anyone help..?
You can use aggregation too in pymongo.
pipeline = {"$set": {"sc_g": { "$toLong": "$sc_g" }}}, {"$out":"collection"}
list(db.collection.aggregate(pipeline))
I am writing a script in python that will query a MongoDB database, parse the results in a format that can be imported into a relational database.
The data is stored in an associative array. I am able to query most of the fields by using dot notation such as $status.state.
Issue:
The issue is that the field $last_seen ISODate is not returning a value when attempting to use dot notation.
# Last seen dot notation does not return a value
"u_updated_timestamp": "$last_seen.date"
Here is the data structure:
{
"status" : {
"state" : "up",
},
"addresses" : {
"ipv4" : "192.168.1.1"
},
"last_seen" : ISODate("2016-04-29T14:06:17.441Z")
}
Here is the code that I am starting with. All of the other fields are returnign in the correct format. However, the last_seen ISO date field is not returning any value at all. What other steps are required in order to retrieve the value?
I tried $dateToString but it did not work (we are running pymongo 2.7).
computers = db['computer'].aggregate([
{"$project" : {
"u_ipv4": "$addresses.ipv4",
"u_status": "$status.state",
# Last seen dot notation does not return a value
"u_updated_timestamp": "$last_seen.date"
}}
])
I also tried simply $last_seen but that returns key and value, I only need the value.
UPDATE: The desired format is flexible. It could be a unix timestamp or mm-dd-yyyy. Any format would be acceptable. The main issue is that there is no date value being returned at all with this query as it stands currently.
I've just started using mongodb via the pyhton drive (pymongo). When I'm posting new data (which actually comes from a MySQL db) some data types appear incorrectly mapped. For example, some single digit number are inserted as long ints, a timestamp is inserted as a string. Also the date which is stored in MySQL as YY-MM-DD is changed to YY-MM-DD 00:00:00 (i.e a time is added).This seems like a waste of space, is this normal procedure for mongodb or should I somehow change the data types which are incorrectly(?) mapped?
ps I did search through the docs as mongodb but I couldn't find anything to match my query.
post = {
"title": video_title,
"ext_id": video_external_id,
"source": video_source,
"date_added": d1,
"views":{
"views_all": views_all,
"views_year": views_yr,
"views_day": views_day,
"views_week": views_wk,
"views_month": views_mo
},
"video_type": 0,
"hd": video_hd,
"features": featured,
"video_thumbs": video_thumbnails,
"video_main_thumb": video_main_thumbnail,
"length": video_length,
"length_sort": video_length,
"rating": {
"rating_all": rating_all,
"rating_year": rating_yr,
"rating_day": rating_day,
"rating_week": rating_wk,
"rating_month": rating_mo
}
}
posts = db.posts
post_id = video_list.insert(post)
For example, some single digit number are inserted as long ints
PyMongo stores python 2.x long as BSON int64 regardless of value, python int as BSON int32 or BSON int64 depending on the value of the int. See the table here for a mapping of python types to BSON types.
a timestamp is inserted as a string
Assuming the timestamp was passed in ISO-8601 format that's correct. If you want to store the timestamp as a BSON datetime pass a python datetime object instead.
Also the date which is stored in MySQL as YY-MM-DD is changed to YY-MM-DD 00:00:00 (i.e a time is added)
BSON (the storage format used by MongoDB) does not have a date type without a time element. It is up to your application to convert dates to datetimes.
I have the next code:
users_table = Table(users_table_name, connection=Core.aws_dynamodb_connection)
users_table.put_item(data={
"login": login,
"password": hashlib.sha256(password.encode("utf-8")).hexdigest(),
"profile": profile,
"registration_date": datetime.now() # PROBLEM IS HERE
})
But when I run it, it fails with error:
TypeError: Unsupported type "< type 'datetime.datetime' >" for value "2015-01-12 05:02:57.053131"
I've tried a lot of ways, but it seems that it isn't possible to save datetime to DynamoDB. Btw it works fine in MongoDB.
Is there any solution?
Okay, I see that DynamoDB does not support any date types. So the only solution is to use unix-like time as integer, or save date as string.
According to alejandro-franco response .isoformat() make the trick.
Just tested and this a working example:
CustomerPreferenceTable.put_item(
Item={
"id": str(uuid4()),
"validAfter": datetime.utcnow().isoformat(),
"validBefore": (datetime.utcnow() + timedelta(days=365)).isoformat(),
"tags": ["potato", "eggplant"]
}
)
According to the documentation:
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/JavaSDKHighLevel.html
Date S (string type). The Date values are stored as ISO-8601 formatted
strings.
old post but maybe still interesting ..
What you can do and how it worked for me:
import datetime
from datetime import datetime
...
now = datetime.now()
x = now.strftime("%m/%d/%Y, %H:%M:%S")
table.put_item(
Item={
'Index': Index,
'Stamp': x,
}
)
And with adaption to the code above:
import datetime
from datetime import datetime
...
now = datetime.now()
x = now.strftime("%m/%d/%Y, %H:%M:%S")
users_table = Table(users_table_name, connection=Core.aws_dynamodb_connection)
users_table.put_item(data={
"login": login,
"password": hashlib.sha256(password.encode("utf-8")).hexdigest(),
"profile": profile,
"registration_date": x,
})
My Output
Reading the documentation lately, I found the right link to the documentation here. The recommended way to store date and time data in DynamoDB is using ISO 8601 strings. so the data type is just string.
I'm not sure why datetime isn't supported in DynamoDB, or in fact I have no experience in it neither.
But if you're so insisted in not converting the datetime to string like people suggested, you can convert the datetime to timestamp, and so you can compare with it.
updated
And you may want to read this SO Question, seemed like numeric comparison is the preferred way.
If you want to use date to find users, you can simply invoke date() function. Like this:
...
users_table = Table(users_table_name, connection=Core.aws_dynamodb_connection)
current = datetime.now()
users_table.put_item(data={
"login": login,
"password": hashlib.sha256(password.encode("utf-8")).hexdigest(),
"profile": profile,
# here use a different name for the entry
"registration_time": current
"registration_date": current.date()
})
...
These are all the supported types for attribute values in DynamoDB as listed in their AWS Docs.
B A Binary data type.
Type: Blob
Required: No
BOOL A Boolean data type.
Type: Boolean
Required: No
BS A Binary Set data type.
Type: array of Blobs
Required: No
L A List of attribute values.
Type: array of AttributeValue objects
Required: No
M A Map of attribute values.
Type: String to AttributeValue object map
Required: No
N A Number data type.
Type: String
Required: No
NS A Number Set data type.
Type: array of Strings
Required: No
NULL A Null data type.
Type: Boolean
Required: No
S A String data type.
Type: String
Required: No
SS A String Set data type.
Type: array of Strings
Required: No
with DDB2, seems like Instant is now supported.
https://github.com/aws/aws-sdk-java-v2/tree/master/services-custom/dynamodb-enhanced#changing-update-behavior-of-attributes
I've been trying to find a way to create an ISODate object whith pyMongo client, but without any success so far.
I use http://pypi.python.org/pypi/pymongo3 client, which is the only serious one available in Python 3 for now, but the problem doesn't seem to come from this specific pymongo version.
I'd like to know if any of you has found a solution to use this MongoDB object type from a pymongo client... thanks for your help !
You just need to store an instance of datetime.datetime.
Inserting from the python shell:
>>> c.test.test.insert({'date': datetime.datetime.utcnow()})
ObjectId('4e8b388367d5bd2de0000000')
>>> c.test.test.find_one()
{u'date': datetime.datetime(2011, 10, 4, 16, 46, 59, 786000), u'_id': ObjectId('4e8b388367d5bd2de0000000')}
Querying in the mongo shell:
> db.test.findOne()
{
"_id" : ObjectId("4e8b388367d5bd2de0000000"),
"date" : ISODate("2011-10-04T16:46:59.786Z")
}
For those who are wondering how to create ISODate from timestamp:
ts = time.time()
isodate = datetime.datetime.fromtimestamp(ts, None)
This will create datetime object with no timezone. When inserted to MongoDB it will get converted to proper ISODate().
Also, I strongly recommend looking at Python TimeTransitionsImage. Note that tuple here is named tuple (equivalent to struct in C). And also note that tuple fields are not the same as in C counterparts, even though the naming is the same (for instance, tm_wday starts with Monday and not Sunday).
Actually that does not work either. When you try to use either utcfromtimestamp or fromtimestamp, the program errors out saying that it needs a float. Just parse the string into a date time object and use that directly in the Mongodb. filter
from_dt = datetime.strptime('2018-04-01','%Y-%m-%d')
#from_dts = datetime.utcfromtimestamp(from_dt)
to_dt = datetime.strptime('2018-04-30','%Y-%m-%d')
#to_dts = datetime.utcfromtimestamp(to_dt)
filterCondition = {
"LastLogin" : { "$lte" : to_dt},
"LastLogin" : { "$gte" : from_dt}
}
And then
db[(colName)].find({ "<colName>" : filterCondition })
Would work...
result = db.objects.insert_one(
{"last_modified": datetime.datetime.utcnow()})
Here utc stands for Universal Time Coordinates.
For create a document with specific date, for example 03/10/1999, run this:
from datetime import datetime
from pymongo import MongoClient
db = MongoClient().db_name
date = datetime(1999, 03, 10)
db.collection.insert_one({'date': date})