I have a collection with date field as long. A sample json format of the collection is as follows:
{
"processID" : "1410449146441-3-8863e29e0055",
"department" : "ABC",
"price" : 9.99,
"unitSold" : 19,
"date" : 1410449146442
},
{
"processID" : "14104491tf-alhb-8863e29e0055",
"department" : "XYZ",
"price" : 19.99,
"unitSold" : 21,
"date" : 1410985985985
}
I need to convert this date field as ISOdate format so that I can extract year, month and day from it. I need to do it in pymongo.
I tried doing it as datetime.datetime.fromtimestamp({"$divide": ["$date", 1e3]}), but got error "TypeError: a float is required"
Can you please let me know a good way of doing it in pymongo
You got the error because fromtimestamp is a Python function not Mongo DB function. .fromtimestamp expects float instead you provided a dict.
Use datetime module to convert while iterating through your cursor:
import datetime
for x in cursor:
timestamp = datetime.datetime.fromtimestamp(x.get("date")/1000)
# timestamp = object of type datetime.
Note: If you don't divide the long value by 1000, you'll get the error of "year not in range".
Related
this is my test json file.
{
"item" : {
"fracData" : [ ],
"fractimeData" : [ {
"number" : "1232323232",
"timePeriods" : [ {
"validFrom" : "2021-08-03"
} ]
} ],
"Module" : [ ]
}
}
This is how I read the json file.
starhist_test_df = spark.read.json("/mapr/xxx/yyy/ttt/dev/rawdata/Test.json", multiLine=True)
starhist_test_df.createOrReplaceTempView("v_test_df")
This query works.
df_test_01 = spark.sql("""
select item.fractimeData.number from v_test_df""")
df_test_01.collect();
Result
[Row(number=['1232323232'])]
But this query doesn't work.
df_test_01 = spark.sql("""
select item.fractimeData.timePeriods.validFrom from v_test_df""")
df_test_01.collect();
Error
cannot resolve 'v_test_df.`item`.`fractimeData`.`timePeriods`['validFrom']' due to data type mismatch: argument 2 requires integral type, however, ''validFrom'' is of string type.; line 3 pos 0;
What do I have to change, to read the validFrom field?
dot notation to access values works with struct or array<struct> types.
The schema for field number in item.fractimeData is string and accessing it via dot notation returns an array<string> since fractimeData is an array.
Similarly, the schema for field timePeriods in item.fractimeData is <array<struct<validFrom>>, and accessing it via dot notation wraps it into another array, resulting in final schema of array<array<struct<validFrom>>>.
The error you get is because the dot notation can work on array<struct> but not on array<array>.
Hence, flatten the result from item.fractimeData.timePeriods to get back an array<struct<validFrom>> and then apply the dot notation.
df_test_01 = spark.sql("""
select flatten(item.fractimeData.timePeriods).validFrom as validFrom from v_test_df""")
df_test_01.collect()
"""
[Row(validFrom=['2021-08-03', '2021-08-03'])]
"""
i 'm actually working on a stream, receiving a bunch of strings and need to make a count of all the strings. the sums is aggragated, that mean for the second record the sum was added to the day before
the output must be some json file looking like
{
"aggregationType" : "day",
"days before" : 2,
"aggregates" : [
{"date" : "2018-03-03",
"sum" : 120},
{"date" :"2018-03-04",
"sum" : 203}
]
}
i created a stream looking like :
val eventStream : DataStream [String] =
eventStream
.addSource(source)
.keyBy("")
.TimeWindow(Time.days(1), Time.days(1))
.trigger(new MyTriggerFunc)
.aggregation(new MyAggregationFunc)
.addSink(sink)
thank you in advance for the help :)
Note on working with JSON in Flink:
Use JSONDeserializationSchema to deserialize the events, which will produce ObjectNodes. You can map the ObjectNode to YourObject for convenience or continue working with the ObjectNode.
Tutorial on working with ObjectNode: http://www.baeldung.com/jackson-json-node-tree-model
Back to your case, you can do it like the following:
val eventStream : DataStream [ObjectNode] =
oneMinuteAgg
.addSource(source)
.windowAll()
.TimeWindow(Time.minutes(1))
.trigger(new MyTriggerFunc)
.aggregation(new MyAggregationFunc)
will output a stream of 1min aggregates
[
{
"date" :2018-03-03
"sum" : 120
},
{
"date" :2018-03-03
"sum" : 120
}
]
then chain another operator to the "oneMinuteAgg" that will add the 1min aggregates into 1day aggregates:
[...]
oneMinuteAgg
.windowAll()
.TimeWindow(Time.days(1))
.trigger(new Whatever)
.aggregation(new YourDayAggF)
that will output what you need
{
"aggregationType" : "day"
"days before" : 4
"aggregates : [{
"date" :2018-03-03
"sum" : 120
},
{
"date" :2018-03-03
"sum" : 120
}]
}
I used windowAll() assuming you don't need to key the stream.
I have a MongoDb document such as
{
"_id" : ObjectId("5731c75196dada69cd52419d"),
"businessName" : "Test",
"createdDate" : "2015-10-04 13:48:55",
"modifiedDate" : "2016-03-03 10:37:48"
}
If you notice the createdDate and modifiedDate column are in YYYY-mm-dd h:i:s format (Which is coming from my Mysql Db). I need to convert the dates into ISO format.
For this, I am using the below query .
db.users.update({"_id" : ObjectId("5731c75196dada69cd52419d")},
{$set : {"createdDate" : ISODate('2015-10-04 13:48:55')}} )
I am getting the desired result.
Now my question is if we are having lot of documents and need to update the field value then how can we do it in iterations in Mongodb itself.
Is that possible ? Updating a single document each time will be very time-consuming. Any help will be appreciated.
Firstly get created at values then convert it into ISODate store it your array and then pass that array to createdDate in $in operator.
You can use toISOString() of javascript to convert values.
db.getCollection('users').update({'_id':{$in:[ObjectId('574d61acf834e66681d37804'),
ObjectId('574d6185f834e66681d37803')]}},
{$set : {"createdDate" :{$in:['your converted date 1','your converted date 2']}},
{multi: true})
I am trying to use JSON in MariaDB. My table have two columns:
ID - INT. PK. Auto increment.
JsonText - BLOB
I have inserted Json into the blob column and it looks like so:
{
"Fields" : {
"CalPeriod" : "1",
"CalYear" : "2014",
"CompanyCode" : "E009",
"CreditDebitInd" : "H",
"FiscalYear" : "2014",
"MaxPostDate" : "2014-01-14T00:00:00",
"MinPostDate" : "2014-01-14T00:00:00"
}
}
However, when I try to select the data and parse it back I get the error which I posted in the title.
Here is my query:
SELECT COLUMN_JSON(JsonText) FROM alliance.jsontest;
I don't understand. I even copied the code exactly from this website and it still threw the same error or it just returned the column with the word BLOB on it.
Better late than never...
The function COLUMN_JSON is meant to be used with dynamics columns.
Inserting a json string into a blob column doesn't make it a dynamic column.
To insert data in this type of column, you need to use COLUMN_CREATE.
e.g.
INSERT INTO alliance.jsontest (JsonText)
VALUES (COLUMN_CREATE(
"CalPeriod", "1",
"CalYear", "2014",
"CompanyCode", "E009",
"CreditDebitInd", "H",
"FiscalYear", "2014",
"MaxPostDate", "2014-01-14T00:00:00",
"MinPostDate", "2014-01-14T00:00:00"
));
Then you can use COLUMN_JSON or select one of the column by using COLUMN_GET.
I was expecting that an object with a Date property would be saved in Mongo as an ISODate from both client or server side, but this is not the case.
When I do
if (Meteor.is_client()){
Collection.insert({text : "Client", number : 1, date : new Date() });
Collection.insert({text : "Client", number : 2, date : (new Date()).getTime() });
}
else {
Collection.insert({text : "Server", number : 1, date : new Date() });
}
In mongo it saves like so
{_id : "xx-xx-xx-xx-xx", text : "Client", number : 1, date : "2012-08-21T18:40:47.446" }
{_id : "xx-xx-xx-xx-xx", text : "Client", number : 2, date : 1345574805367 }
{_id : "xx-xx-xx-xx-xx", text : "Server", number : 1, date : ISODate(2012-08-21T18:40:47.446)
Is there a way to save an object with a Date property from client side as an ISODate?
For me, I don't send timestamps from client side. Instead, I modify the document when inserting through Collection.allow function under auth branch.
I think there're several benefits to do this -
Client-side does not need to insert date field, which saves code.
The timestamp is based on the server time, rather than client-side,
which should be more accurate.
And last, the field value is ISODate, rather than string. (Hate
JSON without native date type supported)