I've been playing around storing tweets inside mongodb, each object looks like this:
{
"_id" : ObjectId("4c02c58de500fe1be1000005"),
"contributors" : null,
"text" : "Hello world",
"user" : {
"following" : null,
"followers_count" : 5,
"utc_offset" : null,
"location" : "",
"profile_text_color" : "000000",
"friends_count" : 11,
"profile_link_color" : "0000ff",
"verified" : false,
"protected" : false,
"url" : null,
"contributors_enabled" : false,
"created_at" : "Sun May 30 18:47:06 +0000 2010",
"geo_enabled" : false,
"profile_sidebar_border_color" : "87bc44",
"statuses_count" : 13,
"favourites_count" : 0,
"description" : "",
"notifications" : null,
"profile_background_tile" : false,
"lang" : "en",
"id" : 149978111,
"time_zone" : null,
"profile_sidebar_fill_color" : "e0ff92"
},
"geo" : null,
"coordinates" : null,
"in_reply_to_user_id" : 149183152,
"place" : null,
"created_at" : "Sun May 30 20:07:35 +0000 2010",
"source" : "web",
"in_reply_to_status_id" : {
"floatApprox" : 15061797850
},
"truncated" : false,
"favorited" : false,
"id" : {
"floatApprox" : 15061838001
}
How would I write a query which checks the created_at and finds all objects between 18:47 and 19:00? Do I need to update my documents so the dates are stored in a specific format?
Querying for a Date Range (Specific Month or Day) in the MongoDB Cookbook has a very good explanation on the matter, but below is something I tried out myself and it seems to work.
items.save({
name: "example",
created_at: ISODate("2010-04-30T00:00:00.000Z")
})
items.find({
created_at: {
$gte: ISODate("2010-04-29T00:00:00.000Z"),
$lt: ISODate("2010-05-01T00:00:00.000Z")
}
})
=> { "_id" : ObjectId("4c0791e2b9ec877893f3363b"), "name" : "example", "created_at" : "Sun May 30 2010 00:00:00 GMT+0300 (EEST)" }
Based on my experiments you will need to serialize your dates into a format that MongoDB supports, because the following gave undesired search results.
items.save({
name: "example",
created_at: "Sun May 30 18.49:00 +0000 2010"
})
items.find({
created_at: {
$gte:"Mon May 30 18:47:00 +0000 2015",
$lt: "Sun May 30 20:40:36 +0000 2010"
}
})
=> { "_id" : ObjectId("4c079123b9ec877893f33638"), "name" : "example", "created_at" : "Sun May 30 18.49:00 +0000 2010" }
In the second example no results were expected, but there was still one gotten. This is because a basic string comparison is done.
To clarify. What is important to know is that:
Yes, you have to pass a Javascript Date object.
Yes, it has to be ISODate friendly
Yes, from my experience getting this to work, you need to manipulate the date to ISO
Yes, working with dates is generally always a tedious process, and mongo is no exception
Here is a working snippet of code, where we do a little bit of date manipulation to ensure Mongo (here i am using mongoose module and want results for rows whose date attribute is less than (before) the date given as myDate param) can handle it correctly:
var inputDate = new Date(myDate.toISOString());
MyModel.find({
'date': { $lte: inputDate }
})
Python and pymongo
Finding objects between two dates in Python with pymongo in collection posts (based on the tutorial):
from_date = datetime.datetime(2010, 12, 31, 12, 30, 30, 125000)
to_date = datetime.datetime(2011, 12, 31, 12, 30, 30, 125000)
for post in posts.find({"date": {"$gte": from_date, "$lt": to_date}}):
print(post)
Where {"$gte": from_date, "$lt": to_date} specifies the range in terms of datetime.datetime types.
db.collection.find({"createdDate":{$gte:new ISODate("2017-04-14T23:59:59Z"),$lte:new ISODate("2017-04-15T23:59:59Z")}}).count();
Replace collection with name of collection you want to execute query
MongoDB actually stores the millis of a date as an int(64), as prescribed by http://bsonspec.org/#/specification
However, it can get pretty confusing when you retrieve dates as the client driver will instantiate a date object with its own local timezone. The JavaScript driver in the mongo console will certainly do this.
So, if you care about your timezones, then make sure you know what it's supposed to be when you get it back. This shouldn't matter so much for the queries, as it will still equate to the same int(64), regardless of what timezone your date object is in (I hope). But I'd definitely make queries with actual date objects (not strings) and let the driver do its thing.
Use this code to find the record between two dates using $gte and $lt:
db.CollectionName.find({"whenCreated": {
'$gte': ISODate("2018-03-06T13:10:40.294Z"),
'$lt': ISODate("2018-05-06T13:10:40.294Z")
}});
Using with Moment.js and Comparison Query Operators
var today = moment().startOf('day');
// "2018-12-05T00:00:00.00
var tomorrow = moment(today).endOf('day');
// ("2018-12-05T23:59:59.999
Example.find(
{
// find in today
created: { '$gte': today, '$lte': tomorrow }
// Or greater than 5 days
// created: { $lt: moment().add(-5, 'days') },
}), function (err, docs) { ... });
db.collection.find({$and:
[
{date_time:{$gt:ISODate("2020-06-01T00:00:00.000Z")}},
{date_time:{$lt:ISODate("2020-06-30T00:00:00.000Z")}}
]
})
##In case you are making the query directly from your application ##
db.collection.find({$and:
[
{date_time:{$gt:"2020-06-01T00:00:00.000Z"}},
{date_time:{$lt:"2020-06-30T00:00:00.000Z"}}
]
})
You can also check this out. If you are using this method, then use the parse function to get values from Mongo Database:
db.getCollection('user').find({
createdOn: {
$gt: ISODate("2020-01-01T00:00:00.000Z"),
$lt: ISODate("2020-03-01T00:00:00.000Z")
}
})
Save created_at date in ISO Date Format then use $gte and $lte.
db.connection.find({
created_at: {
$gte: ISODate("2010-05-30T18:47:00.000Z"),
$lte: ISODate("2010-05-30T19:00:00.000Z")
}
})
use $gte and $lte to find between date data's in mongodb
var tomorrowDate = moment(new Date()).add(1, 'days').format("YYYY-MM-DD");
db.collection.find({"plannedDeliveryDate":{ $gte: new Date(tomorrowDate +"T00:00:00.000Z"),$lte: new Date(tomorrowDate + "T23:59:59.999Z")}})
mongoose.model('ModelName').aggregate([
{
$match: {
userId: mongoose.Types.ObjectId(userId)
}
},
{
$project: {
dataList: {
$filter: {
input: "$dataList",
as: "item",
cond: {
$and: [
{
$gte: [ "$$item.dateTime", new Date(`2017-01-01T00:00:00.000Z`) ]
},
{
$lte: [ "$$item.dateTime", new Date(`2019-12-01T00:00:00.000Z`) ]
},
]
}
}
}
}
}
])
For those using Make (formerly Integromat) and MongoDB:
I was struggling to find the right way to query all records between two dates. In the end, all I had to do was to remove ISODate as suggested in some of the solutions here.
So the full code would be:
"created": {
"$gte": "2016-01-01T00:00:00.000Z",
"$lt": "2017-01-01T00:00:00.000Z"
}
This article helped me achieve my goal.
UPDATE
Another way to achieve the above code in Make (formerly Integromat) would be to use the parseDate function. So the code below will return the same result as the one above :
"created": {
"$gte": "{{parseDate("2016-01-01"; "YYYY-MM-DD")}}",
"$lt": "{{parseDate("2017-01-01"; "YYYY-MM-DD")}}"
}
⚠️ Be sure to wrap {{parseDate("2017-01-01"; "YYYY-MM-DD")}} between quotation marks.
Convert your dates to GMT timezone as you're stuffing them into Mongo. That way there's never a timezone issue. Then just do the math on the twitter/timezone field when you pull the data back out for presentation.
Why not convert the string to an integer of the form YYYYMMDDHHMMSS? Each increment of time would then create a larger integer, and you can filter on the integers instead of worrying about converting to ISO time.
Scala:
With joda DateTime and BSON syntax (reactivemongo):
val queryDateRangeForOneField = (start: DateTime, end: DateTime) =>
BSONDocument(
"created_at" -> BSONDocument(
"$gte" -> BSONDateTime(start.millisOfDay().withMinimumValue().getMillis),
"$lte" -> BSONDateTime(end.millisOfDay().withMaximumValue().getMillis)),
)
where millisOfDay().withMinimumValue() for "2021-09-08T06:42:51.697Z" will be "2021-09-08T00:00:00.000Z"
and
where millisOfDay(). withMaximumValue() for "2021-09-08T06:42:51.697Z" will be "2021-09-08T23:59:99.999Z"
i tried in this model as per my requirements i need to store a date when ever a object is created later i want to retrieve all the records (documents ) between two dates
in my html file
i was using the following format mm/dd/yyyy
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<script>
//jquery
$(document).ready(function(){
$("#select_date").click(function() {
$.ajax({
type: "post",
url: "xxx",
datatype: "html",
data: $("#period").serialize(),
success: function(data){
alert(data);
} ,//success
}); //event triggered
});//ajax
});//jquery
</script>
<title></title>
</head>
<body>
<form id="period" name='period'>
from <input id="selecteddate" name="selecteddate1" type="text"> to
<input id="select_date" type="button" value="selected">
</form>
</body>
</html>
in my py (python) file i converted it into "iso fomate"
in following way
date_str1 = request.POST["SelectedDate1"]
SelectedDate1 = datetime.datetime.strptime(date_str1, '%m/%d/%Y').isoformat()
and saved in my dbmongo collection with "SelectedDate" as field in my collection
to retrieve data or documents between to 2 dates i used following query
db.collection.find( "SelectedDate": {'$gte': SelectedDate1,'$lt': SelectedDate2}})
Related
I have a collection of Promo codes like this:-
{
'promoCode': 'XMAS22',
'country': 'USA',
'isExpirable': true,
'availability' : [
{
'startTimestamp' : startTimeVal1,
'endTimestamp': endTimeVal1,
'timezone': 'America/New_York'
},
{
'startTimestamp' : startTimeVal2,
'endTimestamp': endTimeVal2',
'timezone': 'America/Chicago'
},
{
'startTimestamp' : startTimeVal3,
'endTimestamp': endTimeVal3,
'timezone': 'America/Los_Angeles'
},
]
},
{
'promoCode': 'HAPPYDAY',
'country': 'USA',
'isExpirable': false,
'availability' : null
}
Promo Code XMAS22 is expirable in nature. So it has isExpirable status as true and has anvailability array, containing availability based on three timezones.
Promo code HAPPYDAY has isExpirable status false, so availability is null.
A user sends a timestamp value and a timezone. Something like 1628290101 and 'America/New_York'. The query should retrieve:-
{
'promoCode': 'XMAS22',
'country': 'USA'
},
{
'promoCode': 'HAPPYDAY',
'country': 'USA'
}
Promo code XMAS22 should be retrieved if:-
isExpirable = true and,
timezone value is gte startTimestamp and lte endTimestamp and,
timezone = America/New_York
Promo code HAPPYDAY should be retrieved because:-
isExpirable = false
How can I build up the query?
Query
here we assume the user endered timestamp=5 and timezone="America/New_York"
its the conditions you said, to pass if isExpirable=false or same timezone and timestamp between the start and end time.
finally project to keep only those 2 fields you want, you can remove the _id : 0 also if you want
*this is the find way with only query operators to use the index if you have one on startTimestamp and endTimestamp, we could do the same with aggregation in more programatic way with $filter but index wouldn't be used even if you had it.
Playmongo
find({"$or":
[{"isExpirable": {"$eq": false}},
{"$and":
[{"availability":
{"$elemMatch":
{"$and":
[{"startTimestamp": {"$lte": 5}}, {"endTimestamp": {"$gte": 5}},
{"timezone": {"$eq": "America/New_York"}}]}}}]}]},
{"projection": {"promoCode": 1, "country": 1}})
I am trying to write a validation method for the following object ( associative array ):
{
"10:00": {
discount: 10,
time: "10:00",
},
"11:00": {
discount: 11,
time: "11:00",
},
...
....
}
Using Joi (https://github.com/hapijs/joi) What i got so far is this:
Joi.object().keys(
{time:{
discount: Joi.number(),
time: Joi.string(),
}}
),
which is obviously wrong and failing with : ValidationError: child "discounts" fails because ["10:00" is not allowed, "11:00" is not allowed]
can anyone suggest how to write validation for objects with variable number of keys ( associative array )
sorted it out after reading through : Is there a way to validate dynamic key names?
Joi.object().pattern(/^/, [
Joi.object({
discount: Joi.number(),
time: Joi.string()
})
])
Situation
In a project I have this code to select data from a table. Please note, it is working, I only don't get the result I expect.
serviceSurveyQuestions.find({
query: {
survey_id: this.survey_id,
user_id: this.$store.state.auth.user.id, //TODO move this to the hook!!
//todo make satus also not equal new
$or: [
{ status_id: process.env.mfp.statusSurveyQuestionStarted },
{ status_id: process.env.mfp.statusSurveyQuestionPlanned }
],
$sort: {
survey_question_question: 1
},
$limit: 150,
$select: [
'survey_question_question',
'survey_question_at',
'survey_question_answer',
'survey_question_details',
'survey_question_source_id',
'survey_question_source_answer_id',
'survey_question_source_user_id',
'survey_question_step',
'survey_question_dep_step',
'id'
]
}
}).then(page => {
this.listSurveyQuestions = page;
});
When I see what would be in one item of listSurveyQuestion I will see this:
{
"survey_question_question": "PEN 10 Scope vaststellen",
"survey_question_at": "2017-06-23T06:46:10.038Z",
"survey_question_answer": "",
"survey_question_details": "tester done",
"survey_question_source_id": 83499707,
"survey_question_source_answer_id": 74864,
"survey_question_source_user_id": 83488216,
"survey_question_step": 10,
"survey_question_dep_step": null,
"id": 4651,
"source_user": {
"user_id": 1005
},
"status": {
"status": "Planned"
},
"language": {
"language": "Dutch"
,
"source": {
"source": "MexonInControl - Pob - Dev (local)"
},
"survey_question": [{
"answer_type_id": 1014,
"answer_en": null,
"answer_nl": null,
"answer_explanation_en": null,
"answer_explanation_nl": null,
"survey_question_next_id": 4652
} ]
}
I know the result is comming from the configuration in my get and find hook of the service being called.
Expected Result
What I expect to happen is that the data returned is only the columns defined in the $SELECT. If I leave this as is, it will work but I'm getting to much data from the database which can be seen later as a security breach. Not with this example, but with other tables it will.
** Question **
So what do I need to change to have this functioning as expected. You could adapt the return of the service, but then I can't use the same service in other situations for the columns aren't available. Or can you pass an option to the service which will result in if (parameter = view 1) then return view 1 and so on.
** Solving **
Remark 1:
So I just see the 'cause' is a bit different. The configured hooks returns more columns from the question table which are not shown. So my guess here is that if you don't configure the includes in the find query, it will pass all includes. I need to check that and if this is the case, see if there is a option to not select the 'includes' as well.
Assuming that the hook you are referring to is setting hook.params.sequelize similar to this answer you will have to check if you included properties are also set in the $select query with something like this:
// GET /my-service?include=1
function (hook) {
const include = [];
const select = hook.params.query.$select;
// Go through all properties that are added via includes
['includeProp1', 'includeProp2'].forEach(propertyName => {
// If no $select or the include property is part of the $select
if(!select || select.indexOf(propertyName) !== -1) {
include.push({ model: ModelForIncludeProp1 });
}
});
hook.params.sequelize = { include };
return Promise.resolve(hook);
}
I have a mongodb collection structured like this:
{ "client" : "CLIENTIDHERE", "amount" : 90, "invoice": "SOMEIDHERE", "date" : ISODate("2014-07-09T11:13:49.273Z") }
And need to select somehow all the payments made from a client.
In mysql I would do something like this, but with mongodb I have really no clue.
SELECT SUM(amount) payments,client FROM invoices GROUP BY client;
What I tried:
db.invoices.find({
$group: {
amount: {$sum: "$amount"}
}
}, {
$project:{
amount: "$amount",
client: "$client"
}
})
But it didn't work. What did I do wrong?
EDIT: I get the following error:
error: { "$err" : "Unsupported projection option: amount", "code" : 13097 }
Your $group step doesn't define what to GROUP BY. MongoDB's equivalent for the SQL GROUP BY is the _id field of the $group statement which is missing from your query. Also, when you want to use aggregation, you need to use .aggregate( instead of .find(.
db.invoices.aggregate({
$group: {
_id: "$client",
amount: { $sum: "$amount"}
}
});
This will write the client into the _id field. When you would rather want to have the field named client, you can fix that in the $project step, but otherwise the projection is not required in this case.
new to DOJO..using JsonRest to get data from database...i have given range to display 0-1000 out of 50000 ...but it is displaying full data......requirement is that when that 1000 loaded while scrolling the next request goes to server and rest of the data will display....
please help i tried a lot ......
my code
myStore = dojo.store.Cache(dojo.store.JsonRest({
target : "myip7080/GridExample/string"
}), dojo.store.Memory());
myStore.query({
start: 0,
count: 1000
}).then(function(results){
alert(results);
});
grid = new dojox.grid.DataGrid({
store : dataStore = dojo.data.ObjectStore({
objectStore : myStore
}),
structure : [ {
name : "SNO",
field : "sno",
width : "100px",
editable : true
}, {
name : "SNAME",
field : "sname",
width : "100px",
editable : true
}, {
name : "SALARY",
field : "salary",
width : "200px",
editable : true
} ]
}, "target-node-id"); // make sure you have a target HTML element with this id
grid.startup();
dojo.query("#save").onclick(function() {
dataStore.save();
});
});
dojo/store/Memory::query() takes two parameters:
An object that queries the data
An QueryOptions object containing the options for this query (optional)
You can query specific entries from your data like this:
myStore.query(
{ name: "John Doe" }
);
//returns all rows with name="John Doe"
You don't have to make a specific query. If you want all your data, just do myStore.query("");
If you want to limit the number of rows shown, you need to add a second parameter to represent your query options:
myStore.query(
{ name: "John Doe" },
{ start: 0, count: 1000 } //return 1000 results, starting at the beginning
);
See the documentation for dojo/store/memory::query().