Representing logic as data in JSON - json

For business reasons we need to externalize some conditional logic into external files: preferably JSON.
A simple filter-by scenario could be handled by adding a node as follows:
"filter": [
{
"criteria": "status",
"value": "open",
"condition": "=="
}
]
Multiple conditions could be handled by additional values in the array.
"filter": [
{
"criteria": "status",
"value": "open",
"condition": "=="
},
{
"criteria": "condition2",
"value": "value2",
"condition": "=="
}
]
However, it gets a little confusing when we have handle complex conditions involving ANDs or ORs.
Question: is there a standardized (or even widely accepted) format for representing such logic within JSONs? How would you do it if it were up to you?
NOTE: The first answer has been made an editable wiki so it can be improved by anyone who feels it can be.

If you must implement this using standard JSON, i'd recommend something akin to Lisp's "S-expressions". A condition could be either a plain object, or an array whose first entry is the logical operation that joins them.
For example:
["AND",
{"var1" : "value1"},
["OR",
{ "var2" : "value2" },
{ "var3" : "value3" }
]
]
would represent var1 == value1 AND (var2 == value2 OR var3 == value3).
If you prefer brevity over consistency, you could also allow an object to have multiple properties, which would implicitly be joined by an AND. For example, { "a": "b", "c": "d" } would be equivalent to ["AND", { "a": "b" }, { "c": "d" }]. But there are cases (like the example) where the former syntax can not faithfully represent the condition as written; you'd need additional trickery like translating the condition or using dummy property names. The latter syntax should always work.

I needed a format that would:
Support comparisons other than equality.
Let variables appear in any position, not just be compared to literals.
Be consistent, terse, secure, and extensible.
So I built up a format I'm calling JsonLogic. A rule is a JSON object, with the operator in the key position, and one or an array of arguments in the value position. (Inspired by Amazon CloudFormation functions.) Any argument can be another rule, so you can build arbitrarily deep logic.
I've also written two parsers for it: JsonLogic for JavaScript and JsonLogic for PHP.
cHao's example would be written as
{ "and", [
{"==", [ {"var" : "var1"}, "value1" ]},
{ "or", [
{"==", [ {"var" : "var2"}, "value2" ]},
{"==", [ {"var" : "var3"}, "value3" ]}
]}
]}
var here is the operator to get a property of the "data" object, passed along with the "rule" object to the parser, e.g.:
jsonLogic(
{"==", [{"var":"filling"}, "apple"]} // rule, is this pie apple?
{"filling":"apple", "temperature":100} // data, a pie I'm inspecting
);
// true
There are lots more possible operators (greater than, not-equals, in-array, ternary, etc) and both parsers are available on GitHub (with unit tests and documentation).

By the way, IBM DB2 supports logic statements encoded in JSON.
Boolean operations look like a cross between cHao's solution and Amazon CloudFormation:
{"$and":[{"age":5},{"name":"Joe"}]}
Comparison operations look, to me, like transliterated SQL. (Instead of Amazon or Russellg or cHao's movement toward an abstract syntax tree.)
{"age":{"$lt":3}}

I had a similar need (to build up a sql where clause in javascript). I create dthe following javascript function:
function parseQuery(queryOperation){
var query="";
if (queryOperation.operator == 'and')
query = "(" + parseQuery(queryOperation.leftOp) + ") AND (" + parseQuery(queryOperation.rightOp) + ")";
if (queryOperation.operator == 'or')
query = "(" + parseQuery(queryOperation.leftOp) + ") OR (" + parseQuery(queryOperation.rightOp) + ")";
if (queryOperation.operator == '=')
query = "(" + queryOperation.leftOp +" = "+ queryOperation.rightOp + ")";
return query;
}
I create my queryOperation Like this:
var queryObject = {
operator: 'and',
leftOp: {
leftOp: 'tradedate',
operator: '=',
rightOp: new Date()
},
rightOp: {
operator: 'or',
leftOp: {
leftOp: 'systemid',
operator: '=',
rightOp: 9
},
rightOp: {
leftOp: 'systemid',
operator: '=',
rightOp:10
}
}
};
When I pass my queryOperation to ParseQuery it returns
((tradedate= Thu Jul 24 17:30:37 EDT 2014)) AND (((systemid= 9)) OR ((systemid= 10)))
I need to add some type conversions and other operators, but the basic structure works.

I came up with this format with the primary goal of reading as close as possible to actually SQL.
Here's the Type def in typescript:
type LogicalOperator = 'AND' | 'OR';
type Operator = '=' | '<=' | '>=' | '>' | '<' | 'LIKE' | 'IN' | 'NOT IN';
type ConditionParams = {field: string, opp: Operator, val: string | number | boolean};
type Conditions = ConditionParams | LogicalOperator | ConditionsList;
interface ConditionsList extends Array<Conditions> { }
Or BNF (ish? my cs teachers wouldn't be proud)
WHEREGROUP: = [ CONDITION | ('AND'|'OR') | WHEREGROUP ]
CONDITION: = {field, opp, val}
With the following Parsing Rules:
AND is optional (I typically add it for readability). If logical LogicalOperator is left out between conditions, it will automatically joins them with AND
Inner arrays are parsed as nested groups (EG get wrapped in ())
this type does not restrict multiple logical operators consecutively (unfortunately). I handled this by just using the last one, although I could have thrown a runtime error instead.
Here are some examples (typescript playground link):
1 AND 2 (AND inferred)
[
{ field: 'name', opp: '=', val: '123' },
{ field: 'otherfield', opp: '>=', val: 123 }
]
1 OR 2
[
{ field: 'name', opp: '=', val: '123' },
'OR',
{ field: 'annualRevenue', opp: '>=', val: 123 }
]
(1 OR 2) AND (3 OR 4)
[
[
{ field: 'name', opp: '=', val: '123' },
'OR',
{ field: 'name', opp: '=', val: '456' }
],
'AND',
[
{ field: 'annualRevenue', opp: '>=', val: 123 },
'OR',
{ field: 'active', opp: '=', val: true }
]
]
1 AND (2 OR 3)
[
{ field: 'name', opp: '=', val: '123' },
'AND',
[
{ field: 'annualRevenue', opp: '>=', val: 123 },
'OR',
{ field: 'active', opp: '=', val: true }
]
]
1 AND 2 OR 3
[
{ field: 'name', opp: '=', val: '123' },
'AND',
{ field: 'annualRevenue', opp: '>=', val: 123 },
'OR',
{ field: 'active', opp: '=', val: true }
]
1 OR (2 AND (3 OR 4))
[
{ field: 'name', opp: '=', val: '123' },
'OR',
[
{ field: 'annualRevenue', opp: '>=', val: 123 },
'AND',
[
{ field: 'active', opp: '=', val: true },
'OR',
{ field: 'accountSource', opp: '=', val: 'web' }
]
]
]
As you can see, if you were to remove , and property names, then just replace the [] with (), you'd basically have the condition in SQL format

My colleague suggested this possible solution:
"all OR conditions would be an array while AND conditions would be objects,
For example,OR can match any of the objects in the array:
[
{
"var1":"value1"
},
{
"var2":"value2"
},
{
"var3":"value3"
}
]
AND would be
{
"var1":"val1",
"var2":"val2",
"var3":"val3"
}

Please check out (JSL)[https://www.npmjs.com/package/lib-jsl ].
It seems to fit the description given.
Here is a sample :
var JSL = require('lib-jsl');
var bugs = [
[{ bug : { desc: 'this is bug1 open', status : 'open' } }],
[{ bug : { desc: 'this is bug2 resolved', status : 'resolved' } }],
[{ bug : { desc: 'this is bug3 closed' , status : 'closed' } }],
[{ bug : { desc: 'this is bug4 open', status : 'open' } }],
[{ bug : { desc: 'this is bug5 resolved', status : 'resolved' } }],
[{ bug : { desc: 'this is bug6 open', status : 'open' } }],
[ { workInProgress : '$bug'},
{ bug : '$bug'},
{ $or : [
{ $bind : [ '$bug', { status : 'open'} ] },
{ $bind : [ '$bug', { status : 'resolved'} ] }
] }
]
];
var query = [{workInProgress : '$wip'}]
var transform = '$wip'
var jsl = new JSL ({
rules : bugs,
query : query,
transform : transform
});
var retval = jsl.run();
console.log(JSON.stringify(retval, null,2));
The response is :
[
{
"desc": "this is bug1 open",
"status": "open"
},
{
"desc": "this is bug2 resolved",
"status": "resolved"
},
{
"desc": "this is bug4 open",
"status": "open"
},
{
"desc": "this is bug5 resolved",
"status": "resolved"
},
{
"desc": "this is bug6 open",
"status": "open"
}
]
The main work is done by the query defined in the rule workInProgress :
[ { workInProgress : '$bug'},
{ bug : '$bug'},
{ $or : [
{ $bind : [ '$bug', { status : 'open'} ] },
{ $bind : [ '$bug', { status : 'resolved'} ] }
] }
]
This rule can be read as :
To satisfy the query with workInProgress, we define a variable {workInProgress : '$bug'}, which we then proceed to match against all bugs in the database using the next part of the rule {bug : '$bug'}. This part matches all bugs since the shape of the object (it's keys: 'bug') matches the bug records in the database. The rule further asks the $bug variable to be $bind(ed) against patterns containing relevant status values (open and closed) within a $or. Only those bug records whose status value in $bug satisfies all parts of the rule's body qualify for the result.
The result is finally transformed using the transform specification : transform : '$wip' which literally asks for an array of all values returned in the $wip variable of the query.

Following Jeremy Wadhams comment, I implemented a parser I hope it can help you:
https://play.golang.org/p/QV0FQLrTlyo
The idea is to set all logic operators in special keys with $ character like $and or $lte.
As an example:
{
"$or":[
{
"age":{
"$lte":3
}
},
{
"name":"Joe"
},
{
"$and":[
{
"age":5
},
{
"age ":{
" $nin ":[
1,
2,
3
]
}
}
]
}
]
}
Regards.
Is translated as:
( age <= 3 OR name = Joe OR ( age = 5 AND age NOT IN (1,2,3) ) )

Formula parser + a bit of JS codes to put data into formulas, is another solution described with example in this answer.

We created an npm package json-conditions to handle this. It's not as full featured as some of the others here, but it's easy to translate into a simple UI for non-technically savvy clients as complex rules are possible without nesting and it covers virtually all use cases they can come up with.
Example on runkit
const objectToTest = {
toy: {
engines: 1,
},
batteries: 'AA',
fun: true,
};
const simpleRules = [
// required: true means This first condition must always be satisfied
{ property: 'fun', op: 'eq', value: true, required: true },
{ property: 'toy.engines', op: 'gt', value: 2 },
{ property: 'batteries', op: 'present' },
];
// Returns true
checkConditions({
rules: simpleRules,
satisfy: 'ANY', // or ALL to require all conditions to pass
log: console.log,
}, objectToTest);

Following Jeremy Wadhams comment, I mapped the json by MongoDB logical query operator and MongoDB comparison query operator but MongoDB don't allow $ character in content:
{"and":[
{"age":{"eq": 5}},
{"name":{"eq": "Joe"}
]}

I just wanted to help by defining a parsing logic in JavaScript for the JSON structure mentioned in the answer: https://stackoverflow.com/a/53215240/6908656
This would be helpful for people having a tough time in writing a parsing logic for this.
evaluateBooleanArray = (arr, evaluated = true) => {
if (arr.length === 0) return evaluated;
else if (typeof arr[0] === "object" && !Array.isArray(arr[0])) {
let newEvaluated = checkForCondition(arr[0]);
return evaluateBooleanArray(arr.splice(1), newEvaluated);
} else if (typeof arr[0] === "string" && arr[0].toLowerCase() === "or") {
return evaluated || evaluateBooleanArray(arr.splice(1), evaluated);
} else if (typeof arr[0] === "string" && arr[0].toLowerCase() === "and") {
return evaluated && evaluateBooleanArray(arr.splice(1), evaluated);
} else if (Array.isArray(arr[0])) {
let arrToValuate = [].concat(arr[0]);
return evaluateBooleanArray(
arr.splice(1),
evaluateBooleanArray(arrToValuate, evaluated)
);
} else {
throw new Error("Invalid Expression in Conditions");
}
};
So the param arr here would be an array of conditions defined in the format as described by the attached link.

The first came to mind would be the recurisve
dict1={'$lte':'<','$nin':'not in '}
def updateOp(subdictItem):
for ites in subdictItem:
ops = ites
print dict1.get(ops), subdictItem.get(ops), type(subdictItem.get(ops))
if type(subdictItem.get(ops)) is list:
valuelist=subdictItem.get(ops)
strlist=' ,'.join([str(x) for x in valuelist])
sub = dict1.get(ops) + "(" +strlist +")"
else:
sub = dict1.get(ops) +' ' + str(subdictItem.get(ops))
return sub
def jsonString(input_dict):
items=''
itemslist=[]
list = []
for item in input_dict:
op=item
itemssublist=[]
# print "item",op
for subitem in input_dict.get(op):
# print("subitem",subitem)
for ite in subitem:
if ite not in ('and','or'):
# print('ite_raw',ite,subitem.get(ite))
sub=''
if type(subitem.get(ite)) is dict:
sub=updateOp(subitem.get(ite))
else:
sub='=' + str(subitem.get(ite))
itemssublist.append(ite+sub)
else:
item1=jsonString(subitem)
itemssublist.append(item1)
delimiter=" "+op+ " "
items= "("+delimiter.join(itemssublist)+")"
return items
if __name__ == "__main__":
input_dict={}
with open('ops.json','r') as f:
input_dict=json.load(f)
print input_dict
test= jsonString(input_dict)
#result : (age< 3 or name=Joe or (age=5 and age not in (1 ,2 ,3)))
ops.json file:
{
"or":[
{
"age":{
"$lte":3
}
},
{
"name":"Joe"
},
{
"and":[
{
"age":5
},
{
"age ":{
"$nin":[
1,
2,
3
]
}
}
]
}
]
}

Logic can be implemented with "logicOp": "Operator" on a "set": ["a","b" ...]
For cHau's example:
"var": {
"logicOp": "And",
"set": ["value1",
{
"LogicOp": "Or",
"set": ["value2", "value3"]
}
]
}
There can also be other attributes/operations for the set for example
"val": { "operators": ["min": 0, "max": 2], "set": ["a", "b", "c"] }
For a sundae with two scoops of one or more icecream types, 1 toppings and whipcream
"sundae": {
"icecream": {
"operators": [ "num": 2,
"multipleOfSingleItem": "true"],
"set": ["chocolate", "strawberry", "vanilla"]
},
"topping": {
"operators": ["num": 1],
"set": ["fudge", "caramel"]
},
"whipcream": "true"
}

Use arrays, alternate between OR and AND conditions:
const rule0 = [
[ "ruleA1", "ruleA2", "ruleA3" ],
[ "ruleB5", "ruleB6" ],
[ "ruleB7" ]
];
const rule1 = [
[ "ruleA1", "ruleA2", [ [ "ruleA3A" ], [ "ruleA3B1", "ruleA31B2" ] ] ],
[ "ruleB5", "ruleB6" ],
[ "ruleC7" ]
];
function ruler (rules) {
return "( " +
rules.map(or_rule =>
"( " +
or_rule.map(and_rule =>
Array.isArray(and_rule) ? ruler(and_rule) : and_rule
).join(" AND ") +
" )"
).join(" OR ") + " )";
}
Output:
ruler(rule0)
'( ( ruleA1 AND ruleA2 AND ruleA3 ) OR ( ruleB5 AND ruleB6 ) OR ( ruleB7 ) )'
ruler(rule1)
'( ( ruleA1 AND ruleA2 AND ( ( ruleA3A ) OR ( ruleA3B1 AND ruleA31B2 ) ) ) OR ( ruleB5 AND ruleB6 ) OR ( ruleC7 ) )'

I created a structure thinking about a sequential iteration:
[
{
"operator": null,
"field": "age",
"condition": "eq",
"value": 5
},
{
"operator": "and",
"field": "name",
"condition": "eq",
"value": "Joe"
}
]

Want to throw my own hat in the ring here.
The best serializable predicate you can use to represent logic in JSON format has already largely won the day. It’s called JSON Schema (yes, a schema is a predicate, data will either validate against it or it won’t - true/false - a predicate)
Why do I say it’s win the day? We use it everywhere all the time, open API/swagger, embedded in our IDEs autocompletes, in various dev documentation websites
All you have to do is wrap some logic around how to resolve your data and feed it to a json schema validator with a json schema
https://www.npmjs.com/package/json-schema-rules-engine

Related

How to loop through a JSON object in Perl?

I'm trying to create an api using perl, the api is for a react native, when the user submits a form on the app I'll get the following object,
I'm new to perl and I'm lost trying to loop thro the object :/
{
checkboxes: [
{
id: "1",
fullname: "Name 1",
color: "red",
res: false
},
{
color: "green",
fullname: "Name 2",
id: "2",
res: false
},
{
color: "blue",
id: "3",
fullname: "Name 3",
res: false
}
]
}
my $data_decoded = decode_json($data);
I'm trying this loop, but it only prints the full object.
foreach $a (#data) {
print "value of a: $a\n";
}
You turned your JSON into a reference Perl data structure with decode_json (from somewhere, and Mojo::JSON is such a place):
use Mojo::JSON qw(decode_json);
my $data = ...;
my $data_decoded = decode_json($data);
Now you have to figure out how to access whatever you have in $data_decoded. You can look at its structure by dumping it
use Mojo::Util qw(dumper);
print dumper( $data_decoded );
You'll see that the Perl structure is the same as the JSON structure. You have a hash (JSON's Object) that has a checkboxes key that points to an array. The array elements are hashes.
Using Perl v5.24's postfix dereferencing notation, you get all of the array elements:
# uglier circumfix notation #{$data_decoded->{checkboxes}}
my #elements = $data_decoded->{checkboxes}->#*;
You might put that in a loop:
foreach my $hash ( $data_decoded->{checkboxes}->#* ) {
...
}
Now you get each hash element in $hash and you can do whatever you like with it. That part you haven't told us about yet. :)
The Perl Data Structures Cookbook (perldsc) has a lot of examples of the generation and iteration of various combinations of hash and array references.
You say that you want to do something when the value of the res key is true. In that case, you can use next to skip the items where res is false:
foreach my $hash ( $data_decoded->{checkboxes}->#* ) {
next unless $hash->{res};
say "I'm doing something when res is true";
}
Following demo code demonstrates looping through these particular data
use strict;
use warnings;
use feature 'say';
use JSON;
use Data::Dumper;
my $input = do { local $/; <DATA> };
my $data = from_json($input);
say Dumper($data);
for my $obj ( #{$data->{checkboxes}} ) {
say join(",\t", $obj->#{qw/id fullname color/});
}
__DATA__
{
"checkboxes": [
{
"id": "1",
"fullname": "Name 1",
"color": "red",
"res": false
},
{
"color": "green",
"fullname": "Name 2",
"id": "2",
"res": false
},
{
"color": "blue",
"id": "3",
"fullname": "Name 3",
"res": false
}
]
}
Output
$VAR1 = {
'checkboxes' => [
{
'res' => bless( do{\(my $o = 0)}, 'JSON::PP::Boolean' ),
'color' => 'red',
'fullname' => 'Name 1',
'id' => '1'
},
{
'fullname' => 'Name 2',
'color' => 'green',
'res' => $VAR1->{'checkboxes'}[0]{'res'},
'id' => '2'
},
{
'id' => '3',
'color' => 'blue',
'res' => $VAR1->{'checkboxes'}[0]{'res'},
'fullname' => 'Name 3'
}
]
};
1, Name 1, red
2, Name 2, green
3, Name 3, blue

Modifying JSON in Groovy (or JOLT)

I've a simple JSON look like:
{
"account_login" : "google#gmail.com",
"view_id" : 1868715,
"join_id" : "utm_campaign=toyota&utm_content=multiformat_sites&utm_medium=cpc&utm_source=facebook",
"start_date" : "2020-02-03",
"end_date" : "2020-08-30"
}
With following Groovy script (from this answer):
def content = """
{
"account_login" : "google#gmail.com",
"view_id" : 1868715,
"join_id" : "utm_campaign=toyota&utm_content=multiformat_sites&utm_medium=cpc&utm_source=facebook",
"start_date" : "2020-02-03",
"end_date" : "2020-08-30"
}
"""
def slurped = new JsonSlurper().parseText(content)
def builder = new JsonBuilder(slurped)
builder.content.join_id = builder.content.join_id.split("\\s*&\\s*") //# to array
.collectEntries{
//# convert each item to map entry
String[] utmMarks = it.trim().split("\\s*=\\s*")
utmMarks[0] = [
"utm_medium" : "ga:medium",
"utm_campaign" : "ga:campaign",
"utm_source" : "ga:source",
"utm_content" : "ga:adContent",
"utm_term" : "ga:keyword",
].get( utmMarks[0] )
utmMarks
}
.findAll{
k,v-> k && v!=null //# filter out empty/null keys
}
//builder.content.filters = ...
println(builder.toPrettyString())
I'll get:
{
"account_login": "google#gmail.com",
"view_id": 1868715,
"join_id": {
"ga:campaign": "toyota",
"ga:adContent": "multiformat_sites",
"ga:medium": "cpc",
"ga:source": "facebook"
},
"start_date": "2020-02-03",
"end_date": "2020-08-30"
}
I want to update this script (or write new) and add new property: array filters to modified json above. Expected output:
{
"account_login":"google#gmail.com",
"view_id":1868715,
"join_id":{
"ga:campaign":"toyota",
"ga:adContent":"multiformat_sites",
"ga:medium":"cpc",
"ga:source":"facebook"
},
"start_date":"2020-02-03",
"end_date":"2020-08-30",
"converted_utm_marks":"ga:campaign=toyota&ga:adContent=multiformat_sites&ga:medium=cpc&ga:source=facebook",
"filters":[
{
"dimensionName":"ga:medium",
"operator":"EXACT",
"expressions":[
"cpc"
]
},
{
"dimensionName":"ga:adContent",
"operator":"EXACT",
"expressions":[
"multiformat_sites"
]
},
{
"dimensionName":"ga:campaign",
"operator":"EXACT",
"expressions":[
"toyota"
]
},
{
"dimensionName":"ga:source",
"operator":"EXACT",
"expressions":[
"facebook"
]
}
]
}
But the problem is that the set of filters for each JSON will be different. This set depends directly on the join_id set. If JSON join_id will contain:
"join_id": {
"ga:campaign": "toyota",
"ga:keyword": "car"
}
filters array should be:
[
{
"dimensionName":"ga:campaign",
"operator":"EXACT",
"expressions":[
"toyota"
]
},
{
"dimensionName":"ga:keyword",
"operator":"EXACT",
"expressions":[
"car"
]
}
]
operator is always equals EXACT. Property dimensionName - is a join_id.propety name. Expressions is a join_id.property value. So, property filters based on join_id and I need to loop through join_id property and build filters array with described structure. How to achieve expected output? JOLT configuration appreciated also.
I can't even simple iterate through join_id map:
slurped.join_id.each { println "Key: $it.key = Value: $it.value" }
I got the error:
/home/jdoodle.groovy: 24: illegal colon after argument expression;
solution: a complex label expression before a colon must be parenthesized # line 24, column 28.
.collect { [it.ga:campaign] }
UPDATE
I found out how to build this array:
def array =
[
filters: slurped.join_id.collect {key, value ->
[
dimensionName: key,
operator: "EXACT",
expressions: [
value
]
]
}
]
Seems like i got it:
def slurped = new JsonSlurper().parseText(content)
def builder = new JsonBuilder(slurped)
builder.content.filters = builder.content.join_id.collect {key, value ->
[
dimensionName: key,
operator: "EXACT",
expressions: [
value
]
]
}
Are there any better solutions?
def slurped = new JsonSlurper().parseText(content)
def builder = new JsonBuilder(slurped)
builder.content.filters = builder.content.join_id.collect {key, value ->
[
dimensionName: key,
operator: "EXACT",
expressions: [
value
]
]
}

check for duplicate date from data and assign to new one json

i have a json which contain duplicate date , i want to merge duplicate date into single json object.
Data:
[
{"date":"2017-06-26","mac":"66"},
{"date":"2017-06-26","window":"400"},
{"date":"2017-07-03","mac":"19"},
{"date":"2017-07-03","window":"12"}
]
output should be:
[
{"date":"2017-06-26","mac":"66","window":"400"},
{"date":"2017-07-03","mac":"19","window":"12"}
]
Here a javascript function that does that, you can apply JSON.stringify(...) to the output after passing your array and obtain the new json.
(( a ) => {
// used to check for already inserted dates
let withoutDupes = { };
if(Array.isArray(a)) {
a.forEach( (item) => {
// assuming item has a "date" property inside
if(withoutDupes[item.date]) {
withoutDupes[item.date] = Object.assign( item, withoutDupes[item.date] );
} else {
withoutDupes[item.date] = item;
}
} );
}
return Object.values( withoutDupes );
})( a )
You can try using jq command line parser and its group_by function:
jq '[group_by(.date)|.[]|add]' file
[
{
"date": "2017-06-26",
"mac": "66",
"window": "400"
},
{
"date": "2017-07-03",
"mac": "19",
"window": "12"
}
]

How to get to key in MongoDB? [duplicate]

Suppose you have the following documents in my collection:
{
"_id":ObjectId("562e7c594c12942f08fe4192"),
"shapes":[
{
"shape":"square",
"color":"blue"
},
{
"shape":"circle",
"color":"red"
}
]
},
{
"_id":ObjectId("562e7c594c12942f08fe4193"),
"shapes":[
{
"shape":"square",
"color":"black"
},
{
"shape":"circle",
"color":"green"
}
]
}
Do query:
db.test.find({"shapes.color": "red"}, {"shapes.color": 1})
Or
db.test.find({shapes: {"$elemMatch": {color: "red"}}}, {"shapes.color": 1})
Returns matched document (Document 1), but always with ALL array items in shapes:
{ "shapes":
[
{"shape": "square", "color": "blue"},
{"shape": "circle", "color": "red"}
]
}
However, I'd like to get the document (Document 1) only with the array that contains color=red:
{ "shapes":
[
{"shape": "circle", "color": "red"}
]
}
How can I do this?
MongoDB 2.2's new $elemMatch projection operator provides another way to alter the returned document to contain only the first matched shapes element:
db.test.find(
{"shapes.color": "red"},
{_id: 0, shapes: {$elemMatch: {color: "red"}}});
Returns:
{"shapes" : [{"shape": "circle", "color": "red"}]}
In 2.2 you can also do this using the $ projection operator, where the $ in a projection object field name represents the index of the field's first matching array element from the query. The following returns the same results as above:
db.test.find({"shapes.color": "red"}, {_id: 0, 'shapes.$': 1});
MongoDB 3.2 Update
Starting with the 3.2 release, you can use the new $filter aggregation operator to filter an array during projection, which has the benefit of including all matches, instead of just the first one.
db.test.aggregate([
// Get just the docs that contain a shapes element where color is 'red'
{$match: {'shapes.color': 'red'}},
{$project: {
shapes: {$filter: {
input: '$shapes',
as: 'shape',
cond: {$eq: ['$$shape.color', 'red']}
}},
_id: 0
}}
])
Results:
[
{
"shapes" : [
{
"shape" : "circle",
"color" : "red"
}
]
}
]
The new Aggregation Framework in MongoDB 2.2+ provides an alternative to Map/Reduce. The $unwind operator can be used to separate your shapes array into a stream of documents that can be matched:
db.test.aggregate(
// Start with a $match pipeline which can take advantage of an index and limit documents processed
{ $match : {
"shapes.color": "red"
}},
{ $unwind : "$shapes" },
{ $match : {
"shapes.color": "red"
}}
)
Results in:
{
"result" : [
{
"_id" : ObjectId("504425059b7c9fa7ec92beec"),
"shapes" : {
"shape" : "circle",
"color" : "red"
}
}
],
"ok" : 1
}
Caution: This answer provides a solution that was relevant at that time, before the new features of MongoDB 2.2 and up were introduced. See the other answers if you are using a more recent version of MongoDB.
The field selector parameter is limited to complete properties. It cannot be used to select part of an array, only the entire array. I tried using the $ positional operator, but that didn't work.
The easiest way is to just filter the shapes in the client.
If you really need the correct output directly from MongoDB, you can use a map-reduce to filter the shapes.
function map() {
filteredShapes = [];
this.shapes.forEach(function (s) {
if (s.color === "red") {
filteredShapes.push(s);
}
});
emit(this._id, { shapes: filteredShapes });
}
function reduce(key, values) {
return values[0];
}
res = db.test.mapReduce(map, reduce, { query: { "shapes.color": "red" } })
db[res.result].find()
Another interesing way is to use $redact, which is one of the new aggregation features of MongoDB 2.6. If you are using 2.6, you don't need an $unwind which might cause you performance problems if you have large arrays.
db.test.aggregate([
{ $match: {
shapes: { $elemMatch: {color: "red"} }
}},
{ $redact : {
$cond: {
if: { $or : [{ $eq: ["$color","red"] }, { $not : "$color" }]},
then: "$$DESCEND",
else: "$$PRUNE"
}
}}]);
$redact "restricts the contents of the documents based on information stored in the documents themselves". So it will run only inside of the document. It basically scans your document top to the bottom, and checks if it matches with your if condition which is in $cond, if there is match it will either keep the content($$DESCEND) or remove($$PRUNE).
In the example above, first $match returns the whole shapes array, and $redact strips it down to the expected result.
Note that {$not:"$color"} is necessary, because it will scan the top document as well, and if $redact does not find a color field on the top level this will return false that might strip the whole document which we don't want.
Better you can query in matching array element using $slice is it helpful to returning the significant object in an array.
db.test.find({"shapes.color" : "blue"}, {"shapes.$" : 1})
$slice is helpful when you know the index of the element, but sometimes you want
whichever array element matched your criteria. You can return the matching element
with the $ operator.
db.getCollection('aj').find({"shapes.color":"red"},{"shapes.$":1})
OUTPUTS
{
"shapes" : [
{
"shape" : "circle",
"color" : "red"
}
]
}
The syntax for find in mongodb is
db.<collection name>.find(query, projection);
and the second query that you have written, that is
db.test.find(
{shapes: {"$elemMatch": {color: "red"}}},
{"shapes.color":1})
in this you have used the $elemMatch operator in query part, whereas if you use this operator in the projection part then you will get the desired result. You can write down your query as
db.users.find(
{"shapes.color":"red"},
{_id:0, shapes: {$elemMatch : {color: "red"}}})
This will give you the desired result.
Thanks to JohnnyHK.
Here I just want to add some more complex usage.
// Document
{
"_id" : 1
"shapes" : [
{"shape" : "square", "color" : "red"},
{"shape" : "circle", "color" : "green"}
]
}
{
"_id" : 2
"shapes" : [
{"shape" : "square", "color" : "red"},
{"shape" : "circle", "color" : "green"}
]
}
// The Query
db.contents.find({
"_id" : ObjectId(1),
"shapes.color":"red"
},{
"_id": 0,
"shapes" :{
"$elemMatch":{
"color" : "red"
}
}
})
//And the Result
{"shapes":[
{
"shape" : "square",
"color" : "red"
}
]}
You just need to run query
db.test.find(
{"shapes.color": "red"},
{shapes: {$elemMatch: {color: "red"}}});
output of this query is
{
"_id" : ObjectId("562e7c594c12942f08fe4192"),
"shapes" : [
{"shape" : "circle", "color" : "red"}
]
}
as you expected it'll gives the exact field from array that matches color:'red'.
Along with $project it will be more appropriate other wise matching elements will be clubbed together with other elements in document.
db.test.aggregate(
{ "$unwind" : "$shapes" },
{ "$match" : { "shapes.color": "red" } },
{
"$project": {
"_id":1,
"item":1
}
}
)
Likewise you can find for the multiple
db.getCollection('localData').aggregate([
// Get just the docs that contain a shapes element where color is 'red'
{$match: {'shapes.color': {$in : ['red','yellow'] } }},
{$project: {
shapes: {$filter: {
input: '$shapes',
as: 'shape',
cond: {$in: ['$$shape.color', ['red', 'yellow']]}
}}
}}
])
db.test.find( {"shapes.color": "red"}, {_id: 0})
Use aggregation function and $project to get specific object field in document
db.getCollection('geolocations').aggregate([ { $project : { geolocation : 1} } ])
result:
{
"_id" : ObjectId("5e3ee15968879c0d5942464b"),
"geolocation" : [
{
"_id" : ObjectId("5e3ee3ee68879c0d5942465e"),
"latitude" : 12.9718313,
"longitude" : 77.593551,
"country" : "India",
"city" : "Chennai",
"zipcode" : "560001",
"streetName" : "Sidney Road",
"countryCode" : "in",
"ip" : "116.75.115.248",
"date" : ISODate("2020-02-08T16:38:06.584Z")
}
]
}
Although the question was asked 9.6 years ago, this has been of immense help to numerous people, me being one of them. Thank you everyone for all your queries, hints and answers. Picking up from one of the answers here.. I found that the following method can also be used to project other fields in the parent document.This may be helpful to someone.
For the following document, the need was to find out if an employee (emp #7839) has his leave history set for the year 2020. Leave history is implemented as an embedded document within the parent Employee document.
db.employees.find( {"leave_history.calendar_year": 2020},
{leave_history: {$elemMatch: {calendar_year: 2020}},empno:true,ename:true}).pretty()
{
"_id" : ObjectId("5e907ad23997181dde06e8fc"),
"empno" : 7839,
"ename" : "KING",
"mgrno" : 0,
"hiredate" : "1990-05-09",
"sal" : 100000,
"deptno" : {
"_id" : ObjectId("5e9065f53997181dde06e8f8")
},
"username" : "none",
"password" : "none",
"is_admin" : "N",
"is_approver" : "Y",
"is_manager" : "Y",
"user_role" : "AP",
"admin_approval_received" : "Y",
"active" : "Y",
"created_date" : "2020-04-10",
"updated_date" : "2020-04-10",
"application_usage_log" : [
{
"logged_in_as" : "AP",
"log_in_date" : "2020-04-10"
},
{
"logged_in_as" : "EM",
"log_in_date" : ISODate("2020-04-16T07:28:11.959Z")
}
],
"leave_history" : [
{
"calendar_year" : 2020,
"pl_used" : 0,
"cl_used" : 0,
"sl_used" : 0
},
{
"calendar_year" : 2021,
"pl_used" : 0,
"cl_used" : 0,
"sl_used" : 0
}
]
}
if you want to do filter, set and find at the same time.
let post = await Post.findOneAndUpdate(
{
_id: req.params.id,
tasks: {
$elemMatch: {
id: req.params.jobId,
date,
},
},
},
{
$set: {
'jobs.$[i].performer': performer,
'jobs.$[i].status': status,
'jobs.$[i].type': type,
},
},
{
arrayFilters: [
{
'i.id': req.params.jobId,
},
],
new: true,
}
);
This answer does not fully answer the question but it's related and I'm writing it down because someone decided to close another question marking this one as duplicate (which is not).
In my case I only wanted to filter the array elements but still return the full elements of the array. All previous answers (including the solution given in the question) gave me headaches when applying them to my particular case because:
I needed my solution to be able to return multiple results of the subarray elements.
Using $unwind + $match + $group resulted in losing root documents without matching array elements, which I didn't want to in my case because in fact I was only looking to filter out unwanted elements.
Using $project > $filter resulted in loosing the rest of the fields or the root documents or forced me to specify all of them in the projection as well which was not desirable.
So at the end I fixed all of this problems with an $addFields > $filter like this:
db.test.aggregate([
{ $match: { 'shapes.color': 'red' } },
{ $addFields: { 'shapes': { $filter: {
input: '$shapes',
as: 'shape',
cond: { $eq: ['$$shape.color', 'red'] }
} } } },
])
Explanation:
First match documents with a red coloured shape.
For those documents, add a field called shapes, which in this case will replace the original field called the same way.
To calculate the new value of shapes, $filter the elements of the original $shapes array, temporarily naming each of the array elements as shape so that later we can check if the $$shape.color is red.
Now the new shapes array only contains the desired elements.
for more details refer =
mongo db official referance
suppose you have document like this (you can have multiple document too) -
{
"_id": {
"$oid": "63b5cfbfbcc3196a2a23c44b"
},
"results": [
{
"yearOfRelease": "2022",
"imagePath": "https://upload.wikimedia.org/wikipedia/en/d/d4/The_Kashmir_Files_poster.jpg",
"title": "The Kashmir Files",
"overview": "Krishna endeavours to uncover the reason behind his parents' brutal killings in Kashmir. He is shocked to uncover a web of lies and conspiracies in connection with the massive genocide.",
"originalLanguage": "hi",
"imdbRating": "8.3",
"isbookMark": null,
"originCountry": "india",
"productionHouse": [
"Zee Studios"
],
"_id": {
"$oid": "63b5cfbfbcc3196a2a23c44c"
}
},
{
"yearOfRelease": "2022",
"imagePath": "https://upload.wikimedia.org/wikipedia/en/a/a9/Black_Adam_%28film%29_poster.jpg",
"title": "Black Adam",
"overview": "In ancient Kahndaq, Teth Adam was bestowed the almighty powers of the gods. After using these powers for vengeance, he was imprisoned, becoming Black Adam. Nearly 5,000 years have passed, and Black Adam has gone from man to myth to legend. Now free, his unique form of justice, born out of rage, is challenged by modern-day heroes who form the Justice Society: Hawkman, Dr. Fate, Atom Smasher and Cyclone",
"originalLanguage": "en",
"imdbRating": "8.3",
"isbookMark": null,
"originCountry": "United States of America",
"productionHouse": [
"DC Comics"
],
"_id": {
"$oid": "63b5cfbfbcc3196a2a23c44d"
}
},
{
"yearOfRelease": "2022",
"imagePath": "https://upload.wikimedia.org/wikipedia/en/0/09/The_Sea_Beast_film_poster.png",
"title": "The Sea Beast",
"overview": "A young girl stows away on the ship of a legendary sea monster hunter, turning his life upside down as they venture into uncharted waters.",
"originalLanguage": "en",
"imdbRating": "7.1",
"isbookMark": null,
"originCountry": "United States Canada",
"productionHouse": [
"Netflix Animation"
],
"_id": {
"$oid": "63b5cfbfbcc3196a2a23c44e"
}
},
{
"yearOfRelease": "2021",
"imagePath": "https://upload.wikimedia.org/wikipedia/en/7/7d/Hum_Do_Hamare_Do_poster.jpg",
"title": "Hum Do Hamare Do",
"overview": "Dhruv, who grew up an orphan, is in love with a woman who wishes to marry someone with a family. In order to fulfil his lover's wish, he hires two older individuals to pose as his parents.",
"originalLanguage": "hi",
"imdbRating": "6.0",
"isbookMark": null,
"originCountry": "india",
"productionHouse": [
"Maddock Films"
],
"_id": {
"$oid": "63b5cfbfbcc3196a2a23c44f"
}
},
{
"yearOfRelease": "2021",
"imagePath": "https://upload.wikimedia.org/wikipedia/en/7/74/Shang-Chi_and_the_Legend_of_the_Ten_Rings_poster.jpeg",
"title": "Shang-Chi and the Legend of the Ten Rings",
"overview": "Shang-Chi, a martial artist, lives a quiet life after he leaves his father and the shadowy Ten Rings organisation behind. Years later, he is forced to confront his past when the Ten Rings attack him.",
"originalLanguage": "en",
"imdbRating": "7.4",
"isbookMark": null,
"originCountry": "United States of America",
"productionHouse": [
"Marvel Entertainment"
],
"_id": {
"$oid": "63b5cfbfbcc3196a2a23c450"
}
}
],
"__v": 0
}
=======
mongo db query by aggregate command -
mongomodels.movieMainPageSchema.aggregate(
[
{
$project: {
_id:0, // to supress id
results: {
$filter: {
input: "$results",
as: "result",
cond: { $eq: [ "$$result.yearOfRelease", "2022" ] }
}
}
}
}
]
)
For the new version of MongoDB, it's slightly different.
For db.collection.find you can use the second parameter of find with the key being projection
db.collection.find({}, {projection: {name: 1, email: 0}});
You can also use the .project() method.
However, it is not a native MongoDB method, it's a method provided by most MongoDB driver like Mongoose, MongoDB Node.js driver etc.
db.collection.find({}).project({name: 1, email: 0});
And if you want to use findOne, it's the same that with find
db.collection.findOne({}, {projection: {name: 1, email: 0}});
But findOne doesn't have a .project() method.

How to remove objoct from object by finding in type script

This is my object
"filterValue":[
{"label":"--Select a Member--","value":""},
{"label":"ghi.jkl","value":{"Id":"1",}},
{"label":"abc.def","value":{"Id":"2",}},
{"label":"asd.vdf","value":{"Id":"3",}},
]
from this i want to search where value.Id = 2 and i want to remove that obeject line.
how can i do that..?
note:first value will be empty there is no data in value.
i have tried something like this:
filterValue.splice( filterValue.indexOf(2), 1 );
You can't use indexOf in this case because you are checking a complex object but you can use findIndex like this:
filterValue.splice( filterValue.findIndex(a => a.Id == 2), 1 );
You might want to change the code the check if findIndex actually found something by checking if it returns something larger than (or equal to) 0.
You can use filter to get a new filtered array (filteredArr):
var arr = [
{"label":"--Select a Member--","value":""},
{"label":"ghi.jkl","value":{"Id":"1",}},
{"label":"abc.def","value":{"Id":"2",}},
{"label":"asd.vdf","value":{"Id":"3",}}
];
var filteredArr = arr.filter((x) => JSON.stringify(x.value) !== JSON.stringify({"Id":"2"}));
console.log(filteredArr);
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
You have a couple of subtly traps to avoid with your specific example.
The structure of items differs, so you need to be careful that you don't have a problem with the "--Select a Member--" item, which doesn't have a value.Id.
The example below cheaply solves the type issue (the best common type between the array members doesn't contain the property you are interested in).
const items = [
{ "label": "--Select a Member--", "value": "" },
{ "label": "ghi.jkl", "value": { "Id": "1", } },
{ "label": "abc.def", "value": { "Id": "2", } },
{ "label": "asd.vdf", "value": { "Id": "3", } },
];
const filtered = items.filter((i: any) => !i.value || !i.value.Id || i.value.Id !== '2');
console.log(filtered);
Output:
[
{"label":"--Select a Member--","value":""},
{"label":"ghi.jkl","value":{"Id":"1"}},
{"label":"asd.vdf","value":{"Id":"3"}}
]
const obj = {
filterValue: [
{ label: "--Select a Member--", value: "" },
{ label: "ghi.jkl", value: { Id: "1" } },
{ label: "abc.def", value: { Id: "2" } },
{ label: "asd.vdf", value: { Id: "3" } }
]
};
var changedObj = obj.filterValue.filter((data, index) => {
return data.value.Id != "1";
});
console.log(changedObj);