How to avoid Mongodb from creating "members" fields in Json? - json

I am storing Json Schemas/Objects in a mongodb to query them from a REST client. For accessing the database, I use Spring Boot with MongoTemplates. My problem is that mongodb seems to create several fields like "members", "_class" and "_id" automatically. I was already able to remove the _class and _id fields in the output, but these members seem to be quite persistent, I found no information about how to remove them (or suppress them in the output at least). I also can't just remove them from the objects after querying them because then all the information (title, definitions...) is gone, too.
Anyone out there who had this problem as well and can help? I would appreciate :)
Regards
Object(s) after querying out of mongodb:
[
{
"members": {
"__ID__": {
"value": "c89bae58-8911-45a4-80b0-843278ac72c8"
}
}
},
{
"members": {
"title": {
"value": "LogicalNodes"
},
"__ID__": {
"value": "a09ffc24-d25f-467a-bcd7-9eaed5fbc44e"
}
}
},
{
"members": {
"title": {
"value": "LogicalNodes"
},
"definitions": {
"members": {
"mv": {
"members": {
"type": {
"value": "object"
},
"properties": {
"members": {
"i": {
"members": {
"type": {
"value": "number"
}
}
}
}
}
}
}are
}
},
"__ID__": {
"value": "eb054bd3-2c50-43eb-9ee9-4c8e54a8236d"
}
}
},
{
"members": {
"title": {
"value": "TestObject"
},
"definitions": {
"members": {
"ab": {
"members": {
"type": {
"value": "object"
},
"properties": {
"members": {
"q": {
"members": {
"type": {
"value": "number"
}
}
}
}
}
}
}
}
},
"__ID__": {
"value": "4b5a5813-5596-4c88-9a24-e19ae08b7548"
}
}
},
{
"members": {
"$schema": {
"value": "schema"
},
"title": {
"value": "TestObject"
},
"definitions": {
"members": {
"ab": {
"members": {
"type": {
"value": "object"
},
"properties": {
"members": {
"q": {
"members": {
"type": {
"value": "number"
}
}
}
}
}
}
}
}
},
"__ID__": {
"value": "e2a82f73-709d-410d-a79c-1e997f9fc5a4"
}
}
},
{
"members": {
"$schema": {
"value": "schema"
},
"title": {
"value": "TestObject"
},
"name": {
"value": "test"
},
"__ID__": {
"value": "26089639-8b99-4f47-8696-d8f3e50694b7"
}
}
}
]
NOTE: I have added the "ID" fields by myself. Everything is fine, but there shouldn't be any "members" fields, as they are producing more overhead and make the output worse readable as well as they make the queries more complex.

Related

Why is the json object valid against this conditional schema?

Here is the json object.
{
"payment": {
"account": [
{
"type": "ACCOUNT_INFORMATION",
"identification": "2451114"
},
{
"type": "XXX",
"identification": "2451114"
}
]
}
}
And this is the schema.
{
"if": {
"properties": {
"payment": {
"properties": {
"account": {
"items": {
"properties": {
"type": {
"const": "ACCOUNT_INFORMATION"
}
}
}
}
}
}
}
},
"then": {
"properties": {
"payment": {
"properties": {
"account": {
"items": {
"properties": {
"identification": {
"maxLength": 8,
"minLength": 8
}
}
}
}
}
}
}
}
}
If remove the second account items as follows, the schema gives error.
{
"payment": {
"account": [
{
"type": "ACCOUNT_INFORMATION",
"identification": "2451114"
}
]
}
}
Is this due to the conditional schema cannot be apply to an embedded array?
Validation used https://www.jsonschemavalidator.net/
The first json object returns no error while the second one returns error with violation of minLength constraint.
Should both return error?
To see what's happening, let's break down the schema to focus on the critical part of the if schema.
"items": {
"properties": {
"type": { "const": "ACCOUNT_INFORMATION" }
}
}
Given this schema, the following instance is not valid because not all "type" properties have the value "ACCOUNT_INFORMATION".
[
{
"type": "ACCOUNT_INFORMATION",
"identification": "2451114"
},
{
"type": "XXX",
"identification": "2451114"
}
]
And, this following value is valid because all "type" properties have the value "ACCOUNT_INFORMATION".
[
{
"type": "ACCOUNT_INFORMATION",
"identification": "2451114"
}
]
That difference in validation result is the reason these two values behave differently in your schema. The then schema is applied only when the if schema evaluates to true, which is what you get with the second example and not the first. The then schema is applied on the second example and the minLength constraint causes validation to fail.
It seems like your conditional only applies to the items schema, so you can solve this by moving your conditional into that object only.
{
"properties": {
"payment": {
"properties": {
"account": {
"items": {
"if": {
"properties": {
"type": {
"const": "ACCOUNT_INFORMATION"
}
},
"required": ["type"]
},
"then": {
"properties": {
"identification": {
"maxLength": 8,
"minLength": 8
}
}
}
}
}
}
}
}
}

Json Schema Not working for nested Attributes

I am trying to add some validation to my json schema . I am validating json schema against json using this website https://www.jsonschemavalidator.net/. I am not able to put validation on eventPayload/totalAmount based on value present in eventName. It is not failing when it should fail. Should I give the whole path of eventName attribute as it is not present in eventPayload ? If yes, how to do that.
"totalAmount": {
"type": [
"integer",
"number"
],
"minLength": 1,
"multipleOf": 0.01,
"if": {
"properties": {
"eventName": {
"enum": [
"Test10",
"Test12"
]
}
}
},
"then": {
"properties": {
"totalAmount": {
"exclusiveMinimum": 0
}
}
},
"else": {
"if": {
"properties": {
"eventName": {
"enum": [
"Test1",
"Test2",
"Test3"
]
}
}
},
"then": {
"properties": {
"totalAmount": {
"exclusiveMaximum": 0
}
}
}
}
}
It is not possible to reference values up the tree (e.g. totalAmount is below eventName), you have to define from the top down. Using oneOf (instead of if/then/else) and schema composition, you could solve it as follows (minimal example):
Schema:
{
"$schema": "https://json-schema.org/draft/2019-09/schema",
"type": "object",
"properties": {
"eventName": {
"type": "string",
"enum": ["Test10", "Test12", "Test1", "Test2", "Test3"]
},
"eventPayload": {
"type": "object",
"properties": {
"totalAmount": {
"type": "number"
}
}
}
},
"oneOf": [
{
"properties": {
"eventName": {
"enum": ["Test10", "Test12"]
},
"eventPayload": {
"properties": {
"totalAmount": {
"exclusiveMinimum": 0
}
}
}
}
},
{
"properties": {
"eventName": {
"enum": ["Test1", "Test2", "Test3"]
},
"eventPayload": {
"properties": {
"totalAmount": {
"exclusiveMaximum": 0
}
}
}
}
}
]
}
Here, we first define the general structure in the properties object (no validation yet, just the type of expected objects). Then, in the oneOf array we add the alternatives: If eventName is either "Test10" or "Test12" apply the exclusiveMinimum, if it is one of the others, apply exclusiveMaximum.
Any incoming json has to fulfill both the schema defined in properties and one of the schemas in oneOf. This way of layering schemas is how json schema implements composition. Using this schema and https://www.jsonschemavalidator.net/ we can verify that it
accepts
{
"eventName": "Test12",
"eventPayload": {
"totalAmount": 5
}
}
and
{
"eventName": "Test2",
"eventPayload": {
"totalAmount": -5
}
}
but rejects
{
"eventName": "Test12",
"eventPayload": {
"totalAmount": -5
}
}
and
{
"eventName": "Test2",
"eventPayload": {
"totalAmount": 5
}
}
The properties keyword in your if clause will evaluate to true if property eventName is not present. To ensure that it is, add "required": ["eventName"] to the condition.

How to find value from nested object with key wildcard in Compass MongoDB?

I have one nested object in Compass MongoDB:
{
"data_rm": {
"pembiayaan": {
"name": "pembiayaan",
"value": "asuransi",
"type": "radio",
"title": ""
},
"asuransi": {
"name": "asuransi",
"value": "ADMEDIKA - FWD LIFE INDONESIA (FINANSIAL WIRAMITRA DANADYAKSA, PT)",
"type": "select-one",
"text": "ADMEDIKA - FWD LIFE INDONESIA (FINANSIAL WIRAMITRA DANADYAKSA, PT)",
"title": ""
},
"informasi_diperoleh_dari": {
"name": "informasi_diperoleh_dari",
"value": "pasien",
"type": "radio",
"title": ""
},
"cara_datang": {
"name": "cara_datang",
"value": "sendiri",
"type": "radio",
"title": ""
},
"nama_pengantar": {
"name": "nama_pengantar",
"value": "Tn. BAGUS",
"type": "text",
"title": ""
},
"no_telp_pengantar": {
"name": "no_telp_pengantar",
"value": "0813xxxxxxxx",
"type": "text",
"title": ""
}
}
How did I return the document WHERE type is "select-one"? (I want to find which key in data_rm has the type of select-one)
** EDIT **
The desired output is:
{
data_rm: {
"asuransi": {
"name": "asuransi",
"value": "ADMEDIKA - FWD LIFE INDONESIA (FINANSIAL WIRAMITRA DANADYAKSA, PT)",
"type": "select-one",
"text": "ADMEDIKA - FWD LIFE INDONESIA (FINANSIAL WIRAMITRA DANADYAKSA, PT)",
"title": ""
}
}
}
Here's one way you could do it. In Compass you can enter each stage of the aggregation pipeline.
db.collection.aggregate([
{
"$set": { "rmArray": { "$objectToArray": "$data_rm" } }
},
{
"$set": {
"selOne": {
"$filter": {
"input": "$rmArray",
"cond": { "$eq": [ "$$this.v.type", "select-one" ] }
}
}
}
},
{
"$replaceWith": { "data_rm": {"$arrayToObject": "$selOne" } }
}
])
Try it on mongoplayground.net.

What is wrong with this elastic json query, mapping?

I am trying to use nested JSON to query DB records. Here is my query -
"query": {
"nested": {
"path": "metadata.technical",
"query": {
"bool": {
"must": [
{
"term": {
"metadata.technical.key": "techcolor"
}
},
{
"term": {
"metadata.technical.value": "red"
}
}
]
}
}
}
}
Here is this part in my mapping.json -
"metadata": {
"include_in_parent": true,
"properties": {
"technical": {
"type": "nested",
"properties": {
"key": {
"type": "string"
},
"value": {
"type": "string"
}
}
}
}
}
And I have table that has column 'value' and its content is -
{"technical":
{
"techname22": "test",
"techcolor":"red",
"techlocation": "usa"
}
}
Why I can't get any results with this? FYI I am using ES 1.7. Thanks for any help.
To respect the mapping you've defined your sample document should look like this:
{
"technical": [
{
"key": "techname22",
"value": "test"
},
{
"key": "techcolor",
"value": "red"
},
{
"key": "techlocation",
"value": "usa"
}
]
}
Changing your document with the above structure would make your query work as it is.
The real mapping of this document:
{
"technical": {
"techname22": "test",
"techcolor": "red",
"techlocation": "usa"
}
}
Is more like this:
{
"include_in_parent": true,
"properties": {
"technical": {
"type": "nested",
"properties": {
"techname22": {
"type": "string"
},
"techcolor": {
"type": "string"
},
"techlocation": {
"type": "string"
}
}
}
}
}
If all your keys are dynamic and not known in advance, you can also configure your mapping to be dynamic as well, i.e. don't define any fields in the nested type and new fields will be added if not already present in the mapping:
{
"include_in_parent": true,
"properties": {
"technical": {
"type": "nested",
"properties": {
}
}
}
}

How to use function_score on nested geo_point field

I've been trying to use function_score on nested geo_type but Elasticsearch always returns documents with _score = 1.
My mapping
{
"myindex": {
"mappings": {
"offers": {
"properties": {
"country_id": {
"type": "long"
},
"created_at": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss"
},
"locations": {
"type": "nested",
"properties": {
"city": {
"type": "string",
"store": true,
"fields": {
"city_original": {
"type": "string",
"analyzer": "keyword_analyzer"
}
}
},
"coordinates": {
"type": "geo_point"
}
}
}
}
}
}
}
}
The problem is that some of documents can have empty coordinates. My query:
GET /myindex/offers/_search?explain
{
"query": {
"nested": {
"path": "locations",
"query": {
"function_score": {
"functions": [
{
"filter": {
"exists": {
"field": "locations.coordinates"
}
},
"gauss": {
"locations.coordinates": {
"origin": {
"lat": 49.1,
"lon": 17.03333
},
"scale": "10km",
"offset": "20km"
}
}
}
]
}
}
}
}
}
As I said, Elasticsearch returns documents with _score = 1 and explanation: Score based on child doc range from 21714 to 21714
What am I doing wrong? My version of Elastisearch is 1.7.