I am trying to figure out an approach to delete all entries for a specific property in an elasticsearch index and remove all type mappings for that property.
I have been looking at the following two doc pages: put mapping and delete mapping
From second link:
"Allow to delete a mapping (type) along with its data. The REST
endpoint is /{index}/{type} with DELETE method."
What I think I need is a /{index}/{type}/{property}?
Do I need to recreate the whole index to accomplish this, i.e. moving and manipulating data between types?
For Example, calling GET on the mapping:
curl -XGET 'http://.../some_index/some_type/_mapping'
result:
{
"some_type": {
"properties": {
"propVal1": {
"type": "double",
"index": "analyzed"
},
"propVal2": {
"type": "string",
"analyzer": "keyword"
},
"propVal3": {
"type": "string",
"analyzer": "keyword"
}
}
}
}
after this delete operation on propVal3 would return:
curl -XGET 'http://.../some_index/some_type/_mapping'
result:
{
"some_type": {
"properties": {
"propVal1": {
"type": "double",
"index": "analyzed"
},
"propVal2": {
"type": "string",
"analyzer": "keyword"
}
}
}
}
and all data for propVal3 would be removed through the index.
You can not do that. Just forget that this value exists... ;-)
If you really need to remove it, you will have to reindex your documents.
You can use the new _reindex api for this, you could even PUT a new _mapping to the dest index before running the reindex so you can change the properties of the fields in your index.
To do a reindex and removing a property, you can do this:
POST /_reindex
{
"source": {
"index": "twitter"
},
"dest": {
"index": "new_twitter",
},
"script": {
"inline": "ctx._source.remove('whatever')"
}
}
if you would use this in combination with the _aliases API you can modify indexes without having any 'downtime'
It's not currently possible to remove a property from a mapping. In order to remove all values of a property from all records, you need to reindex all records with this property removed.
You can choose whats documents fields you will reindex to a new index. For example:
POST _reindex
{
"source": {
"index": "my-source-index",
"_source": ["host.hostname", "host.ip", "another_field"]
},
"dest": {
"index": "my-dest-index"
}
}
Reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html#docs-reindex-filter-source
Just want to add another approach for the case of "removing property from index".
POST pulse/_update_by_query
{
"query": {
"match_all": {}
},
"script": {
"source": "ctx._source.remove(\"file_id\")",
"lang": "painless"
}
}
Related
I am trying to delete all documents in my collection infrastructure that have a type.primary property of "pipelines" and a type.secondary property of "oil."
I'm trying to use the following query:
db.infrastructure.deleteMany({$and: [{"properties.type.primary": "pipelines"}, {"properties.type.secondary": "oil"}] }),
That returns: { acknowledged: true, deletedCount: 0 }
I expect my query to work because in MongoDB Compass, I can retrieve 182 documents that match the query {$and: [{"properties.type.primary": "pipelines"}, {"properties.type.secondary": "oil"}] }
My documents appear with the following structure (relevant section only):
properties": {
"optional": {
"description": ""
},
"original": {
"Opername": "ENBRIDGE",
"Pipename": "Lakehead",
"Shape_Leng": 604328.294581,
"Source": "EIA"
},
"required": {
"unit": null,
"viz_dim": null,
"years": []
},
"type": {
"primary": "pipelines",
"secondary": "oil"
}
...
My understanding is that I just need to pass a filter to deleteMany() and that $and expects an array of objects. For some reason the two combined isn't working here.
I realized the simplest answer was the correct one -- I spelled my database name incorrectly.
I need to refer to a sub-schema of certain property (Kind in the example) from a different property in the schema, and then enforce some more conditions on it. Important thing to note is I cannot make those changes where I've defined Kind, I need to refer to it from some other property and then add conditionals on top of it.
{
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"Kind": {
"$id": "#/properties/Kind",
"type": "string",
"enum": [
"Foo",
"Bar"
]
}
},
"allOf": [
{
"if": {
"$ref": "#/properties/Kind",
"const": "Foo"
},
"then": {
"required": [
"MyField"
]
}
}
]
}
A json object like below should fail the validation, because MyField property is absent
{
"Kind": "Foo"
}
I don't want the following solution, since this is just a simplified version and ultimately I want to refer to Kind value from another property. If I do following, then #/properties/Kind is interpreted relative to where I refer Kind so it doesn't refer to the Kind at the top level. I want a solution which uses the $ref and $id keywords.
{
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"Kind": {
"$id": "#/properties/Kind",
"type": "string",
"enum": [
"Foo",
"Bar"
]
}
},
"allOf": [
{
"if": {
"properties": {"Kind":{
"const":"Foo"
}}
},
"then": {
"required": [
"MyField"
]
}
}
]
}
To summarize, let's say I've following JSON structure. The last allOf statement is what I need to add.
- Kind ( enum of One,Two)
- Other
- MyField
- ConditionField
- allOf ( which enforces the required-ness of MyField based on ConditionField)
- allOf ( MyField should be not-required if Kind is One)
[ To add this last conditional, I need to reference the value of Kind.
I'm hoping providing $id to Kind and referring to it with $ref should be my approach,
which doesn't seem to be working]
To summarize even further, I would get my answer if we're able to get the first snippet work using $id and $ref.
There seem to be some misunderstandings that are making it difficult to fully understand the problem here, but one part of the edited question makes enough sense that I think I can start things off and we can iterate on the answer as necessary.
Let's start with some of the things that don't make sense in hopes that it helps clarify possible misunderstandings.
$ref can't change the behavior of a schema. If you can't do something without $ref, then you can't make the schema behave another way by introducing $ref. The only exception to that rule is recursive schemas, which would require an infinitely large and repeating schema without using $ref.
I'm not sure what you are trying to get from $id, but it's pretty safe to say you don't need it for this. In any case, the $id used the question is invalid. An anchor can not have a / in it. Even if it was valid, it would be redundant because you can reference that location with the same JSON Pointer without an anchor.
MyField should be not-required if Kind is One
I'm not sure if "not-required" means forbidden or optional. Everything is optional by default in JSON Schema, so if you meant optional, there is nothing to do here. Therefore, I'll assume for now that you mean forbidden. Here's what that would look like.
{
"type": "object",
"properties": {
"Kind": { "enum": ["One", "Two"] },
"Other": {
"type": "object",
"properties": {
"MyField": {}
}
}
},
"allOf": [
{
"if": {
"properties": {
"Kind": { "const": "One" }
},
"required": ["Kind"]
},
"then": {
"properties": {
"Other": {
"not": { "required": ["MyField"] }
}
}
}
}
]
}
Im trying to make a system monitor, which is highly customisable by user. This customization is achieved by using JSON file for modeling look of system monitor. The JSON could look like this.
{
"_": "WINDOW",
"name": "myWindow",
"children": [
{
"_": "CPU",
"name": "cpuMonitor",
"freq_Unit": "MHZ"
},
{
"_": "NETWORK",
"name": "network",
"unit": "Kb/s"
},
{
"_": "DISK",
"name": "disk"
}
],
"background": "red"
}
As you can see, each object coresponds to this schema.
{
"$schema": "http://json-schema.org/draft-07/schema#",
"name":"Component",
"type": "object",
"properties":{
"_": {
"type": "string"
},
"name":{
"type":"string"
},
"childern":{
"type":"array"
}
},
"required": ["_","name"]
}
But each component has also its own schema definition. I'd like to parse whole JSON and validate each node for different schema (first if its component and then to its corresponding schema).
I had look at rapidJson and other libraries, but I didnt find solution for validating nodes for different schema. Do you know any library which could do that? Or is it even possible to validate JSON in this way?
All feedback on how to solve this will be appreciated.
Edit: Corrected schema :(
There's a simple approach involved with that, use the oneOf pattern declaration to specify the layout of the array elements. Inside these nested declarations, you specify the fixed identifier (probably the content of your _ field) as a constant, so that there is only one nested schema matching each of your panel types.
Notes:
I had to specify the constant type identifier using the enum specifier because the regular constant specifier didn't work with the library I was using. This may also have been an oversight in the revision of the specification that it was based on.
A different approach is to split the the validation steps. You simply verify that the elements of the array are objects and that they have a string field _ containing one of the supported types. When iterating over the array, you then validate each field individually according to its _ field.
In addition to Ulrich's answer, here's an example of what I'd do:
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Component",
"type": "object",
"definitions": {
"base": {
"properties": {
"name": { "type": "string" },
"children": {
"type": "array",
"items": { "$ref": "#" }
}
},
"required": [ "_", "name" ]
},
"cpu": {
"properties": {
"_": { "const": "CPU" },
"freq_Unit": "MHZ"
}
},
"network": {
"properties": {
"_": { "const": "NETWORK" },
"unit": "Kb/s"
}
},
"disk": {
"properties": {
"_": { "const": "DISK" }
}
},
"window": {
"properties": {
"_": { "const": "WINDOW" },
"background": { "enum": [ "red", "orange", "yellow", ... ] }
}
}
},
"allOf": [
{ "$ref": "#/definitions/base" },
{
"oneOf": [
{ "$ref": "#/definitions/cpu" },
{ "$ref": "#/definitions/network" },
{ "$ref": "#/definitions/disk" },
{ "$ref": "#/definitions/window" }
]
}
]
}
First, we require that any instance MUST adhere to base which declares _ and name as required properties. Additionally, we declare a children array property that requires all items also match this schema (giving us a recursive behavior). This doesn't really do much except that it allows us to declare these things in one place instead of having to declare them in the other three definitions.
(Note that we don't declare _ in the properties list. This means that any value will pass for this portion of the schema. We clean it up in the next part. If you want to ensure that future components are declared with strings, then you can add a "type": "string" requirement to that property, but I don't feel it's necessary unless others are authoring those components.)
Second, we declare each of our specific types as separate definitions, using the const keyword to isolate the one we want. This construct is analogous to a switch (or case) statement. If the instance doesn't match one of these explicit options, it fails. If it's missing one of the required base properties, it fails.
This will get you where you want to be.
To take it further, there are two more things you can do:
Add required to the other definitions to say that the specific properties are also required (e.g. freq_Unit for the cpu definition).
Declare each of the definitions in separate files. This would allow you to add a new definition by simply adding a new file and referencing it in the main schema. In my opinion, it's a bit cleaner. Some people prefer to have it all in one file, though.
I am using a alpacajs library to construct my json schema into a form that doesnt support oneOf keyword.
I have a simple requirement where if one field is present then the other one should be hidden.
Is there a way to implement using json schema itself without using oneOf keyword.
Right now i am using dependencies keyword to handle my conditions.
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Company Setup",
"description": "Company Wizard Setup",
"type": "object",
"properties": {
"name": {
"type": "string",
"title": "Company Name"
},
"alternate_names": {
"type": "array",
"title": "Alternate Name",
"items": {
"type": "string"
},
"maxItems": 5
}
},
"dependencies": {
"name": "alternate_names"
}
}
{
"options": {
"fields": {
"name": {
"dependencies": {
"alternate_names": "Value for empty Array?"
}
},
"alternate_names": {
...
}
}
}
}
This is just a stripped down version of my schema for the sake of clarity and i would appreciate if anyone can help me tell if dependencies can be used to remove a field too. I could potentially just use javascript to get this but i wanted to handle this just using schema as i am going to use same for validation too, infact schema has more use as a validator than form itself.
Since none of the field is boolean, i can't just put true and false in value.
Thanks in advance.
Question title pretty much self explanatory. It is possible to create an id-less entity in Orion. An id = .* query returns normally as an id-less, although existing entity. But how can someone delete that entity? This request didn't work obviously:
{
"contextElements": [
{
"type": "",
"isPattern": "false",
"id": ""
}
],
"updateAction": "DELETE"
}
This is the returned query:
{
"contextElement": {
"type": "",
"isPattern": "false",
"id": "",
"attributes": [
{
"name": "temp",
"type": "integer",
"value": "15"
},
{
"name": "pressure",
"type": "integer",
"value": "720"
}
]
},
"statusCode": {
"code": "200",
"reasonPhrase": "OK"
}
}
There is a known bug (now fixed) in Orion that seems to be causing your issue. Basically Orion interprets the final "/" at the end of an URL as an empty element.
For example (as documented in the issue):
v1/contextEntityTypes queries for all types, while
v1/contextEntityTypes/ queries only for the empty type
In your particular case, something similar happens with some REST operations. If you do a GET /v1/contextEntities you will see all entities, including the empty-id one. You can query that particular entity with GET /v1/contextEntity/ (note the final "/").
And then, the DELETE method doesn't seem to use the same pattern. So if you do DELETE /v1/contextEntity/ You get a No context element found.
So, basically, this is another manifestation of a known bug.