I am trying to delete all documents in my collection infrastructure that have a type.primary property of "pipelines" and a type.secondary property of "oil."
I'm trying to use the following query:
db.infrastructure.deleteMany({$and: [{"properties.type.primary": "pipelines"}, {"properties.type.secondary": "oil"}] }),
That returns: { acknowledged: true, deletedCount: 0 }
I expect my query to work because in MongoDB Compass, I can retrieve 182 documents that match the query {$and: [{"properties.type.primary": "pipelines"}, {"properties.type.secondary": "oil"}] }
My documents appear with the following structure (relevant section only):
properties": {
"optional": {
"description": ""
},
"original": {
"Opername": "ENBRIDGE",
"Pipename": "Lakehead",
"Shape_Leng": 604328.294581,
"Source": "EIA"
},
"required": {
"unit": null,
"viz_dim": null,
"years": []
},
"type": {
"primary": "pipelines",
"secondary": "oil"
}
...
My understanding is that I just need to pass a filter to deleteMany() and that $and expects an array of objects. For some reason the two combined isn't working here.
I realized the simplest answer was the correct one -- I spelled my database name incorrectly.
Related
Wondering if I can create a "dynamic mapping" within an elasticsearch index. The problem I am trying to solve is the following: I have a schema that has an attribute that contains an object that can differ greatly between records. I would like to mirror this data within elasticsearch if possible but believe that automatic mapping may get in the way.
Imagine a scenario where I have a schema like the following:
{
name: string
origin: string
payload: object // can be of any type / schema
}
Is it possible to create a mapping that supports this? I do not need to query the records by this payload attribute, but it would be great if I can.
Note that I have checked the documentation but am confused on if what elastic calls dynamic mapping is what I am looking for.
It's certainly possible to specify which queryable fields you expect the payload to contain and what those fields' mappings should be.
Let's say each doc will include the fields payload.livemode and payload.created_at. If these are the only two fields you'll want to perform queries on, and you'd like to disable dynamic, index-time mappings autogenerated by Elasticsearch for the rest of the fields, you can use dynamic templates like so:
PUT my-payload-index
{
"mappings": {
"dynamic_templates": [
{
"variable_payload": {
"path_match": "payload",
"mapping": {
"type": "object",
"dynamic": false,
"properties": {
"created_at": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss"
},
"livemode": {
"type": "boolean"
}
}
}
}
}
],
"properties": {
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
},
"origin": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
}
}
}
}
Then, as you ingest your docs:
POST my-payload-index/_doc
{
"name": "abc",
"origin": "web.dev",
"payload": {
"created_at": "2021-04-05 08:00:00",
"livemode": false,
"abc":"def"
}
}
POST my-payload-index/_doc
{
"name": "abc",
"origin": "web.dev",
"payload": {
"created_at": "2021-04-05 08:00:00",
"livemode": true,
"modified_at": "2021-04-05 09:00:00"
}
}
and verify with
GET my-payload-index/_mapping
no new mappings will be generated for the fields payload.abc nor payload.modified_at.
Not only that — the new fields will also be ignored, as per the documentation:
These fields will not be indexed or searchable, but will still appear in the _source field of returned hits.
Side note: if fields are neither stored nor searchable, they're effectively the opposite of enabled.
The Big Picture
Working with variable contents of a single, top-level object is quite standard. Take for instance the stripe event object — each event has an id, an api_version and a few other shared params. Then there's the data object that's analogous to your payload field.
Now, all is fine, until you need to aggregate on the contents of your payload. See, since the content is variable, so are the data paths / accessors. But wildcards in aggregation paths don't work in Elasticsearch. Scripts do but are onerous to maintain.
Back to stripe. They partially solved it through what they call polymorphic, typed hashes — as discussed in their blog on API design:
A pretty neat approach that's worth emulating.
P.S. I discuss dynamic templates in more detail in the chapter "Mapping Automation" of my ES Handbook.
I have JSON document which is stored under single column of type jsonb inside postgresql which looks like below:
{
"resourceType": "Bundle",
"type": "transaction",
"entry": [
{
"fullUrl": "urn:uuid:100",
"resource": {
"resourceType": "Encounter",
"id": "110",
"status": "planned",
"priority": {
"coding": [
{
"code": "ASAP"
}
]
},
"subject": {
"reference": "Patient/123"
},
"appointment": [
{
"reference": "Appointment/12213#42"
}
],
"diagnosis": [
{
"condition": {
"reference": "Condition/condReferenceValue"
},
"use": {
"coding": [
{
"system": "http://terminology.hl7.org/CodeSystem/diagnosis-role",
"code": "AD"
},
{
"system": "http://terminology.hl7.org/CodeSystem/diagnosis-role",
"code": "DD"
}
]
}
}
],
"hospitalization": {
"preAdmissionIdentifier": {
"system": "https://system.html"
}
},
"location": [
{
"location": {
"display": "Mumbai"
},
"status": "active"
},
{
"status": "planned"
}
]
},
"request": {
"method": "POST",
"url": "Encounter"
}
}
]
}
Now, I want to update value for reference under subject attribute. So, I tried below way but it throws an error:
update fhir.testing set names = jsonb_set(names,'{"subject":{"reference"','"Patient/1"',true) where id = 10;
Error:
SQL Error [22P02]: ERROR: malformed array literal: "{"subject":{"reference""
Detail: Unexpected array element.
I referred this link but didn't work out for me. How can I do it?
I don't use Postgres that much but from what i read in the relative jsonb_set example in the documentation of JSON functions (and since you want to update) shouldn't it be
jsonb_set(names, '{entry,0,subject,reference}','Patient/1', false)
instead of
jsonb_set(names,'{"subject":{"reference"','"Patient/1"',true)
jsonb
jsonb_set(target jsonb, path text[], new_value jsonb [, create_missing
boolean])
Returns target with the section designated by path replaced by
new_value, or with new_value added if create_missing is true (default
is true) and the item designated by path does not exist. As with the
path oriented operators, negative integers that appear in path count
from the end of JSON arrays.
EDIT
To explain the path used in jsonb_set, check this example.
jsonb_set('[{"f1":1,"f2":null},2,null,3]', '{0,f1}','[2,3,4]', false)
returns
[{"f1":[2,3,4],"f2":null},2,null,3]
As i understand if a sub-element in a complex JSON document is an array, you need to specify it's index e.g. 0,1,2,...
EDIT
Always look very carefully the structure of the JSON document. I simply write this because i did not see that subject was a child of resource and that is causing you the error.
So the correct path is actually '{entry,0,resource,subject,reference}'
Correct Query for your requirement is:
update fhir.testing
set names= jsonb_set(names, '{entry,0,resource,subject,reference}', '"Patient/1"' , false)
where id = 10;
Explanation
json_set takes 4 parameter
target_json (jsonb) - which accept jsonb type data. In your case it is names field.
path (text[]) - which accepts a text array. in your case it is '{entry,0,resource,subject,reference}'.
new_value (jsonb) - in your case you want to change it to '"Patient/1"'.
create_missing (boolean) - in your case it should be false. as you want to replace the existing one. if you want to create the reference with given value in case of not found then just mark it true.
the value is not valid json, try this out:
update fhir.testing set names = jsonb_set(names, '{"entry": [{"resource": {"subject":{"reference":"Patient/1"} }}]}',true) where id = 10;
You have to create a valid json, closing every { and every ], yours was
'{"subject":{"reference"'
{
"metadata": {
"id": "2",
"uri": "3",
"type": "2"
},
"Number": "2323600002913",
"Date": "04/21/2009",
"postingDate": "00/00/0000",
"ata": {
"results": [
{
"metadata": {
"id": "r",
"uri": "e2",
"type": "s2"
},
"item": "000010",
"data":"ad"
}
]
}
}
want to remove metadata property from above json message and output should be like below
{
"Number": "2323600002913",
"Date": "04/21/2009",
"postingDate": "00/00/0000",
"ata": {
"results": [
{
"item": "000010",
"data":"ad"
}
]
}
}
I tried with removeProperty() which is working for root level metadata but inside metadata not removed.
how to use replace() in this case or anything else to only remove metadata.
The simplest way is use inline code, cause even with removeProperty() expression to remove the metadata under results, it will return the results array data not the whole json data. Then you will have to combine them, it's not a convenient way.
And with inline code you could refer to my below picture. The variable json is the value from triggerbody, then just delete the node or key and return the json variable. And with this way, even you want to delete many metadata in the array, you could add a for loop to delete it, just think of it as plain js code.
Update:if you want to get value from variable,cause no support expression to get value from variable so use the below expression.
var json =wworkflowContext.actions.Initialize_variable.inputs.variables[0].value;
And about how to loop the array in the json refer to my below pic.
We have a heavily nested json document containing server metrcs, the document contains > 1000 fields some of which are completely irrelevant to us for analytic purposes so i would like to remove them before indexing the document in Elastic.
However i am unable to find the correct filter to use as the fields i want to remove have common names in multiple different objects within the document.
The source document looks like this ( reduced in size for brevity)
[
{
"server": {
"is_master": true,
"name": "MYServer",
"id": 2111
},
"metrics": {
"Server": {
"time": {
"boundary": {},
"type": "TEXT",
"display_name": "Time",
"value": "2018-11-01 14:57:52"
}
},
"Mem_OldGen": {
"used": {
"boundary": {},
"display_name": "Used(mb)",
"value": 687
},
"committed": {
"boundary": {},
"display_name": "Committed(mb)",
"value": 7116
}
"cpu_count": {
"boundary": {},
"display_name": "Cores",
"value": 4
}
}
}
}
]
The data is loaded into logstash using the http_poller input plugin and needs to be processed before sending to Elastic for indexing.
I am trying to remove the fields that are not relevant for us to track for analytical purposes, these include the "display_name" and "boundary" fields from each json object in the different metrics.
I have tried using the mutate filter to remove the fields but because they exist in so many different objects it requires to many coded paths to be added to the logstash config.
I have also looked at the ruby filter, which seems promising as it can look the event, but i am unable to get it to crawl the entire json document, or more importantly actually remove the fields.
Here is what i was trying as a test
filter {
split{
field => "message"
}
ruby {
code => '
event.get("[metrics][Mem_OldGen][used]").to_hash.keys.each { |k|
logger.info("field is:", k)
if k.include?("display_name")
event.remove(k)
end
if k.include?("boundary")
event.remove(k)
end
}
'
}
}
It first splits the input at the message level to create one event per server, then tries to remove the fields from a specific metric.
Any help you be greatly appreciated.
If I get the point, you want to keep just the value key.
So, considering the response hash:
response = {
"server": {
"is_master": true,
"name": "MYServer",
"id": 2111
},
"metrics": {
...
You could do:
response[:metrics].transform_values { |hh| hh.transform_values { |h| h.delete_if { |k,v| k != :value } } }
#=> {:server=>{:is_master=>true, :name=>"MYServer", :id=>2111}, :metrics=>{:Server=>{:time=>{:value=>"2018-11-01 14:57:52"}}, :Mem_OldGen=>{:used=>{:value=>687}, :committed=>{:value=>7116}, :cpu_count=>{:value=>4}}}}
I am storing json data to one of the fields in a table and I am having trouble using REGEXP to return the correct entry
Basically, it matches other attributes in the JSON object, that it should not
Sample JSON
{
"data": {
"en": {
"containers": [
{
"id": 1441530944931,
"template": "12",
"columns": {
"column1": [
"144",
"145",
"148"
],
"column2":[
"135",
"148",
"234"
]
}
}
],
"left": "152",
"right": "151"
},
}
}
Now, I would like to search the columns array against a specific value (ie 148)
Right now I have the below query
WHERE (w.`_attrs` REGEXP '"column[0-9]":.*\\[.*"148".*\\]'
which works just fine
However, if I change the value from 148 to 152 or 151, it also works
For some reason the query matches the attribute left and right as well, but this is not desirable
Any help?
Thanks
Or... Switch to MariaDB 10 and index the components of the JSON.