After asked this question : topic restHeart with a satisfactory answer, i have a new question.
I used this http restHeart request :
PATCH http://test:8081/purge/test3 { rts: [
{
"name": "addRequestProperties",
"phase": "REQUEST",
"scope": "CHILDREN",
"args": { "log": [ "dateTime", "epochTimeStamp" ] } }]}
and now when I insert some json data, mongo db automaticaly add datetime and epochTimeStamp like that :
"invoiceNumber": "6666"
"log": {
"dateTime": "[23/Mar/2016:16:24:24 +0000]"
"epochTimeStamp": 1458750264
}
So my problem is now to make my query.
I tried something like that but does not work :
http://test:8081/purge/test3?filter={"log":{"epochTimeStamp":{"$lte":"1458750378"}}}
Finally my query retrieve nothing...
Version mongo 3.2 / restheart 1.2
Hope you can help me :)
You are passing a string to the $lte operator.
You need to pass a number:
http://test:8081/purge/test3?filter={"log.epochTimeStamp":{"$lte": 1458750378}}
Related
I am able to get a single JSON object in Kibana:
By having this in the filebeat.yml file:
output.elasticsearch:
hosts: ["localhost:9200"]
How can I get the individual elements in the JSON string. So say if I wanted to compare all the "pseudorange" fields of all my JSON objects. How would I:
Select "pseudorange" field from all my JSON messages to compare them.
Compare them visually in kibana. At the moment I can't even find the message let alone the individual fields in the visualisation tab...
I have heard of people using logstash to parse the string somehow but is there no way of doing this simply with filebeat? If there isn't then what do I do with logstash to help filter the individual fields in the json instead of have my message just one big json string that I cannot interact with?
I get the following output from output.console, note I am putting some information in <> to hide it:
"#timestamp": "2021-03-23T09:37:21.941Z",
"#metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.8.14",
"truncated": false
},
"message": "{\n\t\"Signal_data\" : \n\t{\n\t\t\"antenna type:\" : \"GPS\",\n\t\t\"frequency type:\" : \"GPS\",\n\t\t\"position x:\" : 0.0,\n\t\t\"position y:\" : 0.0,\n\t\t\"position z:\" : 0.0,\n\t\t\"pseudorange:\" : 20280317.359730639,\n\t\t\"pseudorange_error:\" : 0.0,\n\t\t\"pseudorange_rate:\" : -152.02620448094211,\n\t\t\"svid\" : 18\n\t}\n}\u0000",
"source": <ip address>,
"log": {
"source": {
"address": <ip address>
}
},
"input": {
"type": "udp"
},
"prospector": {
"type": "udp"
},
"beat": {
"name": <ip address>,
"hostname": "ip-<ip address>",
"version": "6.8.14"
},
"host": {
"name": "ip-<ip address>",
"os": {
<ubuntu info>
},
"id": <id>,
"containerized": false,
"architecture": "x86_64"
},
"meta": {
"cloud": {
<cloud info>
}
}
}
In Filebeat, you can leverage the decode_json_fields processor in order to decode a JSON string and add the decoded fields into the root obejct:
processors:
- decode_json_fields:
fields: ["message"]
process_array: false
max_depth: 2
target: ""
overwrite_keys: true
add_error_key: false
Credit to Val for this. His answer worked however as he suggested my JSON string had a \000 at the end which stops it being JSON and prevented the decode_json_fields processor from working as it should...
Upgrading to version 7.12 of Filebeat (also ensure version 7.12 of Elasticsearch and Kibana because mismatched versions between them can cause issues) allows us to use the script processor: https://www.elastic.co/guide/en/beats/filebeat/current/processor-script.html.
Credit to Val here again, this script removed the null terminator:
- script:
lang: javascript
id: trim
source: >
function process(event) {
event.Put("message", event.Get("message").trim());
}
After the null terminator was removed the decode_json_fields processor did its job as Val suggested and I was able to extract the individual elements of the JSON field which allowed Kibana visualisation to look at the elements I wanted!
I'm trying to write a JsonPath query that selects a specific object based on a condition, but either the syntax fails me or I fail the syntax.
Given the below Json object, how would I select the "Data" object containing Dirk Gently's details based on the fact that he uses the "Stumble" method?
{
"Investigators": [
{
"Type": "Legend",
"Object": {
"Method": "Investigate",
"Data": {
"Name": "Sherlock",
"Surname": "Holmes"
}
}
},
{
"Type": "Visionary",
"Object": {
"Method": "Stumble",
"Data": {
"Name": "Dirk",
"Surname": "Gently"
}
}
}
],
"Version": 1
}
I know that I can get to the Method-field like this:
$.Investigators..Object.Method
I assumed that something like this would work:
$.Investigators..Object[?(#.Method=="Stumble")].Data
I'm testing this using: https://jsonpath.com/ to evaluate the query - and I can't seem to get it right.
Am I trying to do something that is not achievable or am I just making a stupid mistake?
Go to this jsonpath online sandbox and try this experssion:
$.Investigators..Object[?(#.Method=="Stumble")].Data
using either the Jayway or Gatling implementations, the output will be:
[
{
"Name" : "Dirk",
"Surname" : "Gently"
}
]
I have to insert documents in my Mongo database by using Ruby (not on Rails, on Notepad++), many documents have duplicates with some modifications.
I want to write a script which use a json file, read it, import it in MongoDB by checking if each documents do not have any duplicate in the database, and if there is a duplicate I want to combine it if it contains any additional information:
Such as :
Document 1
{ "Name" : "Lila",
"Files":
[
{ "Name": "File1", "Date" : "05-11-2017"},
{ "Name": "File2", "Date" : "26-03-2018"}
]
}
Document 2
{ "Name" : "Lila",
"Files":
[
{ "Name": "File3", "Date" : "26-03-2018"}
]
}
Combine them to have:
{ "Name" : "Lila",
"Files":
[
{ "Name": "File1", "Date" : "05-11-2017"},
{ "Name": "File2", "Date" : "26-03-2018"},
{ "Name": "File3", "Date" : "26-03-2018"}
]
}
I founded that it is possible in mongo shell thanks to the aggregation-accumulation $mergeObjects, but in Ruby it do not seems to exists.
You can use all the operators in ruby, too. You need to get the underlying collection object first.
require 'mongo'
db = Mongo::Connection.new.db("mydb")
coll = db.collection('posts')
coll.aggregate([
{"$project" => {"last_name" => 1, "first_name" => 1 }},
{"$match" => {"last_name" => "Jones"}}
])
This is an example pipeline. You can give the same aggregation pipeline that worked for you on the mongo shell to aggregate.
For more information, refer to the MongoDB Ruby driver documentation:
http://www.rubydoc.info/gems/mongo/1.8.2/Mongo%2FCollection%3Aaggregate
Good day community, I am using LUIS to train a data set to let it classified between different meaning of the words. After I've done trained, I want to import a set of data to let it test. There is a batch testing options for me to import a json file, however it keeps showing this error:
BadArgument: Dataset object cannot be null. Parameter name: dataSet
I have already follow the json format that it gave which is like this:
[
{
"text": "hey dad, are you hungry?",
"intent": "None",
"entities":
[
{
"entity": "FamilyMember",
"startPos": 4,
"endPos": 6
}
]
},
{
.
.
.
}
]
My json file has the format like this:
[
{
"text" : "Hello"
"intent": "Greetings"
},
{
"text" : "I want bread"
"intent": "Request"
}
]
Can anyone tell me what am I doing wrong? The training doesn't include any entities so I did not put it into my json file.
Thank you.
You still need to provide the entities attribute and give it an empty array, otherwise you'll receive a different error. Regarding your format, you're missing commas after your text attributes.
[
{
"text" : "Hello",
"intent": "Greetings",
"entities": []
},
{
"text" : "I want bread",
"intent": "Request",
"entities": []
}
]
When I used the above code the batch test successfully completed for me.
I'm POST'ing the following JSON to asana's "tasks" endpoint.
{
"data": {
"options": {
"fields": [
"name",
"notes"
]
},
"workspace": <valid number>,
"assignee": <valid number>
}
}
It's giving me a "Invalid field" error every time. I've read through the API a few times now and this JOSN looks exactly how the API says it should. Any ideas?
Asana API for those of you who want to help out: Asana API Documentation
(I work for Asana)
The "options" field is a sibling of the "data" field, not a child. This is mentioned in the docs, but perhaps we aren't providing clarifying examples to make it more obvious.
If you change your request to look like this:
{
"options": {
"fields": [
"name",
"notes"
]
},
"data": {
"workspace": <valid number>,
"assignee": <valid number>
}
}
things should work.