I have tried Elasticsearch 2.3.2 and Kibana Version 4.5.0 to store big Json object containing a lot of info of Windows OS, an example of json type is the followed:
{
"_index": "siem",
"_type": "rmd-siem",
"_id": "2b1fc911-85f6-48f3-a7ba-b9e5f6e8b38f",
"_score": null,
"_source": {
"timestamp": "2016-05-08T15:00:37.000Z",
"BIOS": {
"CURRENTLANGUAGE": "en|US|iso8859-1",
"BIOSCHARACTERISTICS": "{4;7;9;11;12;15;16;19;21;22;23;24;27;28;30;32;33;40;41;42;48;56;57;64;65;66;67;68;69;70;71;72;73;74;75;76;77;78;79}",
"NAME": "xxxxx",
"SERIALNUMBER": "xxxxx",
"DESCRIPTION": "xxxcv",
"SMBIOSMINORVERSION": 6,
"INSTALLABLELANGUAGES": 1,
"RELEASEDATE": "24-May-2010 12:00:00 AM",
"STATUS": "OK",
"LISTOFLANGUAGES": "{en|US|iso8859-1}",
"HOST": "GOLFINHO_OSM1",
"SMBIOSPRESENT": "TRUE",
"PRIMARYBIOS": "TRUE",
"VERSION": "DELL - 1",
"TARGETOPERATINGSYSTEM": 0,
"SMBIOSBIOSVERSION": "1.3.6",
"SOFTWAREELEMENTID": "xxxccv",
"SMBIOSMAJORVERSION": 2,
"SOFTWAREELEMENTSTATE": 3,
"TIMESTAMP": "2016-05-08 15:00:37",
"MANUFACTURER": "Dell Inc."
}
},
"fields": {
"timestamp": [
1462719637000
]
},
"sort": [
1462719637000
]
}
I have other big json that they are json array containing json object with many data. So I have few object but with very info.
While I tryed Kibana to show the info I saw that the load data for a few hits (results) is very slow in loading phase.
I think that the issue could be to amount of data (a lot info inside json object) because from the test performed I have seen about 22 seconds to receive data and to render the data inside Kibana others 40 seconds for total of 277 hits.
I ask you if the issue is the big json or there is another reason. Thanks
Related
I am looking for a way to extract data out of an azure environment. Problem i'm currently having is that when I use my API call I receive about 60 lines of json while I only need 4 of those lines. To reduce load, increase efficiency and remove the need for parsing withing the other environment where I need to data, I want to find a way to filter the data in the api call. Currently my call looks like this.
https://management.azure.com/subscriptions/{subscription}/resourceGroups/{resourcegroup}/providers/Microsoft.Web/sites/{application or resource}/providers/microsoft.insights/metrics?api-version=2021-05-01&metricnames=IoWriteBytesPerSecond,IoReadBytesPerSecond&timeSpan=PT1M
now the ouput looks something like this.
{
"cost": 0,
"timespan": "2022-10-11T10:18:00Z/2022-10-11T10:19:00Z",
"interval": "PT1M",
"value": [
{
"id": "/subscriptions//resourceGroups//providers/Microsoft.Web/sites//providers/Microsoft.Insights/metrics/IoWriteBytesPerSecond",
"type": "Microsoft.Insights/metrics",
"name": {
"value": "IoWriteBytesPerSecond",
"localizedValue": "IO Write Bytes Per Second"
},
"displayDescription": "The rate at which the app process is writing bytes to I/O operations. For WebApps and FunctionApps.",
"unit": "BytesPerSecond",
"timeseries": [
{
"metadatavalues": [],
"data": [
{
"timeStamp": "2022-10-11T10:18:00Z",
"total": 288.0
}
]
}
],
"errorCode": "Success"
},
{
"id": "/subscriptions//resourceGroups//providers/Microsoft.Web/sites//providers/Microsoft.Insights/metrics/IoReadBytesPerSecond",
"type": "Microsoft.Insights/metrics",
"name": {
"value": "IoReadBytesPerSecond",
"localizedValue": "IO Read Bytes Per Second"
},
"displayDescription": "The rate at which the app process is reading bytes from I/O operations. For WebApps and FunctionApps.",
"unit": "BytesPerSecond",
"timeseries": [
{
"metadatavalues": [],
"data": [
{
"timeStamp": "2022-10-11T10:18:00Z",
"total": 284.0
}
]
}
],
"errorCode": "Success"
}
],
"namespace": "Microsoft.Web/sites",
"resourceregion": "westeurope"
}
Out of all these lines I only need about 4 objects, Is it possible to use the $filter function within the URL api call? If yes, can someone redirect me to a forum, doc or example where this is used?
Thanks, regards
I am trying to adapt an event calendar vuejs module called dayspan calendar. Current entry object for an event as json is a bit strange and I want to balance the parsing of the payload before the post request and handling of data in DRF serializers. So I can get an optimum and performant client-server rest API communication. Json output is as below before any parsing:
{
"data": {
"title": "YOK",
"description": "",
"location": "",
"color": "#1976d2",
"forecolor": "#ffffff",
"calendar": "",
"busy": true,
"icon": ""
},
"schedule": {
"times": [
"17"
],
"dayOfMonth": [
11
],
"year": [
2021
],
"month": [
5
]
}
}
There are more schedule fields like "dayOFWeek", "duration" etc. that is to be expected for different entries.
What would be the best approach in terms of parsing json payload before posting to DRF and in deserializing stage before saving into database? I appreciate any ideas.
I have been studying official documentation of JMESPath and a few other resources. However I was not successful with the following task:
my data structure is a json from vimeo api (video list):
data array contains lots of objects, each object is the uploaded file that has many attributes and various options.
"data": [
{
"uri": "/videos/00001",
"name": "Video will be added.mp4",
"description": null,
"type": "video",
"link": "https://vimeo.com/00001",
"duration": 9,
"files":[
{
"quality": "hd",
"type": "video/mp4",
"width": 1440,
"height": 1440,
"link": "https://player.vimeo.com/external/4443333.sd.mp4",
"created_time": "2020-09-01T19:10:01+00:00",
"fps": 30,
"size": 10807854,
"md5": "643d9f18e0a63e0630da4ad85eecc7cb",
"public_name": "UHD 1440p",
"size_short": "10.31MB"
},
{
"quality": "sd",
"type": "video/mp4",
"width": 540,
"height": 540,
"link": "https://player.vimeo.com/external/44444444.sd.mp4",
"created_time": "2020-09-01T19:10:01+00:00",
"fps": 30,
"size": 1345793,
"md5": "cb568939bb7b276eb468d9474c1f63f6",
"public_name": "SD 540p",
"size_short": "1.28MB"
},
... other data
]
},
... other uploaded files
]
Filter I need to apply is that duration needs to be less than 10 and width of file needs to be 540 and the result needs to contain a link (url) from files
I have managed to get only one of structure-levels working:
data[].files[?width == '540'].link
I need to extract this kind of list
[
{
"uri": "/videos/111111",
"link": "https://player.vimeo.com/external/4123112312.sd.mp4"
},
{
"uri": "/videos/22222",
"link": "https://player.vimeo.com/external/1231231231.sd.mp4"
},
...other data
]
Since the duration is in your data array, you will have to add this filter at that level.
You will also have to use what is described under the section filtering and selecting nested data because you only care of one specific type of file under the files array, so, you can use the same type of query structure | [0] in order to pull only the first element of the filtered files array.
So on your reduced exemple, the query:
data[?duration < `10`].{ uri: uri, link: files[?width == `540`].link | [0] }
Would yield the expected:
[
{
"uri": "/videos/00001",
"link": "https://player.vimeo.com/external/44444444.sd.mp4"
}
]
I'm trying to use some results exported in JSON of a script called "Mixed Content Scan" (it's a script in order to search on a website if there is some mixed HTTP/HTTPS content and if all your pages are ok in HTTPS).
I'm a beginner with JSON, I read and watched a lot of tutorials in order to understand how to structure JSON data but I'm stumbling on something.
Here is a sample of my data (first 3 lines) :
{"message":"Scanning https://mywebsite.com/","context":[],"level":250,"level_name":"NOTICE","channel":"MCS","datetime":{"date":"2018-10-05 23:48:50.268196","timezone_type":3,"timezone":"America/New_York"},"extra":[]}
{"message":"00000 - https://mywebsite.com/","context":[],"level":400,"level_name":"ERROR","channel":"MCS","datetime":{"date":"2018-10-05 23:48:50.760948","timezone_type":3,"timezone":"America/New_York"},"extra":[]}
{"message":"http://mywebsite.com/wp-content/uploads/2015/03/image.jpg","context":[],"level":300,"level_name":"WARNING","channel":"MCS","datetime":{"date":"2018-10-05 23:48:50.761082","timezone_type":3,"timezone":"America/New_York"},"extra":[]}
I know I need to wrap my data around some {} or [] (tried both), but I think I'm missing something, for example, every JSON data validator websites are telling me that I have an error between 2 lines when I add a "," when I try to have multiple results into it.
How can I upgrade this raw data in order for a JSON validator to validate it?
Thanks!
How's this
[{
"message": "Scanning https://mywebsite.com/",
"context": [],
"level": 250,
"level_name": "NOTICE",
"channel": "MCS",
"datetime": {
"date": "2018-10-05 23:48:50.268196",
"timezone_type": 3,
"timezone": "America/New_York"
},
"extra": []
}, {
"message": "00000 - https://mywebsite.com/",
"context": [],
"level": 400,
"level_name": "ERROR",
"channel": "MCS",
"datetime": {
"date": "2018-10-05 23:48:50.760948",
"timezone_type": 3,
"timezone": "America/New_York"
},
"extra": []
}, {
"message": "http://mywebsite.com/wp-content/uploads/2015/03/image.jpg",
"context": [],
"level": 300,
"level_name": "WARNING",
"channel": "MCS",
"datetime": {
"date": "2018-10-05 23:48:50.761082",
"timezone_type": 3,
"timezone": "America/New_York"
},
"extra": []
}]
Entries in an array need to be separated by commas.
I am trying to create a domain and uploading a sample data which is like :
[
{
"type": "add",
"id": "1371964",
"version": 1,
"lang": "eng",
"fields": {
"id": "1371964",
"uid": "1200983280",
"time": "2013-12-23 13:00:26",
"orderid": "1200983280",
"callerid": "66580662",
"is_called": "1",
"is_synced": "1",
"is_sent": "1",
"allcaller": [
{
"sno": "1085770",
"uid": "1387783883.30547",
"lastfun": null,
"callduration": "00:00:46",
"request_id": "1371964"
}
]
}
}]
when I am uploading sample data while creating a domain, cloudsearch is not taking it.
If I remove allcaller array then it takes it smoothly.
If cloudsearch does not allowing object arrays, then how should I format this json??
Just found after searching on aws forums, cloudsearch doesnot allow nested json (object arrays) :(
https://forums.aws.amazon.com/thread.jspa?messageID=405879񣅷
Time to try Elastic search.