How to fill/change particular unknown strings of some sections of a well formated JSON file from bash keeping the format intact? - json

How to fill/change particular unknown strings of some sections of a well formated .json file from bash keeping the format intact ?
Details:
Part of settings.json :
"profiles":
{
"defaults":
{
// Put settings here that you want to apply to all profiles.
},
"theme": "dark",
"list":
[
{
// Make changes here to the powershell.exe profile.
"guid": "{61c54bbd-c2c6-5271-96e7-009a87ff44bf}",
"name": "Windows PowerShell",
"commandline": "powershell.exe",
"hidden": false,
"acrylicOpacity": 0.9,
"colorScheme" : "Material",
"cursorColor" : "#FFFFFD",
"useAcrylic" : true
},
{
// Make changes here to the cmd.exe profile.
"guid": "{0caa0dad-35be-5f56-a8ff-afceeeaa6101}",
"name": "Command Prompt",
"commandline": "cmd.exe",
"hidden": false,
"acrylicOpacity": 0.2,
"colorScheme" : "Frost",
"cursorColor" : "#000000",
"useAcrylic": true,
},
"schemes": [
{
"name" : "Frost",
"background" : "#ffffff",
"black" : "#3C5712",
"blue" : "#17b2ff",
"brightBlack" : "#749B36",
"brightBlue" : "#27B2F6",
"brightCyan" : "#13A8C0",
"brightGreen" : "#89AF50",
"brightPurple" : "#F2A20A",
"brightRed" : "#F49B36",
"brightWhite" : "#741274",
"brightYellow" : "#991070",
"cyan" : "#3C96A6",
"foreground" : "#000000",
"green" : "#6AAE08",
"purple" : "#991070",
"red" : "#8D0C0C",
"white" : "#6E386E",
"yellow" : "#991070"
},
another file pickcolor.txt:
#FFFAFA
#FFFAFA
#EEE9E9
#FFC1C1
#EEB4B4
#CDC9C9
#F08080
#FF6A6A
#CD9B9B
#EE6363
#BC8F8F
#FF4040
#FF3030
#EE3B3B
#CD5C5C
#CD5555
#EE2C2C
#8B8989
#CD3333
#FF0000
#FF0000
#8B6969
#CD2626
#EE0000
#B22222
#A52A2A
#CD0000
#8B3A3A
#8B2323
Want to create a bash which changes value of fields like "background", "cursorColor" to any random value from pickcolor.txt
Requirements/problems I am facing:
Fields appearing many times gets different values from each other
Value of fields changes from time to time, so simple find and replace do not work
Fields changes there line number every time new contents are added, so line number for fields do not remain same.
Format of .json file should not change
Please note : I do not want anyone to write the complete code ; just hinting/pointing out ways will be sufficient. And thanks in advance !!

I believe what you need is something like this
fields=(background cursorColor)
colors=("#FFFAFA" "#CDC9C9" "#EE3B3B")
size=${#colors[#]}
for f in "${fields[#]}"; do
for line in $(grep -n "\"$f\"" myfile.json | cut -f1 -d:); do
index=$(($RANDOM % $size))
color="${colors[$index]}"
sed -i $line's|\("'"$f"'"\s*:\s*"\).\+"|\1'"$color"'"|' myfile.json
done
done
There is some magic in this solution:
We choose random index from array of colors
For each of fields which should be replaced we create it's own unique sed expression, which
First part (\("'"$f"'"\s*:\s*"\)#.\+" finds anything matching pattern "<field name>"<spaces>:<spaces>"<any color>"
Captures anything until 3rd double quote into group
In second group (\1'"${colors[$index]}"'") replaces matched part with itself, but inserts randomly picked color from array into second double quote group
Weird things with lots of quotes are made to interpolate field name and color into single quotes
NB: this is tested with GNU sed. May need a bit different syntax for BSD

Related

How to mass-replace text in "sounds" : ["test/test"]

I'm trying to make a resource pack in Minecraft, and I'm replacing it so there's only one sound. When I went to go and edit sounds.json in VSC, I want to set all the locations to just one file. It should look like this :
"sounds" : [
"test/test"
],
to test/test for all the "sounds". But I have no idea of how to do this. The sounds.json file is so big it would take more than a day to do all the work by hand. So I checked to see if VSC had any options to replace the text. There wasn't.
I've tried looking around in VSC and there wasn't anything useful.
I've tried replacing all the sounds by pasting .ogg files and renaming them, it took too long, so I realized I could just set all the locations to point at one sound file.
I've gone on Google to do some research but found nothing of use.
"block.enderchest.open": {
"sounds": [
"test/test"
],
"subtitle": "subtitles.block.chest.open"
},
"block.fence_gate.close": {
"sounds": [
"block/fence_gate/close1",
"block/fence_gate/close2"
],
"subtitle": "subtitles.block.fence_gate.toggle"
},
"block.fence_gate.open": {
"sounds": [
"block/fence_gate/open1",
"block/fence_gate/open2"
],
"subtitle": "subtitles.block.fence_gate.toggle"
},
"block.fire.ambient": {
"sounds": [
"fire/fire"
],
I expect a convenient way in order to edit "sounds" : [] []'s all at once.
The actual result is not really a convenient way and a time waster to edit all of the sounds [] values.
One way to do it is with regex, see regex101 demo.
Search for : ("sounds": \[\n)((\s*)[^\]]*\n)*(\s*\],)
Replace with: $1$3"test/test"\n$4

Elastic search, watcher access dotted field names in the result set

I created a query for a elastic search watcher setup. The result set looks like this:
"_index": "transaction_broker-2017.09.15",
"_type": "transaction_broker",
"_id": "AV6Fn_UQ9KbnKce40avY",
"_score": 3.8539968,
"_source": {
"tbroker.workitem.sync_check.tbroker_value": 7000,
"source": "/logs/web/tomcat/tbroker.log.json",
"type": "transaction_broker",
"tbroker.job.instance_id": "lixporta-p00.xxxxxxx.15053054001381505305457198",
"tbroker.workitem.sync_check.backend_total_value": 6995,
"tbroker.appversion": "1.1.135-180",
"#version": 1,
"beat": {
"hostname": "lixporta-p00",
"name": "lixporta-p00",
"version": "5.1.1"
In the action section, I can access the fields by using:
"actions": {
"my-logging-action": {
"logging": {
"text": "There are {{ctx.payload.hits.hits.0._source.....
After the source tag, I use for example the "type" field from the list above. Other example is:
"ctx.payload.hits.hits.0._source.beat.hostname"
This works pretty fine...
But it is not possible to use a field like
"tbroker.workitem.sync_check.tbroker_value"
The parser thinks that this fields are nested, but this is only a fieldname with dots in it.
Is there any possiblity to "escape" this fieldname?
Anyone who also have had this problem ?
Many thanks & best regards
Claus
I think the following should work:
{{#ctx.payload.hits.hits.0._source}}{{tbroker.workitem.sync_check.tbroker_value}}{{/ctx.payload.hits.hits.0._source}}
It is a limitation of Mustache and this is a workaround.
Another example may help - when in a context looping through hits (I have added // comments purely for clarity - they aren't valid Mustache syntax & should be removed):
{{#ctx.payload.hits.hits}}
// This works fine
{{_source.foo}}
// Not working if each hit's _source contains "bar.baz", not nested "bar">"baz"
{{_source.bar.baz}}
{{/ctx.payload.hits.hits}}
Applying the same workaround by adding an extra context/section:
{{#ctx.payload.hits.hits}}
// Put us in the context of [the current hit] > _source
{{#_source}}
// Now both of these work...
{{foo}}
// ...including this one containing a dot (yay!)
{{bar.baz}}
{{/_source}}
{{/ctx.payload.hits.hits}}
There is no way to directly access source fields that have dots in them, but if you apply a transform like this:
"transform": {
"script": {
"inline": "return [ 'host' : ctx.payload.hits.hits[0]._source.host, 'tbroker_value' : ctx.payload.hits.hits[0]._source['tbroker.workitem.sync_check.tbroker_value']]",
"lang": "painless"
}
}
and then you can use {{ctx.payload.host}} and {{ctx.payload.tbroker_value}} in your action.

MongoDB queries return no results

I'm having a problem with querying a MongoDB dataset ("On Street Crime in Camden" from data.gov.uk)
The database name is Crime_Data_in_Camden and the collection name is Street_Crime_Camden. The query to find all records, db.Street_Crime_Camden.find(), works fine but anything else returns nothing at
all. Here is a portion of the metadata:
{
"id" : 509935,
"name" : "Ward Name",
"dataTypeName" : "text",
"fieldName" : "ward_name",
"position" : 13,
"renderTypeName" : "text",
"tableColumnId" : 258836,
"width" : 100,
"cachedContents" : {
"largest" : "West Hampstead",
"non_null" : 79813,
"null" : 0,
"top" : [ {
"item" : "Regent's Park",
"count" : 20
}, {
"item" : "Swiss Cottage",
"count" : 19
}, {
"item" : "Holborn and Covent Garden",
"count" : 18
}
}
}
I've tried 3 attempts at a basic query:
db.Street_Crime_Camden.find({"ward_name":"West Hampstead"});
db.Street_Crime_Camden.find({'meta.ward_name':'West Hampstead'});
db.Street_Crime_Camden.find({meta:{ward_name:"West Hampstead"} });
According to any documentation or tutorial that I've seen any of these approaches should be valid. And I know that there are hundreds of rows (or documents) that match those terms, so why are these queries returning nothing? Advice would be appreciated.
The common theme in the three aproaches you tried is some form of ward_name = West Hampstead but there is no attribute named ward_name in the document you shared with us.
Based on the document you show in your question the only way of addressing an attribute with the value West Hampstead is:
db.Street_Crime_Camden.find({"cachedContents.largest": "West Hampstead"});
For background; you address attributes in your documents by using dot notation so the document you included in your question could be found by any of the following find commands:
db.Street_Crime_Camden.find({"name": "Ward Name"});
db.Street_Crime_Camden.find({"position": 13});
db.Street_Crime_Camden.find({"cachedContents.top.item": "Swiss Cottage"});
db.Street_Crime_Camden.find({"cachedContents.top.1.count": 20});
... etc
These examples might help you to understand how to form find criteria. The MongoDB docs are also useful.

Using logical operators in json path file

Is there a functionality to use logical operators in JSON path file used in copy command.
For example, I have a JSON which can contain a key which can either be
Desc
Or
Description
So in the JSON it would be something like -
{
"Desc": "Hello",
"City" : "City1",
"Age": "21"
}
{
"Description" : "World",
"City" : "City2",
"Age": "25"
}
I'm using copy command to pull the data from the JSON above into my table in redshift. The table has a column named "description_data". This would store values of either "Desc" or "Description". So I want my path file to identify using an "OR" condition.
This is the path file that I'm currently using -
{
"jsonpaths": [
"$['Desc']",
"$['City']",
"$['Age']"
]
}
Which is working fine.
What I'm trying to do is the below (this is where I'm unsure if there is any syntax or functionality to achieve the objective)
{
"jsonpaths": [
"$['Desc']" or "$['Description']",
"$['City']",
"$['Age']"
]
}
No, Redshift doesn't support this.
You can issue two copy commands, one with Desc, and another with Description, to load the data into two temporary tables. After that, you can merge the two into your final table.

How to display 'c' array values alone from the given JSON document below using MongoDB?

I am a newbie to MongoDB. I am experimenting the various ways of extracting fields from a document inside collection.
Here in the below JSON document, I am finding it difficult to get extract it according to my need
{
"_id":1,
"dependencies":{
"a":[
"hello",
"hi"
],
"b":[
"Hmmm"
],
"c":[
"Vanilla",
"Strawberry",
"Pista"
],
"d":[
"Carrot",
"Cauliflower",
"Potato",
"Cabbage"
]
},
"productid":"25",
"date":"Thu Jul 30 11:36:49 PDT 2015"
}
I need to display the following output:
c:[
"Vanilla",
"Strawberry",
"Pista"
]
Can anyone please help me in solving it?
MongoDB Aggregation comes into rescue to get the result you are looking for :
$Project--> Passes along the documents with only the specified fields to the next stage in the pipeline. The specified fields can be existing fields from the input documents or newly computed fields.
db.collection.aggregate( [
{ $project :
{ c: "$dependencies.c", _id : 0 }
}
]).pretty();
As per the output you required, we just need to project ( display) the field "dependencies.c" , so we are creating a new field "c" and assigining the value of the "dependencies.c" into it.
Also by defalut "_id" field will be display along with the result, since you dont need it, so we are suppressing of the _id field by assigining "_id" : <0 or false>, so that it will not display the _id field in the output.
The above query will fetch you the result as below :
"c" : [
"Vanilla",
"Strawberry",
"Pista"
]