I have a dynamically created json file.
I want to know only the "errors" part of "status". Is there any easy way to do so?
...
a lot of lines
...
"status": {
"errors": [
{
"message": "Input contained no data",
"reason": "invalid"
}
],
"state": "DONE"
}
...
a lot of lines
...
I need to use the output in a shell script so awk is preferred.
This might work for you:
sed '/^ "status": {/,/^ }$/!d;/^ "errors": \[/,/^ \],/!d' file.txt
I would use sed to select a range of lines like this:
sed -e '/^ "errors\": \[/,/^ ],/!d' file.txt
result:
"errors": [
{
"message": "Input contained no data",
"reason": "invalid"
}
],
Consider adding expected results if I have misunderstood your question.
HTH
Related
I have some input, which you may recognize as a JSON stream, which I need to process from a simple shell script
I've opted to use sed, because half a dozen other manipulations I need to make are already working great with sed.
Below is the scenario.
input1:
{
"Expr": {
"Criteria": [
{
"Values": [
{
"displayValue": "MyObject",
"value": "UUID"
}
]
}
],
"Type": "SELECTED"
}
}
Intended result 1 (print the clause[s] containing displayValue + value pairs}:
{
"displayValue": "MyObject",
"value": "UUID"
}
Attempt:
$ sed -n -e '/"Expr":/,/"Type": "SELECTED"/ {/"Values":/,/^ }$/ {/^ {/,/^ }/p;};}' /tmp/input1.json
Actual result (good):
{
"displayValue": "MyObject",
"value": "UUID"
}
Problem is with this input2:
{
"Expr": {
"Criteria": [
{
"Values": [
{
"displayValue": "MyObject",
"value": "UUID"
}
]
}
],
"Type": "NOT_SELECTED"
}
}
Result is unexpected (since outer nest is NOT_SELECTED, do not want match unless SELECTED):
$ sed -n -e '/"Expr":/,/"Type": "SELECTED"/ {/"Values":/,/^ }$/ {/^ {/,/^ }/p;};}' /tmp/input2.json
Result 2:
{
"displayValue": "MyObject",
"value": "UUID"
}
Is it me, or is this a sed bug on my platform?
Otherwise, can you suggest a good approach to this problem?
The constraint is I do not want to print "Criteria" lines that are not within the "Type" is "SELECTED" range.
This might work for you (GNU sed):
sed -En '/"Expr"/{:a;n;H;/"Values"/h;/"Type"/!ba;/"SELECTED"/!d;g;s/.*\n((\s*)\{.*\2\}).*/\1/p}' file
Use a loop rather than a range.
Form a loop between "Expr" and "Type" and only proceed if the quoted word "SELECTED" is present.
While executing the loop save Values in the hold space and then manipulate the saved content to just show the parts needed.
N.B. This is a filtering operation so use the -n option.
Remember that sed does not look ahead in the file. So, multiline techniques must be used for this job. My approach is to "slurp" lines between toplevel { and }, which I assume they consist of a single character ({ or }, and no leading and trailing spaces), and examine the resultant pattern space:
sed '
/^{$/,/^}$/{
//!{H;d;}
g
s/.*"Values":.*\(\n[[:space:]]*{[^}]*"displayValue":[^}]*"value":[^}]*}\).*"Type":[[:space:]]"SELECTED".*/\1/p
s/.*//
h
d
}' file
I have a templatized json file called template.json as below:
{
"subject": "Some subject line",
"content": $CONTENT,
}
I have another file called sample.json with the json content as below:
{
"status": "ACTIVE",
"id": 217,
"type": "TEXT",
"name": "string",
"subject": "string",
"url": "contenttemplates/217",
"content": {
"text": "hello ${user_name}",
"variables": [{
"key": "${user_name}",
"value": null
}]
},
"content_footer": null,
"audit": {
"creator": "1000",
"timestamp": 1548613800000,
"product": "2",
"channel": "10",
"party": null,
"event": {
"type": null,
"type_id": "0",
"txn_id": "0"
},
"client_key": "pk6781gsfr5"
}
}
I want to replace $CONTENT from template.json with the content under the tag "content" from the content.json file . I have tried with below sed commands:
sed -i 's/$CONTENT/'$(jq -c '.content' sample.json)'/' template.json
I am getting below error:
sed: -e expression #1, char 15: unterminated `s' command
Can someone please help me to get the right sed command (or any other alternative)?
The jq Cookbook has a section on using jq with templates: https://github.com/stedolan/jq/wiki/Cookbook#using-jq-as-a-template-engine
In the present case, the first technique ("Using jq variables as template variables") matches the already-defined template file (except for the dangling comma), so you could for example write:
jq -n --arg CONTENT "$(jq -c .content sample.json)" '
{"subject": "Some subject line", "content": $CONTENT}'
or use the format:
jq -n --arg CONTENT "$(jq -c .content sample.json)" -f template.jq
(I'd only use the .json suffix for files that hold JSON or JSON streams.)
The output from jq contains spaces, you need to quote them to prevent the shell from tokenizing them.
sed -i 's/$CONTENT/'"$(jq -c '.content' sample.json)/" template.json
See further When to wrap quotes around a shell variable?
With GNU sed:
sed '/$CONTENT/{s/.*/jq -c ".content" sample.json/e}'
Replace the entire line with your command and e (GNU only) to execute the command and replace sed's pattern space with the output of the command.
I have a file:
{
"test_data": [
{
"id": "1",
"pm": "30",
"mp": "40"
}
],
"test": [
"id",
"pm",
"mp"
]
}
I want to extract test_data. Output:
"test_data": [
{
"id": "1",
"pm": "30",
"mp": "40"
}
],
I try this command: cat myFile | sed -n '/^"test_data": \[$/,/^\],$/p'
But it's don't work. An idea ?
Thanks you !
jq seems the right tool for the job :
$ jq '.|{test_data:.test_data}' filename
{
"test_data": [
{
"id": "1",
"pm": "30",
"mp": "40"
}
]
}
Solution 1st: With sed
sed -n '/"test_data"/,/],/p' Input_file
OR: as per OP, OP needs to append a string/data after a line matches:
sed -n '/"test_data"/,/],/p;/],/s/$/"test"/p' Input_file
OR2: If one wants to add an another file's content to a match then following may help in same:
sed -n '/"test_data"/,/],/p;/],/r another_file' Input_file
Solution 2nd: Following simple awk may help you in same.
awk '/test_data/, /],/' Input_file
Output will be as follows.
"test_data": [
{
"id": "1",
"pm": "30",
"mp": "40"
}
],
Logic for above solutions:
For sed: -n option in sed will turn OFF the printing of any line till it is explicitly mentioned to print it, then by doing /"test_data"/,/],/ I am letting sed know that I need to get the data from test_data to till ,/] and mentioning p after that will make sure those lines which are coming in this range are getting printed here/
For awk: Simply mentioning the range from /"test_data"/,/],/ and not mentioning any action so when any line comes into this range condition becomes true and since no action mentioned so by default print of that line happens then.
You can try that with gnu
csplit -s -z infile %test_data%-1 /],/1;rm xx01;echo "Add Text here" >> xx00;cat xx00
The right way is jq tool:
jq 'del(.test)' inputfile
The output:
{
"test_data": [
{
"id": "1",
"pm": "30",
"mp": "40"
}
]
}
I run a command on one of my systems and it spits out JSON "pretty" format like so:
[
{
"server": "servename1",
"i.p": 127.0.0.1,
"domain": "generic",
"OS": "RHEL",
"Version": 7.0
},
{
"server": "servename2",
"i.p": 127.0.0.1,
"domain": "generic",
"OS": "RHEL",
"Version": 7.0
},
{
"server": "servename3",
"i.p": 127.0.0.1,
"domain": "generic",
"OS": "RHEL",
"Version": 7.0
}
]
I need to parse these paragraphs in one liners like so:
[{"server":"servename1","i.p":127.0.0.1,"domain":"generic","OS":"RHEL","Version":7.0},
{"server":"servename2","i.p":127.0.0.1,"domain":"generic","OS":"RHEL","Version":7.0},
{"server":"servename3","i.p":127.0.0.1,"domain":"generic","OS":"RHEL","Version":7.0},]
What is the easiest way to do this? I am tried using SED and JQ but couldn't get it.
You can try this in jq if you're OK with newline-delimited JSON:
$ jq -c ".[]" test.json
{"server":"servename1","i.p":"127.0.0.1","domain":"generic","OS":"RHEL","Version":7}
{"server":"servename2","i.p":"127.0.0.1","domain":"generic","OS":"RHEL","Version":7}
{"server":"servename3","i.p":"127.0.0.1","domain":"generic","OS":"RHEL","Version":7}
Note that I had to quote the IP addresses from your sample, since the JSON you posted is not valid JSON.
If your input is always that regular:
$ awk '{ORS=(/},|\]/?RS:""); gsub(/[[:blank:]]+/,""); sub(/}$/,"},")}1' file
[{"server":"servename1","i.p":127.0.0.1,"domain":"generic","OS":"RHEL","Version":7.0},
{"server":"servename2","i.p":127.0.0.1,"domain":"generic","OS":"RHEL","Version":7.0},
{"server":"servename3","i.p":127.0.0.1,"domain":"generic","OS":"RHEL","Version":7.0},]
If that doesn't work for your real input then edit your question to include more truly representative sample input.
To skip the [ and ] lines:
$ awk '!/^[][]/{ORS=(/},/?RS:""); gsub(/[[:blank:]]+/,""); sub(/}$/,"},\n"); print}' file
{"server":"servename1","i.p":127.0.0.1,"domain":"generic","OS":"RHEL","Version":7.0},
{"server":"servename2","i.p":127.0.0.1,"domain":"generic","OS":"RHEL","Version":7.0},
{"server":"servename3","i.p":127.0.0.1,"domain":"generic","OS":"RHEL","Version":7.0},
I have the following JSON file
I have used awk to get rid of empty spaces, trailing, next line
awk -v ORS= -v OFS= '{$1=$1}1' data.json
I have added a create request at the top of my data.json followed by \n and the rest of my data.
{"create": {"_index":"socteam", "_type":"products"}}
When I issue bulk submit request, I get the following error
CURL -XPUT http://localhost:9200/_bulk
{
"took": 1,
"errors": true,
"items": [
{
"create": {
"_index": "socteam",
"_type": "products",
"_id": "AVQuGPff-1Y7OIPIJaLX",
"status": 400,
"error": {
"type": "mapper_parsing_exception",
"reason": "failed to parse",
"caused_by": {
"type": "not_x_content_exception",
"reason": "Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"
}
}
}
}
]
Any idea on what this error mean? I haven't created any mapping, I'm using vanilla elasticsearch
Accordingly to this doc, you have to specify index and type in URL:
curl -XPUT 'localhost:9200/socteam/products/_bulk?pretty' --data-binary "#data.json"
It works for PUT and POST methods.
And your data.json file should have structure like:
{"index":{"_id":"1"}}
{"name": "John Doe" }
{"index":{"_id":"2"}}
{"name": "Jane Doe" }
Maybe there present another method to import data, but i know just this... Hope it'll help...