I've seen a number of posts on this but can't figure out what I need exactly. I've tried -r and argjson among other things.
I need the newlines to remain as \n and not be escaped to \\n.
I'd also like to be able to use ``` for code blocks but it ignores that section of the string.
FALLBACK_MESSAGE="TEST MESSAGE - $HOSTNAME"
MARKDOWN_MESSAGE="TEST MESSAGE - $HOSTNAME \(0x0a) \(\n) Hi <#U12345789>\n```Can we do a\nmultiline code block```"
JSON_STRING=$( jq -nr \
--arg fallbackMessage "$FALLBACK_MESSAGE" \
--arg sectionType "section" \
--arg markdownType "mrkdwn" \
--arg textMessage "$MARKDOWN_MESSAGE" \
'{
text: $fallbackMessage,
blocks: [
{
type: $sectionType,
text: {
type: $markdownType,
text: $textMessage
}
}
]
}')
echo $JSON_STRING
Outputs:
{ "text": "TEST MESSAGE - devDebug", "blocks": [ { "type": "section", "text": { "type": "mrkdwn", "text": "TEST MESSAGE - devDebug \\(0x0a) \\(\\n) Hi <#U12345789>\\n" } } ] }
Make sure your shell variables contain actual newlines, not \n sequences.
If you want bash to convert escape sequences in a string into the characters they refer to, printf %b can be used for this purpose.
#!/usr/bin/env bash
fallback_message="TEST MESSAGE - $HOSTNAME"
markdown_message="TEST MESSAGE - $HOSTNAME \(0x0a) \(\n) Hi <#U12345789>\n\`\`\`Can we do a\nmultiline code block\`\`\`"
# create markdown_message_unescaped with an unescaped version of markdown_message
printf -v markdown_message_unescaped %b "$markdown_message"
jq -n \
--arg textMessage "$markdown_message_unescaped" \
--arg fallbackMessage "$fallback_message" \
--arg sectionType section --arg markdownType markdown '
{
text: $fallbackMessage,
blocks: [
{
type: $sectionType,
text: {
type: $markdownType,
text: $textMessage
}
}
]
}'
...properly emits as output:
{
"text": "TEST MESSAGE - YOUR_HOSTNAME",
"blocks": [
{
"type": "section",
"text": {
"type": "markdown",
"text": "TEST MESSAGE - YOUR_HOSTNAME (0x0a) (\n)\nHi <#U12345789>\n```\nCan we do a multiline code block\n```"
}
}
]
}
Related
I have a json file with this data:
{
"data": [
{
"name": "table",
"values": [
"This is old data",
"that needs to be",
"replaced."
]
}
]
}
But my challege here is I need to replace that values array with words in a text or csv file:
this
this
this
is
is
an
an
array
My output needs to have (although I could probably get away with the words all on one line...):
"values": [
"this this this",
"is is",
"an an",
"array"
],
Is this possible with only jq? Or would I have to get awk to help out?
I already started down the awk road with:
awk -F, 'BEGIN{ORS=" "; {print "["}} {print $2} END{{print "]"}}' filename
But I know there is still some work here...
And then I came across jq -Rn inputs. But I haven't figured out how or if I can get the desired result.
Thanks for any pointers.
Assuming you have a raw ASCII text file named file and an input JSON file, you could do
jq --rawfile txt file '.data[].values |= ( $txt | split("\n")[:-1] | group_by(.) | map(join(" ")) )' json
produces
{
"data": [
{
"name": "table",
"values": [
"an an",
"array",
"is is",
"this this this"
]
}
]
}
You can use jq and awk.
Given:
$ cat file
{
"data": [
{
"name": "table",
"values": [
"This is old data",
"that needs to be",
"replaced."
]
}
]
}
$ cat replacement
this
this
this
is
is
an
an
array
First create a string for the replacement array (awk is easy to use here):
ins=$(awk '!s {s=last=$1; next}
$1==last{s=s " " $1; next}
{print s; s=last=$1}
END{print s}' replacement | tr '\n' '\t')
Then use jq to insert into the JSON:
jq --rawfile txt <(echo "$ins") '.data[].values |= ( $txt | split("\t")[:-1] )' file
{
"data": [
{
"name": "table",
"values": [
"this this this",
"is is",
"an an",
"array"
]
}
]
}
You can also use ruby to process both files:
ruby -r json -e '
BEGIN{ ar=File.readlines(ARGV[0])
.map{|l| l.rstrip}
.group_by{|e| e}
.values
.map{|v| v.join(" ")}
j=JSON.parse(File.read(ARGV[1]))
}
j["data"][0]["values"]=ar
puts JSON.pretty_generate(j)' txt file
# same output...
I want to create JSON output with jq that looks like this:
{
"records": [
{
"id": "1234",
"song": "Yesterday",
"artist": "The Beatles"
}
]
}
I assumed I have to twiddle with the "filter" of jq whose concept I don't fully get after reading the doc.
This is what I got so far:
$ jq --arg id 1234 \
--arg song Yesterday \
--arg artist "The Beatles" \
'.' \
<<<'{ "records" : [{ "id":"$id", "song":"$song", "artist":"$artist" }] }'
which prints
{
"records": [
{
"id" : "$id",
"song" : "$song",
"artist" : "$artist"
}
]
}
Do I modify the filter? Do I change the input?
An alternate way to your original attempt, on jq-1.6 you can use the $ARGS.positional attribute to construct your JSON from scratch
jq -n '
$ARGS.positional | {
records: [
{
id: .[0],
song: .[1],
artist: .[2]
}
]
}' --args 1234 Yesterday "The Beatles"
As for why your original attempt didn't work, looks you are not modifying your json at all, with your filter '.' you are basically just reading in and printing out "untouched". The arguments set using --arg need to be set to the object inside the filter.
You are looking for something like this:
jq --null-input \
--arg id 1234 \
--arg song Yesterday \
--arg artist "The Beatles" \
'.records[0] = {$id, $song, $artist}'
Each variable reference between curly brackets is converted to a key-value pair where its name is the key, and its value is the value. And assigning the resulting object to .records[0] forces the creation of the surrounding structure.
jq --null-input\
--argjson id 1234\
--arg song Yesterday\
--arg artist "The Beatles"\
'{ "records" : [{ $id, $song, $artist }] }'
gives
{
"records": [
{
"id": 1234,
"song": "Yesterday",
"artist": "The Beatles"
}
]
}
I think you got the JSON/JQ the wrong way round:
This should be your JQ script:
rec.jq
{
records: [
{
id: $id,
song: $song,
artist: $artist
}
]
}
And this should be your JSON (empty):
rec.json
{}
Then:
jq --arg id 123 --arg song "Yesterday" --arg artist "The Beatles" -f rec.jq rec.json
Which produces:
{
"records": [
{
"id": "123",
"song": "Yesterday",
"artist": "The Beatles"
}
]
}
Start with an empty JSON and add the missing bits:
$ jq --arg id 1234 \
--arg song Yesterday \
--arg artist "The Beatles" \
'. | .records[0].id=$id | .records[0].song=$song | .records[0].artist=$artist' \
<<<'{}'
Outputs
{
"records": [
{
"id": "1234",
"song": "Yesterday",
"artist": "The Beatles"
}
]
}
Another, cleaner, approach based on the answer of #Inian could be
jq -n \
--arg id 1234
--arg song Yesterday
--arg artist "The Beatles"
'{records: [{id:$id, song:$song, artist:$artist}]}'
Hey I am using conduit curl method to create tasks from post. It work fine when I run from terminal with hardcoded values. But when I try to execute it with variables it throws an error:
Script:
#!/bin/bash
echo "$1"
echo "$2"
echo "$3"
echo "$4"
echo "$5"
echo '{
"transactions": [
{
"type": "title",
"value": "$1"
},
{
"type": "description",
"value": "$2"
},
{
"type": "status",
"value": "$3"
},
{
"type": "priority",
"value": "$4"
},
{
"type": "owner",
"value": "$5"
}
]
}' | arc call-conduit --conduit-uri https://mydomain.phacility.com/ --conduit-token mytoken maniphest.edit
execution:
./test.sh "test003 ticket from api post" "for testing" "open" "high" "ahsan"
Output:
test003 ticket from api post
for testing
open
high
ahsan
{"error":"ERR-CONDUIT-CORE","errorMessage":"ERR-CONDUIT-CORE: Validation errors:\n - User \"$5\" is not a valid user.\n - Task priority \"$4\" is not a valid task priority. Use a priority keyword to choose a task priority: unbreak, very, high, kinda, triage, normal, low, wish.","response":null}
As you can see in error its reading $4 and $5 as values not variables. And I am failing to understand how to use $variables as input in these arguments.
You're using single quotes around the last echo to so that you can use double-quotes inside the JSON, but that causes echo to print the string without expanding anything. You need to use double quotes for the string, so you'll have to escape the double quotes inside of it.
Replace the last echo with this:
echo "{
\"transactions\": [
{
\"type\": \"title\",
\"value\": \"$1\"
},
{
\"type\": \"description\",
\"value\": \"$2\"
},
{
\"type\": \"status\",
\"value\": \"$3\"
},
{
\"type\": \"priority\",
\"value\": \"$4\"
},
{
\"type\": \"owner\",
\"value\": \"$5\"
}
]
}"
and it'll work. To avoid issues like this you can check http://wiki.bash-hackers.org and http://mywiki.wooledge.org/BashGuide for some general tips for bash newbies. Also, you can use shellcheck with a lot of text editors, which will spot errors like this automatically.
I have a while loop with two variables I have to merge into a single piece of JSON, like so:
#!/bin/bash
while read -r from to
do
# BONUS: Solution would ideally require no quoting at this point
echo { \"From\": \"$from\", \"To\": \"$to\" }
done << EOF
foo bar
what ever
EOF
Which currently outputs invalid JSON:
{ "From": "foo", "To": "bar" }
{ "From": "what", "To": "ever" }
What's the simplest I can create valid JSON like:
[
{ "From": "foo", "To": "bar" },
{ "From": "what", "To": "ever" }
]
I looked at jq but I couldn't figure out how to do it. I'm not looking to do it in shell ideally because I feel adding commas and such is a bit ugly.
With jq:
$ jq -nR '[inputs | split(" ") | {"From": .[0], "To": .[1]}]' <<EOF
foo bar
what ever
EOF
[
{
"From": "foo",
"To": "bar"
},
{
"From": "what",
"To": "ever"
}
]
-n tells jq to not read any input; -R is for raw input so it doesn't expect JSON.
The input is read with inputs, resulting in one string per input line:
$ jq -nR 'inputs' <<EOF
foo bar
what ever
EOF
"foo bar"
"what ever"
These are then split into arrays of words:
$ jq -nR 'inputs | split(" ")' <<EOF
foo bar
what ever
EOF
[
"foo",
"bar"
]
[
"what",
"ever"
]
From this, we construct the objects:
$ jq -nR 'inputs | split(" ") | {"From": .[0], "To": .[1]}' <<EOF
foo bar
what ever
EOF
{
"From": "foo",
"To": "bar"
}
{
"From": "what",
"To": "ever"
}
And finally, we wrap everything in [] to get the final output shown first.
The more intuitive approach of splitting input directly fails because wrapping everything in [] results in one array per input line:
$ jq -R '[split(" ") | { "From": .[0], "To": .[1] }]' <<EOF
foo bar
what ever
EOF
[
{
"From": "foo",
"To": "bar"
}
]
[
{
"From": "what",
"To": "ever"
}
]
Hence the somewhat cumbersome -n/inputs. Notice that inputs requires jq version 1.5.
Here's an all-jq solution that assumes the "From" and "To" values are presented exactly as in your example:
jq -R -n '[inputs | split(" ") | {From: .[0], To: .[1]}]'
You might want to handle additional spaces using gsub/2.
If your jq does not have inputs then you can use this incantation:
jq -R -s 'split("\n")
| map(select(length>1) | split(" ") | {From: .[0], To: .[1]})'
Or you could just pipe the output from your while-loop into jq -s.
I have JSON like this that I'm parsing with jq:
{
"data": [
{
"item": {
"name": "string 1"
},
"item": {
"name": "string 2"
},
"item": {
"name": "string 3"
}
}
]
}
...and I'm trying to get "string 1" "string 2" and "string 3" into a Bash array, but I can't find a solution that ignores the whitespace in them. Is there a method in jq that I'm missing, or perhaps an elegant solution in Bash for it?
Current method:
json_names=$(cat file.json | jq ".data[] .item .name")
read -a name_array <<< $json_names
The below assume your JSON text is in a string named s. That is:
s='{
"data": [
{
"item1": {
"name": "string 1"
},
"item2": {
"name": "string 2"
},
"item3": {
"name": "string 3"
}
}
]
}'
Unfortunately, both of the below will misbehave with strings containing literal newlines; since jq doesn't have support for NUL-delimited output, this is difficult to work around.
On bash 4 (with slightly sloppy error handling, but tersely):
readarray -t name_array < <(jq -r '.data[] | .[] | .name' <<<"$s")
...or on bash 3.x or newer (with very comprehensive error handling, but verbosely):
# -d '' tells read to process up to a NUL, and will exit with a nonzero exit status if that
# NUL is not seen; thus, this causes the read to pass through any error which occurred in
# jq.
IFS=$'\n' read -r -d '' -a name_array \
< <(jq -r '.data[] | .[] | .name' <<<"$s" && printf '\0')
This populates a bash array, contents of which can be displayed with:
declare -p name_array
Arrays are assigned in the form:
NAME=(VALUE1 VALUE2 ... )
where NAME is the name of the variable, VALUE1, VALUE2, and the rest are fields separated with characters that are present in the $IFS (input field separator) variable.
Since jq outputs the string values as lines (sequences separated with the new line character), then you can temporarily override $IFS, e.g.:
# Disable globbing, remember current -f flag value
[[ "$-" == *f* ]] || globbing_disabled=1
set -f
IFS=$'\n' a=( $(jq --raw-output '.data[].item.name' file.json) )
# Restore globbing
test -n "$globbing_disabled" && set +f
The above will create an array of three items for the following file.json:
{
"data": [
{"item": {
"name": "string 1"
}},
{"item": {
"name": "string 2"
}},
{"item": {
"name": "string 3"
}}
]
}
The following shows how to create a bash array consisting of arbitrary JSON texts produced by a run of jq.
In the following, I'll assume input.json is a file with the following:
["string 1", "new\nline", {"x": 1}, ["one\ttab", 4]]
With this input, the jq filter .[] produces four JSON texts -- two JSON strings, a JSON object, and a JSON array.
The following bash script can then be used to set x to be a bash array of the JSON texts:
#!/bin/bash
x=()
while read -r value
do
x+=("$value")
done < <(jq -c '.[]' input.json)
For example, adding this bash expression to the script:
for a in "${x[#]}" ; do echo a="$a"; done
would yield:
a="string 1"
a="new\nline"
a={"x":1}
a=["one\ttab",4]
Notice how (encoded) newlines and (encoded) tabs are handled properly.