Encode JSON value while disabling line wrapping - json

Question

I originally had an issue with a JSON (let's call it "raw_data.json") that looked like this:
[
{
"metadata": {
"project": [
"someProject1",
"someProject2"
],
"table_name": [
"someTable1",
"someTable2"
],
"sys_insertdatetime": "someDate",
"sys_sourcesystem": "someSourceSystem"
},
"data": {
"field1": 63533712,
"field2": "",
"field3": "hello",
"field4": "other",
"field5": 2022,
"field6": "0",
"field7": "0",
"field8": "0",
"field9": "0",
"field10": "0",
"field11": "0"
}
},
{
"metadata": {
"project": [
"someProject1",
"someProject2"
],
"table_name": [
"someTable1",
"someTable2"
],
"sys_insertdatetime": "someDate",
"sys_sourcesystem": "someSourceSystem"
},
"data": {
"field1": 63533713,
"field2": "Y2JjLTIwMjItdzA1LXRyZi1vZmZyZXMtcmVuZm9ydC1jaGVxdWllci13MDU=",
"field3": "A0AVB",
"field4": "other",
"field5": "HJlbmZvcnQgY2hlcXVpZXIgVzA1",
"field6": "",
"field7": "02/02/2022",
"field8": "14/02/2022",
"field9": "Ticket"
}
}
]
The dictionaries were pretty big and usually spanned over several lines on my monitor. That was an issue because I needed to encode the whole thing in Base64 and send the result with a HTTP POST method. But the encoding tended to wrap lines by adding newlines characters, which caused my POST method to fail.
I fortunately found a question with the following solution:
export DATA=$(cat 'raw_data.json')
export ENCODED_DATA=$(echo "$DATA" | jq -c . | base64 -w 0)
Problem is, my JSON has now changed to this:
{
"request_id": 1234,
"data_to_be_encoded": [
{
"metadata": {
"project": [
"someProject1",
"someProject2"
],
"table_name": [
"someTable1",
"someTable2"
],
"sys_insertdatetime": "someDate",
"sys_sourcesystem": "someSourceSystem"
},
"data": {
"field1": 63533712,
"field2": "",
"field3": "hello",
"field4": "other",
"field5": 2022,
"field6": "0",
"field7": "0",
"field8": "0",
"field9": "0",
"field10": "0",
"field11": "0"
}
},
{
"metadata": {
"project": [
"someProject1",
"someProject2"
],
"table_name": [
"someTable1",
"someTable2"
],
"sys_insertdatetime": "someDate",
"sys_sourcesystem": "someSourceSystem"
},
"data": {
"field1": 63533713,
"field2": "Y2JjLTIwMjItdzA1LXRyZi1vZmZyZXMtcmVuZm9ydC1jaGVxdWllci13MDU=",
"field3": "A0AVB",
"field4": "other",
"field5": "HJlbmZvcnQgY2hlcXVpZXIgVzA1",
"field6": "",
"field7": "02/02/2022",
"field8": "14/02/2022",
"field9": "Ticket"
}
}
]
}
Basically I needed to keep the request_id key-value pair as is, while what was inside the data_to_be_encoded key had to be encoded. Again, another post offered a nice solution:
export DATA=$(cat 'raw_data.json')
export ENCODED_DATA=$(echo "$DATA" | jq '.data_to_be_encoded |= #base64')
Except for the fact that this solution adds line wrapping and I haven't found a way to disable that feature like I managed to do with the first solution.
I did try this:
export ENCODED_DATA=$(echo "$DATA" | jq -c .data_to_be_encoded | base64 -w 0)
But it only returns the value inside the data_to_be_encoded key and not the whole JSON. So I'm back to square one.
How can I get the best of both worlds? In other words, how can I disable the line wrapping while at the same time encoding a specific part of my JSON?

Answer 1

Use jq's #base64 builtin, rather than doing the conversion outside because that way you can either convert all or nothing.
jq '.data_to_be_encoded |= #base64'
{
"request_id": 1234,
"data_to_be_encoded": "W3sibWV0YWRhdGEiOnsicHJvamVjdCI6WyJzb21lUHJvamVjdDEiLCJzb21lUHJvamVjdDIiXSwidGFibGVfbmFtZSI6WyJzb21lVGFibGUxIiwic29tZVRhYmxlMiJdLCJzeXNfaW5zZXJ0ZGF0ZXRpbWUiOiJzb21lRGF0ZSIsInN5c19zb3VyY2VzeXN0ZW0iOiJzb21lU291cmNlU3lzdGVtIn0sImRhdGEiOnsiZmllbGQxIjo2MzUzMzcxMiwiZmllbGQyIjoiIiwiZmllbGQzIjoiaGVsbG8iLCJmaWVsZDQiOiJvdGhlciIsImZpZWxkNSI6MjAyMiwiZmllbGQ2IjoiMCIsImZpZWxkNyI6IjAiLCJmaWVsZDgiOiIwIiwiZmllbGQ5IjoiMCIsImZpZWxkMTAiOiIwIiwiZmllbGQxMSI6IjAifX0seyJtZXRhZGF0YSI6eyJwcm9qZWN0IjpbInNvbWVQcm9qZWN0MSIsInNvbWVQcm9qZWN0MiJdLCJ0YWJsZV9uYW1lIjpbInNvbWVUYWJsZTEiLCJzb21lVGFibGUyIl0sInN5c19pbnNlcnRkYXRldGltZSI6InNvbWVEYXRlIiwic3lzX3NvdXJjZXN5c3RlbSI6InNvbWVTb3VyY2VTeXN0ZW0ifSwiZGF0YSI6eyJmaWVsZDEiOjYzNTMzNzEzLCJmaWVsZDIiOiJZMkpqTFRJd01qSXRkekExTFhSeVppMXZabVp5WlhNdGNtVnVabTl5ZEMxamFHVnhkV2xsY2kxM01EVT0iLCJmaWVsZDMiOiJBMEFWQiIsImZpZWxkNCI6Im90aGVyIiwiZmllbGQ1IjoiSEpsYm1admNuUWdZMmhsY1hWcFpYSWdWekExIiwiZmllbGQ2IjoiIiwiZmllbGQ3IjoiMDIvMDIvMjAyMiIsImZpZWxkOCI6IjE0LzAyLzIwMjIiLCJmaWVsZDkiOiJUaWNrZXQifX1d"
}
Demo
If you need all in one line, use the -c option as before:
jq -c '.data_to_be_encoded |= #base64'
{"request_id":1234,"data_to_be_encoded":"W3sibWV0YWRhdGEiOnsicHJvamVjdCI6WyJzb21lUHJvamVjdDEiLCJzb21lUHJvamVjdDIiXSwidGFibGVfbmFtZSI6WyJzb21lVGFibGUxIiwic29tZVRhYmxlMiJdLCJzeXNfaW5zZXJ0ZGF0ZXRpbWUiOiJzb21lRGF0ZSIsInN5c19zb3VyY2VzeXN0ZW0iOiJzb21lU291cmNlU3lzdGVtIn0sImRhdGEiOnsiZmllbGQxIjo2MzUzMzcxMiwiZmllbGQyIjoiIiwiZmllbGQzIjoiaGVsbG8iLCJmaWVsZDQiOiJvdGhlciIsImZpZWxkNSI6MjAyMiwiZmllbGQ2IjoiMCIsImZpZWxkNyI6IjAiLCJmaWVsZDgiOiIwIiwiZmllbGQ5IjoiMCIsImZpZWxkMTAiOiIwIiwiZmllbGQxMSI6IjAifX0seyJtZXRhZGF0YSI6eyJwcm9qZWN0IjpbInNvbWVQcm9qZWN0MSIsInNvbWVQcm9qZWN0MiJdLCJ0YWJsZV9uYW1lIjpbInNvbWVUYWJsZTEiLCJzb21lVGFibGUyIl0sInN5c19pbnNlcnRkYXRldGltZSI6InNvbWVEYXRlIiwic3lzX3NvdXJjZXN5c3RlbSI6InNvbWVTb3VyY2VTeXN0ZW0ifSwiZGF0YSI6eyJmaWVsZDEiOjYzNTMzNzEzLCJmaWVsZDIiOiJZMkpqTFRJd01qSXRkekExTFhSeVppMXZabVp5WlhNdGNtVnVabTl5ZEMxamFHVnhkV2xsY2kxM01EVT0iLCJmaWVsZDMiOiJBMEFWQiIsImZpZWxkNCI6Im90aGVyIiwiZmllbGQ1IjoiSEpsYm1admNuUWdZMmhsY1hWcFpYSWdWekExIiwiZmllbGQ2IjoiIiwiZmllbGQ3IjoiMDIvMDIvMjAyMiIsImZpZWxkOCI6IjE0LzAyLzIwMjIiLCJmaWVsZDkiOiJUaWNrZXQifX1d"}
Demo