I have what I would consider a very messy block of JSON, and I want to read and
modify two values that are deeply nested within (denoted: I want this!) using Go.
Due to the server that I am sending it to,
I cannot change the label names. What makes it especially difficult
for me is that parents have multiple children which are also nested, and since there are so many "value" labels I don't know how to specify which "value" child thatI want to enter.
I got the values in Bash very quickly using this
jq ' .value[0].value[1].value[0].value[1].value[0].value="'"$one"'" | '\ '
.value[0].value[1].value[0].value[1].value[1].value="'"$two"'"'
I tried a format like this in Go initially, but could not get it working because of the issue where children are all named "value" and I want to go into one other than the first. Unfortunately none of those magic JSON to Go struct were able to handle all of the nesting either.
type Bar struct {
Value struct { // Value[0]
Value struct { // Value[1]
Value struct { // Value[0]
Value struct { // Value[1]
Value struct { // Value[1] or Value[2]
}}}}}}
This code converts the JSON into a more struct/map friendly form, and prints the whole thing.
var foo interface{}
err := json.Unmarshal(blob, &foo)
if err != nil {
fmt.Println("error:", err)
}
m := foo.(map[string]interface{})
// print all
fmt.Println("Loop: ", m)
for k, v := range m {
fmt.Printf("%v : %v", k, v)
}
Here is the JSON that I am storing as a variable
var blob = []byte(`{
"tag": "1",
"value": [{
"tag": "2",
"value": [{
"tag": "3",
"value": [{
"tag": "4",
"type": "x",
"value": "x"
}, {
"tag": "5",
"type": "x",
"value": "x"
}
]
}, {
"tag": "6",
"value": [{
"tag": "7",
"value": [{
"tag": "8",
"type": "x",
"value": "x"
}, {
"tag": "9",
"value": [{
"tag": "10",
"type": "x",
"value": "I want this!"
}, {
"tag": "11",
"type": "Thanks for the help mate!",
"value": "I want this!"
}
]
}
]
}
]
}, {
"tag": "12",
"type": "x",
"value": "x"
}
]
}, {
"tag": "13",
"value": [{
"tag": "14",
"type": "x",
"value": "x"
}, {
"tag": "15",
"value": [{
"tag": "16",
"value": [{
"tag": "17",
"type": "x",
"value": "x"
}
]
}
]
}
]
}
]
}`)
I would appreciate any help or advice that you could give me.
Your initial attempt didn't work because you used structs where you need a slice of struct (Value []struct{...}). But that won't work either, because in some cases the values are slices and in some cases they're strings, which encoding/json does not like. Your best bet is one of two options: raw text manipulation, which will probably be quicker but more fragile and error-prone, or decoding as in your second example into a map[string]interface{} and doing some gnarly type assertions:
var foo map[string]interface{}
err := json.Unmarshal(blob, &foo)
if err != nil {
fmt.Println("error:", err)
}
foo["value"].([]interface{})[0].(map[string]interface{})["value"].([]interface{})[1].(map[string]interface{})["value"].([]interface{})[0].(map[string]interface{})["value"].([]interface{})[1].(map[string]interface{})["value"].([]interface{})[0].(map[string]interface{})["value"]
Related
Is there any solution to parse an unstructured json(text) data?
below is a sample response of a web requst that i want to parse and access data (the inner list)
res,err := http.Get("url_of_server")
[[
{
"id": "1",
"text": "sample text",
"user": {
"user_id": "1",
"username": "user1"
},
"created_at_utc": "2022-12-20T16:38:06+00:00",
"status": "Active"
},
{
"id": "2",
"text": "sample text",
"user": {
"user_id": "2",
"username": "user2"
},
"created_at_utc": "2022-12-01T10:15:00+00:00",
"status": "Active"
}
],
"{"code": "hsdvnkvuahudvhafdlfv",
"is_updated": true}",
null
]
what i want to get is:
[
{
"id": "1",
"text": "sample text",
"user": {
"user_id": "1",
"username": "user1"
},
"created_at_utc": "2022-12-20T16:38:06+00:00",
"status": "Active"
},
{
"id": "2",
"text": "sample text",
"user": {
"user_id": "2",
"username": "user2"
},
"created_at_utc": "2022-12-01T10:15:00+00:00",
"status": "Active"
}
]
in python it is possible by easily using res.json()[0]
I have tried using json.Unmarshal() to a map and also struct but does not work,
i don't know how to get rid of this part of response:
"{"code": "hsdvnkvuahudvhafdlfv",
"is_updated": true}",
null
Declare a type for the items:
type Item struct {
ID string `json:"id"`
Text string `json:"text"`
User struct {
UserID string `json:"user_id"`
Username string `json:"username"`
} `json:"user"`
CreatedAtUtc time.Time `json:"created_at_utc"`
Status string `json:"status"`
}
Declare a slice of the items:
var items []Item
Declare a slice representing the entire JSON thing. The first element is the items.
var v = []any{&items}
Unmarshal to v. The items slice will have the values that you are looking for. The second and third elements of v will contain the values you want to ignore.
err := json.Unmarshal(data, &v)
Run the code in the GoLang PlayGround.
Go's standard JSON library is not as flexible as others when it comes to dealing with unexpected or uncontrolled input.
A great alternative is tidwall's gjson.
Example code with gjson:
package main
import (
"fmt"
"github.com/tidwall/gjson"
)
const textInput = `[[
{
"id": "1",
"text": "sample text",
"user": {
"user_id": "1",
"username": "user1"
},
"created_at_utc": "2022-12-20T16:38:06+00:00",
"status": "Active"
},
{
"id": "2",
"text": "sample text",
"user": {
"user_id": "2",
"username": "user2"
},
"created_at_utc": "2022-12-01T10:15:00+00:00",
"status": "Active"
}
],
"{"code": "hsdvnkvuahudvhafdlfv",
"is_updated": true}",
null
]`
func main() {
jsonBody := gjson.Parse(textInput)
fmt.Println(jsonBody.Get("0"))
}
I'm looking to go from a JSON structure that looks something like this:
{
"id": "955559665",
"timestamp": "2022-04-21 00:00:19",
"user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15",
"remote_addr": "123.456.789.012",
"read": "0",
"data": {
"80928111": {
"field": "80928111",
"value": "Z01234567",
"flat_value": "Z01234567",
"label": "ID",
"type": "text"
},
"90924321": {
"field": "90924321",
"value": {
"first": "Jane",
"last": "Doe"
},
"flat_value": "first = Jane\nlast = Doe",
"label": "Name",
"type": "name"
},
"88888770": {
"field": "88888770",
"value": "jdoe001#gmail.com",
"flat_value": "jdoe001#gmail.com",
"label": "Email",
"type": "email"
},
"12345678": {
"field": "12345678",
"value": "https://www.google.com/subdomain/attachment/file.txt",
"flat_value": "https://www.google.com/subdomain/attachment/file.txt",
"label": "Choose File",
"type": "file"
}
}
}
Ultimately to something like this:
{
"name_val":"Name: first = Jane\nlast = Doe\nEmail: jdoe001#gmail.com\n",
"file": {
"id": "12345678C",
"name": "file.txt"
}
}
In the original JSON, the 'data' object represents a form submission. Each sub object represents a field on the submitted form. The only distinction I'm interested in is the 'type' of field identified as 'file'.
Every response that is not of 'file' type, I want to concatenate into one large String value that looks like: 'label1: flat_value1\nlabel2: flat_value2...'
Note, the number of actual fields is variable.
Then I need a second object to show the field of type 'file', by identifying the 'field' id, and the name of the file.
I've gotten pieces of this to work. For example, using pluck and filter, I've been able to separate the types of fields.
Something like this:
%dw 2.0
output application/json
---
[
"fields": payload.data pluck(
{
"field": $."label",
"value": $."flat_value",
"type": $."type"
}
) filter ($."type" != "file") default "",
"files": payload.data pluck(
{
"type": $."type",
"fieldId": $."field"
}
) filter ($."type" == "file") default ""
]
Gives me:
[
{
"fields": [
{
"field": "ID",
"value": "Z01234567",
"type": "text"
},
{
"field": "Name",
"value": "first = Jane\nlast = Doe",
"type": "name"
},
{
"field": "Email",
"value": "jdoe001#gmail.com",
"type": "email"
}
]
},
{
"files": [
{
"type": "file",
"fieldId": "12345678"
}
]
}
]
And playing around with a modified JSON input, I was able to easily see concatenation similar to how I want to see it, but not quite there:
%dw 2.0
output application/json
var inputJson = [
{
"field": "ID",
"value": "Z01234567",
"type": "text"
},
{
"field": "Name",
"value": "first = Jane\nlast = Doe",
"type": "name"
}
]
---
inputJson map ((value, index) -> value.field ++ ': ' ++ value.value)
Gives me:
[
"ID: Z01234567",
"Name: first = Jane\nlast = Doe"
]
But I can't seem to put it all together and go from Beginning to End.
There are several ways to implement this. I recommend to try to encapsulate the parts that you get working and use them as building blocks.
%dw 2.0
output application/json
fun fields(x) = x.data pluck(
{
"field": $."label",
"value": $."flat_value",
"type": $."type"
}
) filter ($."type" != "file") default ""
fun files(x) = x.data pluck(
{
"type": $."type",
"fieldId": $."field"
}
) filter ($."type" == "file") default ""
---
{
name_val: fields(payload) reduce ((item,acc="") -> acc ++ item.field ++ ': ' ++ item.value ++ "\n"),
files: files(payload)[0]
}
Output:
{
"name_val": "ID: Z01234567\nName: first = Jane\nlast = Doe\nEmail: jdoe001#gmail.com\n",
"files": {
"type": "file",
"fieldId": "12345678"
}
}
I am updating the problem statement since input from nipuna, who solved the problem. The question is better described as, "why does unmarshal, then marshal of JSON result in changes to the JSON that are not compatible with a REST API request?" The reason discovered, and that I could not have discovered alone, were that the text fields sometimes needed to be included and sometimes omitted in a JSON array, and the marshal data structure did not support omitting them. The structure did use the term "omitempty" for these fields, so it could be believed this directive would omit them, however that is not sufficient. I also needed for the text fields to be pointers. The original problem description follows below. The final data structure uses both "omitempty" for the text fields and also makes the anonymous struct a pointer. (Please see the accepted answer)
I have a REST request that fails when I use Go Unmarshal / Marshal but it works on the same JSON data from a file. I have verified that the resulting []byte from reading the file is different if I take that exact []byte and roundtrip UnMarshal / Marshal it. As a result, my REST request fails. Here it is diagramatically: file json -> []byte = set A of bytes. file json -> []byte -> UnMarshal -> Marshal -> []byte = set B of bytes. Set A of bytes != set B of bytes. I would not care except that I need to modify the data in the structure of Go, then Marshal it, however the byte changes that I do not understand causes the request to fail.
Here is test code:
type SlackRequestData struct {
Blocks []struct {
BlockID string `json:"block_id"`
Elements []struct {
ActionID string `json:"action_id"`
Style string `json:"style"`
Text struct {
Text string `json:"text"`
Type string `json:"type"`
} `json:"text"`
Type string `json:"type"`
Value string `json:"value"`
} `json:"elements"`
Text struct {
Text string `json:"text"`
Type string `json:"type"`
} `json:"text"`
Type string `json:"type"`
} `json:"blocks"`
}
func MarshalFailTest(){
bytes_io, err := ioutil.ReadFile("../../slack/slack-message.json")
if err != nil {
fmt.Printf("Error: %+v\n", err) // handle err
}
new_req := SlackRequestData{}
err = json.Unmarshal(bytes_io, &new_req)
if err != nil {
fmt.Printf("Error: %+v\n", err) // handle err
}
new_json, err := json.Marshal(new_req)
if err != nil {
fmt.Printf("Error: %+v\n", err) // handle err
}
fmt.Printf("New req:\n%+v\n", new_req)
fmt.Printf("New Json:\n%+v\n", new_json)
fmt.Printf("Bytes:\n%+v\n", bytes_io)
}
Here is the json file data:
{
"blocks": [{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "Service Is Ready. Now?"
}
},
{
"type": "actions",
"block_id": "deploy_id",
"elements": [{
"type": "button",
"action_id": "yes_button",
"text": {
"type": "plain_text",
"text": "Yes"
},
"style": "danger",
"value": "yes"
},
{
"type": "button",
"action_id": "yes_toll_button",
"text": {
"type": "plain_text",
"text": "Yes To All"
},
"style": "danger",
"value": "yes"
},
{
"type": "button",
"action_id": "no_button",
"text": {
"type": "plain_text",
"text": "No"
},
"style": "primary",
"value": "no"
},
{
"type": "button",
"action_id": "no_toall_button",
"text": {
"type": "plain_text",
"text": "No To All"
},
"style": "primary",
"value": "no"
}]
}]
}
Note that I used several converters to take the file json and create the SlackRequestData struct, and the UnMarshal appears to work perfectly as the struct in the debugger has all the data in the right place. There are no errors with Marshal / UnMarshal either. So why does the []byte differ?
The data sets are large, so I'll just share the first line of each set here.
Set A:
[123 10 32 32 34 98 108 111 99 107 115 34 58 32 91 123 10 32 32 32 32 34 116 121 112 101 34 58 32 34 115 101 99 116 105 111 110 34 44 10 32 32 32 32 34
Set B:
[123 34 98 108 111 99 107 115 34 58 91 123 34 98 108 111 99 107 95 105 100 34 58 34 34 44 34 101 108 101 109 101 110 116 115 34 58 110 117 108 108 44
Here are the two sets converted from []byte to strings, as this is quite interesting too. The set A keeps whitespace, the set B has no whitespace but also the elements have been rearranged. I wonder if the struct is not correct? However I've been using this technique of using converters to create structs out of json and they have worked well to this point.
Set A:
{
"blocks": [{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "Service Is Ready. Now?"
}
},
{
"type": "actions",
"block_id": "deploy_id",
"elements": [{
"type": "button",
"action_id": "yes_button",
"text": {
"type": "plain_text",
"text": "Yes"
},
"style": "danger",
"value": "yes"
},
{
"type": "button",
"action_id": "yes_toll_button",
"text": {
"type": "plain_text",
"text": "Yes To All"
},
"style": "danger",
"value": "yes"
},
{
"type": "button",
"action_id": "no_button",
"text": {
"type": "plain_text",
"text": "No"
},
"style": "primary",
"value": "no"
},
{
"type": "button",
"action_id": "no_toall_button",
"text": {
"type": "plain_text",
"text": "No To All"
},
"style": "primary",
"value": "no"
}]
}]
}
Set B:
{"blocks":[{"block_id":"","elements":null,"text":{"text":"Service Is Ready. Now?","type":"mrkdwn"},"type":"section"},{"block_id":"deploy_id","elements":[{"action_id":"yes_button","style":"danger","text":{"text":"Yes","type":"plain_text"},"type":"button","value":"yes"},{"action_id":"yes_toll_button","style":"danger","text":{"text":"Yes To All","type":"plain_text"},"type":"button","value":"yes"},{"action_id":"no_button","style":"primary","text":{"text":"No","type":"plain_text"},"type":"button","value":"no"},{"action_id":"no_toall_button","style":"primary","text":{"text":"No To All","type":"plain_text"},"type":"button","value":"no"}],"text":{"text":"","type":""},"type":"actions"}]}
nipuna suggested I try using beautify of the JSON so I did this and there is the string version of the []byte for each set.
Set A:
{
"blocks": [{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "Service Is Ready. Now?"
}
},
{
"type": "actions",
"block_id": "deploy_id",
"elements": [{
"type": "button",
"action_id": "yes_button",
"text": {
"type": "plain_text",
"text": "Yes"
},
"style": "danger",
"value": "yes"
},
{
"type": "button",
"action_id": "yes_toll_button",
"text": {
"type": "plain_text",
"text": "Yes To All"
},
"style": "danger",
"value": "yes"
},
{
"type": "button",
"action_id": "no_button",
"text": {
"type": "plain_text",
"text": "No"
},
"style": "primary",
"value": "no"
},
{
"type": "button",
"action_id": "no_toall_button",
"text": {
"type": "plain_text",
"text": "No To All"
},
"style": "primary",
"value": "no"
}]
}]
}
Set B:
{
"blocks": [
{
"block_id": "",
"elements": null,
"text": {
"text": "Service Is Ready. Now?",
"type": "mrkdwn"
},
"type": "section"
},
{
"block_id": "deploy_id",
"elements": [
{
"action_id": "yes_button",
"style": "danger",
"text": {
"text": "Yes",
"type": "plain_text"
},
"type": "button",
"value": "yes"
},
{
"action_id": "yes_toll_button",
"style": "danger",
"text": {
"text": "Yes To All",
"type": "plain_text"
},
"type": "button",
"value": "yes"
},
{
"action_id": "no_button",
"style": "primary",
"text": {
"text": "No",
"type": "plain_text"
},
"type": "button",
"value": "no"
},
{
"action_id": "no_toall_button",
"style": "primary",
"text": {
"text": "No To All",
"type": "plain_text"
},
"type": "button",
"value": "no"
}
],
"text": {
"text": "",
"type": ""
},
"type": "actions"
}
]
}
I can see following differences and I have mention some suggestions.
Optional fields
Your data consist of array of blocks, in first block, there is only two fields text and type and in second block there is all other fields but no text. So that text is optional and it need to be omit when it is an empty struct. So you need to define your SlackRequestData with omitempty and pointer type to Text fields.
Suggested SlackRequestData struct is something like below.
type Text struct {
Type string `json:"type,omitempty"`
Text string `json:"text,omitempty"`
}
type SlackRequestData struct {
Blocks []struct {
BlockID string `json:"block_id,omitempty"`
Elements []struct {
ActionID string `json:"action_id,omitempty"`
Style string `json:"style,omitempty"`
Text *Text `json:"text,omitempty"`
Type string `json:"type,omitempty"`
Value string `json:"value,omitempty"`
} `json:"elements,omitempty"`
Text *Text `json:"text,omitempty"`
Type string `json:"type,omitempty"`
} `json:"blocks,omitempty"`
}
Marshalled Data Order
your file json data's field order and SlackRequestData's field order is different. So obviously that marshalled string and file string is different. json marshalling not guarantees that order when marshalling as mentiond in here. So if you need to compare those two, please sort those bytes in someway and compare.
Json Indent
In your file data is somewhat json beautified data and json marshal return compacted output. If you need to get same marshal output, you need to use MarshalIndent and do indent you used in file.
So if you need to compare both things, marshal with omitempty, sort both results (file byte array and json marshalled byte array). and remove whitespaces and then compare. You can get correct same result.
I have a Graphql returning values that look like this
"{\"date\":\"2020-05-21\",\"time\":null,\"changed_at\":\"2020-05-25T16:16:33.201Z\"}"
How would you target just the date to return "2020-05-21"?
Here is a larger picture of the retuned.
"column_values": [
{
"value": null,
"id": "check8",
"title": "Approved?"
},
{
"value": "{\"date\":\"2020-05-21\",\"time\":null,\"changed_at\":\"2020-05-25T16:16:33.201Z\"}",
"id": "due_date2",
"title": "Due Date"
},
{
"value": null,
"id": "qtr",
"title": "QTR"
},
{
"value": null,
"id": "status5",
"title": "Priority"
},
{
"value": "{\"index\":2,\"post_id\":null,\"changed_at\":\"2020-05-22T17:56:59.936Z\"}",
"id": "status",
"title": "Status"
},
{
"value": null,
"id": "progress",
"title": "Progress"
},
{
"value": null,
"id": "link",
"title": "Video Link"
},
{
"value": null,
"id": "formula8",
"title": "DATE_API"
}
]
}
Do you have to parse the JSON or Stringify it first? perhaps Graphql isn't returning usable json. Thank you for any help or bump in the correct direction.
JSON.stringify() is the method used to generate a JSON string. If you apply it to something that's already a JSON string then you'll get a double-encoded JSON string.
What you need to use is the JSON.parse() method:
const jsonObject= '{"foo": "bar"}';
const decoded = JSON.parse(jsonObject);
console.log(decode, typeof decoded);
Result:
{ foo: 'bar' } 'object'
I have got a configuration with content to be exchanged with snippets from separate file. How would I neatly achieve this?
Config file might look like:
# config file
{
"keep": "whatever type of value",
"manipulate": [
{
"foo": "bar",
"cat": {
"color": "grey"
},
"type": "keep",
"detail": "keep whole array element"
},
{
"stuff": "obsolete",
"more_stuff": "obsolete",
"type": "replace",
"detail": "replace whole array element with content from separate file"
},
{
"foz": "baz",
"dog": {
"color": "brown"
},
"type": "keep",
"detail": "keep whole array element"
},
],
"also_keep": "whatever type of value"
}
The content (coming from separate file) to be inserted as a replacement of obsolete array element:
# replacement
{
"stuff": "i want that",
"fancy": "very",
"type": "new"
}
The desired result should look like:
# result
{
"keep": "whatever kind of value",
"manipulate": [
{
"foo": "bar",
"cat": {
"color": "grey"
},
"type": "keep",
"detail": "keep whole array element"
},
{
"stuff": "i want that",
"fancy": "very",
"type": "new"
},
{
"foz": "baz",
"dog": {
"color": "brown"
},
"type": "keep",
"detail": "keep whole array element"
},
],
"also_keep": "whatever kind of value",
}
Requirements:
Content replacement needs to be done based on type key's value.
Preferably use jq and common bash/linux tools.
Array element ordering should stay same
jq solution:
jq --slurpfile repl repl.json '.manipulate=[.manipulate[]
| if .type=="replace" then .=$repl[0] else . end]' config.json
repl.json - json file containing replacement JSON data
--slurpfile repl repl.json - reads all the JSON texts in the named file and binds an array of the parsed JSON values to the given global variable
The output:
{
"keep": "whatever type of value",
"manipulate": [
{
"foo": "bar",
"cat": {
"color": "grey"
},
"type": "keep",
"detail": "keep whole array element"
},
{
"stuff": "i want that",
"fancy": "very",
"type": "new"
},
{
"foz": "baz",
"dog": {
"color": "brown"
},
"type": "keep",
"detail": "keep whole array element"
}
],
"also_keep": "whatever type of value"
}
jq --slurpfile repl repl.json '.manipulate |=
map(if .type=="replace" then $repl[0] else . end)' config.json