Marshal Unmarshal JSON struct problems causing REST request to fail - json

Question

I am updating the problem statement since input from nipuna, who solved the problem. The question is better described as, "why does unmarshal, then marshal of JSON result in changes to the JSON that are not compatible with a REST API request?" The reason discovered, and that I could not have discovered alone, were that the text fields sometimes needed to be included and sometimes omitted in a JSON array, and the marshal data structure did not support omitting them. The structure did use the term "omitempty" for these fields, so it could be believed this directive would omit them, however that is not sufficient. I also needed for the text fields to be pointers. The original problem description follows below. The final data structure uses both "omitempty" for the text fields and also makes the anonymous struct a pointer. (Please see the accepted answer)
I have a REST request that fails when I use Go Unmarshal / Marshal but it works on the same JSON data from a file. I have verified that the resulting []byte from reading the file is different if I take that exact []byte and roundtrip UnMarshal / Marshal it. As a result, my REST request fails. Here it is diagramatically: file json -> []byte = set A of bytes. file json -> []byte -> UnMarshal -> Marshal -> []byte = set B of bytes. Set A of bytes != set B of bytes. I would not care except that I need to modify the data in the structure of Go, then Marshal it, however the byte changes that I do not understand causes the request to fail.
Here is test code:
type SlackRequestData struct {
Blocks []struct {
BlockID string `json:"block_id"`
Elements []struct {
ActionID string `json:"action_id"`
Style string `json:"style"`
Text struct {
Text string `json:"text"`
Type string `json:"type"`
} `json:"text"`
Type string `json:"type"`
Value string `json:"value"`
} `json:"elements"`
Text struct {
Text string `json:"text"`
Type string `json:"type"`
} `json:"text"`
Type string `json:"type"`
} `json:"blocks"`
}
func MarshalFailTest(){
bytes_io, err := ioutil.ReadFile("../../slack/slack-message.json")
if err != nil {
fmt.Printf("Error: %+v\n", err) // handle err
}
new_req := SlackRequestData{}
err = json.Unmarshal(bytes_io, &new_req)
if err != nil {
fmt.Printf("Error: %+v\n", err) // handle err
}
new_json, err := json.Marshal(new_req)
if err != nil {
fmt.Printf("Error: %+v\n", err) // handle err
}
fmt.Printf("New req:\n%+v\n", new_req)
fmt.Printf("New Json:\n%+v\n", new_json)
fmt.Printf("Bytes:\n%+v\n", bytes_io)
}
Here is the json file data:
{
"blocks": [{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "Service Is Ready. Now?"
}
},
{
"type": "actions",
"block_id": "deploy_id",
"elements": [{
"type": "button",
"action_id": "yes_button",
"text": {
"type": "plain_text",
"text": "Yes"
},
"style": "danger",
"value": "yes"
},
{
"type": "button",
"action_id": "yes_toll_button",
"text": {
"type": "plain_text",
"text": "Yes To All"
},
"style": "danger",
"value": "yes"
},
{
"type": "button",
"action_id": "no_button",
"text": {
"type": "plain_text",
"text": "No"
},
"style": "primary",
"value": "no"
},
{
"type": "button",
"action_id": "no_toall_button",
"text": {
"type": "plain_text",
"text": "No To All"
},
"style": "primary",
"value": "no"
}]
}]
}
Note that I used several converters to take the file json and create the SlackRequestData struct, and the UnMarshal appears to work perfectly as the struct in the debugger has all the data in the right place. There are no errors with Marshal / UnMarshal either. So why does the []byte differ?
The data sets are large, so I'll just share the first line of each set here.
Set A:
[123 10 32 32 34 98 108 111 99 107 115 34 58 32 91 123 10 32 32 32 32 34 116 121 112 101 34 58 32 34 115 101 99 116 105 111 110 34 44 10 32 32 32 32 34
Set B:
[123 34 98 108 111 99 107 115 34 58 91 123 34 98 108 111 99 107 95 105 100 34 58 34 34 44 34 101 108 101 109 101 110 116 115 34 58 110 117 108 108 44
Here are the two sets converted from []byte to strings, as this is quite interesting too. The set A keeps whitespace, the set B has no whitespace but also the elements have been rearranged. I wonder if the struct is not correct? However I've been using this technique of using converters to create structs out of json and they have worked well to this point.
Set A:
{
"blocks": [{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "Service Is Ready. Now?"
}
},
{
"type": "actions",
"block_id": "deploy_id",
"elements": [{
"type": "button",
"action_id": "yes_button",
"text": {
"type": "plain_text",
"text": "Yes"
},
"style": "danger",
"value": "yes"
},
{
"type": "button",
"action_id": "yes_toll_button",
"text": {
"type": "plain_text",
"text": "Yes To All"
},
"style": "danger",
"value": "yes"
},
{
"type": "button",
"action_id": "no_button",
"text": {
"type": "plain_text",
"text": "No"
},
"style": "primary",
"value": "no"
},
{
"type": "button",
"action_id": "no_toall_button",
"text": {
"type": "plain_text",
"text": "No To All"
},
"style": "primary",
"value": "no"
}]
}]
}
Set B:
{"blocks":[{"block_id":"","elements":null,"text":{"text":"Service Is Ready. Now?","type":"mrkdwn"},"type":"section"},{"block_id":"deploy_id","elements":[{"action_id":"yes_button","style":"danger","text":{"text":"Yes","type":"plain_text"},"type":"button","value":"yes"},{"action_id":"yes_toll_button","style":"danger","text":{"text":"Yes To All","type":"plain_text"},"type":"button","value":"yes"},{"action_id":"no_button","style":"primary","text":{"text":"No","type":"plain_text"},"type":"button","value":"no"},{"action_id":"no_toall_button","style":"primary","text":{"text":"No To All","type":"plain_text"},"type":"button","value":"no"}],"text":{"text":"","type":""},"type":"actions"}]}
nipuna suggested I try using beautify of the JSON so I did this and there is the string version of the []byte for each set.
Set A:
{
"blocks": [{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "Service Is Ready. Now?"
}
},
{
"type": "actions",
"block_id": "deploy_id",
"elements": [{
"type": "button",
"action_id": "yes_button",
"text": {
"type": "plain_text",
"text": "Yes"
},
"style": "danger",
"value": "yes"
},
{
"type": "button",
"action_id": "yes_toll_button",
"text": {
"type": "plain_text",
"text": "Yes To All"
},
"style": "danger",
"value": "yes"
},
{
"type": "button",
"action_id": "no_button",
"text": {
"type": "plain_text",
"text": "No"
},
"style": "primary",
"value": "no"
},
{
"type": "button",
"action_id": "no_toall_button",
"text": {
"type": "plain_text",
"text": "No To All"
},
"style": "primary",
"value": "no"
}]
}]
}
Set B:
{
"blocks": [
{
"block_id": "",
"elements": null,
"text": {
"text": "Service Is Ready. Now?",
"type": "mrkdwn"
},
"type": "section"
},
{
"block_id": "deploy_id",
"elements": [
{
"action_id": "yes_button",
"style": "danger",
"text": {
"text": "Yes",
"type": "plain_text"
},
"type": "button",
"value": "yes"
},
{
"action_id": "yes_toll_button",
"style": "danger",
"text": {
"text": "Yes To All",
"type": "plain_text"
},
"type": "button",
"value": "yes"
},
{
"action_id": "no_button",
"style": "primary",
"text": {
"text": "No",
"type": "plain_text"
},
"type": "button",
"value": "no"
},
{
"action_id": "no_toall_button",
"style": "primary",
"text": {
"text": "No To All",
"type": "plain_text"
},
"type": "button",
"value": "no"
}
],
"text": {
"text": "",
"type": ""
},
"type": "actions"
}
]
}

Answer 1

I can see following differences and I have mention some suggestions.
Optional fields
Your data consist of array of blocks, in first block, there is only two fields text and type and in second block there is all other fields but no text. So that text is optional and it need to be omit when it is an empty struct. So you need to define your SlackRequestData with omitempty and pointer type to Text fields.
Suggested SlackRequestData struct is something like below.
type Text struct {
Type string `json:"type,omitempty"`
Text string `json:"text,omitempty"`
}
type SlackRequestData struct {
Blocks []struct {
BlockID string `json:"block_id,omitempty"`
Elements []struct {
ActionID string `json:"action_id,omitempty"`
Style string `json:"style,omitempty"`
Text *Text `json:"text,omitempty"`
Type string `json:"type,omitempty"`
Value string `json:"value,omitempty"`
} `json:"elements,omitempty"`
Text *Text `json:"text,omitempty"`
Type string `json:"type,omitempty"`
} `json:"blocks,omitempty"`
}
Marshalled Data Order
your file json data's field order and SlackRequestData's field order is different. So obviously that marshalled string and file string is different. json marshalling not guarantees that order when marshalling as mentiond in here. So if you need to compare those two, please sort those bytes in someway and compare.
Json Indent
In your file data is somewhat json beautified data and json marshal return compacted output. If you need to get same marshal output, you need to use MarshalIndent and do indent you used in file.
So if you need to compare both things, marshal with omitempty, sort both results (file byte array and json marshalled byte array). and remove whitespaces and then compare. You can get correct same result.