How to simplify marshaling of deeply-nested JSON - json

I have a JSON which contain some static and dynamic data. Below is a sample of JSON
{
"request": { /*Static data start */
"data": {
"object": { /*Static data ends here*/
"user": { /*Dynamic data start here */
"userid": "andmmdn",
"ipaddr": "1.1.1.1",
"noofusers": "100",
"qos": "34",
"id": "kldjflkdfjlkdjfkld",
"domain": "xyz.com" /*Dynamic data ends here */
}
}
}
}
}
Below is the code which can create this JSON
package main
import (
"fmt"
"encoding/json"
)
//ReqJSON struct
type ReqJSON struct {
Request Request `json:"request"`
}
//Request struct
type Request struct {
Data Data `json:"data"`
}
//Data struct
type Data struct {
Object Object `json:"object"`
}
//Object struct
type Object struct {
User User `json:"user"`
}
//User struct
type User struct {
UserID string `json:"userid"`
IPAddr string `json:"ipaddr"`
Noofusers string `json:"noofusers"`
Qos string `json:"qos"`
ID string `json:"id"`
Domain string `json:"domain"`
}
func main() {
test := ReqJSON {
Request{
Data: Data{
Object: Object{
User: User{
UserID: "andmmdn",
IPAddr: "1.1.1.1",
Noofusers: "100",
Qos: "34",
ID: "kldjflkdfjlkdjfkld",
Domain: "xyz.com",
},
},
},
},
}
jsonEncode, _ := json.Marshal(test)
jsonIdent, _ := json.MarshalIndent(&test, "", "\t")
fmt.Println(string(jsonEncode))
fmt.Println(string(jsonIdent))
}
As you can see from above it contain struct which doesn't make much sense as they act more like placeholder for nesting the data. So how we make it more optimized. As all the data is being taken care in the last struct. What approach for unmarshaling of the data should be applied as the response will be in same format and want to use the last struct for the same.
Any thoughts on the approach.
Also how to make a generic struct as I multiple API which uses same struct below is an example
//ReqJSON for populating data
type ReqJSON struct {
Request struct {
Data struct {
Object struct {
Auth Auth `json:"auth"`
} `json:"object"`
} `json:"data"`
} `json:"request"`
}
//ReqJSON for populating data
type ReqJSON struct {
Request struct {
Data struct {
Object struct {
Error Error `json:"error"`
} `json:"object"`
} `json:"data"`
} `json:"request"`
}

If you don't need the wrapping types for anything besides marshaling/unmarshaling, you can define them anonymously:
type ReqJSON struct {
Request struct {
Data struct {
Object struct {
User User `json:"user"`
} `json:"object"`
} `json:"data"`
} `json:"request"`
}
type User struct {
UserID string `json:"userid"`
IPAddr string `json:"ipaddr"`
Noofusers string `json:"noofusers"`
Qos string `json:"qos"`
ID string `json:"id"`
Domain string `json:"domain"`
}
And, borowing from icza's answer, you can add accessor methods to ReqJSON:
func (j *ReqJSON) User() User { return j.Request.Data.Object.User }
func (j *ReqJSON) SetUser(u User) { j.Request.Data.Object.User = u }
func main() {
var j ReqJSON
j.SetUser(User{
UserID: "_id",
IPAddr: "1.1.1.1",
Noofusers: "100",
Qos: "34",
ID: "kldjflkdfjlkdjfkld",
Domain: "xyz.com",
})
b, err := json.MarshalIndent(j, "", " ")
fmt.Println(err, string(b))
}

That sounds about right. The solution is a little verbose / redundant, but so is the data format you have to deal with.
To work with that easily, you may create helper functions and use them:
func wrap(u User) *ReqJSON {
return &ReqJSON{Request: Request{Data: Data{Object: Object{User: u}}}}
}
func unwrap(r *ReqJSON) User {
return r.Request.Data.Object.User
}
But other than that, you can't really simplify other things.
So marshaling a User is like:
var u User
data, err := json.Marshal(wrap(u))
Unmarshaling is:
var r *ReqJSON
err := json.Unmarshal(data, &r)
// Check error
u := unwrap(r) // Here we have the user

You can't eliminate the complexity, but you could potentially hide some of it inside of a custom Marshaler:
type Request struct {
UserID string `json:"userid"`
IPAddr string `json:"ipaddr"`
Noofusers string `json:"noofusers"`
Qos string `json:"qos"`
ID string `json:"id"`
Domain string `json:"domain"`
}
func (r *Request) MarshalJSON() ([]byte, error) {
type request struct {
Data struct {
Object struct {
User struct {
Request
} `json:"user"`
} `json:"object"`
} `json:"data"`
}
structure := request{Data: data{Object: object{User: user{r}}}}
return json.Marshal(structure)
}
The same approach can be employed in reverse for UnmarshalJSON, if desired.

Related

Trying to get 2 integers out of a JSON subsection

I'm using the VirusTotal HTTP API in Golang to get the votes on an URL, both malicious and harmless.
I want to get them using structs, and then unmarshaling the URL using data from these structs. But when I try that, this error shows up:
cannot convert "harmless" (untyped string constant) to int
The code I am currently using is:
type Votes struct {
harmless int
malicious int
}
type Attributes struct {
votes []Votes `json:"total_votes"`
}
type data struct {
attributes []Attributes `json:"attributes"`
}
type jsonstruct struct {
data []data `json:"data"`
}
var printdata jsonstruct
fmt.Println(resp)
json.Unmarshal([]byte(body), &printdata)
fmt.Println(printdata.data[0].attributes[0].votes["harmless"])
And the part of the JSON that I want to get is:
{
"data": {
"attributes": {
"last_modification_date": 1642534694,
"last_http_response_cookies": {
"1P_JAR": "2022-01-18-19",
"NID": "511=drq8-0Gwl0gpw2D-iyZhxrizpE--UMOyc_bO381XXkxknypvl_IETvsxRw3p8kMlBtiYEuSbASKK1wHirmgxce79kgzGMg9MryT0PnHox6kWbmEQTe2vsv_HtVZDFDXiLt4HKpcyDczOT8c_OK8bPb_P-f8rbIXJu_xrA0Ce4lw",
"SameSite": "none"
},
"times_submitted": 82289,
"total_votes": {
"harmless": 54,
"malicious": 18
},
As you can see, I want to get the contents of the subsection total_votes, which are integers harmless and malicious. In short, how can I get them without getting the error about them being untyped strings?
You need to define valid data structure that matches json structure. And after Unmarshal you can access votes fields as res.Data.Attributes.TotalVotes.Harmless . For example (https://go.dev/play/p/YCtV1u-KF7Y):
package main
import (
"encoding/json"
"fmt"
"log"
)
type Result struct {
Data struct {
Attributes struct {
LastHTTPResponseCookies struct {
OnePJAR string `json:"1P_JAR"`
Nid string `json:"NID"`
SameSite string `json:"SameSite"`
} `json:"last_http_response_cookies"`
LastModificationDate float64 `json:"last_modification_date"`
TimesSubmitted float64 `json:"times_submitted"`
TotalVotes struct {
Harmless float64 `json:"harmless"`
Malicious float64 `json:"malicious"`
} `json:"total_votes"`
} `json:"attributes"`
} `json:"data"`
}
var input = `{
"data": {
"attributes": {
"last_modification_date": 1642534694,
"last_http_response_cookies": {
"1P_JAR": "2022-01-18-19",
"NID": "pcyDczOT8c_OK8bPb_P-f8rbIXJu_xrA0Ce4lw",
"SameSite": "none"
},
"times_submitted": 82289,
"total_votes": {
"harmless": 54,
"malicious": 18
}
}
}
}
`
func main() {
var res Result
if err := json.Unmarshal([]byte(input), &res); err != nil {
log.Fatal(err)
}
fmt.Printf("%+v", res.Data.Attributes.TotalVotes.Harmless)
}

Golang json unmarshall

I'm new in Go. I have json like this:
{
"3415": {
"age": 25,
"name": "Tommy"
},
"3414": {
"age": 21,
"name": "Billy"
}
}
I want to unmarshall it to struct:
type People struct {
Id map[string]PeopleDetails
}
type PeopleDetails struct {
Age int `json:"age"`
Name string `json:"name"`
}
But while I run it, I see that struct return nil value.
I did read some tutorials, but most of them have predefined keys, as You see here "id" e.g. 3415 is different for every new json.
When you have to deal with a "dynamic" json key, the answer is use a map of struct.
You can use the following code:
package main
import (
"encoding/json"
"fmt"
)
// Use the struct pointed by #Adirio
type People map[string]PeopleDetails
type PeopleDetails struct {
Age int `json:"age"`
Name string `json:"name"`
}
var data string = `{"3415":{"age":25,"name":"Tommy"},"3414":{"age":21,"name":"Billy"}}`
func main() {
var p People
if err := json.Unmarshal([]byte(data), &p); err != nil {
fmt.Println(err)
}
fmt.Println(p)
}
GoPlayground: https://play.golang.org/p/kVzNV56NcTd
Try with these types instead:
type People map[string]PeopleDetails
type PeopleDetails struct {
Age int `json:"age"`
Name string `json:"name"`
}

Unmarshal JSON in go with different types in a list

I have trouble unmarschaling a JSON contruct:
{
"id": 10,
"result": [
{
"bundled": true,
"type": "RM-J1100"
},
[
{
"name": "PowerOff",
"value": "AAAAAQAAAAEAAAAvAw=="
},
{
"name": "Input",
"value": "AAAAAQAAAAEAAAAlAw=="
}
]
]
}
I actually need the second slice item from the result.
My current attempt is
type Codes struct {
Id int32 `json:"id"`
Result []interface{} `json:"result"`
}
type ResultList struct {
Info InfoMap
Codes []Code
}
type InfoMap struct {
Bundled bool `json:"bundled"`
Type string `json:"type"`
}
type Code struct {
Name string `json:"name"`
Value string `json:"value"`
}
the output is like:
{10 {{false } []}}
but I also tried to use this:
type Codes struct {
Id int32 `json:"id"`
Result []interface{} `json:"result"`
}
the output is okay:
{10 [map[type:RM-J1100 bundled:true] [map[name:PowerOff value:AAAAAQAAAAEAAAAvAw==] map[name:Input value:AAAAAQAAAAEAAAAlAw==]]]}
I can also reference the Result[1] index:
[map[name:PowerOff value:AAAAAQAAAAEAAAAvAw==] map[name:Input value:AAAAAQAAAAEAAAAlAw==]]
But I fail to convert the interface type to any other Type which would match. Can anybody tell me how to do interface conversion. And what approach would be the "best".
One option would be to unmarshal the top level thing into a slice of json.RawMessage initially.
Then loop through the members, and look at the first character of each one. If it is an object, unmarshal that into your InfoMap header struct, and if it is an array, unmarshal it into a slice of the Code struct.
Or if it is predictable enough, just unmarshal the first member to the one struct and the second to a slice.
I made a playground example of this approach.
type Response struct {
ID int `json:"id"`
RawResult []json.RawMessage `json:"result"`
Header *Header `json:"-"`
Values []*Value `json:"-"`
}
type Header struct {
Bundled bool `json:"bundled"`
Type string `json:"type"`
}
type Value struct {
Name string `json:"name"`
Value string `json:"value"`
}
func main() {
//error checks ommitted
resp := &Response{}
json.Unmarshal(rawJ, resp)
resp.Header = &Header{}
json.Unmarshal(resp.RawResult[0], resp.Header)
resp.Values = []*Value{}
json.Unmarshal(resp.RawResult[1], &resp.Values)
}
(I will not point out how horrific it is this JSON struct, but as always: sXXt happens)
You can convert your struct like this, by using a cycle of JSON Marshal / Unmarshal. Code follows:
package main
import (
"encoding/json"
"log"
)
const (
inputJSON = `{
"id": 10,
"result": [
{
"bundled": true,
"type": "RM-J1100"
},
[
{
"name": "PowerOff",
"value": "AAAAAQAAAAEAAAAvAw=="
},
{
"name": "Input",
"value": "AAAAAQAAAAEAAAAlAw=="
}
]
]
}`
)
type Codes struct {
Id int32 `json:"id"`
Result [2]interface{} `json:"result"`
}
type Result struct {
Info InfoMap
Codes []Code
}
type InfoMap struct {
Bundled bool `json:"bundled"`
Type string `json:"type"`
}
type Code struct {
Name string `json:"name"`
Value string `json:"value"`
}
func main() {
newCodes := &Codes{}
err := json.Unmarshal([]byte(inputJSON), newCodes)
if err != nil {
log.Fatal(err)
}
// Prints the whole object
log.Println(newCodes)
// Prints the Result array (!)
log.Println(newCodes.Result)
if len(newCodes.Result) != 2 {
log.Fatal("Invalid Result struct")
}
// Marshal and Unmarshal data to obtain the code list
byteCodeList, _ := json.Marshal(newCodes.Result[1])
codeList := make([]Code, 0)
err = json.Unmarshal(byteCodeList, &codeList)
if err != nil {
log.Fatal("Invalid Code list")
}
// Prints the codeList
log.Println(codeList)
}
Test it on playground.

What will be the structure representation of the following json data in golang?

{
"devices": [
{
"id": 20081691,
"targetIp": "10.1.1.1",
"iops": "0.25 IOPS per GB",
"capacity": 20,
"allowedVirtualGuests": [
{
"Name": "akhil1"
},
{
"Name": "akhil2"
}
]
}
]
}
How to write a structure representation of this JSON data so that I can add and delete devices to the list. I tried with the different structure representations but nothing is working. Below is one of the example I have tried with a similar kind of json data. I am not able to add new data to it. The structure representation and the way the append is done might be wrong here
package main
import (
"encoding/json"
"fmt"
)
type Person struct {
ID string `json:"id,omitempty"`
Firstname string `json:"firstname,omitempty"`
Lastname string `json:"lastname,omitempty"`
Address []Address `json:"address,omitempty"`
}
type Address[] struct {
City string `json:"city,omitempty"`
}
func main() {
var people []Person
people = append(people, Person{ID: "1", Firstname: "Nic", Lastname: "Raboy", Address: []Address{{City: "Dublin"},{ City: "CA"}}} )
b, err := json.Marshal(people)
if err != nil {
fmt.Println("json err:", err)
}
fmt.Println(string(b))
}
It will be below. This was generated using the excellent JSON-to-GO tool:
type MyStruct struct {
Devices []struct {
ID int `json:"id"`
TargetIP string `json:"targetIp"`
Iops string `json:"iops"`
Capacity int `json:"capacity"`
AllowedVirtualGuests []struct {
Name string `json:"Name"`
} `json:"allowedVirtualGuests"`
} `json:"devices"`
}
To simplify that though, you can break it into smaller structs for readability. An example is below:
package main
import "fmt"
type VirtualGuest struct {
Name string `json:"Name"`
}
type Device struct {
ID int `json:"id"`
TargetIP string `json:"targetIp"`
Iops string `json:"iops"`
Capacity int `json:"capacity"`
AllowedVirtualGuests []VirtualGuest `json:"allowedVirtualGuests"`
}
type MyStruct struct {
Devices []Device `json:"devices"`
}
func main() {
var myStruct MyStruct
// Add a MyStruct
myStruct.Devices = append(myStruct.Devices, Device{
ID:1,
TargetIP:"1.2.3.4",
Iops:"iops",
Capacity:1,
AllowedVirtualGuests:[]VirtualGuest{
VirtualGuest{
Name:"guest 1",
},
VirtualGuest{
Name:"guest 2",
},
},
})
fmt.Printf("MyStruct: %v\n", myStruct)
}
you can use struct tag like json:"id", try struct below:
type Data struct {
Devices []struct {
Id int `json:"id"`
IP string `json:"targetIp"`
IOPS string `json:"iops"`
Capacity int `json:"capacity"`
AllowedVirtualGuests []struct {
Name string `json:"Name"`
} `json:"allowedVirtualGuests"`
} `json:"devices"`
}

Mapping JSON returned by REST API containing dynamic keys to a struct in Golang

I'm calling a REST API from my Go program which takes n number of hotel ids in the request and returns their data as a JSON. The response is like the following when say I pass 2 ids in the request, 1018089108070373346 and 2017089208070373346 :
{
"data": {
"1018089108070373346": {
"name": "A Nice Hotel",
"success": true
},
"2017089208070373346": {
"name": "Another Nice Hotel",
"success": true
}
}
}
Since I'm new to Golang I using a JSON Go tool available at http://mholt.github.io/json-to-go/ to get the struct representation for the above response. What I get is:
type Autogenerated struct {
Data struct {
Num1017089108070373346 struct {
Name string `json:"name"`
Success bool `json:"success"`
} `json:"1017089108070373346"`
Num2017089208070373346 struct {
Name string `json:"name"`
Success bool `json:"success"`
} `json:"2017089208070373346"`
} `json:"data"`
}
I cannot use the above struct because the actual id values and the number of ids I pass can be different each time, the JSON returned will have different keys. How can this situation be mapped to a struct ?
Thanks
Use a map:
type Item struct {
Name string `json:"name"`
Success bool `json:"success"`
}
type Response struct {
Data map[string]Item `json:"data"`
}
Run it on the playground
Here is some sample code that utilizes Mellow Marmots answer and shows how to iterate over the items in the response.
test.json
{
"data": {
"1018089108070373346": {
"name": "A Nice Hotel",
"success": true
},
"2017089208070373346": {
"name": "Another Nice Hotel",
"success": true
}
}
}
test.go
package main
import (
"encoding/json"
"fmt"
"os"
)
// Item struct
type Item struct {
Name string `json:"name"`
Success bool `json:"success"`
}
// Response struct
type Response struct {
Data map[string]Item `json:"data"`
}
func main() {
jsonFile, err := os.Open("test.json")
if err != nil {
fmt.Println("Error opening test file\n", err.Error())
return
}
jsonParser := json.NewDecoder(jsonFile)
var filedata Response
if err = jsonParser.Decode(&filedata); err != nil {
fmt.Println("Error while reading test file.\n", err.Error())
return
}
for key, value := range filedata.Data {
fmt.Println(key, value.Name, value.Success)
}
}
Which outputs:
1018089108070373346 A Nice Hotel true
2017089208070373346 Another Nice Hotel true