Trying to get 2 integers out of a JSON subsection - json

I'm using the VirusTotal HTTP API in Golang to get the votes on an URL, both malicious and harmless.
I want to get them using structs, and then unmarshaling the URL using data from these structs. But when I try that, this error shows up:
cannot convert "harmless" (untyped string constant) to int
The code I am currently using is:
type Votes struct {
harmless int
malicious int
}
type Attributes struct {
votes []Votes `json:"total_votes"`
}
type data struct {
attributes []Attributes `json:"attributes"`
}
type jsonstruct struct {
data []data `json:"data"`
}
var printdata jsonstruct
fmt.Println(resp)
json.Unmarshal([]byte(body), &printdata)
fmt.Println(printdata.data[0].attributes[0].votes["harmless"])
And the part of the JSON that I want to get is:
{
"data": {
"attributes": {
"last_modification_date": 1642534694,
"last_http_response_cookies": {
"1P_JAR": "2022-01-18-19",
"NID": "511=drq8-0Gwl0gpw2D-iyZhxrizpE--UMOyc_bO381XXkxknypvl_IETvsxRw3p8kMlBtiYEuSbASKK1wHirmgxce79kgzGMg9MryT0PnHox6kWbmEQTe2vsv_HtVZDFDXiLt4HKpcyDczOT8c_OK8bPb_P-f8rbIXJu_xrA0Ce4lw",
"SameSite": "none"
},
"times_submitted": 82289,
"total_votes": {
"harmless": 54,
"malicious": 18
},
As you can see, I want to get the contents of the subsection total_votes, which are integers harmless and malicious. In short, how can I get them without getting the error about them being untyped strings?

You need to define valid data structure that matches json structure. And after Unmarshal you can access votes fields as res.Data.Attributes.TotalVotes.Harmless . For example (https://go.dev/play/p/YCtV1u-KF7Y):
package main
import (
"encoding/json"
"fmt"
"log"
)
type Result struct {
Data struct {
Attributes struct {
LastHTTPResponseCookies struct {
OnePJAR string `json:"1P_JAR"`
Nid string `json:"NID"`
SameSite string `json:"SameSite"`
} `json:"last_http_response_cookies"`
LastModificationDate float64 `json:"last_modification_date"`
TimesSubmitted float64 `json:"times_submitted"`
TotalVotes struct {
Harmless float64 `json:"harmless"`
Malicious float64 `json:"malicious"`
} `json:"total_votes"`
} `json:"attributes"`
} `json:"data"`
}
var input = `{
"data": {
"attributes": {
"last_modification_date": 1642534694,
"last_http_response_cookies": {
"1P_JAR": "2022-01-18-19",
"NID": "pcyDczOT8c_OK8bPb_P-f8rbIXJu_xrA0Ce4lw",
"SameSite": "none"
},
"times_submitted": 82289,
"total_votes": {
"harmless": 54,
"malicious": 18
}
}
}
}
`
func main() {
var res Result
if err := json.Unmarshal([]byte(input), &res); err != nil {
log.Fatal(err)
}
fmt.Printf("%+v", res.Data.Attributes.TotalVotes.Harmless)
}

Related

How to simplify marshaling of deeply-nested JSON

I have a JSON which contain some static and dynamic data. Below is a sample of JSON
{
"request": { /*Static data start */
"data": {
"object": { /*Static data ends here*/
"user": { /*Dynamic data start here */
"userid": "andmmdn",
"ipaddr": "1.1.1.1",
"noofusers": "100",
"qos": "34",
"id": "kldjflkdfjlkdjfkld",
"domain": "xyz.com" /*Dynamic data ends here */
}
}
}
}
}
Below is the code which can create this JSON
package main
import (
"fmt"
"encoding/json"
)
//ReqJSON struct
type ReqJSON struct {
Request Request `json:"request"`
}
//Request struct
type Request struct {
Data Data `json:"data"`
}
//Data struct
type Data struct {
Object Object `json:"object"`
}
//Object struct
type Object struct {
User User `json:"user"`
}
//User struct
type User struct {
UserID string `json:"userid"`
IPAddr string `json:"ipaddr"`
Noofusers string `json:"noofusers"`
Qos string `json:"qos"`
ID string `json:"id"`
Domain string `json:"domain"`
}
func main() {
test := ReqJSON {
Request{
Data: Data{
Object: Object{
User: User{
UserID: "andmmdn",
IPAddr: "1.1.1.1",
Noofusers: "100",
Qos: "34",
ID: "kldjflkdfjlkdjfkld",
Domain: "xyz.com",
},
},
},
},
}
jsonEncode, _ := json.Marshal(test)
jsonIdent, _ := json.MarshalIndent(&test, "", "\t")
fmt.Println(string(jsonEncode))
fmt.Println(string(jsonIdent))
}
As you can see from above it contain struct which doesn't make much sense as they act more like placeholder for nesting the data. So how we make it more optimized. As all the data is being taken care in the last struct. What approach for unmarshaling of the data should be applied as the response will be in same format and want to use the last struct for the same.
Any thoughts on the approach.
Also how to make a generic struct as I multiple API which uses same struct below is an example
//ReqJSON for populating data
type ReqJSON struct {
Request struct {
Data struct {
Object struct {
Auth Auth `json:"auth"`
} `json:"object"`
} `json:"data"`
} `json:"request"`
}
//ReqJSON for populating data
type ReqJSON struct {
Request struct {
Data struct {
Object struct {
Error Error `json:"error"`
} `json:"object"`
} `json:"data"`
} `json:"request"`
}
If you don't need the wrapping types for anything besides marshaling/unmarshaling, you can define them anonymously:
type ReqJSON struct {
Request struct {
Data struct {
Object struct {
User User `json:"user"`
} `json:"object"`
} `json:"data"`
} `json:"request"`
}
type User struct {
UserID string `json:"userid"`
IPAddr string `json:"ipaddr"`
Noofusers string `json:"noofusers"`
Qos string `json:"qos"`
ID string `json:"id"`
Domain string `json:"domain"`
}
And, borowing from icza's answer, you can add accessor methods to ReqJSON:
func (j *ReqJSON) User() User { return j.Request.Data.Object.User }
func (j *ReqJSON) SetUser(u User) { j.Request.Data.Object.User = u }
func main() {
var j ReqJSON
j.SetUser(User{
UserID: "_id",
IPAddr: "1.1.1.1",
Noofusers: "100",
Qos: "34",
ID: "kldjflkdfjlkdjfkld",
Domain: "xyz.com",
})
b, err := json.MarshalIndent(j, "", " ")
fmt.Println(err, string(b))
}
That sounds about right. The solution is a little verbose / redundant, but so is the data format you have to deal with.
To work with that easily, you may create helper functions and use them:
func wrap(u User) *ReqJSON {
return &ReqJSON{Request: Request{Data: Data{Object: Object{User: u}}}}
}
func unwrap(r *ReqJSON) User {
return r.Request.Data.Object.User
}
But other than that, you can't really simplify other things.
So marshaling a User is like:
var u User
data, err := json.Marshal(wrap(u))
Unmarshaling is:
var r *ReqJSON
err := json.Unmarshal(data, &r)
// Check error
u := unwrap(r) // Here we have the user
You can't eliminate the complexity, but you could potentially hide some of it inside of a custom Marshaler:
type Request struct {
UserID string `json:"userid"`
IPAddr string `json:"ipaddr"`
Noofusers string `json:"noofusers"`
Qos string `json:"qos"`
ID string `json:"id"`
Domain string `json:"domain"`
}
func (r *Request) MarshalJSON() ([]byte, error) {
type request struct {
Data struct {
Object struct {
User struct {
Request
} `json:"user"`
} `json:"object"`
} `json:"data"`
}
structure := request{Data: data{Object: object{User: user{r}}}}
return json.Marshal(structure)
}
The same approach can be employed in reverse for UnmarshalJSON, if desired.

Unmarshal with changing JSON attribute

I have a JSON object whose details can contain different types of JSON objects, the rest of the JSON remains the same, in such a case how can I have a single struct in Golang to handle both types of JSON
JSON 1:
{
"field1":"",
"field2":"",
"field3":"",
"field4":"",
"field5":"",
"field6":"",
"field7":"",
"details":{
"detail1":"",
"detail2":[
{
"arr1":"",
"arr2":{
"id":"",
"name":""
},
"list":[
{
"id":"",
"version":1,
"name":""
}
]
}
]
},
"user":{
"id":"",
"name":""
}
}
JSON 2:
{
"field1":"",
"field2":"",
"field3":"",
"field4":"",
"field5":"",
"field6":"",
"field7":"",
"details":{
"anotherdetail1":"",
"anotherdetail2":[
{
"arr7":"",
"arr8":{
"id":"",
"name":""
},
"arr10":{
}
}
]
},
"user":{
"id":"",
"name":""
}
}
My goal is to use a single struct for both these JSON objects. In a language like Java I would create a Parent Class which resembles details in a generic way and have 2 child classes to resemble the type of details that vary and during runtime I would create an object of a child type and assign it to the Parent. I am unsure how this is done in Go.
I am not sure you can have a single struct unless you are ok with a string interface map, but You can prevent the details from being decoded by setting them as a json.RawMessage type int he struct. You can then decode the unknown-typed json data by attempting to decode it into one type, if that returns an error then you try with the next type.
Here is some code, that should give you a better idea to what I am talking about.
https://play.golang.org/p/06owmiJXNaO
package main
import (
"encoding/json"
"fmt"
)
const json1 = `{"name": "foo", "details":[1, 2, 3]}`
const json2 = `{"name": "foo", "details":{"a": [1, 2, 3]}}`
type data struct {
Name string `json:"name"`
Details json.RawMessage `json:"details"`
}
type detailsone []int
type detailstwo struct {
A []int `json:"a"`
}
func main() {
var d1, d2 data
json.Unmarshal([]byte(json1), &d1)
json.Unmarshal([]byte(json2), &d2)
fmt.Printf("%+v\n", d1)
fmt.Printf("%+v\n", d2)
var err error
var b1 detailsone
var b2 detailstwo
// json1
err = json.Unmarshal([]byte(d1.Details), &b1)
if err == nil {
fmt.Printf("d1 is an []int: %+v\n", b1)
}
err = json.Unmarshal([]byte(d1.Details), &b2)
if err == nil {
fmt.Printf("d1 is an detailstwo struct: %+v\n", b2)
}
// json2
err = json.Unmarshal([]byte(d2.Details), &b1)
if err == nil {
fmt.Printf("d2 is an []int: %+v\n", b1)
}
err = json.Unmarshal([]byte(d2.Details), &b2)
if err == nil {
fmt.Printf("d2 is an detailstwo struct: %+v\n", b2)
}
}
type Base struct {
Data map[string]interface{}
Details struct {
*D1
*D2
} `json:"details"
}
type D1 struct {
Detail1 string
Detail2 string
}
type D2 struct {
AnotherDetail1 string
AnotherDetail2 string
}
you can find filled struct by comparing them with nil

Unmarshal JSON in go with different types in a list

I have trouble unmarschaling a JSON contruct:
{
"id": 10,
"result": [
{
"bundled": true,
"type": "RM-J1100"
},
[
{
"name": "PowerOff",
"value": "AAAAAQAAAAEAAAAvAw=="
},
{
"name": "Input",
"value": "AAAAAQAAAAEAAAAlAw=="
}
]
]
}
I actually need the second slice item from the result.
My current attempt is
type Codes struct {
Id int32 `json:"id"`
Result []interface{} `json:"result"`
}
type ResultList struct {
Info InfoMap
Codes []Code
}
type InfoMap struct {
Bundled bool `json:"bundled"`
Type string `json:"type"`
}
type Code struct {
Name string `json:"name"`
Value string `json:"value"`
}
the output is like:
{10 {{false } []}}
but I also tried to use this:
type Codes struct {
Id int32 `json:"id"`
Result []interface{} `json:"result"`
}
the output is okay:
{10 [map[type:RM-J1100 bundled:true] [map[name:PowerOff value:AAAAAQAAAAEAAAAvAw==] map[name:Input value:AAAAAQAAAAEAAAAlAw==]]]}
I can also reference the Result[1] index:
[map[name:PowerOff value:AAAAAQAAAAEAAAAvAw==] map[name:Input value:AAAAAQAAAAEAAAAlAw==]]
But I fail to convert the interface type to any other Type which would match. Can anybody tell me how to do interface conversion. And what approach would be the "best".
One option would be to unmarshal the top level thing into a slice of json.RawMessage initially.
Then loop through the members, and look at the first character of each one. If it is an object, unmarshal that into your InfoMap header struct, and if it is an array, unmarshal it into a slice of the Code struct.
Or if it is predictable enough, just unmarshal the first member to the one struct and the second to a slice.
I made a playground example of this approach.
type Response struct {
ID int `json:"id"`
RawResult []json.RawMessage `json:"result"`
Header *Header `json:"-"`
Values []*Value `json:"-"`
}
type Header struct {
Bundled bool `json:"bundled"`
Type string `json:"type"`
}
type Value struct {
Name string `json:"name"`
Value string `json:"value"`
}
func main() {
//error checks ommitted
resp := &Response{}
json.Unmarshal(rawJ, resp)
resp.Header = &Header{}
json.Unmarshal(resp.RawResult[0], resp.Header)
resp.Values = []*Value{}
json.Unmarshal(resp.RawResult[1], &resp.Values)
}
(I will not point out how horrific it is this JSON struct, but as always: sXXt happens)
You can convert your struct like this, by using a cycle of JSON Marshal / Unmarshal. Code follows:
package main
import (
"encoding/json"
"log"
)
const (
inputJSON = `{
"id": 10,
"result": [
{
"bundled": true,
"type": "RM-J1100"
},
[
{
"name": "PowerOff",
"value": "AAAAAQAAAAEAAAAvAw=="
},
{
"name": "Input",
"value": "AAAAAQAAAAEAAAAlAw=="
}
]
]
}`
)
type Codes struct {
Id int32 `json:"id"`
Result [2]interface{} `json:"result"`
}
type Result struct {
Info InfoMap
Codes []Code
}
type InfoMap struct {
Bundled bool `json:"bundled"`
Type string `json:"type"`
}
type Code struct {
Name string `json:"name"`
Value string `json:"value"`
}
func main() {
newCodes := &Codes{}
err := json.Unmarshal([]byte(inputJSON), newCodes)
if err != nil {
log.Fatal(err)
}
// Prints the whole object
log.Println(newCodes)
// Prints the Result array (!)
log.Println(newCodes.Result)
if len(newCodes.Result) != 2 {
log.Fatal("Invalid Result struct")
}
// Marshal and Unmarshal data to obtain the code list
byteCodeList, _ := json.Marshal(newCodes.Result[1])
codeList := make([]Code, 0)
err = json.Unmarshal(byteCodeList, &codeList)
if err != nil {
log.Fatal("Invalid Code list")
}
// Prints the codeList
log.Println(codeList)
}
Test it on playground.

Mapping JSON returned by REST API containing dynamic keys to a struct in Golang

I'm calling a REST API from my Go program which takes n number of hotel ids in the request and returns their data as a JSON. The response is like the following when say I pass 2 ids in the request, 1018089108070373346 and 2017089208070373346 :
{
"data": {
"1018089108070373346": {
"name": "A Nice Hotel",
"success": true
},
"2017089208070373346": {
"name": "Another Nice Hotel",
"success": true
}
}
}
Since I'm new to Golang I using a JSON Go tool available at http://mholt.github.io/json-to-go/ to get the struct representation for the above response. What I get is:
type Autogenerated struct {
Data struct {
Num1017089108070373346 struct {
Name string `json:"name"`
Success bool `json:"success"`
} `json:"1017089108070373346"`
Num2017089208070373346 struct {
Name string `json:"name"`
Success bool `json:"success"`
} `json:"2017089208070373346"`
} `json:"data"`
}
I cannot use the above struct because the actual id values and the number of ids I pass can be different each time, the JSON returned will have different keys. How can this situation be mapped to a struct ?
Thanks
Use a map:
type Item struct {
Name string `json:"name"`
Success bool `json:"success"`
}
type Response struct {
Data map[string]Item `json:"data"`
}
Run it on the playground
Here is some sample code that utilizes Mellow Marmots answer and shows how to iterate over the items in the response.
test.json
{
"data": {
"1018089108070373346": {
"name": "A Nice Hotel",
"success": true
},
"2017089208070373346": {
"name": "Another Nice Hotel",
"success": true
}
}
}
test.go
package main
import (
"encoding/json"
"fmt"
"os"
)
// Item struct
type Item struct {
Name string `json:"name"`
Success bool `json:"success"`
}
// Response struct
type Response struct {
Data map[string]Item `json:"data"`
}
func main() {
jsonFile, err := os.Open("test.json")
if err != nil {
fmt.Println("Error opening test file\n", err.Error())
return
}
jsonParser := json.NewDecoder(jsonFile)
var filedata Response
if err = jsonParser.Decode(&filedata); err != nil {
fmt.Println("Error while reading test file.\n", err.Error())
return
}
for key, value := range filedata.Data {
fmt.Println(key, value.Name, value.Success)
}
}
Which outputs:
1018089108070373346 A Nice Hotel true
2017089208070373346 Another Nice Hotel true

How to unmarshal two json with same internal structure into one single golang struct?

I have two json files with following structure
{
"cast": [
{
"url": "carey-mulligan",
"name": "Carey Mulligan",
"role": "Actress"
},
{
"url": "leonardo-dicaprio",
"name": "Leonardo DiCaprio",
"role": "Actor"
},
.
.
.
]
}
and
{
"movie": [
{
"url": "carey-mulligan",
"name": "Carey Mulligan",
"role": "Actress"
},
{
"url": "leonardo-dicaprio",
"name": "Leonardo DiCaprio",
"role": "Actor"
},
.
.
.
]
}
as you can see internal structure of the json is same for cast and movie. I want to unmarshel these json file into the same golang structure. But i am not able to give two name tags (cast and movie) for same struct element. I want something like
type Detail struct {
Name string `json:"name"`
Url string `json:"url"`
Role string `json:"role"`
}
type Info struct {
Detail []Detail `json:"cast or movie"`
}
In which case Detail could parse both cast and movie.
Here is my current code
// RIMAGE project main.go
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
)
const (
website = "https://data.moviebuff.com/"
)
func main() {
fmt.Println("Hello World!")
content, err := ioutil.ReadFile("data/great-getsby")
if err != nil {
panic(err)
}
var info Info
err = json.Unmarshal(content, &info)
if err != nil {
panic(err)
}
fmt.Println(info.Detail)
}
type Detail struct {
Name string `json:"name"`
Url string `json:"url"`
Role string `json:"role"`
}
type Info struct {
Detail []Detail `json:"cast" json:"movie"
}
but it only works for first tag "cast" and gives nill in case json contain the movie.
Thanks in advance.
You can use type Info map[string][]Detail instead of your struct.
Try it on the Go playground
Or you can use both types in your structure, and make method Details() which will return right one:
type Info struct {
CastDetails []Detail `json:"cast"`
MovieDetails []Detail `json:"movie"`
}
func (i Info) Details() []Detail {
if i.CastDetails == nil {
return i.MovieDetails
}
return i.CastDetails
}
Try it on the Go playground
Try anonymous field in struct:
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
)
type Detail struct {
Name string `json:"name"`
Url string `json:"url"`
Role string `json:"role"`
}
type Cast struct {
Detail []Detail `json:"cast"`
}
type Movie struct {
Detail []Detail `json:"movie"`
}
type Info struct {
Cast
Movie
}
func (i *Info) getDetails() []Detail {
if len(i.Cast.Detail) > 0 {
return i.Cast.Detail
}
return i.Movie.Detail
}
func main() {
cast, _ := ioutil.ReadFile("./cast.json")
movie, _ := ioutil.ReadFile("./movie.json")
var cInfo Info
err := json.Unmarshal(cast, &cInfo)
fmt.Printf("cast: %+v\n", &cInfo)
fmt.Printf("err: %v\n", err)
fmt.Printf("details: %v\n", cInfo.getDetails())
var mInfo Info
err = json.Unmarshal(movie, &mInfo)
fmt.Printf("movie: %+v\n", &mInfo)
fmt.Printf("err: %v\n", err)
fmt.Printf("details: %v\n", mInfo.getDetails())
}
Things to note:
One more level of indirection: to access 'Details' field, you need to access either 'Cast' or 'Movie' field first in Info first.
Better provide an access function for 'Details' ('getDetail' in this example)
If you dig far enough down into encoding/json you'll get to https://github.com/golang/go/blob/master/src/encoding/json/encode.go and the following:
tag := sf.Tag.Get("json")
if tag == "-" {
continue
}
So it gets one json tag and keeps on going.
You could always go with RoninDev's solution and just copy it over when done.