Extracting Elements from JSON - json

I have a function that returns JSON, and the return type is this:
map[string]interface{}
In my particular case, I know that the returned JSON looks like this:
{
"someNumber": 1,
"someString": "ee",
"someArray": [
"stringInArray",
{
"anotherNumber": 100,
"anotherString": "asdf",
"yetAnotherString": "qwer"
}
]
}
I want to get the value of "stringInArray" and also "anotherString". I've searched for solutions for example from Go by Example and blog.golang.org but I've not been successful.
For example, given that res is the json returned from the function call, I tried this (from the go blog):
var f interface{}
b := []byte(`res` )
err2 := json.Unmarshal(b, &f)
if err2!=nil{
log.Fatalf("Err unmarshalling: %v", err2)
}
m := f.( map[string]interface{} )
for k, v := range m {
switch vv := v.(type) {
case string:
fmt.Println(k, "is string", vv)
case int:
fmt.Println(k, "is int", vv)
case []interface{}:
fmt.Println(k, "is an array:")
for i, u := range vv {
fmt.Println(i, u)
}
default:
fmt.Println(k, "is of a type I don't know how to handle")
}
}
(I know the above would not do exactly what I want but it is a start.)
But when code execution hits b := []byte(res ), I get this error:
Err unmarshalling: invalid character 'r' looking for beginning of value
What is the most efficient method / best practice for obtaining the values? I'm open to any solution, not necessarily the approach above.
#sydnash Here is the code I promised in my response to your comment:
type LoginResponse2 struct {
id float64
jsonrpc string
result struct {
token string
details struct {
a float64
b string
c float64
}
}
}
var resStruct2 LoginResponse2
// Convert result to json
b, _ :=json.Marshal(res)
fmt.Println( string(b) )
// results of Println:
{"id":1,"jsonrpc":"2.0","result":[ "myToken",{"a":someNumber,"b":"some string","c":someOtherNumber} ] }
// Unmarshall json into struct
err2 := json.Unmarshal(b, &resStruct2)
if err2 != nil {
log.Fatalf("Error unmarshalling json: %v", err)
}
fmt.Println("Here is json unmarshalled into a struct")
fmt.Println( resStruct2 )
// results of Println
{0 { {0 0}}}

I think you should use b := []byte(res) instead b :=[]byte[res] and res should be a json string or []byte. res is not a legal json format.
this information maybe help you:
To unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
you can see there is no int but float64 for JSON numbers.

There is an example of how to unmarshal json in the docs on golang packages encoding/json. Essentially your problem is that the un-marshal expects to be putting the json into type. So your code must declare a type that represents the json you are converting from.
Example of the json and the declared type from the linked example:
var jsonBlob = []byte(`[
{"Name": "Platypus", "Order": "Monotremata"},
{"Name": "Quoll", "Order": "Dasyuromorphia"}
]`)
type Animal struct {
Name string
Order string
}
Once you define the type it should all fall into place.

Related

How do you read nested Json in Go?

I have the following JSON to be decoded, its structure can vary therefore I do not want to use structs:
{ "cabinet": "A", "shelve": {"box": "10", "color": "red"} }
Following the Golang blog (https://blog.golang.org/json) I have prepared this program to parse it:
import (
"fmt"
"encoding/json"
)
func main() {
fmt.Println("Hello, playground")
respString := `{ "cabinet": "A", "shelve": {"box": "10", "color": "red"} }`
respBytes := []byte(respString)
var f interface{}
json.Unmarshal(respBytes, &f)
m := f.(map[string]interface{})
for k, v := range m {
switch vv := v.(type) {
case string:
fmt.Println(k, "is string", vv)
case float64:
fmt.Println(k, "is float64", vv)
case []interface{}:
fmt.Println(k, "is an array:")
for i, u := range vv {
fmt.Println(i, u)
}
default:
fmt.Println(k, "is of a type I don't know how to handle")
}
}
}
However, I am wondering how can I access the nested JSON embbedded as value in "shelve".
So far this is the output:
cabinet is string A
shelve is of a type I don't know how to handle
What shall be done to access inner key/values at shelve? Which strategy is appropriate in Go?
A complete executable code can be found in https://play.golang.org/p/AVMMVQVjY__B
shelve in JSON is an object, so a Go map will be created to model it:
case map[string]interface{}:
fmt.Println(k, "is a map:")
for k, v := range vv {
fmt.Println("\t", k, "=", v)
}
With this change output is (try it on the Go Playground):
cabinet is string A
shelve is a map:
color = red
box = 10
See related question:
Accessing Nested Map of Type map[string]interface{} in Golang
Is there any convenient way to get JSON element without type assertion?
Taking a JSON string, unmarshaling it into a map[string]interface{}, editing, and marshaling it into a []byte seems more complicated then it should be

Does json.Unmarshal require your result structure to match exactly the JSON passed in?

I have a JSON string I want to unmarshal:
{
"id":1720,
"alertId":1,
"alertName":"{stats} Test Lambda Alert",
"dashboardId":5,
"panelId":2,
"userId":0,
"newState":"alerting",
"prevState":"ok",
"time":1523983581000,
"text":"",
"regionId":0,
"tags":[],
"login":"",
"email":"",
"avatarUrl":"",
"data":{
"evalMatches":[
{
"metric":"{prod}{stats} Lambda Alert Test",
"tags":null,
"value":16.525333333333332
}
]
}
}
I get the raw stream via a request: bodyBytes, err := ioutil.ReadAll(resp.Body)
I was hoping I could just specify a struct that pulls the values I care about, e.g.,
type Result struct {
ID string `json:"id"`
Time int64 `json:"time"`
}
However, when I try this, I get errors.
type Result struct {
ID string `json:"id"`
Time string `json:"time"`
}
var result Result
err2 := json.Unmarshal(bodyBytes, &result)
if err2 != nil {
log.Fatal(fmt.Sprintf(`Error Unmarshalling: %s`, err2))
}
fmt.Println(result.ID)
Error Unmarshalling: json: cannot unmarshal array into Go value of type main.Result
I suspect this error may be due to what's actually returned from ioutil.ReadAll(), since it has the above JSON string wrapped in [ ] if I do a fmt.Println(string(bodyBytes)), but if I try to json.Unmarshal(bodyBytes[0], &result), I just get compile errors, so I'm not sure.
If I want to unmarshal a JSON string, do I have to specify the full structure in my type Result struct? Is there a way around this? I don't want to be bound to the JSON object I receive (if the API changes upstream, it requires us to modify our code to recognize that, etc.).
You can unmarshal into structs that represent only some fields of your JSON document, but the field types have to match, as the error clearly states:
cannot unmarshal number into Go struct field Result.id of type string
You cannot unmarshal a number into a string. If you define the ID field as any numeric type it'll work just fine:
package main
import (
"encoding/json"
"fmt"
)
var j = []byte(`
{
"id":1720,
"prevState":"ok",
"time":1523983581000,
"text":"",
"regionId":0
}
`)
type Result struct {
ID int `json:"id"` // or any other integer type, or float{32,64}, or json.Number
Time int64 `json:"time"`
}
func main() {
var r Result
err := json.Unmarshal(j, &r)
fmt.Println(r, err)
}
Try it on the playground: https://play.golang.org/p/lqsQwLW2dHZ
Update
You have just edited your question with the actual error you receive. You have to unmarshal JSON arrays into slices. So if the HTTP response in fact returns a JSON array, unmarshal into []Result:
var j = []byte(`
[
{
"id":1720,
"prevState":"ok",
"time":1523983581000,
"text":"",
"regionId":0
}
]
`)
var r []Result
err := json.Unmarshal(j, &r)
fmt.Println(r[0], err)
https://play.golang.org/p/EbOVA8CbcFO
To generate Go types that match your JSON document pretty well, use https://mholt.github.io/json-to-go/.

Is there a way to have json.Unmarshal() select struct type based on "type" property?

I have some JSON of the form:
[{
"type": "car",
"color": "red",
"hp": 85,
"doors": 4
}, {
"type": "plane",
"color": "blue",
"engines": 3
}]
I have types car and plane that satisfy a vehicle interface; I'd like to be able to write:
var v []vehicle
e := json.Unmarshal(myJSON, &v)
... and have JSON fill my slice of vehicles with a car and a plane; instead (and unsurprisingly) I just get "cannot unmarshal object into Go value of type main.vehicle".
For reference, here are suitable definitions of the types involved:
type vehicle interface {
vehicle()
}
type car struct {
Type string
Color string
HP int
Doors int
}
func (car) vehicle() { return }
type plane struct {
Type string
Color string
Engines int
}
func (plane) vehicle() { return }
var _ vehicle = (*car)(nil)
var _ vehicle = (*plane)(nil)
(Note that I'm actually totally uninterested in the t field on car and plane - it could be omitted because this information will, if someone successfully answers this question, be implicit in the dynamic type of the objects in v.)
Is there a way to have the JSON umarhsaller choose which type to use based on some part of the contents (in this case, the type field) of the data being decoded?
(Note that this is not a duplicate of Unmarshal JSON with unknown fields because I want each item in the slice to have a different dynamic type, and from the value of the 'type' property I know exactly what fields to expect—I just don't know how to tell json.Unmarshal how to map 'type' property values onto Go types.)
Taking the answers from the similar question: Unmarshal JSON with unknown fields, we can construct a few ways to unamrshal this JSON object in a []vehicle data structure.
The "Unmarshal with Manual Handling" version can be done by using a generic []map[string]interface{} data structure, then building the correct vehicles from the slice of maps. For brevity, this example does leave out the error checking for missing or incorrectly typed fields which the json package would have done.
https://play.golang.org/p/fAY9JwVp-4
func NewVehicle(m map[string]interface{}) vehicle {
switch m["type"].(string) {
case "car":
return NewCar(m)
case "plane":
return NewPlane(m)
}
return nil
}
func NewCar(m map[string]interface{}) *car {
return &car{
Type: m["type"].(string),
Color: m["color"].(string),
HP: int(m["hp"].(float64)),
Doors: int(m["doors"].(float64)),
}
}
func NewPlane(m map[string]interface{}) *plane {
return &plane{
Type: m["type"].(string),
Color: m["color"].(string),
Engines: int(m["engines"].(float64)),
}
}
func main() {
var vehicles []vehicle
objs := []map[string]interface{}{}
err := json.Unmarshal(js, &objs)
if err != nil {
log.Fatal(err)
}
for _, obj := range objs {
vehicles = append(vehicles, NewVehicle(obj))
}
fmt.Printf("%#v\n", vehicles)
}
We could leverage the json package again to take care of the unmarshaling and type checking of the individual structs by unmarshaling a second time directly into the correct type. This could all be wrapped up into a json.Unmarshaler implementation by defining an UnmarshalJSON method on the []vehicle type to first split up the JSON objects into raw messages.
https://play.golang.org/p/zQyL0JeB3b
type Vehicles []vehicle
func (v *Vehicles) UnmarshalJSON(data []byte) error {
// this just splits up the JSON array into the raw JSON for each object
var raw []json.RawMessage
err := json.Unmarshal(data, &raw)
if err != nil {
return err
}
for _, r := range raw {
// unamrshal into a map to check the "type" field
var obj map[string]interface{}
err := json.Unmarshal(r, &obj)
if err != nil {
return err
}
vehicleType := ""
if t, ok := obj["type"].(string); ok {
vehicleType = t
}
// unmarshal again into the correct type
var actual vehicle
switch vehicleType {
case "car":
actual = &car{}
case "plane":
actual = &plane{}
}
err = json.Unmarshal(r, actual)
if err != nil {
return err
}
*v = append(*v, actual)
}
return nil
}
JSON decoding and encoding in Go is actually surprisingly well at recognizing fields inside embedded structs. E.g. decoding or encoding the following structure works when there is no overlapping fields between type A and type B:
type T struct{
Type string `json:"type"`
*A
*B
}
type A struct{
Baz int `json:"baz"`
}
type B struct{
Bar int `json:"bar"`
}
Be aware that if both "baz" and "bar" are set in the JSON for the example above, both the T.A and T.B properties will be set.
If there is overlapping fields between A and B, or just to be able to better discard invalid combinations of fields and type, you need to implement the json.Unmarshaler interface. To not have to first decode fields into a map, you can extend the trick of using embedded structs.
type TypeSwitch struct {
Type string `json:"type"`
}
type T struct {
TypeSwitch
*A
*B
}
func (t *T) UnmarshalJSON(data []byte) error {
if err := json.Unmarshal(data, &t.TypeSwitch); err != nil {
return err
}
switch t.Type {
case "a":
t.A = &A{}
return json.Unmarshal(data, t.A)
case "b":
t.B = &B{}
return json.Unmarshal(data, t.B)
default:
return fmt.Errorf("unrecognized type value %q", t.Type)
}
}
type A struct {
Foo string `json:"bar"`
Baz int `json:"baz"`
}
type B struct {
Foo string `json:"foo"`
Bar int `json:"bar"`
}
For marshaling back, json.Marshaler must also be implemented if there is overlapping fields.
Full example: https://play.golang.org/p/UHAdxlVdFQQ
The two passes approach works fine, but there is also the option of the mapstructure package, that was created to do exactly this.
I was facing the same problem.
I'm using the lib github.com/mitchellh/mapstructure together the encoding/json.
I first, unmarshal the json to a map, and use mapstructure to convert the map to my struct, e.g.:
type (
Foo struct {
Foo string `json:"foo"`
}
Bar struct {
Bar string `json:"bar"`
}
)
func Load(jsonStr string, makeInstance func(typ string) any) (any, error) {
// json to map
m := make(map[string]any)
e := json.Unmarshal([]byte(jsonStr), &m)
if e != nil {
return nil, e
}
data := makeInstance(m["type"].(string))
// decoder to copy map values to my struct using json tags
cfg := &mapstructure.DecoderConfig{
Metadata: nil,
Result: &data,
TagName: "json",
Squash: true,
}
decoder, e := mapstructure.NewDecoder(cfg)
if e != nil {
return nil, e
}
// copy map to struct
e = decoder.Decode(m)
return data, e
}
Using:
f, _ := Load(`{"type": "Foo", "foo": "bar"}`, func(typ string) any {
switch typ {
case "Foo":
return &Foo{}
}
return nil
})
If the property is a string you can use .(string) for casting the property because the origin is an interface.
You can use it the next way:
v["type"].(string)

Custom JSON Unmarshalling for string-encoded number

I have a struct which contains various currency values, in cents (1/100 USD):
type CurrencyValues struct {
v1 int `json:"v1,string"`
v2 int `json:"v2,string"`
}
I'd like to create a custom json Unmarshaller for currency values with thousand separators. These values are encoded as strings, with one or more thousand separators (,), and possibly a decimal point (.).
For this JSON {"v1": "10", "v2": "1,503.21"}, I'd like to JSON Unmarshal a CurrencyValues{v1: 1000, v2: 150321}.
Following a similar answer here: Golang: How to unmarshall both 0 and false as bool from JSON, I went ahead and created a custom type for my currency fields, which include a custom Unmarshalling function:
type ConvertibleCentValue int
func (cents *ConvertibleCentValue) UnmarshalJSON(data []byte) error {
asString := string(data)
// Remove thousands separators
asString = strings.Replace(asString, ",", "", -1)
// Parse to float, then convert dollars to cents
if floatVal, err := strconv.ParseFloat(asString, 32); err == nil {
*cents = ConvertibleCentValue(int(floatVal * 100.0))
return nil
} else {
return err
}
}
However, when writing unit tests:
func Test_ConvertibleCentValue_Unmarshal(t *testing.T) {
var c ConvertibleCentValue
assert.Nil(t, json.Unmarshal([]byte("1,500"), &c))
assert.Equal(t, 150000, int(c))
}
I encounter this error:
Error: Expected nil, but got: &json.SyntaxError{msg:"invalid character ',' after top-level value", Offset:2}
What am I missing here?
You're trying to unmarshal the string 1,500 which is invalid in JSON. I think what you means is to unmarshal the JSON string "1,500":
assert.Nil(t, json.Unmarshal([]byte(`"1,500"`), &c))
Note the backticks. Here is a simplified example:
b := []byte(`1,500`)
var s string
err := json.Unmarshal(b, &s)
fmt.Println(s, err) // Prints error.
b = []byte(`"1,500"`)
err = json.Unmarshal(b, &s)
fmt.Println(s, err) // Works fine.
Playground: http://play.golang.org/p/uwayOSgmTv.

Golang JSON array of different types reflection: float64 vs int64

Consider this simple example:
package main
import (
"encoding/json"
"fmt"
"log"
"reflect"
)
var args = `[1, 2.5, "aaa", true, false]`
func main() {
var x []interface{}
err := json.Unmarshal([]byte(args), &x)
if err != nil {
log.Fatalf("%s", err.Error())
panic(fmt.Sprintf("%s", err.Error()))
}
for _, arg := range x {
t := reflect.TypeOf(arg).Kind().String()
v := reflect.ValueOf(arg)
if t == "int64" {
fmt.Printf("int64 %v\n", v.Int())
}
if t == "float64" {
fmt.Printf("float64 %v\n", v.Float())
}
if t == "string" {
fmt.Printf("string %v\n", v.String())
}
if t == "bool" {
fmt.Printf("bool %v\n", v.Bool())
}
}
}
The program outputs:
float64 1
float64 2.5
string aaa
bool true
bool false
As you can see, my input is a valid JSON which represents an array with five items:
- integer
- floating point number
- string
- boolean
- boolean
When I unmarshal the valid JSON string into []interface{} and try to check the types with reflection, the integer value from JSON has a type of float64. Any idea why? Is this expected behaviour?
This is documented behavior of Unmarshal. All numbers are unmarshaled into float64.
To unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
This is because JSON does not have integers, every number in JSON is defined to be a 64 bit floating point.
This is the default behaviour of the JSON decoder. You can change it to output json.Number instead by using the UseNumber method.