I need to integrate with an API that returns loads of elements in its response.
Is it possible to cherry-pick just the fields I want with go's json library or do I need to deserialise the entire response?
Yes.
Here's an example of having 2 fields in json and only decoding one:
jsonString := `{"a": 1, "b": 2}`
var rec struct {
A int `json:"a"`
}
err := json.Unmarshal([]byte(jsonString), &rec)
if err != nil {
log.Fatalf("json.Unmarshal() failed with '%s'\n", err)
}
fmt.Printf("rec: %+v\n", rec)
When run it prints:
rec: {A:1}
i.e. field "a" in json was decoded and field "b" was discarded.
See https://play.golang.org/p/89tu-ZC4pR for full example.
Related
I am working with an API that sends JSON data. The problem is that an array of a single element shows up as a single value. For example, consider the following JSON:
{ "names": ["Alice","Bob"] }
The API sends this as an array. But when the names field has a single element, the API sends this:
{ "names": "Alice" }
This is how I would normally decode this of response in Go:
type Response struct {
Names []string `json:"names"`
}
// later
d := &Response{}
_ = json.NewDecoder(resp).Decode(d) // resp would be a http.Response.Body containing the problematic JSON
Go decodes the first JSON correctly. However, after decoding the second JSON, the object contains an empty array.
I don't have any control over the API, so I have to get around this problem. How could I decode this JSON correctly in Go, so that the Names slice contains a single element? Thank you for your help.
You could implement the json.Unmarshaler interface and have it check the 0th raw byte for [ or " to find out whether it's an array or a string respectivelly:
type StringSlice []string
func (ss *StringSlice) UnmarshalJSON(data []byte) error {
if data[0] == '[' {
return json.Unmarshal(data, (*[]string)(ss))
} else if data[0] == '"' {
var s string
if err := json.Unmarshal(data, &s); err != nil {
return err
}
*ss = append(*ss, s)
}
return nil
}
https://play.golang.com/p/2GEJsS2YOLJ
You'll have to decode this into an interface{} and then use a type assertion to check whether the underlying type is a slice or just a string.
type Response struct {
Names interface{} `json:"names"`
}
Then after decoding into d, you'd do something like:
slice, ok := d.Names.([]interface{})
if ok {
// it was a slice. use it.
} else {
// it wasn't a slice - so expect it to be a string
// and use that, etc.
}
Use json.RawMessage as type. RawMessage simply delays the decoding of part of a message, so we can do it ourselves later.
type Response struct {
NamesRaw json.RawMessage `json:"names"`
Names []string
}
First, decode the response and then using json.Unmarshal decode json.RawMessage
x := &Response{}
_ = json.NewDecoder(resp).Decode(x);
x.Names = DecodeName(x.NamesRaw)
DecodeName used for decoding NameRaw data
func DecodeName(nameRaw json.RawMessage) (data []string) {
var s string
if err := json.Unmarshal(nameRaw, &s); err == nil {
v := []string{s}
return v
}
var sn []string
if err := json.Unmarshal(nameRaw, &sn); err == nil {
return sn
}
return
}
I have a JSON string I want to unmarshal:
{
"id":1720,
"alertId":1,
"alertName":"{stats} Test Lambda Alert",
"dashboardId":5,
"panelId":2,
"userId":0,
"newState":"alerting",
"prevState":"ok",
"time":1523983581000,
"text":"",
"regionId":0,
"tags":[],
"login":"",
"email":"",
"avatarUrl":"",
"data":{
"evalMatches":[
{
"metric":"{prod}{stats} Lambda Alert Test",
"tags":null,
"value":16.525333333333332
}
]
}
}
I get the raw stream via a request: bodyBytes, err := ioutil.ReadAll(resp.Body)
I was hoping I could just specify a struct that pulls the values I care about, e.g.,
type Result struct {
ID string `json:"id"`
Time int64 `json:"time"`
}
However, when I try this, I get errors.
type Result struct {
ID string `json:"id"`
Time string `json:"time"`
}
var result Result
err2 := json.Unmarshal(bodyBytes, &result)
if err2 != nil {
log.Fatal(fmt.Sprintf(`Error Unmarshalling: %s`, err2))
}
fmt.Println(result.ID)
Error Unmarshalling: json: cannot unmarshal array into Go value of type main.Result
I suspect this error may be due to what's actually returned from ioutil.ReadAll(), since it has the above JSON string wrapped in [ ] if I do a fmt.Println(string(bodyBytes)), but if I try to json.Unmarshal(bodyBytes[0], &result), I just get compile errors, so I'm not sure.
If I want to unmarshal a JSON string, do I have to specify the full structure in my type Result struct? Is there a way around this? I don't want to be bound to the JSON object I receive (if the API changes upstream, it requires us to modify our code to recognize that, etc.).
You can unmarshal into structs that represent only some fields of your JSON document, but the field types have to match, as the error clearly states:
cannot unmarshal number into Go struct field Result.id of type string
You cannot unmarshal a number into a string. If you define the ID field as any numeric type it'll work just fine:
package main
import (
"encoding/json"
"fmt"
)
var j = []byte(`
{
"id":1720,
"prevState":"ok",
"time":1523983581000,
"text":"",
"regionId":0
}
`)
type Result struct {
ID int `json:"id"` // or any other integer type, or float{32,64}, or json.Number
Time int64 `json:"time"`
}
func main() {
var r Result
err := json.Unmarshal(j, &r)
fmt.Println(r, err)
}
Try it on the playground: https://play.golang.org/p/lqsQwLW2dHZ
Update
You have just edited your question with the actual error you receive. You have to unmarshal JSON arrays into slices. So if the HTTP response in fact returns a JSON array, unmarshal into []Result:
var j = []byte(`
[
{
"id":1720,
"prevState":"ok",
"time":1523983581000,
"text":"",
"regionId":0
}
]
`)
var r []Result
err := json.Unmarshal(j, &r)
fmt.Println(r[0], err)
https://play.golang.org/p/EbOVA8CbcFO
To generate Go types that match your JSON document pretty well, use https://mholt.github.io/json-to-go/.
There is some inconvenience while processing JSON response from a web server.
For example, I don't know the data structure (and don't want to model it) of the JSON in advance, and just want to get the value from it!
So, for Python, I can just write
value = response["body"][4]["data"]["uid"] //response is a dictionary
But for Golang, I need to do the assertion for every element!
value := response["body"].([]interface{})[4].(map[string]interface{})["data"].(map[string]interface{})["uid"]
//response is a map[string]interface{}
This is what I write in golang to get the value I need. Do you have any suggestion on it? It there any useful tips for this kind of case?
If you model your JSON object with a struct and you unmarshal into a value of that, then you don't need those ugly indices and type assertions, you can simply refer to struct fields.
Note that you don't have to be afraid of the response being complex, you only need to model the parts you intend to use. E.g. if the response is an object with a hundred fields but you only need 2, then create a struct containing only those 2 fields.
If you don't want to model your JSON object (or can't because it's dynamic), then you may write a general utility function which gets a value based on the path (series of map keys and slice indices), which you can see in this answer: Taking a JSON string, unmarshaling it into a map[string]interface{}, editing, and marshaling it into a []byte seems more complicated then it should be
And last you may use 3rd party libs which already contain this helper functionality, such as https://github.com/icza/dyno (disclosure: I'm the author).
Using github.com/icza/dyno, it would look like this:
value, err := dyno.Get(response, "body", 4, "data", "uid")
As per icza you can create struct for JSON object. But in case you are getting JSON structure you don't know from start. Then you can create a dynamic parsing using reflections for interface which will be a recursive function to parse JSON data.
func main(){
var data interface{}
err := json.Unmarshal([]bytes(file.json), &data)
if err != nil {
panic(err)
}
var itemData map[string]interface{}
itemsMap := data.(map[string]interface{})
jsonParsedObject := interate(itemsMap)
log.Println(jsonParsedObject)
}
func iterate(data interface{}) interface{} {
if reflect.ValueOf(data).Kind() == reflect.Slice {
d := reflect.ValueOf(data)
tmpData := make([]interface{}, d.Len())
returnSlice := make([]interface{}, d.Len())
for i := 0; i < d.Len(); i++ {
tmpData[i] = d.Index(i).Interface()
}
for i, v := range tmpData {
returnSlice[i] = iterate(v)
}
return returnSlice
} else if reflect.ValueOf(data).Kind() == reflect.Map {
d := reflect.ValueOf(data)
tmpData := make(map[string]interface{})
for _, k := range d.MapKeys() {
typeOfValue := reflect.TypeOf(d.MapIndex(k).Interface()).Kind()
if typeOfValue == reflect.Map || typeOfValue == reflect.Slice {
tmpData[k.String()] = iterate(d.MapIndex(k).Interface())
} else {
tmpData[k.String()] = d.MapIndex(k).Interface()
}
}
return tmpData
}
return data
}
I have a function that returns JSON, and the return type is this:
map[string]interface{}
In my particular case, I know that the returned JSON looks like this:
{
"someNumber": 1,
"someString": "ee",
"someArray": [
"stringInArray",
{
"anotherNumber": 100,
"anotherString": "asdf",
"yetAnotherString": "qwer"
}
]
}
I want to get the value of "stringInArray" and also "anotherString". I've searched for solutions for example from Go by Example and blog.golang.org but I've not been successful.
For example, given that res is the json returned from the function call, I tried this (from the go blog):
var f interface{}
b := []byte(`res` )
err2 := json.Unmarshal(b, &f)
if err2!=nil{
log.Fatalf("Err unmarshalling: %v", err2)
}
m := f.( map[string]interface{} )
for k, v := range m {
switch vv := v.(type) {
case string:
fmt.Println(k, "is string", vv)
case int:
fmt.Println(k, "is int", vv)
case []interface{}:
fmt.Println(k, "is an array:")
for i, u := range vv {
fmt.Println(i, u)
}
default:
fmt.Println(k, "is of a type I don't know how to handle")
}
}
(I know the above would not do exactly what I want but it is a start.)
But when code execution hits b := []byte(res ), I get this error:
Err unmarshalling: invalid character 'r' looking for beginning of value
What is the most efficient method / best practice for obtaining the values? I'm open to any solution, not necessarily the approach above.
#sydnash Here is the code I promised in my response to your comment:
type LoginResponse2 struct {
id float64
jsonrpc string
result struct {
token string
details struct {
a float64
b string
c float64
}
}
}
var resStruct2 LoginResponse2
// Convert result to json
b, _ :=json.Marshal(res)
fmt.Println( string(b) )
// results of Println:
{"id":1,"jsonrpc":"2.0","result":[ "myToken",{"a":someNumber,"b":"some string","c":someOtherNumber} ] }
// Unmarshall json into struct
err2 := json.Unmarshal(b, &resStruct2)
if err2 != nil {
log.Fatalf("Error unmarshalling json: %v", err)
}
fmt.Println("Here is json unmarshalled into a struct")
fmt.Println( resStruct2 )
// results of Println
{0 { {0 0}}}
I think you should use b := []byte(res) instead b :=[]byte[res] and res should be a json string or []byte. res is not a legal json format.
this information maybe help you:
To unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
you can see there is no int but float64 for JSON numbers.
There is an example of how to unmarshal json in the docs on golang packages encoding/json. Essentially your problem is that the un-marshal expects to be putting the json into type. So your code must declare a type that represents the json you are converting from.
Example of the json and the declared type from the linked example:
var jsonBlob = []byte(`[
{"Name": "Platypus", "Order": "Monotremata"},
{"Name": "Quoll", "Order": "Dasyuromorphia"}
]`)
type Animal struct {
Name string
Order string
}
Once you define the type it should all fall into place.
I'm having a little bit of trouble handling types in Golang. I'm making a POST router.
Consider the following struct:
type DataBlob struct {
Timestamp string
Metric_Id int `json:"id,string,omitempty"`
Value float32 `json:"id,string,omitempty"`
Stderr float32 `json:"id,string,omitempty"`
}
This is my POST router using json.Unmarshal() from a decoded stream:
func Post(w http.ResponseWriter, req * http.Request) {
body, err := ioutil.ReadAll(req.Body)
if err != nil {
panic()
}
var t DataBlob
err = json.Unmarshal(body, &t)
if err != nil {
panic()
}
fmt.Printf("%s\n", t.Timestamp)
fmt.Printf("%d\n", int(t.Metric_Id))
fmt.Printf("%f\n", t.Value)
fmt.Printf("%f\n", t.Stderr)
}
It seems that no matter what I make my values in my POST request:
{
"timestamp": "2011-05-16 15:36:38",
"metric_id": "28",
"value": "4.5",
"stderr": "8.5"
}
All the non-String values print as 0 or 0.000000, respectively. It also doesn't matter if I try to type convert inline after-the-fact, as I did with t.Metric_Id in the example.
If I edit my struct to just handle string types, the values print correctly.
I also wrote a version of the POST router using json.NewDecoder():
func Post(w http.ResponseWriter, req * http.Request) {
decoder := json.NewDecoder(req.Body)
var t DataBlob
err := decoder.Decode(&t)
if err != nil {
panic()
}
fmt.Printf("%s\n", t.Timestamp)
fmt.Printf("%d\n", t.Metric_Id)
fmt.Printf("%f\n", t.Value)
fmt.Printf("%f\n", t.Stderr)
}
This is building off of functionality described in this answer, although the solution doesn't appear to work.
I appreciate your help!
You need to change the names of your Datablob values. You've told the JSON decoder that they're all named "id". You should try something like the following. Also, take a look at the json.Marshal description and how it describes the tags for structs and how the json library handles them. https://golang.org/pkg/encoding/json/#Marshal
type DataBlob struct {
Timestamp string
Metric_Id int `json:"metric_id,string,omitempty"`
Value float32 `json:"value,string,omitempty"`
Stderr float32 `json:"stderr,string,omitempty"`
}