I'm totally newbie in Golang and solving problem with parsing JSON. Everything is working, except error handling.
if err := json.Unmarshal(file, &configData); err != nil {
if ute, ok := err.(*json.UnmarshalTypeError); ok {
fmt.Printf("UnmarshalTypeError %v - %v - %v", ute.Value, ute.Type, ute.Offset)
}
}
Here I get error ute.Offset undefined (type *json.UnmarshalTypeError has no field or method Offset) but in Docs of JSON package and also code they have this variable in UnmarshalTypeError struct.
What I'm doing wrong? Thank you
According to the godoc description:
If a JSON value is not appropriate for a given target type, or if a JSON number overflows the target type, Unmarshal skips that field and completes the unmarshalling as best it can. If no more serious errors are encountered, Unmarshal returns an UnmarshalTypeError describing the earliest such error.
Just like string type unmarshals into chan type,it make a UnmarshalTypeError error,just like the following:
package main
import (
"encoding/json"
"fmt"
)
type A struct {
Name string
Chan chan int
}
func main() {
var a A
bs := []byte(`{"Name":"hello","Chan":"chan"}`)
if e := json.Unmarshal(bs, &a); e != nil {
if ute, ok := e.(*json.UnmarshalTypeError); ok {
fmt.Printf("UnmarshalTypeError %v - %v - %v\n", ute.Value, ute.Type, ute.Offset)
} else {
fmt.Println("Other error:", e)
}
}
}
Outputs:
UnmarshalTypeError string - chan int - 29
It is working properly!
Related
I have been trying to work with AWS Go SDK Version 2. I am getting an where I want to convert the response I got from DescribeLoadBalancer() function into a JSON, so that I can parse it and perform further action but when I try to follow how to parse the JSON in go describe here https://blog.alexellis.io/golang-json-api-client/. I get an error in the compiler that I can't convert *elbv2.DescribeLoadBalancersOutput into []byte. I tried different approaches i.e trying to convert *elbv2.DescribeLoadBalancersOutput into string etc but I always get the same error i.e cannot convert *elbv2.DescribeLoadBalancersOutput into the specified type. hence want to understand what would be the best to perform the required action. Here's my code.
package main
import (
"encoding/gob"
"fmt"
"reflect"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/awserr"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/elbv2"
)
type LoadBalancerARN struct {
loadBalancerARN string
}
func main() {
session, error := session.NewSession(&aws.Config{
Region: aws.String("eu-central-1"),
})
if error != nil {
fmt.Printf("Cannot create a session, %v", error)
}
service := elbv2.New(session)
input := &elbv2.DescribeLoadBalancersInput{}
getLoadBalancersJson, error := service.DescribeLoadBalancers(input)
if error != nil {
if aerr, ok := error.(awserr.Error); ok {
switch aerr.Code() {
case elbv2.ErrCodeLoadBalancerNotFoundException:
fmt.Println(elbv2.ErrCodeLoadBalancerNotFoundException, aerr.Error())
default:
fmt.Println(aerr.Error())
}
} else {
fmt.Println(error.Error())
}
return
}
encodinglbJson := gob.NewEncoder(getLoadBalancersJson)
//var arn LoadBalancerARN
fmt.Println(getLoadBalancersJson)
fmt.Println(reflect.TypeOf(getLoadBalancersJson))
}
So, I have figured the above out, best way to do the above is, AWS provide a LoadBalancer function which convert the *elbv2.DescribeLoadBalancersOutput into an object of type *[]elbv2.LoadBalancer and then you can convert that object into the byte array using json.Marshal function and from there you can perform the parsing as needed.
Here's code
type LoadBalancerJSONStructure struct {
LoadBalancerArn string
DNSName string
CanonicalHostedZoneId string
CreatedTime string
LoadBalancerName string
Scheme string
VpcId string
Type string
IpAddressType string
}
response, error := json.Marshal(getLoadBalancers.LoadBalancers)
if error != nil {
fmt.Println("Error, %v", error)
}
fmt.Println(reflect.TypeOf(getLoadBalancers.LoadBalancers))
var lbJson LoadBalancerJSONStructure
err := json.Unmarshal(response, &lbJson)
if err != nil {
fmt.Println(err)
}
fmt.Printf("Struct is: ", lbJson)
I'm trying to create a JSON representation within Go using a map[string]interface{} type. I'm dealing with JSON strings and I'm having a hard time figuring out how to avoid the JSON unmarshaler to automatically deal with numbers as float64s. As a result the following error occurs.
Ex.
"{ 'a' : 9223372036854775807}" should be map[string]interface{} = [a 9223372036854775807 but in reality it is map[string]interface{} = [a 9.2233720368547758088E18]
I searched how structs can be used to avoid this by using json.Number but I'd really prefer using the map type designated above.
The go json.Unmarshal(...) function automatically uses float64 for JSON numbers. If you want to unmarshal numbers into a different type then you'll have to use a custom type with a custom unmarshaler. There is no way to force the unmarshaler to deserialize custom values into a generic map.
For example, here's how you could parse values of the "a" property as a big.Int.
package main
import (
"encoding/json"
"fmt"
"math/big"
)
type MyDoc struct {
A BigA `json:"a"`
}
type BigA struct{ *big.Int }
func (a BigA) UnmarshalJSON(bs []byte) error {
_, ok := a.SetString(string(bs), 10)
if !ok {
return fmt.Errorf("invalid integer %s", bs)
}
return nil
}
func main() {
jsonstr := `{"a":9223372036854775807}`
mydoc := MyDoc{A: BigA{new(big.Int)}}
err := json.Unmarshal([]byte(jsonstr), &mydoc)
if err != nil {
panic(err)
}
fmt.Printf("OK: mydoc=%#v\n", mydoc)
// OK: mydoc=main.MyDoc{A:9223372036854775807}
}
func jsonToMap(jsonStr string) map[string]interface{} {
result := make(map[string]interface{})
json.Unmarshal([]byte(jsonStr), &result)
return result
}
Example - https://goplay.space/#ra7Gv8A5Heh
Related questions - create a JSON data as map[string]interface with the given data
I'm fetching and decoding a large JSON response that has an error in it. Now I need to find where the error is! I read about json.SyntaxError but I am struggling to find out how to use it.
package main
import (
"encoding/json"
"fmt"
"net/http"
"os"
"text/template"
"time"
)
type Movie struct {
Title string `json:"title"`
PublishedAt time.Time `json:"published_at"`
}
func main() {
req, _ := http.NewRequest("GET", "https://s.natalian.org/2016-12-07/debugme2.json", nil)
resp, err := http.DefaultClient.Do(req)
defer resp.Body.Close()
dec := json.NewDecoder(resp.Body)
_, err = dec.Token()
for dec.More() {
var m Movie
if err = dec.Decode(&m); err != nil {
fmt.Println(err)
fmt.Println("Bad", m)
// https://blog.golang.org/error-handling-and-go
if serr, ok := err.(*json.SyntaxError); ok {
fmt.Println("Syntax error", serr)
}
} else {
fmt.Println("Good", m)
}
tmpl := template.Must(template.New("test").Parse("OUTPUT: {{ if .Title }}{{.Title}}{{ if .PublishedAt }} was published at {{.PublishedAt}} {{ end }}{{end}}\n"))
tmpl.Execute(os.Stdout, m)
}
}
What am I missing? Any tools or strategies or suggestions would be much appreciated. My output currently looks like:
Good {foobar 2016-11-24 16:17:12 +0800 SGT}
OUTPUT: foobar was published at 2016-11-24 16:17:12 +0800 SGT
parsing time ""null"" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null"" as "2006"
Bad {barbar 0001-01-01 00:00:00 +0000 UTC}
OUTPUT: barbar was published at 0001-01-01 00:00:00 +0000 UTC
Good { 1999-12-24 16:11:12 +0200 +0200}
OUTPUT:
Good {Something else entirely 2000-01-24 16:11:12 +0200 +0200}
OUTPUT: Something else entirely was published at 2000-01-24 16:11:12 +0200 +0200
But I need something like this in my stderr to better debug the issue:
Line 8: published_at is invalid
And maybe some context of the Title so I can tell the API backend team they have an error in their JSON response.
BONUS question: Furthermore I don't want to print the value 0001-01-01 00:00:00 +0000 UTC as it's actually really empty. I don't actually mind it being missing.
I found some solution:
if err := json.Unmarshal([]byte(data), &myStruct); err != nil {
if jsonErr, ok := err.(*json.SyntaxError); ok {
problemPart := data[jsonErr.Offset-10 : jsonErr.Offset+10]
err = fmt.Errorf("%w ~ error near '%s' (offset %d)", err, problemPart, jsonErr.Offset)
}
}
It will print something like
invalid character 'n' after object key:value pair ~ error near 'rence\","numberOfBil' (offset 14557)
One way to both accept null values, and to not print anything if published_at is null, is to set PublishedAt field to a pointer value :
type Movie struct {
Title string `json:"title"`
PublishedAt *time.Time `json:"published_at"`
}
The input string is valid JSON, so the json package does not raise a SyntaxError.
The json package has some other error types, such as UnmarshalTypeError, which is raised when an error occurs when the json does not match a nuilt-in type (e.g : string, int, array ...).
Unfortunately, when it calls a custom UnmarshalJSON() function, it looks like the json package returns the raw error :
package main
import (
"bytes"
"encoding/json"
"fmt"
"time"
)
// check the full type of an error raised when Unmarshaling a json string
func main() {
var test struct {
Clock time.Time
}
buf := bytes.NewBufferString(`{"Clock":null}`)
dec := json.NewDecoder(buf)
// ask to decode an invalid null value into a flat time.Time field :
err := dec.Decode(&test)
// print the details of the returned error :
fmt.Printf("%#v\n", err)
}
// Output :
&time.ParseError{Layout:"\"2006-01-02T15:04:05Z07:00\"", Value:"null", LayoutElem:"\"", ValueElem:"null", Message:""}
https://play.golang.org/p/fhZxVpOflb
The final error comes straight from the time package, it is not some kind of UnmarshalError from the json package which could at least tell you "this error occured when trying to Unmarshal value at this offset", and the error alone will not give you the context.
You can look specifically for type *time.ParseError in the error :
if terr, ok := err.(*time.ParseError); ok {
// in the example : Movie has one single time.Time field ;
// if a time.ParseError occured, it was while trying to read that field
fmt.Println("Error when trying to read 'published_at' value", terr)
// you can leave the field to its zero value,
// or if you switched to a pointer field :
m.PublishedAt = nil
}
If you happen to have several time fields (e.g : ProducedAt and PublishedAt), you can still look which field was left with its zero value :
if terr, ok := err.(*time.ParseError); ok {
if m.ProducedAt.IsZero() {
fmt.Println("Error when trying to read 'produced_at' value", terr)
}
if m.PublishedAt == zero {
fmt.Println("Error when trying to read 'published_at' value", terr)
}
}
By the way : as specified in the docs, "0001-01-01 00:00:00 UTC" is the zero value that the go team chose for go's time.Time zero value.
Your data for published_at is "null", it is string type, so I think you can define the PublishedAt as string, and you can use code to parse it to time.Time.
This is my test code:
package main
import (
"encoding/json"
"github.com/swanwish/go-common/logs"
"github.com/swanwish/go-common/utils"
)
func main() {
url := `https://s.natalian.org/2016-12-07/debugme2.json`
_, content, err := utils.GetUrlContent(url)
if err != nil {
logs.Errorf("Failed to get content from url %s, the error is %v", url, err)
return
}
movies := []struct {
Title string `json:"title"`
PublishedAt string `json:"published_at"`
}{}
err = json.Unmarshal(content, &movies)
if err != nil {
logs.Errorf("Failed to unmarshal content %s, the error is %v", string(content), err)
return
}
logs.Debugf("The movies are %v", movies)
}
The result is:
The movies are [{foobar 2016-11-24T16:17:12.000+08:00} {barbar null} { 1999-12-24T16:11:12.000+02:00} {Something else entirely 2000-01-24T16:11:12.000+02:00}]
It looks like madness, but it should work:
rawBody := []byte(`{"title":"test", "published_at":"2017-08-05T15:04:05Z", "edited_at":"05.08.2017"}`)
type Movie struct {
Title string `json:"title"`
PublishedAt time.Time `json:"published_at"`
EditedAt time.Time `json:"edited_at"`
}
var msg Movie
if err = json.Unmarshal(rawBody, &msg); err != nil {
if _, ok := err.(*time.ParseError); ok {
value := reflect.ValueOf(msg).Elem()
if value.Kind().String() != "struct" {
return err
}
for i := 0; i < value.NumField(); i++ {
field := value.Field(i)
if t, ok := field.Interface().(time.Time); ok {
if t.IsZero() {
name := value.Type().Field(i).Name
return fmt.Errorf("field: %s, message: %s", strings.ToLower(name), "time is not in RFC 3339 format.")
}
}
}
}
return err
}
This code will return first error happened. If PublishedAt is invalid we will know nothing about EditedAt even if it is valid.
A little over a year ago, Go added an Offset value to the json.UnmarshalTypeError type (see closed issue here for context). The purpose behind the offset value makes sense, but I'm not sure how it can be used when reading a go http response body, which is of type io.ReadCloser.
// An UnmarshalTypeError describes a JSON value that was
// not appropriate for a value of a specific Go type.
type UnmarshalTypeError struct {
Value string // description of JSON value - "bool", "array", "number -5"
Type reflect.Type // type of Go value it could not be assigned to
Offset int64 // error occurred after reading Offset bytes
}
For example:
var body CustomType
decoderErr := json.NewDecoder(response.Body).Decode(&body)
if decoderErr != nil {
if typeError, ok := decoderErr.(*json.UnmarshalTypeError); ok {
// Do something with typeError.Offset here
}
}
At the point of the error getting caught, I've already read from response.Body via json.NewDecoder.... I'm looking for a way to read response.Body again, but only up to the point of the error by using the Offset value in typeError.
Since you want to reuse the request body you should read and store the body before you Unmarshal the body, then if there is a JSON syntax or type error you can return a more useful error using the body you previously stored.
Proof of concept:
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
)
type Hello struct {
Name string `json:"name"`
Message string `json:"message"`
}
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
b, err := ioutil.ReadAll(r.Body)
if err != nil {
http.Error(w, "Error reading body", 400)
return
}
h := &Hello{}
if err := json.Unmarshal(b, &h); err != nil {
var msg string
switch t := err.(type) {
case *json.SyntaxError:
jsn := string(b[0:t.Offset])
jsn += "<--(Invalid Character)"
msg = fmt.Sprintf("Invalid character at offset %v\n %s", t.Offset, jsn)
case *json.UnmarshalTypeError:
jsn := string(b[0:t.Offset])
jsn += "<--(Invalid Type)"
msg = fmt.Sprintf("Invalid value at offset %v\n %s", t.Offset, jsn)
default:
msg = err.Error()
}
http.Error(w, msg, 400)
return
}
w.Write([]byte(`Good to go!`))
})
if err := http.ListenAndServe(":8000", nil); err != nil {
log.Fatal(err)
}
}
I am using a json.Decoder to decode JSON delivered over a network stream. It works fine, but whenever someone sends data that doesn't fit the schema (e.g. sending a negative integer when the struct's field type is unsigned) it returns an error with a vague message that doesn't pinpoint the offending property. This makes debugging more difficult.
Is there any way to extract the JSON that the decoder was trying to unmarshal when it errored? Here's a small reproducable snippet:
package main
import (
"bytes"
"fmt"
"encoding/json"
)
func main() {
buff := bytes.NewBufferString("{\"bar\": -123}")
decoder := json.NewDecoder(buff)
var foo struct{
Bar uint32
}
if err := decoder.Decode(&foo); err != nil {
fmt.Println(err)
fmt.Println("TODO: how to get JSON that caused this error?")
} else {
fmt.Println(foo.Bar)
}
}
Or on playground: https://play.golang.org/p/-WdYBkYEzJ
Some information is in the error, which is of type *json.UnamrshalTypeError
type UnmarshalTypeError struct {
Value string // description of JSON value - "bool", "array", "number -5"
Type reflect.Type // type of Go value it could not be assigned to
Offset int64 // error occurred after reading Offset bytes
}
You can get the offset in the json string, the reflect.Type of the field being unmarshaled into, and the json description of the Value. This can still pose a problem for types that implement their own unmarshaling logic, which is referenced by issue 11858
As of Go 1.8 this is now possible. The UnmarshalTypeError type now contains Struct and Field values which provide the name of the struct and field which caused a type mismatch.
package main
import (
"bytes"
"fmt"
"encoding/json"
)
func main() {
buff := bytes.NewBufferString("{\"bar\": -123}")
decoder := json.NewDecoder(buff)
var foo struct{
Bar uint32
}
if err := decoder.Decode(&foo); err != nil {
if terr, ok := err.(*json.UnmarshalTypeError); ok {
fmt.Printf("Failed to unmarshal field %s \n", terr.Field)
} else {
fmt.Println(err)
}
} else {
fmt.Println(foo.Bar)
}
}
The error message string also was changed to contain this new information.
Go 1.8:
json: cannot unmarshal number -123 into Go struct field .Bar of type uint32
Go 1.7 and earlier:
json: cannot unmarshal number -123 into Go value of type uint32
You can get at each element, key, value, and even delimiters using decode.Token() such as this in the playground, and below (modified from your example):
package main
import (
"bytes"
"encoding/json"
"fmt"
)
func main() {
buff := bytes.NewBufferString(`{"foo": 123, "bar": -123, "baz": "123"}`)
decoder := json.NewDecoder(buff)
for {
t, err := decoder.Token()
if _, ok := t.(json.Delim); ok {
continue
}
fmt.Printf("type:%11T | value:%5v //", t, t)
switch t.(type) {
case uint32:
fmt.Println("you don't see any uints")
case int:
fmt.Println("you don't see any ints")
case string:
fmt.Println("handle strings as you will")
case float64:
fmt.Println("handle numbers as you will")
}
if !decoder.More() {
break
}
if err != nil {
fmt.Println(err)
}
}
}
This will output
type: string | value: foo //handle strings as you will
type: float64 | value: 123 //handle numbers as you will
type: string | value: bar //handle strings as you will
type: float64 | value: -123 //handle numbers as you will
type: string | value: baz //handle strings as you will
type: string | value: 123 //handle strings as you will
You can switch on the type and handle each one as you wish. I have shown a simple example of that as well, each of the "//comments" in the result are conditional based on the type.
You'll also notice that each numbers' type is float64, although they would fit into an int, or even uint in the case of the "foo" value, and I check for those types in the switch but they never get used. You would have to provide your own logic in order to convert the float64 values to the types you wanted if they could be and handle types that wouldn't convert as special cases, or errors or whatever you wanted.