Parsing an unfamiliar YAML/JSON file in Go - json

How do you parse YAML data, in Go, without knowing its structure in advance? All of the examples I've seen assume you want to decode a marshaled map whose keys you already know. What if you don't know the keys? What if it's not a map but a marshaled list, a scalar, or some other common type?
Though I'm principally concerned with YAML, here, it seems like the technique might be generally useful for JSON, etc.. since there's a general pattern for parsing structured data (tagged structs, obviously).

For JSON, unmarshal the data to an interface{} value. Use type assertions to determine what's in the value.
var v interface{}
err := json.Unmarshal(data, &v)
if err != nil {
// handle error
}
switch v := v.(type) {
case string:
fmt.Println("string:", v)
case float64:
fmt.Println("number:", v)
case []interface{}:
fmt.Println("array:", v)
case map[string]interface{}:
fmt.Println("object:", v)
case nil:
fmt.Println("nil")
}

In the case of JSON, the standard library json.Unmarshal function will unmarshal arbitrary JSON if you like, if you pass it a pointer to an uninitialized empty interface. (See this example.)
The official docs even say as much:
To unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
Edit: Though not documented, the same is true of the yaml package; I tested on my workstation, and passing in a pointer to an uninitialized empty interface results in the initialization of correct arrays, maps, and primitives.

Related

How to marshal JSON with bigints?

I have a json that contains a field with a bigint
{"NETWORK_ID": 6000370005980500000071}
The format I have before marshaling is map[string]interface{}
When I marshal it and print to console everything seems to be fine but this field actually creates problems due to its size in other mediums so I want to serialize it as a string.
UseNumber() seems to be for this purpose but it's only for decoding I think.
Is there any way that I can detect this kind of bigint number and make them serialize as strings?
You'll need to create a custom type that implements the json.Marshaler interface, and marshals to a string. Example:
type MyBigInt big.Int
func (i MyBigInt) MarshalJSON() ([]byte, error) {
i2 := big.Int(i)
return []byte(fmt.Sprintf(`"%s"`, i2.String()), nil
}
This will always marshal your custom type as a quoted decimal number.

How to determine the type of json object in go

In gobyexample.com/json, a few examples show how to decode json string into typed objects or dictionary objects, which are declared as map[string]interface{}. But it assumes the result is always a dictionary.
So my question is how to determine the type of json object and what is the best practice to handle that?
Checkout the definition of json.Unmarshal:
func Unmarshal(data []byte, v interface{}) error
So at least you can obtain the underlying type by using reflect.
var v interface{}
json.Unmarshal([]byte(JSON_STR), &v)
fmt.Println(reflect.TypeOf(v), reflect.ValueOf(v))
And switch definitely is a better practice. I suppose below snippet
switch result := v.(type) {
case map[string]interface{}:
fmt.Println("dict:", result)
case []interface{}:
fmt.Println("list:", result)
default:
fmt.Println("value:", result)
}
can basically meet your requirement.

How can we read a json file as json object in golang

I have a JSON file stored on the local machine. I need to read it in a variable and loop through it to fetch the JSON object values. If I use the Marshal command after reading the file using the ioutil.Readfile method, it gives some numbers as an output. These are my few failed attempts,
Attempt 1:
plan, _ := ioutil.ReadFile(filename) // filename is the JSON file to read
var data interface{}
err := json.Unmarshal(plan, data)
if err != nil {
log.Error("Cannot unmarshal the json ", err)
}
fmt.Println(data)
It gave me following error,
time="2016-12-13T22:13:05-08:00" level=error msg="Cannot unmarshal the json json: Unmarshal(nil)"
<nil>
Attempt 2: I tried to store the JSON values in a struct and then using MarshalIndent
generatePlan, _ := json.MarshalIndent(plan, "", " ") // plan is a pointer to a struct
fmt.Println(string(generatePlan))
It give me the output as string. But if I cast the output to string then I won't be able to loop it as JSON object.
How can we read a JSON file as JSON object in golang? Is it possible to do that?
Any help is appreciated. Thanks in advance!
The value to be populated by json.Unmarshal needs to be a pointer.
From GoDoc :
Unmarshal parses the JSON-encoded data and stores the result in the value pointed to by v.
So you need to do the following :
plan, _ := ioutil.ReadFile(filename)
var data interface{}
err := json.Unmarshal(plan, &data)
Your error (Unmarshal(nil)) indicates that there was some problem in reading the file , please check the error returned by ioutil.ReadFile
Also please note that when using an empty interface in unmarshal, you would need to use type assertion to get the underlying values as go primitive types.
To unmarshal JSON into an interface value, Unmarshal stores one of
these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
It is always a much better approach to use a concrete structure to populate your json using Unmarshal.
If you're looking at this in 2022, the ioutil package has been deprecated. You can still use it but you'll get annoying errors.
Instead you can use the os package.
someStruct := SomeStruct{} // Read errors caught by unmarshal
fileBytes, _ := os.ReadFile(filename)
err := json.Unmarshal(fileBytes, spec)
Note, I'm specifically ignoring the error from os.ReadFile since it will also cause an error in json.Unmarshal for the sake of the example.

Unmarshall into an interface type

I expected below code to print an object of type struct J, however it prints a map object of type map[string]interface{}. I can feel why it acts like that, however when I run, reflect.ValueOf(i).Kind(), it returns Struct, so it kinda gives me the impression that Unmarshal method should return type J instead of a map. Could anyone enlighten me ?
type J struct {
Text string
}
func main() {
j := J{}
var i interface{} = j
js := "{\"Text\": \"lala\"}"
json.Unmarshal([]byte(js), &i)
fmt.Printf("%#v", i)
}
The type you're passing into Unmarshal is not *J, you're passing in an *interface{}.
When the json package reflects what the type is of the pointer it received, it sees interface{}, so it then uses the default types of the package to unmarshal into, which are
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
There is almost never a reason to use a pointer to an interface. If you find yourself using a pointer to an interface, and you don't know exactly why, then it's probably a mistake. If you want to unmarshal into J, then pass that in directly. If you need to assign that to an intermediary interface, make sure you use a pointer to the original value, not a pointer to its interface.
http://play.golang.org/p/uJDFKfSIxN
j := J{}
var i interface{} = &j
js := "{\"Text\": \"lala\"}"
json.Unmarshal([]byte(js), i)
fmt.Printf("%#v", i)
This is expected behavior: instead of giving json.Unmarshal a pointer to a properly typed place in memory you give it a pointer to a place in memory with type interface{}. It can essentially store anything in there under the type the JSON defines, so it does just that.
See it like this:
Unmarshal gets a place to store the data v with type interface{}
Unmarshal detects a map encoded as JSON
Unmarshal sees that the target type is of type interface{}, creates a go map from it and stores it in v
If you would have given it a different type than interface{} the process would have looked like this:
Unmarshal gets a place to store the data v with type struct main.J
Unmarshal detects a map encoded as JSON
Unmarshal sees that the target type is struct main.J and begins to recursively fit the data to the type
The main point is, that the initial assignment
var i interface{} = j
is completely ignored by json.Unmarshal.

How to handle index out of range in JSON ( Go)

I'm developing a web service and part of that I read the Request.Body and try to unmarshal it.
if err := json.NewDecoder(body).Decode(r); err !=nil{
log.Error(err)
return err
}
The issue is that sometimes the client is sending an empty body and I get a panic runtime error: index out of range
goroutine 7 [running]:
How am I supposed to mitigate this?
I am decomposing your code:
NewDecoder: -
func NewDecoder(r io.Reader) *Decoder
NewDecoder returns a new decoder that reads from r. The decoder
introduces its own buffering and may read data from r beyond the JSON
values requested.
So NewDecoder only reads data from r. it's not care, is r empty...
Decode:-
func (dec *Decoder) Decode(v interface{}) error
Decode reads the next JSON-encoded value from its input and stores it
in the value pointed to by v.
See the documentation for Unmarshal for details about the conversion of JSON into a Go value.
To unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
If a JSON value is not appropriate for a given target type, or if a
JSON number overflows the target type, Unmarshal skips that field and
completes the unmarshalling as best it can. If no more serious errors
are encountered, Unmarshal returns an UnmarshalTypeError describing
the earliest such error.
The JSON null value unmarshals into an interface, map, pointer, or
slice by setting that Go value to nil. Because null is often used in
JSON to mean “not present,” unmarshaling a JSON null into any other Go
type has no effect on the value and produces no error.
Reading above statement, it's clear that there is not chances, we get run time panic error. I was experimenting with a sample code to reproduce this ERROR. May error coming from inside the JSON package or your own code.
var dummy []byte
dummy = make([]byte, 10)
size, _ := body.Read(dummy)
if size > 0 {
if err := json.NewDecoder(body).Decode(r); err != nil {
log.Error(err)
return err
}
fmt.Fprintf(w, "%s", "Json cannot be empty")// where w is http.ResponseWriter