Json Parsing in Golang - json

I am trying to parse a json from a third party software. It returns a json like this
{
"top1/dir1": "10",
"top1/dir2": "20",
"top1/dir3": "30",
"top2/diff_val1": "40"
}
JSONLint says this is a valid json. But I could not figure how I can parse this with golang.
The code I used to parse the json file above (to be clear I took the code from another stackoverflow post).
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
)
type mytype []map[string]string
func main() {
var data mytype
file, err := ioutil.ReadFile("t1.json")
if err != nil {
log.Fatal(err)
}
err = json.Unmarshal(file, &data)
if err != nil {
log.Fatal(err)
}
fmt.Println(data)
}
When I do a go run main.go, I get the below error
$ go run main.go
2016/06/19 22:53:57 json: cannot unmarshal object into Go value of type main.mytype
exit status 1
I did try to parse this format with another library - "github.com/Jeffail/gabs", but was unsuccessful. Since this is a valid json, I am pretty sure this can be parsed, but I am not sure how.

There is a Go package with methods for decoding JSON strings.
https://golang.org/pkg/encoding/json/#Unmarshal
Here is an example of usage:
package main
import (
"encoding/json"
"fmt"
)
func main() {
var jsonBlob = []byte(`[
{"Name": "Platypus", "Order": "Monotremata"},
{"Name": "Quoll", "Order": "Dasyuromorphia"}
]`)
type Animal struct {
Name string
Order string
}
var animals []Animal
err := json.Unmarshal(jsonBlob, &animals)
if err != nil {
fmt.Println("error:", err)
}
fmt.Printf("%+v", animals)
}
EDIT: As pointed out by Malik, the type of the value whose pointer you pass is wrong. In this case, your type should be map[string]interface{} (preferably, because a JSON field might not store a string) or map[string]string instead of []map[string]string. The brackets at the beginning are wrong: such would be an array of JSON objects.

It's just that you have a small typo in your program. You've declared mytype as a slice of maps, rather than just a map.
Just change:
type mytype []map[string]string
To:
type mytype map[string]string
See https://play.golang.org/p/pZQl8jV5TC for an example.

Jonathan's answer provides a good example of decoding JSON, and links the relevant package. You don't provide much detail on what exactly is going wrong with your parsing, but if I had to take a guess I would say you're probably not creating an appropriate struct to contain the JSON once it is unmarshalled. Because Go is statically typed, it expects data to adhere to explicitly defined formats.
If you don't want to go to the trouble of defining structs, you could just use an empty interface object, which is sort of a catch all in Go. Simply declare a variable with the type []interface{}, and then pass it into the JSON unmarshal function. Hope this helps!

Related

How to read "interfaces" map of json without defining structure in Golang?

Following this tutorial I'm trying to read a json file in Golang. It says there are two ways of doing that:
unmarshal the JSON using a set of predefined structs
or unmarshal the JSON using a map[string]interface{}
Since I'll probably have a lot of different json formats I prefer to interpret it on the fly. So I now have the following code:
package main
import (
"fmt"
"os"
"io/ioutil"
"encoding/json"
)
func main() {
// Open our jsonFile
jsonFile, err := os.Open("users.json")
// if we os.Open returns an error then handle it
if err != nil {
fmt.Println(err)
}
fmt.Println("Successfully Opened users.json")
// defer the closing of our jsonFile so that we can parse it later on
defer jsonFile.Close()
byteValue, _ := ioutil.ReadAll(jsonFile)
var result map[string]interface{}
json.Unmarshal([]byte(byteValue), &result)
fmt.Println(result["users"])
fmt.Printf("%T\n", result["users"])
}
This prints out:
Successfully Opened users.json
[map[type:Reader age:23 social:map[facebook:https://facebook.com twitter:https://twitter.com] name:Elliot] map[name:Fraser type:Author age:17 social:map[facebook:https://facebook.com twitter:https://twitter.com]]]
[]interface {}
At this point I don't understand how I can read the age of the first user (23). I tried some variations:
fmt.Println(result["users"][0])
fmt.Println(result["users"][0].age)
But apparently, type interface {} does not support indexing.
Is there a way that I can access the items in the json without defining the structure?
Probably you want
fmt.Println(result["users"].(map[string]interface{})["age"])
or
fmt.Println(result[0].(map[string]interface{})["age"])
As the JSON is a map of maps the type of the leaf nodes is interface{} and so has to be converted to map[string]interface{} in order to lookup a key
Defining a struct is much easier. My top tip for doing this is to use a website that converts JSON to a Go struct definition, like Json-To-Go

Is Go able to unmarshal to map[string][]interface{}?

Currently, I try to parse JSON to map[string][]interface{}, but unmarshalling returns an error. According to (https://golang.org/pkg/encoding/json/), to unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
-[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
I wonder if golang is able to unmarshal map[string][]interface{}. The following is code snippet. I am new to Golang, thanks for help in advance.
// emailsStr looks like "{"isSchemaConforming":true,"schemaVersion":0,"unknown.0":[{"email_address":"test1#uber.com"},{"email_address":"test2#uber.com"}]}"
emailsRaw := make(map[string][]*entities.Email)
err := json.Unmarshal([]byte(emailsStr), &emailsRaw)
Error message:
&json.UnmarshalTypeError{Value:"number", Type:(*reflect.rtype)(0x151c7a0), Offset:44, Struct:"", Field:""}
The Go encoding/json package will only unmarshal dynamically to a map[string]interface{}. From there, you will need to use type assertions and casting to pull out the values you want, like so:
func main() {
jsonStr := `{"isSchemaConforming":true,"schemaVersion":0,"unknown.0":[{"email_address":"test1#uber.com"},{"email_address":"test2#uber.com"}]}`
dynamic := make(map[string]interface{})
json.Unmarshal([]byte(jsonStr), &dynamic)
firstEmail := dynamic["unknown.0"].([]interface{})[0].(map[string]interface{})["email_address"]
fmt.Println(firstEmail)
}
(https://play.golang.org/p/VEUEIwj3CIC)
Each time, Go's .(<type>) operator is used to assert and cast the dynamic value to a specific type. This particular code will panic if anything happens to be the wrong type at runtime, like if the contents of unknown.0 aren't an array of JSON objects.
The more idiomatic (and robust) way to do this in Go is to annotate a couple structs with json:"" tags and have encoding/json unmarshal into them. This avoids all the nasty brittle .([]interface{}) type casting:
type Email struct {
Email string `json:"email_address"`
}
type EmailsList struct {
IsSchemaConforming bool `json:"isSchemaConforming"`
SchemaVersion int `json:"schemaVersion"`
Emails []Email `json:"unknown.0"`
}
func main() {
jsonStr := `{"isSchemaConforming":true,"schemaVersion":0,"unknown.0":[{"email_address":"test1#uber.com"},{"email_address":"test2#uber.com"}]}`
emails := EmailsList{}
json.Unmarshal([]byte(jsonStr), &emails)
fmt.Printf("%+v\n", emails)
}
(https://play.golang.org/p/iS6e0_87P2J)
A better approach will be to use struct for main schema and then use an slice of email struct for fetching the data for email entities get the values from the same according to requirements. Please find the solution below :-
package main
import (
"fmt"
"encoding/json"
)
type Data struct{
IsSchemaConforming bool `json:"isSchemaConforming"`
SchemaVersion float64 `json:"schemaVersion"`
EmailEntity []Email `json:"unknown.0"`
}
// Email struct
type Email struct{
EmailAddress string `json:"email_address"`
}
func main() {
jsonStr := `{"isSchemaConforming":true,"schemaVersion":0,"unknown.0":[{"email_address":"test1#uber.com"},{"email_address":"test2#uber.com"}]}`
var dynamic Data
json.Unmarshal([]byte(jsonStr), &dynamic)
fmt.Printf("%#v", dynamic)
}

How to access fields of a JSON in GO

Hi everyone I'm trying to see what the proper way of accessing fields of a json object from a http.get request in go.
I first do an http.get call get the JSON and then print it (which works) but is there a way to access just a field?
for example:
response, err:= http.Get("URL")
//Error checking is done between
contents, err:=ioutil.Readall(response.Body)
//Now at this point I have a json that looks like
{"id": "someID",
"name": "someName",
"test": [{"Name":"Test1",
"Result": "Success"},
{"Name":"Test2",
"Result": "Success"},
{...},
]}
Is there a way to only print the "test" of the Json? What is the proper way of accessing that field?
Use encoding/json package to Unmarshal data into struct, like following.
type Result struct {
ID string `json:"id"`
Name string `json:"name"`
Test []interface{} `json:"test"`
}
var result Result
json.Unmarshal(contents, &result)
fmt.Println(result.Test)
You can also parse Test to specific struct.
Same as the previous answer, use encoding/json package to Unmarshal data. But if you don't want to specify the structure, you could use map[string]interface/bson.M{} to receive the data, and get the field, then cast into types your want.
m := make(map[string]interface{})
err := json.Unmarshal(data, &m)
if err != nil {
log.Fatal(err)
}
fmt.Println(m["id"])
You may want to try gabs container, if you are not sure how depth JSON hierarchy can be. Have a look at below resources
https://github.com/Jeffail/gabs
https://godoc.org/github.com/Jeffail/gabs
If you just want to access one field then you can use the jsonq module https://godoc.org/github.com/jmoiron/jsonq
For your example you could get the test object with code similar to
jq.Object("test")
Where jq is a jsonq query object constructed from your JSON above (see the godoc page for instructions on how to create a query object from a JSON stream or string).
You can also use this library for retrieving specific String, Integer, Float and Bool values at an arbitrary depth inside a JSON object.
Since you are starting with a URL, Decode is a better option than Unmarshal:
package main
import (
"encoding/json"
"net/http"
)
func main() {
r, e := http.Get("https://github.com/manifest.json")
if e != nil {
panic(e)
}
defer r.Body.Close()
var s struct { Name string }
json.NewDecoder(r.Body).Decode(&s)
println(s.Name == "GitHub")
}
https://golang.org/pkg/encoding/json#Decoder.Decode
You may check this https://github.com/valyala/fastjson
s := []byte(`{"foo": [123, "bar"]}`)
fmt.Printf("foo.0=%d\n", fastjson.GetInt(s, "foo", "0"))
// Output:
// foo.0=123

How to marshal field starting with underscore to JSON [Golang]

Go's encoding/json package has some brilliant JSON marshalling functionality, and for all intents and purposes it's exactly what I need. But the problem arises when I want to try and marshal something I want to insert into a MongoDB instance.
MongoDB understands _id as an indexed identifier, but Go's JSON package only marshals exported fields so MongoDB creates its own ID for the document when I save, which I do not want, and I haven't even begun to test the implications it will have unmarshalling to a struct.
Is there a way to make the JSON marshaller include fields beginning with an underscore without writing a whole new one?
You can easily rename the fields. The Go name should start with an uppercase to be exported, but the json name can be anything compliant with json.
Here is an example borrowed to the encoding/json package documentation:
package main
import (
"encoding/json"
"fmt"
"os"
)
func main() {
type ColorGroup struct {
ID int `json:"_id"`
Name string
Colors []string
}
group := ColorGroup{
ID: 1,
Name: "Reds",
Colors: []string{"Crimson", "Red", "Ruby", "Maroon"},
}
b, err := json.Marshal(group)
if err != nil {
fmt.Println("error:", err)
}
os.Stdout.Write(b)
}

How to access interface fields on json decode?

I have a json document and I'm using a client which decodes the document in an interface (instead of struct) as below:
var jsonR interface{}
err = json.Unmarshal(res, &jsonR)
How can I access the interface fields? I've read the go doc and blog but my head still can't get it. Their example seem to show only that you can decode the json in an interface but doesn't explain how its fields can be used.
I've tried to use a range loop but it seems the story ends when I reach a map[string]interface. The fields that I need seem to be in the interface.
for k, v := range jsonR {
if k == "topfield" {
fmt.Printf("k is %v, v is %v", k, v)
}
}
The value inside the interface depends on the json structure you're parsing. If you have a json dictionary, the dynamic type of jsonR will be: map[string]interface{}.
Here's an example.
package main
import (
"encoding/json"
"fmt"
"log"
)
func main() {
a := []byte(`{"topfield": 123}`)
var v interface{}
if err := json.Unmarshal(a, &v); err != nil {
log.Fatalf("unmarshal failed: %s", err)
}
fmt.Printf("value is %v", v.(map[string]interface{})["topfield"])
}
Parsing json like this can be very difficult. The default type of a parse is map[string]interface{}. The Problem arises when you have another complex data structure within the main json(like another list or object). The best way to go about decoding json is defining a struct to hold data. Not only will the values be of the correct type but you can extract the specific data you actually care about.
Your struct can look like this:
type Top struct {
Topfield int `json:"topfield"`
}
which can be decoded like this:
a := []byte(`{"topfield": 123}`)
var data Top
json.Unmarshal(a, &data) //parse the json into data
now you can use regular struct operations to access your data like this:
value := data.Topfield
json which contains more complex data can also be easyli decoded. Perhaps you have a list in your data somewhere, you can use a struct like the following to extract it:
type Data struct {
States []string `json:"states"`
PopulationData []Country `json:"popdata"`
}
type Country struct {
Id int `json:"id"`
LastCensusPop int `json:"lcensuspopulation"`
Gdp float64 `json:"gdp"`
}
such a structure can not only parse list but also parse objects withing fields.