I'm having a heck of a time figuring out how to unmarshal a large unstructured json response into a usable object.
Here is a sample response (trimmed to show the part I'm having trouble with)
This has been greatly trimmed as this is a very large json response. I left the struct tags off below as well for simplicity.
{
"responseStatus": "ok",
"responseHeader": {
"status": "ok",
"requestId": "blah"
},
"responseData": {
"records": [
{
"name": "blah",
"site": {
"id": 1,
"name": "west"
},
"somevar1": "someval",
"somevar2": {
"x": 2,
"y": 1
},
"entries": [
{
"model": "100",
},
{
"model": "200",
}
]
},
]
}
So records is a large list of "objects". I need to convert these to a type I defined. The "entries" list also needs to be converted to its object type.
type Record struct {
Name string
Site map[string]string
Somevar1 string
Somevar2 map[string]string
Entries []Entry
}
type Entry struct {
Model string
}
Here I can get the responseData into an object I can iterate over:
results := data["responseData"].(map[string]interface{})
devices := results["records"].([]interface{})
for _, device := range devices {
fmt.Fprintf(os.Stdout, "%T\n", device)
fmt.Fprintf(os.Stdout, "%+v\n", device)
}
Here is a sample output from 1 iteration:
map[string]interface {}
map[name:foo site:map[id:2 name:somewhere somevar1: blah somevar2:map[x:1 y:2] entries:[map[model:100] map[model:200]
This is where I'm stuck. I need to take this line above and get convert it into my type Record while also converting the Entries to []Entry.
Change the Site and Somevar2 fields to map[string]interface{} or to a proper struct, because their corresponding json contains ints, so when you use only map[string]string it will fail.
Other than that your code works https://play.golang.com/p/rTgaXhXD1V6
Related
I've the following struct which works as expected Im getting
data and I was able
type Service struct {
Resources []struct {
Model struct {
Name string `json:"name"`
Credentials struct {
path string `json:"path"`
Vts struct {
user string `json:"user"`
id string `json:"id"`
address string `json:"address"`
} `json:"vts"`
} `json:"credentials"`
} `json:"model"`
} `json:"resources"`
}
service:= Service{}
err := json.Unmarshal([]byte(data), &service
The data is like following,
service1
{
"resources": [
"model": {
"name": "cred",
"credentials": {
"path": "path in fs",
"vts": {
"user": "stephane",
"id": "123",
"address": "some address"
}
},
}
},
Now some services providing additional data under vts for example now we have 3 fields (user/id/address) but some services (service1) can provides additional data like email, secondName ,etc . but the big problem here is that
I need to get it from parameters since (service 2) education, salary etc
Service2
{
"resources": [
"model": {
"name": "cred",
"credentials": {
"path": "path in fs",
"vts": {
"user": "stephane",
"id": "123",
"address": "some address",
"email" : "some email",
"secondName" : "secondName"
}
},
}
},
service N
{
"resources": [
"model": {
"name": "cred",
"credentials": {
"path": "path in fs",
"vts": {
"user": "stephane",
"id": "123",
"address": "some address",
"salary" : "1000000"
}
},
}
},
Of course If I know in advance the fields I can put them all in the struct and use omitempty but I dont know, I just get it as parameter to the function (the new properties names) , some service can provide 10 more fields in this struct (which I should get the properties name of them as args[]to the functions) but I don't know them in advance, this should be dynamic somehow ....is there a nice way to handle it in Golang ?
If you don't know the fields in advance, then don't use a struct but something that is also "dynamic": a map.
type Service struct {
Resources []struct {
Model struct {
Name string `json:"name"`
Credentials struct {
Path string `json:"path"`
Vts map[string]interface{} `json:"vts"`
} `json:"credentials"`
} `json:"model"`
} `json:"resources"`
}
map[sting]interface{} can hold values of any type. If you know all fields will hold a string value, you may also use a map[string]string so it will be easier to work with it.
Example with input JSON:
{
"resources": [
{
"model": {
"name": "cred",
"credentials": {
"path": "path in fs",
"vts": {
"user": "stephane",
"id": "123",
"address": "some address",
"dyn1": "d1value",
"dyn2": "d2value"
}
}
}
}
]
}
Testing it:
service := Service{}
err := json.Unmarshal([]byte(data), &service)
fmt.Printf("%q %v", service, err)
Output (try it on the Go Playground):
{[{{"cred" {"path in fs" map["dyn2":"d2value" "user":"stephane" "id":"123"
"address":"some address" "dyn1":"d1value"]}}}]} <nil>
Now if you want to collect values from the Vts map for a set of keys, this is how you can do it:
args := []string{"dyn1", "dyn2"}
values := make([]interface{}, len(args))
for i, arg := range args {
values[i] = service.Resources[0].Model.Credentials.Vts[arg]
}
fmt.Println(values)
Output of the above will be (try it on the Go Playground):
[d1value d2value]
I am reading in a .json file. It's an array of objects in valid JSON format, example:
[
{
"Id": 13,
"Location": "Australia",
"Content": "Another string"
},
{
"Id": 145,
"Location": "England",
"Content": "SomeString"
},
{
"Id": 12,
"Location": "England",
"Content": "SomeString"
},
{
"Id": 12331,
"Location": "Sweden",
"Content": "SomeString"
},
{
"Id": 213123,
"Location": "England",
"Content": "SomeString"
}
]
I want to filter these objects out - say, removing anything where "Location"doesn't equal "England".
What I've tried so far is creating a custom UnmarshalJSON function. It does unmarshal it, but the objects it produces are empty - and as many as the input.
Sample code:
type languageStruct struct {
ID int `json:"Id"`
Location string `json:"Location"`
Content string `json:"Content"`
}
func filterJSON(file []byte) ([]byte, error) {
var x []*languageStruct
err := json.Unmarshal(file, &x)
check(err)
return json.MarshalIndent(x, "", " ")
}
func (s *languageStruct) UnmarshalJSON(p []byte) error {
var result struct {
ID int `json:"Id"`
Location string `json:"Location"`
Content string `json:"Content"`
}
err := json.Unmarshal(p, &result)
check(err)
// slice of locations we'd like to filter the objects on
locations := []string{"England"} // Can be more
if sliceContains(s.Location, locations) {
s.ID = result.ID
s.Location= result.Location
s.Content = result.Content
}
return nil
}
// helper func to check if a given string, f.e. a value of a key-value pair in a json object, is in a provided list
func sliceContains(a string, list []string) bool {
for _, b := range list {
if b == a {
fmt.Println("it's a match!")
return true
}
}
return false
}
While this runs - the output is wrong. It creates as many objects as comes in - however, the new ones are empty, f.e.:
// ...
[
{
"Id": 0,
"Location": "",
"Content": ""
},
{
"Id": 0,
"Location": "",
"Content": ""
}
]
//...
Whereas my desired output, from the first given input, would be:
[
{
"Id": 145,
"Location": "England",
"Content": "SomeString"
},
{
"Id": 12,
"Location": "England",
"Content": "SomeString"
},
{
"Id": 213123,
"Location": "England",
"Content": "SomeString"
}
]
When languageStruct.UnmarshalJSON() is called, there is already a languageStruct prepared that will be appended to the slice, no matter if you fill its content (fields) or not.
The easiest and my suggested solution is to just unmarshal normally, and post-process the slice: remove elements according to your requirements. This results in clean code, which you can easily adjust / alter in the future. Although it could be implemented as custom marshaling logic on a custom slice type []languageStruct, I would still not create custom marshaling logic for this but implement it as a separate filtering logic.
Here's a simple code unmarshaling, filtering and marshaling it again (note: no custom marshaling is defined / used for this):
var x []*languageStruct
err := json.Unmarshal(file, &x)
if err != nil {
panic(err)
}
var x2 []*languageStruct
for _, v := range x {
if v.Location == "England" {
x2 = append(x2, v)
}
}
data, err := json.MarshalIndent(x2, "", " ")
fmt.Println(string(data), err)
This will result in your desired output. Try it on the Go Playground.
The fastest and most complex solution would be to use event-driven parsing and building a state machine, but the complexity would increase by large. The idea would be to process the JSON by tokens, track where you're at currently in the object tree, and when an object is detected that must be excluded, don't process / add it to your slice. For details and ideas how this can be written, check out this anwser: Go - Decode JSON as it is still streaming in via net/http
The bson name is used when performing pipe in mgo.
Struct :
type Training struct {
Id bson.ObjectId `json:"id" bson:"_id"`
Name string `json:"name" bson:"name"`
Description string `json:"description"`
Level *TrainingLevel `json:"level"`
Preworks []bson.ObjectId `json:"preworks"`
PrePostTests []bson.ObjectId `json:"preposttests" bson:"preposttests"`
TrainingEvaluations []bson.ObjectId `json:"training_evaluations" bson:"training_evaluations"`
TrainerEvaluations []bson.ObjectId `json:"trainer_evaluations" bson:"trainer_evaluations"`
AppCompanyId bson.ObjectId `json:"app_company_id" bson:"app_company_id"`
Company *Company `json:"company"`
}
Function :
func (this *TrainingClass) GetAllTraining() (interface{}, error) {
if !this.tokenInfo.IsAllowed(this.c) {
return nil, tlib.NewTError(common.Error_NoAccess, "You don't have the right!")
}
sess, db := GetDB()
defer sess.Close()
pipeline := []bson.M{
{"$match": bson.M{
"app_company_id": this.tokenInfo.AppCompanyId}},
{"$lookup": bson.M{
"from": "trainingbatch",
"localField": "_id",
"foreignField": "training._id",
"as": "trainingbatches"}},
}
resp := []bson.M{}
db.C(common.C_TRAINING).Pipe(pipeline).All(&resp)
return bson.M{"data": resp}, nil
}
Json result :
{
"data": [
{
"_id": "5995a749dbcfbe4e8cc31378",
"app_company_id": "58b24756e65bd121f6b1a923",
"description": "Description First Training",
"name": "First Training",
"trainingbatches": [
{
"_id": "5995a74adbcfbe4e8cc31379",
"app_company_id": "58b24756e65bd121f6b1a923",
"company": {
"_id": "58b24756e65bd121f6b1a923",
"address": "",
"app_company_id": "58b24756e65bd121f6b1a923",
"fullname": "",
"name": "Tandem",
"phone": ""
},
}
]
}
]
}
As you can see field _id is generated instead of id. That's not happen if I use find or findId. Is there any way to keep using json field no matter what's the query?
The way you're reading the result, it has no idea what the JSON field names are. In order for it to use those tags, it must actually deserialize into the struct where the tags have been specified. When you do:
resp := []bson.M{}
db.C(common.C_TRAINING).Pipe(pipeline).All(&resp)
You're explicitly telling mgo to return BSON results. The object you pass in (a slice of bson.M) has no json tags on it. In order to control the serialization to JSON, you must pass a struct with the JSON tags specified to All:
resp := []Training
db.C(common.C_TRAINING).Pipe(pipeline).All(&resp)
I'm trying to parse the first record in an embedded JSON array and create an object based on a subset of those properties. I have this working, but based on this question, I have to think there is a more elegant/less brittle way of doing this. For a bit more background, this is a result set from a call to the musicbrainz JSON web service, and I am treating the first artists record as the artist I am looking for.
The format of the JSON is like this:
{
"created": "2014-10-08T23:55:54.343Z",
"count": 458,
"offset": 0,
"artists": [{
"id": "83b9cbe7-9857-49e2-ab8e-b57b01038103",
"type": "Group",
"score": "100",
"name": "Pearl Jam",
"sort-name": "Pearl Jam",
"country": "US",
"area": {
"id": "489ce91b-6658-3307-9877-795b68554c98",
"name": "United States",
"sort-name": "United States"
},
"begin-area": {
"id": "10adc6b5-63bf-4b4e-993e-ed83b05c22fc",
"name": "Seattle",
"sort-name": "Seattle"
},
"life-span": {
"begin": "1990",
"ended": null
},
"aliases": [],
"tags": []
},
...
}
Here's the code I have so far. I'd like to be able to use my ArtistCollection type to get around some of the interface{} stuff, but I'm stuck as to how. I also don't want to bother with mapping all of the properties of the artist record, I'm only interested in the "name" and "id" values.
package main
import (
"fmt"
"encoding/json"
)
type Artist struct {
Id string
Name string
}
type ArtistCollection struct {
Artists []Artist
}
func main() {
raw := //json formatted byte array
var topLevel interface{}
err := json.Unmarshal(raw, &topLevel)
if err != nil {
fmt.Println("Uh oh")
} else {
m := topLevel.(map[string]interface{})
//this seems really hacky/brittle, there has to be a better way?
result := (m["artists"].([]interface{})[0]).(map[string]interface{})
artist := new(Artist)
artist.Id = result["id"].(string)
artist.Name = result["name"].(string)
fmt.Println(artist)
}
}
Requisite go playground link
Define a type that matches the structure of the JSON and unmarshal to a value of that type. I use an anonymous type below. Use an array of length one to grab the first artist record:
package main
import (
"encoding/json"
"fmt"
)
type Artist struct {
Id string
Name string
}
func main() {
raw := // JSON formatted byte array
var result struct {
Artists artist
}
err := json.Unmarshal(raw, &result)
if err != nil {
fmt.Println(err)
return
}
fmt.Printf("%#v\n", result.Artists[0])
}
playground
I have the following response from a graph api
{
"data": [
{
"name": "Mohamed Galib",
"id": "502008940"
},
{
"name": "Mebin Joseph",
"id": "503453614"
},
{
"name": "Rohith Raveendranath",
"id": "507482441"
}
],
"paging": {
"next": "https://some_url"
}
}
I have a struct as follows
type Item struct {
Name, Id string
}
I wanted to parse the response and get an array of Item, How do I do that?
You need to update your struct like so:
type Item struct {
Name string `json:"name"`
Id string `json:"id"`
}
and add a struct to represent the wrapper:
type Data struct {
Data []Item `json:"data"`
}
You can then use json.Unmarshal to populate a Data instance.
See the example in the docs.