Retrieve First Record of Nested Json Array - json

I'm trying to parse the first record in an embedded JSON array and create an object based on a subset of those properties. I have this working, but based on this question, I have to think there is a more elegant/less brittle way of doing this. For a bit more background, this is a result set from a call to the musicbrainz JSON web service, and I am treating the first artists record as the artist I am looking for.
The format of the JSON is like this:
{
"created": "2014-10-08T23:55:54.343Z",
"count": 458,
"offset": 0,
"artists": [{
"id": "83b9cbe7-9857-49e2-ab8e-b57b01038103",
"type": "Group",
"score": "100",
"name": "Pearl Jam",
"sort-name": "Pearl Jam",
"country": "US",
"area": {
"id": "489ce91b-6658-3307-9877-795b68554c98",
"name": "United States",
"sort-name": "United States"
},
"begin-area": {
"id": "10adc6b5-63bf-4b4e-993e-ed83b05c22fc",
"name": "Seattle",
"sort-name": "Seattle"
},
"life-span": {
"begin": "1990",
"ended": null
},
"aliases": [],
"tags": []
},
...
}
Here's the code I have so far. I'd like to be able to use my ArtistCollection type to get around some of the interface{} stuff, but I'm stuck as to how. I also don't want to bother with mapping all of the properties of the artist record, I'm only interested in the "name" and "id" values.
package main
import (
"fmt"
"encoding/json"
)
type Artist struct {
Id string
Name string
}
type ArtistCollection struct {
Artists []Artist
}
func main() {
raw := //json formatted byte array
var topLevel interface{}
err := json.Unmarshal(raw, &topLevel)
if err != nil {
fmt.Println("Uh oh")
} else {
m := topLevel.(map[string]interface{})
//this seems really hacky/brittle, there has to be a better way?
result := (m["artists"].([]interface{})[0]).(map[string]interface{})
artist := new(Artist)
artist.Id = result["id"].(string)
artist.Name = result["name"].(string)
fmt.Println(artist)
}
}
Requisite go playground link

Define a type that matches the structure of the JSON and unmarshal to a value of that type. I use an anonymous type below. Use an array of length one to grab the first artist record:
package main
import (
"encoding/json"
"fmt"
)
type Artist struct {
Id string
Name string
}
func main() {
raw := // JSON formatted byte array
var result struct {
Artists artist
}
err := json.Unmarshal(raw, &result)
if err != nil {
fmt.Println(err)
return
}
fmt.Printf("%#v\n", result.Artists[0])
}
playground

Related

Trouble mapping json to golang struct

I have a json stream as follows ...
[
{
"page": 1,
"pages": 7,
"per_page": "2000",
"total": 13200
},
[
{
"indicator": {
"id": "SP.POP.TOTL",
"value": "Population, total"
},
"country": {
"id": "1A",
"value": "Arab World"
},
"value": null,
"decimal": "0",
"date": "2019"
},
{
"indicator": {
"id": "SP.POP.TOTL",
"value": "Population, total"
},
"country": {
"id": "1A",
"value": "Arab World"
},
"value": "419790588",
"decimal": "0",
"date": "2018"
},
...
]
]
And I'm trying to decode it ... so I have the following struct ... but I keep getting
"cannot unmarshal array into Go value of type struct { P struct ... "
type Message []struct {
P struct {
Page int
}
V []struct {
Indicator struct {
Id string
Value string
}
Country struct {
Value string
}
Value string
Decimal string
Date string
}
}
My struct looks to match the json ... but obviously not! Any ideas?
Since your JSON array have two different types first unmarshal them into a slice of json.RawMessage which is []byte as underlying type so that we can unmarshal again JSON array data.
So unmarshal data for P and V struct type using index directly (predict) or detect if object(starting with '{') then unmarshal into P and array(starting with '[') then unmarshal into V. Now prepare your Message using those data.
type Message struct {
PageData P
ValData []V
}
type P struct {
Page int
}
type V struct {
Indicator struct {
Id string
Value string
}
Country struct {
Value string
}
Value string
Decimal string
Date string
}
func main() {
var rawdata []json.RawMessage
json.Unmarshal([]byte(jsonData), &rawdata)
var pageData P
json.Unmarshal(rawdata[0], &pageData)
var valData []V
json.Unmarshal(rawdata[1], &valData)
res := Message{pageData, valData}
fmt.Println(res)
}
var jsonData = `[...]` //your json data
Full code in Go Playground
As poWar said, the JSON you actually have is a list of objects whose types do not conform to each other. You must therefore unmarshal into something capable of holding different object types, such as interface{} or—since there is an outer array—[]interface{}.
You can also, if you like, decode into a []json.RawMessage. The underlying json.RawMessage itself has underlying type []byte so that it's basically the undecoded "inner" JSON. In at least some cases this is going to be more work than just decoding directly to []interface{} and checking each resulting interface, but you can, if you wish, decode to struct once you have the JSON separated out. For instance:
func main() {
var x []json.RawMessage
err := json.Unmarshal(input, &x)
if err != nil {
fmt.Printf("err = %v\n", err)
return
}
if len(x) != 2 {
fmt.Println("unexpected input")
return
}
var page struct {
Page int
}
err = json.Unmarshal(x[0], &page)
if err != nil {
fmt.Printf("unable to unmarshal page part: %v\n", err)
return
}
fmt.Printf("page = %d\n", page.Page)
// ...
}
Here on the Go Playground is a more complete example. See also Eklavya's answer.
Looking at your struct, your corresponding JSON should look something like this.
[
{
"P": {"page": 1},
"V": [
{
"Indicator": {"Id": ...},
"Country": {"Value":""},
"Value": "",
...
}
]
},
...
]
The JSON structure you are trying to Unmarshal looks like a list of objects where each object is not of the same type. You can start unmarshalling them into interfaces and defining each interface based on the object being unmarhsalled.
package main
import (
"encoding/json"
"log"
)
type Message []interface{}
func main() {
data := `[{"page":1,"pages":7,"per_page":"2000","total":13200},[{"indicator":{"id":"SP.POP.TOTL","value":"Population, total"},"country":{"id":"1A","value":"Arab World"},"value":null,"decimal":"0","date":"2019"},{"indicator":{"id":"SP.POP.TOTL","value":"Population, total"},"country":{"id":"1A","value":"Arab World"},"value":"419790588","decimal":"0","date":"2018"}]]`
var m Message
if err := json.Unmarshal([]byte(data), &m); err != nil {
log.Fatalf("could not unmarshal")
}
log.Printf("message: %v", m)
}
Output:
message: [map[page:1 pages:7 per_page:2000 total:13200] [map[country:map[id:1A value:Arab World] date:2019 decimal:0 indicator:map[id:SP.POP.TOTL value:Population, total] value:<nil>] map[country:map[id:1A value:Arab World] date:2018 decimal:0 indicator:map[id:SP.POP.TOTL value:Population, total] value:419790588]]]
[Edit]: Ideally you should change your JSON to be structured better for unmarshalling. If you do not have control on it, then your corresponding Go structure is just embedded maps of string to interfaces, which you will have to manually type cast and access.

What is the most efficient way of "filtering" out JSON objects from a key-value pair?

I am reading in a .json file. It's an array of objects in valid JSON format, example:
[
{
"Id": 13,
"Location": "Australia",
"Content": "Another string"
},
{
"Id": 145,
"Location": "England",
"Content": "SomeString"
},
{
"Id": 12,
"Location": "England",
"Content": "SomeString"
},
{
"Id": 12331,
"Location": "Sweden",
"Content": "SomeString"
},
{
"Id": 213123,
"Location": "England",
"Content": "SomeString"
}
]
I want to filter these objects out - say, removing anything where "Location"doesn't equal "England".
What I've tried so far is creating a custom UnmarshalJSON function. It does unmarshal it, but the objects it produces are empty - and as many as the input.
Sample code:
type languageStruct struct {
ID int `json:"Id"`
Location string `json:"Location"`
Content string `json:"Content"`
}
func filterJSON(file []byte) ([]byte, error) {
var x []*languageStruct
err := json.Unmarshal(file, &x)
check(err)
return json.MarshalIndent(x, "", " ")
}
func (s *languageStruct) UnmarshalJSON(p []byte) error {
var result struct {
ID int `json:"Id"`
Location string `json:"Location"`
Content string `json:"Content"`
}
err := json.Unmarshal(p, &result)
check(err)
// slice of locations we'd like to filter the objects on
locations := []string{"England"} // Can be more
if sliceContains(s.Location, locations) {
s.ID = result.ID
s.Location= result.Location
s.Content = result.Content
}
return nil
}
// helper func to check if a given string, f.e. a value of a key-value pair in a json object, is in a provided list
func sliceContains(a string, list []string) bool {
for _, b := range list {
if b == a {
fmt.Println("it's a match!")
return true
}
}
return false
}
While this runs - the output is wrong. It creates as many objects as comes in - however, the new ones are empty, f.e.:
// ...
[
{
"Id": 0,
"Location": "",
"Content": ""
},
{
"Id": 0,
"Location": "",
"Content": ""
}
]
//...
Whereas my desired output, from the first given input, would be:
[
{
"Id": 145,
"Location": "England",
"Content": "SomeString"
},
{
"Id": 12,
"Location": "England",
"Content": "SomeString"
},
{
"Id": 213123,
"Location": "England",
"Content": "SomeString"
}
]
When languageStruct.UnmarshalJSON() is called, there is already a languageStruct prepared that will be appended to the slice, no matter if you fill its content (fields) or not.
The easiest and my suggested solution is to just unmarshal normally, and post-process the slice: remove elements according to your requirements. This results in clean code, which you can easily adjust / alter in the future. Although it could be implemented as custom marshaling logic on a custom slice type []languageStruct, I would still not create custom marshaling logic for this but implement it as a separate filtering logic.
Here's a simple code unmarshaling, filtering and marshaling it again (note: no custom marshaling is defined / used for this):
var x []*languageStruct
err := json.Unmarshal(file, &x)
if err != nil {
panic(err)
}
var x2 []*languageStruct
for _, v := range x {
if v.Location == "England" {
x2 = append(x2, v)
}
}
data, err := json.MarshalIndent(x2, "", " ")
fmt.Println(string(data), err)
This will result in your desired output. Try it on the Go Playground.
The fastest and most complex solution would be to use event-driven parsing and building a state machine, but the complexity would increase by large. The idea would be to process the JSON by tokens, track where you're at currently in the object tree, and when an object is detected that must be excluded, don't process / add it to your slice. For details and ideas how this can be written, check out this anwser: Go - Decode JSON as it is still streaming in via net/http

How to unmarshal JSON in to array of interface and use

I am having difficulty understanding how to correctly unmarshal some JSON data that goes in to an array of type inteface and then use it. I tried to make this example code as simple as possible to illustrate the problem I am having. The code can be found in the playground here: https://play.golang.org/p/U85J_lBJ7Zr
The output looks like:
[map[ObjectType:chair ID:1234 Brand:Blue Inc.] map[ID:5678
Location:Kitchen ObjectType:table]] { } false { } false
Code
package main
import (
"fmt"
"encoding/json"
)
type Chair struct {
ObjectType string
ID string
Brand string
}
type Table struct {
ObjectType string
ID string
Location string
}
type House struct {
Name string
Objects []interface{}
}
func main() {
var h House
data := returnJSONBlob()
err := json.Unmarshal(data, &h)
if err != nil {
fmt.Println(err)
}
fmt.Println(h.Objects)
s1, ok := h.Objects[0].(Table)
fmt.Println(s1, ok)
s2, ok := h.Objects[0].(Chair)
fmt.Println(s2, ok)
}
func returnJSONBlob() []byte {
s := []byte(`
{
"Name": "house1",
"Objects": [
{
"ObjectType": "chair",
"ID": "1234",
"Brand": "Blue Inc."
},
{
"ObjectType": "table",
"ID": "5678",
"Location": "Kitchen"
}
]
}
`)
return s
}
I'm not sure if this is practical, since this is a simplified version of your scenario. However, one way to do this is combine the two object types to a new one, Object, and then unmarshal them directly to Object instead of using interface{}:
package main
import (
"encoding/json"
"fmt"
)
type Object struct {
ObjectType string
ID string
Brand string
Location string
}
type House struct {
Name string
Objects []Object
}
func returnJSONBlob() []byte {
s := []byte(`
{
"Name": "house1",
"Objects": [
{
"ObjectType": "chair",
"ID": "1234",
"Brand": "Blue Inc."
},
{
"ObjectType": "table",
"ID": "5678",
"Location": "Kitchen"
}
]
}
`)
return s
}
func main() {
var h House
data := returnJSONBlob()
err := json.Unmarshal(data, &h)
if err != nil {
fmt.Println(err)
}
fmt.Println(h.Objects[0].Brand)
fmt.Println(h.Objects[1].Location)
}
Prints:
Blue Inc.
Kitchen
Example here: https://play.golang.org/p/91F4UrQlSjJ

How to unmarshal nested json to a struct, if the outer element is not the same in GO

Hi i wonder if it is possible to unmarshal this given json to a struct
type Movie struct {
Title string
Actors []string
ID int
Length int
RelaseDate string
}
Here is an example of the json
{
"movies": [
{
"movie_title_A": {
"actors": [
"<actorID1123123>",
"<actorID1123123>",
"<actorID1123123>",
"<actorID1123123>",
"<actorID1123123>",
"<actorID1123123>...."
]
},
"ID": 99992,
"length": 120,
"relaseDate": "2.10.2012"
},
{
"movie_title_B": {
"actors": [
"<actorID1123123>",
"<actorID1123123>",
"<actorID1123123>",
"<actorID1123123>",
"<actorID1123123>",
"<actorID1123123>...."
]
},
"ID": 123124,
"length": 90,
"relaseDate": "10.10.2012"
}
]
}
As you can see the Name field can take on any name, since it is the title of the movie. Is there an efficient way to put it into the struct above?
Any help would be nice, thanks
given it's dynamic nature it might be easier use a map[string]interface as you'll not be able to define dynamic keys like asd123 and 2movie23123.
package main
import (
"encoding/json"
"fmt"
)
const j = `{
"movies": [
{
"asd123": {
"actors": [
"<actorID1123123>",
"<actorID1123123>",
"<actorID1123123>",
"<actorID1123123>",
"<actorID1123123>",
"<actorID1123123>"
]
},
"ID": 99992,
"length": 120,
"relaseDate": "2.10.2012"
},
{
"2movie23123": {
"actors": [
"<actorID1123123>",
"<actorID1123123>",
"<actorID1123123>",
"<actorID1123123>",
"<actorID1123123>",
"<actorID1123123>"
]
},
"ID": 123124,
"length": 90,
"relaseDate": "10.10.2012"
}
]
}`
// Movies ...
type Movies struct {
Name string
ID float64
Length float64
ReleaseDate string
Actors []interface{}
}
func main() {
data := map[string]interface{}{}
err := json.Unmarshal([]byte(j), &data)
if err != nil {
panic(err)
}
// printing it out to show you it marshaled
// b, _ := json.MarshalIndent(data, "", " ")
// fmt.Println(string(b))
//
var myMovies []Movies
for _, d := range data {
temp := Movies{}
converting := d.([]interface{})
for _, movie := range converting {
convertingMovie := movie.(map[string]interface{})
temp.Length = convertingMovie["length"].(float64)
temp.ID = convertingMovie["ID"].(float64)
temp.ReleaseDate = convertingMovie["relaseDate"].(string)
// getting rid of these keys so the for loop below doesn't iterate on them
// need the for loop cuz I don't know what the key name is
delete(convertingMovie, "length")
delete(convertingMovie, "ID")
delete(convertingMovie, "relaseDate")
for key, val := range convertingMovie {
temp.Name = key
actors := val.(map[string]interface{})
temp.Actors = actors["actors"].([]interface{})
}
}
myMovies = append(myMovies, temp)
}
b, _ := json.MarshalIndent(myMovies, "", " ")
fmt.Println(string(b))
}
Probably a better way to do it above, but I provided a quick example. The best way would be to organize the json data better so that it fits into a struct better, otherwise the use reflection. Without to much more work, I'd use the for loop above, and add it to a struct in a may that makes sense to me and so that it can access the data easier. Consider above the start of the JSON parser, so now that you can access the json data, fit it into a struct then, change data around.

How to decode json into structs

I'm trying to decode some json in Go but some fields don't get decoded.
See the code running in browser here:
What am I doing wrong?
I need only the MX records so I didn't define the other fields. As I understand from the godoc you don't need to define the fields you don't use/need.
// You can edit this code!
// Click here and start typing.
package main
import "fmt"
import "encoding/json"
func main() {
body := `
{"response": {
"status": "SUCCESS",
"data": {
"mxRecords": [
{
"value": "us2.mx3.mailhostbox.com.",
"ttl": 1,
"priority": 100,
"hostName": "#"
},
{
"value": "us2.mx1.mailhostbox.com.",
"ttl": 1,
"priority": 100,
"hostName": "#"
},
{
"value": "us2.mx2.mailhostbox.com.",
"ttl": 1,
"priority": 100,
"hostName": "#"
}
],
"cnameRecords": [
{
"aliasHost": "pop.a.co.uk.",
"canonicalHost": "us2.pop.mailhostbox.com."
},
{
"aliasHost": "webmail.a.co.uk.",
"canonicalHost": "us2.webmail.mailhostbox.com."
},
{
"aliasHost": "smtp.a.co.uk.",
"canonicalHost": "us2.smtp.mailhostbox.com."
},
{
"aliasHost": "imap.a.co.uk.",
"canonicalHost": "us2.imap.mailhostbox.com."
}
],
"dkimTxtRecord": {
"domainname": "20a19._domainkey.a.co.uk",
"value": "\"v=DKIM1; g=*; k=rsa; p=DkfbhO8Oyy0E1WyUWwIDAQAB\"",
"ttl": 1
},
"spfTxtRecord": {
"domainname": "a.co.uk",
"value": "\"v=spf1 redirect=_spf.mailhostbox.com\"",
"ttl": 1
},
"loginUrl": "us2.cp.mailhostbox.com"
}
}}`
type MxRecords struct {
value string
ttl int
priority int
hostName string
}
type Data struct {
mxRecords []MxRecords
}
type Response struct {
Status string `json:"status"`
Data Data `json:"data"`
}
type apiR struct {
Response Response
}
var r apiR
err := json.Unmarshal([]byte(body), &r)
if err != nil {
fmt.Printf("err was %v", err)
}
fmt.Printf("decoded is %v", r)
}
As per the go documentaiton about json.Unmarshal, you can only decode toward exported fields, the main reason being that external packages (such as encoding/json) cannot acces unexported fields.
If your json doesn't follow the go convention for names, you can use the json tag in your fields to change the matching between json key and struct field.
Exemple:
package main
import (
"fmt"
"encoding/json"
)
type T struct {
Foo string `json:"foo"`
priv string `json:"priv"`
}
func main() {
text := []byte(`{"foo":"bar", "priv":"nothing"}`)
var t T
err := json.Unmarshal(text, &t)
if err != nil {
panic(err)
}
fmt.Println(t.Foo) // prints "bar"
fmt.Println(t.priv) // prints "", priv is not exported
}
You must Uppercase struct fields:
type MxRecords struct {
Value string `json:"value"`
Ttl int `json:"ttl"`
Priority int `json:"priority"`
HostName string `json:"hostName"`
}
type Data struct {
MxRecords []MxRecords `json:"mxRecords"`
}
http://play.golang.org/p/EEyiISdoaE
The encoding/json package can only decode into exported struct fields. Your Data.mxRecords member is not exported, so it is ignored when decoding. If you rename it to use a capital letter, the JSON package will notice it.
You will need to do the same thing for all the members of your MxRecords type.