I have a json in the following format
{
"status_code": 200,
"status_message": "OK",
"response": {
"Messages": [
"CODE_NOT_AVAILABLE"
],
"UnknownDevices": {
"": [
"6",
"7",
"8",
"9",
"10"
]
}
}
}
As we see we are missing one index key, after unknownDevices, i am trying to unmarshal this json using golan in the following way
package main
import (
"encoding/json"
"fmt"
)
type pushWooshResponse struct {
Status int `json:"status_code"`
Status_msg string `json:"status_message"`
Response response
}
type response struct {
Message []string `json:"Messages"`
UnknownDevices devices
}
type devices struct {
Udevices []string `json:""`
}
func main() {
itemInfoR := `{"status_code":200,"status_message":"OK","response":{"Messages":["CODE_NOT_AVAILABLE"],"UnknownDevices":{"devices":["6","7","8","9","10"]}}}`
itemInfoBytes := []byte(itemInfoR)
var ItemInfo pushWooshResponse
er := json.Unmarshal(itemInfoBytes, &ItemInfo)
if er != nil {
fmt.Println("Error", er.Error())
} else {
fmt.Println(ItemInfo)
}
}
Output is
{200 OK {[CODE_NOT_AVAILABLE] {[]}}}
Everything is working fine except for the last part, which i am not able to unmarshall this. Can you help me out to umarshal the last part of the json.
There may be other options, but whenever you see strange JSON you can always fall back to implementing your own custom (un)marshalling by implementing json.Unmarshaler and/or json.Marshaler.
Perhaps something like:
type devices struct {
Udevices []string
}
func (d *devices) UnmarshalJSON(b []byte) error {
var x map[string][]string
err := json.Unmarshal(b, &x)
if err == nil {
// perhaps check that only a single
// key exists in the map as well
d.Udevices = x[""]
}
return err
}
func (d devices) MarshalJSON() ([]byte, error) {
x := map[string][]string{"": d.Udevices}
return json.Marshal(x)
}
Playground
Related
I am very, very new to golang (using this project as a way to learn the language better). I have an api I'd like to pull data from, but I can't seem to create the right type of struct for it.
The full JSON I am trying to encode is located here. https://prices.runescape.wiki/api/v1/osrs/latest
The structure of the JSON is
{
"data": {
"2": {
"high": 182,
"highTime": 1621811749,
"low": 180,
"lowTime": 1621811755
},
"6": {
"high": 186683,
"highTime": 1621811083,
"low": 184528,
"lowTime": 1621811286
},
... REPEATS THOUSANDS OF TIMES ...
}
}
I have tried using the JSON to Go converter for the JSON data, but that returns a struct that is entirely too large. (Go returns an error) I have tried to manually make a struct with a slice as follows
type osrsPrices []struct {
ID struct {
High int `json:"high"`
Hightime int `json:"highTime"`
Low int `json:"low"`
Lowtime int `json:"lowTime"`
} `json:"id"`
}
Whenever I try to run this from the terminal I am presented with the error "json: cannot unmarshal object into Go value of type main.osrsPrices exit status 1"
Here is the entire code I am using
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"time"
)
type osrsPrices []struct {
ID struct {
High int `json:"high"`
Hightime int `json:"highTime"`
Low int `json:"low"`
Lowtime int `json:"lowTime"`
} `json:"id"`
}
func main() {
url := "https://prices.runescape.wiki/api/v1/osrs/latest"
spaceClient := http.Client{
Timeout: time.Second * 2, // Timeout after 2 seconds
}
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
log.Fatal(err)
}
req.Header.Set("User-Agent", "skillerscape:LearningGoLang|priceChecker")
res, getErr := spaceClient.Do(req)
if getErr != nil {
log.Fatal(getErr)
}
if res.Body != nil {
defer res.Body.Close()
}
body, readErr := ioutil.ReadAll(res.Body)
if readErr != nil {
log.Fatal(readErr)
}
items := osrsPrices{}
jsonErr := json.Unmarshal(body, &items)
if jsonErr != nil {
log.Fatal(jsonErr)
}
for _, item := range items {
fmt.Println(item)
}
}
I am pretty sure the error has to do with the way I am trying to process the data, but I cannot figure out the issue directly, nor the solution despite a couple hours of googling. I appreciate any help anyone has on this issue.
At the deepest level, you have:
type Price struct {
High int `json:"high"`
Hightime int `json:"highTime"`
Low int `json:"low"`
Lowtime int `json:"lowTime"`
}
This is in an object with variable keys, so you need:
type Data struct {
Data map[string]Price `json:"data"`
}
Then you can unmarshal into an instance of this struct:
var data Data
json.Unmarshal(input,&data)
Quite new to go, sorry if this question sounds obvious.
I would like to use reflection in order to identify the type of an object while reading a json file.
The use case (please see the code below) is the following: I have two structs BoyGift and GirlGift that contain different fields. I have also a Boolean indicator IsBoy that is true if the recipient of the giftis a boy, false otherwise.
The type that encapsulates this behavior is the type Gift:
//Gift type
type Gift struct {
IsBoy bool `json:"isBoy"`
Payload ??? `json:"payload"`
}
that holds the data. How can I define that type in order the json unmarshal to convert dynamically to the correct type? The "json schema" in this case defines that a Gift should be either a BoyGift or a GirlGift. Is it possible to do this via reflection? How?
Doing unmarshal twice would be great, if the Boolean info is known
package main
import (
"encoding/json"
"fmt"
)
//BoyGift type
type BoyGift struct {
Cars uint32 `json:"cars"`
Balls uint32 `json:"balls"`
}
//GirlGift type
type GirlGift struct {
Dolls uint32 `json:"dolls"`
Lego uint32 `json:"lego"`
}
//Gift type
type Gift struct {
IsBoy bool `json:"isBoy"`
Payload GirlGift `json:"payload"`
}
func main() {
b := []byte(`{
"isBoy": true,
"payload": {
"cars": 1,
"balls": 2
}
}`)
var g Gift
err := json.Unmarshal(b, &g)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(g)
}
You should use json.RawMessage to dynamically unmarshal your data.
You can define Gift's Payliad as json.RawMessage and then delay unmarshaling until you know the value of IsBoy. Below you can find a basic example of how to do it.
package main
import (
"encoding/json"
"fmt"
)
//BoyGift type
type BoyGift struct {
Cars uint32 `json:"cars"`
Balls uint32 `json:"balls"`
}
//GirlGift type
type GirlGift struct {
Dolls uint32 `json:"dolls"`
Lego uint32 `json:"lego"`
}
//Gift type
type Gift struct {
IsBoy bool `json:"isBoy"`
Payload json.RawMessage `json:"payload"`
}
func main() {
b1 := []byte(`{
"isBoy": true,
"payload": {
"cars": 1,
"balls": 2
}
}`)
b2 := []byte(`{
"isBoy": false,
"payload": {
"dolls": 3,
"lego": 4
}
}`)
for _, b := range [][]byte{b1, b2} {
var g Gift
err := json.Unmarshal(b, &g)
if err != nil {
fmt.Println(err)
return
}
if g.IsBoy {
var boyGift BoyGift
err := json.Unmarshal(g.Payload, &boyGift)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(boyGift)
} else {
var girlGift GirlGift
err := json.Unmarshal(g.Payload, &girlGift)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(girlGift)
}
}
}
If you want to use Payload as an interface{} (which can be BoyGift or GirlGift), you could create additional helper struct for unmarshalling. Check extended example in Go Playground: https://play.golang.org/p/q1Hn45bgjsc
I have a third-party json api to work with in go.
It has some endpoints which return data as key-value.
For example here is json for statuses:
{
"result": {
"0": "done",
"1": "incomplete",
"2": "completed",
....
}
}
So as you see it is not an array it is an object.
Is it possible to marshal this kind of json to array of objects like
type Status struct {
Id int
Status string
}
without using additional struct like
type StatusReposne struct {
Result map[string]string `json:"result"`
}
and code to extract values?
As #mkopriva stated in the comment, it's not possible without some extra work. This code does provide means to Marshal/Unmarshal data to/from a slice of Statuss:
func main() {
sr := new(StatusReposne)
json.Unmarshal([]byte(input), sr)
fmt.Printf("%+v\n", sr)
js, _ := json.Marshal(sr)
fmt.Printf("%s\n", js)
}
type StatusReposne struct {
Result []Status `json:"result"`
}
type Status struct {
Id int
Status string
}
func (x *StatusReposne) MarshalJSON() ([]byte, error) {
var buffer struct {
Result map[string]string `json:"result"`
}
buffer.Result = make(map[string]string)
for _, v := range x.Result {
buffer.Result[strconv.Itoa(v.Id)] = v.Status
}
return json.Marshal(&buffer)
}
func (x *StatusReposne) UnmarshalJSON(b []byte) error {
var buffer struct {
Result map[string]string `json:"result"`
}
buffer.Result = make(map[string]string)
json.Unmarshal(b, &buffer)
for k, v := range buffer.Result {
k, _ := strconv.Atoi(k)
x.Result = append(x.Result, Status{Id: k, Status: v})
}
return nil
}
var input = `{
"result": {
"0": "done",
"1": "incomplete",
"2": "completed"
}
}`
type Struct struct {
Value string `json:"value"`
Value1 string `json:"value_one"`
Nest Nested `json:"nest"`
}
type Nested struct {
Something string `json:"something"`
}
I want to add elements which are not in the structs definitions without creating another struct type. For example
Struct.Extra1 = Nested{"yy"}
Struct.Nested.Extra2 = "zz"
Which will result
{
"Value": "xx",
"Value1": "xx",
"Extra1": {
"Something", "yy"
},
"Nest": {
"Something": "xx",
"Extra2": "zz"
}
}
SOLUTION1:
I thought of adding omitempty to achieve this but it makes the structs complex.
type Struct struct {
Value string
Value1 string
Nest Nested
Extra1 Nested `json:"omitempty"`
}
type Nested struct {
Something string
Extra2 string `json:"omitempty"`
}
SOLUTION2:
myextras := make(map[string]interface{})
// get Struct.Nested in map[string]interface{} format
myextras = Struct.Nest
myextras["Extra2"] = "zz"
// get Struct in map[string]interface{} format
struct["Nest"] = myextras
struct["Extra1"] = Nested{"yy"}
// solves the problem with lots of type casting but doesn't support json tag naming
Is there a better solution to add nested elements which are not represented in struct datatype with json-tag support and could be used to output to user.
If someone is not happy with the solution provided:
Try tidwall/sjson. It provides functions for quick JSON editing without having to define any structure. It saved me a bunch of time yesterday :D
Example usage:
value, _ := sjson.Set(`{"name":{"last":"Anderson"}}`, "name.last", "Smith")
println(value)
// Output:
// {"name":{"last":"Smith"}}
Based on this answer: Can I use MarshalJSON to add arbitrary fields to a json encoding in golang?
You could do something like (demo: http://play.golang.org/p/dDiTwxhoNn):
package main
import (
"encoding/json"
"fmt"
"log"
)
type Book struct {
Title string
Author string
// extra is used for additional dynamic element marshalling
extra func() interface{}
}
type FakeBook Book
func (b *Book) SetExtra(fn func() interface{}) {
b.extra = fn
}
func (b *Book) MarshalJSON() ([]byte, error) {
if b.extra == nil {
b.extra = func() interface{} { return *b }
}
return json.Marshal(b.extra())
}
func main() {
ms := &Book{
Title: "Catch-22",
Author: "Joseph Heller",
}
ms.SetExtra(func() interface{} {
return struct {
FakeBook
Extra1 struct {
Something string `json:"something"`
} `json:"extra1"`
}{
FakeBook: FakeBook(*ms),
Extra1: struct {
Something string `json:"something"`
}{
Something: "yy",
},
}
})
out, err := json.MarshalIndent(ms, "", " ")
if err != nil {
log.Fatalln(err)
}
fmt.Println(string(out))
mb := &Book{
Title: "Vim-go",
Author: "Fatih Arslan",
}
mb.SetExtra(func() interface{} {
return struct {
FakeBook
Something string `json:"something"`
}{
FakeBook: FakeBook(*mb),
Something: "xx",
}
})
out, err = json.MarshalIndent(mb, "", " ")
if err != nil {
log.Fatalln(err)
}
fmt.Println(string(out))
mc := &Book{
Title: "Another-Title",
Author: "Fatih Arslan",
}
out, err = json.MarshalIndent(mc, "", " ")
if err != nil {
log.Fatalln(err)
}
fmt.Println(string(out))
}
yes. there is a type json.Raw which not a struct but []byte. you can manage it out of struct, in any marshal/unmarshal way.
UPDATE
use any way to edit json string on the fly
package main
import (
"fmt"
"encoding/json"
"strings"
)
type Struct struct {
Value string `json:"value"`
Value1 string `json:"value_one"`
Nest json.RawMessage`json:"nest"`
}
func main() {
s := Struct{Value1: "struct string"}
buf, _ := json.Marshal(s)
fmt.Println(string(buf))
s2 := strings.ReplaceAll(string(buf), "null", `{"extra2":{"anykey":3.1415926535}}`)
fmt.Println(s2)
}
https://play.golang.org/p/lOCxJBs5iRJ
The map approach is the only sane way to do it, everything else (like json.RawMessage fields would require an extra marshalling step anyway.
if you don't want install extra package, you can change or add new values with this code :
package main
import "fmt"
import "strings"
import "encoding/json"
func main() {
s := []byte(`{ "level1a":"aaa", "level1b":{"level2a":"bbb", "level2b":{"level3":"aaa"} } }`)
var j interface{}
json.Unmarshal(s, &j)
SetValueInJSON(j, "level1a", "new value 1a")
SetValueInJSON(j, "level1b.level2a", "new value 2a")
SetValueInJSON(j, "level1b.level2b.level3", "new value 3")
SetValueInJSON(j, "level1b.level2c", "new key")
s,_ = json.Marshal(j)
fmt.Println(string(s))
// result: {"level1a":"new value 1a","level1b":{"level2a":"new value 2a","level2b":{"level3":"new value 3"},"level2c":"new key"}}
}
func SetValueInJSON(iface interface{}, path string, value interface{}) interface{} {
m := iface.(map[string]interface{})
split := strings.Split(path, ".")
for k, v := range m {
if strings.EqualFold(k, split[0]) {
if len(split) == 1 {
m[k] = value
return m
}
switch v.(type) {
case map[string]interface{}:
return SetValueInJSON(v, strings.Join(split[1:], "."), value)
default:
return m
}
}
}
// path not found -> create
if len(split) == 1 {
m[split[0]] = value
} else {
newMap := make(map[string]interface{})
newMap[split[len(split)-1]] = value
for i := len(split) - 2; i > 0; i-- {
mTmp := make(map[string]interface{})
mTmp[split[i]] = newMap
newMap = mTmp
}
m[split[0]] = newMap
}
return m
}
The out of the box JSON encoding in Go is really nice, but I need to get the output to match a certain format by adding a layer. I've figured out a way, but was hoping that there would be an easier way than the way I'm doing it.
Below is an example of how I'm doing it.
import (
"bytes"
"encoding/json"
"encoding/xml"
"fmt"
)
type Query struct {
XMLName xml.Name `xml:"http://marklogic.com/appservices/search query" json:"-"`
Format int `xml:"-" json:"-"`
Queries []interface{} `xml:",any" json:"queries"`
}
type TermQuery struct {
XMLName xml.Name `xml:"http://marklogic.com/appservices/search term-query" json:"-"`
Terms []string `xml:"http://marklogic.com/appservices/search text" json:"text"`
Weight float64 `xml:"http://marklogic.com/appservices/search weight,omitempty" json:"weight,omitempty"`
}
// use fakeQuery to avoid an infinite loop
type fakeQuery Query
//MarshalJSON for Query struct in a special way to add wraping {"query":...}
func (q Query) MarshalJSON() ([]byte, error) {
return wrapJSON(`query`, fakeQuery(q))
}
// use fakeTermQuery to avoid an infinite loop
type fakeTermQuery TermQuery
//MarshalJSON for TermQuery struct in a special way to add wraping {"term-query":...}
func (q TermQuery) MarshalJSON() ([]byte, error) {
return wrapJSON(`term-query`, fakeTermQuery(q))
}
func wrapJSON(name string, item interface{}) ([]byte, error) {
var buffer bytes.Buffer
b, err := json.Marshal(item)
buffer.Write([]byte(`{"`))
buffer.Write([]byte(name))
buffer.Write([]byte(`":`))
buffer.Write(b)
buffer.Write([]byte(`}`))
return buffer.Bytes(), err
}
I have a lot of defined structures that I would need to do this to, so I'm hoping for a better solution that won't leave me with with 100+ lines of code to just add a wrapper around the JSON object. Ideally I would like something that could peak at the XML element name defined for the XML encoder and use that to wrap the JSON.
In my case I'm using the MarshalJSON functions because these structures can be nested. If it helps I always know that the Query structure is the root structure.
When I started to use Go & Json I had the same problem. I resolved it by that
func wrapJSON(name string, item interface{}) ([]byte, error) {
wrapped := map[string]interface{}{
name: item,
}
converted, err := json.Marshal(wrapped)
return converted
}
Ideally, rename your method wrapJSON to wrap that return an interface and after convert this interface to JSON or XML
Perhaps I am missing something, but is this what you are looking for?
I started off with the same idea as #Manawasp (using a map[string]interface{}) but decided to try to get the name from the struct tag like you asked about... here's what I came up with (*note: there may be unhandled error cases, and this may overcomplicate something that can be handled pretty easily with the other solution)
http://play.golang.org/p/qO6tDZjtXA
package main
import (
"fmt"
"reflect"
"strings"
)
import (
"encoding/json"
"encoding/xml"
"errors"
)
type Query struct {
XMLName xml.Name `xml:"http://marklogic.com/appservices/search query" json:"-"`
Field1 string
Field2 int64
}
type TermQuery struct {
XMLName xml.Name `xml:"http://marklogic.com/appservices/search term-query" json:"-"`
Field3 string
Field4 int64
}
func getXmlName(d interface{}, label string) (string, bool) {
switch reflect.TypeOf(d).Kind() {
case reflect.Struct:
v, _ := reflect.TypeOf(d).FieldByName(label)
parts := strings.Split(v.Tag.Get("xml"), " ")
return parts[1], true
}
return "", false
}
func wrapJson(item interface{}) ([]byte, error) {
if n, ok := getXmlName(item, "XMLName"); ok {
b, err := json.Marshal(map[string]interface{}{n: item})
if err != nil {
return nil, err
}
return b, nil
}
return nil, errors.New("You failed")
}
func main() {
// create a Query and encode it as {"query": {struct}}
q := Query{Field1: "hello", Field2: 42}
wrappedQ, err := wrapJson(q)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(string(wrappedQ))
// create a TermQuery and encode it as {"term-query": {struct}}
tq := TermQuery{Field3: "world", Field4: 99}
wrappedTQ, err := wrapJson(tq)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(string(wrappedTQ))
}
OUTPUT
{"query":{"Field1":"hello","Field2":42}}
{"term-query":{"Field3":"world","Field4":99}}
EDIT
Ok, here is an update now that I can see what your issue is. It might be ugly, and it might not be bullet-proof (error handling, etc)... but for my test it seems to do what you want.
http://play.golang.org/p/8MloLP3X4H
package main
import (
"fmt"
"reflect"
"strings"
)
import (
//"encoding/json"
"encoding/json"
"encoding/xml"
"errors"
)
type Query struct {
XMLName xml.Name `xml:"http://marklogic.com/appservices/search query" json:"-"`
Field1 string
Field2 int64
Queries []interface{} `xml:",any" json:"queries"`
}
type TermQuery struct {
XMLName xml.Name `xml:"http://marklogic.com/appservices/search term-query" json:"-"`
Field3 string
Field4 int64
}
func getXmlName(d interface{}, label string) (string, bool) {
switch reflect.TypeOf(d).Kind() {
case reflect.Struct:
v, _ := reflect.TypeOf(d).FieldByName(label)
parts := strings.Split(v.Tag.Get("xml"), " ")
return parts[1], true
default:
fmt.Println(reflect.TypeOf(d).Kind())
}
return "", false
}
func wrapJson(item interface{}) (map[string]interface{}, error) {
if n, ok := getXmlName(item, "XMLName"); ok {
if k := reflect.ValueOf(item).FieldByName("Queries"); k.IsValid() {
for i := 0; i < k.Len(); i++ {
b, err1 := wrapJson(k.Index(i).Interface())
if err1 != nil {
continue
}
k.Index(i).Set(reflect.ValueOf(b))
}
}
return map[string]interface{}{n: item}, nil
}
return nil, errors.New("You failed")
}
func asJson(i interface{}) []byte {
b, err := json.Marshal(i)
if err != nil {
return []byte(`{"error": "too bad"}`)
}
return b
}
func main() {
// create a TermQuery and encode it as {"term-query": {struct}}
tq := TermQuery{Field3: "world", Field4: 99}
wrappedTQ, err := wrapJson(tq)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(string(asJson(wrappedTQ)))
// create a Query and encode it as {"query": {struct}}
q := Query{
Field1: "hello",
Field2: 42,
Queries: []interface{}{
TermQuery{Field3: "world", Field4: 99},
TermQuery{Field3: "yay, it works!", Field4: 666},
Query{
Field1: "Hi",
Field2: 21,
Queries: []interface{}{
TermQuery{
Field3: "omg",
Field4: 1,
},
},
},
},
}
wrappedQ, err := wrapJson(q)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(string(asJson(wrappedQ)))
}
PRETTY-PRINTED OUTOUT
{
"query": {
"Field1": "hello",
"Field2": 42,
"queries": [
{
"term-query": {
"Field3": "world",
"Field4": 99
}
},
{
"term-query": {
"Field3": "yay, it works!",
"Field4": 666
}
},
{
"query": {
"Field1": "Hi",
"Field2": 21,
"queries": [
{
"term-query": {
"Field3": "omg",
"Field4": 1
}
}
]
}
}
]
}
}
Inside your JSON Marshaler you can define one more more (anonymous) structs with the various JSON tags and add the desired wrapping.
Since you have already defined the JSON Marshaler any tags defined in the original struct might be overriden by your JSON Marshaler implementation.
See Golang Playground.
type Book struct {
Title string
Author string
Language string
Publisher string
}
func (b Book) MarshalJSON() ([]byte, error) {
type BookDetailJSON struct {
Name string `json:"Title"`
Author string
Language string `json:",omitempty"`
Publisher string `json:"-"`
}
type BookJSON struct {
Book BookDetailJSON `json:"Novel"`
}
return json.Marshal(BookJSON{BookDetailJSON{
Name: b.Title,
Author: b.Author,
Language: b.Language,
Publisher: b.Publisher,
}})
}
type MultiMatch struct {
Query string `json:"query"`
Fields []string `json:"fields"`
}
func (m *MultiMatch) MarshalJSON() ([]byte, error) {
w := map[string]interface{}{}
w["multi_match"] = *m
return json.Marshal(w)
}