I'm trying to create a struct based on a response I get.
I have no control over the response, and in it's structure there are field names which use comma as part of the filed name itself
JSON Example:
"date": "2022-09-09 00:00:00 UTC",
"Sum": {
"Change, %": "0.10",
"Price": "254",
"Value, $": "455.26",
}
When trying to create a struct the "regular" way, I get an error since once I use the comma character, reflect.StructTag.Get expects something specific and not the rest of the name.
Struct Example:
Date string `json:"date"`
Sum struct {
Change string `json:"Change, %"`
Price string `json:"Price"`
Value string `json:"Value, $"`
} `json:"Sum"`
The error I receive is:
"struct field tag json:"Value, $" not compatible with reflect.StructTag.Get: suspicious space in struct tag value"
The goal is to later manipulate and use this data (hopefully using my simple names), so "just" printing the JSON or moving it to an array or similar option cannot be the final step.
The 'Price' field is working as expected, but the problem is with the fields using the comma.
I couldn't find a way around it or a similar question.
Would love to understand how to deal with this issue.
Thanks!
You may unmarshal the Sum JSON object into a Go map, and assign the values from the map to the struct fields. The restriction only applies to struct tags (that they may not contain arbitrary characters like comma), but maps may hold any keys.
For example:
type Sum struct {
Change string
Price string
Value string
}
func (s *Sum) UnmarshalJSON(data []byte) error {
var m map[string]string
if err := json.Unmarshal(data, &m); err != nil {
return err
}
s.Change = m["Change, %"]
s.Price = m["Price"]
s.Value = m["Value, $"]
return nil
}
type model struct {
Date string `json:"date"`
Sum Sum `json:"Sum"`
}
func main() {
var m model
if err := json.Unmarshal([]byte(src), &m); err != nil {
panic(err)
}
fmt.Printf("%+v\n", m)
}
const src = `{
"date": "2022-09-09 00:00:00 UTC",
"Sum": {
"Change, %": "0.10",
"Price": "254",
"Value, $": "455.26"
}
}`
This will output (try it on the Go Playground):
{Date:2022-09-09 00:00:00 UTC Sum:{Change:0.10 Price:254 Value:455.26}}
Related
I'm a noob with Golang. I managed to get some things done with lots of effort.
I'm dealing with JSON files containing dates in a nested way.
I came across some workaround to unmarshal dates from JSON data into time.Time but I'm having a hard time dealing with nested ones.
The following code (obtained here in StackOverflow) is easy to understand since creates a user-defined function to parse the time objects first to a string and then to time.Time with time.Parse.
package main
import (
"encoding/json"
"fmt"
"log"
"time"
)
const dateFormat = "2006-01-02"
const data = `{
"name": "Gopher",
"join_date": "2007-09-20"
}`
type User struct {
Name string `json:"name"`
JoinDate time.Time `json:"join_date"`
}
func (u *User) UnmarshalJSON(p []byte) error {
var aux struct {
Name string `json:"name"`
JoinDate string `json:"join_date"`
}
err := json.Unmarshal(p, &aux)
if err != nil {
return err
}
t, err := time.Parse(dateFormat, aux.JoinDate)
if err != nil {
return err
}
u.Name = aux.Name
u.JoinDate = t
return nil
}
func main() {
var u User
err := json.Unmarshal([]byte(data), &u)
if err != nil {
log.Fatal(err)
}
fmt.Println(u.JoinDate.Format(time.RFC3339))
}
So far, so good.
Now I would like to extend it in order to handle the nested date fields in the JSON, like the example below:
[{
"name": "Gopher",
"join_date": "2007-09-20",
"cashflow": [
{"date": "2021-02-25",
"amount": 100},
{"date": "2021-03-25",
"amount": 105}
]
}]
The struct that I would like to get is:
type Record []struct {
Name string `json:"name"`
JoinDate time.Time `json:"join_date"`
Cashflow []struct {
Date time.Time `json:"date"`
Amount int `json:"amount"`
} `json:"cashflow"`
}
Thanks for the help.
To solve this using the patterns you've already got, you can write a separate unmarshalling function for the inner struct. You can do that by hoisting the inner struct to its own named struct, and then writing the function.
type CashflowRec struct {
Date time.Time `json:"date"`
Amount int `json:"amount"`
}
type Record struct {
Name string `json:"name"`
JoinDate time.Time `json:"join_date"`
Cashflow []CashflowRec `json:"cashflow"`
}
You've already shown how to write the unmarshalling function for CashflowRec, it looks almost the same as your User function. The unmarshalling function for Record will make use of that when it calls
func (u *Record) UnmarshalJSON(p []byte) error {
var aux struct {
Name string `json:"name"`
JoinDate string `json:"join_date"`
Cashflow []CashflowRec `json:"cashflow"`
}
err := json.Unmarshal(p, &aux)
Working example: https://go.dev/play/p/1X7BJ4NETM0
aside 1 Something amusing I learned while looking at this: because you've provided your own unmarshalling function, you don't actually need the json tags in your original structs. Those are hints for the unmarshaller that the json package provides. You should probably still leave them in, in case you have to marshal the struct later. Here's it working without those tags: https://go.dev/play/p/G2VWopO_A3t
aside 2 You might find it simpler not to use time.Time, but instead create a new type of your own, and then give that type its own unmarshaller. This gives you the interesting choice for writing only that one unmarshaller, but whether or not this is a win depends on what else you do with the struct later on. Working example that still uses your nested anonymous structs: https://go.dev/play/p/bJUcaw3_r41
type dateType time.Time
type Record struct {
Name string `json:"name"`
JoinDate dateType `json:"join_date"`
Cashflow []struct {
Date dateType `json:"date"`
Amount int `json:"amount"`
} `json:"cashflow"`
}
func (c *dateType) UnmarshalJSON(p []byte) error {
var s string
if err := json.Unmarshal(p, &s); err != nil {
return err
}
t, err := time.Parse(dateFormat, s)
if err != nil {
return err
}
*c = dateType(t)
return nil
}
This is the Go code that I have:
func main(){
s := string(`{"Id": "ABC123",
"Name": "Hello",
"RelatedItems":[
{"RId":"TEST123","RName":"TEST1","RChildren":"Ch1"},
{"RId":"TEST234","RName":"TEST2","RChildren":"Ch2"}]
}`)
var result map[string]interface{}
json.Unmarshal([]byte(s), &result)
fmt.Println("Id:", result["Id"])
Rlist := result["RelatedItems"].([]map[string]interface{})
for key, pist := range pist {
fmt.Println("Key: ", key)
fmt.Println("RID:", pist["RId"])
}
}
The struct is down below
type Model struct {
Id string `json:"Id"`
Name string `json:"ModelName"`
RelatedItems []RelatedItems `json:"RelatedItems"`
}
type RelatedItems struct {
RId string `json:"PCId"`
RName string `json:"PCName"`
RChildren string `json:"string"`
}
How would I get an output that would let me choose a particular field from the above?
eg:
Output
Id: ABC123
key:0
RID:TEST123
key:1
RID:TEST234
I am seeing this error
panic: interface conversion: interface {} is nil, not []map[string]interface {}
Based on the posted content,
I'm clear that you are facing issues retrieving data from the nested JSON string.
I've taken your piece of code and tried to compile and reproduce the issue.
After observing, I have a few suggestions based on the way the code has been written.
When the datatype present in the s is known to be similar to the type Model, the result could have been declared as type Model.
That results in var result Model instead of map[string]interface{}.
When the data that's gonna be decoded from the interface{} is not known, the usage of switch would come into rescue without crashing the code.
Something similar to:
switch dataType := result["RelatedItems"].(type){
case interface{}:
// Handle interface{}
case []map[string]interface{}:
// Handle []map[string]interface{}
default:
fmt.Println("Unexpected-Datatype", dataType)
// Handle Accordingly
When we try to Unmarshal, we make sure to look into the json tags that are provided for the fields of a structure. If the data encoded is not having the tags we provided, the data will not be decoded accordingly.
Hence, the result of decoding the data from s into result would result in {ABC123 [{ } { }]} as the tags of the fields Name, RId, RName, RChildren are given as ModelName, PCId, PCName, string respectively.
By the above suggestions and refining the tags given, the piece of code would be as following which would definitely retrieve data from nested JSON structures.
s := string(`{"Id": "ABC123",
"Name": "Hello",
"RelatedItems":[
{"RId":"TEST123","RName":"TEST1","RChildren":"Ch1"},
{"RId":"TEST234","RName":"TEST2","RChildren":"Ch2"}]
}`)
var result Model
json.Unmarshal([]byte(s), &result)
fmt.Println(result)
type Model struct {
Id string `json:"Id"`
Name string `json:"Name"`
RelatedItems []RelatedItems `json:"RelatedItems"`
}
type RelatedItems struct {
RId string `json:"RId"`
RName string `json:"RName"`
RChildren string `json:"RChildren"`
}
This results in the output: {ABC123 Hello [{TEST123 TEST1 Ch1} {TEST234 TEST2 Ch2}]}
Why would you unmarshal to a map anyway and go through type checks?
type Model struct {
Id string `json:"Id"`
Name string `json:"ModelName"`
RelatedItems []RelatedItems `json:"RelatedItems"`
}
type RelatedItems struct {
RId string `json:"PCId"`
RName string `json:"PCName"`
RChildren string `json:"string"`
}
s := `{"Id": "ABC123",
"Name": "Hello",
"RelatedItems":[
{"RId":"TEST123","RName":"TEST1","RChildren":"Ch1"},
{"RId":"TEST234","RName":"TEST2","RChildren":"Ch2"}]
}`
var result Model
if err := json.Unmarshal([]byte(s), &result); err != nil {
log.Fatal(err.Error())
}
fmt.Println("Id: ", result.Id)
for index, ri := range result.RelatedItems {
fmt.Printf("Key: %d\n", index)
fmt.Printf("RID: %s\n", ri.RId)
}
So I have this struct in Go:
type Car struct {
Name string `json:"name"`
Speed int `json:"speed"`
}
And I have two JSON samples that I unmarshal:
str := `{"name": "", "speed": 0}`
strTwo := `{}`
I do the unmarshaling in this way:
car := Car{}
_ = json.Unmarshal([]byte(str), &car)
carTwo := Car{}
_ = json.Unmarshal([]byte(strTwo), &carTwo)
Now because of the way Go deals with default value types, when I try to print the structure, I get the same result:
car - { 0}
carTwo - { 0}
So I can't see the difference between a missing value in JSON and when a default value is passed. How can I solve this problem?
One way is to use pointers in struct:
type Car struct {
Name *string `json:"name"`
Speed *int `json:"speed"`
}
But I get a very ugly code when using this values, I have to do pointer dereferencing everywhere
Go's primitive data types are not suitable to handle the "all valid values" and an additional "is present" information.
If you do need this, one way is to use pointers, where the nil pointer value corresponds to the "missing" state.
If it is uncomfortable to work with pointers afterwards, do a "post processing": convert your structs with pointer fields to a struct value with non-pointer fields, so you can work with that later on.
You may do this "manually", or write a custom unmarshaler to make this happen automatically.
Here's an example how to do it:
type PCar struct {
Name *string `json:"name"`
Speed *int `json:"speed"`
}
type Car struct {
Name string `json:"-"`
Speed int `json:"-"`
PCar
}
func (c *Car) UnmarshalJSON(data []byte) error {
if err := json.Unmarshal(data, &c.PCar); err != nil {
return err
}
if c.PCar.Name != nil {
c.Name = *c.PCar.Name
}
if c.PCar.Speed != nil {
c.Speed = *c.PCar.Speed
}
return nil
}
Example using it:
sources := []string{
`{"name": "", "speed": 0}`,
`{}`,
`{"name": "Bob", "speed": 21}`,
}
for i, src := range sources {
var c Car
if err := json.Unmarshal([]byte(src), &c); err != nil {
panic(err)
}
fmt.Println("car", i, c)
}
Output (try it on the Go Playground):
car 0 { 0 {0x40c200 0x4140ac}}
car 1 { 0 {<nil> <nil>}}
car 2 {Bob 21 {0x40c218 0x41410c}}
As you can see, car 1 contains 2 non-nil pointers, because the respective fields were present in the input JSON, while car 2 contains 2 nil pointers, because those fields were missing in its input. You may use Car.Name and Car.Speed fields as non-pointers (because they are not pointers). To tell if they were present in the input, you may check the corresponding pointers Car.PCar.Name and Car.PCar.Speed if they are nil.
I have a JSON string I want to unmarshal:
{
"id":1720,
"alertId":1,
"alertName":"{stats} Test Lambda Alert",
"dashboardId":5,
"panelId":2,
"userId":0,
"newState":"alerting",
"prevState":"ok",
"time":1523983581000,
"text":"",
"regionId":0,
"tags":[],
"login":"",
"email":"",
"avatarUrl":"",
"data":{
"evalMatches":[
{
"metric":"{prod}{stats} Lambda Alert Test",
"tags":null,
"value":16.525333333333332
}
]
}
}
I get the raw stream via a request: bodyBytes, err := ioutil.ReadAll(resp.Body)
I was hoping I could just specify a struct that pulls the values I care about, e.g.,
type Result struct {
ID string `json:"id"`
Time int64 `json:"time"`
}
However, when I try this, I get errors.
type Result struct {
ID string `json:"id"`
Time string `json:"time"`
}
var result Result
err2 := json.Unmarshal(bodyBytes, &result)
if err2 != nil {
log.Fatal(fmt.Sprintf(`Error Unmarshalling: %s`, err2))
}
fmt.Println(result.ID)
Error Unmarshalling: json: cannot unmarshal array into Go value of type main.Result
I suspect this error may be due to what's actually returned from ioutil.ReadAll(), since it has the above JSON string wrapped in [ ] if I do a fmt.Println(string(bodyBytes)), but if I try to json.Unmarshal(bodyBytes[0], &result), I just get compile errors, so I'm not sure.
If I want to unmarshal a JSON string, do I have to specify the full structure in my type Result struct? Is there a way around this? I don't want to be bound to the JSON object I receive (if the API changes upstream, it requires us to modify our code to recognize that, etc.).
You can unmarshal into structs that represent only some fields of your JSON document, but the field types have to match, as the error clearly states:
cannot unmarshal number into Go struct field Result.id of type string
You cannot unmarshal a number into a string. If you define the ID field as any numeric type it'll work just fine:
package main
import (
"encoding/json"
"fmt"
)
var j = []byte(`
{
"id":1720,
"prevState":"ok",
"time":1523983581000,
"text":"",
"regionId":0
}
`)
type Result struct {
ID int `json:"id"` // or any other integer type, or float{32,64}, or json.Number
Time int64 `json:"time"`
}
func main() {
var r Result
err := json.Unmarshal(j, &r)
fmt.Println(r, err)
}
Try it on the playground: https://play.golang.org/p/lqsQwLW2dHZ
Update
You have just edited your question with the actual error you receive. You have to unmarshal JSON arrays into slices. So if the HTTP response in fact returns a JSON array, unmarshal into []Result:
var j = []byte(`
[
{
"id":1720,
"prevState":"ok",
"time":1523983581000,
"text":"",
"regionId":0
}
]
`)
var r []Result
err := json.Unmarshal(j, &r)
fmt.Println(r[0], err)
https://play.golang.org/p/EbOVA8CbcFO
To generate Go types that match your JSON document pretty well, use https://mholt.github.io/json-to-go/.
I have a simple type that implements conversion of subtyped integer consts to strings and vice versa in Go. I want to be able to automatically unmarshal strings in JSON to values of this type. I can't, because UnmarshalJSON doesn't give me a way to return or modify the scalar value. It's expecting a struct, whose members are set by UnmarshalJSON. The ",string" method doesn't work either for other than builtin scalar types. Is there a way to implement UnmarshalJSON correctly for a derived scalar type?
Here's an example of what I'm after. I want it to print "Hello Ralph" four times, but it prints "Hello Bob" four times because the PersonID isn't being changed.
package main
import (
"encoding/json"
"fmt"
)
type PersonID int
const (
Bob PersonID = iota
Jane
Ralph
Nobody = -1
)
var nameMap = map[string]PersonID{
"Bob": Bob,
"Jane": Jane,
"Ralph": Ralph,
"Nobody": Nobody,
}
var idMap = map[PersonID]string{
Bob: "Bob",
Jane: "Jane",
Ralph: "Ralph",
Nobody: "Nobody",
}
func (intValue PersonID) Name() string {
return idMap[intValue]
}
func Lookup(name string) PersonID {
return nameMap[name]
}
func (intValue PersonID) UnmarshalJSON(data []byte) error {
// The following line is not correct
intValue = Lookup(string(data))
return nil
}
type MyType struct {
Person PersonID `json: "person"`
Count int `json: "count"`
Greeting string `json: "greeting"`
}
func main() {
var m MyType
if err := json.Unmarshal([]byte(`{"person": "Ralph", "count": 4, "greeting": "Hello"}`), &m); err != nil {
fmt.Println(err)
} else {
for i := 0; i < m.Count; i++ {
fmt.Println(m.Greeting, m.Person.Name())
}
}
}
Use a pointer receiver for the unmarshal method. If a value receiver is used, changes to the receiver are lost when the method returns.
The argument to the unmarshal method is JSON text. Unmarshal the JSON text to get a plain string with all JSON quoting removed.
func (intValue *PersonID) UnmarshalJSON(data []byte) error {
var s string
if err := json.Unmarshal(data, &s); err != nil {
return err
}
*intValue = Lookup(s)
return nil
}
There's a mismatch between the JSON tags an the example JSON. I changed the JSON to match the tag, but you can change it the other way.
if err := json.Unmarshal([]byte(`{"person": "Ralph", "count": 4, "greeting": "Hello"}`), &m); err != nil {
playground example
Here's an answer based on my comment. I'm not sure this is exactly what you want to do as some of your questions wording confuses me however the basic idea is to separate unmarshalling and transformation into two different steps. First unmarshal raw data into a compatible type, after do a transformation to another type or enrich the type you already have like in the example below. You're welcome to hide this behavior in a custom implementation of UnmarshalJSON if you'd like but I would personally advise against it. Here's my two reasons; 1) it's just not consistent with Go's explicit verbose coding style 2) I despise highly obfuscated packages/libraries/languages that do stuff like this for you because sooner or later it bites you in the ass an costs you a lot more than adding that 1 line of extra code in a few places (like hours trying to debug something that makes no sense to you).
type MyType struct {
Id PersonID
Name string `json: "name"`
Count int `json: "count"`
Greeting string `json: "greeting"`
}
func main() {
var m MyType
if err := json.Unmarshal([]byte(`{"name": "Ralph", "count": 4, "greeting": "Hello"}`), &m); err != nil {
fmt.Println(err)
} else {
m.Id = Lookup(m.Name) // see this isn't unmarshalling
// better to take the data as is and do transformation separate
for i := 0; i < m.Count; i++ {
fmt.Println(m.Greeting, m.Person.Name())
}
}
}