The problem I'm currently having is after saving a struct to a json file and then opening the struct from the json file, somehow the properties of the struct have changed slightly.
In the struct N, sometimes A and B can point to the same J. However, after encoding then decoding they point to different Js of the value.
before encoding this returns true (expected). After decoding it, it returns false (not expected)
fmt.Println("is same pointer", n.A[0] == n.B[0])
Is this supposed to happen? Is there a way around this. Thanks.
type N struct {
A []*J
B []*J
C []*J
}
func (n *N) Save(name string) {
name = "radacted.json"
err := os.Remove(name)
file, err := os.Create(name)
defer file.Close()
if err != nil {
fmt.Println(err)
}
bytes, err := json.Marshal(n)
file.Write(bytes)
}
func Open(name string) *N {
bytes, err := ioutil.ReadFile("redacted.json")
if err != nil {
log.Fatal("decode error:", err)
}
var n NeuralNetwork
json.Unmarshal(bytes, &n)
return &n
}
It's expected and documented behaviour
Pointer values encode as the value pointed to.
You can assert values equality
*n.A[0] == *n.B[0] //should stay
fmt.Println("is same pointer", n.A[0] == n.B[0])
you are comparing the address value here so it will not be the same. Let me give you an example
suppose you have struct like this :
type Test struct {
ValueA *int
ValueB *int
}
and on your main function you add the same value but with different address in this case with different variable :
func main() {
hello := 12
hello2 := 12
testObject := Test{ValueA: &hello, ValueB: &hello2}
if *testObject.ValueA == *testObject.ValueB {
fmt.Println("Equal Value")
} else {
fmt.Println("Different Value")
}
}
Notice that the *testObject.ValueA and *testObject.ValueBis getting the exact value not the value address. If you are not using * the result would be different.
so as uvelichitel said you just need to use * when comparing your struct value.
Related
I thought I understood unmarshalling by now, but I guess not. I'm having a little bit of trouble unmarshalling a map in go. Here is the code that I have so far
type OHLC_RESS struct {
Pair map[string][]Candles
Last int64 `json:"last"`
}
type Candles struct {
Time uint64
Open string
High string
Low string
Close string
VWAP string
Volume string
Count int
}
func (c *Candles) UnmarshalJSON(d []byte) error {
tmp := []interface{}{&c.Time, &c.Open, &c.High, &c.Low, &c.Close, &c.VWAP, &c.Volume, &c.Count}
length := len(tmp)
err := json.Unmarshal(d, &tmp)
if err != nil {
return err
}
g := len(tmp)
if g != length {
return fmt.Errorf("Lengths don't match: %d != %d", g, length)
}
return nil
}
func main() {
response := []byte(`{"XXBTZUSD":[[1616662740,"52591.9","52599.9","52591.8","52599.9","52599.1","0.11091626",5],[1616662740,"52591.9","52599.9","52591.8","52599.9","52599.1","0.11091626",5]],"last":15}`)
var resp OHLC_RESS
err := json.Unmarshal(response, &resp)
fmt.Println("resp: ", resp)
}
after running the code, the last field will unmarshal fine, but for whatever reason, the map is left without any value. Any help?
The expedient solution, for the specific example JSON, would be to NOT use a map at all but instead change the structure of OHLC_RESS so that it matches the structure of the JSON, i.e.
type OHLC_RESS struct {
Pair []Candles `json:"XXBTZUSD"`
Last int64 `json:"last"`
}
https://go.dev/play/p/Z9PhJt3wX33
However it's safe to assume, I think, that the reason you've opted to use a map is because the JSON object's key(s) that hold the "pairs" can vary and so hardcoding them into the field's tag is out of the question.
To understand why your code doesn't produce the desired result, you have to realize two things. First, the order of a struct's fields has no bearing on how the keys of a JSON object will be decoded. Second, the name Pair holds no special meaning for the unmarshaler. Therefore, by default, the unmarshaler has no way of knowing that your wish is to decode the "XXBTZUSD": [ ... ] element into the Pair map.
So, to get your desired result, you can have the OHLC_RESS implement the json.Unmarshaler interface and do the following:
func (r *OHLC_RESS) UnmarshalJSON(d []byte) error {
// first, decode just the object's keys and leave
// the values as raw, non-decoded JSON
var obj map[string]json.RawMessage
if err := json.Unmarshal(d, &obj); err != nil {
return err
}
// next, look up the "last" element's raw, non-decoded value
// and, if it is present, then decode it into the Last field
if last, ok := obj["last"]; ok {
if err := json.Unmarshal(last, &r.Last); err != nil {
return err
}
// remove the element so it's not in
// the way when decoding the rest below
delete(obj, "last")
}
// finally, decode the rest of the element values
// in the object and store them in the Pair field
r.Pair = make(map[string][]Candles, len(obj))
for key, val := range obj {
cc := []Candles{}
if err := json.Unmarshal(val, &cc); err != nil {
return err
}
r.Pair[key] = cc
}
return nil
}
https://go.dev/play/p/Lj8a8Gx9fWH
I wish to be able to use .Decode() on a response body to populate a struct without first having to attempt to figure out which type of struct I should decode to.
I have a generic struct Match to hold information about a game that was played e.g. a match in Fortnite. Within this struct, I use MatchData to hold the entirety of the game's match data.
When decoding into the MatchData struct, I'm finding the underlying embedded type is initialised, but with all default values, and not the values from the respose.
type Match struct {
MatchID int `json:"match_id"`
GameType int `json:"game_type"`
MatchData *MatchData `json:"match_data"`
}
type MatchData struct {
MatchGame1
MatchGame2
}
type MatchGame1 struct {
X int `json:"x"`
Y int `json:"y"`
}
type MatchGame2 struct {
X int `json:"x"`
Y int `json:"y"`
}
func populateData(m *Match) (Match, error) {
response, err := http.Get("game1.com/path")
if err != nil {
return nil, err
}
// Here, m.MatchData is set with X and Y equal to 0
// when response contains values > 0
err = json.NewDecoder(response.Body).Decode(&m.MatchData)
if err != nil {
return nil, err
}
return m, nil
}
Edit
Example expected JSON payload.
{
"x": 10,
"y": 20
}
I can solve the issue by checking m.GameType, creating a struct that corresponds and then assigning it to m.MatchData, but if I wanted to add another 100 game APIs, I'd prefer if the function could be agnostic of it.
I'm not sure if this is even possible, but thanks in advance.
The approach in the question will not work because the embedded structs share field names. Try this approach.
Declare a map that associates game type identifiers with the associated Go types. This is only code related to decoding that knows about the hundreds of game types.
var gameTypes = map[int]reflect.Type{
1: reflect.TypeOf(&MatchGame1{}),
2: reflect.TypeOf(&MatchGame2{}),
}
Decode the match data to a raw message. Use the game type to create a match data value and decode to that value.
func decodeMatch(r io.Reader) (*Match, error) {
// Start with match data set to a raw messae.
var raw json.RawMessage
m := &Match{MatchData: &raw}
err := json.NewDecoder(r).Decode(m)
if err != nil {
return nil, err
}
m.MatchData = nil
// We are done if there's no raw message.
if len(raw) == 0 {
return m, nil
}
// Create the required match data value.
t := gameTypes[m.GameType]
if t == nil {
return nil, errors.New("unknown game type")
}
m.MatchData = reflect.New(t.Elem()).Interface()
// Decode the raw message to the match data.
return m, json.Unmarshal(raw, m.MatchData)
}
Run it on the playground.
This question already has answers here:
How to specify default values when parsing JSON in Go
(4 answers)
Closed 3 years ago.
I want to set a default value to a field when unmarshal from a json string.
I know i can set the value i want before unmarshaling, I think it's not a beautiful way.
Is there any way else, like using a "default" tag?
func main() {
in := "{}"
myStruct := StructTest{}
json.Unmarshal([]byte(in), &myStruct)
fmt.Println(myStruct)
}
type StructTest struct {
V int64 `default:1`
}
What you can do is define a custom unmarshal function and decide there if you want to use the default value or not. In case you have other fields in StructTest you will want to create an alias for StructTest in UnmarshalJSON so that other fields will still be treated the same, while V will be overridden.
The snippet below shows one way to do it, also check out this working Go playground example.
type StructTest struct {
V int64
Other string // this field should be unmarshaled the regular way
}
func (st *StructTest) UnmarshalJSON(data []byte) error {
// create alias to prevent endless loop
type Alias StructTest
tmp := struct {
*Alias
V *int64
}{
Alias: (*Alias)(st),
}
// unmarshal into temporary struct
err := json.Unmarshal(data, &tmp)
if err != nil {
return err
}
// check if V was supplied in JSON and set default value if it wasn't
if tmp.V == nil {
st.V = 1 // default
} else {
st.V = *tmp.V
}
return nil
}
EDIT:
Actually for this simple example it can be done even simpler:
func (st *StructTest) UnmarshalJSON(data []byte) error {
st.V = 1 // set default value before unmarshaling
type Alias StructTest // create alias to prevent endless loop
tmp := (*Alias)(st)
return json.Unmarshal(data, tmp)
}
The short answer is no, and for cases like this:
type T struct {
Field1 int
Field2 int
}
...
func foo(data []byte) error {
var x T
if err := json.Unmarshal(data, &x); err != nil {
return err
}
// now set Field1 and/or Field2 to default values if needed
// ... but this is hard to do ...
use(x)
return nil
}
it's simple and easy to do it the other way, i.e.:
func foo(data []byte) error {
x := T{Field1: 99 /* red ballons */, Field2: 42 /* The Answer */ }
if err := json.Unmarshal(data, &x); err != nil {
return err
}
// now Field1 and/or Field2 are already set to default values if needed
use(x)
return nil
}
But what if the default is hard to compute? For instance:
type T2 struct {
Answer int
Question string
}
and function foo should have a default Answer of 42 as before, but the default Question should be the one that the mice were trying to compute, and obviously we don't want to spend a few millenia computing it if we don't have to. So we can't pre-initialize x, and we need to know if a question was provided.1
Yet another alternative, of course, is to decode into a struct with a pointer, then convert that nullable thing to the struct that doesn't have a pointer; we know whether the nullable variant's field was filled in or not because it's non-nil if it was filled-in. This produces code of this sort:
type T_ struct {
Answer int
Question *string
}
and fill in one variable x_ of type T_:
func foo(data []byte) error {
var x T2
x_ := T_{Answer: 42}
if err := json.Unmarshal(data, &x_); err != nil {
return err
}
x.Answer = x_.Answer
if x_.Question = nil {
x.Question = computeTheQuestion(x_.Answer)
} else {
x.Question = *x_.Question
}
use(x)
return nil
}
However, once again I lament (at least slightly) the ability to unmarshal json data with code like this hypothetical interface:
func foo(data []byte) error {
var objectish map[string]interface{}
if err := json.Unmarshal(data, &objectish); err != nil {
return err
}
x := T2{Answer: 42}
if err := json.ReUnmarshal(objectish, &x); err != nil {
return err
}
// We now know that the object-ish decoded JSON has the right
// "shape" for a T, and variable x is filled in with a default
// Answer. Its Question is the empty string if there was an
// empty Question, or if there was *no* Question at all, so
// now let's find out if we need to compute the right Question.
if _, ok := objectish["Question"]; !ok {
x.Question = computeTheQuestion(x.Answer)
}
use(x) // pass x to a hoopy frood
return nil
}
This hypothetical ReUnmarshal—which could actually just be Unmarshal itself, really—would, if given an interface{}, treat its value as resulting from an earlier Unmarshal and just re-type the result. If given a map[string]interface{} it would re-type the object. If given a map[string]map[string]interface{} it would do the obvious thing here as well, and so on. The only change the user sees is that Unmarshal (or ReUnmarshal, if changing the type signature is too rude) now takes:
[]byte, or
string, probably, just for convenience, or
json.RawMessage, because that already does almost the right thing, or
map[string]T for T being any of the types it accepts (including map recursively)
and it does the obvious thing with each of these.
Note that this is pretty similar to using json.RawMessage. However, when using json.RawMessage, we're right back to writing out a variant of the original struct type, with the same field names. See this example in the Go Playground, where we have to declare x_ (rather than objectish) with the type that uses the json.RawMessage argument.
(Alternatively, you can create a type with its own unmarshal function, as in lmazgon's answer. But again you must invent a type, rather than just decoding right into the already-supplied target type.)
1In this case, we could use Question *string, or assume that null strings are not allowed, or something, but this example is pretty reductionist. Suppose instead that we're supposed to compute a bignum with an accurate pi to some number of places, or any other hard but realistic computation. The point here is that there can be situations where pre-loading the defaults is relatively expensive, so we'd like to avoid that.
I have some JSON that I am unmarshalling into various structs so that I can then process the data, seemingly this is turning into the hardest part of the project!!!
The format of this JSON is that if the field is missing then it is essentially nil. This is following on from Default struct values but thought it deserved it's own question on SO.
A zero therefore is a valid value and I need to be able to discern this in my Go code. Is there a way to get Go to unmarshal into this struct with pointers at all?
In the example playground you can see what I mean, it "appears" to work but when I come to print out one of the pointer values it always prints the pointer address and not the actual value.
package main
import "fmt"
import "log"
import "encoding/json"
const inputMissing = `
["AAAAAA", {"testCode" : "Sss"}, 123456]
`
const inputZero = `
["AAAAAA", {"testCode" : "Sss", "missingInt" : 0, "defaultInt" : 0,"missingString" : "", "defaultString" : ""}, 123456]
`
type RawMessage struct {
AlwaysString string
ClientData ClientData
ReceptionTime int
}
type ClientData struct {
TestCode string
MissingInt *int
DefaultInt int
MissingString *string
DefaultString string
}
func main() {
var n RawMessage
if err := json.Unmarshal([]byte(inputMissing), &n); err != nil {
log.Fatal(err)
}
fmt.Printf("%#v\n", n)
var o RawMessage
if err := json.Unmarshal([]byte(inputZero), &o); err != nil {
log.Fatal(err)
}
fmt.Printf("%#v\n", o)
fmt.Printf("Should print the value of the int, not pointer... %i", o.ClientData.MissingInt)
}
func (n *RawMessage) UnmarshalJSON(buf []byte) error {
tmp := []interface{}{&n.AlwaysString, &n.ClientData, &n.ReceptionTime}
wantLen := len(tmp)
if err := json.Unmarshal(buf, &tmp); err != nil {
return err
}
if g, e := len(tmp), wantLen; g != e {
return fmt.Errorf("wrong number of fields in RawMessage: %d != %d", g, e)
}
return nil
}
Your code is correct. Test the pointer against nil and dereference the pointer if you want the value:
fmt.Printf("Should print the value of the int, not pointer... %d", *o.ClientData.MissingInt)
Your confusion arises from the default formatting of the fmt package, which prints the hex value in case of pointer fields, and not the pointed value.
If you're using the %#v verb for printing, you may implement the fmt.GoStringer interface on your struct to override this "behavior":
func (c ClientData) GoString() string {
mi := "<missing>"
if c.MissingInt != nil {
mi = strconv.Itoa(*c.MissingInt)
}
ms := "<missing>"
if c.MissingString != nil {
ms = *c.MissingString
}
return fmt.Sprintf("{TestCode: %s, MissingInt: %s, DefaultInt: %d, MissingString: %s, DefaultString: %s}",
c.TestCode, mi, c.DefaultInt, ms, c.DefaultString)
}
And changing the last printing line to:
fmt.Printf("Should print the value of the int, not pointer... %d",
*o.ClientData.MissingInt)
Your output becomes more readable (try it on the Go Playground):
main.RawMessage{AlwaysString:"AAAAAA", ClientData:{TestCode: Sss, MissingInt: <missing>, DefaultInt: 0, MissingString: <missing>, DefaultString: }, ReceptionTime:123456}
main.RawMessage{AlwaysString:"AAAAAA", ClientData:{TestCode: Sss, MissingInt: 0, DefaultInt: 0, MissingString: , DefaultString: }, ReceptionTime:123456}
Should print the value of the int, not pointer... 0
As you can see from the output above, the MissingXX fields have the static value <missing> displayed.
Note: If you were to use the %v verb, you'd have to implement the fmt.Stringer interface, which is basically the same, just the method name is not GoString() but simply String().
With PHP and JavaScript (and Node) parsing JSON is a very trivial operation. From the looks of it Go is rather more complicated. Consider the example below:
package main
import ("encoding/json";"fmt")
type fileData struct{
tn string
size int
}
type jMapA map[string] string
type jMapB map[string] fileData
func parseMapA(){
var dat jMapA
s := `{"lang":"Node","compiled":"N","fast":"maybe"}`
if err := json.Unmarshal([]byte(s), &dat); err != nil {
panic(err)
}
fmt.Println(dat);
for k,v := range dat{
fmt.Println(k,v)
}
}
func parseMapB(){
var dat jMapB
s := `{"f1":{"tn":"F1","size":1024},"f2":{"tn":"F2","size":2048}}`
if err := json.Unmarshal([]byte(s), &dat); err != nil {
panic(err)
}
fmt.Println(dat);
for k,v := range dat{
fmt.Println(k,v)
}
}
func main() {
parseMapA()
parseMapB()
}
The parseMapA() call obligingly returns:
map[lang:Node Compiled:N fast:maybe]
lang Node
compiled N
fast maybe
However, parseMapB() returns:
map[f1:{ 0}, f2:{ 0}]
f2 { 0}
f1 { 0}
I am into my first few hours with Go so I imagine I am doing something wrong here. However, I have no idea what that might be. More generally, what would the Go equivalent of the Node code
for(p in obj){
doSomethingWith(obj[p]);
}
be in Go?
In Go, unmarshaling works only if a struct has exported fields, ie. fields that begin with a capital letter.
Change your first structure to:
type fileData struct{
Tn string
Size int
}
See http://play.golang.org/p/qO3U7ZNrNs for a fixed example.
Moreover, if you intend to marshal this struct back to JSON, you'll notice that the resulting JSON use capitalized fields, which is not what you want.
You need to add field tags:
type fileData struct{
Tn string `json:"tn"`
Size int `json:"size"`
}
so the Marshal function will use the correct names.