Golang: JSON: How do I unmarshal array of strings into []int64 - json

Golang encoding/json package lets you use ,string struct tag in order to marshal/unmarshal string values (like "309230") into int64 field. Example:
Int64String int64 `json:",string"`
However, this doesn't work for slices, ie. []int64:
Int64Slice []int64 `json:",string"` // Doesn't work.
Is there any way to marshal/unmarshal JSON string arrays into []int64 field?
Quote from https://golang.org/pkg/encoding/json:
The "string" option signals that a field is stored as JSON inside a JSON-encoded string. It applies only to fields of string, floating point, integer, or boolean types. This extra level of encoding is sometimes used when communicating with JavaScript programs:

For anyone interested, I found a solution using a custom type having MarshalJSON() and UnmarshalJSON() methods defined.
type Int64StringSlice []int64
func (slice Int64StringSlice) MarshalJSON() ([]byte, error) {
values := make([]string, len(slice))
for i, value := range []int64(slice) {
values[i] = fmt.Sprintf(`"%v"`, value)
}
return []byte(fmt.Sprintf("[%v]", strings.Join(values, ","))), nil
}
func (slice *Int64StringSlice) UnmarshalJSON(b []byte) error {
// Try array of strings first.
var values []string
err := json.Unmarshal(b, &values)
if err != nil {
// Fall back to array of integers:
var values []int64
if err := json.Unmarshal(b, &values); err != nil {
return err
}
*slice = values
return nil
}
*slice = make([]int64, len(values))
for i, value := range values {
value, err := strconv.ParseInt(value, 10, 64)
if err != nil {
return err
}
(*slice)[i] = value
}
return nil
}
The above solution marshals []int64 into JSON string array. Unmarshaling works from both JSON string and integer arrays, ie.:
{"bars": ["1729382256910270462", "309286902808622", "23"]}
{"bars": [1729382256910270462, 309286902808622, 23]}
See example at https://play.golang.org/p/BOqUBGR3DXm

As you quoted from json.Marshal(), the ,string option only applies to specific types, namely:
The "string" option signals that a field is stored as JSON inside a JSON-encoded string. It applies only to fields of string, floating point, integer, or boolean types.
You want it to work with a slice, but that is not supported by the json package.
If you still want this functionality, you have to write your custom marshaling / unmarshaling logic.
What you presented works, but it is unnecessarily complex. This is because you created your custom logic on slices, but you only want this functionality on individual elements of the slices (arrays). You don't want to change how an array / slice (as a sequence of elements) is rendered or parsed.
So a much simpler solution is to only create a custom "number" type producing this behavior, and elements of slices of this custom type will behave the same.
Our custom number type and the marshaling / unmarshaling logic:
type Int64Str int64
func (i Int64Str) MarshalJSON() ([]byte, error) {
return json.Marshal(strconv.FormatInt(int64(i), 10))
}
func (i *Int64Str) UnmarshalJSON(b []byte) error {
// Try string first
var s string
if err := json.Unmarshal(b, &s); err == nil {
value, err := strconv.ParseInt(s, 10, 64)
if err != nil {
return err
}
*i = Int64Str(value)
return nil
}
// Fallback to number
return json.Unmarshal(b, (*int64)(i))
}
And that's all!
The type using it:
type Foo struct {
Bars []Int64Str `json:"bars"`
}
Testing it the same way as you did yields the same result. Try it on the Go Playground.

Related

Custom unmarshaling a struct into a map of slices

I thought I understood unmarshalling by now, but I guess not. I'm having a little bit of trouble unmarshalling a map in go. Here is the code that I have so far
type OHLC_RESS struct {
Pair map[string][]Candles
Last int64 `json:"last"`
}
type Candles struct {
Time uint64
Open string
High string
Low string
Close string
VWAP string
Volume string
Count int
}
func (c *Candles) UnmarshalJSON(d []byte) error {
tmp := []interface{}{&c.Time, &c.Open, &c.High, &c.Low, &c.Close, &c.VWAP, &c.Volume, &c.Count}
length := len(tmp)
err := json.Unmarshal(d, &tmp)
if err != nil {
return err
}
g := len(tmp)
if g != length {
return fmt.Errorf("Lengths don't match: %d != %d", g, length)
}
return nil
}
func main() {
response := []byte(`{"XXBTZUSD":[[1616662740,"52591.9","52599.9","52591.8","52599.9","52599.1","0.11091626",5],[1616662740,"52591.9","52599.9","52591.8","52599.9","52599.1","0.11091626",5]],"last":15}`)
var resp OHLC_RESS
err := json.Unmarshal(response, &resp)
fmt.Println("resp: ", resp)
}
after running the code, the last field will unmarshal fine, but for whatever reason, the map is left without any value. Any help?
The expedient solution, for the specific example JSON, would be to NOT use a map at all but instead change the structure of OHLC_RESS so that it matches the structure of the JSON, i.e.
type OHLC_RESS struct {
Pair []Candles `json:"XXBTZUSD"`
Last int64 `json:"last"`
}
https://go.dev/play/p/Z9PhJt3wX33
However it's safe to assume, I think, that the reason you've opted to use a map is because the JSON object's key(s) that hold the "pairs" can vary and so hardcoding them into the field's tag is out of the question.
To understand why your code doesn't produce the desired result, you have to realize two things. First, the order of a struct's fields has no bearing on how the keys of a JSON object will be decoded. Second, the name Pair holds no special meaning for the unmarshaler. Therefore, by default, the unmarshaler has no way of knowing that your wish is to decode the "XXBTZUSD": [ ... ] element into the Pair map.
So, to get your desired result, you can have the OHLC_RESS implement the json.Unmarshaler interface and do the following:
func (r *OHLC_RESS) UnmarshalJSON(d []byte) error {
// first, decode just the object's keys and leave
// the values as raw, non-decoded JSON
var obj map[string]json.RawMessage
if err := json.Unmarshal(d, &obj); err != nil {
return err
}
// next, look up the "last" element's raw, non-decoded value
// and, if it is present, then decode it into the Last field
if last, ok := obj["last"]; ok {
if err := json.Unmarshal(last, &r.Last); err != nil {
return err
}
// remove the element so it's not in
// the way when decoding the rest below
delete(obj, "last")
}
// finally, decode the rest of the element values
// in the object and store them in the Pair field
r.Pair = make(map[string][]Candles, len(obj))
for key, val := range obj {
cc := []Candles{}
if err := json.Unmarshal(val, &cc); err != nil {
return err
}
r.Pair[key] = cc
}
return nil
}
https://go.dev/play/p/Lj8a8Gx9fWH

Reading data from mysql json column in go [duplicate]

I am facing a case where I had to store dynamic values in the database with key and value pairs given by the user itself.
User gives the key and value, and I interpret it as
{"key": "user Given", "value": "user Given"}
and I add all such things to an array, and I want this array to be read into Go code where the array of objects is read from the Database table.
You can use the JSON Unmarshaler interface, but depending on how you are retrieving the data from MySql, will vary your implementation. But the idea is the same. For this example, I use https://github.com/go-sql-driver/mysql and assuming you want to store the data in an actual JSON field (Mysql >= 5.7), you could do something like this:
type MyField struct {
Key string `json:"key"`
Value string `json:"value"`
}
type MyModel struct {
ID uint64
MyFieldName MyField `json:"my_field_name"`
}
func (m *MyField) Scan(src interface{}) error {
// The data stored in a JSON field is actually returned as []uint8
val := src.([]uint8)
return json.Unmarshal(val, &m)
}
func main() {
db, err := sql.Open("mysql", "root:password#tcp(127.0.0.1)/dbname")
if err != nil {
panic(err.Error())
}
defer db.Close()
results, err := db.Query("SELECT id, my_field_name FROM my_table")
if err != nil {
panic(err.Error())
}
for results.Next() {
var row MyModel
err := results.Scan(&row.ID, &row.MyFieldName)
if err != nil {
panic(err.Error())
}
fmt.Println(row.MyFieldName.Key)
}
}
A quick hack, not necessarily the most elegant approach, is to use Golang's default JSON Unmarshaler behavior for golang maps:
jstr := `{"key": "userKeyValue", "value": "userValueValue"}`
// declare a map that has a key string and value interface{} so that any values or
// types will be accepted;
jmap := make(map[string]interface{})
err := json.Unmarshal(jstr, &jmap)
if err != nil {
log.Fatal(err)
}
for k, v := range jmap {
fmt.Printf("Key: %v, Value: %v\n", k, v)
// If you want to directly work with v remember it is an interface{}
// Thus to use it as a string you must v.(string)
}
// Will output the following:
// Key: key, Value: userKeyValue
// Key: value, Value: userValueValue
You can now use standard golang map to manipulate or manage the received data... The more elegant approach is to implement the JSON Marshaler and Unmarshaler interfaces for your declared type. These are described in the golang documentation: https://golang.org/pkg/encoding/json/

Decode JSON value which can be either string or number

When I make an HTTP call to a REST API I may get the JSON value count back as a Number or String. I'ld like to marshal it to be an integer in either case. How can I deal with this in Go?.
Use the "string" field tag option to specify that strings should be converted to numbers. The documentation for the option is:
The "string" option signals that a field is stored as JSON inside a JSON-encoded string. It applies only to fields of string, floating point, integer, or boolean types. This extra level of encoding is sometimes used when communicating with JavaScript programs:
Here's an example use:
type S struct {
Count int `json:"count,string"`
}
playground example
If the JSON value can be number or string, then unmarshal to interface{} and convert to int after unmarshaling:
Count interface{} `json:"count,string"`
Use this function to convert the interface{} value to an int:
func getInt(v interface{}) (int, error) {
switch v := v.(type) {
case float64:
return int(v), nil
case string:
c, err := strconv.Atoi(v)
if err != nil {
return 0, err
}
return c, nil
default:
return 0, fmt.Errorf("conversion to int from %T not supported", v)
}
}
// Format of your expected request
type request struct {
ACTIVE string `json:"active"`
CATEGORY string `json:"category"`
}
// struct to read JSON input
var myReq request
// Decode the received JSON request to struct
decoder := json.NewDecoder(r.Body)
err := decoder.Decode(&myReq)
if err != nil {
log.Println( err)
// Handler for invalid JSON received or if you want to decode the request using another struct with int.
return
}
defer r.Body.Close()
// Convert string to int
numActive, err = strconv.Atoi(myReq.ACTIVE)
if err != nil {
log.Println(err)
// Handler for invalid int received
return
}
// Convert string to int
numCategory, err = strconv.Atoi(myReq.CATEGORY)
if err != nil {
log.Println(err)
// Handler for invalid int received
return
}
I had the same problem with a list of values where the values were string or struct. The solution I'm using is to create a helper struct with fields of expected types and parse value into the correct field.
type Flag struct {
ID string `json:"id"`
Type string `json:"type"`
}
type FlagOrString struct {
Flag *Flag
String *string
}
func (f *FlagOrString) UnmarshalJSON(b []byte) error {
start := []byte("\"")
for idx := range start {
if b[idx] != start[idx] {
return json.Unmarshal(b, &f.Flag)
}
}
return json.Unmarshal(b, &f.String)
}
var MainStruct struct {
Vals []FlagOrString
}
Custom Unmarshaller simplifies a code. Personally I prefer this over interface{} as it explicitly states what a developer expects.

Deserialize patterned fields In Golang

I am writing golang struct, which are compatible with some json structure. However, those most of the fields are know, there will be few fields following some specific patterns(like "x-{randomName}") in the json definition, which I also want to get deserialized to a certain field as map[string]interface{} as well.
Is there any descent way to achieve it?
It's less efficient, but you could unmarshal twice to avoid manually mapping the fields. Once to put all the properly tagged fields into the struct, and then again into a map[string]interface{} to get everything else. If you don't care about the duplicate fields, you don't even need to filter the second map.
You can even do this in an UnmarshalJSONmethod to automatically populate the struct
type S struct {
A string `json:"a"`
B string `json:"b"`
All map[string]interface{}
}
func (s *S) UnmarshalJSON(b []byte) error {
// create a new type to hide the UnmarshalJSON method
// otherwise we'll recurse indefinitely.
type ss S
err := json.Unmarshal(b, (*ss)(s))
if err != nil {
return err
}
// now unmarshal again into the All map
err = json.Unmarshal(b, &s.All)
if err != nil {
return err
}
return nil
}
http://play.golang.org/p/VBVlRjNlHy

How to parse both plain and enquoted JSON numbers in Go?

I'm dealing with a third-party JSON-based API. Usually it wraps all numbers in quotes, but sometimes it doesn't. Nothing I can do about it.
I'm trying to come up with a solution, which parses the numbers regardless of whether they're enquoted or not. Standard library provides a ,string field tag, which allows to map numeric fields to enquoted values, but, unfortunately, it then fails to process the value, if it's not in quotes.
func test(s string) {
err := json.Unmarshal([]byte(s), &struct {
F1 float64
F2 float64 `json:",string"`
}{})
if err != nil {
fmt.Println(err)
return
}
fmt.Println("success")
}
func main() {
test(`{"f1": 1.23}`) // success
test(`{"f1": "1.23"}`) // cannot unmarshal string into Go value of type float64
test(`{"f2": 1.23}`) // invalid use of ,string struct tag, trying to unmarshal unquoted value into ...
test(`{"f2": "1.23"}`) // success
}
The Go Playground
Is there a way around this?
The proper solution is to "clone" float64 and define custom MarshalJSON and UnmarshalJSON for it:
type jsonFloat64 float64
func (f jsonFloat64) MarshalJSON() ([]byte, error) {
return json.Marshal(float64(f))
}
func (f *jsonFloat64) UnmarshalJSON(data []byte) error {
if len(data) >= 2 && data[0] == '"' && data[len(data)-1] == '"' {
data = data[1 : len(data)-1]
}
var tmp float64
err := json.Unmarshal(data, &tmp)
if err != nil {
return err
}
*f = jsonFloat64(tmp)
return nil
}
Then you'd be able to do something like this:
func test(s string) {
err := json.Unmarshal([]byte(s), &struct {
F jsonFloat64
}{})
if err != nil {
fmt.Println(err)
return
}
fmt.Println("success")
}
func main() {
test(`{"f": 1.23}`) // success
test(`{"f": "1.23"}`) // success
}
The Go Playground
Feel free to adjust UnmarshalJSON to your needs, mine is pretty strict about the spacing. Credit goes to Dave C, this was heavily inspired by his comment on another question (which also features more variations on the solution above).
Alternatively, you could pre-process the JSON data with a regular expression, but don't do this if the solution above is a viable option, it's much faster.
re := regexp.MustCompile(`(":\s*)([\d\.]+)(\s*[,}])`)
rawJsonByteArray = re.ReplaceAll(rawJsonByteArray, []byte(`$1"$2"$3`))