Reading data from mysql json column in go [duplicate] - mysql

I am facing a case where I had to store dynamic values in the database with key and value pairs given by the user itself.
User gives the key and value, and I interpret it as
{"key": "user Given", "value": "user Given"}
and I add all such things to an array, and I want this array to be read into Go code where the array of objects is read from the Database table.

You can use the JSON Unmarshaler interface, but depending on how you are retrieving the data from MySql, will vary your implementation. But the idea is the same. For this example, I use https://github.com/go-sql-driver/mysql and assuming you want to store the data in an actual JSON field (Mysql >= 5.7), you could do something like this:
type MyField struct {
Key string `json:"key"`
Value string `json:"value"`
}
type MyModel struct {
ID uint64
MyFieldName MyField `json:"my_field_name"`
}
func (m *MyField) Scan(src interface{}) error {
// The data stored in a JSON field is actually returned as []uint8
val := src.([]uint8)
return json.Unmarshal(val, &m)
}
func main() {
db, err := sql.Open("mysql", "root:password#tcp(127.0.0.1)/dbname")
if err != nil {
panic(err.Error())
}
defer db.Close()
results, err := db.Query("SELECT id, my_field_name FROM my_table")
if err != nil {
panic(err.Error())
}
for results.Next() {
var row MyModel
err := results.Scan(&row.ID, &row.MyFieldName)
if err != nil {
panic(err.Error())
}
fmt.Println(row.MyFieldName.Key)
}
}

A quick hack, not necessarily the most elegant approach, is to use Golang's default JSON Unmarshaler behavior for golang maps:
jstr := `{"key": "userKeyValue", "value": "userValueValue"}`
// declare a map that has a key string and value interface{} so that any values or
// types will be accepted;
jmap := make(map[string]interface{})
err := json.Unmarshal(jstr, &jmap)
if err != nil {
log.Fatal(err)
}
for k, v := range jmap {
fmt.Printf("Key: %v, Value: %v\n", k, v)
// If you want to directly work with v remember it is an interface{}
// Thus to use it as a string you must v.(string)
}
// Will output the following:
// Key: key, Value: userKeyValue
// Key: value, Value: userValueValue
You can now use standard golang map to manipulate or manage the received data... The more elegant approach is to implement the JSON Marshaler and Unmarshaler interfaces for your declared type. These are described in the golang documentation: https://golang.org/pkg/encoding/json/

Related

Custom unmarshaling a struct into a map of slices

I thought I understood unmarshalling by now, but I guess not. I'm having a little bit of trouble unmarshalling a map in go. Here is the code that I have so far
type OHLC_RESS struct {
Pair map[string][]Candles
Last int64 `json:"last"`
}
type Candles struct {
Time uint64
Open string
High string
Low string
Close string
VWAP string
Volume string
Count int
}
func (c *Candles) UnmarshalJSON(d []byte) error {
tmp := []interface{}{&c.Time, &c.Open, &c.High, &c.Low, &c.Close, &c.VWAP, &c.Volume, &c.Count}
length := len(tmp)
err := json.Unmarshal(d, &tmp)
if err != nil {
return err
}
g := len(tmp)
if g != length {
return fmt.Errorf("Lengths don't match: %d != %d", g, length)
}
return nil
}
func main() {
response := []byte(`{"XXBTZUSD":[[1616662740,"52591.9","52599.9","52591.8","52599.9","52599.1","0.11091626",5],[1616662740,"52591.9","52599.9","52591.8","52599.9","52599.1","0.11091626",5]],"last":15}`)
var resp OHLC_RESS
err := json.Unmarshal(response, &resp)
fmt.Println("resp: ", resp)
}
after running the code, the last field will unmarshal fine, but for whatever reason, the map is left without any value. Any help?
The expedient solution, for the specific example JSON, would be to NOT use a map at all but instead change the structure of OHLC_RESS so that it matches the structure of the JSON, i.e.
type OHLC_RESS struct {
Pair []Candles `json:"XXBTZUSD"`
Last int64 `json:"last"`
}
https://go.dev/play/p/Z9PhJt3wX33
However it's safe to assume, I think, that the reason you've opted to use a map is because the JSON object's key(s) that hold the "pairs" can vary and so hardcoding them into the field's tag is out of the question.
To understand why your code doesn't produce the desired result, you have to realize two things. First, the order of a struct's fields has no bearing on how the keys of a JSON object will be decoded. Second, the name Pair holds no special meaning for the unmarshaler. Therefore, by default, the unmarshaler has no way of knowing that your wish is to decode the "XXBTZUSD": [ ... ] element into the Pair map.
So, to get your desired result, you can have the OHLC_RESS implement the json.Unmarshaler interface and do the following:
func (r *OHLC_RESS) UnmarshalJSON(d []byte) error {
// first, decode just the object's keys and leave
// the values as raw, non-decoded JSON
var obj map[string]json.RawMessage
if err := json.Unmarshal(d, &obj); err != nil {
return err
}
// next, look up the "last" element's raw, non-decoded value
// and, if it is present, then decode it into the Last field
if last, ok := obj["last"]; ok {
if err := json.Unmarshal(last, &r.Last); err != nil {
return err
}
// remove the element so it's not in
// the way when decoding the rest below
delete(obj, "last")
}
// finally, decode the rest of the element values
// in the object and store them in the Pair field
r.Pair = make(map[string][]Candles, len(obj))
for key, val := range obj {
cc := []Candles{}
if err := json.Unmarshal(val, &cc); err != nil {
return err
}
r.Pair[key] = cc
}
return nil
}
https://go.dev/play/p/Lj8a8Gx9fWH

Parse JSON having sibling dynamic keys alongside with static in Go

I need to parse this json
{
"version": "1.1.29-snapshot",
"linux-amd64": {
"url": "https://origin/path",
"size": 7794688,
"sha256": "14b3c3ad05e3a98d30ee7e774646aec7ffa8825a1f6f4d9c01e08bf2d8a08646"
},
"windows-amd64": {
"url": "https://origin/path",
"size": 8102400,
"sha256": "01b8b927388f774bdda4b5394e381beb592d8ef0ceed69324d1d42f6605ab56d"
}
}
Keys like linux-amd64 are dynamic and theirs amount is arbitrary. I tried something like that to describe it and unmarshal. Obviously it doesn't work. Items is always empty.
type FileInfo struct {
Url string `json:"url"`
Size int64 `json:"size"`
Sha256 string `json:"sha256"`
}
type UpdateInfo struct {
Version string `json:"version"`
Items map[string]FileInfo
}
It's similar to this use case, but has no parent key items. I suppose I can use 3rd party library or map[string]interface{} approach, but I'm interested in knowing how to achieve this with explicitly declared types.
The rest of the parsing code is:
func parseUpdateJson(jsonStr []byte) (UpdateInfo, error) {
var allInfo = UpdateInfo{Items: make(map[string]FileInfo)}
var err = json.Unmarshal(jsonStr, &allInfo)
return allInfo, err
}
Look at the link I attached and you will realize that is not that simple as you think. Also I pointed that I interested in typed approach. Ok, how to declare this map[string]FileInfo to get parsed?
You can create a json.Unmarshaller to decode the json into a map, then apply those values to your struct: https://play.golang.org/p/j1JXMpc4Q9u
type FileInfo struct {
Url string `json:"url"`
Size int64 `json:"size"`
Sha256 string `json:"sha256"`
}
type UpdateInfo struct {
Version string `json:"version"`
Items map[string]FileInfo
}
func (i *UpdateInfo) UnmarshalJSON(d []byte) error {
tmp := map[string]json.RawMessage{}
err := json.Unmarshal(d, &tmp)
if err != nil {
return err
}
err = json.Unmarshal(tmp["version"], &i.Version)
if err != nil {
return err
}
delete(tmp, "version")
i.Items = map[string]FileInfo{}
for k, v := range tmp {
var item FileInfo
err := json.Unmarshal(v, &item)
if err != nil {
return err
}
i.Items[k] = item
}
return nil
}
This answer is adapted from this recipe in my YouTube video on advanced JSON handling in Go.
func (u *UpdateInfo) UnmarshalJSON(d []byte) error {
var x struct {
UpdateInfo
UnmarshalJSON struct{}
}
if err := json.Unmarshal(d, &x); err != nil {
return err
}
var y map[string]json.RawMessage{}
if err := json.Unsmarshal(d, &y); err != nil {
return err
}
delete(y, "version"_ // We don't need this in the map
*u = x.UpdateInfo
u.Items = make(map[string]FileInfo, len(y))
for k, v := range y {
var info FileInfo
if err := json.Unmarshal(v, &info); err != nil {
return err
}
u.Items[k] = info
}
return nil
}
It:
Unmarshals the JSON into the struct directly, to get the struct fields.
It re-unmarshals into a map of map[string]json.RawMessage to get the arbitrary keys. This is necessary since the value of version is not of type FileInfo, and trying to unmarshal directly into map[string]FileInfo will thus error.
It deletes the keys we know we already got in the struct fields.
It then iterates through the map of string to json.RawMessage, and finally unmarshals each value into the FileInfo type, and stores it in the final object.
If you really don't want to unmarshal multiple times, your next best option is to iterate over the JSON tokens in your input by using the json.Decoder type. I've done this in a couple of performance-sensitive bits of code, but it makes your code INCREDIBLY hard to read, and in almost all cases is not worth the effort.

Golang: JSON: How do I unmarshal array of strings into []int64

Golang encoding/json package lets you use ,string struct tag in order to marshal/unmarshal string values (like "309230") into int64 field. Example:
Int64String int64 `json:",string"`
However, this doesn't work for slices, ie. []int64:
Int64Slice []int64 `json:",string"` // Doesn't work.
Is there any way to marshal/unmarshal JSON string arrays into []int64 field?
Quote from https://golang.org/pkg/encoding/json:
The "string" option signals that a field is stored as JSON inside a JSON-encoded string. It applies only to fields of string, floating point, integer, or boolean types. This extra level of encoding is sometimes used when communicating with JavaScript programs:
For anyone interested, I found a solution using a custom type having MarshalJSON() and UnmarshalJSON() methods defined.
type Int64StringSlice []int64
func (slice Int64StringSlice) MarshalJSON() ([]byte, error) {
values := make([]string, len(slice))
for i, value := range []int64(slice) {
values[i] = fmt.Sprintf(`"%v"`, value)
}
return []byte(fmt.Sprintf("[%v]", strings.Join(values, ","))), nil
}
func (slice *Int64StringSlice) UnmarshalJSON(b []byte) error {
// Try array of strings first.
var values []string
err := json.Unmarshal(b, &values)
if err != nil {
// Fall back to array of integers:
var values []int64
if err := json.Unmarshal(b, &values); err != nil {
return err
}
*slice = values
return nil
}
*slice = make([]int64, len(values))
for i, value := range values {
value, err := strconv.ParseInt(value, 10, 64)
if err != nil {
return err
}
(*slice)[i] = value
}
return nil
}
The above solution marshals []int64 into JSON string array. Unmarshaling works from both JSON string and integer arrays, ie.:
{"bars": ["1729382256910270462", "309286902808622", "23"]}
{"bars": [1729382256910270462, 309286902808622, 23]}
See example at https://play.golang.org/p/BOqUBGR3DXm
As you quoted from json.Marshal(), the ,string option only applies to specific types, namely:
The "string" option signals that a field is stored as JSON inside a JSON-encoded string. It applies only to fields of string, floating point, integer, or boolean types.
You want it to work with a slice, but that is not supported by the json package.
If you still want this functionality, you have to write your custom marshaling / unmarshaling logic.
What you presented works, but it is unnecessarily complex. This is because you created your custom logic on slices, but you only want this functionality on individual elements of the slices (arrays). You don't want to change how an array / slice (as a sequence of elements) is rendered or parsed.
So a much simpler solution is to only create a custom "number" type producing this behavior, and elements of slices of this custom type will behave the same.
Our custom number type and the marshaling / unmarshaling logic:
type Int64Str int64
func (i Int64Str) MarshalJSON() ([]byte, error) {
return json.Marshal(strconv.FormatInt(int64(i), 10))
}
func (i *Int64Str) UnmarshalJSON(b []byte) error {
// Try string first
var s string
if err := json.Unmarshal(b, &s); err == nil {
value, err := strconv.ParseInt(s, 10, 64)
if err != nil {
return err
}
*i = Int64Str(value)
return nil
}
// Fallback to number
return json.Unmarshal(b, (*int64)(i))
}
And that's all!
The type using it:
type Foo struct {
Bars []Int64Str `json:"bars"`
}
Testing it the same way as you did yields the same result. Try it on the Go Playground.

JSON single value parsing

In python you can take a json object and grab a specific item from it without declaring a struct, saving to a struct then obtaining the value like in Go. Is there a package or easier way to store a specific value from json in Go?
python
res = res.json()
return res['results'][0]
Go
type Quotes struct {
AskPrice string `json:"ask_price"`
}
quote := new(Quotes)
errJson := json.Unmarshal(content, &quote)
if errJson != nil {
return "nil", fmt.Errorf("cannot read json body: %v", errJson)
}
You can decode into a map[string]interface{} and then get the element by key.
func main() {
b := []byte(`{"ask_price":"1.0"}`)
data := make(map[string]interface{})
err := json.Unmarshal(b, &data)
if err != nil {
panic(err)
}
if price, ok := data["ask_price"].(string); ok {
fmt.Println(price)
} else {
panic("wrong type")
}
}
Structs are often preferred as they are more explicit about the type. You only have to declare the fields in the JSON you care about, and you don't need to type assert the values as you would with a map (encoding/json handles that implicitly).
Try either fastjson or jsonparser. jsonparser is optimized for the case when a single JSON field must be selected, while fastjson is optimized for the case when multiple unrelated JSON fields must be selected.
Below is an example code for fastjson:
var p fastjson.Parser
v, err := p.Parse(content)
if err != nil {
log.Fatal(err)
}
// obtain v["ask_price"] as float64
price := v.GetFloat64("ask_price")
// obtain v["results"][0] as generic JSON value
result0 := v.Get("results", "0")

How do I turn an array of JSON objects into an array of structs with default values in Go?

I'm working on a Go API that can receive POSTs consisting of a JSON array of objects. The structure of the POST will look something like:
[
{
"name":"Las Vegas",
"size":14
},
{
"valid": false,
"name":"Buffalo",
"size":63
}
]
Let's say I have the following struct:
type Data {
Valid bool
Name string
Size float64
}
I want to create a bunch of Datas with Valid set to true anytime it's not actually specified in the JSON as false. If I were doing a single one I could use How to specify default values when parsing JSON in Go, but for doing an unknown number of them the only thing I've been able to come up with is something like:
var allMap []map[string]interface{}
var structs []Data
for _, item := range allMap {
var data Data
var v interface{}
var ok bool
if v, ok := item["value"]; ok {
data.Valid = v
} else {
data.Valid = true
}
id v, ok := item["name"]; ok {
data.Name = v
}
...
structs = append(structs, data)
}
return structs
Right now the struct I'm actually working with has 14 fields, some of them have values I want to assign defaults, others are fine to leave blank, but all of them have to be iterated through using this approach.
Is there a better way?
You can use the json.RawMessage type to defer unmarshaling some JSON text value. If you use this type, then the JSON text will be stored in this without unmarshaling (so you can unmarshal this fragment later on as you wish).
So in your case if you try to unmarshal into a slice of such RawMessage, you can use the technique what you linked in your question, that is you can iterate over the slice of raw values (which are the JSON text for each Data), create a Data struct with values you want as defaults for missing values, and unmarshal a slice element into this prepared struct. That's all.
It looks like this:
allJson := []json.RawMessage{}
if err := json.Unmarshal(src, &allJson); err != nil {
panic(err)
}
allData := make([]Data, len(allJson))
for i, v := range allJson {
// Here create your Data with default values
allData[i] = Data{Valid: true}
if err := json.Unmarshal(v, &allData[i]); err != nil {
panic(err)
}
}
Try it on the Go Playground.
Notes / Variants
For efficiency (to avoid copying structs), you can also make the allData to be a slice of pointers in the above example, which would look like this:
allData := make([]*Data, len(allJson))
for i, v := range allJson {
// Here create your Data with default values
allData[i] = &Data{Valid: true}
if err := json.Unmarshal(v, allData[i]); err != nil {
panic(err)
}
}
If you want to keep using non-pointers, for efficiency you can "prepare" your wished default values in the slice elements itself, which would look like this:
allData := make([]Data, len(allJson))
for i, v := range allJson {
// Here set your default values in the slice elements
// Only set those which defer from the zero values:
allData[i].Valid = true
if err := json.Unmarshal(v, &allData[i]); err != nil {
panic(err)
}
}
You can do a good trick by providing an UnmarshalJSON method on your type to make it transparent and work automatically even if your type is found within structs or slices.
func (d *Data) UnmarshalJSON(j []byte) error {
type _Data Data // Dummy type to avoid infinite recursion in UnmarshalJSON
tmp := _Data{ // Set defaults here
Valid: true,
}
err := json.Unmarshal(j, &tmp)
if err != nil {
return err
}
*d = Data(tmp)
return nil
}
The type _Data exists simply so that we can call json.Unmarshal(j, &tmp) and get the original un-overridden behavior, instead of calling the UnmarshalJSON method that we're already in the middle of. We can set default values on tmp using the trick that you already linked to. And then after the unmarshalling is done, we can just cast tmp to Data because after all Data and _Data are really the same type.
Given this method you can simply
var structs []Data
err := json.Unmarshal(input, &structs)
(or likewise with a json.Decoder) and have it work just the way you want.