Golang: unmarshal json object into a slice - json

I have JSON like
{
"a": {"key": "a", "value": 1,},
"b": {"key": "b", "value": 1,},
}
Is there a way to unmarshal it into []*struct {Key string; Value int}, preserving the order of the structures?
If I unmarshal the JSON into map[string]*struct {Key string; Value int} and then convert the map into a slice, I'll lose the order of the structures.

To preserve order, use Decoder.Token and Decoder.More to walk through the top-level JSON object.
r := strings.NewReader(`
{
"a": {"key": "a", "value": 1},
"b": {"key": "b", "value": 1}
}`)
d := json.NewDecoder(r)
t, err := d.Token()
if err != nil || t != json.Delim('{') {
log.Fatal("expected object")
}
var result []*element
for d.More() {
k, err := d.Token()
if err != nil {
log.Fatal(err)
}
var v element
if err := d.Decode(&v); err != nil {
log.Fatal(err)
}
result = append(result, &v)
fmt.Println("key:", k, "value:", v)
}
Run it on the Go Playground
Change the calls to log.Fatal to the error handling appropriate for your application.
This answer edits the JSON in the question to make the JSON valid.
The field names in the struct element type must be exported.

The easiest way that I found was to use jsonparser.ObjectEach:
import "github.com/buger/jsonparser"
...
var ss []*struct{Key string; Value int}
err = jsonparser.ObjectEach(data, func(key []byte, value []byte, dataType jsonparser.ValueType, offset int) error {
var s struct{Key string; Value int}
if err := json.Unmarshal(value, &s); err != nil {
return err
}
*ss = append(*ss, &s)
return nil
})

You could use the map[string]interface{} to unmarshal the json string.
The code is this
func test() {
jsonStr := `
{
"a": {"key": "a", "value": 1},
"b": {"key": "b", "value": 1}
}`
var mapResult map[string]interface{}
err := json.Unmarshal([]byte(jsonStr), &mapResult)
if err != nil {
fmt.Println("JsonToMapDemo err: ", err)
}
fmt.Println(mapResult)
}
the output is:
map[a:map[key:a value:1] b:map[key:b value:1]]

Related

Add values to array object in json file

I need to add an array of string to a json file.
i.e let's say i have the following json file:
{
"user": {
"role": "admin",
"email": "admin#domain.com"
},
"region": [
{
"location": "EU",
"currency": "EUR",
"countries": [
{
"FR": [
{
"time": "morning",
"cities": []
}
]
}
]
},
{
"location": "NA",
"currency": "USD",
"countries": [
{
"USA": [
{
"time": "evening",
"cities": []
}
]
}
]
}
]
}
Now lets say I have a list of cities that I want to add to region -> countries -> FR
This is my approach, but not sure how to update the cities field.
I have commented the section in the code that need to store to update the list of cities for FA.
func test (jsonFilePath, cityNames string) error {
jsonBytes, err := ioutil.ReadFile(jsonFile)
if err != nil {
return err
}
fmt.Printf("template json file is %s\n", jsonBytes)
var result map[string]interface{}
err = json.Unmarshal(jsonBytes, &result)
if err != nil {
return err
}
if _, exists := result["region"]; exists {
fmt.Printf("Found region\n")
countries := result["region"].([]interface{})
for _, country := range countries {
m := country.(map[string]interface{})
if _, exists := m["FR"]; exists {
/*How to add cityNames to the cities array of the FA?*/
}
}
} else {
errMsg := "region field is missing"
return errors.New(errMsg)
}
jsonBytes, err = json.Marshal(result)
if err != nil {
return err
}
fmt.Printf("the new json file is %s\n", string(jsonBytes))
// Write back to file
err = ioutil.WriteFile("newJson.json", jsonBytes, 0644)
return err
}
Note: The json file has other fields which should not be updated, only Cities filed should be updated; so the preference is to update the Cities field only without defining a struct
Normally it's better (i.e. easier and more efficient) to use structs instead of reflection. There are even websites that can generate a struct from a JSON sample.
But here's how to do it using maps of interfaces:
var result map[string]interface{}
err := json.Unmarshal(jsonBytes, &result)
if err != nil {
return nil, err
}
regions, ok := result["region"].([]interface{})
if !ok {
return nil, errors.New("region not found")
}
for i := range regions {
region, ok := regions[i].(map[string]interface{})
if !ok {
continue
}
countries, ok := region["countries"].([]interface{})
if !ok {
continue
}
for j := range countries {
country, ok := countries[j].(map[string]interface{})
if !ok {
continue
}
fr, ok := country["FR"].([]interface{})
if !ok {
continue
}
for k := range fr {
time, ok := fr[k].(map[string]interface{})
if !ok {
continue
}
cities, ok := time["cities"].([]interface{})
if !ok {
continue
}
time["cities"] = append(cities, cityNames)
}
}
}
Note that in Go maps do not preserve the order, so the order of fields in the output will be different (and also random between runs).

Golang unmarshaling JSON to protobuf generated structs

I would like to reqeive a JSON response withing a client application and unmarshal this response into a struct. To ensure that the struct stays the same accross all client apps using this package, I would like to define the JSON responses as protobuf messages. I am having difficulties unmarshaling the JSON to the protobuf generated structs.
I have the following JSON data:
[
{
"name": "C1",
"type": "docker"
},
{
"name": "C2",
"type": "docker"
}
]
I have modeled my protobuf definitions like this:
syntax = "proto3";
package main;
message Container {
string name = 1;
string type = 2;
}
message Containers {
repeated Container containers = 1;
}
Using this pattern with structs normaly works, but for some reason using these proto definitions causes issues. The below code demonstrates a working and a non-working example. Although one of the versions work, I am unable to use this solution, since []*Container does not satisfy the proto.Message interface.
package main
import (
"encoding/json"
"fmt"
"strings"
"github.com/gogo/protobuf/jsonpb"
)
func working(data string) ([]*Container, error) {
var cs []*Container
return cs, json.Unmarshal([]byte(data), &cs)
}
func notWorking(data string) (*Containers, error) {
c := &Containers{}
jsm := jsonpb.Unmarshaler{}
if err := jsm.Unmarshal(strings.NewReader(data), c); err != nil {
return nil, err
}
return c, nil
}
func main() {
data := `
[
{
"name": "C1",
"type": "docker"
},
{
"name": "C2",
"type": "docker"
}
]`
w, err := working(data)
if err != nil {
panic(err)
}
fmt.Print(w)
nw, err := notWorking(data)
if err != nil {
panic(err)
}
fmt.Print(nw.Containers)
}
Running this gives the following output:
[name:"C1" type:"docker" name:"C2" type:"docker" ]
panic: json: cannot unmarshal array into Go value of type map[string]json.RawMessage
goroutine 1 [running]:
main.main()
/Users/example/go/src/github.com/example/example/main.go:46 +0x1ee
Process finished with exit code 2
Is there a way to unmarshal this JSON to Containers? Or alternatively, make []*Container to satisfy the proto.Message interface?
For the message Containers, i.e.
message Containers {
repeated Container containers = 1;
}
The correct JSON should look like:
{
"containers" : [
{
"name": "C1",
"type": "docker"
},
{
"name": "C2",
"type": "docker"
}
]
}
If you cannot change the JSON then you can utilize the func that you've created
func working(data string) ([]*Container, error) {
var cs []*Container
err := json.Unmarshal([]byte(data), &cs)
// handle the error here
return &Containers{
containers: cs,
}, nil
}
You should use NewDecoder to transfer the data to jsonDecoder and then traverse
the array.The code is this
func main() {
data := `
[
{
"name": "C1",
"type": "docker"
},
{
"name": "C2",
"type": "docker"
}
]`
jsonDecoder := json.NewDecoder(strings.NewReader(data))
_, err := jsonDecoder.Token()
if err != nil {
log.Fatal(err)
}
var protoMessages []*pb.Container
for jsonDecoder.More() {
protoMessage := pb.Container{}
err := jsonpb.UnmarshalNext(jsonDecoder, &protoMessage)
if err != nil {
log.Fatal(err)
}
protoMessages = append(protoMessages, &protoMessage)
}
fmt.Println("%s", protoMessages)
}

Declaring a struct for json objects

I have a JSON response object from a API like that:
[
{
"peoples": {
"1": {"name": "Jhon", "age": 123},
"2": {"name": "Jhon", "age": 123},
"3": {"name": "Jhon", "age": 123},
"4": {"name": "Jhon", "age": 123},
"_object": true,
"_timestamp": "2020-08-05T07:05:55.509Z",
"_writable": false
}
}
]
The parameters: peoples, _object, _timestamp and _writable is fixed. The dynamic values are the 1,2,3,4...n parameters.
The qty of peoples in that struct can be more then 4 or can be 1. Have any elegant solution for create a Struct object or a json.Unmarshal for that?
Borrowing the input example from Sarath Sadasivan Pillai (see comment), here (link to Playground example) is a way to do it with map[string]json.RawMessage and a custom unmarshal function:
package main
import (
"encoding/json"
"fmt"
"strconv"
"time"
)
type Person struct {
Name string `json:"name"`
Age int `json:"age"`
}
type Keywords struct {
Object bool `json:"_object"`
Timestamp time.Time `json:"_timestamp"`
Writable bool `json:"_writable"`
}
type Decoded struct {
People map[string]Person // or perhaps just []Person
Info Keywords
}
var input []byte = []byte(`{
"1": {"name": "Jhon", "age": 123},
"2": {"name": "Jhon", "age": 123},
"3": {"name": "Jhon", "age": 123},
"4": {"name": "Jhon", "age": 123},
"_object": true,
"_timestamp": "2020-08-05T07:05:55.509Z",
"_writable": false
}`)
// Unmarshal json in the input format outlined by
// the example above: a map of numeric strings to Name/Age pair,
// plus some keywords.
func (d *Decoded) UnmarshalJSON(data []byte) error {
if err := json.Unmarshal(data, &d.Info); err != nil {
return err
}
var m map[string]json.RawMessage
if err := json.Unmarshal(data, &m); err != nil {
return err
}
if d.People == nil {
d.People = make(map[string]Person)
}
for k := range m {
// This is the hard part: you must choose how
// to decide whether to decode this as a Person.
// Here, we use strconv.Atoi() as in the example
// by Sarath Sadasivan Pillai, but there are as
// many options as you can think of.
//
// For instance, another method would be to try
// decoding the json.RawMessage as a Person. It's
// also not clear whether the numeric values imply
// some particular ordering, which maps discard.
// (If these come out in order, that's just luck.)
if _, err := strconv.Atoi(k); err != nil {
continue
}
var p Person
if err := json.Unmarshal(m[k], &p); err != nil {
return err
}
d.People[k] = p
}
return nil
}
func main() {
var x Decoded
err := json.Unmarshal(input, &x)
if err != nil {
fmt.Printf("failed: %v\n", err)
} else {
fmt.Printf("succeeded:\n%#v\n", x.Info)
for k := range x.People {
fmt.Printf("person %q = %#v\n", k, x.People[k])
}
}
}

Golang get raw json from postgres

I have stored procedure in my postgres db
SELECT *FROM get_products()
which returns json
[
{
"id": 1,
"name": "one",
"addition": "spec1",
"size": "",
"category_id": 1,
"vendor_id": 1
},
{
"id": 2,
"name": "two",
"addition": "spec2",
"size": "",
"category_id": 1,
"vendor_id": 1
},
/// ...
]
How could I return the result from the procedure without making a struct, cause there can be unknown numbers of fields in json?
I have this code. It works well, when procedure returns table, but not json
func (s ProductController) GetAll(c *gin.Context) {
db, err := sql.Open("postgres", "host=localhost dbname=postgres sslmode=disable user=postgres")
rows, err := db.Query("SELECT *FROM get_products()")
if err != nil {
panic(err)
}
cols, err := rows.Columns()
if err != nil {
panic(err)
}
allgeneric := make([]map[string]interface{}, 0)
colvals := make([]interface{}, len(cols))
for rows.Next() {
colassoc := make(map[string]interface{}, len(cols))
for i, _ := range colvals {
colvals[i] = new(interface{})
}
if err := rows.Scan(colvals...); err != nil {
panic(err)
}
for i, col := range cols {
colassoc[col] = *colvals[i].(*interface{})
}
allgeneric = append(allgeneric, colassoc)
}
err2 := rows.Close()
if err2 !=nil {
panic(err2)
}
fmt.Println(allgeneric)
c.JSON(200, allgeneric)
}
It returns something like this
[
{
"get_products": "W3siaWQiOjEsIm5hbWUiOiJHMTciLCJhZGRpdGlvbiI6IiIsInNpemUiOiJTdGFuZGFyZCIsImNhdGVnb3J5X2lkIjoxLCJ2ZW5kb3JfaWQiOjF9LCB7ImlkIjoyLCJuYW1lIjoiRzE3IiwiYWRkaXRpb24iOiJHZW40Iiwic2l6ZSI6IlN0YW5kYXJkIiwiY2F0ZWdvcnlfaWQiOjEsInZlbmRvcl9pZCI6MX0sIHsiaWQiOjMsIm5hbWUiOiJHMTciLCJhZGRpdGlvbiI6IkdlbjQgIiwic2l6ZSI6IlN0YW5kYXJkIiwiY2F0ZWdvcnlfaWQiOjEsInZlbmRvcl9pZCI6MX0sIHsiaWQiOjQsIm5hbWUiOiJHMTdDIiwiYWRkaXRpb24iOiJHZW40Iiwic2l6ZSI6IlN0YW5kYXJkIiwiY2F0ZWdvcnlfaWQiOjEsInZlbmRvcl9pZCI6MX0sIHsiaWQiOjUsIm5hbWUiOiJHMTciLCJhZGRpdGlvbiI6IkdlbjUiLCJzaXplIjoiU3
But I need to return json, specified above
UPDATE
modified my code to
func (s ProductController) GetAll(c *gin.Context) {
db, err := sql.Open("postgres", "host=localhost dbname=postgres sslmode=disable user=postgres")
rows, err := db.Query("SELECT *FROM get_products()")
if err != nil {
panic(err)
}
var result []interface{}
cols, _ := rows.Columns()
pointers := make([]interface{}, len(cols))
container := make([]json.RawMessage, len(cols))
for i, _ := range pointers {
pointers[i] = &container[i]
}
for rows.Next() {
rows.Scan(pointers...)
result = append(result, container)
}
fmt.Println(container)
c.JSON(200, container)
}
now it returns
[
[
{
"id": 1,
"name": "one",
"addition": "spec1",
"size": "",
"category_id": 1,
"vendor_id": 1
},
{
"id": 2,
"name": "two",
"addition": "spec2",
"size": "",
"category_id": 1,
"vendor_id": 1
},
]
]
Need to remove this inner array
The query returns a single row with a single column containing the JSON. Use this code to read that single value.
row, err := db.QueryRow("SELECT *FROM get_products()")
if err != nil {
// handle error
}
var jsonData []byte
if err := row.Scan(&jsonData); err != nil {
// handle error
}
c.Data(200, "application/json", jsonData)
There's no need to decode and encode the JSON. Use the JSON data returned form the database as is.
I recommend you this solution for this kind of static queries.
First use .Prepare to create a prepared statement.
db := gopg.Connect(&options)
pgStmt, err := pg.Prepare("SELECT *FROM get_products()")
if err != nil {
log.panic(err)
}
Now, you can simply get the result
var jsonData []byte
_, err := pgStmt.QueryOne(gopg.Scan(&jsonData))
if err != nil {
log.panic(err)
}
c.Data(200, "application/json", jsonData)
You can receive json as string also and then convert it into []byte:
var json string
_, err := pgStmt.QueryOne(gopg.Scan(&json))
if err != nil {
log.panic(err)
}
// do something with that json
jsonData := []byte(json)
c.Data(200, "application/json", jsonData)

Custom config loading in Go

I'm using JSON files to store/load my config. Let's say I have the following:
type X interface
// implements interface X
type Y struct {
Value string
}
// implements interface X
type Z struct {
Value string
}
type Config struct {
interfaceInstance X `json:"X"`
}
Config file example:
{
"config1": {
"X": {
"type": "Z",
"Value": "value_1"
}
},
"config2": {
"X": {
"type": "Y",
"Value": "value_2"
}
}
}
I want to be able to define config files something like this example, and be able to dynamically load the JSON as either struct Y or struct Z. Any suggestions on how to accomplish this? I'm using a simple json.Decoder to load the JSON as a struct.
decoder := json.NewDecoder(file)
err = decoder.Decode(&config)
One possible strategy would be to implement json.Unmarshaler for the Config type in such a way that you first unmarshal into a generic object and inspect the "type" attribute then, switching on the type string, unmarshal the same byte array into the known type and assign to the "interfaceInstance" member of the config.
For example (Go Playground):
// Note the slightly different JSON here...
var jsonstr = `{
"config1": {
"type": "Z",
"Value": "value_1"
},
"config2": {
"type": "Y",
"Value": "value_2"
}
}`
func main() {
config := map[string]Config{}
err := json.Unmarshal([]byte(jsonstr), &config)
if err != nil {
panic(err)
}
fmt.Printf("OK: %#v\n", config)
// OK: map[string]main.Config{
// "config1": main.Config{interfaceInstance:main.Z{Value:"value_1"}},
// "config2": main.Config{interfaceInstance:main.Y{Value:"value_2"}},
// }
}
func (c *Config) UnmarshalJSON(bs []byte) error {
// Unmarshal into an object to inspect the type.
var obj map[string]interface{}
err := json.Unmarshal(bs, &obj)
if err != nil {
return err
}
// Unmarshal again into the target type.
configType := obj["type"].(string)
switch configType {
case "Y":
var y Y
if err = json.Unmarshal(bs, &y); err == nil {
c.interfaceInstance = y
}
case "Z":
var z Z
if err = json.Unmarshal(bs, &z); err == nil {
c.interfaceInstance = z
}
default:
return fmt.Errorf("unexpected type %q", configType)
}
return err
}