Add values to array object in json file - json

I need to add an array of string to a json file.
i.e let's say i have the following json file:
{
"user": {
"role": "admin",
"email": "admin#domain.com"
},
"region": [
{
"location": "EU",
"currency": "EUR",
"countries": [
{
"FR": [
{
"time": "morning",
"cities": []
}
]
}
]
},
{
"location": "NA",
"currency": "USD",
"countries": [
{
"USA": [
{
"time": "evening",
"cities": []
}
]
}
]
}
]
}
Now lets say I have a list of cities that I want to add to region -> countries -> FR
This is my approach, but not sure how to update the cities field.
I have commented the section in the code that need to store to update the list of cities for FA.
func test (jsonFilePath, cityNames string) error {
jsonBytes, err := ioutil.ReadFile(jsonFile)
if err != nil {
return err
}
fmt.Printf("template json file is %s\n", jsonBytes)
var result map[string]interface{}
err = json.Unmarshal(jsonBytes, &result)
if err != nil {
return err
}
if _, exists := result["region"]; exists {
fmt.Printf("Found region\n")
countries := result["region"].([]interface{})
for _, country := range countries {
m := country.(map[string]interface{})
if _, exists := m["FR"]; exists {
/*How to add cityNames to the cities array of the FA?*/
}
}
} else {
errMsg := "region field is missing"
return errors.New(errMsg)
}
jsonBytes, err = json.Marshal(result)
if err != nil {
return err
}
fmt.Printf("the new json file is %s\n", string(jsonBytes))
// Write back to file
err = ioutil.WriteFile("newJson.json", jsonBytes, 0644)
return err
}
Note: The json file has other fields which should not be updated, only Cities filed should be updated; so the preference is to update the Cities field only without defining a struct

Normally it's better (i.e. easier and more efficient) to use structs instead of reflection. There are even websites that can generate a struct from a JSON sample.
But here's how to do it using maps of interfaces:
var result map[string]interface{}
err := json.Unmarshal(jsonBytes, &result)
if err != nil {
return nil, err
}
regions, ok := result["region"].([]interface{})
if !ok {
return nil, errors.New("region not found")
}
for i := range regions {
region, ok := regions[i].(map[string]interface{})
if !ok {
continue
}
countries, ok := region["countries"].([]interface{})
if !ok {
continue
}
for j := range countries {
country, ok := countries[j].(map[string]interface{})
if !ok {
continue
}
fr, ok := country["FR"].([]interface{})
if !ok {
continue
}
for k := range fr {
time, ok := fr[k].(map[string]interface{})
if !ok {
continue
}
cities, ok := time["cities"].([]interface{})
if !ok {
continue
}
time["cities"] = append(cities, cityNames)
}
}
}
Note that in Go maps do not preserve the order, so the order of fields in the output will be different (and also random between runs).

Related

`json.Marshal()` converts an array field of a struct to map which it shouldn't

I want to serialize object of types.Project from github.com/compose-spec into JSON
However, a field of ths struct which is supposed to be array, is always serialized into map.
package main
import (
"encoding/json"
types "github.com/compose-spec/compose-go/types"
)
func main() {
out := types.Project{
Name: "foo",
Services: []types.ServiceConfig{ // <- notice this field is an array
{
Name: "bar",
Image: "hello-world",
},
},
}
buf, err := json.Marshal(out)
if err != nil {
panic(err)
}
println(string(buf)) // <- notice the Services field now a map, which is incorrect!
var in types.Project
if err := json.Unmarshal(buf, &in); err != nil {
panic(err)
}
}
The code fails to run:
{"name":"foo","services":{"bar":{"command":null,"entrypoint":null,"image":"hello-world"}}}
panic: json: cannot unmarshal object into Go struct field Project.services of type types.Services
goroutine 1 [running]:
main.main()
/tmp/sandbox1081064727/prog.go:29 +0x168
The out object is serialized as
{
"name": "foo",
"services": {
"bar": {
"command": null,
"entrypoint": null,
"image": "hello-world"
}
}
}
which should really be something like
{
"name": "foo",
"services": [
{
"name": "bar",
"command": null,
"entrypoint": null,
"image": "hello-world"
}
]
}
The behavior you're seeing is correct with respect to the compose file specification, which says:
A Compose file MUST declare a services root element as a map whose keys are string representations of service names, and whose values are service definitions.
The transformation of the list to a map is implemented by the custom marshalers in compose-go/types.go:
// MarshalYAML makes Services implement yaml.Marshaller
func (s Services) MarshalYAML() (interface{}, error) {
services := map[string]ServiceConfig{}
for _, service := range s {
services[service.Name] = service
}
return services, nil
}
// MarshalJSON makes Services implement json.Marshaler
func (s Services) MarshalJSON() ([]byte, error) {
data, err := s.MarshalYAML()
if err != nil {
return nil, err
}
return json.MarshalIndent(data, "", " ")
}

Golang unmarshaling JSON to protobuf generated structs

I would like to reqeive a JSON response withing a client application and unmarshal this response into a struct. To ensure that the struct stays the same accross all client apps using this package, I would like to define the JSON responses as protobuf messages. I am having difficulties unmarshaling the JSON to the protobuf generated structs.
I have the following JSON data:
[
{
"name": "C1",
"type": "docker"
},
{
"name": "C2",
"type": "docker"
}
]
I have modeled my protobuf definitions like this:
syntax = "proto3";
package main;
message Container {
string name = 1;
string type = 2;
}
message Containers {
repeated Container containers = 1;
}
Using this pattern with structs normaly works, but for some reason using these proto definitions causes issues. The below code demonstrates a working and a non-working example. Although one of the versions work, I am unable to use this solution, since []*Container does not satisfy the proto.Message interface.
package main
import (
"encoding/json"
"fmt"
"strings"
"github.com/gogo/protobuf/jsonpb"
)
func working(data string) ([]*Container, error) {
var cs []*Container
return cs, json.Unmarshal([]byte(data), &cs)
}
func notWorking(data string) (*Containers, error) {
c := &Containers{}
jsm := jsonpb.Unmarshaler{}
if err := jsm.Unmarshal(strings.NewReader(data), c); err != nil {
return nil, err
}
return c, nil
}
func main() {
data := `
[
{
"name": "C1",
"type": "docker"
},
{
"name": "C2",
"type": "docker"
}
]`
w, err := working(data)
if err != nil {
panic(err)
}
fmt.Print(w)
nw, err := notWorking(data)
if err != nil {
panic(err)
}
fmt.Print(nw.Containers)
}
Running this gives the following output:
[name:"C1" type:"docker" name:"C2" type:"docker" ]
panic: json: cannot unmarshal array into Go value of type map[string]json.RawMessage
goroutine 1 [running]:
main.main()
/Users/example/go/src/github.com/example/example/main.go:46 +0x1ee
Process finished with exit code 2
Is there a way to unmarshal this JSON to Containers? Or alternatively, make []*Container to satisfy the proto.Message interface?
For the message Containers, i.e.
message Containers {
repeated Container containers = 1;
}
The correct JSON should look like:
{
"containers" : [
{
"name": "C1",
"type": "docker"
},
{
"name": "C2",
"type": "docker"
}
]
}
If you cannot change the JSON then you can utilize the func that you've created
func working(data string) ([]*Container, error) {
var cs []*Container
err := json.Unmarshal([]byte(data), &cs)
// handle the error here
return &Containers{
containers: cs,
}, nil
}
You should use NewDecoder to transfer the data to jsonDecoder and then traverse
the array.The code is this
func main() {
data := `
[
{
"name": "C1",
"type": "docker"
},
{
"name": "C2",
"type": "docker"
}
]`
jsonDecoder := json.NewDecoder(strings.NewReader(data))
_, err := jsonDecoder.Token()
if err != nil {
log.Fatal(err)
}
var protoMessages []*pb.Container
for jsonDecoder.More() {
protoMessage := pb.Container{}
err := jsonpb.UnmarshalNext(jsonDecoder, &protoMessage)
if err != nil {
log.Fatal(err)
}
protoMessages = append(protoMessages, &protoMessage)
}
fmt.Println("%s", protoMessages)
}

How to empty an existing JSON array object without marshalling the entire document

I'm trying to import a large JSON document from a file, empty all arrays matching a specific key or pattern, then output it, without having to marshall the entire document.
It will be run as part of a periodic batch job, so performance/efficiency is not a priority.
Simplicity, and making sure the code is agnostic to the overall JSON structure, is more important.
Is there an easy way to do solve this in Go?
Example input:
{
"panels": [
{
"alert": {
"executionErrorState": "alerting",
"notifications": [
{
"uid": "fRLbH_6Zk"
},
{
"uid": "8gamKl6Waz"
}
]
}
},
{
"alert": {
"executionErrorState": "alerting",
"notifications": [
{
"uid": "DqjrD_6Zk"
}
]
}
}
]
}
Desired output (all entries in 'alert.notifications' in 'panels' removed):
{
"panels": [
{
"alert": {
"executionErrorState": "alerting",
"notifications": []
}
},
{
"alert": {
"executionErrorState": "alerting",
"notifications": []
}
}
]
}
you can use read streams, to read objects one by one. Code will be unmarshall first object, but will have error on the next one. Its like approve of state that code dont read whole file, here example:
package main
import (
"encoding/json"
"fmt"
"log"
"strings"
)
func main() {
const jsonStream = `
[
{"Name": "Ed", "Text": "Knock knock."},
asdasd sadasd,
`
type Message struct {
Name, Text string
}
dec := json.NewDecoder(strings.NewReader(jsonStream))
// read open bracket
t, err := dec.Token()
if err != nil {
log.Fatal(err)
}
fmt.Printf("%T: %v\n", t, t)
// while the array contains values
for dec.More() {
var m Message
// decode an array value (Message)
err := dec.Decode(&m)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%v: %v\n", m.Name, m.Text)
}
// read closing bracket
t, err = dec.Token()
if err != nil {
log.Fatal(err)
}
fmt.Printf("%T: %v\n", t, t)
}

Golang get raw json from postgres

I have stored procedure in my postgres db
SELECT *FROM get_products()
which returns json
[
{
"id": 1,
"name": "one",
"addition": "spec1",
"size": "",
"category_id": 1,
"vendor_id": 1
},
{
"id": 2,
"name": "two",
"addition": "spec2",
"size": "",
"category_id": 1,
"vendor_id": 1
},
/// ...
]
How could I return the result from the procedure without making a struct, cause there can be unknown numbers of fields in json?
I have this code. It works well, when procedure returns table, but not json
func (s ProductController) GetAll(c *gin.Context) {
db, err := sql.Open("postgres", "host=localhost dbname=postgres sslmode=disable user=postgres")
rows, err := db.Query("SELECT *FROM get_products()")
if err != nil {
panic(err)
}
cols, err := rows.Columns()
if err != nil {
panic(err)
}
allgeneric := make([]map[string]interface{}, 0)
colvals := make([]interface{}, len(cols))
for rows.Next() {
colassoc := make(map[string]interface{}, len(cols))
for i, _ := range colvals {
colvals[i] = new(interface{})
}
if err := rows.Scan(colvals...); err != nil {
panic(err)
}
for i, col := range cols {
colassoc[col] = *colvals[i].(*interface{})
}
allgeneric = append(allgeneric, colassoc)
}
err2 := rows.Close()
if err2 !=nil {
panic(err2)
}
fmt.Println(allgeneric)
c.JSON(200, allgeneric)
}
It returns something like this
[
{
"get_products": "W3siaWQiOjEsIm5hbWUiOiJHMTciLCJhZGRpdGlvbiI6IiIsInNpemUiOiJTdGFuZGFyZCIsImNhdGVnb3J5X2lkIjoxLCJ2ZW5kb3JfaWQiOjF9LCB7ImlkIjoyLCJuYW1lIjoiRzE3IiwiYWRkaXRpb24iOiJHZW40Iiwic2l6ZSI6IlN0YW5kYXJkIiwiY2F0ZWdvcnlfaWQiOjEsInZlbmRvcl9pZCI6MX0sIHsiaWQiOjMsIm5hbWUiOiJHMTciLCJhZGRpdGlvbiI6IkdlbjQgIiwic2l6ZSI6IlN0YW5kYXJkIiwiY2F0ZWdvcnlfaWQiOjEsInZlbmRvcl9pZCI6MX0sIHsiaWQiOjQsIm5hbWUiOiJHMTdDIiwiYWRkaXRpb24iOiJHZW40Iiwic2l6ZSI6IlN0YW5kYXJkIiwiY2F0ZWdvcnlfaWQiOjEsInZlbmRvcl9pZCI6MX0sIHsiaWQiOjUsIm5hbWUiOiJHMTciLCJhZGRpdGlvbiI6IkdlbjUiLCJzaXplIjoiU3
But I need to return json, specified above
UPDATE
modified my code to
func (s ProductController) GetAll(c *gin.Context) {
db, err := sql.Open("postgres", "host=localhost dbname=postgres sslmode=disable user=postgres")
rows, err := db.Query("SELECT *FROM get_products()")
if err != nil {
panic(err)
}
var result []interface{}
cols, _ := rows.Columns()
pointers := make([]interface{}, len(cols))
container := make([]json.RawMessage, len(cols))
for i, _ := range pointers {
pointers[i] = &container[i]
}
for rows.Next() {
rows.Scan(pointers...)
result = append(result, container)
}
fmt.Println(container)
c.JSON(200, container)
}
now it returns
[
[
{
"id": 1,
"name": "one",
"addition": "spec1",
"size": "",
"category_id": 1,
"vendor_id": 1
},
{
"id": 2,
"name": "two",
"addition": "spec2",
"size": "",
"category_id": 1,
"vendor_id": 1
},
]
]
Need to remove this inner array
The query returns a single row with a single column containing the JSON. Use this code to read that single value.
row, err := db.QueryRow("SELECT *FROM get_products()")
if err != nil {
// handle error
}
var jsonData []byte
if err := row.Scan(&jsonData); err != nil {
// handle error
}
c.Data(200, "application/json", jsonData)
There's no need to decode and encode the JSON. Use the JSON data returned form the database as is.
I recommend you this solution for this kind of static queries.
First use .Prepare to create a prepared statement.
db := gopg.Connect(&options)
pgStmt, err := pg.Prepare("SELECT *FROM get_products()")
if err != nil {
log.panic(err)
}
Now, you can simply get the result
var jsonData []byte
_, err := pgStmt.QueryOne(gopg.Scan(&jsonData))
if err != nil {
log.panic(err)
}
c.Data(200, "application/json", jsonData)
You can receive json as string also and then convert it into []byte:
var json string
_, err := pgStmt.QueryOne(gopg.Scan(&json))
if err != nil {
log.panic(err)
}
// do something with that json
jsonData := []byte(json)
c.Data(200, "application/json", jsonData)

Making minimal modification to JSON data without a structure in golang

I have a solr response in JSON format which looks like this:
{
"responseHeader": {
"status": 0,
"QTime": 0,
"params": {
"q": "solo",
"wt": "json"
}
},
"response": {
"numFound": 2,
"start": 0,
"docs": [
{
<Large nested JSON element>
},
{
<Large nested JSON element>
}
]
}
}
Now, in my Golang app, I would like to quickly remove the "responseHeader" so that I can return the "response" alone. How can I do this without creating large structures?
Edit 1
The answer by evanmcdonnal was the solution to this problem, but it had some minor typos, this is what I ended up using:
var temp map[string]interface{}
if err := json.Unmarshal(body, &temp); err != nil {
panic(err.Error())
}
result, err := json.Marshal(temp["response"])
Here's a really brief example of how to do this quickly and easily. The steps are; unmarshal into the universal map[string]interface{} then, assuming no errors, marshal only the inner object which you want.
var temp := &map[string]interface{}
if err := json.Unmarshal(input, temp); err != nil {
return err;
}
return json.Marshal(temp["response"])
I wrote a package µjson to do exactly that: performing generic transformations on JSON documents without unmarshalling them.
Run on Go Playground
input := []byte(`
{
"responseHeader": {
"status": 0,
"QTime": 0,
"params": {
"q": "solo",
"wt": "json"
}
},
"response": {
"numFound": 2,
"start": 0,
"docs": [
{ "name": "foo" },
{ "name": "bar" }
]
}
}`)
blacklistFields := [][]byte{
[]byte(`"responseHeader"`), // note the quotes
}
b := make([]byte, 0, 1024)
err := ujson.Walk(input, func(_ int, key, value []byte) bool {
for _, blacklist := range blacklistFields {
if bytes.Equal(key, blacklist) {
// remove the key and value from the output
return false
}
}
// write to output
if len(b) != 0 && ujson.ShouldAddComma(value, b[len(b)-1]) {
b = append(b, ',')
}
if len(key) > 0 {
b = append(b, key...)
b = append(b, ':')
}
b = append(b, value...)
return true
})
if err != nil {
panic(err)
}
fmt.Printf("%s", b)
// Output: {"response":{"numFound":2,"start":0,"docs":[{"name":"foo"},{"name":"bar"}]}}
You can read more about it on the blog post. I put the answer here just in case someone else might need it.