Insert arrays into json object in golang - json

I am processing data from a text file with go and would like to ouput a json record like this:
[{
"ship": "RMS Titanic",
"crew": [{
"name": "Captain Smith"
}, {
"name": "First Officer Murdoch"
}],
"passengers": [{
"name": "Jack Dawson"
}, {
"name": "Rose Dewitt Bukater"
}]
},
{
"ship": "ship2",
"crew": [{
"name": "crew 1"
}, {
"name": "crew 2"
}],
"passengers": [{
"name": "passenger 1"
}, {
"name": "passenger 2"
}]
}
]
Here is a snippet from my code:
package main
import (
"fmt"
"encoding/json"
)
func main() {
var crew []map[string]string
var passengers []map[string]string
s1 := map[string]string{ "name": "RMS Titanic"}
j1, _ := json.Marshal(s1)
fmt.Printf("j1: %s\n", string(j1))
s2 := map[string]string{ "name": "Captain Smith" }
crew = append(crew, s2)
s2 = map[string]string{ "name": "First Officer Murdoch" }
crew = append(crew, s2)
j2, _ := json.Marshal(crew)
fmt.Printf("j2: %s\n", string(j2))
s3 := map[string]string{ "name": "Jack Dawson"}
passengers = append(passengers, s3)
s3 = map[string]string{ "name": "Rose Dewitt Bukater" }
passengers = append(passengers, s3)
j3, _ := json.Marshal(passengers)
fmt.Printf("j3: %s\n", string(j3))
s4 := map[string]string{"crew": string(j2), "passengers": string(j3)}
j4, _ := json.Marshal(s4)
fmt.Printf("j4: %s\n", string(j4))
}
Output:
j1: {"name":"RMS Titanic"}
j2: [{"name":"Captain Smith"},{"name":"First Officer Murdoch"}]
j3: [{"name":"Jack Dawson"},{"name":"Rose Dewitt Bukater"}]
j4: {"crew":"[{\"name\":\"Captain Smith\"},{\"name\":\"First Officer Murdoch\"}]","passengers":"[{\"name\":\"Jack Dawson\"},{\"name\":\"Rose Dewitt Bukater\"}]"}
I am processing the ship data in j1, the crew data in j2 and the passengers data in j3.
I have managed to merge j2 and j3 together into j4, but the quotation mark s are escapaded, how un-escape the quotation marks ?
How to insert j1 data in there so the output match the json output I wish for ?

The solution is not to unescape the string, but to marshal the complete structure you want to serialize to JSON, for example:
ship1 := map[string]interface{}{
"ship": "RMS Titanic",
"crew": crew,
"passengers": passengers,
}
ship1Json, err := json.Marshal(ship1)
if err != nil {
panic(err)
}
fmt.Println("ship1:", string(ship1Json))
Another example with two ships in a slice:
ship2 := map[string]interface{}{
"ship": "ship2",
"crew": crew,
"passengers": passengers,
}
ships := []map[string]interface{}{ship1, ship2}
shipsJson, err := json.Marshal(ships)
if err != nil {
panic(err)
}
fmt.Println("ships:", string(shipsJson))
The result is easier to see if we print the JSON indented:
indented, err := json.MarshalIndent(ships, "", " ")
if err != nil {
panic(err)
}
fmt.Println(string(indented))
Giving:
[
{
"crew": [
{
"name": "Captain Smith"
},
{
"name": "First Officer Murdoch"
}
],
"passengers": [
{
"name": "Jack Dawson"
},
{
"name": "Rose Dewitt Bukater"
}
],
"ship": "RMS Titanic"
},
{
"crew": [
{
"name": "Captain Smith"
},
{
"name": "First Officer Murdoch"
}
],
"passengers": [
{
"name": "Jack Dawson"
},
{
"name": "Rose Dewitt Bukater"
}
],
"ship": "ship2"
}
]
See also on the playground.

Related

How to get all objects with jsonpaths in golang

I am using jsonpath in golang but I can't get all the objects of the following json that contain in type iPhone:
{
"firstName": "John",
"lastName": "doe",
"age": 26,
"address": {
"streetAddress": "naist street",
"city": "Nara",
"postalCode": "630-0192"
},
"phoneNumbers": [
{
"type": "iPhone",
"number": "0123-4567-8888"
},
{
"type": "home",
"number": "0123-4567-8910"
},
{
"type": "iPhone",
"number": "0123-4567-8910"
}
]}
I am working with golang and I know that the following jsonpath works:
$.phoneNumbers[?(#.type == "iPhone")]
The problem I have is that it is a service in which I have input a json path like the following:
$.[*].phoneNumbers[*].type
And the value that I have to look for, I am doing it in the following way:
values, err := jsonpath.Get(jsonPath, data)
for _, value := range values {
if err != nil {
continue
}
if value.(string) == "iPhone" {
}
}
At this point I cant get the output like:
[{
"type": "iPhone",
"number": "0123-4567-8888"
},
{
"type": "iPhone",
"number": "0123-4567-8888"
}]
I cant use the [?(#.)] format it is necessary to make with if.
Any idea?
Thanks
I cooked up an example using Peter Ohler's ojg package. Here's what the implementation looks like:
package main
import (
"fmt"
"github.com/ohler55/ojg/jp"
"github.com/ohler55/ojg/oj"
)
var jsonString string = `{
// Your JSON string
}`
func main() {
obj, err := oj.ParseString(jsonString)
if err != nil {
panic(err)
}
x, err := jp.ParseString(`$.phoneNumbers[?(#.type == "iPhone")]`)
if err != nil {
panic(err)
}
ys := x.Get(obj)
for k, v := range ys {
fmt.Println(k, "=>", v)
}
}
// Output:
// 0 => map[number:0123-4567-8888 type:iPhone]
// 1 => map[number:0123-4567-8910 type:iPhone]
Go Playground

Protobuf custom options not showing in JSON made by protojson library

I'm trying to extract Protobuf custom options from a FileDescriptorSet generated by the protoc compiler. I'm unable to do so using protoreflect. So, I tried to do so using the protojson library.
PS : Importing the Go-generated code is not an option for my use case.
Here's the Protobuf Message I'm testing with :
syntax = "proto3";
option go_package = "./protoze";
import "google/protobuf/descriptor.proto";
extend google.protobuf.FieldOptions {
string Meta = 50000;
}
extend google.protobuf.FileOptions {
string Food = 50001;
}
option (Food) = "cheese";
message X {
int64 num = 1;
}
message P {
string Fname = 1 [json_name = "FNAME"];
string Lname = 2 [json_name = "0123", (Meta) = "Yo"];
string Designation = 3;
repeated string Email = 4;
string UserID = 5;
string EmpID = 6;
repeated X z = 7;
}
// protoc --go_out=. filename.proto
Here's how far I got :
package main
import (
"fmt"
"io/ioutil"
"os/exec"
"google.golang.org/protobuf/encoding/protojson"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/descriptorpb"
)
func main() {
exec.Command("protoc", "-oBinaryFile", "1.proto").Run()
Fset := descriptorpb.FileDescriptorSet{}
byts, _ := ioutil.ReadFile("File")
proto.Unmarshal(byts, &Fset)
byts, _ = protojson.Marshal(Fset.File[0])
fmt.Println(string(byts))
}
And here's the output JSON
{
"name": "1.proto",
"dependency": [
"google/protobuf/descriptor.proto"
],
"messageType": [
{
"name": "X",
"field": [
{
"name": "num",
"number": 1,
"label": "LABEL_OPTIONAL",
"type": "TYPE_INT64",
"jsonName": "num"
}
]
},
{
"name": "P",
"field": [
{
"name": "Fname",
"number": 1,
"label": "LABEL_OPTIONAL",
"type": "TYPE_STRING",
"jsonName": "FNAME"
},
{
"name": "Lname",
"number": 2,
"label": "LABEL_OPTIONAL",
"type": "TYPE_STRING",
"jsonName": "0123",
"options": {}
},
{
"name": "Designation",
"number": 3,
"label": "LABEL_OPTIONAL",
"type": "TYPE_STRING",
"jsonName": "Designation"
},
{
"name": "Email",
"number": 4,
"label": "LABEL_REPEATED",
"type": "TYPE_STRING",
"jsonName": "Email"
},
{
"name": "UserID",
"number": 5,
"label": "LABEL_OPTIONAL",
"type": "TYPE_STRING",
"jsonName": "UserID"
},
{
"name": "EmpID",
"number": 6,
"label": "LABEL_OPTIONAL",
"type": "TYPE_STRING",
"jsonName": "EmpID"
},
{
"name": "z",
"number": 7,
"label": "LABEL_REPEATED",
"type": "TYPE_MESSAGE",
"typeName": ".X",
"jsonName": "z"
}
]
}
],
"extension": [
{
"name": "Meta",
"number": 50000,
"label": "LABEL_OPTIONAL",
"type": "TYPE_STRING",
"extendee": ".google.protobuf.FieldOptions",
"jsonName": "Meta"
},
{
"name": "Food",
"number": 50001,
"label": "LABEL_OPTIONAL",
"type": "TYPE_STRING",
"extendee": ".google.protobuf.FileOptions",
"jsonName": "Food"
}
],
"options": {
"goPackage": "./protoze"
},
"syntax": "proto3"
}
So, data about my custom options showed up in the extensions. But what I really wanted was the value of those Custom Options in the "options" as well. (Which in my case was (Food) = "Cheese" and I want Cheese)
Can someone tell me how I can extract my custom options from the FileDescriptorSet using Protoreflect or by using Protojson.
I tried a lot to try and extract it using Protoreflect but failed !
Although not specifically an answer to how to get the custom options in a generated JSON, I believe I have an answer to what sounds like your underlying question: how to access the custom options without loading the generated Go code. This is thanks to dsnet's answer to my question on the golang issues board. Needless to say all the credit for this tricky solution goes to him. The punchline is to Marshal and then Unmarshal the options using a runtime-populated protoregistry.Types that actually knows about the custom options.
I made a complete demonstration of this approach working in this repo, and the key section (all the guts of which come from dsnet's example) is here:
func main() {
protogen.Options{
}.Run(func(gen *protogen.Plugin) error {
gen.SupportedFeatures = uint64(pluginpb.CodeGeneratorResponse_FEATURE_PROTO3_OPTIONAL)
// The type information for all extensions is in the source files,
// so we need to extract them into a dynamically created protoregistry.Types.
extTypes := new(protoregistry.Types)
for _, file := range gen.Files {
if err := registerAllExtensions(extTypes, file.Desc); err != nil {
panic(err)
}
}
// run through the files again, extracting and printing the Message options
for _, sourceFile := range gen.Files {
if !sourceFile.Generate {
continue
}
// setup output file
outputfile := gen.NewGeneratedFile("./out.txt", sourceFile.GoImportPath)
for _, message := range sourceFile.Messages {
outputfile.P(fmt.Sprintf("\nMessage %s:", message.Desc.Name()))
// The MessageOptions as provided by protoc does not know about
// dynamically created extensions, so they are left as unknown fields.
// We round-trip marshal and unmarshal the options with
// a dynamically created resolver that does know about extensions at runtime.
options := message.Desc.Options().(*descriptorpb.MessageOptions)
b, err := proto.Marshal(options)
if err != nil {
panic(err)
}
options.Reset()
err = proto.UnmarshalOptions{Resolver: extTypes}.Unmarshal(b, options)
if err != nil {
panic(err)
}
// Use protobuf reflection to iterate over all the extension fields,
// looking for the ones that we are interested in.
options.ProtoReflect().Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) bool {
if !fd.IsExtension() {
return true
}
outputfile.P(fmt.Sprintf("Value of option %s is %s",fd.Name(), v.String()))
// Make use of fd and v based on their reflective properties.
return true
})
}
}
return nil
})
}
// Recursively register all extensions into the provided protoregistry.Types,
// starting with the protoreflect.FileDescriptor and recursing into its MessageDescriptors,
// their nested MessageDescriptors, and so on.
//
// This leverages the fact that both protoreflect.FileDescriptor and protoreflect.MessageDescriptor
// have identical Messages() and Extensions() functions in order to recurse through a single function
func registerAllExtensions(extTypes *protoregistry.Types, descs interface {
Messages() protoreflect.MessageDescriptors
Extensions() protoreflect.ExtensionDescriptors
}) error {
mds := descs.Messages()
for i := 0; i < mds.Len(); i++ {
registerAllExtensions(extTypes, mds.Get(i))
}
xds := descs.Extensions()
for i := 0; i < xds.Len(); i++ {
if err := extTypes.RegisterExtension(dynamicpb.NewExtensionType(xds.Get(i))); err != nil {
return err
}
}
return nil
}

Enumerate slice entries

If I have this output
Stuff {
"Items": [
{
"title": "test1",
"id": "1",
},
{
"title": "test",
"id": "2",
},
],
"total": 19
},
}
But want this instead:
stuff {
"Items": [
1:{
"title": "test1",
"id": "1",
},
2:{
"title": "test",
"id": "2",
},
],
"total": 19
},
}
Currently, my structs are build like this:
Stuff struct {
Items []Items `json:"items"`
Total int `json:"total"`
} `json:"stuff"`
type Items struct {
Title string `json:"title"`
ID string `json:"id"`
}
I initiate a slice by:
stuff := make([]Items, 10) // Lets say I have 10 entries
And append by:
Stuff.Items = stuff
Stuff.Total = len(Stuff.Items)
Now I am unsure on how to numerate that. So for every item entries, there should be a number, starting from 1 - 10 (In this example)
Given your Stuff and Items type declarations, here's a simple data structure and its JSON dump:
s := Stuff{Items: []Items{Items{"test1", "1"}, Items{"test2", "2"}}, Total: 10}
j, err := json.MarshalIndent(s, "", " ")
if err != nil {
log.Fatal(err)
}
fmt.Println(string(j))
JSON:
{
"items": [
{
"title": "test1",
"id": "1"
},
{
"title": "test2",
"id": "2"
}
],
"total": 10
}
To see what you want instead, there's no magic json package call to do that. You'll have to create a new data structure that reflects the structure of your desired output.
In this case a simple map will do:
m := make(map[string]Items)
for _, item := range s.Items {
m[item.ID] = item
}
And now if you JSON-dump this map, you get:
{
"1": {
"title": "test1",
"id": "1"
},
"2": {
"title": "test2",
"id": "2"
}
}
Note that I'm not wrapping it with Stuff now, because Stuff has different fields. Go is statically typed, and each struct can only contain the fields you declared it to have.

Parse JSON in Go

This is an example of JSON output when calling 'ListObjects' for AWS S3
{
"Contents": [{
"ETag": "9e2bc2894b23742b7bb688c646c6fee9",
"Key": "DSC-0237.jpg",
"LastModified": "2017-09-06 21:53:15 +0000 UTC",
"Owner": {
"DisplayName": "demo-user",
"ID": "a9e2f170a6880f1d61852df8e523e88ca2a2b7abd093476cc93f1239ab5063c6"
},
"Size": 117904,
"StorageClass": "STANDARD"
}, {
"ETag": "\"9e2bc2894b23742b7bb688c646c6fee9\"",
"Key": "DSC-0238.jpg",
"LastModified": "2017-09-06 21:52:24 +0000 UTC",
"Owner": {
"DisplayName": "demo-user",
"ID": "a9e2f170a6880f1d61852df8e523e88ca2a2b7abd093476cc93f1239ab5063c6"
},
"Size": 117904,
"StorageClass": "STANDARD"
}, {
"ETag": "\"9e2bc2894b23742b7bb688c646c6fee9\"",
"Key": "DSC-0239.jpg",
"LastModified": "2017-09-06 21:53:01 +0000 UTC",
"Owner": {
"DisplayName": "demo-user",
"ID": "a9e2f170a6880f1d61852df8e523e88ca2a2b7abd093476cc93f1239ab5063c6"
},
"Size": 117904,
"StorageClass": "STANDARD"
}],
"IsTruncated": false,
"Marker": "",
"MaxKeys": 5,
"Name": "test-bucket-x011pp3",
"Prefix": ""
}
How do I parse this in Go? Chiefly I am interested in collecting:
Bucket Name
Key
Size
Owner's DisplayName
LastModified
I am coming from Python and in Python it would be something really simple like:
json_result = json.loads(json_string)
bucket_name = json_result['Name']
for idx, obj in enumerate(json_result['Contents']):
key = obj['Key']
size = obj['Size']
lastmod = obj['LastModified']
owner= obj['Owner']['DisplayName']
Thank you for the help!
It is something like this
package main
import (
"bytes"
"encoding/json"
"fmt"
"log"
)
var jsonData = []byte(`
{
"Contents": [{
"ETag": "9e2bc2894b23742b7bb688c646c6fee9",
"Key": "DSC-0237.jpg",
"LastModified": "2017-09-06 21:53:15 +0000 UTC",
"Owner": {
"DisplayName": "demo-user",
"ID": "a9e2f170a6880f1d61852df8e523e88ca2a2b7abd093476cc93f1239ab5063c6"
},
"Size": 117904,
"StorageClass": "STANDARD"
}, {
"ETag": "\"9e2bc2894b23742b7bb688c646c6fee9\"",
"Key": "DSC-0238.jpg",
"LastModified": "2017-09-06 21:52:24 +0000 UTC",
"Owner": {
"DisplayName": "demo-user",
"ID": "a9e2f170a6880f1d61852df8e523e88ca2a2b7abd093476cc93f1239ab5063c6"
},
"Size": 117904,
"StorageClass": "STANDARD"
}, {
"ETag": "\"9e2bc2894b23742b7bb688c646c6fee9\"",
"Key": "DSC-0239.jpg",
"LastModified": "2017-09-06 21:53:01 +0000 UTC",
"Owner": {
"DisplayName": "demo-user",
"ID": "a9e2f170a6880f1d61852df8e523e88ca2a2b7abd093476cc93f1239ab5063c6"
},
"Size": 117904,
"StorageClass": "STANDARD"
}],
"IsTruncated": false,
"Marker": "",
"MaxKeys": 5,
"Name": "test-bucket-x011pp3",
"Prefix": ""
}`,
)
type Response struct {
Contents []*Content
IsTruncated bool
Marker string
MaxKeys int
Name string
Prefix string
}
type Content struct {
ETag string
Key string
LastModified string
Owner *Owner
Size int
StroageClass string
}
type Owner struct {
DisplayName string
ID string
}
func main() {
resp := &Response{}
if err := json.NewDecoder(bytes.NewBuffer(jsonData)).Decode(resp); err != nil {
log.Fatal(err)
}
fmt.Printf("%#v", resp)
}
Try it in the Go Playground
And you should read this https://golang.org/pkg/encoding/json/
See https://golang.org/pkg/encoding/json
You annotate a struct with JSON fields that should be unmarshalled from the buffer of JSON, something like this:
type AWSObject struct {
Size int `json:"Size"`
Key string `json:"key"`
Owner AWSObjectOwner `json:"Owner"`
}
type AWSObjectOwner struct {
DisplayName `json:"DisplayName"`
}
var awsObjects []AWSObject
err = json.Unmarshal(jsonBuffer, &awsObjects)
if err != nil {
fmt.Printf("Error unmarshaling objects: " + err.Error() + "\n")
...
}
Using quicktype, I generated your model, marshaling code, and usage instructions:
// To parse this JSON data, add this code to your project and do:
//
// r, err := UnmarshalListObjectsResponse(bytes)
// bytes, err = r.Marshal()
package main
import "encoding/json"
func UnmarshalListObjectsResponse(data []byte) (ListObjectsResponse, error) {
var r ListObjectsResponse
err := json.Unmarshal(data, &r)
return r, err
}
func (r *ListObjectsResponse) Marshal() ([]byte, error) {
return json.Marshal(r)
}
type ListObjectsResponse struct {
Contents []Content `json:"Contents"`
IsTruncated bool `json:"IsTruncated"`
Marker string `json:"Marker"`
MaxKeys int64 `json:"MaxKeys"`
Name string `json:"Name"`
Prefix string `json:"Prefix"`
}
type Content struct {
ETag string `json:"ETag"`
Key string `json:"Key"`
LastModified string `json:"LastModified"`
Owner Owner `json:"Owner"`
Size int64 `json:"Size"`
StorageClass string `json:"StorageClass"`
}
type Owner struct {
DisplayName string `json:"DisplayName"`
ID string `json:"ID"`
}
As others suggested, it's probably best to use the AWS SDK for Go, but this method may be useful in general the next time you need to parse arbitrary JSON.

How to create custom key in json response using golang

I get a following json response through golang.
[
{
"CreatedOn": "03-22-2015",
"JSONReceived": [
{
"EzpOrderID": "ezp_126",
"FirstName": "Kumar",
"LastName": "S",
"OrderDesc": "Sample"
}
],
"Status": "0",
"id": "80acbdad-8aae-4d6c-ac63-2a02a9db64b4"
},
{
"CreatedOn": "03-22-2015",
"JSONReceived": [
{
"EzpOrderID": "ezp_126",
"FirstName": "Vasanth",
"LastName": "K",
"OrderDesc": "Sample"
}
],
"Status": "0",
"id": "8f7f52a5-793a-45bd-a9b7-ed41495e0ee3"
}
]..
But i need to create with key in response. sample response as follows. How to achieve using golang programming.
{
"returnResponseData": [{
"CreatedOn": "03-22-2015",
"JSONReceived": [{
"EzpOrderID": "ezp_126",
"FirstName": "Kumar",
"LastName": "S",
"OrderDesc": "Sample"
}],
"Status": "0",
"id": "80acbdad-8aae-4d6c-ac63-2a02a9db64b4"
}, {
"CreatedOn": "03-22-2015",
"JSONReceived": [{
"EzpOrderID": "ezp_126",
"FirstName": "Vasanth",
"LastName": "K",
"OrderDesc": "Sample"
}],
"Status": "0",
"id": "8f7f52a5-793a-45bd-a9b7-ed41495e0ee3"
}]
}
Please help me to achieve this task using golang.
Whole Source code as follows:
func orderList(w http.ResponseWriter, request *http.Request) {
rows, err := r.Table("orders").Run(session)
if err != nil {
fmt.Println(err)
return
}
var resultSet []interface{}
err = rows.All(&resultSet)
if err != nil {
fmt.Printf("Error scanning database result: %s", err)
return
}
if origin := request.Header.Get("Origin"); origin != "" {
w.Header().Set("Access-Control-Allow-Origin", origin)
w.Header().Set("Access-Control-Allow-Methods", "GET")
w.Header().Set("Access-Control-Allow-Headers",
"Accept, Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token, Authorization")
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(200)
json.NewEncoder(w).Encode(resultSet)
}
You have to define another type like this:
type Wrapper struct {
ReturnResponseData []interface{} `json:"returnResponseData"`
}
and then encode wrapper, containing your response set:
json.NewEncoder(w).Encode(&Wrapper{ReturnResponseData: resultSet})
Notice, that you have to use property tag to achieve the name "returnResponseData", starting of a small letter (because encoder doesn't encode private properties by default).