How to get all objects with jsonpaths in golang - json

I am using jsonpath in golang but I can't get all the objects of the following json that contain in type iPhone:
{
"firstName": "John",
"lastName": "doe",
"age": 26,
"address": {
"streetAddress": "naist street",
"city": "Nara",
"postalCode": "630-0192"
},
"phoneNumbers": [
{
"type": "iPhone",
"number": "0123-4567-8888"
},
{
"type": "home",
"number": "0123-4567-8910"
},
{
"type": "iPhone",
"number": "0123-4567-8910"
}
]}
I am working with golang and I know that the following jsonpath works:
$.phoneNumbers[?(#.type == "iPhone")]
The problem I have is that it is a service in which I have input a json path like the following:
$.[*].phoneNumbers[*].type
And the value that I have to look for, I am doing it in the following way:
values, err := jsonpath.Get(jsonPath, data)
for _, value := range values {
if err != nil {
continue
}
if value.(string) == "iPhone" {
}
}
At this point I cant get the output like:
[{
"type": "iPhone",
"number": "0123-4567-8888"
},
{
"type": "iPhone",
"number": "0123-4567-8888"
}]
I cant use the [?(#.)] format it is necessary to make with if.
Any idea?
Thanks

I cooked up an example using Peter Ohler's ojg package. Here's what the implementation looks like:
package main
import (
"fmt"
"github.com/ohler55/ojg/jp"
"github.com/ohler55/ojg/oj"
)
var jsonString string = `{
// Your JSON string
}`
func main() {
obj, err := oj.ParseString(jsonString)
if err != nil {
panic(err)
}
x, err := jp.ParseString(`$.phoneNumbers[?(#.type == "iPhone")]`)
if err != nil {
panic(err)
}
ys := x.Get(obj)
for k, v := range ys {
fmt.Println(k, "=>", v)
}
}
// Output:
// 0 => map[number:0123-4567-8888 type:iPhone]
// 1 => map[number:0123-4567-8910 type:iPhone]
Go Playground

Related

How to filter out duplicates while unmarshalling json into struct?

I have this json which I am trying to unmarshall to my struct.
{
"clientMetrics": [
{
"clientId": 951231,
"customerData": {
"Process": [
"ABC"
],
"Mat": [
"KKK"
]
},
"legCustomer": [
8773
]
},
{
"clientId": 1234,
"legCustomer": [
8789
]
},
{
"clientId": 3435,
"otherIds": [
4,
32,
19
],
"legCustomer": [
10005
]
},
{
"clientId": 9981,
"catId": 8,
"legCustomer": [
13769
]
},
{
"clientId": 12124,
"otherIds": [
33,
29
],
"legCustomer": [
12815
]
},
{
"clientId": 8712,
"customerData": {
"Process": [
"College"
]
},
"legCustomer": [
951
]
},
{
"clientId": 23214,
"legCustomer": [
12724,
12727
]
},
{
"clientId": 119812,
"catId": 8,
"legCustomer": [
14519
]
},
{
"clientId": 22315,
"otherIds": [
32
],
"legCustomer": [
12725,
13993
]
},
{
"clientId": 765121,
"catId": 8,
"legCustomer": [
14523
]
}
]
}
I used this tool to generate struct as shown below -
type AutoGenerated struct {
ClientMetrics []ClientMetrics `json:"clientMetrics"`
}
type CustomerData struct {
Process []string `json:"Process"`
Mat []string `json:"Mat"`
}
type ClientMetrics struct {
ClientID int `json:"clientId"`
CustomerData CustomerData `json:"customerData,omitempty"`
LegCustomer []int `json:"legCustomer"`
OtherIds []int `json:"otherIds,omitempty"`
CatID int `json:"catId,omitempty"`
CustomerData CustomerData `json:"customerData,omitempty"`
}
Now my confusion is, I have lot of string or int array so how can I filter out duplicates? I believe there is no set data type in golang so how can I achieve same thing here? Basically when I unmarshall json into my struct I need to make sure there are no duplicates present at all. Is there any way to achieve this? If yes, can someone provide an example how to achieve this for my above json and how should I design my struct for that.
Update
So basically just use like this and change my struct definitions and that's all? Internally it will call UnmarshalJSON and take care of duplicates? I will pass json string and structure to JSONStringToStructure method.
func JSONStringToStructure(jsonString string, structure interface{}) error {
jsonBytes := []byte(jsonString)
return json.Unmarshal(jsonBytes, structure)
}
type UniqueStrings []string
func (u *UniqueStrings) UnmarshalJSON(in []byte) error {
var arr []string
if err := json.Unmarshal(in, arr); err != nil {
return err
}
*u = UniqueStrings(dedupStr(arr))
return nil
}
func dedupStr(in []string) []string {
seen:=make(map[string]struct{})
w:=0
for i:=range in {
if _,s:=seen[in[i]]; !s {
seen[in[i]]=struct{}{}
in[w]=in[i]
w++
}
}
return in[:w]
}
Ideally, you should post-process these arrays to remove duplicates. However, you can achieve this during unmarshaling using a custom type with an unmarshaler:
type UniqueStrings []string
func (u *UniqueStrings) UnmarshalJSON(in []byte) error {
var arr []string
if err:=json.Unmarshal(in,arr); err!=nil {
return err
}
*u=UniqueStrings(dedupStr(arr))
return nil
}
where
func dedupStr(in []string) []string {
seen:=make(map[string]struct{})
w:=0
for i:=range in {
if _,s:=seen[in[i]]; !s {
seen[in[i]]=struct{}{}
in[w]=in[i]
w++
}
}
return in[:w]
}
You may use a similar approach for []ints.
You use the custom types in your structs:
type CustomerData struct {
Process UniqueStrings `json:"Process"`
Mat UniqueStrings `json:"Mat"`
}

Golang iterate over nested struct

i want to iterate over a nested json struct in golang. The issue is, i do not exactly know how nested the structure will be, becuase there are multiple jsons. In this case for example the output should be:
"available": false
"type": "foo"
"name": "foo"
"street": "foo"
....
Is that possible?
{
"informations": {
"available": false,
"provide": {
"informations": {
"customer": {
"type": "foo",
"address": {
"name": "foo",
"street": "foo",
"zipcode": "123",
"city": "foo"
}
}
}
}
}
}
I think you can do this with a map[string]interface{}. Because, as you said that you don't know exactly the structure of the JSON. But, again, you need to know the JSON structure each time while extracting data. Also, you need to do type assertion for inner map[string]interface{}. You can look at the answer from here for type assertion.
Here's the example code that I wrote:
package main
import (
"encoding/json"
"fmt"
)
var j = `{
"informations": {
"available": false,
"provide": {
"informations": {
"customer": {
"type": "foo",
"address": {
"name": "foo",
"street": "foo",
"zipcode": "123",
"city": "foo"
}
}
}
}
}
}`
func main() {
var data map[string]interface{}
err := json.Unmarshal([]byte(j), &data)
if err != nil {
fmt.Println(err.Error())
return
}
fmt.Println(data["informations"])
information := data["informations"]
provide := information.(map[string]interface{})["provide"]
fmt.Println(provide)
}
Output will be:
map[available:false provide:map[informations:map[customer:map[address:map[city:foo name:foo street:foo zipcode:123] type:foo]]]]
map[informations:map[customer:map[address:map[city:foo name:foo street:foo zipcode:123] type:foo]]]
Go Playground

Protobuf custom options not showing in JSON made by protojson library

I'm trying to extract Protobuf custom options from a FileDescriptorSet generated by the protoc compiler. I'm unable to do so using protoreflect. So, I tried to do so using the protojson library.
PS : Importing the Go-generated code is not an option for my use case.
Here's the Protobuf Message I'm testing with :
syntax = "proto3";
option go_package = "./protoze";
import "google/protobuf/descriptor.proto";
extend google.protobuf.FieldOptions {
string Meta = 50000;
}
extend google.protobuf.FileOptions {
string Food = 50001;
}
option (Food) = "cheese";
message X {
int64 num = 1;
}
message P {
string Fname = 1 [json_name = "FNAME"];
string Lname = 2 [json_name = "0123", (Meta) = "Yo"];
string Designation = 3;
repeated string Email = 4;
string UserID = 5;
string EmpID = 6;
repeated X z = 7;
}
// protoc --go_out=. filename.proto
Here's how far I got :
package main
import (
"fmt"
"io/ioutil"
"os/exec"
"google.golang.org/protobuf/encoding/protojson"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/descriptorpb"
)
func main() {
exec.Command("protoc", "-oBinaryFile", "1.proto").Run()
Fset := descriptorpb.FileDescriptorSet{}
byts, _ := ioutil.ReadFile("File")
proto.Unmarshal(byts, &Fset)
byts, _ = protojson.Marshal(Fset.File[0])
fmt.Println(string(byts))
}
And here's the output JSON
{
"name": "1.proto",
"dependency": [
"google/protobuf/descriptor.proto"
],
"messageType": [
{
"name": "X",
"field": [
{
"name": "num",
"number": 1,
"label": "LABEL_OPTIONAL",
"type": "TYPE_INT64",
"jsonName": "num"
}
]
},
{
"name": "P",
"field": [
{
"name": "Fname",
"number": 1,
"label": "LABEL_OPTIONAL",
"type": "TYPE_STRING",
"jsonName": "FNAME"
},
{
"name": "Lname",
"number": 2,
"label": "LABEL_OPTIONAL",
"type": "TYPE_STRING",
"jsonName": "0123",
"options": {}
},
{
"name": "Designation",
"number": 3,
"label": "LABEL_OPTIONAL",
"type": "TYPE_STRING",
"jsonName": "Designation"
},
{
"name": "Email",
"number": 4,
"label": "LABEL_REPEATED",
"type": "TYPE_STRING",
"jsonName": "Email"
},
{
"name": "UserID",
"number": 5,
"label": "LABEL_OPTIONAL",
"type": "TYPE_STRING",
"jsonName": "UserID"
},
{
"name": "EmpID",
"number": 6,
"label": "LABEL_OPTIONAL",
"type": "TYPE_STRING",
"jsonName": "EmpID"
},
{
"name": "z",
"number": 7,
"label": "LABEL_REPEATED",
"type": "TYPE_MESSAGE",
"typeName": ".X",
"jsonName": "z"
}
]
}
],
"extension": [
{
"name": "Meta",
"number": 50000,
"label": "LABEL_OPTIONAL",
"type": "TYPE_STRING",
"extendee": ".google.protobuf.FieldOptions",
"jsonName": "Meta"
},
{
"name": "Food",
"number": 50001,
"label": "LABEL_OPTIONAL",
"type": "TYPE_STRING",
"extendee": ".google.protobuf.FileOptions",
"jsonName": "Food"
}
],
"options": {
"goPackage": "./protoze"
},
"syntax": "proto3"
}
So, data about my custom options showed up in the extensions. But what I really wanted was the value of those Custom Options in the "options" as well. (Which in my case was (Food) = "Cheese" and I want Cheese)
Can someone tell me how I can extract my custom options from the FileDescriptorSet using Protoreflect or by using Protojson.
I tried a lot to try and extract it using Protoreflect but failed !
Although not specifically an answer to how to get the custom options in a generated JSON, I believe I have an answer to what sounds like your underlying question: how to access the custom options without loading the generated Go code. This is thanks to dsnet's answer to my question on the golang issues board. Needless to say all the credit for this tricky solution goes to him. The punchline is to Marshal and then Unmarshal the options using a runtime-populated protoregistry.Types that actually knows about the custom options.
I made a complete demonstration of this approach working in this repo, and the key section (all the guts of which come from dsnet's example) is here:
func main() {
protogen.Options{
}.Run(func(gen *protogen.Plugin) error {
gen.SupportedFeatures = uint64(pluginpb.CodeGeneratorResponse_FEATURE_PROTO3_OPTIONAL)
// The type information for all extensions is in the source files,
// so we need to extract them into a dynamically created protoregistry.Types.
extTypes := new(protoregistry.Types)
for _, file := range gen.Files {
if err := registerAllExtensions(extTypes, file.Desc); err != nil {
panic(err)
}
}
// run through the files again, extracting and printing the Message options
for _, sourceFile := range gen.Files {
if !sourceFile.Generate {
continue
}
// setup output file
outputfile := gen.NewGeneratedFile("./out.txt", sourceFile.GoImportPath)
for _, message := range sourceFile.Messages {
outputfile.P(fmt.Sprintf("\nMessage %s:", message.Desc.Name()))
// The MessageOptions as provided by protoc does not know about
// dynamically created extensions, so they are left as unknown fields.
// We round-trip marshal and unmarshal the options with
// a dynamically created resolver that does know about extensions at runtime.
options := message.Desc.Options().(*descriptorpb.MessageOptions)
b, err := proto.Marshal(options)
if err != nil {
panic(err)
}
options.Reset()
err = proto.UnmarshalOptions{Resolver: extTypes}.Unmarshal(b, options)
if err != nil {
panic(err)
}
// Use protobuf reflection to iterate over all the extension fields,
// looking for the ones that we are interested in.
options.ProtoReflect().Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) bool {
if !fd.IsExtension() {
return true
}
outputfile.P(fmt.Sprintf("Value of option %s is %s",fd.Name(), v.String()))
// Make use of fd and v based on their reflective properties.
return true
})
}
}
return nil
})
}
// Recursively register all extensions into the provided protoregistry.Types,
// starting with the protoreflect.FileDescriptor and recursing into its MessageDescriptors,
// their nested MessageDescriptors, and so on.
//
// This leverages the fact that both protoreflect.FileDescriptor and protoreflect.MessageDescriptor
// have identical Messages() and Extensions() functions in order to recurse through a single function
func registerAllExtensions(extTypes *protoregistry.Types, descs interface {
Messages() protoreflect.MessageDescriptors
Extensions() protoreflect.ExtensionDescriptors
}) error {
mds := descs.Messages()
for i := 0; i < mds.Len(); i++ {
registerAllExtensions(extTypes, mds.Get(i))
}
xds := descs.Extensions()
for i := 0; i < xds.Len(); i++ {
if err := extTypes.RegisterExtension(dynamicpb.NewExtensionType(xds.Get(i))); err != nil {
return err
}
}
return nil
}

Insert arrays into json object in golang

I am processing data from a text file with go and would like to ouput a json record like this:
[{
"ship": "RMS Titanic",
"crew": [{
"name": "Captain Smith"
}, {
"name": "First Officer Murdoch"
}],
"passengers": [{
"name": "Jack Dawson"
}, {
"name": "Rose Dewitt Bukater"
}]
},
{
"ship": "ship2",
"crew": [{
"name": "crew 1"
}, {
"name": "crew 2"
}],
"passengers": [{
"name": "passenger 1"
}, {
"name": "passenger 2"
}]
}
]
Here is a snippet from my code:
package main
import (
"fmt"
"encoding/json"
)
func main() {
var crew []map[string]string
var passengers []map[string]string
s1 := map[string]string{ "name": "RMS Titanic"}
j1, _ := json.Marshal(s1)
fmt.Printf("j1: %s\n", string(j1))
s2 := map[string]string{ "name": "Captain Smith" }
crew = append(crew, s2)
s2 = map[string]string{ "name": "First Officer Murdoch" }
crew = append(crew, s2)
j2, _ := json.Marshal(crew)
fmt.Printf("j2: %s\n", string(j2))
s3 := map[string]string{ "name": "Jack Dawson"}
passengers = append(passengers, s3)
s3 = map[string]string{ "name": "Rose Dewitt Bukater" }
passengers = append(passengers, s3)
j3, _ := json.Marshal(passengers)
fmt.Printf("j3: %s\n", string(j3))
s4 := map[string]string{"crew": string(j2), "passengers": string(j3)}
j4, _ := json.Marshal(s4)
fmt.Printf("j4: %s\n", string(j4))
}
Output:
j1: {"name":"RMS Titanic"}
j2: [{"name":"Captain Smith"},{"name":"First Officer Murdoch"}]
j3: [{"name":"Jack Dawson"},{"name":"Rose Dewitt Bukater"}]
j4: {"crew":"[{\"name\":\"Captain Smith\"},{\"name\":\"First Officer Murdoch\"}]","passengers":"[{\"name\":\"Jack Dawson\"},{\"name\":\"Rose Dewitt Bukater\"}]"}
I am processing the ship data in j1, the crew data in j2 and the passengers data in j3.
I have managed to merge j2 and j3 together into j4, but the quotation mark s are escapaded, how un-escape the quotation marks ?
How to insert j1 data in there so the output match the json output I wish for ?
The solution is not to unescape the string, but to marshal the complete structure you want to serialize to JSON, for example:
ship1 := map[string]interface{}{
"ship": "RMS Titanic",
"crew": crew,
"passengers": passengers,
}
ship1Json, err := json.Marshal(ship1)
if err != nil {
panic(err)
}
fmt.Println("ship1:", string(ship1Json))
Another example with two ships in a slice:
ship2 := map[string]interface{}{
"ship": "ship2",
"crew": crew,
"passengers": passengers,
}
ships := []map[string]interface{}{ship1, ship2}
shipsJson, err := json.Marshal(ships)
if err != nil {
panic(err)
}
fmt.Println("ships:", string(shipsJson))
The result is easier to see if we print the JSON indented:
indented, err := json.MarshalIndent(ships, "", " ")
if err != nil {
panic(err)
}
fmt.Println(string(indented))
Giving:
[
{
"crew": [
{
"name": "Captain Smith"
},
{
"name": "First Officer Murdoch"
}
],
"passengers": [
{
"name": "Jack Dawson"
},
{
"name": "Rose Dewitt Bukater"
}
],
"ship": "RMS Titanic"
},
{
"crew": [
{
"name": "Captain Smith"
},
{
"name": "First Officer Murdoch"
}
],
"passengers": [
{
"name": "Jack Dawson"
},
{
"name": "Rose Dewitt Bukater"
}
],
"ship": "ship2"
}
]
See also on the playground.

How to create custom key in json response using golang

I get a following json response through golang.
[
{
"CreatedOn": "03-22-2015",
"JSONReceived": [
{
"EzpOrderID": "ezp_126",
"FirstName": "Kumar",
"LastName": "S",
"OrderDesc": "Sample"
}
],
"Status": "0",
"id": "80acbdad-8aae-4d6c-ac63-2a02a9db64b4"
},
{
"CreatedOn": "03-22-2015",
"JSONReceived": [
{
"EzpOrderID": "ezp_126",
"FirstName": "Vasanth",
"LastName": "K",
"OrderDesc": "Sample"
}
],
"Status": "0",
"id": "8f7f52a5-793a-45bd-a9b7-ed41495e0ee3"
}
]..
But i need to create with key in response. sample response as follows. How to achieve using golang programming.
{
"returnResponseData": [{
"CreatedOn": "03-22-2015",
"JSONReceived": [{
"EzpOrderID": "ezp_126",
"FirstName": "Kumar",
"LastName": "S",
"OrderDesc": "Sample"
}],
"Status": "0",
"id": "80acbdad-8aae-4d6c-ac63-2a02a9db64b4"
}, {
"CreatedOn": "03-22-2015",
"JSONReceived": [{
"EzpOrderID": "ezp_126",
"FirstName": "Vasanth",
"LastName": "K",
"OrderDesc": "Sample"
}],
"Status": "0",
"id": "8f7f52a5-793a-45bd-a9b7-ed41495e0ee3"
}]
}
Please help me to achieve this task using golang.
Whole Source code as follows:
func orderList(w http.ResponseWriter, request *http.Request) {
rows, err := r.Table("orders").Run(session)
if err != nil {
fmt.Println(err)
return
}
var resultSet []interface{}
err = rows.All(&resultSet)
if err != nil {
fmt.Printf("Error scanning database result: %s", err)
return
}
if origin := request.Header.Get("Origin"); origin != "" {
w.Header().Set("Access-Control-Allow-Origin", origin)
w.Header().Set("Access-Control-Allow-Methods", "GET")
w.Header().Set("Access-Control-Allow-Headers",
"Accept, Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token, Authorization")
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(200)
json.NewEncoder(w).Encode(resultSet)
}
You have to define another type like this:
type Wrapper struct {
ReturnResponseData []interface{} `json:"returnResponseData"`
}
and then encode wrapper, containing your response set:
json.NewEncoder(w).Encode(&Wrapper{ReturnResponseData: resultSet})
Notice, that you have to use property tag to achieve the name "returnResponseData", starting of a small letter (because encoder doesn't encode private properties by default).