Fill struct dynamically from parameters - json

I've the following struct which works as expected Im getting
data and I was able
type Service struct {
Resources []struct {
Model struct {
Name string `json:"name"`
Credentials struct {
path string `json:"path"`
Vts struct {
user string `json:"user"`
id string `json:"id"`
address string `json:"address"`
} `json:"vts"`
} `json:"credentials"`
} `json:"model"`
} `json:"resources"`
}
service:= Service{}
err := json.Unmarshal([]byte(data), &service
The data is like following,
service1
{
"resources": [
"model": {
"name": "cred",
"credentials": {
"path": "path in fs",
"vts": {
"user": "stephane",
"id": "123",
"address": "some address"
}
},
}
},
Now some services providing additional data under vts for example now we have 3 fields (user/id/address) but some services (service1) can provides additional data like email, secondName ,etc . but the big problem here is that
I need to get it from parameters since (service 2) education, salary etc
Service2
{
"resources": [
"model": {
"name": "cred",
"credentials": {
"path": "path in fs",
"vts": {
"user": "stephane",
"id": "123",
"address": "some address",
"email" : "some email",
"secondName" : "secondName"
}
},
}
},
service N
{
"resources": [
"model": {
"name": "cred",
"credentials": {
"path": "path in fs",
"vts": {
"user": "stephane",
"id": "123",
"address": "some address",
"salary" : "1000000"
}
},
}
},
Of course If I know in advance the fields I can put them all in the struct and use omitempty but I dont know, I just get it as parameter to the function (the new properties names) , some service can provide 10 more fields in this struct (which I should get the properties name of them as args[]to the functions) but I don't know them in advance, this should be dynamic somehow ....is there a nice way to handle it in Golang ?

If you don't know the fields in advance, then don't use a struct but something that is also "dynamic": a map.
type Service struct {
Resources []struct {
Model struct {
Name string `json:"name"`
Credentials struct {
Path string `json:"path"`
Vts map[string]interface{} `json:"vts"`
} `json:"credentials"`
} `json:"model"`
} `json:"resources"`
}
map[sting]interface{} can hold values of any type. If you know all fields will hold a string value, you may also use a map[string]string so it will be easier to work with it.
Example with input JSON:
{
"resources": [
{
"model": {
"name": "cred",
"credentials": {
"path": "path in fs",
"vts": {
"user": "stephane",
"id": "123",
"address": "some address",
"dyn1": "d1value",
"dyn2": "d2value"
}
}
}
}
]
}
Testing it:
service := Service{}
err := json.Unmarshal([]byte(data), &service)
fmt.Printf("%q %v", service, err)
Output (try it on the Go Playground):
{[{{"cred" {"path in fs" map["dyn2":"d2value" "user":"stephane" "id":"123"
"address":"some address" "dyn1":"d1value"]}}}]} <nil>
Now if you want to collect values from the Vts map for a set of keys, this is how you can do it:
args := []string{"dyn1", "dyn2"}
values := make([]interface{}, len(args))
for i, arg := range args {
values[i] = service.Resources[0].Model.Credentials.Vts[arg]
}
fmt.Println(values)
Output of the above will be (try it on the Go Playground):
[d1value d2value]

Related

Go unmarshal JSON with unknown field name but known struct

I retrieve a acme.json from traefik tls where traefik stores ssl/tls certificate information.
Now I want to unmarshal with golang the acme.json into my go struct "Traefik". But I don't know how to handle dynamic/unknown json field names because certificateresolver1 and certificateresolver2 are names I don't know at compile time. These names should be dynamic configured in go.
I know the structure of the json (it is always the same) but not know the field name of the certificateresolver.
Does anyone know the best way to do this?
Traefik acme.json
{
"certificateresolver1": {
"Account": {
"Email": "email#example.com",
"Registration": {
"body": {
"status": "valid",
"contact": [
"mailto:email#example.com"
]
},
"uri": "https://acme-v02.api.letsencrypt.org/acme/acct/124448363"
},
"PrivateKey": "PRIVATEKEY",
"KeyType": "4096"
},
"Certificates": [
{
"domain": {
"main": "example.com",
"sans": [
"test.example.com"
]
},
"certificate": "CERTIFICATE",
"key": "KEY",
"Store": "default"
},
{
"domain": {
"main": "example.org"
},
"certificate": "CERTIFICATE",
"key": "KEY",
"Store": "default"
}
]
},
"certificateresolver2": {
"Account": {
"Email": "email#example.com",
"Registration": {
"body": {
"status": "valid",
"contact": [
"mailto:email#example.com"
]
},
"uri": "https://acme-v02.api.letsencrypt.org/acme/acct/126945414"
},
"PrivateKey": "PRIVATEKEY",
"KeyType": "4096"
},
"Certificates": [
{
"domain": {
"main": "example.net"
},
"certificate": "CERTIFICATE",
"key": "KEY",
"Store": "default"
}
]
}
}
Go struct for acme.json
type Traefik struct {
Provider []struct {
Account struct {
Email string `json:"Email"`
Registration struct {
Body struct {
Status string `json:"status"`
Contact []string `json:"contact"`
} `json:"body"`
URI string `json:"uri"`
} `json:"Registration"`
PrivateKey string `json:"PrivateKey"`
KeyType string `json:"KeyType"`
} `json:"Account"`
Certificates []struct {
Domain struct {
Main string `json:"main"`
Sans []string `json:"sans"`
} `json:"domain"`
Certificate string `json:"certificate"`
Key string `json:"key"`
Store string `json:"Store"`
} `json:"Certificates"`
} `json:"certificateresolver"` <-- What to write there? It should fit for certificateresolver1 and certificateresolver2
}
I think something like this will help you:
type ProviderMdl map[string]Provider
type Provider struct {
Account struct {
Email string `json:"Email"`
Registration struct {
Body struct {
Status string `json:"status"`
Contact []string `json:"contact"`
} `json:"body"`
URI string `json:"uri"`
} `json:"Registration"`
PrivateKey string `json:"PrivateKey"`
KeyType string `json:"KeyType"`
} `json:"Account"`
Certificates []struct {
Domain struct {
Main string `json:"main"`
Sans []string `json:"sans"`
} `json:"domain"`
Certificate string `json:"certificate"`
Key string `json:"key"`
Store string `json:"Store"`
} `json:"Certificates"`
}
So you could work with this data in this way:
bres := new(ProviderMdl)
if err := json.Unmarshal(data, bres); err != nil {
panic(err)
}
// fmt.Printf("%+v - \n", bres)
for key, value := range *bres {
fmt.Printf("%v - %v\n", key, value)
}
Full example here

Unwinding nested data responses into structs when a slice is part of the response

I'd like to get some input into how you would all go about unwinding nested response data into custom structs. Below is an example of the data i'm being returned. I'm trying to get to the user data.
{
"_links": {
"self": {
"href": "/api/v2/user-search/default/test?after=1585612800000&limit=20&offset=0&q=johnsmith%40test.com",
"type": "application/json"
}
},
"totalCount": 1,
"items": [
{
"lastPing": "2020-04-30T02:56:10.430867577Z",
"environmentId": "xxxx",
"ownerId": "xxxx",
"user": {
"key": "johnsmith#test.com",
"email": "johnsmith#test.com",
"firstName": "john",
"lastName": "smith"
},
"_links": {
"parent": {
"href": "/api/v2/users/default/test",
"type": "application/json"
},
"self": {
"href": "/api/v2/users/default/test/johnsmith#test.com",
"type": "application/json"
},
"settings": {
"href": "/api/v2/users/default/test/johnsmith#test.com/flags",
"type": "text/html"
},
"site": {
"href": "/default/test/users/johnsmith#test.com",
"type": "text/html"
}
}
}
]
}
Currently I'm doing the below
respData := map[string][]map[string]map[string]interface{}{}
json.Unmarshal(respBody, &respData)
userData := respData["items"][0]["user"]
I'd love to be able to unmarshal it into a custom struct but I can't seem to get it to work. The nested slice that the user object sits within is what keeps throwing me off.
type User struct {
Key string `json:"key"`
Email string `json:"email"`
FirstName string `json:"firstName"`
LastName string `json:"lastName"`
}
type LinkInfo struct {
Href string `json:"href"`
Type string `json:"type"`
}
type Item struct {
LastPing time.Time `json:"lastPing"`
EnvironmentID string `json:"environmentId"`
OwnerID string `json:"ownerId"`
User User `json:"user"`
Links LinkInfo `json:"_links"`
Self LinkInfo `json:"self"`
Settings LinkInfo `json:"settings"`
Parent LinkInfo `json:"parent"`
Site LinkInfo `json:"site"`
}
type ItemDetails struct {
Links LinkInfo `json:"_links"`
TotalCount int `json:"total_count"`
Items []Item
}
Can you try this?
https://play.golang.org/p/S_CUN0XEh-d
From what you mentioned it sounds like you were on the right track. Your JSON is pretty large, so let me give you a smaller example similar to the part you mentioned you're having trouble with (the user object inside the items list).
type response struct {
TotalCount int `json:"totalCount"`
Items []*itemStruct `json:"items"`
}
type itemStruct struct {
LastPing string `json:"lastPing"`
User *userStruct `json:"user"`
}
type userStruct struct {
Key string `json:"key"`
}
Basically to map to a JSON list of objects, just put a field like this in your struct: Objects []*structWhichMapsToMyObject
Edit: Here's the code running in Go Playground: https://play.golang.org/p/EvSvv-2s8y8
If you want this:
"user": {
"key": "johnsmith#test.com",
"email": "johnsmith#test.com",
"firstName": "john",
"lastName": "smith"
}
Declare a matching Go struct:
type User struct {
Key string `json:"key"`
Email string `json:"email"`
FirstName string `json:"firstName"`
LastName string `json:"lastName"`
}
Then, since the user's parent object looks like this:
"items": [
{
"lastPing": "2020-04-30T02:56:10.430867577Z",
"environmentId": "xxxx",
"ownerId": "xxxx",
"user": { ... },
"_links": { ... }
}
]
you also need to declare a matching Go struct for that (you can omit fields you don't need):
type Item struct {
User User `json:"user"`
}
and then the parent of the parent:
{
"_links": {
"self": {
"href": "/api/v2/user-search/default/test?after=1585612800000&limit=20&offset=0&q=johnsmith%40test.com",
"type": "application/json"
}
},
"totalCount": 1,
"items": [ ... ]
}
and the matching Go struct for the grandparent, again, include only the fields you need:
type ResponseData struct {
Items []Item `json:"items"`
}
Once you have this you can decode the json into an instance of ResponseData:
var rd ResponseData
if err := json.Unmarshal(data, &rd); err != nil {
panic(err)
}
for _, item := range rd.Items {
fmt.Println(item.User)
}
https://play.golang.com/p/7yavVSBcHQP

How to struct array in nested json object with the same keys

I am creating host objects in Zabbix via microservices in Golang. I have to serve the following json to
the Zabbix api to create host which is part of multiple groups
{
"jsonrpc": "2.0",
"method": "host.create",
"params": {
"host": "TEST-HOST",
"interfaces": [
{
"type": 2,
"main": 1,
"useip": 1,
"ip": "0.0.0.0",
"dns": "",
"port": "10050"
}
],
"groups": [
{
"groupid": "33"
},
{
"groupid": "27"
}
],
"templates": [
{
"templateid": "12156"
}
],
"inventory_mode": 0
},
"auth": "example_token",
"id": 1
}
This code returns the json object where the groups array is empty.
package main
import (
"bytes"
"encoding/json"
"fmt"
)
type Zabbix struct {
Jsonrpc string `json:"jsonrpc"`
Method string `json:"method"`
Params Params `json:"params"`
Auth string `json:"auth"`
ID int `json:"id"`
}
type Groups struct {
Groupid string `json:"groupid"`
Groupid1 string `json:"groupid"`
}
type Templates struct {
Templateid string `json:"templateid"`
}
type Inventory struct {
Type string `json:"type"`
Tag string `json:"tag"`
TypeFull string `json:"type_full"`
MacaddressA string `json:"macaddress_a"`
MacaddressB string `json:"macaddress_b"`
SerialA string `json:"serialno_a"`
SerialB string `json:"serialno_b"`
}
type Params struct {
Host string `json:"host"`
Interfaces []Interfaces `json:"interfaces"`
Groups []Groups `json:"groups"`
Templates []Templates `json:"templates"`
InventoryMode int `json:"inventory_mode"`
Inventory Inventory `json:"inventory"`
}
type Interfaces struct {
Type int `json:"type"`
Main int `json:"main"`
Useip int `json:"useip"`
IP string `json:"ip"`
DNS string `json:"dns"`
Port string `json:"port"`
}
func main() {
jsonobj := &Zabbix{
Jsonrpc: "2.0",
Method: "host.create",
Params: Params{
Host: "OBU_TEST_123",
Interfaces: []Interfaces{
{
Type: 2,
Main: 1,
Useip: 1,
IP: "10.10.10.10",
DNS: "",
Port: "10050",
},
},
Groups: []Groups{
{
Groupid: "27",
Groupid1: "33",
},
},
Templates: []Templates{
{
Templateid: "12156",
},
},
Inventory: Inventory{
Type: "On Board Unit",
Tag: "test",
TypeFull: "test",
MacaddressA: "test",
MacaddressB: "test",
SerialA: "test",
SerialB: "test",
},
InventoryMode: 0,
},
Auth: "test-token",
ID: 1,
}
byteArray, err := json.Marshal(jsonobj)
if err != nil {
panic(err)
}
fmt.Print(bytes.NewBuffer(byteArray).String())
}
The output:
{
"jsonrpc": "2.0",
"method": "host.create",
"params": {
"host": "OBU_TEST_123",
"interfaces": [
{
"type": 2,
"main": 1,
"useip": 1,
"ip": "10.10.10.10",
"dns": "",
"port": "10050"
}
],
"groups": [
{}
],
"templates": [
{
"templateid": "12156"
}
],
"inventory_mode": 0,
"inventory": {
"type": "On Board Unit",
"tag": "test",
"type_full": "test",
"macaddress_a": "test",
"macaddress_b": "test",
"serialno_a": "test",
"serialno_b": "test"
}
},
"auth": "test-token",
"id": 1
}
What i am missing? Is there any more elegant way to create such a big json object instead of using structs?
type Groups struct {
Groupid string json:"groupid"
Groupid1 string json:"groupid"
}
groupid - Cannot be the same value for both. Change it as follows and it should work.
type Groups struct {
Groupid string json:"groupid"
Groupid1 string json:"groupid1"
}
type Groups struct {
Groupid string `json:"groupid"`
Groupid1 string `json:"groupid"`
}
groupid - Cannot be the same value for both. Change it as follows and it should work.
type Groups struct {
Groupid string `json:"groupid"`
Groupid1 string `json:"groupid1"`
}

MGO return bson field instead of json field

The bson name is used when performing pipe in mgo.
Struct :
type Training struct {
Id bson.ObjectId `json:"id" bson:"_id"`
Name string `json:"name" bson:"name"`
Description string `json:"description"`
Level *TrainingLevel `json:"level"`
Preworks []bson.ObjectId `json:"preworks"`
PrePostTests []bson.ObjectId `json:"preposttests" bson:"preposttests"`
TrainingEvaluations []bson.ObjectId `json:"training_evaluations" bson:"training_evaluations"`
TrainerEvaluations []bson.ObjectId `json:"trainer_evaluations" bson:"trainer_evaluations"`
AppCompanyId bson.ObjectId `json:"app_company_id" bson:"app_company_id"`
Company *Company `json:"company"`
}
Function :
func (this *TrainingClass) GetAllTraining() (interface{}, error) {
if !this.tokenInfo.IsAllowed(this.c) {
return nil, tlib.NewTError(common.Error_NoAccess, "You don't have the right!")
}
sess, db := GetDB()
defer sess.Close()
pipeline := []bson.M{
{"$match": bson.M{
"app_company_id": this.tokenInfo.AppCompanyId}},
{"$lookup": bson.M{
"from": "trainingbatch",
"localField": "_id",
"foreignField": "training._id",
"as": "trainingbatches"}},
}
resp := []bson.M{}
db.C(common.C_TRAINING).Pipe(pipeline).All(&resp)
return bson.M{"data": resp}, nil
}
Json result :
{
"data": [
{
"_id": "5995a749dbcfbe4e8cc31378",
"app_company_id": "58b24756e65bd121f6b1a923",
"description": "Description First Training",
"name": "First Training",
"trainingbatches": [
{
"_id": "5995a74adbcfbe4e8cc31379",
"app_company_id": "58b24756e65bd121f6b1a923",
"company": {
"_id": "58b24756e65bd121f6b1a923",
"address": "",
"app_company_id": "58b24756e65bd121f6b1a923",
"fullname": "",
"name": "Tandem",
"phone": ""
},
}
]
}
]
}
As you can see field _id is generated instead of id. That's not happen if I use find or findId. Is there any way to keep using json field no matter what's the query?
The way you're reading the result, it has no idea what the JSON field names are. In order for it to use those tags, it must actually deserialize into the struct where the tags have been specified. When you do:
resp := []bson.M{}
db.C(common.C_TRAINING).Pipe(pipeline).All(&resp)
You're explicitly telling mgo to return BSON results. The object you pass in (a slice of bson.M) has no json tags on it. In order to control the serialization to JSON, you must pass a struct with the JSON tags specified to All:
resp := []Training
db.C(common.C_TRAINING).Pipe(pipeline).All(&resp)

map[string] struct inside struct

I have a JSON file that looks like this:
{
"jailbreaks": [
{
"jailbroken": false,
"name": "",
"version": "",
"url": "",
"anleitung": [],
"ios": {
"start": "10.2.1"
},
"caveats": "",
"platforms": []
},
{
"jailbroken": true,
"name": "Yalu102",
"version": "beta 6",
"url": "https://domain-dl.tld",
"anleitung": [
{ "blog": "title", "link": "http://domain.tld/" },
{ "blog": "Test", "link": "http://google.at" }
],
"ios": {
"start": "10.2"
},
"caveats": "some text here",
"platforms": [
"Windows",
"OS X",
"Linux"
]
},
And I create the object to work with like this:
type Jailbreak struct {
Jailbroken bool `json:"jailbroken"`
Name string `json:"name"`
Version string `json:"version"`
URL string `json:"url"`
Anleitung map[string]struct {
Name string `json:"blog"`
Link string `json:"link"`
} `json:"anleitung"`
Firmwares struct {
Start string `json:"start"`
End string `json:"end"`
} `json:"ios"`
Platforms []string `json:"platforms"`
Caveats string `json:"caveats"`
}
When I want to build my go program I get an error, that the JSON file cannot be read. But as soon as I delete the map[string]struct I can compile and run the program without any error and everything works fine.
Am I messing around with something or is there an error in my JSON file?
The json provided is not valid (as the array does not have a closing ] and the top level json object lacks another closing }) so let's assume it's like:
{
"jailbreaks": [
{
"jailbroken": false,
"name": "",
"version": "",
"url": "",
"anleitung": [],
"ios": {
"start": "10.2.1",
"end": ""
},
"platforms": [],
"caveats": ""
},
{
"jailbroken": true,
"name": "Yalu102",
"version": "beta 6",
"url": "https://domain-dl.tld",
"anleitung": [
{
"blog": "title",
"link": "http://domain.tld/"
},
{
"blog": "Test",
"link": "http://google.at"
}
],
"ios": {
"start": "10.2",
"end": ""
},
"platforms": [
"Windows",
"OS X",
"Linux"
],
"caveats": "some text here"
}
]
}
The data structure Jailbreaks (first one), marshals-to/unmarshals-from this json properly:
type Jailbreaks struct {
List []Jailbreak `json:"jailbreaks"`
}
type Jailbreak struct {
Jailbroken bool `json:"jailbroken"`
Name string `json:"name"`
Version string `json:"version"`
URL string `json:"url"`
Anleitung []struct {
Name string `json:"blog"`
Link string `json:"link"`
} `json:"anleitung"`
Firmwares struct {
Start string `json:"start"`
End string `json:"end"`
} `json:"ios"`
Platforms []string `json:"platforms"`
Caveats string `json:"caveats"`
}
As you see Anleitung is declared as a slice (not a map).
Use omitempty flag for when your "anleitung" is empty in JSON to be consumed. Beware though, when that is the case, your Jailbreak struct won't have an "anleitung" field.
Change your map's json flag to to;
Anleitung map[string]struct {
Name string `json:"blog"`
Link string `json:"link"`
} `json:"anleitung,omitempty"`
Option 2;
I guess you could also use Anleitung map[string]interface{} but that is better for "holding a map of strings to arbitrary data types". In your case the data is not arbitrary but rather, empty I guess. And looks like that is just temporary.
I'd go for option 1, then I'd check if my struct contains any Anleitung data or not before accessing it.