Trying to parse the JSON and create an extracted JSON - json

I am trying to create a JSON object on the fly based on the data received from the API.
Sample Data received: Unmarshalled the data into CiItems struct given below
{
"class_name": "test",
"configuration_items": [
{
"id": "ea09a24f-01ef-42ad-ab19-e0369341d9b3",
"ci_name": "makk",
"comments": null,
"created_by": "mike",
"updated_by": "sam",
"created": "2019-08-02T21:16:35.656Z",
"updated": "2019-08-02T21:21:08.073Z",
"ci_state_id": "randomid",
"super_ci_id": null,
"ci_attributes": [
{
"attribute_id": "c995c693-b97c-4863-a61b-81a5d904c967",
"df_attribute_value": "xsmall",
"attribute_name": "tname",
"data_type": "string"
},
{
"attribute_id": "58845f48-7d2a-4c8c-8591-eaf59a23d84d",
"df_attribute_value": "vmware",
"attribute_name": "provider",
"data_type": "string"
}
]}]}
Below are the structs created:
type Attribute struct {
AttributeID string `json:"attribute_id "`
DfAttributeValue string `json:"df_attribute_value"`
AttName string `json:"attribute_name"`
DataType string `json:"data_type"`
}
// Attributes - array of Attribute
type Attributes []Attribute
// CiItem - confiuraion item of a VM
type CiItem struct {
ID string `json:"ci_id"`
Created string `json:"created"`
Updated string `json:"updated"`
CreatedBY string `json:"created_by"`
UpdatedBY string `json:"updated_by"`
Atts Attributes `json:"ci_attributes"`
}
// CiItems - array of CiItem
type CiItems struct {
ClassName string `json:"class_name"`
Items []CiItem `json:"configuration_items"`
}
Code to unmarshal the data and create a extracted Json:
func (client *Client) GetList() (CiItems, error) {
var out CiItems
err := client.doJsonRequest("GET", &out)
log.Info(out)
if err != nil {
return out, err
}
var output map[string]interface{}
//parseMap(out.Items[0].(map[string]interface{}))
extractBase(out.Items[0], &output)
return out, nil
}
func extractBase(ci interface{}, output interface{}) {
fields := reflect.TypeOf(ci)
values := reflect.ValueOf(ci)
num := fields.NumField()
for i := 0; i < num; i++ {
field := fields.Field(i)
value := values.Field(i)
if string(field.Name) != "Atts" {
name := string(field.Name)
output[name] = string(value)
}
}
}
I am trying to create a JSON with key value of id, ci_name, created_by, updated_by, attribute_name as below
{
"ci_name": "makk",
"created_by": "mike",
"updated_by": "sam",
"created": "2019-08-02T21:16:35.656Z",
"updated": "2019-08-02T21:21:08.073Z",
"tname": "xsmall",
"provider": "vmware"
}
I have tried using reflect and other methods

Create a map using values from CiItem fields and return it from the function.
func extractBase(ci *CiItem) map[string]interface{} {
result := map[string]interface{}{
"ci_name": ci.Name,
"created_by": ci.CreatedBY,
"updated_by": ci.UpdatedBY,
"created": ci.Created,
"updated": ci.Updated,
}
for _, a := range ci.Atts {
result[a.AttName] = a.DfAttributeValue
}
return result
}

Related

Golang: Validate inner Struct field based on the values of one of its enclosing struct's field using required_if tag

golang version: 1.18.3
validator: github.com/go-playground/validator/v10
I want to validate an incoming JSON payload after loaded into nested struct data structure. Here's my incoming JSON payload,
{
"irn": 1,
"firstName": "Testing",
"lastName": "Test",
"cardType": "RECIPROCAL",
"number": "2248974514",
"cardExpiry": {
"month": "01",
"year": "2025"
}
}
here's my medicare.go file
package main
import (
"encoding/json"
"github.com/go-playground/validator/v10"
)
type Medicare struct {
IRN uint8
FirstName string
MiddleInitial string
LastName string
CardType string `validate:"required,eq=RESIDENT|eq=RECIPROCAL|eq=INTERIM"`
Number string
CardExpiry *CardExpiry `validate:"required"`
}
type CardExpiry struct {
Day string
Month string `validate:"required"`
Year string `validate:"required"`
}
Here's my test function
func TestUserPayload(t *testing.T) {
var m Medicare
err := json.Unmarshal([]byte(jsonData), &m)
if err != nil {
panic(err)
}
validate := validator.New()
err = validate.Struct(m)
if err != nil {
t.Errorf("error %v", err)
}
}
I want to do the following validation using validator/v10's required_if tag.
Here's the validation logic,
if (m.CardType == "RECIPROCAL" || m.CardType == "INTERIM") &&
m.CardExpiry.Day == "" {
//validation error
}
required_if can be used based on the field values which are in the same struct (in this case CardExpiry)
Day string `validate:"required_if=Month 01"`
My question is,
can it be done based on the values of one of its enclosing struct's field (in this case Medicare struct)?
for ex:
Day string `validate:"required_if=Medicare.CardType RECIPROCAL"`
if can, how?
Here's the go playground code
you could write custom validation,
playground-solution, documentation for custom validation is available here https://pkg.go.dev/github.com/go-playground/validator#CustomTypeFunc.
Slightly old question, but valix (https://github.com/marrow16/valix) can do such things 'out-of-the-box' using conditions. Example...
package main
import (
"fmt"
"net/http"
"strings"
"github.com/marrow16/valix"
)
type Medicare struct {
IRN uint8 `json:"irn"`
FirstName string `json:"firstName"`
MiddleInitial string `json:"middleInitial"`
LastName string `json:"lastName"`
// set order on this property so that it is evaluated before 'CardExpiry' object is checked...
// (and set a condition based on its value)
CardType string `json:"cardType" v8n:"order:-1,required,&StringValidToken{['RESIDENT','RECIPROCAL','INTERIM']},&SetConditionFrom{Global:true}"`
Number string `json:"number"`
CardExpiry *CardExpiry `json:"cardExpiry" v8n:"required,notNull"`
}
type CardExpiry struct {
// this property is required when a condition of `RECIPROCAL` has been set (and unwanted when that condition has not been set)...
Day string `json:"day" v8n:"required:RECIPROCAL,unwanted:!RECIPROCAL"`
Month string `json:"month" v8n:"required"`
Year string `json:"year" v8n:"required"`
}
var medicareValidator = valix.MustCompileValidatorFor(Medicare{}, nil)
func main() {
jsonData := `{
"irn": 1,
"firstName": "Testing",
"lastName": "Test",
"cardType": "RECIPROCAL",
"number": "2248974514",
"cardExpiry": {
"month": "01",
"year": "2025"
}
}`
medicare := &Medicare{}
ok, violations, _ := medicareValidator.ValidateStringInto(jsonData, medicare)
// should fail with 1 violation...
fmt.Printf("First ok?: %v\n", ok)
for i, v := range violations {
fmt.Printf("Violation[%d]: Message:%s, Property:%s, Path:%s\n", i+1, v.Message, v.Property, v.Path)
}
jsonData = `{
"irn": 1,
"firstName": "Testing",
"lastName": "Test",
"cardType": "RECIPROCAL",
"number": "2248974514",
"cardExpiry": {
"day": "01",
"month": "01",
"year": "2025"
}
}`
ok, _, _ = medicareValidator.ValidateStringInto(jsonData, medicare)
// should be ok...
fmt.Printf("Second ok?: %v\n", ok)
jsonData = `{
"irn": 1,
"firstName": "Testing",
"lastName": "Test",
"cardType": "INTERIM",
"number": "2248974514",
"cardExpiry": {
"month": "01",
"year": "2025"
}
}`
ok, _, _ = medicareValidator.ValidateStringInto(jsonData, medicare)
// should be ok...
fmt.Printf("Third ok?: %v\n", ok)
jsonData = `{
"irn": 1,
"firstName": "Testing",
"lastName": "Test",
"cardType": "INTERIM",
"number": "2248974514",
"cardExpiry": {
"day": "01",
"month": "01",
"year": "2025"
}
}`
ok, violations, _ = medicareValidator.ValidateStringInto(jsonData, medicare)
fmt.Printf("Fourth ok?: %v\n", ok)
for i, v := range violations {
fmt.Printf("Violation[%d]: Message:%s, Property:%s, Path:%s\n", i+1, v.Message, v.Property, v.Path)
}
// or validate directly from a http request...
req, _ := http.NewRequest("POST", "", strings.NewReader(jsonData))
ok, violations, _ = medicareValidator.RequestValidateInto(req, medicare)
fmt.Printf("Fourth (as http.Request) ok?: %v\n", ok)
for i, v := range violations {
fmt.Printf("Violation[%d]: Message:%s, Property:%s, Path:%s\n", i+1, v.Message, v.Property, v.Path)
}
}
On go-playground
Disclosure: I am the author of Valix

Unmarshal nested JSON with different structure but same keys

I'm trying to unmarshal some nested JSON which looks like:
{
"id": "aRandomId",
"type": "aRandomType",
"aRandomField": {
"type": "someType",
"createdAt": "2020-07-07T15:50:02",
"object": "anObject",
"modifiedAt": "2020-07-07T15:50:02"
},
"aParameter": {
"type": "Property",
"createdAt": "2020-07-07T15:50:02",
"value": "myValue",
"modifiedAt": "2020-07-07T15:50:02"
},
"location": {
"type": "GeoProperty",
"value": {
"type": "Point",
"coordinates": [
7.0054,
40.9999
]
}
},
... other things with type, value ...
"createdAt": "2020-07-07T15:50:02",
"modifiedAt": "2020-07-07T15:50:02",
}
I would like to get all keys and values which are: type, createdAt, Value (also if they are nested)
Actually, I have 2 structs:
type Attribute struct {
Type string `json:"type"`
CreatedAt string `json:"createdAt"`
Value string `json:"value"`
ModifiedAt string `json:"modifiedAt"`
}
type Entity struct {
Id string `json:"id"`
Type string `json:"type"`
CreatedAt string `json:"createdAt"`
Attribute Attribute
}
in := []byte(buf.String())
var entity Entity
err := json.Unmarshal(in, &entity)
if err != nil {
panic(err)
}
frame.Fields = append(frame.Fields,
data.NewField("key", nil, []string{"type : ", "createdAt : ", "name : "}),
)
frame.Fields = append(frame.Fields,
data.NewField("value", nil, []string{entity.Type, entity.CreatedAt, entity.Attribute.Value}),
)
The problem is there can be several different Attribute struct andIi can't provide them all.
I would like to display all key (only type, createdAt and Value) in one frame and all their value in another.
Maybe have something like that?
type Entity struct {
attribute List<Attribute>
}
type Attribute struct{
Type string
CreatedAt string
Value string
ModifiedAt string
}
The problem is there can be several different Attribute struct andIi can't provide them all
It looks like your JSON data can have a set of keys with similar values (Attribute) and you can't know how many of them might be in the data.
For this case, you can use a map[string]json.RawMessage as the starting entity to unmarshal to
var e map[string]json.RawMessage
if err := json.Unmarshal([]byte(jsonData), &e); err != nil {
panic(err)
}
You can then range over the values to see if you can unmarshal them into Attribute type
for k, v := range e {
var a Attribute
if err := json.Unmarshal(v, &a); err == nil {
log.Printf("Got attribute %s: %s", k, string(v))
}
}
Run it on playground

Golang json unmarshal according to key value pair

I have json as following
"data": [
{
"id": "recent_search",
"items": [],
"name": ""
},
{
"id": "popular_search",
"items": [],
"name": ""
},
{
"id": "digital",
"items": [],
"name": "DIGITAL"
}
]
and the structs are as follows:
type universeTypeData struct {
Recent universeSearchInfo
Popular universeSearchInfo
Digital universeSearchInfo
}
type universeSearchInfo struct {
ID string `json:"id"`
Name string `json:"name"`
Items []universeSearchItem `json:"items"`
}
I want to unmarshal my json as "id" with value "recent_search" map to Recent, "id" with value "popular_search" map to Popular. Is there any way of doing this in golang?
My approach of doing it is
for _, v := range result.Data {
if v.ID == "in_category" {
finalResult.Universe.InCategory.ID = v.ID
finalResult.Universe.InCategory.Name = v.Name
for _, abc := range v.Items {
finalResult.Universe.InCategory.Items = append(finalResult.Universe.InCategory.Items, abc)
}
}
if v.ID == "recent_search" {
finalResult.Universe.Recent.ID = v.ID
finalResult.Universe.Recent.Name = v.Name
for _, abc := range v.Items {
finalResult.Universe.Recent.Items = append(finalResult.Universe.Recent.Items, abc)
}
}
if v.ID == "popular_search" {
finalResult.Universe.Popular.ID = v.ID
finalResult.Universe.Popular.Name = v.Name
for _, abc := range v.Items {
finalResult.Universe.Popular.Items = append(finalResult.Universe.Popular.Items, abc)
}
}
Is there any better way of doing it?
You want to unmurshal JSON array into Go struct which is not natural mapping. Any way, you most likely should be first unmurshal in slice and then parse this slice. Some workaround is to use json.Decoder
dec := json.NewDecoder(JSONdataReader)
var res universeTypeData
// read open bracket
dec.Token()
// while the array contains values
for dec.More() {
var m universeSearchInfo
// decode an array value
dec.Decode(&m)
switch m.ID {
case "recent_search":
res.Recent = m
case "popular_search":
res.Popular = m
case "digital":
res.Digital = m
}
}
// read closing bracket
dec.Token()
which allow you to decode on the fly, in one pass, without consuming intermediate slice representation. Working example
Implement Unmarshaler interface:
Unmarshaler is the interface implemented by types that can unmarshal a
JSON description of themselves. The input can be assumed to be a valid
encoding of a JSON value. UnmarshalJSON must copy the JSON data if it
wishes to retain the data after returning.
json unmarshaler interface assign the value from json to struct after parsing the result and applying conditions to fetch the value.
package main
import (
"encoding/json"
"fmt"
)
type Details struct {
Data []universeSearchInfo `json:"data"`
}
type universeTypeData struct {
Recent universeSearchInfo
Popular universeSearchInfo
Digital universeSearchInfo
}
type universeSearchInfo struct {
ID string `json:"id"`
Name string `json:"name"`
Items []string `json:"items"`
}
func main() {
var result universeTypeData
jsonBytes := []byte(`{"data": [
{
"id": "recent_search",
"items": [],
"name": ""
},
{
"id": "popular_search",
"items": [],
"name": ""
},
{
"id": "digital",
"items": [],
"name": "DIGITAL"
}
]}`)
if err := json.Unmarshal(jsonBytes, &result); err != nil {
fmt.Println(err)
}
fmt.Println(result)
}
func (universeData *universeTypeData) UnmarshalJSON(data []byte) error {
var result Details
if err := json.Unmarshal(data, &result); err != nil {
return err
}
for _,value := range result.Data{
switch value.ID {
case "recent_search":
universeData.Recent = value
}
}
return nil
}
Working code on Go Playground

Unmarshalling json arrays as json objects

I have to unmarshal a series of Json objects, but one of the objects contain a json array which is not really structured in a good way.
"labels": [
{
"key": "owner",
"value": "harry"
},
{
"key": "group",
"value": "student"
}
]
I am unmarshalling it using this struct -
type StudentDetails struct {
Id string `json:"id"`
Name string `json:"name"`
Labels []Label `json:"labels,omitempty"`
}
type Label struct {
Key string `json:"key"`
Value string `json:"value"`
}
And I have to access it using x.Labels[0].key == "owner" inside a for loop which is very annoying.
I want to be able to do x.Labels.Owner == "harry" instead. How do I go about achieving this? The rest of JSON is unmarshalled fine using the default unmarshal function, so I don't think writing custom function will be good option.
With the constraints you have here, this is about as close as you will get (run in playground):
package main
import (
"encoding/json"
"fmt"
)
func main() {
j := `
{
"id": "42",
"name": "Marvin",
"labels": [
{
"key": "owner",
"value": "harry"
},
{
"key": "group",
"value": "student"
}
]
}`
d := StudentDetails{}
err := json.Unmarshal([]byte(j), &d)
if err != nil {
panic(err)
}
fmt.Println(d.Labels["owner"])
fmt.Println(d.Labels["group"])
}
type StudentDetails struct {
Id string `json:"id"`
Name string `json:"name"`
Labels Labels `json:"labels"`
}
type Labels map[string]string
func (l *Labels) UnmarshalJSON(b []byte) error {
a := []map[string]string{}
err := json.Unmarshal(b, &a)
if err != nil {
return err
}
t := map[string]string{}
for _, m := range a {
t[m["key"]] = m["value"]
}
*l = t
return nil
}
How about to define custom []Label type and add function on it.
For instance
type Labels []Label
func (l Labels) Owner() string {
if len(l) > 1 {
return l[0].Value
}
return ""
}

Go json.Unmarshall() works with single entity but not with slice

I'm using Go for a simple http client. Here's the entity I'm unmarshalling:
type Message struct {
Id int64
Timestamp int64
Text string
Author User
LastEdited int64
}
type User struct {
Id int64
Name string
}
A single entity looks like this in JSON:
{
"text": "hello, can you hear me?",
"timestamp": 1512964818565,
"author": {
"name": "andrea",
"id": 3
},
"lastEdited": null,
"id": 8
}
Go/json has no problem unmarshalling the single entity:
var m Message
err = json.Unmarshal(body, &m)
if err != nil {
printerr(err.Error())
}
println(m.Text)
However, if the return of the endpoint is multiple entities:
[
{
"text": "hello, can you hear me?",
"timestamp": 1512964800981,
"author": {
"name": "eleven",
"id": 4
},
"lastEdited": null,
"id": 7
}
]
And I change my corresponding Unmarshall to work on a slice of structs, Go throws an error:
var m []Message
err = json.Unmarshal(body, &m)
if err != nil {
printerr(err.Error()) // unexpected end of JSON input
}
for i := 0; i < len(m); i++ {
println(m[i].Text)
}
What gives?
Works fine for me (try it on playground), where are you getting the payload data from? sounds like that's truncating it.
package main
import (
"encoding/json"
"fmt"
)
type Message struct {
Id int64
Timestamp int64
Text string
Author User
LastEdited int64
}
type User struct {
Id int64
Name string
}
func main() {
body := []byte(`[
{
"text": "hello, can you hear me?",
"timestamp": 1512964800981,
"author": {
"name": "eleven",
"id": 4
},
"lastEdited": null,
"id": 7
}
]`)
var m []Message
err := json.Unmarshal(body, &m)
if err != nil {
fmt.Printf("error: %v") // unexpected end of JSON input
}
for i := 0; i < len(m); i++ {
fmt.Println(m[i].Text)
}
}
running it gives this output
hello, can you hear me?