Parsing complex JSON, again - json

I have some JSON that get via an API call and I want to now parse this using JSON, I followed an online tutorial in how to parse JSON using structs, but my actual JSON is a lot more complex than the one they used. Here is an example of the JSON I have:
{
"metadata": {},
"items": [
{
"metadata": {
"name": "run7",
"namespace": "default",
"uid": "e218fcc4",
"creationTimestamp": "2022-01-01T00:00:00Z"
},
"spec": {
"arguments": {}
},
"status": {
"phase": "Succeeded",
"startedAt": "2022-01-01T00:00:00Z",
"finishedAt": "2022-01-01T00:00:00Z"
}
}
]
}
and here is the strucs that I created for it:
type wfSpec struct{
Arguments string
}
type wfStatus struct {
Phase string
StartedAt string
FinishedAt string
}
type wfMetadata struct {
Name string
Namespace string
Uid string
CreationTimestamp string
}
type Metadata []struct {
Data string
}
type Items []struct {
wfMetadata
wfStatus
wfSpec
}
type Workflow struct {
Metadata Metadata
Items Items
}
When I first tried to print a value using fmt.Printf(workflows.Items.wfMetadata.Name) I got the error workflows.Items.Metadata undefined (type Items has no field or method Metadata)so then I tried to just print the whole thing using fmt.Printf(workflows) and I got this error cannot use workflows (type Workflow) as type string in argument to fmt.Printf
The only data I need to parse from the JSON is the
"name": "run7",
"namespace": "default",
"uid": "e218fcc4",
"creationTimestamp": "2022-01-01T00:00:00Z"

First off
The problem I expect you're having is not using the tags. To parse a JSON the names of the structs must match the names in the JSON fields. Read here Golang Marshal
Secondly wfMetadata has a lowecase first letter, meaning it will not be imported.
Thirdly, workflow.metadata and workflow.items[i].spec.arguments is set as a {} and not the emptystring "". I assume they're not supposed to be string. This can be avoided using the open interface{} if you don't know or care, or actually implementing them using the official documentations from the API you're connecting to.
As a note, using []struct seems wrong to me. Instead define it in the usage
Note, by using an IDE like GoLand from jetbrains they first off support converting JSON to a struct by simply pasting the JSON into a .go file. They might be daunting at first but do help a lot, and would do much of this for you in seconds.
Now try this instead, and understand why and how this is better and working.
type Status struct {
Phase string `json:"phase"`
StartedAt string `json:"startedAt"`
FinishedAt string `json:"finishedAt"`
}
type ItemMetadata struct {
Name string `json:"name"`
Namespace string `json:"namespace"`
UID string `json:"uid"`
CreationTimestamp string `json:"creationTimestamp"`
}
type Items struct {
Metadata ItemMetadata `json:"metadata"`
Status Status `json:"status"`
Spec interface{} `json:"spec"`
}
type Workflow struct {
Metadata interface{} `json:"metadata"`
Items []Items `json:"items"`
}
Working example in playground https://go.dev/play/p/d9rT4FZJsGv

Related

Golang nested structs not getting omitted

I want to omit certain structs nested in a JSON request. I've created a rest API on golang which reads a message body from an http request, decodes it into the struct defined in the code and inserts it into Mongo DB
My structs are as follows. Note that for the nested structure C, I use a pointer in order to be able to omit it.
type A struct {
Title string `json:"title"`
Text string `json:"text"`
Data B `json:"data"`
}
type B struct {
Product *C `json:"product,omitempty"`
ExternalLink string `json:"external_link,omitempty"`
}
type C struct {
Name string `json:"name"`
Id int `json:"id"`
}
Here is how I decode it (Didn't go for Json.Unmarshall as I read that for http bodies, decoding should be used over unmarshall)
func NewMessage(req *http.Request) *A {
var newMessage *A
json.NewDecoder(req.Body).Decode(&newMessage)
messageInData := newMessage
return newMessage
}
The "newMessage" upon return is inserted into Mongo directly. However, Even if the request payload contains no such object as the struct C for instance like below
{
"title": "First message from GoLang",
"text": "Hello Dastgyr",
"data": {
"external_link": "some link here"
//no product object (C struct) here
}
}
The object inserted into Mongo still contains the struct C as having a null value as shown below
{
"title": "First message from GoLang",
"text": "Hello Dastgyr",
"data": {
"product": null,
"external_link": "some link here"
}
}
I've also tried using B as a pointer in Struct A but to no avail
type A struct {
Title string `json:"title"`
Text string `json:"text"`
Data *B `json:"data,omitempty"`
}
I want to be able to omit certain nested structs. Despite using pointers, the struct I want is still not omitting. What mistake am I making in defining structs?
Still new to golang so a nudge in the right direction would help
You are using json tags for json un-marshaling, it seems to be un-marshaling correctly (concluding this as you didn't mention any errors, and moved on to MongoDB)
How you add data to MongoDB is a totally different matter, and has nothing to do with your JSON tags. It uses bson tags, and you will need to add them, if you wish to use this same struct as a mongo DB model representation. Something like this:
type A struct {
Title string `json:"title" bson:"title"`
Text string `json:"text" bson:"text"`
Data *B `json:"data,omitempty" bson:"data,omitempty"`
}
Remember that tags in golang, are just some metadata being added with a structure, which some code actually reads and acts on. json library identifies & processes json:"" tag, while official go mongodb library that you might be using, will process the bson:"" tags.

GoLang Gin - Binding JSON data from POST request rewrites all Identifier Fields to 0

I am taking my first steps with GoLang, and currently setting up an API Server, which is able to read JSON file from POST Request and save that to Memory.
I have a JSON File as Following:
[
{
"id": 0,
"name": "kubernetes",
"uri": "https://github.com/kubernetes/kubernetes"
},
{
"id": 1,
"name": "jenkins",
"uri": "https://github.com/jenkinsci/jenkins"
}
]
Which I am POST:ing to the API Server running on local port.
Here is my setupRoutes() - function:
func setupRoutes() {
// Initialize Router
router := gin.Default()
// Initialize Routes
router.GET("/api/projects", getProjects)
router.GET("/api/projects/:id", getProjectByIdentifier)
router.POST("/api/projects", uploadProjects)
// Start the Router
router.Run("localhost:8080")
}
and here is my uploadProjects() - function:
// Reads file from POST request, and saves that to Memory.
func uploadProjects(c *gin.Context) {
// Initialize Object
var obj []Project
// Bind JSON Data to Object
c.BindJSON(&obj)
fmt.Println(obj) // For Testing: What is binded.
// Save Data to Memory
proj = obj
}
and here is the Project Struct:
type Project struct {
Identifier int64 `json: id`
Name string `json: name`
Uri string `json: uri`
}
After executing this - I can print that data out right away, what is being binded or I can fetch that with my GET /api/projects - call, and the result is always:
[{0 kubernetes https://github.com/kubernetes/kubernetes} {0 jenkins https://github.com/jenkinsci/jenkins}]
What I've tried:
I've tried to swap between string, int and int64 types of Identifier Field in my struct.
Googled a Bunch
This is probably something very simple, but I don't really know where to look at this point, so any help is appreciated.
The id field does not match the field name Identifier. Fix by using properly formatted JSON field tags. The field tags used in the question are not recognized by the JSON codec.
type Project struct {
Identifier int64 `json:"id"`
Name string `json:"name"`
Uri string `json:"uri"`
}

how to create user defined struct for array of different user defined object in golang

I am trying to parse json object in golang.
Below is the json structure :
{
"properties": {
"alertType": "VM_EICAR",
"resourceIdentifiers": [{
"azureResourceId": "/Microsoft.Compute/virtualMachines/vm1",
"type": "AzureResource"
},
{
"workspaceId": "f419f624-acad-4d89-b86d-f62fa387f019",
"workspaceSubscriptionId": "20ff7fc3-e762-44dd-bd96-b71116dcdc23",
"workspaceResourceGroup": "myRg1",
"agentId": "75724a01-f021-4aa8-9ec2-329792373e6e",
"type": "LogAnalytics"
}
],
"vendorName": "Microsoft",
"status": "New"
}
}
I have below user defined types.
type AzureResourceIdentifier struct {
AzureResourceId string `json:"azureResourceId"`
Type string `json:"type"`
}
type LogAnalyticsIdentifier struct{
AgentId string `json:"agentId"`
Type string `json:"type"`
WorkspaceId string `json:"workspaceId"`
WorkspaceResourceGroup string `json:"workspaceResourceGroup"`
WorkspaceSubscriptionId string `json:"workspaceSubscriptionId"`
}
I have a top level user defined type as properties as below.
it has resourceIdentifiers as array of two other user defined types(defined above),
AzureResourceIdentifier
LogAnalyticsIdentifier
how can I define type of resourceIdentifiers in properties ?
type Properties struct{
alertType string
resourceIdentifiers ??? `
vendorName string `
status string
}
Note: I have a existing connector and we are provisioned to input the struct of the parsing json, we cannot override or add any function.
There are a couple of options here.
[]map[string]interface{}
Using a map will allow any JSON object in the array to be parsed, but burdens you with having to safely extract information after the fact.
[]interface{}
Each is going to be a map under the hood, unless non-JSON object elements can also be in the JSON array. No reason to use it over map[string]interface{} in your case.
A slice of a new struct type which covers all possible fields: []ResourceIdentifier
type ResourceIdentifier struct {
AzureResourceId string `json:"azureResourceId"`
AgentId string `json:"agentId"`
WorkspaceId string `json:"workspaceId"`
WorkspaceResourceGroup string `json:"workspaceResourceGroup"`
WorkspaceSubscriptionId string `json:"workspaceSubscriptionId"`
Type string `json:"type"`
}
This is probably the laziest solution. It allows you to get where you want to quickly, at the cost of wasting some memory and creating a non self-explanatory data design which might cause confusion if later trying to serialize with it again.
A new struct type + custom unmarshalling implementation.
type ResourceIdentifiers struct {
AzureResourceIdentifiers []AzureResourceIdentifier
LogAnalyticsIdentifiers []LogAnalyticsIdentifier
}
Implement json.Unmarshaler to decide which type of struct to construct and in which slice to put it.

Using struct with API call in Go

I'm new to Go and working hard to follow its style and I'm not sure how to proceed.
I want to push a JSON object to a Geckoboard leaderboard, which I believe requires the following format based on the API doc and the one for leaderboards specifically:
{
"api_key": "222f66ab58130a8ece8ccd7be57f12e2",
"data": {
"item": [
{ "label": "Bob", "value": 4, "previous_value": 6 },
{ "label": "Alice", "value": 3, "previous_value": 4 }
]
}
}
My instinct is to build a struct for the API call itself and another called Contestants, which will be nested under item. In order to use json.Marshall(Contestant1), the naming convention of my variables would not meet fmt's expectations:
// Contestant structure to nest into the API call
type Contestant struct {
label string
value int8
previous_rank int8
}
This feels incorrect. How should I configure my Contestant objects and be able to marshall them into JSON without breaking convention?
To output a proper JSON object from a structure, you have to export the fields of this structure. To do it, just capitalize the first letter of the field.
Then you can add some kind of annotations, to tell your program how to name your JSON fields :
type Contestant struct {
Label string `json:"label"`
Value int8 `json:"value"`
PreviousRank int8 `json:"previous_rank"`
}

Marshal dynamic JSON field tags in Go

I'm trying to generate JSON for a Terraform file. Because I (think I) want to use marshalling instead of rolling my own JSON, I'm using Terraforms JSON format instead of the 'native' TF format.
{
"resource": [
{
"aws_instance": {
"web1": {
"some": "data"
}
}]
}
resource and aws_instance are static identifiers while web1 in this case is the random name. Also it wouldn't be unthinkable to also have web2 and web3.
type Resource struct {
AwsResource AwsResource `json:"aws_instance,omitempty"`
}
type AwsResource struct {
AwsWebInstance AwsWebInstance `json:"web1,omitempty"`
}
The problem however; how do do I generate random/variable JSON keys with Go's field tags?
I have a feeling the answer is "You don't". What other alternatives do I have then?
In most cases where there are names not known at compile time, a map can be used:
type Resource struct {
AWSInstance map[string]AWSInstance `json:"aws_instance"`
}
type AWSInstance struct {
AMI string `json:"ami"`
Count int `json:"count"`
SourceDestCheck bool `json:"source_dest_check"`
// ... and so on
}
Here's an example showing how to construct the value for marshalling:
r := Resource{
AWSInstance: map[string]AWSInstance{
"web1": AWSInstance{
AMI: "qdx",
Count: 2,
},
},
}
playground example