Parsing JSON in go - json

My question is about parsing JSON files in golang.
Specifically, I am trying to parse output of a command "docker network inspect bridge", which happens to be in JSON format. The command is described here. My goal is to get a list of "IPv4Address" for the listed containers.
I have tried to do this but failing to convert map[string]interface{} to map[string]string. My code is here:- https://play.golang.org/p/eO_j996gGb
$ sudo docker network inspect bridge
[
{
"Name": "bridge",
"Id": "b2b1a2cba717161d984383fd68218cf70bbbd17d328496885f7c921333228b0f",
"Scope": "local",
"Driver": "bridge",
"IPAM": {
"Driver": "default",
"Config": [
{
"Subnet": "172.17.42.1/16",
"Gateway": "172.17.42.1"
}
]
},
"Internal": false,
"Containers": {
"bda12f8922785d1f160be70736f26c1e331ab8aaf8ed8d56728508f2e2fd4727": {
"Name": "container2",
"EndpointID": "0aebb8fcd2b282abe1365979536f21ee4ceaf3ed56177c628eae9f706e00e019",
"MacAddress": "02:42:ac:11:00:02",
"IPv4Address": "172.17.0.2/16",
"IPv6Address": ""
},
"f2870c98fd504370fb86e59f32cd0753b1ac9b69b7d80566ffc7192a82b3ed27": {
"Name": "container1",
"EndpointID": "a00676d9c91a96bbe5bcfb34f705387a33d7cc365bac1a29e4e9728df92d10ad",
"MacAddress": "02:42:ac:11:00:01",
"IPv4Address": "172.17.0.1/16",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.bridge.default_bridge": "true",
"com.docker.network.bridge.enable_icc": "true",
"com.docker.network.bridge.enable_ip_masquerade": "true",
"com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
"com.docker.network.bridge.name": "docker0",
"com.docker.network.driver.mtu": "1500"
}
}
]
What is the correct way to parse such JSON files in golang. I did try using type conversion but doesn't help. I have tried many things but finally could only get as far as the code shown in the snippet. How can I extract the IPv4Address field from the "cont" object?
Link to go playground snippet I have is https://play.golang.org/p/eO_j996gGb. Any help is greatly appreciated (willing to try things out, don't need just the code but ideas :) Thanks.

Your code just needs to be modified to further unpackage the data. You're leaving it with each container object represented as an interface{} which makes it so you can't access the fields within it. Another assert on each to make it a map[string]interface{} will allow you to access the fields within by name (like IPv4Address). Here's a working example; https://play.golang.org/p/4OC5axN4Gd
Here's the important code change;
containers := (foo[0]["Containers"]).(map[string]interface{})
//fmt.Printf("containers %+v", containers)
for _, v := range containers {
unwrapped := v.(map[string]interface{})
fmt.Printf("\ncont %+v\n", unwrapped["IPv4Address"])
}
v is just an interface{} so you have no way of accessing the IPv4Address field without asserting again/converting into a type.
Now that works fine, however, I would recommend getting away from the heavy use of interfaces in your code. Your code is full of unsafe operations that create a need for a lot of error handling (like every time you attempt access into the map you have the potential to throw, as well as anytime you do a type assert). There are plenty of examples of how to do this here on SO but if you leave a comment requesting it I can produce another example that will unmarshal into structs. This is a lot more safe and maintainable imo because most errors will be raised on the call to unmarshal, rather than having potentially a lot of code that works with the results that could all cause a panic.
Here is some play code that has all the types in it and uses unmarshal; https://play.golang.org/p/VfHst9GaiR
Your types can be represented with the following;
type DockerInstance struct {
Name string `json:"Name"`
Id string `json:"Id"`
Scope string `json:"Scope"`
Driver string `json:"Driver"`
EnableIPv6 bool `json:"EnableIPv6"`
IPAM IPAM `json:"IPAM"`
Internal bool `json:"Internal"`
Containers map[string]Container `json:"Containers"`
Options map[string]string `json:"Options"`
Labels interface{} `json:"Labels"`
}
type IPAM struct {
Driver string `json:"Driver"`
Options interface{} `json:"Options"`
Config []Conf `json:"Config"`
}
type Conf struct {
Subnet string `json:"Subnet"`
}
type Container struct {
Name string `json:"Name"`
EndPointID string `json:"EndpointID"`
MacAddress string `json:"MacAddress"`
IPv4Address string `json:"IPv4Address"`
IPv6Address string `json:"IPv6Address"`
}
There's a few stuff I still have set to interface, that's just cause the sample json doesn't include any data for them so I can't know what their type should be in the Go source. Here's some basic stuff I also added in your main to test out the objects/make sure I had things defined correctly;
docks := []DockerInstance{} //note you're using an array of the top level type
err := json.Unmarshal([]byte(teststring), &docks)
if err != nil {
fmt.Println(err)
} else {
fmt.Println()
fmt.Println()
fmt.Println(docks[0].Name)
}
for k, v := range docks[0].Containers {
fmt.Println(k)
fmt.Println(v.IPv4Address)
}
for k, v := range docks[0].Options {
fmt.Println(k)
fmt.Println(v)
}

Related

Parsing complex JSON, again

I have some JSON that get via an API call and I want to now parse this using JSON, I followed an online tutorial in how to parse JSON using structs, but my actual JSON is a lot more complex than the one they used. Here is an example of the JSON I have:
{
"metadata": {},
"items": [
{
"metadata": {
"name": "run7",
"namespace": "default",
"uid": "e218fcc4",
"creationTimestamp": "2022-01-01T00:00:00Z"
},
"spec": {
"arguments": {}
},
"status": {
"phase": "Succeeded",
"startedAt": "2022-01-01T00:00:00Z",
"finishedAt": "2022-01-01T00:00:00Z"
}
}
]
}
and here is the strucs that I created for it:
type wfSpec struct{
Arguments string
}
type wfStatus struct {
Phase string
StartedAt string
FinishedAt string
}
type wfMetadata struct {
Name string
Namespace string
Uid string
CreationTimestamp string
}
type Metadata []struct {
Data string
}
type Items []struct {
wfMetadata
wfStatus
wfSpec
}
type Workflow struct {
Metadata Metadata
Items Items
}
When I first tried to print a value using fmt.Printf(workflows.Items.wfMetadata.Name) I got the error workflows.Items.Metadata undefined (type Items has no field or method Metadata)so then I tried to just print the whole thing using fmt.Printf(workflows) and I got this error cannot use workflows (type Workflow) as type string in argument to fmt.Printf
The only data I need to parse from the JSON is the
"name": "run7",
"namespace": "default",
"uid": "e218fcc4",
"creationTimestamp": "2022-01-01T00:00:00Z"
First off
The problem I expect you're having is not using the tags. To parse a JSON the names of the structs must match the names in the JSON fields. Read here Golang Marshal
Secondly wfMetadata has a lowecase first letter, meaning it will not be imported.
Thirdly, workflow.metadata and workflow.items[i].spec.arguments is set as a {} and not the emptystring "". I assume they're not supposed to be string. This can be avoided using the open interface{} if you don't know or care, or actually implementing them using the official documentations from the API you're connecting to.
As a note, using []struct seems wrong to me. Instead define it in the usage
Note, by using an IDE like GoLand from jetbrains they first off support converting JSON to a struct by simply pasting the JSON into a .go file. They might be daunting at first but do help a lot, and would do much of this for you in seconds.
Now try this instead, and understand why and how this is better and working.
type Status struct {
Phase string `json:"phase"`
StartedAt string `json:"startedAt"`
FinishedAt string `json:"finishedAt"`
}
type ItemMetadata struct {
Name string `json:"name"`
Namespace string `json:"namespace"`
UID string `json:"uid"`
CreationTimestamp string `json:"creationTimestamp"`
}
type Items struct {
Metadata ItemMetadata `json:"metadata"`
Status Status `json:"status"`
Spec interface{} `json:"spec"`
}
type Workflow struct {
Metadata interface{} `json:"metadata"`
Items []Items `json:"items"`
}
Working example in playground https://go.dev/play/p/d9rT4FZJsGv

How do I pass through values from an Elasticsearch document in Go?

I'm experimenting with rewriting parts of our system in Go. They're currently written in Python. Some of the data that I want to serve lives in Elasticsearch.
Across our users, we have a few standard fields, but also allow people to create a number of custom fields specific to their environment. E.g., we have a product object that has some common fields like name and price, but we let someone create a field like discount_price or requires_freight to agree with their use case.
In Python, this is easy to accommodate. The JSON is read in, our chosen JSON parser does some reasonable type inference, and we can then return the data after it's processed.
In Go, any data we want to deal with from the Elasticsearch JSON response has to be mapped to a type. Or at least that's my understanding. For example:
import (
"encoding/json"
)
...
type Product struct {
Name string `json:"name"`
Price string `json:"price"`
}
Here's a simplified example of what the data might look like. I've prefixed the names of the nonstandard fields I'd want to pass through with custom:
{
"id": "ABC123",
"name": "Great Product",
"price": 10.99,
"custom_alternate_names": ["Great Product"],
"custom_sellers": [{
"id": "ABC123",
"name": "Great Product LLC",
"emails": ["great#stuff.com"]
}]
}
It would be fine to special case for places where the route actually needs to process or manipulate a custom field. But in most cases, passing through the JSON data unchanged is fine, so the fact we aren't imposing any type mappings for safety isn't adding anything.
Is there a way to set up the struct with an interface (?) that could act as a passthrough for any unmapped fields? Or a way to take the unparsed JSON data and recombine it with the mapped object before the data are returned?
You can do something like this
package main
import (
"encoding/json"
"log"
)
type Product struct {
//embed product to be able to pull the properties without
//requiring a nested object in the json
KnownFields
OtherStuff json.RawMessage
}
//Custom unmarshaller
func (p *Product) UnmarshalJSON(b []byte) error {
var k KnownFields
//unmarshal the known fields
err := json.Unmarshal(b, &k)
if err != nil {
return err
}
p.KnownFields = k
//You can use json.RawMessage or map[string]interface
p.OtherStuff = json.RawMessage(b)
return nil
}
type KnownFields struct {
Name string `json:"name"`
Price json.Number `json:"price"`
}
const JSON = `
{
"id": "ABC123",
"name": "Great Product",
"price": 10.99,
"custom_alternate_names": ["Great Product"],
"custom_sellers": [{
"id": "ABC123",
"name": "Great Product LLC",
"emails": ["great#stuff.com"]
}]
}`
func main() {
var p Product
err := json.Unmarshal([]byte(JSON), &p)
if err != nil {
log.Panic(err)
}
log.Printf("%v", p)
}
If you are going to mutate and marshall the Product you will have to implement a custom marshaller and also you would need to use map[string]interface{} as the OtherStuff to avoid having duplicate entries for the known fields

Making a struct before the unmarshal in golang

Im new in golang and i have a question.
I have 5 structs where i use Json , but the JSON file can have more structs than the ones i have predetermined BUT... the structures of the JSON satisfies the structures in my programing ( lets say i have 5 structs "struct1" , 6 structs "struct 2" , 1 struct "struct 3", and so on...)
My question is , i want to make a function were i take the JSON FILE , read the structs of it and have as an output the number of structs of the JSON file.
I think i could use the map[string]interface{} but i dont understand it.
I hope i have explain myself
Thank u very much!
Without example JSON or structs, the exact question you are asking is a bit hard to decipher, specifically the "output the number of structs" bit in the question, as it could be interpreted several different ways. I will do my best to answer what I think are the most probable questions you are asking.
Interfaces
First off, some basic go knowledge that might be useful, but outside JSON marshaling itself. The interface{} type appears special, but is not a hardwired keyword as it first might appear. What the interface keyword does is describe the requirements that an object must have to fulfill that interface. Because interface{} has no requirements, and because everything is automatically interfaced in go, everything satisfies the interface{} type.
Because of this implementation of interfaces, map[string]interface{} is really map[string]. This allows for the JSON un/marshal to not care about what is on the value side of the map. This exactly lines up with the format of JSON, where you have a string key on one side, and a value that could be any of the JSON datatypes on the other.
How many different objects are in the base JSON object?
let us take an example json
{
"debug": "on",
"window": {
"title": "Sample Konfabulator Widget",
"name": "main_window",
"width": 500,
"height": 500
},
"image": {
"src": "Images/Sun.png",
"name": "sun1",
"hOffset": 250,
"vOffset": 250,
"alignment": "center"
},
"text": {
"data": "Click Here",
"size": 36,
"style": "bold",
"name": "text1",
"hOffset": 250,
"vOffset": 100,
"alignment": "center",
"onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;"
}
}
The answer to the question in this circumstance would be four. debug, window, image, and text.
The process for determining the number would then be:
Load the json into a byte array.
Marshal into an interface{}
Determine type (array vs object etc.) using type switch. see this A Tour of Go page
If you know the type, you can skip this step
Convert to desired type
Get length, or perform any other operation as desired.
package main
import (
"encoding/json"
"fmt"
)
func main() {
myJSON := `<see above>`
var outStruct *interface{}
json.Unmarshal([]byte(myJSON), &outStruct)
outMap := (*outStruct).(map[string]interface{})
fmt.Printf("Num Structs: %d", len(outMap))
}
Go Playground
How many json objects that I do not have structs for are present?
This answer has a very similar answer to the first one, and is really about manipulation of the output map and the struct
Taking almost the entire code from the first one to the second, let us assume that you have the following structs set up
type Image struct {
Name string
//etc
}
type Text struct {
Name string
//etc
}
type Window struct {
Name string
//etc
}
type Base struct {
Image Image
Window Window
Text Text
}
In this case, in addition to the previous steps, you would have to
5. Marshal the json into a base object
6. Go through the map[string]interface{}, and for each key
7. Determine if the key is one of the objects in your base struct
total := 0
for k, _ := range outMap {
if k != "image" && k != "text" && k != "window" && k != "other" {
total++
}
}
fmt.Printf("Total unknown structs: %d\n", total)
How many of my structs are empty?
This last question is also rather simple, and could be done by checking the map for a value given the input key, but for completion's sake, the example code marshals the JSON into a struct, and uses that.
Marshal JSON into base
For each of Window, Item, Text, Other in base, determine if empty.
total = 0
if (Image{}) == outBase.Image {
total++
}
if (Window{}) == outBase.Window {
total++
}
if (Text{}) == outBase.Text {
total++
}
if (Other{}) == outBase.Other {
total++
}
fmt.Printf("Total empty structs: %d\n", total)
Go Playground
See this go blog post for more information on golang JSON.

Error: interface conversion interface {} is []interface {}, not map[string]interface {}

I am building a project that takes a term from the user and then performs a google search and returns a list of titles in json format.
I am using the serpwow API to perform the google search and am trying to parse the response.
However I am getting the error that states:
panic: interface conversion: interface {} is []interface {}, not map[string]interface {}.
I have looked through various forms and have tried to learn how mapping works but I am not sure why in this case, my mapping is not working. The table for organic results looks like this:
"organic_results": [
{
"position": 1,
"title": "The 10 Best Pizza Places in Dublin - TripAdvisor",
"link": "https://www.tripadvisor.ie/Restaurants-g186605-c31-Dublin_County_Dublin.html",
"domain": "www.tripadvisor.ie",
"displayed_link": "https://www.tripadvisor.ie › ... › County Dublin › Dublin",
"snippet": "Best Pizza in Dublin, County Dublin: Find TripAdvisor traveller reviews of Dublin Pizza places and search by price, location, and more.",
"prerender": false,
"snippet_matched": [
"Pizza",
"Pizza"
],
"cached_page_link": "https://webcache.googleusercontent.com/search?q=cache:OS-Ar9hB_ngJ:https://www.tripadvisor.ie/Restaurants-g186605-c31-Dublin_County_Dublin.html+&cd=4&hl=en&ct=clnk&gl=ie",
"related_page_link": "https://www.google.com/search?q=related:https://www.tripadvisor.ie/Restaurants-g186605-c31-Dublin_County_Dublin.html+pizza&tbo=1&sa=X&ved=2ahUKEwicjYKvvNjmAhVoSBUIHa9MBhcQHzADegQIARAH",
"block_position": 2
},
and here is a snip of my code:
package main
import (
"fmt"
"strings"
serpwow "github.com/serpwow/google-search-results-golang"
)
func main() {
// set API key
apiKey := "Key_Hidden"
//read term to search
fmt.Print("What term would you like to search in google? ")
var term string
fmt.Scanln(&term)
// set up our search parameters
parameters := map[string]interface{}{
"q": term,
}
// retrieve the search results as JSON
response, error := serpwow.GetJSON(parameters, apiKey)
// print the response, or error, if one occurred
if error != nil {
fmt.Println(error)
} else {
//extract title from organic results
//result := fmt.Sprintf("%v", response["organic_results"].(map[string]interface{})["title"])
for _, item := range response["organic_results"].([]interface{}) {
fmt.Sprintf("%v", item.(map[string]interface{})["title"])
}
//s := []string{result, "\n"}
//fmt.Printf(strings.Join(s, " "))
}
}
Can someone please help me figure our where my logic is wrong?
response["organic_results"] corresponds to the JSON array "organic_results", hence it is not a map[string]interface{}, but a []interface. There are multiple results, not one.
for _,item:=range respose["organic_results"].([]interface{}) {
fmt.Printf("%v", item.(map[string]interface{})["title"])
}
As you can see in your JSON, organic_results's value is a JSON array, not an object; each element of the array is an object. So you can't assert organic_results to map[string]interface{} because, as the error states, it's not a map, it's a []interface. You could, for example, do:
result := fmt.Sprintf("%v", response["organic_results"].([]interface{})[0].(map[string]interface{})["title"])
But of course this will only get you the title of the first result, and it will crash if the results are empty. You'll have to think of this not as "get the title from the response" and instead as "handling zero or more results returned" - i.e. you'll probably want to loop over organic_results and do something with each result object.
Sometimes, reverse engineering also works, I had the same issue of not able to UnMarshall a custom made JSON file although all the types which I had defined looked very intuitive.
type ResourceConfig struct {
ResourceType map[string]AlphabetType `json:"resourceType"`
RedisConfig RedisConfigT `json:"redisConfig"`
}
type AlphabetType struct {
XXX []string `json:"xxx"`
YYY []string `json:"yyy"`
}
type RedisConfigT struct {
Broker string `json:"broker"`
DB string `json:"db"`
}
For which the json looked like this:
{
"resourceType": [
{"/abcdefg": [{"xxx": ["/vrf", "/lsp", "/vpn"], "yyy": ["/fpc"]}]},
{"/vrf": [{"xxx": [""], "yyy": ["/abcdefg"]}]},
{"/lsp": [{"xxx": [""], "yyy": ["/abcdefg"]}]},
{"/vpn": [{"xxx": [""], "yyy": ["/abcdefg"]}]},
{"/fpc": [{"xxx": ["/abcdefg"], "yyy": [""]}]}
],
"redisConfig": {"broker": "localhost: 6379", "db": 0}
}
When doing UnMarshall it was throwing error of not able to parse.
So I decided to form the required map programmatically first Marshal it into a json, and print it.
{"resourceType":{"/fpc":{"XXX":["/abcdefg"],"YYY":[]},
"/abcdefg":{"XXX":["/vrf","/lsp","/vpn"],"YYY":["/fpc"]},
"/abc":{"XXX":[],"YYY":["/abcdefg"]},
"/vpn":{"XXX":[],"YYY":["/abcdefg"]},
"/vrf":{"XXX":[],"YYY":["/abcdefg"]}},
"redisConfig":{"broker":"localhost:6349","db":"0"}}
Now use the same format in. the json file and UnMarshal it. It will nicely
fit into the types you have defined based on which the Json was generated originally.

Can one run a PutItem in DynamoDB in Golang without using a struct?

All of AWS' examples involve using a struct to organize and optimize your data prior to doing anything with DynamoDB. See: https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/dynamo-example-create-table-item.html
For my project I am getting a JSON string back from a third party API. I actually want to store the entire JSON object in DynamoDB but I can't determine if it's possible without a struct. Because I'm working with a third party API, I can't guarantee what fields I will get, or if the fields will change in the future. Can this be done? In NodeJS it's quite easy, but I understand that Go is statically typed so it may just not be possible.
You will always need a struct for your item static key and/or range. The JSON can go in another field of type map, which can contain items or nested maps. Then let attributevalue.MarshalMap transform the whole struct into a DynamoDB map.
type Record struct {
ID string
JSON map[string]interface{}
}
r := Record{
ID: "ABC123",
JSON: map[string]interface{}{
"Parent": "aa",
"Children": map[string]interface{}{
"Child1": "foo",
"Child2": "bar",
},
},
}
av, err := attributevalue.MarshalMap(r)
if err != nil {
return fmt.Errorf("failed to marshal Record, %w", err)
}
_, err = client.PutItem(context.TODO(), &dynamodb.PutItemInput{
TableName: aws.String(myTableName),
Item: av,
})
if err != nil {
return fmt.Errorf("failed to put Record, %w", err)
}
example partly taken from https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue