Error: interface conversion interface {} is []interface {}, not map[string]interface {} - json

I am building a project that takes a term from the user and then performs a google search and returns a list of titles in json format.
I am using the serpwow API to perform the google search and am trying to parse the response.
However I am getting the error that states:
panic: interface conversion: interface {} is []interface {}, not map[string]interface {}.
I have looked through various forms and have tried to learn how mapping works but I am not sure why in this case, my mapping is not working. The table for organic results looks like this:
"organic_results": [
{
"position": 1,
"title": "The 10 Best Pizza Places in Dublin - TripAdvisor",
"link": "https://www.tripadvisor.ie/Restaurants-g186605-c31-Dublin_County_Dublin.html",
"domain": "www.tripadvisor.ie",
"displayed_link": "https://www.tripadvisor.ie › ... › County Dublin › Dublin",
"snippet": "Best Pizza in Dublin, County Dublin: Find TripAdvisor traveller reviews of Dublin Pizza places and search by price, location, and more.",
"prerender": false,
"snippet_matched": [
"Pizza",
"Pizza"
],
"cached_page_link": "https://webcache.googleusercontent.com/search?q=cache:OS-Ar9hB_ngJ:https://www.tripadvisor.ie/Restaurants-g186605-c31-Dublin_County_Dublin.html+&cd=4&hl=en&ct=clnk&gl=ie",
"related_page_link": "https://www.google.com/search?q=related:https://www.tripadvisor.ie/Restaurants-g186605-c31-Dublin_County_Dublin.html+pizza&tbo=1&sa=X&ved=2ahUKEwicjYKvvNjmAhVoSBUIHa9MBhcQHzADegQIARAH",
"block_position": 2
},
and here is a snip of my code:
package main
import (
"fmt"
"strings"
serpwow "github.com/serpwow/google-search-results-golang"
)
func main() {
// set API key
apiKey := "Key_Hidden"
//read term to search
fmt.Print("What term would you like to search in google? ")
var term string
fmt.Scanln(&term)
// set up our search parameters
parameters := map[string]interface{}{
"q": term,
}
// retrieve the search results as JSON
response, error := serpwow.GetJSON(parameters, apiKey)
// print the response, or error, if one occurred
if error != nil {
fmt.Println(error)
} else {
//extract title from organic results
//result := fmt.Sprintf("%v", response["organic_results"].(map[string]interface{})["title"])
for _, item := range response["organic_results"].([]interface{}) {
fmt.Sprintf("%v", item.(map[string]interface{})["title"])
}
//s := []string{result, "\n"}
//fmt.Printf(strings.Join(s, " "))
}
}
Can someone please help me figure our where my logic is wrong?

response["organic_results"] corresponds to the JSON array "organic_results", hence it is not a map[string]interface{}, but a []interface. There are multiple results, not one.
for _,item:=range respose["organic_results"].([]interface{}) {
fmt.Printf("%v", item.(map[string]interface{})["title"])
}

As you can see in your JSON, organic_results's value is a JSON array, not an object; each element of the array is an object. So you can't assert organic_results to map[string]interface{} because, as the error states, it's not a map, it's a []interface. You could, for example, do:
result := fmt.Sprintf("%v", response["organic_results"].([]interface{})[0].(map[string]interface{})["title"])
But of course this will only get you the title of the first result, and it will crash if the results are empty. You'll have to think of this not as "get the title from the response" and instead as "handling zero or more results returned" - i.e. you'll probably want to loop over organic_results and do something with each result object.

Sometimes, reverse engineering also works, I had the same issue of not able to UnMarshall a custom made JSON file although all the types which I had defined looked very intuitive.
type ResourceConfig struct {
ResourceType map[string]AlphabetType `json:"resourceType"`
RedisConfig RedisConfigT `json:"redisConfig"`
}
type AlphabetType struct {
XXX []string `json:"xxx"`
YYY []string `json:"yyy"`
}
type RedisConfigT struct {
Broker string `json:"broker"`
DB string `json:"db"`
}
For which the json looked like this:
{
"resourceType": [
{"/abcdefg": [{"xxx": ["/vrf", "/lsp", "/vpn"], "yyy": ["/fpc"]}]},
{"/vrf": [{"xxx": [""], "yyy": ["/abcdefg"]}]},
{"/lsp": [{"xxx": [""], "yyy": ["/abcdefg"]}]},
{"/vpn": [{"xxx": [""], "yyy": ["/abcdefg"]}]},
{"/fpc": [{"xxx": ["/abcdefg"], "yyy": [""]}]}
],
"redisConfig": {"broker": "localhost: 6379", "db": 0}
}
When doing UnMarshall it was throwing error of not able to parse.
So I decided to form the required map programmatically first Marshal it into a json, and print it.
{"resourceType":{"/fpc":{"XXX":["/abcdefg"],"YYY":[]},
"/abcdefg":{"XXX":["/vrf","/lsp","/vpn"],"YYY":["/fpc"]},
"/abc":{"XXX":[],"YYY":["/abcdefg"]},
"/vpn":{"XXX":[],"YYY":["/abcdefg"]},
"/vrf":{"XXX":[],"YYY":["/abcdefg"]}},
"redisConfig":{"broker":"localhost:6349","db":"0"}}
Now use the same format in. the json file and UnMarshal it. It will nicely
fit into the types you have defined based on which the Json was generated originally.

Related

How do I pass through values from an Elasticsearch document in Go?

I'm experimenting with rewriting parts of our system in Go. They're currently written in Python. Some of the data that I want to serve lives in Elasticsearch.
Across our users, we have a few standard fields, but also allow people to create a number of custom fields specific to their environment. E.g., we have a product object that has some common fields like name and price, but we let someone create a field like discount_price or requires_freight to agree with their use case.
In Python, this is easy to accommodate. The JSON is read in, our chosen JSON parser does some reasonable type inference, and we can then return the data after it's processed.
In Go, any data we want to deal with from the Elasticsearch JSON response has to be mapped to a type. Or at least that's my understanding. For example:
import (
"encoding/json"
)
...
type Product struct {
Name string `json:"name"`
Price string `json:"price"`
}
Here's a simplified example of what the data might look like. I've prefixed the names of the nonstandard fields I'd want to pass through with custom:
{
"id": "ABC123",
"name": "Great Product",
"price": 10.99,
"custom_alternate_names": ["Great Product"],
"custom_sellers": [{
"id": "ABC123",
"name": "Great Product LLC",
"emails": ["great#stuff.com"]
}]
}
It would be fine to special case for places where the route actually needs to process or manipulate a custom field. But in most cases, passing through the JSON data unchanged is fine, so the fact we aren't imposing any type mappings for safety isn't adding anything.
Is there a way to set up the struct with an interface (?) that could act as a passthrough for any unmapped fields? Or a way to take the unparsed JSON data and recombine it with the mapped object before the data are returned?
You can do something like this
package main
import (
"encoding/json"
"log"
)
type Product struct {
//embed product to be able to pull the properties without
//requiring a nested object in the json
KnownFields
OtherStuff json.RawMessage
}
//Custom unmarshaller
func (p *Product) UnmarshalJSON(b []byte) error {
var k KnownFields
//unmarshal the known fields
err := json.Unmarshal(b, &k)
if err != nil {
return err
}
p.KnownFields = k
//You can use json.RawMessage or map[string]interface
p.OtherStuff = json.RawMessage(b)
return nil
}
type KnownFields struct {
Name string `json:"name"`
Price json.Number `json:"price"`
}
const JSON = `
{
"id": "ABC123",
"name": "Great Product",
"price": 10.99,
"custom_alternate_names": ["Great Product"],
"custom_sellers": [{
"id": "ABC123",
"name": "Great Product LLC",
"emails": ["great#stuff.com"]
}]
}`
func main() {
var p Product
err := json.Unmarshal([]byte(JSON), &p)
if err != nil {
log.Panic(err)
}
log.Printf("%v", p)
}
If you are going to mutate and marshall the Product you will have to implement a custom marshaller and also you would need to use map[string]interface{} as the OtherStuff to avoid having duplicate entries for the known fields

Making a struct before the unmarshal in golang

Im new in golang and i have a question.
I have 5 structs where i use Json , but the JSON file can have more structs than the ones i have predetermined BUT... the structures of the JSON satisfies the structures in my programing ( lets say i have 5 structs "struct1" , 6 structs "struct 2" , 1 struct "struct 3", and so on...)
My question is , i want to make a function were i take the JSON FILE , read the structs of it and have as an output the number of structs of the JSON file.
I think i could use the map[string]interface{} but i dont understand it.
I hope i have explain myself
Thank u very much!
Without example JSON or structs, the exact question you are asking is a bit hard to decipher, specifically the "output the number of structs" bit in the question, as it could be interpreted several different ways. I will do my best to answer what I think are the most probable questions you are asking.
Interfaces
First off, some basic go knowledge that might be useful, but outside JSON marshaling itself. The interface{} type appears special, but is not a hardwired keyword as it first might appear. What the interface keyword does is describe the requirements that an object must have to fulfill that interface. Because interface{} has no requirements, and because everything is automatically interfaced in go, everything satisfies the interface{} type.
Because of this implementation of interfaces, map[string]interface{} is really map[string]. This allows for the JSON un/marshal to not care about what is on the value side of the map. This exactly lines up with the format of JSON, where you have a string key on one side, and a value that could be any of the JSON datatypes on the other.
How many different objects are in the base JSON object?
let us take an example json
{
"debug": "on",
"window": {
"title": "Sample Konfabulator Widget",
"name": "main_window",
"width": 500,
"height": 500
},
"image": {
"src": "Images/Sun.png",
"name": "sun1",
"hOffset": 250,
"vOffset": 250,
"alignment": "center"
},
"text": {
"data": "Click Here",
"size": 36,
"style": "bold",
"name": "text1",
"hOffset": 250,
"vOffset": 100,
"alignment": "center",
"onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;"
}
}
The answer to the question in this circumstance would be four. debug, window, image, and text.
The process for determining the number would then be:
Load the json into a byte array.
Marshal into an interface{}
Determine type (array vs object etc.) using type switch. see this A Tour of Go page
If you know the type, you can skip this step
Convert to desired type
Get length, or perform any other operation as desired.
package main
import (
"encoding/json"
"fmt"
)
func main() {
myJSON := `<see above>`
var outStruct *interface{}
json.Unmarshal([]byte(myJSON), &outStruct)
outMap := (*outStruct).(map[string]interface{})
fmt.Printf("Num Structs: %d", len(outMap))
}
Go Playground
How many json objects that I do not have structs for are present?
This answer has a very similar answer to the first one, and is really about manipulation of the output map and the struct
Taking almost the entire code from the first one to the second, let us assume that you have the following structs set up
type Image struct {
Name string
//etc
}
type Text struct {
Name string
//etc
}
type Window struct {
Name string
//etc
}
type Base struct {
Image Image
Window Window
Text Text
}
In this case, in addition to the previous steps, you would have to
5. Marshal the json into a base object
6. Go through the map[string]interface{}, and for each key
7. Determine if the key is one of the objects in your base struct
total := 0
for k, _ := range outMap {
if k != "image" && k != "text" && k != "window" && k != "other" {
total++
}
}
fmt.Printf("Total unknown structs: %d\n", total)
How many of my structs are empty?
This last question is also rather simple, and could be done by checking the map for a value given the input key, but for completion's sake, the example code marshals the JSON into a struct, and uses that.
Marshal JSON into base
For each of Window, Item, Text, Other in base, determine if empty.
total = 0
if (Image{}) == outBase.Image {
total++
}
if (Window{}) == outBase.Window {
total++
}
if (Text{}) == outBase.Text {
total++
}
if (Other{}) == outBase.Other {
total++
}
fmt.Printf("Total empty structs: %d\n", total)
Go Playground
See this go blog post for more information on golang JSON.

Can one run a PutItem in DynamoDB in Golang without using a struct?

All of AWS' examples involve using a struct to organize and optimize your data prior to doing anything with DynamoDB. See: https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/dynamo-example-create-table-item.html
For my project I am getting a JSON string back from a third party API. I actually want to store the entire JSON object in DynamoDB but I can't determine if it's possible without a struct. Because I'm working with a third party API, I can't guarantee what fields I will get, or if the fields will change in the future. Can this be done? In NodeJS it's quite easy, but I understand that Go is statically typed so it may just not be possible.
You will always need a struct for your item static key and/or range. The JSON can go in another field of type map, which can contain items or nested maps. Then let attributevalue.MarshalMap transform the whole struct into a DynamoDB map.
type Record struct {
ID string
JSON map[string]interface{}
}
r := Record{
ID: "ABC123",
JSON: map[string]interface{}{
"Parent": "aa",
"Children": map[string]interface{}{
"Child1": "foo",
"Child2": "bar",
},
},
}
av, err := attributevalue.MarshalMap(r)
if err != nil {
return fmt.Errorf("failed to marshal Record, %w", err)
}
_, err = client.PutItem(context.TODO(), &dynamodb.PutItemInput{
TableName: aws.String(myTableName),
Item: av,
})
if err != nil {
return fmt.Errorf("failed to put Record, %w", err)
}
example partly taken from https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue

Parse structs from API

I am trying to access some information stored in a json file via Go. I have two related issues. One is that I'm not sure how to organize my structs and secondly how do I access them via a variable. I'll notate my code to make a little more sense
// To be clear, this is dummy info and I'm linting my actual json
// data. It loads fine, I just don't want to get hung up on this side
{
"A": {
"lob": "A",
"url": [
"example.com",
"test.com"]
}
"B": {
"lob": "B",
"url": [
"example2.com",
"test2.com"]
}
}
So the concern is that the structure of the options is identical. I am building this as part of a REST AP. The hope is that users can use http://testapi.com/getdata/A and it will return the urls and name info under A and likewise for B. As is, it loads both of them as separate components of the same struct:
type SiteList struct {
A struct {
Lob string `json:"lob"`
URL []string `json:"url"`
} `json:"test"`
B struct {
Lob string `json:"lob"`
URL []string `json:"url"`
} `json:"test2"`
}
I can do .A or .B by hand but I'm wondering how to handle it when the requests come in so that my API will only return the data under A or B.
If you're going to consume the API via accessing the API via http://testapi.com/getdata/A or http://testapi.com/getdata/B then A and B can be considered the parameters that drive the behavior of your API.
If you're passing A, you basically want to access the site data associated with A and if you're passing B, the site data for B should be returned.
An easy way to organize this data internally is to use a dedicated Go type site which holds Lob and URL and arrange everything in a map via map[string]site, which is initialized on startup of your server.
You can then dynamically access the parameter given to your API (A or B, but can be easily extended), lookup the site information from the map and, in case it's a valid site, return the corresponding data encoded as JSON.
package main
import (
"encoding/json"
"fmt"
"log"
"net/http"
)
type site struct {
Lob string `json:"lob"`
URL []string `json:"url"`
}
var sites = map[string]site{
"A": site{
Lob: "A",
URL: []string{
"example.com",
"test.com",
},
},
"B": site{
Lob: "B",
URL: []string{
"example2.com",
"test2.com",
},
},
}
const endpoint = "/getdata/"
func handler(w http.ResponseWriter, r *http.Request) {
lob := r.URL.Path[len(endpoint):]
s, ok := sites[lob]
if !ok {
w.WriteHeader(http.StatusNotFound)
return
}
resp, err := json.Marshal(s)
if err != nil {
w.WriteHeader(http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "application/json; charset=UTF-8")
w.Write(resp)
}
func main() {
http.HandleFunc(endpoint, handler)
log.Fatal(http.ListenAndServe(":8080", nil))
}
Missing values in the json will be unmarshalled with their zero values. An int will be 0, a string will be "", a map and any pointer will be nil. Structs will be initialized, but their fields will have the zero value.
In your case, when comcast is missing in the json, B will be initialized as a struct, where Lob is "" and URL is an empty slice of string.
If I understand correctly you have a struct SiteList and depending on if they user navigates to /getData/A or /getData/B you want to serve SiteList.A or SiteList.B.
From the json marshal docs here or this SO answer you can leave out some fields of a struct when they are empty. Basically if there's no data in that field then it will not appear in the marshalled json.
The "omitempty" option specifies that the field should be omitted from
the encoding if the field has an empty value, defined as false, 0, a
nil pointer, a nil interface value, and any empty array, slice, map,
or string.
With that in mind, if you have the control in the handlers of the requests to be able to set the values in SiteList according to the path then you could make use of this feature.
If you extract the shared parts of A and B into a struct:
type Inner struct {
Lob string `json:"lob"`
URL []string `json:"url"`
}
// using pointers here for A and B means setting to nil is empty
type SiteList struct {
A *Inner `json:"test,omitempty"`
B *Inner `json:"test2,omitempty"`
}
And then set the one that you do not want to be in the response body to nil. (nil is empty for a pointer, so it will not get marshalled.)

Parsing JSON in go

My question is about parsing JSON files in golang.
Specifically, I am trying to parse output of a command "docker network inspect bridge", which happens to be in JSON format. The command is described here. My goal is to get a list of "IPv4Address" for the listed containers.
I have tried to do this but failing to convert map[string]interface{} to map[string]string. My code is here:- https://play.golang.org/p/eO_j996gGb
$ sudo docker network inspect bridge
[
{
"Name": "bridge",
"Id": "b2b1a2cba717161d984383fd68218cf70bbbd17d328496885f7c921333228b0f",
"Scope": "local",
"Driver": "bridge",
"IPAM": {
"Driver": "default",
"Config": [
{
"Subnet": "172.17.42.1/16",
"Gateway": "172.17.42.1"
}
]
},
"Internal": false,
"Containers": {
"bda12f8922785d1f160be70736f26c1e331ab8aaf8ed8d56728508f2e2fd4727": {
"Name": "container2",
"EndpointID": "0aebb8fcd2b282abe1365979536f21ee4ceaf3ed56177c628eae9f706e00e019",
"MacAddress": "02:42:ac:11:00:02",
"IPv4Address": "172.17.0.2/16",
"IPv6Address": ""
},
"f2870c98fd504370fb86e59f32cd0753b1ac9b69b7d80566ffc7192a82b3ed27": {
"Name": "container1",
"EndpointID": "a00676d9c91a96bbe5bcfb34f705387a33d7cc365bac1a29e4e9728df92d10ad",
"MacAddress": "02:42:ac:11:00:01",
"IPv4Address": "172.17.0.1/16",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.bridge.default_bridge": "true",
"com.docker.network.bridge.enable_icc": "true",
"com.docker.network.bridge.enable_ip_masquerade": "true",
"com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
"com.docker.network.bridge.name": "docker0",
"com.docker.network.driver.mtu": "1500"
}
}
]
What is the correct way to parse such JSON files in golang. I did try using type conversion but doesn't help. I have tried many things but finally could only get as far as the code shown in the snippet. How can I extract the IPv4Address field from the "cont" object?
Link to go playground snippet I have is https://play.golang.org/p/eO_j996gGb. Any help is greatly appreciated (willing to try things out, don't need just the code but ideas :) Thanks.
Your code just needs to be modified to further unpackage the data. You're leaving it with each container object represented as an interface{} which makes it so you can't access the fields within it. Another assert on each to make it a map[string]interface{} will allow you to access the fields within by name (like IPv4Address). Here's a working example; https://play.golang.org/p/4OC5axN4Gd
Here's the important code change;
containers := (foo[0]["Containers"]).(map[string]interface{})
//fmt.Printf("containers %+v", containers)
for _, v := range containers {
unwrapped := v.(map[string]interface{})
fmt.Printf("\ncont %+v\n", unwrapped["IPv4Address"])
}
v is just an interface{} so you have no way of accessing the IPv4Address field without asserting again/converting into a type.
Now that works fine, however, I would recommend getting away from the heavy use of interfaces in your code. Your code is full of unsafe operations that create a need for a lot of error handling (like every time you attempt access into the map you have the potential to throw, as well as anytime you do a type assert). There are plenty of examples of how to do this here on SO but if you leave a comment requesting it I can produce another example that will unmarshal into structs. This is a lot more safe and maintainable imo because most errors will be raised on the call to unmarshal, rather than having potentially a lot of code that works with the results that could all cause a panic.
Here is some play code that has all the types in it and uses unmarshal; https://play.golang.org/p/VfHst9GaiR
Your types can be represented with the following;
type DockerInstance struct {
Name string `json:"Name"`
Id string `json:"Id"`
Scope string `json:"Scope"`
Driver string `json:"Driver"`
EnableIPv6 bool `json:"EnableIPv6"`
IPAM IPAM `json:"IPAM"`
Internal bool `json:"Internal"`
Containers map[string]Container `json:"Containers"`
Options map[string]string `json:"Options"`
Labels interface{} `json:"Labels"`
}
type IPAM struct {
Driver string `json:"Driver"`
Options interface{} `json:"Options"`
Config []Conf `json:"Config"`
}
type Conf struct {
Subnet string `json:"Subnet"`
}
type Container struct {
Name string `json:"Name"`
EndPointID string `json:"EndpointID"`
MacAddress string `json:"MacAddress"`
IPv4Address string `json:"IPv4Address"`
IPv6Address string `json:"IPv6Address"`
}
There's a few stuff I still have set to interface, that's just cause the sample json doesn't include any data for them so I can't know what their type should be in the Go source. Here's some basic stuff I also added in your main to test out the objects/make sure I had things defined correctly;
docks := []DockerInstance{} //note you're using an array of the top level type
err := json.Unmarshal([]byte(teststring), &docks)
if err != nil {
fmt.Println(err)
} else {
fmt.Println()
fmt.Println()
fmt.Println(docks[0].Name)
}
for k, v := range docks[0].Containers {
fmt.Println(k)
fmt.Println(v.IPv4Address)
}
for k, v := range docks[0].Options {
fmt.Println(k)
fmt.Println(v)
}