I have a question regarding the golang unmarshalling . I was trying to unmarshal Json array but it is giving nil result for one decoding while it is successful in the other. I don't understand the reason behind it. Is it a mistake in the code or expected?
package main
import "fmt"
import "encoding/json"
type PublicKey struct {
Id int
Key string
}
type KeysResponse struct {
Collection []PublicKey
}
func main() {
keysBody := []byte(`[{"id": 1,"key": "-"},{"id": 2,"key": "-"},{"id": 3,"key": "-"}]`)
keys := make([]PublicKey,0)
json.Unmarshal(keysBody, &keys)//This works
fmt.Printf("%#v\n", keys)
response := KeysResponse{}
json.Unmarshal(keysBody, &response)//This doesn't work
fmt.Printf("%#v\n", response)
}
http://play.golang.org/p/L9xDG26M8-
That's not expected to work. What you have in the json is an array of type PublicKey. The KeysResponse type would be used for json looking like this;
{
"Collection": [{"id": 1,"key": "-"},{"id": 2,"key": "-"},{"id": 3,"key": "-"}]
}
Which is not what you have. If you want the data to be stored in that type I'd recommend the following; response := KeysResponse{keys} on the line after you unmarshal into keys.
To elaborate on that distinction. In the working case the json is just an array with objects inside of it. The json I wrote above is an object which has a single property named Collection which is of type array and the objects in the array are represented by the PublicKey type (objects with an int called id and a string called key). When working on code to unmarshal json, it's helpful to describe the structure using plain English like this, it tells you precisely what types/structures you need in Go to hold the data.
Related
I have multiple APIs that follow a similar structure on the high level response. It always gives back an answer in that form:
{"data": {"feed":[{...}]}, "success": true}
However, the structure in Feed varies, depending on the concrete API.
I would now like to build an abstract function to process the various APIs. I have the following objects:
type SourceDTO struct { // top level object
Success bool `json:"success"`
Data Feed `json:"data"`
}
type Feed struct {
FeedData []<???> `json:"Feed"`
}
(The real object is more complex, but this shows the idea)
How would be a good way in go to parse this for the different APIs, ut having some common code with some logic based on the high level data (e.g. success)?
EDIT:
I am extending this, to explain more the extend of my question about the "pattern" I am looking for.
I want to create this package that parses the group of APIs. The DTO objects then have to be transferred into some other objects. These 'final' objects are defined in a different package (the entity package) and have then to be persisted.
I am now wondering, how to bring all this together: The 'finaly' entity objects, the transformation functions from DTO to entity, the parsing of the different APIs and their common and different result components.
Where do the transformation functions belong to (package wise)?
EDIT2: Specified FeedData to a slice after digging into the problem (see comments)
You can embed your SourceDTO struct into another struct, like this:
type SourceDTO struct { // top level object
Success bool `json:"success"`
}
type FeedResponse struct {
FeedData YourCustomFeedStruct `json:"feed"`
// Embedded Struct
SourceDTO
}
Now you can access the Success bool from the FeedResponse struct. Also any methods defined on the SourceDTO struct can be accessed from the FeedResponse.
Thanks to #mkopriva for the input for this solution.
In order to have some abstraction in your json unmarshalling it is possible to use interface{} for many use cases.
package main
import (
"encoding/json"
"fmt"
)
type UniversalDTO struct {
Success bool `json:"success"`
Data interface{} `json:"data"`
}
type ConcreteData struct {
Source string `json:"source"`
Site string `json:"site"`
}
func main() {
jsondata := []byte(`{"sucess":"true","data":[{"source":"foo","site":"bar"}]}`)
data := make([]ConcreteData, 0, 10)
dtoToSend := UniversalDTO{Data: &data}
describe(dtoToSend)
describe(dtoToSend.Data)
json.Unmarshal(jsondata, &dtoToSend)
describe(dtoToSend)
describe(dtoToSend.Data)
}
func describe(i interface{}) {
fmt.Printf("(%v, %T)\n", i, i)
}
Test here: https://play.golang.org/p/SSSp_zptMVN
json.Unmarshal expects an object into which the json is being put into. Thus, first we always need an object. Depending on the concrete instance of the target object, the interface{} can be overriden with a concrete struct object (which of course has to be created separately). An important learning here is, that a go interface can also be overridden with a slice. In this way, it is also possible to unmarshal an array into a go object. However, a slice of a struct has to be defined as a slice of pointers to that type.
Currently, I try to parse JSON to map[string][]interface{}, but unmarshalling returns an error. According to (https://golang.org/pkg/encoding/json/), to unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
-[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
I wonder if golang is able to unmarshal map[string][]interface{}. The following is code snippet. I am new to Golang, thanks for help in advance.
// emailsStr looks like "{"isSchemaConforming":true,"schemaVersion":0,"unknown.0":[{"email_address":"test1#uber.com"},{"email_address":"test2#uber.com"}]}"
emailsRaw := make(map[string][]*entities.Email)
err := json.Unmarshal([]byte(emailsStr), &emailsRaw)
Error message:
&json.UnmarshalTypeError{Value:"number", Type:(*reflect.rtype)(0x151c7a0), Offset:44, Struct:"", Field:""}
The Go encoding/json package will only unmarshal dynamically to a map[string]interface{}. From there, you will need to use type assertions and casting to pull out the values you want, like so:
func main() {
jsonStr := `{"isSchemaConforming":true,"schemaVersion":0,"unknown.0":[{"email_address":"test1#uber.com"},{"email_address":"test2#uber.com"}]}`
dynamic := make(map[string]interface{})
json.Unmarshal([]byte(jsonStr), &dynamic)
firstEmail := dynamic["unknown.0"].([]interface{})[0].(map[string]interface{})["email_address"]
fmt.Println(firstEmail)
}
(https://play.golang.org/p/VEUEIwj3CIC)
Each time, Go's .(<type>) operator is used to assert and cast the dynamic value to a specific type. This particular code will panic if anything happens to be the wrong type at runtime, like if the contents of unknown.0 aren't an array of JSON objects.
The more idiomatic (and robust) way to do this in Go is to annotate a couple structs with json:"" tags and have encoding/json unmarshal into them. This avoids all the nasty brittle .([]interface{}) type casting:
type Email struct {
Email string `json:"email_address"`
}
type EmailsList struct {
IsSchemaConforming bool `json:"isSchemaConforming"`
SchemaVersion int `json:"schemaVersion"`
Emails []Email `json:"unknown.0"`
}
func main() {
jsonStr := `{"isSchemaConforming":true,"schemaVersion":0,"unknown.0":[{"email_address":"test1#uber.com"},{"email_address":"test2#uber.com"}]}`
emails := EmailsList{}
json.Unmarshal([]byte(jsonStr), &emails)
fmt.Printf("%+v\n", emails)
}
(https://play.golang.org/p/iS6e0_87P2J)
A better approach will be to use struct for main schema and then use an slice of email struct for fetching the data for email entities get the values from the same according to requirements. Please find the solution below :-
package main
import (
"fmt"
"encoding/json"
)
type Data struct{
IsSchemaConforming bool `json:"isSchemaConforming"`
SchemaVersion float64 `json:"schemaVersion"`
EmailEntity []Email `json:"unknown.0"`
}
// Email struct
type Email struct{
EmailAddress string `json:"email_address"`
}
func main() {
jsonStr := `{"isSchemaConforming":true,"schemaVersion":0,"unknown.0":[{"email_address":"test1#uber.com"},{"email_address":"test2#uber.com"}]}`
var dynamic Data
json.Unmarshal([]byte(jsonStr), &dynamic)
fmt.Printf("%#v", dynamic)
}
I've got a problem with my json parsing in golang.
I used some code to parse the json into a map[string]interface{}{} but when I try to iterate through a nested field, the error (type interface {} does not support indexing) is triggered.
I'd like to get the following informations :
iterate through each response->blogs and then get the url of original_size photo that lays in response->posts->blog_n->photos->original_size
meta->status
response->blog->total_posts and response->blog->name
Here's a link to a playground
Thank you for your help !
Is there a reason you want to use a map? To do the indexing you're talking about, with maps, I think you would need nested maps as well.
Have you considered using nested structs, as described here, Go Unmarshal nested JSON structure and Unmarshaling nested JSON objects in Golang ?
For your JSON data, here is a sample -- working but limited -- struct. Go will ignore the fields you don't use.
func main() {
//Creating the maps for JSON
data := &Header{}
//Parsing/Unmarshalling JSON encoding/json
err := json.Unmarshal([]byte(input), &data)
if err != nil {
panic(err)
}
fmt.Printf("%+v\n", m)
}
type Header struct {
Response struct {
Blog struct {
Title string `json:"title"`
} `json:"blog"`
Posts []struct {
Id int64 `json:"id"`
} `json:"posts"`
} `json:"response"`
}
To get JSON with Go to do what you want, you have to brush up on JSON and it's relation to Go types.
Notice posts:
slices of structs are, in JSON, arrays of objects, [{..},{..}]
In Go, only exported fields will be filled.
this error appear because your map[key-type]val-type, and you to try get value as nested map.
you can use Type Assertion to get value.
result := m["response"].(map[string]interface{})
fmt.Printf("%+v\n", result["blog"])
Hi everyone I'm trying to see what the proper way of accessing fields of a json object from a http.get request in go.
I first do an http.get call get the JSON and then print it (which works) but is there a way to access just a field?
for example:
response, err:= http.Get("URL")
//Error checking is done between
contents, err:=ioutil.Readall(response.Body)
//Now at this point I have a json that looks like
{"id": "someID",
"name": "someName",
"test": [{"Name":"Test1",
"Result": "Success"},
{"Name":"Test2",
"Result": "Success"},
{...},
]}
Is there a way to only print the "test" of the Json? What is the proper way of accessing that field?
Use encoding/json package to Unmarshal data into struct, like following.
type Result struct {
ID string `json:"id"`
Name string `json:"name"`
Test []interface{} `json:"test"`
}
var result Result
json.Unmarshal(contents, &result)
fmt.Println(result.Test)
You can also parse Test to specific struct.
Same as the previous answer, use encoding/json package to Unmarshal data. But if you don't want to specify the structure, you could use map[string]interface/bson.M{} to receive the data, and get the field, then cast into types your want.
m := make(map[string]interface{})
err := json.Unmarshal(data, &m)
if err != nil {
log.Fatal(err)
}
fmt.Println(m["id"])
You may want to try gabs container, if you are not sure how depth JSON hierarchy can be. Have a look at below resources
https://github.com/Jeffail/gabs
https://godoc.org/github.com/Jeffail/gabs
If you just want to access one field then you can use the jsonq module https://godoc.org/github.com/jmoiron/jsonq
For your example you could get the test object with code similar to
jq.Object("test")
Where jq is a jsonq query object constructed from your JSON above (see the godoc page for instructions on how to create a query object from a JSON stream or string).
You can also use this library for retrieving specific String, Integer, Float and Bool values at an arbitrary depth inside a JSON object.
Since you are starting with a URL, Decode is a better option than Unmarshal:
package main
import (
"encoding/json"
"net/http"
)
func main() {
r, e := http.Get("https://github.com/manifest.json")
if e != nil {
panic(e)
}
defer r.Body.Close()
var s struct { Name string }
json.NewDecoder(r.Body).Decode(&s)
println(s.Name == "GitHub")
}
https://golang.org/pkg/encoding/json#Decoder.Decode
You may check this https://github.com/valyala/fastjson
s := []byte(`{"foo": [123, "bar"]}`)
fmt.Printf("foo.0=%d\n", fastjson.GetInt(s, "foo", "0"))
// Output:
// foo.0=123
I have JSON from an API that I want to save to MongoDB using the mgo package. My JSON looks like this:
{
"something": "value"
"collection": [
{ "obj1": "value" }
// ... (Variable number of objects here)
]
}
To save this data I've created a new type in my Go application that looks like this:
type MyData struct {
Something string
Collection []string // This actually contains more than strings but I'll be happy if I can just get strings saved
}
mongoSess, err := mgo.Dial("localhost:27017")
if err != nil {
panic(err)
}
defer mongoSess.Close()
c := mongoSess.DB("mydatabase").C("mycollection")
insertErr := c.Insert(&MyData{something, collection})
This code works but the problem is that it isn't saving anything in my collection field which should be an array of JSON objects. Nothing at all. I get the keys in my database and they are the right type but they have no data. Here's what the Mongo output is:
{ "_id" : ObjectId("5520c535a236d8a9a215d096"), "something" : "value", "collection" : [ ] }
Can anyone spot what it is I'm doing wrong? I'm obviously new to Go and having trouble with types.
Solution
The answers here really helped me a lot. I'm at fault for not properly explaining things as the answers sent me on the right track but didn't solve the issue directly. Here's what the actual solution is.
package main
import (
"encoding/json"
"github.com/bitly/go-simplejson"
"gopkg.in/mgo.v2"
//"gopkg.in/mgo.v2/bson"
// Other packages are used as well
)
type MyData struct {
Something int
Collection []interface{}
}
func main() {
// I'm using SimpleJson for parsing JSON
collection, cerr := jsonFromExternalApi.Get("collection").Array()
if cerr != nil {
logger.Debug.Fatalln(cerr)
}
// Save response to Mongo (leaving out all the connection code)
c := mongoSess.DB("mydb").C("mycollection")
insertErr := c.Insert(&Producer{npn, licenses })
}
The issue was that SimpleJSON was returning the array of objects from my API call as a []interface{}. I was not aware I could simply declare part of a struct to be an interface so instead of just correcting what Go's compiler was telling me was wrong I was making it way harder than it should have been.
Coming from loosely typed scripting languages, stuff like this really trips me up and sometimes its hard to see the benefit but hopefully this helps someone out one day.
Looks you have the wrong data structure.
insertErr := c.Insert(&MyData{something, collection})
something => string and collection => slice
You code should be like this:
insertErr := c.Insert(&MyData{"something", []string{"obj1", "value"}})
Here is the working code.
[{Something:something Collection:[obj1 value]} {Something:something Collection:[obj1 value]} {Something:something Collection:[obj1 value]} {Something:something Collection:[obj1 value]}]
For further reference, mgo has great documentation and you can find sample code and details about running mongodb queries here.
Here you are not defining proper type to Collection in your MyData struct.
Collection []string //This is what you are defining.
But from Json you are not getting string array,you are getting map[string]interface{}
This is the reason you are not filling Mydata struct properly
Correct MyData struct will be
type MyData struct {
Something string
Collection map[string]string // This actually contains more than strings but I'll be happy if I can just get strings saved
}