How to insert array of objects into MongoDB using Go - json

I have JSON from an API that I want to save to MongoDB using the mgo package. My JSON looks like this:
{
"something": "value"
"collection": [
{ "obj1": "value" }
// ... (Variable number of objects here)
]
}
To save this data I've created a new type in my Go application that looks like this:
type MyData struct {
Something string
Collection []string // This actually contains more than strings but I'll be happy if I can just get strings saved
}
mongoSess, err := mgo.Dial("localhost:27017")
if err != nil {
panic(err)
}
defer mongoSess.Close()
c := mongoSess.DB("mydatabase").C("mycollection")
insertErr := c.Insert(&MyData{something, collection})
This code works but the problem is that it isn't saving anything in my collection field which should be an array of JSON objects. Nothing at all. I get the keys in my database and they are the right type but they have no data. Here's what the Mongo output is:
{ "_id" : ObjectId("5520c535a236d8a9a215d096"), "something" : "value", "collection" : [ ] }
Can anyone spot what it is I'm doing wrong? I'm obviously new to Go and having trouble with types.
Solution
The answers here really helped me a lot. I'm at fault for not properly explaining things as the answers sent me on the right track but didn't solve the issue directly. Here's what the actual solution is.
package main
import (
"encoding/json"
"github.com/bitly/go-simplejson"
"gopkg.in/mgo.v2"
//"gopkg.in/mgo.v2/bson"
// Other packages are used as well
)
type MyData struct {
Something int
Collection []interface{}
}
func main() {
// I'm using SimpleJson for parsing JSON
collection, cerr := jsonFromExternalApi.Get("collection").Array()
if cerr != nil {
logger.Debug.Fatalln(cerr)
}
// Save response to Mongo (leaving out all the connection code)
c := mongoSess.DB("mydb").C("mycollection")
insertErr := c.Insert(&Producer{npn, licenses })
}
The issue was that SimpleJSON was returning the array of objects from my API call as a []interface{}. I was not aware I could simply declare part of a struct to be an interface so instead of just correcting what Go's compiler was telling me was wrong I was making it way harder than it should have been.
Coming from loosely typed scripting languages, stuff like this really trips me up and sometimes its hard to see the benefit but hopefully this helps someone out one day.

Looks you have the wrong data structure.
insertErr := c.Insert(&MyData{something, collection})
something => string and collection => slice
You code should be like this:
insertErr := c.Insert(&MyData{"something", []string{"obj1", "value"}})
Here is the working code.
[{Something:something Collection:[obj1 value]} {Something:something Collection:[obj1 value]} {Something:something Collection:[obj1 value]} {Something:something Collection:[obj1 value]}]
For further reference, mgo has great documentation and you can find sample code and details about running mongodb queries here.

Here you are not defining proper type to Collection in your MyData struct.
Collection []string //This is what you are defining.
But from Json you are not getting string array,you are getting map[string]interface{}
This is the reason you are not filling Mydata struct properly
Correct MyData struct will be
type MyData struct {
Something string
Collection map[string]string // This actually contains more than strings but I'll be happy if I can just get strings saved
}

Related

go JSON nightmare - Is there a simple [POSH] ConvertFrom-Json equivalent?

In powershell, if I make a REST call and receive any kind of json response, I can easily $json | ConvertFrom-Json into a proper object so I can make modifications, render specific values, whatever.
It seems like in Go I have to either define a struct, or "dynamically" convert using a map[string]interface{}.
The issue with a struct is that I am writing a rest handler for a platform that, depending on the endpoint, serves wildly different JSON responses, like most REST APIs. I don't want to define a struct for all of the dozens of possible responses.
The problem with map[string]interface{} is that it pollutes the data by generating a string with a bunch of ridiculous 'map' prefixes and unwanted [brackets].
ala: [map[current_user_role:admin id:1]]
Is there a way to convert a JSON response like:
{
"current_user_role": "admin",
"id": 1
}
To return a basic:
current_user_role: admin
id: 1
... WITHOUT defining a struct?
Your approach of using a map is right if you don't wish to specify the structure of the data you're receiving. You don't like how it is output from fmt.Println, but I suspect you're confusing the output format with the data representation. Printing them out in the format you find acceptable takes a couple of lines of code, and is not as convenient as in python or powershell, which you may find annoying.
Here's a working example (playground link):
package main
import (
"encoding/json"
"fmt"
"log"
)
var data = []byte(`{
"current_user_role": "admin",
"id": 1
}`)
func main() {
var a map[string]interface{}
if err := json.Unmarshal(data, &a); err != nil {
log.Fatal(err)
}
for k, v := range a {
fmt.Printf("%s: %v\n", k, v)
}
}

How to access fields of a JSON in GO

Hi everyone I'm trying to see what the proper way of accessing fields of a json object from a http.get request in go.
I first do an http.get call get the JSON and then print it (which works) but is there a way to access just a field?
for example:
response, err:= http.Get("URL")
//Error checking is done between
contents, err:=ioutil.Readall(response.Body)
//Now at this point I have a json that looks like
{"id": "someID",
"name": "someName",
"test": [{"Name":"Test1",
"Result": "Success"},
{"Name":"Test2",
"Result": "Success"},
{...},
]}
Is there a way to only print the "test" of the Json? What is the proper way of accessing that field?
Use encoding/json package to Unmarshal data into struct, like following.
type Result struct {
ID string `json:"id"`
Name string `json:"name"`
Test []interface{} `json:"test"`
}
var result Result
json.Unmarshal(contents, &result)
fmt.Println(result.Test)
You can also parse Test to specific struct.
Same as the previous answer, use encoding/json package to Unmarshal data. But if you don't want to specify the structure, you could use map[string]interface/bson.M{} to receive the data, and get the field, then cast into types your want.
m := make(map[string]interface{})
err := json.Unmarshal(data, &m)
if err != nil {
log.Fatal(err)
}
fmt.Println(m["id"])
You may want to try gabs container, if you are not sure how depth JSON hierarchy can be. Have a look at below resources
https://github.com/Jeffail/gabs
https://godoc.org/github.com/Jeffail/gabs
If you just want to access one field then you can use the jsonq module https://godoc.org/github.com/jmoiron/jsonq
For your example you could get the test object with code similar to
jq.Object("test")
Where jq is a jsonq query object constructed from your JSON above (see the godoc page for instructions on how to create a query object from a JSON stream or string).
You can also use this library for retrieving specific String, Integer, Float and Bool values at an arbitrary depth inside a JSON object.
Since you are starting with a URL, Decode is a better option than Unmarshal:
package main
import (
"encoding/json"
"net/http"
)
func main() {
r, e := http.Get("https://github.com/manifest.json")
if e != nil {
panic(e)
}
defer r.Body.Close()
var s struct { Name string }
json.NewDecoder(r.Body).Decode(&s)
println(s.Name == "GitHub")
}
https://golang.org/pkg/encoding/json#Decoder.Decode
You may check this https://github.com/valyala/fastjson
s := []byte(`{"foo": [123, "bar"]}`)
fmt.Printf("foo.0=%d\n", fastjson.GetInt(s, "foo", "0"))
// Output:
// foo.0=123

How do I do this? Get an struct send to a func as interface and get it back as struct?

I'm new in golang development and have some question regarding something related to this question.
As a learning exercise, I'm trying to create a simple library to handle json based configuration file. As a configuration file to be used for more then one app, it should be able to handle different parameters. Then I have created a type struct Configuration that has the filename and a data interface. Each app will have a struct based on its configuration needs.
In the code bellow, I put all together (lib and "main code") and the "TestData struct" is the "app parameters".
If it doesn't exists, it will set a default values and create the file, and it is working. But when I try to read the file. I try to decode the json and put it back into the data interface. But it is giving me an error and I couldn't figure out how to solve this. Can someone help on this?
[updated] I didn't put the targeted code before, because I though that it would be easier to read in in all as a single program. Bellow is the 'targeted code' for better view of the issue.
As I will not be able to use the TestData struct inside the library, since it will change from program to program, the only way to handle this was using interface. Is there a better way?
library config
package config
import (
"encoding/json"
"fmt"
"os"
)
// Base configuration struct
type Configuration struct {
Filename string
Data interface{}
}
func (c *Configuration) Create(cData *Configuration) bool {
cFile, err := os.Open(cData.Filename)
defer cFile.Close()
if err == nil {
fmt.Println("Error(1) trying to create a configuration file. File '", cData.Filename, "' may already exist...")
return false
}
cFile, err = os.Create(cData.Filename)
if err != nil {
fmt.Println("Error(2) trying to create a configuration file. File '", cData.Filename, "' may already exist...")
return false
}
buffer, _ := json.MarshalIndent(cData.Data, "", "")
cFile.Write(buffer)
return true
}
func (c *Configuration) Read(cData *Configuration) bool {
cFile, err := os.Open(cData.Filename)
defer cFile.Close()
if err != nil {
fmt.Println("Error(1) trying to read a configuration file. File '", cData.Filename, "' may not already exist...")
return false
}
jConfig := json.NewDecoder(cFile)
jerr := jConfig.Decode(&cData.Data)
if jerr != nil {
panic(jerr)
}
return true
}
program using library config
package main
import (
"fmt"
"./config"
)
// struct basic para configuração
type TestData struct {
URL string
Port string
}
func main() {
var Config config.Configuration
Config.Filename = "config.json"
if !Config.Read(&Config) {
Config.Data = TestData{"http", "8080"}
Config.Create(&Config)
}
fmt.Println(Config.Data)
TestData1 := &TestData{}
TestData1 = Config.Data.(*TestData) // error, why?
fmt.Println(TestData1.URL)
}
NEW UPDATE:
I have made some changes after JimB comment about I'm not clear about some concepts and I tried to review it. Sure many things aren't clear for me yet unfortunately. The "big" understanding I believe I got, but what mess my mind up is the "ins" and "outs" of values and formats and pointers, mainly when it goes to other libraries. I'm not able yet to follow the "full path" of it.
Yet, I believe I had some improvement on my code.
I think that I have corrected some points, but still have some big questions:
I stopped sending "Configuration" as a parameter as all "data" were already there as they are "thenselfs" in the instance. Right?
Why do I have use reference in the line 58 (Config.Data = &TestData{})
Why to I have to use pointer in the line 64 (tmp := Config.Data.(*TestData)
Why I CANNOT use reference in line 69 (Config.Data = tmp)
Thanks
The reason you are running into an error is because you are trying to decode into an interface{} type. When dealing with JSON objects, they are decoded by the encoding/json package into map[string]interface{} types by default. This is causing the type assertion to fail since the memory structure for a map[string]interface{} is much different than that of a struct.
The better way to do this is to make your TestData struct the expected data format for your Configuration struct:
// Base configuration struct
type Configuration struct {
Filename string
Data *TestData
}
Then when Decoding the file data, the package will unmarshal the data into the fields that match the closest with the data it finds.
If you need more control over the data unmarshaling process, you can dictate which JSON fields get decoded into which struct members by using struct tags. You can read more about the json struct tags available here: https://golang.org/pkg/encoding/json/#Marshal
You are trying to assert that Config.Data is of type *TestData, but you're assigning it to TestData{"http", "8080"} above. You can take the address of a composite literal to create a pointer:
Config.Data = &TestData{"http", "8080"}
If your config already exsits, your Read method is going to fill in the Data field with the a default json data type, probably a map[string]interface{}. If you assign a pointer of the correct type to Data first, it will decode into the expected type.
Config.Data = &TestData{}
Ans since Data is an interface{}, you do not want to ever use a pointer to that value, so don't use the & operator when marshaling and unmarshaling.

Unmarshalling a json array in golang

I have a question regarding the golang unmarshalling . I was trying to unmarshal Json array but it is giving nil result for one decoding while it is successful in the other. I don't understand the reason behind it. Is it a mistake in the code or expected?
package main
import "fmt"
import "encoding/json"
type PublicKey struct {
Id int
Key string
}
type KeysResponse struct {
Collection []PublicKey
}
func main() {
keysBody := []byte(`[{"id": 1,"key": "-"},{"id": 2,"key": "-"},{"id": 3,"key": "-"}]`)
keys := make([]PublicKey,0)
json.Unmarshal(keysBody, &keys)//This works
fmt.Printf("%#v\n", keys)
response := KeysResponse{}
json.Unmarshal(keysBody, &response)//This doesn't work
fmt.Printf("%#v\n", response)
}
http://play.golang.org/p/L9xDG26M8-
That's not expected to work. What you have in the json is an array of type PublicKey. The KeysResponse type would be used for json looking like this;
{
"Collection": [{"id": 1,"key": "-"},{"id": 2,"key": "-"},{"id": 3,"key": "-"}]
}
Which is not what you have. If you want the data to be stored in that type I'd recommend the following; response := KeysResponse{keys} on the line after you unmarshal into keys.
To elaborate on that distinction. In the working case the json is just an array with objects inside of it. The json I wrote above is an object which has a single property named Collection which is of type array and the objects in the array are represented by the PublicKey type (objects with an int called id and a string called key). When working on code to unmarshal json, it's helpful to describe the structure using plain English like this, it tells you precisely what types/structures you need in Go to hold the data.

Parsing complex JSON into goweb REST Create() function

I apologise in advance for a very long question. Hopefully you'll bear with me.
I'm working with the goweb library, and experimenting with the example web app.
I've been trying to modify the RESTful example code, which defines a Thing as:
type Thing struct {
Id string
Text string
}
A Thing is created by sending an HTTP Post request, with an appropriate JSON body, to http://localhost:9090/things. This is handled in the example code in the Create function, specifically the lines:
dataMap := data.(map[string]interface{})
thing := new(Thing)
thing.Id = dataMap["Id"].(string)
thing.Text = dataMap["Text"].(string)
This is all well and good, and I can run the example server (which listens on http://localhost:9090/) and the server operates as expected.
For example:
curl -X POST -H "Content-Type: application/json" -d '{"Id":"TestId","Text":"TestText"}' http://localhost:9090/things
returns with no error, and then I GET that Thing with
curl http://localhost:9090/things/TestId
and it returns
{"d":{"Id":"TestId","Text":"TestText"},"s":200}
So far, so good.
Now, I would like to modify the Thing type, and add a custom ThingText type, like so:
type ThingText struct {
Title string
Body string
}
type Thing struct {
Id string
Text ThingText
}
This in itself isn't an issue, and I can modify the Create function like so:
thing := new(Thing)
thing.Id = dataMap["Id"].(string)
thing.Text.Title = dataMap["Title"].(string)
thing.Text.Body = dataMap["Body"].(string)
and the run the previous curl POST request with the JSON set to:
{"Id":"TestId","Title":"TestTitle","Title":"TestBody"}
and it returns with no error.
Once again I can GET the Thing URL and it returns:
{"d":{"Id":"TestId","Text":{"Title":"TestTitle","Body":"TestBody"}},"s":200}
Again, so far, so good.
Now, my question:
how do I modify the Create function to allow me to POST complex JSON to it?
For example, that last returned JSON string above includes {"Id":"TestId","Text":{"Title":"TestTitle","Body":"TestBody"}}. I'd like to be able to POST that exact JSON to the endpoint and have the Thing created.
I've followed the code back, and it seems that the data variable is of type Context.RequestData() from https://github.com/stretchr/goweb/context, and the internal Map seems to be of type Object.Map from https://github.com/stretchr/stew/, described as "a map[string]interface{} with additional helpful functionality." in particular, I noticed "Supports dot syntax to set deep values."
I can't work out how I can set up the thing.Text.Title = dataMap... statement so that the correct JSON field is parsed into it. I can't seem to use anything other than string types in the dataMap, and if I try that JSON it gives an error similar to:
http: panic serving 127.0.0.1:59113: interface conversion: interface is nil, not string
Once again, sorry for the ridiculously long question. I really appreciate you reading, and any help you may have to offer. Thanks!
As the JSON package documentation and JSON and Go introduction describe, JSON data can be parsed either generically through interface{} / string maps or unmarshalling directly to the struct types.
The example code you linked to and you based your changes on seems to use the generic string-map approach, dataMap := data.(map[string]interface{}).
As your desired JSON data is an object in an object, it’s simply a map within a map.
So you should be able to
dataMap := data.(map[string]interface{})
subthingMap := dataMap["Text"].(map[string]interface{})
thing.Text.Title = subthingMap["Title"].(string)
thing.Text.Body = subthingMap["Body"].(string)
I’m not sure why that code uses casts and generic types over type-safe unmarshalling directly from JSON to struct types (abstraction I guess). Using the json packages unmarshalling to struct types would go something like
type ThingText struct {
Title string
Body string
}
type Thing struct {
Id string
Text ThingText
}
…
decoder := json.NewDecoder(body)
var thingobj Thing
for {
if err := decoder.Decode(&thingobj); err == io.EOF {
break
} else if err != nil {
log.Fatal(err)
}
fmt.Println(thingobj)
}
where body is a io.Reader - in simple/most cases from http.Response.Body.