Equivalent of PHP json_decode in Golang
Thanks to the following references :
PHP Manual : json_decode — Decodes a JSON string
php2golang : GoLang alternatives for PHP functions
A Guide to JSON in Golang
To convert JSON data (strings or bytes) into Go types like structs, arrays, and slices, as well as unstructured data like maps and empty interfaces.
..
If you don’t know the structure of your JSON properties beforehand, you cannot use structs to unmarshal your data ..
Looking for an equivalent of PHP json_decode in Golang I go immediately to : Decoding JSON to Maps - Unstructured Data
With the code below :
package main
import (
"fmt"
"os"
"encoding/json"
)
func main() {
json := `{"key_0":{"key_1":"value_1","key_2":"value_2"},"key_3":"value_3"}`
json_decoded, err := Json_decode(json)
fmt.Println(json_decoded)
fmt.Println(json_decoded["key_0"])
return
}
func Json_decode(data string) (map[string]interface{}, error) {
var dat map[string]interface{}
err := json.Unmarshal([]byte(data), &dat)
return dat, err
}
We got this output :
map[key_0:map[key_1:value_1 key_2:value_2] key_3:value_3]
map[key_1:value_1 key_2:value_2]
If we add this line :
fmt.Println(json_decoded["key_0"]["key_1"])
We got this error :
invalid operation: json_decoded["key_0"]["key_1"] (type interface {} does not support indexing)
To get value_1 we have to add these lines :
key_0 := json_decoded["key_0"].(map[string]interface{})
fmt.Println(key_0["key_1"])
But we don’t know the structure of the JSON
Lets say the JSON is a PHP array encoded with json_encode
The key can either be an int or a string and the value can be of any type.
So we may have to add these lines :
key_0 := json_decoded["key_0"].([]interface{})
fmt.Println(key_0[O])
Looking for an equivalent of PHP json_decode in Golang what do you suggest ?
encoding/json requires you to know your data structures,
you can use map[string]interface{} but it will be very slow and hard to manage.
package main
import (
//"encoding/json"
"jsonparser"
)
Alternative JSON parser for Go (10x times faster standard library)
Related
In powershell, if I make a REST call and receive any kind of json response, I can easily $json | ConvertFrom-Json into a proper object so I can make modifications, render specific values, whatever.
It seems like in Go I have to either define a struct, or "dynamically" convert using a map[string]interface{}.
The issue with a struct is that I am writing a rest handler for a platform that, depending on the endpoint, serves wildly different JSON responses, like most REST APIs. I don't want to define a struct for all of the dozens of possible responses.
The problem with map[string]interface{} is that it pollutes the data by generating a string with a bunch of ridiculous 'map' prefixes and unwanted [brackets].
ala: [map[current_user_role:admin id:1]]
Is there a way to convert a JSON response like:
{
"current_user_role": "admin",
"id": 1
}
To return a basic:
current_user_role: admin
id: 1
... WITHOUT defining a struct?
Your approach of using a map is right if you don't wish to specify the structure of the data you're receiving. You don't like how it is output from fmt.Println, but I suspect you're confusing the output format with the data representation. Printing them out in the format you find acceptable takes a couple of lines of code, and is not as convenient as in python or powershell, which you may find annoying.
Here's a working example (playground link):
package main
import (
"encoding/json"
"fmt"
"log"
)
var data = []byte(`{
"current_user_role": "admin",
"id": 1
}`)
func main() {
var a map[string]interface{}
if err := json.Unmarshal(data, &a); err != nil {
log.Fatal(err)
}
for k, v := range a {
fmt.Printf("%s: %v\n", k, v)
}
}
I have a JSON file stored on the local machine. I need to read it in a variable and loop through it to fetch the JSON object values. If I use the Marshal command after reading the file using the ioutil.Readfile method, it gives some numbers as an output. These are my few failed attempts,
Attempt 1:
plan, _ := ioutil.ReadFile(filename) // filename is the JSON file to read
var data interface{}
err := json.Unmarshal(plan, data)
if err != nil {
log.Error("Cannot unmarshal the json ", err)
}
fmt.Println(data)
It gave me following error,
time="2016-12-13T22:13:05-08:00" level=error msg="Cannot unmarshal the json json: Unmarshal(nil)"
<nil>
Attempt 2: I tried to store the JSON values in a struct and then using MarshalIndent
generatePlan, _ := json.MarshalIndent(plan, "", " ") // plan is a pointer to a struct
fmt.Println(string(generatePlan))
It give me the output as string. But if I cast the output to string then I won't be able to loop it as JSON object.
How can we read a JSON file as JSON object in golang? Is it possible to do that?
Any help is appreciated. Thanks in advance!
The value to be populated by json.Unmarshal needs to be a pointer.
From GoDoc :
Unmarshal parses the JSON-encoded data and stores the result in the value pointed to by v.
So you need to do the following :
plan, _ := ioutil.ReadFile(filename)
var data interface{}
err := json.Unmarshal(plan, &data)
Your error (Unmarshal(nil)) indicates that there was some problem in reading the file , please check the error returned by ioutil.ReadFile
Also please note that when using an empty interface in unmarshal, you would need to use type assertion to get the underlying values as go primitive types.
To unmarshal JSON into an interface value, Unmarshal stores one of
these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
It is always a much better approach to use a concrete structure to populate your json using Unmarshal.
If you're looking at this in 2022, the ioutil package has been deprecated. You can still use it but you'll get annoying errors.
Instead you can use the os package.
someStruct := SomeStruct{} // Read errors caught by unmarshal
fileBytes, _ := os.ReadFile(filename)
err := json.Unmarshal(fileBytes, spec)
Note, I'm specifically ignoring the error from os.ReadFile since it will also cause an error in json.Unmarshal for the sake of the example.
I'm new in golang development and have some question regarding something related to this question.
As a learning exercise, I'm trying to create a simple library to handle json based configuration file. As a configuration file to be used for more then one app, it should be able to handle different parameters. Then I have created a type struct Configuration that has the filename and a data interface. Each app will have a struct based on its configuration needs.
In the code bellow, I put all together (lib and "main code") and the "TestData struct" is the "app parameters".
If it doesn't exists, it will set a default values and create the file, and it is working. But when I try to read the file. I try to decode the json and put it back into the data interface. But it is giving me an error and I couldn't figure out how to solve this. Can someone help on this?
[updated] I didn't put the targeted code before, because I though that it would be easier to read in in all as a single program. Bellow is the 'targeted code' for better view of the issue.
As I will not be able to use the TestData struct inside the library, since it will change from program to program, the only way to handle this was using interface. Is there a better way?
library config
package config
import (
"encoding/json"
"fmt"
"os"
)
// Base configuration struct
type Configuration struct {
Filename string
Data interface{}
}
func (c *Configuration) Create(cData *Configuration) bool {
cFile, err := os.Open(cData.Filename)
defer cFile.Close()
if err == nil {
fmt.Println("Error(1) trying to create a configuration file. File '", cData.Filename, "' may already exist...")
return false
}
cFile, err = os.Create(cData.Filename)
if err != nil {
fmt.Println("Error(2) trying to create a configuration file. File '", cData.Filename, "' may already exist...")
return false
}
buffer, _ := json.MarshalIndent(cData.Data, "", "")
cFile.Write(buffer)
return true
}
func (c *Configuration) Read(cData *Configuration) bool {
cFile, err := os.Open(cData.Filename)
defer cFile.Close()
if err != nil {
fmt.Println("Error(1) trying to read a configuration file. File '", cData.Filename, "' may not already exist...")
return false
}
jConfig := json.NewDecoder(cFile)
jerr := jConfig.Decode(&cData.Data)
if jerr != nil {
panic(jerr)
}
return true
}
program using library config
package main
import (
"fmt"
"./config"
)
// struct basic para configuração
type TestData struct {
URL string
Port string
}
func main() {
var Config config.Configuration
Config.Filename = "config.json"
if !Config.Read(&Config) {
Config.Data = TestData{"http", "8080"}
Config.Create(&Config)
}
fmt.Println(Config.Data)
TestData1 := &TestData{}
TestData1 = Config.Data.(*TestData) // error, why?
fmt.Println(TestData1.URL)
}
NEW UPDATE:
I have made some changes after JimB comment about I'm not clear about some concepts and I tried to review it. Sure many things aren't clear for me yet unfortunately. The "big" understanding I believe I got, but what mess my mind up is the "ins" and "outs" of values and formats and pointers, mainly when it goes to other libraries. I'm not able yet to follow the "full path" of it.
Yet, I believe I had some improvement on my code.
I think that I have corrected some points, but still have some big questions:
I stopped sending "Configuration" as a parameter as all "data" were already there as they are "thenselfs" in the instance. Right?
Why do I have use reference in the line 58 (Config.Data = &TestData{})
Why to I have to use pointer in the line 64 (tmp := Config.Data.(*TestData)
Why I CANNOT use reference in line 69 (Config.Data = tmp)
Thanks
The reason you are running into an error is because you are trying to decode into an interface{} type. When dealing with JSON objects, they are decoded by the encoding/json package into map[string]interface{} types by default. This is causing the type assertion to fail since the memory structure for a map[string]interface{} is much different than that of a struct.
The better way to do this is to make your TestData struct the expected data format for your Configuration struct:
// Base configuration struct
type Configuration struct {
Filename string
Data *TestData
}
Then when Decoding the file data, the package will unmarshal the data into the fields that match the closest with the data it finds.
If you need more control over the data unmarshaling process, you can dictate which JSON fields get decoded into which struct members by using struct tags. You can read more about the json struct tags available here: https://golang.org/pkg/encoding/json/#Marshal
You are trying to assert that Config.Data is of type *TestData, but you're assigning it to TestData{"http", "8080"} above. You can take the address of a composite literal to create a pointer:
Config.Data = &TestData{"http", "8080"}
If your config already exsits, your Read method is going to fill in the Data field with the a default json data type, probably a map[string]interface{}. If you assign a pointer of the correct type to Data first, it will decode into the expected type.
Config.Data = &TestData{}
Ans since Data is an interface{}, you do not want to ever use a pointer to that value, so don't use the & operator when marshaling and unmarshaling.
I have a question regarding the golang unmarshalling . I was trying to unmarshal Json array but it is giving nil result for one decoding while it is successful in the other. I don't understand the reason behind it. Is it a mistake in the code or expected?
package main
import "fmt"
import "encoding/json"
type PublicKey struct {
Id int
Key string
}
type KeysResponse struct {
Collection []PublicKey
}
func main() {
keysBody := []byte(`[{"id": 1,"key": "-"},{"id": 2,"key": "-"},{"id": 3,"key": "-"}]`)
keys := make([]PublicKey,0)
json.Unmarshal(keysBody, &keys)//This works
fmt.Printf("%#v\n", keys)
response := KeysResponse{}
json.Unmarshal(keysBody, &response)//This doesn't work
fmt.Printf("%#v\n", response)
}
http://play.golang.org/p/L9xDG26M8-
That's not expected to work. What you have in the json is an array of type PublicKey. The KeysResponse type would be used for json looking like this;
{
"Collection": [{"id": 1,"key": "-"},{"id": 2,"key": "-"},{"id": 3,"key": "-"}]
}
Which is not what you have. If you want the data to be stored in that type I'd recommend the following; response := KeysResponse{keys} on the line after you unmarshal into keys.
To elaborate on that distinction. In the working case the json is just an array with objects inside of it. The json I wrote above is an object which has a single property named Collection which is of type array and the objects in the array are represented by the PublicKey type (objects with an int called id and a string called key). When working on code to unmarshal json, it's helpful to describe the structure using plain English like this, it tells you precisely what types/structures you need in Go to hold the data.
I have JSON from an API that I want to save to MongoDB using the mgo package. My JSON looks like this:
{
"something": "value"
"collection": [
{ "obj1": "value" }
// ... (Variable number of objects here)
]
}
To save this data I've created a new type in my Go application that looks like this:
type MyData struct {
Something string
Collection []string // This actually contains more than strings but I'll be happy if I can just get strings saved
}
mongoSess, err := mgo.Dial("localhost:27017")
if err != nil {
panic(err)
}
defer mongoSess.Close()
c := mongoSess.DB("mydatabase").C("mycollection")
insertErr := c.Insert(&MyData{something, collection})
This code works but the problem is that it isn't saving anything in my collection field which should be an array of JSON objects. Nothing at all. I get the keys in my database and they are the right type but they have no data. Here's what the Mongo output is:
{ "_id" : ObjectId("5520c535a236d8a9a215d096"), "something" : "value", "collection" : [ ] }
Can anyone spot what it is I'm doing wrong? I'm obviously new to Go and having trouble with types.
Solution
The answers here really helped me a lot. I'm at fault for not properly explaining things as the answers sent me on the right track but didn't solve the issue directly. Here's what the actual solution is.
package main
import (
"encoding/json"
"github.com/bitly/go-simplejson"
"gopkg.in/mgo.v2"
//"gopkg.in/mgo.v2/bson"
// Other packages are used as well
)
type MyData struct {
Something int
Collection []interface{}
}
func main() {
// I'm using SimpleJson for parsing JSON
collection, cerr := jsonFromExternalApi.Get("collection").Array()
if cerr != nil {
logger.Debug.Fatalln(cerr)
}
// Save response to Mongo (leaving out all the connection code)
c := mongoSess.DB("mydb").C("mycollection")
insertErr := c.Insert(&Producer{npn, licenses })
}
The issue was that SimpleJSON was returning the array of objects from my API call as a []interface{}. I was not aware I could simply declare part of a struct to be an interface so instead of just correcting what Go's compiler was telling me was wrong I was making it way harder than it should have been.
Coming from loosely typed scripting languages, stuff like this really trips me up and sometimes its hard to see the benefit but hopefully this helps someone out one day.
Looks you have the wrong data structure.
insertErr := c.Insert(&MyData{something, collection})
something => string and collection => slice
You code should be like this:
insertErr := c.Insert(&MyData{"something", []string{"obj1", "value"}})
Here is the working code.
[{Something:something Collection:[obj1 value]} {Something:something Collection:[obj1 value]} {Something:something Collection:[obj1 value]} {Something:something Collection:[obj1 value]}]
For further reference, mgo has great documentation and you can find sample code and details about running mongodb queries here.
Here you are not defining proper type to Collection in your MyData struct.
Collection []string //This is what you are defining.
But from Json you are not getting string array,you are getting map[string]interface{}
This is the reason you are not filling Mydata struct properly
Correct MyData struct will be
type MyData struct {
Something string
Collection map[string]string // This actually contains more than strings but I'll be happy if I can just get strings saved
}