Can I modify json.RawMessage? - json

This is a follow up to JSON sometimes array sometimes object
In the original question, I asked how to deal with: "I am consuming a json API that might return a string for a variable or might return an array for a variable"
I have a solution but I was wondering, is there a way to modify json.RawMessage?
Rather then if/then looking at the RawMessage for a [ or { char to determine if the object is an array or an string, what if I always took a RawMessage variable string and turned it into an array?
This way, I don't have to code all of the accessors for BOTH strings AND arrays. I could simply deal with arrays.
So my question is: Is there a way to modify the json.RawMessage?
eg:
Turn this:
{
"net": {
"comment": {
"line":
{
"$": "All abuse issues will only be responded to by the Abuse",
"#number": "0"
}
}
}
Into This:
{
"net": {
"comment": {
"line": [
{
"$": "All abuse issues will only be responded to by the Abuse",
"#number": "0"
}
]
}
}
So, that way, when I unmarshal into my struct, there is only 1 type of comment.line, Just line[] vs line[] AND line.
Thanks in advance.
I am a golang neophyte and I'm just getting my head wrapped around the difficulties of unmarshaling into an strongly typed language.

Yes, you can edit json.RawMessage type as it is simply an alias for []byte.
That said, you don't need to keep raw type, just make your own implementation of the array type and in your custom Unmarshal function, make scalars an array.
Here's an example (on Play).
All we do here is see if the bytes for MagicArray starts with '[', if so, just unmarshal as normal. Otherwise, Unmarshal and append to slice.
you will have to implement custom array for each type you want to work like this, but that's still probably better than trying to safely manipulate the json binary to try coerce the scalars into arrays.
Another side benefit to this approach is you can it with the streaming json decoder, as in json.NewDecoder(reader).Decode(&obj)
package main
import "encoding/json"
import "log"
type MagicArray []interface{}
func (ma *MagicArray) UnmarshalJSON(b []byte) error {
if b[0] == '[' {
return json.Unmarshal(b, (*[]interface{})(ma))
} else {
var obj interface{}
if err := json.Unmarshal(b, &obj); err != nil {
return err
}
*ma = append(*ma, obj)
}
return nil
}
func main() {
myStruct := struct {
A MagicArray
B MagicArray
}{}
err := json.Unmarshal(jsonToDecode, &myStruct)
if err != nil {
log.Println("Fail:", err)
} else {
log.Println(myStruct)
}
}
var jsonToDecode = []byte(`
{
"A": "I am not an array",
"B":["I am an array"]
}
`)

I think that David has a good (better) answer, but to answer your question directly: yes, you can modify a json.RawMessage if you're careful. it's declared as type json.RawMessage []byte, meaning it's just another name for []byte under the hood. You can cast it to []byte or string, modify it, and cast it back.
Doing string options on serialized data isn't the kind of thing you should do without thinking about the consequences, but in the case of wrapping [ and ] around a JSON object, it's not too hard to prove that it should be safe. If msg is a json.RawMessage representing an object, then
json.RawMessage("[" + string(msg) + "]")
is what I would consider a readable approach to making a RawMessage representing an array containing that object :)

Related

Go: is it possible to marshal a cycled json

I'm coding Go to create a Circular Singly Linked List: https://www.geeksforgeeks.org/circular-singly-linked-list-insertion/
Here is my code:
type Node struct {
Name string `json:"name"`
Next *Node `json:"next"`
}
func main() {
n1 := Node{Name: "111", Next: nil}
n2 := Node{Name: "222", Next: &n1}
n3 := Node{Name: "333", Next: &n2}
n1.Next = &n3
}
However, when I try to marshal, I get an error: json: unsupported value: encountered a cycle via *main.Node
res, err := json.Marshal(n1)
if err != nil {
fmt.Println(err)
} else {
fmt.Println(res)
}
Well, I understand why I get this error. So I want to know if there is some method to allow me to convert the struct of Node to JSON.
From the documentation of encoding/json.Marshal
JSON cannot represent cyclic data structures and Marshal does not handle them. Passing cyclic structures to Marshal will result in an error.
The main part is "JSON cannot represent cyclic data structures" so the answer is a clear No: It is not possible to have JSON which represents your value. You must redesign.

Unmarshal Inconsistent JSON in Go

I'm working with JSON that returns three different object types 'items','categories' and 'modifiers'. An example of the JSON can be viewed here. I created models for the three types of objects. But when I unmarshal I have selected one of the types to unmarshal the entire JSON to.(I know this cant be the correct way...) I then try to parse out the different items depending on what their type is identified as in the json field 'Type' and then append that object to a slice of the proper type. I am having errors because I don't know how to unmarshal JSON that has different types in it that have different fields.
What is the proper method to unmarshal JSON that contains different objects, each with their own respective fields?
Is the solution to create a "super model" which contains all possible fields and then unmarshal to that?
I'm still fairly new and would appreciate any advice. Thanks!
If you implement json.Unmarshaler, you can define a struct that parses each item type into it's relevant struct.
Example:
// Dynamic represents an item of any type.
type Dynamic struct {
Value interface{}
}
// UnmarshalJSON is called by the json package when we ask it to
// parse something into Dynamic.
func (d *Dynamic) UnmarshalJSON(data []byte) error {
// Parse only the "type" field first.
var meta struct {
Type string
}
if err := json.Unmarshal(data, &meta); err != nil {
return err
}
// Determine which struct to unmarshal into according to "type".
switch meta.Type {
case "product":
d.Value = &Product{}
case "post":
d.Value = &Post{}
default:
return fmt.Errorf("%q is an invalid item type", meta.Type)
}
return json.Unmarshal(data, d.Value)
}
// Product and Post are structs representing two different item types.
type Product struct {
Name string
Price int
}
type Post struct {
Title string
Content string
}
Usage:
func main() {
// Parse a JSON item into Dynamic.
input := `{
"type": "product",
"name": "iPhone",
"price": 1000
}`
var dynamic Dynamic
if err := json.Unmarshal([]byte(input), &dynamic); err != nil {
log.Fatal(err)
}
// Type switch on dynamic.Value to get the parsed struct.
// See https://tour.golang.org/methods/16
switch dynamic.Value.(type) {
case *Product:
log.Println("got a product:", dynamic.Value)
case *Post:
log.Println("got a product:", dynamic.Value)
}
}
Output:
2009/11/10 23:00:00 got a product: &{iPhone 1000}
Try it in the Go Playground.
Tip: if you have a list of dynamic objects, just parse into a slice of Dynamic:
var items []Dynamic
json.Unmarshal(`[{...}, {...}]`, &items)
Example output:
[&{iPhone 1000} &{A Good Post Lorem ipsum...}]
I think https://github.com/mitchellh/mapstructure also fits into your use case.

Unmarshaling nested custom same-type JSON in Go

Given the following JSON
{
"some": "value"
"nested": {
"some": "diffvalue",
"nested": {
"some": "innervalue"
}
}
}
which roughly translates to this struct:
type Envelope struct {
some string `json:"some"`
nested InnerEnvelope `json:"nested"`
}
where InnerEnvelope is: type InnerEnvelope map[string]interface{}
Running a simple json.Unmarshal([]byte value, &target) does not help here, because of the recursive type nature of the original JSON.
I do not know up front how deeply and under which keys the inner maps will exist, so I cannot declare the types upfront.
The idea is, that using map[string]interface{} as the type is not good enough, since I need the values in the InnerEnvelope to be somehow transformed & typed. Details are not important, but image, I need to cast every value inside the NestedEnvelope of a bool type as a string saying "true" or "false" as opposed of having an actual bool type.
I turned to UnmarshalJSON interface to solve this problem. I can easily do it at the top level like so:
func (m *Envelope) UnmarshalJSON(b []byte) error {
var stuff noBoolMap
if err := json.Unmarshal(b, &stuff); err != nil {
return err
}
for key, value := range stuff {
switch value.(type) {
case bool:
stuff[key] = strconv.FormatBool(value.(bool))
}
}
return nil
}
But since the inner json.Unmarshal will already have inner maps parsed as map[string]interface{}, I would need to yet-again to traverse the inner maps, cast them to appropriate type and perform my value transformations.
So my question is: In this case, what is the way this would be approached in Go, and preferably do it in a single-pass?
The expected result of the JSON example above would be:
Envelope {
some: string
nested: InnerEnvelope {
some: string {
nested: InnerEnvelope {
some: string
}
}
}
Given your json, you can do this:
type Envelope struct {
Some string `json:"some"`
Nested json.RawMessage `json:"nested"`
}
json.RawMessage is a rather hidden gem, and more people seem to go for the map[string]interface{}.
Using json.RawMessage will result in the nested json to be represented by this RawMessage, which you then can process again as a normal json (unmarshal it into Envelope).
This is more elegant than the map[string]interface{} approach.

UnmarshalJSON on struct containing interface list

I would like to UnmarshalJSON a struct containing an interface as follows:
type Filterer interface {
Filter(s string) error
}
type FieldFilter struct {
Key string
Val string
}
func (ff *FieldFilter) Filter(s string) error {
// Do something
}
type Test struct {
Name string
Filters []Filterer
}
My idea was to send a json like so:
{
"Name": "testing",
"Filters": [
{
"FieldFilter": {
"Key": "key",
"Val": "val"
}
}
]
}
However, when sending this json to the unmarshaler, the following exception returns: json: cannot unmarshal object into Go struct field Test.Filters of type Filterer
I understand the problem fully, but do not know how to approach this problem wisely. Looking for advice on an idiomatic way to solving this problem in go.
Following my own question, I researched how one could implement UnmarshalJSON for interface lists. Ultimately this led me to publish a blog post on how to do this properly. Basically there are 2 main solutions:
Parse the required JSON string into a map[string]*json.RawMessage and work your way from there.
Make an alias for the interface list and implement UnmarshalJSON for that alias. However, you'll still need to work with map[string]*json.RawMessage and some manual work. Nothing comes without a price!
I highly suggest taking the seconds approach. While these two solutions may result in the same amount of code lines, taking advantage of type aliasing and being less dependent on json.RawMessage types will make a more easy to manage code, especially when it is required to support multiple interfaces on the UnmarshalJSON implementation
To directly answer the question, start with making a type alias for the interface list:
type Filterers []Filterer
Now continue with implementing the decoding of the JSON:
func (f *Filterers) UnmarshalJSON(b []byte) error {
var FilterFields map[string]*json.RawMessage
if err := json.Unmarshal(b, &FilterFields); err != nil {
return err
}
for LFKey, LFValue := range FilterFields {
if LFKey == "FieldFilter" {
var MyFieldFilters []*json.RawMessage
if err := json.Unmarshal(*LFValue, &MyFieldFilters); err != nil {
return err
}
for _, MyFieldFilter := range MyFieldFilters {
var filter FieldFilter
if err := json.Unmarshal(*MyFieldFilter, &filter); err != nil {
return err
}
*f = append(*f, &filter)
}
}
}
return nil
}
A detailed explanation (with some examples and a full working code snippets) of the second approach is available on my own blog
There is no way for Unmarshal to know what type it should use. The only case where it can just "make something up" is if it's asked to unmarshal into an interface{}, in which case it will use the rules in the documentation. Since none of those types can be put into a []Filterer, it cannot unmarshal that field. If you want to unmarshal into a struct type, you must specify the field to be of that type.
You can always unmarshal into an intermediate struct or map type, and then do your own conversion from that into whatever types you want.

Json Parsing in Golang

I am trying to parse a json from a third party software. It returns a json like this
{
"top1/dir1": "10",
"top1/dir2": "20",
"top1/dir3": "30",
"top2/diff_val1": "40"
}
JSONLint says this is a valid json. But I could not figure how I can parse this with golang.
The code I used to parse the json file above (to be clear I took the code from another stackoverflow post).
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
)
type mytype []map[string]string
func main() {
var data mytype
file, err := ioutil.ReadFile("t1.json")
if err != nil {
log.Fatal(err)
}
err = json.Unmarshal(file, &data)
if err != nil {
log.Fatal(err)
}
fmt.Println(data)
}
When I do a go run main.go, I get the below error
$ go run main.go
2016/06/19 22:53:57 json: cannot unmarshal object into Go value of type main.mytype
exit status 1
I did try to parse this format with another library - "github.com/Jeffail/gabs", but was unsuccessful. Since this is a valid json, I am pretty sure this can be parsed, but I am not sure how.
There is a Go package with methods for decoding JSON strings.
https://golang.org/pkg/encoding/json/#Unmarshal
Here is an example of usage:
package main
import (
"encoding/json"
"fmt"
)
func main() {
var jsonBlob = []byte(`[
{"Name": "Platypus", "Order": "Monotremata"},
{"Name": "Quoll", "Order": "Dasyuromorphia"}
]`)
type Animal struct {
Name string
Order string
}
var animals []Animal
err := json.Unmarshal(jsonBlob, &animals)
if err != nil {
fmt.Println("error:", err)
}
fmt.Printf("%+v", animals)
}
EDIT: As pointed out by Malik, the type of the value whose pointer you pass is wrong. In this case, your type should be map[string]interface{} (preferably, because a JSON field might not store a string) or map[string]string instead of []map[string]string. The brackets at the beginning are wrong: such would be an array of JSON objects.
It's just that you have a small typo in your program. You've declared mytype as a slice of maps, rather than just a map.
Just change:
type mytype []map[string]string
To:
type mytype map[string]string
See https://play.golang.org/p/pZQl8jV5TC for an example.
Jonathan's answer provides a good example of decoding JSON, and links the relevant package. You don't provide much detail on what exactly is going wrong with your parsing, but if I had to take a guess I would say you're probably not creating an appropriate struct to contain the JSON once it is unmarshalled. Because Go is statically typed, it expects data to adhere to explicitly defined formats.
If you don't want to go to the trouble of defining structs, you could just use an empty interface object, which is sort of a catch all in Go. Simply declare a variable with the type []interface{}, and then pass it into the JSON unmarshal function. Hope this helps!