Replace characters in go serialization by using custom MarshalJSON method - json

As far as I saw, I just did a customized MarshalJSON method in order to replace these characters:\u003c and \u003e: https://go.dev/play/p/xJ-qcMN9QXl
In the example above, i marshaled the similar struct by sending to marshal from an aux struct that contains the same fields and last step is to replace the fields that I actually need and the return.
As you can see in the print placed before returning from MarshalJSON method, the special characters were replaced, but after calling the json.Marshal func, the special characters remains the same.
Something I'm missing here but cannot figure it out. Appreciate your help.
Thankies :)

In the Marshal documentation of the json package https://pkg.go.dev/encoding/json#Marshal you will find the following paragraph:
String values encode as JSON strings coerced to valid UTF-8, replacing invalid bytes with the Unicode replacement rune. So that the JSON will be safe to embed inside HTML tags, the string is encoded using HTMLEscape, which replaces "<", ">", "&", U+2028, and U+2029 are escaped to "\u003c","\u003e", "\u0026", "\u2028", and "\u2029". This replacement can be disabled when using an Encoder, by calling SetEscapeHTML(false).
So try it using a Encoder, example:
package main
import (
"bytes"
"encoding/json"
"fmt"
)
type Foo struct {
Name string
Surname string
Likes map[string]interface{}
Hates map[string]interface{}
newGuy bool //rpcclonable
}
func main() {
foo := &Foo{
Name: "George",
Surname: "Denkin",
Likes: map[string]interface{}{
"Sports": "volleyball",
"Message": "<Geroge> play volleyball <usually>",
},
}
buf := &bytes.Buffer{} // or &strings.Builder{} as from the example of #mkopriva
enc := json.NewEncoder(buf)
enc.SetEscapeHTML(false)
err := enc.Encode(foo)
if err != nil {
return
}
fmt.Println(buf.String())
}

Related

Preserve trailing zero when using json.Unmarshal()

This is not a duplicate of Stop json.Marshal() from stripping trailing zero from floating point number because I wish to unmarshal (i.e. parse the JSON-encoded data).
The trailing .0 is stripped out in the code below: https://play.golang.org/p/jrQ3OSvoTt5.
package main
import (
"encoding/json"
"fmt"
)
func main() {
var result map[string]interface{}
json.Unmarshal([]byte(`{"foo": 1.0}`), &result)
// map[foo:1]
fmt.Println(result)
}
The structure of the JSON is only known at runtime, so I must parse it into map[string]interface{}.
While I appreciate that 1 and 1.0 are numerically equivalent, I need to be able to distinguish between the two.
The default unmarshaler uses float64. You have to use a Decoder with UseNumber, then the numbers will be unmarshaled as json.Number and you can process them.
func main() {
var result map[string]interface{}
dec := json.NewDecoder(strings.NewReader(`{"foo": 1.0}`))
dec.UseNumber()
dec.Decode(&result)
// map[foo:1]
fmt.Println(result)
fmt.Println(result["foo"].(json.Number).String())
fmt.Println(result["foo"].(json.Number).Float64())
}

How to enforce float in decimal format when encoding to JSON in Go

I have a big.float which I'm encoding into JSON . However the JSON always end up showing the float in scientific notation rater than decimal notation. I can fix this by changing the JSON to be a string rather than a number and using float.Text('f'), however I would really prefer to keep the type as a number.
I was a taking a look at float.Format but I don't believe this is suitable.
A really condensed gist of what I'm doing is below. I do a lot more modification of the value of supply before encoding it to json.
type TokenSupply struct {
TotalSupply *big.Float `json:"totalSupply, omitempty"`
}
supply := Float.NewFloat(1000000)
json.NewEncoder(w).Encode(TokenSupply{supply})
This returns:
{"totalSupply":"1e+06"}
big.Float is marshaled to string when converted to a JSON type
https://golang.org/pkg/encoding/json/#Marshal
Marshal traverses the value v recursively. If an encountered value implements the Marshaler interface and is not a nil pointer, Marshal calls its MarshalJSON method to produce JSON. If no MarshalJSON method is present but the value implements encoding.TextMarshaler instead, Marshal calls its MarshalText method and encodes the result as a JSON string. The nil pointer exception is not strictly necessary but mimics a similar, necessary exception in the behavior of UnmarshalJSON.
https://golang.org/pkg/math/big/#Float.MarshalText
func (x *Float) MarshalText() (text []byte, err error)
What can you do about it?
since your float may be more than 64 bits it won't play well with other languages that have to read the JSON value as a number. I'd suggest you keep the number as a string.
Caveats about encoding numbers that don't fit into 64 bits aside, here is how you could marshal a big.Float as a JSON number by wrapping it in a custom type that implements json.Marshaler. The key is that you can implement theMarshalJSON method anyway you like, as long as it emits valid JSON:
package main
import (
"encoding/json"
"fmt"
"math/big"
)
type BigFloatNumberJSON struct{ *big.Float }
func (bfn BigFloatNumberJSON) MarshalJSON() ([]byte, error) {
// Use big.Float.String() or any other string converter
// that emits a valid JSON number here...
return []byte(bfn.String()), nil
}
func main() {
totalSupply := new(big.Float).SetFloat64(1000000)
obj := map[string]interface{}{
"totalSupply": BigFloatNumberJSON{totalSupply},
}
bytes, err := json.Marshal(&obj)
if err != nil {
panic(err)
}
fmt.Println(string(bytes))
// => {"totalSupply":1000000}
}

Json Parsing in Golang

I am trying to parse a json from a third party software. It returns a json like this
{
"top1/dir1": "10",
"top1/dir2": "20",
"top1/dir3": "30",
"top2/diff_val1": "40"
}
JSONLint says this is a valid json. But I could not figure how I can parse this with golang.
The code I used to parse the json file above (to be clear I took the code from another stackoverflow post).
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
)
type mytype []map[string]string
func main() {
var data mytype
file, err := ioutil.ReadFile("t1.json")
if err != nil {
log.Fatal(err)
}
err = json.Unmarshal(file, &data)
if err != nil {
log.Fatal(err)
}
fmt.Println(data)
}
When I do a go run main.go, I get the below error
$ go run main.go
2016/06/19 22:53:57 json: cannot unmarshal object into Go value of type main.mytype
exit status 1
I did try to parse this format with another library - "github.com/Jeffail/gabs", but was unsuccessful. Since this is a valid json, I am pretty sure this can be parsed, but I am not sure how.
There is a Go package with methods for decoding JSON strings.
https://golang.org/pkg/encoding/json/#Unmarshal
Here is an example of usage:
package main
import (
"encoding/json"
"fmt"
)
func main() {
var jsonBlob = []byte(`[
{"Name": "Platypus", "Order": "Monotremata"},
{"Name": "Quoll", "Order": "Dasyuromorphia"}
]`)
type Animal struct {
Name string
Order string
}
var animals []Animal
err := json.Unmarshal(jsonBlob, &animals)
if err != nil {
fmt.Println("error:", err)
}
fmt.Printf("%+v", animals)
}
EDIT: As pointed out by Malik, the type of the value whose pointer you pass is wrong. In this case, your type should be map[string]interface{} (preferably, because a JSON field might not store a string) or map[string]string instead of []map[string]string. The brackets at the beginning are wrong: such would be an array of JSON objects.
It's just that you have a small typo in your program. You've declared mytype as a slice of maps, rather than just a map.
Just change:
type mytype []map[string]string
To:
type mytype map[string]string
See https://play.golang.org/p/pZQl8jV5TC for an example.
Jonathan's answer provides a good example of decoding JSON, and links the relevant package. You don't provide much detail on what exactly is going wrong with your parsing, but if I had to take a guess I would say you're probably not creating an appropriate struct to contain the JSON once it is unmarshalled. Because Go is statically typed, it expects data to adhere to explicitly defined formats.
If you don't want to go to the trouble of defining structs, you could just use an empty interface object, which is sort of a catch all in Go. Simply declare a variable with the type []interface{}, and then pass it into the JSON unmarshal function. Hope this helps!

Why are string and []bytes treated differently when unmarshaling JSON?

My understanding from reading the documentation was that string is essentially an immutable []byte and that one can easily convert between the two.
However when unmarshaling from JSON this doesn't seem to be true. Take the following example program:
package main
import (
"encoding/json"
"fmt"
)
type STHRaw struct {
Hash []byte `json:"hash"`
}
type STHString struct {
Hash string `json:"hash"`
}
func main() {
bytes := []byte(`{"hash": "nuyHN9wx4lZL2L3Ir3dhZpmggTQEIHEZcC3DUNCtQsk="}`)
stringHead := new(STHString)
if err := json.Unmarshal(bytes, &stringHead); err != nil {
return
}
rawHead := new(STHRaw)
if err := json.Unmarshal(bytes, &rawHead); err != nil {
return
}
fmt.Printf("String:\t\t%x\n", stringHead.Hash)
fmt.Printf("Raw:\t\t%x\n", rawHead.Hash)
fmt.Printf("Raw to string:\t%x\n", string(rawHead.Hash[:]))
}
This gives the following output:
String: 6e7579484e397778346c5a4c324c3349723364685a706d67675451454948455a63433344554e437451736b3d
Raw: 9eec8737dc31e2564bd8bdc8af77616699a0813404207119702dc350d0ad42c9
Raw to string: 9eec8737dc31e2564bd8bdc8af77616699a0813404207119702dc350d0ad42c9
Instead I would have expected to receive the same value each time.
What is the difference?
The designers of the encoding/json package made the decision that applications must provide valid UTF-8 text in string values and that applications can put arbitrary byte sequences in []byte values. The package base64 encodes []byte values to ensure that the resulting string is valid UTF-8.
The encoding of []byte values is described in the Marshal function documentation.
This decision was not dictated by the design of the Go language. The string type can contain arbitrary byte sequences. The []byte type can contain valid UTF-8 text.
The designers could have used a flag in the field tag to indicate that a string or []byte value should be encoded and which encoder to use, but that's not what they did.

How to unmarshal an escaped JSON string

I am using Sockjs with Go, but when the JavaScript client send json to the server it escapes it, and send's it as a []byte. I'm trying to figure out how to parse the json, so that i can read the data. but I get this error.
json: cannot unmarshal string into Go value of type main.Msg
How can I fix this? html.UnescapeString() has no effect.
val, err := session.ReadMessage()
if err != nil {
break
}
var msg Msg
err = json.Unmarshal(val, &msg)
fmt.Printf("%v", val)
fmt.Printf("%v", err)
type Msg struct {
Channel string
Name string
Msg string
}
//Output
"{\"channel\":\"buu\",\"name\":\"john\", \"msg\":\"doe\"}"
json: cannot unmarshal string into Go value of type main.Msg
You might want to use strconv.Unquote on your JSON string first :)
Here's an example, kindly provided by #gregghz:
package main
import (
"encoding/json"
"fmt"
"strconv"
)
type Msg struct {
Channel string
Name string
Msg string
}
func main() {
var msg Msg
var val []byte = []byte(`"{\"channel\":\"buu\",\"name\":\"john\", \"msg\":\"doe\"}"`)
s, _ := strconv.Unquote(string(val))
err := json.Unmarshal([]byte(s), &msg)
fmt.Println(s)
fmt.Println(err)
fmt.Println(msg.Channel, msg.Name, msg.Msg)
}
You need to fix this in the code that is generating the JSON.
When it turns out formatted like that, it is being JSON encoded twice. Fix that code that is doing the generating so that it only happens once.
Here's some JavaScript that shows what's going on.
// Start with an object
var object = {"channel":"buu","name":"john", "msg":"doe"};
// serialize once
var json = JSON.stringify(object); // {"channel":"buu","name":"john","msg":"doe"}
// serialize twice
json = JSON.stringify(json); // "{\"channel\":\"buu\",\"name\":\"john\",\"msg\":\"doe\"}"
Sometimes, strconv.Unquote doesn't work.
Heres an example shows the problem and my solution.
(The playground link: https://play.golang.org/p/Ap0cdBgiA05)
Thanks for #Crazy Train's "encodes twice" idea, I just decoded it twice ...
package main
import (
"encoding/json"
"fmt"
"strconv"
)
type Wrapper struct {
Data string
}
type Msg struct {
Photo string
}
func main() {
var wrapper Wrapper
var original = `"{\"photo\":\"https:\/\/www.abc.net\/v\/t1.0-1\/p320x320\/123.jpg\"}"`
_, err := strconv.Unquote(original)
fmt.Println(err)
var val []byte = []byte("{\"data\":"+original+"}")
fmt.Println(string(val))
err = json.Unmarshal([]byte(val), &wrapper)
fmt.Println(wrapper.Data)
var msg Msg
err = json.Unmarshal([]byte(wrapper.Data), &msg)
fmt.Println(msg.Photo)
}
As Crazy Train pointed out, it appears that your input is doubly escaped, thus causing the issue. One way to fix this is to make sure that the function session.ReadMessasge() returns proper output that is escaped appropriately. However, if that's not possible, you can always do what x3ro suggested and use the golang function strconv.Unquote.
Here's a playground example of it in action:
http://play.golang.org/p/GTishI0rwe
The data shown in the problem is stringified for some purposes, in some cases you can even have \n in your string representing break of line in your json.
Let's understand the easiest way to unmarshal/deserialize this kind of data using the following example:
Next line shows the data you get from your sources and want to derserialize
stringifiedData := "{\r\n \"a\": \"b\",\r\n \"c\": \"d\"\r\n}"
Now, remove all new lines first
stringifiedData = strings.ReplaceAll(stringifiedData, "\n", "")
Then remove all the extra quotes that are present in your string
stringifiedData = strings.ReplaceAll(stringifiedData, "\\"", "\"")
Let's now convert the string into a byte array
dataInBytes := []byte(stringifiedData)
Before doing unmarshal, let's define structure of our data
jsonObject := &struct{
A string `json:"a"`
C string `json:"c"`
}
Now, you can dersialize your values into jsonObject
json.Unmarshal(dataInBytes, jsonObject)}
You got into infamous escaped string pitfall from JavaScript. Quite often people face (as I did) the same issue in Go, when serializing JSON string with json.Marshal, e.g.:
in := `{"firstName":"John","lastName":"Dow"}`
bytes, err := json.Marshal(in)
json.Marshal escapes double quotes, producing the same issue when you try to unmarshal bytes into struct.
If you faced the issue in Go, have a look at How To Correctly Serialize JSON String In Golang post which describes the issue in details with solution to it.