golang format data to JSON format in one pass - json

My original data format:
id=1, name=peter, age=12
I converted it to JSON string:
{"id" : "1", "name" : "peter", "age" : "12"}
I use the following golang statement to do the conversion:
Regex, err = regexp.Compile(`([^,\s]*)=([^,\s]*)`)
JSON := fmt.Sprintf("{%s}", Regex.ReplaceAllString(inp, `"$1" : "$2"`))
inp is the variable that holds the original data format.
However, now I get a new format:
id=1 name=peter age=12
and I also want to convert to JSON string using similar method that I used above, i.e., use regex to do a one pass formatting.
{"id"="1", "name"="peter", "age"="12"}
How can I achieve that?
UPDATE: One additional requirement. if the input format is
id=1, name=peter, age="12"
I need to get rid of the "" to be or escape \" so I can process in the next step. The double quote can appear at the beginning and the end of any value field.

There are two parts to the question: The easy part is serialising to JSON, Go has standard library methods for doing that. I would use that library rather than trying to encode the JSON myself.
The slightly trickier part of your question is parsing the input into a struct or map that can be easily serialised out, and making it flexible enough to accept different input formats.
I would do it with a general interface for converting text to a struct or map, and then implementing the interface for parsing each new input type.
Sample code: (You can run it here)
package main
import (
"encoding/json"
"errors"
"fmt"
"strings"
)
// parseFn describes the function for converting input into a map.
// This could be a struct or something else if the format is well known.
// In real code this would return map[string]interface{}, but for this
// demo I'm just using string
type parseFn func(string) (map[string]string, error)
// parseFormat1 is for fields separated by commas
func parseFormat1(in string) (map[string]string, error) {
data := map[string]string{}
fields := strings.Split(in, ",")
for _, field := range fields {
pair := strings.Split(field, "=")
if len(pair) != 2 {
return nil, errors.New("invalid input")
}
data[strings.Trim(pair[0], ` "`)] = strings.Trim(pair[1], ` "`)
}
return data, nil
}
// parseFormat2 is for lines with no commas
func parseFormat2(in string) (map[string]string, error) {
data := map[string]string{}
fields := strings.Split(in, " ")
for _, field := range fields {
pair := strings.Split(field, "=")
if len(pair) != 2 {
return nil, errors.New("invalid input")
}
data[strings.Trim(pair[0], ` "`)] = strings.Trim(pair[1], ` "`)
}
return data, nil
}
// nullFormat is what we fall back on when we just don't know
func nullFormat(in string) (map[string]string, error) { return nil, errors.New("invalid format") }
// classify just tries to guess the parser to use for the input
func classify(in string) parseFn {
switch {
case strings.Count(in, ", ") > 1:
return parseFormat1
case strings.Count(in, " ") > 1:
return parseFormat2
default:
return nullFormat
}
}
func main() {
testCases := []string{
`id=1, name=peter, age=12`,
`id=1, name=peter, age="12"`,
`id=1 name=peter age=12`,
`id=1;name=peter;age="12"`,
}
for ix, tc := range testCases {
pfn := classify(tc)
d, err := pfn(tc)
if err != nil {
fmt.Printf("\nerror parsing on line %d: %v\n", ix, err)
continue
}
b, err := json.Marshal(d)
if err != nil {
fmt.Printf("\nerror marshaling on line %d: %v\n", ix, err)
continue
}
fmt.Printf("\nSuccess on line %d:\n INPUT: %s\nOUTPUT: %s\n", ix, tc, string(b))
}
}

Related

Add a field to JSON ( struct + interface ) golang [duplicate]

This question already has answers here:
Adding Arbitrary fields to json output of an unknown struct
(2 answers)
Closed 1 year ago.
Here's the response interface :
type Response interface{}
It's satisfied by a struct like this :
type CheckResponse struct {
Status string `json:"status"`
}
I am getting out []Response as an output which is to be consumed elsewhere.
I want to add a Version string to this JSON, before it's being sent. I've tried using anonymous structs ( but in vain ) :
for _, d := range out {
outd := struct {
Resp Response `json:",inline"`
Version string `json:",inline"`
}{
Resp: d,
Version: "1.1",
}
data, _ := json.Marshal(outd)
log.Infof("response : %s", data)
}
The output I am getting is :
response : {"Resp":{"status":"UP"},"Version":"1.1"}
What I want is
{"status":"UP","Version":"1.1"}
i.e. one single flat JSON.
Assert your d to CheckResponse type and then define dynamic struct like this
outd := struct {
Resp string `json:"status,inline"`
Version string `json:",inline"`
}
This is the full code for this.
package main
import (
"encoding/json"
"fmt"
)
type Response interface {}
type CheckResponse struct {
Status string `json:"status"`
}
func main() {
out := []Response{
CheckResponse{Status: "UP"},
}
for _, d := range out {
res, ok := d.(CheckResponse)
if !ok {
continue
}
outd := struct {
Resp string `json:"status,inline"`
Version string `json:",inline"`
}{
Resp: res.Status,
Version: "1.1",
}
data, _ := json.Marshal(outd)
fmt.Printf("response : %s", data)
}
}
You can run here
inline tag is not supported by encoding/json and embedding interfaces will also not produce the result you want. You'll have to declare a type for the out value and have that type implement the json.Marshaler interface, you can then customize how its fields are marshaled, for example you could marshal the two fields Resp and Version separately and then "merge the result" into a single json object.
type VersionedResponse struct {
Resp Response
Version string
}
func (r VersionedResponse) MarshalJSON() ([]byte, error) {
out1, err := json.Marshal(r.Resp)
if err != nil {
return nil, err
}
out2, err := json.Marshal(struct{ Version string }{r.Version})
if err != nil {
return nil, err
}
// NOTE: if Resp can hold something that after marshaling
// produces something other than a json object, you'll have
// to be more careful about how you gonna merge the two outputs.
//
// For example if Resp is nil then out1 will be []byte(`null`)
// If Resp is a slice then out1 will be []byte(`[ ... ]`)
out1[len(out1)-1] = ',' // replace '}' with ','
out2 = out2[1:] // remove leading '{'
return append(out1, out2...), nil
}
https://play.golang.org/p/66jIYXGUtWJ
One way that will work for sure is simply use a map[string]interface{}, iterate over fields in Response via reflect or use a library like structs, update your map with response fields, append your version field to map, and then marshal.
Here is an example
package main
import (
"encoding/json"
"fmt"
"github.com/fatih/structs"
)
type Response interface{}
type CheckResponse struct {
Status string `json:"status"`
}
func main() {
resp := CheckResponse{Status: "success"}
m := structs.Map(resp)
m["Version"] = "0.1"
out, _ := json.Marshal(m)
fmt.Println(string(out))
}

How to check if a json matches a struct / struct fields

Is there an easy way to check if each field of myStruct was mapped by using json.Unmarshal(jsonData, &myStruct).
The only way I could image is to define each field of a struct as pointer, otherwise you will always get back an initialized struct.
So every jsonString that is an object (even an empty one {}) will return an initialized struct and you cannot tell if the json represented your struct.
The only solution I could think of is quite uncomfortable:
package main
import (
"encoding/json"
"fmt"
)
type Person struct {
Name *string `json:name`
Age *int `json:age`
Male *bool `json:male`
}
func main() {
var p *Person
err := json.Unmarshal([]byte("{}"), &p)
// handle parse error
if err != nil {
return
}
// handle json did not match error
if p.Name == nil || p.Age == nil || p.Male == nil {
return
}
// now use the fields with dereferencing and hope you did not forget a nil check
fmt.Println("Hello " + *p.Name)
}
Maybe one could use a library like govalidator and use SetFieldsRequiredByDefault. But then you still have to execute the validation and still you are left with the whole pointer dereferencing for value retrieval and the risk of nil pointer.
What I would like is a function that returns my unmarshaled json as a struct or an error if the fields did not match. The only thing the golang json library offers is an option to fail on unknown fields but not to fail on missing fields.
Any idea?
Another way would be to implement your own json.Unmarshaler which uses reflection (similar to the default json unmarshaler):
There are a few points to consider:
if speed is of great importance to you then you should write a benchmark to see how big the impact of the extra reflection is. I suspect its negligible but it can't hurt to write a small go benchmark to get some numbers.
the stdlib will unmarshal all numbers in your json input into floats. So if you use reflection to set integer fields then you need to provide the corresponding conversion yourself (see TODO in example below)
the json.Decoder.DisallowUnknownFields function will not work as expected with your type. You need to implement this yourself (see example below)
if you decide to take this approach you will make your code more complex and thus harder to understand and maintain. Are you actually sure you must know if fields are omitted? Maybe you can refactor your fields to make good usage of the zero values?
Here a fully executable test of this approach:
package sandbox
import (
"encoding/json"
"errors"
"reflect"
"strings"
"testing"
)
type Person struct {
Name string
City string
}
func (p *Person) UnmarshalJSON(data []byte) error {
var m map[string]interface{}
err := json.Unmarshal(data, &m)
if err != nil {
return err
}
v := reflect.ValueOf(p).Elem()
t := v.Type()
var missing []string
for i := 0; i < t.NumField(); i++ {
field := t.Field(i)
val, ok := m[field.Name]
delete(m, field.Name)
if !ok {
missing = append(missing, field.Name)
continue
}
switch field.Type.Kind() {
// TODO: if the field is an integer you need to transform the val from float
default:
v.Field(i).Set(reflect.ValueOf(val))
}
}
if len(missing) > 0 {
return errors.New("missing fields: " + strings.Join(missing, ", "))
}
if len(m) > 0 {
extra := make([]string, 0, len(m))
for field := range m {
extra = append(extra, field)
}
// TODO: consider sorting the output to get deterministic errors:
// sort.Strings(extra)
return errors.New("unknown fields: " + strings.Join(extra, ", "))
}
return nil
}
func TestJSONDecoder(t *testing.T) {
cases := map[string]struct {
in string
err string
expected Person
}{
"Empty object": {
in: `{}`,
err: "missing fields: Name, City",
expected: Person{},
},
"Name missing": {
in: `{"City": "Berlin"}`,
err: "missing fields: Name",
expected: Person{City: "Berlin"},
},
"Age missing": {
in: `{"Name": "Friedrich"}`,
err: "missing fields: City",
expected: Person{Name: "Friedrich"},
},
"Unknown field": {
in: `{"Name": "Friedrich", "City": "Berlin", "Test": true}`,
err: "unknown fields: Test",
expected: Person{Name: "Friedrich", City: "Berlin"},
},
"OK": {
in: `{"Name": "Friedrich", "City": "Berlin"}`,
expected: Person{Name: "Friedrich", City: "Berlin"},
},
}
for name, c := range cases {
t.Run(name, func(t *testing.T) {
var actual Person
r := strings.NewReader(c.in)
err := json.NewDecoder(r).Decode(&actual)
switch {
case err != nil && c.err == "":
t.Errorf("Expected no error but go %v", err)
case err == nil && c.err != "":
t.Errorf("Did not return expected error %v", c.err)
case err != nil && err.Error() != c.err:
t.Errorf("Expected error %q but got %v", c.err, err)
}
if !reflect.DeepEqual(c.expected, actual) {
t.Errorf("\nWant: %+v\nGot: %+v", c.expected, actual)
}
})
}
}
You could compare p with a empty struct, instead of comparing each field with nil.
// handle json did not match error
if p == Person{} {
return
}
Since Person{} will initialize with the 0 value of each field, this will result in each property that is pointers to be nil, strings will be "", ints will be 0, and so on.

Show implicit assignment of unexported field 'data' in simplejson.Json literal when use bitly's go-simplejson

When I use like &simplejson.Json{v} (v is a interface read from file and it's actual data structure is map[string]interface{}), then show this error. Details:
A json file named abcd
{
"pids": [
{
"pid": 168043,
"target_regions": [
40,
25,
43,
299,
240
]
},
{
"pid": 168044,
"target_regions": [
63,
65,
68
]
}
]
}
And the go file is
package main
import (
"fmt"
"io/ioutil"
sjson "github.com/bitly/go-simplejson"
)
type pidInfo struct {
Pid uint64 `json:"pid"`
TargetRegions []uint32 `json:"target_regions"`
}
type pidUnitInfo struct {
Pid2Info map[uint64]*pidInfo
}
func build() error {
content, _ := ioutil.ReadFile("./abcd")
json, err := sjson.NewJson(content)
if err != nil {
return err
}
newPidUnitInfo(json)
return nil
}
func newPidUnitInfo(json *sjson.Json) (*pidUnitInfo, error) {
newInfo := new(pidUnitInfo)
newInfo.buildPid2Info(json)
return nil, nil
}
func (pui *pidUnitInfo) buildPid2Info(json *sjson.Json) error {
raw, ok := json.CheckGet("pids")
if !ok {
return fmt.Errorf("not found json key %v", "pids")
}
pids, err := raw.Array()
if err != nil {
return err
}
pui.Pid2Info = make(map[uint64]*pidInfo, len(pids))
for _, v := range pids {
fmt.Println(v)
m := &sjson.Json{v}
fmt.Println(m)
}
return nil
}
func main() {
build()
}
When I execute it, show implicit assignment of unexported field 'data' in simplejson.Json literal at this line m := &sjson.Json{v}.
This line:
m := &sjson.Json{v}
Tries to create a value (and take the address) of the struct type Json from package go-simplejson. The type declaration is:
type Json struct {
data interface{}
}
It has one field: data which is unexported. That means packages other than go-simplejson cannot refer to this field. When you use a struct literal &sjson.Json{v}, it would try to initialize the Json.data field with value v which is a violation of this. You cannot do this.
The Json type is not designed for you to specify the internal data, it is designed so that the data will be the placeholder of some decoded JSON data (see the NewFromReader() and NewJson() constructor-like functions).
This data field is handled internally by the go-simplejson package, you cannot set it yourself. You may use sjson.New() to obtain a new *Json value which will initialize this data field with an empty map (map[string]interface{}). You may also use Json.Map() method which asserts that data is a map and returns it like that:
js := sjson.New()
m, err := js.Map()
if err != nil {
// Handle error
}
// Now you have a map of type map[string]interface{}
Or to populate the data inside a Json, you can use its Json.Set() method.

Custom JSON Unmarshalling for string-encoded number

I have a struct which contains various currency values, in cents (1/100 USD):
type CurrencyValues struct {
v1 int `json:"v1,string"`
v2 int `json:"v2,string"`
}
I'd like to create a custom json Unmarshaller for currency values with thousand separators. These values are encoded as strings, with one or more thousand separators (,), and possibly a decimal point (.).
For this JSON {"v1": "10", "v2": "1,503.21"}, I'd like to JSON Unmarshal a CurrencyValues{v1: 1000, v2: 150321}.
Following a similar answer here: Golang: How to unmarshall both 0 and false as bool from JSON, I went ahead and created a custom type for my currency fields, which include a custom Unmarshalling function:
type ConvertibleCentValue int
func (cents *ConvertibleCentValue) UnmarshalJSON(data []byte) error {
asString := string(data)
// Remove thousands separators
asString = strings.Replace(asString, ",", "", -1)
// Parse to float, then convert dollars to cents
if floatVal, err := strconv.ParseFloat(asString, 32); err == nil {
*cents = ConvertibleCentValue(int(floatVal * 100.0))
return nil
} else {
return err
}
}
However, when writing unit tests:
func Test_ConvertibleCentValue_Unmarshal(t *testing.T) {
var c ConvertibleCentValue
assert.Nil(t, json.Unmarshal([]byte("1,500"), &c))
assert.Equal(t, 150000, int(c))
}
I encounter this error:
Error: Expected nil, but got: &json.SyntaxError{msg:"invalid character ',' after top-level value", Offset:2}
What am I missing here?
You're trying to unmarshal the string 1,500 which is invalid in JSON. I think what you means is to unmarshal the JSON string "1,500":
assert.Nil(t, json.Unmarshal([]byte(`"1,500"`), &c))
Note the backticks. Here is a simplified example:
b := []byte(`1,500`)
var s string
err := json.Unmarshal(b, &s)
fmt.Println(s, err) // Prints error.
b = []byte(`"1,500"`)
err = json.Unmarshal(b, &s)
fmt.Println(s, err) // Works fine.
Playground: http://play.golang.org/p/uwayOSgmTv.

Golang Decoding/Unmarshaling invalid unicode in JSON

I am fetching JSON files in go that are not formatted homogeneously.
For Example, I can have the following:
{"email": "\"blah.blah#blah.com\""}
{"email": "robert#gmail.com"}
{"name": "m\303\203ead"}
We can see that there will be a problem with the escaping character.
Using json.Decode:
With:
{"name": "m\303\203ead"}
I get the error: invalid character '3' in string escape code
I have tried several approaches to normalise my data for example by passing by a string array (it works but there is too many edge cases), or even to filter escape characters.
Finally, I came through this article: (http://blog.golang.org/normalization)
And the solution they proposed seemed very interesting.
I have tried the following
isMn := func(r rune) bool {
return unicode.Is(unicode.Mn, r)
}
t := transform.Chain(norm.NFC, transform.RemoveFunc(isMn), norm.NFD)
fileReader, err := bucket.GetReader(filename)
transformReader := transform.NewReader(fileReader, t)
decoder := json.NewDecoder(tReader)
for {
var dataModel Model
if err := decoder.Decode(&kmData); err == io.EOF {
break
} else {
// DO SOMETHING
}
}
With Model being:
type Model struct {
Name string `json:"name" bson:"name"`
Email string `json:"email" bson:"email"`
}
I have tried several variations of it, but haven't been able to have it working.
So my question is how to easily handle decoding/unmarshaling JSON data with different encodings? Knowing, that I have no control on those JSON files.
If you are reading this, thank you anyway.
You can use json.RawMessage instead of string, that way json.Decode won't try to decode the invalid characters.
playground : http://play.golang.org/p/fB-38KGAO0
type Model struct {
N json.RawMessage `json:"name" bson:"name"`
}
func (m *Model) Name() string {
return string(m.N)
}
func main() {
s := "{\"name\": \"m\303\203ead\"}"
r := strings.NewReader(s)
d := json.NewDecoder(r)
m := Model{}
fmt.Println(d.Decode(&m))
fmt.Println(m.Name())
}
Edit: Well, you can use regex, not sure how viable that is for you http://play.golang.org/p/VYJKTKmiYm:
func cleanUp(s string) string {
re := regexp.MustCompile(`\b(\\\d\d\d)`)
return re.ReplaceAllStringFunc(s, func(s string) string {
return `\u0` + s[1:]
})
}
func main() {
s := "{\"name\": \"m\303\203ead\"}"
s = cleanUp(s)
r := strings.NewReader(s)
d := json.NewDecoder(r)
m := Model{}
fmt.Println(d.Decode(&m))
fmt.Println(m.Name())
}