I am using Google's go-jsonnet library to evaluate some jsonnet files.
I have a function, like so, which renders a Jsonnet document:
// Takes a list of jsonnet files and imports each one and mixes them with "+"
func renderJsonnet(files []string, param string, prune bool) string {
// empty slice
jsonnetPaths := files[:0]
// range through the files
for _, s := range files {
jsonnetPaths = append(jsonnetPaths, fmt.Sprintf("(import '%s')", s))
}
// Create a JSonnet VM
vm := jsonnet.MakeVM()
// Join the slices into a jsonnet compat string
jsonnetImport := strings.Join(jsonnetPaths, "+")
if param != "" {
jsonnetImport = "(" + jsonnetImport + ")" + param
}
if prune {
// wrap in std.prune, to remove nulls, empty arrays and hashes
jsonnetImport = "std.prune(" + jsonnetImport + ")"
}
// render the jsonnet
out, err := vm.EvaluateSnippet("file", jsonnetImport)
if err != nil {
log.Panic("Error evaluating jsonnet snippet: ", err)
}
return out
}
This function currently returns a string, because the jsonnet EvaluateSnippet function returns a string.
What I now want to do is render that result JSON using the go-prettyjson library. However, because the JSON i'm piping in is a string, it's not rendering correctly.
So, some questions:
Can I convert the returned JSON string to a JSON type, without knowing beforehand what struct to marshal it into
if not, can I render the json in a pretty manner some other way?
Is there an option, function or method I'm missing here to make this easier?
Can I convert the returned JSON string to a JSON type, without knowing beforehand what struct to marshal it into
Yes. It's very easy:
var jsonOut interface{}
err := json.Unmarshal([]byte(out), &jsonOut)
if err != nil {
log.Panic("Invalid json returned by jsonnet: ", err)
}
formatted, err := prettyjson.Marshal([]byte(jsonOut))
if err != nil {
log.Panic("Failed to format jsonnet output: ", err)
}
More info here: https://blog.golang.org/json-and-go#TOC_5.
Is there an option, function or method I'm missing here to make this easier?
Yes. The go-prettyjson library has a Format function which does the unmarshalling for you:
formatted, err := prettyjson.Format([]byte(out))
if err != nil {
log.Panic("Failed to format jsonnet output: ", err)
}
can I render the json in a pretty manner some other way?
Depends on your definition of pretty. Jsonnet normally outputs every field of an object and every array element on a separate line. This is usually considered pretty printing (as opposed to putting everything on the same line with minimal whitespace to save a few bytes). I suppose this is not good enough for you. You can write your own manifester in jsonnet which formats it to your liking (see std.manifestJson as an example).
Related
I am parsing rather simple json where the slash in the date format string is escaped when it arrives in response.
however, when you try to unmarshall the string, it fails on "invalid syntax" error.
So I googled and we should use strconv.Unquote to replace the escaped characters first. Did that, and now the unquote function fails on the "unknown escape" error. However JSON RFC 8259 shows it is a valid case. So why valid JSON is causing failures in Go unmarshaller?
This is the simple JSON
{
"created_at": "6\/30\/2022 21:51:49"
}
var dst string
err := json.Unmarshal([]byte("6\/30\/2022 21:51:49"), dst) // this fails on ivnalid quote
fmt.Println(dst, err)
s, err := strconv.Unquote(`"6\/30\/2022 21:51:49"`) // this fails on ivnalid syntax
fmt.Println(s, err)
How does one proceed with such cases when the JSON is valid, but none of the built-in functions of Go actually parse it? What is the right way, apart from writing simple find and replace pattern in json []byte manually?
EDIT:
ok I think I should have asked a better question or provided a specific example from the scenario rather than just a portion of it. Garbage in garbage out, my fault.
Here is the thing. Let's say we have the specific, non standard time format like in the above json example.
{
"created_at": "6\/30\/2022 21:51:49"
}
I need to parse it into the struct with go time type. Because the standard parser for time would fail for that format I create a custom type.
type Request struct {
CreatedAt Time `json:"created_at"`
}
type Time time.Time
unc (d *Time) UnmarshalJSON(b []byte) error {
s := strings.Trim(string(b), "\"")
parse, err := time.Parse("1/2/2006 15:04:05", s) // this fails because of the escaped backslashes
if err != nil {
return err
}
*d = Time(parse)
return nil
}
Here is an example in Playground: https://go.dev/play/p/bE3AWQeV-ug
panic: parsing time "6\\/30\\/2022 21:51:49" as "1/2/2006 15:04:05": cannot parse "\\/30\\/2022 21:51:49" as "/"
This is the reason I am trying to put the Unquote into the custom unmarshal function, but that is not the right approach as it was already pointed out. How does the default string json unmarshaller for example removes those escaped slashes? Oh and I have no control about the side that writes the json, it comes in with single escaped slashes like that in []byte from *http.Response
Go has a great built-in function to convert a JSON string to a Go string: json.Unmarshal. Here is how you can integrate it with a custom UnmarshalJSON method:
func (d *Time) UnmarshalJSON(b []byte) error {
var s string
if err := json.Unmarshal(b, &s); err != nil {
return err
}
// Remove this line: s := strings.Trim(string(b), "\"")
// The rest of the code is unchanged
Unquote is designed for cases when you have a double quoted string like "\"My name\"", you don't really need that in this case.
The problem with that code is that you're trying to unmarshal a byte slice that isn't JSON. You're also unmarshaling into an string which isn't going to work. You need to unmarshal into a struct that has json tags or unmarshal into a map.
var dest map[string]string
someJsonString := `{
"created_at": "6\/30\/2022 21:51:49"
}`
err := json.Unmarshal([]byte(someJsonString), &dest)
if err != nil {
panic(err)
}
fmt.Println(dest)
if you wanted to do Unmarshal into a struct, you can do it like so
type SomeStruct struct {
CreatedAt string `json:"created_at"`
}
someJsonString := `{
"created_at": "6\/30\/2022 21:51:49"
}`
var data SomeStruct
err := json.Unmarshal([]byte(someJsonString), &data)
if err != nil {
panic(err)
}
fmt.Println(data)
I am writing a program in Golang that interfaces with a modified version of the barefoot mapmatching library which returns results in json via netcat.
My in my actual code json.Unmarshal will only parse the response to the nil value of the struct. But if print the json to console (see code snippet below) and copy paste into goplayground it behaves as expected.
I am wondering if this is an encoding issue that is bypassed when I copy paste from the console as a result.
How do I get my code to process the same string as it is received from barefoot as when it is copy pasted from the console?
Here is the relevant code snippet (structs are identical to goplayground)
body := io_func(conn, cmd)
var obvs []Json_out
json.Unmarshal([]byte(body), &obvs)
fmt.Println(body)
fmt.Println(obvs)
and io_func() if relevant (the response is two lines, with a message on the first and a json string on the second)
func io_func(conn net.Conn, cmd string) string {
fmt.Fprintf(conn, cmd+"\n")
r := bufio.NewReader(conn)
header, _ := r.ReadString('\n')
if header == "SUCCESS\n" {
resp, _ := r.ReadString('\n')
return resp
} else {
return ""
}
}
Following Cerise Limón's advice to properly handle error messages I determined the osm_id value in the JSON was being parsed by json.Unmarshall as number when taking the string from io_func(), although it wasn't doing so when the string was passed in manually in the playground example. Although I don't understand why this is so I would have picked it up with proper error handling.
I altered barefoot code to return the osm_id explicitly in inverted commas since, although only ever composed of digits, I only use it as a string. It now works as expected. Equally I could have changed the type in the struct and convert in Go as needed.
The io_func function creates and discards a bufio.Reader and data the reader may have buffered. If the application calls io_func more than once, then the application may be discarding data read from the network. Fix by creating a single bufio.Reader outside the function and pass that single reader to each invocation of io_func.
Always check and handle errors. The error returned from any of these functions may point you in the right direction for a fix.
func io_func(r *bufio.Reader, conn net.Conn, cmd string) (string, error) {
fmt.Fprintf(conn, cmd+"\n")
header, err := r.ReadString('\n')
if err != nil {
return "", err
}
if header == "SUCCESS\n" {
return r.ReadString('\n')
}
return "", nil
}
...
r := bufio.NewReader(conn)
body, err := io_func(r, conn, cmd)
if err != nil {
// handle error
}
var obvs []Json_out
err = json.Unmarshal([]byte(body), &obvs)
if err != nil {
// handle error
}
fmt.Println(body)
fmt.Println(obvs)
// read next
body, err = io_func(r, conn, cmd)
if err != nil {
// handle error
}
The application uses newline to terminate the JSON body, but newline is valid whitespace in JSON. If the peer includes a newline in the JSON, then the application will read a partial message.
Is there a simple way in Go to check whether given JSON is either an Object {} or array []?
The first thing that comes to mind is to json.Unmarshal() into an interface, and then see if it becomes a map, or a slice of maps. But that seems quite inefficient.
Could I just check if the first byte is a { or a [? Or is there a better way of doing this that already exists.
Use the following to detect if JSON text in the []byte value data is an array or object:
// Get slice of data with optional leading whitespace removed.
// See RFC 7159, Section 2 for the definition of JSON whitespace.
x := bytes.TrimLeft(data, " \t\r\n")
isArray := len(x) > 0 && x[0] == '['
isObject := len(x) > 0 && x[0] == '{'
This snippet of code handles optional leading whitespace and is more efficient than unmarshalling the entire value.
Because the top-level value in JSON can also be a number, string, boolean or nil, it's possible that isArray and isObject both evaluate to false. The values isArray and isObject can also evaluate to false when the JSON is invalid.
Use a type switch to determine the type. This is similar to Xay's answer, but simpler:
var v interface{}
if err := json.Unmarshal(data, &v); err != nil {
// handle error
}
switch v := v.(type) {
case []interface{}:
// it's an array
case map[string]interface{}:
// it's an object
default:
// it's something else
}
Do step-by-step parsing of your JSON, using json.Decoder. This has the advantage over the other answers of:
Being more efficient than decoding the entire value
Using the official JSON parsing rules, and generating standard errors if you get invalid input.
Note, this code isn't tested, but should be enough to give you the idea. It can also be easily expanded to check for numbers, booleans, or strings, if desired.
func jsonType(in io.Reader) (string, error) {
dec := json.NewDecoder(in)
// Get just the first valid JSON token from input
t, err := dec.Token()
if err != nil {
return "", err
}
if d, ok := t.(json.Delim); ok {
// The first token is a delimiter, so this is an array or an object
switch (d) {
case '[':
return "array", nil
case '{':
return "object", nil
default: // ] or }, shouldn't be possible
return "", errors.New("Unexpected delimiter")
}
}
return "", errors.New("Input does not represent a JSON object or array")
}
Note that this consumed the first few bytes of in. It is an exercise for the reader to make a copy, if necessary. If you're trying to read from a byte slice ([]byte), convert it to a reader first:
t, err := jsonType(bytes.NewReader(myValue))
Go playground
I have an application that consumes data from a third-party api. I need to decode the json into a struct, which requires the struct to have json tags of the "incoming" json fields. The outgoing json fields have a different naming convention, so I need different json tags for the encoding.
I will have to do this with many different structs, and each struct might have many fields.
What is the best way to accomplish this without repeating a lot of code?
Example Structs:
// incoming "schema" field names
type AccountIn struct {
OpenDate string `json:"accountStartDate"`
CloseDate string `json:"cancelDate"`
}
// outgoing "schema" field names
type AccountOut struct {
OpenDate string `json:"openDate"`
CloseDate string `json:"closeDate"`
}
Maybe the coming change on Go 1.8 would help you, it will allow to 'cast' types even if its JSON tags definition is different: This https://play.golang.org/p/Xbsoa8SsEk works as expected on 1.8beta, I guess this would simplify your current solution
A bit an uncommon but probably quite well working method would be to use a intermediate format so u can use different readers and writers and therefore different tags. For example https://github.com/mitchellh/mapstructure which allows to convert a nested map structure into struct
types. Pretty similar like json unmarshal, just from a map.
// incoming "schema" field names
type AccountIn struct {
OpenDate string `mapstructure:"accountStartDate" json:"openDate"`
CloseDate string `mapstructure:"cancelDate" json:"closeDate"`
}
// from json to map with no name changes
temporaryMap := map[string]interface{}{}
err := json.Unmarshal(jsonBlob, &temporaryMap)
// from map to structs using mapstructure tags
accountIn := &AccountIn{}
mapstructure.Decode(temporaryMap, accountIn)
Later when writing (or reading) u will use directly the json functions which will then use the json tags.
If it's acceptable to take another round trip through json.Unmarshal and json.Marshal, and you don't have any ambiguous field names within your various types, you could translate all the json keys in one pass by unmarshaling into the generic structures used by the json package:
// map incoming to outgoing json identifiers
var translation = map[string]string{
"accountStartDate": "openDate",
"cancelDate": "closeDate",
}
func translateJS(js []byte) ([]byte, error) {
var m map[string]interface{}
if err := json.Unmarshal(js, &m); err != nil {
return nil, err
}
translateKeys(m)
return json.MarshalIndent(m, "", " ")
}
func translateKeys(m map[string]interface{}) {
for _, v := range m {
if v, ok := v.(map[string]interface{}); ok {
translateKeys(v)
}
}
keys := make([]string, 0, len(m))
for k := range m {
keys = append(keys, k)
}
for _, k := range keys {
if newKey, ok := translation[k]; ok {
m[newKey] = m[k]
delete(m, k)
}
}
}
https://play.golang.org/p/nXmWlj7qH9
This might be a Naive Approach but is fairly easy to implement:-
func ConvertAccountInToAccountOut(AccountIn incoming) (AccountOut outcoming){
var outcoming AccountOut
outcoming.OpenDate = incoming.OpenDate
outcoming.CloseDate = incoming.CloseDate
return outcoming
}
var IncomingJSONData AccountIn
resp := getJSONDataFromSource() // Some method that gives you the Input JSON
err1 := json.UnMarshall(resp,&IncomingJSONData)
OutGoingJSONData := ConvertAccountInToAccountOut(IncomingJSONData)
if err1 != nil {
fmt.Println("Error in UnMarshalling JSON ",err1)
}
fmt.Println("Outgoing JSON Data: ",OutGoingJSONData)
I am using gin as my http server and sending back an empty array in json as my response:
c.JSON(http.StatusOK, []string{})
The resulting json string I get is "[]\n". The newline is added by the json Encoder object, see here.
Using goconvey, I could test my json like
So(response.Body.String(), ShouldEqual, "[]\n")
But is there a better way to generate the expected json string than just adding a newline to all of them?
You should first unmarshal the body of the response into a struct and compare against the resulting object. Example:
result := []string{}
if err := json.NewDecoder(response.Body).Decode(&result); err != nil {
log.Fatalln(err)
}
So(len(result), ShouldEqual, 0)
You may find jsonassert useful.
It has no dependencies outside the standard library and allows you to verify that JSON strings are semantically equivalent to a JSON string you expect.
In your case:
// white space is ignored, no need for \n
jsonassert.New(t).Assertf(response.Body().String(), "[]")
It can handle any form of JSON, and has very friendly assertion error messages.
Disclaimer: I wrote this package.
Unmarshal the body into a struct and the use Gocheck's DeepEquals
https://godoc.org/launchpad.net/gocheck
I made it this way. Because I don't want to include an extra library.
tc := testCase{
w: httptest.NewRecorder(),
wantResponse: mustJson(t, map[string]string{"message": "unauthorized"}),
}
...
if tc.wantResponse != tc.w.Body.String() {
t.Errorf("want %s, got %s", tt.wantResponse, tt.w.Body.String())
}
...
func mustJson(t *testing.T, v interface{}) string {
t.Helper()
out, err := json.Marshal(v)
if err != nil {
t.Fatal(err)
}
return string(out)
}