I have written following code snippet in trial.go:
type Mine struct{
A string `json:"a"`
}
func main(){
s := Mine{}
v := os.Args[1]//`{"a":"1"}`
fmt.Println(v)
fmt.Println(reflect.TypeOf(v))
json.Unmarshal([]byte(v), &s)
fmt.Println(s)
}
I am running this file as below:
go run trial.go `{"A":"1"}`
But I don't get anything in s. It is always a blank struct.
What am I doing wrong here?
First check errors returned by json.Unmarshal().
Next your json tag uses small "a" as the JSON key, however the encoding/json package will recognize the capital "A" too.
And last passing such arguments in the command line may be OS (shell) specific. The backtick and quotes usually have special meaning, try passing it like this:
go run trial.go {\"a\":\"1\"}
Also you should check the length of os.Args before indexing it, if the user does not provide any arguments, os.Args[1] will panic.
As you mentioned, you should find another way to test input JSON documents, this becomes unfeasible if the JSON text is larger, and also this is OS (shell) specific. A better way would be to read from the standard input or read from a file.
Related
I'm trying to encode and decode structs, I've searched around quite a bit and a lot of the questions regarding this topic is usually people who want to encode primitives, or simple structs. What I want is to encode a struct that could look like this:
Name string
Id int
file *os.File
keys *ecdsa.PrivateKey
}
The name and the ID is no problem, and I can encode them using either gob or json marshalling. However when I want to encode a file for example using gob, I'd usegob.Register(os.File{}) I get an error that file has no exported fields, due to the fields in the file struct being lower case. I would use a function like this
buf := bytes.Buffer{}
enc := gob.NewEncoder(&buf)
gob.Register(big.Int{})
...
err := enc.Encode(&p)
if err != nil {
log.Fatal(err)
}
fmt.Println("uncompressed size (bytes): ", len(buf.Bytes()))
return buf.Bytes()
}
I'm not sure if it's correct to register within the encode function, however it seems odd that I have to register all structs that is being referenced to for the one specific struct i want to encode. For example with a file, I would have to register a ton of interfaces, it doesn't seem to be the correct way to do it. Is there a simple way to encode and decode structs that have a bit more complexity.
If I use json marshalling to do this it will always return nil if I use a pointer to another struct. Is there a way to get all the information I want?
Thanks!
Imagine your struct ponts to a file in /foo/bar/baz.txt and you serialize your struct. The you send it to another computer (perhaps in a different operational system) and re-create the struct. What do you expect?
What if you serialize, delete the file (or update the content) and re-create the struct in the same computer?
One solution is store the content of the file.
Another solution is to store the path to the file and, when you deserialize the struct you can try to reopen the file. You can add a security layer by storing the hash of the content, size and other metadata to check if the file is the same.
The answer will guide you to the best implementation
Problem
Found many similar questions (title) but none solved my problem, so here is it.
I have a JSON string that contains some known fields (should always be present) plus any number of unknown/arbitrary fields.
Example
{"known1": "foo", "known2": "bar", "unknown1": "car", "unknown2": 1}
In this example known1 and known2 are known fields. unknown1 and unknown2 are arbitrary/unknown fields.
The unknown fields can have any name (key) and any value. The value type can be either a string, bool, float64 or int.
What I want is to find the simplest and most elegant (idiomatic) way to parse a message like this.
My solution
I've used the following struct:
type Message struct {
Known1 string `json:"known1"`
Known2 string `json:"known2"`
Unknowns []map[string]interface{}
}
Expected result
With this struct and the above sample JSON message I want to achieve a Message like the following (output from fmt.Printf("%+v", msg)):
{Known1:foo Known2:bar Unknowns:[map[unknown1:car] map[unknown2:1]]}
Attempts
1. Simple unmarshal
https://play.golang.org/p/WO6XlpK_vJg
This doesn't work, Unknowns is not filled with the remaining unknown key/value pairs as expected.
2. Double unmarshal
https://play.golang.org/p/Mw6fOYr-Km8
This works but I needed two unmarshals, one to fill the known fields (using an alias type to avoid an infinite loop) and a second one to get all fields as a map[string]interface{} and process the unknowns.
3. Unmarshal and type conversion
https://play.golang.org/p/a7itXObavrX
This works and seems the best option among my solutions.
Any other option?
Option 2 and 3 work but I'm curious if anyone has a simpler/more elegant solution.
TMTOWTDI, and I think you found a reasonable solution. Another option you could consider -- which I guess is similar to your option 2 -- is to unmarshal it onto a map[string]interface{}.
Then range over the keys and values and do whatever you need to with the data, running type assertions on the values as necessary. Depending on what you need to do, you may skip the struct entirely, or still populate it.
package main
import (
"encoding/json"
"fmt"
)
func main() {
jsonMsg := `{"known1": "foo", "known2": "bar", "unknown1": "car", "unknown2": 1}`
var msg map[string]interface{}
fmt.Println(json.Unmarshal([]byte(jsonMsg), &msg))
fmt.Printf("%+v", msg)
}
prints
<nil>
map[known1:foo known2:bar unknown1:car unknown2:1]
https://play.golang.org/p/Bl99Cq5tFWW
Probably there is no existing solution just for your situation.
As an addition to your solutions you may try to use raw parsing libraries like:
https://github.com/buger/jsonparser
https://github.com/json-iterator/go
https://github.com/nikandfor/json
Last one is mine and it has some unfinished features, but raw parsing is done.
You may also use reflection with libraries above to set known fields in the struct.
Using Python I can do following:
r = requests.get(url_base + url)
jsonObj = json.loads(r.content.decode('raw_unicode_escape'))
print(jsonObj["PartDetails"]["ManufacturerPartNumber"]
Is there any way to perform same thing using Golang?
Currently I need following:
json.Unmarshal(body, &part_number_json)
fmt.Println("\r\nPartDetails: ", part_number_json.(map[string]interface{})["PartDetails"].(map[string]interface{})["ManufacturerPartNumber"])
That is to say I need to use casting for each field of JSON what tires and makes the code unreadable.
I tried this using reflection but it is not comphortable too.
EDIT:
currently use following function:
func jso(json interface{}, fields ...string) interface{} {
res := json
for _, v := range fields {
res = res.(map[string]interface{})[v]
}
return res
and call it like that:
fmt.Println("PartDetails: ", jso( part_number_json, "PartDetails", "ManufacturerPartNumber") )
There are third-party packages like gjson that can help you do that.
That said, note that Go is Go, and Python is Python. Go is statically typed, for better and worse. It takes more code to write simple JSON manipulation, but that code should be easier to maintain later since it's more strictly typed and the compiler helps you check against error. Types also serve as documentation - simply nesting dicts and arrays is completely arbitrary.
I have found the following resource very helpful in creating a struct from json. Unmarshaling should only match the fields you have defined in the struct, so take what you need, and leave the rest if you like.
https://mholt.github.io/json-to-go/
To give some background, I am reading device logs from android and ios devices using adb logcat and idevicesyslog. The specific logs I am after are swift/c#/java/etc dictionaries converted to strings via adb logcat / idevicesyslog. I hope to take these logs that contain JSON like strings and convert those to valid JSON. This works for the most part no problem.
However, sometimes these logs/string outputs contain characters like (\134, \M, \t, etc etc) that cause issues when unmarshalling into JSON. I unmarshal them to JSON to send them elsewhere.
For example a raw device log may have something like the following:
{"foo":"bar","foo":"bar\134/\134/bar\134/bar\134/bar"}
{"foo":"bar","foo":"bar\M/\134/bar\134/bar\M/bar"}
These result in errors like "panic: invalid character 'M' in string escape code" when attempting to unmarshal
The majority of logs do not contain these characters and so it's not a problem. However, a few edge cases contain these and it creates problems.
Is there a proper name for these types of characters? (c# escape characters?) Is there a golang package that can remove them from a string? Currently, I am just deleting the specific ones i come across if they appear in a string but I feel like there's a better way to do it. Adding the ones I come across to a list of deletable characters is not a good practice.
To summarize,
idevicesyslog log gives me a string like so:
{"foo":"bar","foo":"bar\134/\134/bar\134/bar\134/bar"}
This can not be unmarshalled.
idevicesyslog log gives me a string like so:
{"foo":"bar","foo":"bar bar bar bar"}
This can be unmarshalled.
Current solution: add new ones I come across to a list and remove them before unmarshaling
Hopeful solution: detect automatically and remove
Use a regexp to replace the invalid octal escape sequences with a space:
var octalEscapePat = regexp.MustCompile(`\\[0-7]{3}`)
func fix(src string) string {
return octalEscapePat.ReplaceAllString(src, " ")
}
You can also parse the octal value and convert to a valid JSON escape sequence:
func fix(src string) string {
return octalEscapePat.ReplaceAllStringFunc(src, func(s string) string {
// Parse octal value
n, _ := strconv.ParseInt(s[1:], 8, 0)
// Convert to string and marshal to JSON to handle any escaping
b, _ := json.Marshal(string(n))
// return string with surrounding quotes removed
return string(b[1 : len(b)-1])
})
}
The \M can be handled in a similar way.
https://play.golang.org/p/-gtxrvnBSrx
I want to parse and validate (custom) JSON configuration files within Go. I would like to be able to parse the file into a struct and validate that:
no unexpected keys are present in the JSON file (in particular to detect typos)
certain keys are present and have non-empty values
In case the validation fails (or in case of a syntax error), I want to print an error message to the user that explains as detailed as possible where in the file the error happened (e.g. by stating the line number if possible).
The JSON parser built into Go seems to just silently ignore unexpected keys. I also tried using jsonpb (Protobuf) to deserialize the JSON, which returns an error in case of an unexpected key, but does not report the position.
To check for non-empty values, I could use an existing validation library, but I haven't seen any that reports detailed error messages. Alternatively, I could write custom code that validates the data returned by the built-in JSON parser, but it would be nice if there was a generic way.
Is there a simple way to get the desired behaviour?
Have you looked at JSON schema?
JSON Schema describes your JSON data format.
I believe it is in Draft stage, but a lot of languages have validation libraries. Here's a Go implementation:
https://github.com/xeipuuv/gojsonschema
You can also use the encoding/json JSON Decoder and force errors when unexpected keys are found. It won't tell you the line number, but it's a start and you don't require any external package.
package main
import (
"bytes"
"encoding/json"
"fmt"
)
type MyType struct {
ExpectedKey string `json:"expected_key"`
}
func main() {
jsonBytes := []byte(`{"expected_key":"a", "unexpected_key":"b"}`)
var typePlaceholder MyType
// Create JSON decoder
dec := json.NewDecoder(bytes.NewReader(jsonBytes))
// Force errors when unexpected keys are present
dec.DisallowUnknownFields()
if err := dec.Decode(&typePlaceholder); err != nil {
fmt.Println(err.Error())
}
}
You can see that working in playground here