I am new to golang and json and currently struggle to parse the json out from a system.
I've read a couple of blog posts on dynamic json in go and also tried the tools like json2GoStructs
Parsing my json file with this tools just gave me a huge structs which I found a bit messy. Also I had no idea how to get the info im interested in.
So, here are my problems:
How do I get to the info I am interested in?
What is the best approach to parse complex json?
I am only interested into the following 3 json fields:
Name
Guid
Task -> Property -> Name: Error
I'm thankful for every tip, code snippet or explanation!
This is what I got so far (mostly from a tutorial):
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
)
func checkErr(err error) {
if err != nil {
panic(err)
}
}
func readFile(filePath string) []byte {
data, err := ioutil.ReadFile(filePath)
checkErr(err)
return data
}
func main() {
path := "/Users/andi/Documents/tmp/wfsJob.json"
data := readFile(path)
var f interface{}
err := json.Unmarshal(data, &f)
checkErr(err)
m := f.(map[string]interface{})
for k, v := range m {
switch vv := v.(type) {
case string:
fmt.Println(k, "is string", vv)
case int:
fmt.Println(k, "is int", vv)
case []interface{}:
fmt.Println(k, "is an array:")
for i, u := range vv {
fmt.Println(i, u)
}
default:
fmt.Println(k, "is of a type I don't know how to handle")
}
}
}
I can offer you this easy way to using JSON in Golang. With this tool you don't need to parse the whole json file, and you can use it without struct.
Gjson is a great solution for fetching a few fields from JSON string. But it may become slow when many (more than 2) fields must be fetched from distinct parts of the JSON, since it re-parses the JSON on each Get call. Additionally, it requires calling gjson.Valid for validating the incoming JSON, since other methods assume the caller provides valid JSON.
There is an alternative package - fastjson. Like gsjon, it is fast and has nice API. Unlike gjson it validates the input JSON and works faster when many unrelated fields must be obtained from the JSON. Here is a sample code for obtaining fields from the original question:
var p fastjson.Parser
v, err := p.ParseBytes(data)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Name: %s", v.GetStringBytes("Name"))
fmt.Printf("Guid: %s", v.GetStringBytes("Guid"))
fmt.Printf("Error: %s", v.GetStringBytes("Task", "Property", "Name"))
Related
I am parsing rather simple json where the slash in the date format string is escaped when it arrives in response.
however, when you try to unmarshall the string, it fails on "invalid syntax" error.
So I googled and we should use strconv.Unquote to replace the escaped characters first. Did that, and now the unquote function fails on the "unknown escape" error. However JSON RFC 8259 shows it is a valid case. So why valid JSON is causing failures in Go unmarshaller?
This is the simple JSON
{
"created_at": "6\/30\/2022 21:51:49"
}
var dst string
err := json.Unmarshal([]byte("6\/30\/2022 21:51:49"), dst) // this fails on ivnalid quote
fmt.Println(dst, err)
s, err := strconv.Unquote(`"6\/30\/2022 21:51:49"`) // this fails on ivnalid syntax
fmt.Println(s, err)
How does one proceed with such cases when the JSON is valid, but none of the built-in functions of Go actually parse it? What is the right way, apart from writing simple find and replace pattern in json []byte manually?
EDIT:
ok I think I should have asked a better question or provided a specific example from the scenario rather than just a portion of it. Garbage in garbage out, my fault.
Here is the thing. Let's say we have the specific, non standard time format like in the above json example.
{
"created_at": "6\/30\/2022 21:51:49"
}
I need to parse it into the struct with go time type. Because the standard parser for time would fail for that format I create a custom type.
type Request struct {
CreatedAt Time `json:"created_at"`
}
type Time time.Time
unc (d *Time) UnmarshalJSON(b []byte) error {
s := strings.Trim(string(b), "\"")
parse, err := time.Parse("1/2/2006 15:04:05", s) // this fails because of the escaped backslashes
if err != nil {
return err
}
*d = Time(parse)
return nil
}
Here is an example in Playground: https://go.dev/play/p/bE3AWQeV-ug
panic: parsing time "6\\/30\\/2022 21:51:49" as "1/2/2006 15:04:05": cannot parse "\\/30\\/2022 21:51:49" as "/"
This is the reason I am trying to put the Unquote into the custom unmarshal function, but that is not the right approach as it was already pointed out. How does the default string json unmarshaller for example removes those escaped slashes? Oh and I have no control about the side that writes the json, it comes in with single escaped slashes like that in []byte from *http.Response
Go has a great built-in function to convert a JSON string to a Go string: json.Unmarshal. Here is how you can integrate it with a custom UnmarshalJSON method:
func (d *Time) UnmarshalJSON(b []byte) error {
var s string
if err := json.Unmarshal(b, &s); err != nil {
return err
}
// Remove this line: s := strings.Trim(string(b), "\"")
// The rest of the code is unchanged
Unquote is designed for cases when you have a double quoted string like "\"My name\"", you don't really need that in this case.
The problem with that code is that you're trying to unmarshal a byte slice that isn't JSON. You're also unmarshaling into an string which isn't going to work. You need to unmarshal into a struct that has json tags or unmarshal into a map.
var dest map[string]string
someJsonString := `{
"created_at": "6\/30\/2022 21:51:49"
}`
err := json.Unmarshal([]byte(someJsonString), &dest)
if err != nil {
panic(err)
}
fmt.Println(dest)
if you wanted to do Unmarshal into a struct, you can do it like so
type SomeStruct struct {
CreatedAt string `json:"created_at"`
}
someJsonString := `{
"created_at": "6\/30\/2022 21:51:49"
}`
var data SomeStruct
err := json.Unmarshal([]byte(someJsonString), &data)
if err != nil {
panic(err)
}
fmt.Println(data)
How can I convert grpc/protobuf3 message to JSON where the enum is represented as string?
For example, the protobuf message:
enum Level {
WARNING = 0;
FATAL = 1;
SEVERE = 2;
...
}
message Http {
string message = 1;
Level level = 2;
}
Is converted by:
j, _ := json.MarshalIndent(protoMessage, "", "\t")
To:
{
"message": "Hello world!",
"level": 2,
}
I wish to get:
{
"message": "Hello world!",
"level": "SEVERE",
}
Thanks
I found out that I should use the protobuf/jsonpb package and not the standard json package.
so:
j, _ := json.MarshalIndent(protoMessage, "", "\t")
Should be:
m := jsonpb.Marshaler{}
result, _ := m.MarshalToString(protoMessage)
Update
As noted bellow, jsonpb is depricated and the new solution is to use protojson
I found some of these modules (jsonpb) to be deprecated. What worked for me was the google encoding version:
import "google.golang.org/protobuf/encoding/protojson"
jsonString := protojson.Format(protoMessage)
Level is not a string though, it is an emum. There are really only two choices I see.
Write a custom marshaller that does this for you
Generate code that does this for you.
For #2, gogoprotobuf has an extension (still marked as experimental) that let's you do exactly this:
https://godoc.org/github.com/gogo/protobuf/plugin/enumstringer and
https://github.com/gogo/protobuf/blob/master/extensions.md
For my use case, I wanted to write it to a file. Using the most recent packages as of this date, this was as close to a regular encoding/json marshal as I could get.
I used the google.golang.org/protobuf/encoding/protojson package and the .ProtoReflect().Interface() methods of my protocol buffer data structure.
package main
import (
"io/ioutil"
"log"
"google.golang.org/protobuf/encoding/protojson"
"myproject/proto"
)
func main() {
myProtoStruct := proto.MyType{}
data, err := protojson.Marshal(myProtoStruct.ProtoReflect().Interface())
if err != nil {
log.Fatalf("Failed to JSON marhsal protobuf.\nError: %s", err.Error())
}
err = ioutil.WriteFile("my.proto.dat", data, 0600)
if err != nil {
log.Fatalf("Failed to write protobuf data to file.\nError: %s", err.Error())
}
log.Println("Written to file.")
}
When it comes to serialize the json object, this would be helpful.
var msg bytes.Buffer
m := jsonpb.Marshaler{}
err := m.Marshal(&msg, event)
msg.Bytes() converts msg to byte stream.
this is a very old question, but I will post my solution, maybe someone on the future will also face the same issue.
firstly, jsonpb is deprecated.
secondly, when you use protojson, you can set your options, which will affect the outcome JSON. In your case, the option UseEnumNumbers says how you want to marshall enums, strings or ints. Like so:
arr, err := protojson.MarshalOptions{
UseEnumNumbers: true, // uses enums as int, not as strings
}.Marshal(a)
I'm new to Golang and I am trying to parse large json like the ones you get from an API which have lots of data. The documentation explains how to do this with any json:
b := []byte(`{"Name":"Wednesday","Age":6,"Parents":["Gomez","Morticia"]}`)
var f interface{}
err := json.Unmarshal(b, &f)
m := f.(map[string]interface{})
This works fine, but when I use a json that I get from the Twitter API, like the one at the end of the reference on the Twitter dev site I get this error:
interface conversion: interface {} is []interface {}, not map[string]interface {}
I know that there are similar questions but I couldn't find the answer. Can someone recommend me the better way to solve this?
My go version go1.6.2 linux/amd64.
Thanks !
In this case you are not unmarshalling a single JSON object, but an array of JSON objects which is why you're having an issue parsing the API response. The error your seeing this is telling you the actual type of f. The other example worked because it is a single JSON object, which can be mapped as a map[string]interface{} Take a look at this:
var f []interface{}
err := json.Unmarshal(str, &f)
if err != nil {
fmt.Println(err)
return
}
for _, v := range f {
z := v.(map[string]interface{})
for k2, v2 := range z {
fmt.Println("Key:", k2, "Value:", v2)
}
}
f should be of type []interface{} an array of interfaces. Depending on how you plan on parsing the API response, you can do something like I'm doing in the for loop to present each key, value pair.
I have an application that consumes data from a third-party api. I need to decode the json into a struct, which requires the struct to have json tags of the "incoming" json fields. The outgoing json fields have a different naming convention, so I need different json tags for the encoding.
I will have to do this with many different structs, and each struct might have many fields.
What is the best way to accomplish this without repeating a lot of code?
Example Structs:
// incoming "schema" field names
type AccountIn struct {
OpenDate string `json:"accountStartDate"`
CloseDate string `json:"cancelDate"`
}
// outgoing "schema" field names
type AccountOut struct {
OpenDate string `json:"openDate"`
CloseDate string `json:"closeDate"`
}
Maybe the coming change on Go 1.8 would help you, it will allow to 'cast' types even if its JSON tags definition is different: This https://play.golang.org/p/Xbsoa8SsEk works as expected on 1.8beta, I guess this would simplify your current solution
A bit an uncommon but probably quite well working method would be to use a intermediate format so u can use different readers and writers and therefore different tags. For example https://github.com/mitchellh/mapstructure which allows to convert a nested map structure into struct
types. Pretty similar like json unmarshal, just from a map.
// incoming "schema" field names
type AccountIn struct {
OpenDate string `mapstructure:"accountStartDate" json:"openDate"`
CloseDate string `mapstructure:"cancelDate" json:"closeDate"`
}
// from json to map with no name changes
temporaryMap := map[string]interface{}{}
err := json.Unmarshal(jsonBlob, &temporaryMap)
// from map to structs using mapstructure tags
accountIn := &AccountIn{}
mapstructure.Decode(temporaryMap, accountIn)
Later when writing (or reading) u will use directly the json functions which will then use the json tags.
If it's acceptable to take another round trip through json.Unmarshal and json.Marshal, and you don't have any ambiguous field names within your various types, you could translate all the json keys in one pass by unmarshaling into the generic structures used by the json package:
// map incoming to outgoing json identifiers
var translation = map[string]string{
"accountStartDate": "openDate",
"cancelDate": "closeDate",
}
func translateJS(js []byte) ([]byte, error) {
var m map[string]interface{}
if err := json.Unmarshal(js, &m); err != nil {
return nil, err
}
translateKeys(m)
return json.MarshalIndent(m, "", " ")
}
func translateKeys(m map[string]interface{}) {
for _, v := range m {
if v, ok := v.(map[string]interface{}); ok {
translateKeys(v)
}
}
keys := make([]string, 0, len(m))
for k := range m {
keys = append(keys, k)
}
for _, k := range keys {
if newKey, ok := translation[k]; ok {
m[newKey] = m[k]
delete(m, k)
}
}
}
https://play.golang.org/p/nXmWlj7qH9
This might be a Naive Approach but is fairly easy to implement:-
func ConvertAccountInToAccountOut(AccountIn incoming) (AccountOut outcoming){
var outcoming AccountOut
outcoming.OpenDate = incoming.OpenDate
outcoming.CloseDate = incoming.CloseDate
return outcoming
}
var IncomingJSONData AccountIn
resp := getJSONDataFromSource() // Some method that gives you the Input JSON
err1 := json.UnMarshall(resp,&IncomingJSONData)
OutGoingJSONData := ConvertAccountInToAccountOut(IncomingJSONData)
if err1 != nil {
fmt.Println("Error in UnMarshalling JSON ",err1)
}
fmt.Println("Outgoing JSON Data: ",OutGoingJSONData)
I'm trying to decode dynamic/random JSON responses in GO, with nested data
body, _ := ioutil.ReadAll(response.Body)
resp := make(map[string]interface{})
err = json.Unmarshal(body, &resp)
fmt.Printf("BODY: %T<\n", body)
fmt.Printf("BODY: %s<\n", body)
fmt.Printf("RESP: %s<\n", resp)
fmt.Printf("RESP: %T<\n", resp)
fmt.Printf("RESP[results]: %T<\n", resp["results"])
fmt.Printf("RESP[results]: %s<\n", resp["results"])
body is the JSON result from the HTTP server and I unmarshall it and the result looks to be a slice of bytes.
BODY: []uint8
BODY: {"results":[{"code":500.0,"errors":["Configuration file 'c2-web-2.conf' already exists."],"status":"Object could not be created."}]}
So I unmarshall it into resp and that works as expected.
RESP: map[string]interface {}
RESP: map[results:[map[code:%!s(float64=500) errors:[Configuration file 'c2-web-2.conf' already exists.] status:Object could not be created.]]]<
I'm able to access the map with the key results.
RESP[results]: []interface {}
RESP[results]: [map[code:%!s(float64=500) errors:[Configuration file 'conf.d/hosts/c2-web-2.conf' already exists.] status:Object could not be created.]]<
Now what i want to access it the "code", "errors" and "status" which is in resp["results"] This looks like an array or slice and I've tried indexing it but I get the error at compile time
./create_host.go:62: invalid operation: resp["results"][0] (type interface {} does not support indexing)
I've done a lot of googling, tried unmarshalling the data within resp["results"] etc, but after a few days I have not made much progress.
How should I access the map which seems to be a member of an array? The data structure is not guaranteed so I can't create a structure and unmarshall into that.
Thanks
A co-worker provided the code fragement below which made it possible to access the map entries I was looking for.
respBody, _ := ioutil.ReadAll(response.Body)
var rsp interface{}
if err := json.Unmarshal(respBody, &rsp); err != nil {
log.Fatal(err)
}
resultMap := rsp.(map[string]interface{})["results"].([]interface{})[0].(map[string]interface{})
fmt.Printf("test: %s<\n", resultMap["errors"] )
test: [Configuration file 'c2-web-2.conf' already exists.]<
I believe you need to do a type assertion. You have an interface{}, but you need some sort of slice to index into. Try resp["results"].([]interface{})[0]? (Sorry, haven't had a chance to test this myself.)