I'm trying to check if a Json string matches an Avro schema. I don't care about doing serialization of the data, just getting a bool result of isValidJson=true/false.
I'll go with every golang library.
I've tried to write something with this goavro lib, but it didn't work for me, maybe because I'm new to golang.
Desired pseudo code:
func main() {
avroSchema :=
`{"type":"record","name":"raw","namespace":"events","fields":[{"name":"my_int","type":["null","int"],"default":null},{"name":"my_string","type":["null","string"],"default":"null"},{"name":"my_string2","type":null}]}`
jsonString := `{"my_int": 3, "my_string": "foo", "my_string2": null}`
ok ;= isValidJson(jsonString, avroSchema)
}
Any idea how to implement the isValidJson(..) method?
Your schema json is invalid, it's missing the terminating }, so goavro.NewCodec returns an error.
Then your json string definitely doesn't match the schema, the json values must be a {type: value}.
You can use the following corrected schema and example string to validate it.
func main() {
avroSchema := `
{
"type":"record",
"name":"raw",
"namespace":"events",
"fields":[
{
"name":"my_int",
"type":[
"null",
"int"
],
"default":null
},
{
"name":"my_string",
"type":[
"null",
"string"
],
"default":null
},
{
"name":"my_string2",
"type":"null"
}
]
}`
codec, err := goavro.NewCodec(avroSchema)
if err != nil {
log.Fatalf("Codec error: %v", err)
}
jsonString := `{"my_int": {"int":3}, "my_string": {"string":"foo"}, "my_string2": null}`
decoded, _, err := codec.NativeFromTextual([]byte(jsonString))
if err != nil {
log.Fatalf("NativeFromTextual error: %v", err)
}
log.Println("Decoded:", decoded)
}
This prints:
Decoded: map[my_int:map[int:3] my_string:map[string:foo]
my_string2:]
Related
I want to serialize object of types.Project from github.com/compose-spec into JSON
However, a field of ths struct which is supposed to be array, is always serialized into map.
package main
import (
"encoding/json"
types "github.com/compose-spec/compose-go/types"
)
func main() {
out := types.Project{
Name: "foo",
Services: []types.ServiceConfig{ // <- notice this field is an array
{
Name: "bar",
Image: "hello-world",
},
},
}
buf, err := json.Marshal(out)
if err != nil {
panic(err)
}
println(string(buf)) // <- notice the Services field now a map, which is incorrect!
var in types.Project
if err := json.Unmarshal(buf, &in); err != nil {
panic(err)
}
}
The code fails to run:
{"name":"foo","services":{"bar":{"command":null,"entrypoint":null,"image":"hello-world"}}}
panic: json: cannot unmarshal object into Go struct field Project.services of type types.Services
goroutine 1 [running]:
main.main()
/tmp/sandbox1081064727/prog.go:29 +0x168
The out object is serialized as
{
"name": "foo",
"services": {
"bar": {
"command": null,
"entrypoint": null,
"image": "hello-world"
}
}
}
which should really be something like
{
"name": "foo",
"services": [
{
"name": "bar",
"command": null,
"entrypoint": null,
"image": "hello-world"
}
]
}
The behavior you're seeing is correct with respect to the compose file specification, which says:
A Compose file MUST declare a services root element as a map whose keys are string representations of service names, and whose values are service definitions.
The transformation of the list to a map is implemented by the custom marshalers in compose-go/types.go:
// MarshalYAML makes Services implement yaml.Marshaller
func (s Services) MarshalYAML() (interface{}, error) {
services := map[string]ServiceConfig{}
for _, service := range s {
services[service.Name] = service
}
return services, nil
}
// MarshalJSON makes Services implement json.Marshaler
func (s Services) MarshalJSON() ([]byte, error) {
data, err := s.MarshalYAML()
if err != nil {
return nil, err
}
return json.MarshalIndent(data, "", " ")
}
I would like to reqeive a JSON response withing a client application and unmarshal this response into a struct. To ensure that the struct stays the same accross all client apps using this package, I would like to define the JSON responses as protobuf messages. I am having difficulties unmarshaling the JSON to the protobuf generated structs.
I have the following JSON data:
[
{
"name": "C1",
"type": "docker"
},
{
"name": "C2",
"type": "docker"
}
]
I have modeled my protobuf definitions like this:
syntax = "proto3";
package main;
message Container {
string name = 1;
string type = 2;
}
message Containers {
repeated Container containers = 1;
}
Using this pattern with structs normaly works, but for some reason using these proto definitions causes issues. The below code demonstrates a working and a non-working example. Although one of the versions work, I am unable to use this solution, since []*Container does not satisfy the proto.Message interface.
package main
import (
"encoding/json"
"fmt"
"strings"
"github.com/gogo/protobuf/jsonpb"
)
func working(data string) ([]*Container, error) {
var cs []*Container
return cs, json.Unmarshal([]byte(data), &cs)
}
func notWorking(data string) (*Containers, error) {
c := &Containers{}
jsm := jsonpb.Unmarshaler{}
if err := jsm.Unmarshal(strings.NewReader(data), c); err != nil {
return nil, err
}
return c, nil
}
func main() {
data := `
[
{
"name": "C1",
"type": "docker"
},
{
"name": "C2",
"type": "docker"
}
]`
w, err := working(data)
if err != nil {
panic(err)
}
fmt.Print(w)
nw, err := notWorking(data)
if err != nil {
panic(err)
}
fmt.Print(nw.Containers)
}
Running this gives the following output:
[name:"C1" type:"docker" name:"C2" type:"docker" ]
panic: json: cannot unmarshal array into Go value of type map[string]json.RawMessage
goroutine 1 [running]:
main.main()
/Users/example/go/src/github.com/example/example/main.go:46 +0x1ee
Process finished with exit code 2
Is there a way to unmarshal this JSON to Containers? Or alternatively, make []*Container to satisfy the proto.Message interface?
For the message Containers, i.e.
message Containers {
repeated Container containers = 1;
}
The correct JSON should look like:
{
"containers" : [
{
"name": "C1",
"type": "docker"
},
{
"name": "C2",
"type": "docker"
}
]
}
If you cannot change the JSON then you can utilize the func that you've created
func working(data string) ([]*Container, error) {
var cs []*Container
err := json.Unmarshal([]byte(data), &cs)
// handle the error here
return &Containers{
containers: cs,
}, nil
}
You should use NewDecoder to transfer the data to jsonDecoder and then traverse
the array.The code is this
func main() {
data := `
[
{
"name": "C1",
"type": "docker"
},
{
"name": "C2",
"type": "docker"
}
]`
jsonDecoder := json.NewDecoder(strings.NewReader(data))
_, err := jsonDecoder.Token()
if err != nil {
log.Fatal(err)
}
var protoMessages []*pb.Container
for jsonDecoder.More() {
protoMessage := pb.Container{}
err := jsonpb.UnmarshalNext(jsonDecoder, &protoMessage)
if err != nil {
log.Fatal(err)
}
protoMessages = append(protoMessages, &protoMessage)
}
fmt.Println("%s", protoMessages)
}
I'm trying to import a large JSON document from a file, empty all arrays matching a specific key or pattern, then output it, without having to marshall the entire document.
It will be run as part of a periodic batch job, so performance/efficiency is not a priority.
Simplicity, and making sure the code is agnostic to the overall JSON structure, is more important.
Is there an easy way to do solve this in Go?
Example input:
{
"panels": [
{
"alert": {
"executionErrorState": "alerting",
"notifications": [
{
"uid": "fRLbH_6Zk"
},
{
"uid": "8gamKl6Waz"
}
]
}
},
{
"alert": {
"executionErrorState": "alerting",
"notifications": [
{
"uid": "DqjrD_6Zk"
}
]
}
}
]
}
Desired output (all entries in 'alert.notifications' in 'panels' removed):
{
"panels": [
{
"alert": {
"executionErrorState": "alerting",
"notifications": []
}
},
{
"alert": {
"executionErrorState": "alerting",
"notifications": []
}
}
]
}
you can use read streams, to read objects one by one. Code will be unmarshall first object, but will have error on the next one. Its like approve of state that code dont read whole file, here example:
package main
import (
"encoding/json"
"fmt"
"log"
"strings"
)
func main() {
const jsonStream = `
[
{"Name": "Ed", "Text": "Knock knock."},
asdasd sadasd,
`
type Message struct {
Name, Text string
}
dec := json.NewDecoder(strings.NewReader(jsonStream))
// read open bracket
t, err := dec.Token()
if err != nil {
log.Fatal(err)
}
fmt.Printf("%T: %v\n", t, t)
// while the array contains values
for dec.More() {
var m Message
// decode an array value (Message)
err := dec.Decode(&m)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%v: %v\n", m.Name, m.Text)
}
// read closing bracket
t, err = dec.Token()
if err != nil {
log.Fatal(err)
}
fmt.Printf("%T: %v\n", t, t)
}
So I am trying to fetch the analytics of an app by pinging and endpoint. I make the GET request which is successfull (no errors there) but I am unable to decode the JSON
I need to to decode the following json into structs
{
"noResultSearches": {
"results": [
{
"count": 1,
"key": "\"note 9\""
},
{
"count": 1,
"key": "nokia"
}
]
},
"popularSearches": {
"results": [
{
"count": 4,
"key": "6"
},
{
"count": 2,
"key": "\"note 9\""
},
{
"count": 1,
"key": "nokia"
}
]
},
"searchVolume": {
"results": [
{
"count": 7,
"key": 1537401600000,
"key_as_string": "2018/09/20 00:00:00"
}
]
}
}
For which I am using the following structs
type analyticsResults struct {
Count int `json:"count"`
Key string `json:"key"`
}
type analyticsVolumeResults struct {
Count int `json:"count"`
Key int64 `json:"key"`
DateAsStr string `json:"key_as_string"`
}
type analyticsPopularSearches struct {
Results []analyticsResults `json:"results"`
}
type analyticsNoResultSearches struct {
Results []analyticsResults `json:"results"`
}
type analyticsSearchVolume struct {
Results []analyticsVolumeResults `json:"results"`
}
type overviewAnalyticsBody struct {
NoResultSearches analyticsNoResultSearches `json:"noResultSearches"`
PopularSearches analyticsPopularSearches `json:"popularSearches"`
SearchVolume analyticsSearchVolume `json:"searchVolume"`
}
I make a GET request to an endpoint and then use the response body to decode the json but I get an error. Following is a part of the code that stays in my ShowAnalytics function
func ShowAppAnalytics(app string) error {
spinner.StartText("Fetching app analytics")
defer spinner.Stop()
fmt.Println()
req, err := http.NewRequest("GET", "<some-endpoint>", nil)
if err != nil {
return err
}
resp, err := session.SendRequest(req)
if err != nil {
return err
}
spinner.Stop()
var res overviewAnalyticsBody
dec := json.NewDecoder(resp.Body)
err = dec.Decode(&res)
if err != nil {
return err
}
fmt.Println(res)
return nil
}
json: cannot unmarshal array into Go struct field
overviewAnalyticsBody.noResultSearches of type
app.analyticsNoResultSearches
What am I doing wrong here? Why do I get this error?
EDIT: After you edited, your current code works as-is. Check it out here: Go Playground.
Original answer follows.
There is some inconsistency between the code you posted and the error you get.
I tried it on the Go Playground (here's your version), and I get the following error:
json: cannot unmarshal number into Go struct field analyticsVolumeResults.key of type string
We get this error because in the JSON searchVolume.results.key is a number:
"key": 1537401600000,
And you used string in the Go model:
Key string `json:"key"`
If we change it to int64:
Key int64 `json:"key"`
It works, and prints (try it on the Go Playground):
{{[{1 "note 9"} {1 nokia}]} {[{4 6} {2 "note 9"} {1 nokia}]} {[{7 1537401600000 2018/09/20 00:00:00}]}}
If that key may sometimes be a number and sometimes a string, you may also use json.Number in the Go model:
Key json.Number `json:"key"`
I am trying to unmarshal JSON of the following format:
{
"fixedString": {
"uselessStuff": {},
"alsoUseless": {},
"dynamicField": [
{ "name": "jack" }
],
"dynamicToo": [
{ "name": "jill" }
]
}
}
I would like to drop the fields "uselessStuff" and "alsoUseless", and get everything else. The other keys are user-defined and can take any value.
I can remove the fields I don't want using a custom UnmarshalJSON (based on this answer), but I have a feeling that this is unnecessarily complicated:
package main
import (
"encoding/json"
"fmt"
)
type Response struct {
Things map[string]interface{} `json:"fixedString"`
}
type _R Response
func (f *Response) UnmarshalJSON(bs []byte) (err error) {
foo := _R{}
if err = json.Unmarshal(bs, &foo); err == nil {
*f = Response(foo)
}
delete(f.Things, "uselessStuff")
delete(f.Things, "alsoUseless")
return err
}
func main() {
j := []byte(`{ "fixedString": { "uselessStuff": {}, "alsoUseless": {}, "dynamicField": [ { "name": "jack" } ], "dynamicToo": [ { "name": "jill" } ] } }`)
var r Response
err := json.Unmarshal(j, &r)
if err != nil {
panic(err.Error())
}
for x, y := range r.Things {
fmt.Println(x, y)
}
}
Is there a way to ignore those two keys using annotations, rather than deleting them in a custom UnmarshalJSON function (and having to add the extra type alias _R to avoid a stack overflow)?
You could remove your "uselessStuff" and "alsoUseless" from the map and use them as unexported (lowercase) fields in your struct. Most likely not interface{}
json package ignores unexported fields
type Response struct {
Things map[string]interface{} `json:"fixedString"`
uselessStuff interface{}
alsoUseless interface{}
}