Unmarshalling JSON with Duplicate Fields - json

I'm still learning the go language, but I've been trying to find some practical things to work on to get a better handle on it. Currently, I'm trying to build a simple program that goes to a youtube channel and returns some information by taking the public JSON and unmarshalling it.
Thus far I've tried making a completely custom struct that only has a few fields in it, but that doesn't seem to pull in any values. I've also tried using tools like https://mholt.github.io/json-to-go/ and getting the "real" struct that way. The issue with that method is there are numerous duplicates and I don't know enough to really assess how to tackle that.
This is an example JSON (I apologize for its size) https://pastebin.com/6u0b39tU
This is the struct that I get from the above tool: https://pastebin.com/3ZCu96st
the basic pattern of code I've tried is:
jsonFile, err := os.Open("test.json")
if err != nil {
fmt.Println("Couldn't open file", err)
}
defer jsonFile.Close()
bytes, _ := ioutil.ReadAll(jsonFile)
var channel Autogenerated
json.Unmarshal(bytes, &Autogenerated)
if err != nil {
fmt.Println("Failed to Unmarshal", err)
}
fmt.Println(channel.Fieldname)
Any feedback on the correct approach for how to handle something like this would be great. I get the feeling I'm just completely missing something.

In your code, you are not unmarshaling into the channel variable. Furthermore, you can optimize your code to not use ReadAll. Also, don't forget to check for errors (all errors).
Here is an improvement to your code.
jsonFile, err := os.Open("test.json")
if err != nil {
log.Fatalf("could not open file: %v", err)
}
defer jsonFile.Close()
var channel Autogenerated
if err := json.NewDecoder(jsonFile).Decode(&channel); err != nil {
log.Fatalf("failed to parse json: %v", err)
}
fmt.Println(channel.Fieldname)
Notice how a reference to channel is passed to Decode.

Related

Do I need to add nil check after decoding a pointer value with the json package?

I have been writing in Go for a long time and recently while rewriting the code I came across a strange thing. I did a couple of tests and if request == nil check never worked. Previously, I was always afraid of getting a nil pointer exception, and so I inserted checks everywhere. But in this case, the json decode error handler seems to cover all cases.
var (
request *models.Owner
err error
)
err = json.NewDecoder(r.Body).Decode(&request)
if err != nil {
render.Render(w, r, response.ErrInvalidRequest(err))
return
}
if request == nil {
render.Render(w, r, response.ErrInvalidRequest("request is nil"))
return
}
if request == nil is it possible to catch this? Perhaps this check is unnecessary, and if I remove this check in my project, the code will become cleaner.
It is possible that nil error will be returned and request will still be nil, but only if the input JSON is the JSON null value.
For example:
type Owners struct {
Name string
}
var request *Owners
if err := json.Unmarshal([]byte("null"), &request); err != nil {
panic(err)
}
fmt.Println(request == nil)
fmt.Println(request)
This will output (try it on the Go Playground):
true
<nil>
This is documented at json.Unmarshal():
To unmarshal JSON into a pointer, Unmarshal first handles the case of the JSON being the JSON literal null. In that case, Unmarshal sets the pointer to nil. Otherwise, Unmarshal unmarshals the JSON into the value pointed at by the pointer. If the pointer is nil, Unmarshal allocates a new value for it to point to.

Json Marshalling straight to stdout

I am trying to learn Golang and while doing that I wrote below code (part of bigger self learning project) and took the code review from strangers, one of the comment was, "you could have marshalled this straight to stdout, instead of marshalling to heap, then converting to string and then streaming it to stdout"
I have gone through the documentation of encoding/json package and io but not able to piece together the change which is required.
Any pointers or help would be great.
// Marshal the struct with proper tab indent so it can be readable
b, err := json.MarshalIndent(res, "", " ")
if err != nil {
log.Fatal(errors.Wrap(err, "error marshaling response data"))
}
// Print the output to the stdout
fmt.Fprint(os.Stdout, string(b))
EDIT
I just found below code sample in documentation:
var out bytes.Buffer
json.Indent(&out, b, "=", "\t")
out.WriteTo(os.Stdout)
But again it writes to heap first before writing to stdout. It does remove one step of converting it to string though.
Create and use a json.Encoder directed to os.Stdout. json.NewEncoder() accepts any io.Writer as its destination.
res := map[string]interface{}{
"one": 1,
"two": "twotwo",
}
if err := json.NewEncoder(os.Stdout).Encode(res); err != nil {
panic(err)
}
This will output (directly to Stdout):
{"one":1,"two":"twotwo"}
If you want to set indentation, use its Encoder.SetIndent() method:
enc := json.NewEncoder(os.Stdout)
enc.SetIndent("", " ")
if err := enc.Encode(res); err != nil {
panic(err)
}
This will output:
{
"one": 1,
"two": "twotwo"
}
Try the examples on the Go Playground.

Go Lang Help - Accessing Array/Slice of interfaces

I'm trying to decode dynamic/random JSON responses in GO, with nested data
body, _ := ioutil.ReadAll(response.Body)
resp := make(map[string]interface{})
err = json.Unmarshal(body, &resp)
fmt.Printf("BODY: %T<\n", body)
fmt.Printf("BODY: %s<\n", body)
fmt.Printf("RESP: %s<\n", resp)
fmt.Printf("RESP: %T<\n", resp)
fmt.Printf("RESP[results]: %T<\n", resp["results"])
fmt.Printf("RESP[results]: %s<\n", resp["results"])
body is the JSON result from the HTTP server and I unmarshall it and the result looks to be a slice of bytes.
BODY: []uint8
BODY: {"results":[{"code":500.0,"errors":["Configuration file 'c2-web-2.conf' already exists."],"status":"Object could not be created."}]}
So I unmarshall it into resp and that works as expected.
RESP: map[string]interface {}
RESP: map[results:[map[code:%!s(float64=500) errors:[Configuration file 'c2-web-2.conf' already exists.] status:Object could not be created.]]]<
I'm able to access the map with the key results.
RESP[results]: []interface {}
RESP[results]: [map[code:%!s(float64=500) errors:[Configuration file 'conf.d/hosts/c2-web-2.conf' already exists.] status:Object could not be created.]]<
Now what i want to access it the "code", "errors" and "status" which is in resp["results"] This looks like an array or slice and I've tried indexing it but I get the error at compile time
./create_host.go:62: invalid operation: resp["results"][0] (type interface {} does not support indexing)
I've done a lot of googling, tried unmarshalling the data within resp["results"] etc, but after a few days I have not made much progress.
How should I access the map which seems to be a member of an array? The data structure is not guaranteed so I can't create a structure and unmarshall into that.
Thanks
A co-worker provided the code fragement below which made it possible to access the map entries I was looking for.
respBody, _ := ioutil.ReadAll(response.Body)
var rsp interface{}
if err := json.Unmarshal(respBody, &rsp); err != nil {
log.Fatal(err)
}
resultMap := rsp.(map[string]interface{})["results"].([]interface{})[0].(map[string]interface{})
fmt.Printf("test: %s<\n", resultMap["errors"] )
test: [Configuration file 'c2-web-2.conf' already exists.]<
I believe you need to do a type assertion. You have an interface{}, but you need some sort of slice to index into. Try resp["results"].([]interface{})[0]? (Sorry, haven't had a chance to test this myself.)

Golang - Parsing JSON string arrays from Twitch TV RESTful service

I've been working on parsing a JSON object that I retrieve through an HTTP GET request using Go's built in HTTP library. I initially tried using the default JSON library in Go in order to do this, but I was having a difficult time (I am a novice in Go still). I eventually resorted to using a different library and had little trouble after that, as shown below:
package main
import (
"github.com/antonholmquist/jason"
"fmt"
"net/http"
)
func main() {
resp, err := http.Get("http://tmi.twitch.tv/group/user/deernadia/chatters")
if nil != err {
panic(err)
}
defer resp.Body.Close()
body, err := jason.NewObjectFromReader(resp.Body)
chatters, err := body.GetObject("chatters")
if nil != err {
panic(err)
}
moderators, err := chatters.GetStringArray("moderators")
if nil != err {
panic(err)
}
for _, moderator := range moderators {
fmt.Println(moderator)
}
}
Where github.com/antonholmquist/jason corresponds to the custom JSON library I used.
This code produces something similar to the following output when run in a Linux shell (the RESTful service will update about every 30 seconds or so, which means the values in the JSON object will potentially change):
antwan250
bbrock89
boxception22
cmnights
deernadia
fartfalcon
fijibot
foggythought
fulc_
h_ov
iceydefeat
kingbobtheking
lospollogne
nightbot
nosleeptv
octaviuskhan
pateyy
phosphyg
poisonyvie
shevek18
trox94
trox_bot
uggasmesh
urbanelf
walmartslayer
wift3
And the raw JSON looks similar to this (with some of the users removed for brevity):
{
"_links": {},
"chatter_count": 469,
"chatters": {
"moderators": [
"antwan250",
"bbrock89",
"boxception22",
"cmnights",
"deernadia",
"fartfalcon",
"fijibot",
"foggythought",
"fulc_",
"h_ov",
"iceydefeat",
"kingbobtheking",
"lospollogne",
"nightbot",
"nosleeptv",
"octaviuskhan",
"pateyy",
"phosphyg",
"poisonyvie",
"shevek18",
"trox94",
"trox_bot",
"uggasmesh",
"urbanelf",
"walmartslayer",
"wift3"
],
"staff": [
"tnose"
],
"admins": [],
"global_mods": [],
"viewers": [
"03xuxu30",
"0dominic0",
"3389942",
"812mfk",
"910dan",
"aaradabooti",
"admiralackbar99",
"adrian97lol",
"aequitaso_o",
"aethiris",
"afropigeon",
"ahhhmong",
"aizaix",
"aka_magosh",
"akitoalexander",
"alex5761",
"allenhei",
"allou_fun_park",
"amilton_tkm",
"... more users that I removed...",
"zachn17",
"zero_x1",
"zigslip",
"ziirbryad",
"zonato83",
"zorr03body",
"zourtv"
]
}
}
As I said before, I'm using a custom library hosted on Github in order to accomplish what I needed, but for the sake of learning, I'm curious... how would I accomplish this same thing using Go's built in JSON library?
To be clear, what I'd like to do is be able to harvest the users from each JSON array embedded within the JSON object returned from the HTTP GET request. I'd also like to be able to get the list of viewers, admins, global moderators, etc., in the same way, but I figured that if I can see the moderator example using the default Go library, then reproducing that code for the other user types will be trivial.
Thank you in advance!
If you want to unmarshal moderators only, use the following:
var v struct {
Chatters struct {
Moderators []string
}
}
if err := json.Unmarshal(data, &v); err != nil {
// handle error
}
for _, mod := range v2.Chatters.Moderators {
fmt.Println(mod)
}
If you want to get all types of chatters, use the following:
var v struct {
Chatters map[string][]string
}
if err := json.Unmarshal(data, &v); err != nil {
handle error
}
for kind, users := range v1.Chatters {
for _, user := range users {
fmt.Println(kind, user)
}
}
run the code on the playground

goroutine channels over a for loop

My main function reads json from a file, unmarshals it into a struct, converts it into another struct type and spits out formatted JSON through stdout.
I'm trying to implement goroutines and channels to add concurrency to my for loop.
func main() {
muvMap := map[string]string{"male": "M", "female": "F"}
fileA, err := os.Open("serviceAfileultimate.json")
if err != nil {
panic(err)
}
defer fileA.Close()
data := make([]byte, 10000)
count, err := fileA.Read(data)
if err != nil {
panic(err)
}
dataBytes := data[:count]
var servicesA ServiceA
json.Unmarshal(dataBytes, &servicesA)
var servicesB = make([]ServiceB, servicesA.Count)
goChannels := make(chan ServiceB, servicesA.Count)
for i := 0; i < servicesA.Count; i++ {
go func() {
reflect.ValueOf(&servicesB[i]).Elem().FieldByName("Address").SetString(Merge(&servicesA.Users[i].Location))
reflect.ValueOf(&servicesB[i]).Elem().FieldByName("Date_Of_Birth").SetString(dateCopyTransform(servicesA.Users[i].Dob))
reflect.ValueOf(&servicesB[i]).Elem().FieldByName("Email").SetString(servicesA.Users[i].Email)
reflect.ValueOf(&servicesB[i]).Elem().FieldByName("Fullname").SetString(Merge(&servicesA.Users[i].Name))
reflect.ValueOf(&servicesB[i]).Elem().FieldByName("Gender").SetString(muvMap[servicesA.Users[i].Gender])
reflect.ValueOf(&servicesB[i]).Elem().FieldByName("Phone").SetString(servicesA.Users[i].Cell)
reflect.ValueOf(&servicesB[i]).Elem().FieldByName("Username").SetString(servicesA.Users[i].Username)
goChannels <- servicesB[i]
}()
}
for index := range goChannels {
json.NewEncoder(os.Stdout).Encode(index)
}
}
It compiles but is returning messages like:
goroutine 1 [chan receive]: main.main() C://.....go.94 +0x55b.
You're printing the channels info, not the data it contains. You don't want a loop, you just want to receive then print.
json := <-index
json.NewEncoder(os.Stdout).Encode(json)
Now I do I need to point out, that code is not going to block. If you want to keep reading until all work is done you need some kind of locking/coordination mechanism.
You'll often see things like
for {
select {
case json := <-jsonChannel:
// do stuff
case <-abort:
// get out of here
}
}
To deal with that. Also, just fyi you're initializing your channel with a default capacity (meaning it's a buffered channel) which is pretty odd. I'd recommend reviewing some tutorials on the topic cause overall your design needs some work actually be an improvement of non-concurrent implementations. Lastly you can find libraries to abstract some of this work for you and most people would probably recommend you do. Here's an example; https://github.com/lytics/squaredance