I've been working on parsing a JSON object that I retrieve through an HTTP GET request using Go's built in HTTP library. I initially tried using the default JSON library in Go in order to do this, but I was having a difficult time (I am a novice in Go still). I eventually resorted to using a different library and had little trouble after that, as shown below:
package main
import (
"github.com/antonholmquist/jason"
"fmt"
"net/http"
)
func main() {
resp, err := http.Get("http://tmi.twitch.tv/group/user/deernadia/chatters")
if nil != err {
panic(err)
}
defer resp.Body.Close()
body, err := jason.NewObjectFromReader(resp.Body)
chatters, err := body.GetObject("chatters")
if nil != err {
panic(err)
}
moderators, err := chatters.GetStringArray("moderators")
if nil != err {
panic(err)
}
for _, moderator := range moderators {
fmt.Println(moderator)
}
}
Where github.com/antonholmquist/jason corresponds to the custom JSON library I used.
This code produces something similar to the following output when run in a Linux shell (the RESTful service will update about every 30 seconds or so, which means the values in the JSON object will potentially change):
antwan250
bbrock89
boxception22
cmnights
deernadia
fartfalcon
fijibot
foggythought
fulc_
h_ov
iceydefeat
kingbobtheking
lospollogne
nightbot
nosleeptv
octaviuskhan
pateyy
phosphyg
poisonyvie
shevek18
trox94
trox_bot
uggasmesh
urbanelf
walmartslayer
wift3
And the raw JSON looks similar to this (with some of the users removed for brevity):
{
"_links": {},
"chatter_count": 469,
"chatters": {
"moderators": [
"antwan250",
"bbrock89",
"boxception22",
"cmnights",
"deernadia",
"fartfalcon",
"fijibot",
"foggythought",
"fulc_",
"h_ov",
"iceydefeat",
"kingbobtheking",
"lospollogne",
"nightbot",
"nosleeptv",
"octaviuskhan",
"pateyy",
"phosphyg",
"poisonyvie",
"shevek18",
"trox94",
"trox_bot",
"uggasmesh",
"urbanelf",
"walmartslayer",
"wift3"
],
"staff": [
"tnose"
],
"admins": [],
"global_mods": [],
"viewers": [
"03xuxu30",
"0dominic0",
"3389942",
"812mfk",
"910dan",
"aaradabooti",
"admiralackbar99",
"adrian97lol",
"aequitaso_o",
"aethiris",
"afropigeon",
"ahhhmong",
"aizaix",
"aka_magosh",
"akitoalexander",
"alex5761",
"allenhei",
"allou_fun_park",
"amilton_tkm",
"... more users that I removed...",
"zachn17",
"zero_x1",
"zigslip",
"ziirbryad",
"zonato83",
"zorr03body",
"zourtv"
]
}
}
As I said before, I'm using a custom library hosted on Github in order to accomplish what I needed, but for the sake of learning, I'm curious... how would I accomplish this same thing using Go's built in JSON library?
To be clear, what I'd like to do is be able to harvest the users from each JSON array embedded within the JSON object returned from the HTTP GET request. I'd also like to be able to get the list of viewers, admins, global moderators, etc., in the same way, but I figured that if I can see the moderator example using the default Go library, then reproducing that code for the other user types will be trivial.
Thank you in advance!
If you want to unmarshal moderators only, use the following:
var v struct {
Chatters struct {
Moderators []string
}
}
if err := json.Unmarshal(data, &v); err != nil {
// handle error
}
for _, mod := range v2.Chatters.Moderators {
fmt.Println(mod)
}
If you want to get all types of chatters, use the following:
var v struct {
Chatters map[string][]string
}
if err := json.Unmarshal(data, &v); err != nil {
handle error
}
for kind, users := range v1.Chatters {
for _, user := range users {
fmt.Println(kind, user)
}
}
run the code on the playground
Related
I'm still learning the go language, but I've been trying to find some practical things to work on to get a better handle on it. Currently, I'm trying to build a simple program that goes to a youtube channel and returns some information by taking the public JSON and unmarshalling it.
Thus far I've tried making a completely custom struct that only has a few fields in it, but that doesn't seem to pull in any values. I've also tried using tools like https://mholt.github.io/json-to-go/ and getting the "real" struct that way. The issue with that method is there are numerous duplicates and I don't know enough to really assess how to tackle that.
This is an example JSON (I apologize for its size) https://pastebin.com/6u0b39tU
This is the struct that I get from the above tool: https://pastebin.com/3ZCu96st
the basic pattern of code I've tried is:
jsonFile, err := os.Open("test.json")
if err != nil {
fmt.Println("Couldn't open file", err)
}
defer jsonFile.Close()
bytes, _ := ioutil.ReadAll(jsonFile)
var channel Autogenerated
json.Unmarshal(bytes, &Autogenerated)
if err != nil {
fmt.Println("Failed to Unmarshal", err)
}
fmt.Println(channel.Fieldname)
Any feedback on the correct approach for how to handle something like this would be great. I get the feeling I'm just completely missing something.
In your code, you are not unmarshaling into the channel variable. Furthermore, you can optimize your code to not use ReadAll. Also, don't forget to check for errors (all errors).
Here is an improvement to your code.
jsonFile, err := os.Open("test.json")
if err != nil {
log.Fatalf("could not open file: %v", err)
}
defer jsonFile.Close()
var channel Autogenerated
if err := json.NewDecoder(jsonFile).Decode(&channel); err != nil {
log.Fatalf("failed to parse json: %v", err)
}
fmt.Println(channel.Fieldname)
Notice how a reference to channel is passed to Decode.
I am having a bit of an issue. So I am writing a tool in go that works with some json files.
The way the tool works is the devops member who is using it is supposed to upload their json file into the specified folder within the project and then that json file is used from there to deploy an api in api-gateway (the json is actually a swagger with extensions but that isn't particularly important to my question)
The issue I am having is I need to update ONE line in the json. Each file passed in will be different, but it is guaranteed to have a url in the same spot every time, just due to the nature of the project. I need to update this url in an automated fashion.
Due to the fact the json files are different, setting up hard coded structs and unmarshalling in order to edit is out of the question. The objective is for the devops members to not even have to go into the code, but rather just to deploy their files, which is the reason I was hoping for this to be automated.
So far my research has yielded nothing. It appears that Go only supports editing json if it is first unmarshaled into structs (see Modifying JSON file using Golang). Is there a way to edit without the structs if i know for a fact what I am looking for will always be available within the json, despite each file being different?
This is only my first month using go, so there may be a simple solution. I have seen some mention of scanners from the megajson library, but I cannot seem to get that to work either
{
"paths": {
"/account": {
"post": {
"something": "body",
"api": {
"uri": "http://thisiswhereineedtoedit.com"
}
}
}
}
}
Unmarshal to interface{}. Walk down nested objects to find the object with the value to set. Set the value. Marshal back to JSON.
var root interface{}
if err := json.Unmarshal(d, &root); err != nil {
log.Fatal(err)
}
// Walk down path to target object.
v := root
var path = []string{"paths", "/account", "post", "api"}
for i, k := range path {
m, ok := v.(map[string]interface{})
if !ok {
log.Fatalf("map not found at %s", strings.Join(path[:i+1], ", "))
}
v, ok = m[k]
if !ok {
log.Fatalf("value not found at %s", strings.Join(path[:i+1], ", "))
}
}
// Set value in the target object.
m, ok := v.(map[string]interface{})
if !ok {
log.Fatalf("map not found at %s", strings.Join(path, ", "))
}
m["uri"] = "the new URI"
// Marshal back to JSON. Variable d is []byte with the JSON
d, err := json.Marshal(root)
if err != nil {
log.Fatal(err)
}
Replace calls to log.Fatal with whatever error handling is appropriate for your application.
playground example
One way you can solve this is by reading the file and changing it.
Lets say you have the file as you mentioned:
example.json
{
"paths": {
"/account": {
"post": {
"something": "body",
"api": {
"uri": "http://thisiswhereineedtoedit.com"
}
}
}
}
}
And we want to change the line with "uri" in it.
You should be more specific then I was in this example, make a placeholder or something - to avoid changing the wrong line.
You can use a small program that would look something like this:
package main
import (
"io/ioutil"
"log"
"strings"
)
func main() {
file := "./example.json"
url := "\"uri\": \"supreme-uri\""
// Read the file
input, err := ioutil.ReadFile(file)
if err != nil {
log.Fatalln(err)
}
// Split it into lines
lines := strings.Split(string(input), "\n")
// Find the line that contains our "placeholder" / "uri"
for i, line := range lines {
if strings.Contains(line, "\"uri\":") {
// Replace the line
lines[i] = "\"uri\": " + url
}
}
// Join lines and write to file
output := strings.Join(lines, "\n")
err = ioutil.WriteFile(file, []byte(output), 0644)
if err != nil {
log.Fatalln(err)
}
}
And after running the program our example.json file now looks like this:
{
"paths": {
"/account": {
"post": {
"something": "body",
"api": {
"uri": "supreme-uri"
}
}
}
}
}
Hope you find this solution useful, Good luck! :]
You can try filepath pkg from cross-plane runtime. You specify the JSON path and get or set the result you want like the example in the above link.
How can I convert grpc/protobuf3 message to JSON where the enum is represented as string?
For example, the protobuf message:
enum Level {
WARNING = 0;
FATAL = 1;
SEVERE = 2;
...
}
message Http {
string message = 1;
Level level = 2;
}
Is converted by:
j, _ := json.MarshalIndent(protoMessage, "", "\t")
To:
{
"message": "Hello world!",
"level": 2,
}
I wish to get:
{
"message": "Hello world!",
"level": "SEVERE",
}
Thanks
I found out that I should use the protobuf/jsonpb package and not the standard json package.
so:
j, _ := json.MarshalIndent(protoMessage, "", "\t")
Should be:
m := jsonpb.Marshaler{}
result, _ := m.MarshalToString(protoMessage)
Update
As noted bellow, jsonpb is depricated and the new solution is to use protojson
I found some of these modules (jsonpb) to be deprecated. What worked for me was the google encoding version:
import "google.golang.org/protobuf/encoding/protojson"
jsonString := protojson.Format(protoMessage)
Level is not a string though, it is an emum. There are really only two choices I see.
Write a custom marshaller that does this for you
Generate code that does this for you.
For #2, gogoprotobuf has an extension (still marked as experimental) that let's you do exactly this:
https://godoc.org/github.com/gogo/protobuf/plugin/enumstringer and
https://github.com/gogo/protobuf/blob/master/extensions.md
For my use case, I wanted to write it to a file. Using the most recent packages as of this date, this was as close to a regular encoding/json marshal as I could get.
I used the google.golang.org/protobuf/encoding/protojson package and the .ProtoReflect().Interface() methods of my protocol buffer data structure.
package main
import (
"io/ioutil"
"log"
"google.golang.org/protobuf/encoding/protojson"
"myproject/proto"
)
func main() {
myProtoStruct := proto.MyType{}
data, err := protojson.Marshal(myProtoStruct.ProtoReflect().Interface())
if err != nil {
log.Fatalf("Failed to JSON marhsal protobuf.\nError: %s", err.Error())
}
err = ioutil.WriteFile("my.proto.dat", data, 0600)
if err != nil {
log.Fatalf("Failed to write protobuf data to file.\nError: %s", err.Error())
}
log.Println("Written to file.")
}
When it comes to serialize the json object, this would be helpful.
var msg bytes.Buffer
m := jsonpb.Marshaler{}
err := m.Marshal(&msg, event)
msg.Bytes() converts msg to byte stream.
this is a very old question, but I will post my solution, maybe someone on the future will also face the same issue.
firstly, jsonpb is deprecated.
secondly, when you use protojson, you can set your options, which will affect the outcome JSON. In your case, the option UseEnumNumbers says how you want to marshall enums, strings or ints. Like so:
arr, err := protojson.MarshalOptions{
UseEnumNumbers: true, // uses enums as int, not as strings
}.Marshal(a)
I am new to golang and json and currently struggle to parse the json out from a system.
I've read a couple of blog posts on dynamic json in go and also tried the tools like json2GoStructs
Parsing my json file with this tools just gave me a huge structs which I found a bit messy. Also I had no idea how to get the info im interested in.
So, here are my problems:
How do I get to the info I am interested in?
What is the best approach to parse complex json?
I am only interested into the following 3 json fields:
Name
Guid
Task -> Property -> Name: Error
I'm thankful for every tip, code snippet or explanation!
This is what I got so far (mostly from a tutorial):
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
)
func checkErr(err error) {
if err != nil {
panic(err)
}
}
func readFile(filePath string) []byte {
data, err := ioutil.ReadFile(filePath)
checkErr(err)
return data
}
func main() {
path := "/Users/andi/Documents/tmp/wfsJob.json"
data := readFile(path)
var f interface{}
err := json.Unmarshal(data, &f)
checkErr(err)
m := f.(map[string]interface{})
for k, v := range m {
switch vv := v.(type) {
case string:
fmt.Println(k, "is string", vv)
case int:
fmt.Println(k, "is int", vv)
case []interface{}:
fmt.Println(k, "is an array:")
for i, u := range vv {
fmt.Println(i, u)
}
default:
fmt.Println(k, "is of a type I don't know how to handle")
}
}
}
I can offer you this easy way to using JSON in Golang. With this tool you don't need to parse the whole json file, and you can use it without struct.
Gjson is a great solution for fetching a few fields from JSON string. But it may become slow when many (more than 2) fields must be fetched from distinct parts of the JSON, since it re-parses the JSON on each Get call. Additionally, it requires calling gjson.Valid for validating the incoming JSON, since other methods assume the caller provides valid JSON.
There is an alternative package - fastjson. Like gsjon, it is fast and has nice API. Unlike gjson it validates the input JSON and works faster when many unrelated fields must be obtained from the JSON. Here is a sample code for obtaining fields from the original question:
var p fastjson.Parser
v, err := p.ParseBytes(data)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Name: %s", v.GetStringBytes("Name"))
fmt.Printf("Guid: %s", v.GetStringBytes("Guid"))
fmt.Printf("Error: %s", v.GetStringBytes("Task", "Property", "Name"))
My main function reads json from a file, unmarshals it into a struct, converts it into another struct type and spits out formatted JSON through stdout.
I'm trying to implement goroutines and channels to add concurrency to my for loop.
func main() {
muvMap := map[string]string{"male": "M", "female": "F"}
fileA, err := os.Open("serviceAfileultimate.json")
if err != nil {
panic(err)
}
defer fileA.Close()
data := make([]byte, 10000)
count, err := fileA.Read(data)
if err != nil {
panic(err)
}
dataBytes := data[:count]
var servicesA ServiceA
json.Unmarshal(dataBytes, &servicesA)
var servicesB = make([]ServiceB, servicesA.Count)
goChannels := make(chan ServiceB, servicesA.Count)
for i := 0; i < servicesA.Count; i++ {
go func() {
reflect.ValueOf(&servicesB[i]).Elem().FieldByName("Address").SetString(Merge(&servicesA.Users[i].Location))
reflect.ValueOf(&servicesB[i]).Elem().FieldByName("Date_Of_Birth").SetString(dateCopyTransform(servicesA.Users[i].Dob))
reflect.ValueOf(&servicesB[i]).Elem().FieldByName("Email").SetString(servicesA.Users[i].Email)
reflect.ValueOf(&servicesB[i]).Elem().FieldByName("Fullname").SetString(Merge(&servicesA.Users[i].Name))
reflect.ValueOf(&servicesB[i]).Elem().FieldByName("Gender").SetString(muvMap[servicesA.Users[i].Gender])
reflect.ValueOf(&servicesB[i]).Elem().FieldByName("Phone").SetString(servicesA.Users[i].Cell)
reflect.ValueOf(&servicesB[i]).Elem().FieldByName("Username").SetString(servicesA.Users[i].Username)
goChannels <- servicesB[i]
}()
}
for index := range goChannels {
json.NewEncoder(os.Stdout).Encode(index)
}
}
It compiles but is returning messages like:
goroutine 1 [chan receive]: main.main() C://.....go.94 +0x55b.
You're printing the channels info, not the data it contains. You don't want a loop, you just want to receive then print.
json := <-index
json.NewEncoder(os.Stdout).Encode(json)
Now I do I need to point out, that code is not going to block. If you want to keep reading until all work is done you need some kind of locking/coordination mechanism.
You'll often see things like
for {
select {
case json := <-jsonChannel:
// do stuff
case <-abort:
// get out of here
}
}
To deal with that. Also, just fyi you're initializing your channel with a default capacity (meaning it's a buffered channel) which is pretty odd. I'd recommend reviewing some tutorials on the topic cause overall your design needs some work actually be an improvement of non-concurrent implementations. Lastly you can find libraries to abstract some of this work for you and most people would probably recommend you do. Here's an example; https://github.com/lytics/squaredance