goroutine channels over a for loop - json

My main function reads json from a file, unmarshals it into a struct, converts it into another struct type and spits out formatted JSON through stdout.
I'm trying to implement goroutines and channels to add concurrency to my for loop.
func main() {
muvMap := map[string]string{"male": "M", "female": "F"}
fileA, err := os.Open("serviceAfileultimate.json")
if err != nil {
panic(err)
}
defer fileA.Close()
data := make([]byte, 10000)
count, err := fileA.Read(data)
if err != nil {
panic(err)
}
dataBytes := data[:count]
var servicesA ServiceA
json.Unmarshal(dataBytes, &servicesA)
var servicesB = make([]ServiceB, servicesA.Count)
goChannels := make(chan ServiceB, servicesA.Count)
for i := 0; i < servicesA.Count; i++ {
go func() {
reflect.ValueOf(&servicesB[i]).Elem().FieldByName("Address").SetString(Merge(&servicesA.Users[i].Location))
reflect.ValueOf(&servicesB[i]).Elem().FieldByName("Date_Of_Birth").SetString(dateCopyTransform(servicesA.Users[i].Dob))
reflect.ValueOf(&servicesB[i]).Elem().FieldByName("Email").SetString(servicesA.Users[i].Email)
reflect.ValueOf(&servicesB[i]).Elem().FieldByName("Fullname").SetString(Merge(&servicesA.Users[i].Name))
reflect.ValueOf(&servicesB[i]).Elem().FieldByName("Gender").SetString(muvMap[servicesA.Users[i].Gender])
reflect.ValueOf(&servicesB[i]).Elem().FieldByName("Phone").SetString(servicesA.Users[i].Cell)
reflect.ValueOf(&servicesB[i]).Elem().FieldByName("Username").SetString(servicesA.Users[i].Username)
goChannels <- servicesB[i]
}()
}
for index := range goChannels {
json.NewEncoder(os.Stdout).Encode(index)
}
}
It compiles but is returning messages like:
goroutine 1 [chan receive]: main.main() C://.....go.94 +0x55b.

You're printing the channels info, not the data it contains. You don't want a loop, you just want to receive then print.
json := <-index
json.NewEncoder(os.Stdout).Encode(json)
Now I do I need to point out, that code is not going to block. If you want to keep reading until all work is done you need some kind of locking/coordination mechanism.
You'll often see things like
for {
select {
case json := <-jsonChannel:
// do stuff
case <-abort:
// get out of here
}
}
To deal with that. Also, just fyi you're initializing your channel with a default capacity (meaning it's a buffered channel) which is pretty odd. I'd recommend reviewing some tutorials on the topic cause overall your design needs some work actually be an improvement of non-concurrent implementations. Lastly you can find libraries to abstract some of this work for you and most people would probably recommend you do. Here's an example; https://github.com/lytics/squaredance

Related

json.Unmarshal file data works but json.NewDecoder().Decode() does not

The following correctly unmarshals the struct:
func foo() {
d, err := os.ReadFile("file.json")
var t T
if err := json.Unmarshal(d, &t); err != nil {
panic(err)
}
}
but this doesn't work and throws a bunch of a classic JSON parsing errors i.e. EOF, unexpected token 't', etc.
func foo() {
f, err := os.Open("file.json")
var t T
if err := json.NewDecoder(f).Decode(&t); err != nil {
panic(err)
}
}
Any idea why? The os.File or []byte is used in two goroutines at once, and the JSON is of the following structure (with some fields omitted):
{
"data": [
{
"field": "stuff",
"num": 123,
},
...
]
}
The os.File or []byte is used in two goroutines at once...
That's the issue. os.File has an internal file pointer, a position where the next read happens. If 2 unrelated entities keep reading from it, they will likely not read overlapping data. Bytes read by the first entity will not be repeated for the second.
Also, os.File is not safe for concurrent use (the documentation doesn't state explicitly that it's safe): calling its methods from multiple, concurrent goroutines may result in a data race.
When you pass a []byte to multiple functions / goroutines which read "from it", there's no shared pointer or index variable. Each function / goroutine will maintain its index, separately, and reading a variable from multiple goroutines is OK (which in this case would be the (fields of the) slice header and the slice elements).

Passing a csv.NewWriter() to another func in Golang to Write to file Asynchronously

I am making API calls (potentially thousands in a single job) and as they return and complete, I'd like to be able to write them to a single shared file (say CSV for simplicity) instead of waiting for all of them to complete before writing.
How could I share a single csv.Writer() in a way that effectively writes to a single file shared by many threads. This may be too daunting of a task, but I was curious if there was a way to go about it.
package main
import (
"encoding/csv"
"os"
)
type Row struct {
Field1 string
Field2 string
}
func main () {
file, _ := os.Create("file.csv")
w := csv.NewWriter(file)
// Some operations to create a slice of Row structs that will contain the rows
// To write
var rowsToWrite []Row
// Now lets iterate over and write to file
// Ideally, I'd like to do this in a goroutine but not entirely sure about thread safe writes
for _, r := range rowsToWrite {
go func(row, writer) {
err := writeToFile(row, writer)
if err != nil {
// Handle error
}
}(r, w)
}
}
func writeToFile(row Row, writer ???) error {
// Use the shared writer to maintain where I am at in the file so I can append to the CSV
if err := w.Write(row); err != nil {
return err
}
return nil
}
Lots of back and forth on this one for me 🙂
I originally thought you could use the Write() method on a csv.Writer in a goroutine, but there are issues when the buffer flushes to disk as the buffer is being written to... not exactly sure.
Anyways, to get back to what you were originally asking for...
Still using the same setup to download Todo objects from https://jsonplaceholder.typicode.com, as an example:
type Todo struct {
UserID int `json:"userId"`
ID int `json:"id"`
Title string `json:"title"`
Completed bool `json:"completed"`
}
// toRecord converts Todo struct to []string, for writing to CSV.
func (t Todo) toRecord() []string {
userID := strconv.Itoa(t.UserID)
id := strconv.Itoa(t.ID)
completed := strconv.FormatBool(t.Completed)
return []string{userID, id, t.Title, completed}
}
// getTodo gets endpoint and unmarshalls the response JSON into todo.
func getTodo(endpoint string) (todo Todo) {
resp, err := http.Get(endpoint)
if err != nil {
log.Println("error:", err)
}
defer resp.Body.Close()
json.NewDecoder(resp.Body).Decode(&todo)
return
}
The following:
Will start one "parent" goroutine to start filling the todos channel:
inside that routine, goroutines will be started for each HTTP request and will send the response Todo on todos
the parent will wait till all the request routines are done
when they're done, the parent will close the todos channel
Meanwhile, main has moved on and is ranging over todos, picking a Todo off one-at-a-time and writing it to the CSV.
When the original, "parent" goroutine finally closes todos, the for-loop will break, the writer does a final Flush(), and the program will complete.
func main() {
todos := make(chan Todo)
go func() {
const nAPICalls = 200
var wg sync.WaitGroup
wg.Add(nAPICalls)
for i := 0; i < nAPICalls; i++ {
s := fmt.Sprintf("https://jsonplaceholder.typicode.com/todos/%d", i+1)
go func(x string) {
todos <- getTodo(x)
wg.Done()
}(s)
}
wg.Wait()
close(todos)
}()
w := csv.NewWriter(os.Stdout)
w.Write([]string{"UserID", "ID", "Title", "Completed"})
for todo := range todos {
w.Write(todo.toRecord())
}
w.Flush()
}
I would (personally) not have the same file open for writing at two separate points in the code. Depending on how the OS handles buffered writes, etc., you can end up with "interesting" things happening.
Given how you've described your goals, one might do something like (this is off the top of my head and not rigorously tested):
Create a channel to queue blocks of text (I assume) to be written - make(chan []byte, depth) - depth could be tuneable based on some tests you'd run, presumably.
Have a goroutine open a filehandle for writing on your file, then read from that queueing channel, writing whatever it gets from the channel to that file
you could then have n goroutines writing to the queueing channel, and as long as you don't exceed the channel capacity (outrun your ability to write), you might never need to worry about locks.
If you did want to use locks, then you'd need a sync.Mutex shared between the goroutines responsible for enqueueing.
Season to taste, obviously.

Unmarshalling JSON with Duplicate Fields

I'm still learning the go language, but I've been trying to find some practical things to work on to get a better handle on it. Currently, I'm trying to build a simple program that goes to a youtube channel and returns some information by taking the public JSON and unmarshalling it.
Thus far I've tried making a completely custom struct that only has a few fields in it, but that doesn't seem to pull in any values. I've also tried using tools like https://mholt.github.io/json-to-go/ and getting the "real" struct that way. The issue with that method is there are numerous duplicates and I don't know enough to really assess how to tackle that.
This is an example JSON (I apologize for its size) https://pastebin.com/6u0b39tU
This is the struct that I get from the above tool: https://pastebin.com/3ZCu96st
the basic pattern of code I've tried is:
jsonFile, err := os.Open("test.json")
if err != nil {
fmt.Println("Couldn't open file", err)
}
defer jsonFile.Close()
bytes, _ := ioutil.ReadAll(jsonFile)
var channel Autogenerated
json.Unmarshal(bytes, &Autogenerated)
if err != nil {
fmt.Println("Failed to Unmarshal", err)
}
fmt.Println(channel.Fieldname)
Any feedback on the correct approach for how to handle something like this would be great. I get the feeling I'm just completely missing something.
In your code, you are not unmarshaling into the channel variable. Furthermore, you can optimize your code to not use ReadAll. Also, don't forget to check for errors (all errors).
Here is an improvement to your code.
jsonFile, err := os.Open("test.json")
if err != nil {
log.Fatalf("could not open file: %v", err)
}
defer jsonFile.Close()
var channel Autogenerated
if err := json.NewDecoder(jsonFile).Decode(&channel); err != nil {
log.Fatalf("failed to parse json: %v", err)
}
fmt.Println(channel.Fieldname)
Notice how a reference to channel is passed to Decode.

Insert a slice result JSON into MongoDB

I'm using the mgo driver for MongoDB, with the Gin framework.
type Users struct {
User_id *string `json:"id user" bson:"id user"`
Images []string `json:"images" bson:"images"`
}
I have this function which tries to convert the slice into JSON.
The slice here is UsersTotal
func GetUsersApi(c *gin.Context) {
UsersTotal, err := GetUsers()
if err != nil {
fmt.Println("error:", err)
}
c.JSON(http.StatusOK, gin.H{
"Count Users": len(UsersTotal),
"Users Found ": UsersTotal,
})
session, err := mgo.Dial(URL)
if err == nil {
fmt.Println("Connection to mongodb established ok!!")
cc := session.DB("UsersDB").C("results")
err22 := cc.Insert(&UsersTotal)
if err22 != nil {
fmt.Println("error insertion ", err22)
}
}
session.Close()
}
Running it I get the following error:
error insertion Wrong type for documents[0]. Expected a object, got a array.
Inserting multiple documents is the same as inserting a single one because the Collection.Insert() method has a variadic parameter:
func (c *Collection) Insert(docs ...interface{}) error
One thing you should note is that it expects interface{} values. Value of any type qualifies "to be" an interface{}. Another thing you should note is that only the slice type []interface{} qualifies to be []interface{}, a user slice []User does not. For details, see Type converting slices of interfaces in go
So simply create a copy of your users slice where the copy has a type of []interface{}, and that you can directly pass to Collection.Insert():
docs := make([]interface{}, len(UsersTotal))
for i, u := range UsersTotal {
docs[i] = u
}
err := cc.Insert(docs...)
// Handle error
Also please do not connect to MongodB in your handler. Do it once, on app startup, store the global connection / session, and clone / copy it when needed. For details see mgo - query performance seems consistently slow (500-650ms); and too many open files in mgo go server.

Golang - Parsing JSON string arrays from Twitch TV RESTful service

I've been working on parsing a JSON object that I retrieve through an HTTP GET request using Go's built in HTTP library. I initially tried using the default JSON library in Go in order to do this, but I was having a difficult time (I am a novice in Go still). I eventually resorted to using a different library and had little trouble after that, as shown below:
package main
import (
"github.com/antonholmquist/jason"
"fmt"
"net/http"
)
func main() {
resp, err := http.Get("http://tmi.twitch.tv/group/user/deernadia/chatters")
if nil != err {
panic(err)
}
defer resp.Body.Close()
body, err := jason.NewObjectFromReader(resp.Body)
chatters, err := body.GetObject("chatters")
if nil != err {
panic(err)
}
moderators, err := chatters.GetStringArray("moderators")
if nil != err {
panic(err)
}
for _, moderator := range moderators {
fmt.Println(moderator)
}
}
Where github.com/antonholmquist/jason corresponds to the custom JSON library I used.
This code produces something similar to the following output when run in a Linux shell (the RESTful service will update about every 30 seconds or so, which means the values in the JSON object will potentially change):
antwan250
bbrock89
boxception22
cmnights
deernadia
fartfalcon
fijibot
foggythought
fulc_
h_ov
iceydefeat
kingbobtheking
lospollogne
nightbot
nosleeptv
octaviuskhan
pateyy
phosphyg
poisonyvie
shevek18
trox94
trox_bot
uggasmesh
urbanelf
walmartslayer
wift3
And the raw JSON looks similar to this (with some of the users removed for brevity):
{
"_links": {},
"chatter_count": 469,
"chatters": {
"moderators": [
"antwan250",
"bbrock89",
"boxception22",
"cmnights",
"deernadia",
"fartfalcon",
"fijibot",
"foggythought",
"fulc_",
"h_ov",
"iceydefeat",
"kingbobtheking",
"lospollogne",
"nightbot",
"nosleeptv",
"octaviuskhan",
"pateyy",
"phosphyg",
"poisonyvie",
"shevek18",
"trox94",
"trox_bot",
"uggasmesh",
"urbanelf",
"walmartslayer",
"wift3"
],
"staff": [
"tnose"
],
"admins": [],
"global_mods": [],
"viewers": [
"03xuxu30",
"0dominic0",
"3389942",
"812mfk",
"910dan",
"aaradabooti",
"admiralackbar99",
"adrian97lol",
"aequitaso_o",
"aethiris",
"afropigeon",
"ahhhmong",
"aizaix",
"aka_magosh",
"akitoalexander",
"alex5761",
"allenhei",
"allou_fun_park",
"amilton_tkm",
"... more users that I removed...",
"zachn17",
"zero_x1",
"zigslip",
"ziirbryad",
"zonato83",
"zorr03body",
"zourtv"
]
}
}
As I said before, I'm using a custom library hosted on Github in order to accomplish what I needed, but for the sake of learning, I'm curious... how would I accomplish this same thing using Go's built in JSON library?
To be clear, what I'd like to do is be able to harvest the users from each JSON array embedded within the JSON object returned from the HTTP GET request. I'd also like to be able to get the list of viewers, admins, global moderators, etc., in the same way, but I figured that if I can see the moderator example using the default Go library, then reproducing that code for the other user types will be trivial.
Thank you in advance!
If you want to unmarshal moderators only, use the following:
var v struct {
Chatters struct {
Moderators []string
}
}
if err := json.Unmarshal(data, &v); err != nil {
// handle error
}
for _, mod := range v2.Chatters.Moderators {
fmt.Println(mod)
}
If you want to get all types of chatters, use the following:
var v struct {
Chatters map[string][]string
}
if err := json.Unmarshal(data, &v); err != nil {
handle error
}
for kind, users := range v1.Chatters {
for _, user := range users {
fmt.Println(kind, user)
}
}
run the code on the playground