Looking for ideas to export JSON read structs into some csv format while preserving the hierarchy.
https://play.golang.org/p/jf2DRL1hC5K
/*
Expected output in excel for data wrangling:
A Key | B Key | C Key | D Key
SomethingA SomethingB SomethingC SomethingF
SomethingA SomethingB SomethingC SomethingG
SomethingA SomethingB SomethingC [1,2,3]
*/
I have tried iterating through the struct as following
for _, value := range mymodel {
fmt.Println(value)
/* could not iterate over structs */
}
Adding a method to your RootModel for both header and individual rows (so you can range over the type and only print the header once):
type RootModel struct {
A string
B string
C string
D factors
}
type factors struct {
F string
G string
H []int
}
func (*RootModel) CSVheader(w io.Writer) {
cw := csv.NewWriter(w)
cw.Write([]string{"A Key", "B Key", "C Key", "D Key"})
cw.Flush()
}
func (rm *RootModel) CSVrow(w io.Writer) {
cw := csv.NewWriter(w)
cw.Write([]string{rm.A, rm.B, rm.C, rm.D.F})
cw.Write([]string{rm.A, rm.B, rm.C, rm.D.G})
is, _ := json.Marshal(rm.D.H)
cw.Write([]string{rm.A, rm.B, rm.C, string(is)})
cw.Flush()
}
Playground: https://play.golang.org/p/c8UQVQ8tQTX
Output:
A Key,B Key,C Key,D Key
SomethingA,SomethingB,SomethingC,SomethingF
SomethingA,SomethingB,SomethingC,SomethingG
SomethingA,SomethingB,SomethingC,"[1,2,3]"
Note: if you are dealing with a slice of RootModel, you may want to put the CSV writer logic at that level - so it can handle the single render of the header-row and then the subsequent data rows.
Related
I'm working with the following topology (Kafka Streams):
employeeKStream
.selectKey { _, v -> v.payload.employeeId.toString() }
.join(
contractKTable,
Joiners.employeeContractJoiner(),
Joined.with(
Serdes.String(),
JsonSerde(EmployeePersistedV1::class.java),
JsonSerde(ContractChangedV1::class.java),
)
)
When it tries to apply the Stream to KTable join I receive the following error:
Can't serialize data [{"event_metadata": null, "payload": ...}] for topic [(*)-KSTREAM-KEY-SELECT-0000000014-repartition]
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Not an array
For any reason, seems like I'm not using the right Serde and it expect to handle an array. What's the problem?
Here the definition of contract table:
val contractKTable: KTable<String, ContractChangedV1> =
table(contractTopic, ContractChangedV1::class.java).toStream()
.selectKey { _, v -> v.payload.employeeId.toString() }.toTable()
Here the stream config:
fun kStreamsConfig(): KafkaStreamsConfiguration = KafkaStreamsConfiguration(
mapOf(
APPLICATION_ID_CONFIG to appName,
BOOTSTRAP_SERVERS_CONFIG to bootstrapAddress,
DEFAULT_KEY_SERDE_CLASS_CONFIG to Serdes.String().javaClass.name,
DEFAULT_VALUE_SERDE_CLASS_CONFIG to JsonSerde::class.java.name
)
)
So I am trying to get data from my database i.e MySQL. I am able to finish that step accessing the database but, the problem is I want to get the output in JSON format, but i did some research but didn't got the result so anyone can guide or hep me in this, getting the MySQL data in json by using GORM.
Here is the sample of my code which i written.
package main
import (
"fmt"
"github.com/jinzhu/gorm"
_ "github.com/jinzhu/gorm/dialects/mysql" //This must be introduced! !
)
type k_movie struct {
Id uint32
Title string `gorm:"default:''"`
Url_name string `gorm:"default:''"`
K_score string ``
Poster_url string ``
}
func main() {
db, errDb := gorm.Open("mysql", "root:xyz#123#(127.0.0.1)/dbdump?charset=utf8mb4&loc=Local")
if errDb != nil {
fmt.Println(errDb)
}
defer db.Close() //Close the database connection after use up
db.LogMode(true) //Open sql debug mode
//SELECT * FROM `k_movies` WHERE (id>0 and id<.....)
var movies []k_movie
db.Where("id>? and id<?", 0, 103697).Limit(3).Find(&movies)
fmt.Println(movies)
//Get the number
total := 0
db.Model(&k_movie{}).Count(&total)
fmt.Println(total)
var infos []k_movie //Define an array to receive multiple results
db.Where("Id in (?)", []uint32{1, 2, 3, 4, 5, 6, 7, 8}).Find(&infos)
fmt.Println(infos)
fmt.Println(len(infos)) //Number of results
var notValue []k_movie
db.Where("id=?", 3).Find(¬Value)
if len(notValue) == 0 {
fmt.Println("No data found!")
} else {
fmt.Println(notValue)
}
}
And the output I'm getting in this format.
kumardivyanshu#Divyanshus-MacBook-Air ~/myproject/src/github.com/gorm_mysql % go run test.go
(/Users/kumardivyanshu/myproject/src/github.com/gorm_mysql/test.go:31)
[2021-05-13 08:59:45] [3.89ms] SELECT * FROM `k_movies` WHERE (id>0 and id<103697) LIMIT 3
[3 rows affected or returned ]
[{1 Golmaal: Fun Unlimited golmaal-fun-unlimited 847 https://movieassetsdigital.sgp1.cdn.digitaloceanspaces.com/thumb/32b7385e1e616d7ba3d11e1bee255ecce638a136} {2 Dabangg 2 dabangg-2 425 https://movieassetsdigital.sgp1.cdn.digitaloceanspaces.com/thumb/1420c4d6f817d2b923cd8b55c81bdb9d9fd1eca0} {3 Force force 519 https://movieassetsdigital.sgp1.cdn.digitaloceanspaces.com/thumb/cd1dc247da9d16e194f4bfb09d99f4dedfb2de00}]
(/Users/kumardivyanshu/myproject/src/github.com/gorm_mysql/test.go:36)
[2021-05-13 08:59:45] [20.22ms] SELECT count(*) FROM `k_movies`
[0 rows affected or returned ]
103697
(/Users/kumardivyanshu/myproject/src/github.com/gorm_mysql/test.go:40)
[2021-05-13 08:59:45] [2.32ms] SELECT * FROM `k_movies` WHERE (Id in (1,2,3,4,5,6,7,8))
[8 rows affected or returned ]
[{1 Golmaal: Fun Unlimited golmaal-fun-unlimited 847 https://movieassetsdigital.sgp1.cdn.digitaloceanspaces.com/thumb/32b7385e1e616d7ba3d11e1bee255ecce638a136} {2 Dabangg 2 dabangg-2 425 https://movieassetsdigital.sgp1.cdn.digitaloceanspaces.com/thumb/1420c4d6f817d2b923cd8b55c81bdb9d9fd1eca0} {3 Force force 519 https://movieassetsdigital.sgp1.cdn.digitaloceanspaces.com/thumb/cd1dc247da9d16e194f4bfb09d99f4dedfb2de00} {4 Eega eega 906 https://movieassetsdigital.sgp1.cdn.digitaloceanspaces.com/thumb/08aef7d961d4699bf2d12a7c854b6b32d1445247} {5 Fukrey fukrey 672 https://movieassetsdigital.sgp1.cdn.digitaloceanspaces.com/thumb/5d14bd2fb0166f4bb9ab919e31b69f2605f366aa} {6 London Paris New York london-paris-new-york 323 https://movieassetsdigital.sgp1.cdn.digitaloceanspaces.com/thumb/222d8a6b5c76b1d3cfa0b93d4bcf1a1f16f5e199} {7 Bhaag Milkha Bhaag bhaag-milkha-bhaag 963 https://movieassetsdigital.sgp1.cdn.digitaloceanspaces.com/thumb/efa8b86c753ae0110cc3e82006fadabb06f1486c} {8 Bobby Jasoos bobby-jasoos 244 https://movieassetsdigital.sgp1.cdn.digitaloceanspaces.com/thumb/0e9540d4c962ec33d8b63c0c563e7b64169122e0}]
8
(/Users/kumardivyanshu/myproject/src/github.com/gorm_mysql/test.go:45)
[2021-05-13 08:59:45] [1.56ms] SELECT * FROM `k_movies` WHERE (id=3)
[1 rows affected or returned ]
[{3 Force force 519 https://movieassetsdigital.sgp1.cdn.digitaloceanspaces.com/thumb/cd1dc247da9d16e194f4bfb09d99f4dedfb2de00}]
You need to define the json tag on your struct, so you can use the json.Marshal to grab a []byte slice that presents a json object.
Example taken from Go by example:
type Response2 struct {
Page int `json:"page"`
Fruits []string `json:"fruits"`
}
res2D := &Response2{
Page: 1,
Fruits: []string{"apple", "peach", "pear"}}
res2B, _ := json.Marshal(res2D)
fmt.Println(string(res2B))
That would print:
{"page":1,"fruits":["apple","peach","pear"]}
I receive a JSON like this
{
"raw_content":"very long string"
"mode":"ML",
"user_id":"4000008367",
"user_description":"John Doe",
"model":3,
"dest_contact":"test#email.it",
"order_details":[
"ART.: 214883 PELL GRANI 9 ESPR.BAR SKGR 1000 SGOC.: 1000 GR\nVS.ART: 305920132 COMPOS. PALLET\n36 COLLI PEZ.RA: 6 TOT.PEZZI: 216 B: 12 T: 6\nEU C.L.: 24,230\nCO SCAP- : 16,500 CA SCAP- : 15,000 CO SCCP- : 0,000\nCO SAGV- : 0,00000\nC.N. : 17,200SCAD.MIN.: 25/01/22\nCONDIZIONI PAGAMENTO : 60GG B.B. DT RIC FT FINEMESE ART62\n",
"ART.: 287047 PELLINI BIO100%ARABICALTGR 250 SGOC.: 250 GR\nVS.ART: 315860176 COMPOS. PALLET\n36 COLLI PEZ.RA: 6 TOT.PEZZI: 216 B: 12 T: 3\nEU C.L.: 8,380\nCO SCAP- : 16,500 PR SCAP- : 15,000 CO SCCP- : 0,000\nCO SAGV- : 0,00000\nC.N. : 5,950SCAD.MIN.: 25/01/22\nCONDIZIONI PAGAMENTO : 60GG B.B. DT RIC FT FINEMESE ART62\n",
"ART.: 3137837 CAFFE PELLINI TOP LTGR 250 SGOC.: 250 GR\nVS.ART: 315850175 COMPOS. PALLET\n30 COLLI PEZ.RA: 12 TOT.PEZZI: 360 B: 6 T: 5\nEU C.L.: 6,810\nCO SCAP- : 16,500 PR SCAP- : 12,000 CO SCCP- : 0,000\nCO SAGV- : 0,00000\nC.N. : 5,000SCAD.MIN.: 18/08/21\nCONDIZIONI PAGAMENTO : 60GG B.B. DT RIC FT FINEMESE ART62\n",
"ART.: 7748220 ESPRES.SUP.TRADIZ. MOKPKGR 500 SGOC.: 500 GR\nVS.ART: 315930114 COMPOS. PALLET\n80 COLLI PEZ.RA: 10 TOT.PEZZI: 800 B: 10 T: 6\nEU C.L.: 7,580\nCO SCAP- : 16,500 PR SCAP- : 27,750 CO SCCP- : 0,000\nCO SAGV- : 0,00000\nC.N. : 4,570SCAD.MIN.: 25/01/22\nCONDIZIONI PAGAMENTO : 60GG B.B. DT RIC FT FINEMESE ART62\n"
],
"order_footer":"\nPALLET DA CM. 80X120\nT O T A L E C O L L I 182\n\n- EX D.P.R. 322 - 18/05/82 NON SI ACCETTANO TERMINI MINIMI DI\nCONSERVAZIONE INFERIORI A QUELLI INDICATI\n- CONSEGNA FRANCO BANCHINA, PALLET MONOPRODOTTO\nCOME DA PALLETTIZZAZIONE SPECIFICATA\nCONDIZIONI DI PAGAMENTO : COME DA ACCORDI\n"
}
and I want to reorder it to this
{
id: "32839ds8a32jjdas93193snkkk32jhds-k2j1", // generated, see my implementation
rawContent: "very long string",
parsedContent: {"mode":"ML", "user_id":"4000008367", "user_description":"John Doe", "order_details":[ "....." ], ... } // basically all the fields except raw content
}
How can I do this? I'm trying to work it with maps:
var output map[string]interface{}
var message processing.Document // message is of type struct {ID string, Raw string, Parsed string}
err = json.Unmarshal([]byte(doc), &output)
if err != nil {
// error handling
}
message.ParsedContent = "{"
for key, data := range output {
if key == "raw_content" {
hash := md5.Sum([]byte(data.(string)))
message.ID = hex.EncodeToString(hash[:])
message.RawContent = base64.RawStdEncoding.EncodeToString([]byte(data.(string)))
} else {
temp := fmt.Sprintf("\"%s\": \"%s\", ", key, data)
message.ParsedContent = message.ParsedContent + temp
}
}
message.ParsedContent = message.ParsedContent + "}"
msg, err := json.Marshal(message)
if err != nil {
// error handling
}
fmt.Println(string(msg))
There's a few problem with this. If it was only strings it would be ok, but there are integers and the sprintf doesn't work (the output I get, for example on the field "model" is "model": "%!s(float64=3)". I could do an if key == model and parse it as an int, but as I said the fields are not always the same and there are other integers that sometimes are there and sometimes are not there.
Also, the field "order_footer" has escaped new lines which are somehow deleted in my parsing, and this breaks the validity of the resulting JSON.
How can I solve this issues?
EDIT: As suggested, hand-parsing JSON is a bad idea. I could parse it into a struct, the field "model" actually tells me which struct to use. The struct for "model": 3 for example is:
type MOD3 struct {
Raw string `json:"raw_content"`
Mode string `json:"mode"`
UserID string `json:"user_id"`
UserDes string `json:"user_description"`
Model int `json:"model"`
Heading string `json:"legal_heading"`
DestContact string `json:"dest_contact"`
VendorID string `json:"vendor_id"`
VendorLegal string `json:"vendor_legal"`
OrderID string `json:"order_id"`
OrderDate int64 `json:"order_date"`
OrderReference string `json:"order_reference"`
DeliveryDate int64 `json:"delivery_date"`
OrderDestination string `json:"order_destination"`
OrderDestinationAddress string `json:"order_destination_address"`
Items []string `json:"order_details"`
OrderFooter string `json:"order_footer"`
}
At this point, how can I parse specific fields to the output format?
You should never, ever, ever try to generate JSON by hand. The steps should be:
Get JSON and parse it in a model object.
Create a copy of the model object with all the changes you want.
Convert the copied model object to JSON.
You don't know enough about JSON to modify it on the fly. I know enough, and the only reasonable way is complete parsing, and writing back the complete changes.
I've been trying to get the go yaml package to parse a file with jsonlines entries.
Below is a simple example with three options of data to be parsed.
Option one is a multi-doc yaml example. Both docs parse ok.
Option two is a two jsonline example. The first line parses ok, but the second is missed.
Option three is a two jsonline example, but I've put yaml doc separators in between, to force the issue. Both of these parse ok.
From reading the yaml and json specs, I believe the second option, multiple jsonlines, ought to be handled by a yaml parser.
My questions are:
Should a YAML parser cope with jsonlines?
Am I using the go yaml package correctly?
package main
import (
"bytes"
"fmt"
"reflect"
"strings"
"gopkg.in/yaml.v2"
)
var testData = []string{
`
---
option_one_first_yaml_doc: ok_here
---
option_one_second_yaml_doc: ok_here
`,
`
{option_two_first_jsonl: ok_here}
{option_two_second_jsonl: missing}
`,
`
---
{option_three_first_jsonl: ok_here}
---
{option_three_second_jsonl: ok_here}
`}
func printVal(v interface{}, depth int) {
typ := reflect.TypeOf(v)
if typ == nil {
fmt.Printf(" %v\n", "<null>")
} else if typ.Kind() == reflect.Int || typ.Kind() == reflect.String {
fmt.Printf("%s%v\n", strings.Repeat(" ", depth), v)
} else if typ.Kind() == reflect.Slice {
fmt.Printf("\n")
printSlice(v.([]interface{}), depth+1)
} else if typ.Kind() == reflect.Map {
fmt.Printf("\n")
printMap(v.(map[interface{}]interface{}), depth+1)
}
}
func printMap(m map[interface{}]interface{}, depth int) {
for k, v := range m {
fmt.Printf("%sKey: %s Value(s):", strings.Repeat(" ", depth), k.(string))
printVal(v, depth+1)
}
}
func printSlice(slc []interface{}, depth int) {
for _, v := range slc {
printVal(v, depth+1)
}
}
func main() {
m := make(map[interface{}]interface{})
for _, data := range testData {
yamlData := bytes.NewReader([]byte(data))
decoder := yaml.NewDecoder(yamlData)
for decoder.Decode(&m) == nil {
printMap(m, 0)
m = make(map[interface{}]interface{})
}
}
}
jsonlines is newline delimited JSON. That means the individual lines are JSON, but not multiple lines and certainly not a whole file of multiple lines.
You will need to read the jsonlines input a line at a time, and those lines you should be able to process with go yaml, since YAML is a superset of JSON.
Since you also seem to have YAML end of indicator (---) lines in your test, you
need to process those as well.
I'm trying to traverse some JSON response I'm getting from the OpenWeatherMap API but I'm getting some issues to retrieve some values. Here is my code:
{-# LANGUAGE OverloadedStrings #-}
import Control.Lens
import Data.Aeson.Lens (_String, key)
import Network.Wreq
myAPIKey :: String
myAPIKey = "my_api_key_here"
conditionsQuery :: String -> String -> String -> String
conditionsQuery city country key =
"https://api.openweathermap.org/data/2.5/forecast?q=" ++ city ++ "," ++ country ++ "&appid=" ++ key
main = do
print "What's the city?"
city <- getLine
print "And the country?"
country <- getLine
r <- get (conditionsQuery city country myAPIKey)
print $ r ^. responseBody . key "name" . _String
print $ r ^. responseBody . key "cod" . _String
print $ r ^. responseBody . key "id" . _String
The issue is that only the value of "cod" is returned ("200" in that case). The values for "name" and "id" appear as "", if we try with London,GB, Chicago, US (for instance). Yet the response body looks like:
{
...
"id": 2643743,
"name": "London",
"cod": 200
}
I first thought it was a type mismatch, but 200 is an Int there (unless I'm mistaken?) so I am not sure where the issue lies? "" seems to indicate that those 2 keys (id and name) do not exist, but they do.
Any ideas? Thanks in advance.
The response body does not look like that.
According to https://openweathermap.org/forecast5, the key "cod" appears at the outermost level of the JSON object, but "id" and "name" do not.
{
"city":{
"id":1851632,
"name":"Shuzenji",
...
}
"cod":"200",
...
}