Unmarshal CSV record into struct in Go - csv

The problem how to automatically deserialize/unmarshal record from CSV file into Go struct.
For example, I have
type Test struct {
Name string
Surname string
Age int
}
And CSV file contains records
John;Smith;42
Piter;Abel;50
Is there an easy way to unmarshal those records into struct except by using "encoding/csv" package for reading record and then doing something like
record, _ := reader.Read()
test := Test{record[0],record[1],atoi(record[2])}

There is gocarina/gocsv which handles custom struct in the same way encoding/json does.
You can also write custom marshaller and unmarshaller for specific types.
Example:
type Client struct {
Id string `csv:"client_id"` // .csv column headers
Name string `csv:"client_name"`
Age string `csv:"client_age"`
}
func main() {
in, err := os.Open("clients.csv")
if err != nil {
panic(err)
}
defer in.Close()
clients := []*Client{}
if err := gocsv.UnmarshalFile(in, &clients); err != nil {
panic(err)
}
for _, client := range clients {
fmt.Println("Hello, ", client.Name)
}
}

Seems I've done with automatic marshaling of CSV records into structs (limited to string and int). Hope this would be useful.
Here is a link to playground: http://play.golang.org/p/kwc32A5mJf
func Unmarshal(reader *csv.Reader, v interface{}) error {
record, err := reader.Read()
if err != nil {
return err
}
s := reflect.ValueOf(v).Elem()
if s.NumField() != len(record) {
return &FieldMismatch{s.NumField(), len(record)}
}
for i := 0; i < s.NumField(); i++ {
f := s.Field(i)
switch f.Type().String() {
case "string":
f.SetString(record[i])
case "int":
ival, err := strconv.ParseInt(record[i], 10, 0)
if err != nil {
return err
}
f.SetInt(ival)
default:
return &UnsupportedType{f.Type().String()}
}
}
return nil
}
I'll try to create github package is someone needs this implementation.

You could bake your own. Perhaps something like this:
package main
import (
"fmt"
"strconv"
"strings"
)
type Test struct {
Name string
Surname string
Age int
}
func (t Test) String() string {
return fmt.Sprintf("%s;%s;%d", t.Name, t.Surname, t.Age)
}
func (t *Test) Parse(in string) {
tmp := strings.Split(in, ";")
t.Name = tmp[0]
t.Surname = tmp[1]
t.Age, _ = strconv.Atoi(tmp[2])
}
func main() {
john := Test{"John", "Smith", 42}
fmt.Printf("john:%v\n", john)
johnString := john.String()
fmt.Printf("johnString:%s\n", johnString)
var rebornJohn Test
rebornJohn.Parse(johnString)
fmt.Printf("rebornJohn:%v\n", rebornJohn)
}

Using csvutil it is possible to give column header see example.
In your case, this could be :
package main
import (
"encoding/csv"
"fmt"
"io"
"os"
"github.com/jszwec/csvutil"
)
type Test struct {
Name string
Surname string
Age int
}
func main() {
csv_file, _ := os.Open("test.csv")
reader := csv.NewReader(csv_file)
reader.Comma = ';'
userHeader, _ := csvutil.Header(Test{}, "csv")
dec, _ := csvutil.NewDecoder(reader, userHeader...)
var users []Test
for {
var u Test
if err := dec.Decode(&u); err == io.EOF {
break
}
users = append(users, u)
}
fmt.Println(users)
}

Related

Unmarshall only approved fields

I'm trying to give permissions to users to edit certain fields of a struct.
This fields will vary depending on the action, the context, and the role the current user has.
At the moment I'm doing this in an imperative way but it's rather tedious and not scalable.
I thought having a list of approved_fields could be a nice and more scalable solution but I have no idea how to go through with this. I think reflection is the way to go but I don't know enough yet.
Any inspiration would help.
type Foo struct {
Bar1 int
Bar2 int
Bar3 int
}
foo := Foo{}
approved_fields := []string{"Bar1", "Bar2"}
decode(json_data, &foo, approved_fields)
foo2 := Foo{}
approved_fields := []string{"Bar1", "Bar3"}
decode(json_data, &foo2, approved_fields)
Here's how I would think about solving it. There may be something more efficient since I am unmarshalling the whole item before selectively choosing fields.
import (
"encoding/json"
"log"
"reflect"
"github.com/pkg/errors"
)
func fieldByName(s string, v interface{}) (reflect.Value, error) {
el := reflect.ValueOf(v).Elem()
fbn := el.FieldByName(s)
if !fbn.IsValid() {
return fbn, errors.Errorf("does not have field named %s", s)
}
return fbn, nil
}
func decode(data []byte, v interface{}, approvedFields []string) error {
typeOf := reflect.TypeOf(v)
if typeOf.Kind() == reflect.Ptr {
typeOf = typeOf.Elem()
}
if typeOf.Kind() == reflect.Slice {
return errors.New("does not support slices")
}
newItem := reflect.New(typeOf)
newItemInterface := newItem.Interface()
if err := json.Unmarshal(data, &newItemInterface); err != nil {
return errors.Wrap(err, "json unmarshall")
}
for _, approvedField := range approvedFields {
fbn, err := fieldByName(approvedField, v)
if err != nil {
return errors.Wrap(err, "field by name")
}
val, _ := fieldByName(approvedField, newItemInterface)
fbn.Set(val)
}
return nil
}
a test:
func TestBar1Bar2(t *testing.T) {
var json_data []byte
{
f := &Foo{
Bar1: 1,
Bar2: 2,
Bar3: 3,
}
json_data, _ = json.Marshal(f)
}
approved_fields := []string{"Bar1", "Bar2"}
f := &Foo{}
err := decode(json_data, f, approved_fields)
if err != nil {
t.Fatal(err)
}
assert.Equal(t, f.Bar1, 1)
assert.Equal(t, f.Bar2, 2)
assert.Equal(t, f.Bar3, 0)
}

Unmarshalling to same struct but different json name

I am trying to unmarshal a particular json data, perform some data transformations and then marshal the data and send it. However I want to marshal it with different json variable name.
Can I marshal the data to another json name like have xyz instead of abc
{"abc":1}
to
{"xyz":1}
package main
import (
"fmt"
"encoding/json"
)
type SomeStruct struct {
SomeField int `json:"abc"`
}
func main(){
jsonData := []byte(`{"abc":1}`)
strct := SomeStruct{}
json.Unmarshal([]byte(jsonData), &strct)
fmt.Println(strct)
x, err := json.Marshal(strct)
if err != nil {
fmt.Println("errs", err)
}
finalValue := string(x)
fmt.Println(finalValue)
}
Define two structures, one for your input, one for your output, and convert like this:
package main
import (
"fmt"
"encoding/json"
)
type inStruct struct {
SomeField int `json:"abc"`
}
type outStruct struct {
SomeField int `json:"xyz"`
}
func main(){
jsonData := []byte(`{"abc":1}`)
strct := inStruct{}
if err := json.Unmarshal(jsonData, &strct); err != nil {
log.Fatal(err)
}
// Edit to reflect mkopriva's comment
// outStruct := &outStruct{SomeField: strct.SomeField}
outStruct := outStruct(strct)
x, err := json.Marshal(outStruct)
if err != nil {
log.Fatal(err)
}
finalValue := string(x)
fmt.Println(finalValue)
}
You can potentially implement a custom ToJSON method for the struct, as follows:
package main
import (
"encoding/json"
"fmt"
)
type SomeStruct struct {
SomeField int `json:"abc"`
}
func (s *SomeStruct) ToJSON() string {
return fmt.Sprintf("{\"xyz\":%d}", s.SomeField)
}
func main() {
jsonData := []byte(`{"abc":1}`)
strct := SomeStruct{}
json.Unmarshal([]byte(jsonData), &strct)
fmt.Println(strct)
x, err := json.Marshal(strct)
if err != nil {
fmt.Println("errs", err)
}
finalValue := string(x)
fmt.Println(finalValue)
fmt.Println("custom ToJSON", strct.ToJSON())
}
Playground link: https://play.golang.org/p/mjW0dBPN59Q
It might not be flexible in the long run though. Personally, for requirement like this, I would prefer the solution posted by #Clément

Getting json dynamic key name as string?

For example:
{"id":
{"12345678901234":
{"Account":"asdf",
"Password":"qwerty"
"LastSeen":"1397621470",
}
}
}
A program I've been trying to make needs to get the id as a string and then later use it to check the time in LastSeen.
I've tried using simplejson and jsonq,but still cant figure out how to do that.
You can use RawMessage and make it much simpiler (play with it) :
package main
import (
"encoding/json"
"fmt"
)
var data []byte = []byte(`{"id": {"12345678901234": {"Account":"asdf", "Password":"qwerty", "LastSeen":"1397621470"}}}`)
type Message struct {
Id string
Info struct {
Account string
Password string
LastSeen string
}
}
func main() {
var (
tmpmsg struct {
Data map[string]json.RawMessage `json:"id"`
}
msg Message
)
if err := json.Unmarshal(data, &tmpmsg); err != nil {
panic(err) //you probably wanna use or something instead
}
for id, raw := range tmpmsg.Data {
msg.Id = id
if err := json.Unmarshal(raw, &msg.Info); err != nil {
panic(err)
}
}
fmt.Printf("%+v\n", msg)
}
Looking at the Golang blog post on JSON here it can be done using the encoding/json package. I created a small program to do this as follows:
package main
import (
"encoding/json"
"fmt"
)
var data []byte = []byte(`{"id": {"12345678901234": {"Account":"asdf", "Password":"qwerty", "LastSeen":"1397621470"}}}`)
type Message struct {
id string
LastSeen int64
}
var m Message
func main() {
var i interface {}
err := json.Unmarshal(data, &i)
if err != nil {
println("Error decoding data")
fmt.Printf("%s", err.Error())
return
}
m := i.(map[string]interface{})
for k, v := range m {
println(k)
im := v.(map[string]interface{})
for ik, iv := range im {
println("\t", ik)
jm := iv.(map[string]interface{})
for jk, jv := range jm {
println("\t\t", jk, ": ", jv.(string))
}
}
}
}
I apologise if this is poor in terms of Go best practices and such, I am new to the language. And I know that some elements of this aren't entirely necessary like the Message type definition but this works, at least on your data.

Golang Converting JSON

map[key:2073933158088]
I need to grab the key out of this data structure as a string, but I can't seem to figure out how!
Help with this overly simple question very much appreciated.
The value above is encapsulated in the variable named data.
I have tried: data.key, data[key], data["key"], data[0] and none of these seem to be appropriate calls.
To define data I sent up a JSON packet to a queue on IronMQ. I then pulled the message from the queue and manipulated it like this:
payloadIndex := 0
for index, arg := range(os.Args) {
if arg == "-payload" {
payloadIndex = index + 1
}
}
if payloadIndex >= len(os.Args) {
panic("No payload value.")
}
payload := os.Args[payloadIndex]
var data interface{}
raw, err := ioutil.ReadFile(payload)
if err != nil {
panic(err.Error())
}
err = json.Unmarshal(raw, &data)
Design your data type to match json structure. This is how can you achieve this:
package main
import (
"fmt"
"encoding/json"
)
type Data struct {
Key string `json:"key"`
}
func main() {
data := new(Data)
text := `{ "key": "2073933158088" }`
raw := []byte(text)
err := json.Unmarshal(raw, data)
if err != nil {
panic(err.Error())
}
fmt.Println(data.Key)
}
Since the number in the json is unquoted, it's not a string, Go will panic if you try to just handle it as a string (playground: http://play.golang.org/p/i-NUwchJc1).
Here's a working alternative:
package main
import (
"fmt"
"encoding/json"
"strconv"
)
type Data struct {
Key string `json:"key"`
}
func (d *Data) UnmarshalJSON(content []byte) error {
var m map[string]interface{}
err := json.Unmarshal(content, &m)
if err != nil {
return err
}
d.Key = strconv.FormatFloat(m["key"].(float64), 'f', -1, 64)
return nil
}
func main() {
data := new(Data)
text := `{"key":2073933158088}`
raw := []byte(text)
err := json.Unmarshal(raw, data)
if err != nil {
panic(err.Error())
}
fmt.Println(data.Key)
}
You can see the result in the playground: http://play.golang.org/p/5hU3hdV3kK

Go- Copy all common fields between structs

I have a database that stores JSON, and a server that provides an external API to whereby through an HTTP post, values in this database can be changed. The database is used by different processes internally, and as such have a common naming scheme.
The keys the customer sees are different, but map 1:1 with the keys in the database (there are unexposed keys). For example:
This is in the database:
{ "bit_size": 8, "secret_key": false }
And this is presented to the client:
{ "num_bits": 8 }
The API can change with respect to field names, but the database always has consistent keys.
I have named the fields the same in the struct, with different flags to the json encoder:
type DB struct {
NumBits int `json:"bit_size"`
Secret bool `json:"secret_key"`
}
type User struct {
NumBits int `json:"num_bits"`
}
I'm using encoding/json to do the Marshal/Unmarshal.
Is reflect the right tool for this? Is there an easier way since all of the keys are the same? I was thinking some kind of memcpy (if I kept the user fields in the same order).
Couldn't struct embedding be useful here?
package main
import (
"fmt"
)
type DB struct {
User
Secret bool `json:"secret_key"`
}
type User struct {
NumBits int `json:"num_bits"`
}
func main() {
db := DB{User{10}, true}
fmt.Printf("Hello, DB: %+v\n", db)
fmt.Printf("Hello, DB.NumBits: %+v\n", db.NumBits)
fmt.Printf("Hello, User: %+v\n", db.User)
}
http://play.golang.org/p/9s4bii3tQ2
buf := bytes.Buffer{}
err := gob.NewEncoder(&buf).Encode(&DbVar)
if err != nil {
return err
}
u := User{}
err = gob.NewDecoder(&buf).Decode(&u)
if err != nil {
return err
}
Here's a solution using reflection. You have to further develop it if you need more complex structures with embedded struct fields and such.
http://play.golang.org/p/iTaDgsdSaI
package main
import (
"encoding/json"
"fmt"
"reflect"
)
type M map[string]interface{} // just an alias
var Record = []byte(`{ "bit_size": 8, "secret_key": false }`)
type DB struct {
NumBits int `json:"bit_size"`
Secret bool `json:"secret_key"`
}
type User struct {
NumBits int `json:"num_bits"`
}
func main() {
d := new(DB)
e := json.Unmarshal(Record, d)
if e != nil {
panic(e)
}
m := mapFields(d)
fmt.Println("Mapped fields: ", m)
u := new(User)
o := applyMap(u, m)
fmt.Println("Applied map: ", o)
j, e := json.Marshal(o)
if e != nil {
panic(e)
}
fmt.Println("Output JSON: ", string(j))
}
func applyMap(u *User, m M) M {
t := reflect.TypeOf(u).Elem()
o := make(M)
for i := 0; i < t.NumField(); i++ {
f := t.FieldByIndex([]int{i})
// skip unexported fields
if f.PkgPath != "" {
continue
}
if x, ok := m[f.Name]; ok {
k := f.Tag.Get("json")
o[k] = x
}
}
return o
}
func mapFields(x *DB) M {
o := make(M)
v := reflect.ValueOf(x).Elem()
t := v.Type()
for i := 0; i < v.NumField(); i++ {
f := t.FieldByIndex([]int{i})
// skip unexported fields
if f.PkgPath != "" {
continue
}
o[f.Name] = v.FieldByIndex([]int{i}).Interface()
}
return o
}
Using struct tags, the following would sure be nice,
package main
import (
"fmt"
"log"
"hacked/json"
)
var dbj = `{ "bit_size": 8, "secret_key": false }`
type User struct {
NumBits int `json:"bit_size" api:"num_bits"`
}
func main() {
fmt.Println(dbj)
// unmarshal from full db record to User struct
var u User
if err := json.Unmarshal([]byte(dbj), &u); err != nil {
log.Fatal(err)
}
// remarshal User struct using api field names
api, err := json.MarshalTag(u, "api")
if err != nil {
log.Fatal(err)
}
fmt.Println(string(api))
}
Adding MarshalTag requires just a small patch to encode.go:
106c106,112
< e := &encodeState{}
---
> return MarshalTag(v, "json")
> }
>
> // MarshalTag is like Marshal but marshalls fields with
> // the specified tag key instead of the default "json".
> func MarshalTag(v interface{}, tag string) ([]byte, error) {
> e := &encodeState{tagKey: tag}
201a208
> tagKey string
328c335
< for _, ef := range encodeFields(v.Type()) {
---
> for _, ef := range encodeFields(v.Type(), e.tagKey) {
509c516
< func encodeFields(t reflect.Type) []encodeField {
---
> func encodeFields(t reflect.Type, tagKey string) []encodeField {
540c547
< tv := f.Tag.Get("json")
---
> tv := f.Tag.Get(tagKey)
The following function use reflect to copy fields between two structs. A src field is copied to a dest field if they have the same field name.
// CopyCommonFields copies src fields into dest fields. A src field is copied
// to a dest field if they have the same field name.
// Dest and src must be pointers to structs.
func CopyCommonFields(dest, src interface{}) {
srcType := reflect.TypeOf(src).Elem()
destType := reflect.TypeOf(dest).Elem()
destFieldsMap := map[string]int{}
for i := 0; i < destType.NumField(); i++ {
destFieldsMap[destType.Field(i).Name] = i
}
for i := 0; i < srcType.NumField(); i++ {
if j, ok := destFieldsMap[srcType.Field(i).Name]; ok {
reflect.ValueOf(dest).Elem().Field(j).Set(
reflect.ValueOf(src).Elem().Field(i),
)
}
}
}
Usage:
func main() {
type T struct {
A string
B int
}
type U struct {
A string
}
src := T{
A: "foo",
B: 5,
}
dest := U{}
CopyCommonFields(&dest, &src)
fmt.Printf("%+v\n", dest)
// output: {A:foo}
}
You can cast structures if they have same field names and types, effectively reassigning field tags:
package main
import "encoding/json"
type DB struct {
dbNumBits
Secret bool `json:"secret_key"`
}
type dbNumBits struct {
NumBits int `json:"bit_size"`
}
type User struct {
NumBits int `json:"num_bits"`
}
var Record = []byte(`{ "bit_size": 8, "secret_key": false }`)
func main() {
d := new(DB)
e := json.Unmarshal(Record, d)
if e != nil {
panic(e)
}
var u User = User(d.dbNumBits)
println(u.NumBits)
}
https://play.golang.org/p/uX-IIgL-rjc
Here's a solution without reflection, unsafe, or a function per struct. The example is a little convoluted, and maybe you wouldn't need to do it just like this, but the key is using a map[string]interface{} to get away from a struct with field tags. You might be able to use the idea in a similar solution.
package main
import (
"encoding/json"
"fmt"
"log"
)
// example full database record
var dbj = `{ "bit_size": 8, "secret_key": false }`
// User type has only the fields going to the API
type User struct {
// tag still specifies internal name, not API name
NumBits int `json:"bit_size"`
}
// mapping from internal field names to API field names.
// (you could have more than one mapping, or even construct this
// at run time)
var ApiField = map[string]string{
// internal: API
"bit_size": "num_bits",
// ...
}
func main() {
fmt.Println(dbj)
// select user fields from full db record by unmarshalling
var u User
if err := json.Unmarshal([]byte(dbj), &u); err != nil {
log.Fatal(err)
}
// remarshal from User struct back to json
exportable, err := json.Marshal(u)
if err != nil {
log.Fatal(err)
}
// unmarshal into a map this time, to shrug field tags.
type jmap map[string]interface{}
mInternal := jmap{}
if err := json.Unmarshal(exportable, &mInternal); err != nil {
log.Fatal(err)
}
// translate field names
mExportable := jmap{}
for internalField, v := range mInternal {
mExportable[ApiField[internalField]] = v
}
// marshal final result with API field names
if exportable, err = json.Marshal(mExportable); err != nil {
log.Fatal(err)
}
fmt.Println(string(exportable))
}
Output:
{ "bit_size": 8, "secret_key": false }
{"num_bits":8}
Edit: More explanation. As Tom notes in a comment, there's reflection going on behind the code. The goal here is to keep the code simple by using the available capabilities of the library. Package json currently offers two ways to work with data, struct tags and maps of [string]interface{}. The struct tags let you select fields, but force you to statically pick a single json field name. The maps let you pick field names at run time, but not which fields to Marshal. It would be nice if the json package let you do both at once, but it doesn't. The answer here just shows the two techniques and how they can be composed in a solution to the example problem in the OP.
"Is reflect the right tool for this?" A better question might be, "Are struct tags the right tool for this?" and the answer might be no.
package main
import (
"encoding/json"
"fmt"
"log"
)
var dbj = `{ "bit_size": 8, "secret_key": false }`
// translation from internal field name to api field name
type apiTrans struct {
db, api string
}
var User = []apiTrans{
{db: "bit_size", api: "num_bits"},
}
func main() {
fmt.Println(dbj)
type jmap map[string]interface{}
// unmarshal full db record
mdb := jmap{}
if err := json.Unmarshal([]byte(dbj), &mdb); err != nil {
log.Fatal(err)
}
// build result
mres := jmap{}
for _, t := range User {
if v, ok := mdb[t.db]; ok {
mres[t.api] = v
}
}
// marshal result
exportable, err := json.Marshal(mres)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(exportable))
}
An efficient way to achieve your goal is to use the gob package.
Here an example with the playground:
package main
import (
"bytes"
"encoding/gob"
"fmt"
)
type DB struct {
NumBits int
Secret bool
}
type User struct {
NumBits int
}
func main() {
db := DB{10, true}
user := User{}
buf := bytes.Buffer{}
err := gob.NewEncoder(&buf).Encode(&db)
if err != nil {
panic(err)
}
err = gob.NewDecoder(&buf).Decode(&user)
if err != nil {
panic(err)
}
fmt.Println(user)
}
Here the official blog post: https://blog.golang.org/gob