Unmarshal JSON into slice of maps with unique elements - json

Question

I'm unmarshaling some json files into []map[string]string{}
but very often the source is dirty with many repeated equal objects.
The input looks like:
[{"sa1":"8172"},{"sa3":"8175"},{"sa1":"8172"},{"sa3":"8175"},{"sa3":"8175"},{"sa3":"8175"},{"sa1":"8172"},{"sa3":"8175"},{"sa3":"8175"}]
Resulting in:
map[sa1:8172]
([]map[string]string) (len=9 cap=9) {
(map[string]string) (len=1) {
(string) (len=3) "sa1": (string) (len=4) "8172"
},
(map[string]string) (len=1) {
(string) (len=3) "sa3": (string) (len=4) "8175"
},
(map[string]string) (len=1) {
(string) (len=3) "sa1": (string) (len=4) "8172"
},
(map[string]string) (len=1) {
(string) (len=3) "sa3": (string) (len=4) "8175"
},
(map[string]string) (len=1) {
(string) (len=3) "sa3": (string) (len=4) "8175"
},
(map[string]string) (len=1) {
(string) (len=3) "sa3": (string) (len=4) "8175"
},
(map[string]string) (len=1) {
(string) (len=3) "sa1": (string) (len=4) "8172"
},
(map[string]string) (len=1) {
(string) (len=3) "sa3": (string) (len=4) "8175"
},
(map[string]string) (len=1) {
(string) (len=3) "sa3": (string) (len=4) "8175"
}
}
How could I clean the slice of maps to contain only unique elements?

Answer 1

One option is to unmarshal the key value pairs directly into a comparable type, like a struct:
type Elem struct {
k string
v string
}
func (e *Elem) UnmarshalJSON(d []byte) error {
m := map[string]string{}
if err := json.Unmarshal(d, &m); err != nil {
return err
}
for k, v := range m {
e.k = k
e.v = v
return nil
}
return nil
}
Once you can compare the elements individually, you could also wrap that in a collection which filters the elements while unmarshaling. Whether to do this implicitly here, or after the fact is a matter of opinion. It may be a better separation of concerns to make filtering its own method, but I included it in UnmarshalJSON for brevity.
type Elems []Elem
func (e *Elems) UnmarshalJSON(d []byte) error {
tmp := []Elem{}
err := json.Unmarshal(d, &tmp)
if err != nil {
return err
}
seen := map[Elem]bool{}
for _, elem := range tmp {
if seen[elem] {
continue
}
seen[elem] = true
*e = append(*e, elem)
}
return nil
}
Then you can unmarshal into Elems:
elems := Elems{}
err := json.Unmarshal(js, &elems)
if err != nil {
log.Fatal(err)
}
fmt.Println(elems)
Which will give you the two unique pairs: [{sa1 8172} {sa3 8175}]
https://go.dev/play/p/U0iqBAjvz-1