Write Struct to Json File using Struct Fields (not json keys) - json

How can I read a json file into a struct, and then Marshal it back out to a json string with the Struct fields as keys (rather than the original json keys)?
(see Desired Output to Json File below...)
Code:
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
)
type Rankings struct {
Keyword string `json:"keyword"`
GetCount uint32 `json:"get_count"`
Engine string `json:"engine"`
Locale string `json:"locale"`
Mobile bool `json:"mobile"`
}
func main() {
var jsonBlob = []byte(`
{"keyword":"hipaa compliance form", "get_count":157, "engine":"google", "locale":"en-us", "mobile":false}
`)
rankings := Rankings{}
err := json.Unmarshal(jsonBlob, &rankings)
if err != nil {
// nozzle.printError("opening config file", err.Error())
}
rankingsJson, _ := json.Marshal(rankings)
err = ioutil.WriteFile("output.json", rankingsJson, 0644)
fmt.Printf("%+v", rankings)
}
Output on screen:
{Keyword:hipaa compliance form GetCount:157 Engine:google Locale:en-us Mobile:false}
Output to Json File:
{"keyword":"hipaa compliance form","get_count":157,"engine":"google","locale":"en-us","mobile":false}
Desired Output to Json File:
{"Keyword":"hipaa compliance form","GetCount":157,"Engine":"google","Locale":"en-us","Mobile":false}

If I understand your question correctly, all you want to do is remove the json tags from your struct definition.
So:
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
)
type Rankings struct {
Keyword string
GetCount uint32
Engine string
Locale string
Mobile bool
}
func main() {
var jsonBlob = []byte(`
{"keyword":"hipaa compliance form", "get_count":157, "engine":"google", "locale":"en-us", "mobile":false}
`)
rankings := Rankings{}
err := json.Unmarshal(jsonBlob, &rankings)
if err != nil {
// nozzle.printError("opening config file", err.Error())
}
rankingsJson, _ := json.Marshal(rankings)
err = ioutil.WriteFile("output.json", rankingsJson, 0644)
fmt.Printf("%+v", rankings)
}
Results in:
{Keyword:hipaa compliance form GetCount:0 Engine:google Locale:en-us Mobile:false}
And the file output is:
{"Keyword":"hipaa compliance form","GetCount":0,"Engine":"google","Locale":" en-us","Mobile":false}
Running example at http://play.golang.org/p/dC3s37HxvZ
Note: GetCount shows 0, since it was read in as "get_count". If you want to read in JSON that has "get_count" vs. "GetCount", but output "GetCount" then you'll have to do some additional parsing.
See Go- Copy all common fields between structs for additional info about this particular situation.

Try to change the json format in the struct
type Rankings struct {
Keyword string `json:"Keyword"`
GetCount uint32 `json:"Get_count"`
Engine string `json:"Engine"`
Locale string `json:"Locale"`
Mobile bool `json:"Mobile"`
}

An accourance happened by just using json.Marshal() / json.MarshalIndent().
It overwrites the existing file, which in my case was suboptimal. I just wanted to add content to current file, and keep old content.
This writes data through a buffer, with bytes.Buffer type.
This is what I gathered up so far:
package srf
import (
"bytes"
"encoding/json"
"os"
)
func WriteDataToFileAsJSON(data interface{}, filedir string) (int, error) {
//write data as buffer to json encoder
buffer := new(bytes.Buffer)
encoder := json.NewEncoder(buffer)
encoder.SetIndent("", "\t")
err := encoder.Encode(data)
if err != nil {
return 0, err
}
file, err := os.OpenFile(filedir, os.O_RDWR|os.O_CREATE, 0755)
if err != nil {
return 0, err
}
n, err := file.Write(buffer.Bytes())
if err != nil {
return 0, err
}
return n, nil
}
This is the execution of the function, together with the standard json.Marshal() or json.MarshalIndent() which overwrites the file
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
minerals "./minerals"
srf "./srf"
)
func main() {
//array of Test struct
var SomeType [10]minerals.Test
//Create 10 units of some random data to write
for a := 0; a < 10; a++ {
SomeType[a] = minerals.Test{
Name: "Rand",
Id: 123,
A: "desc",
Num: 999,
Link: "somelink",
People: []string{"John Doe", "Aby Daby"},
}
}
//writes aditional data to existing file, or creates a new file
n, err := srf.WriteDataToFileAsJSON(SomeType, "test2.json")
if err != nil {
log.Fatal(err)
}
fmt.Println("srf printed ", n, " bytes to ", "test2.json")
//overrides previous file
b, _ := json.MarshalIndent(SomeType, "", "\t")
ioutil.WriteFile("test.json", b, 0644)
}
Why is this useful?
File.Write() returns bytes written to the file! So this is perfect if you want to manage memory or storage.
WriteDataToFileAsJSON() (numberOfBytesWritten, error)

Related

Unmarshal JSON to Struct of Map

I have below program. I need to convert JSON data into object type cache which contains a single field of map[string]string type. There is something that I am doing wrong w.r.t. initialising map and unmarshalling JSON but unable to identify the syntax issue.
Note: I have marshalled data just for convenience sake to have sample JSON data.
package main
import (
"fmt"
"encoding/json"
"strconv"
)
const data = `{"Mode":{"ID-1":"ON","ID-2":"OFF","ID-3":"ON"}}`
type Cache struct {
Mode map[string]string `json:"Mode"`
}
func main() {
jsonData, _ := json.Marshal(data)
fmt.Println(strconv.Unquote(string(jsonData)))
var c Cache
c.Mode = make(map[string]string) //I want to initialise map so that I can store data in next step, but this is wrong I know
c.Mode["ID-4"] = "ON" //Want to store data like this
json.Unmarshal(jsonData, &c)
fmt.Println(c) //I am getting output as nil map i.e. {map[]}
for k, v := range c.Mode {
fmt.Println(k, v) //I am getting NO output i.e. blank
}
}
Here is your your code fixed https://play.golang.org/p/5ftaiz_Q5wl (attaching blow the copy)
package main
import (
"encoding/json"
"fmt"
"log"
)
const data = `{"Mode":{"ID-1":"ON","ID-2":"OFF","ID-3":"ON"}}`
type Cache struct {
Mode map[string]string `json:"Mode"`
}
func main() {
c := Cache{}
err := json.Unmarshal([]byte(data), &c)
if err != nil {
log.Println(err)
}
c.Mode["ID-4"] = "ON" //Want to store data like this
fmt.Println(c)
for k, v := range c.Mode {
fmt.Println(k, v)
}
}
Here is the output:
{map[ID-1:ON ID-2:OFF ID-3:ON ID-4:ON]}
ID-2 OFF
ID-3 ON
ID-4 ON
ID-1 ON

Convert JSON file into csv in golang

this following code is working successfully, it converts my json file into a csv file. But I would like to add titles to each of my columns into the csv file. Unfortunately, I can't find out how to do that. If anyone has an idea it would be very helpful.
Regards
package main
import (
"encoding/json"
"encoding/csv"
"fmt"
"io/ioutil"
"os"
"net/http"
"strconv"
)
type People struct {
Name string
Craft string
}
type General struct {
People []People
Number int
Message string
}
func main() {
// Reading data from JSON File
response, err := http.Get("http://api.open-notify.org/astros.json")
if err != nil {
fmt.Printf("The Http request failed with error %s\n", err)
}
data,_ := ioutil.ReadAll(response.Body)
//fmt.Println(string(data))
// Unmarshal JSON data
var general General
json.Unmarshal([]byte(data), &general)
//fmt.Printf("First person: %s, Message: %s", general.People[0].Name, general.Message)
// Create a csv file
csvdatafile, err := os.Create("./astros.csv")
if err != nil {
fmt.Println(err)
}
defer csvdatafile.Close()
// Write Unmarshaled json data to CSV file
w := csv.NewWriter(csvdatafile)
for _, obj := range general.People {
fmt.Println("Are you going into the for ?")
var record []string
record = append(record, strconv.Itoa(general.Number), general.Message)
record = append(record, obj.Name, obj.Craft)
w.Write(record)
fmt.Println("Are you coming here")
record = nil
}
w.Flush()
fmt.Println("Appending succed")
}
Well if you want it just for this example can you just write the column headers to the file before the for statement, that is:
w := csv.NewWriter(csvdatafile)
//new code
var header []string
header = append(header, "Number")
header = append(header, "Message")
header = append(header, "Name")
header = append(header, "Craft")
w.Write(header)
for _, obj := range general.People {

Read extern JSON file

I am trying to read the following JSON file:
{
"a":1,
"b":2,
"c":3
}
I have tried this but I found that I had to write each field of the JSON file into a struct but I really don't want to have all my JSON file in my Go code.
import (
"fmt"
"encoding/json"
"io/ioutil"
)
type Data struct {
A string `json:"a"`
B string `json:"b"`
C string `json:"c"`
}
func main() {
file, _ := ioutil.ReadFile("/path/to/file.json")
data := Data{}
if err := json.Unmarshal(file ,&data); err != nil {
panic(err)
}
for _, letter := range data.Letter {
fmt.Println(letter)
}
}
Is there a way to bypass this thing with something like json.load(file) in Python?
If you only want to support integer values, you could unmarshal your data into a map[string]int. Note that the order of a map is not defined, so the below program's output is non-deterministic for the input.
package main
import (
"fmt"
"encoding/json"
"io/ioutil"
)
func main() {
file, _ := ioutil.ReadFile("/path/to/file.json")
var data map[string]int
if err := json.Unmarshal(file ,&data); err != nil {
panic(err)
}
for letter := range data {
fmt.Println(letter)
}
}
You can unmarshal any JSON data in this way:
var data interface{}
if err := json.Unmarshal(..., &data); err != nil {
// handle error
}
Though, in this way you should handle all the reflection-related stuffs
since you don't know what type the root data is, and its fields.
Even worse, your data might not be map at all.
It can be any valid JSON data type like array, string, integer, etc.
Here's a playground link: https://play.golang.org/p/DiceOv4sATO
It's impossible to do anything as simple as in Python, because Go is strictly typed, so it's necessary to pass your target into the unmarshal function.
What you've written could otherwise be shortened, slightly, to something like this:
func UnmarshalJSONFile(path string, i interface{}) error {
f, err := os.Open(path)
if err != nil {
return err
}
defer f.Close()
return json.NewDecoder(f).Decode(i)
}
But then to use it, you would do this:
func main() {
data := Data{}
if err := UnmarshalJSONFile("/path/to/file.json", &data); err != nil {
panic(err)
}
}
But you can see that the UnmarshalJSONFile is so simple, it hardly warrants a standard library function.

Saving Read/Write/Save to json from Struct of Map

I've been trying to have a " working " file to which i save certain basic state of my application instead of having them in Ram since they would need to be saved everyday, i've decided on creating file per day, this part is working but i've stripped it from the code for more clarity.
Now i'm able to initialise my file with false value for the informations struct and then unmarshalling and reading from it.
The problem arise when i'm trying to update the "file" after it's been unmarshalled before i save it back to the text file.
The isImportStarted does work (when removing the erronous line obv ) but i can't seem to update the file properly i get this error :
./test.go:62:34: cannot assign to struct field
TheList[symbol].ImportStarted in map
./test.go:71:3: cannot take the address of
TheList[symbol].ImportStarted
./test.go:71:34: cannot assign to &TheList[symbol].ImportStarted
My code :
package main
import (
"encoding/json"
"fmt"
"os"
"io/ioutil"
"log"
)
type Informations struct {
ImportStarted bool
ImportDone bool
}
var MyList = map[string]*Informations{
"test": &Informations{ImportStarted: false,ImportDone:false},
"test2": &Informations{ImportStarted: false,ImportDone:false},
}
func ReadFile(filename string) []byte{
data, err := ioutil.ReadFile(filename)
if err != nil {
log.Panicf("failed reading data from file: %s", err)
}
return data
}
func writeFile(json string,filename string){
file, err := os.OpenFile(filename, os.O_APPEND|os.O_WRONLY, os.ModeAppend)
defer file.Close()
if err != nil {
fmt.Println(err)
}
_,err2 := file.WriteString(json)
fmt.Println(err2)
}
func main() {
isImportStarted("test")
ImportStart("test")
}
func ImportStart(symbol string){
filename := "test.txt"
_, err := os.Stat(filename)
if err != nil {
if os.IsNotExist(err) {
fmt.Println("File does not exist creating it...")
file, err := os.Create(filename)
jsonString, _ := json.Marshal(MyList)
writeFile(string(jsonString),filename)
if err != nil {
fmt.Println(err)
}
fmt.Println("reading from file"+filename )
x := ReadFile(filename)
var TheList = map[string]Informations{}
json.Unmarshal(x,&TheList )
TheList[symbol].ImportStarted = true
defer file.Close()
//wanting to save afterwards...
}
} else {
fmt.Println("reading from file "+ filename)
x := ReadFile(filename)
var TheList = map[string]Informations{}
json.Unmarshal(x,&TheList )
&TheList[symbol].ImportStarted = true
}
}
func isImportStarted(symbol string) bool{
filename := "test.txt"
x := ReadFile(filename)
var TheList = map[string]Informations{}
json.Unmarshal(x,&TheList )
return TheList[symbol].ImportStarted
}
I've tried the Why do I get a "cannot assign" error when setting value to a struct as a value in a map? question but it doesn't fit my use case at all as it would effectivly initialize all my structs with nil instead of {false,false}
Any ideas?
Try var TheList = map[string]*Informations{}, why you cannot assign a value in a map please refer to why-do-i-get-a-cannot-assing-error or access-struct-in-map-without-copying

How can I pretty-print JSON using Go?

Does anyone know of a simple way to pretty-print JSON output in Go?
I'd like to pretty-print the result of json.Marshal, as well as formatting an existing string of JSON so it's easier to read.
MarshalIndent will allow you to output your JSON with indentation and spacing. For example:
{
"data": 1234
}
The indent argument specifies the series of characters to indent with. Thus, json.MarshalIndent(data, "", " ") will pretty-print using four spaces for indentation.
The accepted answer is great if you have an object you want to turn into JSON. The question also mentions pretty-printing just any JSON string, and that's what I was trying to do. I just wanted to pretty-log some JSON from a POST request (specifically a CSP violation report).
To use MarshalIndent, you would have to Unmarshal that into an object. If you need that, go for it, but I didn't. If you just need to pretty-print a byte array, plain Indent is your friend.
Here's what I ended up with:
import (
"bytes"
"encoding/json"
"log"
"net/http"
)
func HandleCSPViolationRequest(w http.ResponseWriter, req *http.Request) {
body := App.MustReadBody(req, w)
if body == nil {
return
}
var prettyJSON bytes.Buffer
error := json.Indent(&prettyJSON, body, "", "\t")
if error != nil {
log.Println("JSON parse error: ", error)
App.BadRequest(w)
return
}
log.Println("CSP Violation:", string(prettyJSON.Bytes()))
}
For better memory usage, I guess this is better:
var out io.Writer
enc := json.NewEncoder(out)
enc.SetIndent("", " ")
if err := enc.Encode(data); err != nil {
panic(err)
}
I was frustrated by the lack of a fast, high quality way to marshal JSON to a colorized string in Go so I wrote my own Marshaller called ColorJSON.
With it, you can easily produce output like this using very little code:
package main
import (
"fmt"
"encoding/json"
"github.com/TylerBrock/colorjson"
)
func main() {
str := `{
"str": "foo",
"num": 100,
"bool": false,
"null": null,
"array": ["foo", "bar", "baz"],
"obj": { "a": 1, "b": 2 }
}`
var obj map[string]interface{}
json.Unmarshal([]byte(str), &obj)
// Make a custom formatter with indent set
f := colorjson.NewFormatter()
f.Indent = 4
// Marshall the Colorized JSON
s, _ := f.Marshal(obj)
fmt.Println(string(s))
}
I'm writing the documentation for it now but I was excited to share my solution.
Edit Looking back, this is non-idiomatic Go. Small helper functions like this add an extra step of complexity. In general, the Go philosophy prefers to include the 3 simple lines over 1 tricky line.
As #robyoder mentioned, json.Indent is the way to go. Thought I'd add this small prettyprint function:
package main
import (
"bytes"
"encoding/json"
"fmt"
)
//dont do this, see above edit
func prettyprint(b []byte) ([]byte, error) {
var out bytes.Buffer
err := json.Indent(&out, b, "", " ")
return out.Bytes(), err
}
func main() {
b := []byte(`{"hello": "123"}`)
b, _ = prettyprint(b)
fmt.Printf("%s", b)
}
https://go-sandbox.com/#/R4LWpkkHIN or http://play.golang.org/p/R4LWpkkHIN
Here's what I use. If it fails to pretty print the JSON it just returns the original string. Useful for printing HTTP responses that should contain JSON.
import (
"encoding/json"
"bytes"
)
func jsonPrettyPrint(in string) string {
var out bytes.Buffer
err := json.Indent(&out, []byte(in), "", "\t")
if err != nil {
return in
}
return out.String()
}
package cube
import (
"encoding/json"
"fmt"
"github.com/magiconair/properties/assert"
"k8s.io/api/rbac/v1beta1"
v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"testing"
)
func TestRole(t *testing.T) {
clusterRoleBind := &v1beta1.ClusterRoleBinding{
ObjectMeta: v1.ObjectMeta{
Name: "serviceaccounts-cluster-admin",
},
RoleRef: v1beta1.RoleRef{
APIGroup: "rbac.authorization.k8s.io",
Kind: "ClusterRole",
Name: "cluster-admin",
},
Subjects: []v1beta1.Subject{{
Kind: "Group",
APIGroup: "rbac.authorization.k8s.io",
Name: "system:serviceaccounts",
},
},
}
b, err := json.MarshalIndent(clusterRoleBind, "", " ")
assert.Equal(t, nil, err)
fmt.Println(string(b))
}
Here is my solution:
import (
"bytes"
"encoding/json"
)
const (
empty = ""
tab = "\t"
)
func PrettyJson(data interface{}) (string, error) {
buffer := new(bytes.Buffer)
encoder := json.NewEncoder(buffer)
encoder.SetIndent(empty, tab)
err := encoder.Encode(data)
if err != nil {
return empty, err
}
return buffer.String(), nil
}
//You can do it with json.MarshalIndent(data, "", " ")
package main
import(
"fmt"
"encoding/json" //Import package
)
//Create struct
type Users struct {
ID int
NAME string
}
//Asign struct
var user []Users
func main() {
//Append data to variable user
user = append(user, Users{1, "Saturn Rings"})
//Use json package the blank spaces are for the indent
data, _ := json.MarshalIndent(user, "", " ")
//Print json formatted
fmt.Println(string(data))
}
Another example with http.ResponseWriter.
import (
"encoding/json"
"net/http"
)
func main() {
var w http.ResponseWriter
type About struct {
ProgName string
Version string
}
goObj := About{ProgName: "demo", Version: "0.0.0"}
beautifulJsonByte, err := json.MarshalIndent(goObj, "", " ")
if err != nil {
panic(err)
}
_, _ = w.Write(beautifulJsonByte)
}
output
{
"ProgName": "demo",
"Version": "0.0.0"
}
If you want to create a commandline utility to pretty print JSON
package main
import ("fmt"
"encoding/json"
"os"
"bufio"
"bytes"
)
func main(){
var out bytes.Buffer
reader := bufio.NewReader(os.Stdin)
text, _ := reader.ReadString('\n')
err := json.Indent(&out, []byte(text), "", " ")
if err != nil {
fmt.Println(err)
}
fmt.Println(string(out.Bytes()))
}
echo "{\"boo\":\"moo\"}" | go run main.go
will produce the following output :
{
"boo": "moo"
}
feel free to build a binary
go build main.go
and drop it in /usr/local/bin
A simple off the shelf pretty printer in Go. One can compile it to a binary through:
go build -o jsonformat jsonformat.go
It reads from standard input, writes to standard output and allow to set indentation:
package main
import (
"bytes"
"encoding/json"
"flag"
"fmt"
"io/ioutil"
"os"
)
func main() {
indent := flag.String("indent", " ", "indentation string/character for formatter")
flag.Parse()
src, err := ioutil.ReadAll(os.Stdin)
if err != nil {
fmt.Fprintf(os.Stderr, "problem reading: %s", err)
os.Exit(1)
}
dst := &bytes.Buffer{}
if err := json.Indent(dst, src, "", *indent); err != nil {
fmt.Fprintf(os.Stderr, "problem formatting: %s", err)
os.Exit(1)
}
if _, err = dst.WriteTo(os.Stdout); err != nil {
fmt.Fprintf(os.Stderr, "problem writing: %s", err)
os.Exit(1)
}
}
It allows to run a bash commands like:
cat myfile | jsonformat | grep "key"
i am sort of new to go, but this is what i gathered up so far:
package srf
import (
"bytes"
"encoding/json"
"os"
)
func WriteDataToFileAsJSON(data interface{}, filedir string) (int, error) {
//write data as buffer to json encoder
buffer := new(bytes.Buffer)
encoder := json.NewEncoder(buffer)
encoder.SetIndent("", "\t")
err := encoder.Encode(data)
if err != nil {
return 0, err
}
file, err := os.OpenFile(filedir, os.O_RDWR|os.O_CREATE, 0755)
if err != nil {
return 0, err
}
n, err := file.Write(buffer.Bytes())
if err != nil {
return 0, err
}
return n, nil
}
This is the execution of the function, and just standard
b, _ := json.MarshalIndent(SomeType, "", "\t")
Code:
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
minerals "./minerals"
srf "./srf"
)
func main() {
//array of Test struct
var SomeType [10]minerals.Test
//Create 10 units of some random data to write
for a := 0; a < 10; a++ {
SomeType[a] = minerals.Test{
Name: "Rand",
Id: 123,
A: "desc",
Num: 999,
Link: "somelink",
People: []string{"John Doe", "Aby Daby"},
}
}
//writes aditional data to existing file, or creates a new file
n, err := srf.WriteDataToFileAsJSON(SomeType, "test2.json")
if err != nil {
log.Fatal(err)
}
fmt.Println("srf printed ", n, " bytes to ", "test2.json")
//overrides previous file
b, _ := json.MarshalIndent(SomeType, "", "\t")
ioutil.WriteFile("test.json", b, 0644)
}
Use json.MarshalIndent with string
This easyPrint function accepts argument data (any type of data) to print it into the intended (pretty) JSON format.
import (
"encoding/json"
"log"
)
func easyPrint(data interface{}) {
manifestJson, _ := json.MarshalIndent(data, "", " ")
log.Println(string(manifestJson))
}
With name argument.
TODO: make argument name optional.
func easyPrint(data interface{}, name string) {
manifestJson, _ := json.MarshalIndent(data, "", " ")
log.Println(name + " ->", string(manifestJson))
}