Convert JSON file into csv in golang - json

this following code is working successfully, it converts my json file into a csv file. But I would like to add titles to each of my columns into the csv file. Unfortunately, I can't find out how to do that. If anyone has an idea it would be very helpful.
Regards
package main
import (
"encoding/json"
"encoding/csv"
"fmt"
"io/ioutil"
"os"
"net/http"
"strconv"
)
type People struct {
Name string
Craft string
}
type General struct {
People []People
Number int
Message string
}
func main() {
// Reading data from JSON File
response, err := http.Get("http://api.open-notify.org/astros.json")
if err != nil {
fmt.Printf("The Http request failed with error %s\n", err)
}
data,_ := ioutil.ReadAll(response.Body)
//fmt.Println(string(data))
// Unmarshal JSON data
var general General
json.Unmarshal([]byte(data), &general)
//fmt.Printf("First person: %s, Message: %s", general.People[0].Name, general.Message)
// Create a csv file
csvdatafile, err := os.Create("./astros.csv")
if err != nil {
fmt.Println(err)
}
defer csvdatafile.Close()
// Write Unmarshaled json data to CSV file
w := csv.NewWriter(csvdatafile)
for _, obj := range general.People {
fmt.Println("Are you going into the for ?")
var record []string
record = append(record, strconv.Itoa(general.Number), general.Message)
record = append(record, obj.Name, obj.Craft)
w.Write(record)
fmt.Println("Are you coming here")
record = nil
}
w.Flush()
fmt.Println("Appending succed")
}

Well if you want it just for this example can you just write the column headers to the file before the for statement, that is:
w := csv.NewWriter(csvdatafile)
//new code
var header []string
header = append(header, "Number")
header = append(header, "Message")
header = append(header, "Name")
header = append(header, "Craft")
w.Write(header)
for _, obj := range general.People {

Related

Unmarshal JSON to Struct of Map

I have below program. I need to convert JSON data into object type cache which contains a single field of map[string]string type. There is something that I am doing wrong w.r.t. initialising map and unmarshalling JSON but unable to identify the syntax issue.
Note: I have marshalled data just for convenience sake to have sample JSON data.
package main
import (
"fmt"
"encoding/json"
"strconv"
)
const data = `{"Mode":{"ID-1":"ON","ID-2":"OFF","ID-3":"ON"}}`
type Cache struct {
Mode map[string]string `json:"Mode"`
}
func main() {
jsonData, _ := json.Marshal(data)
fmt.Println(strconv.Unquote(string(jsonData)))
var c Cache
c.Mode = make(map[string]string) //I want to initialise map so that I can store data in next step, but this is wrong I know
c.Mode["ID-4"] = "ON" //Want to store data like this
json.Unmarshal(jsonData, &c)
fmt.Println(c) //I am getting output as nil map i.e. {map[]}
for k, v := range c.Mode {
fmt.Println(k, v) //I am getting NO output i.e. blank
}
}
Here is your your code fixed https://play.golang.org/p/5ftaiz_Q5wl (attaching blow the copy)
package main
import (
"encoding/json"
"fmt"
"log"
)
const data = `{"Mode":{"ID-1":"ON","ID-2":"OFF","ID-3":"ON"}}`
type Cache struct {
Mode map[string]string `json:"Mode"`
}
func main() {
c := Cache{}
err := json.Unmarshal([]byte(data), &c)
if err != nil {
log.Println(err)
}
c.Mode["ID-4"] = "ON" //Want to store data like this
fmt.Println(c)
for k, v := range c.Mode {
fmt.Println(k, v)
}
}
Here is the output:
{map[ID-1:ON ID-2:OFF ID-3:ON ID-4:ON]}
ID-2 OFF
ID-3 ON
ID-4 ON
ID-1 ON

How to save a JSON file into csv

I am trying to save this astros.json file into csv file.
Everything is working I don't have any error but my code is not going into my for loop.
So the astros.csv file is created successfully but there are no datas inside.
package main
import (
"encoding/json"
"encoding/csv"
"fmt"
"io/ioutil"
"os"
"net/http"
"strconv"
)
type People struct {
Name string
Craft string
}
type General struct {
//People []People
Number int
Message string
}
func main() {
// Reading data from JSON File
response, err := http.Get("http://api.open-notify.org/astros.json")
if err != nil {
fmt.Printf("The Http request failed with error %s\n", err)
}
data,_ := ioutil.ReadAll(response.Body)
//fmt.Println(string(data))
// Unmarshal JSON data
var general []General
json.Unmarshal([]byte(data), &general)
//fmt.Printf("First person: %s, Message: %s", general.People[0].Name, general.Message)
// Create a csv file
csvdatafile, err := os.Create("./astros.csv")
if err != nil {
fmt.Println(err)
}
defer csvdatafile.Close()
// Write Unmarshaled json data to CSV file
w := csv.NewWriter(csvdatafile)
for _, obj := range general {
fmt.Println("Are you going into the for ?")
var record []string
//record = append(record, obj.People)
record = append(record, strconv.Itoa(obj.Number), obj.Message)
record = append(record, obj.Message)
//record = append(record, []string{People})
w.Write(record)
fmt.Println("Are you coming here")
record = nil
}
w.Flush()
fmt.Println("Appending succed")
}
It is important to always check for errors; if you change your code to:
err = json.Unmarshal([]byte(data), &general)
if err != nil {
fmt.Printf("Error in unmarshall %s\n", err)
return
}
Then you would see the problem; json.Unmarshal is returning cannot unmarshal object into Go value of type []main.General. This means that general is nil so your loop is never entered.
The reason for this is apparent when you compare the JSON:
{
"people": [{
"name": "Christina Koch",
"craft": "ISS"
}, {
"name": "Alexander Skvortsov",
"craft": "ISS"
}, {
"name": "Luca Parmitano",
"craft": "ISS"
}, {
"name": "Andrew Morgan",
"craft": "ISS"
}, {
"name": "Oleg Skripochka",
"craft": "ISS"
}, {
"name": "Jessica Meir",
"craft": "ISS"
}
],
"number": 6,
"message": "success"
}
With the variable you are unmarshalling into; []General where General is:
type General struct {
//People []People
Number int
Message string
}
It looks like you had the right idea at some point with the []People but then commented it out.
If you uncomment []People, unmarshal into a General (rather than []General) then make your loop for _, obj := range general.People { it should work as expected. This assumes that you want to iterate through the people; general.Message and general.Number are also available (but you dont need a loop to access these).
See this example (have removed file operation as the playground does not support that).
you could try something like this:
package main
import (
"encoding/csv"
"encoding/json"
"fmt"
"net/http"
"os"
"strconv"
)
type People struct {
Name string
Craft string
}
type Response struct {
People []People
Number int
Message string
}
func main() {
// Reading data from JSON File
r, err := http.Get("http://api.open-notify.org/astros.json")
if err != nil {
fmt.Printf("The Http request failed with error %s\n", err)
}
var response Response
err = json.NewDecoder(r.Body).Decode(&response)
if err != nil {
fmt.Printf("decoding error%s\n", err)
return
}
// Create a csv file
csvdatafile, err := os.Create("./astros.csv")
if err != nil {
fmt.Println(err)
}
defer csvdatafile.Close()
// Write Unmarshaled json data to CSV file
w := csv.NewWriter(csvdatafile)
for _, obj := range response.People {
fmt.Println("Are you going into the for ?")
var record []string
//record = append(record, obj.People)
record = append(record, strconv.Itoa(response.Number), response.Message)
record = append(record, obj.Name)
// record = append(record, []string{obj})
w.Write(record)
fmt.Println("Are you coming here")
record = nil
}
w.Flush()
fmt.Println("Appending succed")
}

Saving Read/Write/Save to json from Struct of Map

I've been trying to have a " working " file to which i save certain basic state of my application instead of having them in Ram since they would need to be saved everyday, i've decided on creating file per day, this part is working but i've stripped it from the code for more clarity.
Now i'm able to initialise my file with false value for the informations struct and then unmarshalling and reading from it.
The problem arise when i'm trying to update the "file" after it's been unmarshalled before i save it back to the text file.
The isImportStarted does work (when removing the erronous line obv ) but i can't seem to update the file properly i get this error :
./test.go:62:34: cannot assign to struct field
TheList[symbol].ImportStarted in map
./test.go:71:3: cannot take the address of
TheList[symbol].ImportStarted
./test.go:71:34: cannot assign to &TheList[symbol].ImportStarted
My code :
package main
import (
"encoding/json"
"fmt"
"os"
"io/ioutil"
"log"
)
type Informations struct {
ImportStarted bool
ImportDone bool
}
var MyList = map[string]*Informations{
"test": &Informations{ImportStarted: false,ImportDone:false},
"test2": &Informations{ImportStarted: false,ImportDone:false},
}
func ReadFile(filename string) []byte{
data, err := ioutil.ReadFile(filename)
if err != nil {
log.Panicf("failed reading data from file: %s", err)
}
return data
}
func writeFile(json string,filename string){
file, err := os.OpenFile(filename, os.O_APPEND|os.O_WRONLY, os.ModeAppend)
defer file.Close()
if err != nil {
fmt.Println(err)
}
_,err2 := file.WriteString(json)
fmt.Println(err2)
}
func main() {
isImportStarted("test")
ImportStart("test")
}
func ImportStart(symbol string){
filename := "test.txt"
_, err := os.Stat(filename)
if err != nil {
if os.IsNotExist(err) {
fmt.Println("File does not exist creating it...")
file, err := os.Create(filename)
jsonString, _ := json.Marshal(MyList)
writeFile(string(jsonString),filename)
if err != nil {
fmt.Println(err)
}
fmt.Println("reading from file"+filename )
x := ReadFile(filename)
var TheList = map[string]Informations{}
json.Unmarshal(x,&TheList )
TheList[symbol].ImportStarted = true
defer file.Close()
//wanting to save afterwards...
}
} else {
fmt.Println("reading from file "+ filename)
x := ReadFile(filename)
var TheList = map[string]Informations{}
json.Unmarshal(x,&TheList )
&TheList[symbol].ImportStarted = true
}
}
func isImportStarted(symbol string) bool{
filename := "test.txt"
x := ReadFile(filename)
var TheList = map[string]Informations{}
json.Unmarshal(x,&TheList )
return TheList[symbol].ImportStarted
}
I've tried the Why do I get a "cannot assign" error when setting value to a struct as a value in a map? question but it doesn't fit my use case at all as it would effectivly initialize all my structs with nil instead of {false,false}
Any ideas?
Try var TheList = map[string]*Informations{}, why you cannot assign a value in a map please refer to why-do-i-get-a-cannot-assing-error or access-struct-in-map-without-copying

Write Struct to Json File using Struct Fields (not json keys)

How can I read a json file into a struct, and then Marshal it back out to a json string with the Struct fields as keys (rather than the original json keys)?
(see Desired Output to Json File below...)
Code:
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
)
type Rankings struct {
Keyword string `json:"keyword"`
GetCount uint32 `json:"get_count"`
Engine string `json:"engine"`
Locale string `json:"locale"`
Mobile bool `json:"mobile"`
}
func main() {
var jsonBlob = []byte(`
{"keyword":"hipaa compliance form", "get_count":157, "engine":"google", "locale":"en-us", "mobile":false}
`)
rankings := Rankings{}
err := json.Unmarshal(jsonBlob, &rankings)
if err != nil {
// nozzle.printError("opening config file", err.Error())
}
rankingsJson, _ := json.Marshal(rankings)
err = ioutil.WriteFile("output.json", rankingsJson, 0644)
fmt.Printf("%+v", rankings)
}
Output on screen:
{Keyword:hipaa compliance form GetCount:157 Engine:google Locale:en-us Mobile:false}
Output to Json File:
{"keyword":"hipaa compliance form","get_count":157,"engine":"google","locale":"en-us","mobile":false}
Desired Output to Json File:
{"Keyword":"hipaa compliance form","GetCount":157,"Engine":"google","Locale":"en-us","Mobile":false}
If I understand your question correctly, all you want to do is remove the json tags from your struct definition.
So:
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
)
type Rankings struct {
Keyword string
GetCount uint32
Engine string
Locale string
Mobile bool
}
func main() {
var jsonBlob = []byte(`
{"keyword":"hipaa compliance form", "get_count":157, "engine":"google", "locale":"en-us", "mobile":false}
`)
rankings := Rankings{}
err := json.Unmarshal(jsonBlob, &rankings)
if err != nil {
// nozzle.printError("opening config file", err.Error())
}
rankingsJson, _ := json.Marshal(rankings)
err = ioutil.WriteFile("output.json", rankingsJson, 0644)
fmt.Printf("%+v", rankings)
}
Results in:
{Keyword:hipaa compliance form GetCount:0 Engine:google Locale:en-us Mobile:false}
And the file output is:
{"Keyword":"hipaa compliance form","GetCount":0,"Engine":"google","Locale":" en-us","Mobile":false}
Running example at http://play.golang.org/p/dC3s37HxvZ
Note: GetCount shows 0, since it was read in as "get_count". If you want to read in JSON that has "get_count" vs. "GetCount", but output "GetCount" then you'll have to do some additional parsing.
See Go- Copy all common fields between structs for additional info about this particular situation.
Try to change the json format in the struct
type Rankings struct {
Keyword string `json:"Keyword"`
GetCount uint32 `json:"Get_count"`
Engine string `json:"Engine"`
Locale string `json:"Locale"`
Mobile bool `json:"Mobile"`
}
An accourance happened by just using json.Marshal() / json.MarshalIndent().
It overwrites the existing file, which in my case was suboptimal. I just wanted to add content to current file, and keep old content.
This writes data through a buffer, with bytes.Buffer type.
This is what I gathered up so far:
package srf
import (
"bytes"
"encoding/json"
"os"
)
func WriteDataToFileAsJSON(data interface{}, filedir string) (int, error) {
//write data as buffer to json encoder
buffer := new(bytes.Buffer)
encoder := json.NewEncoder(buffer)
encoder.SetIndent("", "\t")
err := encoder.Encode(data)
if err != nil {
return 0, err
}
file, err := os.OpenFile(filedir, os.O_RDWR|os.O_CREATE, 0755)
if err != nil {
return 0, err
}
n, err := file.Write(buffer.Bytes())
if err != nil {
return 0, err
}
return n, nil
}
This is the execution of the function, together with the standard json.Marshal() or json.MarshalIndent() which overwrites the file
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
minerals "./minerals"
srf "./srf"
)
func main() {
//array of Test struct
var SomeType [10]minerals.Test
//Create 10 units of some random data to write
for a := 0; a < 10; a++ {
SomeType[a] = minerals.Test{
Name: "Rand",
Id: 123,
A: "desc",
Num: 999,
Link: "somelink",
People: []string{"John Doe", "Aby Daby"},
}
}
//writes aditional data to existing file, or creates a new file
n, err := srf.WriteDataToFileAsJSON(SomeType, "test2.json")
if err != nil {
log.Fatal(err)
}
fmt.Println("srf printed ", n, " bytes to ", "test2.json")
//overrides previous file
b, _ := json.MarshalIndent(SomeType, "", "\t")
ioutil.WriteFile("test.json", b, 0644)
}
Why is this useful?
File.Write() returns bytes written to the file! So this is perfect if you want to manage memory or storage.
WriteDataToFileAsJSON() (numberOfBytesWritten, error)

Getting json dynamic key name as string?

For example:
{"id":
{"12345678901234":
{"Account":"asdf",
"Password":"qwerty"
"LastSeen":"1397621470",
}
}
}
A program I've been trying to make needs to get the id as a string and then later use it to check the time in LastSeen.
I've tried using simplejson and jsonq,but still cant figure out how to do that.
You can use RawMessage and make it much simpiler (play with it) :
package main
import (
"encoding/json"
"fmt"
)
var data []byte = []byte(`{"id": {"12345678901234": {"Account":"asdf", "Password":"qwerty", "LastSeen":"1397621470"}}}`)
type Message struct {
Id string
Info struct {
Account string
Password string
LastSeen string
}
}
func main() {
var (
tmpmsg struct {
Data map[string]json.RawMessage `json:"id"`
}
msg Message
)
if err := json.Unmarshal(data, &tmpmsg); err != nil {
panic(err) //you probably wanna use or something instead
}
for id, raw := range tmpmsg.Data {
msg.Id = id
if err := json.Unmarshal(raw, &msg.Info); err != nil {
panic(err)
}
}
fmt.Printf("%+v\n", msg)
}
Looking at the Golang blog post on JSON here it can be done using the encoding/json package. I created a small program to do this as follows:
package main
import (
"encoding/json"
"fmt"
)
var data []byte = []byte(`{"id": {"12345678901234": {"Account":"asdf", "Password":"qwerty", "LastSeen":"1397621470"}}}`)
type Message struct {
id string
LastSeen int64
}
var m Message
func main() {
var i interface {}
err := json.Unmarshal(data, &i)
if err != nil {
println("Error decoding data")
fmt.Printf("%s", err.Error())
return
}
m := i.(map[string]interface{})
for k, v := range m {
println(k)
im := v.(map[string]interface{})
for ik, iv := range im {
println("\t", ik)
jm := iv.(map[string]interface{})
for jk, jv := range jm {
println("\t\t", jk, ": ", jv.(string))
}
}
}
}
I apologise if this is poor in terms of Go best practices and such, I am new to the language. And I know that some elements of this aren't entirely necessary like the Message type definition but this works, at least on your data.