I am trying to create a JSON file with Golang. I have little knowledge of JSON Files and creating them but I have created a program that creates them. In this program, it takes form data from a website and then puts the data into a JSON struct, then adds the information into a folder. I have the 2 sections of data here. I put a comment where the error occurs along with the error
{
"cl":"[v1]",
"gr":"[A]",
"cr":"[8]"
} // End Of File Expected
{
"cl":"[v2]",
"gr":"[Z]",
"cr":"[8]"
}
So my questions are (1) What does the error mean, and (2) How/can I fix this when creating a JSON file with Golang? I can supply the Golang if needed.
So other than the json not being formatted correctly, here is an example of how to create json using a struct and json struct tags.
Proper JSON
[
{key:value, key value},
{key:value, key value}
]
What you have is
{key:value, key value}
{key:value, key value}
Which is two separate objects instead of one object in an array.
If you are reading this from file and the data is returned like your example then you might have to split on the newline to separate each object and unmarshal them separately.
Otherwise the below should server as an example.
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"strconv"
)
type j struct {
Cl []string `json:"cl"`
Gr []string `json:"gr"`
Cr []string `json:"cr"`
}
func main() {
// create an instance of j as a slice
var data []j
// using a for loop for create dummy data fast
for i := 0; i < 5; i++ {
v := strconv.Itoa(i)
data = append(data, j{
Cl: []string{"foo " + v},
Gr: []string{"bar " + v},
Cr: []string{"foobar " + v},
})
}
// printing out json neatly to demonstrate
b, _ := json.MarshalIndent(data, "", " ")
fmt.Println(string(b))
// writing json to file
_ = ioutil.WriteFile("file.json", b, 0644)
// to append to a file
// create the file if it doesn't exists with O_CREATE, Set the file up for read write, add the append flag and set the permission
f, err := os.OpenFile("/var/log/debug-web.log", os.O_CREATE|os.O_RDWR|os.O_APPEND, 0660)
if err != nil {
log.Fatal(err)
}
// write to file, f.Write()
f.Write(b)
// if you are doing alot of I/O work you may not want to write out to file often instead load up a bytes.Buffer and write to file when you are done... assuming you don't run out of memory when loading to bytes.Buffer
}
Related
I have a 8 gigs CSV file that i need to unmarshal to a list of struct
package main
import (
"encoding/csv"
"fmt"
"io"
"os"
gocsv "github.com/gocarina/gocsv"
dto "github.com/toto/GeoTransport/import/dto"
)
// Put in parameter json the csv names
func importAdresse() {
var adressesDB []dto.GeoAdresse
clientsFile, err := os.OpenFile("../../../data/geo/public.geo_adresse.csv", os.O_RDWR|os.O_CREATE, os.ModePerm)
if err != nil {
panic(err)
}
gocsv.SetCSVReader(func(in io.Reader) gocsv.CSVReader {
r := csv.NewReader(in)
r.Comma = ';'
return r // Allows use pipe as delimiter
})
if err = gocsv.UnmarshalFile(clientsFile, &adressesDB); err != nil { // Load clients from file
panic(err)
}
var i int
i = 0
for _, adresse := range adressesDB {
fmt.Println("adresse.Numero")
fmt.Printf("%+v\n", adresse)
fmt.Println(adresse.Numero)
i++
if i == 3 {
break
}
}
}
func init() {
}
func main() {
importAdresse()
}
Actually I am using go csv to unmarshall it but I have some memory error.
The program quit because it does not have enough ram.
I would like to know how to read the csv line by line and unmarshal it to a struct.
One of the solution will be to split the CSV file with some unix command.
But I would like to know how to do it with only Go.
It looks like the parsing method you're using attempts to read the entire CSV file into memory. You might try using the standard CSV reader package directly, or using another CSV-to-struct library that allows for line-by-line decoding like this one. Does the example code on those pages show what you're looking for?
Another thing to try would be running wc -l ../../../data/geo/public.geo_adresse.csv to get the number of lines in your CSV file, then write this:
var adressesDB [<number of lines in your CSV>]dto.GeoAdresse
If the runtime raises the out of memory exception on that line, it means that the unmarshalled CSV data exceeds your RAM capacity and you'll have to read it in chunks.
In powershell, if I make a REST call and receive any kind of json response, I can easily $json | ConvertFrom-Json into a proper object so I can make modifications, render specific values, whatever.
It seems like in Go I have to either define a struct, or "dynamically" convert using a map[string]interface{}.
The issue with a struct is that I am writing a rest handler for a platform that, depending on the endpoint, serves wildly different JSON responses, like most REST APIs. I don't want to define a struct for all of the dozens of possible responses.
The problem with map[string]interface{} is that it pollutes the data by generating a string with a bunch of ridiculous 'map' prefixes and unwanted [brackets].
ala: [map[current_user_role:admin id:1]]
Is there a way to convert a JSON response like:
{
"current_user_role": "admin",
"id": 1
}
To return a basic:
current_user_role: admin
id: 1
... WITHOUT defining a struct?
Your approach of using a map is right if you don't wish to specify the structure of the data you're receiving. You don't like how it is output from fmt.Println, but I suspect you're confusing the output format with the data representation. Printing them out in the format you find acceptable takes a couple of lines of code, and is not as convenient as in python or powershell, which you may find annoying.
Here's a working example (playground link):
package main
import (
"encoding/json"
"fmt"
"log"
)
var data = []byte(`{
"current_user_role": "admin",
"id": 1
}`)
func main() {
var a map[string]interface{}
if err := json.Unmarshal(data, &a); err != nil {
log.Fatal(err)
}
for k, v := range a {
fmt.Printf("%s: %v\n", k, v)
}
}
Following this tutorial I'm trying to read a json file in Golang. It says there are two ways of doing that:
unmarshal the JSON using a set of predefined structs
or unmarshal the JSON using a map[string]interface{}
Since I'll probably have a lot of different json formats I prefer to interpret it on the fly. So I now have the following code:
package main
import (
"fmt"
"os"
"io/ioutil"
"encoding/json"
)
func main() {
// Open our jsonFile
jsonFile, err := os.Open("users.json")
// if we os.Open returns an error then handle it
if err != nil {
fmt.Println(err)
}
fmt.Println("Successfully Opened users.json")
// defer the closing of our jsonFile so that we can parse it later on
defer jsonFile.Close()
byteValue, _ := ioutil.ReadAll(jsonFile)
var result map[string]interface{}
json.Unmarshal([]byte(byteValue), &result)
fmt.Println(result["users"])
fmt.Printf("%T\n", result["users"])
}
This prints out:
Successfully Opened users.json
[map[type:Reader age:23 social:map[facebook:https://facebook.com twitter:https://twitter.com] name:Elliot] map[name:Fraser type:Author age:17 social:map[facebook:https://facebook.com twitter:https://twitter.com]]]
[]interface {}
At this point I don't understand how I can read the age of the first user (23). I tried some variations:
fmt.Println(result["users"][0])
fmt.Println(result["users"][0].age)
But apparently, type interface {} does not support indexing.
Is there a way that I can access the items in the json without defining the structure?
Probably you want
fmt.Println(result["users"].(map[string]interface{})["age"])
or
fmt.Println(result[0].(map[string]interface{})["age"])
As the JSON is a map of maps the type of the leaf nodes is interface{} and so has to be converted to map[string]interface{} in order to lookup a key
Defining a struct is much easier. My top tip for doing this is to use a website that converts JSON to a Go struct definition, like Json-To-Go
I have seen lots of posts on here about converting from XML to JSON, and I've recently wrote a program to do so, but I was also curious how you would go about converting from JSON to XML?
Sample JSON:
"version":"0.1",
"termsofService":"http://www.wunderground.com/weather/api/d/terms.html",
"features": {
"conditions": 1
}
}
, "current_observation": {
"image": {
"url":"http://icons.wxug.com/graphics/wu2/logo_130x80.png",
"title":"Weather Underground",
"link":"http://www.wunderground.com"
},
"display_location": {
"full":"Kearney, MO",
"city":"Kearney",
"state":"MO",
"state_name":"Missouri",
I'm not sure if it'd be any use to you, but i'll post my JSON to XML program.
package main
import (
"fmt"
"net/url"
"encoding/xml"
"net/http"
"log"
"io/ioutil"
"encoding/json"
)
type reportType struct{
Version xml.CharData `xml:"version"`
TermsOfService xml.CharData `xml:"termsofService"
`
Features xml.CharData `xml:"features>feature"`
Full string `xml:"current_observation>display_location>full"`
StateName string `xml:"current_observation>display_location>state_name"`
WindGust string `xml:"current_observation>observation_location>full"`
Problem myErrorType `xml:"error"`
}
type myErrorType struct{
TypeOfError xml.CharData `xml:"type"`
Desciption xml.CharData `xml:"description"`
}
type reportTypeJson struct{
Version string `json:"version"`;
TermsOfService string `json:"termsofService"`;
Features map[string]string `json:"features"`;
CurrentObservation map[string]map[string]string `json:"current_observation"`
}
func main() {
fmt.Println("data is from WeatherUnderground.")
fmt.Println("https://www.wunderground.com/")
var state, city string
str1 := "What is your state?"
str2 := "What is your city?"
fmt.Println(str1)
fmt.Scanf("%s", &state)
fmt.Println(str2)
fmt.Scanf("%s", &city)
baseURL := "http://api.wunderground.com/api/";
apiKey := "Nunna"
var query string
//set up the query
query = baseURL+apiKey +
"/conditions/q/"+
url.QueryEscape(state)+ "/"+
url.QueryEscape(city)+ ".xml"
fmt.Println("The escaped query: "+query)
response, err := http.Get(query)
doErr(err, "After the GET")
var body []byte
body, err = ioutil.ReadAll(response.Body)
doErr(err, "After Readall")
fmt.Println(body);
fmt.Printf("The body: %s\n",body)
//Unmarshalling
var report reportType
xml.Unmarshal(body, &report)
fmt.Printf("The Report: %s\n", report)
fmt.Printf("The description is [%s]\n",report.Problem.Desciption)
//Now marshal the data out in JSON
var data []byte
var output reportTypeJson
output.Version = string(report.Version);
output.TermsOfService = string(report.TermsOfService)
output.Features= map[string]string{"feature":string(report.Features)} // allocate a map, add the 'features' value to it and assign it to output.Features
output.CurrentObservation = map[string]map[string]string {
"display_location": map[string]string {
"full": report.Full,
"state_name": report.StateName,
},"observation_location":map[string]string {"full": report.WindGust},
}
data,err = json.MarshalIndent(output,""," ")
doErr(err, "From marshalIndent")
fmt.Printf("JSON output nicely formatted: \n%s\n",data)
}
func doErr( err error, message string){
if err != nil{
log.Panicf("ERROR: %s %s \n", message, err.Error())
}
}
//OUTPUT:
//JSON output nicely formatted:
//{
// "version": "0.1",
// "termsofService": "http://www.wunderground.com/weather/api/d/terms.html",
// "features": {
// "feature": "conditions"
// },
// "current_observation": {
// "display_location": {
// "full": "Kearney, MO",
// "state_name": "Missouri"
// },
// "observation_location": {
// "full": "HOMESTEAD HILLS, Kearney, Missouri"
// }
// }
//}
This is the same process as going in the other direction. Define structs/objects to model the input (be it json in this case), create method to assign all the values from the struct modeling the input to the one you're using for output, then marshal the output to get a string. So to give a practice example using one of the more conceptually difficult fields from you type, having unmarshalled some json into an instance of reportTypeJson I can assign to reportType used to model xml like so;
report.StateName = jsonReport.CurrentObservation["display_location"]["state_name"]
The biggest difference here is the struct representing your xml is flat (like it has Something>InnerSomething>InnerInnerSomething to represent it's nested values while the struct in golang has no nesting) where as with the json, your structures in golang tend to have the same amount of nesting (like having a map[string]map[string][string] meaning items are nested 3 levels inside of the main struct). You can observe this by the amount of indirection when you access fields, like in the example above, there is one level of indirection to access CurrentObservation but that is a map of maps, so then I index into it with the display_location key which yields a map[string]string, since I'm looking for the statename value, I have to index into that with state_name to access that value.
Note that in an actual program, a number of checks would be required because these operations are unsafe. For example, if the json read did not contain a display_location object then jsonReport.CurrentObservation["display_location"] would return nil and the attempt to access ["state_name" would result in a panic.
Also, another side note, in your program I would recommend adding two functions, one called NewJsonReport(report *reportType) *reportTypeJson, err and one called NewXmlReport(report *reportTypeJson) *reportType, err in which you initialize/allocate the a new instance of the return type and return it as to avoid code duplication and make your programs main more readable. Doing this type of assignment in more than one location in a program is sloppy coding which will add a lot of maintenance cost and is likely to result in bugs down the line (like if something changes on one of the models or the input you have to fix every reference to it throughout the program rather than just updating the functions I mentioned above).
I'm new in golang development and have some question regarding something related to this question.
As a learning exercise, I'm trying to create a simple library to handle json based configuration file. As a configuration file to be used for more then one app, it should be able to handle different parameters. Then I have created a type struct Configuration that has the filename and a data interface. Each app will have a struct based on its configuration needs.
In the code bellow, I put all together (lib and "main code") and the "TestData struct" is the "app parameters".
If it doesn't exists, it will set a default values and create the file, and it is working. But when I try to read the file. I try to decode the json and put it back into the data interface. But it is giving me an error and I couldn't figure out how to solve this. Can someone help on this?
[updated] I didn't put the targeted code before, because I though that it would be easier to read in in all as a single program. Bellow is the 'targeted code' for better view of the issue.
As I will not be able to use the TestData struct inside the library, since it will change from program to program, the only way to handle this was using interface. Is there a better way?
library config
package config
import (
"encoding/json"
"fmt"
"os"
)
// Base configuration struct
type Configuration struct {
Filename string
Data interface{}
}
func (c *Configuration) Create(cData *Configuration) bool {
cFile, err := os.Open(cData.Filename)
defer cFile.Close()
if err == nil {
fmt.Println("Error(1) trying to create a configuration file. File '", cData.Filename, "' may already exist...")
return false
}
cFile, err = os.Create(cData.Filename)
if err != nil {
fmt.Println("Error(2) trying to create a configuration file. File '", cData.Filename, "' may already exist...")
return false
}
buffer, _ := json.MarshalIndent(cData.Data, "", "")
cFile.Write(buffer)
return true
}
func (c *Configuration) Read(cData *Configuration) bool {
cFile, err := os.Open(cData.Filename)
defer cFile.Close()
if err != nil {
fmt.Println("Error(1) trying to read a configuration file. File '", cData.Filename, "' may not already exist...")
return false
}
jConfig := json.NewDecoder(cFile)
jerr := jConfig.Decode(&cData.Data)
if jerr != nil {
panic(jerr)
}
return true
}
program using library config
package main
import (
"fmt"
"./config"
)
// struct basic para configuraĆ§Ć£o
type TestData struct {
URL string
Port string
}
func main() {
var Config config.Configuration
Config.Filename = "config.json"
if !Config.Read(&Config) {
Config.Data = TestData{"http", "8080"}
Config.Create(&Config)
}
fmt.Println(Config.Data)
TestData1 := &TestData{}
TestData1 = Config.Data.(*TestData) // error, why?
fmt.Println(TestData1.URL)
}
NEW UPDATE:
I have made some changes after JimB comment about I'm not clear about some concepts and I tried to review it. Sure many things aren't clear for me yet unfortunately. The "big" understanding I believe I got, but what mess my mind up is the "ins" and "outs" of values and formats and pointers, mainly when it goes to other libraries. I'm not able yet to follow the "full path" of it.
Yet, I believe I had some improvement on my code.
I think that I have corrected some points, but still have some big questions:
I stopped sending "Configuration" as a parameter as all "data" were already there as they are "thenselfs" in the instance. Right?
Why do I have use reference in the line 58 (Config.Data = &TestData{})
Why to I have to use pointer in the line 64 (tmp := Config.Data.(*TestData)
Why I CANNOT use reference in line 69 (Config.Data = tmp)
Thanks
The reason you are running into an error is because you are trying to decode into an interface{} type. When dealing with JSON objects, they are decoded by the encoding/json package into map[string]interface{} types by default. This is causing the type assertion to fail since the memory structure for a map[string]interface{} is much different than that of a struct.
The better way to do this is to make your TestData struct the expected data format for your Configuration struct:
// Base configuration struct
type Configuration struct {
Filename string
Data *TestData
}
Then when Decoding the file data, the package will unmarshal the data into the fields that match the closest with the data it finds.
If you need more control over the data unmarshaling process, you can dictate which JSON fields get decoded into which struct members by using struct tags. You can read more about the json struct tags available here: https://golang.org/pkg/encoding/json/#Marshal
You are trying to assert that Config.Data is of type *TestData, but you're assigning it to TestData{"http", "8080"} above. You can take the address of a composite literal to create a pointer:
Config.Data = &TestData{"http", "8080"}
If your config already exsits, your Read method is going to fill in the Data field with the a default json data type, probably a map[string]interface{}. If you assign a pointer of the correct type to Data first, it will decode into the expected type.
Config.Data = &TestData{}
Ans since Data is an interface{}, you do not want to ever use a pointer to that value, so don't use the & operator when marshaling and unmarshaling.