This question already has answers here:
How to iterate through a map in Golang in order?
(5 answers)
Closed 4 years ago.
How to parse this json using Go?
timelinedata := '{
"2016-08-17T00:00:00.000Z": 4,
"2016-11-02T00:00:00.000Z": 1,
"2017-08-30T00:00:00.000Z": 1
} '
I want the dates and the values in separate variables by looping over the json.
Currently I am doing it in this way
var timeline map[string]int
json.Unmarshal([]byte(timelinedata),
for k, v := range timeline {
new_key := k
new_val := v
println("val--->>", new_key, new_val)
}
The problem is that the output is not in proper order as like the json input is. Every time I run the loop the output order varies. I want to map the json in exact order as like the input. I think I am not maping it in a proper way---
You should not assume that the key order in a JSON object means anything:
From the introduction of RFC 7159 (emphasis mine):
An object is an unordered collection of zero or more name/value pairs, where a name is a string and a value is a string, number, boolean, null, object, or array.
An array is an ordered sequence of zero or more values.
In addition to that, you shouldn't assume that the producer of the JSON document has been in control of the key/value order; maps are unordered in most languages, so it mostly comes down to the used encoding library. If the producer did care about the order, they would use an array.
That being said, if you really are interested in the order of the JSON keys, you have to decode the object piece by piece, using json.Decoder.Token:
package main
import (
"encoding/json"
"fmt"
"log"
"strings"
)
func main() {
j := `{
"2016-08-17T00:00:00.000Z": 4,
"2016-11-02T00:00:00.000Z": 1,
"2017-08-30T00:00:00.000Z": 1
}`
dec := json.NewDecoder(strings.NewReader(j))
for dec.More() {
t, err := dec.Token()
if err != nil {
log.Fatal(err)
}
switch t := t.(type) {
case json.Delim:
// no-op
case string:
fmt.Printf("%s => ", t)
case float64:
fmt.Printf("%.0f\n", t)
case json.Number:
fmt.Printf(" %s\n", t)
default:
log.Fatalf("Unexpected type: %T", t)
}
}
}
Try it on the Playground: https://play.golang.org/p/qfXcOfOvKws
Related
This question already has answers here:
How to convert bson to json effectively with mongo-go-driver?
(1 answer)
convert result into JSON without structs using mongo-go-driver
(1 answer)
How can I get JSON from BSON without my keys all being named "Key"?
(1 answer)
Closed 3 years ago.
I am struggling to create a valid JSON string from a BSON document in Go for an API.
Let's say I have an object like this:
type MyObject struct {
Name string
}
I call my database which returns to me a cursor containing many documents as: [{"Name": "object_name"}, ...]
I am able to retrieve all the documents via a loop like
for cur.Next(ctx) {
var obj MyObject
err := cur.Decode(&obj)
//then display error if there's one
}
And now I would like to end up with a JSON string containing all the documents my database returned in order to send it via HTTP.
Because, if use I fmt.Println(obj)I end up with something like this: [{object1_name} {object2_name} ...] which is, according to me, not a valid format that I can use for an API.
I know json.Marshal(obj) can actually encode to valid JSON and I can decode it with os.Stdout.Write(obj) but I didn't manage to store this valid string in a variable. How can I manage to do this?
From Golang documentation for json package
package main
import (
"encoding/json"
"fmt"
)
func main() {
type ColorGroup struct {
ID int `json:"id"`
Name string `json:"name"`
Colors []string `json:"colors"`
}
group := ColorGroup{
ID: 1,
Name: "Reds",
Colors: []string{"Crimson", "Red", "Ruby", "Maroon"},
}
b, err := json.Marshal(group)
if err != nil {
fmt.Println("error:", err)
} else {
str := string(b)
fmt.Println("stringified json is:", str)
}
}
Output
stringified json is: {"id":1,"name":"Reds","colors":["Crimson","Red","Ruby","Maroon"]}
The json.Marshal return two values - a byte array and error
If error is nil then you can obtain the string by converting byte array to string using
str := string(b)
I have an application that consumes data from a third-party api. I need to decode the json into a struct, which requires the struct to have json tags of the "incoming" json fields. The outgoing json fields have a different naming convention, so I need different json tags for the encoding.
I will have to do this with many different structs, and each struct might have many fields.
What is the best way to accomplish this without repeating a lot of code?
Example Structs:
// incoming "schema" field names
type AccountIn struct {
OpenDate string `json:"accountStartDate"`
CloseDate string `json:"cancelDate"`
}
// outgoing "schema" field names
type AccountOut struct {
OpenDate string `json:"openDate"`
CloseDate string `json:"closeDate"`
}
Maybe the coming change on Go 1.8 would help you, it will allow to 'cast' types even if its JSON tags definition is different: This https://play.golang.org/p/Xbsoa8SsEk works as expected on 1.8beta, I guess this would simplify your current solution
A bit an uncommon but probably quite well working method would be to use a intermediate format so u can use different readers and writers and therefore different tags. For example https://github.com/mitchellh/mapstructure which allows to convert a nested map structure into struct
types. Pretty similar like json unmarshal, just from a map.
// incoming "schema" field names
type AccountIn struct {
OpenDate string `mapstructure:"accountStartDate" json:"openDate"`
CloseDate string `mapstructure:"cancelDate" json:"closeDate"`
}
// from json to map with no name changes
temporaryMap := map[string]interface{}{}
err := json.Unmarshal(jsonBlob, &temporaryMap)
// from map to structs using mapstructure tags
accountIn := &AccountIn{}
mapstructure.Decode(temporaryMap, accountIn)
Later when writing (or reading) u will use directly the json functions which will then use the json tags.
If it's acceptable to take another round trip through json.Unmarshal and json.Marshal, and you don't have any ambiguous field names within your various types, you could translate all the json keys in one pass by unmarshaling into the generic structures used by the json package:
// map incoming to outgoing json identifiers
var translation = map[string]string{
"accountStartDate": "openDate",
"cancelDate": "closeDate",
}
func translateJS(js []byte) ([]byte, error) {
var m map[string]interface{}
if err := json.Unmarshal(js, &m); err != nil {
return nil, err
}
translateKeys(m)
return json.MarshalIndent(m, "", " ")
}
func translateKeys(m map[string]interface{}) {
for _, v := range m {
if v, ok := v.(map[string]interface{}); ok {
translateKeys(v)
}
}
keys := make([]string, 0, len(m))
for k := range m {
keys = append(keys, k)
}
for _, k := range keys {
if newKey, ok := translation[k]; ok {
m[newKey] = m[k]
delete(m, k)
}
}
}
https://play.golang.org/p/nXmWlj7qH9
This might be a Naive Approach but is fairly easy to implement:-
func ConvertAccountInToAccountOut(AccountIn incoming) (AccountOut outcoming){
var outcoming AccountOut
outcoming.OpenDate = incoming.OpenDate
outcoming.CloseDate = incoming.CloseDate
return outcoming
}
var IncomingJSONData AccountIn
resp := getJSONDataFromSource() // Some method that gives you the Input JSON
err1 := json.UnMarshall(resp,&IncomingJSONData)
OutGoingJSONData := ConvertAccountInToAccountOut(IncomingJSONData)
if err1 != nil {
fmt.Println("Error in UnMarshalling JSON ",err1)
}
fmt.Println("Outgoing JSON Data: ",OutGoingJSONData)
I am new to golang and json and currently struggle to parse the json out from a system.
I've read a couple of blog posts on dynamic json in go and also tried the tools like json2GoStructs
Parsing my json file with this tools just gave me a huge structs which I found a bit messy. Also I had no idea how to get the info im interested in.
So, here are my problems:
How do I get to the info I am interested in?
What is the best approach to parse complex json?
I am only interested into the following 3 json fields:
Name
Guid
Task -> Property -> Name: Error
I'm thankful for every tip, code snippet or explanation!
This is what I got so far (mostly from a tutorial):
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
)
func checkErr(err error) {
if err != nil {
panic(err)
}
}
func readFile(filePath string) []byte {
data, err := ioutil.ReadFile(filePath)
checkErr(err)
return data
}
func main() {
path := "/Users/andi/Documents/tmp/wfsJob.json"
data := readFile(path)
var f interface{}
err := json.Unmarshal(data, &f)
checkErr(err)
m := f.(map[string]interface{})
for k, v := range m {
switch vv := v.(type) {
case string:
fmt.Println(k, "is string", vv)
case int:
fmt.Println(k, "is int", vv)
case []interface{}:
fmt.Println(k, "is an array:")
for i, u := range vv {
fmt.Println(i, u)
}
default:
fmt.Println(k, "is of a type I don't know how to handle")
}
}
}
I can offer you this easy way to using JSON in Golang. With this tool you don't need to parse the whole json file, and you can use it without struct.
Gjson is a great solution for fetching a few fields from JSON string. But it may become slow when many (more than 2) fields must be fetched from distinct parts of the JSON, since it re-parses the JSON on each Get call. Additionally, it requires calling gjson.Valid for validating the incoming JSON, since other methods assume the caller provides valid JSON.
There is an alternative package - fastjson. Like gsjon, it is fast and has nice API. Unlike gjson it validates the input JSON and works faster when many unrelated fields must be obtained from the JSON. Here is a sample code for obtaining fields from the original question:
var p fastjson.Parser
v, err := p.ParseBytes(data)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Name: %s", v.GetStringBytes("Name"))
fmt.Printf("Guid: %s", v.GetStringBytes("Guid"))
fmt.Printf("Error: %s", v.GetStringBytes("Task", "Property", "Name"))
This question already has answers here:
JSON and dealing with unexported fields
(2 answers)
(un)marshalling json golang not working
(3 answers)
json.Marshal(struct) returns "{}"
(3 answers)
Printing Empty Json as a result [duplicate]
(1 answer)
Parsing JSON in Golang doesn't Populate Object [duplicate]
(1 answer)
Closed 10 months ago.
I'm a new Go programmer (From Java) and I would like to reproduce a generic way which is esay to use in Java.
I want to create some function which allow me to do an Unmarshal on a JSON string in order to avoid code duplicity.
This is my current code which is not working :
type myStruct1 struct {
id string
name string
}
func (obj myStruct1) toString() string {
var result bytes.Buffer
result.WriteString("id : ")
result.WriteString(obj.id)
result.WriteString("\n")
result.WriteString("name : ")
result.WriteString(obj.name)
return result.String()
}
func main() {
content := `{id:"id1",name="myName"}`
object := myStruct1{}
parseJSON(content, object)
fmt.Println(object.toString())
}
func parseJSON(content string, object interface{}) {
var parsed interface{}
json.Unmarshal([]byte(content), &parsed)
}
This code, on run, returns me this :
id :
name :
Do you have any idea ?
Thanks
The issue is you want to write to a generic type? You probably want a string map. This works with BSON anyways:
var anyJson map[string]interface{}
json.Unmarshal(bytes, &anyJson)
You'll be able to access the fields like so:
anyJson["id"].(string)
Don't forget to type assert your values, and they must be the correct type or they'll panic. (You can read more about type assertions on the golang site)
To parse "generic JSON" when you have no idea what schema it has:
var parsed any
err := json.Unmarshal(jsonText, &parsed)
The returned any in parsed will be a map[string]any or []any or nil or single values float64, bool, string.
You can test the type and react accordingly.
import (
"encoding/json"
"fmt"
)
func test(jsonText []byte) {
// parsing
var parsed any
err := json.Unmarshal(jsonText, &parsed)
if err != nil {
panic(err) // malformed input
}
// type-specific logic
switch val := parsed.(type) {
case nil:
fmt.Println("json specifies null")
case map[string]any:
fmt.Printf("id:%s name:%s\n", val["id"], val["name"])
case []any:
fmt.Printf("list of %d items\n", len(val))
case float64:
fmt.Printf("single number %f\n", val)
case bool:
fmt.Printf("single bool %v\n", val)
case string:
fmt.Printf("single string %s\n", val)
default:
panic(fmt.Errorf("type %T unexpected", parsed))
}
}
Unmarshal will only set exported fields of the struct.
Which means you need to modify the json struct to use capital case letters:
type myStruct1 struct {
Id string
Name string
}
The reason behind this is that the json library does not have the ability to view fields using reflect unless they are exported.
You have to export your fields:
type myStruct1 struct {
Id string
Name string
}
See Exported Identifiers from documentation.
There are a few changes you need to make in your code to make it work:
The function json.Unmarshal can only set variables inside your struct which are exported, that is, which start with capital letters. Use something like ID and Name for your variable names inside myStruct1.
Your content is invalid JSON. What you actually want is {"ID":"id1","Name":"myName"}.
You're passing object to parseJSON but you're using parsed instead, not object. Make parseJSON receive a *myStruct (instead of an interface{}), and use that variable instead of parsed when unmarshalling the string. Also, always handle the error returns, like err := json.Unmarshal(content, object), and check err.
I'd suggest you to do the Golang Tour ;)
You can also set the file as an Object with dynamic properties inside another struct. This will let you add metadata and you read it the same way.
type MyFile struct {
Version string
Data map[string]interface{}
}
I have a Json string that I want to unmarshal.
This is working:
jsonString := []byte(`{"my_int": 3, "my_string": null}`)
var data map[string]interface{}
err := json.Unmarshal(jsonString, &data)
if err != nil {
fmt.Println(err)
}
//avroJson := make(map[string]interface{})
for k, v := range data {
fmt.Printf("%v, %T\n", k, v)
}
My issue is: the value of my_int which is 3 is returned as float64.
My question is: how to parse a json string with the "minimum type" so that 3 will return int32 and not the maximum type 3 => float64?
Assumption: my Json is huge and only have primitive types and I want a minimum value that is really float64 to continue to show float64.
Clarification:
A "minimum type" means that if 3 can be considered both int32 and float64 the "minimum type" will be int32, which is the exact type you'll get when running this:
reflect.TypeOf(3).string()
Since you are unmarshaling into a map of interface{}, the following section of the golang json.Unmarshal documentation pertains:
To unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
...
float64, for JSON numbers
string, for JSON strings
...
As such, to unmarshal your sample data into your desired types you should define a struct type which contains the desired field/type mappings, for example:
type MyType struct {
MyInt int `json:"my_int"`
MyString *string `json:"my_string"`
}
foo := MyType{}
jsonstr := `{"my_int": 3, "my_string": null}`
err := json.Unmarshal([]byte(jsonstr), &foo)
if err != nil {
panic(err)
}
// foo => main.MyType{MyInt:3, MyString:(*string)(nil)}
Since you cannot describe your data in a struct then your options are to:
Use a json.Decoder to convert the values to your desired types as they are parsed.
Parse the document into a generic interface and post-process the value types.
Option #1 is the most flexible and can likely be implemented to be more performant than the other option since parsing and transformation could be performed in a single pass of the data.
Option #2 might be simpler but will require two passes over the data. Here is an example of what the post-processing step might look like:
func TransformValueTypes(o map[string]interface{}) {
for k, v := range o {
// Convert nil values to *string type.
if v == interface{}(nil) {
o[k] = (*string)(nil)
}
// Convert numbers to int32 if possible
if x, isnumber := v.(float64); isnumber {
if math.Floor(x) == x {
if x >= math.MinInt32 && x <= math.MaxInt32 {
o[k] = int32(x)
}
// Possibly check for other integer sizes here?
}
// Possibly check if float32 is possible here?
}
// Check for maps and slices here...
}
}
So if you call TransformValueTypes(data) then your types will look like:
// my_int -> 3 (int32)
// my_string -> <nil> (*string)
// my_string2 -> "foo" (string)
// my_float -> 1.23 (float64)
Of course, your transform function could also apply type transformation logic based on the key name.
Importantly, note that if your document might have additional structure not mentioned in your question (such as nested objects or arrays) then your transform function will need to account for them by more value type checking, recursive calls, and iteration.