How can I provide JSON array as argument to cobra cli - json

I'm building a CLI using Go and Cobra library. I've the following JSON that needs to be deserialized in the corresponding struct.
Argument as JSON array:
"[
(stringA, stringB),
stringC
]"
Struct
type MyStruct struct {
StringArray []string
}
I'm using Cobra's StringSicceVarP as follows to
cmd.PersistentFlags().StringSliceVarP(&opts.StringParam, "paramname", "", nil, `this is the description`)
But cobra is reading the incoming json as one single string [(stringA, stringB), stringC] whereas I want the array to be of length 2, such that
StringArray[0]: (stringA, stringB) and StringArray[1]:stringC.
I can't use the StringSliceVarP as it will split based on , which I don't want as my array string could itself has a ,.
How can I achieve this?

I personally advice you against this option. Supplying formatted data is conventionally done through reading STDIN or from a file. Such solution is usually more flexible by allowing you to add flags to specify the file's format (JSON, XML, etc.).
Supplying a filename instead of the raw JSON string in the arguments adds better interoperability with other software, and other benefits such as using the computer's disk for buffering data instead of the computer's memory/RAM.
My personal recommendations is that:
Use flags for options and configs, similar to a HTTP's query parameters.
Use stdin/file handles for data, similar to a HTTP's request body.
However, if you insist on using a flag:
Cobra does not have built-in support for JSON structures. However, the pflag package (the flag library used by Cobra) allows you to define custom value types to be used as flags through the pflag.(*FlagSet).Var() method. You have to make a new type that implements the pflag.Value interface:
type Value interface {
String() string
Set(string) error
Type() string
}
To make a custom JSON-parsing type, you could write the following to use the built-in encoding/json package:
import (
"encoding/json"
)
type JSONFlag struct {
Target interface{}
}
// String is used both by fmt.Print and by Cobra in help text
func (f *JSONFlag) String() string {
b, err := json.Marshal(f.Target)
if err != nil {
return "failed to marshal object"
}
return string(b)
}
// Set must have pointer receiver so it doesn't change the value of a copy
func (f *JSONFlag) Set(v string) error {
return json.Unmarshal([]byte(v), f.Target)
}
// Type is only used in help text
func (f *JSONFlag) Type() string {
return "json"
}
Then to use this new pflag.Value-compatible type, you may write something like this:
import (
"fmt"
"github.com/spf13/cobra"
)
type MyStruct struct {
StringArray []string
}
func init() {
var flagMyStringArray []string
var myCmd = &cobra.Command{
Use: "mycmd",
Short: "A brief description of your command",
Run: func(cmd *cobra.Command, args []string) {
myStruct := MyStruct{StringArray: flagMyStringArray}
fmt.Printf("myStruct.StringArray contains %d elements:\n", len(myStruct.StringArray))
for i, s := range myStruct.StringArray {
fmt.Printf("idx=%d: %q", i, s)
}
},
}
rootCmd.AddCommand(myCmd)
myCmd.Flags().Var(&JSONFlag{&flagMyStringArray}, "paramname", `this is the description`)
}
Example usage:
$ go run . mycmd --paramname 'hello'
Error: invalid argument "hello" for "--paramname" flag: invalid character 'h' looking for beginning of value
Usage:
test mycmd [flags]
Flags:
-h, --help help for mycmd
--paramname json this is the description
exit status 1
$ go run . mycmd --paramname '["(stringA, stringB)", "stringC"]'
myStruct.StringArray contains 2 elements:
idx=0: "(stringA, stringB)"
idx=1: "stringC"

Related

Preserve json.RawMessage through multiple marshallings

Background
I'm working with JSON data that must be non-repudiable.
The API that grants me this data also has a service to verify that the data originally came from them.
As best as I can tell, they do this by requiring that the complete JSON they originally sent needs to be supplied to them inside another JSON request, with no byte changes.
Issue
I can't seem to preserve the original JSON!
Because I cannot modify the original JSON, I have carefully preserved it as a json.RawMessage when unmarshalling:
// struct I unmarshal my original data into
type SignedResult struct {
Raw json.RawMessage `json:"random"`
Signature string `json:"signature"`
...
}
// struct I marshal my data back into
type VerifiedSignatureReq {
Raw json.RawMessage `json:"random"`
Signature string `json:"signature"`
}
// ... getData is placeholder for function that gets my data
response := SignedResult{}
x, _ := json.Unmarshal(getData(), &response)
// do some post-processing with SignedResult that does not alter `Raw` or `Signature`
// trouble begins here - x.Raw started off as json.RawMessage...
y := json.Marshal(VerifiedSignatureReq{Raw: x.Raw, Signature: x.Signature}
// but now y.Raw is base64-encoded.
The problem is that []bytes / RawMessages are base64-encoded when marshaled. So I can't use this method, because it completely alters the string.
I'm unsure how to ensure this string is correctly preserved. I had assumed that the json.RawMessage specification in my struct would survive the perils of marshaling an already marshaled instance because it implements the Marshaler interface, but I appear mistaken.
Things I've Tried
My next attempt was to try:
// struct I unmarshal my original data into
type SignedResult struct {
Raw json.RawMessage `json:"random"`
Signature string `json:"signature"`
...
}
// struct I marshal my data back into
type VerifiedSignatureReq {
Raw map[string]interface{} `json:"random"`
Signature string `json:"signature"`
}
// ... getData is placeholder for function that gets my data
response := SignedResult{}
x, _ := json.Unmarshal(getData(), &response)
// do some post-processing with SignedResult that does not alter `Raw` or `Signature`
var object map[string]interface{}
json.Unmarshal(x.Raw, &object)
// now correctly generates the JSON structure.
y := json.Marshal(VerifiedSignatureReq{Raw: object, Signature: x.Signature}
// but now this is not the same JSON string as received!
The issue with this approach is that there are minor byte-wise differences in the spacing between the data. It no longer looks exactly the same when catted to a file.
I cannot use string(x.Raw) either because it escapes certain characters when marshaled with \.
You will need a custom type with its own marshaler, in place of json.RawMessage for your VerifiedSignatureReq struct to use. Example:
type VerifiedSignatureReq {
Raw RawMessage `json:"random"`
Signature string `json:"signature"`
}
type RawMessage []byte
func (m RawMessage) MarshalJSON() ([]byte, error) {
return []byte(m), nil
}

XML into JSON Multiple Nesting

I'm attempting to write code to translate XML to JSON. The XML I'm trying to translate is as follows...
(Just a snippet)
`<version>0.1</version>
<termsofService>http://www.wunderground.com/weather/api/d/terms.html</termsofService>
<features>
<feature>conditions</feature>
</features>
<current_observation>
<image>
<url>http://icons.wxug.com/graphics/wu2/logo_130x80.png</url>
<title>Weather Underground</title>
<link>http://www.wunderground.com</link>
</image>
<display_location>
<full>Kearney, MO</full>
<city>Kearney</city>
<state>MO</state>
<state_name>Missouri</state_name>`
Current Code:
`package main
import (
"fmt"
"net/url"
"encoding/xml"
"net/http"
"log"
"io/ioutil"
"encoding/json"
)
type reportType struct{
Version xml.CharData `xml:"version"`
TermsOfService xml.CharData `xml:"termsofService"
`
Features xml.CharData `xml:"features>feature"`
Full xml.CharData `xml:"current_observation>display_location>full"`
StateName xml.CharData `xml:"current_observation>display_location>state_name"`
WindGust xml.CharData `xml:"current_observation>observation_location>full"`
Problem myErrorType `xml:"error"`
}
type myErrorType struct{
TypeOfError xml.CharData `xml:"type"`
Desciption xml.CharData `xml:"description"`
}
type reportTypeJson struct{
Version string `json:"version"`;
TermsOfService string `json:"termsofService"`;
Features map[string]string `json:"features"`;
Full map[string]string `json:"display_location"`;
WindGust map[string]string `json:"observation_location"`
}
func main() {
fmt.Println("data is from WeatherUnderground.")
fmt.Println("https://www.wunderground.com/")
var state, city string
str1 := "What is your state?"
str2 := "What is your city?"
fmt.Println(str1)
fmt.Scanf("%s", &state)
fmt.Println(str2)
fmt.Scanf("%s", &city)
baseURL := "http://api.wunderground.com/api/";
apiKey := "3hunna"
var query string
//set up the query
query = baseURL+apiKey +
"/conditions/q/"+
url.QueryEscape(state)+ "/"+
url.QueryEscape(city)+ ".xml"
fmt.Println("The escaped query: "+query)
response, err := http.Get(query)
doErr(err, "After the GET")
var body []byte
body, err = ioutil.ReadAll(response.Body)
doErr(err, "After Readall")
fmt.Println(body);
fmt.Printf("The body: %s\n",body)
//Unmarshalling
var report reportType
xml.Unmarshal(body, &report)
fmt.Printf("The Report: %s\n", report)
fmt.Printf("The description is [%s]\n",report.Problem.Desciption)
//Now marshal the data out in JSON
var data []byte
var output reportTypeJson
output.Version = string(report.Version);
output.TermsOfService = string(report.TermsOfService)
output.Features= map[string]string{"feature":string(report.Features)} // allocate a map, add the 'features' value to it and assign it to output.Features
output.Full=map[string]string{"full":string(report.Full),"state_name":string(report.StateName)}
output.WindGust=map[string]string{"full":string(report.WindGust)}
data,err = json.MarshalIndent(output,""," ")
doErr(err, "From marshalIndent")
fmt.Printf("JSON output nicely formatted: \n%s\n",data)
}
func doErr( err error, message string){
if err != nil{
log.Panicf("ERROR: %s %s \n", message, err.Error())
}
}
As you can see, I'm using maps to map one level nesting such as in the features case. But for in two level nesting cases such as xml:"current_observation>display_location>state_name", I can't figure out how to create the very first level, in this case current_observations. Would there be a way to somehow create a map of maps of sorts? Any and all ideas are much appreciated because I am very confused at the moment, Thanks for your time!
And the Output:
JSON output nicely formatted:
{
"version": "0.1",
"termsofService": "http://www.wunderground.com/weather/api/d/terms.html",
"features": {
"feature": "conditions"
},
"display_location": {
"full": "Kearney, MO",
"state_name": "Missouri"
},
"observation_location": {
"full": "Stonecrest, Kearney, Missouri"
}
}
You could use either structs or a map of maps. I'll give some examples of both, starting with the map of maps. The type would be declares as;
CurrentObservation map[string]map[string]string `json:"current_observation"`
In this case you have a map with strings as the keys and the value is another map that has string for both key and value. As a result when you marshal your json you will end up with something like;
"current_observation" {
"image": { // first level where the value is a map[string]string
"title":"Weather Underground" // inner map, string to string
}
}
If say you wanted to just print the title, you would do;
fmt.Println(reportJson.CurrentObservation["image"]["title"])
Since the data there looks fairly static you could also use structs instead. In which case you'd use something like this;
CurrentObservation CurrentObservation `json:"current_observation"`
type CurrentObservation struct {
Image Image `json:"image"`
}
type Image struct {
Url string `json:"url"`
Title string `json:"title"`
Link string `json:"link"`
}
Both options produce the same output though they could behave differently for different inputs. For example, if another version of current_observation were received as input that for example has another nested item in it call it... previous_observation then the map option would automatically unmarhsal this data as well where the struct options would exclude it since there would be no mapping to any object/type in Go.
Personally I prefer the struct route when possible but it varies from case to case. For your application the map is likely better since you're not working with the input (it comes in as xml) and you just want to print it, you don't really have to deal with the details of current_observation, if it has 3 objects inside it, they'll all be output as expected, if it were 5 it would be the same. With the structs you have to explicitly define every single field which isn't really necessary if you're just transforming the input. The advantage of the struct is more for use later on where you have type safety although in this case, I would say they're still fairly equivalent because for example anytime you want to access something in image, like CurrentObservation.Image.Title you'll have to perform a check to ensure Image is not nil, like;
if CurrentObservation.Image != nil {
fmt.Println(CurrentObservation.Image.Title)
}
With the map you basically have the same overhead, only you're checking for the existence of a key rather than checking if one of the inner structs is nil or not.
EDIT: example of initializing the map of maps using composite literal syntax;
reportJson.CurrentObservation := map[string]map[string]string {
"display_location": map[string]string {
"full": report.Full,
"state_name": report.StateName,
},
}

unmarshal generic json in Go [duplicate]

This question already has answers here:
JSON and dealing with unexported fields
(2 answers)
(un)marshalling json golang not working
(3 answers)
json.Marshal(struct) returns "{}"
(3 answers)
Printing Empty Json as a result [duplicate]
(1 answer)
Parsing JSON in Golang doesn't Populate Object [duplicate]
(1 answer)
Closed 10 months ago.
I'm a new Go programmer (From Java) and I would like to reproduce a generic way which is esay to use in Java.
I want to create some function which allow me to do an Unmarshal on a JSON string in order to avoid code duplicity.
This is my current code which is not working :
type myStruct1 struct {
id string
name string
}
func (obj myStruct1) toString() string {
var result bytes.Buffer
result.WriteString("id : ")
result.WriteString(obj.id)
result.WriteString("\n")
result.WriteString("name : ")
result.WriteString(obj.name)
return result.String()
}
func main() {
content := `{id:"id1",name="myName"}`
object := myStruct1{}
parseJSON(content, object)
fmt.Println(object.toString())
}
func parseJSON(content string, object interface{}) {
var parsed interface{}
json.Unmarshal([]byte(content), &parsed)
}
This code, on run, returns me this :
id :
name :
Do you have any idea ?
Thanks
The issue is you want to write to a generic type? You probably want a string map. This works with BSON anyways:
var anyJson map[string]interface{}
json.Unmarshal(bytes, &anyJson)
You'll be able to access the fields like so:
anyJson["id"].(string)
Don't forget to type assert your values, and they must be the correct type or they'll panic. (You can read more about type assertions on the golang site)
To parse "generic JSON" when you have no idea what schema it has:
var parsed any
err := json.Unmarshal(jsonText, &parsed)
The returned any in parsed will be a map[string]any or []any or nil or single values float64, bool, string.
You can test the type and react accordingly.
import (
"encoding/json"
"fmt"
)
func test(jsonText []byte) {
// parsing
var parsed any
err := json.Unmarshal(jsonText, &parsed)
if err != nil {
panic(err) // malformed input
}
// type-specific logic
switch val := parsed.(type) {
case nil:
fmt.Println("json specifies null")
case map[string]any:
fmt.Printf("id:%s name:%s\n", val["id"], val["name"])
case []any:
fmt.Printf("list of %d items\n", len(val))
case float64:
fmt.Printf("single number %f\n", val)
case bool:
fmt.Printf("single bool %v\n", val)
case string:
fmt.Printf("single string %s\n", val)
default:
panic(fmt.Errorf("type %T unexpected", parsed))
}
}
Unmarshal will only set exported fields of the struct.
Which means you need to modify the json struct to use capital case letters:
type myStruct1 struct {
Id string
Name string
}
The reason behind this is that the json library does not have the ability to view fields using reflect unless they are exported.
You have to export your fields:
type myStruct1 struct {
Id string
Name string
}
See Exported Identifiers from documentation.
There are a few changes you need to make in your code to make it work:
The function json.Unmarshal can only set variables inside your struct which are exported, that is, which start with capital letters. Use something like ID and Name for your variable names inside myStruct1.
Your content is invalid JSON. What you actually want is {"ID":"id1","Name":"myName"}.
You're passing object to parseJSON but you're using parsed instead, not object. Make parseJSON receive a *myStruct (instead of an interface{}), and use that variable instead of parsed when unmarshalling the string. Also, always handle the error returns, like err := json.Unmarshal(content, object), and check err.
I'd suggest you to do the Golang Tour ;)
You can also set the file as an Object with dynamic properties inside another struct. This will let you add metadata and you read it the same way.
type MyFile struct {
Version string
Data map[string]interface{}
}

How do I do this? Get an struct send to a func as interface and get it back as struct?

I'm new in golang development and have some question regarding something related to this question.
As a learning exercise, I'm trying to create a simple library to handle json based configuration file. As a configuration file to be used for more then one app, it should be able to handle different parameters. Then I have created a type struct Configuration that has the filename and a data interface. Each app will have a struct based on its configuration needs.
In the code bellow, I put all together (lib and "main code") and the "TestData struct" is the "app parameters".
If it doesn't exists, it will set a default values and create the file, and it is working. But when I try to read the file. I try to decode the json and put it back into the data interface. But it is giving me an error and I couldn't figure out how to solve this. Can someone help on this?
[updated] I didn't put the targeted code before, because I though that it would be easier to read in in all as a single program. Bellow is the 'targeted code' for better view of the issue.
As I will not be able to use the TestData struct inside the library, since it will change from program to program, the only way to handle this was using interface. Is there a better way?
library config
package config
import (
"encoding/json"
"fmt"
"os"
)
// Base configuration struct
type Configuration struct {
Filename string
Data interface{}
}
func (c *Configuration) Create(cData *Configuration) bool {
cFile, err := os.Open(cData.Filename)
defer cFile.Close()
if err == nil {
fmt.Println("Error(1) trying to create a configuration file. File '", cData.Filename, "' may already exist...")
return false
}
cFile, err = os.Create(cData.Filename)
if err != nil {
fmt.Println("Error(2) trying to create a configuration file. File '", cData.Filename, "' may already exist...")
return false
}
buffer, _ := json.MarshalIndent(cData.Data, "", "")
cFile.Write(buffer)
return true
}
func (c *Configuration) Read(cData *Configuration) bool {
cFile, err := os.Open(cData.Filename)
defer cFile.Close()
if err != nil {
fmt.Println("Error(1) trying to read a configuration file. File '", cData.Filename, "' may not already exist...")
return false
}
jConfig := json.NewDecoder(cFile)
jerr := jConfig.Decode(&cData.Data)
if jerr != nil {
panic(jerr)
}
return true
}
program using library config
package main
import (
"fmt"
"./config"
)
// struct basic para configuraĆ§Ć£o
type TestData struct {
URL string
Port string
}
func main() {
var Config config.Configuration
Config.Filename = "config.json"
if !Config.Read(&Config) {
Config.Data = TestData{"http", "8080"}
Config.Create(&Config)
}
fmt.Println(Config.Data)
TestData1 := &TestData{}
TestData1 = Config.Data.(*TestData) // error, why?
fmt.Println(TestData1.URL)
}
NEW UPDATE:
I have made some changes after JimB comment about I'm not clear about some concepts and I tried to review it. Sure many things aren't clear for me yet unfortunately. The "big" understanding I believe I got, but what mess my mind up is the "ins" and "outs" of values and formats and pointers, mainly when it goes to other libraries. I'm not able yet to follow the "full path" of it.
Yet, I believe I had some improvement on my code.
I think that I have corrected some points, but still have some big questions:
I stopped sending "Configuration" as a parameter as all "data" were already there as they are "thenselfs" in the instance. Right?
Why do I have use reference in the line 58 (Config.Data = &TestData{})
Why to I have to use pointer in the line 64 (tmp := Config.Data.(*TestData)
Why I CANNOT use reference in line 69 (Config.Data = tmp)
Thanks
The reason you are running into an error is because you are trying to decode into an interface{} type. When dealing with JSON objects, they are decoded by the encoding/json package into map[string]interface{} types by default. This is causing the type assertion to fail since the memory structure for a map[string]interface{} is much different than that of a struct.
The better way to do this is to make your TestData struct the expected data format for your Configuration struct:
// Base configuration struct
type Configuration struct {
Filename string
Data *TestData
}
Then when Decoding the file data, the package will unmarshal the data into the fields that match the closest with the data it finds.
If you need more control over the data unmarshaling process, you can dictate which JSON fields get decoded into which struct members by using struct tags. You can read more about the json struct tags available here: https://golang.org/pkg/encoding/json/#Marshal
You are trying to assert that Config.Data is of type *TestData, but you're assigning it to TestData{"http", "8080"} above. You can take the address of a composite literal to create a pointer:
Config.Data = &TestData{"http", "8080"}
If your config already exsits, your Read method is going to fill in the Data field with the a default json data type, probably a map[string]interface{}. If you assign a pointer of the correct type to Data first, it will decode into the expected type.
Config.Data = &TestData{}
Ans since Data is an interface{}, you do not want to ever use a pointer to that value, so don't use the & operator when marshaling and unmarshaling.

How to not marshal an empty struct into JSON with Go?

I have a struct like this:
type Result struct {
Data MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
But even if the instance of MyStruct is entirely empty (meaning, all values are default), it's being serialized as:
"data":{}
I know that the encoding/json docs specify that "empty" fields are:
false, 0, any nil pointer or interface value, and any array,
slice, map, or string of length zero
but with no consideration for a struct with all empty/default values. All of its fields are also tagged with omitempty, but this has no effect.
How can I get the JSON package to not marshal my field that is an empty struct?
As the docs say, "any nil pointer." -- make the struct a pointer. Pointers have obvious "empty" values: nil.
Fix - define the type with a struct pointer field:
type Result struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
Then a value like this:
result := Result{}
Will marshal as:
{}
Explanation: Notice the *MyStruct in our type definition. JSON serialization doesn't care whether it is a pointer or not -- that's a runtime detail. So making struct fields into pointers only has implications for compiling and runtime).
Just note that if you do change the field type from MyStruct to *MyStruct, you will need pointers to struct values to populate it, like so:
Data: &MyStruct{ /* values */ }
As #chakrit mentioned in a comment, you can't get this to work by implementing json.Marshaler on MyStruct, and implementing a custom JSON marshalling function on every struct that uses it can be a lot more work. It really depends on your use case as to whether it's worth the extra work or whether you're prepared to live with empty structs in your JSON, but here's the pattern I use applied to Result:
type Result struct {
Data MyStruct
Status string
Reason string
}
func (r Result) MarshalJSON() ([]byte, error) {
return json.Marshal(struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}{
Data: &r.Data,
Status: r.Status,
Reason: r.Reason,
})
}
func (r *Result) UnmarshalJSON(b []byte) error {
decoded := new(struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
})
err := json.Unmarshal(b, decoded)
if err == nil {
r.Data = decoded.Data
r.Status = decoded.Status
r.Reason = decoded.Reason
}
return err
}
If you have huge structs with many fields this can become tedious, especially changing a struct's implementation later, but short of rewriting the whole json package to suit your needs (not a good idea), this is pretty much the only way I can think of getting this done while still keeping a non-pointer MyStruct in there.
Also, you don't have to use inline structs, you can create named ones. I use LiteIDE with code completion though, so I prefer inline to avoid clutter.
Data is an initialized struct, so it isn't considered empty because encoding/json only looks at the immediate value, not the fields inside the struct.
Unfortunately, returning nil from json.Marshaler doesn't currently work:
func (_ MyStruct) MarshalJSON() ([]byte, error) {
if empty {
return nil, nil // unexpected end of JSON input
}
// ...
}
You could give Result a marshaler as well, but it's not worth the effort.
The only option, as Matt suggests, is to make Data a pointer and set the value to nil.
There is an outstanding Golang proposal for this feature which has been active for over 4 years, so at this point, it is safe to assume that it will not make it into the standard library anytime soon. As #Matt pointed out, the traditional approach is to convert the structs to pointers-to-structs. If this approach is infeasible (or impractical), then an alternative is to use an alternate json encoder which does support omitting zero value structs.
I created a mirror of the Golang json library (clarketm/json) with added support for omitting zero value structs when the omitempty tag is applied. This library detects zeroness in a similar manner to the popular YAML encoder go-yaml by recursively checking the public struct fields.
e.g.
$ go get -u "github.com/clarketm/json"
import (
"fmt"
"github.com/clarketm/json" // drop-in replacement for `encoding/json`
)
type Result struct {
Data MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
j, _ := json.Marshal(&Result{
Status: "204",
Reason: "No Content",
})
fmt.Println(string(j))
// Note: `data` is omitted from the resultant json.
{
"status": "204"
"reason": "No Content"
}