I have to write unit tests for several functions with similar signature and return values (an object and an error), which must pass similar test conditions.
I would like to avoid writing:
func TestFunc1(t *testing.T) {
// tests on return values
}
func TestFunc2(t *testing.T) {
// tests identical for Func1
}
func TestFunc3(t *testing.T) {
// tests identical for Func1
}
...
(See this go playground example for a more complete context)
(yes, go playground doesn't support yet go test, only go run, and issue 6511 is there to request that feature)
How would you use reflection (reflect package) in order to write only one test which would:
call each function in turn?
test their return value?
I have seen:
"How to properly use .Call in reflect package, Golang?", using Value.Call
"Selecting a function from a list of functions in Golang"
But I miss a complete example for calling functions and using the returned values in a test.
Once I understood that everything must use or return the type Value, here is what I came up with.
The trick is to use:
ValueOf in order to get a value of the receiver
Value.MethodByName to find a function of that receiver value
Value.IsNil to test for nil returned value.
Main extract of the test code:
var funcNames = []string{"Func1", "Func2", "Func3"}
func TestFunc(t *testing.T) {
stype := reflect.ValueOf(s)
for _, fname := range funcNames {
fmt.Println(fname)
sfunc := stype.MethodByName(fname)
// no parameter => empty slice of Value
ret := sfunc.Call([]reflect.Value{})
val := ret[0].Int()
// That would panic for a nil returned err
// err := ret[1].Interface().(error)
err := ret[1]
if val < 1 {
t.Error(fname + " should return positive value")
}
if err.IsNil() == false {
t.Error(fname + " shouldn't err")
}
}
}
See a runnable example in go playground.
Note that if you are calling that test function with a non-existent function name, that will panic.
See that example here.
runtime.panic(0x126660, 0x10533140)
/tmp/sandbox/go/src/pkg/runtime/panic.c:266 +0xe0
testing.func·005()
/tmp/sandbox/go/src/pkg/testing/testing.go:383 +0x180
----- stack segment boundary -----
runtime.panic(0x126660, 0x10533140)
/tmp/sandbox/go/src/pkg/runtime/panic.c:248 +0x160
reflect.flag.mustBe(0x0, 0x13)
/tmp/sandbox/go/src/pkg/reflect/value.go:249 +0xc0
reflect.Value.Call(0x0, 0x0, 0x0, 0xfeef9f28, 0x0, ...)
/tmp/sandbox/go/src/pkg/reflect/value.go:351 +0x40
main.TestFunc(0x10546120, 0xe)
/tmpfs/gosandbox-3642d986_9569fcc1_f443bbfb_73e4528d_c874f1af/prog.go:34 +0x240
Go playground recover from that panic, but your test program might not.
That is why I added to the test function above:
for _, fname := range funcNames {
defer func() {
if x := recover(); x != nil {
t.Error("TestFunc paniced for", fname, ": ", x)
}
}()
fmt.Println(fname)
That produces (see example) a much nicer output:
Func1
Func2
Func3
Func4
--- FAIL: TestFunc (0.00 seconds)
prog.go:48: Func2 should return positive value
prog.go:51: Func3 shouldn't err
prog.go:32: TestFunc paniced for Func4 : reflect: call of reflect.Value.Call on zero Value
FAIL
Related
Is there a simple way in Go to check whether given JSON is either an Object {} or array []?
The first thing that comes to mind is to json.Unmarshal() into an interface, and then see if it becomes a map, or a slice of maps. But that seems quite inefficient.
Could I just check if the first byte is a { or a [? Or is there a better way of doing this that already exists.
Use the following to detect if JSON text in the []byte value data is an array or object:
// Get slice of data with optional leading whitespace removed.
// See RFC 7159, Section 2 for the definition of JSON whitespace.
x := bytes.TrimLeft(data, " \t\r\n")
isArray := len(x) > 0 && x[0] == '['
isObject := len(x) > 0 && x[0] == '{'
This snippet of code handles optional leading whitespace and is more efficient than unmarshalling the entire value.
Because the top-level value in JSON can also be a number, string, boolean or nil, it's possible that isArray and isObject both evaluate to false. The values isArray and isObject can also evaluate to false when the JSON is invalid.
Use a type switch to determine the type. This is similar to Xay's answer, but simpler:
var v interface{}
if err := json.Unmarshal(data, &v); err != nil {
// handle error
}
switch v := v.(type) {
case []interface{}:
// it's an array
case map[string]interface{}:
// it's an object
default:
// it's something else
}
Do step-by-step parsing of your JSON, using json.Decoder. This has the advantage over the other answers of:
Being more efficient than decoding the entire value
Using the official JSON parsing rules, and generating standard errors if you get invalid input.
Note, this code isn't tested, but should be enough to give you the idea. It can also be easily expanded to check for numbers, booleans, or strings, if desired.
func jsonType(in io.Reader) (string, error) {
dec := json.NewDecoder(in)
// Get just the first valid JSON token from input
t, err := dec.Token()
if err != nil {
return "", err
}
if d, ok := t.(json.Delim); ok {
// The first token is a delimiter, so this is an array or an object
switch (d) {
case '[':
return "array", nil
case '{':
return "object", nil
default: // ] or }, shouldn't be possible
return "", errors.New("Unexpected delimiter")
}
}
return "", errors.New("Input does not represent a JSON object or array")
}
Note that this consumed the first few bytes of in. It is an exercise for the reader to make a copy, if necessary. If you're trying to read from a byte slice ([]byte), convert it to a reader first:
t, err := jsonType(bytes.NewReader(myValue))
Go playground
I've tried to create a function in Go that used to retry any fail query functions (usually because serialization issue).
func retryer(functionA func(interface{}) (interface{}, []error), maxRetry int, waitBetween time.Duration) interface{} {
//when no error in functionA, retryer returns whatever functionA returns
//when maxRetry is reached, returns nil
}
The functions I want to retry are looked like this
func GetTopStudent(classId string) ([]Student, []error) {
//queries top 10 students for class with classId
}
func GetAverageStudentScores(classId string, from time.Time, until time.Time) ([]Pair, []error) {
//queries all average score using aggregate, grouping by studentId
//Pair behaves like C++ pair<string,string>
}
But, the results is a compile error
cannot use GetTopStudent (type func(string) ([]Student, []error)) as type func(interface{}) (interface {}, []error) in argument to retryer
I've tried to modify it a little and I got another compile error
cannot use GetTopStudent (type func(string) ([]Student, []error)) as type func(string) (interface {}, []error) in argument to retryer
Can anyone help me creating a general function to wrap a function to retry on error?
A better way to solve your problem would be to use closures.
For example, change the type of retryer:
func retryer(f func() error, maxRetry int, waitBetween time.Duration) error {
// retry and wait logic
err := f()
// error handling, retry, and wait logic
return err
}
Now call functions to be retried as:
// ...
classId := "some value"
// ...
var st []Student
var errors []error
err := retryer(func() error {
st, errors = GetTopStudent(classId)
// handle errors
return nil
}, numTries, waitTime)
// use st here
I have a struct that looks like
type Request struct {
Name string `json:"name"`
Parameters []Parameter `json:"parameters"`
}
and
type Parameter struct {
Attached bool `json:"attached"`
Script string `json:"script"`
}
Now, I have unmarshalled the json into the struct, and the Script variable has an http location "http://localhost/helloworld.sh". What I am trying to do is, to change the struct variable Parameter.Script from http://localhost/helloworld.sh with the actual content of the script, which is a plain ascii shell script. I wrote a method for the inner struct like
func (p *Parameter) SetScript(script string) {
p.Script = script
}
using pointer to Parameter,
and in the GetScript function, try to call that method after getting the response body.
func GetScript(params *Request) {
for _, i := range params.Parameters {
switch i.Attached {
case false:
client := new(http.Client)
req, _ := http.NewRequest("GET", i.Script, nil)
resp, _ := client.Do(req)
defer resp.Body.Close()
reader, _ := ioutil.ReadAll(resp.Body)
i.SetScript(string(reader))
}
}
}
However, when I print the struct after calling this function, it has not modified the variables, and prints the http://localhost/helloworld.sh only.
I am able to get the response body, which is the actual content of the script, but I am not able to replace the struct variable from within the GetScript function.
Could someone please point out the right way to do this?
Thank you.
The problem is that you are using a for _, i := range loop, and you modify the loop variable inside the loop:
for _, i := range params.Parameters {
switch i.Attached {
case false:
// ...
i.SetScript(string(reader))
}
}
The loop variable i is a copy of the slice elements you range over. So if you do any modification to it, you will only modify the copy and not the element in the slice. (Note that it's true that the SetScript() method has a pointer receiver, but it will receive the address of the copy, so it can and will modify only the copy.)
One workaround is use an index-only range, and refer to the slice element using indexing (replace all occurrence of i with params.Parameters[i]):
for i := range params.Parameters {
switch params.Parameters[i].Attached {
case false:
// ...
params.Parameters[i].SetScript(string(reader))
}
}
You can simplify the above code by assigning the slice to a local variable (this will copy only the slice header not its elements, and it will refer to the same underlying array), and use if statement instead of that ugly switch:
p := params.Parameters
for i := range p {
if !p[i].Attached {
// ...
p[i].SetScript(string(reader))
}
}
An alternative simplification / improvement is to take the address of the index expression, and use that (so you can omit repeating it multiple times):
for i := range params.Parameters {
p := ¶ms.Parameters[i]
if !p.Attached {
// ...
p.SetScript(string(reader))
}
}
I want to Marshal / Unmarshal Golang object (json) with a custom tag.
Like
type Foo struct {
Bar string `json:"test" es:"bar"`
}
data, _ := json.MarshalWithESTag(Foo{"Bar"})
log.Println(string(data)) // -> {"foo":"bar"}
In other words, I whan to use the encoding/json library with a different tag here: https://github.com/golang/go/blob/master/src/encoding/json/encode.go#L1033
Thanks :)
I think the way you wrote your example might have been a bit incorrect?
When I run your code using Marshal() inplace of MarshalWithESTag() I get {"test":"Bar"} not {"foo":"test"} as I think your example would imply.
Here is that code running in the Go Playground to illustrate the output:
package main
import (
"encoding/json"
"fmt"
)
type Foo struct {
Bar string `json:"test" es:"bar"`
}
func main() {
data, _ := json.Marshal(Foo{"Bar"})
fmt.Println(string(data))
}
Assuming I am correct about what you wanted then that would imply what you really wanted was for your output to be {"bar":"Bar"} when you call json.MarshalWithESTag().
Based on that assumption you could accomplish with the following code — which you can see in the Go Playground — after which I will explain the code. (If my assumption was not correct I will address that too):
You cannot add a MarshalWithESTag() method to the the json package because Go does not allow for safe monkey patching. However, you can add a MarshalWithESTag() method to your Foo struct, and this example also shows you how to call it:
func (f Foo) MarshalWithESTag() ([]byte, error) {
data, err := json.Marshal(f)
return data,err
}
func main() {
f := &Foo{"Bar"}
data, _ := f.MarshalWithESTag()
log.Println(string(data)) // -> {"bar":"Bar"}
}
Next you need to add a MarshalJSON() method to your Foo struct. This will get called when you call json.Marshal() and pass an instance of Foo to it.The following is a simple example that hard-codes a return value of {"hello":"goodbye"} so you can see in the playground how adding a MarshalJSON() to Foo affects json.Marshal(Foo{"Bar"}):
func (f Foo) MarshalJSON() ([]byte, error) {
return []byte(`{"hello":"goodbye"}`),nil
}
The output for this will be:
{"hello":"goodbye"}
Inside the MarshalJSON() method we need to produce JSON with the es tags instead of the json tags meaning we will need to generate JSON within the method because Go does not provide us with the JSON; it expects us to generate it. And the easiest way to generate JSON in Go is to use json.Marshal(). However, if we use json.Marshal(f) where f is an instance of Foo that gets passed as the receiver when calling MarshalJson() it will end up in an infinite recursive loop!The solution is to create a new struct type based on and identical to the existing type of Foo, except for its identity. Creating a new type esFoo based on Foo is as easy as:
type esFoo Foo
Since we have esFoo we can now cast our instance of Foo to be of type esFoo to break the association with our custom MarshalJSON(). This works because our method was specific to the type with the identity of Foo and not with the type esFoo. Passing an instance of esFoo to json.Marshal() allows us to use the default JSON marshalling we get from Go.
To illustrate, here you can see an example that uses esFoo and sets its Bar property to "baz" giving us output of {"test":"baz"} (you can also see it run in the Go playground):
type esFoo Foo
func (f Foo) MarshalJSON() ([]byte, error) {
es := esFoo(f)
es.Bar = "baz"
_json,err := json.Marshal(es)
return _json,err
}
The output for this will be:
{"test":"baz"}
Next we process and manipulate the JSON inside MarshalJSON(). This can be done by using json.Unmarshal() to an interface{} variable which we can then use a type assertion to treat the variable as a map.Here is a standalone example unrelated to the prior examples that illustrates this by printing map[maker:Chevrolet model:Corvette year:2021] (Again you can see it work in the Go Playground):
package main
import (
"encoding/json"
"fmt"
)
type Car struct {
Maker string `json:"maker" es:"fabricante"`
Model string `json:"model" es:"modelo"`
Year int `json:"year" es:"año"`
}
var car = Car{
Maker:"Chevrolet",
Model:"Corvette",
Year:2021,
}
func main() {
_json,_ := json.Marshal(car)
var intf interface{}
_ = json.Unmarshal(_json, &intf)
m := intf.(map[string]interface{})
fmt.Printf("%v",m)
}
The output for this will be:
map[maker:Chevrolet model:Corvette year:2021]
Our next challenge is to access the tags. Tags are accessible using Reflection. Go provides reflection functionality in the standard reflect package.
Using our Car struct from above, here is a simple example that illustrates how to use Reflection. It uses the reflect.TypeOf() function to retrieve the type as a value and then introspects that type to retrieve the tags for each field. The code for retrieving each tag is t.Field(i).Tag.Lookup("es"), which is hopefully somewhat self-explanatory (and again, check it out in the Go Playground):
func main() {
t := reflect.TypeOf(car)
for i:=0; i<t.NumField();i++{
tag, _ := t.Field(i).Tag.Lookup("es")
fmt.Printf("%s\n",tag)
}
}
The output for this will be:
fabricante
modelo
año
Now that we have covered all the building blocks we can bring it all together into a working solution. The only addition worth mentioning are the creation of a new map variable _m of the same length as m to allow us to store the values using the es tags:
func (f Foo) MarshalJSON() ([]byte, error) {
es := esFoo(f)
_json,err := json.Marshal(es)
{
if err != nil {
goto end
}
var intf interface{}
err = json.Unmarshal(_json, &intf)
if err != nil {
goto end
}
m := intf.(map[string]interface{})
_m := make(map[string]interface{},len(m))
t := reflect.TypeOf(f)
i := 0
for _,v := range m {
tag, found := t.Field(i).Tag.Lookup("es")
if !found {
continue
}
_m[tag] = v
i++
}
_json,err = json.Marshal(_m)
}
end:
return _json,err
}
However, there is still one detail left undone. With all the above code f.MarshalWithESTag() will generate JSON for the es tags, but so will json.Marshal(f) and we want the latter to return its use of the json tags.
So address that we just need to:
a. Add a local package variable useESTags with an initial value of false,
b. Modify f.MarshalWithESTag() to set useESTags to true before calling json.Marshal(), and then
c. To set useESTags back to false before returning, and
d. Lastly modify MarshalJSON() to only perform the logic required for the es tags if useESTags is set to true:
Which brings us to the final code — with a second property in Foo to provide a better example (and finally, you can of course see here in the Go Playground):
package main
import (
"encoding/json"
"log"
"reflect"
)
type Foo struct {
Foo string `json:"test" es:"bar"`
Bar string `json:"live" es:"baz"`
}
type esFoo Foo
var useESTags = false
func (f Foo) MarshalWithESTag() ([]byte, error) {
useESTags = true
data, err := json.Marshal(f)
useESTags = false
return data,err
}
func (f Foo) MarshalJSON() ([]byte, error) {
es := esFoo(f)
_json,err := json.Marshal(es)
if useESTags {
if err != nil {
goto end
}
var intf interface{}
err = json.Unmarshal(_json, &intf)
if err != nil {
goto end
}
m := intf.(map[string]interface{})
_m := make(map[string]interface{},len(m))
t := reflect.TypeOf(f)
i := 0
for _,v := range m {
tag, found := t.Field(i).Tag.Lookup("es")
if !found {
continue
}
_m[tag] = v
i++
}
_json,err = json.Marshal(_m)
}
end:
return _json,err
}
func main() {
f := &Foo{"Hello","World"}
data, _ := json.Marshal(f)
log.Println(string(data)) // -> {"test":"Hello","live":"World"}
data, _ = f.MarshalWithESTag()
log.Println(string(data)) // -> {"bar":"Hello","baz":"World"}
}
Epilogue
If my assumption was wrong I think I can at least assume this code I provided gives you enough to achieve your objective. You should be able to swap the keys and values in your output if that is actually what you want given the techniques shown. If not, please comment asking for help.
Finally, I would be remiss not to mention that reflection can be slow and that this example uses reflection multiple times per object to achieve your desired output. For many use-cases the time required to process JSON this way won't be significant. However, for many other use-cases the execution time can be a deal-killer. Several commented that you should approach this a different way; if performance matters and/or using a more idiomatic Go approach is important, you might want to seriously consider their recommendations.
With the following code, if no file argument is given, a panic is thrown for line 9 panic: runtime error: index out of range as expected.
How can I 'catch' this panic and handle it when directly when passing something to it (os.Args[1]) that causes the panic? Much like try/catch in PHP or try/except in Python.
I've had a search here on StackOverflow but I've not found anything that answers this as such.
package main
import (
"fmt"
"os"
)
func main() {
file, err := os.Open(os.Args[1])
if err != nil {
fmt.Println("Could not open file")
}
fmt.Printf("%s", file)
}
A panicking program can recover with the builtin recover() function:
The recover function allows a program to manage behavior of a panicking goroutine. Suppose a function G defers a function D that calls recover and a panic occurs in a function on the same goroutine in which G is executing. When the running of deferred functions reaches D, the return value of D's call to recover will be the value passed to the call of panic. If D returns normally, without starting a new panic, the panicking sequence stops. In that case, the state of functions called between G and the call to panic is discarded, and normal execution resumes. Any functions deferred by G before D are then run and G's execution terminates by returning to its caller.
The return value of recover is nil if any of the following conditions holds:
panic's argument was nil;
the goroutine is not panicking;
recover was not called directly by a deferred function.
Here is an example of how to use this:
// access buf[i] and return an error if that fails.
func PanicExample(buf []int, i int) (x int, err error) {
defer func() {
// recover from panic if one occured. Set err to nil otherwise.
if (recover() != nil) {
err = errors.New("array index out of bounds")
}
}()
x = buf[i]
}
Notice that more often than not, panicking is not the right solution. The Go paradigm is to check for errors explicitly. A program should only panic if the circumstances under which it panics do not happen during ordinary program executing. For instance, not being able to open a file is something that can happen and should not cause a panic while running out of memory is worth a panic. Nevertheless, this mechanism exists to be able to catch even these cases and perhaps shut down gracefully.
Go is not python, you should properly check for args before you use it:
func main() {
if len(os.Args) != 2 {
fmt.Printf("usage: %s [filename]\n", os.Args[0])
os.Exit(1)
}
file, err := os.Open(os.Args[1])
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s", file)
}
Some Golang official packages use panic/defer+recover as throw/catch, but only when they need to unwind a large call stack. In Golang's json package using panic/defer+recover as throw/catch is the most elegant solution.
from http://blog.golang.org/defer-panic-and-recover
For a real-world example of panic and recover, see the json package from the Go standard library. It decodes JSON-encoded data with a set of recursive functions. When malformed JSON is encountered, the parser calls panic to unwind the stack to the top-level function call, which recovers from the panic and returns an appropriate error value (see the 'error' and 'unmarshal' methods of the decodeState type in decode.go).
Search for d.error(
at http://golang.org/src/encoding/json/decode.go
In your example the "idiomatic" solution is to check the parameters before using them, as other solutions have pointed.
But, if you want/need to catch anything you can do:
package main
import (
"fmt"
"os"
)
func main() {
defer func() { //catch or finally
if err := recover(); err != nil { //catch
fmt.Fprintf(os.Stderr, "Exception: %v\n", err)
os.Exit(1)
}
}()
file, err := os.Open(os.Args[1])
if err != nil {
fmt.Println("Could not open file")
}
fmt.Printf("%s", file)
}
First: You wouldn't want to do this. Try-catch-style error handling is no error handling. In Go you would check len(os.Args) first and access element 1 only if present.
For the rare cases you need to catch panics (and your case is not one of them!) use defer in combination with recover. See http://golang.org/doc/effective_go.html#recover
We can manage panic without halting process using recover. By calling recover in any function using defer it will return the execution to calling function. Recover returns two values one is boolean and other one is interface to recover. Using type assertion we can get underlying error value
You can also print underlying error using recover.
defer func() {
if r := recover(); r != nil {
var ok bool
err, ok = r.(error)
if !ok {
err = fmt.Errorf("pkg: %v", r)
}
}
}()
I had to catch panics in a test case. I got redirected here.
func.go
var errUnexpectedClose = errors.New("Unexpected Close")
func closeTransaction(a bool) {
if a == true {
panic(errUnexpectedClose)
}
}
func_test.go
func TestExpectedPanic() {
got := panicValue(func() { closeTransaction(true) })
a, ok := got.(error)
if a != errUnexpectedClose || !ok {
t.Error("Expected ", errUnexpectedClose.Error())
}
}
func panicValue(fn func()) (recovered interface{}) {
defer func() {
recovered = recover()
}()
fn()
return
}
Used from https://github.com/golang/go/commit/e4f1d9cf2e948eb0f0bb91d7c253ab61dfff3a59 (ref from VonC)
Note that the recover treatment of a panic Execution error (such as attempting to index an array out of bounds trigger) might change with go 1.7 after issue 14965
See CL 21214 and its test:
runtime: make execution error panic values implement the Error interface
Make execution panics implement Error as mandated by Run-time panics (specs), instead of panics with strings.
When you recover a panic error, you would be able to do:
if _, ok := recovered.(runtime.Error); !ok {
This is still being evaluated, and as Dave Cheney. mentions:
I don't know what people are currently doing but from my POV this has been broken for a long time and nobody has complained so they are either explicitly relying on the broken behaviour, or nobody cares. Either way I think it's a good idea to avoid making this change.