In C when I return a pointer of a stack-created variable from a function, the memory discards after the function is returned, thus making the pointer impossible to dereference. But in Go, the compiler is not giving me any errors. Does that mean that this is safe to do?
package main
import (
"fmt"
)
func main() {
fmt.Println(*(something()))
}
func something() *string {
s := "a"
return &s
}
Yes, this is safe and a normal pattern in Go programming. Go uses escape analysis to move any values with pointers that escape the stack to the heap automatically. You don't need to be concerned with where values are allocated.
From the Go FAQ: "How do I know whether a variable is allocated on the heap or the stack?"
if the compiler cannot prove that the variable is not referenced after the function returns, then the compiler must allocate the variable on the garbage-collected heap to avoid dangling pointer errors
You can see these optimization choices during compilation by using the -gcflags -m option.
Yes, in Golang it is fine to return a pointer to a local variable. Golang will manage the objects lifetime for you and free it when all pointers to it are gone.
In another answer I point out all the differences between C/C++ pointers and Golang pointers: What is the meaning of '*' and '&' in Golang?
Related
Here is a simplified code
func MyHandler(a int) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteCode(a)
})
}
Whenever a http request comes MyHandler will be called, and it will return a function which will be used to handle the request. So whenever a http request comes a new function object will be created. Function is taken as the first class in Go. I'm trying to understand what actually happened when you return a function from memory's perspective. When you return a value, for example an integer, it will occupy 4 bytes in stack. So how about return a function and lots of things inside the function body? Is it an efficient way to do so? What's shortcomings?
If you're not used to closures, they may seem a bit magic. They are, however, easy to implement in compilers:
The compiler finds any variables that must be captured by the closure. It puts them into a working area that will be allocated and remain allocated as long as the closure itself exists.
The compiler then generates the inner function with a secret extra parameter, or some other runtime trickery,1 such that calling the function activates the closure.
Because the returned function accesses its closure variables through the compile-time arrangement, there's nothing special needed. And since Go is a garbage-collected language, there's nothing else needed either: the pointer to the closure keeps the closure data alive until the pointer is gone because the function cannot be called any more, at which point the closure data evaporates (well, at the next GC).
1GCC sometimes uses trampolines to do this for C, where trampolines are executable code generated at runtime. The executable code may set a register or pass an extra parameter or some such. This can be expensive since something treated as data at runtime (generated code) must be turned into executable code at runtime (possibly requiring a system call and potentially requiring that some supervisory code "vet" the resulting runtime code).
Go does not need any of this because the language was defined with closures in mind, so implementors don't, er, "close off" any easy ways to make this all work. Some runtime ABIs are defined with closures in mind as well, e.g., register r1 is reserved as the closure-variables pointer in all pointer-to-function types, or some such.
Actual function size is irrelevant. When you return a function like this, memory will be allocated for the closure, that is, any variables in the scope that the function uses. In this case, a pointer will be returned containing the address of the function and a pointer to the closure, which will contain a reference to the variable a.
I've been playing around with the reflect package, and I notice how limited the functionality of functions are.
package main
import (
"fmt"
"reflect"
"strings"
)
func main() {
v := reflect.ValueOf(strings.ToUpper)
fmt.Printf("Address: %v\n", v) // 0xd54a0
fmt.Printf("Can set? %d\n", v.CanSet()) // False
fmt.Printf("Can address? %d\n", v.CanAddr()) // False
fmt.Printf("Element? %d\n", v.Elem()) // Panics
}
Playground link here.
I've been taught that functions are addresses to memory with a set of instructions (hence v prints out 0xd54a0), but it looks like I can't get an address to this memory address, set it, or dereference it.
So, how are Go functions implemented under the hood? Eventually, I'd ideally want to manipulate the strings.ToUpper function by making the function point to my own code.
Disclaimers:
I've only recently started to delve deeper into the golang compiler, more specifically: the go assembler and mapping thereof. Because I'm by no means an expert, I'm not going to attempt explaining all the details here (as my knowledge is most likely still lacking). I will provide a couple of links at the bottom that might be worth checking out for more details.
What you're trying to do makes very, very little sense to me. If, at runtime, you're trying to modify a function, you're probably doing something wrong earlier on. And that's just in case you want to mess with any function. The fact that you're trying to do something with a function from the strings package makes this all the more worrying. The reflect package allows you to write very generic functions (eg a service with request handlers, but you want to pass arbitrary arguments to those handlers requires you to have a single handler, process the raw request, then call the corresponding handler. You cannot possibly know what that handler looks like, so you use reflection to work out the arguments required...).
Now, how are functions implemented?
The go compiler is a tricky codebase to wrap ones head around, but thankfully the language design, and the implementation thereof has been discussed openly. From what I gather, golang functions are essentially compiled in pretty much the same way as a function in, for example, C. However, calling a function is a bit different. Go functions are first-class objects, that's why you can pass them as arguments, declare a function type, and why the reflect package has to allow you to use reflection on a function argument.
Essentially, functions are not addressed directly. Functions are passed and invoked through a function "pointer". Functions are effectively a type like similar to a map or a slice. They hold a pointer to the actual code, and the call data. In simple terms, think of a function as a type (in pseudo-code):
type SomeFunc struct {
actualFunc *func(...) // pointer to actual function body
data struct {
args []interface{} // arguments
rVal []interface{} // returns
// any other info
}
}
This means that the reflect package can be used to, for example, count the number of arguments and return values the function expects. It also tells you what the return value(s) will be. The overall function "type" will be able to tell you where the function resides, and what arguments it expects and returns, but that's about it. IMO, that's all you really need though.
Because of this implementation, you can create fields or variables with a function type like this:
var callback func(string) string
This would create an underlying value that, based on the pseudo code above, looks something like this:
callback := struct{
actualFunc: nil, // no actual code to point to, function is nil
data: struct{
args: []interface{}{string}, // where string is a value representing the actual string type
rVal: []interface{}{string},
},
}
Then, by assigning any function that matches the args and rVal constraints, you can determine what executable code the callback variable points to:
callback = strings.ToUpper
callback = func(a string) string {
return fmt.Sprintf("a = %s", a)
}
callback = myOwnToUpper
I hope this cleared 1 or 2 things up a bit, but if not, here's a bunch of links that might shed some more light on the matter.
Go functions implementation and design
Introduction to go's ASM
Rob Pike on the go compiler written in go, and the plan 9 derived asm mapping
Writing a JIT in go asm
a "case study" attempting to use golang ASM for optimisation
Go and assembly introduction
Plan 9 assembly docs
Update
Seeing as you're attempting to swap out a function you're using for testing purposes, I would suggest you not use reflection, but instead inject mock functions, which is a more common practice WRT testing to begin with. Not to mention it being so much easier:
type someT struct {
toUpper func(string) string
}
func New(toUpper func(string) string) *someT {
if toUpper == nil {
toUpper = strings.ToUpper
}
return &someT{
toUpper: toUpper,
}
}
func (s *someT) FuncToTest(t string) string {
return s.toUpper(t)
}
This is a basic example of how you could inject a specific function. From within your foo_test.go file, you'd just call New, passing a different function.
In more complex scenario's, using interfaces is the easiest way to go. Simply implement the interface in the test file, and pass the alternative implementation to the New function:
type StringProcessor interface {
ToUpper(string) string
Join([]string, string) string
// all of the functions you need
}
func New(sp StringProcessor) return *someT {
return &someT{
processor: sp,
}
}
From that point on, simply create a mock implementation of that interface, and you can test everything without having to muck about with reflection. This makes your tests easier to maintain and, because reflection is complex, it makes it far less likely for your tests to be faulty.
If your test is faulty, it could cause your actual tests to pass, even though the code you're trying to test isn't working. I'm always suspicious if the test code is more complex than the code you're covering to begin with...
Underneath the covers, a Go function is probably just as you describe it- an address to a set of instructions in memory, and parameters / return values are filled in according to your system's linkage conventions as the function executes.
However, Go's function abstraction is much more limited, on purpose (it's a language design decision). You can't just replace functions, or even override methods from other imported packages, like you might do in a normal object-oriented language. You certainly can't do dynamic replacement of functions under normal circumstances (I suppose you could write into arbitrary memory locations using the unsafe package, but that's willful circumvention of the language rules, and all bets are off at that point).
Are you trying to do some sort of dependency injection for unit testing? If so, the idiomatic way to do this in Go is to define interface that you pass around to your functions/methods, and replace with a test version in your tests. In your case, an interface may wrap the call to strings.ToUpper in the normal implementation, but a test implementation might call something else.
For example:
type Upper interface {
ToUpper(string) string
}
type defaultUpper struct {}
func (d *defaultUpper) ToUpper(s string) string {
return strings.ToUpper(s)
}
...
// normal implementation: pass in &defaultUpper{}
// test implementation: pass in a test version that
// does something else
func SomethingUseful(s string, d Upper) string {
return d.ToUpper(s)
}
Finally, you can also pass function values around. For example:
var fn func(string) string
fn = strings.ToUpper
...
fn("hello")
... but this won't let you replace the system's strings.ToUpper implementation, of course.
Either way, you can only sort of approximate what you want to do in Go via interfaces or function values. It's not like Python, where everything is dynamic and replaceable.
Why C++ hasn't placement delete that directly corresponds to the placement new, i.e. calls the destructor and calls appropriate placement delete operator?
For example:
MyType *p = new(arena) MyType;
...
//current technique
p->~MyType();
operator delete(p, arena);
//proposed technique
delete(arena) p;
operator delete is unique in being a non-member or static member function that is dynamically dispatched. A type with a virtual destructor performs the call to its own delete from the most derived destructor.
struct abc {
virtual ~abc() = 0;
};
struct d : abc {
operator delete() { std::cout << "goodbye\n"; }
};
int main() {
abc *p = new d;
delete p;
}
(Run this example.)
For this to work with placement delete, the destructor would have to somehow pass the additional arguments to operator delete.
Solution 1: Pass the arguments through the virtual function. This requires a separate virtual destructor for every static member and global operator delete overload with different arguments.
Solution 2: Let the virtual destructor return a function pointer to the caller specifying what operator delete should be called. But if the destructor does lookup, this hits the same problem of requiring multiple virtual function definitions as #1. Some kind of abstract overload set would have to be created, which the caller would resolve.
You have a perfectly good point, and it would be a nice addition to the language. Retrofitting it into the existing semantics of delete is probably even possible, in theory. But most of the time we don't use the full functionality of delete and it suffices to use a pseudo-destructor call followed by something like arena.release(p).
Probably because there was syntax for explicitly calling a destructor without deallocation (exactly as in your question), but no syntax for explicit construction in raw memory?
Actually there is a placement delete which is called by the implementation for an object that was "allocated" using placement new if the constructor threw an exception.
From Wikipedia.
The placement delete functions are called from placement new expressions. In particular, they are called if the constructor of the object throws an exception. In such a circumstance, in order to ensure that the program does not incur a memory leak, the placement delete functions are called.
The whole point of placement new is to separate object creation from its memory management. So it makes no sense to tie it back during object destruction.
If memory for your objects is from heap and you want same lifetime for objects and their memory just use operator new and operator delete, maybe overriding them if you want any special behavior.
Placement new is good for example in vector, which keeps a large chunk of raw memory and creates and destroys object inside of it, but without releasing memory.
I am still learning from the DLL process and since no Delphi FTDI Libmpsse library exist, I must work my own way through with the DLL from scratch.
However, I'm far from perfect in Delphi!
I've been using it for few years without touching pointers.
This is the particular line I'm now stuck with (This is what I've tried to translate):
function I2C_DeviceWrite(
handle :dword;
deviceAddress :uint32;
sizeToTransfer :uint32;
out buffer :byte;
out sizeTransferred :uint32;
options :uint32
):FT_Result; cdecl; external 'libmpsse.dll';
From the API User guide it is describe the C way like this:
FT_STATUS I2C_DeviceWrite(
FT_HANDLE handle,
uint32 deviceAddress,
uint32 sizeToTransfer,
uint8 *buffer,
uint32 *sizeTransferred,
uint32 options )
Asterix means pointer in C from what I understand.
I don't know how to declare them properly yet.
Any hints?
NB: I'm still reading (XE2 Foundations) and searching about pointers and buffers the most I can!
Thank you!
BTW, API Guide is here for consulation:PDF from FTDI
EDIT:
var
Buffer: array of Byte;
written,towrite:cardinal;
begin
SetLength(Buffer,I2C_DEVICE_BUFFER_SIZE);
buffer[0] := $07;
towrite := 1;
i2c_openchannel(1,FT_HANDLE); //open channel 1
i2c_initchannel(FT_HANDLE,I2C_CLOCK_STANDARD_MODE and LATENCY_TIMER and 0);
i2c_devicewrite(FT_HANDLE,I2C_DEVICE_ADDRESS,towrite,buffer,written,I2C_TRANSFER_OPTIONS_START_BIT);
i2c_closechannel(FT_HANDLE); //close device
So far so good everything else work except the i2c_devicewrite line!It gives me E2033:unable to compile actual and formal var parameters must be identical
You've translated the byte array parameter incorrectly. You used
out buffer: byte
which is a single byte, passed out by reference. It should be
buffer: PByte
which here means a pointer to array of byte.
Then pass the address of the first element of the array. Like this:
#Buffer[0]
or like this
PByte(Buffer)
Note that I've not checked anything other than the byte array parameter.
FWIW, a constant length array may be simpler here than a dynamic array.
I'd also comment that you appear to be ignoring return values. Never do that. Always check return values for errors.
I'd like to ask for an advice. I am working with small embedded uP.
I'd like to assign my various functions to myfunctions struct. How to do that correctly?
Then I want to place this myfunctions (struct of function pointers) to specific memory address (e.g. 0x1000). Whats is the best procedure to achieve this?
typedef void (*fptr)(void);
typedef struct FCN_IMAGE
{
fptr fcn1;
fptr fcn2;
fptr fcn3;
} FUNC_T;
FUNC_T myfunctions;
Actually it should be sort of jump table.
Secondly I want to read this function pointers from within other program - directly from specified address location (e.g. 0x1000).
It means the first code should assign the struct of function pointers to specific memory location and other independent code should read this table from specific memory. Interconnection between those two programs should be
#define FCN_BASE_ADDRESS (0x1000)
Any ideas what is the best way to achieve it?
P.S. This will run on embedded processor - not PC.
Locating objects at specific locations is usually most easily performed by the use of compiler specific extension; there is no standard method defined by the language. It may also be possible to locate a global object at a specific location by modifying the linker script, but that will be specific to your particular tool-chain
What compiler/tool-chain are you using? Refer to its documentation.
Maybe the following will help you:
// assign my various functions to myfunctions struct
myfunctions.fcn1 = &YourFunction1;
myfunctions.fcn2 = &YourFunction2;
myfunctions.fcn3 = &YourFunction3;
// assign the struct of function pointers to specific memory location
memcpy((void*)FCN_BASE_ADDRESS, &myfunctions, sizeof(myfunctions));
// read this table from specific memory
memcpy(&myfunctions, (void*)FCN_BASE_ADDRESS, sizeof(myfunctions));
This is based on my guess on what you actually want to do.
This is the best way to solve it in a portable manner:
typedef void (*fptr)(void);
#define FCN_BASE_ADDRESS ((volatile fptr*)0x1000)
/* Make myfunctions an array, not a struct.
Structs can have padding and aren't portable.
It doesn't look like you need a struct in this case.
*/
fptr myfunctions [N] =
{
fptr fcn1;
fptr fcn2;
fptr fcn3;
};
memcpy(&FCN_BASE_ADDRESS, myfunctions, sizeof(myfunctions));
Though if you are using Codewarrior, you could probably use a #pragma to allocate them where you want them. Here is an example assuming they are stored in read/write RAM and a 32-bit address bus.
// .prm file
SECTIONS
MEM_FCN_BASE_ADDRESS = READ_WRITE 0x2000 TO 0x200C;
END
PLACEMENT
FCN_BASE_ADDRESS INTO MEM_FCN_BASE_ADDRESS;
END
// .c file
#pragma DATA_SEG FCN_BASE_ADDRESS
fptr myfunctions[N] = ...;
#pragma DATA_SEG DEFAULT
If they should be stored in ROM/flash, for example a vector table, then it must be done differently with READ_ONLY sections, #pragma CONST_SEG and const fptr. (Note that the const keyword in C behaves in irrational ways when combined with typedef:ed pointers. In this case I believe it would give a constant pointer to a non-constant function and thus it should end up in NVM as desired.)