How to debug/dump Go variable while building with cgo? - mysql

I'm trying to write a MySQL UDF in Go with cgo, in which I have a basic one functioning, but there's little bits and pieces that I can't figure out because I have no idea what some of the C variables are in terms of Go.
This is an example that I have written in C that forces the type of one of the MySQL parameters to an int
my_bool unhex_sha3_init(UDF_INIT *initid, UDF_ARGS *args, char *message) {
if (args->arg_count != 2) {
strcpy(message, "`unhex_sha3`() requires 2 parameters: the message part, and the bits");
return 1;
}
args->arg_type[1] = INT_RESULT;
initid->maybe_null = 1; //can return null
return 0;
}
And that works fine, but then I try to do the same/similar thing with this other function in Go like this
//export get_url_param_init
func get_url_param_init(initid *C.UDF_INIT, args *C.UDF_ARGS, message *C.char) C.my_bool {
if args.arg_count != 2 {
message = C.CString("`get_url_param` require 2 parameters: the URL string and the param name")
return 1
}
(*args.arg_type)[0] = C.STRING_RESULT
(*args.arg_type)[1] = C.STRING_RESULT
initid.maybe_null = 1
return 0
}
With this build error
./main.go:24: invalid operation: (*args.arg_type)[0] (type uint32 does
not support indexing)
And I'm not totally sure what that means. Shouldn't this be a slice of some sort, not a uint32?
And this is where it'd be super helpful have some way of dumping the args struct somewhere somehow (maybe even in Go syntax as a super plus) so that I can tell what I'm working with.
Well I used spew to dump the variable contents to a tmp file inside the init function (commenting out the lines that made it not compile) and I got this
(string) (len=3) "%#v"
(*main._Ctype_struct_st_udf_args)(0x7ff318006af8)({
arg_count: (main._Ctype_uint) 2,
_: ([4]uint8) (len=4 cap=4) {
00000000 00 00 00 00 |....|
},
arg_type: (*uint32)(0x7ff318006d18)(0),
args: (**main._Ctype_char)(0x7ff318006d20->0x7ff3180251b0)(0),
lengths: (*main._Ctype_ulong)(0x7ff318006d30)(0),
maybe_null: (*main._Ctype_char)(0x7ff318006d40)(0),
attributes: (**main._Ctype_char)(0x7ff318006d58->0x7ff318006b88)(39),
attribute_lengths: (*main._Ctype_ulong)(0x7ff318006d68)(2),
extension: (unsafe.Pointer) <nil>
})

Alright so huge help with #JimB who stuck with me even though I'm clearly less adept with Go (and especially CGO) but I've got a working version of my UDF, which is an easy and straight forward (and fast) function that pulls a single parameter out of a URL string and decodes it correctly and what not (e.g. %20 gets returned as a space, basically how you would expect it to work).
This seemed incredibly tricky with a pure C UDF because I don't really know C (as well as I know other languages), and there's a lot that can go wrong with URL parsing and URL parameter decoding, and native MySQL functions are slow (and there's not really a good, clean way to do the decoding either), so Go seemed like the better-than-perfect candidate for this kind of problem, for strong performance, ease of writing, and wide variety of easy to use built ins & third party libraries.
The full UDF and it's installation/usage instructions are here https://github.com/StirlingMarketingGroup/mysql-get-url-param/blob/master/main.go
First problem was debugging output. And I did that by Fprintfing to a tmp file instead of the standard output, so that I could check the file to see variable dumps.
t, err := ioutil.TempFile(os.TempDir(), "get-url-param")
fmt.Fprintf(t, "%#v\n", args.arg_type)
And then after I got my output (I was expecting args.arg_type to be an array like it is in C, but instead was a number) I needed to convert the data referenced by that number (the pointer to the start of the C array) to a Go array so I could set it's values.
argsTypes := *(*[2]uint32)(unsafe.Pointer(args.arg_type))
argsTypes[0] = C.STRING_RESULT
argsTypes[1] = C.STRING_RESULT

Related

While Iterator in groovy

I'm trying to create a loop to read, for example, 4200 users from 1000 to 1000 but I can't get it to cut when it reaches the end. I tried it with if, for and I couldn't do it.
I have programmed in JAVA but with Groovy I see that the structure is different.
urlUsers = urlUsers.concat("/1/1000");
List<UserConnectorObject> usersList = null;
while({
gesdenResponse = GesdenUtils.sendHttpRequest(urlUsers, "LOOKUP", null,
request.getMetaData()?.getLogin(), request.getMetaData()?.getPassword());
log.info("Users data in JSON: "+gesdenResponse.getOutput())
usersList = GesdenUtils.fromJSON(gesdenResponse.getOutput(), GesdenConstants.USER_IDENTITY_KEY);
usersList.size() == 10;
log.info("List size in JSON "+usersList.size());
}()) continue
Groovy has lots of loop structures, but it is crucial to separate the regular ones (lang built-ins) and the api functions which take closure as an argument
take closure - no plain way to escape
If you want to iterate from A to B users, you can use, for instance,
(10..20).each { userNo -> // Here you will have all 10 iterations
if ( userNo == 5) {
return
}
}
If something outrageous happens in the loop body and you cannot use return to escape, as loop boddy is a closure (separate function) and this resurn just exits this closure. Next iteration will happen just after.
use regular lang built-in loop structures - make use of break/continue
for (int userNo in 1..10) { // Here you will have only 5 iterations
if (userNo == 5) {
break
}
}
It looks like your closure always return falsy because there is no explicit return, and the last statement evaluated is the call to log.info(String) which returns void.
Use an explicit return or move/delete the log statement.

Solidity and Web3 sha3() methods return something else

In my contract, I have a function that returns the sha3 hash of a certain set of values. While running some tests I found that the value returned from this function differs from the hash value generated by web3.utils.sha3() (with identical arguments).
Here is the code:
Solidity
function hashInfo() public onlyOwner view returns (bytes32) {
bytes32 hash = sha3(
'0x969A70A4fa9F69D2D655E4B743abb9cA297E5328',
'0x496AAFA2960f3Ff530716B5334c9aFf4612e3c27',
'jdiojd',
'oidjoidj',
'idjodj',
12345
)
return hash;
}
JS (web3)
async function testHash(instance){
const contractHash = await instance.methods.hashInfo().call({from: '0x969A70A4fa9F69D2D655E4B743abb9cA297E5328'});
const localHash = web3.utils.sha3(
'0x969A70A4fa9F69D2D655E4B743abb9cA297E5328',
'0x496AAFA2960f3Ff530716B5334c9aFf4612e3c27',
'jdiojd',
'oidjoidj',
'idjodj',
12345
)
console.log(contractHash);
console.log(localHash);
console.log('local == contract: ' + (contractHash == localHash));
}
The resulting console output is:
0xe65757c5a99964b72d217493c192c073b9a580ec4b477f40a6c1f4bc537be076
0x3c23cebfe35b4da6f6592d38876bdb93f548085baf9000d538a1beb31558fc6d
local == contract: false
Any ideas? Does this have something to do with passing multiple arguments to the functions? I have also tried to convert everything to a string and concatenate them into one single string, but also without success.
Thanks in advance!
UPDATE
I found out there also if a web3 method called web3.utils.soliditySha3(). This too did not work and gave the following result:
0xe65757c5a99964b72d217493c192c073b9a580ec4b477f40a6c1f4bc537be076
0x0cf65f7c81dab0a5d414539b0e2f3807526fd9c15e197eaa6c7706d27aa7a0f8
local == contract: false
I'm happy I came after your update as I was just gonna suggest solditySHA3. Now that you've got the right function your problem is most likely with Soldity packing it's parameters.
As you can see here, sha3 is an alias to keccak256 which tightly packs it's arguments. Following the link on that page takes you here which fully explains how it's handled. Basically just take the inputs to soliditySHA3 and pack the bits as if they were the sizes of the variables you used. So if you hashed two uint32s (32 bits each, 64 total) you need to take the 2 64 bit Javascript numbers and compress them into 1 Javascript number.
For cases where more than 64 bits are needed I believe you can pass sequential ints (sets of 64 bits) to soliditySHA3 or you could use a BigInt. Personally, I usually try to only hash 256 bit variables together to avoid having to manually pack my bits on the JS end, but we all know that space constraints are huge in Solidity. I hope I helped, and let me know if you have further questions.

Calling std::vector constructor when containing class manually allocated

I'm afraid to ask questions in case it turns out to be stupid... But I tried to search and don't see the same situation.
I'm retrofitting a std::vector into some existing legacy code that is mostly C style. Our next major release which isn't due for a year or two will jettison a lot of the legacy code. But for now, the way we work is, every project gets recompiled for the customer, depending on specs. Some projects are on Visual Studio 2008, some 2010, etc. My added std::vector code I'm working on has no visible problems when compiled with 2013, but, I get crashes within the STL code when running VS 2008 SP1.
The existing code has a struct, and a fixed size array in it:
#define MAX_REMOTE_CONN 75
typedef struct {
int rno;
int adrs;
bool integ_pending;
} RTUref;
typedef struct {
char device[64];
int port;
RTUref RTU[MAX_REMOTE_CONN];
// more stuff...
} Connect_Info;
So, my basic goal is to get rid of the hard coded size limit to the RTU array. So, I have revised it like this:
class{
public:
int rno;
int adrs;
bool integ_pending;
} RTUref;
typedef std::vector <RTUref> RTUlist;
typedef struct {
char device[64];
int port;
RTUlist RTU;
// more stuff...
} Connect_Info;
The Connect_Info structs are allocated using our own memory manager. Don't know much about it other than it is supposed to be more efficient than use malloc() and free(). I'm guessing that the constructor for RTU doesn't get called since the struct it is contained in data allocated by our own memory manager?
Nevertheless, the code where I size the array, put values into the array all at least seem to work okay. But, when I call .clear() I get a crash from within the STL. And as I said, only if I use 2008. If I use 2013, I don't get that crash.
Assuming pct is a pointer to an allocated Connect_Info structure, the the line:
pct->RTU.clear();
Generates a crash on VS 2008. I am able to resize and add elements to the array. And I even tried to add a check that I don't clear unless the size is greater than zero like so:
if (pct->RTU.size() > 0)
pct->RTU.clear();
And I still get the crash on the clear.
So, I made the educated guess that I need to call a constructor. But, I wasn't quite sure of how to do it. But, in the code where the Connect_Info struct is allocated, I tried to add contructor code like this:
pct->RTU = RTUlist();
It compiles. But, I then get a crash in the STL on that line.
I haven't yet tried to build a small contained test program, as I'm not even sure that I will be able to reproduce the problem without our memory manager. But, I will try if that is what I need to do. I thought maybe someone might see obviously what I'm doing wrong. I'm fairly novice to the STL.
A little background: there is a term in C++ called "POD Type" (or "Plain Old Data Type").
There are verbose rules, but basically things that may do special things on allocations, deallocations, or copies are not POD types. The original Connect_Info was a POD type since it didn't do special things at those times and didn't have any non-POD members.
However, since you added a std::vector (which is not a POD type because it has to do special things at allocation, deallocation, copy, etc (because it allocates memory)), Connect_Info is not a POD type.
POD types can be allocated safely with malloc and deallocated with free since they don't do special things. However, non-POD types cannot (except in exceedingly rare cases which you'll first see after several years of programming C++) be allocated like that.
C only has POD types, so malloc is perfectly acceptable. There are a few options you can do:
int main ( ... )
{
Connect_Info * info = new Connect_Info() ;
std::cout << info->port << std::endl ;
delete info ;
}
Or
Connect_Info * makeOne ()
{
void * ptr = malloc ( sizeof(Connect_Info) ) ;
if ( ! ptr ) return 0 ;
return new (ptr) Connect_Info () ; // "In-Place constructor"
}
void deleteOne ( Connect_Info * info )
{
if ( ! ptr ) return ;
info = info->~Connect_Info() ; // manually call its destructor with the weirdest syntax ever
// Note: I'm not 100% sure this call to 'free' is right because the in-place new can return a different pointer, but I don't know how to the get the original back
free ( static_cast<void*>(info) ) ;
}
int main ( ... )
{
Connect_Info * info = makeOne ()
std::cout << info->port << std::endl ;
deleteOne ( info ) ;
}
If you have boost available (or C++11, which you probably don't), this is a MUCH better option (and only uses header components of boost):
boost::shared_ptr<Connect_Info> makeOne ()
{
return boost::make_shared<Connect_Info> () ;
}
int main ( ... )
{
boost::shared_ptr<Connect_Info> info = makeOne ()
std::cout << info->port << std::endl ;
// nothing else: shared_ptr takes care of that for you
}
(If you have C++11, use std::shared_ptr and std::make_shared)

golang return multiple values issue

I was wondering why this is valid go code:
func FindUserInfo(id string) (Info, bool) {
it, present := all[id]
return it, present
}
but this isn't
func FindUserInfo(id string) (Info, bool) {
return all[id]
}
is there a way to avoid the temporary variables?
To elaborate on my comment, the Effective Go mentions that the multi-value assignment from accessing a map key is called the "comma ok" pattern.
Sometimes you need to distinguish a missing entry from a zero value. Is there an entry for "UTC" or is that the empty string because it's not in the map at all? You can discriminate with a form of multiple assignment.
var seconds int
var ok bool
seconds, ok = timeZone[tz]
For obvious reasons this is called the “comma ok” idiom. In this example, if tz is present, seconds will be set appropriately and ok will be true; if not, seconds will be set to zero and ok will be false.
Playground demonstrating this
We can see that this differs from calling a regular function where the compiler would tell you that something is wrong:
package main
import "fmt"
func multiValueReturn() (int, int) {
return 0, 0
}
func main() {
fmt.Println(multiValueReturn)
asgn1, _ := multiValueReturn()
asgn2 := multiValueReturn()
}
On the playground this will output
# command-line-arguments
/tmp/sandbox592492597/main.go:14: multiple-value multiValueReturn() in single-value context
This gives us a hint that it may be something the compiler is doing. Searching the source code for "commaOk" gives us a few places to look, including types.unpack
At the time of writing this it this the method's godoc reads:
// unpack takes a getter get and a number of operands n. If n == 1, unpack
// calls the incoming getter for the first operand. If that operand is
// invalid, unpack returns (nil, 0, false). Otherwise, if that operand is a
// function call, or a comma-ok expression and allowCommaOk is set, the result
// is a new getter and operand count providing access to the function results,
// or comma-ok values, respectively. The third result value reports if it
// is indeed the comma-ok case. In all other cases, the incoming getter and
// operand count are returned unchanged, and the third result value is false.
//
// In other words, if there's exactly one operand that - after type-checking
// by calling get - stands for multiple operands, the resulting getter provides
// access to those operands instead.
//
// If the returned getter is called at most once for a given operand index i
// (including i == 0), that operand is guaranteed to cause only one call of
// the incoming getter with that i.
//
The key bits of this being that this method appears to determine whether or not something is actually a "comma ok" case.
Digging into that method tells us that it will check to see if the mode of the operands is indexing a map or if the mode is set to commaok (where this is defined does give us many hints on when it's used, but searching the source for assignments to commaok we can see it's used when getting a value from a channel and type assertions). Remember the bolded bit for later!
if x0.mode == mapindex || x0.mode == commaok {
// comma-ok value
if allowCommaOk {
a := [2]Type{x0.typ, Typ[UntypedBool]}
return func(x *operand, i int) {
x.mode = value
x.expr = x0.expr
x.typ = a[i]
}, 2, true
}
x0.mode = value
}
allowCommaOk is a parameter to the function. Checking out where unpack is called in that file we can see that all callers pass false as an argument. Searching the rest of the repository leads us to assignments.go in the Checker.initVars() method.
l := len(lhs)
get, r, commaOk := unpack(func(x *operand, i int) { check.expr(x, rhs[i]) }, len(rhs), l == 2 && !returnPos.IsValid())
Since it seems that we can only use the "comma ok" pattern to get two return values when doing a multi-value assignment this seems like the right place to look! In the above code the length of the left hand side is checked, and when unpack is called the allowCommaOk parameter is the result of l == 2 && !returnPos.IsValid(). The !returnPos.IsValid() is somewhat confusing here as that would mean that the position has no file or line information associated with it, but we'll just ignore that.
Further down in that method we've got:
var x operand
if commaOk {
var a [2]Type
for i := range a {
get(&x, i)
a[i] = check.initVar(lhs[i], &x, returnPos.IsValid())
}
check.recordCommaOkTypes(rhs[0], a)
return
}
So what does all of this tell us?
Since the unpack method takes an allowCommaOk parameter that's hardcoded to false everywhere except in assignment.go's Checker.initVars() method, we can probably assume that you will only ever get two values when doing an assignment and have two variables on the left-hand side.
The unpack method will determine whether or not you actually do get an ok value in return by checking if you are indexing a slice, grabbing a value from a channel, or doing a type assertion
Since you can only get the ok value when doing an assignment it looks like in your specific case you will always need to use variables
You may save a couple of key strokes by using named returns:
func FindUserInfo(id string) (i Info, ok bool) {
i, ok = all[id]
return
}
But apart from that, I don't think what you want is possible.
Simply put: the reason why your second example isn't valid Go code is because the language specification says so. ;)
Indexing a map only yields a secondary value in an assignment to two variables. Return statement is not an assignment.
An index expression on a map a of type map[K]V used in an assignment or initialization of the special form
v, ok = a[x]
v, ok := a[x]
var v, ok = a[x]
yields an additional untyped boolean value. The value of ok is true if the key x is present in the map, and false otherwise.
Furthermore, indexing a map is not a "single call to a multi-valued function", which is one of the three ways to return values from a function (the second one, the other two not being relevant here):
There are three ways to return values from a function with a result type:
The return value or values may be explicitly listed in the "return" statement. Each expression must be single-valued and assignable to the corresponding element of the function's result type.
The expression list in the "return" statement may be a single call to a multi-valued function. The effect is as if each value returned from that function were assigned to a temporary variable with the type of the respective value, followed by a "return" statement listing these variables, at which point the rules of the previous case apply.
The expression list may be empty if the function's result type specifies names for its result parameters. The result parameters act as ordinary local variables and the function may assign values to them as necessary. The "return" statement returns the values of these variables.
As for your actual question: the only way to avoid temporary variables would be using non-temporary variables, but usually that would be quite unwise - and probably not much of an optimization even when safe.
So, why doesn't the language specification allow this kind of special use of map indexing (or type assertion or channel receive, both of which can also utilize the "comma ok" idiom) in return statements? That's a good question. My guess: to keep the language specification simple.
I'm no Go expert but I believe you are getting compile time error when you are trying to return the array i.e. return all[id]. The reason could be because the functions return type is specially mentioned as (Info, bool) and when you are doing return all[id] it can't map the return type of all[id] to (Info, bool).
However the solution mentioned above, the variables being returned i and ok are the same that are mentioned in the return type of the function (i Info, ok bool) and hence the compiler knows what it's returning as opposed to just doing (i Info, ok bool).
By default, maps in golang return a single value when accessing a key
https://blog.golang.org/go-maps-in-action
Hence, return all[id] won't compile for a function that expects 2 return values.

JPA2 Criteria API .as(String.class) casting to char(1) - How do I work around this?

Using the criteria api, I have a query which does something like:
CriteriaBuilder cb = entityManager.getCriteriaBuilder();
CriteriaQuery<SourcePath> pathQuery = cb.createQuery(SourcePath.class);
Root<SourcePath> pathRoot = pathQuery.from(SourcePath.class);
pathQuery.where(cb.equal(cb.literal(0x00010002).as(String.class), (pathRoot.get(SourcePath_.path))));
TypedQuery<SourcePath> query = entityManager.createQuery(pathQuery);
query.getResultList();
The resulting sql query results something like:
select ...snip aliases... from SourcePath where cast(x'00010002', char(1)) = path;
(path would be some nasty old alias, but that's irrelevant).
This query is incorrect. Particularly, the cast: cast(x'00010002', char(1)) is not a cast to as string, as specified by .as(String.class), instead it should be either cast(x'00010002', char), or cast(x'00010002', char(N) where N is a suitably big enough number.
I've isolated the cause of this cast failure to the MySqlDialect provided by org.hibernate. Particularly:
public String getCastTypeName(int code) {
if ( code==Types.INTEGER ) {
return "signed";
}
else if ( code==Types.VARCHAR ) {
return "char";
}
...snip...
}
Which farther down the chain is interpreted as a char, which is registered by the dialog: registerColumnType( Types.CHAR, "char(1)" );.
Finally, to my question. How can I work around this behaviour? Do I report it as a bug to Hibernate? Do I extend the Dialog and correct the returned type from getCastTypeName? Is there an alternative to .as which will appropriately cast? Or, should I use strings everywhere I'm currently using hex-values to avoid touching edge cases of hibernate's implementation?
Thanks
idb
IMHO, you should use a String literal like cb.literal("\u0001\u0002"). Reasons:
It's the same thing as cb.literal(0x00010002).as(String.class), but less verbose.
Does not hit any "edge case"
It's clearer: does "0x00010002" be treated in Big Endian or LE? Which encoding should be used?
You can improve legibility using constants (e.g.: cb.literal(ASCII_SOH + ASCII_STX) where SOH="\u0001" and STX="\u0002").
EDIT: Adding a better description, since I didn't see the "Or, should I use strings everywhere I'm currently using hex-values"