i got a problem and its driving me nuts.
in a function called cargo_id_whitelist(MYSQL * conexion,struct info * data)
i perform a mysql Query. but every time i call the mysql api funcion
mysql_use_res(conexion) it stomp over used memory, ruining data (mostly of the data structure)
example
printf("-kind-> %d \n",conf_var->next->next->id) //work its display the third node id info;
res=( MYSQL_RES *)mysql_use_result(conexion); //this break my memory
printf("puntero %p \n",res);
printf("-kind-> %d \n",conf_var->next->next->id); //segfault
conf_var is a linked list.
theres is something i need to know?
If, as in your comment, your code is as follows:
MYSQL *conn;
if (!mysql_real_connect(conn, blah, blah, blah)) {
return 1;
}
then you're breaking the rules and all bets are off. The MySQL documentation page for mysql_real_connect() states:
The first parameter should be the address of an existing MYSQL structure. Before calling mysql_real_connect() you must call mysql_init() to initialize the MYSQL structure.
The statement MYSQL *conn; (assuming it's not static storage duration) simply creates a pointer that points to an arbitrary location and, unless you init it, using it is likely to cause you a great deal of grief.
The fix is probably simply replacing:
MYSQL *conn;
with:
MYSQL *conn = mysql_init (NULL);
That will give you a new object, properly initialised, that you can then pass to mysql_real_connect().
Related
I have Lambda that uses RDS. I wanted to improve it and use the Lambda connection caching. I have found several articles, and implemented it on my side, best to my knowledge. But now, I am not sure it is this the rigth way to go.
I have Lambda (running Node 8), which has several files used with require. I will start from the main function, until I reach the MySQL initializer, which is exact path. All will be super simple, showing only to flow of the code that runs MySQL:
Main Lambda:
const jobLoader = require('./Helpers/JobLoader');
exports.handler = async (event, context) => {
const emarsysPayload = event.Records[0];
let validationSchema;
const body = jobLoader.loadJob('JobName');
...
return;
...//
Job Code:
const MySQLQueryBuilder = require('../Helpers/MySqlQueryBuilder');
exports.runJob = async (params) => {
const data = await MySQLQueryBuilder.getBasicUserData(userId);
MySQLBuilder:
const mySqlConnector = require('../Storage/MySqlConnector');
class MySqlQueryBuilder {
async getBasicUserData (id) {
let query = `
SELECT * from sometable WHERE id= ${id}
`;
return mySqlConnector.runQuery(query);
}
}
And Finally the connector itself:
const mySqlConnector = require('promise-mysql');
const pool = mySqlConnector.createPool({
host: process.env.MY_SQL_HOST,
user: process.env.MY_SQL_USER,
password: process.env.MY_SQL_PASSWORD,
database: process.env.MY_SQL_DATABASE,
port: 3306
});
exports.runQuery = async query => {
const con = await pool.getConnection();
const result = con.query(query);
con.release();
return result;
};
I know that measuring performance will show the actual results, but today is Friday, and I will not be able to run this on Lambda until the late next week... And really, it would be awesome start of the weekend knowing I am in right direction... or not.
Thank for the inputs.
First thing would be to understand how require works in NodeJS. I do recommend you go through this article if you're interested in knowing more about it.
Now, once you have required your connection, you have it for good and it won't be required again. This matches what you're looking for as you don't want to overwhelm your database by creating a new connection every time.
But, there is a problem...
Lambda Cold Starts
Whenever you invoke a Lambda function for the first time, it will spin up a container with your function inside it and keep it alive for approximately 5 mins. It's very likely (although not guaranteed) that you will hit the same container every time as long as you are making 1 request at a time. But what happens if you have 2 requests at the same time? Then another container will be spun up in parallel with the previous, already warmed up container. You have just created another connection on your database and now you have 2 containers. Now, guess what happens if you have 3 concurrent requests? Yes! One more container, which equals one more DB connection.
As long as there are new requests to your Lambda functions, by default, they will scale out to meet demand (you can configure it in the console to limit the execution to as many concurrent executions as you want - respecting your Account limits)
You cannot safely make sure you have a fixed amount of connections to your Database by simply requiring your code upon a Function's invocation. The good thing is that this is not your fault. This is just how Lambda functions behave.
...one other approach is
to cache the data you want in a real caching system, like ElasticCache, for example. You could then have one Lambda function be triggered by a CloudWatch Event that runs in a certain frequency of time. This function would then query your DB and store the results in your external cache. This way you make sure your DB connection is only opened by one Lambda at a time, because it will respect the CloudWatch Event, which turns out to run only once per trigger.
EDIT: after the OP sent a link in the comment sections, I have decided to add a few more info to clarify what the mentioned article wants to say
From the article:
"Simple. You ARE able to store variables outside the scope of our
handler function. This means that you are able to create your DB
connection pool outside of the handler function, which can then be
shared with each future invocation of that function. This allows for
pooling to occur."
And this is exactly what you're doing. And this works! But the problem is if you have N connections (Lambda Requests) at the same time. If you don't set any limits, by default, up to 1000 Lambda functions can be spun up concurrently. Now, if you then make another 1000 requests simultaneously in the next 5 minutes, it's very likely you won't be opening any new connections, because they have already been opened on previous invocations and the containers are still alive.
Adding to the answer above by Thales Minussi but for a Python Lambda. I am using PyMySQL and to create a connection pool I added the connection code above the handler in a Lambda that fetches data. Once I did this, I was not getting any new data that was added to the DB after an instance of the Lambda was executed. I found bugs reported here and here that are related to this issue.
The solution that worked for me was to add a conn.commit() after the SELECT query execution in the Lambda.
According to the PyMySQL documentation, conn.commit() is supposed to commit any changes, but a SELECT does not make changes to the DB. So I am not sure exactly why this works.
When using MySQL in C you free the memory using the MySQL API, like so:
MYSQL* connection = NULL;
connection = mysql_init(NULL);
// Stuff...
mysql_close(connection);
But Splint doesn't know that mysql_close is in fact freeing memory, so I get this error:
Fresh storage connection not released before return
A memory leak has been detected. Storage allocated locally is
not released before the last reference to it is lost. (Use
-mustfreefresh to inhibit warning)
How do I tell Splint that mysql_close is deallocating memory? A special annotation to the mysql.h file?
EDIT: OK, maybe the releases *p annotation, if that can be used in a header file. Will try.
EDIT 2: Added /*#releases *sock#*/ to mysql.h, but now get this error:
Releases clauses includes *sock of non-dynamically allocated
type MYSQL
A declaration uses an invalid annotation. (Use -annotationerror to inhibit
warning)
This is the signaure of mysql_close:
void STDCALL mysql_close(/*#notnull#*/ MYSQL *sock) /*#releases *sock#*/;
I believe that proper annotation would be:
void STDCALL mysql_close(/*#special#*/ /*#notnull#*/ MYSQL *sock)
/*#releases sock#*/;
The key, that you have missed is /*#special#*/ annotation, that is required to "activate" so called state clauses. From Splint's documentation, 7.4 State Clauses:
The /*#special#*/ annotation is used to mark a parameter, global
variable, or return value that is described using state clauses.
Here's some simplified code in my web application:
sub insert {
my $pid = fork();
if ($pid > 0) {
return;
}
else {
&insert_to_mysql();
my $last_id = &get_last_inserted(); # call mysql last_inserted_id
exit(0);
}
}
for my $i (1..10) {
&insert();
}
Since insert is called in a multiprocessing environment, the order of get_last_inserted might be uncertain. Will it always return a correct last id corresponding to insert_to_mysql subroutine? I read some documents saying that as long as the processes don't share the same mysql connection, the returned id will be always the right one. However, these processes are spawned from the same session, so I'm not sure if they share the mysql connection or not. Thanks in advance.
these processes are spawned from the same session
Are you saying you're forking and using the same connection in more than one process? That doesn't work at all, never mind LAST_INSERT_ID(). You can't have two processes reading and writing from the same connection! The response for one could end up in the other, assuming the two clients didn't clobber each other's request.
Does last_insert_id return the correct auto_increment id in a multiprocessing environment?
According to MySQL's documentation for LAST_INSERT_ID(),
The ID that was generated is maintained in the server on a per-connection basis.
It would be useless otherwise. Since connections can't shared across processes, yes, it's safe.
I don't know about MySql and perl, but in PHP that's quite the same issue, since it depends on the environment and not on the language. In PHP, last_insert_id expects one parameter: current connection! As long as multiple instances do not share the same connection ressource, passing the connection resource to the current mysql session should do the trick.
That's what I've found googling around: http://www.xinotes.org/notes/note/179/
I don't have a good knowledge of SSL principles, but just want the encryption to work for me.
I have a DB and a user with "REQUIRE X509" specified.
The necessary certificates have been created as described in MySQL docs, and work well - i can connect to the server from Windows command line.
The problem arises, when i try to do the same from my program using MySQL Client API (without SSL, the program also works fine).
The unit used is: http://www.audio-data.de/mysql.html.
These are my action paths:
1) if i just add mysql_ssl_set() call (with proper params) before mysql_real_connect(), the last one gives generic SSL Connection Error.
2) the MySQL docs in en/mysql-ssl-set.html say, that the function always returns 0. But when i checked that, it appeared that the result is the number 11150848. Then i wrote it like that:
showmessage(inttostr(mysql_ssl_set(mys, '.\certs\client-key.pem', '.\certs\client-cert.pem', '.\certs\ca-cert.pem', nil)));
...and repeated the line 8 times.
Each time it returned a slightly greater number - 11158528, 11158784, 11159040, ... and two zeroes for the last two calls.
After which mysql_real_connect() was finally successful! The program even managed to execute some queries, return proper results for them (i know the data), but then it crashed with an Access Violation: write of address ... at some place.
The crash point varied between runs and slight changes to code.
It looks much like a version incompatibility issue. I tried libraries from both MySQL 5.0 and 5.1 Windows installations (the server is 5.1 and runs under Linux remotely; however, 5.0 mysql-client programs do not have troubles when SSL-connecting to it), but with no success.
Is anybody familiar with the issue? Thanks a lot for the help & sorry for the mistakes in the question.
As I see the mysql_ssl_set declaration is incorrect. It is declared:
function mysql_ssl_set(_mysql: PMYSQL; key, cert, ca, capath: PAnsiChar): longint; stdcall;
But the mysql.h contains:
my_bool STDCALL mysql_ssl_set(MYSQL *mysql, const char *key,
const char *cert, const char *ca,
const char *capath, const char *cipher);
That explains the garbage in return value, AV's and so on.
I'm using latest version of Xampp on 64bit Win7.
The problem is that, when I use mysql_connect with "bool $new_link" set to true like so:
mysql_connect('localhost', 'root', 'my_password', TRUE);
script execution time increases dramatically (about 0,5 seconds per connection, and when I have 4 diffirent objects using different connections, it takes ~2 seconds).
Is setting "bool $new_link" to true, generally a bad idea or could it just be some problem with my software configuration.
Thank you.
//Edit:
I'm using new link, because I have multiple objects, that use mysql connections (new objects can be created inside already existing objects and so on). In the end, when it comes to unsetting objects (I have mysql_close inside my __destruct() functions), I figured, that the only way to correctly clean up loose ends would be that all objects have their own connection variables.
I just formated my PC so configuration should be default conf.
Don't open a new connection unless you have a need for one (for instance, accessing multiple databases simultaneously).
Also you don't have to explicitly call mysql_close. I usually just include a function to quickly retrieve an existing db link (or a new one if none exists yet).
function &getDBConn() {
global $DBConn;
if(!$DBConn) $DBConn = mysql_connect(...);
return $DBConn;
}
// now you can just call $dbconn = getDBConn(); whenever you need it
Use "127.0.0.1" instead of "localhost". It improved my performance with mysql_connect from ~1 sek to a couple of milliseconds.
Article about php/mysql_connect and IPv6 on windows: http://www.bluetopazgames.com/uncategorized/php-mysql_connect-is-slow-1-second-for-localhost-windows-7/