I have a web-application written in CodeIgniter, and for the most part everything seems fairly reasonable. However, I've noticed extremely high CPU usage when a user logs out. My logout function in my auth controller is as follows:
function logout()
{
$goto = $SERVER['HTTP-REFERER'];
$this->session->sess_destroy();
if (!$goto)
$goto = "/";
header('location: '.$goto);
}
Normally, this is perfectly fine and fast. The strange thing is that when linked to from a particular sub-page this function takes about 5-6 seconds my of mysqld running at 100% cpu. How can I see what it's doing, and why?
Taking a shot in the dark (if you're sure this function is causing the slowness):
First, you could turn on MySQL's slow query log:
http://dev.mysql.com/doc/refman/5.1/en/slow-query-log.html
Then, if $sess_use_database is TRUE you might try optimizing your session table. You could have some overhead causing issues.
Apart from that, the only other thing I can think of is that there is an issue with your DB server. You might try running the MySQL Tuner to see if you can improve things a bit:
https://github.com/rackerhacker/MySQLTuner-perl
Hope that helps!
FYI
Here is the code that is run when the OP runs sess_destroy() (from v2.0.2):
/**
* Destroy the current session
*
* #access public
* #return void
*/
function sess_destroy()
{
// Kill the session DB row
if ($this->sess_use_database === TRUE AND isset($this->userdata['session_id']))
{
$this->CI->db->where('session_id', $this->userdata['session_id']);
$this->CI->db->delete($this->sess_table_name);
}
// Kill the cookie
setcookie(
$this->sess_cookie_name,
addslashes(serialize(array())),
($this->now - 31500000),
$this->cookie_path,
$this->cookie_domain,
0
);
}
Related
I have an artisan command, in which I am cleaning up some data which has gone bad. Before I actually delete the data, I want to do a dry run and show some of the implications that deleting that data may present.
The essesnce of my command is:
public function handle()
{
...
$this->dryRun($modelsToDelete); // Prints info to user
if ($this->confirm('Are you sure you want to delete?') {
$modelsToDelete->each->forceDelete();
}
...
}
public function dryRun($modelsToDelete)
{
...
DB::connection($connection)->beginTransaction();
$before = $this->findAllOrphans($models);
$modelsToDelete->each(function ($record) use ($bar) {
$record->forceDelete();
});
$after = $this->findAllOrphans($models);
DB::connection($connection)->rollBack();
// Print info about diff
...
}
The problem is that when I do the dry run, and confirm to delete, the actual operation is not persisting in the database. If I comment out the dry run and do the command, the operation does persist. I have checked DB::transactionLevel() before and after the dry run and real operation, and everything seems correct.
I have also tried using DB::connection($connection)->pretend(...), but still the same issue. I also tried doing DB::purge($connection) and DB::reconnect($connection) after rolling back.
Does anyone have any thoughts as to what is going on?
(Using Laravel v6.20.14)
after digging the source code, I found out that laravel set property "exists" to false after you call delete on model instance and it will not perform delete query again. you can reference:
https://github.com/laravel/framework/blob/9edd46fc6dcd550e4fd5d081bea37b0a43162165/src/Illuminate/Database/Eloquent/Model.php#L1173
https://github.com/laravel/framework/blob/9edd46fc6dcd550e4fd5d081bea37b0a43162165/src/Illuminate/Database/Eloquent/Model.php#L1129
and to make model instance can be deleted after dryRun, you should pass a deep copy to dryRun, for example:
$this->dryRun(unserialize(serialize($modelsToDelete)));
note: don't use php clone because it's create a shallow copy
Turn on MySQL's "General log".
Run the experiment that is giving you trouble.
Turn off that log.
The problem may be obvious in the log; if not show us the log.
I have Lambda that uses RDS. I wanted to improve it and use the Lambda connection caching. I have found several articles, and implemented it on my side, best to my knowledge. But now, I am not sure it is this the rigth way to go.
I have Lambda (running Node 8), which has several files used with require. I will start from the main function, until I reach the MySQL initializer, which is exact path. All will be super simple, showing only to flow of the code that runs MySQL:
Main Lambda:
const jobLoader = require('./Helpers/JobLoader');
exports.handler = async (event, context) => {
const emarsysPayload = event.Records[0];
let validationSchema;
const body = jobLoader.loadJob('JobName');
...
return;
...//
Job Code:
const MySQLQueryBuilder = require('../Helpers/MySqlQueryBuilder');
exports.runJob = async (params) => {
const data = await MySQLQueryBuilder.getBasicUserData(userId);
MySQLBuilder:
const mySqlConnector = require('../Storage/MySqlConnector');
class MySqlQueryBuilder {
async getBasicUserData (id) {
let query = `
SELECT * from sometable WHERE id= ${id}
`;
return mySqlConnector.runQuery(query);
}
}
And Finally the connector itself:
const mySqlConnector = require('promise-mysql');
const pool = mySqlConnector.createPool({
host: process.env.MY_SQL_HOST,
user: process.env.MY_SQL_USER,
password: process.env.MY_SQL_PASSWORD,
database: process.env.MY_SQL_DATABASE,
port: 3306
});
exports.runQuery = async query => {
const con = await pool.getConnection();
const result = con.query(query);
con.release();
return result;
};
I know that measuring performance will show the actual results, but today is Friday, and I will not be able to run this on Lambda until the late next week... And really, it would be awesome start of the weekend knowing I am in right direction... or not.
Thank for the inputs.
First thing would be to understand how require works in NodeJS. I do recommend you go through this article if you're interested in knowing more about it.
Now, once you have required your connection, you have it for good and it won't be required again. This matches what you're looking for as you don't want to overwhelm your database by creating a new connection every time.
But, there is a problem...
Lambda Cold Starts
Whenever you invoke a Lambda function for the first time, it will spin up a container with your function inside it and keep it alive for approximately 5 mins. It's very likely (although not guaranteed) that you will hit the same container every time as long as you are making 1 request at a time. But what happens if you have 2 requests at the same time? Then another container will be spun up in parallel with the previous, already warmed up container. You have just created another connection on your database and now you have 2 containers. Now, guess what happens if you have 3 concurrent requests? Yes! One more container, which equals one more DB connection.
As long as there are new requests to your Lambda functions, by default, they will scale out to meet demand (you can configure it in the console to limit the execution to as many concurrent executions as you want - respecting your Account limits)
You cannot safely make sure you have a fixed amount of connections to your Database by simply requiring your code upon a Function's invocation. The good thing is that this is not your fault. This is just how Lambda functions behave.
...one other approach is
to cache the data you want in a real caching system, like ElasticCache, for example. You could then have one Lambda function be triggered by a CloudWatch Event that runs in a certain frequency of time. This function would then query your DB and store the results in your external cache. This way you make sure your DB connection is only opened by one Lambda at a time, because it will respect the CloudWatch Event, which turns out to run only once per trigger.
EDIT: after the OP sent a link in the comment sections, I have decided to add a few more info to clarify what the mentioned article wants to say
From the article:
"Simple. You ARE able to store variables outside the scope of our
handler function. This means that you are able to create your DB
connection pool outside of the handler function, which can then be
shared with each future invocation of that function. This allows for
pooling to occur."
And this is exactly what you're doing. And this works! But the problem is if you have N connections (Lambda Requests) at the same time. If you don't set any limits, by default, up to 1000 Lambda functions can be spun up concurrently. Now, if you then make another 1000 requests simultaneously in the next 5 minutes, it's very likely you won't be opening any new connections, because they have already been opened on previous invocations and the containers are still alive.
Adding to the answer above by Thales Minussi but for a Python Lambda. I am using PyMySQL and to create a connection pool I added the connection code above the handler in a Lambda that fetches data. Once I did this, I was not getting any new data that was added to the DB after an instance of the Lambda was executed. I found bugs reported here and here that are related to this issue.
The solution that worked for me was to add a conn.commit() after the SELECT query execution in the Lambda.
According to the PyMySQL documentation, conn.commit() is supposed to commit any changes, but a SELECT does not make changes to the DB. So I am not sure exactly why this works.
I need to update or create data in a mysql table from a large array (few 1000s objects) with sequelize.
When I run the following code it uses up almost all my cpu power of my db server (vserver 2gb ram / 2cpu) and clogs my app for a few minutes until it's done.
Is there a better way to do this with sequelize? Can this be done in the background somehow or as a bulk operation so it doesn't effect my apps performance?
data.forEach(function(item) {
var query = {
'itemId': item.id,
'networkId': item.networkId
};
db.model.findOne({
where: query
}).then(function(storedItem) {
try {
if(!!storedItem) {
storedItem.update(item);
} else {
db.model.create(item);
}
} catch(e) {
console.log(e);
}
});
});
Your first line of your sample code data.forEach()... makes a whole mess of calls to your function(item){}. Your code in that function fires off, in turn, a whole mess of asynchronously completing operations.
Try using the async package https://caolan.github.io/async/docs.htm and doing this
async = require('async');
...
async.mapSeries(data, function(item){...
It should allow each iteration of your function (which iterates once per item in your data array) to complete before starting the next one. Paradoxically enough, doing them one at a time will probably make them finish faster. It will certainly avoid soaking up your resources.
Weeks later I found the actual reason for this. (And unfortunately using async didn't really help after all) It was as simple as stupid: I didn't have an MYSQL index for itemId so with every iteration the whole table was queried which caused the high CPU load (obviously).
Given: extension that has a properly configured cronjob (let's say, every 5 minutes) in Config.xml. Also, the system cron is set to run Magento's cron.sh. The cronjob has to run a couple of times after the extension installed, and when it has no more data to process then it becomes obsolete.
Problem: the job isn't needed after it had processed all the data. However, its setup in Config.xml causes it to run every 5 minutes forever, just to check that there is no more data and die.
Question: is there any proper way (maybe with the cron_schedule table...) to 'dismiss' the cronjob programmatically from its own PHP when it sees that there is no more data? Or any other way?
The cron is used since the extension installation process shouldn't be interrupted. Maybe it's possible to schedule some PHP code in some other way than cron (but within Magento)? Thought about threading but since there is no guarantee that this feature will be built in, this doesn't seem to be the option....
Thanks in advance!
So, I found 2 possible solutions: 1) it seems to be possible to create/remove crontabs via core_config_data table without config.xml; 2) remove the crontab node from config.xml after all data is processed + clean the cache + remove all pending tasks. I've managed to implement the 2nd, and it works (I know that the 1st approach is much better, but I just had no time to dig it out).
The 2nd looks like:
if ($more_data) {
// processing...
} else { // Dismissing the cron
$config_xml_path = Mage::getModuleDir('etc', 'the_extension') . '/config.xml';
$config_xml = simplexml_load_file($config_xml_path) or die("Error: Cannot create object");
if (isset($config_xml) && isset($config_xml->crontab)) {
unset($config_xml->crontab);
$config_xml->asXML($config_xml_path);
}
// Cleaning
Mage::app()->cleanCache();
$schedule = Mage::getModel('cron/schedule');
$sch_col = $schedule->getCollection()
->addFilter('job_code', 'the_extension_cronFunc')
->addFilter('status', 'pending');
foreach ($sch_col as $s) {
$s->delete();
}
}
For a small chat server thingie I'm making, I decided to use D; finding myself
with a very neat example from listener.d to get a kick start I decided to
pretty much take the example over! However, I'm stuck on a bug I can't truly
wrap my finger around. Most likely it's my own fault and I'm doing something
wrong, but considering I took the code pretty much from the example I am more
inclined to believe the example is broken.
I'll explain what happens:
List item
I start up my server (nothing wrong, it's running as it should and listening)
I telnet to it. My server accepts the connection.
I use telnet to send some information. Server handles the information
properly, again, no issue.
I quit telnet by using ^] and then writing quit. Breaking the connection
rather ungracefully.
The server properly recognises this isn't a clean disconnect and executes
the code to remove the socket.
I then get a range violation.
This is the main process and it's loop:
https://github.com/JGBrands/BlaatServer/blob/master/source/bserver.d
This is the server class, the code where it deletes the socket is at the
complete bottom in the function void destroySocket(int index);
https://github.com/JGBrands/BlaatServer/blob/master/source/server.d
Actually let me copy paste that. :-)
void destroySocket(int index) {
this.reads[index].close(); /* release resources. */
/* Remove the socket now. We don't want this around! It'll crash us! */
if (index != this.reads.length -1)
this.reads[index] = this.reads[this.reads.length -1];
this.reads = this.reads[0 .. this.reads.length -1];
writeln("Total connections: " ~ to!string(this.reads.length));
}
The code is primarily taken over from the listener.d example like I said, the
error I get is this:
core.exception.RangeError#server(61): Range violation
----------------
----------------
I'm lead to believe the function is deleting something it shouldn't, for those
interested, this is line 61 in server.d:
if (this.sset.isSet(this.reads[i])) {
Hopefully you guys can help me make more sense about this, am I missing something really obvious here?
As ratchet points out:
if (index != this.reads.length -1)
This does not verify that index is within range. It only validates that index is not the last element. This is fine as long as index has already been verified to be within range.
void destroySocket(int index)
in {
assert(index > -1);
assert(index < this.reads.length);
} body {
{