I have an artisan command, in which I am cleaning up some data which has gone bad. Before I actually delete the data, I want to do a dry run and show some of the implications that deleting that data may present.
The essesnce of my command is:
public function handle()
{
...
$this->dryRun($modelsToDelete); // Prints info to user
if ($this->confirm('Are you sure you want to delete?') {
$modelsToDelete->each->forceDelete();
}
...
}
public function dryRun($modelsToDelete)
{
...
DB::connection($connection)->beginTransaction();
$before = $this->findAllOrphans($models);
$modelsToDelete->each(function ($record) use ($bar) {
$record->forceDelete();
});
$after = $this->findAllOrphans($models);
DB::connection($connection)->rollBack();
// Print info about diff
...
}
The problem is that when I do the dry run, and confirm to delete, the actual operation is not persisting in the database. If I comment out the dry run and do the command, the operation does persist. I have checked DB::transactionLevel() before and after the dry run and real operation, and everything seems correct.
I have also tried using DB::connection($connection)->pretend(...), but still the same issue. I also tried doing DB::purge($connection) and DB::reconnect($connection) after rolling back.
Does anyone have any thoughts as to what is going on?
(Using Laravel v6.20.14)
after digging the source code, I found out that laravel set property "exists" to false after you call delete on model instance and it will not perform delete query again. you can reference:
https://github.com/laravel/framework/blob/9edd46fc6dcd550e4fd5d081bea37b0a43162165/src/Illuminate/Database/Eloquent/Model.php#L1173
https://github.com/laravel/framework/blob/9edd46fc6dcd550e4fd5d081bea37b0a43162165/src/Illuminate/Database/Eloquent/Model.php#L1129
and to make model instance can be deleted after dryRun, you should pass a deep copy to dryRun, for example:
$this->dryRun(unserialize(serialize($modelsToDelete)));
note: don't use php clone because it's create a shallow copy
Turn on MySQL's "General log".
Run the experiment that is giving you trouble.
Turn off that log.
The problem may be obvious in the log; if not show us the log.
Related
I'm belatedly writing some PHPUnit feature tests for a project. One of these hits a number of routes which modify the database - and until now, I've been using database transactions in some of these routes.
If I use either the RefreshDatabase or DatabaseTransactions trait in my test, it fails with a strange error:
Illuminate\Database\Eloquent\ModelNotFoundException: No query results for model [App\Models\ClassM]
If I remove the traits (so the data will persist), the test passes.
I went through and removed all my database transactions from the relevant routes, and the test now does pass with RefreshDatabase.
My suspicion is that the problem is that MySQL doesn't support nested transactions, and both traits use transactions. But that would mean that if I want to run tests with RefreshDatabase (by far the best option!), I can't use transactions at all in my code.
Is that correct? It seems a major limitation.
I've found a workaround - and it seems a little strange. I was mostly using the "manual" transaction methods - e.g.
DB::beginTransaction();
try {
$user = User::create([...]); etc.
DB::commit();
}
catch (...) {
DB::rollBack();
}
I simply switched to the closure method:
DB::transaction(function () {
$user = User::create([...]); etc.
});
No idea why that would have fixed it!
does anyone know anything about debugging with knexjs and mysql? I'm trying to do a lot of things and test out stuff and I keep polluting my test database with random data. ideally, I'd just like to do things and see what the output query would be instead of running it against the actual database to see if it actually worked.
I can't find anything too helpful in their docs. they mention passing {debug: true} as one of the options in your initialize settings but it doesn't really explain what it does.
I am a junior developer, so maybe some of this is not meant to be understood by juniors but at the end of the day Is just not clear at all what steps I should take to be able to just see what queries would have been ran instead of running the real queries and polluting my db.
const result = await db().transaction(trx =>
trx.insert(mapToSnakeCase(address), 'id').into('addresses')
.then(addressId =>
trx.insert({ addresses_id: addressId, display_name: displayName }, 'id')
.into('chains')).toString();
You can build a knex query, but until you attach a .then() or awiat() (or run . asCallback((error,cb)=>{})), the query is just an object.
So you could do
let localVar = 8
let query = knex('table-a').select().where('id', localVar)
console.log(query.toString())
// outputs a string 'select * from table-a where id = 8'
This does not hit the database, and is synchronous. Make as many of these as you want!
As soon as you do await query or query.then(rows => {}) or query.asCallback( (err,rows)=>{} ) you are awaiting the db results, starting the promise chain or defining the callback. That is when the database is hit.
Turning on debug: true when initializing just writes the results of query.toSQL() to the console as they run against the actual DB. Sometimes an app might make a lot of queries and if one goes wrong this is a way to see why a DB call failed (but is VERY verbose so typically is not on all the time).
In our app's tests, we do actually test against the database because unit testing this type of stuff is a mess. We use knex's migrations on a test database that is brought down and up every time the tests run. So it always starts clean (or with known seed data), and if a test fails the DB is in the same state to be manually inspected. While we create a lot of test data in a testing run, it's cleaned up before the next test.
I'm trying to add some lua functionality to my existing conky setup so that repetitive "code" in my conky text can be cleaned up. For example, I have information for each mounted FS, each core, etc. where each row displayed in my panel differs ONLY by one parameter.
My first skeletal, attempt at using lua functions for this seems to run but displays nothing in my panel. I've only found very simple examples to base this on, so I may have made a simple error, but I don't even know how to diagnose it. My code here is modeled after what I HAVE been able to find regarding writing functions, such as this How to implement a basic Lua function in Conky? , but that's about all the depth I've found on the topic except for drawing and cairo examples.
Here's the code added to my conky config, as well as the contents of my functions.lua file
conky.config = {
...
lua_load = '/home/conky-manager/MyConky/functions.lua',
};
conky.text = [[
...
${voffset 5}${lua conky_test 'test'}
...
]]
file - functions.lua
function conky_test(parm1)
return 'result text'
end
What I would expect is to see is "result text" displayed in my panel at the location where that function call appears, but nothing shows.
Is there a log created by conky as it runs, or a way to provide some debug output? Even if I'd made a simple error here, I'd still like to have the ability to diagnose things as my code gets more complex.
Success!
After cobbling info from several articles together, I figured out my basic flaws -
1. Missing a 'conky_main' function,
2. Missing a 'lua_draw_hook_post' to invoke it, and
3. Realizing that if I invoke conky from a terminal, print statements in lua would appear there.
So, for anyone who sees this question and has the same issues, here's the corrected code.
conky.config = {
...
lua_load = '/home/conky-manager/MyConky/functions.lua',
lua_draw_hook_post = "main",
};
conky.text = [[
...
${lua conky_test 'test'}
...
]]
and the proper basics in my functions.lua file
function conky_test(parm1)
return 'result text'
end
function conky_main()
if conky_window == nil then
return
end
end
A few notes:
I still haven't determined if using 'lua_draw_hook_pre' instead of 'lua_draw_hook_post' makes any difference, but it doesn't seem to in this example.
Also, some examples showed actually calling this 'test' function instead of writing a 'main', but the 'main' seemed to have value in checking to see if conky_window existed.
Some examples seemed to state that naming functions with the prefix 'conky_' was required, but then showed examples of calling those functions without the prefix, so I assume the prefix is inferred during the call.
a major note: you should run conky from the directory containing the lua scripts.
Given: extension that has a properly configured cronjob (let's say, every 5 minutes) in Config.xml. Also, the system cron is set to run Magento's cron.sh. The cronjob has to run a couple of times after the extension installed, and when it has no more data to process then it becomes obsolete.
Problem: the job isn't needed after it had processed all the data. However, its setup in Config.xml causes it to run every 5 minutes forever, just to check that there is no more data and die.
Question: is there any proper way (maybe with the cron_schedule table...) to 'dismiss' the cronjob programmatically from its own PHP when it sees that there is no more data? Or any other way?
The cron is used since the extension installation process shouldn't be interrupted. Maybe it's possible to schedule some PHP code in some other way than cron (but within Magento)? Thought about threading but since there is no guarantee that this feature will be built in, this doesn't seem to be the option....
Thanks in advance!
So, I found 2 possible solutions: 1) it seems to be possible to create/remove crontabs via core_config_data table without config.xml; 2) remove the crontab node from config.xml after all data is processed + clean the cache + remove all pending tasks. I've managed to implement the 2nd, and it works (I know that the 1st approach is much better, but I just had no time to dig it out).
The 2nd looks like:
if ($more_data) {
// processing...
} else { // Dismissing the cron
$config_xml_path = Mage::getModuleDir('etc', 'the_extension') . '/config.xml';
$config_xml = simplexml_load_file($config_xml_path) or die("Error: Cannot create object");
if (isset($config_xml) && isset($config_xml->crontab)) {
unset($config_xml->crontab);
$config_xml->asXML($config_xml_path);
}
// Cleaning
Mage::app()->cleanCache();
$schedule = Mage::getModel('cron/schedule');
$sch_col = $schedule->getCollection()
->addFilter('job_code', 'the_extension_cronFunc')
->addFilter('status', 'pending');
foreach ($sch_col as $s) {
$s->delete();
}
}
I have a question about testing the queries in a transaction. I've been using MySQL transactions for quite some time now, and everytime I do this, I use something like:
$doCommit = true;
$error = "";
mysql_query("BEGIN");
/* repeat this part with the different queries in the transaction
this often involves updating of and inserting in multiple tables */
$query = "SELECT, UPDATE, INSERT, etc";
$result = mysql_query($query);
if(!$result){
$error .= mysql_error() . " in " . $query . "<BR>";
$doCommit = false;
}
/* end of repeating part */
if($doCommit){
mysql_query("COMMIT");
} else {
echo $error;
mysql_query("ROLLBACK");
}
Now, it often happens that I want to test my transaction, so I change mysql_query("COMMIT"); to mysql_query("ROLLBACK");, but I can imagine this is not a very good way to test this kind of stuff. It's usually not really feasable to copy every table to a temp_table and update and insert into those tables and delete them afterwards (for instance because tables maybe very large). Of course, when the code goes into production relevant error-handling (instead of just printing the error) is put into place.
What's the best way to do stuff like this?
First of all, there is a bug in your implementation. If a query errors out, the current transaction is automatically rolled back and then closed. So as you continue to execute queries, they will not be within a transaction (they will be commited to the DB). Then, when you execute Rollback, it'll silently fail. From the MySQL docs:
Rolling back can be a slow operation that may occur implicitly without the user
having explicitly asked for it (for example, when an error occurs).
The explicit command ROLLBACK should only be used if you determine in the application that you need to rollback (for reasons other than a query error). For example, if you're deducting funds from an account, you'd explicitly rollback if you found out the user didn't have enough funds to complete the exchange...
As far as testing the transactions, I do copy the database. I create a new database and install a set of "dummy data". Then I run all the tests using an automated tool. The tool will actually commit the transactions and force rollbacks, and check that the expected database state is maintained throughout the tests. Since it's harder to programatically know the end state from a transaction if you have an unknown input to the transaction, testing off of live (or even copied-from-live) data is not going to be easy. You can do it (and should), but don't depend upon those results for determining if your system is working. Use those results to build new test cases for the automated tester...
Maybe you could refactor your first example and use some DB access wrapper class?
In that wrapper class you can have a variable $normalCommit = true;
and a method SetCommitMode() which sets that $normalCommit variable.
And you have a method Commit() which commits if($normalCommit == true)
Or even have a variable $failTransaction which calls mysql_query("ROLLBACK"); if you wish (so you could pass/fail many sequential tests).
Then when you run the test, you can set somewhere in the test code file:
$myDBClass->SetCommitMode(false);
or
$myDBClass->RollBackNextOperation(true);
before the operation which you wish to fail, and it will just fail. In such a way the code which you are testing will not contain those fail/commit checks, only the DB class will contain them.
And normally ONLLY the test code (especially if you do unit testing) should call those SetCommitMode and RollBackNextOperation methods, so you accidentally do not leave those calls in the production code.
Or you could pass some crazy data to your method (if you are testing a method), like negative variables to save in UNSIGNED fields, and then your transaction should fail 100% if your code does not do commit after such an SQL error (but it should not).
Generally I use something like (I use pdo for my example):
$db->beginTransaction();
try {
$db->exec('INSERT/DELETE/UPDATE');
$db->commit();
}
catch (PDOException $e) {
$db->rollBack();
// rethrow the error or
}
Or if you have your own exception handler, use a special clause for your PDOExceptions, where to rollback the execution. Example:
function my_exception_handler($exception) {
if($exception instanceof PDOException) {
// assuming you have a registry class
Registry::get('database')->rollBack();
}
}