I attempted a fix on ConfirmAccount extension, but my fix did not seem to work. Any suggestions?
Details:
"pruning of old requests will not trigger often, so old rejected requests may persist."
https://www.mediawiki.org/wiki/Extension:ConfirmAccount#Known_issues
This behavior will prevent rejected emails from requesting an account again. After Admin rejected an account request, the same username/email could not submit another request. Error on 2nd attempt:
Username is already in use in a pending account request.
We want to enable re-requests. To fix, i want to force prune after every rejection, to clear the request cache.
It appears that, currently, pruning occurs in file \extensions\ConfirmAccount\frontend\specialpages\actions\ConfirmAccount_body.php
# Every 30th view, prune old deleted items
if ( 0 == mt_rand( 0, 29 ) ) {
ConfirmAccount::runAutoMaintenance();
}
Therefor, the function runAutoMaintenance appears to be the pruning function. runAutoMaintenance lives in \ConfirmAccount\backend\ConfirmAccount.class.php
class ConfirmAccount {
/** * Move old stale requests to rejected list. Delete old rejected requests. */
public static function runAutoMaintenance() {
...
In order to call runAutoMaintenance after every reject-action, I think the call to runAutoMaintenance should be placed in function rejectRequest, in file \extensions\ConfirmAccount\business\AccountConfirmSubmission.php
Specifically, i think it can go directly under:
# Clear cache for notice of how many account requests there are
ConfirmAccount::clearAccountRequestCountCache();
Maybe pruning should also happen after Accept, Hold, and Spam actions. Unsure. For now, pruning after Reject should handle the original problem.
I attempted the above fix, and it did not seem to work. I'm at a loss.
Can someone help determine why this fix did not work?
Original code:
protected function rejectRequest( IContextSource $context ) {
....
# Clear cache for notice of how many account requests there are
ConfirmAccount::clearAccountRequestCountCache();
....
New code:
protected function rejectRequest( IContextSource $context ) {
....
# Clear cache for notice of how many account requests there are
ConfirmAccount::clearAccountRequestCountCache();
# Prune
ConfirmAccount::runAutoMaintenance();
....
On 2nd request, still getting "Username is already in use in a pending account request."
Looks like i solved it. There are 2 steps:
Set rejected items to expire immediately (in LocalSettings.php)
Prune rejected items at beginning of Request action (in RequestAccount_body.php)
Details:
in LocalSettings.php, after required declaration, set Rejected-Age to 0. That ensures rejected requests will be removed on prune-action:
require_once "$IP/extensions/ConfirmAccount/ConfirmAccount.php";
$wgRejectedAccountMaxAge = 0;
Add Prune code to the function that shows the Request form, in /ConfirmAccount/frontend/specialpages/actions/RequestAccount_body.php, function showForm. Add very last command in the function:
old code:
$out->addWikiMsg( 'requestaccount-footer' );
}
new code:
$out->addWikiMsg( 'requestaccount-footer' );
# PRUNE
ConfirmAccount::runAutoMaintenance();
}
Related
I have an artisan command, in which I am cleaning up some data which has gone bad. Before I actually delete the data, I want to do a dry run and show some of the implications that deleting that data may present.
The essesnce of my command is:
public function handle()
{
...
$this->dryRun($modelsToDelete); // Prints info to user
if ($this->confirm('Are you sure you want to delete?') {
$modelsToDelete->each->forceDelete();
}
...
}
public function dryRun($modelsToDelete)
{
...
DB::connection($connection)->beginTransaction();
$before = $this->findAllOrphans($models);
$modelsToDelete->each(function ($record) use ($bar) {
$record->forceDelete();
});
$after = $this->findAllOrphans($models);
DB::connection($connection)->rollBack();
// Print info about diff
...
}
The problem is that when I do the dry run, and confirm to delete, the actual operation is not persisting in the database. If I comment out the dry run and do the command, the operation does persist. I have checked DB::transactionLevel() before and after the dry run and real operation, and everything seems correct.
I have also tried using DB::connection($connection)->pretend(...), but still the same issue. I also tried doing DB::purge($connection) and DB::reconnect($connection) after rolling back.
Does anyone have any thoughts as to what is going on?
(Using Laravel v6.20.14)
after digging the source code, I found out that laravel set property "exists" to false after you call delete on model instance and it will not perform delete query again. you can reference:
https://github.com/laravel/framework/blob/9edd46fc6dcd550e4fd5d081bea37b0a43162165/src/Illuminate/Database/Eloquent/Model.php#L1173
https://github.com/laravel/framework/blob/9edd46fc6dcd550e4fd5d081bea37b0a43162165/src/Illuminate/Database/Eloquent/Model.php#L1129
and to make model instance can be deleted after dryRun, you should pass a deep copy to dryRun, for example:
$this->dryRun(unserialize(serialize($modelsToDelete)));
note: don't use php clone because it's create a shallow copy
Turn on MySQL's "General log".
Run the experiment that is giving you trouble.
Turn off that log.
The problem may be obvious in the log; if not show us the log.
I try to sync changes from GDrive, So I'm using the changes api. It works like a charm when I'm using it with restrictToMyDrive=true.
But when I tried to expand it to also track shared files restrictToMyDrive=false I encounter some major drawback. I have inconsist results - the parents field is missing sporadically.
Let say that we have user A that share that folder to user B:
rootSharedFolder => subFolder => subSubFolder => File
If user B calls the changes API relatively close to the time that user A share the rootSharedFolder , Then in 3/10 times some of the inner folder will be received without the parents field.
Even trying to use the files.get API on the received changed item , results in empty parents field. But if I wait a minute or two and then call it again , the parents field does exist in the result.
Anyone else encounter this problem , and maybe have a workaround for it?
Thanks!
this behavior happens only when calling the changes api close to the time that the other user share you the items
Parents field may be missing due two factors:
A propagation delay
Insufficient permissions to see the parents
In both cases you'll notice the same: the parents field for the file is missing, at first there is no way to tell in which case you are:
A propagation issue:
This may happen when you request some file details related to a shared file not owned by you right after it was shared to you.
You may not be able to find it's related parents at first glance, this is because changes are still propagating on the file system, this is call a propagation issue, it should not last long and it should be possible to identify and solve this inconvenience by retrieving this field data a couple of minutes after the permission changes.
Not having access to the parents:
In this case you may have access to a certain file, but not to it's parent folder, thus, you can not get to know what parent it has, because it's not been shared with you.
This is on the documentation:
parents — A parent does not appear in the parents list if the requesting user is a not a member of the shared drive and does not have access to the parent. In addition, with the exception of the top level folder, the parents list must contain exactly one item if the file is located within a shared drive.
Side note: you may be interested on using SharedDrives, where files are owned by a organization rather than individual users, simplifying the sharing process and maybe avoiding the problems you are facing here.
https://developers.google.com/drive/api/v3/enable-shareddrives
How to know which case is?
A way to go is to implement an exponential back-off algorithm to try to retrieve the missing parents field, if after a max number of attempts it does not retrieve we are probably on the second case:
exponentialBackoff(getParent, 7, 300, function(result) {
console.log('the result is',result);
});
// A function that keeps trying, "getParent" until it returns true or has
// tried "max" number of times. First retry has a delay of "delay".
// "callback" is called upon success.
function exponentialBackoff(toTry, max, delay, callback) {
console.log('max',max,'next delay',delay);
var result = toTry();
if (result) {
callback(result);
} else {
if (max > 0) {
setTimeout(function() {
exponentialBackoff(toTry, --max, delay * 2, callback);
}, delay);
} else {
console.log('we give up');
}
}
}
function getParent() {
var percentFail = 0.8;
return Math.random() >= 0.8;
}
I have a route in my API, as an example lets call it /users/:userId/updateBalance. This route will fetch the users current balance, add whatever comes from the request and then update the balance with the newly calculated balance. A request like this comes into the server for a specific user every 30 minutes, so until recently, I thought a concurrency issue to be impossible.
What ended up happening is that somewhere, a sent request failed and was only sent again 30 minutes later, about within a second of the other request. The result was that, as I can see it in the database, both of these requests fetched the same balance from the DB and both added their respective amounts. Essentially, the second request actually read a stale balance, as normally it should execute after request 1 has executed.
To give a numerical example for some more clarity, lets say request 1 was to add $2 to the balance, and request 2 was to add $5, and the user had a balance of $10. If the requests act in parallel, the users balance would end at either $12 or $15 depending on whether request 1 or request 2 finished first respectively, because both requests fetch a balance of $10 from the DB. However, obviously the expected behaviour is that we want request 1 to execute, update the users balance to $12, and then request 2 to execute and update the balance from $12 to $17.
To give some better perspective of the overall execution of this process: the request is received, a function is called, the function has to wait for the balance from the DB, the function then calculates the new balance and updates the db, after which execution is completed.
So I have a few questions on this. The first being, how does node handle incoming requests when it is waiting for an asynchronous request like a MySQL database read. Given the results I have observed, I assume that when the first request is waiting for the DB, the second request can commence being processed? Otherwise I am uncertain of how such asynchronous behaviour is experienced within a single threaded environment like node.
Secondly, how do I go about controlling this and preventing it. I had wanted to use a MySQL transaction with a forUpdate lock, but it turns out it seems not possible due to the way the code is currently written. Is there a way to tell node that a certain block of code can not be executed "in parallel"? Or any other alternatives?
You are right, while node waits for the database query to return, it will handle any incoming requests and start that requests database call before the first one finishes.
The easiest way to prevent this IMO would be to use queues. Instead of processing the balance update directly in the route handler, that route handler could push an event to a queue (in Redis, in AWS SQS, in RabbitMQ etc) and somewhere else in your app (or even in a completely different service) you would have a consumer that listens to new events in that queue. If an update fails, add it back to the beginning of the queue, add some wait time, and then try again.
This way, no matter how many times your first request fails, your balance will be correct, and pending changes to that balance will be in the correct order. In case of an event in the queue failing repeatedly, you could even send an email or a notification to someone to have a look at it, and while the problem is fixed pending changes to the balance will be added to the queue, and once it's fixed, everything will be processed correctly.
You could even read that queue and display information to your user, for instance tell the user the balance has pending updates so it might not be accurate.
Hope this helps!
The first being, how does node handle incoming requests when it is waiting for an asynchronous request like a MySQL database read
The event loop of nodejs makes this happens, otherwise you'll have a totally sync programm with super-low performances.
Every single async function invocked in a context will be executed after the context itself has been executed.
Between the finish of execution of the context and the execution of the async function, other async functions can be scheduled for been executed (this "insertion" is managed by the event loop).
If an async function is awaited, the remaining code of the context is scheduled somewhere after the execution of the async function.
Is more clear when playing with it. Example 1:
// Expected result: 1, 3, 4, 2
function asyncFunction(x) {
// setTimeout as example of async operation
setTimeout(() => console.log(x), 10)
}
function context() {
console.log(1)
asyncFunction(2)
console.log(3)
}
context()
console.log(4)
Example 2:
// Expected result: 1, 2, 3
function asyncFunction(x) {
// Promise as example of async operation
return new Promise((resolve) => {
console.log(x)
resolve()
})
}
async function context() {
console.log(1)
await asyncFunction(2)
console.log(3)
}
context()
Example 3 (more similar to your situation):
// Expected result: 1, 2, 4, 5, 3, 6
function asyncFunction(x) {
// Promise as example of async operation
return new Promise((resolve) => {
console.log(x)
resolve()
})
}
async function context(a, b, c) {
console.log(a)
await asyncFunction(b)
console.log(c)
}
context(1, 2, 3)
context(4, 5, 6)
In your example:
when the server receive a connection, the execution of the handler is scheduled
when the handler is executed, it schedule the execution of the query, and the remaining portion of the handler context is scheduled after that
In between scheduled executions everything can happen.
I've added a 'product' content type to my Bolt installation. I want to track and update inventory when a purchase is made, so I added an integer field 'available'. I set up a test controller to modify the record. Everything appears to work, but the update never happens. What am I missing?
<?php
namespace Bundle\Site;
class CartController extends \Bolt\Controller\Base
{
public function addRoutes(\Silex\ControllerCollection $c)
{
$c->match('/test-save', [$this,'testSave']);
return $c;
}
public function testSave()
{
$result=false;
$repo = $this->app['storage']->getRepository('products');
$content = $repo->find(1);
//Error log output confirms that this is the correct record
error_log(get_class($this).'::'.__FUNCTION__.': '.json_encode($content));
$content->set('available',15);
$content->setDatechanged('now');
$result=$repo->save($content); //returns 1
return new \Symfony\Component\HttpFoundation\Response(json_encode($result), \Symfony\Component\HttpFoundation\Response::HTTP_OK);
}
}
It turns out that my code works. I was checking the result by reloading the back-end edit form, which had apparently cached its data, so I didn't see the value changing.
The same thing happens if I load the edit form in two browser tabs and update one, then refresh the other, so it looks like this is the best I can do.
I have faced a similar problem. Either the updated codes didn't show up or the result in the output is unchanged. Figured though this happened due to setting debug: false in the config.yml file. When debug is true your in bolt-data/app/ all cache files increases but this thing get sorted.
In production it is suggested that to keep debug: false but this problem checks-in too, hence the way you are dealing with it comes out to be the easiest way out without changing the configuration much.
In Windows and Android Google Chrome browser, (haven't tested for others yet) response time from a service worker increases linearly to number of items stored in that specific cache storage when you use Cache.match() function with following option;
ignoreSearch = true
Dividing items in multiple caches helps but not always convenient to do so. Plus even a small amount of increase in items stored makes a lot of difference in response times. According to my measurements response time is roughly doubled for every tenfold increase in number of items in the cache.
Official answer to my question in chromium issue tracker reveals that the problem is a known performance issue with Cache Storage implementation in Chrome which only happens when you use Cache.match() with ignoreSearch parameter set to true.
As you might know ignoreSearch is used to disregard query parameters in URL while matching the request against responses in cache. Quote from MDN:
...whether to ignore the query string in the url. For example, if set to
true the ?value=bar part of http://example.com/?value=bar would be ignored
when performing a match.
Since it is not really convenient to stop using query parameter match, I have come up with following workaround, and I am posting it here in hopes of it will save time for someone;
// if the request has query parameters, `hasQuery` will be set to `true`
var hasQuery = event.request.url.indexOf('?') != -1;
event.respondWith(
caches.match(event.request, {
// ignore query section of the URL based on our variable
ignoreSearch: hasQuery,
})
.then(function(response) {
// handle the response
})
);
This works great because it handles every request with a query parameter correctly while handling others still at lightning speed. And you do not have to change anything else in your application.
According to the guy in that bug report, the issue was tied to the number of items in a cache. I made a solution and took it to the extreme, giving each resource its own cache:
var cachedUrls = [
/* CACHE INJECT FROM GULP */
];
//update the cache
//don't worry StackOverflow, I call this only when the site tells the SW to update
function fetchCache() {
return Promise.all(
//for all urls
cachedUrls.map(function(url) {
//add a cache
return caches.open('resource:'url).then(function(cache) {
//add the url
return cache.add(url);
});
});
);
}
In the project we have here, there are static resources served with high cache expirations set, and we use query parameters (repository revision numbers, injected into the html) only as a way to manage the [browser] cache.
It didn't really work to use your solution to selectively use ignoreSearch, since we'd have to use it for all static resources anyway so that we could get cache hits!
However, not only did I dislike this hack, but it still performed very slowly.
Okay, so, given that it was only a specific set of resources I needed to ignoreSearch on, I decided to take a different route;
just remove the parameters from the url requests manually, instead of relying on ignoreSearch.
self.addEventListener('fetch', function(event) {
//find urls that only have numbers as parameters
//yours will obviously differ, my queries to ignore were just repo revisions
var shaved = event.request.url.match(/^([^?]*)[?]\d+$/);
//extract the url without the query
shaved = shaved && shaved[1];
event.respondWith(
//try to get the url from the cache.
//if this is a resource, use the shaved url,
//otherwise use the original request
//(I assume it [can] contain post-data and stuff)
caches.match(shaved || event.request).then(function(response) {
//respond
return response || fetch(event.request);
})
);
});
I had the same issue, and previous approaches caused some errors with requests that should be ignoreSearch:false. An easy approach that worked for me was to simply apply ignoreSearch:true to a certain requests by using url.contains('A') && ... See example below:
self.addEventListener("fetch", function(event) {
var ignore
if(event.request.url.includes('A') && event.request.url.includes('B') && event.request.url.includes('C')){
ignore = true
}else{
ignore = false
}
event.respondWith(
caches.match(event.request,{
ignoreSearch:ignore,
})
.then(function(cached) {
...
}