In Windows and Android Google Chrome browser, (haven't tested for others yet) response time from a service worker increases linearly to number of items stored in that specific cache storage when you use Cache.match() function with following option;
ignoreSearch = true
Dividing items in multiple caches helps but not always convenient to do so. Plus even a small amount of increase in items stored makes a lot of difference in response times. According to my measurements response time is roughly doubled for every tenfold increase in number of items in the cache.
Official answer to my question in chromium issue tracker reveals that the problem is a known performance issue with Cache Storage implementation in Chrome which only happens when you use Cache.match() with ignoreSearch parameter set to true.
As you might know ignoreSearch is used to disregard query parameters in URL while matching the request against responses in cache. Quote from MDN:
...whether to ignore the query string in the url. For example, if set to
true the ?value=bar part of http://example.com/?value=bar would be ignored
when performing a match.
Since it is not really convenient to stop using query parameter match, I have come up with following workaround, and I am posting it here in hopes of it will save time for someone;
// if the request has query parameters, `hasQuery` will be set to `true`
var hasQuery = event.request.url.indexOf('?') != -1;
event.respondWith(
caches.match(event.request, {
// ignore query section of the URL based on our variable
ignoreSearch: hasQuery,
})
.then(function(response) {
// handle the response
})
);
This works great because it handles every request with a query parameter correctly while handling others still at lightning speed. And you do not have to change anything else in your application.
According to the guy in that bug report, the issue was tied to the number of items in a cache. I made a solution and took it to the extreme, giving each resource its own cache:
var cachedUrls = [
/* CACHE INJECT FROM GULP */
];
//update the cache
//don't worry StackOverflow, I call this only when the site tells the SW to update
function fetchCache() {
return Promise.all(
//for all urls
cachedUrls.map(function(url) {
//add a cache
return caches.open('resource:'url).then(function(cache) {
//add the url
return cache.add(url);
});
});
);
}
In the project we have here, there are static resources served with high cache expirations set, and we use query parameters (repository revision numbers, injected into the html) only as a way to manage the [browser] cache.
It didn't really work to use your solution to selectively use ignoreSearch, since we'd have to use it for all static resources anyway so that we could get cache hits!
However, not only did I dislike this hack, but it still performed very slowly.
Okay, so, given that it was only a specific set of resources I needed to ignoreSearch on, I decided to take a different route;
just remove the parameters from the url requests manually, instead of relying on ignoreSearch.
self.addEventListener('fetch', function(event) {
//find urls that only have numbers as parameters
//yours will obviously differ, my queries to ignore were just repo revisions
var shaved = event.request.url.match(/^([^?]*)[?]\d+$/);
//extract the url without the query
shaved = shaved && shaved[1];
event.respondWith(
//try to get the url from the cache.
//if this is a resource, use the shaved url,
//otherwise use the original request
//(I assume it [can] contain post-data and stuff)
caches.match(shaved || event.request).then(function(response) {
//respond
return response || fetch(event.request);
})
);
});
I had the same issue, and previous approaches caused some errors with requests that should be ignoreSearch:false. An easy approach that worked for me was to simply apply ignoreSearch:true to a certain requests by using url.contains('A') && ... See example below:
self.addEventListener("fetch", function(event) {
var ignore
if(event.request.url.includes('A') && event.request.url.includes('B') && event.request.url.includes('C')){
ignore = true
}else{
ignore = false
}
event.respondWith(
caches.match(event.request,{
ignoreSearch:ignore,
})
.then(function(cached) {
...
}
Related
I try to sync changes from GDrive, So I'm using the changes api. It works like a charm when I'm using it with restrictToMyDrive=true.
But when I tried to expand it to also track shared files restrictToMyDrive=false I encounter some major drawback. I have inconsist results - the parents field is missing sporadically.
Let say that we have user A that share that folder to user B:
rootSharedFolder => subFolder => subSubFolder => File
If user B calls the changes API relatively close to the time that user A share the rootSharedFolder , Then in 3/10 times some of the inner folder will be received without the parents field.
Even trying to use the files.get API on the received changed item , results in empty parents field. But if I wait a minute or two and then call it again , the parents field does exist in the result.
Anyone else encounter this problem , and maybe have a workaround for it?
Thanks!
this behavior happens only when calling the changes api close to the time that the other user share you the items
Parents field may be missing due two factors:
A propagation delay
Insufficient permissions to see the parents
In both cases you'll notice the same: the parents field for the file is missing, at first there is no way to tell in which case you are:
A propagation issue:
This may happen when you request some file details related to a shared file not owned by you right after it was shared to you.
You may not be able to find it's related parents at first glance, this is because changes are still propagating on the file system, this is call a propagation issue, it should not last long and it should be possible to identify and solve this inconvenience by retrieving this field data a couple of minutes after the permission changes.
Not having access to the parents:
In this case you may have access to a certain file, but not to it's parent folder, thus, you can not get to know what parent it has, because it's not been shared with you.
This is on the documentation:
parents — A parent does not appear in the parents list if the requesting user is a not a member of the shared drive and does not have access to the parent. In addition, with the exception of the top level folder, the parents list must contain exactly one item if the file is located within a shared drive.
Side note: you may be interested on using SharedDrives, where files are owned by a organization rather than individual users, simplifying the sharing process and maybe avoiding the problems you are facing here.
https://developers.google.com/drive/api/v3/enable-shareddrives
How to know which case is?
A way to go is to implement an exponential back-off algorithm to try to retrieve the missing parents field, if after a max number of attempts it does not retrieve we are probably on the second case:
exponentialBackoff(getParent, 7, 300, function(result) {
console.log('the result is',result);
});
// A function that keeps trying, "getParent" until it returns true or has
// tried "max" number of times. First retry has a delay of "delay".
// "callback" is called upon success.
function exponentialBackoff(toTry, max, delay, callback) {
console.log('max',max,'next delay',delay);
var result = toTry();
if (result) {
callback(result);
} else {
if (max > 0) {
setTimeout(function() {
exponentialBackoff(toTry, --max, delay * 2, callback);
}, delay);
} else {
console.log('we give up');
}
}
}
function getParent() {
var percentFail = 0.8;
return Math.random() >= 0.8;
}
I've added a 'product' content type to my Bolt installation. I want to track and update inventory when a purchase is made, so I added an integer field 'available'. I set up a test controller to modify the record. Everything appears to work, but the update never happens. What am I missing?
<?php
namespace Bundle\Site;
class CartController extends \Bolt\Controller\Base
{
public function addRoutes(\Silex\ControllerCollection $c)
{
$c->match('/test-save', [$this,'testSave']);
return $c;
}
public function testSave()
{
$result=false;
$repo = $this->app['storage']->getRepository('products');
$content = $repo->find(1);
//Error log output confirms that this is the correct record
error_log(get_class($this).'::'.__FUNCTION__.': '.json_encode($content));
$content->set('available',15);
$content->setDatechanged('now');
$result=$repo->save($content); //returns 1
return new \Symfony\Component\HttpFoundation\Response(json_encode($result), \Symfony\Component\HttpFoundation\Response::HTTP_OK);
}
}
It turns out that my code works. I was checking the result by reloading the back-end edit form, which had apparently cached its data, so I didn't see the value changing.
The same thing happens if I load the edit form in two browser tabs and update one, then refresh the other, so it looks like this is the best I can do.
I have faced a similar problem. Either the updated codes didn't show up or the result in the output is unchanged. Figured though this happened due to setting debug: false in the config.yml file. When debug is true your in bolt-data/app/ all cache files increases but this thing get sorted.
In production it is suggested that to keep debug: false but this problem checks-in too, hence the way you are dealing with it comes out to be the easiest way out without changing the configuration much.
I attempted a fix on ConfirmAccount extension, but my fix did not seem to work. Any suggestions?
Details:
"pruning of old requests will not trigger often, so old rejected requests may persist."
https://www.mediawiki.org/wiki/Extension:ConfirmAccount#Known_issues
This behavior will prevent rejected emails from requesting an account again. After Admin rejected an account request, the same username/email could not submit another request. Error on 2nd attempt:
Username is already in use in a pending account request.
We want to enable re-requests. To fix, i want to force prune after every rejection, to clear the request cache.
It appears that, currently, pruning occurs in file \extensions\ConfirmAccount\frontend\specialpages\actions\ConfirmAccount_body.php
# Every 30th view, prune old deleted items
if ( 0 == mt_rand( 0, 29 ) ) {
ConfirmAccount::runAutoMaintenance();
}
Therefor, the function runAutoMaintenance appears to be the pruning function. runAutoMaintenance lives in \ConfirmAccount\backend\ConfirmAccount.class.php
class ConfirmAccount {
/** * Move old stale requests to rejected list. Delete old rejected requests. */
public static function runAutoMaintenance() {
...
In order to call runAutoMaintenance after every reject-action, I think the call to runAutoMaintenance should be placed in function rejectRequest, in file \extensions\ConfirmAccount\business\AccountConfirmSubmission.php
Specifically, i think it can go directly under:
# Clear cache for notice of how many account requests there are
ConfirmAccount::clearAccountRequestCountCache();
Maybe pruning should also happen after Accept, Hold, and Spam actions. Unsure. For now, pruning after Reject should handle the original problem.
I attempted the above fix, and it did not seem to work. I'm at a loss.
Can someone help determine why this fix did not work?
Original code:
protected function rejectRequest( IContextSource $context ) {
....
# Clear cache for notice of how many account requests there are
ConfirmAccount::clearAccountRequestCountCache();
....
New code:
protected function rejectRequest( IContextSource $context ) {
....
# Clear cache for notice of how many account requests there are
ConfirmAccount::clearAccountRequestCountCache();
# Prune
ConfirmAccount::runAutoMaintenance();
....
On 2nd request, still getting "Username is already in use in a pending account request."
Looks like i solved it. There are 2 steps:
Set rejected items to expire immediately (in LocalSettings.php)
Prune rejected items at beginning of Request action (in RequestAccount_body.php)
Details:
in LocalSettings.php, after required declaration, set Rejected-Age to 0. That ensures rejected requests will be removed on prune-action:
require_once "$IP/extensions/ConfirmAccount/ConfirmAccount.php";
$wgRejectedAccountMaxAge = 0;
Add Prune code to the function that shows the Request form, in /ConfirmAccount/frontend/specialpages/actions/RequestAccount_body.php, function showForm. Add very last command in the function:
old code:
$out->addWikiMsg( 'requestaccount-footer' );
}
new code:
$out->addWikiMsg( 'requestaccount-footer' );
# PRUNE
ConfirmAccount::runAutoMaintenance();
}
When i write view in couchbase to return whole doc it also return _sync":{} data, is there any way to remove this data from response.
here is my view function:-
function map(doc, meta) {
if (doc.type == 'user' && doc.user_id) {
emit(doc.user_id, doc);
}
}
first off, you shouldn't ever have to emit the whole doc. This makes the index bigger on disk, and it's redudant since you can get the whole doc easily from a view row (the doc id is always included, and the SDKs will transparently fetch it for you usually).
in your case though, you may actually need that second part of emit. Select the attributes you are interested in and emit them in an array (like emit(doc.user_id, [doc.attributeA, doc.attributeB])), thus effectively "filtering out" _sync.
only problem is that if later on you add an attributeC to your users, it won't automatically be included (so it filters out attributes not explicitly listed in the map function). does that make sense?
I like the user experience of cubism, and would like to use this on top of a backend we have.
I've read the API doc's and some of the code, most of this seems to be extracted away. How could I begin to use other data sources exactly?
I have a data store of about 6k individual machines with 5 minute precision on around 100 or so stats.
I would like to query some web app with a specific identifier for that machine and then render a dashboard similar to cubism via querying a specific mongo data store.
Writing the webapp or the querying to mongo isn't the issue.
The issue is more in line with the fact that cubism seems to require querying whatever data store you use for each individual data point (say you have 100 stats across a window of a week...expensive).
Is there another way I could leverage this tool to look at data that gets loaded using something similar to the code below?
var data = [];
d3.json("/initial", function(json) { data.concat(json); });
d3.json("/update", function(json) { data.push(json); });
Cubism takes care of initialization and update for you: the initial request is the full visible window (start to stop, typically 1,440 data points), while subsequent requests are only for a few most recent metrics (7 data points).
Take a look at context.metric for how to implement a new data source. The simplest possible implementation is like this:
var foo = context.metric(function(start, stop, step, callback) {
d3.json("/data", function(data) {
if (!data) return callback(new Error("unable to load data"));
callback(null, data);
});
});
You would extend this to change the "/data" URL as appropriate, passing in the start, stop and step times, and whatever else you want to use to identify a metric. For example, both Cube and Graphite use a metric expression as an additional query parameter.