I'm planning to use the HTML LocalStorage capability of browsers (FF especially) to store data of our intranet application.
I've read about it, how to use, the per domain isolation of data, how the user may increase it's quota, and so on.
But the question is if this quota is shared? Or does each domain has 5Mb to use?
(*) And, if I need to use, say 25Mb, and I ask my clients to increase their browser quota, does that means that all other domains may have this 25MB at disposal?
Or, if another domain uses all of the quota, is there no space left to my app?
UPDATE:
There's a (sub)question related to change browser config to allow more space (*). Althrough the below seems to be a recomendation, does anyone knows if is that implemented on FF?
11.3 Disk space
User agents should limit the total amount of space allowed for storage
areas, because hostile authors could otherwise use this feature to
exhaust the user's available disk space.
User agents should guard against sites storing data under their
origin's other affiliated sites, e.g. storing up to the limit in
a1.example.com, a2.example.com, a3.example.com, etc, circumventing the
main example.com storage limit.
User agents may prompt the user when quotas are reached, allowing the
user to grant a site more space. This enables sites to store many
user-created documents on the user's computer, for instance.
User agents should allow users to see how much space each domain is
using.
A mostly arbitrary limit of five megabytes per origin is suggested.
Implementation feedback is welcome and will be used to update this
suggestion in the future.
For predictability, quotas should be based on the uncompressed size of
data stored.
No, the quota is not shared across domains.
The reason for this is as you said - one domain might use up the browser's entire quota, and leave nothing for other sites.
To see how much storage is provided across different browsers, and to test your own, you can visit http://dev-test.nemikor.com/web-storage/support-test/
Related
Making localStorage and/or IndexedDB data offline permanent?
Is it possible to make localStorage and/or IndexedDB offline data permanent?
I am planning to make a completely offline HTML5 app and want the data to never get deleted, unless and otherwise the user knowingly does so.
I do not want the data to be deleted even after the app is closed, system is shutdown.
...want the data to never get deleted, unless and otherwise the user
knowingly does so.
The Storage API allow requesting a "persistent" permission for your origin. If granted, the semantics are that the data stored in certain APIs (local storage, Indexed DB, Cache, etc) is preserved until an explicit user action (like clearing storage), which appears to match your needs.
Documentation:
https://developer.mozilla.org/en-US/docs/Web/API/StorageManager/persist
Note that this is a new web platform API. It was recently added to Firefox (57) and Chrome (55), but is not present in Edge or Safari. Also note that Chrome does not currently show prompts for this permission request; it uses heuristics about the web app to determine if the request should be granted or not rather than asking the user to make the decision.
The specification is at: https://storage.spec.whatwg.org
The spec gives a background on how storage could work in browsers, i.e. by default an origin's data is "best effort", meaning preserved until the browser needs to clear it (due to space constraints, etc). The actual behavior of browsers varies significantly.
A few questions regard HTML5's sessionStorage:
Does the 5MB limit on localStorage include sessionStorage? (ie. is it really a 5MB limit on the WebStorage API)
If not does sessionStorage have a maximum size limit similar to localStorage?
I found this site http://dev-test.nemikor.com/web-storage/support-test/ in another SO questions, I'm wondering if the data is still relevant?
Does the 5MB limit on localStorage include sessionStorage? (Answer: NO)
is it really a 5MB limit on the WebStorage API (Answer: NO; Supported By )
does sessionStorage have a maximum size limit similar to localStorage? (Answer: NO; Supported By )
Hope I answered all your questions in a simplified manner. Here is a particularly informative section for you:
QUOTAS
You can imagine the chaos if any website was allowed to
populate unsuspecting hard drives with gigabytes of data! Thus,
browsers impose limits on storage capacity. When your app attempts to
exceed that limit, the browser will typically show a dialog to let the
user confirm the increase. You might expect the browser to enforce a
single limit for all storage an origin can use, but the major browsers
are actually enforcing limits separately for each storage mechanism.
This may change in the future, but for now, you should think of the
browser as maintaining a 2-D matrix, with "origin" in one dimension
and "storage" in the other. For example, "http://abc.example.com" is
allowed to store up to 5MB of Web Storage, 25MB of Web SQL Database
Storage, and forbidden to use Indexed Database. Another welcome
enhancement in this area would be user interfaces to let users view
and control how much space they have allocated for each origin. There
are also environments where the user can see upfront how much storage
will be used, e.g. in the case of the Chrome Web Store, when a user
installs an app, they will be prompted upfront to accept its
permissions, which include storage limits. One possible value is
"unlimited_storage".
The above text taken from http://www.html5rocks.com/en/tutorials/offline/storage/
I am learning about localStorage, and I understand that the limit is 5MB per domain. The first thought that crossed my mind is that an average web user visits hundred of websites regularly. If each one of these sites used their 5MB quota, the user could have multiple Gigabytes of storage on his disk dedicated to localStorage for his browser, which seems kind of crazy, especially for an ignorant user who might not be aware.
Now, I found my localStorage file in Firefox and it is only 2MB (and I've been recycling my user profile directory for years, so I expected it to be huge), so apparently either not many sites use it, or they clear it out often, or I don't know what. But hypothetically it seems like a lot more space could get used, and in some cases wasted.
Is this a valid concern? I couldn't find any discussion of this topic anywhere...
Whilst people may visit many websites daily, most of them would not be regular. Google, stack overflow, facebook, gmail, youtube, x newspaper and 9gag may be an average casual browsing session. Provided websites are using local storage responsibly, that is storing data for a valid purpose and only when a user "registers" to a site (or specifically requests local storage) rather than every passing visitor, then I think it's a non-issue. It should be the responsibility of the browser to keep an eye on the storage size, and if necessary prompt the user to clear infrequently used data. Web browsers have been caching images from every website you happen to have visited in the history of the universe for a while now, I wouldn't say we've brought about a massive storage crisis. I would also question the impact of reaching a gigabyte of local storage anyway, given even budget laptops these days come with half a terabyte of hard disk space.
How long can I expect data to be kept in localStorage. How long will an average user's localStorage data persist? If the user doesn't clear it, will it last till a browser re-install?
Is this consistent across browsers?
localStorage is also known as Web Storage, HTML5 Storage, and DOM Storage (these all mean the same thing).
localStorage.setItem('bob', varMyData);
sessionStorage.setItem('bob', varMyData);
localStorage is similar to sessionStorage, except that data stored in localStorage has no expiration time, while data stored in sessionStorage gets cleared when the browsing session ends (i.e. when the browser / browser tab is closed). (See Limitations section below for up-to-date storage size limitations.)
Session storage is used much less often than localStorage, and exists only within the current browser tab - even two tabs loaded with the same website will have different sessionStorage data. sessionStorage data survives page refresh, but not closing/opening the tab. LocalStorage data, on the other hand, is shared between all tabs and windows from the same origin. LocalStorage data does not expire; it remains after the browser is restarted and even after OS reboot.
localStorage is available on all browsers, but persistence is not consistently implemented. In particular, localStorage can be cleared by user action and may be cleared inadvertently (who would think that clearing all cookies also clears localStorage?).
In Firefox, localStorage is cleared when these three conditions are met: (a) user clears recent history, (b) cookies are selected to be cleared, (c) time range is "Everything" -- or when LocalStorage is full - see "Limitations" section below.
In Chrome, localStorage is cleared when these conditions are met: (a) clear browsing data, (b) "cookies and other site data" is selected, (c) timeframe is "from beginning of time" -- or when LocalStorage is full (see "Limitations" section below). In Chrome, it is also now possible to delete localStorage for one specific site.
In IE, to clear localStorage: (a) Tools--Internet Options, (b) General tab, (c) delete browsing history on exit, (d) ensure "Cookies and website data" (or "temporary internet files and website files") is selected, (e) consider unchecking "Preserve Favorites website data" at the top
In Safari: (a) Click Safari (b) Preferences (c) Select the Privacy tab (d) Click Remove all website data (e) Click Remove Now
Opera: Despite excellent articles on localStorage from the Opera site, I haven't yet found clear (non-programmatic) instructions to users on how to clear localStorage. If anyone finds, please leave a comment below this answer with reference link.
Limitations:
TOTAL localStorage is limited to 50% of free disk space.
ALSO, the localStorage for any one "origin" (domain + any subdomains) is (theoretically) limited to 20% of total localStorage - in practice, though, the localStorage for one domain (as of Oct/2022) is:
minimum: 10Mb
maximum: 2Gb Source
actual: 5Mb (limit on my system with 6Gb free space, per a modified version of this script )
(Test System: i7 / 32Gb / 500Gb SSD w 6Gb free / Brave browser:
Version 1.45.133 Chromium: 107.0.5304.141 (Official Build) (64-bit)
When the TOTAL localStorage is full, then the browser will start clearing out data (called "origin eviction") based on an LRU policy — the Least Recently Used domain will be deleted first, then the next one, until the browser is no longer over the limit.
Note that this origin eviction process will delete an entire domain's worth of data until the storage amount goes under the limit again. Deletion of a domain's localStorage data is "all-or-nothing" -- there is no trimming effect put in place to delete parts of origins (domains) because partial data could be much worse than no data.
The Opera dev site has an excellent summary of localStorage:
The current way of storing data on the client-side — cookies — is a
problem:
Low size: Cookies generally have a maximum size of around 4 KB, which
is not much good for storing any kind of complex data
It’s difficult for cookies to keep track of two or more transactions on the same
site, which might be happening in two or more different tabs
Cookies
can be exploited using techniques such as cross site scripting,
resulting in security breaches
Other (less popular) alternatives to
cookies include techniques involving query strings, hidden form
fields, flash based local shared objects, etc. Each with their own set
of problems related to security, ease of use, size restrictions etc.
So up until now we have been using pretty bad ways of storing data on
the user’s end. We need a better way, which is where Web Storage comes
in.
Web Storage
The W3C Web Storage specification was designed as a better way of
storing data on the client-side. It has two different types of
storage: Session Storage and Local Storage.
Both Session and Local Storage will typically be able to store around
5 MB of data per domain, which is significantly more than cookies. NOTE THAT although MDN's numbers were updated (Oct 2022) and now say: minimum: 10Mb / Maximum: 2Gb, this author is unable to exceed 5Mb per domain/origin. M.D.N. / test script
Resources:
https://developer.mozilla.org/en-US/docs/Web/API/Window/sessionStorage
MDN - Browser_storage_limits_and_eviction_criteria
https://javascript.info/localstorage
https://dev.opera.com/articles/web-storage/
http://www.quirksmode.org/html5/storage.html
http://www.ghacks.net/2015/02/05/how-to-clear-web-storage-in-your-browser-of-choice/
https://nakedsecurity.sophos.com/2014/11/05/how-to-clear-out-cookies-flash-cookies-and-local-storage/
http://www.opera.com/dragonfly/documentation/storage/
DOMStorage article on MDN (written by John Resig)
http://ejohn.org/blog/dom-storage/
W3C draft says this
User agents should expire data from the local storage areas only for security reasons or when requested to do so by the user. User agents should always avoid deleting data while a script that could access that data is running.
So if browsers follow the spec it should persist untill the user removes it on all browsers, I have not found any that have deleted on any off my projects.
A good article to read is also http://ejohn.org/blog/dom-storage/
Duration
Unlimited. The data persists through browser & OS restarts.
Capacity
Each domain can store minimum of 5MB of data in LocalStorage.
For some browsers you can store up to 1GB of data.
In Chrome while performing 'clear browsing data' , if you choose 'Cookies and other site and plugin data' option then sessionStorage data will be erased.
The content in localstorage is persistent as long as the user chooses to clear the storage (entirely or a single value inside it)
About the consistency across browser, localstorage is currently available on every major browser, including IE8+ (see http://caniuse.com/#feat=namevalue-storage)
It is impossible to identify a user or request as unique since duping is trivial.
However, there are a handful of methods that, combined, can hamper cheating attempts and give a user quasi-unique status.
I know of the following:
IP Address - store the IP address of each visitor in a database of some sort
Can be faked
Multiple computers/users can have the same address
Users with dynamic IP addresses (some ISP issue them)
Cookie tracking - store a cookie per visitor. Visitors that don't have it are considered "unique"
Can be faked
Cookies can be blocked or cleared via browser
Are there more ways to track non-authorized (non-login, non-authentication) website visitors?
There are actually many ways you can detect a "unique" user. Many of these methods are used by our marketing friends. It get's even easier when you have plugins enabled such as Java, Flash etc.
Currently my favorite presentation of cookie based tracking is evercookie (http://samy.pl/evercookie/). It creates a "permanent" cookie via multiple storage mechanisms, the average user is not able to flush, specifically it uses:
Standard HTTP Cookies
Local Shared Objects (Flash Cookies)
Silverlight Isolated Storage
Storing cookies in RGB values of
auto-generated, force-cached PNGs
using HTML5 Canvas tag to read pixels
(cookies) back out
Storing cookies in Web History
Storing cookies in HTTP ETags
Storing cookies in Web cache
window.name caching
Internet Explorer userData storage
HTML5 Session Storage
HTML5 Local Storage
HTML5 Global Storage
HTML5 Database Storage via SQLite
I can't remember the URL, but there is also a site which tells you how "anonymous" you are based on everything it can gather from your web browser: What plugins you have loaded, what version, what language, screensize, ... Then you can leverage the plugins I was talking about earlier (Flash, Java, ...) to find out even more about the user. I'll edit this post when I find the page whcih showed you "how unique you are" or maybe somebody knows »» actually it looks as if every user is in a way unique!
--EDIT--
Found the page I was talking about: Panopticlick - "How Unique and trackable is your browser".
It collects stuff like User Agent, HTTP_ACCEPT headers, Browser Plugins, Time Zone, Screen Size and Depth, System Fonts (via Java?), Cookies...
My result: Your browser fingerprint appears to be unique among the 1,221,154 tested so far.
Panopticlick has a quite refined method for checking for unique users using fingerprinting. Apart from IP-adress and user-agent it used things like timezone, screen resolution, fonts installed on the system and plugins installed in the browser etc, so it comes up with a very distinct ID for each and every user without storing anything in their computers. False negatives (finding two different users with the exact same fingerprint) are very rare.
A problem with that approach is that it can yield some false positive, i.e. it considers the same user to be a new one if they've installed a new font for example. If this is ok or not depends on your application I suppose.
Yes, it's impossible to tell anonymous visitors apart with 100% certainty. The best that you can do is to gather the information that you have, and try to tell as many visitors apart as you can.
There is one more piece of infomration that you can use:
Browser string
It's not unique, but in combination with the other information it increases the resolution.
If you need to tell the visitors apart with 100% certainty, then you need to make them log in.
There is no sure-fire way to achieve this, in my view. Of your options, cookies are the most likely to yield a reasonably realistic number. NATing and proxy servers can mask the IP addresses of a large number of users, and dynamic IP address allocation will confuse the results for a lot of others
Have you considered using e.g Google Analytics or similar? They do unique visitor tracking as part of their service, and they probably have a lot more money to throw at finding heuristic solutions to this problem than you or I. Just a thought!