With the (deprecated, Chrome-proprietary) navigator.webkitPersistentStorage.requestQuota() method I can request persistent storage in Chrome.
Does this also apply to IndexedDB data or only for the filesystem API?
Background: I want to make IndexedDB really persistent, make it immune to any browser garbage collection / eviction (except manual user deletion) in order to save important data.
No.
Chrome's old PersistentStorage mechanism only applies to the FileSystem API. The intent was to apply it to all storage types but the proposal fizzled and was not adopted by other browsers. (Like the FileSystem API itself.)
...
There's a newer proposal at: https://storage.spec.whatwg.org/
... whereby script can call navigator.storage.persist() to request the "persistent" permission. If granted, it makes the entire origin's storage (Indexed DB, Cache API, localStorage, etc) persistent, i.e. can only be cleared by explicit user action. Chrome is currently (circa August 2016) experimenting with this proposal - if you'd like to try it out, see: https://developers.google.com/web/updates/2016/06/persistent-storage?hl=en
Related
Making localStorage and/or IndexedDB data offline permanent?
Is it possible to make localStorage and/or IndexedDB offline data permanent?
I am planning to make a completely offline HTML5 app and want the data to never get deleted, unless and otherwise the user knowingly does so.
I do not want the data to be deleted even after the app is closed, system is shutdown.
...want the data to never get deleted, unless and otherwise the user
knowingly does so.
The Storage API allow requesting a "persistent" permission for your origin. If granted, the semantics are that the data stored in certain APIs (local storage, Indexed DB, Cache, etc) is preserved until an explicit user action (like clearing storage), which appears to match your needs.
Documentation:
https://developer.mozilla.org/en-US/docs/Web/API/StorageManager/persist
Note that this is a new web platform API. It was recently added to Firefox (57) and Chrome (55), but is not present in Edge or Safari. Also note that Chrome does not currently show prompts for this permission request; it uses heuristics about the web app to determine if the request should be granted or not rather than asking the user to make the decision.
The specification is at: https://storage.spec.whatwg.org
The spec gives a background on how storage could work in browsers, i.e. by default an origin's data is "best effort", meaning preserved until the browser needs to clear it (due to space constraints, etc). The actual behavior of browsers varies significantly.
How long can I expect data to be kept in localStorage. How long will an average user's localStorage data persist? If the user doesn't clear it, will it last till a browser re-install?
Is this consistent across browsers?
localStorage is also known as Web Storage, HTML5 Storage, and DOM Storage (these all mean the same thing).
localStorage.setItem('bob', varMyData);
sessionStorage.setItem('bob', varMyData);
localStorage is similar to sessionStorage, except that data stored in localStorage has no expiration time, while data stored in sessionStorage gets cleared when the browsing session ends (i.e. when the browser / browser tab is closed). (See Limitations section below for up-to-date storage size limitations.)
Session storage is used much less often than localStorage, and exists only within the current browser tab - even two tabs loaded with the same website will have different sessionStorage data. sessionStorage data survives page refresh, but not closing/opening the tab. LocalStorage data, on the other hand, is shared between all tabs and windows from the same origin. LocalStorage data does not expire; it remains after the browser is restarted and even after OS reboot.
localStorage is available on all browsers, but persistence is not consistently implemented. In particular, localStorage can be cleared by user action and may be cleared inadvertently (who would think that clearing all cookies also clears localStorage?).
In Firefox, localStorage is cleared when these three conditions are met: (a) user clears recent history, (b) cookies are selected to be cleared, (c) time range is "Everything" -- or when LocalStorage is full - see "Limitations" section below.
In Chrome, localStorage is cleared when these conditions are met: (a) clear browsing data, (b) "cookies and other site data" is selected, (c) timeframe is "from beginning of time" -- or when LocalStorage is full (see "Limitations" section below). In Chrome, it is also now possible to delete localStorage for one specific site.
In IE, to clear localStorage: (a) Tools--Internet Options, (b) General tab, (c) delete browsing history on exit, (d) ensure "Cookies and website data" (or "temporary internet files and website files") is selected, (e) consider unchecking "Preserve Favorites website data" at the top
In Safari: (a) Click Safari (b) Preferences (c) Select the Privacy tab (d) Click Remove all website data (e) Click Remove Now
Opera: Despite excellent articles on localStorage from the Opera site, I haven't yet found clear (non-programmatic) instructions to users on how to clear localStorage. If anyone finds, please leave a comment below this answer with reference link.
Limitations:
TOTAL localStorage is limited to 50% of free disk space.
ALSO, the localStorage for any one "origin" (domain + any subdomains) is (theoretically) limited to 20% of total localStorage - in practice, though, the localStorage for one domain (as of Oct/2022) is:
minimum: 10Mb
maximum: 2Gb Source
actual: 5Mb (limit on my system with 6Gb free space, per a modified version of this script )
(Test System: i7 / 32Gb / 500Gb SSD w 6Gb free / Brave browser:
Version 1.45.133 Chromium: 107.0.5304.141 (Official Build) (64-bit)
When the TOTAL localStorage is full, then the browser will start clearing out data (called "origin eviction") based on an LRU policy — the Least Recently Used domain will be deleted first, then the next one, until the browser is no longer over the limit.
Note that this origin eviction process will delete an entire domain's worth of data until the storage amount goes under the limit again. Deletion of a domain's localStorage data is "all-or-nothing" -- there is no trimming effect put in place to delete parts of origins (domains) because partial data could be much worse than no data.
The Opera dev site has an excellent summary of localStorage:
The current way of storing data on the client-side — cookies — is a
problem:
Low size: Cookies generally have a maximum size of around 4 KB, which
is not much good for storing any kind of complex data
It’s difficult for cookies to keep track of two or more transactions on the same
site, which might be happening in two or more different tabs
Cookies
can be exploited using techniques such as cross site scripting,
resulting in security breaches
Other (less popular) alternatives to
cookies include techniques involving query strings, hidden form
fields, flash based local shared objects, etc. Each with their own set
of problems related to security, ease of use, size restrictions etc.
So up until now we have been using pretty bad ways of storing data on
the user’s end. We need a better way, which is where Web Storage comes
in.
Web Storage
The W3C Web Storage specification was designed as a better way of
storing data on the client-side. It has two different types of
storage: Session Storage and Local Storage.
Both Session and Local Storage will typically be able to store around
5 MB of data per domain, which is significantly more than cookies. NOTE THAT although MDN's numbers were updated (Oct 2022) and now say: minimum: 10Mb / Maximum: 2Gb, this author is unable to exceed 5Mb per domain/origin. M.D.N. / test script
Resources:
https://developer.mozilla.org/en-US/docs/Web/API/Window/sessionStorage
MDN - Browser_storage_limits_and_eviction_criteria
https://javascript.info/localstorage
https://dev.opera.com/articles/web-storage/
http://www.quirksmode.org/html5/storage.html
http://www.ghacks.net/2015/02/05/how-to-clear-web-storage-in-your-browser-of-choice/
https://nakedsecurity.sophos.com/2014/11/05/how-to-clear-out-cookies-flash-cookies-and-local-storage/
http://www.opera.com/dragonfly/documentation/storage/
DOMStorage article on MDN (written by John Resig)
http://ejohn.org/blog/dom-storage/
W3C draft says this
User agents should expire data from the local storage areas only for security reasons or when requested to do so by the user. User agents should always avoid deleting data while a script that could access that data is running.
So if browsers follow the spec it should persist untill the user removes it on all browsers, I have not found any that have deleted on any off my projects.
A good article to read is also http://ejohn.org/blog/dom-storage/
Duration
Unlimited. The data persists through browser & OS restarts.
Capacity
Each domain can store minimum of 5MB of data in LocalStorage.
For some browsers you can store up to 1GB of data.
In Chrome while performing 'clear browsing data' , if you choose 'Cookies and other site and plugin data' option then sessionStorage data will be erased.
The content in localstorage is persistent as long as the user chooses to clear the storage (entirely or a single value inside it)
About the consistency across browser, localstorage is currently available on every major browser, including IE8+ (see http://caniuse.com/#feat=namevalue-storage)
How do I setup the basic switching of offline storage modes (offline/online) in Web SQL? I know there's javascript
window.navigator.onLine. I can check the mode and then go through a process...
//All GET/POST performed with AJAX
//On Startup pulldown entire accessible database into offline storage (Doesn't seem secure IMO)
//if(read) pull from offline
//if(create, update, delete and online) pull from standard db, mark changes with offline expiration flag
//if(create, update, delete and offline) perform operation on offline storage, persist with POST when next online (change flag)
I'm asking if there is any OOB integration for these standard tasks?
The navigator.online property generally isn't very useful - in a desktop browser all it does is hook into the File -> Work Offline menu. It may be more useful on an iPad, I don't know because I don't have one, and I'm guessing there's not a File menu, but I would recommend you test.
A common approach to this issue is to set up two easily distinguishable files in the fallback section of your manifest. Every time you want to connect back to the server attempt to fetch the file with AJAX and, in the callback, check it to see if you got the online file or the fallback, then branch accordingly.
You shouldn't be using Web SQL as that spec was nixed a new months ago. You should be using Localstorage. Unless you are specifically coding for something like the iphone, but even then you dont know how long the spec will be in webkit.
I have an extension where users maintain a list of links. It would be nice to have this data synchronized between computers (at work and at home). What are the possible solutions?
Chrome has extension synchronization option but I am not sure if it synchronizes data or not (I would be surprised if yes). Even if it does, not everyone would want all their other extensions be synced.
I can store my links in a special bookmark folder and use built-in bookmark synchronization, but in this case all bookmarks would be synchronized too (not all users would want that either I think).
Any external sites I can use? Something easy to use and linked to a google account?
(I don't want to build my own site for this)
Edit: As of Chrome 20 and above you can use chrome.storage module to save to the cloud.
chrome.experimental.storage.sync.set({'settingAlwaysOn': true}, function() {
console.log('Saved option in the cloud');
});
Before Chrome 20
You're right, the Chrome Sync for extensions options (in settings) does not synchronize extension data. The only way to synchronize those data is through a third party.
Since you ruled out the usage of Bookmarks, which makes sense if users don't want bookmarks to be synchronized.
Everytime you persist data through storage (Web SQL Storage, localStorage, IndexDB), you grab that object, and serialize it into JSON (via JSON.stringify), and you send it to some online service such as Google Docs.
That would be quite tricky for Web SQL Storage and IndexDB, you would have to do your own importer and exporter. For localStorage it is pretty simple, since its a key/value pair.
It requires some work to link it to a Google Account (such as Docs) you would have to use OAuth and do the plumbing to connect your extension to the service. Once your connected, it is not that difficult to maintain the state.
Good luck :)
Chrome 20 supports chrome.storage.sync API. It seems to fit your requirements perfectly.
For storing data offline WebApp can use:
session storage, "advanced version of cookies"
key/value based Web Storage (AKA local/global/offline/DOM storage)
sql-based Web SQL Database (deprecated) and Indexed Database API
FileReader and FileWriter API (requires user to select files each time the application loads)
But apparently there is no File Storage. Of course, there is a manifest-based caching, but it's just a cache and is not supposed to be used as a user data storage.
Does it mean that the user of WebApp is forced to use some sort of a cloud file storage?
Is there any way to save large files on user's local machine? Or maybe some way to select a local folder web application can use to store user data?
Edit. Security. HTML5 already has the ability to write big portions of data to user's local machine. I don't see any security issues if a browser will provide another, file-based abstraction to store data. It can be some virtual machine, virtual filesystem, whatever.
Hm, I think, it is possible to write JS filesystem and store it as a blob in SQL...
Similar questions.
Update:
Hm... recently I've found this and this. Maybe it is what I'm looking for... Yes, it is! See the answer below.
At last, I've found it! Here's the answer:
I’ll have the DOMFileSystem with a side of read/write access please wrote:
Eric Uhrhane of Google has been
working on the working draft of the
File API: Directories and System specification which defines a set of
APIs to create a sandboxed filesystem
where a web app can read and write
data to.
Wow! I'm so excited!
Why not use localStorage while the user is editing a document and the FileWriter API when they want to save it to disk? Most people are used to seeing a save dialog pop up when saving a document.
The only scenario I can think of that warrants userless access to the FileWriter API is an autosave feature, but autosaving to localStorage can be just as good.
There is a way to save relatively large files to a users hard drive if you are willing to use Flash. Look into Downloadify:
http://www.bitrepository.com/downloadify-client-side-file-generation.html
Downloadify allows you to send data to a SWF and have that SWF create a file on the users machine. My recommendation would be to store the data via one of the methods you listed, Webstorage, sqlite database, etc. Put all your assets, including the SWF in the manifest file so everything is cached locally to the browser. You can then pull information from your db or webstorage and use the SWF to create the files you need.
I'm not sure if you will be able to read these files back into your web application.
Another option to save data is by using link tags with the data URI scheme. However, I'm not sure if it is supported in all the major browsers at the moment.
For security reasons you can't write files to a user's local filesystem in case it gets used for nefarious purposes by evil people.
That's not likely to change, and that's a good thing.
The HTML5 FileSystem API started landing in Chrome 8 and is fairly complete as of now (Chrome 11).
There's a nice tutorial on it here: http://www.html5rocks.com/tutorials/file/filesystem/
http://fsojs.com wraps the FileSystem API effectively, if you want an easy solution
As mentioned by others here, the FileWriter and FileSystem APIs can be used to store files on a client's machine from the context of a browser tab/window.
However, there are several things pertaining to these APIs which you should be aware of:
Implementations of the APIs currently exist only in Chromium-based browsers (Chrome & Opera)
Both of the APIs were taken off of the W3C standards track on April 24, 2014, and as of now are proprietary
Removal of the (now proprietary) APIs from implementing browsers in the future is a possibility
A sandbox (a location on disk outside of which files can produce no effect) is used to store the files created with the APIs
A virtual file system (a directory structure which does not necessarily exist on disk in the same form that it does when accessed from within the browser) is used represent the files created with the APIs
IsolatedStorage, which hasn't been mentioned as of yet, also allows for file i/o from a tab/window context, but it is made available through solely through Silverlight and requires the use of managed code to access. It, like FileSystem, also exists in a sandbox and makes use of a virtual file system.
Given the high market penetration of both Chromium-based browsers and Silverlight (support for which, interestingly enough has been dropped by such browsers), you may find a solution which uses the first of the above approaches available on a client machine satisfactory.
BakedGoods, a Javascript library that establishes a uniform interface that can be used to conduct common storage operations in all native (including FileSystem), and some non-native (including IsolatedStorage) storage facilities, is an example of such a solution:
//Write file to first of either FileSystem or IsolatedStorage
bakedGoods.set({
data: [{key: "testFile", value: "Hello world!", dataFormat: "text/plain"}],
storageTypes: ["fileSystem", "silverlight"],
options: {fileSystem:{storageType: Window.PERSISTENT}},
complete: function(byStorageTypeStoredItemRangeDataObj, byStorageTypeErrorObj){}
});
Just for the sake of complete transparency, BakedGoods is maintained by this guy right here :) .