I am trying to store an array in localStorage, It's working for 3000 records, but when records increases to 5-10 thousand code breaks.
Is there is anything so I can store huge data in localStorage.
LocalStorage has size limits that vary depending on the browser. This is to prevent malicious scripts from filling a user's hard drive.
You can test your browser's localStorage limits here: https://arty.name/localstorage.html
The simple answer is, you shouldn't try to store more than 5MB-10MB of data on the client, depending on the browser. Needing to store that much local data is a sign that you probably need to come up with a better solution.
One other possibility for storing data locally is IndexedDB, which has reasonable compatibility across modern browsers. It's a object store which acts a lot like document databases such as MongoDB. You can store objects without converting them to strings and you can query those objects the way you would a database.
Most browsers seem to have a "soft" limit of around 5MB on IndexedDB storage. It's a soft limit because it's not necessarily enforced so you can go store much more if the browser allows it. Your mileage may vary.
Max size for localstorage is 5MB for preventing malicious scripts from filling a user's hard drive. You can go for IndexedDB which is compatible with all modern browsers. The minimum or soft limit is 5MB- the browser will ask for permission to store the data. Maximum storage is the limit of your hardrive disk, as all of the data is stored locally on your machine disk. Basically if you have 20GB free storage than you can use all of the storage for IndexedDB.
Related
The main purpose is to store data locally so it can be accessed without internet connection.
In my React application I will need to fetch JSON data (such as images, text and videos) from the internet and display it for a certain amount of time.
To add flexibility, this should work offline as well.
I've read about options such as localStorage and Firebase but all of them so far require either access to the Internet, or are limited to 10Mb which is too low for what I'll need.
What would be my best option to persist data in some sort of offline
database or file trough react?
I'd also be thankful if you could point me to a good tutorial about
any provided solution.
To store large amounts of data on client side you can use indexedDB.
IndexedDB is a low-level API for client-side storage of significant amounts of structured data, including files/blobs.
You can read more about indexedDB api here
I'm looking to store a video in a database as a BLOB type.
The video will be playing back using progressive downloading.
Are there any problems that I might face?
Using the html5 video player it would look like this:
<video width="800" height="450" src=BLOB type=""/></video>
By storing video files in the database it means your data-tier is invoked every time there's a request for the video which will be very expensive from an IO perspective, also given that (AFAIK) most database libraries don't let you access binary information from an offset it means you won't be able to provide seek support to consumers (they'd all have to download the video fully first).
Storing video-files in a database can be acceptable depending on the load (how many requests will be made for this video?), if you're employing any form of caching so it doesn't have to hit the database so often (e.g. storing the whole video in memcached), how big the videos are (if they're just 6-second Vine-style videos then you'll get away with it, if these are 20GB 4K hi-def documentary videos then probably not).
Out of curiosity, why not use a third-party video hosting service like YouTube or use a CDN to host your video files?
Some downside of storing Blobs
When your DB grows bigger and bigger it will become harder to backup. Restoring a backup of a table with over 100 GB of data is not something that makes you happy.
Another thing that get is that all the table management functions get slower and slower as the dataset grows.
But this can be overcome by making your data table just contain 2 fields: ID and BLOB.
Retrieving data (by primary key) will likely only become a problem long after you hit a wall with backing up the dataset.
We had similar situation in pas project where we saved video and audio on disk and researched if there are any benefits to keep them in database. What we've found is that file system is easiest and primitive solution, whenever you have complex requirement your choice is database, but keep in mind these:
Saving on disk offers simplicity, i.e. you can later change the structure of folders and move files between them relatively easy. Typically you would want to distinguish between audio and video files, corrupted files, formats of files, group files by date, etc. and keep those in sub-folders. So whenever you need more folders you can easily add them
Think in advance about retain policies when for example, newest media you would keep for 30 days, then move into different disk or whatever. You need clear understanding how you would achieve this with database
CDN. It was really easy to spread out files through servers. With database it depends on your scenario and might be easier or more challenging
Database will add overhead as you can not just access the file through URI and need some middle-ware code that will communicate to db
Database will be quite helpful if you need analysis of data or Business Intelligence support, transactions and indexing
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
database for huge files like audio and video
I'm seeking for the best (or at least good enough) way of storing large sets of binary data (images, videos, documents, etc.). The solution has to be scalable and can't get stuck after X amount of data.
I would like to have a one place for example MySQL database where all the data is kept. When one of web front ends needs it (on request) It can acquire it from the the DB and cache it permanently for later.
From this what I can see on http://dev.mysql.com/doc/refman/5.0/en/table-size-limit.html MySQL table can't store more then 4TB per table. Is there something more appropriate like perhaps nosql databases or perhaps it's better to store everything in files on one server and propagate it to all web frontends?
You typically don't want to store large files in a relational database -- it's not what they're designed for. I would also advise against using a NoSQL solution, since they're also typically not designed for this, although there are a few exceptions (see below).
Your last idea, storing the files on the filesystem (do note that this is what filesystems are designed for ;) is most likely the right approach. This can be somewhat difficult depending on what your scalability requirements are, but you will likely want to go with one of the following:
SAN. SANs provide redundant, highly-available storage solutions within a network. Multiple servers can be attached to storage provided by a SAN and share files between each other. Note that this solution is typically enterprise-oriented and fairly expensive to implement reliably (you'll need physical hardware for it as well as RAID controllers and a lot of disks, at minimum).
CDN. A content delivery network is a remote, globally distributed system for serving files to end users over the Internet. You typically put a file in a location on your server that is then replicated to the CDN for actual distribution. The way a CDN works is that if it doesn't have the file a user is requesting, it'll automatically try to fetch it from your server; once it has a copy of the file once, it caches the file for some period of time. It can be really helpful if you're normally constrained by bandwidth costs or processing overhead from serving up a huge number of files concurrently.
Cloud offering (Amazon S3, Rackspace Cloud Files). These are similar to a CDN, but work well with your existing cloud infrastructure, if that's something you're using. You issue a request to the cloud API to store your file, and it subsequently becomes available over the Internet, just like with a CDN. The major difference is that you have to issue any storage requests (create, delete, or update) manually.
If the number of files you're serving is small, you can also go with an in-house solution. Store files on two or three servers (perhaps have a larger set of servers and use a hash calculation for sharding if space becomes an issue). Build a small API for your frontend servers to request files from your storage servers, falling back to alternate servers if one is unavailable.
One solution that I almost forgot (although I haven't ever used beyond research purposes) is Riak's Luwak project. Luwak is an extension of Riak, which is an efficient distributed key/value store, that provides large file support by breaking the large files into consistently-sized segments and then storing those segments in a tree structure for quick access. It might be something to look into, because it gives you the redundancy, sharding, and API that I mentioned in the last paragraph for free.
I work as a (volunteer) developer on a fairly large website - we have some 2GB of images in 14000 images [that's clearly nowhere near a "world record"], and a database of 150MB of database. Image files are stored as separate files instead of as database objects, partly because we resize images for different usages - thumbnails, medium and large images are created programattically from the stored image (which may be larger than the "large" size we use for the site).
Whilst it's possible to store "blobs" (Binary Large Objects) in SQL databases, I don't believe it's the best solution. Storing a reference in the database, so that you can make a path/filename combination for the actual stored file [and possibly hiding the actual image behind some sort of script - php, jsp, ruby or whatever you prefer] would be a better solution.
When users log into our site we retrieve an object from our database that contains various settings that are used throughout the site. In order to reduce server load from going back to our database each time the user interacts with our site, we are trying to think of alternative ways. (We serialize and de-serialize the object, when needed). The object is likely to be <1MB but could vary.
How big of an object can we have in a session without significantly affecting performance?
How big of an object can we store in a cookie?
Are there any other alternatives (other, than, retrieving the data from our DB)?
The maximum allowed cookie size depends on the client. For example, a MSDN article from 2005 says that the whole cookie may have at least 4096 bytes available (including expiry date etc). The RFC mentioned in the same article contains some more information regarding limitations:
6.3 Implementation Limits
Practical user agent implementations have limits on the number and
size of cookies that they can store. In general, user agents' cookie
support should have no fixed limits. They should strive to store as
many frequently-used cookies as possible. Furthermore, general-use
user agents should provide each of the following minimum capabilities
individually, although not necessarily simultaneously:
at least 300 cookies
at least 4096 bytes per cookie (as measured by the size of the
characters that comprise the cookie non-terminal in the syntax
description of the Set-Cookie header)
at least 20 cookies per unique host or domain name
If your session data is not valuable (as in "shouldn't be lost in case of e.g. a reboot"), consider storing it in memcached. This is pretty fast and avoids accessing the DB just to get session data. You might actually want to consider using a mix of both: You could create a small cookie containing the session id and login information. Then a loss of your server-side sessions would not result in users being logged out so the impact would be pretty low.
An alternative to cookies is html5 local storage. It's not supported by old browsers, but if that doesn't matter to you its a good option for user preferences. Keep in mind the following:
1) The default limit is 5MB per domain (I think)
2) If you store settings-type data in local storage, you still need to sync with a server, or else changing browsers will result in user settings not being present in the new browser.
I'm building a demo, and find that I am storing lots of data in localStorage, almost constantly writing and reading values, parsing and stringifying JSON, etc. etc.
Do you find yourself relying on localStorage much?
What are the dangers of overusing it?
Are there dangers in regularly translating variables from and to JSON for storage?
It seems to me that if I did a db storage of lots of this data, I'd significantly increase the number and size of queries to my db.
The demo is very user focused, so I'm storing stuff like items the user has selected and input the user has provided. The data stored locally is only of value/interest to the user.
If you were building something like a todo list with scheduled end dates and reminders, would you use localStorage? Why or why not?
I know one limitation is that the user would only be able to view this content on one browser on one machine, but that isn't an issue for now.
You should really only write data to local storage that should persist across pages. If you're constantly parsing/stringifying JSON, then your code is likely much slower than it needs to be. Aside from reducing performance, there's a limited amount of space available in local storage, so you should use it judiciously.
Ask yourself: "How much of this data needs to stick around after the user leaves this page?"