This question already has answers here:
Storing Images in DB - Yea or Nay?
(56 answers)
Closed 8 years ago.
I'd like to know what's the best method to store images into a MySQL database with the best performance. (assume a big database)
Solution A: Use LONGBLOB type and put the binary content
Solution B: Store the files on the hosting and save the URL how a VARCHAR type
Thanks in advance
If you are building a system that needs to scale up to serve many images: there is an enormous disadvantage to putting the image contents in your DBMS.
Web servers can be clustered around large common file storage systems and can serve images very efficiently. This sort of file-serving architecture has been highly optimized by the various web server products like Apache, nginx and IIS.
But if you put your images in database BLOBs, fetching them from the database becomes a bottleneck. Plus you'll need a script (php, .net, whatever) to run to serve each image.
Almost all production web sites serve their images directly from files. You can store, in your DBMS, the locations on your file store, then convert them to URLs in the HTML pages you send out.
For example, if the imgloc column contains u/10234/abcde.img, and the table also has width and height columns, your web app can emit something like
<img src="/content/u/10234/abcde.img" width="300" height="200">
Then the client will fetch the image from your content store.
I'd say rather than store the url, store the image name like "image1.png" and host the images on your server. That should help reduce db size and call times a bit. Then programmaticly prefix the url path and reference that.
Its probably also better to just save the file name in case the image location changes. that way you won't have to worry about updating the entire table and you can just fix the one line of code with the url path
I tend to lean toward putting them in the database. That way when you do backups of the DB, it's all packaged up together. If you ever move servers, then you don't have to worry about moving files from the file system.
Unless you're talking about large files(GBs), i doubt you would see much of a difference from file system vs.db performance.
Either way is going to work for you, just a matter of preference and what works best for your particular situation.
There are many answers already on SO and the internet.
Storing Images in DB - Yea or Nay?
Related
I need to store the X most requested images on my website. Each image is about 20-50KB. It will be easy at first as I have 0 images, but once I go X+1 I'll need to remove the least requested images to bring me back to X.
I was thinking of using a database on each image request. Logging the file name and then either loading the image locally or loading the image remotely. If I'm under X and I don't have the image locally then I'll load the image remotely and save a copy locally for future use. If I'm over X I'll delete the least requested image from the database and locally.
Then it hit me, I could store the image content inside the database. I have to connect to the database anyways, and it would make purging the file really easy.
How much of a performance sin would storing images in a database be? And if its bad, is there a better approach?
I don't know the exact performance impact as it depends on a lot of factors, why not give it a try yourself? Add 10000 images (same image for example) and check which manages to perform the given task faster.
You can store the image in the DB with 2 columns, one being the header image/png and one being the body which are the image bytes as a base64 encoded string.
Having the image in the database would also mean that you have to invoke some server side logic and establish a db connection each time you want to fetch an image, something that you can skip with regular files.
But again, these days computers are powerful and the difference might be insignificant for you.
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
database for huge files like audio and video
I'm seeking for the best (or at least good enough) way of storing large sets of binary data (images, videos, documents, etc.). The solution has to be scalable and can't get stuck after X amount of data.
I would like to have a one place for example MySQL database where all the data is kept. When one of web front ends needs it (on request) It can acquire it from the the DB and cache it permanently for later.
From this what I can see on http://dev.mysql.com/doc/refman/5.0/en/table-size-limit.html MySQL table can't store more then 4TB per table. Is there something more appropriate like perhaps nosql databases or perhaps it's better to store everything in files on one server and propagate it to all web frontends?
You typically don't want to store large files in a relational database -- it's not what they're designed for. I would also advise against using a NoSQL solution, since they're also typically not designed for this, although there are a few exceptions (see below).
Your last idea, storing the files on the filesystem (do note that this is what filesystems are designed for ;) is most likely the right approach. This can be somewhat difficult depending on what your scalability requirements are, but you will likely want to go with one of the following:
SAN. SANs provide redundant, highly-available storage solutions within a network. Multiple servers can be attached to storage provided by a SAN and share files between each other. Note that this solution is typically enterprise-oriented and fairly expensive to implement reliably (you'll need physical hardware for it as well as RAID controllers and a lot of disks, at minimum).
CDN. A content delivery network is a remote, globally distributed system for serving files to end users over the Internet. You typically put a file in a location on your server that is then replicated to the CDN for actual distribution. The way a CDN works is that if it doesn't have the file a user is requesting, it'll automatically try to fetch it from your server; once it has a copy of the file once, it caches the file for some period of time. It can be really helpful if you're normally constrained by bandwidth costs or processing overhead from serving up a huge number of files concurrently.
Cloud offering (Amazon S3, Rackspace Cloud Files). These are similar to a CDN, but work well with your existing cloud infrastructure, if that's something you're using. You issue a request to the cloud API to store your file, and it subsequently becomes available over the Internet, just like with a CDN. The major difference is that you have to issue any storage requests (create, delete, or update) manually.
If the number of files you're serving is small, you can also go with an in-house solution. Store files on two or three servers (perhaps have a larger set of servers and use a hash calculation for sharding if space becomes an issue). Build a small API for your frontend servers to request files from your storage servers, falling back to alternate servers if one is unavailable.
One solution that I almost forgot (although I haven't ever used beyond research purposes) is Riak's Luwak project. Luwak is an extension of Riak, which is an efficient distributed key/value store, that provides large file support by breaking the large files into consistently-sized segments and then storing those segments in a tree structure for quick access. It might be something to look into, because it gives you the redundancy, sharding, and API that I mentioned in the last paragraph for free.
I work as a (volunteer) developer on a fairly large website - we have some 2GB of images in 14000 images [that's clearly nowhere near a "world record"], and a database of 150MB of database. Image files are stored as separate files instead of as database objects, partly because we resize images for different usages - thumbnails, medium and large images are created programattically from the stored image (which may be larger than the "large" size we use for the site).
Whilst it's possible to store "blobs" (Binary Large Objects) in SQL databases, I don't believe it's the best solution. Storing a reference in the database, so that you can make a path/filename combination for the actual stored file [and possibly hiding the actual image behind some sort of script - php, jsp, ruby or whatever you prefer] would be a better solution.
This question already has answers here:
Closed 11 years ago.
Possible Duplicate:
Storing Images in DB - Yea or Nay?
Images in database vs file system
I've been developing a web application using RIA technologies (Flex + PHP + MySQL + Ajax) and now I'm in a dilemma about image files.
I use some images in my Flex app, so I think "it could be awesome if I store them into database, and then retrieve from it; consecuently, maintain process should be more easy". But, here is my dilemma:
Should I store the physical URL of my images, or is going to be better if I store directly the image?
For example, should my Cars table looks like:
ID (autonumeric) | Source (text)
or like this?
ID (autonumeric) | Image (longblob or blob)
I know that here are cool people that can answer me this question, explaining me which is better and why :)
I personally recommend to Store Images in the database. Of course it both advantages and disadvantages.
Advantages of storing BLOB data in the database:
It is easier to keep the BLOB data synchronized with the remaining items in the row.
BLOB data is backed up with the database. Having a single storage system can ease administration.
BLOB data can be accessed through XML support in MySQL, which can return a base 64–encoded representation of the data in the XML stream.
MySQL Full Text Search (FTS) operations can be performed against columns that contain fixed or variable-length character (including Unicode) data. You can also perform FTS operations against formatted text-based data contained within image fields—for example, Microsoft Word or Microsoft Excel documents.
Disadvantages of Storing BLOB Data in the Database:
Carefully consider what resources might be better stored on the file system rather than in a database. Good examples are images that are typically referenced via HTTP HREF. This is because:
Retrieving an image from a database incurs significant overhead compared to using the file system.
Disk storage on database SANs is typically more expensive than storage on disks used in Web server farms.
As a general rule you wan't to keep your databases small, so they perform better (and backup better too). So if you can store only a filesystem reference (path + filename) or URL in the DB, that would be better.
Its probably a question of personal preference.
As a general rule its better to keep the database small. However when you come to enterprise applications they regulary add the images directly to the database. If you place them on the file system the db and your file system can get out of sync.
Larger CMS will regulary place those files in the db. However be aware that this requires a larger DB sizing when everything is growing...
When you are saving the url and name only, be sure that these won't change in the future.
With files stored in the database you can implement security easier and you don't have to worry about duplicate filenames.
I used to store the path into the URL, but then adding an additional web server to the mix proved less than ideal. For one thing, you'll have to share the path to where the images are stored. We were using NFS and it became slow after a while. We tried syncing the files from one web server to another but the process became cumbersome.
Having said that, I would store them in the DB. I've since moved all my image/file storage over to MongoDB. I know this doesn't satisfy your needs but we've tried it all (even S3) and we weren't happy with the other solutions. If we had to, I would definite throw them inside MySQL.
Personally, I've always stored the URL.
There's no real reason not to store the image directly in the database, but there are benefits to not storing it in the database.
You get more flexibility when you don't store the image in the database. You can easily move it around and just update the URL in the file. So, if you wanted to move the image from your webserver to a service such as Flickr or Amazon Web Services, it would just be as easy as updating the link to the new files. That also gives you easy access to content delivery networks so that the images are delivered to end users quicker.
I'd store the url, it's less data and that means a smaller database and faster data fetching from it ;)
I have an application where every user gets there own database and runs from the same file system folder. (the database is determined by sub domain)
Storing in the filesystem could lead to conflict. I'd imagine the images upload would be small. (I would scale them down before storing)
Is it ok in this case to store in database?
(I know this has been asked a lot)
I also want to make my application easy to install and creating a writable folder is hard for some people)
To take the contrary view from Nathanial -- I find it easier to use the data base to store opaque data like images. When you back up the data base, you automatically get a backup of the images. Also, you can retrieve, update, or delete the image along with all the other data in integrated SQL queries; keeping the files separately means writing much more complex code that has to go out to the file system to maintain data integrity every time you issue certain SQL queries. Locking can be a big problem, and transaction processing (especially rollback) even bigger.
Seems like you've already sort of talked yourself into it, but in my experience it's better to store files in a filesystem and data in a database. Use GUID's for the file names if you are worried about a conflict.
Pasting my answer from a similar post: I have implemented both solutions (file system and database-persisted images) in previous projects. In my opinion, you should store images in your database. Here's why:
File system storage is more complicated when your app servers are
clustered. You have to have shared storage. Even if your current
environment is not clustered, this makes it more difficult to scale
up when you need to
You should be using a CDN for your static
content anyways, and set your app up as the origin. This means that
your app will only be hit once for a given image, then it will be
cached on the CDN. CloudFront is dirt cheap and simple to set
up...there's no reason not to use it. Save your bandwidth for your
dynamic content.
It's much quicker (and thus cheaper) to develop
database persisted images
You get referential integrity with
database persisted images. If you're storing images on the file
system, you will inevitably have orphan files with no matching
database records, or you'll have database records with broken file
links. This WILL happen...it's just a matter of time. You'll have to
write something to clean these up.
Anyways, my two cents.
I am implementing a project that deals with a significant amount of images.
In your opinion what are the cons/pros of the following two approaches:
I need to store thousands of items, each item as several string properties and an image.
Each item as an ID (integer)
MyISAM tables
How would you store the images:
approach 1: store images into a directory and each image named as ID.jpg
approach 2: store images into the database as a binary BLOB
Using approach 1 I can access the image directly and that's it
<img src="same_directory/10.jpg" />
Using approach 2, I can still use the above HTML, but need to redirect that jpg access to a PHP script which will return the real image from the DB.
In terms of performance which one do you think its faster?
I am keen to approach 1.
advantages of approach 1:
Retrieving the flat file form webserver is more faster.
most of the web hosts likely to follow this approach.
the file system is faster for flat file storage.
advantages of approach 2:
All your data is kept in one place, if you migrate your
website/database the images will just be there
Its easier to sort/delete/etc...
Since you have to serve it via a PHP script, you can perform
additional things such as security if required, or image processing
(obviously you can do this with flat file too, but you have to make
sure the security cant be bypassed by leaving the images in a public
directory).
considering performance approach 1 is best to proceed.
Storing on filesystem is faster.
I'm be tempted to use the first approach as there's no real value in cluttering up the database with image data. (Fetching the data from the database will also be significantly slower than simply loading it off disk.)
However, as an suggestion you might not want to store the full path on disk to the image in the database table, to aid portability in the future. (i.e.: Just store the portion of the path and filename off a 'known' base folder.)
Keep the image files as image files on the server to lower your DB load and allow the server to handle caching etc.
Overall it really depends on the kind of images we're talking about. Small thumbnails (e.g. for file icons) wouldn't be that bad, but I wouldn't store whole images in the DB. In general I guess the file system approach would be faster.
Lets investigate problem on web browser.
When you load page with 10 pictures saved in database. You browser send new http request to the server. Each request init DB connection and server side script. Or just read static image from the file system.
what will be faster?
Other part - get data from file system or database. If we do not use cache for the database (but for 10 GB of images you should have 10 GB RAM to cache this data). Database and HTTP server reads data from file system in any case. But I think HTTP browser reads data faster then Database server.
Only one thing cons for the Database storage - very easy to migrate data from one server to other. But this is not matter for system performance.
And do not forget make path for images like /a/b/c/abc.jpg - it will be faster for big amount of images, then put all images in one directory.