I need to store the X most requested images on my website. Each image is about 20-50KB. It will be easy at first as I have 0 images, but once I go X+1 I'll need to remove the least requested images to bring me back to X.
I was thinking of using a database on each image request. Logging the file name and then either loading the image locally or loading the image remotely. If I'm under X and I don't have the image locally then I'll load the image remotely and save a copy locally for future use. If I'm over X I'll delete the least requested image from the database and locally.
Then it hit me, I could store the image content inside the database. I have to connect to the database anyways, and it would make purging the file really easy.
How much of a performance sin would storing images in a database be? And if its bad, is there a better approach?
I don't know the exact performance impact as it depends on a lot of factors, why not give it a try yourself? Add 10000 images (same image for example) and check which manages to perform the given task faster.
You can store the image in the DB with 2 columns, one being the header image/png and one being the body which are the image bytes as a base64 encoded string.
Having the image in the database would also mean that you have to invoke some server side logic and establish a db connection each time you want to fetch an image, something that you can skip with regular files.
But again, these days computers are powerful and the difference might be insignificant for you.
Related
I've a website that all of it's images are loading from MySQL database.
Sometimes when many clients connect to the site, it slows down & I'm doing some optimizations on my server & codes to increase total performance.
As a candidate of change, I want to know that moving files out of the database & loading from static files instead of dynamically generated contents, can cause any significant improvement on my performance?
If yes, is there any benchmark available about it?
Storing images in a database is generally a bad idea, yet you see lots of people doing it without any good reason.
In 99% of cases I would recommend only storing file path references to the images in the database and have the images stored statically.
Here are some reasons why:
You don't tie up both the application server and the databases server transmitting images to the browser, you can offload this to web server itself which is more optimized for this.
If you have a sizeable site, you would eventually want to move static images onto a CDN anyway. You can't do this with files in database
You application will be slower when trying to insert images into the database, as you basically have to upload file to application server, then turn around and write into the DB as opposed to simply writing the path reference.
You DB itself could grow in size at a significant rate with enough images. You don't want to tie up your DB file system with a bunch of files that can be stored at low cost in other ways (like distributed file storage services like Amazon S3)
I have a similar situation to yours. The solution is simple: cache the content.
When you run the first time the query to get an image, ie:
SELECT * FROM images WHERE id = 1
Then simply cache the result to a file:
file_put_contents("image1.png",$row['data']);
Next time simply check whenever there is the file, this will avoid to query the database
This question already has answers here:
Storing Images in DB - Yea or Nay?
(56 answers)
Closed 8 years ago.
I'd like to know what's the best method to store images into a MySQL database with the best performance. (assume a big database)
Solution A: Use LONGBLOB type and put the binary content
Solution B: Store the files on the hosting and save the URL how a VARCHAR type
Thanks in advance
If you are building a system that needs to scale up to serve many images: there is an enormous disadvantage to putting the image contents in your DBMS.
Web servers can be clustered around large common file storage systems and can serve images very efficiently. This sort of file-serving architecture has been highly optimized by the various web server products like Apache, nginx and IIS.
But if you put your images in database BLOBs, fetching them from the database becomes a bottleneck. Plus you'll need a script (php, .net, whatever) to run to serve each image.
Almost all production web sites serve their images directly from files. You can store, in your DBMS, the locations on your file store, then convert them to URLs in the HTML pages you send out.
For example, if the imgloc column contains u/10234/abcde.img, and the table also has width and height columns, your web app can emit something like
<img src="/content/u/10234/abcde.img" width="300" height="200">
Then the client will fetch the image from your content store.
I'd say rather than store the url, store the image name like "image1.png" and host the images on your server. That should help reduce db size and call times a bit. Then programmaticly prefix the url path and reference that.
Its probably also better to just save the file name in case the image location changes. that way you won't have to worry about updating the entire table and you can just fix the one line of code with the url path
I tend to lean toward putting them in the database. That way when you do backups of the DB, it's all packaged up together. If you ever move servers, then you don't have to worry about moving files from the file system.
Unless you're talking about large files(GBs), i doubt you would see much of a difference from file system vs.db performance.
Either way is going to work for you, just a matter of preference and what works best for your particular situation.
There are many answers already on SO and the internet.
Storing Images in DB - Yea or Nay?
I am using MySQL and i want to store some images. I read on forum that instead of storing directly images as 'BLOB', store images in directory and there path in database. But what if user will delete any image from directory(or by some other means image gets deleted). Is there any way to handle that situation ? And why its not preferd to store images directly in db?
EDIT: The database is not related to website development. I am developing a software in which user will take some images using camera and after playing with it, I want to store a record for that image in database with some other parameters(in which one column is for storing image). The software and database is going to run on users PC. And I am going to generate (probably in excel) final report of all records.
Well, while storing image in mysql can lead to increasing size and load for the database. If your site have a high volume, traffic site, this is not a good idea. Instead, i usually prefer to store the image files in a folder & update its path in the database. The main thing you need to take care is setting a unique name (such as img1_2014_04_20_16_30_25.JPG) in order to prevent the file getting overwritten. In the above mentioned name can be made unique using the time functions easily. Then there is no direct way of handling such error with mysql as far as i know, all you can do with your front end is to make sure the image exist or put a default no image as display.
You can never be 100% sure that the images won't get deleted. Of course for extra security you can give permission only to a certain user to the directory and the files inside. Downside of storing blobs - as much as MySQL is capable of handling quite big databases, images can add up ridiculous amount of size to the database and it will be much harder to manage, maintain and optimize if it gets really big. Migration will be really difficult if you decide to swap the engine, and even more painful if you decide to go with a noSQL solution(couchdb, mongo, etc.). Perhaps you could set up a cron to iterate through all the records and check if the file exists. If it doesn't, drop the record from the table. Then again I'm not sure what you are trying to do exactly. If a user has the option to delete an image, I assume it's going to happen through some interface. You will need to implement a logic so that:
1) The file gets deleted
2) The record is deleted from the database.
I have an application where every user gets there own database and runs from the same file system folder. (the database is determined by sub domain)
Storing in the filesystem could lead to conflict. I'd imagine the images upload would be small. (I would scale them down before storing)
Is it ok in this case to store in database?
(I know this has been asked a lot)
I also want to make my application easy to install and creating a writable folder is hard for some people)
To take the contrary view from Nathanial -- I find it easier to use the data base to store opaque data like images. When you back up the data base, you automatically get a backup of the images. Also, you can retrieve, update, or delete the image along with all the other data in integrated SQL queries; keeping the files separately means writing much more complex code that has to go out to the file system to maintain data integrity every time you issue certain SQL queries. Locking can be a big problem, and transaction processing (especially rollback) even bigger.
Seems like you've already sort of talked yourself into it, but in my experience it's better to store files in a filesystem and data in a database. Use GUID's for the file names if you are worried about a conflict.
Pasting my answer from a similar post: I have implemented both solutions (file system and database-persisted images) in previous projects. In my opinion, you should store images in your database. Here's why:
File system storage is more complicated when your app servers are
clustered. You have to have shared storage. Even if your current
environment is not clustered, this makes it more difficult to scale
up when you need to
You should be using a CDN for your static
content anyways, and set your app up as the origin. This means that
your app will only be hit once for a given image, then it will be
cached on the CDN. CloudFront is dirt cheap and simple to set
up...there's no reason not to use it. Save your bandwidth for your
dynamic content.
It's much quicker (and thus cheaper) to develop
database persisted images
You get referential integrity with
database persisted images. If you're storing images on the file
system, you will inevitably have orphan files with no matching
database records, or you'll have database records with broken file
links. This WILL happen...it's just a matter of time. You'll have to
write something to clean these up.
Anyways, my two cents.
I am implementing a project that deals with a significant amount of images.
In your opinion what are the cons/pros of the following two approaches:
I need to store thousands of items, each item as several string properties and an image.
Each item as an ID (integer)
MyISAM tables
How would you store the images:
approach 1: store images into a directory and each image named as ID.jpg
approach 2: store images into the database as a binary BLOB
Using approach 1 I can access the image directly and that's it
<img src="same_directory/10.jpg" />
Using approach 2, I can still use the above HTML, but need to redirect that jpg access to a PHP script which will return the real image from the DB.
In terms of performance which one do you think its faster?
I am keen to approach 1.
advantages of approach 1:
Retrieving the flat file form webserver is more faster.
most of the web hosts likely to follow this approach.
the file system is faster for flat file storage.
advantages of approach 2:
All your data is kept in one place, if you migrate your
website/database the images will just be there
Its easier to sort/delete/etc...
Since you have to serve it via a PHP script, you can perform
additional things such as security if required, or image processing
(obviously you can do this with flat file too, but you have to make
sure the security cant be bypassed by leaving the images in a public
directory).
considering performance approach 1 is best to proceed.
Storing on filesystem is faster.
I'm be tempted to use the first approach as there's no real value in cluttering up the database with image data. (Fetching the data from the database will also be significantly slower than simply loading it off disk.)
However, as an suggestion you might not want to store the full path on disk to the image in the database table, to aid portability in the future. (i.e.: Just store the portion of the path and filename off a 'known' base folder.)
Keep the image files as image files on the server to lower your DB load and allow the server to handle caching etc.
Overall it really depends on the kind of images we're talking about. Small thumbnails (e.g. for file icons) wouldn't be that bad, but I wouldn't store whole images in the DB. In general I guess the file system approach would be faster.
Lets investigate problem on web browser.
When you load page with 10 pictures saved in database. You browser send new http request to the server. Each request init DB connection and server side script. Or just read static image from the file system.
what will be faster?
Other part - get data from file system or database. If we do not use cache for the database (but for 10 GB of images you should have 10 GB RAM to cache this data). Database and HTTP server reads data from file system in any case. But I think HTTP browser reads data faster then Database server.
Only one thing cons for the Database storage - very easy to migrate data from one server to other. But this is not matter for system performance.
And do not forget make path for images like /a/b/c/abc.jpg - it will be faster for big amount of images, then put all images in one directory.