I have a website with a lot of small images (no larger than 300 kb). They're all in a single folder, and now they're around 40.000, but they keep increasing very fast.
When I open a page of my website, each page will load 6 of those images. I can have also 1000 or more people connected together on my website.
Do you think this way of storing small files will slow down performances of my server? I thought about these possible solutions, but I'm not sure which of those is the best:
- Store these pictures in folders, maybe a folder for each day, so we'll have no more than 5000 picture in each folder;
- Store pictures in my DB in blob fields.
My server uses CentOS 6.5 as operating system.
Which solution do you think will be the best?
Related
I've a website that all of it's images are loading from MySQL database.
Sometimes when many clients connect to the site, it slows down & I'm doing some optimizations on my server & codes to increase total performance.
As a candidate of change, I want to know that moving files out of the database & loading from static files instead of dynamically generated contents, can cause any significant improvement on my performance?
If yes, is there any benchmark available about it?
Storing images in a database is generally a bad idea, yet you see lots of people doing it without any good reason.
In 99% of cases I would recommend only storing file path references to the images in the database and have the images stored statically.
Here are some reasons why:
You don't tie up both the application server and the databases server transmitting images to the browser, you can offload this to web server itself which is more optimized for this.
If you have a sizeable site, you would eventually want to move static images onto a CDN anyway. You can't do this with files in database
You application will be slower when trying to insert images into the database, as you basically have to upload file to application server, then turn around and write into the DB as opposed to simply writing the path reference.
You DB itself could grow in size at a significant rate with enough images. You don't want to tie up your DB file system with a bunch of files that can be stored at low cost in other ways (like distributed file storage services like Amazon S3)
I have a similar situation to yours. The solution is simple: cache the content.
When you run the first time the query to get an image, ie:
SELECT * FROM images WHERE id = 1
Then simply cache the result to a file:
file_put_contents("image1.png",$row['data']);
Next time simply check whenever there is the file, this will avoid to query the database
This question already has answers here:
Storing Images in DB - Yea or Nay?
(56 answers)
Closed 8 years ago.
I'd like to know what's the best method to store images into a MySQL database with the best performance. (assume a big database)
Solution A: Use LONGBLOB type and put the binary content
Solution B: Store the files on the hosting and save the URL how a VARCHAR type
Thanks in advance
If you are building a system that needs to scale up to serve many images: there is an enormous disadvantage to putting the image contents in your DBMS.
Web servers can be clustered around large common file storage systems and can serve images very efficiently. This sort of file-serving architecture has been highly optimized by the various web server products like Apache, nginx and IIS.
But if you put your images in database BLOBs, fetching them from the database becomes a bottleneck. Plus you'll need a script (php, .net, whatever) to run to serve each image.
Almost all production web sites serve their images directly from files. You can store, in your DBMS, the locations on your file store, then convert them to URLs in the HTML pages you send out.
For example, if the imgloc column contains u/10234/abcde.img, and the table also has width and height columns, your web app can emit something like
<img src="/content/u/10234/abcde.img" width="300" height="200">
Then the client will fetch the image from your content store.
I'd say rather than store the url, store the image name like "image1.png" and host the images on your server. That should help reduce db size and call times a bit. Then programmaticly prefix the url path and reference that.
Its probably also better to just save the file name in case the image location changes. that way you won't have to worry about updating the entire table and you can just fix the one line of code with the url path
I tend to lean toward putting them in the database. That way when you do backups of the DB, it's all packaged up together. If you ever move servers, then you don't have to worry about moving files from the file system.
Unless you're talking about large files(GBs), i doubt you would see much of a difference from file system vs.db performance.
Either way is going to work for you, just a matter of preference and what works best for your particular situation.
There are many answers already on SO and the internet.
Storing Images in DB - Yea or Nay?
We have about 60 million webpages in a compressed format. We would like to de-compress and work with these files individually.
Here are my questions!
First, if I decompress them into the file system, would the FS cope with such number of files. My file system is ext4. (I've 4 different file systems so I can divide the data between them like 15 M pages for each file system)
Secondly, Would storing these files into a relational database be a better option? assuming that all the hassle of cleaning html text is done before inserting them into the database.
Thanks,
If you extract them into a single directory you may exceed the maximum allocated indices in that folder. If you extract them into multiple directories you will fair better.
60 Million is definitely a fair amount, if you plan on doing any indexing on them or searching then a database would be your best option, you can do indexing on files using something like lucene it all depends on what you want to do with the files After they ave been extracted.
I currently have a similar issue with images on a large user site, the way I got around this issue was to give each image a GUID and for each byte in the guid assign it to a different directory, then the next byte under a subdirectory (down to 8 bytes) if my fill ratio goes up I'll create more subdirectories to compensate, it also means I can spread it across different net storage boxes.
I am implementing a project that deals with a significant amount of images.
In your opinion what are the cons/pros of the following two approaches:
I need to store thousands of items, each item as several string properties and an image.
Each item as an ID (integer)
MyISAM tables
How would you store the images:
approach 1: store images into a directory and each image named as ID.jpg
approach 2: store images into the database as a binary BLOB
Using approach 1 I can access the image directly and that's it
<img src="same_directory/10.jpg" />
Using approach 2, I can still use the above HTML, but need to redirect that jpg access to a PHP script which will return the real image from the DB.
In terms of performance which one do you think its faster?
I am keen to approach 1.
advantages of approach 1:
Retrieving the flat file form webserver is more faster.
most of the web hosts likely to follow this approach.
the file system is faster for flat file storage.
advantages of approach 2:
All your data is kept in one place, if you migrate your
website/database the images will just be there
Its easier to sort/delete/etc...
Since you have to serve it via a PHP script, you can perform
additional things such as security if required, or image processing
(obviously you can do this with flat file too, but you have to make
sure the security cant be bypassed by leaving the images in a public
directory).
considering performance approach 1 is best to proceed.
Storing on filesystem is faster.
I'm be tempted to use the first approach as there's no real value in cluttering up the database with image data. (Fetching the data from the database will also be significantly slower than simply loading it off disk.)
However, as an suggestion you might not want to store the full path on disk to the image in the database table, to aid portability in the future. (i.e.: Just store the portion of the path and filename off a 'known' base folder.)
Keep the image files as image files on the server to lower your DB load and allow the server to handle caching etc.
Overall it really depends on the kind of images we're talking about. Small thumbnails (e.g. for file icons) wouldn't be that bad, but I wouldn't store whole images in the DB. In general I guess the file system approach would be faster.
Lets investigate problem on web browser.
When you load page with 10 pictures saved in database. You browser send new http request to the server. Each request init DB connection and server side script. Or just read static image from the file system.
what will be faster?
Other part - get data from file system or database. If we do not use cache for the database (but for 10 GB of images you should have 10 GB RAM to cache this data). Database and HTTP server reads data from file system in any case. But I think HTTP browser reads data faster then Database server.
Only one thing cons for the Database storage - very easy to migrate data from one server to other. But this is not matter for system performance.
And do not forget make path for images like /a/b/c/abc.jpg - it will be faster for big amount of images, then put all images in one directory.
This question already has answers here:
Closed 12 years ago.
Possible Duplicates:
Storing Images in DB - Yea or Nay?
Saving images in database mysql
I have a site where user can upload profile picture and photo albums. Now i'm starting to wonder was I suppose to save pictures into a database or is file operation still ok as it gets bigger?
I'm thinking in a huge scale like social networks, hundred of millions of pictures stored in files seem to me crazy so im sure it must be stored some other way?
Any advice or guidance.
Thanks
Storing them on your webserver and having the URL in your database would be a better idea. It makes it easier to modify the images, too.
There are pros and cons to both approaches. Databases are not really great for storing large blobs of data. Doing so will tend to use up lots of resources on the database (CPU, memory, hard drive space and network bandwidth between the database and web server). It will make the database backups much larger.
However, the big problem with storing the images on a separate file system (say on the web server) is that it is much more difficult to sync it with the data that is in the database. If a user changes a photo while a database backup is going on, who knows what state you will end up in. If you ever need to restore the database from a backup, it will be very difficult to sync up the images. You could also get orphaned images hanging around wasting disk space if you aren't careful.
IMHO, it is too much effort to maintain data integrity between the database and a file system, so I would store them in the database throw some money are the hardware so that it can keep up.
Save you pictures on you web server as you are going to develop social networking site so it will be a huge burden on database. Because some times you have to handle pictures of ++MB. So store only link in BD.
From File you can easily play with pics and even can migrate them easily without any effect to database.