We have a system that has many machines working together. At some point they need to create a image and upload it to a media server. This media server is just a web server providing static files. We dont want to display the incremental id of each image so we need to obfuscate the image name.
Because each machine has a unique user and we keep track only of the generated url we decided to use HMAC-SHA256(secret, user_id) to generate filenames for each image.
This results in images with this kind of names: b3dbaea7adef41e8ecbeaa0cd8ec02d57f0334a6e003271d5e334ae2e46b4285.jpg
Is not the most beautiful solution but it does works, for now.
Because some business changes now we need this image to be generated more frequent and we need to create folders for each user.
How can I keep the unique file path without it being so large and, at the same time, avoid collision between image names or between user folders
I want to avoid this kind of path: b3dbaea7adef41e8ecbeaa0cd8ec02d57f0334a6e003271d5e334ae2e46b4285/b3dbaea7adef41e8ecbeaa0cd8ec02d57f0334a6e003271d5e334ae2e46b4285.jpg
MD5, as opposed to SHA256, will generate strings about half as long; if you used it for both the directory & file names, you'd have paths about as long as your current implementation.
Another thought: the character space used is much smaller (16 characters) than file and directory names would allow (either 36 or 62, depending on if you allow both cases or not, and that's sticking to alphanumerics), so you can map your hashes to smaller strings without raising the likelihood of collisions.
Related
I want to allow file upload(public/per user), but not sure about how to properly do it.
I've read that it is not recommended to use MySQL for this and instead should use file system for files and indexing them in the database. I remember reading some popular(with many votes) Q/A about this in SO, but can't find it(please send a link if you can).
So how should I do it? I should use some inaccessible folder and store files there with an ID as their name and this name(ID of the file) in the table of files with user_id, and when the user request a file, I should check auth and then send the corresponding file to the user?
There's no single answer to this question. It depends how you want your application to work. It could be perfectly fine, for example, to put the files in a directory that is http-accessible, if you don't need to restrict access. Then you don't need to pass the file through any code, you just link directly to it.
There are also legitimate reasons to store a file in the database. For instance, it's automatically included in backups, it is guaranteed to be deleted when you delete the database row, it obeys transaction semantics, and so on.
This has been asked frequently on Stack Overflow. Here are a few links to ones that I have answered.
Should I use MySQL blob field type? (2009)
What is difference between storing data in a blob, vs. storing a pointer to a file? (2012)
Save file as blob in MYSQL database or as file path (2018)
I also cover this in the chapter "Phantom Files" in my book, SQL Antipatterns Volume 1: Avoiding the Pitfalls of Database Programming.
tl;dr
In my node.js application I create pdf documents. What is the best/right way to save them? Right now I use node.js fileserver and shell.js to do it.
I am working on a node.js web application to manage apartments and tenants for learning purpose and on some point I create PDF Documents that I want to save under a path
/documents/building_name/apartment_name/tenant_name/year/example.pfd
Now if the user wants to change the building, apartment or tenant name via an http PUT request I change the database but also the want to change the path.
Well both works but I can't write good tests for these functions.
Now a friend told me that it's a bad practice to save documents on a file server and I better should use BLOB.
On the other side google doesn't really agree on using blobs
So what is the right way to save documents?
Thanks
Amit
You should first define a source of truth. Unless you're legally obliged to keep copies of those files and they are not being accessed very often, I wouldn't even bother storing those at all and just generate them upon request.
If not, keep the DB clean, blobs will make it huge. Put them into cold storage (again assuming they are not being accessed too frequently) without those paths. If the paths are reliant on often changing information, that can't be performant for neither the file server nor your system.
Instead store a revision number in your DB that the file can be found under and limit the path structure to information that rarely change.
Like {building}/{apartment}/{tenant}_{revision}.pfd
That - depending on your backup structure - will allow you to time-travel if necessary and doesn't force a re-index all the time.
Note: I don't know too much about your use case.
I have to display images on website and I can store image in the folder on my website and also I can store the image in image column of SQL Server.
So which way of storing image is better : in folder or in Image column of SQL Server.
1. Which way of storing image and retrieving it is faster
With SQL Server 2008, while you can store BLOB data, it's best to avoid it. I've done it in the past, grudgingly, and it did have negative performance implications. Unless you have some constraint which prevents it, use the file system. That's what it's built for, and it's much faster.
As #Martin Smith pointed out you could use FileStream. We started storing our files using FileStream so that we could also add full-text indexing and allow the users to not only search the data, but the files on our site. It is also nice because we can easily move our files along with teh Database to other environments (Dev, Test).
nice file stream Article: Here
Also, please use varbinary(max) if you are going to store in the DB. The image column is going to be deprecated in future versions.
--S
I am trying to prepare a sample question paper preparation app.
I am using tinyMCE editor and ajax-file-uploader plugin with it - as some questions may need images along with text.
How do I store my questions that have both image and text into MySql using PhP?
I would suggest storing the image on some sort of NAS or some other location, and store the path to the image in the database along with the other data in respective fields.
You can store the image in the DB but it is not good idea to retrieve and present the image from the database to the user (It doesn't perform that great either). There might be a performance hit
MySQL handles images very well. You insert them in BLOBs. On the other hand you could store the image file name and path or a link, as text in your DB.
Which solution is the best depends on the requirements of your application. In general if you have a huge amount of images, your database will become huge and backing up will be slow. There might be a similar impact on your file system in order to store a huge amount of images.
Here is the interesting Microsoft To BLOB or not to BLOB paper, that will give you more information on the topic and even some metrics.
I would echo the answers already given by #harigm and #Costis Aivalis but if you really wanted to go "all out" and store both the HTML content and the images in the same BLOB why not have a look at RFC 2557 which allows you to place binary data (like images) in the document itself using the url scheme data:. To make this work you will need to parse your HTML once it gets back to your server and base64 encode all the images to be placed in the HTML, quite a lot of work for what would probably turn out to be little reward.
I am using Sencha Touch to capture data from a user on an iPad. This includes a standard form (name, email, etc.) as well as the customer's signature (see the plugin here).
Essentially, the plugin takes the coordinates from the user's signature and gives me back Base64 PNG data.
Once I have the signature data, I want to store it. My two questions are:
Should I store the Base64 data in my
(MySQL) database along with the rest
of the user's information, or should
I create a static file and link as
necessary?
If storing in the
database is the way to go, what data
type should I use?
There's no need to base64 encode the image. MySQL's perfectly capable of storing binary data. Just make sure you use a 'blob' field type, and not 'text'. text fields are subject to character set translation, which could trash your .png data. blob fields are not translated.
As well, base64 encoding increases the size of text by around 35%, so you'd be wasting a large chunk of space for no benefit whatsoever.
However, it's generally a bad idea to store images in the database. You do have the advantage of the image being "right there" always, but makes for absolutely huge dumps at backup time and all kinds of fun trying to get the image out and displayed in your app/web page.
it's invariably better to store it externally in a file named after the record's primary key for ease of access/verfication.
Just save files in BLOB field. Such PNG file shouldn't be larger than 1KB if you turn some optimizations (grayscale or B/W).
Storing files outside DB seems easy but there are things to consider:
backup,
additional replication if multi-server
security - access rights to files dir, but also to files,
no transactions - e.g. DB insert ok but file write fails,
need to distribute files within multiple directories to avoid large dir listings (depends on filesystem capabilities)
Blob will store Base64. It will get you what you need. Storing it in the database gives you built in relational capabilities that you would have to code yourself if you stored it in a static file. Hope this helps. Good luck sir.
Edit: mark's right about binary v. base 64
Set your field as Blob data type, it stores perfectly base64EncodedString