I want to insert a file in MYSQL database residing on a remote webserver using a webservice.
My question is: What type of table column (e.g. varchar, etc.) will store a file? And will the insert statement be somewhat different in case of a file?
File size by MySQL type:
TINYBLOB 255 bytes = 0.000255 Mb
BLOB 65535 bytes = 0.0655 Mb
MEDIUMBLOB 16777215 bytes = 16.78 Mb
LONGBLOB 4294967295 bytes = 4294.97 Mb = 4.295 Gb
Yet, in most cases, I would NOT recommend storing big blobs of bytes in database, even if it supports it, because it will increase overall database size & may cause real performance issues. You can read more on topic here. Many databases that care about consistent performance won't even let you do such thing. Like e.g. AWS DynamoDB, which is known to perform extremely well at any scale, limits single item record to 400KB. MongoDB does allow 16MB, which is also already too much, imo. MySQL allows all 4GB if you wish. But again, think twice before doing that. The case where you may be OK to store big blob of data with these column types would be - you have small traffic database and you just want to save all the stuff in one place for faster development. Like internal system in a small company.
The BLOB datatype is best for storing files.
See: How to store .pdf files into MySQL as BLOBs using PHP?
The MySQL BLOB reference manual has some interesting comments
The other answers will give you a good idea how to accomplish what you have asked for....
However
There are not many cases where this is a good idea. It is usually better to store only the filename in the database and the file on the file system.
That way your database is much smaller, can be transported around easier and more importantly is quicker to backup / restore.
You need to use BLOB, there's TINY, MEDIUM, LONG, and just BLOB, as with other types, choose one according to your size needs.
TINYBLOB 255
BLOB 65535
MEDIUMBLOB 16777215
LONGBLOB 4294967295
(in bytes)
The insert statement would be fairly normal. You need to read the file using fread and then addslashes to it.
Related
I have a data table which has to be read often. I need to store in it strings and binary data of variable length. I could store data as BLOB or TEXT, but the way I understand MySql, those types are stored on the hard drive instead of memory, and if I use them, the speed of reading the table is going to be low.
Are there any alternative variable length types which I could use? Or, maybe, is there a way to tell MySql to hold the data in columns of those types in memory?
Is this 'data table' the only place that the strings are stored? If so, you need the 'persistence' of storing it on disk. However, MySQL will "cache" the data, so reads will almost always be from RAM.
If each element of data is not 'too' big, you could use ENGINE=MEMORY for the table; that would leave the data only in RAM. A system crash would lose the data.
But if you don't need persistence, there are many flavors of caching outside MySQL. Please describe where the data comes from, what language is using the data, how big the data is, etc.
I have about 2k of raw binary data that I need to store in a table, but don't know whether to choose the Varbinary or Blob type. I have read through the descriptions in the MySQL docs but didn't find any contract and compare descriptions. I also read that varbinary only supports up to 255 characters, but I successfully created a varbinary(2048) field, so I'm a bit confused.
The binary data does not need to be indexed, nor will I need to query on it. Is there an advantage to using one type over the other from PHP?
Thanks!
VARBINARY is bound to 255 bytes on MySQL 5.0.2 and below, to 65kB on 5.0.3 and above.
BLOB is bound to 65kB.
Ultimately, VARBINARY is virtually the same as BLOB (from the perspective of what can be stored in it), unless you want to preserve compatibility with "old" versions of MySQL. The MySQL Documentation says:
In most respects, you can regard a BLOB column as a VARBINARY column that can be as large as you like.
Actually blob can be bigger (there are tinyblob, blob, mediumblob & longblob http://dev.mysql.com/doc/refman/5.0/en/storage-requirements.html) with up to 2^32 -1 on size limit.
Also blob storage grows "outside" of the row, while max varbinary size is tied by amount of free row size available (so it can actually be less than 64Kb).
There are some minor differences between both
1) With Index scripting (blob needs a prefix size on indexes,
varbinary doesn't) http:/en/column-indexes.html
CREATE TABLE test (blob_col BLOB, INDEX(blob_col(10)));
2) As already mentioned there are trailling space issues managed
differently between varbinary & blob at MySql 5.0.x or earlier
versions: http:///en/blob.html
http:///en/binary-varbinary.html
(truncating the links, since stackoverflow thinks too many links are spam)
One significant difference is blob types are stored in secondary storage, while varbinaries are stored inline in the row in the same way as varchars and other "simple" types.
This can have an impact on performance in a busy system, where the additional lookup to fetch and manipulate the blob data can be expensive.
It is worth to point that Memory storage engine does not support BLOB/TEXT but it works with VARBINARY.
I am just looking at a test app that stores around 5k binary data in a column. It initially used varbinary but since it is so slow I decided to try blob. Well I'm looking at disk write speed with atop and can't see any difference.
The only significant difference I read in mysql manual is that blobs are unsupported by the memory engine so any temporary tables you create with queries (see when mysql uses temp tables) will be created on-disk and that is much slower.
So you better bet on varbinary/binary if it is a short enough to fit into a row (at the moment 64k total for all columns).
Is there a way to edit a blob with mysql, for example to delete from 39th to 48 byte or to insert some bytes (characters), at some position? Are there any such commands.
If your blob is extremely large, entering it into the database in the first place is very difficult.
MySQL only allows the maximum packet size to be sent as a command - which makes using large blobs difficult.
If your blobs are big enough that you care, you probably need to create some schema where they're stored as several rows with chunks, to allow them to be created in a sensible fashion.
BUT storing very large blobs is probably not a good idea in MySQL, as it's effectively a massive waste of your innodb_buffer_pool.
NB: By "Very large" I mean > 10M or so.
I understand the size/storage constraints of MySQL TEXT and MEDIUMTEXT fields, but I just wanted to make absolutely sure (before I sign off on a change) that I'm not looking at any adverse effects from converting a field with existing data from TEXT to MEDIUMTEXT.
My concerns are mainly performance, integrity, and disk storage.
Thanks
With regard to the performance, integrity, and disk storage in the database layer, I wouldn't worry about it.
Variable-length data like varchar, text, and blob is stored without padding.
I don't know any issues with integrity. All data types are treated atomically by the database engine.
Of course if you have really long text data then it will take more storage and more time for disk I/O and network bandwidth when you fetch that data. But if that's the data you need to put in the database, then that's what you have to do.
I can think of one possible impact:
Some client interface libraries pre-allocate a buffer to hold results, and they allocate enough memory for the largest possible value. The client doesn't know the length of the data before it fetches the data, so it must allocate enough space assuming the data might be as long as the data type supports.
Therefore the library would allocate 16MB per mediumtext while it would allocate 64KB for a text. This is something to watch out for if you have a low memory limit in your client layer. For instance, PHP has a memory_limit config parameter for scripts, and the buffer allocated for data result sets would count toward this.
How can I insert huge BLOBs into a MySQL database (InnoDB)?
Fields of type LONGBLOB support data sizes of up to 4 GB according to the MySQL manual. But how does data of such a huge size get into the database?
I tried to use
INSERT INTO table (bindata) VALUES ( LOAD_FILE('c:/tmp/hugefile') );
which fails if the size of hugefile is bigger than about 500 MB. I have set max_allowed_packet to an appropriate size; the value of innodb_buffer_pool_size doesn't seem to have an influence.
My server machine runs Windows Server 2003 and has 2 GB RAM. I'm using MySQL 5.0.74-enterprise-nt.
BLOBs are cached in memory, that's why you will have 3 copies of a BLOB as you are inserting it into a database.
Your 500 MB BLOB occupies 1,500 MB in RAM, which seems to hit your memory limit.
I do not know which client/api you use, but when trying to use blobs from own Java and Objective-C clients it seems, MySQL does not really support streaming of blobs. You need enough memory to hold the whole blob as byte array in ram (server and client side) more than once! Moving to a 64 bit linux helps, but is not desired solution.
MySQL ist not made for BLOB handling (ok for small BOBs :-). It occupies twice or three times the ram to be stored/read in the "BLOB".
You have to use an other database like PostgreSQL to get real BLOB support, sorry.