I have a project where I need to store a large number of values.
The data is a dataset holding 1024 2Byte Unsigned integer values. Now I store one value at one row together with a timestamp and a unik ID.
This data is continously stored based on a time trigger.
What I would like to do, is store all 1024 values in one field. So would it be possible to do some routine that stores all the 1024 2byte integer values in one field as binary. Maybe a blobfield.
Thanks.
Br.
Enghoej
Yes. You can serialize your data into a byte array, and store it in a BLOB. 2048 bytes will be supported in a BLOB in most databases.
One big question to ask yourself is "how will I need to retrieve this data?" Any reports or queries such as "what IDs have value X set to Y" will have to load all rows from the table and parse the data AFAIK. For instance, if this were user configuration data, you might need to know which users had a particular setting set incorrectly.
In SQL Server, I'd suggest considering using an XML data type and storing a known schema, since this can be queried with XPath. MySQL did not support this as of 2007, so that may not be an option for you.
I would definitely consider breaking out any data that you might possibly need to query in such a manner into separate columns.
Note also that you will be unable to interpret BLOB data without a client application.
You always want to consider reporting. Databases often end up with multiple clients over the years.
Related
I will generate a big matrix(data frame) in R whose size is about 1300000*10000, about 50 GB. I want to store this matrix in a appropriate format, so later I can feed the data into Python or other program codes to make some analysis. Of course I cannot feed the data one time, so I have to subset the matrix and feed them little by little.
But I don't know how to store the matrix. I think of two ways, but I think neither is appropriate:
(1) plain text(including csv or excel table), because it is very hard to subset(e.g. if I just want some columns and some rows of the data)
(2) database, I have searched information about mysql and sqlite, but it seems that the number of columns is limited in sql database(1024).
So I just want to know if there are any good strategies to store the data, so that I can subset the data by row/column indexes or name.
Have separate column(s) for each of the few columns you need to search/filter on. Then put the entire 10K columns into some data format that is convenient for the client code to parse. JSON is one common possibility.
So the table would 1.3M rows and perhaps 3 columns: an id (auto_increment, primary key), the column search on, and the JSON blob - as datatype JSON or TEXT (depending on software version) for the many data values.
So i have a user based application, which multiple (around 500 users in total) users will be accessing.
Every user will have like 500KB of encoded JSON string in each column (total of 10 cols) that makes 5MB of data for each user.
For 500 users this will be 5*500MB of data in total.
I won't be doing any filtering and searching in the JSON. Moreover the data will be totally different across all the JSONs, so i can not think of storing only the unique content.
I can not use No SQL Databases. Is this a good architecture, if yes what should be the column type?
You may try using mysql 5.7+ and JSON column type, internally it will be stored more efficiently and also you can query it.
I know I can store images in mysql as data type blob. Wandering if i can compare the blob values in my query?
E.g. select answer from mtTable where imageBlobValue = image010.
Here imageBlobValue and image010 data type is blob.
TIA
Well, while that should be theoretically possible, a more optimal way to handle this would be to store an image hash or some other unique identifier alongside the blob to use for query purposes.
You want to minimize the amount of times the database has to handle that blob in memory and the best way to do that is to have a suitable key to search on.
This is a question of converting strings from DB2 to SQL Server.
On DB2 you can have a column that contains a mix of strings and binary data (e.g. using REDEFINS in COBOL to combine string and decimal values into a DB2 column).
This will have unpredictable results during data replication as the binary zero (0x00) is treated as string-terminator (in the C family of software languages).
Both SQL Server and DB2 are able to store binary zero in the middle of fixed length char columns without any issue.
Has anyone any experiences with this problem? The way I see it, the only way to fix it, is to amend the COBOL program and the database schema, so if you have a column of 14 chars, where the first 10 is a string and the last 4 a decimal, split this up into two columns containing one "part" each.
If you want to just transfer the data 1:1, I'd just create a binary(x) field of equal length, of varbinary(x) in case the length differs.
If you need to easily access the stored string and decimal values, you could create a number of computed columns that extract the string/decimal values from the binary(x) field and represents them as normal columns. This would allow you to do an easy 1:1 migration while having simple and strongly typed access to the contents.
The optimal way would be to create strongly typed columns on the SQL Server database and then perform the actual migration either in COBOL or whatever script/system is used to perform the one time migration. You could still store a binary(x) to save the original value, in case a conversion error occurs, or you need to present the original value to the COBOL system.
I have a situation where I have to create tables dynamically. Depending on some criteria I am going to vary the size of the columns of a particular table.
For that purpose I need to calculate the size of one row.
e.g.
If I am going to create a following table
CREATE TABLE sample(id int, name varchar(30));
so that formula would give me the size of a single row for the table above considering all overheads for storing a row in a mysql table.
Is possible to do so and Is it feasible to do so?
It depends on the storage engine you use and the row format chosen for that table, and also your indexes. But it is not a very useful information.
Edit:
I suggest going against normalization only when you know exactly what you're doing. A DBMS is created to deal with large amount of data. You probably don't need to serialize your strctured data into a single field.
Keep in mind that your application layer then has to tokenie (or worse) the serialized field data to get the original meaning back, which has certainly larger overhead than getting the data already in a structured form, from the DB.
The only exeption I can think of is a client-heavy architcture, when moving processing to the client side actually takes burden off the server, and you would serialize our data anyway for the sake of the transfer. - In server-side code (like php) it is not a good practive to save serialized stye data into the DB.
(Though, using php's built in serialization may be a good idea in some cases. Your current project does not seem to benefit from it.)
The VARCHAR is a variable-length data type, it has a length property, but the value can be empty; calculation may be not exact. Have a look at 'Avg_row_length' field in information_schema.tables.