I'm writing a low level tool to scan and recover data from damaged Jet4 MDB files. I'm scanning through pages, parsing Rows, and decoding columns.
If I have the raw 8 byte value for Datetime fields, how can I convert this to a string representation of the date such as "MM-DD-YY HH:MM:SS"?
If I have the raw 4 byte value for a Number field with Single field size and 3 decimal places, how can I convert this value to a float/double?
Are there any documents that describe how all of the access fields are encoded and stored on disk?
Thank you.
If I have the raw 8 byte value for Datetime fields, how can I convert this to a string representation of the date such as "MM-DD-YY HH:MM:SS"?
Access stores Date/Time values as 64 bit (8 byte) Double values in little-endian format. The integer portion represents the number of days before or after the Access "epoch" (1899-12-30 00:00:00) and the absolute value of the fractional portion represents the time portion for that day (e.g., 0.5 = Noon). So, for example, in Python we would convert the bytes into a datetime value like so:
from datetime import datetime, timedelta
import struct
# bytes as retrieved from .accdb or .mdb file
d_as_bytes = b'\x35\x07\x2F\x2C\x93\x63\xDD\xC0'
d_as_double = struct.unpack('<d', d_as_bytes)[0] # -30094.29957175926
d_integer_part = int(d_as_double) # -30094
d_fractional_part = abs(d_as_double - d_integer_part) # 0.29957175926
access_epoch = datetime(1899, 12, 30)
d = access_epoch + timedelta(days=d_integer_part) + timedelta(days=d_fractional_part)
print(d) # 1817-08-08 07:11:23
No. The JET format is proprietary of Microsoft and not documented.
As for the Date data type, this is simply a double.
Neither double nor single carry a format. As you mention, these a standard floating point values.
Related
I need to store an array of unsigned TINYINT in a MySQL table for each user.
The array length is constant and I don't need to do any search or sorting in it.
Only its values are changed over time.
My goal is to have the values stored in a way that the data size remains as close as N x TINYINT for each line and hopefully readable.
I was considering 2 solutions:
Solution 1:
| user_id | TINYINT_1 | TINYINT_... | TINYINT_N |
Solution 2:
| user_id | JSON array [TINYINT_1, TINYINT_..., TINYINT_N] |
The second seems cleaner as I don't need to give N useless names, but from what I understand I have have no control on the type of value used to store data in a JSON array and I'm afraid that it increase the final memory size way more than N x TINYINT per line.
Is there way to control the type of values or some other smarter ways to do it?
Thanks for your advises.
One TINYINT takes one byte. The only way to ensure the storage of N TINYINTs is N x TINYINT bytes is to store them as a BINARY(N) up to N of 255, and BLOB if it's longer. That is, each TINYINT gets one byte in a binary string. That's not readable at all, but it is the most compact way to store it.
Because you would be responsible for interpreting this string byte-by-byte, there is no way it could be misinterpreted, or the elements treated as some other data type. That's entirely up to your code that unpacks the string.
MySQL does not have an array type (like for example PostgreSQL does).
If you want the array of TINYINT to be stored in a readable fashion, you could store it as a string of hex digits, with two digits per TINYINT. This takes exactly twice the space of the BINARY(N) solution.
You can also store the numbers as a string of comma-separated text numbers in base 10, which is more readable, but takes more space.
You can also use JSON, which allows for an array of digits, but it takes even more space because it stores numbers in base 10, and there need to be [ ] array delimiters. And you already thought of the possibility that JSON allows arbitrary types for the array elements. MySQL supports JSON schema, but not automatically. You'd have to write a CHECK constraint. There's an example here: https://dev.mysql.com/doc/refman/8.0/en/json-validation-functions.html
I am new to dynamo db binary data. I have a hash key + range key(both are byte[]). Now I am trying to get a list of items by querying on range key(ex: le, ge or between). I am able to do put and get operations fine.
However I am getting errors while doing this. My question is can dynamodb do this comparison? I am passing a byte[]. Can dynamodb check if existing rangekey(byte[]) is lesser or greater than this?
Yes, DynamoDB does support the byte array type well, and also allows comparison between them in conditions, done lexicographically, so what you want to do should and does work.
You didn't say which "errors" you are getting. You should be aware that DynamoDB treats the bytes of the byte arrays as unsigned bytes. For example, the byte 128 comes after byte 127. I don't know which language you are using to test this, but some languages have signed bytes - meaning that the byte 128 is treated as "-1" and will come before, not after, byte 127 in the sort order. DynamoDB doesn't do that, because it uses unsigned bytes.
I have a table of user entries, and for every entry I have an array of (2-byte) integers to store (15-25, sporadically even more). The array elements will be written and read all at the same time, it is never needed to update or to access them individually. Their order matters. It makes sense to think of this as an array object.
I have many millions of these user entries and want to store this with the minimum possible amount of disk space. I'm however struggling with MySQL's lack of Array datatype.
I've been considering the following options.
Do it the MySQL way. Make a table my_data with columns user_id, data_id and data_int. To make this efficient, one needs an index on user_id, totalling well over 10 bytes per integer.
Store the array in text format. This takes ~6.5 bytes per integer.
making 35-40 columns ("enough") and having -32768 be 'empty' (since this value cannot occur in my data). This takes 3.5-4 bytes per integer, but is somewhat ugly (as I have to impose a strict limit on the number of elements in the array).
Is there a better way to do this in MySQL? I know MySQL has an efficient varchar type, so ideally I'd store my 2-byte integers as 2-byte chars in a varchar (or a similar approach with blob), but I'm not sure how to do that. Is this possible? How should this be done?
You could store them as separate SMALLINT NULL columns.
In MyISAM this this uses 2 bytes of data + 1 bit of null indicator for each value.
In InnoDB, the null indicators are encoded into the column's field start offset, so they don't take any extra space, and null values are not actually stored in the row data. If the rows are small enough that all the offsets are 1 byte, then this uses 3 bytes for every existing value (1 byte offset, 2 bytes data), and 1 byte for every nonexistent value.
Either of these would be better than using INT with a special value to indicate that it doesn't exist, since that would be 4 bytes of data for every value.
See NULL in MySQL (Performance & Storage)
The best answer was given in the comments, so I'll repost it here with some use-ready code, for further reference.
MySQL has a varbinary type that works really well for this: you can simply use PHP's pack/unpack functions to convert them to and from binary form, and store that binary form in the database using varbinary. Example code for the conversion is below.
function pack24bit($n) { //input: 24-bit integer, output: binary string of length 3 bytes
$b3 = $n%256;
$b2 = $n/256;
$b1 = $b2/256;
$b2 = $b2%256;
return pack('CCC',$b1,$b2,$b3);
}
function unpack24bit($packed) { //input: binary string of 3 bytes long, output: 24-bit int
$arr = unpack('C3b',$packed);
return 256*(256*$arr['b1']+$arr['b2'])+$arr['b3'];
}
I have a task to deliver a numeric/decimal value as part of a fixed length text file. These values will take the following form:
10 chars, w/last 5 chars representing the decimal portion of the string. These will all be positive numbers.
A few examples:
0.123 = "0000012300"
1.0 = "0000100000"
123.456 = "0012345600"
234 = "0023400000"
The numeric data resides in an Access database formatted as numbers (double).
My current thought is:
Retain the orignal numeric data in one table
Convert to TEXT strings via query, save to a second table
Export to a fixed width flat file using MSAccess export function
Can anyone suggest a reasonable approach to produce the necessary 10 character TEXT conversion?
Thanks!
Perhaps just multiply by 100000 and format?
Format(x * 100000, "0000000000")
Please view for example.
I don't understand SQL Server data types. It has a nvarchar data type column.
I want to import that data in MySql.
What is this this column. Md5 or Base64 ?
Do you have an idea ?
Thanks.
As per msdn variable-length Unicode string data. n defines the string length and can be a value from 1 through 4,000. max indicates that the maximum storage size is 2^31-1 bytes (2 GB). The storage size, in bytes, is two times the actual length of data entered + 2 bytes. The ISO synonyms for nvarchar are national char varying and national character varying.
So it's a plain string datatype with Unicode support. I hope now it will be easy for you to find the matching datatype in MySQL.