How do you compress image data for LZW encoding for .GIF files? - gif

I am having trouble understanding how to compress image data for the 89a specification for .gif files. Say for example I am trying to make a 3x2 .GIF. Let me construct a sample color code table and walk through an example [of what I think is correct].
Color code | Color
------------------
0 | Brown
1 | Red
2 | Green
3 | Black
The image I want to create is this.
3x2 pixels (6 pixels total)
----------
Br Br Br
Br R Br
Compressing with LZW walks me through this process. This is the final code table I get.
Code table
----------
# | code
0 | 0
1 | 1
2 | 2
3 | 3
4 | clear
5 | eoi // end of information
6 | 0 0
7 | 0 0 0
8 | 0 1
9 | 1 0
With an eventual value of 4 0 6 0 1 0 5 that are my codes. Because I wrote out a code 0 0 0, this code value equals 7, so I had to increase my code size from 3 > 4 bits for subsequent codes. So, here are the bytes of my image data (from my code table).
100 - 4
000 - 0
110 - 6
0000 - 0
0001 - 1
0000 - 0
0101 - 5
I end up encoding my image data as
10000100 - 132
00100001 - 33
10100000 - 160
00000000 - 0
Which ends up looking like this in my final .gif file (I've put brackets around the values that correspond to the image data)
47 49 46 38 39 61 03 00 02 00 f1 00 00 b9 7a 56
ff 00 00 00 ff 00 00 00 00 21 ff 0b 4e 45 54 53
43 41 50 45 32 2e 30 03 01 ff ff 00 21 f9 04 04
64 00 00 00 2c 00 00 00 00 03 00 02 00 00 [02 04
84 21 a0 00 00] 3b
// Explanation
02 - Minimum LZW code size
04 - Data sub-block of 4 bytes
84 - 132 in decimal
21 - 33 in decimal
a0 - 160 in decimal
00 - 0 in decimal
00 - Termination byte
My image looks something like this (why is there green in here instead of red?). I blew the image up since 2x3 pixels is a bit hard to read.
Is there something fundamental that I am missing? I appreciate your time to look at this with me.

Found the error, it lies in the code size when compressing LZW image data.
When you are creating the code table when compressing image data with LZW, you need to increment your code size when you've added a code that equals to 2^(code size). So, instead of incrementing the code size by one after adding code 7 | 0 0 0 (as shown in the table above), I needed to instead increment the code size by one after adding 8 | 0 1 (because 8 = 2^(code size == 3)).
This is how the image data changes by incrementing the code size as described
100 - 4
000 - 0
110 - 6
000 - 0
0001 - 1
0000 - 0
0101 - 5
And then, how the resulting image data bytes has changed.
10000100 - 132
00010001 - 17
01010000 - 80
I've put brackets around the data to show a comparison from the full .gif data to show what has changed (after applying the fix). This is the same .gif file from above.
47 49 46 38 39 61 03 00 02 00 f1 00 00 b9 7a 56
ff 00 00 00 ff 00 00 00 00 21 ff 0b 4e 45 54 53
43 41 50 45 32 2e 30 03 01 ff ff 00 21 f9 04 04
64 00 00 00 2c 00 00 00 00 03 00 02 00 00 [02 03
84 11 50 00] 3b
// Explanation
02 - Minimum LZW code size
03 - Data sub-block of 3 bytes
84 - 132 in decimal
11 - 17 in decimal
50 - 80 in decimal
00 - Termination byte

Related

Reverse CRC16 calculation

I'm trying to understand how is calculated the CRC at the end of a radio packet.
Here are a few examples:
11 00 01 0D 30 10 05 1F 11 ED 7E 01 00 01 B9 33
11 00 01 0D 30 10 05 1F 11 ED 7E 01 00 00 B9 32
11 00 01 1D 30 10 05 1F 11 ED 7E 01 00 00 EA CC
11 00 01 2D 30 10 05 1F 11 ED 7E 01 00 00 1E CE
The 4th byte is a sequence number. All other bytes are constant. The last 2 bytes definitely look like a CRC16, as these are the only ones changing when the sequence byte increases. The last 2 bytes are not related to the time, as I can reproduce that exact same sequence anytime.
Here are a few more examples, from the same device but with a different command:
16 00 01 60 20 10 05 1F 11 ED 7E 01 02 00 04 00 02 00 65 32 CC
16 00 01 CB 20 10 31 53 11 ED 7E 01 42 00 04 00 02 00 65 B4 B9
This time again, the last 2 bytes look like a CRC16.
I've tried many CRC calculations, using online calculators like crccalc.com.
I've also used the RevEng tool, but got no results.
I can't figure out the method of calculation, so I must be missing something.
Any help to determine the calculation would be welcome.
Thanks!
It is the CRC-16/XMODEM, computed on your examples with the first three bytes and the last two bytes before the CRC removed, and then, oddly, that CRC exclusive-or'ed with the two bytes that precede it (those that were excluded from the CRC calculation). The resulting 16-bit value is appended in big-endian order.

SQL split hexastring in different bytes

I've got a google bigquery looking like this:
#standardSQL
SELECT
timestamp,
CAN_Frame,
TRIM(SPLIT(CAN_Frame)[OFFSET(4)]) AS bytes
FROM
`data.source`
WHERE
LENGTH(CAN_Frame) > 1 and
SUBSTR(TRIM(SPLIT(CAN_Frame)[OFFSET(4)]),1,2) IN ('83', '84')
ORDER BY
timestamp DESC
LIMIT
8000
or like this
#standardSQL
SELECT
*
FROM (
SELECT
timestamp,
CAN_Frame,
REGEXP_EXTRACT(CAN_Frame, r', ([^,]+)$') AS bytes_string,
FROM_HEX(REPLACE(REGEXP_EXTRACT(CAN_Frame, r', ([^,]+)$'), ' ', '')) AS
bytes
FROM `data.source`
)
WHERE SUBSTR(bytes, 1, 1) IN (b'\x83', b'\x84')
ORDER BY timestamp DESC
LIMIT 8000
with the resulting tables:
Row timestamp CAN_Frame bytes
1 2017-09-29 14:31:02 UTC S,48778,410,8, 84 10 00 25 00 21 00 4F 84 10 00
25 00 21 00 4F
2 2017-09-29 14:30:42 UTC S,35847,480,8, 83 80 00 01 00 03 00 0D 83 80 00
01 00 03 00 0D
3 2017-09-29 14:30:40 UTC S,34612,4B2,8, 84 B2 00 27 00 08 00 03 84 B2 00
27 00 08 00 03
or
Row timestamp CAN_Frame bytes_string bytes
1 2017-09-29 14:31:02 UTC S,48778,410,8, 84 10 00 25 00 21 00 4F 84 10 00 25 00 21 00 4F hBAAJQAhAE8=
2 2017-09-29 14:30:42 UTC S,35847,480,8, 83 80 00 01 00 03 00 0D 83 80 00 01 00 03 00 0D g4AAAQADAA0=
3 2017-09-29 14:30:40 UTC S,34612,4B2,8, 84 B2 00 27 00 08 00 03 84 B2 00 27 00 08 00 03 hLIAJwAIAAM=
4 2017-09-29 14:30:39 UTC S,34314,4C0,8, 84 C0 00 1C 00 15 00 07 84 C0 00 1C 00 15 00 07 hMAAHAAVAAc=
My problem and question is now how to split the 8 byte hexa string in a way that let me have the 6th and 7th byte of string beginning with 83, the 8th byte from 83 and the 3rd byte from 84 and the 4th and 5th byte of the string beginning with 84. these datapairs are values with lsb msb in unsigned int that i need to read.
i hope somebody can help me or at least understand my problem.
best regards
#standardSQL
WITH `data.source` AS (
SELECT 'S,0,2B3,8, C2 B3 00 00 00 00 03 DE' AS frame UNION ALL
SELECT 'S,0,3FA,6, 00 E0 04 A5 00 0B' UNION ALL
SELECT 'S,0,440,8, 83 40 4E A5 00 47 00 64' UNION ALL
SELECT 'S,0,450,8, 84 50 01 12 01 19 01 B3' UNION ALL
SELECT 'S,0,4B0,8, 84 B0 4E A5 00 43 00 64'
)
SELECT
frame, bytes, STRING_AGG(b, ' ' ORDER BY p) AS selected_bytes
FROM (
SELECT frame, TRIM(SPLIT(frame)[OFFSET(4)]) AS bytes, SUBSTR(TRIM(SPLIT(frame)[OFFSET(4)]), 1, 2) AS f
FROM `data.source`
WHERE SUBSTR(TRIM(SPLIT(frame)[OFFSET(4)]), 1, 2) IN ('83', '84')
), UNNEST(SPLIT(bytes, ' ')) AS b WITH OFFSET AS p
WHERE CASE f WHEN '83' THEN p IN (5, 6, 7) WHEN '84' THEN p IN (2, 3, 4) END
GROUP BY frame, bytes
-- ORDER BY frame
result is:
frame bytes selected_bytes
S,0,440,8, 83 40 4E A5 00 47 00 64 83 40 4E A5 00 47 00 64 47 00 64
S,0,450,8, 84 50 01 12 01 19 01 B3 84 50 01 12 01 19 01 B3 01 12 01
S,0,4B0,8, 84 B0 4E A5 00 43 00 64 84 B0 4E A5 00 43 00 64 4E A5 00
Update for:
byte 6 and 7 from the string beginning with 83 called Aiout
one column contains byte 8 from the string beginning with 83 and byte 3 from the string beginning with 84 called Biout
one column contains byte 4 and 5 from the string beginning with 84 called Avout
Bvout containing byte 6 and 7 from the string beginning with 84
#standardSQL
WITH `data.source` AS (
SELECT 'S,0,2B3,8, C2 B3 00 00 00 00 03 DE' AS frame UNION ALL
SELECT 'S,0,3FA,6, 00 E0 04 A5 00 0B' UNION ALL
SELECT 'S,0,440,8, 83 40 4E A5 00 47 00 64' UNION ALL
SELECT 'S,0,450,8, 84 50 01 12 01 19 01 B3' UNION ALL
SELECT 'S,0,4B0,8, 84 B0 4E A5 00 43 00 64'
)
SELECT
frame, bytes,
STRING_AGG(CASE WHEN f='83' AND p IN (5, 6) THEN b ELSE '' END, ' ' ORDER BY p) AS Aiout,
STRING_AGG(CASE WHEN (f='83' AND p=7) OR (f='84' AND p=2) THEN b ELSE '' END, ' ' ORDER BY p) AS Biout,
STRING_AGG(CASE WHEN f='84' AND p IN (3, 4) THEN b ELSE '' END, ' ' ORDER BY p) AS Avout,
STRING_AGG(CASE WHEN f='84' AND p IN (5, 6) THEN b ELSE '' END, ' ' ORDER BY p) AS Bvout
FROM (
SELECT frame, TRIM(SPLIT(frame)[OFFSET(4)]) AS bytes, SUBSTR(TRIM(SPLIT(frame)[OFFSET(4)]), 1, 2) AS f
FROM `data.source`
WHERE SUBSTR(TRIM(SPLIT(frame)[OFFSET(4)]), 1, 2) IN ('83', '84')
), UNNEST(SPLIT(bytes, ' ')) AS b WITH OFFSET AS p
GROUP BY frame, bytes
ORDER BY frame
with output as
frame bytes Aiout Biout Avout Bvout
S,0,440,8, 83 40 4E A5 00 47 00 64 83 40 4E A5 00 47 00 64 47 00 64
S,0,450,8, 84 50 01 12 01 19 01 B3 84 50 01 12 01 19 01 B3 01 12 01 19 01
S,0,4B0,8, 84 B0 4E A5 00 43 00 64 84 B0 4E A5 00 43 00 64 4E A5 00 43 00

pep8 binary to decimal

I got stuck on this assignment:
Write a program in object code that will convert a 3-bit binary number to a decimal number. For example, if input is 101, output should be 5; if input is 011, output should be 3.
This is my code so far:
49 00 50 - Input char 1
49 00 51 - Input char 2
49 00 53 - Input char 3
c0 00 00 - 0 memory
d1 00 50 - Load char 1
80 00 48 - subtract 48 cose 0 is 48 in dec
1c - shift to left
1c - shift to left again cose 3rd position in binary is x4
e1 00 60 - store int 1
d1 00 51 - load char 2
80 00 48 - subtract 48 cose 0 is 48 in dec
1c - shift to left cose 2nd position in binary is x2
e1 00 62 - store int 2
d1 00 53 - load char 3
80 00 48 - subtract 48 cose 0 is 48 in dec
71 00 60 - add int 1
71 00 62 - add int 2
f1 00 64 - store char
51 00 64 - output character
00
zz
Can someone help me and guide me how to do this, keep in mind that I just started with pep8 2 weeks ago.
Thank you
04 00 06 00 00 00 49 00 03 D1 00 03 80 00 30 1C
1C E1 00 04 49 00 03 D1 00 03 80 00 30 1C 71 00
04 E1 00 04 49 00 03 D1 00 03 80 00 30 71 00 04
E1 00 04 39 00 04 00 zz
This code is written for only 3 bits binary values

MySQL - Escaping ampersand (&) in fulltext searches

We are using a fulltext search to search for the name of a company and all is going well until we have a company with an ampersand in its name, e.g. 'M&S'.
SELECT name FROM company WHERE MATCH (name) against ('M&S' IN BOOLEAN MODE);
This fails to return any results as MySQL is treating the ampersand as a boolean operator. The boolean mode is desired so it can't simply be turned off.
What I'm looking for is a way to escape the ampersand so that MySQL treats it correctly and finds the record.
Ditching fulltext search in favour of LIKEs isn't exactly an option either
Thanks for your help
Seems like & isn't considered a word character in the collation you use for your fulltext search.
so you have to create your own collation (or recompile your MySQL server) where you add & to the list of word characters like i found out in the MySQL docs (
http://dev.mysql.com/doc/refman/5.0/en/fulltext-fine-tuning.html) :
If you want to change the set of characters that are considered word
characters, you can do so in several ways, as described in the
following list. After making the modification, you must rebuild the
indexes for each table that contains any FULLTEXT indexes. Suppose
that you want to treat the hyphen character ('-') as a word character.
Use one of these methods:
Modify the MySQL source: In myisam/ftdefs.h, see the true_word_char()
and misc_word_char() macros. Add '-' to one of those macros and
recompile MySQL.
Modify a character set file: This requires no recompilation. The
true_word_char() macro uses a “character type” table to distinguish
letters and numbers from other characters. . You can edit the contents
of the array in one of the character set XML files to
specify that '-' is a “letter.” Then use the given character set for
your FULLTEXT indexes. For information about the array
format, see Section 10.3.1, “Character Definition Arrays”.
Add a new collation for the character set used by the indexed columns,
and alter the columns to use that collation. For general information
about adding collations, see Section 10.4, “Adding a Collation to a
Character Set”. For an example specific to full-text indexing, see
Section 12.9.7, “Adding a Collation for Full-Text Indexing”.
UPDATE: in case you are using latin1 collation, open your XML file which is at mysql/share/charsets/latin1.xml. and find the corresponding character code in a map - in this case you can take the map for lower case or upper case because this doesn't matter for the ampersand symbol:
<lower>
<map>
00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F
20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F
30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F
40 61 62 63 64 65 66 67 68 69 6A 6B 6C 6D 6E 6F
70 71 72 73 74 75 76 77 78 79 7A 5B 5C 5D 5E 5F
60 61 62 63 64 65 66 67 68 69 6A 6B 6C 6D 6E 6F
70 71 72 73 74 75 76 77 78 79 7A 7B 7C 7D 7E 7F
80 81 82 83 84 85 86 87 88 89 8A 8B 8C 8D 8E 8F
90 91 92 93 94 95 96 97 98 99 9A 9B 9C 9D 9E 9F
A0 A1 A2 A3 A4 A5 A6 A7 A8 A9 AA AB AC AD AE AF
B0 B1 B2 B3 B4 B5 B6 B7 B8 B9 BA BB BC BD BE BF
E0 E1 E2 E3 E4 E5 E6 E7 E8 E9 EA EB EC ED EE EF
F0 F1 F2 F3 F4 F5 F6 D7 F8 F9 FA FB FC FD FE DF
E0 E1 E2 E3 E4 E5 E6 E7 E8 E9 EA EB EC ED EE EF
F0 F1 F2 F3 F4 F5 F6 F7 F8 F9 FA FB FC FD FE FF
</map>
</lower>
the ampersand's unicode is U+0026 and in utf-8 encoding it's 0x26, so search for 26 in the map - which is in the 3rd row, 7th column.
then in the ctype-map change the type of the character from 10 which means punctuation to 01 which means small letter:
<ctype>
<map>
00
20 20 20 20 20 20 20 20 20 28 28 28 28 28 20 20
20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
48 10 10 10 10 10 01 10 10 10 10 10 10 10 10 10
84 84 84 84 84 84 84 84 84 84 10 10 10 10 10 10
10 81 81 81 81 81 81 01 01 01 01 01 01 01 01 01
01 01 01 01 01 01 01 01 01 01 01 10 10 10 10 10
10 82 82 82 82 82 82 02 02 02 02 02 02 02 02 02
02 02 02 02 02 02 02 02 02 02 02 10 10 10 10 20
10 00 10 02 10 10 10 10 10 10 01 10 01 00 01 00
00 10 10 10 10 10 10 10 10 10 02 10 02 00 02 01
48 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10
10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10
01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01
01 01 01 01 01 01 01 10 01 01 01 01 01 01 01 02
02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02
02 02 02 02 02 02 02 10 02 02 02 02 02 02 02 02
</map>
</ctype>
restart your MySQL server and the corresponding collation is handling & like it was a small letter.
of course it's better to first copy and rename your new collation XML-file and to also copy and paste the corresponding lines in the Index.xml (don't forget to use a new unused id in the XML tags there) and link them to your new collation XML-file so you don't lose your original collation.
you can find the full documentation where i got most of the information from here:
http://dev.mysql.com/doc/refman/5.0/en/full-text-adding-collation.html
Note - For all those working with Mysql 5.7 version use an unused collation id. The mysql article http://dev.mysql.com/doc/refman/5.0/en/fulltext-fine-tuning.html is for Mysql 5.5 version. To get maximum collation Id use following Query -
SELECT MAX(ID) FROM INFORMATION_SCHEMA.COLLATIONS;
EDIT: so the & is splitting it into two separate words... since they are 1 letter it is not returning anything. I tested with "Ma&Sa".. my ft_min_word_len = 4... and it didn't return anything so since the length of that string > 4 but its not returning it has to be splitting it into two words... it looks like the suggestion northkildonan made is what you have to do.
So this may or may not be an answer.. but I hope it is helpful for figuring this out.. try this.
first: run this statement -- SHOW VARIABLES LIKE 'ft_min_word_len'; and affirm that the length is actually = 2
if it is i'm not sure how it is any different than a word that is longer than a length of 4
Second: I did this and got results.
SET UP:
I set up a sample table on my localhost database...
create table company(
`id` int,
`name` varchar(55)
);
insert into company
(`id`, `name`)
values
(1, 'oracle'),
(2, 'microsoft'),
(3, 'M&S'),
(4, 'dell');
TESTS:
tested when ft_min_word_len = 4 and obviously it didn't return anything.
SELECT `name` FROM company WHERE MATCH (`name`) against ("M&S" IN BOOLEAN MODE);
I didn't want to try restarting my localhost database to reset the length to 2 (incase I accidentally mess something up because I use it a lot)..
but I got the idea of trying to look for the name of a company that was longer than a length of 4 with the & in it.
MORE SETUP:
insert into company
(`id`, `name`)
values
(5, 'Mary&Sasha');
ANOTHER TEST:
SELECT `name` FROM company WHERE MATCH (`name`) against ("Mary&Sasha" IN BOOLEAN MODE);
this returned http://screencast.com/t/Rx8mh98OUp
I also did this just incase the collation was messing it up but I doubt that was the problem..
COLLATION STUFF:
ALTER TABLE company MODIFY
`name` VARCHAR(55)
CHARACTER SET latin1
COLLATE latin1_german2_ci;
you can also check your tables collation with:
SHOW TABLE STATUS;
hope this is at least some help :)
& is not a special character in mysql therefore you are able to store and search for the expression &
you can test that as followed
SELECT name FROM `testing` WHERE name LIKE '%&%'
also please try somthing like the following to replace the &.
SET #searchstring = 'M&S';
SET #searchstring = REPLACE(#searchstring,'&','&');
SELECT name FROM company WHERE MATCH (name) against (#searchstring IN BOOLEAN MODE);
You may also take a look at regexp.
http://dev.mysql.com/doc/refman/5.1/en/regexp.html
Here the & is used as followed.
mysql> SELECT '&' REGEXP '[[.ampersand.]]';
The following query is also getting you the result
SELECT *
FROM `testing`
WHERE `name` REGEXP CONVERT( _utf8 'M&S'
USING latin1 ) COLLATE latin1_german2_ci
LIMIT 0 , 30
please also read this thread, maybe you can understand it better then me. This is SQL but they seem to have solved the problem
http://forums.asp.net/t/1073707.aspx?Full+text+search+and+sepcial+characters+like+ampersand+
sorry I couldn´t help more

It's ok to use HEX as BCD function?

I'm wondering if it's ok to use HEX as BCD function? and for decoding we can take each character and decode it as decimal or byte..
Hex and BCD are to different things. One is numbers base 16 and the other base ten. What may confuse is you can see ether one with hexa representation of memory.
Here is an example of memory representation:
004044FC : 33 31 1E 00 - 33 31 1E 00 - 32 38 1B 00 - 33 31 1E 00
0040450C : 33 30 1D 00 - 33 31 1E 00 - 33 30 1D 00 - 33 31 1E 00
0040451C : 33 31 1E 00 - 33 30 1D 00 - 48 65 6C 6C - 6F 20 54 68
0040452C : 69 73 20 69 - 73 20 74 68 - 65 20 74 65 - 73 74 20 6F