I'm trying to understand how is calculated the CRC at the end of a radio packet.
Here are a few examples:
11 00 01 0D 30 10 05 1F 11 ED 7E 01 00 01 B9 33
11 00 01 0D 30 10 05 1F 11 ED 7E 01 00 00 B9 32
11 00 01 1D 30 10 05 1F 11 ED 7E 01 00 00 EA CC
11 00 01 2D 30 10 05 1F 11 ED 7E 01 00 00 1E CE
The 4th byte is a sequence number. All other bytes are constant. The last 2 bytes definitely look like a CRC16, as these are the only ones changing when the sequence byte increases. The last 2 bytes are not related to the time, as I can reproduce that exact same sequence anytime.
Here are a few more examples, from the same device but with a different command:
16 00 01 60 20 10 05 1F 11 ED 7E 01 02 00 04 00 02 00 65 32 CC
16 00 01 CB 20 10 31 53 11 ED 7E 01 42 00 04 00 02 00 65 B4 B9
This time again, the last 2 bytes look like a CRC16.
I've tried many CRC calculations, using online calculators like crccalc.com.
I've also used the RevEng tool, but got no results.
I can't figure out the method of calculation, so I must be missing something.
Any help to determine the calculation would be welcome.
Thanks!
It is the CRC-16/XMODEM, computed on your examples with the first three bytes and the last two bytes before the CRC removed, and then, oddly, that CRC exclusive-or'ed with the two bytes that precede it (those that were excluded from the CRC calculation). The resulting 16-bit value is appended in big-endian order.
I've got a google bigquery looking like this:
#standardSQL
SELECT
timestamp,
CAN_Frame,
TRIM(SPLIT(CAN_Frame)[OFFSET(4)]) AS bytes
FROM
`data.source`
WHERE
LENGTH(CAN_Frame) > 1 and
SUBSTR(TRIM(SPLIT(CAN_Frame)[OFFSET(4)]),1,2) IN ('83', '84')
ORDER BY
timestamp DESC
LIMIT
8000
or like this
#standardSQL
SELECT
*
FROM (
SELECT
timestamp,
CAN_Frame,
REGEXP_EXTRACT(CAN_Frame, r', ([^,]+)$') AS bytes_string,
FROM_HEX(REPLACE(REGEXP_EXTRACT(CAN_Frame, r', ([^,]+)$'), ' ', '')) AS
bytes
FROM `data.source`
)
WHERE SUBSTR(bytes, 1, 1) IN (b'\x83', b'\x84')
ORDER BY timestamp DESC
LIMIT 8000
with the resulting tables:
Row timestamp CAN_Frame bytes
1 2017-09-29 14:31:02 UTC S,48778,410,8, 84 10 00 25 00 21 00 4F 84 10 00
25 00 21 00 4F
2 2017-09-29 14:30:42 UTC S,35847,480,8, 83 80 00 01 00 03 00 0D 83 80 00
01 00 03 00 0D
3 2017-09-29 14:30:40 UTC S,34612,4B2,8, 84 B2 00 27 00 08 00 03 84 B2 00
27 00 08 00 03
or
Row timestamp CAN_Frame bytes_string bytes
1 2017-09-29 14:31:02 UTC S,48778,410,8, 84 10 00 25 00 21 00 4F 84 10 00 25 00 21 00 4F hBAAJQAhAE8=
2 2017-09-29 14:30:42 UTC S,35847,480,8, 83 80 00 01 00 03 00 0D 83 80 00 01 00 03 00 0D g4AAAQADAA0=
3 2017-09-29 14:30:40 UTC S,34612,4B2,8, 84 B2 00 27 00 08 00 03 84 B2 00 27 00 08 00 03 hLIAJwAIAAM=
4 2017-09-29 14:30:39 UTC S,34314,4C0,8, 84 C0 00 1C 00 15 00 07 84 C0 00 1C 00 15 00 07 hMAAHAAVAAc=
My problem and question is now how to split the 8 byte hexa string in a way that let me have the 6th and 7th byte of string beginning with 83, the 8th byte from 83 and the 3rd byte from 84 and the 4th and 5th byte of the string beginning with 84. these datapairs are values with lsb msb in unsigned int that i need to read.
i hope somebody can help me or at least understand my problem.
best regards
#standardSQL
WITH `data.source` AS (
SELECT 'S,0,2B3,8, C2 B3 00 00 00 00 03 DE' AS frame UNION ALL
SELECT 'S,0,3FA,6, 00 E0 04 A5 00 0B' UNION ALL
SELECT 'S,0,440,8, 83 40 4E A5 00 47 00 64' UNION ALL
SELECT 'S,0,450,8, 84 50 01 12 01 19 01 B3' UNION ALL
SELECT 'S,0,4B0,8, 84 B0 4E A5 00 43 00 64'
)
SELECT
frame, bytes, STRING_AGG(b, ' ' ORDER BY p) AS selected_bytes
FROM (
SELECT frame, TRIM(SPLIT(frame)[OFFSET(4)]) AS bytes, SUBSTR(TRIM(SPLIT(frame)[OFFSET(4)]), 1, 2) AS f
FROM `data.source`
WHERE SUBSTR(TRIM(SPLIT(frame)[OFFSET(4)]), 1, 2) IN ('83', '84')
), UNNEST(SPLIT(bytes, ' ')) AS b WITH OFFSET AS p
WHERE CASE f WHEN '83' THEN p IN (5, 6, 7) WHEN '84' THEN p IN (2, 3, 4) END
GROUP BY frame, bytes
-- ORDER BY frame
result is:
frame bytes selected_bytes
S,0,440,8, 83 40 4E A5 00 47 00 64 83 40 4E A5 00 47 00 64 47 00 64
S,0,450,8, 84 50 01 12 01 19 01 B3 84 50 01 12 01 19 01 B3 01 12 01
S,0,4B0,8, 84 B0 4E A5 00 43 00 64 84 B0 4E A5 00 43 00 64 4E A5 00
Update for:
byte 6 and 7 from the string beginning with 83 called Aiout
one column contains byte 8 from the string beginning with 83 and byte 3 from the string beginning with 84 called Biout
one column contains byte 4 and 5 from the string beginning with 84 called Avout
Bvout containing byte 6 and 7 from the string beginning with 84
#standardSQL
WITH `data.source` AS (
SELECT 'S,0,2B3,8, C2 B3 00 00 00 00 03 DE' AS frame UNION ALL
SELECT 'S,0,3FA,6, 00 E0 04 A5 00 0B' UNION ALL
SELECT 'S,0,440,8, 83 40 4E A5 00 47 00 64' UNION ALL
SELECT 'S,0,450,8, 84 50 01 12 01 19 01 B3' UNION ALL
SELECT 'S,0,4B0,8, 84 B0 4E A5 00 43 00 64'
)
SELECT
frame, bytes,
STRING_AGG(CASE WHEN f='83' AND p IN (5, 6) THEN b ELSE '' END, ' ' ORDER BY p) AS Aiout,
STRING_AGG(CASE WHEN (f='83' AND p=7) OR (f='84' AND p=2) THEN b ELSE '' END, ' ' ORDER BY p) AS Biout,
STRING_AGG(CASE WHEN f='84' AND p IN (3, 4) THEN b ELSE '' END, ' ' ORDER BY p) AS Avout,
STRING_AGG(CASE WHEN f='84' AND p IN (5, 6) THEN b ELSE '' END, ' ' ORDER BY p) AS Bvout
FROM (
SELECT frame, TRIM(SPLIT(frame)[OFFSET(4)]) AS bytes, SUBSTR(TRIM(SPLIT(frame)[OFFSET(4)]), 1, 2) AS f
FROM `data.source`
WHERE SUBSTR(TRIM(SPLIT(frame)[OFFSET(4)]), 1, 2) IN ('83', '84')
), UNNEST(SPLIT(bytes, ' ')) AS b WITH OFFSET AS p
GROUP BY frame, bytes
ORDER BY frame
with output as
frame bytes Aiout Biout Avout Bvout
S,0,440,8, 83 40 4E A5 00 47 00 64 83 40 4E A5 00 47 00 64 47 00 64
S,0,450,8, 84 50 01 12 01 19 01 B3 84 50 01 12 01 19 01 B3 01 12 01 19 01
S,0,4B0,8, 84 B0 4E A5 00 43 00 64 84 B0 4E A5 00 43 00 64 4E A5 00 43 00
My program was able to craft and send raw transactions to geth v1.3.3 before, but after I upgrade to geth v1.4.0, calling sendRawTransaction over RPC always returns invalid sender error.
Is transaction serialization (i.e. RLP) changed somehow from v1.3.3 to v1.4.0? Here is a dump of by raw transaction that triggers an invalid user error:
0x0000: F8 CA 80 85 0B A4 3B 74 00 83 01 5F 90 94 08 BE ......;t..._....
0x0010: 24 CD 8D CF 73 F8 FA 5D B4 2B 85 5B 43 70 BD 5C $...s..].+.[Cp.\
0x0020: 44 8B 80 B8 64 B0 70 B9 BA 00 00 00 00 00 00 00 D...d.p.........
0x0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x0040: 00 00 00 00 00 00 00 00 01 87 44 2E B8 96 6A 07 ..........D...j.
0x0050: 0C 31 C1 E8 AE A3 60 F5 35 32 47 81 13 34 31 D4 .1....`.52G..41.
0x0060: 4B FA 0A 0B 1B 9F 13 C6 F5 00 00 00 00 00 00 00 K...............
0x0070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x0080: 00 00 00 00 00 00 00 00 00 1B A0 DE A4 6B 8C E8 .............k..
0x0090: 72 5A 31 49 92 EC 6B 6F C6 89 8C BB D7 A4 B9 8A rZ1I..ko........
0x00A0: 10 D2 F7 9E CE 6B D5 0F C5 19 E9 A0 8F 74 57 C2 .....k.......tW.
0x00B0: 1C DA CB 7D 7A 2B 46 58 98 53 31 C3 4B CF 50 1F ...}z+FX.S1.K.P.
0x00C0: 17 CE 16 80 95 30 38 9B 98 3C 5B B8 .....08..<[.
A more machine readable version of my transaction is:
F8CA80850BA43B740083015F909408BE24CD8DCF73F8FA5DB42B855B4370BD5C448B80B864B070B9BA000000000000000000000000000000000000000000000000000000000000000187442EB8966A070C31C1E8AEA360F535324781133431D44BFA0A0B1B9F13C6F500000000000000000000000000000000000000000000000000000000000000001BA0DEA46B8CE8725A314992EC6B6FC6898CBBD7A4B98A10D2F79ECE6BD50FC519E9A08F7457C21CDACB7D7A2B4658985331C34BCF501F17CE16809530389B983C5BB8
Log from geth gives
I0504 20:22:27.392581 9768 types.go:106] Generated response: *shared.ErrorResponse &{%!s(float64=1) 2.0 %!s(*shared.ErrorObject=&{-32603 Invalid sender})}
I0504 20:22:27.392886 9768 http.go:157] Sending payload: {
"id": 1,
"jsonrpc": "2.0",
"error": {
"code": -32603,
"message": "Invalid sender"
}
}
I believe the JSON RPC stuff changed in geth v1.4.0. I can't tell why this is happening without seeing the full sendRawTransaction you are calling but check out the docs: https://github.com/ethereum/wiki/wiki/JSON-RPC#eth_sendtransaction
Also, this should be migrated to https://ethereum.stackexchange.com/
I understand that PK is part of every secondary index in InnoDB, but what about this situation:
I have table structure with two columns
a,b
I choose PK as a+b
and create INDEX on b
Question is, how looks structure on the secondary index ?
Is it stored as
b a
or
b a+b
Or another, more detailed example:
table: street , city , state , zip , fk_customers
PK: fk_customers + street + city + zip
INDEX (city)
INDEX (street)
Are the secondary indexes stored like this:
city fk_customers + street + zip
street fk_customers + city + zip
or like this:
city fk_customers + street + city + zip
street fk_customers + street + city + zip
That's easy to verify. My gut feeling told the secondary key on b would be b a b, but InnoDB is smart enough to omit the redundant b:
mysql> create table t1 (a varchar(32), b varchar(32), primary key (a,b));
mysql> alter table add index(b);
mysql> insert into t1 values('aaa','bbb');
Query OK, 1 row affected (0.00 sec)
Here's a hexdump of a secondary index page:
00004000 07 11 7d 29 00 00 00 04 ff ff ff ff ff ff ff ff |..})............|
00004010 00 00 00 00 ff ee cf d5 45 bf 00 00 00 00 00 00 |........E.......|
00004020 00 00 00 00 09 31 00 02 00 85 80 03 00 00 00 00 |.....1..........|
00004030 00 7f 00 05 00 00 00 01 00 00 00 00 00 01 e0 7e |...............~|
00004040 00 00 00 00 00 00 00 00 18 1a 00 00 09 31 00 00 |.............1..|
00004050 00 02 02 72 00 00 09 31 00 00 00 02 01 b2 01 00 |...r...1........|
00004060 02 00 1c 69 6e 66 69 6d 75 6d 00 02 00 0b 00 00 |...infimum......|
00004070 73 75 70 72 65 6d 75 6d 03 03 00 00 10 ff f1 62 |supremum.......b|
00004080 62 62 61 61 61 00 00 00 00 00 00 00 00 00 00 00 |bbaaa...........|
00004090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
As you see, column a goes right after key column b. There is no trailing b
The answer above by akuzminsky is correct, but to add a bit more discussion: This is actually discussed in my blog post "The physical structure of records in InnoDB" under the heading "Secondary indexes":
For example, if a table has a PRIMARY KEY (a, b, c) and a secondary index KEY (a, d), the secondary key in the index will be as expected, (a, d) but the PKVs will contain only (b, c).
You can also confirm this by creating a table and using the page-dump or record-dump modes along with --trace to see the actual bytes encountered when reading the records
I have a binary file that I want to open as (or to convert to) jpeg image.
The file is only supposed by me to be a jpeg within a binary format, therefore I include a part of it so that You can tell me if I'm wrong ('cause unfortunately I never saw one of those before):
0000000000 0E 03 13 01 00 10 00 00 EC B0 00 1E 00 01 00 00 [................]
0000000016 00 CA 00 00 00 5C 00 0F 00 01 00 00 01 26 00 00 [.....\.......&..]
0000000032 00 00 01 2F 00 01 00 00 01 26 00 00 83 B2 00 6A [.../.....&.....j]
0000000048 00 01 00 00 84 D8 00 00 00 04 01 2C 00 01 00 00 [...........,....]
0000000064 84 DC 00 00 00 14 01 32 00 01 00 00 84 F0 00 00 [.......2........]
0000000080 00 08 07 AD 00 02 00 00 84 F8 00 00 00 19 00 0F [................]
0000000096 00 02 00 00 85 11 00 00 00 00 01 2F 00 02 00 00 [.........../....]
0000000112 85 11 00 00 0D D3 01 2C 00 02 00 00 92 E4 00 00 [.......,........]
0000000128 00 14 01 32 00 02 00 00 92 F8 00 00 00 08 07 AD [...2............]
0000000144 00 03 00 00 93 00 00 00 00 19 00 0F 00 03 00 00 [................]
0000000160 93 19 00 00 00 00 01 2F 00 03 00 00 93 19 00 00 [......./........]
0000000176 59 7B 01 2C 00 03 00 00 EC 94 00 00 00 14 01 32 [Y{.,...........2]
0000000192 00 03 00 00 EC A8 00 00 00 08 00 00 00 04 00 00 [................]
0000000208 00 02 00 00 00 00 4E 43 53 41 20 48 44 46 20 56 [......NCSA HDF V]
0000000224 65 72 73 69 6F 6E 20 34 2E 32 20 52 65 6C 65 61 [ersion 4.2 Relea]
0000000240 73 65 20 30 2C 20 44 65 63 65 6D 62 65 72 20 32 [se 0, December 2]
0000000256 2C 20 32 30 30 33 00 00 00 00 00 00 00 00 00 00 [, 2003..........]
0000000272 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [................]
0000000288 00 00 00 00 00 00 FF D8 FF E0 00 10 4A 46 49 46 [............JFIF]
0000000304 00 01 01 00 00 01 00 01 00 00 FF DB 00 43 00 10 [.............C..]
0000000320 0B 0C 0E 0C 0A 10 0E 0D 0E 12 11 10 13 18 28 1A [..............(.]
0000000336 18 16 16 18 31 23 25 1D 28 3A 33 3D 3C 39 33 38 [....1#%.(:3=<938]
0000000352 37 40 48 5C 4E 40 44 57 45 37 38 50 6D 51 57 5F [7#H\N#DWE78PmQW_]
0000000368 62 67 68 67 3E 4D 71 79 70 64 78 5C 65 67 63 FF [bghg>Mqypdx\egc.]
0000000384 DB 00 43 01 11 12 12 18 15 18 2F 1A 1A 2F 63 42 [..C......./../cB]
0000000400 38 42 63 63 63 63 63 63 63 63 63 63 63 63 63 63 [8Bcccccccccccccc]
0000000416 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 [cccccccccccccccc]
0000000432 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 63 [cccccccccccccccc]
0000000448 63 63 63 63 FF C0 00 11 08 01 A0 01 C0 03 01 22 [cccc..........."]
0000000464 00 02 11 01 03 11 01 FF C4 00 1F 00 00 01 05 01 [................]
0000000480 01 01 01 01 01 00 00 00 00 00 00 00 00 01 02 03 [................]
0000000496 04 05 06 07 08 09 0A 0B FF C4 00 B5 10 00 02 01 [................]
How can I convert it to jpeg to view its content (that is supposed to be an image)?
Is this conversion necessary or would You reccomend a software that can open it as I like?
I hope someone could help me, thanks in advance.
ANSWER
(As I can't answer my own question right now, I'm doing it here...)
At the end I came up with a solution thanks to the input information of unwind.
I downloaded HEX editor (http://www.hhdsoftware.com/free-hex-editor) to edit my binary file. Then I searched for the string where the 0xff 0xd8 was (in may case in line 0000000288). This is supposed to be the beginning of a JPEG file. Then I deleted everything that came before of that (also the six pair of zeros within the same line). Then I saved m edits and tried again to open it with an image processing program (in my case, I'm usin ENVI), and....IT WORKS! Now the binary file is red as an image file!
The problem now is that I have plenty of those files (302), and I need to edit all of them. Moreover, they each contain more than one jpeg, so I need to modify each one ore times. Guess I need to improve my programming knowledge...
Well, according to this page JPEG files begin with the byte pair 0xff 0xd8, so you could search forwards for that sequence, and throw away the data before it.
In your file, it happens on the line starting 0000000288.