The JSON specification states that control characters that must be escaped are only with codes from U+0000 to U+001F:
7. Strings
The representation of strings is similar to conventions used in the C
family of programming languages. A string begins and ends with
quotation marks. All Unicode characters may be placed within the
quotation marks, except for the characters that must be escaped:
quotation mark, reverse solidus, and the control characters (U+0000
through U+001F).
Main idea of escaping is to don't damage output when printing JSON document or message on terminal or paper.
But there other control characters like [DEL] from C0 and other control characters from C1 set (U+0080 through U+009F). Shouldn't be they also escaped in JSON strings?
From the JSON specification:
8. String and Character Issues
8.1. Character Encoding
JSON text SHALL be encoded in UTF-8, UTF-16, or UTF-32.
In UTF-8, all codepoints above 127 are encoded in multiple bytes. About half of those bytes are in the C1 control character range. So in order to avoid having those bytes in a UTF-8 encoded JSON string, all of those code points would need to be escaped. This effectively eliminates the use of UTF-8 and the JSON string might as well be encoded in ASCII. As ASCII is a subset of UTF-8 this is not disallowed by the standard. So if you are concerned with putting C1 control characters in the byte stream just escape them, but requiring every JSON representation to use ASCII would be wildly inefficient in anything but an english environment.
UTF-16 and UTF-32 could not possibly be parsed by something that uses the C1 (or even C0) control characters so the point is rather moot for those encodings.
Related
Yesterday I upgraded an html page from "4.01 strict" to html5.
* http://r0k.us/rock/games/CoH/HallsOfHeroes/
The character encoding is iso-8859-1. The http://validator.w3.org fails and won't even parse it when utf-8 is specified as charset, apparently because I use footnote characters such as ² . They are in the upper 128 bytes of the character set. What confuses me is that I keep reading that the first 256 bytes of utf-8 is 8859-1.
Does anyone know why the page won't validate as utf-8 ?
Actually, only the first 128 code points are encoded in UTF-8 as ASCII, but UTF-8 is not ASCII, in particular, the next 128 code points differ.
You need to re-save the files as UTF-8 if you want them to be served as UTF-8.
The character ² ("SUPERSCRIPT TWO") is represented by the number 0xb2 (178 decimal) -- but it's represented differently in 8859-1 and UTF-8.
In 8859-1, it's represented as a single byte with the value 0xb2.
In UTF-8, it's represented as two consecutive bytes with the values 0xc2, 0xb2. See here for an explanation of the encoding.
(8859-1 is more compact that UTF-8 for files containing 8-bit characters, but it's incapable of representing anything past 255. UTF-8 is compatible with ASCII and with 8859-1 for 7-bit characters, is reasonably compact for most text, and can represent more than a million distinct characters.)
A file containing only 7-bit characters can be interpreted either as ASCII, 8859-1, or UTF-8. A file containing 8-bit characters cannot; it has to be translated.
If you're on a Unix-like system with the iconv command installed, this:
iconv -f iso-8859-1 -t utf-8
will perform the appropriate translation.
I want to test if a certain BLOB starts with character 255 (\xff). Is there a way to encode this character in a literal string?
Table 9.1. Special Character Escape Sequences gives certain special characters but not a way to encode arbitrary characters.
Failing a way to encode characters, is there a different workaround?
Use a hexadecimal literal, e.g. X'FF'. See http://dev.mysql.com/doc/refman/5.0/en/hexadecimal-literals.html
Im reading one chapter from the W3C HTML Document Representation
In the 5.1 says this:
User agents must also know the specific character encoding that was used to transform the document character stream into a byte stream.
Then in the 5.2 says this:
The "charset" parameter identifies a character encoding, which is a method of converting a sequence of bytes into a sequence of characters.
Char-Bytes
Bytes-Char
So im wrong or there are 2 encodings between the representation...
A "character encoding" such as UTF-8 is, strictly speaking, a specification for representing characters as a sequence of bytes. But the encodings are always reversible, so we can speak of a (single) character encoding as going both ways.
Other character encodings used in practice are UTF-16 ad UTF-32.
Each of these are specifications under which you can encode text as bytes and decode bytes into characters. Two parts of the same specification.
How to distinguish UTF-8 (no BOM) and ASCII files?
If the file contains any bytes with the top bit set, then it is not ASCII.
So if the only possibilities are ASCII or UTF-8, then it's UTF-8.
If the file contains only bytes with the top bit clear, then it's meaningless to distinguish whether it's ASCII or UTF-8, since it represents exactly the same series of characters either way. But you can call it ASCII.
Of course this doesn't distinguish UTF-8 from ISO Latin or CP1252, and neither does it confirm that the so-called UTF-8 is actually valid.
http://msdn.microsoft.com/en-us/library/dd318672%28v=vs.85%29.aspx
IsTextUnicode Function
Determines if a buffer is likely to contain a form of Unicode text.
RFC 4627 on Json reads:
Encoding
JSON text SHALL be encoded in Unicode. The default encoding is
UTF-8.
Since the first two characters of a JSON text will always be ASCII
characters [RFC0020], it is possible to determine whether an octet
stream is UTF-8, UTF-16 (BE or LE), or UTF-32 (BE or LE) by looking
at the pattern of nulls in the first four octets.
What does it mean "Since the first two characters of a JSON text will always be ASCII characters [RFC0020]"? I've looked at RFC0020 but couldn't find anything about it. JSON could be {" or { " (ie whitespace before the quote.
It means that since JSON will always start with ASCII characters (non-ASCII is only permitted in strings, which cannot be the root object), it is possible to determine from the start of the stream/file what encoding it is in.
UTF-16 and UTF-32 should have a BOM that appears at the start of the stream and by finding out what it is, you can determine the exact encoding. This is possible as one can determine if the first characters are JSON or not.
I assume the spec specifically mentions this as for many other text streams/files, this is not always possible (as most text files can start with any two characters and the two first bytes of the actual file are not known in advance).
RFC 4627 requires a JSON document to represent either an object or an array. So, the first characters must be (with any amount of JSON whitespace characters) [ followed by a value or { followed by ". Values are null, true, false, or a string ("…), object or array. So, since JSON whitespace characters, [, {, n, t, f, and " are in the C0 Controls and Basic Latin block, they are also in the ASCII character set [by the design of Unicode]. (Not sure why the standard is fixated on "ASCII" when it says, "JSON text SHALL be encoded in Unicode." Future standards drop the reference.)
UTF-32 has four bytes per character. UTF-16 has two. So, to distinguish between UTF-16 and UTF-32, you need 4 bytes. In both of those encodings, characters from the C0 Controls and Basic Latin block are encoded with at most one non-zero byte (a byte with a value of 0 is sometimes called a "null byte"). Also, U+0000 (which is encoded as 0x00 0x00 0x00 0x00 in UTF-32 and 0x00 0x00 in UTF-16) is not valid JSON whitespace. So, the pattern of 0x00 bytes can be used to determine which of the allowed encodings, a valid JSON document uses.
RFC 7159 changed JSON to allow a JSON document to represent any value, not just an object or array. So, the statement in the previous standard is no longer valid. Therefore, the character detection algorithm was broken and removed from the standard.
For accurate detection, you need to see the beginning and the end of the document. 0x22 0x00 0x00 0x00 at the beginning could be any of UTF-8, UTF-16LE, and UTF-32LE; It's the start of a string with zero or more U+0000 characters. In this case, you need the number of 0x00 bytes at the end to tell which.
RFC 8259 changed JSON to require UTF-8 (for JSON "exchanged between systems that are not part of a closed ecosystem"). Out of practically, a JSON reader would still accept UTF-16 and UTF-32.
In the end, some popular JSON parsers leave character decoding up to the caller, having APIs that accept only the "native" string type for the programming environment. (This opens up the very common hazard of using the wrong character encoding for reading text files or HTTP bodies.)