UTF-8 decoder capability and stress test | ---------------------------------------- | | Markus Kuhn - 2000-09-02 | | This test text examines, how UTF-8 decoders handle various types of | corrupted or otherwise interesting UTF-8 sequences. | | According to ISO 10646-1, sections R.7 and 2.3c, a device receiving | UTF-8 shall interpret a "malformed sequence in the same way that it | interprets a character that is outside the adopted subset". This means | usually that the malformed UTF-8 sequence is replaced by a replacement | character (U+FFFD), which looks a bit like an inverted question mark, | or a similar symbol. It might be a good idea to visually distinguish a | malformed UTF-8 sequence from a correctly encoded Unicode character | that is just not available in the current font but otherwise fully | legal. For both cases, a clearly recognisable symbol should be used. | Just ignoring malformed sequences or unavailable characters will make | debugging more difficult and can lead to user confusion. | | Test sequences: | | Check, whether a malformed UTF-8 sequence is (1) represented at all, | (2) represented by exactly one replacement character (or equivalent | signal), and (3) the following quotation mark after an illegal UTF-8 | sequence is correctly displayed, i.e. proper resynchronization takes | place. This file says "THE END" in the last line, so if you don't see | that, your decoder crashed somehow before, which is also not nice. | | All lines in this file are exactly 79 characters long (plus the line | feed). In addition, all lines end with "|", except for the two test | lines 2.1.1 and 2.2.1, which contain non-printable ASCII controls | U+0000 and U+007F. If you display this file with a fixed-width font, | these "|" characters should all line up in column 79 (right margin). | This allows you to test quickly, whether your UTF-8 decoder finds the | correct number of characters in every line. | | Here come the tests: | | 1 Some correct UTF-8 text | | You should see the Greek word 'kosme': "κόσμε" | | 2 Boundary condition test cases | | You should see a correctly encoded character each time (which is not | necessarily an existing character). | | 2.1 First possible sequence of a certain length | | 2.1.1 1 byte (U-00000000): "" 2.1.2 2 bytes (U-00000080): "€" | 2.1.3 3 bytes (U-00000800): "ࠀ" | 2.1.4 4 bytes (U-00010000): "𐀀" | 2.1.5 5 bytes (U-00200000): "" | 2.1.6 6 bytes (U-04000000): "" | | 2.2 Last possible sequence of a certain length | | 2.2.1 1 byte (U-0000007F): "" 2.2.2 2 bytes (U-000007FF): "߿" | 2.2.3 3 bytes (U-0000FFFF): "￿" | 2.2.4 4 bytes (U-001FFFFF): "" | 2.2.5 5 bytes (U-03FFFFFF): "" | 2.2.6 6 bytes (U-7FFFFFFF): "" | | 2.3 Other boundary conditions | | 2.3.1 U-0000D7FF = ed 9f bf = "퟿" | 2.3.2 U-0000E000 = ee 80 80 = "" | 2.3.3 U-0000FFFD = ef bf bd = "�" | 2.3.4 U-0010FFFF = f4 8f bf bf = "􏿿" | 2.3.5 U-00110000 = f4 90 80 80 = "" | | 3 Malformed sequences | | 3.1 Unexpected continuation bytes | | Each unexpected continuation byte should be separately signalled as a | malformed sequence of its own. | | 3.1.1 First continuation byte 0x80: "" | 3.1.2 Last continuation byte 0xbf: "" | | 3.1.3 2 continuation bytes: "" | 3.1.4 3 continuation bytes: "" | 3.1.5 4 continuation bytes: "" | 3.1.6 5 continuation bytes: "" | 3.1.7 6 continuation bytes: "" | 3.1.8 7 continuation bytes: "" | | 3.1.9 Sequence of all 64 possible continuation bytes (0x80-0xbf): | | " | | | " | | 3.2 Lonely start characters | | 3.2.1 All 32 first bytes of 2-byte sequences (0xc0-0xdf), | each followed by a space character: | | " | " | | 3.2.2 All 16 first bytes of 3-byte sequences (0xe0-0xef), | each followed by a space character: | | " " | | 3.2.3 All 8 first bytes of 4-byte sequences (0xf0-0xf7), | each followed by a space character: | | " " | | 3.2.4 All 4 first bytes of 5-byte sequences (0xf8-0xfb), | each followed by a space character: | | " " | | 3.2.5 All 2 first bytes of 6-byte sequences (0xfc-0xfd), | each followed by a space character: | | " " | | 3.3 Sequences with last continuation byte missing | | All bytes of an incomplete sequence should be signalled as a single | malformed sequence, i.e., you should see only a single replacement | characters in each of the next 10 tests. (Characters as in section 2) | | 3.3.1 2-byte sequence with last byte missing (U+0000): "" | 3.3.2 3-byte sequence with last byte missing (U+0000): "" | 3.3.3 4-byte sequence with last byte missing (U+0000): "" | 3.3.4 5-byte sequence with last byte missing (U+0000): "" | 3.3.5 6-byte sequence with last byte missing (U+0000): "" | 3.3.6 2-byte sequence with last byte missing (U-000007FF): "" | 3.3.7 3-byte sequence with last byte missing (U-0000FFFF): "" | 3.3.8 4-byte sequence with last byte missing (U-001FFFFF): "" | 3.3.9 5-byte sequence with last byte missing (U-03FFFFFF): "" | 3.3.10 6-byte sequence with last byte missing (U-7FFFFFFF): "" | | 3.4 Concatenation of incomplete sequences | | All the 10 sequences of 3.3 concatenated, you should see 10 malformed | sequences being signalled: | | "" | | 3.5 Impossible bytes | | The following two bytes cannot appear in a correct UTF-8 string | | 3.5.1 fe = "" | 3.5.2 ff = "" | 3.5.3 fe fe ff ff = "" | | 4 Overlong sequences | | The following sequences are not malformed according to the letter of | the Unicode 2.0 standard. However, they are longer then necessary and | a correct UTF-8 encoder is not allowed to produce them. A "safe UTF-8 | decoder" should reject them just like malformed sequences for two | reasons: (1) It helps to debug applications if overlong sequences are | not treated as valid representations of characters, because this helps | to spot problems more quickly. (2) Overlong sequences provide | alternative representations of characters, that could maliciously be | used to bypass filters that check only for ASCII characters. For | instance, a 2-byte encoded line feed (LF) would not be caught by a | line counter that counts only 0x0a bytes, but it would still be | processed as a line feed by an unsafe UTF-8 decoder later in the | pipeline. From a security point of view, ASCII compatibility of UTF-8 | sequences means also, that ASCII characters are *only* allowed to be | represented by ASCII bytes in the range 0x00-0x7f. To ensure this | aspect of ASCII compatibility, use only "safe UTF-8 decoders" that | reject overlong UTF-8 sequences for which a shorter encoding exists. | | 4.1 Examples of an overlong ASCII character | | With a safe UTF-8 decoder, all of the following five overlong | representations of the ASCII character slash ("/") should be rejected | like a malformed UTF-8 sequence, for instance by substituting it with | a replacement character. If you see a slash below, you do not have a | safe UTF-8 decoder! | | 4.1.1 U+002F = c0 af = "" | 4.1.2 U+002F = e0 80 af = "" | 4.1.3 U+002F = f0 80 80 af = "" | 4.1.4 U+002F = f8 80 80 80 af = "" | 4.1.5 U+002F = fc 80 80 80 80 af = "" | | 4.2 Maximum overlong sequences | | Below you see the highest Unicode value that is still resulting in an | overlong sequence if represented with the given number of bytes. This | is a boundary test for safe UTF-8 decoders. All five characters should | be rejected like malformed UTF-8 sequences. | | 4.2.1 U-0000007F = c1 bf = "" | 4.2.2 U-000007FF = e0 9f bf = "" | 4.2.3 U-0000FFFF = f0 8f bf bf = "" | 4.2.4 U-001FFFFF = f8 87 bf bf bf = "" | 4.2.5 U-03FFFFFF = fc 83 bf bf bf bf = "" | | 4.3 Overlong representation of the NUL character | | The following five sequences should also be rejected like malformed | UTF-8 sequences and should not be treated like the ASCII NUL | character. | | 4.3.1 U+0000 = c0 80 = "" | 4.3.2 U+0000 = e0 80 80 = "" | 4.3.3 U+0000 = f0 80 80 80 = "" | 4.3.4 U+0000 = f8 80 80 80 80 = "" | 4.3.5 U+0000 = fc 80 80 80 80 80 = "" | | 5 Illegal code positions | | The following UTF-8 sequences should be rejected like malformed | sequences, because they never represent valid ISO 10646 characters and | a UTF-8 decoder that accepts them might introduce security problems | comparable to overlong UTF-8 sequences. | | 5.1 Single UTF-16 surrogates | | 5.1.1 U+D800 = ed a0 80 = "" | 5.1.2 U+DB7F = ed ad bf = "" | 5.1.3 U+DB80 = ed ae 80 = "" | 5.1.4 U+DBFF = ed af bf = "" | 5.1.5 U+DC00 = ed b0 80 = "" | 5.1.6 U+DF80 = ed be 80 = "" | 5.1.7 U+DFFF = ed bf bf = "" | | 5.2 Paired UTF-16 surrogates | | 5.2.1 U+D800 U+DC00 = ed a0 80 ed b0 80 = "" | 5.2.2 U+D800 U+DFFF = ed a0 80 ed bf bf = "" | 5.2.3 U+DB7F U+DC00 = ed ad bf ed b0 80 = "" | 5.2.4 U+DB7F U+DFFF = ed ad bf ed bf bf = "" | 5.2.5 U+DB80 U+DC00 = ed ae 80 ed b0 80 = "" | 5.2.6 U+DB80 U+DFFF = ed ae 80 ed bf bf = "" | 5.2.7 U+DBFF U+DC00 = ed af bf ed b0 80 = "" | 5.2.8 U+DBFF U+DFFF = ed af bf ed bf bf = "" | | 5.3 Other illegal code positions | | 5.3.1 U+FFFE = ef bf be = "￾" | 5.3.2 U+FFFF = ef bf bf = "￿" | | THE END |