Insert Japanese characters into latin1_swedish_ci collated mysql table column - mysql

Question

Japanese characters are getting replaced by ??? I am not allowed to change the collation for the table/column. How can I insert these values?
MariaDB [company]> show full columns from test_table_latin1;
+-------+-------------+-------------------+------+-----+---------+-------+---------------------------------+---------+
| Field | Type | Collation | Null | Key | Default | Extra | Privileges | Comment |
+-------+-------------+-------------------+------+-----+---------+-------+---------------------------------+---------+
| id | int(5) | NULL | YES | | NULL | | select,insert,update,references | |
| data | varchar(20) | latin1_swedish_ci | YES | | NULL | | select,insert,update,references | |
+-------+-------------+-------------------+------+-----+---------+-------+---------------------------------+---------+
2 rows in set (0.00 sec)
MariaDB [company]> insert into test_table_latin1 values (4,'Was sent 検索キーワード - 自然');
Query OK, 1 row affected, 1 warning (0.00 sec)
MariaDB [company]> select * from test_table_latin1 where id=4;
+------+----------------------+
| id | data |
+------+----------------------+
| 4 | Was sent ??????? - ? |
+------+----------------------+
1 row in set (0.00 sec)

Answer 1

Japanese data is already there
It can't be, or if it is, it is garbled beyond recognition. For one thing, DB throws warnings if you try (INSERT INTO test_table_latin1 (data) VALUES ('キーワード'); with "Incorrect string value: '\xE3\x82\xAD\xE3\x83\xBC...' for column 'data'".
Same if you force it (CONVERT('キーワード' USING latin1)), you get the question marks as it does the best it can with an impossible request. It tried to warn you when you were doing it accidentally, but now that you're doing it explicitly it will comply, and just mark the problem spots with '?'. The data is lost, the Japanese is no longer there, and there's nothing you can do to convert ????? to キーワード.
The best of horrible options is pretending all is well: INSERT INTO test_table_latin1 (data) VALUES (CONVERT('キーワード' USING binary)), which gets you キーワード. Total garbage, but garbage that can be converted back to original: SELECT CONVERT(CONVERT(data USING binary) USING utf8) FROM test_table_latin1; should give you `キーワード'. Problem is, this only works when there's no actual Swedish, because either you encode the characters above 0x7f as if they were Unicode (which they are not), or if you avoid them, then you break UTF8 and you won't be able to convert back. So it's again a very bad case.
Finally, you could make your own way of signifying "treat this part differently", like "Was sent [[Base64:UTF8:5qSc57Si44Kt44O844Ov44O844OJ]] - [[Base64:UTF8:6Ieq54S2]]" and decode it on the client.
All of these are bad, bad alternatives to the single correct one: make the column Unicode. I understand that you might be unable to do so (company policy, legacy, compatibility, whatever), but it doesn't change the facts that anything else is no longer suited for this multicultural world we live in.