How to convert large UTF-8 encoded char* string to CStringW (UTF-16)? - html

Question

I have a problem with converting a UTF-8 encoded string to a UTF-16 encoded CStringW.
Here is my source code:
CStringW ConvertUTF8ToUTF16( __in const CHAR * pszTextUTF8 )
{
_wsetlocale( LC_ALL, L"Korean" );
if ( (pszTextUTF8 == NULL) || (*pszTextUTF8 == '\0') )
{
return L"";
}
const size_t cchUTF8Max = INT_MAX - 1;
size_t cchUTF8;
HRESULT hr = ::StringCbLengthA( pszTextUTF8, cchUTF8Max, &cchUTF8 );
if ( FAILED( hr ) )
{
AtlThrow( hr );
}
++cchUTF8;
int cbUTF8 = static_cast<int>( cchUTF8 );
int cchUTF16 = ::MultiByteToWideChar(
CP_UTF8,
MB_ERR_INVALID_CHARS,
pszTextUTF8,
-1,
NULL,
0
);
CString strUTF16;
strUTF16.GetBufferSetLength(cbUTF8);
WCHAR * pszUTF16 = new WCHAR[cchUTF16];
int result = ::MultiByteToWideChar(
CP_UTF8,
0,
pszTextUTF8,
cbUTF8,
pszUTF16,
cchUTF16
);
ATLASSERT( result != 0 );
if ( result == 0 )
{
AtlThrowLastWin32();
}
strUTF16.Format(_T("%s"), pszUTF16);
return strUTF16;
}
pszTextUTF8 is htm file's content in UTF-8.
When htm file's volume is less than 500kb, this code works well.
but, when converting over 500kb htm file, (ex 648KB htm file that I have.)
pszUTF16 has all content of file, but strUTF16 is not. (about half)
I guess File open is not wrong.
In strUTF16 m_pszData has all content how to I get that?
strUTF16.Getbuffer(); dosen't work.

Answer 1

The code in the question is stock full of bugs, somewhere in the order of 1 bug per 1-2 lines of code.
Here is a short summary:
_wsetlocale( LC_ALL, L"Korean" );
Changing a global setting in a conversion function is unexpected, and will break code calling that. It's not even necessary either; you aren't using the locale for the encoding conversion.
HRESULT hr = ::StringCbLengthA( pszTextUTF8, cchUTF8Max, &cchUTF8 );
This is passing the wrong cchUTF8Max value (according to the documentation), and counts the number of bytes (vs. the number of characters, i.e. code units). Besides all that, you do not even need to know the number of code units, as you never use it (well, you are, but that is just another bug).
int cbUTF8 = static_cast<int>( cchUTF8 );
While that fixes the prefix (count of bytes as opposed to count of characters), it won't save you from using it later on for something that has an unrelated value.
strUTF16.GetBufferSetLength(cbUTF8);
This resizes the string object that should eventually hold the UTF-16 encoded characters. But it doesn't use the correct number of characters (the previous call to MultiByteToWideChar would have provided that value), but rather chooses a completely unrelated value: The number of bytes in the UTF-8 encoded source string.
But it doesn't just stop there, that line of code also throws away the pointer to the internal buffer, that was ready to be written to. Failure to call ReleaseBuffer is only a natural consequence, since you decided against reading the documentation.
WCHAR * pszUTF16 = new WCHAR[cchUTF16];
While not a bug in itself, it needlessly allocates another buffer (this time passing the correct size). You already allocated a buffer of the required size (albeit wrong) in the previous call to GetBufferSetLength. Just use that, that's what the member function is for.
strUTF16.Format(_T("%s"), pszUTF16);
That is probably the anti-pattern associated with the printf family of functions. It is the convoluted way to write CopyChars (or Append).
Now that that's cleared up, here is the correct way to write that function (or at least one way to do it):
CStringW ConvertUTF8ToUTF16( __in const CHAR * pszTextUTF8 ) {
// Allocate return value immediately, so that (N)RVO can be applied
CStringW strUTF16;
if ( (pszTextUTF8 == NULL) || (*pszTextUTF8 == '\0') ) {
return strUTF16;
}
// Calculate the required destination buffer size
int cchUTF16 = ::MultiByteToWideChar( CP_UTF8,
MB_ERR_INVALID_CHARS,
pszTextUTF8,
-1,
nullptr,
0 );
// Perform error checking
if ( cchUTF16 == 0 ) {
throw std::runtime_error( "MultiByteToWideChar failed." );
}
// Resize the output string size and use the pointer to the internal buffer
wchar_t* const pszUTF16 = strUTF16.GetBufferSetLength( cchUTF16 );
// Perform conversion (return value ignored, since we just checked for success)
::MultiByteToWideChar( CP_UTF8,
MB_ERR_INVALID_CHARS, // Use identical flags
pszTextUTF8,
-1,
pszUTF16,
cchUTF16 );
// Perform required cleanup
strUTF16.ReleaseBuffer();
// Return converted string
return strUTF16;
}