Encode String in XQuery - csv

Question

Wrote a function in my web application based on eXist-db to export some xml elements to csv with XQuery. Everything works fine but I have some umlauts like ü, ä or ß in my elements which are displayed the wrong way in my csv. I tried to encode the content by using fn:normalize-unicode but this is not working.
Here is a minimalized example of my code snippet:
let $input =
<root>
<number>1234</number>
<name>Aufmaß</name>
</root>
let $csv := string-join(
for $ta in $input
return concat($ta/number/text(), fn:normalize-unicode($ta/name/text())))
let $csv-ueber-string := concat($csv-ueber, string-join($massnahmen, $nl))
let $set-content-type := response:set-header('Content-Type', 'text/csv')
let $set-accept := response:set-header('Accept', 'text/csv')
let $set-file-name := response:set-header('Content-Disposition', 'attachment; filename="export.csv"')
return response:stream($csv, '')

Answer 1

It's very unlikely indeed that there's anything wrong with your query, or that there's anything you can do in your query to correct this.
The problem is likely to be either
(a) the input data being passed to your query is in a different character encoding from what the query processor thinks it is
(b) the output data from your query is in a different character encoding from what the recipient of the output thinks it is.
A quick glance at your query suggests that it doesn't actually have any external input other that the query source code itself. But the source code is one of the inputs, and that's a possible source of error. A good way to eliminate this possibility might be to see what happens if you replace
<name>Aufmaß</name>
by
<name>Aufma{codepoints-to-string(223)}</name>
If that solves the problem, then your query source text is not in the encoding that the query compiler thinks it is.
The other possibility is that the problem is on the output side, and frankly, this seems more likely. You seem to be producing an HTTP response stream as output, and constructing the HTTP headers yourself. I don't see any evidence that you are setting any particular encoding in the HTTP response headers. The response:stream() function is vendor-specific and I'm not familiar with its details, but I suspect that you need to ensure it encodes the content in UTF-8 and that the HTTP headers say it is in UTF-8; this may be by extra parameters to the function, or by external configuration options.

Answer 2

As you might expect, eXist is serializing the CSV as Unicode (UTF-8). But when you open the resulting export.csv file directly in Excel (i.e., via File > Open), Excel will try its best to guess the encoding of the CSV file. But CSV files lack any way of declaring their encoding, so applications may well guess wrong, as it sounds like Excel did in your case. On my computer, Excel guesses wrong too, mangling the encoding of Aufmaß as Aufmaß. Here's the way to force Excel to use the encoding of a UTF-8 encoded CSV file such as the one produced by your query.
In Excel, start a new spreadsheet via File > New
Select File > Import to bring up a series of dialogs that let you specify how to import the CSV file.
In the first dialog, select "CSV file" as the type of file.
In the next dialog, titled "Text Import Wizard -
Step 1 of 3", select "Unicode (UTF-8)" as the "File origin." (At least these are the titles/order in my copy of MS Excel for Mac 2016).
Proceed through the remainder of the dialogs, keeping the default values.
Excel will then place the contents of your export.csv in the new spreadsheet.
Lastly, let me provide the following query I used to test and confirm that the CSV file produced by eXist does open as expected when following the directions above. The query is essentially the same as yours but fixes some problems in your query that prevented me from running it directly. I saved this query at /db/csv-test.xq and called it via http://localhost:8080/exist/rest/db/csv-test.xq,
xquery version "3.1";
let $input :=
<root>
<number>1234</number>
<name>Aufmaß</name>
</root>
let $cell-separator := ","
let $column-headings := $input/*/name()
let $header-row := string-join($column-headings, $cell-separator)
let $body-row := string-join($input/*/string(), $cell-separator)
let $newline := '
'
let $csv := string-join(($header-row, $body-row), $newline)
return
response:stream-binary(
util:string-to-binary($csv),
"text/csv",
"export.csv"
)