\x00 is coming between each char in Pyspark dataframe - csv

Question

I am reading.CSV UTF-8 file into Pyspark dataframe. In dataframe getting \x00 in each char of each column in dataframe.
For example
In csv-
Username
Xyz
In dataframe, it's coming in like square box. When collect() it show like below. So square box =\x00
\x00U\x00S\x00E....
\x00X\x00y\x00Z\x00
Can you please help

Answer 1

Issue
Your issue may be that you are not reading the file using the correct encoding.
Recommendation
You may achieve this using the encoding option when reading from your csv file. #JosefZ suggested using utf_16_BE and utf_16_LE which is a good start for determining what the real encoding is being used by your file. However, while these options may work in python, pyspark is looking for the following string encoding values:
US-ASCII
ISO-8859-1
UTF-8
UTF-16BE
UTF-16LE
UTF-16
You may learn more about string encodings here.
For your specific use case, you could try using .option("encoding","UTF-16") or simply replace and test the encoding with the options listed above eg:
df = (
spark.read
.format("csv")
.option("header",False) # optional
.option("encoding","UTF-16")
.schema("username string") # optional
.load("testfile.d") # - replace with your actual file name
)
Full Reproducible Example
# Creating test csv file
with open("testfile.d","wb") as fp:
fp.write(b'\x00X\x00y\x00Z') # writing encoded content
# reading and displaying content without specifying encoding
spark.read.format("csv").option("header",False).schema("username string").load("testfile.d").show()
+--------+
|username|
+--------+
| �X�y�Z|
+--------+
# reading and displaying content by specifying encoding
spark.read.format("csv").option("header",False).option("encoding","UTF-16").schema("username string").load("testfile.d").show()
+--------+
|username|
+--------+
| XyZ|
+--------+