How to understand the base64 encoding in vtk binary file format - binary

Question

I have a problem with how to understand the binary DataArray. The problem because of the base64 encoding.
The manual said if the format of DataArray is binary,
The data are encoded in base64 and listed contiguously inside the
DataArray element. Data may also be compressed before encoding in base64. The byte-
order of the data matches that specified by the byte_order attribute of the VTKFile element.
I can not fully understand that, so I have obtain ascii file and binary file for same model.
ASCII file
<?xml version="1.0"?>
<VTKFile type="UnstructuredGrid" version="0.1" byte_order="LittleEndian" header_type="UInt32" compressor="vtkZLibDataCompressor">
<UnstructuredGrid>
<Piece NumberOfPoints="4" NumberOfCells="1">
<PointData>
</PointData>
<CellData>
</CellData>
<Points>
<DataArray type="Float32" Name="Points" NumberOfComponents="3" format="ascii" RangeMin="0" RangeMax="1.4142135624">
0 0 0 1 0 0
1 1 0 0 1 1
</DataArray>
</Points>
<Cells>
<DataArray type="Int64" Name="connectivity" format="ascii" RangeMin="0" RangeMax="3">
0 1 2 3
</DataArray>
<DataArray type="Int64" Name="offsets" format="ascii" RangeMin="4" RangeMax="4">
4
</DataArray>
<DataArray type="UInt8" Name="types" format="ascii" RangeMin="10" RangeMax="10">
10
</DataArray>
</Cells>
</Piece>
</UnstructuredGrid>
</VTKFile>
Binary file
<?xml version="1.0"?>
<VTKFile type="UnstructuredGrid" version="0.1" byte_order="LittleEndian" header_type="UInt32" compressor="vtkZLibDataCompressor">
<UnstructuredGrid>
<Piece NumberOfPoints="4" NumberOfCells="1">
<PointData>
</PointData>
<CellData>
</CellData>
<Points>
<DataArray type="Float32" Name="Points" NumberOfComponents="3" format="binary" RangeMin="0" RangeMax="1.4142135624">
AQAAAACAAAAwAAAAEQAAAA==eJxjYEAGDfaobEw+ADwjA7w=
</DataArray>
</Points>
<Cells>
<DataArray type="Int64" Name="connectivity" format="binary" RangeMin="0" RangeMax="3">
AQAAAACAAAAgAAAAEwAAAA==eJxjYIAARijNBKWZoTQAAHAABw==
</DataArray>
<DataArray type="Int64" Name="offsets" format="binary" RangeMin="4" RangeMax="4">
AQAAAACAAAAIAAAACwAAAA==eJxjYYAAAAAoAAU=
</DataArray>
<DataArray type="UInt8" Name="types" format="binary" RangeMin="10" RangeMax="10">
AQAAAACAAAABAAAACQAAAA==eJzjAgAACwAL
</DataArray>
</Cells>
</Piece>
</UnstructuredGrid>
</VTKFile>
When I looked at the the DataArray, using the last one as an example, I can not create the relationship between AQAAAACAAAABAAAACQAAAA==eJzjAgAACwAL and 10.
My understanding can be expressed using follow code, but it obtain CggAAA==.
#include "base64.h" // https://github.com/superwills/NibbleAndAHalf/blob/master/NibbleAndAHalf/base64.h
#include <iostream>
int main()
{
int x = 10;
int len;
// first arg: binary buffer
// second arg: length of binary buffer
// third arg: length of ascii buffer
char *ascii = base64((char *)&x, sizeof(int), &len);
std::cout << ascii << std::endl;
std::cout << len << std::endl;
free(ascii);
return 0;
}
Can someone give me an explanation of how to convert?
Another relate topic can be see in
https://discourse.vtk.org/t/error-when-writing-binary-vtk-files/4487/7
Thanks for your time.

Answer 1

Solution can be found in disscusstion.
https://discourse.vtk.org/t/how-to-understand-binary-dataarray-in-xml-vtk-output/4489
The long extra data comes from the compressor header.

Answer 2

I have found the solution and written the answer in a VTK support question, but I am writing it here in case anyone comes here looking for the same issue as us two.
Note that I program in Python, but I believe there are base64 and zlib functions in C++. Also, I use numpy to define arrays, but I believe std::vector can be equivalently used in C++.
So, suppose we want to write the single precision float32 array called "Points" in your example. If we suppose that a header type of "UInt32" is used, then in Python, we would do:
import numpy as np
import zlib
import base64
# write the float array.
arr = np.array([0, 0, 0, 1, 0, 0,
1, 1, 0, 0, 1, 1], dtype='float32')
# generate a zlib compressed array. This outputs a python byte type
arr_comp = zlib.compress(arr)
# generate the uncompressed header
header = np.array([ 1, # apparently this is always the case, I think
2**15, # from what I have read, this is true in general
arr.nbytes, # the size of the array `arr` in bytes
len(arr_comp)], # the size of the compressed array
dtype='uint32') # because of header_type="UInt32"
# use base64 encoding when writing to file
# `.decode("utf-8")` transforms the python byte type to a string
print((base64.b64encode(header_arr) + base64.b64encode(arr_comp)).decode("utf-8"))
The output is as expected:
AQAAAACAAAAwAAAAEQAAAA==eJxjYEAGDfaobEw+ADwjA7w=
The 2**15 is the argument that controls the size of the history buffer (or the “window size”) used when compressing data, according to the zlib python docs. Not sure what that means though...
Edit: The above code only works if the size in bytes of the array is less than or equal to 2**15. In the VTK support question I have expanded for the case where the array is larger. You have to divide it into chunks.