Upload XML into Exact Online exceeding maximum size or response time - exact-online

Question

I have several XML files generated by a industry solution with new data to be uploaded into Exact Online using the XML API, either directly or using the UploadXmlTopics table in Invantive SQL with the Exact Online driver.
However, the Exact Online XML API poses a limit of approx. 10 MB per upload and even then the load time can be long when the system is heavily loaded.
When the load time exceeds ten minutes, part of the transactions have been applied and a part has failed. With a timeout there is no message returned which states what remains to be loaded.
I can not change the XML files since they are automatically generated by the industry solution.
What is the best way to reliably upload the XML files into Exact Online?

Answer 1

Of course it is best to have the XML files changed, but there are various alternatives. Please note first of all that performance of Exact Online varies across the day and weekend. Best time to upload massive amounts of data is Sunday between 13:00 and 23:00 based upon experience.
When uploading manually, you can split the XML files by hand into several XML files. Always split on the main topic at the path /eExact/TOPIC.
When uploading through UploadXmlTopics table, you can use two approaches:
Calculated
Automated
Calculated XML size for Exact Online
The calculated approach is the only one available in older versions. It works as follows:
In a in memory table or a file based table put a record per XML topic that you wish to upload. I normally name them 'xml' as column name.
Then determine how many fragments you need, for instance using:
select ceil(log(xmlsize / 10000, 16)) + 1
from ( select sum(length(xml)) xmlsize from xmlaccounts#inmemorystorage )
Replace 10000 by the maximum fragment size. Choose a smaller one during periods of heavy load and 1000000 for the weekend.
Memorize the outcome using for instance:
local define xmlaccountsparts "${outcome:0,0}"
Then construct the new XML to insert into UploadXmlTopics as follows:
select filenamepostfix, xml
from ( select filenamepostfix, listagg(xml, '') xml
from ( select substr(md5(xml), 1, ${xmlaccountsparts}) filenamepostfix
, xml
from xmlaccounts#inmemorystorage
)
group
by filenamepostfix
)
And insert this payload into Exact Online using UploadXMLTopics.
What it effectively does is to first determine approximately how many files you need using the logarithmic function with 16 as base. Then use MD5 to associate somewhat randomly distributed hexadecimal (16 base) values for each XML topic to upload. Take a number of left-side characters of the MD5 values equal to the logarithmic function outcome to get approximately this number of files, each with approximately the same payload size. And then reconstruct the XML.
Automatic XML size for Exact Online
Newer releases have an auto fragment option which do the heavy lifting for you. Use a SQL like:
insert into UploadXMLTopics#eol
--
-- Upload seed data into Exact Online.
--
( topic
, payload
, division_code
, orig_system_reference
, fragment_payload_flag
, fragment_max_size_characters
)
select topic
, filecontents
, division_code
, filename
, true
, 10000 /* This one is in characters. You can also specify in number. */
from ...
The loaded fragments can be queried using:
select *
--
-- Check results and reload.
--
from UploadXMLTopicFragments#eol
And looks something like: