Using a query as below, you get output in the XML format required for Exact Online.
These XML messages can be put in many files on disk using:
local export documents in filecontents to "c:\temp\in" filename column filename
And then loaded into Exact Online's current division using:
local eol batch import xml "Accounts" in "c:\temp\in\Accounts" success "c:\temp\success" fail "c:\temp\fail"
This works fine for one division, but not when you load 100 divisions/companies.
How can I directly load data from the SQL into multiple companies/divisions at the same time instead of using an individual local eol batch import xml statement and use DIVISIONCODE statement?
Query sample:
select 'Accounts\010-Accounts.xml'
filename
, stg.fileprefix
|| chr(13)
|| '<Accounts>'
|| xml
|| chr(13)
|| '</Accounts>'
|| stg.filepostfix
filecontents
from ( select listagg
( chr(13)
|| '<Account code="'
|| xmlencode(custsupid)
|| '" status="'
...
You can upload into multiple companies/divisions with one statement using the following syntax as described on UploadXMLTopics:
insert into UploadXMLTopics#eol
( topic
, payload
, division_code
)
select topic
, filecontents
, division_code
from ...
Even when one or more of the uploads fail, the insert will not report an error and process all XML messages.
You can see the results afterwards using:
select *
from UploadXMLTopics
The column SUCCESSFUL indicates whether the payload column was loaded succesfully.
In RESULT column you will find the return message of Exact Online, such as:
<?xml version="1.0" encoding="utf-8"?>
<eExact xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="eExact-XML.xsd">
<Messages>
<Message type="0">
<Topic code="FinYears" node="FinYear" />
<Date>2016-12-09T09:02:36</Date>
<Description>Kan boekjaar niet aanmaken: 2014 vanwege ontbrekende boekjaren. Het laagste boekjaar in Exact Online is 2016.</Description>
</Message>
<Message type="2">
<Topic code="FinYears" node="FinYear">
<Data key="2015" keyAlt="2015" />
</Topic>
<Date>2016-12-09T09:02:37</Date>
<Description>Aangemaakt</Description>
</Message>
</Messages>
</eExact>
Related
I've been trying to download a certain data on my graph and it returns this error :
Neo.ClientError.Statement.SyntaxError: Type mismatch: expected List<Node> but was Node (line 2, column 27 (offset: 77))"CALL apoc.export.csv.data(c,[], "contrib.csv",{})"
This is the query I did :
MATCH (c:Contrib) WHERE c.nationality CONTAINS "|" CALL apoc.export.csv.data(c,[], "contrib.csv",{}) YIELD file, source, format, nodes, relationships, properties, time, rows, batchSize, batches, done, data RETURN file, source, format, nodes, relationships, properties, time, rows, batchSize, batches, done, data
What went wrong ? :(
Thanks
The syntax for the function: apoc.export.csv.data is
apoc.export.csv.data(nodes,rels,file,config)
exports given nodes and relationships as csv to the provided file
The nodes is a collection of nodes rather than a node.
OLD: MATCH (c:Contrib) WHERE c.nationality CONTAINS "|"
CALL apoc.export.csv.data(c,[], "contrib.csv",{})
NEW: MATCH (c:Contrib) WHERE c.nationality CONTAINS "|"
WITH collect(c) as contribs
CALL apoc.export.csv.data(contribs, [], "contrib.csv", {})
I have millions of files with the following (poor) JSON format:
{
"3000105002":[
{
"pool_id": "97808",
"pool_name": "WILDCAT (DO NOT USE)",
"status": "Zone Permanently Plugged",
"bhl": "D-12-10N-05E 902 FWL 902 FWL",
"acreage": ""
},
{
"pool_id": "96838",
"pool_name": "DRY & ABANDONED",
"status": "Zone Permanently Plugged",
"bhl": "D-12-10N-05E 902 FWL 902 FWL",
"acreage": ""
}]
}
I've tried to generate an Athena DDL that would accommodate this type (especially the api field) of structure with this:
CREATE EXTERNAL TABLE wp_info (
api:array < struct < pool_id:string,
pool_name:string,
status:string,
bhl:string,
acreage:string>>)
LOCATION 's3://foo/'
After trying to generate a table with this, the following error is thrown:
Your query has the following error(s):
FAILED: ParseException line 2:12 cannot recognize input near ':' 'array' '<' in column type
What is a workable solution to this issue? Note that the api string is different for every one of the million files. The api key is not actually within any of the files, so I hope there is a way that Athena can accommodate just the string-type value for these data.
If you don't have control over the JSON format that you are receiving, and you don't have a streaming service in the middle to transform the JSON format to something simpler, you can use regex functions to retrieve the relevant data that you need.
A simple way to do it is to use Create-Table-As-Select (CTAS) query that will convert the data from its complex JSON format to a simpler table format.
CREATE TABLE new_table
WITH (
external_location = 's3://path/to/ctas_partitioned/',
format = 'Parquet',
parquet_compression = 'SNAPPY')
AS SELECT
regexp_extract(line, '"pool_id": "(\d+)"', 1) as pool_id,
regexp_extract(line, ' "pool_name": "([^"])",', 1) as pool_name,
...
FROM json_lines_table;
You will improve the performance of the queries to the new table, as you are using Parquet format.
Note that you can also update the table when you can new data, by running the CTAS query again with external_location as 's3://path/to/ctas_partitioned/part=01' or any other partition scheme
I have a few processes that utilize the CFSpreadsheet tag to import and then manipulate Excel data. This works great for .XLS & .XLSX files, however, it doesn't work if the data is sent as a .CSV file since CFSpreadsheet apparently was never updated to import .CSV files. At the end of the day I just want a simple pre-processor that takes a .CSV file and re-writes it as an .XLSX file so that my other process can take it from there.
My environment is the developer edition of Coldfusion 2018 and I've tried importing the data manually (which can work if I know all of the column definitions---but I won't always know that). My latest attempt has been with Ben Nadel's CSVToArray function ( https://www.bennadel.com/blog/2041-update-parsing-csv-data-files-in-coldfusion-with-csvtoarray.htm ) which works---I can easily get the .CSV file into an array---but I can't figure out how to go from that array to something like a query that I can write a spreadsheet with using CFSpreadsheet.
Here's an EXAMPLE:
<!--- This include is the function from Ben Nadel referenced above --->
<cfinclude template="Function_CSVtoArray.cfm">
<cfset result = csvToArray(file="TEST_File.csv") />
<cfdump var="#result#" label="TESTING">
<!--- *** The above WORKS up to this point ***--->
<!--- Create a new query. --->
<cfset qPartsTwo = QueryNew( "" ) />
<!--- Loop over keys in the struct. --->
<cfloop index="strKey" list="#StructKeyList(result)#" delimiters=",">
<!--- Add column to new query with default values. --->
<cfset QueryAddColumn(qPartsTwo,strKey,"VARCHAR",objParts[strKey]) />
</cfloop>
<!--- This code FAILS with a "You have attempted to dereference a scalar variable of type class coldfusion.runtime.Array as a structure with members" error message --->
I'd like to end up at something like this (although right now "result" is an array of some kind and not a query):
<cfspreadsheet action="write" filename="<my path>\TEST.xlsx" query="result">
Any ideas would be appreciated!
It looks like your UDF returns a multi-dimensional array, not an array of structures. Instead of trying to coerce the array into a query object, try using spreadsheet functions to write the array data to an xlsx file.
DEMO / Sample data
result = [ ["First Name", "Last Name", "Address"]
, ["John", "Doe", "123 Anywhere Ave"]
, ["Mary", "Smith", "456 Somewhere Street"]
, ["Charles", "Doe", "789 Anywhere Court"]
];
Code:
// create spreadsheet
xlsx = SpreadSheetNew("The Results", true);
// populate with array data
SpreadSheetAddRows( xlsx, result );
// save to file
SpreadSheetWrite( xlsx, "c:/path/to/test.xlsx", true );
.. or as James A Mohler suggested, you could also use member functions:
xlsx = SpreadSheetNew("The Results", true);
xlsx.addRows( result );
xlsx.write( "c:/path/to/test.xlsx", true );
I bet you could do something like this
<cfinclude template="Function_CSVtoArray.cfm">
<cfset result = csvToArray(file="TEST_File.csv") />
<cfdump var="#result#" label="TESTING">
<!--- setup the columns that you need --->
<cfset qPartsTwo = queryNew("id,name,amount","Integer,Varchar,Integer", result) />
<cfspreadsheet action="write" filename="<my path>\TEST.xlsx" query="result">
CSVToArray() looks like it makes an array of structs.
If you already have an array of structures from CSVToArray. Can you then use the ArrayOfStructuresToQuery function: https://cflib.org/udf/ArrayOfStructuresToQuery
I want to read arbitrary ABAP data into an iXML document object which contains the JSON-XML representation of these data.
The only way I see is a double application of the id transformation which is not very efficient:
data(lo_aux1) = cl_sxml_string_writer=>create( if_sxml=>co_xt_json ).
call transformation id
source data = ls_some_abap_data
result xml lo_aux1.
data(lv_aux2) = lo_aux1->get_output( ).
data(lo_result) = cl_ixml=>create( )->create_document( ).
call transformation id
source xml lv_aux2
result xml lo_result.
Now lo_result is an iXML DOM representation of the ABAP data in the JSON-XML format, as required. Is it possible to obtain it in a more direct way?
Note: I am not interested in result objects of the sXML family, as I want to manipulate / extend the resulting JSON-XML document with the usual XML DOM methods, which is impossible for an sXML writer object (sXML writers are so simple they can only write everything they have into an output object, but don't allow editing of parts of the object that they already contain).
I am sitting at a proxy and want to enrich the incoming JSON payload by some ABAP data, before passing it to the endpoint. Strategy: parse the incoming JSON into an JSON-XML doc, read the (complex) ABAP data into a second XML doc, then add XML subtrees of the second to the first before finally producing the result JSON from the first JSON-XML doc
I don't understand the need of iXML at all here. Converting complex ABAP structure into XML to merge it with JSON is redundant here. Let's assume you received some JSON data from Web-Service:
{
"main":{
"PASSENGERS":[
{
"NAME":"Horst",
"TITLE":"Herr",
"AGE":30
},
{
"NAME":"Jutta",
"TITLE":"Frau",
"AGE":35
},
{
"NAME":"Ingo",
"TITLE":"Herr",
"AGE":31
}
]
}
}
And you want enrich each passenger data with flight data from SFLIGHT table. You can manipulate the nodes and attributes with cl_sxml_string_writer just like this:
DATA(lv_json) = CONV string( '{"main": {"PASSENGERS":[ {"NAME":"Horst","TITLE":"Herr","AGE":30}, {"NAME":"Jutta","TITLE":"Frau","AGE":35}, {"NAME":"Ingo","TITLE":"Herr","AGE":31} ]}}' ).
DATA open_element TYPE REF TO if_sxml_open_element.
DATA value TYPE REF TO if_sxml_value_node.
DATA(o_json) = cl_abap_codepage=>convert_to( lv_json ).
DATA(reader) = cl_sxml_string_reader=>create( o_json ).
SELECT DISTINCT connid, fldate, planetype FROM sflight INTO TABLE #DATA(lt_flight).
DATA(xml) = cl_abap_codepage=>convert_to( lv_json ).
DATA(out) = cl_demo_output=>new( )->begin_section( 'Original JSON' )->write_xml( xml ).
DATA(writer) = CAST if_sxml_writer( cl_sxml_string_writer=>create( ) ).
open_element = writer->new_open_element( name = 'flights' nsuri = reader->nsuri ).
writer->write_node( open_element ).
DATA(i) = 1.
DO.
DATA(node) = reader->read_next_node( ).
IF node IS INITIAL.
EXIT.
ENDIF.
IF node IS INSTANCE OF if_sxml_value_node.
DATA(value_node) = CAST if_sxml_value_node( node ).
value_node->set_value( to_upper( value_node->get_value( ) ) ).
ENDIF.
writer->write_node( node ).
IF node->type = if_sxml_node=>co_nt_element_open.
DATA(op) = CAST if_sxml_open_element( node ).
CHECK op->qname-name = 'object' AND op->get_attributes( ) IS INITIAL.
open_element = writer->new_open_element( name = 'flight' nsuri = reader->nsuri ).
open_element->set_attribute( name = 'FLIGHT_DATE'
value = | { lt_flight[ i ]-fldate } | ).
open_element->set_attribute( name = 'PLANE_TYPE'
value = | { lt_flight[ i ]-planetype } | ).
writer->write_node( open_element ).
value = writer->new_value( ).
value->set_value( | { lt_flight[ i ]-connid } | ).
writer->write_node( value ).
writer->write_node( writer->new_close_element( ) ).
i = i + 1.
ENDIF.
ENDDO.
writer->write_node( writer->new_close_element( ) ).
out->next_section( 'Modified JSON'
)->write_xml(
CAST cl_sxml_string_writer( writer )->get_output( )
)->display( ).
DATA(result_json) = CAST cl_sxml_string_writer( writer )->get_output( ).
Resulted JSON will be dumped into result_json variable and you can push it further wherever you want.
Here is resulted JSON, with capitalized passenger values and extended with flight node, which contains flight number and attributed with data and plane type:
<flights>
<object>
<flight FLIGHT_DATE=" 20160610 " PLANE_TYPE=" A380‑800 "> 0002 </flight>
<object name="main">
<array name="PASSENGERS">
<object>
<flight FLIGHT_DATE=" 20160712 " PLANE_TYPE=" A380‑800 "> 0002 </flight>
<str name="NAME">HORST</str>
<str name="TITLE">HERR</str>
<num name="AGE">30</num>
</object>
<object>
<flight FLIGHT_DATE=" 20160813 " PLANE_TYPE=" A380‑800 "> 0002 </flight>
<str name="NAME">JUTTA</str>
<str name="TITLE">FRAU</str>
<num name="AGE">35</num>
</object>
<object>
<flight FLIGHT_DATE=" 20160914 " PLANE_TYPE=" A380‑800 "> 0002 </flight>
<str name="NAME">INGO</str>
<str name="TITLE">HERR</str>
<num name="AGE">31</num>
</object>
</array>
</object>
</object>
</flights>
I have XML which is structured similar to the example below and I've written an XQuery in MarkLogic to export this to a CSV (see below the XML).
What I need help with is formatting the output so that when I open the CSV file, instead of having all of the output across 1 I'd like it to be grouped "columns" so to speak.
Let's say for the sample below, I'd like to output all of the DataTime and Source element values and have the values in their own columns like this:
2012-02-15T00:58:26 a
2012-02-15T00:58:26 b
2012-02-15T00:58:26 c
How would I go about that?
Would welcome any reference points or help. Thank you in advance.
Here's the sample XML:
<Document xmlns="http://fakeexample.org/schemas">
<Information>
<ItemId>1f28cb0c2c4f4eb7b13c4abf998e391e</ItemId>
<MediaType>Text</MediaType>
<DocDateTime>2012-02-15T00:58:26</DocDateTime>
</Information>
<FilingData>
<DateTime>2012-02-15T00:58:26</DateTime>
<Source>a</Source>
</FilingData>
<FilingData>
<DateTime>2012-02-15T00:58:27</DateTime>
<Source>b</Source>
</FilingData>
<FilingData>
<DateTime>2012-02-15T00:58:28</DateTime>
<Source>c</Source>
</FilingData>
</Document>
Here's the sample XQuery:
xquery version "1.0-ml";
declare default function namespace "http://www.w3.org/2005/xpath-functions";
declare namespace xdmp="http://marklogic.com/xdmp";
declare namespace exam="http://fakeexample.org/schemas";
declare function local:getDocument($url)
{
let $response := xdmp:document-get($url,
<options xmlns="xdmp:document-get">
<repair>full</repair>
<format>xml</format>
</options>)
return $response
};
xdmp:set-response-content-type("text/csv"),
xdmp:add-response-header(
"Content-disposition",
fn:concat("attachment;filename=", "output", fn:current-time(), ".csv")
),
(
let $q := cts:element-value-query(xs:QName("exam:ItemId"), ("1f28cb0c2c4f4eb7b13c4abf998e391e"))
let $results := cts:search(fn:doc(), $q)
for $result in $results
return fn:string-join((xs:string($result//exam:DateTime),
xs:string($result//exam:Source)
), "," )
)
Replace your for loop with this:
return
string-join(
for $result in $results//FilingData
return fn:string-join((xs:string($result//exam:DateTime),
xs:string($result//exam:Source)
), "," )
, "
")
That should about do the trick..
Edit: note that I added //FilingData behind $results. That makes sure DateTime and Source of each FilingData is joined separately, and returned as separate strings of the for loop. That allows the outer string-join to add the required line ends between them.
Note:
should be translated to OS specific line endings automatically.
Building on the answer by #grtjn:
string-join(..., "
")
Line endings can be treated differently depending on OS or application. You could try alternative characters (either or both):
"
" (LF)
"
" (CR)
Also, this could be thwarted by the application used to view the CSV. For example, most versions of Microsoft Excel will convert all whitespace within a cell - newlines included - into plain spaces.