How do you ignore the 1st two lines of a CSV file in Dataweave? - csv

I am using Anypoint Studio 6.1 with Mule 3.8.1 and in the Transform Message component I have an input CSV file which has fieldnames on the 1st line and datatypes for that filed on the 2nd line. I want to start processing the records from line 3 onwards, but cannot configure the Reader Configuration to miss the 2nd line containing the datatypes.
Is there a way I can do this so the 1st line is used for fieldnames, the 2nd line is ignored and the 3rd line onwards are the records that are processed?
Test CSV:
Product,Price
string,decimal(10,2)
phone,99
tv,499
XML flow:
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:dw="http://www.mulesoft.org/schema/mule/ee/dw" xmlns:metadata="http://www.mulesoft.org/schema/mule/metadata" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:spring="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/ee/dw http://www.mulesoft.org/schema/mule/ee/dw/current/dw.xsd">
<flow name="csvreaderFlow">
<dw:transform-message metadata:id="08c0a5d4-9d7d-44ad-b72b-bac83ab6b6c1" doc:name="Transform Message">
<dw:input-payload doc:sample="sample_data\list_csv.csv" mimeType="application/csv">
<dw:reader-property name="bodyStartLineNumber" value="2"/>
</dw:input-payload>
<dw:set-payload><![CDATA[%dw 1.0
%output application/json
---
payload map ((payload01 , indexOfPayload01) -> {
products: payload01.Product
})]]></dw:set-payload>
</dw:transform-message>
</flow>
</mule>
Even if I increase the body line start number to 10, it still shows the 2 line and the 2 records to process. I thought this would cause an error.
Thanks

In the Input reader configuration for CSV on the Dataweave, they have a setting "bodyStartLineNumber" (set that to '3' in your case). Let me know if you have issues,
Here are the steps to be followed on the Dataweave,
1. Right click on the "Payload" label (left panel).
2. Click on the "Reader Configuration" option.
3. GUI opens with multiple options to be set.
4. Configure the "Body Start Line Number"

Related

How to skip specific Execution Plan Steps?

In the TaskBlockService there is a POST call that one or more steps should be skipped. There is not a good example given how the posted XML (List of String) the paths of the steps to skip.
Tried the following content for the POSTed data:
curl -X POST https://xldeploy.company.com/deployit/tasks/v2/5e917094-d054-4cc7-940e-89d851ca225a/skip
File remove-steps.xml content - sample 1:
<list>
<string>0_1_1</string>
</list>
File remove-steps.xml content - sample 2:
<list>
<string>0-1-1</string>
</list>
The first format you list is right, but you have to make sure you're using a step path and not just the path to a block.
Lets say you get the blocks from your deployment plan with this call.
curl -uadmin:password http://localhost:4516/deployit/tasks/v2/28830810-5104-4ab9-9826-22f66dee265d
This will produce the result:
<task id="28830810-5104-4ab9-9826-22f66dee265d" failures="0" state="PENDING" owner="admin">
<description>Initial deployment of Environments/local/TestApp001</description>
<activeBlocks/>
<metadata>
<environment>local</environment>
<taskType>INITIAL</taskType>
<environment_id>Environments/local</environment_id>
<application>TestApp001</application>
<version>1.0</version>
</metadata>
<block id="0" state="PENDING" description="" root="true">
<block id="0_1" state="PENDING" description="Deploy" phase="true">
<block id="0_1_1" state="PENDING" description="Deploy TestApp001 1.0 on environment local"/>
</block>
</block>
<dependencies/>
If you want to see the steps in block 0_1_1 then you can use this rest call to get the steps.
curl -uadmin:password http://local6/deployit/tasks/v2/28830810-5104-4ab9-9826-22f66dee265d/block/0_1_1/step
<block id="0_1_1" state="PENDING" description="Deploy TestApp001 1.0 on environment local" current="0">
<step failures="0" state="PENDING" description="Execute Command"/>
<step failures="0" state="PENDING" description="Copy File001.txt to Infrastructure/localhost"/>
The steps are numbered within the block starting from 1. So if you are want to skip the step - Copy File001.txt to Infrastructure/localhost the step path is 0_1_1_2. Your XML will look like:
<list>
<string>0_1_1_2</string>
</list>

Is it possible to define optional fields in a Smooks CSV reader

I want to read CSV entries with a Smooks CSV reader.
But my problem is how to define a field as optional.
<?xml version="1.0"?>
<smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd" xmlns:csv="http://www.milyn.org/xsd/smooks/csv-1.2.xsd">
<csv:reader fields="field1,field2,OPTONAL_FIELD3,$ignore$+" />
</smooks-resource-list>
Is there any way to define such a configuration?
The data I want to read looks like this:
123,4,opt1
456,7
If field 3 is declared as normal in the configuration, the line must always ends with a ',' like this:
456,7,

Solr 4.7.1 uploading CSV Document is missing mandatory uniqueKey field id

I'm new to solr (4.7.1). I unzip the solr code and copied the schemaless example to its own directory. I then used start.jar passing it -Dsolr.solr.home=" to the new location. Jetty came up and everything appears to be working on that front.
Now I wanted to upload/update a csv file to it. Here's what I used:
curl http://localhost:8983/solr/update/csv --data-binary #c:\solrschemaless\data.csv -H "Content-type:text/csv; charset=utf-8"
I received the following:
<response>
<lst name="responseHeader">
<int name="status">400</int>
<int name="QTime">0</int>
</lst>
<lst name="error">
<str name="msg">Document is missing mandatory uniqueKey field: id</str>
<int name="code">400</int>
</lst>
</response>
The csv file has a column named XXXXID. I changed to "id", "id_s", "id_i" but still getting the same error. There are a lot of post on SO and elsewhere but thus far I didn't see one for the schemaless model.
EDIT: I reduced down my csv file to this:
id,Contact,Address,Focus,Type
2,97087,1170,NULL,1
and I'm still getting the same error message of missing mandatory uniqueKey.
I'm on Windows 8.
Any ideas?
Figured it out. For some reason the column id cannot be the first column on the list. When I change
id,Contact,Address,Focus,Type
2,97087,1170,NULL,1
to this
foo,id,Contact,Address,Focus,Type
bar,2,97087,1170,NULL,1
it works.

what's the purpose of the .database file in the ssis bids solutions?

I Cannot find a link with more details.
I think it's not necessary to check in that (same as .suo). It does not seem to change changing the structure of the solution files Ex adding new ssis packages or similar so I want to know it I can avoid to check in, as it might be recreated automatically each time
Mario
If you look at the file association for .database, you can see some of SSIS and SSAS (analysis services) shared project history. On my machine, the file type for .database is "Analysis Services Database" even though we're in an SSIS project.
The only difference I see between a 2005, 2008 and a 2012 file are the attributes in the header row (the 200s, 300s and 400s)
2005 declaration
<Database xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:ddl2="http://schemas.microsoft.com/analysisservices/2003/engine/2"
xmlns:ddl2_2="http://schemas.microsoft.com/analysisservices/2003/engine/2/2"
xmlns:ddl100_100="http://schemas.microsoft.com/analysisservices/2008/engine/100/100"
xmlns:dwd="http://schemas.microsoft.com/DataWarehouse/Designer/1.0"
dwd:design-time-name="bfed880d-636a-441b-9f13-55d813b61715"
xmlns="http://schemas.microsoft.com/analysisservices/2003/engine">
<ID>SSISHackAndSlash</ID>
<Name>SSISHackAndSlash</Name>
<CreatedTimestamp>0001-01-01T00:00:00Z</CreatedTimestamp>
<LastSchemaUpdate>0001-01-01T00:00:00Z</LastSchemaUpdate>
<LastProcessed>0001-01-01T00:00:00Z</LastProcessed>
<State>Unprocessed</State>
<LastUpdate>0001-01-01T00:00:00Z</LastUpdate>
<DataSourceImpersonationInfo>
<ImpersonationMode>Default</ImpersonationMode>
<ImpersonationInfoSecurity>Unchanged</ImpersonationInfoSecurity>
</DataSourceImpersonationInfo>
</Database>
2008 declaration
<Database xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:ddl2="http://schemas.microsoft.com/analysisservices/2003/engine/2"
xmlns:ddl2_2="http://schemas.microsoft.com/analysisservices/2003/engine/2/2"
xmlns:ddl100_100="http://schemas.microsoft.com/analysisservices/2008/engine/100/100"
xmlns:ddl200="http://schemas.microsoft.com/analysisservices/2010/engine/200"
xmlns:ddl200_200="http://schemas.microsoft.com/analysisservices/2010/engine/200/200"
xmlns:dwd="http://schemas.microsoft.com/DataWarehouse/Designer/1.0"
dwd:design-time-name="0d4fe615-edae-41b8-a46b-e62b3ba25a4a"
xmlns="http://schemas.microsoft.com/analysisservices/2003/engine">
<ID>SSISHackAndSlash2008</ID>
<Name>SSISHackAndSlash2008</Name>
<CreatedTimestamp>0001-01-01T00:00:00Z</CreatedTimestamp>
<LastSchemaUpdate>0001-01-01T00:00:00Z</LastSchemaUpdate>
<LastProcessed>0001-01-01T00:00:00Z</LastProcessed>
<State>Unprocessed</State>
<LastUpdate>0001-01-01T00:00:00Z</LastUpdate>
<DataSourceImpersonationInfo>
<ImpersonationMode>Default</ImpersonationMode>
<ImpersonationInfoSecurity>Unchanged</ImpersonationInfoSecurity>
</DataSourceImpersonationInfo>
</Database>
2012 declaration
<Database xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:ddl2="http://schemas.microsoft.com/analysisservices/2003/engine/2"
xmlns:ddl2_2="http://schemas.microsoft.com/analysisservices/2003/engine/2/2"
xmlns:ddl100_100="http://schemas.microsoft.com/analysisservices/2008/engine/100/100"
xmlns:ddl200="http://schemas.microsoft.com/analysisservices/2010/engine/200"
xmlns:ddl200_200="http://schemas.microsoft.com/analysisservices/2010/engine/200/200"
xmlns:ddl300="http://schemas.microsoft.com/analysisservices/2011/engine/300"
xmlns:ddl300_300="http://schemas.microsoft.com/analysisservices/2011/engine/300/300"
xmlns:ddl400="http://schemas.microsoft.com/analysisservices/2012/engine/400"
xmlns:ddl400_400="http://schemas.microsoft.com/analysisservices/2012/engine/400/400"
xmlns:dwd="http://schemas.microsoft.com/DataWarehouse/Designer/1.0"
dwd:design-time-name="14d580fb-02bf-43e0-a9c8-ebe99ba02d52"
xmlns="http://schemas.microsoft.com/analysisservices/2003/engine">
<ID>EzAPIDemo</ID>
<Name>EzAPIDemo</Name>
<CreatedTimestamp>0001-01-01T00:00:00Z</CreatedTimestamp>
<LastSchemaUpdate>0001-01-01T00:00:00Z</LastSchemaUpdate>
<LastProcessed>0001-01-01T00:00:00Z</LastProcessed>
<State>Unprocessed</State>
<LastUpdate>0001-01-01T00:00:00Z</LastUpdate>
<DataSourceImpersonationInfo>
<ImpersonationMode>Default</ImpersonationMode>
<ImpersonationInfoSecurity>Unchanged</ImpersonationInfoSecurity>
</DataSourceImpersonationInfo>
</Database>
I always add that to source control, even though as you have observed, it does nothing.

Erro loading schemes from Eclipse - Mule IDE

i am attempting to create my first mule server but i get an error for any external scheme i try and include,
my Config file is as follows (working on Eclipse Indigo with mule standalone 3.2 installation):
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns="http://www.mulesoft.org/schema/mule/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:spring="http://www.springframework.org/schema/beans"
xmlns:http="http://www.mulesoft.org/schema/mule/http"
xmlns:vm="http://www.mulesoft.org/schema/mule/vm"
xmlns:quartz="http://www.mulesoft.org/schema/mule/quartz"
xsi:schemaLocation="
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/3.2/mule.xsd
http://www.mulesoft.org/schema/mule/quartz http://www.mulesoft.org/schema/mule/quartz/current/mule-quartz.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/3.2/mule-http.xsd
http://www.mulesoft.org/schema/mule/vm http://www.mulesoft.org/schema/mule/vm/3.2/mule-vm.xsd">
<flow name="ChatListener">
<quartz:inbound-endpoint jobName="eventTimer" repeatInterval="2000">
<quartz:event-generator-job>
<quartz:payload>Poll Chat DB</quartz:payload>
</quartz:event-generator-job>
</quartz:inbound-endpoint>
<component>
<singleton-object class="com.TimeLineListener.ChatListener" />
</component>
<vm:outbound-endpoint path="ChatMsgs" exchange-pattern="one-way"/>
</flow>
<flow name="TimeLineMsgSender">
<composite-source>
<!-- Incoming Chat Msgs -->
<vm:inbound-endpoint path="ChatMsgs" exchange-pattern="one-way"/>
<!-- Incoming SIEM Msgs -->
<vm:inbound-endpoint path="SIEMMsgs" exchange-pattern="one-way"/>
<!-- Incoming NMS Msgs -->
<vm:inbound-endpoint path="NMSMsgs" exchange-pattern="one-way"/>
</composite-source>
<!-- Tested OutPut endpoint -->
<stdio:outbound-endpoint system="OUT"/>
</flow>
</mule>
and the errors i recieve are:
1.
The prefix "stdio" for element "stdio:outbound-endpoint" is not bound. mule-config.xml ‪/ChatTester‬ line 41 XML Problem
2.
cvc-complex-type.2.4.a: Invalid content was found starting with element 'vm:inbound-endpoint'. One of '{"http://www.mulesoft.org/schema/mule/core":abstract-inbound-endpoint}' is expected. mule-config.xml ‪/ChatTester‬ line 31 XML Problem
3.
cvc-complex-type.2.4.a: Invalid content was found starting with element 'quartz:inbound-endpoint'. One of '{"http://www.mulesoft.org/schema/mule/core":description, "http://www.mulesoft.org/schema/mule/core":composite-source, "http://www.mulesoft.org/schema/mule/core":abstract-inbound-endpoint, "http://www.mulesoft.org/schema/mule/core":abstract-message-processor, "http://www.mulesoft.org/schema/mule/core":abstract-outbound-endpoint, "http://www.mulesoft.org/schema/mule/core":response}' is expected. mule-config.xml ‪/ChatTester‬ line 17 XML Problem
any idea what i"m doing wrong?
1.
The prefix "stdio" for element "stdio:outbound-endpoint" is not bound. mule-config.xml ‪/ChatTester‬ line 41 XML Problem
This one is easy: you're missing the stdio namespace declaration.
2.
cvc-complex-type.2.4.a: Invalid content was found starting with element 'vm:inbound-endpoint'. One of '{"http://www.mulesoft.org/schema/mule/core":abstract-inbound-endpoint}' is expected. mule-config.xml ‪/ChatTester‬ line 31 XML Problem
3.
cvc-complex-type.2.4.a: Invalid content was found starting with element 'quartz:inbound-endpoint'. One of '{"http://www.mulesoft.org/schema/mule/core":description, "http://www.mulesoft.org/schema/mule/core":composite-source, "http://www.mulesoft.org/schema/mule/core":abstract-inbound-endpoint, "http://www.mulesoft.org/schema/mule/core":abstract-message-processor, "http://www.mulesoft.org/schema/mule/core":abstract-outbound-endpoint, "http://www.mulesoft.org/schema/mule/core":response}' is expected. mule-config.xml ‪/ChatTester‬ line 17 XML Problem
For these ones: I don't know. Maybe due to the mix of "current" and "3.2" you're using in the namespace locations? Try only with "3.2" instead of current to see if it helps.
Otherwise, nothing visibly crazy in your config :)
Actually, this is a problem of Eclipse and doesn't relate to your configuration. Hope this helps:
"Since Mule’s schemas are not split over multiple schema files, it’s safe to turn off this feature. In Eclipse’s preferences, go to XML > XML Files > Validation and clear the Honour all schema locations check box. Mule config files should now validate without errors again."-MuleSolf Blog.
For more details: http://blogs.mulesoft.org/overcoming-xml-validation-errors-in-eclipse-35/