I am trying to load CSV file into SQLserver table using Liquibase change log set.
When saved XLSX file as CSV file, column containing comma saved in double quotes (please see 3rd value below), this is fine as per standards but liquibase is ignoring double quotes and considering comma inside the double-quotes.
13,OV,"Diabetes outpatient self-management training services individual,per 30 minutes",77.82,1,0,1/4/2016,,G0108
Error messgae from command line terminal:
CSV file v2.1/r21/TestData20212021.csv Line 21 has 10 values defined, Header has 9. Numbers MUST be equal (check for unquoted string with embedded commas)
<changeSet author="sprint-developer" id="sprint1-09">
<loadData
file="v2.1/r21/TestData2021.csv"
tableName = "tbl_Votes" encoding="UTF-8" >
<column header="VcenarioID" name="VcenarioID" type="numeric"/>
<column header="venefitCode" name="venefitCode" type="string"/>
<column header="KostDescription" name="KostDescription" type="string"/>
<column header="Kost" name="Kost" type="NUMERIC"/>
<column header="OcKurrences" name="OKcurrences" type="numeric"/>
<column header="KostIsPerIncident" name="KostIsPerIncident" type="boolean"/>
<column header="KostDate" name="KostDate" type="date"/>
<column header="VundleId" name="VundleId" type="NUMERIC"/>
<column header="VillingCode" name="VillingCode" type="string"/>
</loadData>
<rollback>Delete from tbl_Votes where VcenarioID=13 </rollback>
</changeSet>
Try adding quotchar='"' to your changeSet. This should tell liqbuiase to treat everything inside "" as one single value.
Check out loadData docs.
So your changeSet could look like this:
<changeSet author="sprint-developer" id="sprint1-09">
<loadData
file="v2.1/r21/TestData2021.csv"
tableName = "tbl_Votes" encoding="UTF-8" quotchar='"'>
<column header="VcenarioID" name="VcenarioID" type="numeric"/>
<column header="venefitCode" name="venefitCode" type="string"/>
<column header="KostDescription" name="KostDescription" type="string"/>
<column header="Kost" name="Kost" type="NUMERIC"/>
<column header="OcKurrences" name="OKcurrences" type="numeric"/>
<column header="KostIsPerIncident" name="KostIsPerIncident" type="boolean"/>
<column header="KostDate" name="KostDate" type="date"/>
<column header="VundleId" name="VundleId" type="NUMERIC"/>
<column header="VillingCode" name="VillingCode" type="string"/>
</loadData>
<rollback>Delete from tbl_Votes where VcenarioID=13 </rollback>
</changeSet>
Related
Please find my two tables
CREATE TABLE "DBS_P2P"."KW_PAYMENT_IMAGEE_TEST"(IMAGE_KEY INTEGER PRIMARY KEY NOT NULL GENERATED ALWAYS AS IDENTITY ( START WITH +1 INCREMENT BY +1 MINVALUE +1 MAXVALUE +2147483647 NO CYCLE CACHE 20 NO ORDER),REQUEST_ID INTEGER,IMAGE_CONTENT CLOB(5120000),FOREIGN KEY(REQUEST_ID) REFERENCES KW_PAYMENT_LINK_MASTER_TEST(REQUEST_ID))
CREATE TABLE "DBS_P2P"."KW_PAYMENT_IMAGEE_TEST"(IMAGE_KEY INTEGER PRIMARY KEY NOT NULL GENERATED ALWAYS AS IDENTITY ( START WITH +1 INCREMENT BY +1 MINVALUE +1 MAXVALUE +2147483647 NO CYCLE CACHE 20 NO ORDER),REQUEST_ID INTEGER,IMAGE_CONTENT CLOB(5120000),FOREIGN KEY(REQUEST_ID) REFERENCES KW_PAYMENT_LINK_MASTER_TEST(REQUEST_ID))
<hibernate-mapping>
<class name="xyz" table="KW_PAYMENT_LINK_MASTER_TEST" catalog="xyz">
<id name="requestId" type="int">
<column name="REQUEST_ID"/>
<generator class="identity" />
</id>
<property name="referenceWalletId" type="string">
<column name="REFERENCE_ID"/>
</property>
</class>
<hibernate-mapping>
<class name="com.kony.p2p.bo.KwPaymentImagetest" table="KW_PAYMENT_IMAGEE_TEST" catalog="DBS_P2P">
<id name="imageKey" type="int">
<column name="IMAGE_KEY" />
<generator class="identity" />
</id>
<many-to-one name="KwPaymentLinkMastertest" class="com.kony.p2p.bo.KwPaymentLinkMastertest" fetch="select" lazy="false">
<column name="REQUEST_ID" not-null="false" />
</many-to-one>
<property name="imageContent" type="string">
<column name="IMAGE_CONTENT" not-null="false" />
</property>
</class>
<filter-def name="myFilter">
Im able to insert but not able to retrieve the result .
session.createCriteria(KwPaymentLinkMastertest.class).add(Restrictions.eq("requestId", Integer.parseInt(transactionReferenceNumber) )).uniqueResult();
KwPaymentImagetest image = (KwPaymentImagetest) session.createCriteria(KwPaymentImagetest.class).add(Restrictions.eq("kwPaymentLinkMastertest", kwPaymentLinkMastertest)).uniqueResult();
Getting the following error
org.hibernate.QueryException: could not resolve property: kwPaymentLinkMastertest of: com.kony.p2p.bo.KwPaymentImagetest
Please help fix
I think you have typing mistake here.
<many-to-one name="KwPaymentLinkMastertest" with K is upper case
while Restrictions.eq("kwPaymentLinkMastertest" with k is lower case.
Ya,
You are right .
The problem was with small letter and capital letter , this helped to fix my problem.
Thanks.
I am defining a Derived Column transformation in BIML but I am having trouble referencing the output from the previous Excel Source in my Derived Column transformation.
I receive the error upon opening the package after successfully generating the SSIS package and it suggests that it the Derived Transformation cannot find the output from the Excel Source.
Error 2 Error loading AFR_ShareTableBIML.dtsx: The object
"/DTS:Executable/DTS:Executables/DTS:Executable/DTS:ObjectData/pipeline/components/component/inputs/input/inputColumns/inputColumn/properties/property"
references ID "#{Package\Data Flow {Import Share Table CSV}\Source
{Flat File Share Table}.Outputs[Output].Columns[Div c per share]}",
but no object in the package has this ID.
Here is a code snippet:
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<FileFormats>
<FlatFileFormat Name="FFF_AFRShareTable" ColumnNamesInFirstDataRow="true"
FlatFileType="Delimited" IsUnicode="false" TextQualifer="None" HeaderRowsToSkip="6">
<Columns>
<Column Name="Quote Buy" ColumnType="Delimited" DataType="AnsiString" Length ="50" Delimiter=","></Column>
<Column Name="Quote Sell" ColumnType="Delimited" DataType="AnsiString" Length ="50" Delimiter=","></Column>
<Column Name="Div c per share" ColumnType="Delimited" DataType="AnsiString" Length ="50" Delimiter=","></Column>
</Columns>
</FlatFileFormat>
</FileFormats>
<Connections>
<FlatFileConnection Name="FF_AFRShareTable" FileFormat="FFF_AFRShareTable"
FilePath="C:\Temp\Stocks.csv"></FlatFileConnection>
<OleDbConnection Name="CMD DB"
ConnectionString="Data Source=Localhost;Initial Catalog=DB;Provider=SQLNCLI11.1;Integrated Security=SSPI;" CreateInProject="true">
</OleDbConnection>
</Connections>
<Packages>
<Package Name="AFR_ShareTableBIML" ConstraintMode="Linear" ProtectionLevel="DontSaveSensitive">
<Tasks>
<ExecuteSQL Name="SQLTask {OLE_DB} Truncate Security Share Table" ConnectionName="CMD DB">
<DirectInput>truncate table Staging.SecurityShareTable</DirectInput>
</ExecuteSQL>
<Dataflow Name="Data Flow {Import Share Table CSV}">
<Transformations>
<FlatFileSource Name="Source {Flat File Share Table}" ConnectionName="FF_AFRShareTable"></FlatFileSource>
<DerivedColumns Name="DER_NullifyColumns">
<Columns>
<Column Name ="DER_DPS" DataType = "Decimal" Precision="4">
[Div c per share] == "-" ? NULL(DT_DECIMAL, 4) : (DT_DECIMAL, 4)[Div c per share]
</Column>
</Columns>
</DerivedColumns>
</Transformations>
</Dataflow>
</Tasks>
</Package>
</Packages>
I have already defined the column name via the FlatFileFormat and I have confirmed that the expression in the DER_DPS column is is syntactically correct. I found that through replacing the square brackets "[" and "]" with double apostrophes, the SSIS package can be opened. For example:
"Div c per share" == "-" ? NULL(DT_DECIMAL, 4) : (DT_DECIMAL, 4) "Div c per share"
However there are derived column transformation errors on incorrect syntax. Are square brackets special characters in BIML that I need to escape?
That was ... interesting.
It appears that your use of curly braces in your component names causes the Biml expansion to go haywire.
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<FileFormats>
<FlatFileFormat Name="FFF_AFRShareTable" ColumnNamesInFirstDataRow="true"
FlatFileType="Delimited" IsUnicode="false" TextQualifer="None" HeaderRowsToSkip="6">
<Columns>
<Column Name="Quote Buy" ColumnType="Delimited" DataType="AnsiString" Length ="50" Delimiter=","></Column>
<Column Name="Quote Sell" ColumnType="Delimited" DataType="AnsiString" Length ="50" Delimiter=","></Column>
<!-- Change -->
<Column Name="Div c per share" ColumnType="Delimited" DataType="AnsiString" Length ="50" Delimiter="CRLF"></Column>
</Columns>
</FlatFileFormat>
</FileFormats>
<Connections>
<FlatFileConnection Name="FF_AFRShareTable" FileFormat="FFF_AFRShareTable"
FilePath="C:\ssisdata\so\input\Stocks.csv"></FlatFileConnection>
<OleDbConnection Name="CMD DB"
ConnectionString="Data Source=Localhost\dev2014;Initial Catalog=tempdb;Provider=SQLNCLI11.1;Integrated Security=SSPI;"
CreateInProject="false">
</OleDbConnection>
</Connections>
<Packages>
<Package Name="so_37641290_AFR_ShareTableBIML" ConstraintMode="Linear" ProtectionLevel="DontSaveSensitive">
<Tasks>
<ExecuteSQL Name="SQLTask OLE_DB Truncate Security Share Table" ConnectionName="CMD DB">
<DirectInput>truncate table Staging.SecurityShareTable</DirectInput>
</ExecuteSQL>
<Dataflow Name="Data Flow Import Share Table CSV">
<Transformations>
<FlatFileSource Name="Source Flat File Share Table" ConnectionName="FF_AFRShareTable"></FlatFileSource>
<DerivedColumns Name="DER_NullifyColumns">
<Columns>
<Column Name="DER_DPS" DataType="Decimal" Precision="4"><![CDATA[[Div c per share] == "-" ? NULL(DT_DECIMAL, 4) : (DT_DECIMAL, 4)[Div c per share]]]></Column>
</Columns>
</DerivedColumns>
</Transformations>
</Dataflow>
</Tasks>
</Package>
</Packages>
</Biml>
The above biml works for me. Changes I made:
removed { and } from the tasks and component names
updated the last Column definition within your FlatFileFormat Columns collection to have a delimiter of CRLF instead of ,
I used the CDATA tag for the expression. Not needed here but if you had a > or < in there, then you'd need to escape them as either < or the CDATA approach as I used.
I also cleaned up the Derived Column's entity assignments. There were spaces around the equals and I don't believe those are supposed to be there.
Path updates for flat file + OLE DB to work with my setup.
Source data
0
1
2
3
4
5
Quote Buy,Quote Sell,Div c per share
1,1,1
2,2,2
3,3,-
Results
I try to insert a row into mysql using liquibase in a spring-boot app. Something similar to this:
<databaseChangeLog
xmlns="http://www.liquibase.org/xml/ns/dbchangelog/1.9"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog/1.9
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-1.9.xsd">
<changeSet author="laxika" id="20160508-2">
<insert tableName="news">
<column name="title" value="Hello world"/>
<column name="release_date" value="2016-05-09 00:00:00"/>
<column name="icon" value="update"/>
<column name="message" value="
a
b
c
d
"/>
</insert>
</changeSet>
</databaseChangeLog>
For some strange reason liquibase doesn't add newlines to the sql properly so I end up having one long string in the db. How can I force liquibase to insert with newlines?
Here is the schema for the table I'm trying to insert into:
<databaseChangeLog
xmlns="http://www.liquibase.org/xml/ns/dbchangelog/1.9"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog/1.9
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-1.9.xsd">
<changeSet author="laxika" id="20150922-4">
<createTable tableName="news">
<column name="id" type="smallint unsigned" autoIncrement="true">
<constraints primaryKey="true" nullable="false" />
</column>
<column name="release_date" type="date">
<constraints nullable="false" />
</column>
<column name="title" type="varchar(256)">
<constraints nullable="false" />
</column>
<column name="message" type="varchar(2048)">
<constraints nullable="false" />
</column>
<column name="icon" type="varchar(32)">
<constraints nullable="false" />
</column>
</createTable>
<modifySql>
<append value="ENGINE=INNODB DEFAULT CHARSET=utf8"/>
</modifySql>
</changeSet>
</databaseChangeLog>
I tried all above answers, eventually I see an answer on the community forum of liquibase which has the answer:
<column name="message"><![CDATA[
a
b
c
d
]]></column>
Try usind CDATA in the XML :
<column name="message" value="<![CDATA[
a
b
c
d
]]>"/>
<databaseChangeLog
xmlns="http://www.liquibase.org/xml/ns/dbchangelog/1.9"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog/1.9
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-1.9.xsd">
<changeSet author="laxika" id="20160508-2">
<insert tableName="news">
<column name="title" value="Hello world"/>
<column name="release_date" value="2016-05-09 00:00:00"/>
<column name="icon" value="update"/>
<column name="message" valueComputed="(SELECT '\na\nb\nc\nd\n' from dual)"/>
</insert>
</changeSet>
</databaseChangeLog>
Worked for me:
<insert tableName="AUDIT_ATTR">
<column name="AUDIT_ID" valueNumeric="601"/>
<column name="KEY" value="FINDINGS_IN_CATEGORY_NCW_TEXT"/>
<column name="VALUE">
tra1
la2
la3
</column>
</insert>
I have a talend job which reads a db table into a tMap and stores the result in another db table. The destination table is set to "insert" only and before running the job I drop the destination table so its always starting fresh. I have an error logger to handle problems writing to the destination table
I process about 150k rows and it kicks 41k of those saying there was a duplicate entry for key PRIMARY.
While I do expect there to be some duplicates when I check the error logger there are rows where a duplicate key was detected but no matching value is found in the destination table.
Any ideas what might be going on? Its hard to understand as there is also error handling on the tMap component but it seems to be an issue actually writing to the db table once approved rows are sent to it from tMap
For what its worth. I using Talend DI 5.6.1 reading and writing to the same database schema but different tables on MySQL db 5.6.17
Incase its useful I have a number of subjobs that use the same source and destinations but just refine the city value a bit differently on each execution
Source Schema
<?xml version="1.0" encoding="UTF-8"?>
<schema dbmsId="mysql_id">
<column comment="" default="" key="true" label="Column22"
length="50" nullable="false" originalDbColumnName="Column22"
originalLength="0" pattern="" precision="0"
talendType="id_String" type="VARCHAR"/>
<column comment="" default="" key="false" label="city" length="50"
nullable="true" originalDbColumnName="city" originalLength="0"
pattern="" precision="0" talendType="id_String" type="VARCHAR"/>
<column comment="" default="" key="false" label="CountryCode"
length="3" nullable="true" originalDbColumnName="CountryCode"
originalLength="0" pattern="" precision="0"
talendType="id_String" type="VARCHAR"/>
<column comment="" default="" key="false" label="svc_area_code"
length="20" nullable="true" originalDbColumnName="svc_area_code"
originalLength="0" pattern="" precision="0"
talendType="id_String" type="VARCHAR"/>
</schema>
Destination Schema
<?xml version="1.0" encoding="UTF-8"?>
<schema dbmsId="mysql_id">
<column comment="" default="" key="true" label="Column22"
length="50" nullable="true" originalDbColumnName="Column22"
originalLength="0" pattern="" precision="0"
talendType="id_String" type="VARCHAR"/>
<column comment="" default="" key="false" label="city" length="50"
nullable="true" originalDbColumnName="city" originalLength="0"
pattern="" precision="0" talendType="id_String" type="VARCHAR"/>
<column comment="" default="" key="false" label="CountryCode"
length="3" nullable="true" originalDbColumnName="CountryCode"
originalLength="3" pattern="" precision="0"
talendType="id_String" type="VARCHAR"/>
<column comment="" default="" key="false" label="svc_area_code"
length="20" nullable="true" originalDbColumnName="svc_area_code"
originalLength="20" pattern="" precision="0"
talendType="id_String" type="VARCHAR"/>
</schema>
You can always try to do merge insert, which will update row on key duplicate.
I am using Spring HibernateTemplate, OpenSessionInViewFilter(actually I extended this class and created my own to switch to FLUSH.AUTO Mode) and Mysql for implementing hibernate many-to-many association. However when I save an object, corresponding many-to-many table's values are not inserted. Does anybody can help me? Thank you.
here is the mapping xml
<hibernate-mapping>
<class name="com.intelli.epub.domain.Content" table="CONTENT">
<id name="id" type="java.lang.Long">
<column name="ID" />
<generator class="native" />
</id>
<property name="title" type="java.lang.String">
<column name="TITLE" />
</property>
<property name="text" type="java.lang.String">
<column name="TEXT" />
</property>
<many-to-one name="writer" class="com.intelli.epub.domain.User" fetch="join">
<column name="WRITER" />
</many-to-one>
<property name="createdDate" type="java.util.Date">
<column name="CREATEDDATE" />
</property>
<set name="menus" table="MENU_CONTENT" cascade="all">
<key column="CONTENT_ID"></key>
<many-to-many column="MENU_ID" class="com.intelli.epub.domain.Menu"/>
</set>
</class>
</hibernate-mapping>
another one:
<hibernate-mapping>
<class name="com.intelli.epub.domain.Menu" table="MENU">
<id name="id" type="java.lang.Long">
<column name="ID" />
<generator class="native" />
</id>
<property name="text" type="java.lang.String">
<column name="TEXT" />
</property>
<set name="contents" table="MENU_CONTENT" inverse="true">
<key column="MENU_ID"></key>
<many-to-many column="CONTENT_ID" class="com.intelli.epub.domain.Content"/>
</set>
</class>
and when saving like this:
Content content = new Content();
content.setCreatedDate(new Date());
content.setWriter(some user here);
content.setText("some text here");
Menu menu1 = new Menu("menu1");
Menu menu2 = new Menu("menu2");
Set<Menu> menus = new HashSet();
menus.add(menu1);
menus.add(menu2);
content.setMenus(menus);
contentDao.saveOrUpdate(content);
Now menu1 and menu2 would be saved in the MENU table, However nothing happens to MENU_CONTENT table; MENU_CONTENT table doesn't have a primary key field, instead MENU_ID and CONTENT_ID are primary key together. I don't know if it's the problem. Please help me. Thank you.
I found a solution. Instead of using Spring HibernateTemplate. I wrapped it in regular session and transaction like this.
Session session = contentDao.getHibernateTemplate().getSessionFactory().getCurrentSession();
transaction = session.beginTransaction();
Content content = new Content();
content.setCreatedDate(new Date());
content.setWriter(some user here);
content.setText("some text here");
Menu menu1 = new Menu("menu1");
Menu menu2 = new Menu("menu2");
Set<Menu> menus = new HashSet();
menus.add(menu1);
menus.add(menu2);
content.setMenus(menus);
session.save(content);
transaction.commit();
session.close();
And here is my session filter which inherited from OpenSessionInViewFilter
public class SessionFilter extends OpenSessionInViewFilter {
protected Session getSession(SessionFactory sessionFactory)
throws DataAccessResourceFailureException {
Session session = super.getSession(sessionFactory);
session.setFlushMode(FlushMode.AUTO);
return session;
}
}
Does anybody know a way to handle this without bothering to write session management myself?