I am trying to migrate quote details using the Kingswaysoft SSIS CRM migration package from CRM2016 on premise to D365.
To read the data I am using Fetch XML, on the quotedetails entity and want to link to the quote entity to retreive quotenumber and revisionnumber.
My current query looks like the following -
<fetch mapping='logical'>
<entity name="quotedetail">
<attribute name="baseamount" />
<link-entity name="quote" from="quoteid" to="quoteid" link-type="inner" alias="quote">
<attribute name="quotenumber" />
<attribute name="revisionnumber" />
<filter>
<condition attribute='revisionnumber' operator='eq' value='1'/>
</filter>
</link-entity>
</entity>
</fetch>
Using the data viewer shows that all values in the linked entity are returned as null, and therefore I cannot use these as a lookup for the quoteid.
Data Viewer Results Image
Using this FetchXML statement in XRMtoolbox however does show the fields correctly.
According to KingswaySoft official "Known Limitation", this is a known issue with a workaround:
In the CDS/CRM Source Component, with the FetchXML option, if you are using linked entities in your FetchXML queries, the CRM component returns NULL values for the fields from linked entities. To avoid this, provide an alias for each linked entity.
I.e. you should provide an alias in the FetchXML to avoid this issue. I got it working in my case by providing the alias on the attribute, meaning you would have to do the following:
<fetch mapping='logical'>
<entity name="quotedetail">
<attribute name="baseamount" />
<link-entity name="quote" from="quoteid" to="quoteid" link-type="inner">
<attribute name="quotenumber" alias="quote_quotenumber"/>
<attribute name="revisionnumber" alias="quote_revisionnumber" />
<filter>
<condition attribute='revisionnumber' operator='eq' value='1'/>
</filter>
</link-entity>
</entity>
</fetch>
Reference: https://www.kingswaysoft.com/products/ssis-integration-toolkit-for-microsoft-dynamics-365/help-manual/crm/Known-Limitations
Related
I'm learning SSRS and stuck in a scenario where I want to display some records (containing null values) in a multi-value parameter. I'm using Dynamics 365 Fetch query. For a simple drop-down allow null works but for multi-values it's not working. Is there any way how can I make it work?
Remember, I'm able to retrieve the null values but what I want is to display those null values as string, like 'Null', in the multi-value parameter list
This is my main query:
<fetch version="1.0" output-format="xml-platform" mapping="logical" distinct="false">
<entity name="contact">
<attribute name="fullname"/>
<attribute name="telephone1"/>
<attribute name="jobtitle"/>
<attribute name="emailaddress1"/>
<attribute name="creditlimit"/>
<attribute name="parentcustomerid"/>
<attribute name="birthdate"/>
<attribute name="address1_stateorprovince"/>
<attribute name="address1_country"/>
<attribute name="address1_city"/>
<order attribute="fullname" descending="false"/>
<filter>
<condition attribute="telephone1" operator="not-null"/>
</filter>
<filter>
<condition attribute="emailaddress1" operator="not-null"/>
</filter>
<filter type="or">
<condition attribute="jobtitle" operator="null">
<value>Null</value>
</condition>
<condition attribute="jobtitle" operator="in" value="#jobtitle"/>
</filter>
</entity>
</fetch>
Parameter gets values through this:
<fetch distinct="true">
<entity name="contact">
<attribute name="jobtitle" />
<filter>
<condition attribute="telephone1" operator="not-null" />
</filter>
<filter>
<condition attribute="emailaddress1" operator="not-null" />
</filter>
</entity>
</fetch>
Although null values exists there but I want them to show as string 'Null'...
I want to use Solr with MongoDB and MySQL together and need to combine in single core.
For example, I have a MongoDB collection which has depends on MySQL's one table,
I tried both with separate Solr core it's working fine but i want it in single core, i don't know its possible or not, if its possible then how we can use?
Updated
Here my DIHs: (Data import Handler)
- Solr with MySQL
<dataConfig>
<dataSource
name="MySQl"
type="JdbcDataSource"
driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost/test"
user="root" password="root"
batchSize="-1"/>
<document>
<entity
query="select * from master_table"
name="master">
</entity>
</document>
</dataConfig>
- Solr with MongoDB
<dataConfig>
<dataSource
name="MyMongo"
type="MongoDataSource"
database="test" />
<document>
<entity
processor="MongoEntityProcessor"
query=""
collection="MarketCity"
datasource="MyMongo"
transformer="MongoMapperTransformer"
name="sample_entity">
<field column="_id" name="id" mongoField="_id" />
<field column="keyName" name="keyName" mongoField="keyName"/>
</entity>
</document>
</dataConfig>
So i want to do with the single core.
You can read the data from Mysql and MongoDB. Merge this records in single record and the index the same into solr.
To get the data from MySql, use any programming language and fetch the data.
For example you can use Java and fetch the data from mysql.
Apply the same logic to MongoDB. Get all the required records from mongoDB using Java.
Now By using the SolrJ apis create the solrDocument. Read more about the SolrDOcument and other apis here
Once your create the instance of SolrDocument then add the data that you fetched from Mysql and MongoDB into it using the below method.
addField(String name, Object value)
This will add a field to the document.
You can prepare the document something like this.
SolrInputDocument document = new SolrInputDocument();
document.addField("id", "123456");
document.addField("name", "Kevin Ross");
document.addField("price", "100.00");
solr.add(document);
solr.commit();
Get a solr instance of HttpSolrClient.
Once the SolrDocument is ready, index it to solr.
I have set up Solr's data import handler as instructed in manual. Solr reads the records from a MySQL database. The database has large number of records (expected is milliards/billions).
I have read that batch size does not work for MySQL because the JDBC driver does not support it. I have tried setting it up to -1. In this case, Solr performs one select and gets all records from the DB and indexes them.
Now, I have problem, since a timeout occurred while indexing and caused it to stop. I see that Solr hasn't written any id value in the properties file after the exception occurred. I am not sure how to proceed with indexing the rest of the records.
Can anyone suggest to me how to set up Solr with MySQL for a proper data import?
Below is data config I am currently using.
<dataConfig>
<dataSource type="JdbcDataSource" name="ds-2" driver="com.mysql.jdbc.Driver" url="jdbc:mysql://localhost:3306/myowndb" batchSize="-1" />
<document name="statuses">
<entity name="status" query="select s.*, ti.id2, ti.value2 from tblTable1 s inner join tblTable2 ti on s.table2Id = ti.id;">
<field column="id" name="id" />
<field column="statusID" name="statusId" />
<field column="type" name="type" />
<field column="date" name="date" />
<field name="id2" column="id2" />
<field name="value2" column="value2" />
</entity>
</document>
</dataConfig>
EDIT:
Based on my tests today, it looks like batchSize is working. If batchSize is set to -1, it will make single request to MySQL retrieving all rows at once. If set to some value greater than 0, it will put every record in memory before processing.
New question is next: how to set up data import handler so it can index in batches? Not only to perform batch select from database, but to index collected set before collecting next one.
EDIT: Specified question
New question that came up from reading is next: is it possible to mark row in database as processed? There are only two events available in DIH, onImportStart and onImportEnd.
Current flow in ideas lead me to implement EntityProcessor. If it would be possible to know when some row is indexed, it would also be easy to mark isIndexed flag in database for indexed row. This is in case I implement custom EntityProcessor.
Thanks, I edited my data-config.xml file
It's like this
<dataConfig>
<dataSource type="JdbcDataSource"
driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost/lol"
user="root"
password="n"/>
<document name="content">
<entity name="id">
query="SELECT id from foo"
</entity>
</document>
</dataConfig>
When I run
http://localhost:8983/solr/dataimport?command=full-import
On the browser I get this
<response><lst name="responseHeader"><int name="status">0</int><int name="QTime">1</int></lst><lst name="initArgs"><lst name="defaults"><str name="config">data-config.xml</str></lst></lst><str name="status">idle</str><str name="importResponse"/><lst name="statusMessages"><str name="Time Elapsed">0:0:6.299</str><str name="Total Requests made to DataSource">1</str><str name="Total Rows Fetched">0</str><str name="Total Documents Processed">0</str><str name="Total Documents Skipped">0</str><str name="Full Dump Started">2013-06-28 11:17:34</str><str name="">Indexing failed. Rolled back all changes.</str><str name="Rolledback">2013-06-28 11:17:34</str></lst><str name="WARNING">This response format is experimental. It is likely to change in the future.</str></response>
I believe that configuration should look more like (note, query should be an attribute of the entity element):
<document name="content">
<entity name="id" query="SELECT id from foo">
<!--I assume you have a field in Solr and a column in MySQL, both of which are named "id"-->
</entity>
</document>
I want to index MySQL table with Solr4.0 row by row . I have installed the necessary java my database is called 'twitter_db' and the table i want ot index called "tweets"
and i login using user : root and no password
the schema is so i added it in the :
<dataConfig>
<dataSource driver="org.hsqldb.jdbcDriver" url="jdbc:mysql://localhost/twitter_db" user="root" password="" />
<document name="tweet">
<entity name="tweet" query="select * from tweets">
<field column="tweet_id" name="tweet_id" />
<field column="text" name="text" />
<field column="user" name="user" />
<field column="tweet_time" name="tweet_time" />
<field column="topic_kw" name="topic_kw" />
<field column="timestamp" name="timestamp" />
</entity>
</document>
</dataConfig>
and the solrconfig changes is:
<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
<lst name="defaults">
<str name="config">db-data-config.xml</str>
</lst>
</requestHandler>
and when i hit [root]:8983/solr/db/dataimport?command=full-import
for full import
it fails : error message in the GUI is :
Indexing failed. Rolled back all changes.
and the the part of the error message in the log is :
SEVERE: Exception while processing: tweet document : SolrInputDocument[]:org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to execute query: select * from tweets Processing Document # 1
at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:71)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:252)
at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:209)
i wonder if it's a wrong data in selecting the database of the table
i tried a similar Question it's a similar problem but i didn't find the answer there
Check driver="org.hsqldb.jdbcDriver" it is hsqldb rather it should be pointing to MySQL driver class. Try updating the driver class to appropriate driver for MySQL & you can run it in debug mode.