Hi Am trying to index mysql db to solr. But its only indexing id. I tried a lot. But no success. But I tried to index geo location in the same db for testing and its indexing all the cities successfully. I just what to know what am missing. This query contains some descriptions of the products like "Solr not returning results indexed from MySQL".
My db config
<entity name="geo"
pk="id"
query="select id, title from `inlive`"
deltaImportQuery="select id, title from `inlive`"
deltaQuery="select id, title from `inlive`"
>
<field column="id" name="id"/>
<field column="title" name="title"/>
</entity>
Schema
<?xml version="1.0" encoding="UTF-8" ?>
<uniqueKey>id</uniqueKey>
<schema name="geo" version="1.5">
<field name="_version_" type="long" indexed="true" stored="true"/>
<field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />
<field name="title" type="string" indexed="true" stored="true" multiValued="false" required="true" />
</schema>
Result
{
"responseHeader":{
"status":0,
"QTime":3,
"params":{
"q":"*:*",
"indent":"on",
"wt":"json",
"_":"1506369405758"}},
"response":{"numFound":2,"start":0,"docs":[
{
"id":"1",
"_version_":1579543508130201600},
{
"id":"2",
"_version_":1579543508218281984}]
}}
Debug
{
"responseHeader": {
"status": 0,
"QTime": 275
},
"initArgs": [
"defaults",
[
"config",
"data-config.xml"
]
],
"command": "full-import",
"mode": "debug",
"documents": [
{
"id": [
1
],
"_version_": [
1579544391043776500
]
},
{
"id": [
2
],
"_version_": [
1579544391046922200
]
}
],
"verbose-output": [
"entity:geo",
[
"document#1",
[
"query",
"select id, title from `inlive`",
"time-taken",
"0:0:0.10",
null,
"----------- row #1-------------",
"id",
1,
"title",
"test title",
null,
"---------------------------------------------"
],
"document#2",
[
null,
"----------- row #1-------------",
"id",
2,
"title",
"test title 2",
null,
"---------------------------------------------"
],
"document#3",
[]
]
],
"status": "idle",
"importResponse": "",
"statusMessages": {
"Total Requests made to DataSource": "1",
"Total Rows Fetched": "2",
"Total Documents Processed": "2",
"Total Documents Skipped": "0",
"Full Dump Started": "2017-09-25 20:21:57",
"": "Indexing completed. Added/Updated: 2 documents. Deleted 0 documents.",
"Committed": "2017-09-25 20:21:57",
"Optimized": "2017-09-25 20:21:57",
"Time taken": "0:0:0.255"
}
}
Finally I found the issue for this solr.
Am using solr 6.6
So I've to rename managed-schema to schema as per this tutorial in this link http://makble.com/how-to-index-mysql-data-in-solr. Also I made a test run with geo cities. By default it took managed schema as main schema it took my old test schema fields. Its not getting schema.xml.
So the solution is need to define schema.xml
<schemaFactory class="ClassicIndexSchemaFactory"/>
Reference link : Data-config.xml and mysql - I can load only "id" column
or rename managed-schema to schema.xml or delete managed-schema
Related
We are trying to build a JasperReport for an estimate via JSON. The header and lines (multiple) are fine. However, there is one more node in the JSON file which requires to be looped in the detail table. We are not able to achieve it using subdataset. Currently, we are using JSONQL to iterate the lines.
We are unable to loop the taxes node via subdataset.
Below is the sample JSON input:
{
"taxes": [
{
"tax_slab_name": "VAT Exempt",
"tax_amount": "AED 20.00"
},
{
"tax_slab_name": "VAT 0%",
"tax_amount": "AED 30.00"
},
{
"tax_slab_name": "VAT 5 %",
"tax_amount": "AED 50.00"
}
],
"header": [
{
"estimate_no": "EST-000054",
"bill_to_address": "111 Stamm Cliffs Suite 285, Heaney Ville, Jebel Ali, Dubai, United Arab Emirates (UAE)",
"ship_to_address": "66015 شارع بلال السهلي, ممر ميسر مدني, Jebel Ali, Dubai, United Arab Emirates (UAE)",
"estimate_date": "2020-05-08",
"expiry_date": "2020-05-23",
"sub_total": 14000,
"discount": 0,
"tax_amt": 450,
"total": 14450,
"conditions": "All charges are in United States Dollar (USD) unless otherwise stated. E. and O.E.",
"status": "SAVED",
"contact_name": "Kerluke, Bartell and Dickinson (مجموعة الداوود وأولاده)"
}
],
"lines": [
{
"item_name": "Enterprise Product Development - Java/JSF/PF",
"tax_slab": 1,
"uom": "box",
"quantity": 1,
"rate": 1000,
"amount": 1000
},
{
"item_name": "Server Setup",
"tax_slab": 2,
"uom": "cm",
"quantity": 2,
"rate": 2000,
"amount": 4000
},
{
"item_name": "Backup and DR Service",
"tax_slab": 2,
"uom": "cm",
"quantity": 2,
"rate": 2000,
"amount": 4000
}
]
}
Datasource Type: JSONQL
JasperReport Version: 6.12.2 (latest)
Never mind. Got it working with sub-data-set.
<subDataset name="TaxSDS" uuid="1701ce05-88ab-4435-85c5-3ad38e0b941b">
<queryString language="jsonql">
<![CDATA[..taxes]]>
</queryString>
<field name="tax_slab_name" class="java.lang.String">
<property name="net.sf.jasperreports.jsonql.field.expression" value="tax_slab_name"/>
<fieldDescription><![CDATA[Tax Slabs]]></fieldDescription>
</field>
<field name="tax_slab_amt" class="java.lang.String">
<property name="net.sf.jasperreports.jsonql.field.expression" value="tax_amount"/>
<fieldDescription><![CDATA[Tax Amount]]></fieldDescription>
</field>
</subDataset>
<subDataset name="LinesSDS" uuid="1701ce05-88ab-4435-85c5-3ad38e0b941b">
<queryString language="jsonql">
<![CDATA[..lines]]>
</queryString>
<field name="item_name" class="java.lang.String">
<property name="net.sf.jasperreports.jsonql.field.expression" value="item_name"/>
<fieldDescription><![CDATA[Item]]></fieldDescription>
</field>
<field name="uom" class="java.lang.String">
<property name="net.sf.jasperreports.jsonql.field.expression" value="uom"/>
<fieldDescription><![CDATA[UOM]]></fieldDescription>
</field>
</subDataset>
<queryString language="jsonql">
<![CDATA[]]>
</queryString>
<field name="invoiceNo" class="java.lang.String">
<property name="net.sf.jasperreports.jsonql.field.expression" value="header.estimate_no"/>
<fieldDescription><![CDATA[Estimate No]]></fieldDescription>
</field>
I have the following json file
[
{
"id": 1,
"name": "Ramesh",
"designation": "SE",
"salary": 25000.00
},
{
"id": 2,
"name": "Anjali",
"designation": "SE",
"salary": 20000.00
},
{
"id": 2,
"name": "Harrish",
"designation": "PM",
"salary": 120000.00
},
{
"id": 4,
"name": "Rahul",
"designation": "TA",
"salary": 150000.00
}
]
Here I want to insert either all the records or none of them. Means all the four records will be inserted to employee database or none of them above. But here you can see the third record have the duplicate id 2 as second record's id hold the same id and when I run the following mule application MySQLIntegrityConstraintViolationException error is occur but first 2 records got inserted into DB as error comes in records no 3. But here what I want is to prevent from insert of records or rollback changes if all records did not get inserted into DB (means any error comes in between).
Below are my mule xml config file to insert json data into DB:
<sub-flow name="insert-fulltime-employee-details-subflow">
<dw:transform-message metadata:id="35bfe913-8de7-4b3c-9ba9-98f375a2873e" doc:name="Transform Message">
<dw:input-payload mimeType="application/json"/>
<dw:set-payload><![CDATA[%dw 1.0
%output application/java
---
payload map ((payload01 , indexOfPayload01) -> {
id: payload01.id,
name: payload01.name,
designation: payload01.designation,
salary: payload01.salary
})]]></dw:set-payload>
</dw:transform-message>
<foreach doc:name="For Each">
<db:insert config-ref="MySQL_Configuration" doc:name="Database-Insert">
<db:parameterized-query><![CDATA[insert into fulltime_employee values (#[payload.id], #[payload.name], #[payload.designation], #[payload.salary])]]></db:parameterized-query>
</db:insert>
</foreach>
<set-payload value="{"message": "All data successfully inserted into database"}" mimeType="application/json" doc:name="Set Payload"/>
</sub-flow>
How can I achieve this?
Wrap the block that has to be rolled back in the "Transactional scope". Like the below,
<transactional action="ALWAYS_BEGIN" doc:name="Transactional">
<db:insert config-ref="MySQL_Configuration" doc:name="Database-Insert">
<db:parameterized-query><![CDATA[insert into fulltime_employee values (#[payload.id], #[payload.name], #[payload.designation], #[payload.salary])]]></db:parameterized-query>
</db:insert>
</transactional>
I am learning solr (5.5.0) and running in a standalone mode.
Here is the solr-data-config.xml:
<dataConfig>
<dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver" url="jdbc:mysql://127.0.0.1/DBN" user="root" password="root"/>
<document>
<entity name="planexample" query="select * from plan where userid=60 limit 3">
<field column="planid" name="id" />
<field column="userid" name="userid" />
<field column="planname" name="planname" />
<field column="del" name="del" />
</entity>
</document>
</dataConfig>
And I add fields accordingly into managed-schema:
<field name="userid" type="int" indexed="true" stored="true" required="true" multiValued="true"/>
<field name="planname" type="string" indexed="true" stored="true" required="true" multiValued="true"/>
<field name="del" type="int" indexed="true" stored="true" required="true" multiValued="true"/>
<field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />
But when I do query on the webpage of solr, for example, ip:8983/solr/corename/select?q=del%3A1&wt=json&indent=true or ip:8983/solr/corename/select?q=userid%3A60&wt=json&indent=true, results only show the id except userid,planname,del, how to make those fields visible in results? And what does the score field mean in results?
One example of results:
{
"responseHeader": {
"status": 0,
"QTime": 1,
"params": {
"indent": "true",
"q": "del:1",
"_": "1461050814466",
"wt": "json"
}
},
"response": {
"numFound": 10,
"start": 0,
"maxScore": 0.90468985,
"docs": [
{
"id": "985963",
"score": 0.90468985
},
{
"id": "1033572",
"score": 0.90468985
},
{
"id": "1763216",
"score": 0.90468985
}
]
}
}
You need to provide a fl parameter, to get all stored fields,
do &fl=*, or a comma seperated list of fields you want i.e.
&fl=id,userid,planname,del for example.
I'm trying to make a custom YQL table to access a user's likes through the Tumblr API. I made the following table in the YQL editor and saved it as tumblr.likes:
<?xml version="1.0" encoding="UTF-8"?>
<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd">
<meta>
<author>nonphoto</author>
<documentationURL>http://www.tumblr.com/docs/api</documentationURL>
<sampleQuery>select * from {table} where username='XXX' api_key='XXX'</sampleQuery>
</meta>
<bindings>
<select itemPath="response.liked_posts" produces="JSON">
<urls>
<url>http://api.tumblr.com/v2/blog/{username}.tumblr.com/likes</url>
</urls>
<inputs>
<key id="username" type="xs:string" paramType="path" required="true" />
<key id="api_key" type="xs:string" paramType="query" required="true" />
</inputs>
</select>
</bindings>
</table>
If this is correct then I should be able to type this query into the YQL console and get a JSON response back from Tumblr:
use "XXX" as tumblr.likes;
select * from tumblr.likes where username='XXX' and api_key='XXX';
But null appears in the results entry of the response, even if debug and diagnostics are checked to prevent caching. The response even shows the correct URL, which works if I just copy and paste it into my browser. Am I missing something? Here's an example response:
{
"query": {
"count": 0,
"created": "2016-01-15T21:44:36Z",
"lang": "en-US",
"diagnostics": {
"url": [
{
"execution-start-time": "2",
"execution-stop-time": "8",
"execution-time": "6",
"id": "579e13ad-a7c3-4eea-81d9-41fda5caf243",
"content": "http://sherpa-bcp5903.dht.yahoo.com:4080/YDHTWebService/V1/get/yql.global/store%3A%2F%2FoSSGByQMlFLQhMqNCwUcp1"
},
{
"execution-start-time": "14",
"execution-stop-time": "1137",
"execution-time": "1123",
"id": "ffab25db-521f-4795-9220-a82e2ac33a9d",
"content": "http://api.tumblr.com/v2/blog/XXX.tumblr.com/likes?api_key=XXX"
}
],
"publiclyCallable": "true",
"user-time": "1146",
"service-time": "1129",
"build-version": "0.2.942"
},
"results": null
}
}
I need to index a patent catalog that has the following data structure:
"cpc": [
{
"class": "61",
"section": "A",
"sequence": "1",
"subclass": "K",
"subgroup": "06",
"main-group": "45",
"classification-value": "I"
},
{
"class": "61",
"section": "A",
"sequence": "2",
"subclass": "K",
"subgroup": "506",
"main-group": "31",
"classification-value": "I"
}
]
I was wondering what is the right approach here. I might be able to use cpc.class and combine it with multiValued="true".
I would like to find documents that match a certain CPC code. The CPC code can be partial. Right now my solution is simply use a nested reference with multiValued=true. Is there a better way of doing this?
<field name="cpc.class" type="int" indexed="true" stored="true" multiValued="true" />
<field name="cpc.section" type="string" indexed="true" stored="true" multiValued="true" />
<field name="cpc.sequence" type="int" indexed="true" stored="true" multiValued="true" />
<field name="cpc.subclass" type="string" indexed="true" stored="true" multiValued="true" />
<field name="cpc.subgroup" type="int" indexed="true" stored="true" multiValued="true" />
<field name="cpc.main-group" type="int" indexed="true" stored="true" multiValued="true" />
<field name="cpc.classification-value" type="string" indexed="true" stored="true" multiValued="true" />
The problem with this implementation is that it returns documents not actually matching the search criteria. Example:
"cpc.section:A",
"cpc.class:61",
"cpc.subclass:Q",
"cpc.main-group:8"
I get documents not having this combination. I think the current way implements the search so that every field is a list and matching values in any combination are returned. I need to narrow it down so only the right combinations are returned.
The best way to index this with Solr is to split out the nested data structures (cpcs) to flat documents and have the patent_id included there. That way an arbitrary combination of partial cpcs can be searched.