Importing data from CSV, XML or Json files to solr using dataimport - json

I am trying to import data from files like CSV, XML and Json to solr core, I am new to solr so this might be a straightforward to some but for me I have tried many online suggestions but not getting the desired result.
So I have a json file and I have enabled dataimport by adding the following requestHandler to solrconfig.xml:
<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
<lst name="defaults">
<str name="config">solr-data-config.xml</str>
</lst>
</requestHandler>
And in the solr-data-config.xml:
<dataConfig>
<dataSource name="dfs" type="FileDataSource"/>
<document>
<entity name="sourcefile" processor="FileListEntityProcessor" fileName=".*" rootEntity="false" baseDir="${solr.install.dir}/example/exampledocs">
<entity name="entryline" processor="LineEntityProcessor" url="${sourcefile.fileAbsolutePath}" rootEntity="true" dataSource="fds" separator=","/>
</entity>
</document>
The updated version:
<dataConfig>
<script><![CDATA[
function CategoryPieces(row) {
var pieces = row.get('manu_id_s').split('/');
var arr = new Array();
for (var i=0; i < pieces.length; i++) {
row.put('manu_id_s' + i, pieces[i].trim());
arr[i] = pieces[i].trim();
}
row.put('manu_id_s', (pieces.length - 1).toFixed());
row.put('manu_id_s', arr.join('/'));
row.put('manu_id_s', arr.join('/'));
return row;
}
]]></script>
<dataSource type="FileDataSource" />
<document>
<entity
name="document"
processor="FileListEntityProcessor"
baseDir="${solr.install.dir}/example/exampledocs/khaled"
fileName=".*.xml$"
recursive="false"
rootEntity="false"
dataSource="null">
<entity
name="test"
processor="XPathEntityProcessor"
transformer="script:CategoryPieces"
url="${document.fileAbsolutePath}"
useSolrAddSchema="true"
stream="true">
</entity>
</entity>
</document>
</dataConfig>
And I have added a json, csv and xml files to the path, in the ui of solr when I use the dataimport it says that e.g. Requests: 0 , Fetched: 26 , Skipped: 0 , Processed: 0 and nothing in the logs, can someone advice how to add the data in the file to solr?
Here is my sample data in the xml file:
<add>
<doc>
<field name="id">USD</field>
<field name="name">One Dollar</field>
<field name="manu">Bank of America</field>
<field name="manu_id_s">boa</field>
<field name="cat">currency</field>
<field name="features">Coins and notes</field>
<field name="price_c">1,USD</field>
<field name="inStock">true</field>
</doc>
<doc>
<field name="id">EUR</field>
<field name="name">One Euro</field>
<field name="manu">European Union</field>
<field name="manu_id_s">eu</field>
<field name="cat">currency</field>
<field name="features">Coins and notes</field>
<field name="price_c">1,EUR</field>
<field name="inStock">true</field>
</doc>
<doc>
<field name="id">GBP</field>
<field name="name">One British Pound</field>
<field name="manu">U.K.</field>
<field name="manu_id_s">uk</field>
<field name="cat">currency</field>
<field name="features">Coins and notes</field>
<field name="price_c">1,GBP</field>
<field name="inStock">true</field>
</doc>
<doc>
<field name="id">NOK</field>
<field name="name">One Krone</field>
<field name="manu">Bank of Norway</field>
<field name="manu_id_s">nor</field>
<field name="cat">currency</field>
<field name="features">Coins and notes</field>
<field name="price_c">1,NOK</field>
<field name="inStock">true</field>
</doc>
</add>
Now the data returned from the query looks like:
{
"responseHeader":{
"status":0,
"QTime":0,
"params":{
"q":"*:*",
"_":"1547556805546"}},
"response":{"numFound":4,"start":0,"docs":[
{
"manu_id_s":"boa",
"id":"USD",
"_version_":1622731088078045184},
{
"manu_id_s":"eu",
"id":"EUR",
"_version_":1622731088081190912},
{
"manu_id_s":"uk",
"id":"GBP",
"_version_":1622731088081190913},
{
"manu_id_s":"nor",
"id":"NOK",
"_version_":1622731088082239488}]
}}

Related

Hide string name from field in odoo 12

When you define a field in odoo, by default, it displays the string for that value and next to it, the value.
If you don't put the field between group tags, the string value, doesn't appear.
The problem is that I am defining a tree view which doesn't contain group tags:
<record model="ir.ui.view" id="field_partida_fertilizer_tree">
<field name="name">field.diary.partida.fertilizer.tree.view</field>
<field name="model">field.diary.partida.fertilizer</field>
<field name="priority" eval="17"/>
<field name="arch" type="xml">
<tree string="Fertilizante">
<field string= 'Nombre' name="fertilizer"/>
<field string='N' name="nitrato" options='{"show_string": False}' widget="percentpie"/>
<field string='P' name="fosforo" options='{"fg_color": "white"}' widget="percentpie"/>
<field string='K' name="potasio" options='{"fg_color": "white"}' widget="percentpie"/>
<field string='Cantidad (kg)' name="quantity"/>
</tree>
</field>
</record>
By using widget percentpie, I can display this:
I want to hide the letter N next to the graph but not the column and I can't find any property of the widget to do that. The only way I could trick this is by giving the color white to the text so it disappears with the background, but the column cant be centered.
I have tried with the option show_string I found on GitHub but it doesn't work.
Any help would be awesome...
Thank you!
Odoo defines a condition in the PercentPie template to show the string attribute vaue:
<span t-if="widget.string"><t t-esc="widget.string"/></span>
You can alter the template and define a new condition base on a value (show_string ) passed through the options attribute:
<?xml version="1.0" encoding="UTF-8"?>
<templates id="template" xml:space="preserve">
<t t-extend="FieldPercentPie">
<t t-jquery="span" t-operation="attributes">
<attribute name="t-if">widget.string and (widget.attrs.options.show_string === undefined or widget.attrs.options.show_string)</attribute>
</t>
</t>
</templates>
Create a new XML file with the above content and add it under the qweb section of the manifest file.
Example:
<record model="ir.ui.view" id="field_partida_fertilizer_tree">
<field name="name">field.diary.partida.fertilizer.tree.view</field>
<field name="model">field.diary.partida.fertilizer</field>
<field name="priority" eval="17"/>
<field name="arch" type="xml">
<tree string="Fertilizante">
<field string='Nombre' name="fertilizer"/>
<field name="nitrato" options='{"show_string": False}' widget="percentpie"/>
<field name="fosforo" options='{"show_string": False}' widget="percentpie"/>
<field name="potasio" options='{"show_string": False}' widget="percentpie"/>
<field string='Cantidad (kg)' name="quantity"/>
</tree>
</field>

XPATH _ Extract values XML

How Can I extract the characteristics values for the following XML code using XPATH?
Example: Extract displayName, firsName, lastName, etc.
This is my XML data:
<directory>
<fieldset>
<field id="displayName">Display name</field>
<field id="firstName">First name</field>
<field id="lastName">Last name</field>
<field id="preferredName">Preferred name</field>
<field id="jobTitle">Job title</field>
<field id="workPhone">Work Phone</field>
<field id="canUploadPhoto">Can Upload Photo</field>
</fieldset>
<employees>
<employee id="229">
<field id="displayName">Susan</field>
<field id="firstName">TestName</field>
<field id="canUploadPhoto">no</field>
</employee>
</employees>
</directory>
You can get actual values along the following XPath expression:
/directory/employees/employee/field[#id="displayName"]/text()

Wordpress - add featured post with image using MySQL

I am trying to add featured post with image in Wordpress using MySQL. I want to do it only by MySQL(no php, only SQL). I tried to add post and then I added meta for the post too. Still, I cannot see featured post with image. Please advise where I am wrong? I have generated XML for records, please check and let me know where I am wrong.
Post Data:
<?xml version="1.0" encoding="utf-8"?>
<table_data name="wp_posts">
<row>
<field name="ID">62</field>
<field name="post_author">1</field>
<field name="post_date">2018-04-24 00:00:00</field>
<field name="post_date_gmt">2018-04-24 00:00:00</field>
<field name="post_content"><img class="alignnone size-medium wp-image-26" src="http://test.com/wp-content/uploads/2018/photos/testing2.jpg" alt="test" width="300" height="219"></field>
<field name="post_title">Manual post</field>
<field name="post_excerpt"></field>
<field name="post_status">publish</field>
<field name="comment_status">open</field>
<field name="ping_status">open</field>
<field name="post_password"></field>
<field name="post_name"></field>
<field name="to_ping"></field>
<field name="pinged"></field>
<field name="post_modified">0000-00-00 00:00:00</field>
<field name="post_modified_gmt">0000-00-00 00:00:00</field>
<field name="post_content_filtered"></field>
<field name="post_parent">0</field>
<field name="guid"></field>
<field name="menu_order">0</field>
<field name="post_type">post</field>
<field name="post_mime_type"></field>
<field name="comment_count">0</field>
</row>
<row>
<field name="ID">63</field>
<field name="post_author">1</field>
<field name="post_date">2018-04-24 00:00:00</field>
<field name="post_date_gmt">2018-04-24 00:00:00</field>
<field name="post_content"></field>
<field name="post_title">testing2</field>
<field name="post_excerpt"></field>
<field name="post_status">inherit</field>
<field name="comment_status">closed</field>
<field name="ping_status">open</field>
<field name="post_password"></field>
<field name="post_name">testing2</field>
<field name="to_ping"></field>
<field name="pinged"></field>
<field name="post_modified">0000-00-00 00:00:00</field>
<field name="post_modified_gmt">0000-00-00 00:00:00</field>
<field name="post_content_filtered"></field>
<field name="post_parent">62</field>
<field name="guid">http://test.com/wp-content/uploads/2018/photos/testing2.jpg</field>
<field name="menu_order">0</field>
<field name="post_type">attachment</field>
<field name="post_mime_type">image/jpeg</field>
<field name="comment_count">0</field>
</row>
</table_data>
Meta of Post
<table_data name="wp_postmeta">
<row>
<field name="meta_id">211</field>
<field name="post_id">62</field>
<field name="meta_key">_thumbnail_id</field>
<field name="meta_value">63</field>
</row>
</table_data>
What is missing and how to add using MySQL?
Thanks

Whys is my MySQL query returning rows, but no values?

I'm using this query:
LOAD XML LOCAL INFILE '\\MC_Library_Video.xml' INTO TABLE video ROWS IDENTIFIED BY '<Item>';
On this XML file (abbreviated)
<Item>
<field name="Filename">V:\13 Assassins\13 Assassins.mkv</field>
<field name="Name">13 Assassins</field>
<field name="File Type">mkv</field>
<field name="Genre">Adventure; Drama; Foreign; Martial Arts</field>
<field name="Date">40662</field>
<field name="Bitrate">7501</field>
<field name="Image File">13 Assassins.jpg</field>
<field name="Media Type">Video</field>
<field name="File Size">7041096346</field>
<field name="Duration">7509</field>
<field name="Date Created">1322620528</field>
<field name="Date Modified">1322620528</field>
<field name="Date Imported">1348075046</field>
<field name="Keywords">...</field>
<field name="Description">...</field>
<field name="Media Sub Type">Movie</field>
<field name="Sample Rate">48000</field>
<field name="Channels">6</field>
<field name="Bit Depth">16</field>
<field name="Compression">mkv video (video: AVC1, audio: DTS Audio)</field>
<field name="Width">1280</field>
<field name="Height">528</field>
<field name="MPAA Rating">R</field>
<field name="Director">Takashi Miike</field>
<field name="Actors">...</field>
<field name="Gross Revenue">$17,054,213</field>
<field name="Budget">$6,000,000</field>
<field name="FPS">23.9760400000000012</field>
<field name="Genres">Adventure; Drama; Foreign; Martial Arts</field>
</Item>
And I'm getting this result:
285 row(s) affected Records: 285 Deleted: 0 Skipped: 0 Warnings: 0
But when I look at the data for the table (SELECT * FROM media.video;) I get nothing. There's no data there. Am I missing something? The result makes it appear as if the data was read in, but the table is still blank.

Solr data indexing , returns only one field

I am trying to use solr for indexing data from my data base.
After I index data, when I query *.*
I get just the id field in result. not all the fields which I had in my query.
My data-config.xml
<document name="content">
<entity name="documen" query="SELECT indexId ,brand_id, category_id, product_name from Production">
<field column="indexId" name="id" />
<field column="category_id" name="categoryid" />
<field column="brand_id" name="brandid" />
<field column="product_name" name="id" />
</entity>
</document>
My schema.xml looks like this :
<field name="id" type="int" indexed="true" stored="true" required="true"/>
<field name="categoryid" type="int" indexed="true" stored="true"/>
<field name="brandid" type="int" indexed="true" stored="true" />
<field name="productname" type="string" indexed="true" stored="true"/>
When I query using *.* I get
<doc>
<str name="id">1</str>
<long name="_version_">1426653005792411648</long></doc>
<doc>
<str name="id">2</str>
<long name="_version_">1426653005793460224</long></doc>
<doc>
I get only "id" field as result.
Actually, whatever field is in "uniquekey" tag is returned as query result