Strip, store and index HTML files in Solr - html

Question

I'm trying to search a collection of HTML files and also provide excerpts in Solr 6.4.1. And since the highlighting needs to return clean readable text, the HTML needs to be stripped down to bare text and stored.
But no matter what I change in the core's configuration, the field I'm specifying does not get returned in the result and highlighting for the document is always empty {}.
managed-schema:
<fieldType name="text_en_splitting_html" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100">
<analyzer type="index">
<charFilter class="solr.HTMLStripCharFilterFactory"/>
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.StopFilterFactory" words="lang/stopwords_en.txt" ignoreCase="true"/>
<filter class="solr.WordDelimiterFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" catenateAll="0" catenateWords="1"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
<filter class="solr.PorterStemFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.SynonymFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/>
<filter class="solr.StopFilterFactory" words="lang/stopwords_en.txt" ignoreCase="true"/>
<filter class="solr.WordDelimiterFilterFactory" catenateNumbers="0" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" catenateAll="0" catenateWords="0"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
<filter class="solr.PorterStemFilterFactory"/>
</analyzer>
</fieldType>
<field name="content1" type="text_en_splitting_html" multiValued="true" indexed="true" stored="true"/>
solrconfig.xml is the default one, with the default /update/extract requestHandler. The response I'm getting is:
{
"responseHeader":{
"status":0,
"QTime":4,
"params":{
"q":"*:*",
"hl":"on",
"indent":"on",
"hl.fl":"content1",
"wt":"json",
"_":"1488077854581"}},
"response":{"numFound":100,"start":0,"docs":[
{
"id":"/home/me/files/d1/test.html",
"stream_size":[62963],
"x_parsed_by":["org.apache.tika.parser.DefaultParser",
"org.apache.tika.parser.html.HtmlParser"],
"creator":["createhtml"],
"stream_content_type":["text/html"],
"viewport":["width=device-width, initial-scale=1"],
"dc_title":["A nice read"],
"content_encoding":["UTF-8"],
"resourcename":["/home/me/files/d1/test.html"],
"title":["A nice read"],
"creator_url":["http://createhtml.net"],
"content_type":["text/html; charset=UTF-8"],
"_version_":1560362957551960064}
...
},
"highlighting":{
"/home/me/files/d1/test.html":{},
...
I'm indexing with
/opt/solr/bin/post -c mycollection -filetypes html files/
I've also tried with the Tika extract handler
<requestHandler name="/update/extract" class="org.apache.solr.handler.extraction.ExtractingRequestHandler">
<lst name="defaults">
<str name="fmap.Last-Modified">last_modified</str>
</lst>
</requestHandler>
but with limited success. A "content" field now appears in the response and it contains what appears to be a poorly stripped and badly formatted version of the initial document. Highlighting appears to work but it's not clean.
So what I need Solr to do is:
clean up my HTML entirely (no tags, class names, or inline styles - just like JavaScript's .text() method)
perform the search on the stripped content
return the stripped content if I ask it to
return the highlighting on the stripped content
It seems that no matter what I change (except Tika above), "content1" is ignored.
All I'm trying to do here, simply put, is be able to search HTML files and provide excerpts like any other search engine.

Answer 1

I was unable to make this work and Tika would not correctly strip the HTML, so I fixed this by using the Solarium PHP Client for Solr and PHPQuery to parse, strip, extract data, then form my own document to post directly to Solr.
The problem was the ERH (ExtractRequestHandler) defined in solrconfig.xml which was enforcing the use of Tika. By using Solarium, the ERH was bypassed so all fields I defined in managed-schema started being used by the /update request handler.