I have to write my own custom request handler in solr but i am getting error like org.apache.solr.common.SolrException: Error loading class 'QPRequestHandler'
Here QPRequestHandler is my custom handler that i want to plug into my SOLR 3.4
Here is what i did so far:
Created new directory lib in apache-solr-3.4.0/example/solr/
In solrconfig.xml(path: apache-solr-3.4.0/example/solr/conf/solrconfig.xml) i have added this line <lib dir="./lib" />
In solrconfig.xml i have added my custom handler like this:
<requestHandler name="/mytesthandler" class="QPRequestHandler">
<!-- initialization args may optionally be defined here -->
<lst name="defaults">
<str name="d1">d1 value</str>
</lst>
<lst name="invariants">
<str name="i1">i1 value</str>
</lst>
The sample code of QPRequestHandler is this:
public class QPRequestHandler extends RequestHandlerBase {
/**
* #param args
*/
public static void main(String[] args) {
// TODO Auto-generated method stub
}
#Override
public void init(NamedList args) {
// do nothing
}
#Override
public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse rsp) throws Exception {
rsp.add("key1", "value1");
}
}
I made a JAR file of this QPRequestHandler.java and put it in lib directory, path: apache-solr-3.4.0/example/solr/lib/
6 After restarting solr: i am getting error like org.apache.solr.common.SolrException: Error loading class 'QPRequestHandler'
Problem might be that i am creating jar file of java file and not the class file or is there any path issues or configuration error.
You would need to package the class into the jar file, which would be loaded by Solr.
You can compile the java file, by setting the classpath with the solr dependant jars.
Related
Based on the WordCount Example, I am trying to read my own json data (instead of the shakespear txts).
I am running the pipeline with:
mvn compile exec:java -Dexec.mainClass=myPkg.myClass -Dexec.args=" \
--project=myProj \
--stagingLocation=gs://myBkt/stage \
--runner=BlockingDataflowPipelineRunner \
--output=gs://myBkt/output/out \
--defaultWorkerLogLevel=DEBUG"
the output from the console is as follows:
<date> com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 68 files. Enable logging at DEBUG level to see which files will be staged.
<date> myPkg$GroupPublished apply
<date> myPkg$GroupPublished apply
INFO: GroupPublished/JsonToDatePosPlatKeyFn.out [PCollection]
<date> myPkg main
main
static void main(String[] args) {
...
Pipeline p = Pipeline.create(options);
p.apply(TextIO.Read.named("ReadJson").from(options.getInputFile()))
.apply(new GroupPublished())
.apply(ParDo.of(new FormatAsStringFn()))
.apply(TextIO.Write.named("WriteCounts").to(options.getOutput()));
}
GroupPublished transformation
static class GroupPublished extends PTransform<PCollection<String>,
PCollection<KV<DatePosPlatKey, Long>>> {
#Override
public PCollection<KV<DatePosPlatKey, Long>> apply(PCollection<String> lines) {
PCollection<DatePosPlatKey> keyList
= lines.apply(ParDo.of(new JsonToDatePosPlatKeyFn()));
PCollection<KV<DatePosPlatKey, Long>> keysCounted =
keyList.apply(Count.<DatePosPlatKey>perElement());
return keysCounted;
}
}
json row processing
static class JsonToDatePosPlatKeyFn extends DoFn<String, DatePosPlatKey>{
#Override
public void processElement(ProcessContext c) throws Exception {
JsonNode root = mapper.readTree(c.element());
for (JsonNode jsonFact : root) {
DatePosPlatKey key = new DatePosPlatKey(...construct...);
...manipulate...
c.output(key);
}
}
}
data class
#DefaultCoder(AvroCoder.class)
public static class DatePosPlatKey { ... }
stuff I've checked so far:
adding defaultWorkerLogLevel doesn't seem to make any difference to the console output
renaming the json file throws an error, so I know its been seen by TextIO
the json files have data in the format: {...}\n{...}\n...
no logging or dataflow job appears in the google cloud console
how can I better debug a complete lack of data?
can you see what I've done wrong?
Upon offline discussion it turned out the code was missing a call to p.run(), so the pipeline was only constructed but not executed.
I've been trying to insert a XML file into mongoDB with camel and I can't manage to make it work.
I've followed this tutorial for the first steps:
http://www.pretechsol.com/2014/09/apache-camel-mongodb-component-example.html
In my route, I convert it in JSON then use 'convertBodyTo(string.class) for mongo to recognize the file.
The code works well with regular route (sending the file to another folder for example). But when I run it for mongoDB, all I get in the console is my Process message again and again with my databased never being filled.As I don't receive any error message, I don't know how to find where the problem come from.
The mongoDB name, ip, users, password have been already checked multiple times.
I would be very grateful if someone could help me on this one. Here is the files I am using. (I will spare you the process file).
camel-context.xml:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://camel.apache.org/schema/spring
http://camel.apache.org/schema/spring/camel-spring.xsd">
<bean id="myDb" class="com.mongodb.Mongo">
<constructor-arg index="0">
<bean class="com.mongodb.MongoURI">
<constructor-arg index="0"
value="mongodb://username:password#192.168.3.29:27017/db" />
</bean>
</constructor-arg>
</bean>
<bean id="mongodb" class="org.apache.camel.component.mongodb.MongoDbComponent"></bean>
<camelContext xmlns="http://camel.apache.org/schema/spring">
<routeBuilder ref="camelRoute" />
</camelContext>
<bean id="camelRoute" class="infotel.camel.project01.CamelRoute" />
Here is my RoutingFile:
#Component
public class CamelRoute extends SpringRouteBuilder {
final Processor myProcessor = new MyProcessor();
final Processor myProcessorMongo = new MyProcessorMongo();
final XmlJsonDataFormat xmlJsonFormat = new XmlJsonDataFormat();
#Override
public void configure() {
xmlJsonFormat.setForceTopLevelObject(true);
from("file:xml_files?noop=true").marshal(xmlJsonFormat).convertBodyTo(String.class).process(myProcessorMongo)
.to("mongodb:myDb?database=test_bignav&collection=doc&operation=insert");
}
}
And finally here is my main:
public class MyMain {
public static void main(String[] args) throws Exception {
ApplicationContext context =
new ClassPathXmlApplicationContext("META-INF/spring/camel-context.xml");
}
}
Thanks a lot.
Edit:
Here is MyProcessorMongo edited to get the error:
public class MyProcessorMongo implements Processor{
public void process(Exchange exchange) throws Exception {
System.out.println("\n file transfered to mongo: "+ exchange.getIn().getHeader("CamelFileName"));
exchange.getProperty(Exchange.EXCEPTION_CAUGHT, Exception.class).printStackTrace();
}
}
Enable tracing with trace="true":
<camelContext trace="true" xmlns="http://camel.apache.org/schema/spring">
Dirty but quick, to get the error you can add this to you configure() method before your from :
.onException(Exception.class).handled(true).process(new Processor() {
#Override
public void process(Exchange exchange) {
exchange.getProperty(Exchange.EXCEPTION_CAUGHT, Exception.class).printStackTrace();
}
})
The handled(true) prevents your message from being processed again and again.
thanks for your help I have been able to get the error message.
The problem actually came from mongoDB itself and not camel or code. With the change on users the connection works and I'm able to insert document inside a collection.
Remove the ".process(myProcessorMongo)" from route configuration . Input xml-> json conversion->string conversion -> Mongodb. Above route will work. And you are passing the exchange object to myProcessorMongo but Out message is null so nothing will be inserted into MongoDB . Put exchange.getOut().getBody(); in the myProcessorMongo and print it.If its coming as null u have to get the input message from exchange Obj and set it back it in to Out message property in the exchange Object.
I have heard alot of success integration story when comes to Apache Camel with Fuse. HEnce. here Im just starting to explore the Fuse IDE, with just a simple task on top of my head, i would like to achieve:
Read a fix length file
Parse the fix length file
persist it to mysql database table
I am only able to get as far as:
Read the fix length file (with Endpoint "file:src/data/Japan?noop=true")
Define a Marshal with Bindy and Define a POJO package model with #FixedLengthRecord annotation
then i am stuck... HOW TO persist the POJO into mysql database table? I can see some JDBC, IBatis and JPA end point, but how to accomplish that in Fuse IDE?
My POJO package:
package com.mbww.model;
import org.apache.camel.dataformat.bindy.annotation.DataField;
import org.apache.camel.dataformat.bindy.annotation.FixedLengthRecord;
#FixedLengthRecord(length=91)
public class Japan {
#DataField(pos=1, length=10)
private String TNR;
#DataField(pos=11, length=10)
private String ATR;
#DataField(pos=21, length=70)
private String STR;
}
Well you can use all of the following components to actually read and write from the database:
JDBC
IBATIS
MyBATIS
SPRING-JDBC
SQL
Custom Processor
I am going to show you how to use the custom processor to insert the rows into a table. The main reason for this is that you will get to work with the messages and exchange and this will give you more of a insight into Camel. All of the other components can be used by following the documentation on the camel site.
So lets review what you have. You are reading the file and converting the body to a bindy object. So for each line in your text file Camel will send a bindy object of class com.mbww.model.JAPAN to the next end point. This next end point needs to talk to the database. There is one problem I can spot immediately you are using a marshal you should be using a unmarshal.
The documentation clearly states: If you receive a message from one of the Camel Components such as File, HTTP or JMS you often want to unmarshal the payload into some bean so that you can process it using some Bean Integration or perform Predicate evaluation and so forth. To do this use the unmarshal word in the DSL in Java or the Xml Configuration.
Your bindy class looks good but it is missing getters and setters modify the class to look like this:
package com.mbww.model;
import org.apache.camel.dataformat.bindy.annotation.DataField;
import org.apache.camel.dataformat.bindy.annotation.FixedLengthRecord;
#FixedLengthRecord(length=91)
public class Japan {
#DataField(pos=1, length=10)
private String TNR;
#DataField(pos=11, length=10)
private String ATR;
#DataField(pos=21, length=70)
private String STR;
public String getTNR() {
return TNR;
}
public void setTNR(String tNR) {
TNR = tNR;
}
public String getATR() {
return ATR;
}
public void setATR(String aTR) {
ATR = aTR;
}
public String getSTR() {
return STR;
}
public void setSTR(String sTR) {
STR = sTR;
}
}
First you need to create a data source to your database in your route. First thing is to add the mysql driver jar to your maven dependencies open your pom.xml file and add the following dependency to it.
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<!-- use this version of the driver or a later version of the driver -->
<version>5.1.25</version>
</dependency>
Right now we need to declare a custom processor to use in the route that will use this driver and insert the received body into a table.
So lets create a new class in Fuse IDE called PersistToDatabase code below:
package com.mbww.JapanData;
import java.sql.DriverManager;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.util.Map;
import org.apache.camel.Body;
import org.apache.camel.Exchange;
import org.apache.camel.Handler;
import org.apache.camel.Headers;
import com.mbww.model.Japan;
import com.mysql.jdbc.Statement;
public class PersistToDatabase {
#Handler
public void PersistRecord
(
#Body Japan msgBody
, #Headers Map hdr
, Exchange exch
) throws Exception
{
try {
Class.forName("com.mysql.jdbc.Driver");
} catch (ClassNotFoundException e) {
System.out.println("Where is your MySQL JDBC Driver?");
e.printStackTrace();
return;
}
System.out.println("MySQL JDBC Driver Registered!");
Connection connection = null;
try {
connection = DriverManager.getConnection("jdbc:mysql://localhost:3306/databasename","root", "password");
} catch (SQLException e) {
System.out.println("Connection Failed! Check output console");
e.printStackTrace();
return;
}
if (connection != null) {
System.out.println("You made it, take control your database now!");
} else {
System.out.println("Failed to make connection!");
}
try {
PreparedStatement stmt=connection.prepareStatement("INSERT INTO JapanDate(TNR,ATR,STR) VALUES(?,?,?)");
stmt.setString(1, msgBody.getTNR());
stmt.setString(2, msgBody.getATR());
stmt.setString(1, msgBody.getSTR());
int rows = stmt.executeUpdate();
System.out.println("Number of rows inserted: "+Integer.toString(rows));
}
catch(Exception e){
System.out.println("Error in executing sql statement: "+e.getMessage() );
throw new Exception(e.getMessage());
}
}
}
This class is a POJO nothing fancy except the #Handler annotation on the PersistRecord. This annotation tells camel that the PersistRecord method/procedure will handle the message exchange. You will also notice that the method PersistRecord has a parameter of type Japan. As mentioned earlier when you call the conversion bean in your camel route it translates each line into a Japan object and passes it along the route.
The rest of the code is just how to handle the JDBC connection and calling a insert statement.
We are almost done just one last thing to do. We need to declare this class in our camel route xml. This file will typically be called camel-route.xml or blueprint.xml depending on your arch type. Open the source tab and add the following line <bean id="JapanPersist" class="com.mbww.JapanData.PersistToDatabase"/> before the <camelContext> tag.
This declares a new spring bean called JapanPersist based on the class we just added to the camel route. You can now reference this bean inside your camel route.
Thus the final route xml file should look something like this:
<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:camel="http://camel.apache.org/schema/blueprint"
xsi:schemaLocation="
http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd
http://camel.apache.org/schema/blueprint http://camel.apache.org/schema/blueprint/camel-blueprint.xsd">
<bean id="JapanPersist" class="com.mbww.JapanData.PersistToDatabase"/>
<camelContext trace="false" id="blueprintContext" xmlns="http://camel.apache.org/schema/blueprint">
<route id="JapanDataFromFileToDB">
<from uri="file:src/data/japan"/>
<unmarshal ref="Japan"/>
<bean ref="JapanPersist"/>
</route>
</camelContext>
</blueprint>
Or see screen shot below:
Once you understand this technique you can start scaling the solution by using a splitter, connection pooling and threading to do massive amount of concurrent inserts etc.
Using the technique above you learned how to inject your own beans into a camel route which give you the ability to work with the messages directly in code.
I have not tested the code so there will probably be a bug or two but the idea should be clear.
We have a file on our server that's accessible directly through the URL, but it's a security issue at this point.
Our system opens the file in a pop-up window, but you can also get directly to the page by navigating directly to its URL.
How can we prevent this and only allow access to the file through a redirect?
Set a Session variable on the page that opens the popup:
Session["MainPageVisited"] = true;
And on the popup page check this value:
if (Session["MainPageVisited"] == null || !Session["MainPageVisited"])
{
Response.Redirect("http://www.example.com/", true);
}
For this solution to work your html file will need to be served as an aspx. Alternatively, you could create a HTTP Module if you need it to be an actual html:
Create Module
using System;
using System.Web;
public class HelloWorldModule : IHttpModule
{
public HelloWorldModule()
{
}
public String ModuleName
{
get { return "HelloWorldModule"; }
}
// In the Init function, register for HttpApplication
// events by adding your handlers.
public void Init(HttpApplication application)
{
application.BeginRequest +=
(new EventHandler(this.Application_BeginRequest));
}
private void Application_BeginRequest(Object source,
EventArgs e)
{
// Create HttpApplication and HttpContext objects to access
// request and response properties.
HttpApplication application = (HttpApplication)source;
HttpContext context = application.Context;
string filePath = context.Request.FilePath;
string fileExtension =
VirtualPathUtility.GetExtension(filePath);
if (fileExtension.Equals(".html"))
{
if (Session["MainPageVisited"] == null || !Session["MainPageVisited"])
{
// Handle it
}
}
}
public void Dispose() { }
}
To register the module for IIS 6.0 and IIS 7.0 running in Classic mode
<configuration>
<system.web>
<httpModules>
<add name="HelloWorldModule" type="HelloWorldModule"/>
</httpModules>
</system.web>
</configuration>
To register the module for IIS 7.0 running in Integrated mode
<configuration>
<system.webServer>
<modules>
<add name="HelloWorldModule" type="HelloWorldModule"/>
</modules>
</system.webServer>
</configuration>
Note, this was created without testing but it should put you on the right track. Make sure that all requests are mapped through ASP.NET for this to work (Integrated mode or set wildcard application mappings).
I have an ASP.NET 3.5 SP1 web application that uses a custom JavaScriptConverter. The code used to work at some time in the past, but has stopped working. I do not know what changes have happened in the middle server side. The problem we are seeing now is that the converter is not being invoked, so we are getting errors that System.Data.DataRow cannot be serialized.
The following is the relevant portion of web.config:
<system.web.extensions>
<scripting>
<webServices>
<jsonSerialization>
<converters>
<add name="DataSetConverter" type="Microsoft.Web.Preview.Script.Serialization.Converters.DataSetConverter, Microsoft.Web.Preview" />
<add name="DataRowConverter" type="WebUI.DataRowConverter, WebUI.DataRowConverter, Version=1.1.0.323, Culture=neutral" />
<add name="DataTableConverter" type="Microsoft.Web.Preview.Script.Serialization.Converters.DataTableConverter, Microsoft.Web.Preview" />
</converters>
</jsonSerialization>
</webServices>
</scripting>
</system.web.extensions>
A trimmed version of the class is as follows (trimmed only to avoid wasting space on unnecesary implementation):
using System;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.Data;
using System.Web.Script.Serialization;
namespace WebUI {
public class DataRowConverter : JavaScriptConverter {
private ReadOnlyCollection<Type> _supportedTypes = new ReadOnlyCollection<Type>(new Type[] { typeof(DataRow) });
public override object Deserialize(IDictionary<string, object> dictionary, Type type, JavaScriptSerializer serializer) {
// stuff
return dr;
}
public override IDictionary<string, object> Serialize(object obj, JavaScriptSerializer serializer) {
// stuff
return dictionary;
}
public override IEnumerable<Type> SupportedTypes {
get {
return this._supportedTypes;
}
}
}
}
What seems to happen is that the class is indeed being loaded (if we take it out of web.config and the project references, no breakpoints are available; put it back into web.config and copy the DLL/PDB by hand or add it to the project, breakpoints are available), but it's not being used propertly. No breakpoint anywhere in the class is hit, and no exceptions (including one thrown in a constructor added to see what happens) are thrown. It seems like the class is being loaded but never called.
This is on IIS 7.5 and IIS 7.0 in Integrated mode, if it matters.
Anyone have any ideas?
OK, just in case anyone else hits this, when calling web services through the automatically generated test pages, the custom serializers are not invoked - they are bypassed. This is apparently by design.