I am trying to import a simple mySQL table into Solr and keep failing.
Having ploughed through:
https://gist.github.com/maxivak/3e3ee1fca32f3949f052
https://wiki.apache.org/solr/DataImportHandler
How to import data from mysql to solr
and half a dozen other posts that are related to Solr4, I am in desperate need of some community help.
I need specific instructions for Solr 6, it's installed on RedHat to connect to a mySQL DB. Thanks a lot.
Please find the steps below..
Install Java Runtime Environment (JRE) version 1.8 for Solr 6.1
Set Java home "export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_101/"
Download solr 6.1 "solr-6.1.0.tgz" from http://www-eu.apache.org/dist/lucene/solr/6.1.0/
Extract tgz file using "tar zxf solr-6.1.0.tgz".
Open the port 8983 as to communicate with solr.
Go to the path ../solr-6.1.0/ and start the server with "bin/solr start"
Create a config folder named myConfig under configsets directory , move the conf folder into it. Add the data-config.xml into "/myConfig/conf".
Modified the schema.xml under the folder "/myConfig/conf" and add the data-config.xml.
add teh entry in solconfig.xml
data-config.xml
Fire the url to create the core.
http://localhost:8983/solr/admin/cores?action=CREATE&name=mycore&instanceDir=my_instance&configSet=myConfig
Add the jars "solr-dataimporthandler-6.1.0.jar", "solr-dataimporthandler-extras-6.1.0.jar" to path "/home/abhijit/Downloads/solr-6.1.0/server/lib".
Add the "solr-core-6.1.0.jar" jar to the path "/home/abhijit/Downloads/solr-6.1.0/server/lib".
Add the "solr-solrj-6.1.0.jar" jar to the path "/home/abhijit/Downloads/solr-6.1.0/server/lib".
If you do the above steps you will ge the error like :
"Error Instantiating requestHandler, org.apache.solr.handler.dataimport.DataImportHandler failed to instantiate org.apache.solr.request.SolrRequestHandler"
This problem occurred because dataimporthandler and requesthandler are loaded by two different class loaders. To solve it, ensure that solr loads its jars only from the same class loader.
Option to steps 5,6,7 create a folder lib inside the solr folder at path "/home/abhijit/Downloads/solr-6.1.0/server/solr"
and add the entry of the same to solrConfig.xml as "".
Comment the other lib entries from solrConfig.xml
the jars "solr-dataimporthandler-6.1.0.jar", "solr-dataimporthandler-extras-6.1.0.jar" to path "/home/abhijit/Downloads/solr-6.1.0/server/solr/lib".
Commented the updateRequestProcessorChain in solrConfig.xml.
Command for full import
http://localhost:8983/solr/mycore/dataimport?command=full-import&clean=true&commit=true
if you have any params to pass on then do something like this
http://localhost:8983/solr/mycore/dataimport?command=full-import&clean=true&commit=true&cabinetId=5654174
Related
I'm running an EMR with a spark cluster on AWS.
Spark version is 1.6
When running the folllowing command:
proxy = sqlContext.read.load("/user/zeppelin/ProxyRaw.csv",
format="com.databricks.spark.csv",
header="true",
inferSchema="true")
I get the following error:
Py4JJavaError: An error occurred while calling o162.load.
: java.lang.ClassNotFoundException: Failed to find data source: com.databricks.spark.csv. Please find packages at
http://spark-packages.org
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:77)
How can I solve this? I assume I should add a package but how do I install it and where?
There is many way to add packages in Zeppelin :
One of them is to actually change the conf/zeppelin-env.sh configuration file adding the packages you need e.g com.databricks:spark-csv_2.10:1.4.0 in your case to the submit options since Zeppelin uses the spark-submit command under the hood :
export SPARK_SUBMIT_OPTIONS="--packages com.databricks:spark-csv_2.10:1.4.0"
But let's say that you don't have actually access to those configuration. You can then use Dynamic Dependency Loading via %dep interpreter (deprecated) :
%dep
z.load("com.databricks:spark-csv_2.10:1.4.0")
This will require that you load the dependencies before launching or restarting the interpreter.
Another way to do it is do add the dependency you need via the interpreter dependency manager as described in the following link : Dependency Management for Interpreter.
Well,
First you need to download the CSV liv from Maven repository:
https://mvnrepository.com/artifact/com.databricks/spark-csv_2.10/1.5.0
Check the scala version that you are using. If is 2.10 or 2.11.
When you call spark-shell our spark-submit or pyspark. Or even your Zeppelin you need to add the option --jars and the path to your lib.
Like this:
pyspark --jars /path/to/jar/spark-csv_2.10-1.5.0.jar
Than you can call it as you did above.
You can see other close issue here: How to add third party java jars for use in pyspark
I am using ofbiz in my organization. I want to migrate ofbiz from derby to MySQL.
I refer the steps from
(https://cwiki.apache.org/confluence/display/OFBIZ/How+to+migrate+OfBiz+from+Derby+to+MySQL+database) here, but at the i got stuck at the end.
At the end when I type (java -jar ofbiz.jar -install) this command I am getting an exception,
C:\Users\sagar_vinod_khanke\Sagar\Apache OFBiz\Ofbiz\13.07>java -jar ofbiz.jar -
install
Exception in thread "main" org.ofbiz.base.start.StartupException: Couldn't not f
etch config instance
at org.ofbiz.base.start.Start.init(Start.java:202)
at org.ofbiz.base.start.Start.main(Start.java:127)
Caused by: java.io.IOException: Cannot load configuration properties : org/ofbiz
/base/start/-install.properties
at org.ofbiz.base.start.Config.getPropertiesFile(Config.java:229)
at org.ofbiz.base.start.Config.readConfig(Config.java:297)
at org.ofbiz.base.start.Config.getInstance(Config.java:58)
at org.ofbiz.base.start.Start.init(Start.java:200)
... 1 more
Can you please help me?
don't use - with install.
See revised Step-V
Step-V
1. Run the following command from command prompt:
java -jar ofbiz.jar install
2. Start OfBiz
3. Use webtools to import all data from XML:
a. Navigate to http://localhost:8080/catalog/
b. Go to Applications>WebTools
c. Go to section 'Entity XML Tools' and click the link 'XML Data Import Dir' -> In the 'Absolute directory path:' enter the full path of the directory where you exported the data in Step - II
I would like to ask, how can I load new module in Kamailio 4.1.2?
Actually, I have an issue, when I tried to compile my kamaiio.cfg
I've got error:
root#kamailio:/usr/local/# kamailio -c kamailio.cfg
loading modules under /usr/local/lib64/kamailio/modules/
0(25392) ERROR: <core> [sr_module.c:587]: load_module(): ERROR: load_module: could not find module <websocket> in </usr/local/lib64/kamailio/modules/>
0(25392) : <core> [cfg.y:3408]: yyerror_at(): parse error in config file /usr/local/etc/kamailio/kamailio.cfg, line 323, column 12-25: failed to load module
0(25392) ERROR: <core> [cfg.y:3272]: yyparse(): cfg. parser: failed to find command ws_handle_handshake
0(25392) : <core> [cfg.y:3411]: yyerror_at(): parse error in config file /usr/local/etc/kamailio/kamailio.cfg, line 1083, column 27: unknown command, missing loadmodule?
ERROR: bad config file (2 errors)
when I look physically to the /usr/local/lib64/kamailio/modules/ there is some modules, but websocket.so is missing.
So, how can I get and load module in Kamailio?
Thank you for help!
You haven't installed the module websocket. Edit modules.lst file in the source code directory and add websocket to include_modules variable. If you don't have modules.lst, just do:
make cfg
Alternative is to do:
make cfg include_modules="websocket"
By default, the build system for kamailio compiles and install only the modules that have the same dependencies as the core of the application. For websocket you need to install libunistring and openssl (libssl) devel packages.
Steps to load new module to Kamailio server. (Try if above answer is not working for you)
Check the modules is exist in the default module directly /usr/local/lib64/kamailio/modules.
If found, add loadmodule "module_name.so" in load module section in kamailio.cfg file.
If the module is not found in default module directory, you can check for the source code of that module in the default module source code directory /usr/local/src/kamailio-4.4/kamailio/modules.
If source code found, enter to the module directory. Then create modules' shared object file(.so) by following commands.
./configure
make
make test
make install
Then you will get a shared object file(.so). Copy that file into the default module directory. and load this module from the kamailio.cfg file as mentioned in step 1.
If module source code does not exist in the default source code directory, You need to download the source code from the web. And follow step 3 and 4.
I got a problem using MySQL with MonkeyRunner, need a help with MySQL Driver Name. The starting of my script is:
#coding : utf-8
from com.android.monkeyrunner import MonkeyRunner,MonkeyDevice
from com.ziclix.python.sql import zxJDBC
db = zxJDBC.connect('jdbc:mysql://localhost/android','dani','123456','com.mysql.jdbc.Driver')
.
.
etc
Each time I got an error that com.mysql.jdbc.Driver not found even when using org.git.mm.mysql.Driver. I set classpath as following :
export CLASSPATH=$CLASSPATH:/usr/share/java/mysql-connector-java.5.1.10.jar
Any tips or hits?
Regards,,,
You need to include the relevant jar in the correct location. There should be a libs folder where you need to put the jar. It should be in the top level of your project folder, at the same level as the src and res folders.
I need to Insert something in the DB. im using JDBC as a connector, jython the script, mysql the DB and the script is running in CentOS.
my code looks something like this:
from com.android.monkeyrunner import MonkeyRunner, MonkeyDevice, MonkeyImage
from com.ziclix.python.sql import zxJDBC
db=zxJDBC.connect("jdbc:mysql://XXX.XXX.XXX.XXX:3306/dbname","USER","PASSWORD","org.gjt.mm.mysql.Driver")
c=db.cursor() c.execute("INSERT INTO tablename values ('X','X','X')")
before that, I downloaded and decompressed the file from here (in the desktop)
I added the path to classpath by doing this
export PATH=/home/XX/Desktop/mysql-connector-java-5.1.22
and when I ran the script, it gave me this error
zxJDBC.DatabaseError.driver [org.gjt.mm.mysql.Driver] not found
what have I done wrong? is the name of the driver name correct? because I just copied it in one of the tutorials that I've seen. or probably did I install the driver correctly?
Thanks.
this is how I managed to solve the error:
Download the JDBC driver here
Extract the tar.gz file anywhere you want.
You will find mysql-connector-java-5.1.22-bin.jar inside that folder. Copy that and paste to (in my case) /%android-sdk%/tools/lib
Add the new location of mysql-connector-java-5.1.22-bin.jar to classpath
do the script like this
from com.android.monkeyrunner import MonkeyRunner, MonkeyDevice,
MonkeyImage
from com.ziclix.python.sql import zxJDBC
db=zxJDBC.connect("jdbc:mysql://XXX.XXX.XXX.XXX:3306/dbname","USER","PASSWORD","com.mysql.jdbc.Driver")
c=db.cursor()
c.execute("INSERT INTO tablename values ('X','X','X')")
db.commit()
Hope this helps to those who will need it in the future. :)
How are you running jython? If you're using the standalone install, i.e. java -jar jython.jar, then from the Java Documentation ...
-jar
When you use this option, the JAR file is the source of all user classes, and other user class path settings are ignored.
... you can't add anything to the classpath. Repackaging the required classes into the jython jar is one approach or this answer has an alternative solution - basically add the jython.jar to the classpath too (either using -cp or CLASSPATH) and run the org.python.util.jython class directly.
I got the sample problem in windows7,I slove this problem by this:
download the JDBC driver
add the mysql-connector-java-ver-bin.jar to envionment variables:
such as:
CLASSPATH : C:\xxx-path\mysql-connector-java-5.1.41-bin.jar
then I slove this problem