Solr return 400(bad request) while indexing the json file from localhost - json

I am new in Solr . I have setup solr in local pc. I am facing problem with the indexing of json file. I have one json file in local pc which i want to index in solr. it show some error which i have mention below.
Error
D:\Solr\Example\exampledocs>java -Durl=http://localhost:8983/solr/update -Dtype=application/json -jar post.jar timeline.json
SimplePostTool version 1.5
Posting files to base url http://localhost:8983/solr/update using content-type application/json..
POSTing file timeline.json
SimplePostTool: WARNING: Solr returned an error #400 (Bad Request) for url: http://localhost:8983/solr/update
SimplePostTool: WARNING: Response: {"responseHeader":{"status":400,"QTime":0},"error":{"msg":"Unknown command: Name [8]","code":400}}
SimplePostTool: WARNING: IOException while reading response: java.io.IOException: Server returned HTTP response code: 400 for URL: http://localhost:8983/solr/update
1 files indexed.
COMMITting Solr index changes to http://localhost:8983/solr/update..
Time spent: 0:00:00.167
Please help me , how can i solve it? Thanks in advance.
Log show
ERROR - 2014-07-30 18:38:52.330; org.apache.solr.common.SolrException; org.apache.solr.common.SolrException: Unexpected character '{' (code 123) in prolog; expected '<'
at [row,col {unknown-source}]: [1,1]
at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:176)
at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1962)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Unknown Source)
Caused by: com.ctc.wstx.exc.WstxUnexpectedCharException: Unexpected character '{' (code 123) in prolog; expected '<'
at [row,col {unknown-source}]: [1,1]
at com.ctc.wstx.sr.StreamScanner.throwUnexpectedChar(StreamScanner.java:648)
at com.ctc.wstx.sr.BasicStreamReader.nextFromProlog(BasicStreamReader.java:2047)
at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1069)
at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:213)
at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:174)
... 32 more
Json file
{ "Name" : "Matches and Schedule", "timestamp" : { "$date" : 1400825267792 }, "_id" : { "$oid" : "537ee50494" } }
{ "Name" : "meet Modi", "timestamp" : { "$date" : 1401449841192 }, "_id" : { "$oid" : "53886d3a2c" } }

Can't comment yet.
Please post a snippet of your JSON file so we can comment.
Also, in these cases, Solr log files are your best friend, they will tell you exactly what it doesn't like about the data you are posting.
Also, you don't have to specify the host if you are sending to localhost, it is the default.
EDIT: Your JSON doesn't look correct. What do you expect it to do with stuff like:
"timestamp" : { "$date" : 1400825267792 }
First of, if you need timestamp is solr to be a date/time field it needs to be in UTC format. Second, Solr doesn't support nested elements.
Finally, if you are posting multiple documents via json, the format needs to look like this:
[ {"id":"doc1","field2":"val2"} , {"id":"doc2","field2":"val3"} ]
Note that all documents are enclosed in square brackets and separated by a comma.

Related

JSON parse error: Unexpected character while running curl statement to upload file to neptune database from EC2

Any idea what's wrong with the following curl statement? I am using this to upload files to a neptune database from an EC2 instance.
curl -X POST \
-H 'Content-Type: application/json' \
https://*my neptune endpoint*:8182/loader -d '
{​​​​
"source" : "s3://<file path>/<file name>.nq",
"format" : "nquads",
"iamRoleArn" : "arn:aws:iam::##########:role/NeptuneLoadFromS3",
"region" : "us-east-1",
"failOnError" : "FALSE",
"parallelism" : "MEDIUM",
"updateSingleCardinalityProperties" : "FALSE",
"queueRequest" : "TRUE"
}​​​​'
I have used this command template multiple times before without issue. The only things that i have changed here are the neptune endpoint and the file location on s3. When i run it now, i get the following error:
{"detailedMessage":"Json parse error: Unexpected character ('​' (code 8203 / 0x200b)): was expecting double-quote to
start field name\n at [Source: (String)\"{​​​​\n \"source\" : \"s3://<file path>/<file name>.nq\",\n \"format\"
: \"nquads\",\n \"iamRoleArn\" : \"arn:aws:iam::#########:role/NeptuneLoadFromS3\",\n \"region\"
: \"us-east-1\",\n \"failOnError\" : \"FALSE\",\n \"parallelism\" : \"MEDIUM\",\n
\"updateSingleCardinalityProperties\" : \"FALSE\",\n \"queueRequest\" : \"TRUE\"\n }​​​​\"; line: 1, column: 3]",
"requestId":"4ebb82c9-107d-8578-cf84-8056817e504e","code":"BadRequestException"}
Anything that i change in the statement does not seem to have an effect on the outcome. Is there something really obvious that i am missing here?

Log in JSON format to cloudfoundry logstash

We are using Swisscom application cloud (based on Cloudfoundry) and the provided Kibana/Logstash/Elasticsearch service. Now we d like to log in JSON format from our applications to logstash.
Thats why we integrated Logstash formatter into our wildfly swarm apps and since then they log in JSON format like:
{"#version":1,"#timestamp":"2018-07-24T18:28:51.291+0200","sequence":15299,"loggerClassName":"org.jboss.as.server.logging.ServerLogger_$logger","loggerName":"org.jboss.as.server.deployment","level":"INFO","threadName":"MSC service thread 1-2","message":"WFLYSRV0027: Starting deployment of \"hospush.war\" (runtime-name: \"hospush.war\")","threadId":31,"mdc":{},"ndc":""}
I also added a filter.conf to the logstash app on swisscom appcloud with the following content:
filter {
json {
source => "message"
}
}
When I check the logs of logstash now I can see that it throws an error and no logs are transferred to Kibana.
2018-07-24 20:33:15 [APP/PROC/WEB/0] OUT [2018-07-24T18:33:15,202][WARN ][logstash.filters.json ] Error parsing json {:source=>"message", :raw=>"<14>1 2018-07-24T18:33:15.018187+00:00 HosPush.demo-test.demo-test 3694b57f-bc05-459a-880a-17c174fc6d7c [APP/PROC/WEB/0] - - {\"#version\":1,\"#timestamp\":\"2018-07-24T20:33:15.017+0200\",\"sequence\":3796,\"loggerClassName\":\"org.slf4j.impl.Slf4jLogger\",\"loggerName\":\"com.hospush.business.escalation.EscalationService\",\"level\":\"INFO\",\"threadName\":\"EJB default - 3\",\"message\":\"Found 0 patientNeeds with no open notification.\",\"threadId\":154,\"mdc\":{},\"ndc\":\"\"}\n", :exception=>#<LogStash::Json::ParserError: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
2018-07-24 20:33:15 [APP/PROC/WEB/0] OUT at [Source: (byte[])"<14>1 2018-07-24T18:33:15.018187+00:00 HosPush.demo-test.demo-test 3694b57f-bc05-459a-880a-17c174fc6d7c [APP/PROC/WEB/0] - - {"#version":1,"#timestamp":"2018-07-24T20:33:15.017+0200","sequence":3796,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"com.hospush.business.escalation.EscalationService","level":"INFO","threadName":"EJB default - 3","message":"Found 0 patientNeeds with no open notification.","threadId":154,"mdc":{},"ndc":""}
2018-07-24 20:33:30 [APP/PROC/WEB/0] OUT [2018-07-24T18:33:30,117][WARN ][logstash.filters.json ] Error parsing json {:source=>"message", :raw=>"<14>1 2018-07-24T18:33:30.002777+00:00 HosPush.demo-test.demo-test 3694b57f-bc05-459a-880a-17c174fc6d7c [APP/PROC/WEB/0] - - {\"#version\":1,\"#timestamp\":\"2018-07-24T20:33:30.001+0200\",\"sequence\":3797,\"loggerClassName\":\"org.slf4j.impl.Slf4jLogger\",\"loggerName\":\"com.hospush.business.escalation.OrphanEscalationScheduler\",\"level\":\"INFO\",\"threadName\":\"EJB default - 4\",\"message\":\"orphan escalation scheduler called...\",\"threadId\":155,\"mdc\":{},\"ndc\":\"\"}\n", :exception=>#<LogStash::Json::ParserError: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
2018-07-24 20:33:30 [APP/PROC/WEB/0] OUT at [Source: (byte[])"<14>1 2018-07-24T18:33:30.002777+00:00 HosPush.demo-test.demo-test 3694b57f-bc05-459a-880a-17c174fc6d7c [APP/PROC/WEB/0] - - {"#version":1,"#timestamp":"2018-07-24T20:33:30.001+0200","sequence":3797,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"com.hospush.business.escalation.OrphanEscalationScheduler","level":"INFO","threadName":"EJB default - 4","message":"orphan escalation scheduler called...","threadId":155,"mdc":{},"ndc":""}
My guess is, that because of source => "message" logstash parses the message property as JSON which failes. What it should do is parsing the whole "root object" as json instead of only the message property.
Could that be and if yes, how do I need to adjust the filter.conf to make it work?
Thx a lot for your help guys.
Probably not relevant, but I guess the filter here is incorrect.
This says that their is a JSON structure in the message field that should be parsed as JSON. As I see, I do not see any message field, so this will not work.

How to Index a Json file into a Solr with multivalued fields

When I'm trying to index my json file into solr I face the following error:
SimplePostTool: WARNING: Solr returned an error #400 (Bad Request) for url: http
://localhost:8983/solr/ontoneo/update
SimplePostTool: WARNING: Response: {
"responseHeader":{
"status":400,
"QTime":16},
"error":{
"metadata":[
"error-class","org.apache.solr.common.SolrException",
"root-error-class","org.apache.solr.common.SolrException"],
"msg":"Error parsing JSON field value. Unexpected OBJECT_START at [138], fie
ld=http://www.w3.org/2002/07/owl#intersectionOf",
"code":400}}
SimplePostTool: WARNING: IOException while reading response: java.io.IOException
: Server returned HTTP response code: 400 for URL: http://localhost:8983/solr/on
toneo/update
1 files indexed.
COMMITting Solr index changes to http://localhost:8983/solr/ontoneo/update...
Time spent: 0:00:00.146
The line 138 of my json file is shown as follow:
{
"#id" : "_:genid117",
"#type" : [ "http://www.w3.org/2002/07/owl#Class" ],
"http://www.w3.org/2002/07/owl#intersectionOf" : [ {
"#list" : [ {
"#id" : "_:genid121"
}, {
"#id" : "_:genid119"
} ]
} ]
},
I have done some research about it and found that the problem is the field list that is multivalued. Trying to fix it I have added the valued field into the file managed-schema.xml from my collection. The line of code is shown bellow:
<field name="list" type="text_general" multiValued="true" indexed="true" stored="true"/>
But when I run the solr to index, the problem persists. What is going wrong?
Seems like you defined the field of type "text_general" but then passing JSON data to index. Did you mean to index "JSON string" instead of JSON object?

npm unexpected token / in config.json at position 98

I have seen several unexpected token errors related to npm but this one is yet again unique to me. I could not find any similar issues posted by others. My config file just has connection objects to my local mongodb, rabbitmq "amqp", wordpress plugin and mysql DB.
Could anyone help me with this error in case they've seen it before?
C:\xampp\htdocs\livetrader\chat\node_modules\nconf\lib\nconf\stores\file.js:160
throw new Error("Error parsing your configuration file: [" + this.file + ']: ' + ex.message);
^
Error: Error parsing your configuration file:
[C:\xampp\htdocs\livetrader\chat/config.json]: Unexpected token / in JSON at position 98
Updating this post with the config.json file referenced in this error
{
"port": 1337,
"mongoose": {
"uri": "mongodb://localhost:27017/livetrader"
},
"amqp":{
"uri": "amqp://guest:guest#localhost:15672",
"queue": "TickData"
},
"sql":{
"host":"localhost",
"database":"db_main",
"user":"root",
"password":""
},
"wp":{
"host":"http://forextrader.dev",
"LOGGED_IN_KEY":"MY_LOGGED_IN_KEY",
"LOGGED_IN_SALT":"MY_LOGGED_IN_SALT",
"prefix":"wp_",
"avatarPath":"http://localhost/livetrader/wp-content/uploads/avatars",
"myCreedApi":"MY_API_KEY"
},
"guest":{
"cookie_name": "trading-forex-guest",
"ttl": 180
}
}
Updating the back-end config path
"use strict";
var nconf = require('nconf');
nconf.argv()
.env()
.file({
file: process.cwd() + '/config.json'
});
module.exports = nconf;
Your JSON file is property formatted. JSON.parse() takes it, no problem. Character 98 is not a slash, either.
There is, however, a suspicious slash in your path:
C:\xampp\htdocs\livetrader\chat/config.json
The final slash is unix-style, not windows-style. Could that be causing the problem?
If that's not it, you must be reading the wrong file, or passing it the wrong way to a function. Try using console.log to figure out what exactly is reaching JSON.parse().
Giving objects to JSON.parse() can cause unexpected errors, since .toString() is invoked. For example:
> JSON.parse({a:1})
SyntaxError: Unexpected token o in JSON at position 1
> ({a:1}).toString()
'[object Object]'
So, to recap, you must be:
Reading the wrong file
Passing an object that's not a String to JSON.parse()
Having a strange problem with the inverted slash in the path

Jdbc river stops on MapperParsingException

I am using Elastic search version 1.2.0, Jdbc river version 1.2.0.1.
Following is my Jdbc river command.
curl -XPUT 'localhost:9200/_river/tbl_messages/_meta' -d '{
"type" : "jdbc",
"jdbc" : {
"strategy" : "simple",
"url" : "jdbc:mysql://localhost:3306/messageDB",
"user" : "username",
"password" : "password",
"sql" : "select messageAlias.id as _id,messageAlias.subject as subject from tbl_messages messageAlias",
"index" : "MessageDb",
"type" : "tbl_messages",
"maxbulkactions":1000,
"maxconcurrentbulkactions" : 4,
"autocommit" : true,
"schedule" : "0 0-59 0-23 ? * *"
}
}'
Subject column's index meta data
subject: {
type: string
}
This table has 2 Million records and subject field contains arbitrary strings. Some sample data are "You're invited ","{New York:45} We rock!!","{Invitation:27}" so on.
My problem is that when jdbc river encounters one such record with {anything inside of this}, It stalls the river and throws parsing exception. It never moves on to index next records.
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [subject]
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:418)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:537)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:479)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:515)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:462)
at org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:394)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:413)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:155)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:534)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:433)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: unknown property [Inivitation]
at org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateFieldForString(StringFieldMapper.java:332)
at org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:278)
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:408)
... 12 more
Deleting this record in db,clearing data inside ES_HOME/data and recreating the river seems to be the only way to proceed until it encounter the above said formatted record again.
How do I make it to continue indexing irrespective of exception when parsing few records?
It is related to Elastic search and not the river.
https://github.com/jprante/elasticsearch-river-jdbc/issues/258
https://github.com/elasticsearch/elasticsearch/issues/2898