I have a json object like this,
{
"id" : "123",
"name" : "aaa",
"loc" : {
"lat" : "",
"lon" : ""
},
}
In solr i added a field like
<field name="loc" type="string" indexed="true" stored="true" multiValued="true"/>
Then i tried adding the json doc to solr
But i couldn't see the 'loc' object being indexed in solr.
Am i doing wrong? Do i need to add anything in schema.xml
Can someone help me out of this pls?
I afraid you have to flatten the nested object to become something like
loc_lat = value of loc.lat
loc_lon = value of loc.lon
And of course, two new fields to store the data.
How are you adding your JSON? Because it does not look like the format required by Solr.
You need to conform to that. Also, if those lat/longs are really geographic coordinates, Solr has a dedicated geospatial support. Though you will need to get data into the appropriate shape either on the client side or with something like UpdateRequestProcessor, most likely ConcatFieldUpdateProcessorFactory.
post to
/solr/update/json?commit=true?split=/&f=txt:/**
see http://lucidworks.com/blog/schemaless-solr-part-1/
Related
I have converted some columns to JSON using the columns to json node. The output from that is:
{
"Material" : 101,
"UOM" : "GRAM",
"EAN" : 7698,
"Description" : "CHALK BOX"
}
I would like to add the value of the material property as a key to each JSON object. So, my desired output is:
"101": {
"Material" : 101,
"UOM" : "GRAM",
"EAN" : 7698,
"Description" : "CHALK BOX"
}
I have tried entering the following expression in the JSON transformer node but all I get is a question mark in the new column it generates:
$Material$:{"Material":$Material$,"UOM":$UOM$,"EAN":$EAN$,"Description":$Description$}
I have also tried replacing the $Material$ with "Material" but got the same result.
How would I go about this, please?
In case you convert the Material column to String (for example with String Manipulator), you can easily configure the Columns to JSON:
As you can see the Data bound key is the important part.
The String Manipulator node configuration (string($Material$)):
I finally managed to solve this by a different method.
I split the JSON data into several columns, then used the join function to create a string in the required order. I put the resulting string through the string to JSON node to create the new JSON object.
Thanks for all your tips and comments !
I would like to log API request/response as json format.
The expected LogEntry is something like
{
"timestamp" : "...",
"level" : "DEBUG",
"headers" : [
"header1" : "value1",
"header2" : "value2",
"header3" : "value3"
],
"requestPayload" : "<Request Json>" // prefereablly as sub-document, worst case string is fine.
"labels" : [ //key fields which can be used for searching the logentry
"searchField1" : "....",
"searchField2" : "....",
"searchField3" : "...."
]
}
My Question is :
Using Logback, How to log nested fields (e.g. headers, labels, requestPaylod in above example), as json sub-document. I tried MDC, but its limited to
Map of 'String, String' only, and considers all fields after first level as String.
I hate to write my custom logger for this, and would like to use the goodness of proven logging frameworks(logback/log4j) to control the logging level, time stamping log event, etc.
For request/response logging try Logbook and logstash-logback-encoder. You'll be able to use Markers to add structured contents.
For adding 'your own' fields in a JSON structure, I've written a code generator you might find interesting, it adds builder support: json-log-domain
You can use logback-contrib's JsonLayout inside any Logback appender. For example:
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<layout class="ch.qos.logback.contrib.json.classic.JsonLayout">
<jsonFormatter class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
<prettyPrint>false</prettyPrint>
</jsonFormatter>
<timestampFormat>yyyy-MM-dd' 'HH:mm:ss.SSS</timestampFormat>
<appendLineSeparator>true</appendLineSeparator>
</layout>
</appender>
Given log statements like this ...
MDC.put("header1", "headerValue1");
logger.info("hello!");
logger.info("good bye!");
... the use of JsonLayout would result in Logback writing this:
{"timestamp":"2017-08-15 09:06:41.813","level":"INFO","thread":"main","mdc":{"header1":"headerValue1"},"logger":"com.stackoverflow.logback.LogbackTest","message":"hello!","context":"default"}
{"timestamp":"2017-08-15 09:06:41.887","level":"INFO","thread":"main","mdc":{"header1":"headerValue1"},"logger":"com.stackoverflow.logback.LogbackTest","message":"good bye!","context":"default"}
I think this ticks the boxes of writing your log events as JSON documents whilst still retaining Logback's behaviour such as "control the logging level, time stamping log event, etc". There is some built-in support for changing the JSON format (e.g. you can include/exclude context, logger name etc) but the JsonLayout class provides an extension point which would allow you to change the attribute names in the resulting JSON by extending JsonLayout and overriding toJsonMap().
Edit 1: addressing your reply ("my question was more on how to achieve json subdocument/nesting as mentioned. MDC is limited to map of only") ... you could serialise your complex MDC values to JSON and add the serialised JSON representations to MDC. For example:
Map<String, Object> complexMdcValue = new HashMap<>();
Map<String, Object> childMdcValue = new HashMap<>();
childMdcValue.put("name", "Joe");
childMdcValue.put("type", "Martian");
complexMdcValue.put("child", childMdcValue);
complexMdcValue.put("category", "etc");
MDC.put("complexNestedValue", objectMapper.writeValueAsString(complexMdcValue));
logger.info("hello!");
Would produce this output (in which the MVC logged key "complexNestedValue" is JSON containing a sub document):
{"timestamp":"2017-08-27 18:03:46.706","level":"INFO","thread":"main","mdc":{"complexNestedValue":"{\"category\":\"etc\",\"child\":{\"name\":\"Joe\",\"type\":\"Martian\"}}"},"logger":"com.stackoverflow.logback.LogbackTest","message":"hello!","context":"default"}
I'm wondering if there is a JSON format for editing JSON ?
eg if I had some json
{ "first name" : "Joe" }
and + another json file
{ "action" : "add",
"dest" : "root" -- json pointer maybe
"value" : { "surname : "Blogs" }
}
would get me =
{ "first name" : "Joe", "surname : "Blogs" }
similarly a delete , and change ..
Is there a JSON format that does this? It may be part of a noSQL db or may not be or some javascript library - but i'm not after a JS library more is there a JSON format to do this, One would assume someone has done this before!
It doesn't work like that. JSON is just a data-interchange format (http://www.json.org/). There is no transactional semantic or anything that goes with it.
Goal:
Given the myInfoObject definition below, I wish to be able to do this:
println new groovy.xml.MarkupBuilder(myInfoObject).toPrettyString()
Premise:
The following is one of the most amazing and convenient features of Groovy for my use cases: Brilliant dynamic serializing of complex nested objects into sensible JSON. Just pass the object, and get the JSON.
Example - A simple Map within a Map
import groovy.json.*
def myInfoMap = [
firstname : 'firstname',
lastname : 'lastname',
relatives : [
mother : "mom",
father : "dad"
]
]
myInfoJson = new JsonBuilder(myInfoMap)
//One line, straight to JSON object, no string writer/parser conversions
//Works on any object, extremely elegant, even handles deep nesting
//Alternatively, add .toPrettyString() for the string representation
Returns:
{
"firstname": "firstname",
"lastname": "lastname",
"relatives": {
"mother": "mom",
"father": "dad"
}
}
I have read through all the MarkupBuilder examples and docs I could find, and there does not seem to be any equivalent for XML. Here is the closest I could find, it is not nearly the same.
http://www.leveluplunch.com/groovy/examples/build-xml-from-map-with-markupbuilder/
XML and JSON are fundamentally different, but it's still common for objects to be represented by XML in a similar fashion. An XML equivalent would require at least one optional parameter specifying how the data should be represented, but I think a sensible default would be something like:
<myInfoMap>
<firstname>firstname</firstname>
<lastname>lastname</lastname>
<relatives>
<relative>
<mother>mom</mother>
</relative>
<relative>
<father>dad</father>
</relative>
</relatives>
</myInfoMap>
...Which has to be built manually with intimate knowledge of the structure like so...
def writer = new StringWriter()
def builder = new groovy.xml.MarkupBuilder(writer)
builder.myInfoMap {
myInfoMap.each{key, value ->
if (value instanceof Map){
"${key}"{
value.each{key2, value2 ->
"${key[0..key.size()-2]}"{
"${key2}" "${value2}"
}
}
}
}else{
"${key}" "${value}"
}
}
}
println writer.toString()
I even tried to be clever and make it a bit dynamic, but you can see how far from the JSONBuilder example it is, even in a simple case.
If this is currently impossible and not on anybody's radar, I will submit my first JIRA ticket to the Groovy project as a feature request. Just want to be sure before I do. Please just comment if you think this is the next step.
Try grails.converters.XML. In your case:
def myInfoMap = [
firstname: 'firstname',
lastname : 'lastname',
relatives: [
mother: "mom",
father: "dad"
]
]
println new grails.converters.XML(myInfoMap)
would result in:
<?xml version="1.0" encoding="UTF-8"?>
<map>
<entry key="firstname">firstname</entry>
<entry key="lastname">lastname</entry>
<entry key="relatives">
<entry key="mother">mom</entry>
<entry key="father">dad</entry>
</entry>
</map>
Sorry I may be wrong but XmlUtil.serialize I think covers what you want.
MrHaki: groovy-goodness-pretty-print-xml
I did some work a while back that I was going to release as a plugin, I didn't release it in the end. Got side tracked with dns on the fly update which didn't end up working. Anyhow process xml controller and follow gsp view for it. But I think MrHaki has put it a lot more elegantly and has some really good examples
I am trying to process a Json data in Java. I have the data in below format (it is nested data structure with arrays etc.)
person.name,person.friend[0],person.friend[1],person.address.city,person.address.country
1,x,y,kolkata,india
2,a,b,london,uk
The first line is header denoting the nested object hierarchy. I want a json in below format,
{
"data" : [
{
"name" : "1",
"friend" : ["x","y"],
"address" : { "city" : "kolkata", "country" : "india" }
},
{
"name" : "2",
"friend" : ["a","b"],
"address" : { "city" : "london", "country" : "uk" }
} ]
}
The object structure is dynamic and I dont know the columns or header in advance, i.e. I can not use any predefined POJO to get populated with the data. In this example, it "Person" object but it may be any object structure.
I have gone through Jackson or Gson API, but none seems to fulfill this requirement. Is there any API that can help? or any other wayout?
Thanks
You need to do it in 2 steps.
First, you have to parse your CSV. I recommend superCSV. Parsing CSV may be fancy sometimes, so I really recommend you to use a library for that.
Second, you can serialize into JSON. Then you can use GSON, jackson, flexjson, whatever.
After a long Google...I found that the only option is to represent a collection based object structure in flat file is repeated rows,
person.name,person.friends,person.address.city,person.address.country
1,x,kolkata,india
1,y,kolkata,india
2,a,london,uk
2,b,london,uk
where the non-array elements repeats. We need to form a json from this, then need to filter or club the same object by its ID (here person.name)