We recently migrated from Splunk to ELK. We wanted to log our message as json for better searchability in Kibana.
Our application was using vert.x 3.9. I came across https://reactiverse.io/reactiverse-contextual-logging but that requires vertx-core to be updated to 4.x. This will be a major change for us and I looked for other options. Also, I joined this team recently and new to Vert.x
I came across net.logstash.logback:logstash-logback-encoder and able to log the messages as json. I used io.vertx.core.json.JsonObject to convert my message and key values to convert to json. I created a wrapper class that returns the string for given message and key/values as shown below.
public class KeyValueLogger {
public static String getLog(final String message) {
return new JsonObject().put("message", message).encode();
}
public static String getLog(final String message, final Map<String, Object> params) {
return new JsonObject().put("message", message).mergeIn(new JsonObject(params)).encode();
}
}
Every log message will call the above KeyValueLogger.getLog to get json message. Since Vert.x is a reactive application, is there a better solution to convert the log messages to json? Most of the log messages are in worker threads. I am afraid if any is in event loop thread then it might impact the performance.
Thanks in advance!
I am able to use logstash-logback-encoder by referring to https://github.com/logfellow/logstash-logback-encoder/tree/logstash-logback-encoder-6.6#readme in section Examples using Markers: . This eliminated custom code to convert the message and key values conversion to json. I tested the performance and it is really good and no need of reactive/async for my use case.
Sample Code
import java.util.Map;
import static net.logstash.logback.marker.Markers.appendEntries;
#Test
public void logTest(){
Map<String, String> map = ImmutableMap.of("p1", "v1", "p2", "v2");
log.info(appendEntries(map), "startTest 100");
log.info("startTest {} , key1={}, key2= {}", 101, "v1", "v2");
log.info(appendEntries(map), "startTest {} , key1={}, key2= {}", 101, "v1", "v2");
log.info("startTest 102");
log.error(appendEntries(map), "error occured", new RuntimeException());
}
Sample logback configuration
Note: <logstashMarkers/> is required to log the Map as key values in the json log output.
<appender name="console" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
<providers>
<timestamp/>
<logLevel/>
<contextName/>
<threadName>
<fieldName>thread</fieldName>
</threadName>
<callerData>
<fieldName>src</fieldName>
<classFieldName>class</classFieldName>
<methodFieldName>method</methodFieldName>
<fileFieldName>file</fileFieldName>
<lineFieldName>line</lineFieldName>
</callerData>
<logstashMarkers/>
<pattern>
<pattern>
{
"message": "%message"
}
</pattern>
</pattern>
<stackTrace>
<throwableConverter class="net.logstash.logback.stacktrace.ShortenedThrowableConverter">
<maxDepthPerThrowable>30</maxDepthPerThrowable>
<maxLength>2048</maxLength>
<shortenedClassNameLength>20</shortenedClassNameLength>
<exclude>^sun\.reflect\..*\.invoke</exclude>
<exclude>^net\.sf\.cglib\.proxy\.MethodProxy\.invoke</exclude>
<rootCauseFirst>true</rootCauseFirst>
</throwableConverter>
</stackTrace>
</providers>
</encoder>
</appender>
Performance test
Added the following performance test and it took ~ 0.08286483 milli seconds = 82864.8304 nano seconds per log message on average.
private static Map<String, Object> getMap(){
int count = 5;
Map<String, Object> map = new HashMap<String, Object>();
for(int i=0; i<count; ++i){
map.put("key"+i, UUID.randomUUID());
}
return map;
}
#Test
public void logPerformanceTest(){
long startTime = System.nanoTime();
final int COUNT = 10000;
for(int i=0; i<COUNT; i++) {
log.info(appendEntries(getMap()), "startTest 100");
}
long time = System.nanoTime() - startTime;
System.out.println("###### TOOK " + time + " (ns) ");
}
There might be a better solution to convert the log messages to JSON in Vert.x. One option could be using the Vert.x event bus to send log messages from worker threads to a dedicated logger verticle. This way, the logger verticle can handle the JSON conversion and logging, freeing up the worker threads from performing additional tasks.
This approach also ensures that the logging process does not impact the performance of the event loop thread, as it's running in a separate verticle.
First, you need to create a logger verticle that will handle the JSON conversion and logging. This verticle can subscribe to a specific address on the event bus to receive log messages. The logger verticle can look like this:
public class LoggerVerticle extends AbstractVerticle {
#Override
public void start() {
EventBus eventBus = vertx.eventBus();
eventBus.consumer("logger.address", message -> {
JsonObject logMessage = (JsonObject) message.body();
// Convert log message to JSON format and log it
System.out.println(logMessage.encodePrettily());
});
}
}
Next, in your worker threads, you can send log messages to the logger verticle using the event bus. To send a log message, you can create a JsonObject with the log message and other key-value pairs, and then send it to the logger.address on the event bus.
Map<String, Object> params = new HashMap<>();
params.put("timestamp", System.currentTimeMillis());
params.put("worker-thread", Thread.currentThread().getName());
JsonObject logMessage = new JsonObject(params);
logMessage.put("message", "Log message from worker thread");
EventBus eventBus = vertx.eventBus();
eventBus.send("logger.address", logMessage);
Finally, you need to deploy the logger verticle in your Vert.x application, so that it starts receiving log messages.
Vertx vertx = Vertx.vertx();
vertx.deployVerticle(new LoggerVerticle());
This way, you can ensure that log messages are converted to JSON format before logging and that the logging process does not impact the performance of the event loop thread.
It's like having a dedicated chef to cook the dishes and serve them, freeing up the waiter to focus on taking orders and managing the restaurant.
When I throw a WebApplicationException in my service like the following, I get a 404 response code which is what I'm expecting
throw new WebApplicationException(Response.Status.NOT_FOUND);
When I attempt to put a little more information in my exception via a ResponseBuilder, I'm getting a 500 response code. Here's my enhanced code and the response message I get back:
List<Error> errors = new ArrayList<>();
errors.add(new Error(ErrorCode.PDF_GENERATION_ERROR));
ErrorResponse errResponse = new ErrorResponse();
errResponse.setErrors(errors);
throw new WebApplicationException(Response.status(Response.Status.NOT_FOUND).entity(errResponse).build());
javax.ws.rs.WebApplicationException:
com.sun.jersey.api.MessageExceptio n: A message body writer for Java
class idexx.ordering.rest.common.types.ErrorRe sponse, and Java type
class idexx.ordering.rest.common.types.ErrorResponse, and MIME media
type application/octet-stream was not found
My REST service has the following method signature:
#Produces("application/pdf")
#GET
#Path("/{id}")
public Response getPdfById(#PathParam("id") final Long id) {
Is there anyway I can get that extra error code information into the response when I throw a WebApplicationException?
Try this:
String errResponse = "convert your errors to text";
throw new WebApplicationException(Response.status(Response.Status.NOT_FOUND).entity(errResponse).build());
Probably this will work because JAX-RS can convert basic types, but to convert List<Error> you need to register a custom MessageBodyWriter.
Hi I am trying to write small app with REST Json. I have some method that returns ArrayList of entity objects. And I am doing that:
#RequestMapping(value="/workers/", method = RequestMethod.GET)
public #ResponseBody ArrayList<Workers> showAllEmployes() throws Exception
{
ArrayList<Workers> workers = new ArrayList<Workers>();
workers = (ArrayList<Workers>) spiroService.getAllWorkers();
return workers;
}
And after this I got:
HTTP Status 500. The server encountered an internal error that prevented it from fulfilling this request.
When I try to return primitive data type then all is ok. I have nothing in server logs. And I have necessary imports. Please some tip.
Seems you have issue in produce json format, try this.
#RequestMapping(value = "/workers/", method = RequestMethod.GET,
produces={MediaType.APPLICATION_JSON_VALUE})
I'm using a Restful web service (Jersy implementation) with a JSF application and used Json to get the data as follows:
carObjectDao = new GenericDAO<carObject>(carObject.class);
List<carObject> allCars = carObjectDao.readAll();
Gson gson = new Gson();
String carString = gson.toJson(allCars);
System.err.println(carString );
return carString ;
i run the application in debug mode and allCars is filled with the data correctly, but after that an exception is thrown :
java.lang.UnsupportedOperationException: Attempted to serialize java.lang.Class: org.hibernate.proxy.HibernateProxy. Forgot to register a type adapter?
i don't know the root cause of the exception
This is a known problem: Could not serialize object cause of HibernateProxy
JSon can't deserialize HibernateProxy objects, so you either unproxy or remove em.
Or, you can eager fetch the lazy data.
Try parsing through ObjectMapper as
carObjectDao = new GenericDAO<carObject>(carObject.class);
List<carObject> allCars = carObjectDao.readAll();
String carString = new ObjectMapper().writeValueAsString(allCars);
System.err.println(carString );
return carString ;
I am trying to host a WCF service that responds to incoming requests by providing a json output stream. I have the following type
[DataContract]
[KnownType(typeof(List<HubCommon>))]
[KnownType(typeof(Music))]
[KnownType(typeof(AppsAndPlugins))]
[KnownType(typeof(Notifications))]
[KnownType(typeof(Scenes))]
[KnownType(typeof(Skins))]
[KnownType(typeof(Ringtones))]
[KnownType(typeof(Alarms))]
[KnownType(typeof(Widgets))]
[KnownType(typeof(Wallpapers))]
[KnownType(typeof(Soundsets))]
public class HubCommon{}
In my *.svc.cs file I do the following
List<HubCommon> hubContent = _ldapFacade.GetResults(query);
MemoryStream stream = new MemoryStream();
DataContractJsonSerializer serializer = new DataContractJsonSerializer(typeof(HubCommon));
serializer.WriteObject(stream,hubContent);
So essentially I am trying to serialize a List to Json but I get the following error on the "WriteObject" execution:-
The server encountered an error processing the request. The exception message is 'Type 'System.Collections.Generic.List`1[[HubContentCore.Domain.HubCommon, HubContentCore, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]]' with data contract name 'ArrayOfHubCommon:http://schemas.datacontract.org/2004/07/HubContentCore.Domain' is not expected. Add any types not known statically to the list of known types - for example, by using the KnownTypeAttribute attribute or by adding them to the list of known types passed to DataContractSerializer.'
What am I missing here ?
Thanks in advance.
The type of your DataContractJsonSerializer is HubCommon but you are writing an object of type List<HubCommon> and HubCommon is not added to the KnownTypAttribute