I'm trying to serialize flink Row to kafka, I don't have json schema with me, but have columns names, also Row can be accessed with index and fields, with plain json below code is working fine, however with nested json, for type Row, it is printing rowking and arity. I'm using JsonRowSerializationSchema with withTypeInfo builder.
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.api.java.typeutils.RowTypeInfo;
import org.apache.flink.connector.kafka.sink.KafkaRecordSerializationSchema;
import org.apache.flink.formats.json.JsonRowSerializationSchema;
import org.apache.flink.types.Row;
public class JsonSerializerBuilder {
private final String[] columnNames;
public JsonSerializerBuilder(String[] columnNames) {
this.columnNames = columnNames;
}
public KafkaRecordSerializationSchema<Row> build() {
String outputTopic = "test_topic";
TypeInformation<Row> opTypeInfo = getTypeInformation();
JsonRowSerializationSchema jsonRowSerializationSchema =
JsonRowSerializationSchema.builder().withTypeInfo(opTypeInfo).build();
return KafkaRecordSerializationSchema.builder()
.setValueSerializationSchema(jsonRowSerializationSchema)
.setTopic(outputTopic)
.build();
}
private TypeInformation<Row> getTypeInformation() {
TypeInformation[] typeInformationArray = new TypeInformation[columnNames.length];
Arrays.fill(typeInformationArray, TypeInformation.of(Object.class));
return new RowTypeInfo(typeInformationArray, columnNames);
}
}
input: {"id": 1,"name":"mike","school_details":{"location": "uk","name": "test"}}
output: {"id": 1,"name":"mike","school_details":{"kind":"INSERT","arity":2}}
so basically Object TypeInformation is not working with Row type objects, how can I fix it.
Related
I am trying json parsing with gson in a small java applicaiton. I have a json string which comes from .Net business layer, has a field as "1999-08-24T00:00:00". In my model like User model, I have java.time.Instant birthDay field. With gson i am trying to get json string to my user model. Also I have a InstantDeserializer class. But when I try to convert it I got a message like java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)..
Before the instant type I was using Date class. I wrote DateDeserializer class but I know Date class is deprecated. I googled to much page. I tried many things but i didn't how to figure out. So i just want to ask where I am making mistakes. What sould I do? How can I make my code more clear or what is the best approch? If you could give some code examples, I can understand better.
Any advice is appreciated..
Here is my code..
JSON String :
{
"Value":{
"ID":"123",
"NAME":"John",
"SURNAME":"Concept",
"BIRTHDAY":"1999-08-24T00:00:00",
"PAYMENTINFORMATION":[
{
"ID":"1",
"PAYMENTINFO":"RECIEVED"
}
]
},
"Succued": true
}
UserModel class
package Models;
import java.lang.reflect.Array;
import java.util.ArrayList;
import java.util.List;
import java.time.LocalDateTime;
import java.util.Date;
import com.google.gson.annotations.SerializedName;
import java.time.Instant;
public class UserModel {
private long id;
private String name;
private String surname;
private Instant birthday;
private List<PaymentModel> paymentInformation;
//GETTER SETTER
public UserModel() {
paymentInformation= new ArrayList<>();
}
}
InstantDeserializer class
package Util;
import com.google.gson.JsonDeserializationContext;
import com.google.gson.JsonDeserializer;
import com.google.gson.JsonElement;
import com.google.gson.JsonParseException;
import java.lang.reflect.Type;
import java.time.Instant;
import java.time.LocalDateTime;
import java.time.ZoneId;
public class InstantDeSerializer implements JsonDeserializer<Instant> {
#Override
public Instant deserialize(JsonElement jelement, Type type, JsonDeserializationContext jdc) throws JsonParseException {
Instant insObj= Instant.ofEpochMilli(jelement.getAsJsonPrimitive().getAsLong());
return insObj;
}
}
And Main class
public class JSONTryMe {
public static void main(String[] args) throws Exception {
JSONObject responseJSON = new JSONObject(jsonString);
if (responseJSON.isNull("Value")) {
return;
}
GsonBuilder build = new GsonBuilder();
build.registerTypeAdapter(Instant.class, new InstantDeSerializer());
Gson gObj = build.create();
UserModel user = gObj.fromJson(responseJSON.getJSONObject("Value").toString(), UserModel.class);
System.out.println(user.getBirthday().toString());
}
}
Ant the error stackTrace is
java.lang.NumberFormatException: For input string: "1999-08-24T00:00:00"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:589)
at java.lang.Long.parseLong(Long.java:631)
at com.google.gson.JsonPrimitive.getAsLong(JsonPrimitive.java:206)
at Util.InstantDeSerializer.deserialize(InstantDeSerializer.java:25)
at Util.InstantDeSerializer.deserialize(InstantDeSerializer.java:21)
at com.google.gson.internal.bind.TreeTypeAdapter.read(TreeTypeAdapter.java:69)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:131)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:222)
at com.google.gson.Gson.fromJson(Gson.java:932)
at com.google.gson.Gson.fromJson(Gson.java:897)
at com.google.gson.Gson.fromJson(Gson.java:846)
at com.google.gson.Gson.fromJson(Gson.java:817)
at Source.JSONTryMe.main(JSONTryMe.java:85)
Here are several conceptual flaws:
Birth dates should use LocalDate
Your JSON input provides ISO datetime, but your deserializer tries to read milliseconds since epoch. Use LocalDate#parse() for this
I am getting a result from my unit test that I don't quite understand.
Controller Code
package com.rk.capstone.controllers;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;
import com.rk.capstone.model.domain.User;
import com.rk.capstone.model.services.user.IUserService;
/**
* REST Controller for /register endpoint
*/
#RestController
#RequestMapping("/register")
public class RegisterController {
private final IUserService userService;
public RegisterController(IUserService userService) {
this.userService = userService;
}
#RequestMapping(value = "/user", method = RequestMethod.POST)
public ResponseEntity<User> registerNewUser(#RequestBody User user) {
if (userService.findByUserName(user.getUserName()) == null) {
user = userService.saveUser(user);
return ResponseEntity.status(HttpStatus.CREATED).body(user);
} else {
return ResponseEntity.status(HttpStatus.CONFLICT).body(null);
}
}
}
Unit Test Code:
package com.rk.capstone.controllers;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTest;
import org.springframework.boot.test.mock.mockito.MockBean;
import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
import org.springframework.test.context.junit4.SpringRunner;
import org.springframework.test.web.servlet.MockMvc;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.rk.capstone.model.dao.UserDao;
import com.rk.capstone.model.domain.User;
import com.rk.capstone.model.services.user.IUserService;
import static org.mockito.BDDMockito.given;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.post;
import static org.springframework.test.web.servlet.result.MockMvcResultHandlers.print;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;
/**
* Class Provides Unit Testing for RegisterController
*/
#RunWith(SpringRunner.class)
#WebMvcTest(RegisterController.class)
public class RegisterControllerTest {
#MockBean
private IUserService userService;
#Autowired
private MockMvc mockMvc;
private User user;
private String userJson;
#Before
public void setup() {
user = new User("rick", "k", "rick#email.com", "rkow", "abc123");
ObjectMapper objectMapper = new ObjectMapper();
try {
userJson = objectMapper.writeValueAsString(user);
} catch (JsonProcessingException e) {
e.printStackTrace();
}
}
#Test
public void testRegisterNewUserPostResponse() throws Exception {
given(this.userService.findByUserName(user.getUserName())).willReturn(null);
given(this.userService.saveUser(user)).willReturn(user);
Assert.assertNotNull("Mocked UserService is Null", this.userService);
this.mockMvc.perform(post("/register/user").content(userJson).
contentType(MediaType.APPLICATION_JSON)).
andExpect(status().isCreated()).
andDo(print()).andReturn();
}
}
The result of the print() is below, I do not understand why the Body is empty. I have tried numerous things I've read on other posts and blogs and no matter what I try the Body is always empty. Adding a Content-Type header in the controller response makes no difference.
MockHttpServletResponse:
Status = 201
Error message = null
Headers = {}
Content type = null
Body =
Forwarded URL = null
Redirected URL = null
Cookies = []
What is confounding me even more, is when I run the actual application and perform a POST using PostMan to the /register/user endpoint the response contains the body and status code I expect, a User represented via JSON, e.g.
Status Code: 201 Created
Response Body
{
"userId": 1,
"firstName": "rick",
"lastName": "k",
"emailAddress": "rick#email.com",
"userName": "rk",
"password": "abc123"
}
Any help or ideas is appreciated, using SpringBoot 1.4.0.RELEASE.
UPDATE: For some reason the following mocked method call is returning null in the controller under test.
given(this.userService.saveUser(user)).willReturn(user);
This thread ultimately turned me on to a solution:
Mockito when/then not returning expected value
Changed this line:
given(this.userService.saveUser(user)).willReturn(user);
to
given(this.userService.saveUser(any(User.class))).willReturn(user);
How an application which serves resources (.js; .css...) could return JSON entities if an error occurs ?
I wrote a ControllerExceptionHandler according to this blog:
package com.my.rest;
import com.my.rest.errors.ErrorMessage;
import com.my.rest.errors.ErrorMessageFactory;
import com.my.service.errors.NotFoundException;
import com.my.service.errors.ValidationException;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.ControllerAdvice;
import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.bind.annotation.ResponseBody;
import org.springframework.web.bind.annotation.ResponseStatus;
import org.springframework.web.context.request.WebRequest;
import org.springframework.web.servlet.mvc.method.annotation.ResponseEntityExceptionHandler;
import static org.springframework.http.HttpStatus.NOT_FOUND;
#ControllerAdvice
#ResponseBody
public class GlobalControllerExceptionHandler extends ResponseEntityExceptionHandler {
#Autowired
private ErrorMessageFactory messageFactory;
#ResponseStatus(NOT_FOUND)
#ExceptionHandler({NotFoundException.class})
public ResponseEntity<Object> handleServiceException(RuntimeException e, WebRequest request) {
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON_UTF8);
return handleExceptionInternal(e, messageFactory.generateMessage(e), headers, HttpStatus.NOT_FOUND, request);
}
}
If the request URL is: https://my-server/resources/a.js, but a.js is not found, this ExceptionHandler leads to a HttpMediaTypeNotAcceptableException thrown by AbstractMessageConverterMethodProcessor because ContentNegotiationManager uses a strategy based on the extension .js, and determines that ErrorMessage is not applicable to the media type application/javascript.
My question is: Is there a way to ignore the check of media type to always send JSON error responses so that the client-side error management can parse it ?
Thank you
In my entities I have some hibernate annotations for validation, like #NotEmpty, #Pattern.. and others
In my controller, on save action, it has an #Valid parameter.
But if any entity has any required field, and there is no annotation I will have problems.
So I would like to test each entity, to ensure they have the necessary notes.
Something like:
#Test(expect=IllegalArgumentException.class)
public void testAllNull() {
Person p = new Persson(); // Person name has an #NotEmpty
validator.validate(p);
}
But how to validate it? Who is called to check #Valid?
Thanks.
I found out how to check:
#Autowired
private LocalValidatorFactoryBean validator;
...
validator.validateProperty(object, propertyName)
Here is a Spring v4.1.x based example of a test validating presence and processing of the #Valid annotation and building of custom JSON response in case of an error.
jUnit
import com.fasterxml.jackson.core.type.TypeReference;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.context.annotation.Bean;
import org.springframework.http.MediaType;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.test.web.servlet.MvcResult;
import org.springframework.test.web.servlet.setup.MockMvcBuilders;
import javax.inject.Inject;
import java.util.List;
import static org.abtechbit.miscboard.util.JsonUtils.toJson;
import static org.hamcrest.CoreMatchers.is;
import static org.hamcrest.CoreMatchers.notNullValue;
import static org.junit.Assert.assertThat;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.post;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.content;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(classes = {
RegistrationValidationTest.MockDependencies.class,
})
public class RegistrationValidationTest {
#Inject
MockMvc mvc;
#Test
public void validatesRegistration() throws Exception {
Registration registration = ... //build an invalid Registration object
MvcResult result = mvc.perform(post(RegistrationController.CONTEXT_REGISTER).
contentType(MediaType.APPLICATION_JSON).
content(toJson(registration))).
andExpect(status().isBadRequest()).
andExpect(content().contentType(MediaType.APPLICATION_JSON)).
andReturn();
assertThat(result.getResolvedException(), is(notNullValue()));
String content = result.getResponse().getContentAsString();
assertThat(content, is(notNullValue()));
List<Message> messages = JsonUtils.fromJson(content, new TypeReference<List<Message>>() {
});
assertThat(messages.size(), is(1));
}
public static class MockDependencies {
#Bean
public MockMvc mvc() {
return MockMvcBuilders.standaloneSetup(new RegistrationController()).build();
}
}
}
Controller
import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.MethodArgumentNotValidException;
import org.springframework.web.bind.annotation.*;
import javax.validation.Valid;
import java.util.List;
import java.util.stream.Collectors;
#Controller
public class RegistrationController
{
public static final String CONTEXT_REGISTER = "/register";
#RequestMapping(value = CONTEXT_REGISTER, method = RequestMethod.POST)
#ResponseBody
public String register(#RequestBody #Valid Registration registration) {
//perform registration
}
#ExceptionHandler(MethodArgumentNotValidException.class)
public ResponseEntity<List> handleValidationException(MethodArgumentNotValidException ex) {
//Build a list of custom Message{String message;} objects
List<Message> messages = ex.getBindingResult().getAllErrors().
stream().map(e->new Message(e.getDefaultMessage())).collect(Collectors.toList());
return ResponseEntity.status(HttpStatus.BAD_REQUEST).contentType(MediaType.APPLICATION_JSON).body(messages);
}
}
Spring MVC Test Framework might be a good choice. By using this, you can be assured that validations in your tests runs codes as Spring #MVC actually works.
Actually, the #Valid annotation is detected by HandlerMethodInvoker, which processes annotations on the handler methods of Spring controllers. Internally, the actual validation logic is delegated to the Validator bean depending on your application context settings. (Hibernate Validator is widely used.)
By default configuration (e.g. <mvc:annotation-driven />), LocalValidatorFactoryBean is used internally to process #Valid annotation as #Falci noted, but it may differ time to time. Instead, Spring MVC Test Framework provides the same environment as the main application uses, hence a good choice.
If I get the following json from a RESTful client, how do I elegantly unmarshal the java.util.Date? (Is it possible without providing (aka. hard-coding) the format, that's what I mean by elegantly...)
{
"class": "url",
"link": "http://www.empa.ch",
"rating": 5,
"lastcrawl" : "2009-06-04 16:53:26.706 CEST",
"checksum" : "837261836712xxxkfjhds",
}
The cleanest way is probably to register a custom DataBinder for possible date formats.
import java.beans.PropertyEditorSupport;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
public class CustomDateBinder extends PropertyEditorSupport {
private final List<String> formats;
public CustomDateBinder(List formats) {
List<String> formatList = new ArrayList<String>(formats.size());
for (Object format : formats) {
formatList.add(format.toString()); // Force String values (eg. for GStrings)
}
this.formats = Collections.unmodifiableList(formatList);
}
#Override
public void setAsText(String s) throws IllegalArgumentException {
if (s != null)
for (String format : formats) {
// Need to create the SimpleDateFormat every time, since it's not thead-safe
SimpleDateFormat df = new SimpleDateFormat(format);
try {
setValue(df.parse(s));
return;
} catch (ParseException e) {
// Ignore
}
}
}
}
You'd also need to implement a PropertyEditorRegistrar
import org.springframework.beans.PropertyEditorRegistrar;
import org.springframework.beans.PropertyEditorRegistry;
import grails.util.GrailsConfig;
import java.util.Date;
import java.util.List;
public class CustomEditorRegistrar implements PropertyEditorRegistrar {
public void registerCustomEditors(PropertyEditorRegistry reg) {
reg.registerCustomEditor(Date.class, new CustomDateBinder(GrailsConfig.get("grails.date.formats", List.class)));
}
}
and create a Spring-bean definition in your grails-app/conf/spring/resources.groovy:
beans = {
"customEditorRegistrar"(CustomEditorRegistrar)
}
and finally define the date formats in your grails-app/conf/Config.groovy:
grails.date.formats = ["yyyy-MM-dd HH:mm:ss.SSS ZZZZ", "dd.MM.yyyy HH:mm:ss"]
Be aware that the new version of Grails 2.3+ supports this type of feature out of the box.
See Date Formats for Data Binding
With that said, if you are forced to use a version of Grails prior to 2.3, the CustomEditorRegistrar
can be updated using the following code to eliminate the deprecation warning, and also uses the #Component annotation, which allows you to remove / skip the step of adding the bean directly in resources.groovy.
Also not that I changed the grails configuration property name to grails.databinding.dateFormats, which matches the property now supported in Grails 2.3+. Finally, my version is a .groovy, not .java file.
import javax.annotation.Resource
import org.codehaus.groovy.grails.commons.GrailsApplication
import org.springframework.beans.PropertyEditorRegistrar
import org.springframework.beans.PropertyEditorRegistry
import org.springframework.stereotype.Component
#Component
public class CustomEditorRegistrar implements PropertyEditorRegistrar {
#Resource
GrailsApplication grailsApplication
public void registerCustomEditors(PropertyEditorRegistry reg){
def dateFormats = grailsApplication.config.grails.databinding.dateFormats as List
reg.registerCustomEditor(Date.class, new CustomDateBinder(dateFormats))
}
}