I have a Java Iterable object, Iterable records. And I want to pass it to the Beam pipeline. I tried
PCollection csvRecordPC = p.apply("Create collection", Create.of(records));
It caused an error
An exception occured while executing the Java class. Can not determine a default Coder for a 'Create' PTransform that has no elements. Either add elements, call Create.empty(Coder), Create.empty(TypeDescriptor), or call 'withCoder(Coder)' or 'withType(TypeDescriptor)' on the PTransform.
Which Coder should I use? Or how can I write my custom coder?
I found a solution using FileIO.
p.apply(FileIO.match().filepattern(options.getInputFile()))
.apply(FileIO.readMatches())
.apply(ParDo.of(new CsvParser()))
The CsvPaser() is
public class CsvParser extends DoFn<ReadableFile, CSVRecord> {
#DoFn.ProcessElement
public void processElement(#Element ReadableFile element, DoFn.OutputReceiver<CSVRecord> receiver) throws IOException {
InputStream is = Channels.newInputStream(element.open());
Reader reader = new InputStreamReader(is);
Iterable<CSVRecord> records = CSVFormat.EXCEL.withFirstRecordAsHeader().parse(reader);
for (CSVRecord record : records) {
receiver.output(record);
}
}
}
Related
I am used to JAX-RS and would like to have similar comfort when sending requests using Spring MVC and working with the responses, i.e. on the client side inside my tests.
On the server (controller) side I'm quite happy with the automatic conversion, i.e. it suffices to just return an object instance and have JSON in the resulting HTTP response sent to the client.
Could you tell me how to work around the manual process of converting objectInstance to jsonString or vice versa in these snippets? If possible, I'd also like to skip configuring the content type manually.
String jsonStringRequest = objectMapper.writeValueAsString(objectInstance);
ResultActions resultActions = mockMvc.perform(post(PATH)
.contentType(MediaType.APPLICATION_JSON)
.content(jsonStringRequest)
)
String jsonStringResponse = resultActions.andReturn().getResponse().getContentAsString();
Some objectInstanceResponse = objectMapper.readValue(jsonStringResponse, Some.class);
For comparison, with JAX-RS client API I can easily send an object using request.post(Entity.entity(objectInstance, MediaType.APPLICATION_JSON_TYPE) and read the response using response.readEntity(Some.class);
if you have lot's of response objects, you could create some generic JsonToObject mapper-factory. It could be then used to detect the object type from a generic response (all response objects inherit from the same generic class) and respond/log properly from a bad mapping attempt.
I do not have a code example at hand, but as a pseudocode:
public abstract GenericResponse {
public String responseClassName = null;
// get/set
}
In the server code, add the name of the actual response object to this class.
The JsonToObject factory
public ConverterFactory<T> {
private T objectType;
public ConverterFactory(T type) {
objectType = type;
}
public T convert(String jsonString) {
// Type check
GenericResponse genResp = mapper.readValue(result.getResponse().getContentAsString(),
GenericResponse.class);
if (objectType.getClass().getSimpleName().equals(genResp.getResponseClassName())) {
// ObjectMapper code
return mapper.readValue(result.getResponse().getContentAsString(),
objectType.class);
} else {
// Error handling
}
}
}
I think this could be extended to be used with annotation to do more automation magic with the response. (start checking with BeanPostProcessor)
#Component
public class AnnotationWorker implements BeanPostProcessor {
#Override
public Object postProcessBeforeInitialization(final Object bean, String name) throws BeansException {
ReflectionUtils.doWithFields(bean.getClass(), field -> {
// make the field accessible if defined private
ReflectionUtils.makeAccessible(field);
if (field.getAnnotation(MyAnnotation.class) != null) {
field.set(bean, log);
}
});
return bean;
}
}
The above code snippet is copied from my current project and it injects to fields, you need to change it so, that it works for methods, eg ... where you may need it.
Having this all implemented may be tricky and can't say it necessarily works even, but it's something to try if you don't mind a bit of educative work.
I am trying to use PowerMockito to mock the creation of the java.net.URL class in my code that I'm testing. Basically, I want to prevent the real HTTP request from occurring and instead 1) check the data when the request is made and 2) supply my own test data back on a mocked response. This is what I'm trying:
#RunWith(PowerMockRunner.class)
#PrepareForTest({ URL.class, MockedHttpConnection.class })
public class Test {
URL mockedURL = PowerMockito.mock(URL.class);
MockedHttpConnection mockedConnection = PowerMockito.mock(MockedHttpConnection.class);
...
PowerMockito.whenNew(URL.class).withParameterTypes(String.class).withArguments("MyURLString").thenReturn(mockedURL);
PowerMockito.when(mockedURL.openConnection()).thenReturn(mockedConnection);
...
}
The code that I want to test looks like this:
URL wlInvokeUrl = new URL(wlInvokeUrlString);
connection = (HttpURLConnection) wlInvokeUrl.openConnection();
Earlier in my test scenario I mock the wlInvokeUrlString to match "MyURLString". I've also tried using various other forms of the whenNew line, trying to inject the mock. No matter what I try, it never intercepts the constructor. All I want to do is "catch" the call to openConnection() and have it return my mocked HTTP connection instead of the real one.
I have mocked other classes ahead of this one in the same script and these are working as expected. Either I need a second pair of eyes (probably true) or there is something unique about the URL class. I did notice that if I use "whenNew(URL.class).withAnyArguments()" and change the "thenReturn" to "thenAnswer" I could get it to trigger. Only problem is I never get the URL call for my code. What I see is an invocation of the 3-argument constructor for URL.class with all nulls for the parameters. Could it be this class is from the Java runtime and is bootstrapped by the test runner? Any help is much appreciated.
It's a common mistake when use PowerMockito.whenNew.
Note that you must prepare the class creating the new instance of MyClass for test, not the MyClass itself. E.g. if the class doing new MyClass() is called X then you'd have to do #PrepareForTest(X.class) in order for whenNew to work
From Powermock wiki
So, you need a bit change your test and add to #PrepareForTesta class which create a new instance of URLlike:
#RunWith(PowerMockRunner.class)
#PrepareForTest({ URL.class, MockedHttpConnection.class , ConnectionUser.class})
public class URLTest {
public class URLTest {
private ConnectionUser connectionUser;
#Before
public void setUp() throws Exception {
connectionUser = new ConnectionUser();
}
#Test
public void testName() throws Exception {
URL mockedURL = PowerMockito.mock(URL.class);
MockedHttpConnection mockedConnection = PowerMockito.mock(MockedHttpConnection.class);
PowerMockito.whenNew(URL.class).withParameterTypes(String.class).withArguments("MyURLString").thenReturn(mockedURL);
PowerMockito.when(mockedURL.openConnection()).thenReturn(mockedConnection);
connectionUser.open();
assertEquals(mockedConnection, connectionUser.getConnection());
}
}
where:
public class ConnectionUser {
private String wlInvokeUrlString = "MyURLString";
private HttpURLConnection connection;
public void open() throws IOException {
URL wlInvokeUrl = new URL(wlInvokeUrlString);
connection = (HttpURLConnection) wlInvokeUrl.openConnection();
}
public HttpURLConnection getConnection() {
return connection;
}
}
I'm not sure the difference between .withParameterTypes(x) and .withArguments(x) but I believe you need to set it up as follows for your code to work. Give it a try and let me know if this doesn't help.
PowerMockito.when(mockedURL.openConnection()).thenReturn(mockedConnection);
PowerMockito.whenNew(URL.class).withArguments(Mockito.anyString()).thenReturn(mockedURL);
The problem is that the arguments of the real call are not matching with the expected in your mock.
PowerMockito.whenNew(URL.class).withParameterTypes(String.class).withArguments("MyURLString").thenReturn(mockedURL) will return mockedURL only the call is new URL("MyURLString").
If you change it to:
PowerMockito.whenNew( URL.class ).withParameterTypes( String.class )
.withArguments( org.mockito.Matchers.any( String.class ) ).thenReturn( mockedURL );
It will catch any string passed to the constructor URL(String) (even null) and return your mock.
When you tried
"whenNew(URL.class).withAnyArguments()" and change the "thenReturn" to
"thenAnswer" I could get it to trigger. Only problem is I never get
the URL call for my code. What I see is an invocation of the
3-argument constructor for URL.class with all nulls for the
parameters.
PowerMock will try to mock all constructors (org.powermock.api.mockito.internal.expectation.DelegatingToConstructorsOngoingStubbing.InvokeStubMethod at line 122) then it calls the first one (with 3 arguments) and mock its answer. But the subsequent calls will return the already mocked one because you told it to mock for any arguments. That's why you see just one call with null, null, null in your Answer.
I have got the following code:
Collection<String> errors = ...;
try (InputStream stream = My.class.getResourceAsStream(resource)) {
// do stuff
}
catch(IOException ex) {
errors.add("Fail");
}
I'm trying with Byteman Junit Runner to trigger an IOException when the (valid) input stream I give is supposedly closed:
#RunWith(BMUnitRunner.class)
public class MyTest {
private My my = new My();
#BMRule(
name = "force_read_error",
targetClass = "java.io.InputStream",
targetMethod = "close()",
action = "throw new IOException(\"bazinga\")"
)
#Test
public void catches_read_error() throws IOException {
Collection<String> errors = my.foo("/valid-resource-in-classpath");
assertThat(errors).containsExactly("Fail");
}
}
My test fails: errors is always empty, which means the Byteman rule obviously isn't executed (it's well loaded by the agent, so I don't understand what's going on).
How can I trigger an IOException on close method called via try-with-resources?
Your rule ist not firing, because the class of the stream object received when calling
InputStream stream = My.class.getResourceAsStream(resource)
is not a "java.io.InputStream" class. It is a class extending "java.io.InputStream", most likely a "BufferedInputStream".
To tell byteman to "trigger rule for any class extending java.io.InputStream", you need to put a '^' before the class name:
targetClass = "^java.io.InputStream"
This change might have the unwanted side effect, that the rule gets triggered also when other objects extending "java.io.InputStream" get closed. To prevent this from happening, a condition should be added to the rule to only get triggered, when the caller matches the "foo" method of the "My" class. Byteman has a helper method for that called "callerMatches" (Please see also the advanced tutorial)
A working condition for your case would be:
condition = "callerMatches(\".+My.foo\",true,true)"
The complete Byteman rule definition as BMRule annotation should look like:
#BMRule(
name = "force_read_error",
targetClass = "^java.io.InputStream",
targetMethod = "close()",
condition = "callerMatches(\".+My.foo\",true,true)",
action = "throw new java.io.IOException(\"bazinga\")"
)
I'm trying to persist Maps of properties as single JSON-encoded columns, as shown in this question.
The problem I'm having is that apparently transient properties cannot be set in the default map constructor. Given any transient field:
class Test {
//...
String foo
static transients = ['foo']
}
It seems that the map constructor (which Grails overrides in various ways) simply discards transient fields:
groovy:000> t = new Test(foo:'bar')
===> Test : (unsaved)
groovy:000> t.foo
===> null
While direct assignment (through the setter method) works as expected:
groovy:000> c.foo = 'bar'
===> bar
groovy:000> c.foo
===> bar
Is there a way to make the map constructor accept transient fields?
Or rather: is there a better way to persist a Map as a single JSON-encoded DB field, rather than the method shown in the linked question?
Here's the complete example:
import grails.converters.JSON
class JsonMap {
Map data
String dataAsJSON
static transients = ['data']
def afterLoad() { data = JSON.parse(dataAsJSON) }
def beforeValidate() { dataAsJSON = data as JSON }
}
I can set data using the setter (which will then be converted into dataAsJSON) but not using the map constructor.
The map constructor in GORM uses the data binding mechanism, and transient properties are not data-bindable by default. But you can override this using the bindable constraint
class Test {
//...
String foo
static transients = ['foo']
static constraints = {
foo bindable:true
}
}
I've also replied to your original question, that you don't need json conversion to achieve what you need. However, If you need json conversion badly, why don't you implement it in your getters/setters?
class Test {
String propsAsJson
static transients = ['props']
public Map getProps() {
return JSON.parse(propsAsJson)
}
public void setProps(Map props) {
propsAsJson = props as JSON
}
}
//So you can do
Test t = new Test(props : ["foo" : "bar"])
t.save()
In this way you encapsulate the conversion stuff, and in DB you have your properties as Json.
You can simplify your case by adding the JSON-conversion methods to your domain class, they should have nothing to do with GORMing:
class Test {
String title
void titleFromJSON( json ){
title = json.toStringOfSomeKind()
}
def titleAsJSON(){
new JSON( title )
}
}
I'm working on MySQL using .Net Connector 6.3.6 and created Entity models on VS 2010. I'm planning to write a generic method that would add an EntityObject to its corresponding table. Here is how it looks:
public void AddToTable(ObjectContext dataContext, string tableName, EntityObject tableObj)
{
try
{
Type type = dataContext.GetType();
string methodName = "AddTo" + tableName;
MethodInfo methodInfo = type.GetMethod(methodName);
PropertyInfo propInfo = dataContext.GetType().GetProperty(tableName);
Object[] parameters = new Object[] { Convert.ChangeType(tableObj, propInfo.PropertyType) };
methodInfo.Invoke(dataContext, parameters);
}
catch (Exception e)
{
edit://gonna handle it appropriately here!
}
}
ObjectContext will be the actual ObjectContext class.
But I'm getting exception saying "object must implement IConvertible" when I use Covert.ChangeType() on an EntityObject.
How to overcome this problem?
Edit: FYI, my main intention is to make write a method which is as generic as possible so that no casting to a particular table type would be required.
Thanks,
Alerter
You're reinventing the wheel.
public void AddToTable<TEntity>(ObjectContext dataContext, TEntity tableObj)
{
dataContext.CreateObjectSet<TEntity>().AddObject(tableObj);
}
And please don't eat exceptions.
Followed the following generalized repository pattern:
[link]http://www.codeproject.com/Articles/37155/Implementing-Repository-Pattern-With-Entity-Framew[link] It is very intuitive and fits my requirement :)