org.apache.spark.sql.catalyst.TableIdentifier cannot be resolved error in SnappySQLJob - snappydata

I have an compile-time error trying to write a SnappySQLJob. Am I missing a dependency?
The error message is:
The type org.apache.spark.sql.catalyst.TableIdentifier cannot be resolved. It is indirectly referenced from required .class files
#Override
public Object runJob(Object sparkContext, Config jobConfig) {
SnappyContext snappyContext = (SnappyContext)sparkContext;
String fileResource = "data.csv";
DataFrame dataFrame = snappyContext.read()
.format("com.databricks.spark.csv")
.option("header", "true")
.option("inferSchema", "true")
.load(fileResource);
// Compile-Time error is on this line
dataFrame.write().insertInto("example_table_col");
return null;
}
Here is my pom.xml dependencies:
<dependency>
<groupId>io.snappydata</groupId>
<artifactId>snappy-core_2.10</artifactId>
<version>0.2.1-PREVIEW</version>
</dependency>
<dependency>
<groupId>io.snappydata</groupId>
<artifactId>snappy-tools_2.10</artifactId>
<version>0.2.1-PREVIEW</version>
<exclusions>
<exclusion>
<artifactId>jdk.tools</artifactId>
<groupId>jdk.tools</groupId>
</exclusion>
<exclusion>
<artifactId>logback-classic</artifactId>
<groupId>ch.qos.logback</groupId>
</exclusion>
</exclusions>
</dependency>

This old release seems to be missing spark-catalyst dependency. I will suggest upgrading to 0.5 release version instead (snappy-tools is now called snappy-cluster) and the snappydata cluster should also be upgraded to 0.5
For the 0.2.1 release, below should correct the problem:
<dependency>
<groupId>io.snappydata</groupId>
<artifactId>snappy-spark-catalyst_2.10</artifactId>
<version>1.6.0-BETA</version>
</dependency>
<dependency>
<groupId>io.snappydata</groupId>
<artifactId>snappy-spark-sql_2.10</artifactId>
<version>1.6.0-BETA</version>
</dependency>

Related

Failed to configure a DataSource: 'url' attribute is not specified and no embedded datasource could be configured

***************************
APPLICATION FAILED TO START
***************************
Description:
Failed to configure a DataSource: 'url' attribute is not specified and no embedded datasource could be configured.
Reason: Failed to determine a suitable driver class
Action:
Consider the following:
If you want an embedded database (H2, HSQL or Derby), please put it on the classpath.
If you have database settings to be loaded from a particular profile you may need to activate it (no profiles are currently active).
I have tried using
spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.MySQL5InnoDBDialect
spring.autoconfigure.exclude=org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
in application properties but still it didn't work.
And even in have tried using
#SpringBootApplication(exclude = {DataSourceAutoConfiguration.class }) or
#EnableAutoConfiguration (exclude = {DataSourceAutoConfiguration.class })
then im not gettin any error but the table is getting created in the database.
pom.xml
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-thymeleaf</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<scope>runtime</scope>
<optional>true</optional>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>org.junit.vintage</groupId>
<artifactId>junit-vintage-engine</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-test</artifactId>
<scope>test</scope>
</dependency>
```
Application properties
````
spring.datasource.url=jdbc:mysql://localhost:3306/emp
spring.datasource.username=root1
spring.datasource.password=root#123
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.MySQL5Dialect
spring.jpa.generate-ddl=true
spring.jpa.hibernate.ddl-auto = create
````
```
package net.codejava;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration;
#SpringBootApplication
public class SpringBootRegistrationLoginApplication2 {
public static void main(String[] args) {
SpringApplication.run(SpringBootRegistrationLoginApplication2.class, args);
}`your text`
}
```

jersey + json parsing

I have the following controller:
#POST
#Path("endpoint")
#Consumes({MediaType.APPLICATION_JSON+ ";charset=utf-8"})
#Produces({MediaType.APPLICATION_JSON + ";charset=utf-8", "application/javascript;charset=utf-8"})
public void postData(#Context HttpHeaders httpHeaders,
#Context HttpServletRequest request, #Context UriInfo uriInfo,
JSONData postedData) {
}
My related pom dependencies are:
<dependency>
<groupId>org.glassfish.jersey.containers</groupId>
<artifactId>jersey-container-servlet</artifactId>
</dependency>
<dependency>
<groupId>org.glassfish.jersey.media</groupId>
<artifactId>jersey-media-json-jackson</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.glassfish.jersey.core</groupId>
<artifactId>jersey-server</artifactId>
<exclusions>
<exclusion>
<groupId>org.glassfish.jersey.media</groupId>
<artifactId>jersey-media-jaxb</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.glassfish.jersey.media</groupId>
<artifactId>jersey-media-json-jackson</artifactId>
<scope>runtime</scope>
</dependency>
The version resolved in the dependencies is 2.17 for all the dependencies.
I am registering the JacksonFeature class in the properties
jersey.config.server.provider.classnames=org.glassfish.jersey.jackson.JacksonFeature
And I get the following exception when performing a request: (the JSONData class is very simple by now, only a String property annotated with lombok to generate getter and setters)
2020-12-07 12:31:56,910 ERROR [jersey-server-managed-async-executor-0]
JsonExceptionMapper - at
es.colbenson.sb.ws.JsonExceptionMapper.toResponse(JsonExceptionMapper.java:28)
Exception: java.lang.NoSuchMethodError java.lang.NoSuchMethodError:
com.fasterxml.jackson.databind.util.BeanUtil.okNameForSetter(Lcom/fasterxml/jackson/databind/introspect/AnnotatedMethod;)Ljava/lang/String;
at
com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector.findNameForDeserialization(JaxbAnnotationIntrospector.java:875)
~[jackson-module-jaxb-annotations-2.3.2.jar:2.3.2]
I've also tried registering the JacksonFeature class on my application constructor:
register(JacksonFeature)
But I can get this working. Any thoughts?
Also I can see this in the exception Stack Trace, which seems to be pointing to a jackson-databind-2.9.8 inside, but I'm not sure if it is a kind of conflict...
at
com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addSetterMethod(POJOPropertiesCollector.java:620)
~[jackson-databind-2.9.8.jar:2.9.8] at
com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addMethods(POJOPropertiesCollector.java:535)
~[jackson-databind-2.9.8.jar:2.9.8] at
com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.collectAll(POJOPropertiesCollector.java:309)
~[jackson-databind-2.9.8.jar:2.9.8] at
com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.getPropertyMap(POJOPropertiesCollector.java:287)
~[jackson-databind-2.9.8.jar:2.9.8] at
com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.getProperties(POJOPropertiesCollector.java:170)
~[jackson-databind-2.9.8.jar:2.9.8]
I had to remove this dependency:
<dependency>
<groupId>org.glassfish.jersey.media</groupId>
<artifactId>jersey-media-json-jackson</artifactId>
<scope>runtime</scope>
</dependency>
And add this other
<dependency>
<groupId>com.fasterxml.jackson.jaxrs</groupId>
<artifactId>jackson-jaxrs-json-provider</artifactId>
<version>2.9.8</version>
</dependency>
Then register this provider:
jersey.config.server.provider.classnames=com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider
It seems a matter of using the proper jackson binder for this version of jersey

Camel json validator support for json draft 07

We are using Apache Camel Java DSL and the json-validator component to validate json requests against a json schema. The current camel version is 2.22.0, according to the camel documentation this supports JSON schema v4, v6, v7 and v2019-09 using the NetworkNT JSON Schema library. However, when I try a JSON schema draft 07, I get an error when running tests "Caused by: com.networknt.schema.JsonSchemaException: Unknown Metaschema: http://json-schema.org/draft-07/schema#".
When i revert back to json schema draft 04 it works fine.
Any ideas on how to get this working?
The default draft is the 4th one, but you can override the schema validator (JsonSchemaLoader), by defining a bean.
#Bean(name = "mySchemaLoader")
public JsonSchemaLoader mySchemaLoader() {
return (camelContext, schemaStream) -> JsonSchemaFactory.getInstance(SpecVersion.VersionFlag.V7)
.getSchema(schemaStream);
}
with that we just create a bean wich will return a V7 schema validator, if you want to override the default configuration
...
ObjectMapper mapper = new ObjectMapper();
JsonSchemaFactory validatorFactory = JsonSchemaFactory.builder(JsonSchemaFactory.getInstance(SpecVersion.VersionFlag.V7)).objectMapper(mapper).build();
....
So having the bean, you just have to said to camel that you are going to usea that bean via query param
public void configureRemote() throws Exception {
from("direct:getPrescripciones")
.recipientList(
simple"${header.url}?bridgeEndpoint=true"))
.to("json-validator:deliveryReport.schema.json?schemaLoader=#bean:mySchemaLoader")
.end();
}
that's all
here the dependencies I used
<properties>
<java.version>1.8</java.version>
<camel.version>3.4.0</camel.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.camel.springboot</groupId>
<artifactId>camel-json-validator-starter</artifactId>
<version>${camel.version}</version>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-jackson</artifactId>
<version>${camel.version}</version>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-http</artifactId>
<version>${camel.version}</version>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-jolt</artifactId>
<version>${camel.version}</version>
</dependency>
<dependency>
<groupId>org.apache.camel.springboot</groupId>
<artifactId>camel-servlet-starter</artifactId>
<version>${camel.version}</version>
</dependency>
<dependency>
<groupId>org.apache.camel.springboot</groupId>
<artifactId>camel-swagger-java-starter</artifactId>
<version>${camel.version}</version>
</dependency>
<dependency>
<groupId>org.apache.camel.springboot</groupId>
<artifactId>camel-google-pubsub-starter</artifactId>
<version>${camel.version}</version>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-google-pubsub</artifactId>
<version>${camel.version}</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>org.junit.vintage</groupId>
<artifactId>junit-vintage-engine</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<scope>runtime</scope>
</dependency>
</dependencies>
The camel version supporting the v7 is 3.4.0
The documentation you're looking for is on 2.x and in that case it's correctly noted that only v4 is supported: https://camel.apache.org/components/2.x/json-validator-component.html

posting JSON to Spring rest controller

I got stuck on sending POST to my rest controller. When I try to update my entry, I get 415 status. I have this error in my web application, written in angular JS and chrome tool for rest API.
Controller code:
#RequestMapping(value = "/update", method = RequestMethod.POST)
public void updateEntry(#RequestBody Entry entry) {
System.out.println("editing: " + entry.toString());
}
My pom.xml:
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
<version>4.1.2.RELEASE</version>
<exclusions>
<exclusion>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-webmvc</artifactId>
<version>4.1.2.RELEASE</version>
<exclusions>
<exclusion>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-web</artifactId>
<version>4.1.2.RELEASE</version>
<exclusions>
<exclusion>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
</exclusions>
</dependency>
<!--JSON-->
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<version>1.9.13</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-core-asl</artifactId>
<version>1.9.13</version>
</dependency>
I'm trying to send something like this:
{"id":"5471db49b0ca59357d6beded","name":"EADS 3 Sigma Nearchos"}
My entry contains only two fields: String id, String name.
Spring Framework raised library requirements, including Jackson. You should update to Jackson 2.1+ when using Spring 4.1.x.
Check out the migration page on github for more details.

RestEasy Jettison/Jackson customization

I'm trying to use my objects with JAXB annotations for application/json output with my JAX-RS resource. I'm running on JBoss AS7 with RestEasy (both lastest versions - 7.1.1.Final and 2.3.4.Final). The issue is that I would like to customize my JSON output. I must note that I don't care if I will use Jettison or Jackson, but I was able only make Jettison work (deploy application) without errors. I also would like to stick only with JAXB annotations on my objects if possible - but it is not necessairly needed.
1) I want to omit "#" within XmlAttribute annotated fields. I found out property how to do it with Jettison, but I don't know how to configure it on JBoss AS7. Didn't find any ContextResolver example.
2) I would like to have "normal" JSON arrays, e.g.
#XmlRootElement(name = "root")
#XmlAccessorType(XmlAccessType.FIELD)
public class Root {
#XmlElementRef(type = Entry.class, required = false)
// no difference with #XmlElement
private Set<Entry> entries;
}
serializes into
{"entries":
{"entry":[{...},{...},{...}]}
}
and I would expect
{"entries":
[
{"entry":{...}},
{"entry":{...}},
{"entry":{...}}
]
}
or just (omit XmlRootElement)
{"entries":
[{...},{...},{...}]
}
3) As I noted, I don't care what provider (Jettison/Jackson) will I use but it is hard to find working example how to correctly set maven dependencies for application to be deployable without errors. So far I'm using:
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-jaxrs</artifactId>
<version>${resteasyVersion}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-jaxb-provider</artifactId>
<version>${resteasyVersion}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-jettison-provider</artifactId>
<version>${resteasyVersion}</version>
<!--<scope>provided</scope>-->
<exclusions>
<exclusion>
<groupId>javax.xml.bind</groupId>
<artifactId>jaxb-api</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.xml.bind</groupId>
<artifactId>jaxb-impl</artifactId>
</exclusion>
<exclusion>
<groupId>javax.xml.stream</groupId>
<artifactId>stax-api</artifactId>
</exclusion>
</exclusions>
</dependency>
Thanks for all answers
So far I managed to make Jackson working and use its provider-specific annotations (#JsonProperty ... and so) which solved my problem. I tried to find only JAXB solution but I can accept also this.
Btw my dependencies
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-jaxrs</artifactId>
<version>${resteasyVersion}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-jaxb-provider</artifactId>
<version>${resteasyVersion}</version>
</dependency>
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-jackson-provider</artifactId>
<version>${resteasyVersion}</version>
</dependency>