My job(Spark-Scala) is running fine in emr-5.36.0 using spark 2.4.8 and scala 2.11.
But I am getting below error while running the same job in emr-6.6.0 using spark 3.2.0 and scala 2.12.15
Exception in thread "main" java.lang.IncompatibleClassChangeError: Class org.json.JSONArray does not implement the requested interface java.lang.Iterable
As per my understanding below part of code is causing issue
*import scala.collection.JavaConverters._
val configJSON: JSONArray = fetchConfigs(configServerUri, configFiles)
configJSON.asScala.foreach(propertySource => {
propertySource.asInstanceOf[JSONObject].getJSONObject("source").toMap.asScala.foreach(kv => {
sparkAppConf.set(configReMap.getOrElse(kv._1, kv._1), kv._2.toString)
})*
I am using below dependency in pom for json:
<dependency>
<groupId>org.json</groupId>
<artifactId>json</artifactId>
<version>20220924</version>
</dependency>
And JSONArray is implementing the Iterable:
**public class JSONArray implements Iterable<Object>**
Note:The same code is running fine in local with spark 3.2.0 and scala 2.12.15.
Related
Jackson Object Mapper class given an error Spring mvc
All required jar already added no error in project like jackson-core-asl, jackson-core-2.2.3 and jackson-all 1.9.0 still getting error
Remove the unnecessary jackson-all and jackson-core-asl dependencies.
Include the only one jackson-databind that automatically de/serializes the objects.
<jackson.version>2.8.9</jackson.version>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>${jackson.version}</version>
</dependency>
Don't forget to return the whole List<Demo> instead of String.
#RequestMapping(value="/getJson", method = RequestMethod.GET)
public List<Demo> listJsonList {
// ...
return alldatalist;
}
And please, next time copy-paste the source code as a text, not as the image :)
1.When i upgrade my play framework from 2.2.6 to 2.3.8 and securesocial from 2.1.4 to 3.0-M3. I got the following error. How to fix it?
[error] /Work/test.scala:81: not found: value SecuredAction
[error] def findStats = SecuredAction {
2.The following line used to work but now it gets an error message:trait SecureSocial takes type parameters.How to fix this?
object ToolbarHandler extends Controller with SecureSocial with CookieLang{
3.Request needs a parameter.How to add the parameter though?
def findStats = SecuredAction {
implicit request =>
You have to use the snapshot version.
As stated here : http://securesocial.ws/guide/getting-started.html
add:
resolvers += "Sonatype OSS Snapshots" at "https://repo.typesafe.com/typesafe/snapshots/"
then add the library dependency:
"ws.securesocial" % "securesocial_2.11" % "master-SNAPSHOT"
Hope that helps.
Is there a way and/or library to automatically create Kotlin Data class from Json like it is works in Scala Json.Spray?
Something like this:
data class User(id: Int, name: String)
class DataClassFactory(val json: String) {
fun getUser(): User {
//some reflection
return User(10, "Kirill")
}
}
fun main(args: Array<String>): Unit {
val json = "{id: 10, name: Kirill}"
val usr = DataClassFactory(json).getUser()
println(usr)
}
You can use the Jackson module for Kotlin to serialize/deserialize easily from any format that Jackson supports (including JSON). This is the easiest way, and supports Kotlin data classes without annotations. See https://github.com/FasterXML/jackson-module-kotlin for the module which includes the latest information for using from Maven and Gradle (you can infer IVY and download JARs from the Maven repositories as well)
Alternatives exist such as Boon, but it has no specific support for Kotlin (usually a problem with not having a default constructor) and uses some unsafe direct access to internal JVM classes for performance. In doing so, it can crash on some VM's, and in cases where you extend Boon from Kotlin with custom serializer/deserializer it makes assumptions about the classes that do not hold true in Kotlin (the String class is wrapped for example) which I have seen core dump. Boon is lightening fast, just be careful of these issues and test first before using.
(note: I am the creator of the Jackson-Kotlin module)
This is very clean and easy in Kotlin.
import com.fasterxml.jackson.module.kotlin.*
data class User(val id: Int, val name: String)
fun main(args: Array<String>) {
val mapper = jacksonObjectMapper()
val json = """{"id": 10, "name": "Kirill"}"""
val user = mapper.readValue<User>(json)
println(user)
}
produces this output:
User(id=10, name=Kirill)
you only have to add this to your pom.xml
<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
<artifactId>jackson-module-kotlin</artifactId>
<version>2.6.3-4</version>
</dependency>
Why not use Jackson or any other serializer? It should work..
What about this
This is a translator that translate JSON string into kotlin data class ,it make it throught a plugin,see next
https://plugins.jetbrains.com/plugin/9960-jsontokotlinclass
http://www.json2kotlin.com converts your json response to kotlin data classes online, without the need to install any plugin. Optionally, you can generate gson annotations too. (Disclosure: I created this utility)
The following code snippet:
GraphDatabaseService graphDb = new EmbeddedGraphDatabase("var/geo");
// Wrap it as a spatial db service
SpatialDatabaseService spatialDb = new SpatialDatabaseService(graphDb);
// Create the layer to store our spatial data
EditableLayer runningLayer = (EditableLayer) spatialDb.getOrCreateLayer("running", SimplePointEncoder.class, EditableLayerImpl.class, "lon:lat");
fails with the error:
Exception in thread "main" java.lang.ClassCastException: org.neo4j.collections.graphdb.impl.EmbeddedGraphDatabase cannot be cast to org.neo4j.kernel.GraphDatabaseAPI
at org.neo4j.cypher.ExecutionEngine.<init>(ExecutionEngine.scala:113)
at org.neo4j.cypher.javacompat.ExecutionEngine.<init>(ExecutionEngine.java:53)
at org.neo4j.cypher.javacompat.ExecutionEngine.<init>(ExecutionEngine.java:43)
at org.neo4j.collections.graphdb.ReferenceNodes.getReferenceNode(ReferenceNodes.java:60)
at org.neo4j.gis.spatial.SpatialDatabaseService.getSpatialRoot(SpatialDatabaseService.java:76)
at org.neo4j.gis.spatial.SpatialDatabaseService.getLayer(SpatialDatabaseService.java:108)
at org.neo4j.gis.spatial.SpatialDatabaseService.getOrCreateLayer(SpatialDatabaseService.java:202)
at com.bmt.contain.spatial.test.SpatialTest.main(SpatialTest.java:47)
I was trying to get the sample code from here to work, I have included the relevant import statements below, in case I am somehow importing the wrong version of a function.
import org.neo4j.collections.graphdb.impl.EmbeddedGraphDatabase;
import org.neo4j.graphdb.GraphDatabaseService;
import org.neo4j.gis.spatial.EditableLayer;
import org.neo4j.gis.spatial.EditableLayerImpl;
import org.neo4j.gis.spatial.Layer;
import org.neo4j.gis.spatial.encoders.SimplePointEncoder;
Can someone advise me?
Also, its 2.0.1 for both neo4j and v13 for spatial.
<dependency>
<groupId>org.neo4j</groupId>
<artifactId>neo4j-spatial</artifactId>
<version>0.13-neo4j-2.0.1</version>
</dependency>
<dependency>
<groupId>org.neo4j</groupId>
<artifactId>neo4j</artifactId>
<version>2.0.1</version>
</dependency>
So the answer to this question is that you need to use
GraphDatabaseService graphDb = new GraphDatabaseFactory().newEmbeddedDatabase("var/geo");
instead of
GraphDatabaseService graphDb = new EmbeddedGraphDatabase("var/geo");
Somewhere under the hood this creates a different type of Embedded GDbS which doesnt cause a class cast exception.
The following example works in a Java EE6 (Glassfish3) project of mine but failed after I switched to Java EE7 (Glassfish4). The HTTP request returns "500 Internal Error" without any message in the Glassfish server log. The project was setup using NetBeans8 as Maven Web Project and has no special dependencies, beans.xml or other configuration.
#RequestScoped
#Path("generic")
public class GenericResource {
#GET
#Path("ping")
#Produces(APPLICATION_JSON)
public List<String> debugPing() {
return Arrays.asList("pong");
}
And then:
$ curl -v http://localhost:8080/mavenproject2/webresources/generic/ping
> GET /mavenproject2/webresources/generic/ping HTTP/1.1
...
< HTTP/1.1 500 Internal Server Error
As as I understand, all REST handling is done by the Jackson reference implementation and that Jackson uses Jersey as underlaying JSON library. One of the two is supposed to have some kind of provider for all basic data types. Only custom made classes need a self written ObjectMapper. Are these concepts still correct?
It took me some hours but I finally solved this question myself.
First fact is that the Glassfish4 JAX-RS implementation "Jersey" as switched its underlying JSON library from Jackson 1.x to Eclipselink MOXy. The latter seems not be able to convert Lists, Arrays and arbitrary POJOs to JSON out of the box. Therefore I tried to force JAX-RS to use Jackson 2.x and disable MOXy.
import java.util.HashMap;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;
import javax.ws.rs.ApplicationPath;
import javax.ws.rs.core.Application;
// This is Jackson 2.x, Jackson 1.x used org.codehaus.jackson!
import com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
#ApplicationPath("rest")
public class RestConfig extends Application {
private final static Logger log = LoggerFactory.getLogger(RestConfig.class);
#Override
public Set<Object> getSingletons() {
Set<Object> set = new HashSet<>();
log.info("Enabling custom Jackson JSON provider");
set.add(new JacksonJsonProvider() /* optionally add .configure(SerializationFeature.INDENT_OUTPUT, true) */);
return set;
}
#Override
public Map<String, Object> getProperties() {
Map<String, Object> map = new HashMap<>();
log.info("Disabling MOXy JSON provider");
map.put("jersey.config.disableMoxyJson.server", true);
return map;
}
#Override
public Set<Class<?>> getClasses() {
Set<Class<?>> resources = new java.util.HashSet<>();
// ... add your own REST enabled classes here ...
return resources;
}
}
My pom.xml contains:
<dependency>
<!-- REST (Jackson as JSON mapper) -->
<groupId>com.fasterxml.jackson.jaxrs</groupId>
<artifactId>jackson-jaxrs-json-provider</artifactId>
<version>2.2.3</version>
</dependency>
<dependency>
<!-- REST (Jackson LowerCaseWithUnderscoresStrategy etc.) -->
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.2.3</version>
</dependency>
Hope this helps someone!