I am trying to work with jerkson in play and with scala 2.10.
However, i want to load data fixtures based on a json files. for this prcoedure I'm trying to load the json with the "parse" command from jerkson.
That ultimatly fails.
I'm doing this in the "override def onStart(app: Application)" function. The error:
NoClassDefFoundError: Could not initialize class com.codahale.jerkson.Json$
Any guesses why this is happening ? I have the following libs in my deps.:
"com.codahale" % "jerkson_2.9.1" % "0.5.0",
"com.cloudphysics" % "jerkson_2.10" % "0.6.3"
my parsing command is:
com.codahale.jerkson.Json.parse[Map[String,Any]](json)
Thanks in advance
A NoClassDefFoundError generally means there is some sort of issues with the classpath. For starters, if you are running on scala 2.10, I would remove the following line from your sbt file:
"com.codahale" % "jerkson_2.9.1" % "0.5.0"
Then, make sure the com.cloudphysics jerkson jar file is available in your apps classpath and try your test again.
Related
I am trying to read a simple json file using spark scala using below code
val data =spark.read.option("multiLine",true).json(jsonpath)
However I am getting error while reading it
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.11.2
Here is the sbt file
"org.apache.spark" %% "spark-core" % "2.4.0",
"org.apache.spark" %% "spark-sql" % "2.4.0",
"org.apache.spark" %% "spark-streaming" % "2.4.0",
"com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.6.7.1",
"com.fasterxml.jackson.core" % "jackson-databind" % "2.6.7",
"com.fasterxml.jackson.core" % "jackson-core" % "2.6.7"
I have tried with different version. I am not able to find the compatible version for spark 2.4
Can someone please help me with this?
Spark comes with its own version of Jackson. And Jackson is not compatible between minor releases. Thus you cannot have a mix of Jackson version dependencies in your project.
The easiest way to avoid trouble is to use the same version as Spark and mark the dependency as Provided in your build definition.
For Spark 2.4.x, it's Jackson 2.6.x to be used. See for instance on https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.12/2.4.8
EDIT: you should also make sure that you don't have another dependency that pulls another Jackson version. You can use SBT dependencyOverrides to achieve this.
Note that there's also the possibility to include another version of Jackson and force Spark to use it by using the "userClasspathFirst" property. I'll let the reader find information about this if needed.
I have written a small code for custom producer in Kafka using scala and it is giving the below error. I have attached the code in code section. I have attached some code for reference.
Name: Compile Error
Message: <console>:61: error: not found: type KafkaProducer
val producer = new KafkaProducer[String, String](props)
^
I think I need to import a relevant package. I tried importing the packages but could not get the correct one.
val producer = new KafkaProducer[String, String](props)
for( i <- 1 to 10) {
//producer.send(new ProducerRecord[String, String]("jin", "test",
"test"));
val record = new ProducerRecord("jin", "key", "the end ")
producer.send(record)
I can't install a scala kernel for jupyter right now, but based on this github you should add Kafka as a dependency, then the library might be recognized
%%configure -f
{
"conf": {
"spark.jars.packages": "org.apache.spark:spark-streaming_2.11:2.1.0,org.apache.bahir:spark-streaming-twitter_2.11:2.1.0,org.apache.spark:spark-streaming-kafka-0-8_2.10:2.1.0,com.google.code.gson:gson:2.4",
"spark.jars.excludes": "org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.11"
}
}
If this doesn't work, try downloading the whole notebook from the git, and fire it yourself, to see if something else is needed
#Arthur , Magic command %%configure -f did not work in jupyter notebook. I tried downloading the Whole notebook from the git but that also does not work. Luckily I was
reading the apache toree documentation for adding the dependencies and found a command %%addDeps. After putting dependencies in the below format into jupyter notebook,
I managed to run the code.
%AddDeps org.apache.kafka kafka-clients 1.0.0
%AddDeps org.apache.spark spark-core_2.11 2.3.0
Just for the information of others, when we compile the code using SBT, we need to comment this code from jupyter notebook as we will add these in build.sbt file.
Thanks Arthur for showing the direction !
I am attempting to use the Eclipse Scala-IDE for 2.11 (downloaded the prepackaged bundle from the web-site). I have been using the Scala Worksheet to work with a SaaS API returning JSON. I've been pushing through just using String methods. I decided to begin using json4s. I went to http://mvnrepository.com/ and obtained the following libraries:
json4s-core-2.11-3.2.10.jar
json4s-native-2.11-3.2.10.jar
paranamer-2.6.jar
I have added all three jars to the Project's Build Path. And they appear under the project's "Referenced Libraries".
I have the following code in a Scala Worksheet:
package org.public_domain
import org.json4s._
import org.json4s.native.JsonMethods._
object WorkSheet6 {
println("Welcome to the Scala worksheet")
parse(""" { "numbers" : [1, 2, 3, 4] } """)
println("Bye")
}
I am receiving the following two compilation errors:
bad symbolic reference to org.json4s.JsonAST.JValue encountered in class file 'package.class'. Cannot access type JValue in value org.json4s.JsonAST. The current classpath may be missing a definition for org.json4s.JsonAST.JValue, or package.class may have been compiled against a version that's incompatible with the one found on the current classpath.
bad symbolic reference to org.json4s.JsonAST.JValue encountered in class file 'package.class'. Cannot access type JValue in value org.json4s.JsonAST. The current classpath may be missing a definition for org.json4s.JsonAST.JValue, or package.class may have been compiled against a version that's incompatible with the one found on the current classpath.
When I go look in the org.json4s package in the json4s-core-2.11-3.2.10.jar file, there is in fact no .class file indicating any sort of compiled object JsonAST.
This is a showstopper. Any help on this would be greatly appreciated.
Your classpath is incomplete. You are missing a dependency of json4s-core.
bad symbolic reference to org.json4s.JsonAST.JValue encountered in class file 'package.class'. Cannot access type JValue in value org.json4s.JsonAST. The current classpath may be missing a definition for org.json4s.JsonAST.JValue, or package.class may have been compiled against a version that's incompatible with the one found on the current classpath.
The simplest way to consume Scala or Java libraries is to use sbt or maven. They bring in (transitive) dependencies for you. If you check the pom definition of json4s-core library, you notice it depends on json4s-ast. You should add that jar to your build path as well.
Environment Details
Mac OS X 10.9
Oracle JDK 1.7.0_55 64-bit
jython-standalone-2.5.3.jar
junit-4.11
What I have done so far
I have added the junit jar to /Library/Java/Extensions.
I invoked Jython as follows java -jar jython-standalone-2.5.3.jar
In the Jython interpreter, I imported the following import org.junit.Assert, and this import was successful.
Problem
When I tried to use assertTrue, I got a NameError in the interpreter. Why is this so?
I understand that assertTrue is a static method. Not sure what implication this has when I try to use it in Jython.
Additional Context
I am using XMLUnit in Jython. Was able to successfully import the Diff class from org.custommonkey.xmlunit in Jython. Also able to use the methods in this class, and call them on a Diff object. The result of this method call is what I am trying to pass to assertTrue, when it throws the error.
from org.custommonkey.xmlunit import Diff
import org.junit.Assert
xml1 = ...some XML string...
xml2 = ...some XML string...
myDiff = Diff(xml1, xml2)
assertTrue(myDiff.similar())
Hope this additional information is useful in identifying a solution to this problem.
Latest Status
I narrowed it down to setting this property python.security.respectJavaAccessibility = false, since the Assert() constructor is protected.
Still trying to get it to work. Any help is greatly appreciated.
Figured it out.
In addition to junit.jar file, the hamcrest-core.jar file also needed to be copied to /Library/Java/Extensions.
Then I got rid of the jython.jar file, and instead installed it using the jython installer.
After the installation was completed, I updated the registry file in the installation folder, specifically setting this property python.security.respectJavaAccessibility = false.
Now I am able to see the assertTrue method, and no longer getting a NameError.
I have problems while generating .java files with jruby 1.7.3. Here is an example:
class Duck
def quack()
puts "quack!";
end
end
def quack_it(duck)
duck.quack
end
a = Duck.new
quack_it(a)
when I execute
jrubyc --java Test.rb I get the following compilation error:
Failure during compilation of file DuckExample_simple.rb:
undefined method `new_method' for nil:NilClass.
Therefore, I have 2 questions:
What is wrong here?
I want to generate .java files in order to see how the JRuby code is translated into the bytecode and instead of reading the bytecode itself I thought to read the java code. Does the generated java code correspond 1 to 1 to the bytecode generated by AOT jruby compiler, or it's better to read the bytecode itself? I actually want to see how jruby handles dynamic method dispatch at the bytecode level. Any hints would be appreciated.
i don't use jruby so i am not really the best guy to talk to, but here are my 2 cents anyways.
if you just put a simple class into the file, it will work. so try
class Duck
def quack()
puts "quack!"
end
end
it will create a Duck.java file as you would expect, which answeres the second question you had. there is also a nice writeup about the generated file here: http://rhnh.net/2012/10/20/guice-in-your-jruby
i guess that the command is somewhat broken. it would be best to open an issue at the jruby issue tracker: http://jira.codehaus.org/browse/JRUBY