Error while creating a custom producer in scala - kafka-producer-api

I have written a small code for custom producer in Kafka using scala and it is giving the below error. I have attached the code in code section. I have attached some code for reference.
Name: Compile Error
Message: <console>:61: error: not found: type KafkaProducer
val producer = new KafkaProducer[String, String](props)
^
I think I need to import a relevant package. I tried importing the packages but could not get the correct one.
val producer = new KafkaProducer[String, String](props)
for( i <- 1 to 10) {
//producer.send(new ProducerRecord[String, String]("jin", "test",
"test"));
val record = new ProducerRecord("jin", "key", "the end ")
producer.send(record)

I can't install a scala kernel for jupyter right now, but based on this github you should add Kafka as a dependency, then the library might be recognized
%%configure -f
{
"conf": {
"spark.jars.packages": "org.apache.spark:spark-streaming_2.11:2.1.0,org.apache.bahir:spark-streaming-twitter_2.11:2.1.0,org.apache.spark:spark-streaming-kafka-0-8_2.10:2.1.0,com.google.code.gson:gson:2.4",
"spark.jars.excludes": "org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.11"
}
}
If this doesn't work, try downloading the whole notebook from the git, and fire it yourself, to see if something else is needed

#Arthur , Magic command %%configure -f did not work in jupyter notebook. I tried downloading the Whole notebook from the git but that also does not work. Luckily I was
reading the apache toree documentation for adding the dependencies and found a command %%addDeps. After putting dependencies in the below format into jupyter notebook,
I managed to run the code.
%AddDeps org.apache.kafka kafka-clients 1.0.0
%AddDeps org.apache.spark spark-core_2.11 2.3.0
Just for the information of others, when we compile the code using SBT, we need to comment this code from jupyter notebook as we will add these in build.sbt file.
Thanks Arthur for showing the direction !

Related

Fsharp CSV parsing

I'm trying to get this to work
http://fsharp.github.io/FSharp.Data/library/CsvFile.html
But to me it seems like tha CsvFile class and the CSVextionson are removed of the Data lib
I'm running it on Manjaro (Arclinux system)
I'm using the mono compiler (fsharpc)
I got the dotnet core lib
So nothing should be wrong
please help
EDITED
Code
open FSharp.Data
let msft = CsvFile.Load("https://github.com/kam1986/Data-Science-project/blob/master/news_sample.csv")
It is placed in a .fsx file
get hte error message
Fake News.fsx(3,14): error FS0039: The value, namespace, type or module 'CsvFile' is not defined.

Jenkins: import external package from Jenkinsfile using declarative syntax

I had a groovy code wich contains "import groovy.json.JsonSlurper".
I have spent a day testing and i dont know how to load external libraries using declarative syntax.
This is my code:
pipeline {
agent any
import groovy.json.JsonSlurper
stages {
stage("test") {
steps {
}
}
}
}
I have read the jenkins documentation, and i have tried to use the next but without success:
#Grab('groovy.json.JsonSlurper')
import groovy.json.JsonSlurper
both import and #Grab is not recognized. Some idea?
Thanks!
What #Daniel Majano says is true about the import syntax, but the #Grab syntax I found holds differences of behavior between a Pipeline script maintained directly in Jenkins vs Pipeline script from SCM.
When I placed a Grab command in the Pipeline script for a tester pipeline job I found that it didn't make any difference whether the Grab command was there or if it was commented out.
However when used from a Pipeline script from SCM it would throw the following exception...
java.lang.RuntimeException: No suitable ClassLoader found for grab
I removed it from the SCM script and everything worked out in the end.
Additional Background
I'm not sure why the grab was choking in the SCM version, but there's definitely some working parts to the groovy editor because if you define a partial grab command it will give you some validation errors pointing to the broken line as you see in the red X box below, with the error The missing attribute "module" is required in #Grab annotations:
Therefore the script validator is aware of the Grab annotation as it calls it and that it has both a group and module attribute. I'm using the so called shorthand notation in this example.

ceylon run: Module default/unversioned not found

Today i installed the intelliJ ceylon IDE on my macbook. When compiling my project I get the following message
/Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/bin/java "-Dceylon.system.repo=/Users/Laust/Library/ApplicationSupport/IdeaIC2016.3/CeylonIDEA/classes/embeddedDist/repo" -Didea.launcher.port=7533 "-Didea.launcher.bin.path=/Applications/IntelliJ IDEA CE.app/Contents/bin" -Dfile.encoding=UTF-8 -classpath "/Users/Laust/Library/Application Support/IdeaIC2016.3/CeylonIDEA/classes/embeddedDist/lib/ceylon-bootstrap.jar:/Applications/IntelliJ IDEA CE.app/Contents/lib/idea_rt.jar" com.intellij.rt.execution.application.AppMain com.redhat.ceylon.launcher.Bootstrap run --run main default/unversioned
ceylon run: Module default/unversioned not found in the following repositories:
/Users/Laust/Library/Application Support/IdeaIC2016.
3/CeylonIDEA/classes/embeddedDist/repo
/Users/Laust/.ceylon/cache
https://modules.ceylon-lang.org/repo/1
[Maven] Aether
[NPM] npm
Process finished with exit code 1
The code executes fine on my other computer (windows 7).
the folder 'modules' contains the following:
default
default.car
default.car.sha1
default.src
default.src.sha1
and my build configuration looks as follows.
this is my code (in the file source/main.ceylon)
shared void main() {
print("Generating pretty sweet g-code:");
{Gcommand+} myGcommands = {
G00( Vector3(0.0, 0.0, 0.0) ),
G00( Vector3(9.0, 0.0, 0.0) ),
G00( Vector3(9.0, 9.0, 0.0) ),
G00( Vector3(0.0, 9.0, 0.0) ),
G00( Vector3(0.0, 0.0, 0.0) )
};
GcodeProgram myGcodeProgram = GcodeProgram( *myGcommands );
print(myGcodeProgram.toString());
}
"A carthesian coordinate class"
alias X => Float;
alias Y => Float;
alias Z => Float;
class Vector3(shared X x, shared Y y, shared Z z) {
}
"An abstract spec class for all G-code command classes"
abstract class Gcommand() {
shared formal String toString();
}
"G-code command for moving in a straight line at rapid speed"
class G00( Vector3 endPoint ) extends Gcommand() {
toString() => "G0 " + "X" + endPoint.x.string
+ "Y" + endPoint.y.string
+ "Z" + endPoint.z.string + "\n";
}
class GcodeProgram( Gcommand+ gcommands ) {
variable String stringifiedGcodeProgram = "";
shared String toString() {
for (gcommand in gcommands) {
stringifiedGcodeProgram = stringifiedGcodeProgram + gcommand.toString();
}
return stringifiedGcodeProgram;
}
}
The screenshot you provided shows that the run configuration isn't based on any IntelliJ module (Use classpath of module is set to [none]). This means that the configuration will not be run in your project folder where the modules directory lives. That directory contains the compiled code, and ceylon run will look for that directory when you ask it to run the default module.
Generally speaking, you should avoid creating run configurations manually. By clicking on the green arrow next to a runnable function's name, Ceylon IDE will automatically create and configure a correct run configuration.
To fix your existing run configuration, simply select the IntelliJ module that contains your code in the field labeled Use classpath of module.
See also the getting started guide for more information on how to get started with Ceylon IDE for IntelliJ.
That might be a bug with the IntelliJ plugin not handling "default" modules correctly. We tend not to use default modules much because they're more limited than regular modules.
Try creating a module and moving your code to it. THat will most likely fix the problem. If so you can then open an issue to have this bug fixed here: https://github.com/ceylon/ceylon-ide-intellij/issues/new
There appears to be something messed up in the project setup here. Note the list of repos that are being searched:
Module default/unversioned not found in the following repositories:
/Users/Laust/Library/Application Support/IdeaIC2016.3/CeylonIDEA/classes/embeddedDist/repo
/Users/Laust/.ceylon/cache
https://modules.ceylon-lang.org/repo/1
[Maven] Aether
[NPM] npm
I would expect to see a repo of form your-project-dir/modules as the second entry in that list, but it's not there.
That is to say, ceylon run is not looking in the modules directory where the compiled .car is. So the question is why that repo is missing from the list.
What do you see in Project Structure > Modules > Ceylon > Repositories?
In this question, the first (and only) answer tells how to create a new module.
I have a few comments to that answer:
when beginning on a new project, you probably don't need an intricate nested naming hierarchy for your modules. You will get that, if you use periods in your module name (eg. my.ceylon.example), so I suggest you stick to a simple name such as main.
when creating your new module, you will (among other things) be asked to specify a 'Runnable unit name'. The purpose of this field is to tell IntelliJ which of your modules' classes it should execute when starting your program. In other words, this becomes the entry point to your program. A suitable name for this could (also) be main.
Ceylon projects are divided into modules, modules are divided into packages, and packages are divided into classes and top-level functions. When you create a module, a package is automatically created under this module. The path for your code files under this module will be 'source/moduleName/packageName'. When creating a new module, you don't get to specify the name for the first package in the module. Instead the package is given the same name as your module name. Thus a module named 'main' would have this path: source/main/main as the path for it's code files.
In your new modules folder (eg. source/main/main) three new files will be created. Find the file that is named after the 'Runnable unit name' you chose earlier. Your code should go into this file. Also, your code should have a class with the exact same name that you chose as your 'Runnable unit name'.
the answer used the fancy term 'runnable unit', by which he just means a file containing Ceylon code.
remember to delete the file containing your old 'default' module, before trying to run your new module.
a module name cannot start with a capital letter.
modules/ is the output directory where compiled code goes. It is automatically recreated from the code in source/ when the project is built.

Using JUnit in Jython - NameError for assertTrue

Environment Details
Mac OS X 10.9
Oracle JDK 1.7.0_55 64-bit
jython-standalone-2.5.3.jar
junit-4.11
What I have done so far
I have added the junit jar to /Library/Java/Extensions.
I invoked Jython as follows java -jar jython-standalone-2.5.3.jar
In the Jython interpreter, I imported the following import org.junit.Assert, and this import was successful.
Problem
When I tried to use assertTrue, I got a NameError in the interpreter. Why is this so?
I understand that assertTrue is a static method. Not sure what implication this has when I try to use it in Jython.
Additional Context
I am using XMLUnit in Jython. Was able to successfully import the Diff class from org.custommonkey.xmlunit in Jython. Also able to use the methods in this class, and call them on a Diff object. The result of this method call is what I am trying to pass to assertTrue, when it throws the error.
from org.custommonkey.xmlunit import Diff
import org.junit.Assert
xml1 = ...some XML string...
xml2 = ...some XML string...
myDiff = Diff(xml1, xml2)
assertTrue(myDiff.similar())
Hope this additional information is useful in identifying a solution to this problem.
Latest Status
I narrowed it down to setting this property python.security.respectJavaAccessibility = false, since the Assert() constructor is protected.
Still trying to get it to work. Any help is greatly appreciated.
Figured it out.
In addition to junit.jar file, the hamcrest-core.jar file also needed to be copied to /Library/Java/Extensions.
Then I got rid of the jython.jar file, and instead installed it using the jython installer.
After the installation was completed, I updated the registry file in the installation folder, specifically setting this property python.security.respectJavaAccessibility = false.
Now I am able to see the assertTrue method, and no longer getting a NameError.

play, scala and jerkson noClassDefFound error

I am trying to work with jerkson in play and with scala 2.10.
However, i want to load data fixtures based on a json files. for this prcoedure I'm trying to load the json with the "parse" command from jerkson.
That ultimatly fails.
I'm doing this in the "override def onStart(app: Application)" function. The error:
NoClassDefFoundError: Could not initialize class com.codahale.jerkson.Json$
Any guesses why this is happening ? I have the following libs in my deps.:
"com.codahale" % "jerkson_2.9.1" % "0.5.0",
"com.cloudphysics" % "jerkson_2.10" % "0.6.3"
my parsing command is:
com.codahale.jerkson.Json.parse[Map[String,Any]](json)
Thanks in advance
A NoClassDefFoundError generally means there is some sort of issues with the classpath. For starters, if you are running on scala 2.10, I would remove the following line from your sbt file:
"com.codahale" % "jerkson_2.9.1" % "0.5.0"
Then, make sure the com.cloudphysics jerkson jar file is available in your apps classpath and try your test again.