SparkStreaming tasks fails with NullPointerException - kafka-producer-api

I'm running spark job with kafka-clients:2.2.1 and
spark-streaming-kafka-0-10_2.11:2.4.3
The job is running in the following mode:
val scc: StreamingContext = new StreamingContext(spark, "180 seconds")
val kafkaSink: Broadcast[KafkaSink] = ...
val stream = KafkaUtils.createDirectStream(...)
stream //. do some logic
.foreachRDD { (rdd, time) =>
p.foreach { row =>
val rowJson = convertRowToJSON(row)
kafkaSink.value.send(kafkaOptions.topic, rowJson) }
}
The job is running but for some mini-batches, there are failed tasks with following exceptions:
java.lang.NullPointerException
at org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata(KafkaProducer.java:1010)
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:916)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:897)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:774)
at com.outbrain.recs.rtap.kafka.KafkaSink.send(KafkaSink.scala:42)
at
What could be the reason for those exceptions?

Related

Groovy/Jenkins: how to prettify json string?

How do you capture a JSON object as a prettified string using a Jenkins declarative-syntax pipeline?
pipeline {
agent any
stages {
stage( "Set up" ) {
steps {
script {
hostname = "bld-machine"
reply_email = "jenkins#${hostname}.company.com"
actor_email = "user#company.com"
status_json = initStatusJson()
}
}
}
/** Try figure out the difference between "global" and "env." variables. */
stage( "Capture variables" ) {
steps {
script {
status_json.env["var"] = "${env.var}" as String
status_json.env["var2"] = "${var}" as String
}
}
}
}
post {
always {
script {
def pretty_json = writeJSON( returnText: true, json: status_json )
}
emailext( subject: "CI/CD | ${currentBuild.currentResult}",
from: "${reply_email}",
to: "${actor_email}",
mimeType: "text/plain",
body: "${pretty_json}" )
}
}
}
def initStatusJson() {
def json_obj = readJSON text: '{}'
json_obj.job = readJSON text: '{}'
json_obj.env = [:]
json_obj.job.name = "${JOB_BASE_NAME}" as String
json_obj.job.number = "${BUILD_ID}" as String
json_obj.job.server = "${JENKINS_URL}" as String
json_obj.job.visualization = "${JENKINS_URL}/blue/organizations/jenkins/${JOB_BASE_NAME}/detail/${JOB_BASE_NAME}/${BUILD_ID}/pipeline" as String
return json_obj
}
The def pretty_json =... statement in the above Jenkinsfile triggers the following error:
WARNING: Unknown parameter(s) found for class type WARNING: Unknown parameter(s) found for class type 'org.jenkinsci.plugins.pipeline.utility.steps.json.WriteJSONStep': returnText
[Pipeline] }
[Pipeline] // script
Error when executing always post condition:
java.lang.IllegalArgumentException: You have to provided a file for writeJSON.
at org.jenkinsci.plugins.pipeline.utility.steps.json.WriteJSONStepExecution.run(WriteJSONStepExecution.java:61)
at org.jenkinsci.plugins.pipeline.utility.steps.json.WriteJSONStepExecution.run(WriteJSONStepExecution.java:43)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
What I have tried:
The def pretty_json = writeJSON( returnText: true, json: status_json ) statement is inspired by these resources:
Jenkinsfile pipeline construct JSON object and write to file
https://www.jenkins.io/doc/pipeline/steps/pipeline-utility-steps/#writejson-write-json-to-a-file-in-the-workspace
I also tried def pretty_json = writeJSON returnText: true, json: status_json which resulted in an identical error.
status_json.toString() returns a valid, but non-prettified JSON string.
I tried def pretty_json = JsonOutput.toJson(status_json) based on Create JSON strings from Groovy variables in Jenkins Pipeline, and it generates this error:
Error when executing always post condition:
groovy.lang.MissingPropertyException: No such property: JsonOutput for class: groovy.lang.Binding
Tried def pretty_json = groovy.json.JsonOutput.prettyPrint(status_json) based on https://gist.github.com/osima/1161966, and it generated this error:
Error when executing always post condition:
groovy.lang.MissingMethodException: No signature of method: java.lang.Class.prettyPrint() is applicable for argument types: (net.sf.json.JSONObject)
Update: Attempted #daggett's solution as follows:
post {
always {
script {
def pretty_json = status_json.toString(2)
}
emailext( subject: "CI/CD | ${currentBuild.currentResult}",
from: "${reply_email}",
to: "${actor_email}",
mimeType: "text/plain",
body: "${pretty_json}" )
}
}
...and also tried some variations like pretty_json = ... (instead of def pretty_json = ...) and also moving the pretty_json assignment outside the script{...} scope...but none worked.
Inside the script{...} context, the .toString(2) generated this error:
Scripts not permitted to use method net.sf.json.JSON toString int.
Outside the script{...} context, it generated what I interpret to be a "syntax error":
WorkflowScript: 79: Expected a step # line 79, column 7.
pretty_json = status_json.toString(2)
According to last error message
groovy.lang.MissingMethodException:
No signature of method: java.lang.Class.prettyPrint()
is applicable for argument types: (net.sf.json.JSONObject)
You have net.sf.json.JSONObject in status_json variable.
that's really strange - seems you are getting status_json not in a standard way for jenkins
however according to documentation of this class
http://json-lib.sourceforge.net/apidocs/jdk15/net/sf/json/JSONObject.html#toString(int)
just do following to make pretty json:
def pretty_json = status_json.toString(2)
If you have Scripts not permitted to use method XYZ exception:
for security reasons a lot of non-standard methods are disabled in jenkins.
refer this answer to resolve this kind of issue: https://stackoverflow.com/a/39412951/1276664
and finally - almost every case from your question should work:
writeJSON( returnText: true, json: status_json ) :
update pipeline-utility-steps jenkins plugin to the latest version to support returnText parameter
the same as above
...
JsonOutput.toJson(status_json) : JsonOutput class located in groovy.json package. you could import this package at t
he beginning of the script import groovy.json or call it like this: groovy.json.JsonOutput.toJson(status_json). note that this method returns non-formatted json.
groovy.json.JsonOutput.prettyPrint(status_json) : check the documentation for JsonOutput.prettyPrint - it could be called for string and not for object. so this could work: groovy.json.JsonOutput.prettyPrint(status_json.toString()) but only in case when status_json.toString() returns a valid json and JsonOutput.prettyPrint allowed to be called in jenkins admin.
I just did a test and it gave results :
def pretty_json = writeJSON( returnText: true, json: status_json , pretty: 4)
Note : Ensure you have the plugin Pipeline Utility Steps installed. Or reinstall it again.
Below is the script example:
#!groovy
import hudson.model.Result
import groovy.json.*
pipeline
{
agent any
stages
{
stage ('Set up')
{
steps
{
script
{
hostname = "bld-machine"
reply_email = "jenkins#${hostname}.company.com"
actor_email = "user#company.com"
status_json = initStatusJson()
println (status_json)
}
}
}
stage ('Capture variables')
{
steps
{
script
{
// Added just for test
status_json.env["var"] = "Alt" as String
status_json.env["var2"] = "su" as String
println (status_json)
}
}
}
}
post {
always {
script {
def pretty_json = writeJSON( returnText: true, json: status_json , pretty: 4)
println (pretty_json)
emailext( subject: "CI/CD | ${currentBuild.currentResult}",
from: "${reply_email}",
to: "${actor_email}",
mimeType: "text/plain",
body: "${pretty_json}" )
}
}
}
}
def initStatusJson() {
def json_obj = readJSON text: '{}'
json_obj.job = readJSON text: '{}'
json_obj.env = [:]
json_obj.job.name = "${JOB_BASE_NAME}" as String
json_obj.job.number = "${BUILD_ID}" as String
json_obj.job.server = "${JENKINS_URL}" as String
json_obj.job.visualization = "${JENKINS_URL}/blue/organizations/jenkins/${JOB_BASE_NAME}/detail/${JOB_BASE_NAME}/${BUILD_ID}/pipeline" as String
return json_obj
}
Output log :

How to load data class that include MutableList<Uri> from SharedPreference?

I'm trying to use SharedPreference and error occur when i load data. I think the problem is that my data class include MutableList, but i don't know how to handle it.
private lateinit var albumData:MutableList<Album>
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
loadData()
connectAdapter()
btnAddAlbum.setOnClickListener {
if(isStoragePermitted()) {
openGallery()
}
}
}
fun saveData() {
val sharedPreferences = getSharedPreferences("shared preferences", Context.MODE_PRIVATE)
val editor = sharedPreferences.edit()
val gson = GsonBuilder().create()
val json = gson.toJson(albumData)
editor.putString("Album data", json)
editor.apply()
}
fun loadData() {
val sharedPreferences = getSharedPreferences("shared preferences", Context.MODE_PRIVATE)
val gson = GsonBuilder().create()
val json = sharedPreferences.getString("Album data", null)
var test = mutableListOf<Album>()
if(json != null) {
Log.d("Log_show", "data load 1")
val type = object : TypeToken<MutableList<Album>>() {}.type
Log.d("Log_show", "data load 2")
albumData = gson.fromJson<MutableList<Album>>(json, type)
Log.d("Log_show", "data load 3")
}else{
albumData = mutableListOf()
}
}
override fun onDestroy() {
saveData()
Log.d("Log_show", "shut down")
super.onDestroy()
}
Above is part of my functions in main activity.
data class Album (
var albumImages : MutableList<Uri>,
var albumTitle : String,
var pictureCount : Int
)
This is my data class "Album"
When i first open the app, it save data well and no error occur.
But when i close the app and open it again, the error occur in process of loading data
This is part of my log of "Log_show" and "data load 3" didn't appear
2020-11-04 23:12:33.418 15720-15720/kr.co.ddophi.autochangingwallpaper D/log: data load 1
2020-11-04 23:12:33.420 15720-15720/kr.co.ddophi.autochangingwallpaper D/log: data load 2
the full log is like this
2020-11-05 14:48:19.319 6956-6956/kr.co.ddophi.autochangingwallpaper D/Log_show: data load 1
2020-11-05 14:48:19.321 6956-6956/kr.co.ddophi.autochangingwallpaper D/Log_show: data load 2
2020-11-05 14:48:19.331 6956-6956/kr.co.ddophi.autochangingwallpaper D/AndroidRuntime: Shutting down VM
2020-11-05 14:48:19.336 6956-6956/kr.co.ddophi.autochangingwallpaper E/AndroidRuntime: FATAL EXCEPTION: main
Process: kr.co.ddophi.autochangingwallpaper, PID: 6956
java.lang.RuntimeException: Unable to start activity ComponentInfo{kr.co.ddophi.autochangingwallpaper/kr.co.ddophi.autochangingwallpaper.MainActivity}: java.lang.RuntimeException: Failed to invoke private android.net.Uri() with no args
at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2957)
at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3032)
at android.app.ActivityThread.-wrap11(Unknown Source:0)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1696)
at android.os.Handler.dispatchMessage(Handler.java:105)
at android.os.Looper.loop(Looper.java:164)
at android.app.ActivityThread.main(ActivityThread.java:6944)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.Zygote$MethodAndArgsCaller.run(Zygote.java:327)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1374)
Caused by: java.lang.RuntimeException: Failed to invoke private android.net.Uri() with no args
at com.google.gson.internal.ConstructorConstructor$3.construct(ConstructorConstructor.java:113)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:212)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:41)
at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:82)
at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:61)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:131)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:222)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:41)
at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:82)
at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:61)
at com.google.gson.Gson.fromJson(Gson.java:927)
at com.google.gson.Gson.fromJson(Gson.java:892)
at com.google.gson.Gson.fromJson(Gson.java:841)
at kr.co.ddophi.autochangingwallpaper.MainActivity.loadData(MainActivity.kt:174)
at kr.co.ddophi.autochangingwallpaper.MainActivity.onCreate(MainActivity.kt:43)
at android.app.Activity.performCreate(Activity.java:7183)
at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1220)
at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2910)
at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3032) 
at android.app.ActivityThread.-wrap11(Unknown Source:0) 
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1696) 
at android.os.Handler.dispatchMessage(Handler.java:105) 
at android.os.Looper.loop(Looper.java:164) 
at android.app.ActivityThread.main(ActivityThread.java:6944) 
at java.lang.reflect.Method.invoke(Native Method) 
at com.android.internal.os.Zygote$MethodAndArgsCaller.run(Zygote.java:327) 
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1374) 
Caused by: java.lang.InstantiationException: Can't instantiate abstract class android.net.Uri
at java.lang.reflect.Constructor.newInstance0(Native Method)
at java.lang.reflect.Constructor.newInstance(Constructor.java:334)
at com.google.gson.internal.ConstructorConstructor$3.construct(ConstructorConstructor.java:110)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:212) 
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:41) 
at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:82) 
at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:61) 
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:131) 
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:222) 
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:41) 
at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:82) 
at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:61) 
at com.google.gson.Gson.fromJson(Gson.java:927) 
at com.google.gson.Gson.fromJson(Gson.java:892) 
at com.google.gson.Gson.fromJson(Gson.java:841) 
at kr.co.ddophi.autochangingwallpaper.MainActivity.loadData(MainActivity.kt:174) 
at kr.co.ddophi.autochangingwallpaper.MainActivity.onCreate(MainActivity.kt:43) 
at android.app.Activity.performCreate(Activity.java:7183) 
at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1220) 
at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2910) 
at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3032) 
at android.app.ActivityThread.-wrap11(Unknown Source:0) 
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1696) 
at android.os.Handler.dispatchMessage(Handler.java:105) 
at android.os.Looper.loop(Looper.java:164) 
at android.app.ActivityThread.main(ActivityThread.java:6944) 
at java.lang.reflect.Method.invoke(Native Method) 
at com.android.internal.os.Zygote$MethodAndArgsCaller.run(Zygote.java:327) 
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1374) 
Any help would be appreciated
Thank you
You should custom TypeAdapter for serialize and deserialize Uri.
class UriTypeAdapter : TypeAdapter<Uri>() {
override fun write(out: JsonWriter?, value: Uri?) {
value?.let {
out?.value(it.toString())
} ?: run {
out?.nullValue()
}
}
override fun read(reader: JsonReader?): Uri {
return if (reader?.peek() == JsonToken.NULL) {
reader.nextNull()
Uri.EMPTY
} else {
val uriString = reader?.nextString()
uriString?.let {
Uri.parse(it)
} ?: run {
Uri.EMPTY
}
}
}
}
and setup Gson like this:
val gson = GsonBuilder()
.registerTypeHierarchyAdapter(Uri::class.java, UriTypeAdapter())
.create()

Batch json strings processing using playframework in Scala

I use API which might either return a single json string, a batch of json strings or the string PROCESSING_TIMEOUT, e.g.:
{"id":123,"field1":"test1"}
or:
{"id":123,"field1":"test1"}
{"id":456,"field2":"test2"}
{"id":789,"field3":"test3"}
or (in case of asynchronous processing timeout in remote API)
PROCESSING_TIMEOUT
In the function getRestContent I want to be able to correcty process all these possible outputs, including also the timeout errors. In the current version of the function I lack the possibility to process batch json strings. I think that the best option would be that the function returns List<JsValue> instead of JsValue.
How can I do this modification using Play Framework.
def getRestContent(url:String,param:String,paramValue:String): JsValue = {
var output : JsValue = null
var httpOutput : String = null
try {
val response: HttpResponse[String] = Http(url).timeout(connTimeoutMs = 10000000, readTimeoutMs = 10000000).param(param,paramValue).asString
httpOutput = response.body
} catch
{
case ex: Exception => {
println("Failed connection with remote API")
}
}
if (!httpOutput.contains("PROCESSING_TIMEOUT") && httpOutput != null)
{
try {
output = Json.parse(httpOutput)
}
catch
{
case ex: Exception => {
println("Failed to process a document")
}
}
}
else
{
println("Asynchronous processing timeout")
}
if (output != null)
{
return output
}
else {
return new JsObject(Map("empty" -> JsNumber(0)))
}
}
Disregarding Play Framework your code is not Scala-flavored in general.
Below is a simple example of how better to approach such task in more Scala-way.
Highlights: use scala.util.Try wrapper to normally work with exceptions in a functional way, use pattern matching and monadic map and other methods of collections.
I hope the code below is self-explanatory. I mimicked the API. It returns Strings and not JSON but your 3 basic cases are sufficiently emulated: single line, multiple lines and exceptional case.
Try to run the program few times and you will see that all 3 cases are handled correctly:
import scala.util.{Failure, Success, Try, Random}
/**
* Created by Alex on 3/10/2016.
*/
object Temp {
case class Item(name:String, value:Int)
object API{
def getTimeOutResponse:String = throw new TimeoutException("no luck this time")
def getSingleLineResponse = "{name: \"Alex\", value: 1}"
def getMultiLineResponse = "{name: \"Alex\", value: 1}\n{name: \"HackerDuck\", value: 2}"
def getRandomResponse = (Math.abs(Random.nextInt() % 3)) match{
case 0 => getTimeOutResponse
case 1 => getSingleLineResponse
case 2 => getMultiLineResponse
}
}
def getResults:List[Item]={
Try(API.getRandomResponse) match{
case Success(s) =>{
s.split("\n").toList.map{item =>
val parts = item.split(", ")
Item(parts(0).replace("{name: ", "").replace("\"", ""), parts(1).replace("value: ", "").replace("}", "").toInt)
}
}
case Failure(_) =>{
println("API timeout happened")
List.empty[Item]
}
}
}
def main(args:Array[String])={
println(getResults)
println(getResults)
println(getResults)
}
}
A sample output from my console:
API timeout happened
List()
List(Item(Alex,1))
List(Item(Alex,1), Item(HackerDuck,2))
Process finished with exit code 0
The code becomes less cluttered without all these if statements and !=null and alike. Also you can streamline conversion of your elements sequence to List in one place.

Slick: Return inserted row with auto increment id

I'm trying to make an insert into a MySQL table and to return the row with the auto increment id. My code is below:
private val Log = TableQuery[GCMLogTable]
def save(log: GCMLog): Try[GCMLog] = Try {
val newId = (Log returning Log.map(_.id)) += log
log.copy(id = newId)
}
But my compilation fails for my code with the below error:
type mismatch;
found : slick.profile.FixedSqlAction[Long,slick.dbio.NoStream,slick.dbio.Effect.Write]
required: Long
Also tried
def save(log: GCMLog): Try[GCMLog] = Try {
(Log returning Log.map(_.id)
into ((log, newId) => log.copy(id = newId))
) += log
}
But still fails with
type mismatch;
found : slick.profile.FixedSqlAction[models.GCMLog,slick.dbio.NoStream,slick.dbio.Effect.Write]
required: models.GCMLog
[I referred the SO question How to catch slick postgres exceptions for duplicate key value violations and Slick documentation here http://slick.typesafe.com/doc/3.1.1/queries.html ]
Much appreciated if someone can tell me what's going on and how this can be fixed.
Thanks!
def save(log: GCMLog): Try[GCMLog] = Try {
(Log returning Log.map(_.id))
into ((log, newId) => log.copy(id = newId))
) += log
}
UPDATE:
It looks db.run needs to be perform to convert that Action into result.
slick supports it:
def create(objectToCreate: MyCaseClass): MyCaseClass = {
db.withSession {
(self returning self) += objectToCreate
}
}
Try this
private val Log = TableQuery[GCMLogTable]
private val db = Database.forConfig("mysql")
def save(log: GCMLog): Try[GCMLog] = Try {
val newId = db.run((Log returning Log.map(_.id) into ((Log,id) => Log.copy(id=id))) += log)
Await.result(newId, Duration.Inf)
}

Conditional port in a Chisel Module

I have a selectable feature which is not normally required. However to support this feature, some I/O ports should be added to the origin Module I/O port.
I am doing it in this way:
import Chisel._
class TestModule extends Module {
class IOBundle extends Bundle {
val i = Bool(INPUT)
val o = Bool(OUTPUT)
}
class IOBundle_EXT extends IOBundle {
val o_ext = Bool(OUTPUT)
}
val io = if(true) new IOBundle_EXT else new IOBundle;
io.o := io.i
io.o_ext := io.i
}
After running sbt "run TestModule --backend c --compile --test --genHarness", the compiler complains:
[error] xxxx/test/condi_port.scala:17: value o_ext is not a member of TestModule.this.IOBundle
[error] io.o_ext := io.i
[error] ^
[error] one error found
[error] (compile:compile) Compilation failed
So the if statement has no effect. val io is still assigned to IOBundle, rather than the extended IOBoundle_EXT, which makes no sense to me.
Chisel now supports Options in IO bundles.
As an example, I explored Options here (https://github.com/ucb-bar/riscv-boom/commit/da6edcb4b7bec341e31a55567ee04c8a1431d659), but here's a summary:
class MyBundle extends Bundle
{
val my_ext = if (SOME_SWITCH) Some(ExtBundle) else None
}
...
io.my_ext match
{
case Some(b: ExtBundle) =>
my_ext.a := Bool(false)
...
case _ => require (!SOME_SWITCH)
}
It's incredibly verbose, but I was able to get it working even when doing bulk connects and hiding bundles within bundles, etc.
Even though the compiler could determine that only one result is possible (the expression is always true), the type system sets the type of the result equal to the greatest common subtype of the two possible (true or false) sub-expressions.
You can verify this trivially with the following:
scala> val result = if (true) 1 else "one"
result: Any = 1
Try this:
import Chisel._
class TestModule(val useEXT : Boolean) extends Module {
class IOBundle extends Bundle {
val i = Bool(INPUT)
val o = Bool(OUTPUT)
}
class IOBundle_EXT extends IOBundle {
val o_ext = Bool(OUTPUT)
}
val io = {
if(useEXT) {
val res = new IOBundle_EXT; res.o_ext := res.i; res
} else {
new IOBundle }};
io.o := io.i
}