Compiling Modules Separately and Linking - chisel

Question

I'm compiling some very large projects in chisel3 with lots of wires and connections. Right now, the top level object and all of its submodules are in the same Scala package, and I run sbt "runMain top.Instantiator --verilog". However, it crashes in FIRRTL with:
[info] Done packaging.
[info] Running top.Instantiator --verilog --testArgs vcs
[info] [0.005] Elaborating design...
[info] [68.146] Done elaborating.
[error] (run-main-0) java.lang.NegativeArraySizeException
[error] java.lang.NegativeArraySizeException
[error] at java.util.Arrays.copyOf(Arrays.java:3332)
[error] at org.antlr.v4.runtime.ANTLRInputStream.load(ANTLRInputStream.java:101)
[error] at org.antlr.v4.runtime.ANTLRInputStream.<init>(ANTLRInputStream.java:64)
[error] at org.antlr.v4.runtime.ANTLRInputStream.<init>(ANTLRInputStream.java:60)
[error] at org.antlr.v4.runtime.ANTLRInputStream.<init>(ANTLRInputStream.java:68)
[error] at firrtl.Parser$$anonfun$1.apply(Parser.scala:35)
[error] at firrtl.Parser$$anonfun$1.apply(Parser.scala:29)
[error] at firrtl.Utils$.time(Utils.scala:135)
[error] at firrtl.Parser$.parse(Parser.scala:29)
[error] at firrtl.Driver$$anonfun$execute$1.apply(Driver.scala:171)
[error] at firrtl.Driver$$anonfun$execute$1.apply(Driver.scala:140)
[error] at logger.Logger$$anonfun$makeScope$1.apply(Logger.scala:129)
[error] at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
[error] at logger.Logger$.makeScope(Logger.scala:127)
[error] at firrtl.Driver$.execute(Driver.scala:140)
[error] at chisel3.Driver$.execute(Driver.scala:180)
[error] at chisel3.Driver$.execute(Driver.scala:200)
[error] at fringe.CommonMain$class.main(CommonMain.scala:64)
[error] at top.Instantiator$.main(Instantiator.scala:17)
[error] at top.Instantiator.main(Instantiator.scala)
[error] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error] at java.lang.reflect.Method.invoke(Method.java:498)
[error] Nonzero exit code: 1
[error] (Compile / runMain) Nonzero exit code: 1
My guess is that the huge number of wires and connections in the project are causing some overflow. Does anyone have experience with this kind of error and advice on how to go about debugging it? Right now, my ideas are to
1) refactor the chisel code to make it easier to compile
2) figure out a way to generate verilog for each module separately and then link them together at the end. Is there a way to compile non-recursively so I don't run into the same error when I get to compiling top.Instantiator?
3) separate the packages/projects somehow so that sbt works on each one separately
Thanks

Answer 1

Adding idea 4) Use protobuf serialization between chisel and firrtl.
I can see that firrtl is blowing up in the parser. The protobuf support is fairly new, but I believe it is much less memory intensive than the antlr based parser.
Here's an example of producing a firrtl protobuf serialization from chisel3
import chisel3._
class XX extends Module {
val io = IO(new Bundle {
val i = Input(UInt(1.W))
val o = Output(UInt(1.W))
})
io.o := io.i
}
object PB {
def main(args: Array[String]): Unit = {
val c = Driver.elaborate(() => new XX)
Driver.dumpProto(c, None)
}
}
You can then generate the verilog with the firrtl compiler
firrtl -i XX.pb -X verilog -o XX.v
Or build your own compiler something like
import firrtl._
object PB {
def main(args: Array[String]): Unit = {
Driver.execute(Array("-i", "XX.pb", "-X", "verilog", "-o", "XX.v"))
}
}
Your other options may be worth pursuing but this might be an easier quick fix.