What are these `a.bits.user.amba_prot` signals and why are they only uninitialized conditionally in my HarnessBinder? - chisel

Question

Context: I began using Chipyard about a month back to facilitate the building of a quick prototype using RISC-V cores on the VCU118. Chipyard was perfect, but required me to step up and learn Chisel and Rocket-chip tools to extend interconnection to my design.
The first piece of hardware to hook up was the PCIe to AXI Bridge IP provided by Xilinx known as XDMA. fpga-shells provides a wrapper, shell, and overlay for this IP already, so with some studying of the Chipyard System, HarnessBinders, and IOBinders, I managed to hook it up. The overlay was placed like so:
val overlayOutput = dp(PCIeOverlayKey).last.place(PCIeDesignInput(wrangler=dutWrangler.node, corePLL=harnessSysPLL)).overlayOutput
val (pcieNode: TLNode, pcieIntNode: IntOutwardNode) = (overlayOutput.pcieNode, overlayOutput.intNode)
val (pcieSlaveTLNode: TLIdentityNode, pcieMasterTLNode: TLAsyncSinkNode) = (pcieNode.inward, pcieNode.outward)
There are two slaves, but I'll only show the IOBinders and HarnessBinders for one. I'm assuming my other Port mixin is functioning correctly since it's exactly like the CanHaveMasterTLMMIO port, but with a separate key and different address ranges. I realize this is inefficient, but was easier than creating an external bus. Here is the IOBinder which takes advantage of CanHaveMasterTLMMIOPort to introduce a master MMIO port to the system bus.
class WithXDMASlaveIOPassthrough extends OverrideIOBinder({
(system: CanHaveMasterTLMMIOPort) => {
val io_xdma_slave_pins_temp = IO(DataMirror.internal.chiselTypeClone[HeterogeneousBag[TLBundle]](system.mmio_tl)).suggestName("tl_slave_mmio")
io_xdma_slave_pins_temp <> system.mmio_tl
(Seq(io_xdma_slave_pins_temp), Nil)
}
})
I retrieve and connect the slave nodes in the TestHarness like so:
val inParamsMMIOPeriph = topDesign match { case td: ChipTop =>
td.lazySystem match { case lsys: CanHaveMasterTLMMIOPort =>
lsys.mmioTLNode.edges.in(0)
}
}
val inParamsControl = topDesign match {case td: ChipTop =>
td.lazySystem match { case lsys: CanHaveMasterTLCtrlPort =>
lsys.ctrlTLNode.edges.in(0)
}
}
val pcieClient = TLClientNode(Seq(inParamsMMIOPeriph.master))
val pcieCtrlClient = TLClientNode(Seq(inParamsControl.master))
val connectorNode = TLIdentityNode()
// pcieSlaveTLNode should be driven for both the control slave and the axi bridge slave
connectorNode := pcieClient
connectorNode := pcieCtrlClient
pcieSlaveTLNode :=* connectorNode
Finally, I connect the node to a harness binder. I followed the way DDR was connected in Chipyard for this step.
class WithPCIeClient extends OverrideHarnessBinder({
(system: CanHaveMasterTLMMIOPort, th: BaseModule with HasHarnessSignalReferences, ports: Seq[HeterogeneousBag[TLBundle]]) => {
require(ports.size == 1)
th match { case vcu118th: FCMVCU118FPGATestHarnessImp => {
val bundles = vcu118th.fcmOuter.pcieClient.out.map(_._1)
val pcieClientBundle = Wire(new HeterogeneousBag(bundles.map(_.cloneType)))
pcieClientBundle <> DontCare // Some signals aren't being driven to this bundle, but it's hard to know how critical that is. Only happens when myPeripheral is on.
bundles.zip(pcieClientBundle).foreach{case (bundle, io) => bundle <> io}
pcieClientBundle <> ports.head
} }
}
})
I added the pcieClientBundle <> DontCare to surpress these undriven signals. This problem only affects the signals driven to both slaves.
Signals such as:
a.bits.user.amba_prot.fetch
a.bits.user.amba_prot.secure
a.bits.user.amba_prot.modifiable
a.bits.user.amba_prot.privileged
are undriven (resulting in a $RefNotInitializedException in FIRRTL pass through)
in both wires. I recognize these come from TLToAXI4 but what's weird is that they are only undriven when I connect another peripheral to the bus. This peripheral does not leave ChipTop. It has three master AXI buses connected to the system like so:
( peirpheralMasterNode1
:= TLBuffer(BufferParams.default)
:= TLWidthWidget(8)
:= AXI4ToTL()
:= AXI4UserYanker(capMaxFlight=Some(16))
:= AXI4Fragmenter()
:= AXI4IdIndexer(idBits=3)
:= AXI4Buffer()
:= peripheralTop.Master1)
fbus.fromPort(Some("PERIPHERAL_MASTER_1"))() := peripheralMasterNode1
The peripheral's AXI Lite slave is where I suspect the problem could be. Its node is declared like so:
val regCfgSlv = AXI4SlaveNode(Seq(AXI4SlavePortParameters(
slaves = Seq(AXI4SlaveParameters(
address = AddressSet.misaligned(0xf000E0000L, 0x1000L),
resources = regCfgDevice.reg("config"),
// executable = true, // Determines whether processor can execute from this memory.
supportsWrite = TransferSizes(1, 4),
supportsRead = TransferSizes(1, 4),
interleavedId = Some(0))),
beatBytes = 4
)))
And it is connected to the system bus using toSlave
sbus.toSlave(Some(portName)){
(peripheralTop.regCfgSlv
:= AXI4Buffer()
:= AXI4UserYanker(capMaxFlight=Some(2))
:= TLToAXI4()
// := TLWidthWidget(sbus.beatBytes))
:= TLFragmenter(4,
p(CacheBlockBytes),
// sbus.beatBytes,
holdFirstDeny = true)
:= TLWidthWidget(sbus.beatBytes))
}
When I include this mixin peripheral in my config by setting myPeripheral's key, I get the error about a.bits.user.amba_prot signals not being driven for both HarnessBinders Client Bundles [val pcieClientBundle = Wire(new HeterogeneousBag(bundles.map(_.cloneType)))]. When I only use one of myPeripheral or XDMA, the $RefNotInitializedException goes away.
Here are my configs. I have tried moving WithMyPeripheral around to no avail.
class CustomRocketConfig extends Config(
new WithPCIeMMIOPort ++ // add default external master port
new WithControlPort ++ // add control port for pcie cfg. // TODO: Crossbar this on MMIO Port? Move both MMIO and Control to a port on System Bus?
new freechips.rocketchip.subsystem.WithDefaultSlavePort ++ // add default external slave port
new WithMyPeripheral(MyPeripheralParams()) ++ // Link up myPeripheral
new freechips.rocketchip.subsystem.WithNBigCores(2) ++
new freechips.rocketchip.subsystem.WithNExtTopInterrupts(3) ++
new chipyard.config.AbstractConfig)
class WithPCIeTweaks extends Config (
new WithPCIeClient ++
new WithPCIeManager ++
new WithPCIeCtrlClient ++ // Same for these harness binders - ME
new WithXDMAMasterIOPassthrough ++
new WithXDMASlaveIOPassthrough ++
new WithXDMACtrlIOPassthrough // I imagine these three IOBinders can be combined into one - ME
//TODO: Probably need harness binder and io binder for interrupt if we use it
)
class myRocketConfig extends Config (
// new WithMyPeripheral(MyPeripheralParams()) ++ // Link up myPeripheral
new WithPCIeTweaks ++
new WithVCU118Tweaks ++
new WithMyVCU118System ++
new CustomRocketConfig
)
I hope my problem is clear and interesting. What are a.user.amba_prot signals and why are they undriven when I hook up both XDMA and my peripheral? Why is it that I can declare either myPeripheral or the XDMA, but, when I hook up both, these signals don't have drivers? I realize this is a difficult question with a lot of moving parts in an already scarcely viewed tag. If you took the time to read this and have suggestions, your kindness and expertise are greatly appreciated.
Edit: I think the issue might be parameter negotiation failing between Test Harness's diplomacy region and ChipTop's diplomacy region.
This is the control node in XDMA.
val control = AXI4SlaveNode(Seq(AXI4SlavePortParameters(
slaves = Seq(AXI4SlaveParameters(
address = List(AddressSet(c.control, c.ecamMask)),
resources = device.reg("control"),
supportsWrite = TransferSizes(1, 4),
supportsRead = TransferSizes(1, 4),
interleavedId = Some(0))), // AXI4-Lite never interleaves responses
beatBytes = 4)))
This is the port the system sees.
val mmioTLNode = TLManagerNode(
mmioPortParamsOpt.map(params =>
TLSlavePortParameters.v1(
managers = Seq(TLSlaveParameters.v1(
address = AddressSet.misaligned(params.base, params.size),
resources = device.ranges,
executable = params.executable,
supportsGet = TransferSizes(1, sbus.blockBytes),
supportsPutFull = TransferSizes(1, sbus.blockBytes),
supportsPutPartial = TransferSizes(1, sbus.blockBytes))),
beatBytes = params.beatBytes)).toSeq)
mmioPortParamsOpt.map { params =>
sbus.coupleTo(s"port_named_$portName") {
(mmioTLNode
:= TLBuffer()
:= TLSourceShrinker(1 << params.idBits)
:= TLWidthWidget(sbus.beatBytes)
:= _ )
}
}
param.beatBytes is currently set to 8 (site(MemoryBusKey).beatBytes). But the control configuration slave node is 4.

Answer 1

I have determined a somewhat satisfactory way around this error. My understanding of diplomacy might be flawed here, so if this helps someone craft a better answer, I'd love to hear it.
What are a.user.amba_prot signals?
These signals are a result of bridging protocols between TL and AXI4. If the master is originally AXI4, then it has ar and aw channel signals labelled prot and cache.
AxCACHE[0], Bufferable (B) bit
AxCACHE[1], Cacheable (C) bit (this is the "modifiable" bit)
AxCACHE[2], Read-allocate (RA) bit
AxCACHE[3], Write-allocate (WA) bit
AxPROT[0], Privileged Access bit
AxPROT[1], Secure Access bit
AxPROT[2], Data/Instruction Access bit
These signals are driven from AXI masters to AXI slaves.
Why is it that I can declare either myPeripheral or the XDMA, but, when I hook up both, these signals don't have drivers?
It's important to keep in mind the following point when using Chipyard: the Chipyard system is one diplomatic region. The Test Harness is another. So from Chipyard system perspective, XDMA appears completely as TL connections. It has a TL master driving its slave port and a TL slave on its master port. This is because XDMA in the PCIeOverlay uses Diplomacy to connect the AXI nodes to TileLink within the Test Harness diplomatic region. The system's diplomatic region can only see these TileLink master and slave ports. When AXI masters and slaves are connected, the TileLink client bundles now need amba_prot signals because the system can see it needs to connect to AXI slaves.
The workaround for this is to expose the AXI ports directly for XDMA. This way the system can see the AXI slaves and masters from XDMA and the bridge to TL can happen on the system side.