How to display OpenCV webcam capture frame in JavaFX app - swing

Question

I'm working on creating desktop app using JavaFX, which allows you to scan qr codes from a webcam.
I decided to choose JavaCV to handle webcam capturning. However, the problem is that the CanvasFrame class creates a Swing JFrame. My main goal is to find the best way to integrate this with JavaFX components.
My question is whether it is possible to create CanvasFrame in JPanel(or other Swing/JavaFx component), not in JFrame. In this option I would wrap JPanel into SwingNode - it's solve my integration problem.
I'm also asking for other suggestions that solves JavaFX with JavaCV integration problem in my case.
Maybe there is a direct way to embed a camera screen into a JavaFx component.
I'm pasting the test code below. My code is written in kotlin, but it doesn't affect the problem:
import com.google.zxing.*
import com.google.zxing.client.j2se.BufferedImageLuminanceSource
import com.google.zxing.common.HybridBinarizer
import org.bytedeco.javacv.*
import java.awt.image.BufferedImage
import java.util.*
import java.util.concurrent.Executors
class Test {
companion object {
#JvmStatic
fun main(args: Array<String>) {
Executors.newSingleThreadExecutor().execute { testWebcam() }
}
private fun testWebcam() {
val grabber: OpenCVFrameGrabber = OpenCVFrameGrabber(0);
val canvasFrame: CanvasFrame = CanvasFrame("Cam")
grabber.start()
while (canvasFrame.isVisible) {
val frame: Frame = grabber.grabFrame() ?: break
canvasFrame.showImage(frame)
decodeQrCode(grabber)
}
}
private fun decodeQrCode(grabber: OpenCVFrameGrabber) {
val java2DFrameConverter: Java2DFrameConverter = Java2DFrameConverter()
val frame: Frame = grabber.grabFrame()
val image = java2DFrameConverter.getBufferedImage(frame)
val decodedQr = parseQr(image)
println(decodedQr)
}
private fun parseQr(image: BufferedImage): String? {
val reader: MultiFormatReader = MultiFormatReader()
val binaryBitmap: BinaryBitmap =
BinaryBitmap(HybridBinarizer(BufferedImageLuminanceSource(image)))
val hints: Hashtable<DecodeHintType, Any> = Hashtable()
hints[DecodeHintType.CHARACTER_SET] = "UTF-8"
hints[DecodeHintType.POSSIBLE_FORMATS] = listOf(BarcodeFormat.QR_CODE)
return try {
reader.decode(binaryBitmap, hints).text
} catch (e: NotFoundException) {
null;
}
}
}
}

Answer 1

There's a project at https://github.com/rladstaetter/javacv-webcam that has examples of using javacv with Swing, JavaFX, and a newer method of using a shared memory buffer between OpenCV and JavaFX's PixelBuffer.
Instead of using a CanvasFrame, you can use JavaFX's ImageView backed by a shared ByteBuffer. The pseudo-code is:
import java.nio.ByteBuffer
import javafx.scene.image._
import org.bytedeco.javacv.Frame
import org.bytedeco.opencv.global.opencv_imgproc._
import org.bytedeco.opencv.opencv_core.Mat
val videoView: ImageView = ImageView()
val grabber: OpenCVFrameGrabber = OpenCVFrameGrabber(0)
grabber.start()
// Fire off a thread to grab frames while the camera is active
// Each frame will ber passed to the updateView method below
// ... timer/thread omitted for brevity
val javaCVMat = Mat()
/** create buffer only once saves much time! */
val buffer: ByteBuffer = javaCVMat.createBuffer()
val formatByte: WritablePixelFormat<ByteBuffer> = PixelFormat.getByteBgraPreInstance()
fun updateView(frame: Frame): Unit = {
val w = frame.imageWidth()
val h = frame.imageHeight()
val mat = javaCVConv.convert(frame)
cvtColor(mat, javaCVMat, COLOR_BGR2BGRA)
val pb = PixelBuffer(w, h, buffer, formatByte)
val wi = WritableImage(pb)
videoView.setImage(wi)
}

Answer 2

I solved my problem. In my case, the best solution was to use Java2DFrameConverter:
import javafx.application.Application
import javafx.embed.swing.SwingFXUtils
import javafx.scene.Scene
import javafx.scene.image.ImageView
import javafx.scene.image.WritableImage
import javafx.scene.layout.VBox
import javafx.stage.Stage
import org.bytedeco.javacv.Frame
import org.bytedeco.javacv.Java2DFrameConverter
import org.bytedeco.javacv.OpenCVFrameGrabber
import java.awt.image.BufferedImage
import java.util.concurrent.Executors
class StackOverflow : Application() {
private val java2DFrameConverter: Java2DFrameConverter = Java2DFrameConverter()
companion object {
#JvmStatic
fun main(args: Array<String>) {
launch(StackOverflow::class.java)
}
}
override fun start(primaryStage: Stage) {
val grabber: OpenCVFrameGrabber = OpenCVFrameGrabber(0)
grabber.start()
val imageView: ImageView = ImageView()
Executors.newSingleThreadExecutor().execute {
while (true) {
val frame = grabber.grabFrame()
imageView.image = frameToImage(frame)
}
}
val scene: Scene = Scene(VBox(imageView), 800.0, 800.0)
primaryStage.scene = scene
primaryStage.show()
}
private fun frameToImage(frame: Frame): WritableImage {
val bufferedImage: BufferedImage = java2DFrameConverter.getBufferedImage(frame)
return SwingFXUtils.toFXImage(bufferedImage, null)
}
}