Displaying rgb8 pixel data in-browser - html

Question

I have found similar questions to mine on SO, but have not yet come across an answer to this problem. I have a rgb8 encoded image that I am trying to display in-browser, either in an img or canvas element. I am unsure how to convert this pixel data into an image properly, and was looking for any insight.
For context, the source of this rgb8 data is from a ROS topic with type sensor_msgs/Image. When subscribing to this topic using roslibjs, I am given the following object:
{
data: “MT4+CR…”, (of length 1228800)
encoding: "rgb8",
header: {
frame_id: “camera_color_optical_frame”,
seq: 1455,
stamp: ...timestamp info
},
height: 480,
is_bigendian: 0,
step: 1920,
width: 640
}
With the data string, I have tried displaying it on canvas, converting it to base64, etc. but have not been able to. I know about web_video_server in ROS to help send these images over a port, but that is not an option for me unfortunately - I need to work directly with the data.
Is there a way I can go about displaying this rgb8 data in the browser? Based on the documentation on here, data should be represented as a uint8[] (if that helps).
Thank you so much!

Answer 1

First create a canvas of the correct size and obtain a CanvasRenderingContext2D
// Assuming that imgMes is the image message as linked in question
const can = document.createElement("canvas");
can.width = imgMes.width;
can.height = imgMes.height;
const ctx = can.getcontext("2d");
Then create an image buffer to hold the pixels
const imgData = ctx.createImageData(0, 0, imgMes.width, imgMes.height);
const data = imgData.data;
const inData = imgMes.data;
Then read the data from the image message. Making sure to use the correct order as defined in the flag is_bigendian
var i = 0, j, y = 0, x;
while (y < imgMes.height) {
j = y * imgMes.step;
for (x = 0; x < imgMes.width; x ++) {
if (imgMes.is_bigendian) {
data[i] = inData[j]; // red
data[i + 1] = inData[j + 1]; // green
data[i + 2] = inData[j + 2]; // blue
} else {
data[i + 2] = inData[j]; // blue
data[i + 1] = inData[j + 1]; // green
data[i] = inData[j + 2]; // red
}
data[i + 3] = 255; // alpha
i += 4;
j += 3;
}
y++;
}
The put the pixel data into the canvas;
ctx.putImageData(imgData, 0, 0);
And add the canvas to your HTML
document.body.appendChild(can);
And you are done.
Note that I may have is_bigendian the wrong way around. If so just change the line if (imgMes.is_bigendian) { to if (!imgMes.is_bigendian) {
UPDATE
With more information regarding the data format i was able to extract the image.
I used atob to decode the Base64 string. This returns another string. I then iterated each character in the string, getting the character code to add to each pixel.
It is unclear where the endianess is. My guess is that it is in the decoded string and thus the code swaps bytes for each char code as it makes no sense to have endianess on multiples of 3 bytes
const can = document.createElement("canvas");
can.width = imgMes.width;
can.height = imgMes.height;
const ctx = can.getContext("2d");
const imgData = ctx.createImageData(imgMes.width, imgMes.height);
const data = imgData.data;
const inData = atob(imgMes.data);
var j = 0; i = 4; // j data in , i data out
while( j < inData.length) {
const w1 = inData.charCodeAt(j++); // read 3 16 bit words represent 1 pixel
const w2 = inData.charCodeAt(j++);
const w3 = inData.charCodeAt(j++);
if (!imgMes.is_bigendian) {
data[i++] = w1; // red
data[i++] = w2; // green
data[i++] = w3; // blue
} else {
data[i++] = (w1 >> 8) + ((w1 & 0xFF) << 8);
data[i++] = (w2 >> 8) + ((w2 & 0xFF) << 8);
data[i++] = (w3 >> 8) + ((w3 & 0xFF) << 8);
}
data[i++] = 255; // alpha
}
ctx.putImageData(imgData, 0, 0);
document.body.appendChild(can);
From the example data I got an image of some paving near a road.