Direct/system command for reading the display buffer of EV3 bricks?

Question

I recall being able to read in the display buffer of NXT 2.0 bricks over Bluetooth/USB by issuing Read IO Map system commands. I've looked through the bytecodes and communication documentation for the EV3, but it's not clear how to do this with the EV3. Does anyone have any pointers?

Answer 1

I think you should be able to do this. The screen is represented by a framebuffer device. In Linux, everything is a file, so even the framebuffer is a file, namely /dev/fb0.

So, you should be able to read the display by reading the /dev/fb0 file using something like this:

EDIT: Using direct commands didn't work out so well, so here is a sample using system commands.

Getting file content
--------------------

Used to upload datalog files - file handle is only closed when reaching EOF
and file is not open for writing

BEGIN_GETFILE:

  Bytes send to the brick:

  xxxxxxxx0196xxxxxxx
  bbbbmmmmttssllllnnn...

  bbbb = Bytes in massage, mmmm = message counter, tt = type of command,
  ss = system command, llll = max bytes to read, nnnn.... = path


  Bytes send to the PC:

  xxxxxxxx039600xxxxxxxx00xxx
  bbbbmmmmttssrrllllllllhhppp...

  bbbb = bytes ion massage, mmmm = message counter, tt = type of command,
  ss = system command, rr = return status, llllllll = File size, hh = Handle,
  ppp... = payload


CONTINUE_GETFILE:

  Bytes send to the brick:

  0700xxxx019700xxxx
  bbbbmmmmttsshhllll

  bbbb = bytes in massage, mmmm = message counter, tt = type of command,
  ss = system command, hh = handle, llll = max bytes to read


  Bytes send to the PC:

  xxxxxxxx039700xxxxxxxx00xxx
  bbbbmmmmttssrrllllllllhhppp...

  bbbb = bytes in message, mmmm = message counter, tt = type of command,
  ss = system command, rr = return status, llllllll = File size,
  hh = Handle, ppp... = payload

Opcodes come from here System commands come from here. The size returned may be 0 since this is not a real file, but the framebuffer size is 7680 bytes (comes from here, i.e. (178+2)/3*128). Also, the maximum number of bytes allowable for max bytes to read depends on the connection type (USB, Bluetooth or Wi-Fi). 1000 should be a safe number.

Now, hopefully you have the data, but you still need to decode it. The framebuffer has an unusual format where there are 3 pixels per byte (see datasheet).

Assuming you have packed all of the data read into a single 7680 byte array you can use nested for loops to convert the data to whatever image format you like. For simplicity, I am going to use a .pbm. Just doing this in pseudo code...

// framebuffer data - 60 bytes/row * 128 rows
buffer[60*128]

// write the PBM header
write_line("P1")
write_line("178 128")
for (row = 0; row < 128; row++) {
    // last byte in each row only has 1 pixel, so only loop first 59 bytes
    for (offset = 0; offset < 59; offset++) {
        var b = buffer[row * 60 + offset];
        // 3 pixels per byte

        // bit 7 (or 6) is pixel 0
        write((b & 128) ? "1 " : "0 ")
        // bit 4 (or 3) is pixel 1
        write((b & 16) ? "1 " : "0 ")
        // bit 1 (or 0) is pixel 2
        write((b & 2) ? "1 " : "0 ")
    }
    // the last pixel
    b = buffer[row * 60 + offset]
    write_line((b & 128) ? "1" : "0")
}