Slow refresh rate streaming the camera preview to a display

yokonav

Changing the mode, it reduces to 70ms per frame!!!

#define TFT_SPI_MODE SPI_MODE1

jens6151 0 1 1

@yokonav
Thanks for the hints for SPI_MODE1 and the time consumption of the swap bytes

It looks like Spresense uses the 2D Graphics hardware acceleration for the conversion, so we cannot hook in there.

However I found this. Seems like our issue is not new.
https://stackoverflow.com/questions/41675438/fastest-way-to-swap-alternate-bytes-on-arm-cortex-m4-using-gcc

So changing the swap to this halfs the time.

inline uint32_t Rev16(uint32_t a) {
  asm("rev16 %1,%0" : "=r"(a) : "r"(a));
  return a;
}

void swapColors(uint16_t *colors, uint32_t len) {
  len = len / 2;
  uint32_t *data = (uint32_t *)colors;
  for (uint32_t i = 0; i < len; i++) {
    data[i] = Rev16(data[i]);
  }
}

Though it looks like hacky. It seems the loop consumes some time and it is possible to squees a little more performance out with this. Don't ask why. It was empirical try & error.

void swapColors(uint16_t *colors, uint32_t len) {
  // works only if length is dividable!! Is the case here.
  len = len / 64;
  uint32_t *data = (uint32_t *)colors;
  while ( len-- ) {
    *(data) = Rev16(*(data));
    *(data + 1) = Rev16(*(data + 1));
    *(data + 2) = Rev16(*(data + 2));
    *(data + 3) = Rev16(*(data + 3));
    *(data + 4) = Rev16(*(data + 4));
    *(data + 5) = Rev16(*(data + 5));
    *(data + 6) = Rev16(*(data + 6));
    *(data + 7) = Rev16(*(data + 7));
    *(data + 8) = Rev16(*(data + 8));
    *(data + 9) = Rev16(*(data + 9));
    *(data + 10) = Rev16(*(data + 10));
    *(data + 11) = Rev16(*(data + 11));
    *(data + 12) = Rev16(*(data + 12));
    *(data + 13) = Rev16(*(data + 13));
    *(data + 14) = Rev16(*(data + 14));
    *(data + 15) = Rev16(*(data + 15));
    *(data + 16) = Rev16(*(data + 16));
    *(data + 17) = Rev16(*(data + 17));
    *(data + 18) = Rev16(*(data + 18));
    *(data + 19) = Rev16(*(data + 19));
    *(data + 20) = Rev16(*(data + 20));
    *(data + 21) = Rev16(*(data + 21));
    *(data + 22) = Rev16(*(data + 22));
    *(data + 23) = Rev16(*(data + 23));
    *(data + 24) = Rev16(*(data + 24));
    *(data + 25) = Rev16(*(data + 25));
    *(data + 26) = Rev16(*(data + 26));
    *(data + 27) = Rev16(*(data + 27));
    *(data + 28) = Rev16(*(data + 28));
    *(data + 29) = Rev16(*(data + 29));
    *(data + 30) = Rev16(*(data + 30));
    *(data + 31) = Rev16(*(data + 31));
    data += 32;
  }
}

jens6151 0 1 1

@jens6151-0-1-1
There seems to be an issue with the loop that some data is skipped. Need to check that maybe later.

yokonav

@jens6151-0-1-1 said in Slow refresh rate streaming the camera preview to a display:

There seems to be an issue with the loop that some data is skipped. Need to check that maybe later.

Thanks for your time! By the way I was checking the function call imageproc_convert_yuv2rgb which seems using the hardware to convert YUV to RGB. They are setting some register value ROT_RGB_ALIGNMENT which looks like the order but I am not sure. If we can directly output BGR instead of RGB, would it be same as colors swapping?

jens6151 0 1 1

@yokonav
I do not think that ROT_RGB_ALIGNMENT is related to the byte order we need.
I assume it is for the RGB 888 or BGR 888 format. (1 byte per color)
We have 5-6-5 bit for the color channels. That makes up 2 bytes or 16 bit.
Byte order means that 1st byte (bit 0-7) and 2nd byte (8-15) is exchanged for a 2 byte number.

yokonav

@jens6151-0-1-1
I think it is RGB565. See the code snippet below from the Arduino SDK.

CamErr CamImage::convertPixFormat(CAM_IMAGE_PIX_FMT to_fmt)
{
  CAM_IMAGE_PIX_FMT from_fmt = getPixFormat();
  int               width    = getWidth();
  int               height   = getHeight();
  uint8_t           *buff    = getImgBuff();

  if (buff == NULL)
    {
      return CAM_ERR_NOT_PERMITTED;
    }

  switch (from_fmt)
    {
      case CAM_IMAGE_PIX_FMT_YUV422:
        switch (to_fmt)
          {
            case CAM_IMAGE_PIX_FMT_RGB565:
              imageproc_convert_yuv2rgb(buff, width, height);

The code converts YUV422 (16bit) input to RGB565 (16bit) output and overwrites the input buffer with the output by the HW.

jens6151 0 1 1

@yokonav
I tried to change this line to use 1 or 2. But I do not see any difference. I hope I did not make any mistake. I recompiled, replaced the sdk, cleaned by deleting all intemerdiate files ...

  putreg32(0, ROT_RGB_ALIGNMENT);

Without any documentation difficult to say. Even the https://www.sony-semicon.co.jp/products/common/pdf/CXD5602_user_manual.pdf did not mention the registers.
@CamilaSouza Is it possible that the 2D Graphics hardware acceleration returns "display byte order"?

jens6151 0 1 1

@yokonav
Just had the idea to look up the display handling inside the Spresense SDK.

This looks promising but might not be supported.
spresense-sdk/spresense/nuttx/include/nuttx/spi/spi.h

#ifdef CONFIG_SPI_HWFEATURES
  /* If there are multiple SPI drivers, some may not support hardware
   * feature selection.
   */

...
   *   Bit 4: HWFEAT_LSBFIRST
   *          Data transferred LSB first (default is MSB first)
   */
...

Need to continue to investigate on the sdk samples later.

yokonav

@jens6151-0-1-1

This should be working.

putreg32(1, ROT_RGB_ALIGNMENT);

@jens6151-0-1-1

There is an entry for the register in the https://github.com/sonydevworld/spresense/blob/master/sdk/tools/SVD/rot.xml file.

<register>
      <name>RGB_ALIGNMENT</name>
      <description>RGB format selector</description>
      <addressOffset>0x38</addressOffset>
      <access>read-write</access>
      <resetValue>0x00000000</resetValue>
      <resetMask>0x00000001</resetMask>

      <fields>
        <field>
          <name>FORMAT</name>
          <description>RGB Format</description>
          <bitRange>[0:0]</bitRange>
          <enumeratedValues>
            <enumeratedValue><name>RGB</name><value>0</value></enumeratedValue>
            <enumeratedValue><name>BGR</name><value>1</value></enumeratedValue>
          </enumeratedValues>
        </field>
      </fields>
    </register>

jens6151 0 1 1

@yokonav
It seems like imageproc_convert_yuv2rgb is not called. It is only called if you convert formats, but not called if you set the desired format in the first place.
I tried forcing a call by converting formats. Unfortunately it was slow. I assume the asm optimized conversion is the best trade off for now.

jens6151 0 1 1

@yokonav
I used the imageproc for rotation and found out that it automatically converts byte order, even if you set to no change. This was my code with rotation.
It took about 1-2ms.

  putreg32(1, ROT_INTR_ENABLE);
  putreg32(0, ROT_INTR_DISABLE);

  putreg32(hsize, ROT_SET_SRC_HSIZE);
  putreg32(vsize, ROT_SET_SRC_VSIZE);
  putreg32(CXD56_PHYSADDR(ibuf), ROT_SET_SRC_ADDRESS);

  putreg32(hsize, ROT_SET_SRC_PITCH);
  putreg32(CXD56_PHYSADDR(obuf), ROT_SET_DST_ADDRESS);

  putreg32(hsize, ROT_SET_DST_PITCH);

  putreg32(0, ROT_CONV_CTRL);     // 0:NOCONVERT 1:YCBCR422_RGB565 2:RGB565_YCBCR422
  putreg32(0, ROT_RGB_ALIGNMENT); // 0:RGB; 1:BGR
  putreg32(2, ROT_SET_DIRECTION); // No rotation  Right 90 degrees  Right 180 degrees   Right 270 degrees
  putreg32(1, ROT_COMMAND);       // start rotation

jens6151 0 1 1

short addition.
Input and Output is different in my case.

Upcoming maintenance

Slow refresh rate streaming the camera preview to a display