I tried to implement a learning model for image recognition created by the Neural Network Console (NNC) in spresense.

Thank you for visiting our site.

In conclusion, the jpg image cannot be used as it is for entering the DNNRT of the Presense.

NNC decodes jpg data into pixel data and enters it into a neural network, so it seems to be able to handle jpg as well. (I have never used jpg for learning data, so I guess.)

The reason why I can say that with confidence is obvious at a glance by looking at the definition of the "input layer" of a neural network.

"If the parameter of ""input layer"" is ""1,28,28"", it means to enter the matrix of (1,28,28)."In other words, you have to deploy it to 28x28 pixel data in black and white to enter it into the network.

For color images, the input layer is defined in 3,28,28.This means entering the (3,28,28) matrix, or 28x28 pixel data in RGB.Therefore, the jpg data must be deployed to RGB images in advance.

In the case of Spresense, it is built-in, so it is not as kind as the NNC.If you enter DNNRT, you must expand the pre-compressed image (and enter data on a pixel-by-pixel basis, between 0.0 and 1.0).

If you want to handle jpg images, you need to include the jpg decoder in your Spresense and deploy the jpg data to pixel data (although it's hard to include).Not recommended)

Use caution when entering additional color images into DNNRT.Color data must be entered into DNNRT in blocks of R, G, and B data.In other words, RGB pixel data needs to be broken down into R data, G data, and B data.You will understand this considering that the matrix structure of the input layer is (3,28,28).

For example, if you're dealing with RGB 565 (16 bits) data on a camera, this is what it looks like.

uint16_t*buf=(uint16_t*)img.getImgBuff();

  DNNVariable input (INPUT_WIDTH*INPUT_HEIGHT*3);  
  float*r=input.data();
  float*g=r+INPUT_WIDTH*INPUT_HEIGHT;
  float*b=g+INPUT_WIDTH*INPUT_HEIGHT;
  for(inti=0;i<INPUT_WIDTH*INPUT_HEIGHT;++i){
   *(r++)=(float)(*buf>>11)&0x1F)/31.0;//0x1F=31
   *(g++)=(float)(*buf>>5)&0x3F)/63.0;//0x3F=64
   *(b++)=(float)(*buf)&0x1F)/31.0;//0x1F=31
   ++buf;
  }
  dnnrt.inputVariable(input,0);
  dnnrt.forward();
  DNNVariable output = dnnrt.outputVariable(0);

Because of this background, when dealing with image files in the Presense, the simple format PNM is said to be good.However, using only PNM is inconvenient for working on a PC.

So I created a library where I could handle BMPs in Spresense.

https://github.com/YoshinoTaro/BmpImage_ArduinoLib

As for this library, due to the structure of the BMP, Spresense can only handle small images because it uses pallets unnecessarily. (Originally, DNNRT only handles small images, and it is memory optimized.)

If necessary, please try using it.However, since it's fresh from the oven, I think there are a lot of implementation problems.Please use it at your own risk.