[OpenR8 solution] Image_PCB_RestNetSSD512_Caffe (Image analysis uses SSD 512 algorithm and Caffe framework for PCB object detection)
  1. Image_PCB_ResNetSSD512_Caffe

This Image_PCB_ResNetSSD512_Caffe uses the Caffe framework, which is more than twice as fast as Keras. Unlike other Image_PCB_SSD512_Caffe, it uses the resNet network architecture, and uses the SSD (Single Shot MultiBox Detector) deep learning algorithm to train the model. A good model detects the capacitance on the PCB. The size of this training picture is 512 × 512.

 

The main process is shown in Fig. 1.

 

First, we need to prepare the image that we want the model to learn, label the bounding box in the image. The purpose is to let the model know what the object in the image belongs to.

 

Then, through the flow file in OpenR8 Image_PCB_ResNetSSD512_Caffe, two txt list files are automatically generated to let the model know which files are for training and testing, and which categories are divided into these files, after doing these pre-processing, perform the training flow file, after the training is completed, inference has been trained model.

 

Fig. 1. process.png

Fig. 1. process

 

 

  1. Step1: Pre-processing - bounding box

 

Fig. 1.1 process.png

 

  1. Purpose:

Prepare the bounding box to mark the area of interest.

 

  1. Introduction:

Frame the objects you want the model to learn in the image. For example, if you want the model to learn the capacitor, prepare the image of the capacitor, and then select the capacitor in the image and mark it as a capacitor.

 

In this way, the model will know that the bounding box of the area by this image is the capacitor. By analogy, if you want the model to learn something, you are ready to mark the file with the image and frame selected.

 

So the file we need to prepare contains the following two:

(1) Image (images that want the model to learn, images that want to test the accuracy of the model)

 

(2) xml file archive that labels the target location of the image (After the label is completed, the file will be automatically generated)

  1. Example:

In this Image_PCB_ResNetSSD512_Caffe solution example:

 

To detect the capacitance in the PCB, we mark the target capacitor in the PCB image by labelImg.exe (Fig. 2). The red box in Fig. 3 is the bounding box capacitor and the category labeled capacitor. After labeling, the xml file will be automatically generated when it is saved.

 

Place the image in the specified folder location. As shown in Fig. 4,Fig. 5.

【training】

Please place the image in “solution/Image_PCB_ResNetSSD512_Caffe/data/train_image”

 

Please place the xml in “solution/Image_PCB_ResNetSSD512_Caffe/data/train_annotation”

 

【test】 Please place the image in “solution/Image_PCB_ResNetSSD512_Caffe/data/test_image”

 

Please place the image in solution/Image_PCB_ResNetSSD512_Caffe/data/test_annotation

 

  1. Additional instructions:

 

The labelImg.exe has been attached to the OpenR8 file with the file path openR8 > solution > Image_PCB_ResNetSSD512_Caffe > labelImg.exe, as shown in Fig. 2. How to use it can refer to the Open Robot Club https://tw.openrobot.club/article/index?sn=10976.

 

Fig. 2. Location of the label tool.png

Fig. 2. Location of the label tool.

 

Fig. 3. Marking the target object in the image.png

Fig. 3. Marking the target object in the image.

 

Fig. 4. Labeled image and xml file placement location - training.png

Fig. 4. Labeled image and xml file placement location - training.

 

Fig. 5. Labeled image and xml file placement location - test.png

Fig. 5. Labeled image and xml file placement location - test.

 

 

  1. Step2: Pre-processing - create a txt file for the file location

 

Fig. 5. 1Labeled image and xml file placement location - test.png

  1. Purpose:

Define the image of [training] and [test] and the location of their label.

  1. Instructions for use:

【training】train.txt

step1:Execute R8.exe.

 

Fig. 6. Execute R8..png

Fig. 6. Execute R8.exe

 

step2: Choosing and opening a solution.

 

Fig. 7. Open the 1_prepare_train_txt.flow solution.png

Fig. 7. Open the“1_prepare_train_txt.flow” solution.

 

step3: Execute the solution to produce train.txt.

 

Fig. 8. Execute the 1_prepare_train_txt.flow solution.png

Fig. 8. Execute the “1_prepare_train_txt.flow” solution

 

Fig. 9. produce train.png

Fig. 9. produce train.txt.

 

【test】

step1: Execution R8.exe, as shown in Fig. 6.

step2: Choosing and opening the solution, as shown in Fig. 10.

 

Fig. 10. Choosing the 2_prepare_test_txt.flow solution.png

Fig. 10. Choosing the “2_prepare_test_txt.flow” solution.

 

step3: Execute the solution to produce test.txt.

 

Fig. 11. Execute the solution.png

Fig. 11. Execute the solution.

 

Fig. 12. Produce test.png

Fig. 12. Produce test.txt.

 

 

  1. Step3: Pre-processing - Create a label category txt file

 

Fig. 12. 1Produce test.png

 

  1. Purpose:

Create a category txt file, which is to tell the model which categories of our data to divide into, so that the model can automatically learn to classify images.

  1. Introduction:

Please create a "labelmap.prototxt" file under the

OpenR8/solution/Image_PCB_ResNetSSD512_Caffe/data path. The file content format is as follows:

item {

name: "none_of_the_above"

label: 0

display_name: "background"

}

item {

name: "Category label name"

label: 1

display_name: "Category label name"

}

……

item {

name: "other category label name"

label: n

display_name: "other category label name"

}

Please note: The above "name" "display_name" must be marked with the     "mark area of interest Mark" The category name and quantity of the xml     match.     For example: when marking, divided into two categories of cats and dogs, then     the above"name" and "display_name" must be the same name as the category     labeled Cat and Dog, as shown in Fig. 13。

 

Fig. 13. Examples of labelmap.prototxt.png

Fig. 13. Examples of labelmap.prototxt.

 

  1. Example:

Labelmap.prototxt in the OpenR8/solution/Image_PCB_ResNetSSD512_Caffe/data folder, can turn on the file by notepad++ or Notepad.As shown in Fig. 14.

The contents of the file are shown in Fig. 15, and we can see that we classify the images into none_of_the_above (none of the above) and capacitor (capacitors), of which individual volume labels are numbered 0 and 1.

If you want to add a new category yourself, add the category name and label number, as shown in Fig. 15 (right).

 

Fig. 14. labelmap.prototxt path.png

Fig. 14. “labelmap.prototxt” path

 

Fig. 15. labelmap content left add category right .png

Fig. 15. “labelmap” content (left), add category (right).

 

 

  1. Step4: Pre-processing - annoset_to_lmdb

 

Fig. 15. 1 labelmap content left add category right .png

This step is to convert the txt that has been defined in the previous two steps into a Lmdb file that is required for subsequent training and test of the model.

 

  1. Instructions for use: Open and execute 3_annoset_to_lmdb flow files

(1) Please “execute R8.exe” => click “File” => “Open” => find “solution > Image_PCB_ResNetSSD512_Caffe” => selete “3_annoset_to_lmdb.flow”.As shown in Fig. 16,Fig. 17,Fig. 18.

(2) Please click "Run". When the old Lmdb file exists, if you want to jump out of the deleted file, click “Yes(Y)”, as shown in Fig. 19. After the execution is complete, train_lmdb and Test_lmdb will be generated. As shown in Fig. 20,Fig. 21.

For any questions about from turning on the software to the load solution, refer to the "OpenR8 Manual.doc"

 

Fig. 16. R8.png

Fig. 16. R8.exe.

 

Fig. 17. Open 3_annoset_to_lmdb.flow .png

Fig. 17. Open “3_annoset_to_lmdb.flow”

 

Fig. 18. load 3_annoset_to_lmdb.flow .png

Fig. 18. load “3_annoset_to_lmdb.flow”

 

Fig. 19. Delete lmdb file.png

Fig. 19. Delete lmdb file.

 

Fig. 20. Execution complete.png

Fig. 20. Execution complete.

 

Fig. 21. Produce train_lmdb and test_lmdb.png

Fig. 21. Produce train_lmdb and test_lmdb.

 

  1. annoset_to_lmdb parameter description

 

Fig. 22. Input file left turn to output file right .png

Fig. 22. Input archive(left)Convert to output file(right)

 

This is the path to set the input file and the path to the output lmdb. The file of the file is shown in Fig. 22.

The following is a description of each process sequence for this flow file, from top to bottom:

Caffe_Init: Caffe framework initialization.

File_DeleteFile(Fig. 23): When the old Lmdb folder exists in the folder, you need to remove the old file before you start generating a new lmdb. (For training)

As shown in Fig. 23,the Flow area clicks “File_DeleteDir” to appear in the variable list, the function list appears the parameters of the process, click "dirName" edit, this folder name variable and display in the variable area, this area of "Value", "Datatrain_Lmdb "is the folder for Lmdb.

 

Fig. 23. File_DeleteDir.png

Fig. 23. File_DeleteDir.

 

Caffe_ObjectDefect_CreateTrainData(Fig. 24):Produce lmdb File. (For training)

 

Fig. 24. Caffe_ObjectDefect_CreateTrainData.png

Fig. 24. Caffe_ObjectDefect_CreateTrainData.

 

The following last two "File_DeleteDir" and

"Caffe_ObjectDetection_CreateTrainData":In order to produce the lmdb used in the test model, set the way such as the above generated training lmdb.

Additional Instructions

The “enableShuffle” field in "Caffe_objectdetection_createtraindata", if its value is True, will randomly disrupt the sorting of data, the effect of training will be better, it is recommended to be set to True,

The position of the enbleShuffle is as Fig. 25.

 

Fig. 25..png

Fig. 25. enableShuffle position of "Caffe_ObjectDetect_CreateTrainData"

 

 

  1. Step5: training model

 

Fig. 25.1.png

 

The introduction is divided into two parts, the first describes how to start training, and then if you are interested in knowing that each process block can be viewed in the 2nd to describe the details of this process.

  1. Instructions for use: Training model.

First, use the “OpenR8” open “4_train.flow” file to load “4_train.flow”, such as Fig. 26,Fig. 27。

Next, click "Run" to start the training model, and this step takes a little time to wait for the program to build the model. such as Fig. 28.

When this execution is completed, a trained model will be generated, and its model file generation will be placed, as shown in Fig. 29.

Path: OpenR8/solution/Image_PCB_ResNetSSD512_Caffe/data/

File name 1:xxx.Caffemodel

File name 2:xxx. Solverstate

If the appliance does not support the NVIDIA GPU. An error message appears.Please follow the Fig. 30,Fig. 31 operation.

 

Fig. 26. 4_train.flow path.png

Fig. 26. “4_train.flow” path.

 

Fig. 27. load 4_train.png

Fig. 27. load 4_train.

 

Fig. 28. Perform 4_train and its execution process.png

Fig. 28. Perform 4_train and its execution process diagram.

 

Fig. 29. The production model is finished after training.png

Fig. 29. The production model is finished after training.

 

Fig. 30. No GPU error message - Training.png

Fig. 30. No GPU error message - Training.

 

Fig. 31. No GPU the way set it up - Training.png

Fig. 31. No GPU the way set it up - Training.

 

  1. 4_train.flow introduction to processes and parameters:

Caffe_Init:Caffe framework Initialization.

 Caffe_Train:

【CaffeObject】The setting that follows the previous initialization.

【GPU】Whether the device used to train the model needs to use GPU acceleration, if so, change the value to all; Conversely, fill in the blanks. (If the selection is, make sure the appliance has a GPU)

【solverPath】The main parameter. This is the input file “Solver.prototxt” required for Caffe training, which contains the number of training iterations, training parameters, and a list of samples. The following Fig. 32,Fig. 33.

【continueTrainModelPath】Continue training with pretrained models. Fill in the value of the model path that you want to train, and the model extension must be .caffemodel.

 

Fig. 32. 4_train.flow - Caffe_Train.png

Fig. 32. 4_train.flow - Caffe_Train.

 

Fig. 33. solver.prototxt content.png

Fig. 33. solver.prototxt content.

 

 

  1. Step6: Testing models for each training phase

 

  1. Purpose: To test the accuracy of each stage model, to find out the appropriate model, to avoid the training of excessive preparation of training samples, resulting in poor test sample effect.
  2. Instructions for use:

(1) Training samples: “Open OpenR8” => “Open and load 5_train_result.flow” =>  “execute flow” => “produce train_result.txt”

(2) Test samples: “Open OpenR8” => “Open and load 6_test_result.flow” =>  “execute flow” => “produce test_result.txt”

(3) Choose the appropriate training model: As with Fig. 34, two content from train_result.txt and test_result.txt can be observed, the results of the model at a certain training frequency stage are acceptable, or have reached convergence after a certain stage.

Therefore, it is more appropriate for us to choose the model of this number.

 

Fig. 34. train_result.txt and test_result.txt content.png

Fig. 34. train_result.txt and test_result.txt content.

 

 

  1. Step7: Inference using a trained model - inference

 

Fig. 34. 1train_result.txt and test_result.txt content.png

 

When the model has been trained, you can use this step to make inferences using a trained model. First use OpenR8 to open "7_inference.flow" and load the file.  such as Fig. 35,Fig. 36.

  1. Instructions for use:

►Select the image you want to test. The following Fig. 37.

►Select the model you have trained. Such as Fig. 38.

►View the result message. Press execute to detect the coordinates of the target in the object and its type through the trained model, as shown in Fig. 39.

►View the results messages and images. If you want to see the results box detected above the image, press "Debug" to display the results of object detection. PCB images are detected with capacitors and their bounding boxes, such as Fig. 40.

If the appliance does not support the NVIDIA GPU. An error message appears. Please follow the Fig. 41,Fig. 42 operation.

 

Fig. 35. 7_inference.png

Fig. 35. 7_inference.

 

Fig. 36. load 7_inference.png

Fig. 36. load 7_inference.

 

Fig. 37. 7_inference - Choosing a test image.png

Fig. 37. 7_inference - Choosing a test image.

 

Fig. 38. 7_inference - Select the model you want to make inferences.png

Fig. 38. 7_inference - Select the model you want to make inferences.

 

Fig. 39. 7_inference - Result of execution.png

Fig. 39. 7_inference - Result of execution.

 

Fig. 40. 7_inference debug result.png

Fig. 40. 7_inference debug result.

 

Fig. 41. Error message - not GPU.png

Fig. 41. Error message - not GPU.

 

Fig. 42. not NVIDIA GPU setting method.png

Fig. 42. not NVIDIA GPU setting method.

 

  1. Introduction to the Content 

►Caffe_Init:Caffe framework initialization。

► Image_Open:

“imageFileName (String)”: To infer the image path of the test.

“image (Image)”: After reading the image to be inferred according to the image file case name above, save this variable.

►Caffe_ObjectDetect_ReadNet

“CaffeObject (Object)”

“GPU (Int)”: Whether to open GPU Accelerated.

“deployPath (String)”: Read the network structure model. “data/deploy.prototxt”

“caffeModelPath (String)”: Read into a trained model (Extension is caffemodel )

“labelPath (String)”: Read in use labelImg.exe The category file that is used when marking. “data/predefined_classes.txt”

“meanFilePath (Sting)”

►Caffe_ObjectDetect_InferenceImage

“CaffeObject (Object)”

“OutputResult (Json)”: The results after the image is deduced are stored in the message to this variable.

“GPU (Int)”: Whether to open GPU Accelerated.

“image (Image)”: An image to infer.

“ConfidenceThreshold (Float)”

► Json_Print

“json (Json)”: Show Caffe_ObjectDetect_InferenceImage the test results of the message.

► Image_DrawRectJson

“image (Image)”: An image to infer.

“json (Json)”: Inference of the result message.

“imageDrawRectJson (Image) 】Draw the inferred result message to the image to be inferred.

► Debug_Image

“image (Image)”: Displays an image of a message that has been drawn on the inference result.

“displayPercentage (Int)”: Displays the percentage of the graph. Value is 200 Said 200 % (The picture is magnified a few times), 50 Said 50 % (The picture narrows in half), and so on, zoom out and shrink.

 

 

  1. Step8: Inference - 8_inference_webcam

 

  1. Instructions for use:

When the model has been trained, it can also be done through the 8_inference_webcam.flow File for Webcam Real-time image detection and identification of target objects.

 

Start "8_inference_webcam" with OpenR8 first. Load the file. Check that the "Opencv_VideoCapture_Open" Webcam "Device Code" (The notebook webcam defaults to 0, the added webcam is 1, and so on). Press Run. The following Fig. 43,Fig. 44,Fig. 45,Fig. 46.

 

If the appliance does not support the NVIDIA GPU. An error message appears. Please take the Fig. 41,Fig. 42 operation.

 

Fig. 43. Choose 8_inference_webcam.png

Fig. 43. Choose 8_inference_webcam.

 

Fig. 44. Load 8_inference_webcam.png

Fig. 44. Load 8_inference_webcam.

 

Fig. 45. 8_inference_webcam check webcam device code.png

Fig. 45. 8_inference_webcam check webcam device code.

 

Fig. 46. Run 8_inference_webcam.png

Fig. 46. Run 8_inference_webcam.

 

After execution, the webcam camera is turned on for continuous detection and identification, and the capture image is not stopped until the “Enter” key is pressed.


Recommended Article

1.
OpenR8 - AI Software for Everyone (Download)