[OpenR8 solution] Image_PCB_RestNetSSD512_Caffe (Image analysis uses SSD 512 algorithm and Caffe framework for PCB object detection)
  1. Image_PCB_ResNetSSD512_Caffe

This Image_PCB_ResNetSSD512_Caffe uses the Caffe framework, which is more than twice as fast as Keras. Unlike other Image_PCB_SSD512_Caffe, it uses the resNet network architecture, and uses the SSD (Single Shot MultiBox Detector) deep learning algorithm to train the model. A good model detects the capacitance on the PCB. The size of this training picture is 512 × 512.

 

The main process is shown in Fig. 1.

 

First, we need to prepare the image that we want the model to learn, label the bounding box in the image. The purpose is to let the model know what the object in the image belongs to.

 

Then, through the flow file in OpenR8 Image_PCB_ResNetSSD512_Caffe, two txt list files are automatically generated to let the model know which files are for training and testing, and which categories are divided into these files, after doing these pre-processing, perform the training flow file, after the training is completed, inference has been trained model.

 

Fig. 1. process.png

Fig. 1. process

 

 

  1. Step1: Pre-processing - bounding box

 

Fig. 1. 1process.png

 

  1. Purpose:

Prepare the bounding box to mark the area of interest.

 

  1. Introduction:

Frame the objects you want the model to learn in the image. For example, if you want the model to learn the capacitor, prepare the image of the capacitor, and then select the capacitor in the image and mark it as a capacitor.

 

In this way, the model will know that the bounding box of the area by this image is the capacitor. By analogy, if you want the model to learn something, you are ready to mark the file with the image and frame selected.

 

So the file we need to prepare contains the following two:

(1) Image (images that want the model to learn, images that want to test the accuracy of the model)

 

(2) xml file archive that labels the target location of the image (After the label is completed, the file will be automatically generated)

  1. Example:

In this Image_PCB_ResNetSSD512_Caffe solution example:

 

To detect the capacitance in the PCB, we mark the target capacitor in the PCB image by labelImg.exe (Fig. 2). The red box in Fig. 3 is the bounding box capacitor and the category labeled capacitor. After labeling, the xml file will be automatically generated when it is saved.

 

Place the image in the specified folder location. As shown in Fig. 4,Fig. 5.

【training】

Please place the image in “solution/Image_PCB_ResNetSSD512_Caffe/data/train_image”

 

Please place the xml in “solution/Image_PCB_ResNetSSD512_Caffe/data/train_annotation”

 

【test】

Please place the image in “solution/Image_PCB_ResNetSSD512_Caffe/data/test_image”

 

Please place the image in solution/Image_PCB_ResNetSSD512_Caffe/data/test_annotation

 

  1. Additional instructions:

 

The labelImg.exe has been attached to the OpenR8 file with the file path openR8 > solution > Image_PCB_ResNetSSD512_Caffe > labelImg.exe, as shown in Fig. 2. How to use it can refer to the Open Robot Club https://tw.openrobot.club/article/index?sn=10976.

 

Fig. 2. Location of the label tool.png

Fig. 2. Location of the label tool.

 

Fig. 3. Marking the target object in the image.png

Fig. 3. Marking the target object in the image.

 

Fig. 4. Labeled image and xml file placement location - training.png

Fig. 4. Labeled image and xml file placement location - training.

 

Fig. 5. Labeled image and xml file placement location - test.png

Fig. 5. Labeled image and xml file placement location - test.

 

 

  1. Step2: Pre-processing - create a txt file for the file location

 

Fig. 5.1 Labeled image and xml file placement location - test.png

 

  1. Purpose:

Define the image of [training] and [test] and the location of their label.

 

  1. Instructions for use:

【training】train.txt

step1:Run R8_Python3.6_CPU.bat or R8_Python3.6_GPU.bat.

 

If you want to use GPU acceleration, you can execute R8_Python3.6_GPU.bat. If you use CPU acceleration, you can execute R8_Python3.6_CPU.bat.

 

Fig. 6. Run R8 software.png

Fig. 6. Run R8 software.

 

step2: Choosing and opening a solution.

 

Fig. 7. Open the 1_prepare_train_txt.py solution.png

Fig. 7. Open the “1_prepare_train_txt.py” solution.

 

step3: Execute the solution to produce train.txt.

 

Fig. 8. Execute the 1_prepare_train_txt.py solution.png

Fig. 8. Execute the “1_prepare_train_txt.py” solution.

 

Fig. 9. produce train.png

Fig. 9. produce train.txt.

 

【test】

step1: Run R8_Python3.6_CPU.bat or R8_Python3.6_GPU.bat, as shown in Fig. 6.

If you want to use GPU acceleration, you can execute R8_Python3.6_GPU.bat. If you use CPU acceleration, you can execute R8_Python3.6_CPU.bat.

 

step2: Choosing and opening the solution, as shown in Fig. 10.

 

Fig. 10. Choosing the 2_prepare_test_txt.py solution.png

Fig. 10. Choosing the “2_prepare_test_txt.py” solution.

 

step3: Execute the solution to produce test.txt.

 

Fig. 11. Run the solution.png

Fig. 11. Run the solution.

 

Fig. 12. Produce test.png

Fig. 12. Produce test.txt.

 

 

  1. Step3: Pre-processing - Create a label category txt file

 

Fig. 12.1Produce test.png

 

  1. Purpose:

Create a category txt file, which is to tell the model which categories of our data to divide into, so that the model can automatically learn to classify images.

 

  1. Introduction:

Please create a "labelmap.prototxt" file under the OpenR8/solution/Image_PCB_ResNetSSD512_Caffe/data path. The file content format is as follows:

 

item {

name: "none_of_the_above"

label: 0

display_name: "background"

}

item {

name: ""

label: 1

display_name: "Category label name"

}

……

item {

name: "other category label name"

label: n

display_name: "other category label name"

}

 

Please note: The above "name" "display_name" must be marked with the "mark area of interest Mark" The category name and quantity of the xml match. For example: when marking, divided into two categories of cats and dogs, then the above"name" and "display_name" must be the same name as the category labeled Cat and Dog, as shown in Fig. 13.

 

Fig. 13. Examples of labelmap.png

Fig. 13. Examples of labelmap.prototxt.

 

  1. Example:

Labelmap.prototxt in the OpenR8/solution/Image_PCB_ResNetSSD512_Caffe/data folder, can turn on the file by notepad++ or Notepad.As shown in Fig. 14.

 

The contents of the file are shown in Fig. 15, and we can see that we classify the images into none_of_the_above (none of the above) and capacitor (capacitors), of which individual volume labels are numbered 0 and 1.

 

If you want to add a new category yourself, add the category name and label number, as shown in Fig. 15 (right).

 

Fig. 14..png

Fig. 14. “labelmap.prototxt” path.

 

Fig. 15. labelmap content left add category right .png

Fig. 15. “labelmap” content (left), add category (right).

 

 

  1. Step4: Pre-processing - annoset_to_lmdb

 

Fig. 15. 1 labelmap content left add category right .png

 

This step is to convert the txt that has been defined in the previous two steps into a Lmdb file that is required for subsequent training and test of the model.

 

  1. Instructions for use: Open and execute 3_annoset_to_lmdb python file

(1) Please “execute run R8_Python3.6_CPU.bat or R8_Python3.6_GPU.bat” => click “File” => “Open” => find “solution > Image_PCB_ResNetSSD512_Caffe” => select “3_annoset_to_lmdb.py”.As shown in Fig. 16,Fig. 17,Fig. 18.

(2) Please click "Run". When the old Lmdb file exists, if you want to jump out of the deleted file, click “Yes(Y)”, as shown in Fig. 19. After the execution is complete, train_lmdb and Test_lmdb will be generated. As shown in Fig. 20,Fig. 21.

 

For any questions about from turning on the software to the load solution, refer to the "OpenR8 Manual"

 

Fig. 16. R8 software.png

Fig. 16. R8 software.

 

Fig. 17. Open.png

Fig. 17. Open “3_annoset_to_lmdb.py”.

 

Fig. 18. load.png

Fig. 18. load “3_annoset_to_lmdb.py”.

 

Fig. 19. Delete lmdb file.png

Fig. 19. Delete lmdb file.

 

Fig. 20. Execution complete.png

Fig. 20. Execution complete.

 

Fig. 21. Produce train_lmdb and test_lmdb.png

Fig. 21. Produce train_lmdb and test_lmdb.

 

  1. annoset_to_lmdb parameter description

 

Fig. 22. Input file left turn to output file right .png

Fig. 22. Input file (left) turn to output file (right)

 

This is the path to set the input file and the path to the output lmdb. The file of the file is shown in Fig. 22.

 

The following is a description of each process sequence for this python file, from top to bottom:

 

►Caffe_Init: Caffe framework initialization.

 

►File_DeleteFile(Fig. 23): When the old Lmdb folder exists in the folder, you need to remove the old file before you start generating a new lmdb. (For training)  As shown in Fig. 23,the Flow area clicks “File_DeleteDir” to appear in the variable list, the function list appears the parameters of the process, click "dirName" edit, this folder name variable and display in the variable area, this area of "Value", "Data/train_Lmdb/ "is the folder for Lmdb.

 

Fig. 23. File_DeleteDir.png

Fig. 23. File_DeleteDir.

 

►Caffe_ObjectDefect_CreateTrainData(Fig. 24):Produce lmdb File. (For training)

 

Fig. 24. Caffe_ObjectDefect_CreateTrainData.png

Fig. 24. Caffe_ObjectDefect_CreateTrainData.

 

►The following last two "File_DeleteDir" and "Caffe_ObjectDetection_CreateTrainData": In order to produce the lmdb used in the test model, set the way such as the above generated training lmdb.

 

※ Additional Instructions:The “enableShuffle” field in "Caffe_objectdetection_createtraindata", if its value is True, will randomly intermingle the sorting of data, the effect of training will be better, it is recommended to be set to True, the position of the enbleShuffle is as Fig. 25.

 

Fig. 25. enableShuffle position of.png

Fig. 25. enableShuffle position of "Caffe_ObjectDetect_CreateTrainData".

 

 

  1. Step5: training model

 

Fig. 25. 1enableShuffle position of.png

 

The introduction is divided into two parts, the first describes how to start training, and then if you are interested in knowing that each process block can be viewed in the 2nd to describe the details of this process.

 

  1. Instructions for use: Training model.

First, use the “OpenR8 html” open “4_train.py” file to load “4_train.py”, such as Fig. 26,Fig. 27.

 

Next, click "Run" to start the training model, and this step takes a little time to wait for the program to build the model. such as Fig. 28.

 

When this execution is completed, a trained model will be generated, and its model file generation will be placed, as shown in Fig. 29.

 

Path: OpenR8/solution/Image_PCB_ResNetSSD512_Caffe/data/

File name 1:xxx.Caffemodel

File name 2:xxx. Solverstate

 

If the appliance does not support the NVIDIA GPU. An error message appears. Please follow the Fig. 30,Fig. 31 operation.

 

Fig. 26..png

Fig. 26. “4_train.py” path.

 

Fig. 27. load 4_train.png

Fig. 27. load 4_train.

 

Fig. 28. Perform 4_train and its execution process.png

Fig. 28. Perform 4_train and its execution process.

 

Fig. 29. The production model is finished after training.png

Fig. 29. The production model is finished after training.

 

Fig. 30. No GPU error message - Training.png

Fig. 30. No GPU error message - Training.

 

Fig. 31. No GPU the way set it up - Training.png

Fig. 31. No GPU the way set it up - Training.

 

  1. 4_train.py introduction to processes and parameters:

 

►Caffe_Init:Caffe framework Initialization.

 

►Caffe_Train:

【CaffeObject】The setting that follows the previous initialization.

【GPU】Whether the device used to train the model needs to use GPU acceleration, if so, change the value to all; Conversely, fill in the blanks. (If the selection is, make sure the appliance has a GPU)

【solverPath】The main parameter. This is the input file “Solver.prototxt” required for Caffe training, which contains the number of training iterations, training parameters, and a list of samples. The following Fig. 32,Fig. 33.

【continueTrainModelPath】Continue training with pretrained models. Fill in the value of the model path that you want to train, and the model extension must be .caffemodel.

 

Fig. 32. 4_train.py - Caffe_Train.png

Fig. 32. 4_train.py - Caffe_Train.

 

Fig. 33. solver.prototxt content.png

Fig. 33. solver.prototxt content.

 

 

  1. Step6: Testing models for each training phase

 

  1. Purpose: To test the accuracy of each stage model, to find out the appropriate model, to avoid the training of excessive preparation of training samples, resulting in poor test sample effect.

 

  1. Instructions for use:

(1) Training samples: “Open OpenR8” => “Open and load 5_train_result.py” =>  “execute py” => “produce train_result.txt”

(2) Test samples: “Open OpenR8” => “Open and load 6_test_result.py” =>  “execute py” => “produce test_result.txt”

(3) Choose the appropriate training model: As with Fig. 34, two content from train_result.txt and test_result.txt can be observed, the results of the model at a certain training frequency stage are acceptable or have reached convergence after a certain stage.

 

Therefore, it is more appropriate for us to choose the model of this number.

 

Fig. 34. train_result.txt and test_result.txt content.png

Fig. 34. train_result.txt and test_result.txt content.

 

 

  1. Step7: Inference using a trained model - inference

 

Fig. 34. 1train_result.txt and test_result.txt content.png

 

When the model has been trained, you can use this step to make inferences using a trained model. First use OpenR8 to open "7_inference.py" and load the file.  such as Fig. 35,Fig. 36.

 

  1. Instructions for use:

 

►Select the image you want to test. The following Fig. 37.

 

►Select the model you have trained. Such as Fig. 38.

 

►View the result message. Press execute to detect the coordinates of the target in the object and its type through the trained model, as shown in Fig. 39.

 

►View the results messages and images. If you want to see the results box detected above the image, press "Debug" to display the results of object detection. PCB images are detected with capacitors and their bounding boxes, such as Fig. 40.

 

If the appliance does not support the NVIDIA GPU. An error message appears. Please follow the Fig. 41,Fig. 42 operation.

 

Fig. 35. 7_inference.png

Fig. 35. 7_inference.

 

Fig. 36. load 7_inference.png

Fig. 36. load 7_inference.

 

Fig. 37. 7_inference - Enter the test image file path.png

Fig. 37. 7_inference - Enter the test image file path.

 

Fig. 38. 7_inference - Select the model you want to make inferences.png

Fig. 38. 7_inference - Select the model you want to make inferences.

 

Fig. 39. 7_inference - Result of execution.png

Fig. 39. 7_inference - Result of execution.

 

Fig. 40. 7_inference debug result.png

Fig. 40. 7_inference debug result.

 

Fig. 41. Error message - not GPU.png

Fig. 41. Error message - not GPU.

 

Fig. 42. not NVIDIA GPU setting method.png

Fig. 42. not NVIDIA GPU setting method.

 

  1. Introduction to the Content

 

►Caffe_Init:Caffe framework initialization.

 

►Image_Open:

“imageFileName (String)”: To infer the image path of the test.

“image (Image)”: After reading the image to be inferred according to the image file case name above, save this variable.

 

►Caffe_ObjectDetect_ReadNet

“CaffeObject (Object)”

“GPU (Int)”: Whether to open GPU Accelerated.

“deployPath (String)”: Read the network structure model.

“data/deploy.prototxt”

“caffeModelPath (String)”: Read into a trained model (Extension is caffemodel )

“labelPath (String)”: Read in use labelImg.exe The category file that is used when marking.

“data/predefined_classes.txt”

“meanFilePath (Sting)”

 

►Caffe_ObjectDetect_InferenceImage

“CaffeObject (Object)”

“OutputResult (Json)”: The results after the image is deduced are stored in the message to this variable.

“GPU (Int)”: Whether to open GPU Accelerated.

“image (Image)”: An image to infer.

“ConfidenceThreshold (Float)”

 

►Json_Print

“json (Json)”: Show Caffe_ObjectDetect_InferenceImage the test results of the message.

 

►Image_DrawRectJson

“image (Image)”: An image to infer.

“json (Json)”: Inference of the result message.

“imageDrawRectJson (Image)” Draw the inferred result message to the image to be inferred.

 

►Debug_Image “image (Image)”: Displays an image of a message that has been drawn on the inference result.

“displayPercentage (Int)”: Displays the percentage of the graph. Value is 200 Said 200 % (The picture is magnified a few times), 50 Said 50 % (The picture narrows in half), and so on, zoom out and shrink.

 

 

  1. Step8: Inference - 8_inference_webcam

 

  1. Instructions for use:

When the model has been trained, it can also be done through the 8_inference_webcam.py File for Webcam Real-time image detection and identification of target objects.

 

Start "8_inference_webcam" with OpenR8 first. Load the file. Check that the "Opencv_VideoCapture_Open" Webcam "Device Code" (The notebook webcam defaults to 0, the added webcam is 1, and so on). Press Run. The following Fig. 43,Fig. 44,Fig. 45,Fig. 46.

 

If the appliance does not support the NVIDIA GPU. An error message appears. Please take the Fig. 41,Fig. 42 operation.

 

Fig. 43. Choose 8_inference_webcam.png

Fig. 43. Choose 8_inference_webcam.

 

Fig. 44. Load 8_inference_webcam.png

Fig. 44. Load 8_inference_webcam.

 

Fig. 45. 8_inference_webcam check webcam device code.png

Fig. 45. 8_inference_webcam check webcam device code.

 

Fig. 46. Run 8_inference_webcam.png

Fig. 46. Run 8_inference_webcam.

 

After execution, the webcam camera is turned on for continuous detection and identification, and the capture image is not stopped until the “Enter” key is pressed.


Recommended Article

1.
OpenR8 Community Edition - AI Software for Everyone (Free Download)