[OpenR8 solution] Image_PCB_ResNetSSD300_Caffe (Using SSD 300 algorithm and Caffe framework for PCB object detection)
  1. Image_PCB_ResNetSSD300_Caffe

 

Image_PCB_ResNetSSD300_Caffe uses the Caffe framework, which is more than twice as fast as Keras. Unlike other Image_PCB_SSD300_Caffe, it uses the resNet network architecture, and uses the SSD (Single Shot MultiBox Detector) deep learning algorithm to train the model. A good model detects the capacitance on the PCB. The size of this training image is 300 × 300.

 

The main process is shown in Fig. 1.

 

First, we need to prepare the image that we want the model to learn, label the bounding box in the image. The purpose is to let the model know what the object in the image belongs to.

 

Then, through the py file in OpenR8 Image_PCB_ResNetSSD300_Caffe, two txt list files are automatically generated to let the model know which files are for training and testing, and which categories are divided into these files, after doing these pre-processing, perform the training py file, after the training is completed, inference has been trained model.

 

Fig. 1. process.png

Fig. 1. process.

 

 

  1. Step1: Pre-processing - bounding box

 

Fig. 1. 1process.png

 

  1. Purpose:

Prepare the bounding box to label the area of interest.

 

  1. Introduction: In the Image bounding box, label the object that you want the model to learn. For example, you want the model to learn the capacitance to prepare the capacitance image, and then the capacitance box in the image to be labeled as a capacitor. As a result, the model will know that the area of the image is capacitance. And so on, to let the model learn something, prepare the image and bounding box to label the xml file.

 

So the files we need to prepare contain the following two items:

(1). Image (Images that want the model to learn, images that want to infer the accuracy of the model)

(2). xml file archive that labels the target location of the image (After the label is completed, the file will be automatically generated)

 

  1. Example:

In this Image_PCB_ResNetSSD512_Caffe solution example:

To detect the capacitance in the PCB, we mark the target capacitor in the PCB image by labelImg.exe (Fig. 2). The red box in Fig. 3 is the bounding box capacitor and the category labeled capacitor. After labeling, the xml file will be automatically generated when it is saved.

Place the image in the specified folder location. As shown in Fig. 4, Fig. 5.

 

【training】

Please place the image in “solution/Image_PCB_ResNetSSD300_Caffe/data/train_image”

Please place the xml in “solution/Image_PCB_ResNetSSD300_Caffe/data/train_annotation”

 

【test】

Please place the image in “solution/Image_PCB_ResNetSSD300_Caffe/data/test_image”

Please place the xml in “solution/Image_PCB_ResNetSSD300_Caffe/data/test_annotation”

 

  1. Additional instructions:

The labelImg.exe has been attached to the OpenR8 file with the file path openR8 > solution > Image_PCB_ResNetSSD300_Caffe > labelImg.exe, as shown in Fig. 2. How to use it can refer to the Open Robot Club https://tw.openrobot.club/article/index?sn=10976.

 

Fig. 2. Location of the label tool.png

Fig. 2. Location of the label tool.

 

Fig. 3. Labeling the target object in the image.png

Fig. 3. Labeling the target object in the image.

 

Fig. 4. Labeled image and xml file placement location - training.png

Fig. 4. Labeled image and xml file placement location - training.

 

Fig. 5. Labeled image and xml file placement location - test.png

Fig. 5. Labeled image and xml file placement location - test.

 

 

  1. Step2: Pre-processing - create a txt file for the file location

 

Fig. 5.1 Labeled image and xml file placement location - test.png

 

  1. Purpose:

Defines the location of the images for training and test with their labels.

 

  1. Instructions for use:

 

【training】train.txt

step1: Run R8_Python3.6_CPU.bat or R8_Python3.6_GPU.bat.

(If you want to use GPU acceleration, you can execute R8_Python3.6_GPU.bat. If you use CPU acceleration, you can execute R8_Python3.6_CPU.bat.)

 

Fig. 6. Run R8 software.png

Fig. 6. Run R8 software.

 

step2: Select and open solution.

 

Fig. 7. Open the 1_prepare_train_txt.py solution.png

Fig. 7. Open the “1_prepare_train_txt.py” solution.

 

step3: Run solution to produce train.txt.

 

Fig. 8. Run 1_prepare_train_txt.py solution.png

Fig. 8. Run “1_prepare_train_txt.py” solution.

 

Fig. 9. Produce train.png

Fig. 9. Produce train.txt.

 

【test】

step1: Open the OpenR8, run R8 software, as shown in Fig. 6.

step2: Selecting and opening the solution, as shown in Fig. 10.

 

Fig. 10. Selecting the 2_prepare_test_txt.py solution.png

Fig. 10. Selecting the “2_prepare_test_txt.py” solution.

 

step3: Run the solution to produce test.txt.

 

Fig. 11. Run the solution.png

Fig. 11. Run the solution.

 

Fig. 12. 1Produce test.png

Fig. 12. Produce test.txt.

 

 

  1. Step3: Pre-processing - Create a label category txt file

 

Fig. 12. 1Produce test.png

 

  1. Purpose:

Create a category txt file, which is to tell the model which categories of our data to divide into, so that the model can automatically learn to classify images.

 

  1. Introduction:

Please create a "labelmap.prototxt" file under the OpenR8/solution/Image_PCB_ResNetSSD300_Caffe/data path. The file content format is as follows:

 

item {

name: "none_of_the_above"

label: 0

display_name: "background"

}

item {

name: " Category label name "

label: 1

display_name: " Category label name "

}

……

item {

name: " Category label name "

label: n

display_name: " Category label name "

}

 

Please note: The above "name" "display_name" must be marked with the  "mark area of interest Mark" The category name and quantity of the xml match.     For example: when marking, divided into two categories of cats and dogs, then the above"name" and "display_name" must be the same name as the category labeled Cat and Dog, as shown in Fig. 13.

 

  1. Example:

Labelmap.prototxt in the OpenR8/solution/Image_PCB_ResNetSSD512_Caffe/data folder, can turn on the file by notepad++ or Notepad. As shown in Fig. 14.

 

The contents of the file are shown in Fig. 15, and we can see that we classify the images into none_of_the_above (none of the above) and capacitor (capacitors), of which individual volume labels are numbered 0 and 1.

 

If you want to add a new category yourself, add the category name and label number, as shown in Fig. 15 (right).

 

Fig. 14. labelmap.prototxt file path.png

Fig. 14. labelmap.prototxt file path.

 

Fig. 15. labelmap content left add category right .png

Fig. 15. “labelmap” content (left), add category (right).

 

 

  1. Step4: Pre-processing - annoset_to_lmdb

 

Fig. 15.1 labelmap content left add category right .png

 

This step is to convert the txt that has been defined in the previous two steps into a lmdb file that is required for subsequent training and test of the model.

 

  1. Instructions for use:Open and execute annoset_to_lmdb py file

(1) Please“execute R8_Python3.6_CPU.bat or R8_Python3.6_GPU.bat” => click “File” => “Open” => find “solution > Image_PCB_ResNetSSD300_Caffe” => select “3_annoset_to_lmdb.py”. As shown in Fig. 16, Fig. 7, Fig. 18.

(If you want to use GPU acceleration, you can execute R8_Python3.6_GPU.bat. If you use CPU acceleration, you can execute R8_Python3.6_CPU.bat.)

(2) Please click "Run". When the old Lmdb file exists, if you want to jump out of the deleted file, click “Yes(Y)”, as shown in Fig. 19. After the execution is complete, train_lmdb and Test_lmdb will be generated. As shown in Fig. 20, Fig. 21.

 

For any questions about from turning on the software to the load solution, refer to the "OpenR8 Manual.doc"

 

Fig. 16. R8 software.png

Fig. 16. R8 software.

 

Fig. 17. Open annoset_to_lmdb.py.png

Fig. 17. Open annoset_to_lmdb.py.

 

Fig. 18. Load annoset_to_lmdb.png

Fig. 18. Load annoset_to_lmdb.

 

Fig. 19. Delete lmdb files.png

Fig. 19. Delete lmdb files.

 

Fig. 20. Execution complete.png

Fig. 20. Execution complete.

 

Fig. 21. Produce train_lmdb and test_lmdb.png

Fig. 21. Produce train_lmdb and test_lmdb.

 

  1. annoset_to_lmdb parameter description

 

Fig. 22. Input file left turn to output file right .png

Fig. 22. Input file (left) turn to output file (right).

 

This is the path to set the input file and the path to the output lmdb. The content of the file is shown in Fig. 22

The following is a description of each process sequence for this py file, from top to bottom:

 

Caffe_Init: Caffe framework initialization.

 

►File_DeleteFile(Fig. 23): When the old Lmdb folder exists in the folder, you need to remove the old file before you start generating a new lmdb. (For training) As shown in Fig. 23, the Flow area clicks “File_DeleteDir” to appear in the variable list, the function list appears the parameters of the process, click "dirName" edit, this folder name variable and display in the variable area, this area of "Value", "data/train_Lmdb/ "is the folder for Lmdb.

 

Fig. 23. File_DeleteFile.png

Fig. 23. File_DeleteFile.

 

►Caffe_ObjectDetect_CreateTrainData (Fig. 24): Produce lmdb File. (For training)

 

Fig. 24. Caffe_ObjectDetect_CreateTrainData.png

Fig. 24. Caffe_ObjectDetect_CreateTrainData.

 

►The following last two "File_DeleteDir" and "Caffe_ObjectDetection_CreateTrainData": In order to produce the lmdb used in the test model, set the way such as the above generated training lmdb.

 

※ Additional Instructions:

The “enableShuffle” field in "Caffe_objectdetection_createtraindata", if its value is True, will randomly intermingle the sorting of data, the effect of training will be better, it is recommended to be set to True, the position of the enbleShuffle is as Fig. 25.

 

Fig. 25. enableShuffle position of.png

Fig. 25. enableShuffle position of "Caffe_ObjectDetect_CreateTrainData".

 

 

  1. Step5: Training Models - train

 

Fig. 25. 1enableShuffle position of.png

 

The introduction is divided into two parts, the first describes how to start training, and then if you are interested in knowing that each process block can be viewed in the 2nd to describe the details of this process.

 

  1. Instructions for use: Training model.

First, use the “OpenR8” open “4_train.py” file to load “4_train.py”, such as Fig. 26, Fig. 27.

Next, click "Run" to start the training model, and this step takes a little time to wait for the program to build the model. such as Fig. 28.

When this execution is completed, a trained model will be generated, and its model file generation will be placed, as shown in Fig. 29.

Path: OpenR8/solution/Image_PCB_ResNetSSD300_Caffe/data/

File name 1:xxx.Caffemodel

File name 2:xxx.Solverstate

 

If the appliance does not support the NVIDIA GPU. An error message appears. Please follow the Fig. 30, Fig. 31 operation.

 

Fig. 26. 4_train file path.png

Fig. 26. 4_train file path.

 

Fig. 27. Load 4_train.png

Fig. 27. Load 4_train.

 

Fig. 28. Perform 4_train and its execution process.png

Fig. 28. Perform 4_train and its execution process.

 

Fig. 29. The production model is finished after training.png

Fig. 29. The production model is finished after training.

 

Fig. 30. No GPU error message-Training.png

Fig. 30. No GPU error message-Training.

 

Fig. 31. No GPU setup Way-Training.png

Fig. 31. No GPU setup Way-Training.

 

  1. 4_train.py introduction to processes and parameters:

 

►Caffe_Init: Caffe framework initialization.

 

►Caffe_Train:

【CaffeObject】The setting that follows the previous initialization.

【GPU】Whether the device used to train the model needs to use GPU acceleration, if so, change the value to all; Conversely, fill in the blanks. (If the selection is, make sure the appliance has a GPU)

【SolverPath】The main parameter. This is the input file “Solver.prototxt” required for Caffe training, which contains the number of training iterations, training parameters, and a list of samples. The following Fig. 32, Fig. 33.

【continueTrainModelPath】Continue training with pretrained models. Fill in the value of the model path that you want to train, and the model extension must be .caffemodel.

 

Fig. 32. 4_train.py - Caffe_Train.png

Fig. 32. 4_train.py - Caffe_Train.

 

Fig. 33. solver.prototxt content.png

Fig. 33. solver.prototxt content.

 

 

  1. Step6: Testing models for each training phase

 

  1. Purpose: To test the accuracy of each stage model, to find out the appropriate model, to avoid the training of excessive preparation of training samples, resulting in poor test sample effect.

 

  1. Instructions for use:

(1) Training samples: “Open OpenR8” => “Open and load 5_train_result.py” =>  “run py” => “produce train_result.txt”

 

(2) Test samples: “Open OpenR8” => “Open and load 6_test_result.py” =>  “run py” => “produce test_result.txt”

 

(3) Choose the appropriate training model:

As shown in Fig. 34, two content from Train_result.txt and test_result.txt can be observed that the results of the model at a certain training number stage are acceptable or have reached convergence after a certain stage. Therefore, it is more appropriate for us to choose the model of this number.

 

Fig. 34. train_result.txt and test_result.txt content.png

Fig. 34. train_result.txt and test_result.txt content.

 

 

  1. Step7: Inference using a trained model - inference

 

Fig. 34. 1train_result.txt and test_result.txt content.png

 

When the model has been trained, you can use this step to make inferences using a trained model. First use OpenR8 to open "7_inference.py" and load the file.

 

such as Fig. 35, Fig. 36.

  1. Instructions for use:

►Select the image you want to test. The following Fig. 37.

 

►Select the model you have trained. Such as Fig. 38.

 

►View the result message. Press execute to detect the coordinates of the target in the object and its type through the trained model, as shown in Fig. 39.

 

►View the results messages and images. If you want to see the results box detected above the image, press "Debug" to display the results of object detection. PCB images are detected with capacitors and their bounding boxes, such as Fig. 40.

 

If the appliance does not support the NVIDIA GPU. An error message appears. Please follow the Fig. 42 operation.

 

Fig. 35. 7_inference.png

Fig. 35. 7_inference.

 

Fig. 36. Load 7_inference.png

Fig. 36. Load 7_inference.

 

Fig. 37. 7_inference - Choosing a test image.png

Fig. 37. 7_inference - Choosing a test image.

 

Fig. 38. 7_inference - Select the model you want to infer.png

Fig. 38. 7_inference - Select the model you want to infer.

 

Fig. 39. 7_inference - Result of execution.png

Fig. 39. 7_inference - Result of execution.

 

Fig. 40. 7_inference debug result.png

Fig. 40. 7_inference debug result.

 

Fig. 41. Error message - not GPU.png

Fig. 41. Error message - not GPU.

 

Fig. 42. Not NVIDIA GPU setting method.png

Fig. 42. Not NVIDIA GPU setting method.

 

  1. Introduction to the Content

►Caffe_Init:Caffe framework initialization.

 

►Image_Open:

“imageFileName (String)”: To infer the image path of the test.

“image (Image)”: After reading the image to be inferred according to the image file case name above, save this variable.

 

►Caffe_ObjectDetect_ReadNet

“CaffeObject (Object)”

“GPU (Int)”: Whether to use GPU accelerated.

“deployPath (String)”: Read the network structure model.

“data/deploy.prototxt”

“caffeModelPath (String)”: Read into a trained model (Extension is caffemodel )

“labelPath (String)”: Read in use labelImg.exe The category file that is used when      marking.

“data/predefined_classes.txt”

“meanFilePath (Sting)”

 

►Caffe_ObjectDetect_InferenceImage

“CaffeObject (Object)”

“OutputResult (Json)”: The results after the image is deduced are stored in the message to this variable.

“GPU (Int)”: Whether to use GPU Accelerated.

“image (Image)”: An image to infer.

“ConfidenceThreshold (Float)”

 

►Json_Print

“json (Json)”: Show Caffe_ObjectDetect_InferenceImage the test results of the message.

 

►Image_DrawRectJson

“image (Image)”: An image to infer.

“json (Json)”: Inference of the result message.

“imageDrawRectJson (Image)

“Draw the inferred result message to the image to be inferred.

 

►Debug_Image

“image (Image)”: Displays an image of a message that has been drawn on the inference result.

“displayPercentage (Int)”: Displays the percentage of the graph. Value is 200 Said 200 % (The picture is magnified a few times), 50 Said 50 % (The picture narrows in half), and so on, zoom out and shrink.

 

Infer all image in the folder:

Use OpenR8 to open "7_inference_folder" and load the file. As shown in Fig. 43, Fig. 44

  1. Instructions for use:

 

►Select the model you have trained. The following Fig. 46.

 

►Select the location of the image folder you want to infer. The following Fig. 45.

 

►View the result. Click execute to detect the coordinate position of the target in the object and its category, through a trained model,as shown in Fig. 47.

 

►View the results and images. If you want to see the results box detected above the image, press Debug to display the results of object detection. The capacitance of the PCB is detected on the image and marked, as shown in Fig. 48. Press any key to continue to view the next image result.

 

If the appliance does not support the NVIDIA GPU. An error message appears. Please follow the 0, Fig. 49 operation.

 

Fig. 43. 7_inference_folder.png

Fig. 43. 7_inference_folder.

 

Fig. 44. Load 7_inference_folder.png

Fig. 44. Load 7_inference_folder.

 

Fig. 45. 7_inference_folder - Select the model you want to infer.png

Fig. 45. 7_inference_folder - Select the model you want to infer.

 

Fig. 46. 7_inference_folder select test image folder.png

Fig. 46. 7_inference_folder select test image folder.

 

Fig. 47. 7_inference_folder result.png

Fig. 47. 7_inference_folder result.

 

Fig. 48. 7_inference_folder debug result.png

Fig. 48. 7_inference_folder debug result.

 

49.png

Fig. 49. error massage - not GPU.

 

Fig. 50. Not NVIDIA GPU setting way.png

Fig. 50. Not NVIDIA GPU setting way.

 

  1. Introduction to the Content

►Caffe_Init:Caffe framework initialization.

 

►File_ReadFilesFromFolder:

“filesInFolder (Object)”: Output the image folder path that you want to infer.

“numberOfFiles (Int)”: Output the number of files in the image folder to infer.

“folderPath (String)”: Select the image folder location where you want to infer.

“fileFilter”: Filter out the files for a specific extension and read all the files in the folder when not filled in.

 

►Caffe_ObjectDetect_ReadNet

“CaffeObject (Object)”

“GPU (Int)”: Whether to use GPU acceleration.

“deployPath (String)”: read caffe model.datadeploy.prototxt.

“caffeModelPath (String)”: Read training model (Extension is caffemodel)

“labelPath (String)”: Read in use labelImg.exe The category file that is used when marking. data/predefined_classes.txt.

“meanFilePath (String)”

 

►Caffe_ObjectDetect_ InferenceImage

“CaffeObject (Object)”

“OutputResult (Json)”: The results after the image is deduced are stored in the message to this variable.

“GPU (Int)”: Whether to use GPU Accelerated.

“image (Image)”: An image to infer.

“ConfidenceThreshold (Float)”

 

►Json_Print

“json (Json)”: Show Caffe_ObjectDetect_InferenceImage the test results of the message.

 

►Image_DrawRectJson

“image (Image)”: An image to infer.

“json (Json)”: Inference of the result message.

“imageDrawRectJson (Image)” Draw the inferred result message to the image to be inferred.

 

►Debug_Image

“image (Image)”: Displays an image of a message that has been drawn on the inference result.

“displayPercentage (Int)”: Displays the percentage of the graph. Value is 200 Said 200 % (The picture is magnified a few times), 50 Said 50 % (The picture narrows in half), and so on, zoom out and shrink.

 

 

  1. Step8: Infer - 8_inference_webcam

 

Fig. 50.1 Not NVIDIA GPU setting way.png

 

  1. Instructions for use:

When the model has been trained, it can also be done through the 8_inference_webcam.py file for Webcam Real-time image detection and identification of target objects.

Start "8_inference_webcam" with OpenR8 first. Load the file. Check that the "Opencv_VideoCapture_Open" Webcam "Device Code" (The notebook webcam defaults to 0, the added webcam is 1, and so on). Press Run. The following Fig. 50, Fig. 51, Fig. 52, Fig. 53.

 

If the appliance does not support the NVIDIA GPU. An error message appears. Please take the Fig. 42 operation.

 

Fig. 51. Select 8_inference_webcam.png

Fig. 51. Select 8_inference_webcam.

 

Fig. 52. Load 8_inference_webcam.png

Fig. 52. Load 8_inference_webcam.

 

Fig. 53. 8_inference_webcam - check webcam device code.png

Fig. 53. 8_inference_webcam - check webcam device code.

 

Fig. 54. Run 8_inference_webcam.png

Fig. 54. Run 8_inference_webcam.

 

After execution, the webcam camera is turned on for continuous detection and identification, and the capture image is not stopped until the “Enter” key is pressed.


Recommended Article

1.
OpenR8 Community Edition - AI Software for Everyone (Free Download)