[OpenR8 solution] Image_PCB_DefectDetection_SSD512_Caffe (Using deep learning Caffe framework and SSD 512 algorithm for defect detection on PCB)
  1. Image_PCB_DefectDetection_SSD512_Caffe

 

This image_pcb_defectdetection_ssd512_caffe is the use of the Caffe framework. Speed is more than twice times faster than Keras. First Use the SSD (Single Shot multibox detector) deep learning algorithm to train the model, and then through the trained model to detect the defects above on PCB. This training image size is 512x512.

The main process is shown in Fig. 1 below.  First, we need to prepare the image that we want the model to learn, select the Target box in the image and record its category. The goal is to let the model know which category of objects in this image belong to. Then use OpenR8 to execute the flow file to automatically generate two TXT list files to let the model know which files are being trained and tested, and which categories are grouped in these files.

After the pre-processing is ready. Then through six flow files in Image_PCB_DefectDetection_SSD512_Caffe. The training and testing can be completed by executing sequentially.

 

Fig. 1. Flow chart..png

Fig. 1. Flow chart.

 

 

  1. Step1:Pre_processing - label the area of interest

 

Fig. 1. 1Flow chart..png

 

  1. Objective :

Box to select the preparation with the labeled region of interest.

  1. Description :

In the Image box, select the object that you want the model to learn. For example, you want the model to learn the capacitance to prepare the capacitance image, and then the Capacitance box in the image to be selected, marked as a capacitor.As a result, the model will know that the area marked by this image is capacitance. And so on, if you want the model to learn something, prepare its image and markup category box.

So the files we need to prepare contain the following two items

(1) Images (images that you want the model to learn, images that you want to test the accuracy of the model)

(2) An XML file that stores the target location of an image (files that are automatically generated when the box is selected)

  1. Example :

 In this Image_PCB_DefectDetection_SSD512_Caffe solution example:

To detect the capacitance in the PCB, we mark the defect in the PCB image through LabelImg.exe (Fig. 2 below), as shown in Fig. 3 red box is the box to select the defective capacitor and mark it as the [Ng_capacitor] category. When the box is selected, the. xml file is automatically generated when stored.

Places the image in the specified folder location. Fig. 4 below.

【Training】

Image path : openR8/solution/Image_PCB_DefectDetection_SSD512_Caffe/data/ train_image

Xml path : openR8/solution/Image_PCB_DefectDetection_SSD512_Caffe/data/ train_annotation

  1. Additional notes :

The tagged software labelImg.exe has been attached with the OPENR8 software and its file path is OpenR8 > Solution > Image_PCB_DefectDetection_SSD512_Caffe > LabelImg.exe. Fig. 2 below, which can be used refer to the Open Source Robot Club [Ezai simple AI] labelimg usage method (Windows version).

 

Fig. 2. The file path of LabelImg.exe..png

Fig. 2. The file path of LabelImg.exe.

 

Fig. 3. Mark the target object in the image..png

Fig. 3. Mark the target object in the image.

 

Fig. 4. The file path of image and xml training..png

Fig. 4. The file path of image and xml training.

 

 

  1. Step 2 : Pre-processing- establish train.txt files to record file locations

 

Fig. 4. 1The file path of image and xml training..png

 

  1. Objective :

The txt files are for defining which data would be used for training and testing respectively.

  1. Procedures :Open and Run Annoset_to_lmdb flow Under the Path

OpenR8solutionImage_PCB_DefectDetection_SSD512_Caffedata folder, a train.txt is generated.

(1) Please "mouse double-click R8.exe" => click "File" => "open" => "OpenR8 > solution Image_PCB_DefectDetection_SSD512_Caffe" => "1_prepare_train_ Txt.flow". The diagram is shown in Fig. 5、Fig. 6、Fig. 7.

(2) Please click run. When execution is complete, a train.txt is generated. Fig. 8  below, located in Fig. 9 , the path is:

【train.txt】openR8/solution/ Image_PCB_DefectDetection_SSD512_Caffe/data/train.txt The TXT content is indicated in Fig. 10 below

For any questions about turning on the software to the load solution, refer to the "OpenR8 manual".

 

Fig. 5. R8.png

Fig. 5. R8.exe.

 

Fig. 6. Open 1_prepare_train_txt.flow..png

Fig. 6. Open 1_prepare_train_txt.flow.

 

Fig. 7. Load 1_prepare_train_txt..png

Fig. 7. Load 1_prepare_train_txt.

 

Fig. 8. Running complete..png

Fig. 8. Running complete.

 

Fig. 9. The file location of Train.txt..png

Fig. 9. The file location of Train.txt.

 

Fig. 10. .png

Fig. 10. Train.txt's content diagram.

 

 

  1. Step 3: Pre-processing- establish a TXT file for the Record Volume label category

 

Fig. 10. 1.png

 

  1. Objective :

Create a category TXT to let the model know what categories the data is divided into, so that the model can learn to classify the images on its own.

  1. Procedures:

Please create a "labelmap.prototxt" file under the openR8/solution/Image_PCB_DefectDetection_SSD512_Caffe /data path.

The file content format is as follows:

item {

name: "none_of_the_above"

label: 0

display_name: "background"

}

item {

           name: " The name of the category"

label: 1

display_name: "display the name of category"

}

……

item {

name: " The name of the category "

label: n

display_name: " display the name of category "

}

Please note that the name and display_name above must be associated with the The category name and number of classifications within the XML file.

 

  1. Example:

In the openR8/solution/ Image_PCB_DefectDetection_SSD512_Caffe /data folder Labelmap.prototxt, you can open the file by notepad++ or Notepad, as shown in Fig. 11 below.

The contents of the file are shown in Fig. 12 below, and we can see that we classify the images into none_of_the_above (none of the above) and Pass (normal), Ng_capacitor, Ng_scratch four categories, of which individual volume labels are numbered 0 and 1, 2, 3, 4.

If you want to add a new category yourself, please add the category name and label number, as shown in Fig. 12 (right).

 

Fig. 11. The file path of Labelmap.prototxt..png

Fig. 11. The file path of Labelmap.prototxt.

 

Fig. 12. Labelmap content Diagram left add category diagram right ..png

Fig. 12. Labelmap content Diagram left add category diagram right .

 

 

  1. Step 4 : pre-processing - annoset_to_lmdb

 

Fig. 12. 1Labelmap content Diagram left add category diagram right ..png

This step is the Lmdb file required to train and test the model for the TXT transformation that has been defined in the previous two steps.

  1. Description: Open and Run Annoset_to_lmdb flow.

(1) Please "mouse double-click R8.exe" => click "File" => "open" => "OpenR8 > solution Image_PCB_DefectDetection_SSD512_Caffe" => "2_annoset_to_ Lmdb.flow. " The diagram is shown in Fig. 13, Fig. 14, Fig. 15.

(2) Please click Execute. If there is an old Lmdb file, you will jump out and remove it, click Yes (Y), as shown in Fig. 16 below. When the execution is complete, train_lmdb and Test_lmdb are generated.Fig. 17, Fig. 18 below.

For any questions about turning on the software to the load solution, refer to the "OpenR8 manual".

 

Fig. 13. R8.png

Fig. 13. R8.exe.

 

Fig. 14. Open the annoset_to_lmdb.flow..png

Fig. 14. Open the annoset_to_lmdb.flow.

 

Fig. 15. Load the annoset_to_lmdb..png

Fig. 15. Load the annoset_to_lmdb.

 

Fig. 16. delete the lmdb file..png

Fig. 16. delete the lmdb file.

 

Fig. 17. Running complete..png

Fig. 17. Running complete.

 

Fig. 18. Create the train_lmdb and test_lmdb..png

Fig. 18. Create the train_lmdb and test_lmdb.

 

  1. annoset_to_lmdb Parameter description

 

Fig. 19. Input file left turn to output file right .png

Fig. 19. Input file (left) turn to output file (right)

 

The following is the explanation of functions used in this flow.

Caffe_Init : Initialize Caffe framework.

File_DeleteDir (Fig. 20) : When there is an old lmdb file in the folder, you need to remove the old file before starting to generate a new lmdb file. Press File_DeleteDir in Flow area. The function details will be shown in the Function area. Press Edit to edit the value of dirName in the Variable area. 

 

Fig. 20. File_DeleteDir..png

Fig. 20. File_DeleteDir.

 

Caffe_ObjectDetect_CreateTrainData: generate lmdb file.

 

Fig. 21. Caffe_ObjectDetect_CreateTrainData..png

Fig. 21. Caffe_ObjectDetect_CreateTrainData.

 

 

  1. Step 5: Training

 

Fig. 21. 1Caffe_ObjectDetect_CreateTrainData..png

The introduction is divided into two parts, the 1th first describes how to start training, and then if you are interested in knowing that each process block can be viewed in the 2nd to describe the details of this process.

 

  1. Description :Training model.

First, use the OpenR8 "open 3_train.flow" file and load the 3_train.flow, Fig. 22 below Fig. 23.

Next, press "execute" to start the training model, which takes a little time to wait for the program to build the model. Figure 24 below.

When this run is complete, a trained model is generated, and its model file generation is placed, as shown in Figure 25 below.

Path : OpenR8/solution/ Image_PCB_DefectDetection_SSD512_Caffe/data/

File name1 : xxx.Caffemodel

File name2 : xxx. solverstate

 

Fig. 22. The path of 3_train.flow..png

Fig. 22. The path of 3_train.flow.

 

Fig. 23. Load the 3_train.flow..png

Fig. 23. Load the 3_train.flow.

 

Fig. 24. Run 3_train.flow..png

Fig. 24. Run 3_train.flow.

 

Fig. 25. The production model is finished after training..png

Fig. 25. The production model is finished after training.

 

  1. The Parameters Description of 3_train.flow:

►Caffe_Init: Initialize Caffe framework.

►Caffe_Train:

【GPU】This variable is for setting GPU acceleration. Set the value to “all” to use GPU or set to “null” not to use GPU. (If you fill in “all”, please confirm whether the device has GPU)

【solverPath】Set the path of solver.prototxt which is the configuration of training.

【continueTrainModelPath】Set the path of pre-trained model.

 

Fig. 26. The Caffe_Train of 3_train.flow..png

Fig. 26. The Caffe_Train of 3_train.flow.

 

Fig. 27. solver.prototxt..png

Fig. 27. solver.prototxt.

 

 

  1. Step 6: Test the model of each training stage

 

  1. Objective : Used to test the accuracy of each stage model, from which to find the appropriate model, to avoid over fitting of training samples during training, resulting in poor test sample results.
  2. Description :

(1)    Training sample: "Turn on OpenR8" => "Open and load 4_train_result.flow" => "Run Flow" => "train_result.txt generation".

(2)    Select the appropriate training number model:

As shown in Fig. 28, it can be observed from the content of train_result.txt that the results of the model at a certain training number stage are acceptable or have reached convergence after a certain stage. Therefore, it is more appropriate for us to choose the model of this number.

 

Fig. 28. Detail of train_result.txt. .png

Fig. 28. Detail of train_result.txt.

 

 

  1. Step 7: Inference using a trained model - inference

 

Fig. 28.1 Detail of train_result.txt. .png

When the model has been trained, you can use this step to infer using a trained model.

View the resulting image after the inference. If you want to see the results box detected above the image, press Debug. Displays the results of object detection. PCB images are marked with detected capacitors. Figure 34 Below

 

  1. How to use the process :

►First use OpenR8 to open "5_inference" and load the file. See Fig. 29, Fig. 30

►Select the image you want to test. See Fig. 31.

►Choosing a model that has been trained。See Fig. 32。

►View the results of the test message. Press run to detect the target coordinate position and its category in the image through a trained model. See Fig. 33.

►View the resulting image after the inference. If you want to see the results box detected above the image, press Debug. Displays the results of object detection. PCB images are marked with detected capacitors. See Fig. 34.

If the hardware does not support the NVIDIA GPU. An error message appears. FollowFig. 35, Fig. 36 below.

 

Fig. 29. 5_inference..png

Fig. 29. 5_inference.

 

Fig. 30. Load 5_inference..png

Fig. 30. Load 5_inference.

 

Fig. 31. 5_inference - Select test Images..png

Fig. 31. 5_inference - Select test Images.

 

Fig. 32. 5_inference - select the model you want to infer..png

Fig. 32. 5_inference - select the model you want to infer.

 

Fig. 33. 5_inference running result..png

Fig. 33. 5_inference running result.

 

Fig. 34. 5_inference - Debugging results..png

Fig. 34. 5_inference - Debugging results.

 

Fig. 35. Error message-show no GPU..png

Fig. 35. Error message-show no GPU.

 

Fig. 36. How to set up without NVIDIA GPU..png

Fig. 36. How to set up without NVIDIA GPU.

 

  1. Function details:

►Caffe_Init: Initialize Caffe.

►Image_Open:

【imageFileName (String) 】The path of inference image.

【image (Image) 】After reading the image for inference according to the image file name above, save it to this variable.

►Caffe_ObjectDetect_ReadNet

【CaffeObject (Object) 】The Caffe object.

【GPU (String) 】Whether to enable GPU acceleration.

【deployPath (String) 】Read data/deploy.prototxt.

【caffeModelPath (String) 】Read trained model. (.caffemodel)

【labelPath (String) 】Read the class file data/predefined_classes.txt.

【meanFilePath (字串) 】

►Caffe_ObjectDetect_InferenceImage

【CaffeObject (Object) 】The Caffe object.

【OutputResult (Json) 】The outcome of inference.

【GPU (Int) 】Whether to enable GPU acceleration.

【image (Image) 】

【ConfidenceThreshold (Float) 】The confidence threshold of object recognition. ►Json_Print

【json (Json) 】Print the outcome of inference.

►Image_DrawRectJson

【image (Image) 】

【json (Json) 】The outcome of inference.

【imageDrawRectJson (Image) 】Draw the result to the inference image.

►Debug_Image

【image (Image) 】Show the inference image.

【displayPecentage (Int) 】The showing percentage of inference image. A value of 200 means 200 (the picture size is magnified one times), 50 means 50 (the picture size is halved), and so on, zoom in and out.

 

 

  1. Step8: Using webcam for inference

 

Fig. 36. 1How to set up without NVIDIA GPU..png

After the model has been trained, the trained model can also be tested by capturing the image from the webcam in this step which is “8_inference_webcam.flow”.(See Fig. 37, Fig. 38)

  1. Procedure:

►[OpenCV_VideoCapture_Open] Make sure the deviceNumber is set (default is 0). (See Fig. 39)

►Check if the computer supports GPU acceleration. Click edit of the "Caffe_ObjectDetect_ReadNet" GPU. Replace "All" in "value" of the variable area with "" To cancel the GPU acceleration.

►Select a model with good training effect to detect the picture. Click caffeModelPath "..."  of "Caffe_ObjectDetect_ReadNet" to select the training model to use (. caffemodel file).

►Press Run to detect the objects through the webcam. (See Fig. 40)

 

Fig. 37. 6_inference_webcam.flow..png

Fig. 37. 6_inference_webcam.flow.

 

Fig. 38. Load 6_inference_webcam.flow..png

Fig. 38. Load 6_inference_webcam.flow.

 

Fig. 39. 6_inference_webcam - Check the deviceNumber of webcam..png

Fig. 39. 6_inference_webcam - Check the deviceNumber of webcam.

 

Fig. 40. 6_inference_webcam running result..png

Fig. 40. 6_inference_webcam running result.

 

  1. Function details:

►OpenCV_VideoCapture_Init: Initialize video capture.

►OpenCV_VideoCapture_Open: Open selected webcam.

►OpenCV_VideoCapture_Grab: Grab the image from the webcam.

►OpenCV_VideoCapture_Retrieve: Retrieve the image from the webcam.

►OpenCV_VideoCapture_Release:Close the webcam. 


Recommended Article

1.
OpenR8 - AI Software for Everyone (Download)