[OpenR8 solution] Image_Pill_MaskRCNN_Keras (Using MaskRCNN and Keras framework for Pill)
  1. Image_Pill_MaskRCNN_Keras Introduction 


Mask R-CNN is an extended application of Faster R-CNN, adding a branch more than Faster r-cnn. The target pixels are segmented while the target is being detected.


This solution uses Mask r-cnn to determine the type and location of the pill. When marking, the target uses polygons to depict the outline of the object, labeled its category.


Fig. 1.Process for marking training and testing samples.png

Fig. 1.Process for marking training and testing samples.



  1. Image_Pill_maskrcnn_keras Folder Introduction


Image_pill_maskrcnn_keras is located in the solution folder of OpenR8, which contains:

  1. Folders:【data folder】、【logs folder】、【src folder】、【tool folder】.
  2. Flow files:【1_train.flow】、【2_inference.flow】.


※For the first time, it is recommended that you change only the file content of the pill folder in data, and then make your own changes to the desired location.


Fig. 2.Image_pill_maskrcnn_keras location.png

Fig. 2.Image_pill_maskrcnn_keras location.



Use and function


data folder

Store training sample images, categories, test images, categories, and model files.





mask_rcnn_coco.h5 (Used as a sample model training (Use do not delete for the first time))

logs folder

Store the model files that have been trained to complete.


mask_rcnn_mask_0020.h5 (Well-trained model files)

src folder

Python files used in training, of which surgery.py are mainly for training and testing purposes.









tool folder

Use a web page to mark a profile.



A solution for training samples.



A solution for testing samples.


Table 1. Image_Pill_MaskRCNN_Keras folder introduction.



  1. Prepare training sample image and tag categories


When we have to train, we have to decide the right direction first.


In this file, for example, we want to test the type of pill and where it is located, so we will sample picture one by one of their category (pill).


Step 1:Open the tag web site interface

Open the via.html page in the via-2.0.2 folder in the tool folder to mark the sample categories we want to train.


Fig. 3.Tagging web site interfaces.png

Fig. 3.Tagging web site interfaces.


Step 2:Select sample picture storage folder

Click open dir to open the folder location where the picture sample is placed.

Take the solution here, for example, to train the picture to be placed in data/pill/train, so press "Add Files" to prepare to mark the picture, as shown in Fig. 4, Fig. 5.


Fig. 4.Press Add Files to add a new file.png

Fig. 4.Press Add Files to add a new file.


Fig. 5.Select and open the profile you want to train.png

Fig. 5.Select and open the profile you want to train.


Step 3:Box selection categories

Use polygons to depict the area you want to identify.


Fig. 6.Use the Polygon box to select pills.png

Fig. 6.Use the Polygon box to select pills.


If you use the polygon box to finish, double-click the left button to end the polygon box selection, as shown in Fig. 7、Fig. 8.


Fig. 7.Use the polygon box to select pills.png

Fig. 7.Use the polygon box to select pills.



Fig. 8.Use the polygon box to select pills.


Step 4:Box to select a sample picture and mark a category

As shown in Fig. 9, enter the name in the Attributes of attribute name field. As shown in Fig. 10, fill in the category name according to the number of the box. Take this document as an example, it is to judge the pill, because there are different kinds of pills, so fill in "pill1", "pill2", "pill3" and so on, fill out can press X to close.


Continue to select the next sample picture until all sample pictures are marked with a category.


Fig. 9.New name.png

Fig. 9.New name.


Fig. 10.Enter a category name.png

Fig. 10.Enter a category name.


Step 5:Output tag category file

To output the tagged file after all the tags have been completed, as shown in Fig. 11. Press the export annotations (as JSON) output in the upper annotation to mark the JSON file for all picture categories.


Fig. 11.Output tag JSON file.png

Fig. 11.Output tag JSON file.


Step 6:Place the output JSON file in the data/pill/train folder

Place the JSON file you just output in the data/pill/train folder and confirm that the file name is "via_region_data.json" and, if not, rename it "via_region_data.json", as shown in Fig. 12.


Fig. 12.Place.png

Fig. 12.Place the output tag JSON file in the data/pill/train folder.


※ To test the sample in the datapillval folder, do the same as the first step to sixth step.



  1. Perform 1_train.flow start training


At the beginning, please open the "OpenR8 program", if the computer has installed display adapter, please click "R8_Python3.6_GPU.bat" to execute the file. If not, click "R8_Python3.6_CPU.bat" to execute the file, as shown in Fig. 13. After you start the "OpenR8 program", please click "File" => "Open" => "go to OpenR8 under the solution folder" => "select Image_pill_maskrcnn_keras folder" => "select 1_train.flow open", as shown in Fig. 14 and Fig. 15.


Fig. 13.Open the OpenR8 program.png

Fig. 13.Open the OpenR8 program.


Fig. 14.Select 1_train.png

Fig. 14.Select 1_train.py.


Fig. 15.Open 1_train.png

Fig. 15.Open 1_train.py.


※If the sample image is not placed in "data/pill/", an additional dataset_path is required, as shown in Fig. 16.


Fig. 16.Set dataset_path.png

Fig. 16.Set dataset_path.


※Before running, if you do not want to follow the previous model, please delete all h5 files (but keep mask_rcnn_coco.h5), unfamiliar suggestions are not deleted.


※Before executing, if you want to change the "training model name", "training number", "classification category" ... such as parameter setting, please see chapter 6 introduction.


Press perform to start the training sample until you jump out of "press any key to continue...".



  1. Perform 2_inference.flow to see the results of the training


At the end of 1_train.flow training, open 2_inference.flow to test the picture, as shown in Fig. 17 and Fig. 18.


Fig. 17.Select 2_inference.png

Fig. 17.Select 2_inference.py


Fig. 18.Open 2_inference.png

Fig. 18.Open 2_inference.py.


Fill in the sample path to be tested with the trained h5 file path, as shown in Fig. 19 and Fig. 20.


※If you have performed a 1_train.flow and successfully trained the model, be sure to confirm the Fig. 20 of the h5 file names are consistent.


Fig. 19.Fill in the sample path to test.png

Fig. 19.Fill in the sample path to test.


Fig. 20.Fill in the h5 sample path to be trained.png

Fig. 20.Fill in the h5 sample path to be trained.


Press execution to see the results, MASK_R_CNN and other ways to display the results are not the same, if there is a judgment to the category, that area will be marked as a color and frame up to show the category and similarity, on the contrary, if nothing is caught, there will be no mark color, as shown in Fig. 21. The position of the pill is marked separately in different colors, representing the being caught.


Fig. 21.Test results for 2_inference.png

Fig. 21.Test results for 2_inference.py.



  1. Parameter introduction


※ Change read h5 file name:Fig. 22.

※ Set the category name in "data/predefined_classes.txt":Fig. 23.

※ Change the JSON name:Fig. 24.

※ Set the number of GPUs:Fig. 25.


Fig. 22.Change the read h5 file name in surgery.png

Fig. 22.Change the read h5 file name in surgery.py.


Fig. 23.Set the category name.png

Fig. 23.Set the category name in "data/predefined_classes.txt".


Fig. 24.Change the read category JSON name in surgery.png

Fig. 24.Change the read category JSON name in surgery.py.


Fig. 25.Increase GPU number settings in surgery.png

Fig. 25.Increase GPU number settings in surgery.py.


Recommended Article

OpenR8 Community Edition - AI Software for Everyone (Free Download)