[OpenR8 solution] KVM (Use the http connection to capture the remote computer screen and analyze and remotely control the remote computer with artificial intelligence)-
  1. Objective 

 

We use OpenR8 's httpd approach to manipulate remote computers to simulate KVM functionality. The example is to automatically categorize pictures of cats and dogs into designated folders through remote manipulation, as follows Fig. 1.

 

Fig. 1. Flow chart..png

Fig. 1. Flow chart.

 

 

  1. Folder Introduction-KVM

 

Fig. 2. KVM folder..png

Fig. 2. KVM folder.

 

As above, the contents of the folder under KVM are included from the up to bottom: 

 

  1. data folder :

1.1 testImage :

The content is an image of the cat and dog used for training.

1.2 deploy.prototxt :

The network model of the classification.

1.3 labelmap_voc.prototxt and predefined_classes.txt :

Categories of classifications.

1.4 VGG_SSD_300x300_iter_120000.caffemodel :

The model that has been trained to complete.

 

  1. KVM_Httpd_Client.flow :

Run on the control side. It asks the Server for a screenshot..Then read the mouse event from the displayed window. Send the event to the Server to control the mouse on the server side.

 

  1. KVM_Httpd_Client_Caffe_DogOrCat_Example.flow :

When you run this example, you need to open the test folder for the cat and dog picture in Server. Client receives screenshot from server. Then cut the fixed position. Pass the location image to Caffe to identify whether it is a cat or a dog.

According to the identification result "cat, dog, empty" to "send the mouse drag event to drag the image to the Cat folder, send the mouse drag event to drag the image to the Dog folder, end program" action. Because the [cropped position, the towed coordinate position] is a fixed value, the relevant values need to be modified when running a device at a different resolution.

 

  1. KVM_Httpd_Client_HaveKeyBoard.flow :

With the function of KVM_Httpd_Client.flow and Remote keyboard control function.

The parts that the keyboard can perform are 0~9, a~z, enter and SPACEBAR, and so on. You can't read shift, Alt, control, F1~F12, and up or down.

 

  1. KVM_Httpd_Server.flow :

A device running on a remotely controlled end. It builds a Httpd server. After receiving the specific instruction, according to the instruction screenshot, control the keyboard mouse.

 

 

  1. KVM Sample Program Description 

 

  1. On a remote (remote-controlled) computer, turn on OpenR8 execution KVM_Httpd_Server.flow to run Httpd Server

 

On the remote (remote control) computer, turn on the OpenR8 execution KVM_Httpd_Server.flow file to run Httpd Server, as shown in Fig. 3 and Fig. 4, if there is a bounce out of the Windows Security Alert window, select the Allow Access button, as shown in Fig. 5.

 

Fig. 3. Open KVM_Httpd_Server.flow..png

Fig. 3. Open KVM_Httpd_Server.flow.

 

Fig. 4. Open the window for httpd server..png

Fig. 4. Open the window for httpd server.

 

Fig. 5. Click the Allow Access button to unlock the Windows Defender firewall..png

Fig. 5. Click the Allow Access button to unlock the Windows Defender firewall.

 

  1. Open KVM/data/testImage folder

 

When Httpd server is turned on, open the Path KVM/data/testimage folder and zoom in to the full screen, as shown in Fig. 6, so that the client side controls the remote computer's mouse to categorize the cat and dog images in the folder through the Caffe, Then drag the mouse to the cat or dog's folder.

 

Fig. 6. Open the Path Kvm.jpg

Fig. 6. Open the Path Kvm/data/testimage folder. Zoom in on this window to the full screen.

 

  1. Open OpenR8 execution KVM_Httpd_Client_Caffe_DogOrCat_Example.flow file on client computer 

 

Turn on the OpenR8 execution Kvm_httpd_client_caffe_dogorcat_example.flow file on the client-side computer, as shown in Fig. 7. Before clicking the Execute button, you need to set the trained Caffe model, remote IP location, mouse click on the coordinate position of the picture, drag the picture to the “mouse coordinate position of the specified folder”, as described below:

 

Fig. 7. Open KVM_Httpd_Client_Caffe_DogOrCat_Example.flow..png

Fig. 7. Open KVM_Httpd_Client_Caffe_DogOrCat_Example.flow.

 

3.1 Load a trained Caffe model  Start by selecting the " caffeModelPath (string)” of the 2nd function block "Caffe_ ObjectDetect_ReadNet". Click the "..." button to select the trained Caffe model, as shown in Fig. 8.

 

Fig. 8. Select a trained Caffe model..png

Fig. 8. Select a trained Caffe model.

 

3.2 Set the IP of the remote (manipulated) computer  Select the " serverURL_pre " parameter for the 3rd function "String_Add" within "string A (String)". By clicking the "Edit" button, you only need to modify the IP portion of the remote (manipulated) computer, such as "http://192.168.88.11:9000/?cmd=". Just modify the "192.168.88.11" section and change it to a remote (manipulated) computer IP location, as shown in Fig. 9.

 

Fig. 9. Modify remote manipulated computer IP..png

Fig. 9. Modify remote (manipulated) computer IP.

 

3.3 Set the screen cut coordinates for opening a picture of a cat and dog from a remote (manipulative) computer.

 

Select the 18th function "Image_Crop", modify the "x, y, w, h" parameters inside, click the "edit" button to modify, as shown in Fig. 10. You can pull 1 function "Image_Save" before the 18th function "Image_Crop" to store the full image of the remote (manipulating) computer, as shown in Fig. 11. Use the painter to edit this image to see the "x, y, w, and h" parameters, as shown in Fig. 12. A image of a cat and dog that is turned on by a remote (manipulative) computer can be cropped to the correct image information in order to classify successfully, as shown in Fig. 13.

 

Fig. 10. Set the screen cut coordinates for opening a picture of a cat and dog from a remote manipulative computer..png

Fig. 10. Set the screen cut coordinates for opening a picture of a cat and dog from a remote (manipulative) computer.

 

Fig. 11. Store the full image of the remote manipulating computer..png

Fig. 11. Store the full image of the remote (manipulating) computer.

 

Fig. 12..png

Fig. 12. Use the painter to edit this image to see "x, y, w, h" parameters.

 

Fig. 13. cropping and cutting remote images..png

Fig. 13. cropping and cutting remote images.

 

3.4 Set the coordinate position of the mouse to click on the picture.

 

Select the "serverURL_clickDown" parameter for " serverURL (string)” in the 31st function Httpd_HttpGetString", click the "Edit" button, just modify the remote (control) computer IP and x, y.

 

For example:

"http://192.168.88.11:9000/?cmd=mouse&x=205&y=151&event=1&flag=1". Only need to modify the "192.168.88.11", "x=205" and "y=151" parts, change to remote (control) Computer IP location and click on the image x, y coordinate position, as shown in Fig. 14.

 

Fig. 14. Set the coordinate position of the mouse to click on the image..png

Fig. 14. Set the coordinate position of the mouse to click on the image.

 

3.5 Drag a picture to the mouse coordinate position of the specified folder  Select the " serverURL_drop " and " serverURL_moveToCat " parameters for the " serverURL (string) of Httpd_HttpGetString” in the 33rd and 35th function. Click the "Edit" button, only need to modify the remote (control) computer IP and x, y parts.

 

For example :

"http://192.168.88.11:9000/?cmd=mouse&x=366&y=150&event=4&flag=0" only need to modify "192.168.88.11", "x=366" and "y= 150" part, change to remote (control) the computer IP location and drag the picture to the specified folder x and y coordinate position, as shown in Fig. 15.

 

Fig. 15. Drag a image to the mouse coordinate position of the specified folder..png

Fig. 15. Drag a image to the mouse coordinate position of the specified folder.

 

3.6 Move the mouse away the coordinate position of the specified folder

Select the "serverURL_moveToOut" parameter of " serverURL (string)” in the 37th function " Httpd_HttpGetString ", click the "Edit" button, just modify the remote (control) computer IP and x, y parts.

 

For example :

http://192.168.88.11:9000/?cmd=mouse&x=700&y=300&event=0&flag=0 only need to modify "192.168.88.11", "x=700" and "y= 300"part, change to the remote (control) computer IP location and remove the specified folder xy coordinate position, as shown in Fig. 16.

 

Fig. 16. Move the mouse away the coordinate position of the specified folder..png

Fig. 16. Move the mouse away the coordinate position of the specified folder.

 

3.7 The section described above is a folder that sets the picture to be sorted to the cat. You need to set the classification to the Dog folder again, so to modify the 41, 43, 45 and 47 function “serverURL (string) “ of " Httpd_HttpGetString ", modify the steps please follow 3.4~3.6.

 

3.8 When all settings are set, click the "Execute" button to start running the classification, as shown in Fig. 17. 

 

Fig. 17..jpg

Fig. 17. Click the "Execute" button to start running the classification.

 

  1. Simulate KVM function to automatically classify pictures of cats and dogs 

 

As shown in Fig. 18, to run the classification process, identify the captured image as a cat or dog, classified to a cat or dog folder, as shown in Fig. 19.

 

Fig. 18. Running the classification process..jpg

Fig. 18. Running the classification process.

 

Fig. 19. Identify the capture image as a cat or dog classified as a cat or dog.jpg

Fig. 19. Identify the capture image as a cat or dog, classified as a cat or dog folder.


Recommended Article

1.
OpenR8 - AI Software for Everyone (Download)