Data Mining & AI Introduction

Congratulations on purchasing the Data Mining & AI Kit from PHYTEC. In this guide, you will be taken through the process of setting up and working with this kit.

The Data Mining & AI kit serves two purposes. First, it enables the user to get familiar with the Microsoft Azure cloud and its possibilities.

Second, the Data Mining & AI kit empowers the user to send their data directly to the Microsoft Azure cloud where the user can make use of the many possibilities from the Azure services. These include everything from classic analytic tools to virtual machines designed for deep learning.

The user has the advantage of directly starting their own data recording and analyzing it in the Azure cloud. This can be done as we set up the environment to ensure smooth data mining. The use of Azure services is also explained in this guide. The possibility of running autonomous analysis on the hardware is also provided. We prepared a straight forward model/algorithm embedding via the Azure IoT Edge solution. How to embed the model or algorithm is explained in this guide.

If you are looking for a solution with your own cloud or another cloud provider, please contact us. At the moment, only the Azure cloud is embedded in the kit. If asked, we can find an individual solution for you. For any additional questions, please do not hesitate to contact us at www.phytec.de.

What is Artificial Intelligence and how can I utilize it?

When we talk about artificial intelligence (AI) in an industrial application, we are referring to the automation of intelligently acting machines and algorithms. In most cases, when we use the term AI, we specifically mean machine learning or deep learning, both of which are subsections of AI.

Figure 1 below shows the overlap of the different relationships:

Machine learning and deep learning both use input data to learn the key features of that input data. The resulting model is then capable of identifying those key features in unseen input data. There is, however, a big difference between these two types of learning. 

Machine learning is very effective with limited data and lower complexity and can outperform the latest deep learning algorithms within those boundaries. Deep learning demands quite a bit more training data, calculation performance, and time. The potential to outperform classic machine learning and integrate highly complex and non-linear relations between multiple features is much greater.

Generally, machine learning needs more input from the developer for successful training, while deep learning is very successful in automatically updating its internal parameter until optimization is reached. Next to the impressive results in speech and image classification, the limited demand for developer input is one of the keys to the success of deep learning.

Figure 2 shows which algorithm to choose for your specific requirements.

When starting to use machine learning or deep learning, consider that a crucial part of the whole process is data preparation.

Generally, it should be noted that the quality of the input data is very important for succeeding in classifying anything. "Garbage In Garbage Out" is an essential point to keep in mind. Therefore, sufficient time should be planned for data preparation within an AI project. If you have running data collection in your projects already, make sure that the data is organized, stored, and tagged properly. This will reduce your amount of preparation by a great margin.

With the following guide, you will learn how to implement and use AI for image recognition. Future projects can be built on this knowledge and how to use AI.

Quickstart

For instructions on how to connect, boot up, and begin the demo on your kit, head to: Data Mining and AI Kit Quickstart.

AI Demo Application Usage

The kit comes with a graphical application preinstalled. This application is written in Python and uses the Qt5 framework to draw the window and user interface elements. The application is run in a Docker container, separated by the actual AI model, which also runs in its own container.

The displayed video is a live stream which comes directly from the camera. As soon as you press theCapturebutton, an image is captured with the camera. This image is then used as the input for the AI model and is analyzed. The AI model outputs a prediction of the input image with the most likely matching tag. A tag is a simple name for each of the different possible image classifications. For example, an image representing an open hand may have the tag: "hand-open". The tags are defined by the user before the AI model is created. With the matching tag, the application then displays information text and the confidence of the prediction on the left side of the graphical user interface.

Next to theCapturebutton, you will find a dropdown menu where you can select the endpoint the application is using. "Local" means that the local, exported AI model is being used and processing is done on the IoT Edge device. This is the provided PHYTEC hardware in the kit that is running the application. No internet connection is required for this use-case. The "remote" endpoint utilizes the Azure Cloud processing power and the Custom Vision service to do the prediction. The advantage with this endpoint is that processing times can be cut down by a large margin if the hardware is not powerful enough but has a decent broadband connection.

Under the hood, the graphical application uses a part of a custom wrapper for managing Custom Vision projects. This wrapper is a simple command line tool, also written in Python. See later chapters for a detailed description on how to use the different functions of this command line tool.

The demo application can be controlled either by using an attached mouse on the kit hardware or by using the following shortcuts on a keyboard:

Setting up Azure IoT Edge

Initially, the containers are started by a custom systemd service and the Azure IoT Edge service is disabled. This is so you can run the demo without signing up for an Azure account. But this also implies that other features that Azure IoT Edge offers, do not work.

First, create an Azure IoT Hub and add an edge device. Detailed instructions on how to do this can be found in the Azure Documentation. Then create a deployment for the edge device using the helper script from our Git repository git.phytec.de/aidemo-customvision:

python3 prepare-deployment.py deployment-template.json
az iot edge set-modules --device-id mydevice --hub-name myhub --content deployment.json

This process is described in more detail in a later section of this guide.

To do any work on the PHYTEC AI Kit board, connect to it using SSH. This requires that your computer and AI board are on the same network.

ssh root@phycore-rk3288-4

Assuming you have created an Azure IoT Hub instance with an IoT Edge device, setting up Azure IoT Edge on your PHYTEC edge device is as simple as entering the connection string to /etc/iotedge/config.yaml.

vi /etc/iotedge/config.yaml

In the editor vi, enter the connection string by pressing the i key and replace the text <ADD DEVICE CONNECTION STRING HERE>.

provisioning:
  source: "manual"
  device_connection_string: "<ADD DEVICE CONNECTION STRING HERE>"

After you are done, press ESC and save and quit the file with :wq. Make sure that the custom systemd service for launching the demo application is disabled and stopped. Then enable and start the IoT Edge service:

systemctl stop aikit-docker-image
systemctl disable aikit-docker-image
systemctl enable iotedge
systemctl start iotedge

The Azure IoT Edge service should start the containers and the demo application should be visible after a short amount of time. You can inspect the status of all containers with:

docker ps -a

For a detailed logging output of the IoT Edge service use journalctl:

journalctl -fu iotedge

Setting up remote processing in the demo application

With a running demo application, attach to the Docker container with:

docker exec -it demo bash

There, edit the configuration file of the demo application:

nano env.json

The variables prediction_key, project_id and published_iteration_name must be replaced with the values from your Custom Vision service instance. Also make sure that the endpoint is the one you specified, as this may be different depending on your location.

{
  "endpoint": "https://westeurope.api.cognitive.microsoft.com",
  "endpoint_local": "http://model",
  "training_key": "YOUR_TRAINING_KEY",
  "prediction_key": "YOUR_PREDICTION_KEY",
  "project_id": "YOUR_PROJECT_ID",
  "published_iteration_name": "YOUR_PUBLISHED_ITERATION_NAME",
  "capture_path": "/tmp/aidemo-customvision-capture.jpg",
  "descriptions_path": "data/descriptions/hands/",
  "prediction_threshold": 0
}

After you are done, press Ctrl+X and confirm your changes with y and Enter. Now the remote endpoint can be used in the demo application for predictions as well.

Creating your own AI Model with the Custom Vision Service

In this guide, we will be implementing a hand gesture recognition application. We will use theCustom Visionservice from Microsoft for the first steps. Although not in the scope of this guide, you may also use your own model implementation instead. As the kit already has a preinstalled model which can recognize open and closed hands, the following instructions will result in a model recognizing 3 hand gestures: Closed hand, open hand, and index finger pointing out.

Creating a New Project

1. Visit the Custom Vision web page
2. Sign in with the same account that you use to access your Azure resources
3. Select New project
4. Create your project with the values below
5. Select Create project

Obtaining and Uploading the Images for Training

We store the images and the application code we use to train and deploy our AI models within a Git repository on our FTP server.

1. Clone the repository to your local machine:

   git clone https://git.phytec.de/aidemo-customvision

2. Download the images from our FTP server: https://download.phytec.de/Software/Linux/Applications/aidemo-customvision-hands-pictures.tar.gz

   Tip

   An updated SD-card image with the hand model has been uploaded to our FTP: https://download.phytec.de/Software/Linux/BSP-Yocto-IoTEdge/BSP-Yocto-IoTEdge-RK3288-v1.2/images/phycore-rk3288-4/phytec-iotedge-image-phycore-rk3288-4.sdcard

3. Return to your Custom Vision project on the Custom Vision website and selectAdd images.
4. The Custom Vision API will prompt you to give the images a tag. Tag the image with, "hand-open", "hand-closed", or "hand-finger" depending on the image type.
5. Repeat this process for all image types. You can also add additional images of open and closed hands to improve the performance of the classifier with the specific background you will be using. You can also add new hand gestures like thumb up to increase the usability of the application.
6. After all the images are uploaded, select Trainto train the classifier. After the training is finished, you will get a summary of the precision of the generated model. You can, at any time, add more images and retrain the AI model. Each training iteration can be reviewed under the Performance tab in your Custom Vision project. To get the best results, use training images which show the object in different locations, lighting conditions, and backgrounds as well as from different angles.
7. When you are happy with the results of the training, test the model using the Quick Test button at the top. Specify an image to upload or a URL to an image you want to test.

For real predictions, you can use the endpoint of the Custom Vision project. This endpoint is a REST API you can send commands to. This can be done manually by issuing curl commands, or you use the Azure SDK which is available in multiple languages. For convenience, we recommend using the Azure Python SDK or the command line tool we provide with the Git repository in: modules/demo/app/cli.py (which is just a wrapper for common function using the Azure SDK).

Deploying the Model to the Board

1. Export the AI model from the Custom Vision website using the Export button in the iteration.
2. Select Dockerfileas your platform.
3. SelectARMas the version.
4. After exporting, download the model. You will see that the downloaded ZIP file contains not only the model but also some Python application code that will start aFlaskweb server that makes the aforementioned REST API available. We have prepared this step for the extended hand gesture recognition demo (hand closed, open, and pointing) inmodules/model/. The directory contains all the files you need to build a deployable AI model.

5. To deploy the module containing the AI model, you must build the Docker image and upload it to your Azure Container Registry. Go to the directory of the model:

   cd modules/model/

6. Build the Docker image. Make sure to use the correct address when specifying the address to the container registry (that means you have to replace the "mycontainerregistry" part to match your own Azure Container Registry):

   docker build \
       -t mycontainerregistry.azurecr.io/model:0.1.0-arm32v7 \
       -f arm32v7.Dockerfile .

   Note

   You have to specify the build argument model as well. This variable must contain the exact base filename of the model. All models are placed in modules/model/models/ and their corresponding list of labels in modules/model/labels/.

7. After the image has been built, you should see it in the list of locally available Docker images:

   docker image ls

8. Upload the Docker image to your container registry:

   docker login mycontainerregistry.azurecr.io
   docker push mycontainerregistry.azurecr.io/model:0.1.0-arm32v7

   The image should be visible in the Azure Portal in your container registry under the tab Repositories. If you want to use the module containing the AI model without the preinstalled IoT Edge Agent on PHYTEC boards, you can simply download the image from your container registry and run it manually there.

   Tip

   Do not forget to expose port 8877 when running the container with the AI model.

   The argument--name is needed for the graphical application to detect the AI model container on the same bridge network.

   The IoT Edge Agent automatically does most of the manual managing we did with the Docker commands. For deployment to an IoT Edge device, that is our PHYTEC hardware from the kit, we only specify the modules and their properties in a deployment file. The Git repository contains a template deployment.template.json and a parser scriptparse-deployment.pyfor this job.

9. Change all necessary variables in the JSON file env.jsonin the main directory. Then, simply execute the parser script:

   python3 prepare-deployment.py deployment-template.json

   This will output the final deployment file deployment.json. Use this file to deploy all modules to your IoT Edge device:

   az iot edge set-modules --device-id mydevice --hub-name myhub --content deployment.json

   For detailed instructions on how to modify the deployment template, see the Azure Documentation.

The Command Line Tool

For convenience, we provide a command line tool in modules/demo/app/cli.py to easily execute common commands for Azure Custom Vision projects. The command line tool is written in Python 3 and makes use of the Azure SDK. The demo module that is deployed on the IoT Edge device of this kit has this command line tool integrated and is used by the graphical demo application.

Prerequisites

Usepip to install the Azure SDK for Python. Be sure to use Python 3 for all installations made using pipand the application code.

sudo apt install python3 python3-pip python3-requests
pip3 install azure azureml azure-cognitiveservices-vision-customvision

You also need a set of images to train your Azure Custom Vision model. See the Azure documentation for detailed information on how to correctly capture images for training.

Usage

You can get help on all available CLI commands using the -hflag:

# Get general help and possible commands
python3 app/cli.py -h
# Get help about a specific subcommand (also named positional arguments in the help), e.g. about creating a new project
python3 app/cli.py create-project -h

1. Create a new project:

   python3 app/cli.py create-project "My Project"

2. Verify the creation of your project by listing all projects in your account:

   python3 app/cli.py list-projects

3. Now the images can be added:

   python3 app/cli.py add-images {PROJECT_ID} {PATH_TO_IMAGES} {TAG_NAME}

   Replace the project ID with the unique identifier from the project list. The path to the images can be a wildcard, e.g. path/to/images/*.jpg, or a list of individual file paths. The tag name should be a descriptor of what the images' content is, for example "banana" for a set of images of different bananas. Repeat this step for each set of images.

4. After uploading all your images, the AI model can be trained:

   python3 app/cli.py train-project {PROJECT_ID}

5. The iteration that results from the training must be published to the corresponding prediction resource:

   python3 app/cli.py publish-iteration {PROJECT_ID} {ITERATION_ID} {PREDICTION_RESOURCE} {PUBLISH_NAME}

6. Now the published iteration can be used to classify images:

   python3 app/cli.py classify-remote {PROJECT_ID} {PUBLISH_NAME} path/to/image.jpg

If you want to use the AI model locally, you can export and download it. By listing the available projects, you also get a list of iterations and their IDs.

python3 app/cli.py export-iteration {PROJECT_ID} {ITERATION_ID}
python3 app/cli.py download-exports {PROJECT_ID} {ITERATION_ID}

The model will be downloaded and packaged with the corresponding application and model files to create a Docker image to the current directory.

Further Use of the Kit

The Data Mining & AI kit comes with limited functionality as the model is trained only for the previously described hand gestures. In principle, almost all image classification solutions can be implemented with this kit within the limitations of the Custom Vision service.

Further ideas for the usage of this kit in your next application may be:

• Train the kit to play Tic-tac-toe.
• Automated recognition of your products
• Recognize if a machine is occupied by a worker
• Determine free parking spaces on your company premise
• Recognize persons in a specific area and sound an alarm for unauthorized trespassing
• Count people or goods
• Identify if a process is finished
• etc.

All of the above-mentioned examples can be implemented by writing your own application which takes the classification output to activate an alarm, counter, or any other function.

Setting Up Azure Resources

Company Azure Account

For the Kit to run, you do not need a full company Azure account. If you want to use the kit more extensively, we strongly recommend subscribing to Azure with a company account.

For any paid services you can use the 200$ free credit. To get an overview of the costs associated with Azure, refer to their pricing information: https://azure.microsoft.com/en-us/pricing/

If you already have a Microsoft Office 365 subscription, simply follow this short guide to link an Azure account to your Office 365 subscription: https://docs.microsoft.com/en-us/azure/billing/billing-use-existing-office-365-account-azure-subscription

Otherwise, you will have to create an Office 365 subscription for your company: https://docs.microsoft.com/en-us/office365/admin/setup/setup?view=o365-worldwide&tabsy.

IoT Hub

The Azure IoT Hub is needed to make your Edge device known in Azure, monitor your device, and establish a save connection.

1. Select Create a resource on the Azure portal in the upper left corner (see Figure 3).
   
2. Select Internet of things and following IoT Hub (Figure 4).
   
3. Now you can choose your subscription type, Research Group, Region, and IoT Hub Name.
4. Choose Next: Size and scale to choose the F1: Free tier to use the Kit. This can be changed at any time.
5. Click Review + Create

Create Registry

Next, we need to create an Azure Container Registry to be able to store a Docker container in the Azure cloud. At a later stage, the Docker container will hold the trained model which can be deployed on the Kit hardware.

1. We start again by creating a new resource.
   
2. Choose Containers and then following Container Registry (Figure 6).
   
3. Here, choose the unit Basic under SKU. It provides enough scale for using the Kit. This can be changed at any time.
4. Click Create

Custom Vision Service

Now we add a cognitive service to be able to run the image recognition. The service is called "Custom Vision".

1. First, create a new resource:
   
2. Alternatively, you can use the search bar to search for the custom vision module. Choose the Marketplace result (Figure 8).
   
3. Fill in the needed information and choose the F0 pricing tier as this is enough to use this Kit for now.

Blob Storage

To be able to easily save accessible data in the cloud, we have to set up Blob Storage.

1. Create a new resource:
   
2. Choose Storageand then Storage Account(Figure 10).
   
3. In the creating menu, add all your information regarding subscription, location, etc. (Figure 11),
4. For Account, choose Blob Storage. Blob Storage is an easily accessible storage format.
5. For Replication, choose Locally only for this trial. As Geo redundancy is not needed at this point, you may want to change those settings for future projects.
6. You can also choose between hot and cold. As we are accessing the data more frequently for the training and this tutorial, choose hot.
7. Then click Review + Create:
   

To link your data storage, you need the keys. Go to the Blob Storage via the "all resources button" and select your storage account (Figure 12).

Here you can access the key management which is needed to connect your kit to the Blob Storage. If you are deciding to further use PHYTEC hardware in your projects and will be including more than one module, we will provide the key handling via Trusted Platform Modules (TPM). TPM can refer to a standard for securely storing keys used to authenticate the platform, or it can refer to the I/O interface used to interact with the modules implementing the standard.

Setting up an IoT Edge Device

The kit hardware runs as an IoT Edge device. The necessary runtime binaries are already integrated in our BSP and do not need to be installed manually. Only the connection string in /etc/iotedge/config.yaml has to be set. See the Azure Documentation for detailed information on how to do this.

Adding IoT Edge Devices to the IoT Hub

1. Select your IoT Hub in the portal (Figure 13).
   
2. Select IoT Edge and Add an IoT Edge device (Figure 14).
   
3. Provide a descriptive device ID. Use the default settings to auto-generate authentication keys and connect the new device to your hub.
4. Select Save

The Azure Environment

Azure is a Microsoft service with a variety of cloud solutions. The Azure cloud offers everything from the storage of data to classic data analysis tools and deep learning solutions. Most of the services are supported by the Azure SDK which is available for different languages such as C, Python, C#, Go, and Java to name a few.

To interact with the services, you can use the Azure CLI. Use this if you prefer the command line as your main working environment or the Azure Portal if you need different people with a variety of skill-levels to access the services. Figure 15 shows a general overview of a data mining application which focuses on machine learning.

The Azure Cognitive Services offer pre-built AI solutions which can be used for quick and powerful AI implementations. One example is the kit's own implemented image recognition application, which was created using the Custom Vision service from Microsoft Azure. Figure 16 shows the possible paths based on your desired experience.

Your data analysis function or trained model can be deployed to several instances, from SQL servers to hardware deployment via IoT Edge. The deployed model can run self-sufficiently on the PHYTEC hardware and operate independently from a cloud service, reducing costs for bandwidth and virtual machine usage.

Figure 17 shows a graph with different options when using Azure machine learning, Apache Spark-based analytics Databricks or SQL applications.

References

For more information, tutorials, and case studies, check out the following links:

1. Github, Custom Vision + Azure IoT Edge on a Raspberry Pi 3, https://github.com/Azure-Samples/Custom-vision-service-iot-edge-raspberry-pi
2. Microsoft, Tutorial: Perform image classification at the edge with Custom Vision Service, https://docs.microsoft.com/en-us/azure/iot-edge/tutorial-deploy-custom-vision
3. Microsoft, Deploy Azure IoT Edge modules with Azure CLI, https://docs.microsoft.com/en-us/azure/iot-edge/how-to-deploy-modules-cli
4. Microsoft, Learn how to deploy modules and establish routes in IoT Edge, https://docs.microsoft.com/en-us/azure/iot-edge/module-composition
5. Microsoft, Properties of the IoT Edge agent and IoT Edge hub module twins, https://docs.microsoft.com/en-us/azure/iot-edge/module-edgeagent-edgehub
6. Microsoft, Automate OS and framework patching with ACR Tasks, https://docs.microsoft.com/en-us/azure/container-registry/container-registry-tasks-overview
7. Stack Exchange, Qt camera example doesn't work, https://stackoverflow.com/questions/37650773/qt-camera-example-doesnt-work
8. Docker, Create a container, https://docs.docker.com/engine/api/v1.32/#operation/ContainerCreate
9. Microsoft, Build in Azure with ACR Tasks, https://docs.microsoft.com/en-us/azure/container-registry/container-registry-tutorial-quick-task#build-in-azure-with-acr-tasks
10. Azure pricing information, https://azure.microsoft.com/en-us/pricing/
11. Link Office 356 to your Azure account, https://docs.microsoft.com/en-us/azure/billing/billing-use-existing-office-365-account-azure-subscription
12. Setup Office 365 for your Business, https://docs.microsoft.com/en-us/office365/admin/setup/setup?view=o365-worldwide&tabs=BusPremium
13. Install Azure IoT Edge runtime on Linux (ARM32v7/armhf), https://docs.microsoft.com/en-us/azure/iot-edge/how-to-install-iot-edge-linux-arm