Training Azure Cognitive Services to recognize karting images with Custom Vision service

For this we will use the public preview of Custom Vision Service Training API.

We previously saw how to use the Azure Cognitive Services Vison API named Computer Vision to analyze a karting image.

Today thanks to the public preview of Custom Vision Service we will train Azure Cognitive Services to recognize karting images. It means that we take a different approach because here we will use our own data to recognize images as opposed to using Vison API Microsoft's data.

To start we will create a project with the .NET Console App template in Visual Studio.

Before starting you will need to have a Training Key. You can get it from www.customvision.ai. If you have an Azure account get it by creating a Custom Vision Service in the Azure Portal or as seen in the previous article via ARM template.

Creation

The goal here is to build a console application allowing us to create, edit and delete Custom Vision Service projects. Then for each project we want to be able to add karting images and train the project.

The first thing you will need is to add the following NuGet package to your project: Microsoft.Cognitive.CustomVision.Training

Let's create the main method of our console application and a start method:

...
using Microsoft.Cognitive.CustomVision.Training;
using Microsoft.Rest;
using System;
using System.Linq;
using System.Threading.Tasks;
...

namespace VisionRacing.TrainingRacingImages
{
    class Program
    {
        static void Main(string[] args)
        {
            var trainingKey = "Your Custom Vision training key.";

            Start(trainingKey).Wait();
        }

        private static async Task Start(string trainingKey)
        {
            var projectName = " ";
            var trainingApi = GetTrainingApi(trainingKey);

            while (!string.IsNullOrEmpty(projectName))
            {
                try
                {
                    Console.Clear();
                    await ListProjects(trainingApi);

                    Console.WriteLine("Please enter a project name or press enter to exit:");
                    projectName = Console.ReadLine();

                    if (!string.IsNullOrEmpty(projectName))
                    {
                        await WorkOnProject(trainingApi, projectName);
                    }
                }
                catch (Exception ex)
                {
                    Console.ForegroundColor = ConsoleColor.Red;
                    Console.WriteLine($"An error occurred: {Environment.NewLine}{ex.Message}");

                    if (ex is HttpOperationException)
                    {
                        Console.WriteLine(((HttpOperationException)ex).Response.Content);
                    }

                    Console.ResetColor();
                    Console.WriteLine();
                    Console.WriteLine();
                    Console.WriteLine("Press any key to continue");
                    Console.ReadLine();
                }
            }
        }

        ...

        private static TrainingApi GetTrainingApi(string trainingKey)
        {
            return new TrainingApi
            {
                ApiKey = trainingKey
            };
        }

        private static async Task ListProjects(TrainingApi trainingApi)
        {
            var projects = await trainingApi.GetProjectsAsync();

            if (projects.Any())
            {
                Console.WriteLine($"Existing projects: {Environment.NewLine}{string.Join(Environment.NewLine, projects.Select(p => p.Name))}{Environment.NewLine}");
            }
        }
        ...
    }
}

The start method will allow us to list all the projects associated to our account. You can then enter the name of an existing project or a new project.

We also create two other methods:

GetTrainingApi: this will return a TrainingApi object using the provided Training Key.
ListProjects: it will call the TrainingApi method GetProjectsAsync to list all the projects.

When a project name is entered, the WorkOnProject method is called:

...
using Microsoft.Cognitive.CustomVision.Training;
using Microsoft.Cognitive.CustomVision.Training.Models;
using System;
using System.Linq;
using System.Threading.Tasks;
...

namespace VisionRacing.TrainingRacingImages
{
    class Program
    {
        ...
        private static async Task WorkOnProject(TrainingApi trainingApi, string name)
        {
            var option = " ";

            while (!string.IsNullOrEmpty(option))
            {
                Console.Clear();

                var project = await GetOrCreateProject(trainingApi, name);
                Console.WriteLine($"  --- Project {project.Name} ---");
                Console.WriteLine();

                await ListProjectTags(trainingApi, project.Id);

                Console.WriteLine("Type an option number:");
                Console.WriteLine("  1: Create Karting images");
                Console.WriteLine("  2: Create F1 images");
                Console.WriteLine("  3: Create MotoGP images");
                Console.WriteLine("  4: Create Rally images");
                Console.WriteLine("  5: Train project");
                Console.WriteLine("  6: Delete project");
                Console.WriteLine();
                Console.WriteLine($"Press any other key to exit project {name}");
                option = Console.ReadLine();

                switch (option)
                {
                    case "1":
                        await CreateTagImages(trainingApi, project.Id, ImageType.Karting);
                        break;
                    case "2":
                        await CreateTagImages(trainingApi, project.Id, ImageType.F1);
                        break;
                    case "3":
                        await CreateTagImages(trainingApi, project.Id, ImageType.MotoGP);
                        break;
                    case "4":
                        await CreateTagImages(trainingApi, project.Id, ImageType.Rally);
                        break;
                    case "5":
                        await TrainProject(trainingApi, project.Id);
                        break;
                    case "6":
                        await DeleteProject(trainingApi, project.Id);
                        option = string.Empty;
                        break;
                    default:
                        option = string.Empty;
                        break;
                }
            }
        }

        ...

        private static async Task<Project> GetOrCreateProject(TrainingApi trainingApi, string name)
        {
            var projects = await trainingApi.GetProjectsAsync();

            var project = projects.Where(p => p.Name.ToUpper() == name.ToUpper()).SingleOrDefault();

            if (project == null)
            {
                project = await trainingApi.CreateProjectAsync(name);
            }

            return project;
        }

        ...

        private static async Task ListProjectTags(TrainingApi trainingApi, Guid projectId)
        {
            var tagList = await trainingApi.GetTagsAsync(projectId);

            if (tagList.Tags.Any())
            {
                Console.WriteLine($"Tags: {Environment.NewLine}{string.Join(Environment.NewLine, tagList.Tags.Select(t => $"  {t.Name} (Image count: {t.ImageCount})"))}{Environment.NewLine}");
            }
            else
            {
                Console.WriteLine($"No tags yet...{Environment.NewLine}");
            }
        }
        
        ...

        private enum ImageType
        {
            F1,
            Karting,
            MotoGP,
            Rally
        }
    }
}

We will also use two other methods:

GetOrCreateProject: will try to retrieve an existing project. Create it if doesn't exists.
ListProjectTags: list all the existing tags of a project.

The workOnProject method will get a project using the name provided, list the project tags and then show several options.

The first four options will allow to add different kind of images to the project (Karting, F1, Moto GP, and Rally images). Option five is to train the project and six to delete it.

Here is the code that shows how to add images with tags to a project:

...
using Microsoft.Cognitive.CustomVision.Training;
using Microsoft.Cognitive.CustomVision.Training.Models;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
...

namespace VisionRacing.TrainingRacingImages
{
    class Program
    {
        ...
        private static async Task CreateTagImages(TrainingApi trainingApi, Guid projectId, ImageType imageType)
        {
            Console.Clear();

            var imageTag = await GetOrCreateTag(trainingApi, projectId, imageType.ToString());
            var racingTag = await GetOrCreateTag(trainingApi, projectId, "Racing");
            var imageTypeCount = GetImageCountPerImageType(imageType);

            if (imageTag.ImageCount != imageTypeCount)
            {
                Console.WriteLine($"{imageType} images creation in progress...");

                var images = new List<ImageUrlCreateEntry>();

                for (int i = 1; i <= imageTypeCount; i++)
                {
                    images.Add(new ImageUrlCreateEntry($"https://github.com/vivienchevallier/Article-AzureCognitive.Vision-Racing/raw/master/Images/{imageType}/{imageType}%20({i}).jpg"));
                }

                var tags = new List<Guid>() { imageTag.Id, racingTag.Id };
                var response = await trainingApi.CreateImagesFromUrlsAsync(projectId, new ImageUrlCreateBatch(images, tags));

                Console.Clear();
                Console.WriteLine($"{imageType} images successfully created.");
            }
            else
            {
                Console.WriteLine($"{imageType} images already created.");
            }

            Console.WriteLine();
            Console.WriteLine("Press any key to continue");
            Console.ReadLine();
        }

        private static int GetImageCountPerImageType(ImageType imageType)
        {
            switch (imageType)
            {
                case ImageType.F1:
                    return 7;
                case ImageType.Karting:
                    return 35;
                case ImageType.MotoGP:
                    return 7;
                case ImageType.Rally:
                    return 6;
                default:
                    return 0;
            }
        }

        ...

        private static async Task<Tag> GetOrCreateTag(TrainingApi trainingApi, Guid projectId, string name)
        {
            var tagList = await trainingApi.GetTagsAsync(projectId);

            var tag = tagList.Tags.Where(t => t.Name.ToUpper() == name.ToUpper()).SingleOrDefault();

            if (tag == null)
            {
                tag = await trainingApi.CreateTagAsync(projectId, name);
            }

            return tag;
        }
        ...
    }
}

Here are more details about the other methods used:

CreateTagImages: using the provided ImageType, add images to the project. The images are public on GitHub. Each image is associated with two tags: Racing and the value of the ImageType.
GetImageCountPerImageType: get the number of images on GitHub for the specified ImageType.
GetOrCreateTag: will try to retrieve an existing tag. Create it if doesn't exists.

For each image we associate two tags, this is the minimum required by the service to be able to train an image.

And to finish, here the code to delete and train a project:

...
using Microsoft.Cognitive.CustomVision.Training;
using System;
using System.Threading;
using System.Threading.Tasks;
...

namespace VisionRacing.TrainingRacingImages
{
    class Program
    {
        ...
        private static async Task DeleteProject(TrainingApi trainingApi, Guid projectId)
        {
            Console.Clear();

            await trainingApi.DeleteProjectAsync(projectId);

            Console.WriteLine("Project deleted... Press any key to continue");
            Console.ReadLine();
        }

        ...

        private static async Task TrainProject(TrainingApi trainingApi, Guid projectId)
        {
            var iteration = await trainingApi.TrainProjectAsync(projectId);

            while (iteration.Status == "Training")
            {
                Console.Clear();
                Console.WriteLine("Training in progress...");

                Thread.Sleep(1000);

                iteration = await trainingApi.GetIterationAsync(projectId, iteration.Id);
            }

            iteration.IsDefault = true;
            trainingApi.UpdateIteration(projectId, iteration.Id, iteration);

            Console.WriteLine();
            Console.WriteLine("Project successfully trained... Press any key to continue");
            Console.ReadLine();
        }
        ...
    }
}

DeleteProject: a straightforward method to delete a project.
TrainProject: send a request to the service to start training the project. Each time a project is trained is called an iteration. Training a project is not immediate, that's why we loop to reach for the status of the iteration every second.

Example of use

The console application is now ready to run, let's execute it:

Existing projects:
Test Project

Please enter a project name or press enter to exit:

When starting it will list the existing projects and allow to enter a project name, new or existing.

  --- Project Test Project ---

No tags yet...

Type an option number:
  1: Create Karting images
  2: Create F1 images
  3: Create MotoGP images
  4: Create Rally images
  5: Train project
  6: Delete project

Press any other key to exit project test project

At first no images have been created. Select option one to create karting images.

Karting images creation in progress...

Karting images successfully created.

Press any key to continue

Once the images created, the project will display the following:

  --- Project Test Project ---

Tags:
  Karting (Image count: 35)
  Racing (Image count: 35)

Type an option number:

...

Now you can train the project by selecting option five.

Training in progress...

Project successfully trained... Press any key to continue

Now that your project is trained with the Training API, you could start using the Prediction API.

To go further

If you try to train a project and don't have at least two tags, here the error message you'll get:

An error occurred:
Operation returned an invalid status code 'BadRequest'
{"Code":"BadRequestTrainingValidationFailed","Message":"Not enough tags for training"}

If you train your project and didn't add anything since the previous training, you'll get the following error:

An error occurred:
Operation returned an invalid status code 'BadRequest'
{"Code":"BadRequestTrainingNotNeeded","Message":"Nothing changed since last training"}

Summary

We have seen how to train Azure Cognitive Services to recognize karting images thanks to Custom Vision Service Training API in a .NET console application.

Now we want to use our trained service, how do to that?

Well in my next article about Azure Cognitive Services we will discover how to use the Custom Vision Service Prediction API!

You can get the project source code here:

Browse the GitHub repository

(Note that the project uses .NET Framework 4.7)

Please feel free to comment or contact me if you have any question about this article.

Comments

Thursday, August 01, 2019 11:50 AM - Kelly Techno

English

Awesome article, Thanks for sharing.

Add a comment

Language

Name

E-mail (Will not be published)

Website url

Message

Back to articles