Using the content moderator API, we can add monitoring to user-generated content. The API is created to assist with flags and to assess and filter offensive and unwanted content.
We will quickly go through the key features of the moderation APIs in this section.
Note
A reference to the documentation for all APIs can be found at https://docs.microsoft.com/nb-no/azure/cognitive-services/content-moderator/api-reference.
The image moderation API allows you to moderate images for adult and inappropriate content. It can also extract textual content and detect faces in images.
When using the API to evaluate inappropriate content, the API will take an image as input. Based on the image, it will return a Boolean value, indicating whether the image is appropriate or not. It will also contain a corresponding confidence score between 0 and 1. The Boolean value is set based on a set of default thresholds.
If the image contains any text, the API will use OCR to extract the text. It will then look for the same adult or racy content as text moderation, which we will get to shortly.
Some content-based applications may not want to display any personally identifiable information, in which case it can be wise to detect faces in images. Based on the information retrieved in the face-detection evaluation, you can ensure that no user content contains images of people.
Using the text moderation API, you can screen text against custom and shared lists of text. It is able to detect personally identifiable information and profanity in text. In this case, personally identifiable information is the presence of information such as email addresses, phone numbers, and mailing addresses.
When you submit a text to be moderated, the API can detect the language used, if it is not stated. Screening text will automatically correct any misspelled words (to catch deliberately misspelled words). The results will contain the location of profanities and personal identifiable information in the text, as well as the original text, autocorrected text, and the language. Using these results, you can moderate content appropriately.
There are three ways to moderate content, enabled by the content moderator:
The common scenario used is the last one. This is where machine learning is used to automate the moderation process and teams of people can review the moderation. Microsoft have created a review tool to ease this process. This allows you to see through all the items for review in a web browser while using the APIs in your application. We will look into this tool in the following section.
l, head over to https://contentmoderator.cognitive.microsoft.com/. From here, you can sign in using your Microsoft account. On your first sign-in, you will need to register by adding your name to the account. You will then go on to create a review team, as shown in the following screenshot:
You can do this by selecting the region and entering a team name. You can optionally enter the email addresses of other people who should be part of the team. Click on Create Team.
Once in, you will be presented with the following dashboard:
You will be presented with the total number of images and textual content that are for review. You will also be presented with the total number of completed and pending reviews. The dashboard also lists the users that have completed reviews, as well as any tags used for content.
By selecting the Try option in the menu, you have the option to upload images or text to execute moderation online. Do this by either uploading an image or entering sample text in the textbox. Once done, you can select the Review option, where you will be presented with the following screen:
If the given content is either adult content or racist, you can click on the a or r buttons, respectively. For text, any profanities will be displayed. Once you are done marking reviews, click on Next. This will go through a process of moderating the given content.
Apart from the APIs and the review tool, there are two other tools you can use, as follows:
To use any of these APIs, or to use the moderator APIs, you can make calls to specific REST APIs. To do so, you will need to use an API key and a base URL. These settings can be found under Settings | Credentials on the review tool website, as shown in the following screenshot:
