Chapter 15. Testing and Qualification

Ideas must be put to the test. That’s why we make things; otherwise, they would no more than ideas. There is often a huge difference between an idea and its realisation. I’ve had what I thought were great ideas that just didn’t work.

—Andy Goldsworthy

Before releasing a Bluetooth low energy product that implements one or more services and profiles, it must be tested and then qualified. The Bluetooth Special Interest Group (SIG) requires this and codifies it in the membership agreements that all Bluetooth SIG members sign. This agreement allows products that comply with the published specifications to be able to utilize and rely on the necessary claims of the intellectual property that any other members may have. This means that a product that has implemented one or more Bluetooth specifications, as determined by a Declaration of Compliance (DoC) signed by that member, has a royalty-free license to use that technology. This implies that there is a great benefit for declaring compliance. Proving that your device is compliant is where the testing and qualification come into the picture.

As illustrated in Figure 15–1, the qualification program is a multistep process that spans product conception through to a qualified and listed product. Part of this process requires that testing is performed. Some steps require that a monetary investment is made; others require decisions to be made about what the product will do.

Figure 15–1. The Bluetooth testing and qualification process

15.1. Starting a Project

The very first step is the easiest. You start a project. This requires you to log in to the Bluetooth SIG Web site¹ at bluetooth.org and start a new Test Plan Generator (TPG) project. You need a project name and the date that you expect to qualify. Don’t worry about this information too much. You can change these names and dates in a later step.

The nice thing about starting a project is that it is free. You can start as many projects as you want. If you want to test out what might be required to do something, you can just start a project, configure the project, and see what the resulting testing burden is. You can also delete projects at any time, so if a project no longer looks viable, you can remove it from your current list of active projects.

The project that you create in this initial step will be the same all the way through to the point of qualification. This means that although the project is free to create, before the project can become an officially listed, qualified product, a fee will have to be paid.

To start the project, you go to the Create New Project page on the bluetooth.org Web site. The information you will need includes the expected qualification date, the TPG Release Version, a project name, product type, and a description of the hardware and software versions.

For a Bluetooth low energy product, the TPG Release Version should be 4.0 or later. By doing so, the project takes advantage of all the low energy layers in the system that were only introduced in the v4.0 core specification.

The next value to select is the product type. There are eight types of products that can be selected, depending on the type of component or subsystem that is being qualified.

• End Product This is an actual product that is available for public consumption.

• Host Subsystem Product This is a product that is only composed of the host parts of the core specification, such as Logical Link Control and Adaptation Protocol (L2CAP), Attribute Protocol (ATT), Generic Attribute Profile (GATT), Security Manager (SM), and Generic Access Profile (GAP). This needs to be combined with a controller subsystem and one or more profile subsystems to make an end product.

• Controller Subsystem Product This is a product that is only composed of the controller parts of the core specification, such as the radio and link layers. This needs to be combined with a host subsystem and one or more profile subsystems to make an end product.

• Profile Subsystem Product This is a profile subsystem that contains an implementation of one or more profiles and/or services. This will be combined with a host subsystem and a controller subsystem, and possibly additional profile subsystem products to make an end product.

• Component Subsystem Product This is an individual part of a subsystem. For example, the implementation of your SM might be separately listed as a component subsystem product.

• Development Tool This is a special product type that you use during the development of Bluetooth products. This type of product cannot be sold to the end consumer, but it can be distributed to other members to assist the rapid creation of end products.

• Test Equipment This is a special type of product designed for the small set of test equipment manufacturers who need to have access to the royalty-free license. Because the test equipment might need to do things outside the specification to enable testing of real products, these tools have a special product type.

The interesting thing about this set of product types is how they can be combined. At the most discrete level, everything can be combined together into an end product as a whole. This holistic approach can cause lots of problems because the complexity of the testing will be huge. Instead, most components would be tested individually and listed as individual component subsystems. These can then be combined together without any additional testing into an end product.

Components can also be combined into a controller or host subsystem. By doing this, whole parts of the product’s design can be combined into a single listing. For example, when using a controller chip, it would be listed as a controller subsystem so that it can be combined with your host subsystem to make an end product.

15.2. Selecting Features

The next step in the process of qualifying a product is to select what features your product is going to support. This is declared in the summary template. Here, you can select the set of core layers and profiles and services that your device will support. For example, a low energy proximity device would select RF PHY, Link Layer, L2CAP, ATT, SM, and GATT in the core section, and Link Loss Service, Immediate Alert Service, and the Tx Power Service in the host section.

After this, you need to delve deeper into the details of what is being claimed. This is where you match the features that you plan to implement with the feature set for which you will later be claiming compliance. For every feature claimed, a set of tests will be required to be run, and test evidence will need to be collected before the DoC can be signed.

For each layer that was selected in the summary template, the detailed Implementation Conformance Statement (ICS) is filled in. For each ICS, each and every possible mandatory or optional feature has a line where you can identify whether your product will support this feature. There is an ICS for each layer of the core specification and each profile or service being claimed.

15.3. Consistency Check

Once all the ICSs have been filled in, the validity of the selections can be checked. There are many rules as to what features you must have if another feature has been selected. If any of these selections are invalid, the line will be marked as invalid. Once these errors have been corrected, the consistency check can be refreshed. For example, if a service has optional notification of a characteristic, then if you select this line, the consistency check will ensure that GATT also has the ability to write characteristic descriptors. This is because without the ability to write characteristic descriptors, it would be impossible to configure a characteristic to enable notifications.

Sometimes, the set of features selected will be inconsistent. If they are inconsistent, the set of features selected will need to be changed and the consistency check attempted again. Once all the consistency checks have completed, it is possible to move to the next step.

It is worth noting that occasionally the test plan generator (TPG) has errors for new specfications. If you believe you are correct and the TPG signals an inconsistency, this can be raised with the Bluetooth Qualification Administrator (BQA), who can arrange for the TPG issues to be fixed.

15.4. Generating a Test Plan

Each layer of the core specification, or service or profile specification, has an associated test specification. Each test within these test specifications has a unique identifier. For your product, many of these tests will not be required, depending on the configuration of your ICS that was determined from the previous step. Therefore, the next step is to generate your test plan for your product as it pertains to the ICS that you have just completed. The test plan includes all the tests that need to be run to prove that your product has implemented all the required functionality that you are claiming.

The test plan that is generated can be downloaded to your local computer from the Web site; this provides you with a list of tests to use during your product development. Once the product has been developed and a prototype is available for testing, there is one final step before the qualification testing can begin. This is the creation of a compliance folder.

15.5. Creating a Compliance Folder

A compliance folder is a single location where all the information about a product, that product’s feature set and capabilities, and all test evidence are stored. The compliance folder can be a collection of electronic documents or paper documents. The folder must be stored securely and be easily accessible for each product that you have qualified. If later on a problem is found with your device, the Bluetooth SIG can audit that product’s compliance; thus, it must be given access to the compliance folder to be able to perform this audit.

The compliance folder contains the following items:

• Product Information This includes information about the product, such as the product name, a description of the product, and hardware and software version information. It also includes, where relevant, the user manual or user guide, a block diagram and technical description of the product, and notes on how the Bluetooth component is integrated into the design of the product.

• Testing Information This includes all the information about how the device was tested, including the test plan that was generated, a test declaration that has the result of each test, test reports that are required for formal validated tests, and any extra information that might be required to perform the tests so that the testing can be replicated at a later date.

• Design Information This includes detailed design information that describes the design of the product where relevant to the Bluetooth functionality. This could include elements, such as the circuit schematics, printed circuit board layouts, component layouts, and a bill of materials that describes the physical external components and their tolerances. This could also include things such as the software design documentation and sufficient information to extract the precise software used during testing from a software version control system.

• Implementation Conformance Statements This includes the ICS documents for each layer in the system and each profile or service.

The compliance folder must be stored securely while the product is offered for sale or distribution. Only after the product has been removed from sale for over a year can the compliance folder be destroyed.

15.6. Qualification Testing

Once the prototype is ready for testing, a test plan has been generated, and a compliance folder has been created to store all the information about the product and its testing information, the testing can begin.

For some tests, there exists validated test equipment that can be used to run the tests. There are three types of validated testers: RF testers are used to run the physical layer qualification tests; protocol testers are used to run the protocol layer tests; and the Profile Tuning Suite (PTS) is used to run the profile tests.

Each test is assigned to one of four different categories:

Category A Cat A tests are the best-defined test cases and must be done by a Bluetooth Qualified Test Facility (BQTF) or a Bluetooth Recognized Test Facility (BRTF). The tests that are run will generate a test report that will be stored in the compliance folder.

Category B Cat B tests are run by the product company according to the test requirements. The test setup, test evidence, and test results are stored in the compliance folder.

Category C Cat C tests are also run by the product company, but the documentation requirements are significantly reduced. Cat C tests have to be declared that they have been performed and the test result must be captured, but no test evidence is required.

Category D Cat D tests are optional. There is no requirement to run these tests and there is no need to document that they have been run, that the results have been captured, or to store any test evidence.

Category D tests might appear to be mostly pointless, but all tests can move between categories as the tests mature or problems are found. For example, a new specification might start with all tests being at Category C, but once the maturity increases, these can move to Category B. Category B tests might be implemented by a test-equipment manufacturer that validates these tests, and, therefore, these might move up to Category A.

A test might be moved to Category D when it has known unresolved issues, and thus, the SIG cannot reasonably require that it be run. However, you should keep in mind that when you qualify, you declare compliance to the specification. Test cases are merely a useful way to check that compliance. So, even if a test case is broken, products must still correctly comply with the feature it was designed to test.

It is possible to raise test case errata on all the tests in the qualification program if the test case or the expected outcomes are invalid. The core specifications, not the test specifications, are the determining factor in deciding whether something is a valid test. It is therefore possible to take a Category A test, prove that it does not implement what is required in the core specification, and still qualify your product because you can still provide the test evidence required. This doesn’t happen very often, but it is a useful safety valve.

15.7. Qualify Your Design

Once the testing has been completed, the compliance folder should be full of test reports. The next step is to qualify your design. To do this, you need a Qualified Design Identifier (QD ID). This is a unique number that refers to your design. This does not refer to an end product; this will come later. Instead, you qualify a design that can be used in multiple end products. For example, you might design a product that can be sold under multiple brands, with each brand using its own packaging. The qualification of the design allows the parts of the product that use Bluetooth wireless technology to be used by many end products. It does cost money to qualify a design, but this design can be used in many end products with no additional qualification charges.

Once the QD ID has been obtained, the test declaration is uploaded to the Bluetooth SIG Web site. This test declaration includes all the test reports from the validated testers as well as any additional Category C or Category D tests that you performed directly. Also, the profile tuning suite reports should be uploaded separately. These are directly parsed and the set of profiles and services that your device will support is immediately verified.

15.8. Declaring Compliance

Once the Bluetooth SIG has all the information about the design—what it does, the tests that have been run, and the test results—the design can be qualified. The member company that has designed the product now completes a Declaration of Compliance (DoC). This is a legally binding document that states that the member declares that the design complies with the Bluetooth specifications referenced, that the product implements the reference design, and that the product meets the testing and documentation requirements of the defining process document. This whole process is defined by the Qualification Program Reference Document (PRD). Essentially, by signing this declaration of compliance, the member has said that it has done everything necessary according to the process.

Because the DoC is a legal document, it can only be signed by someone who has the authority to enter into a legal commitment for the company. This means an officer of the company or someone who has been delegated the authority to make legal commitments by an officer of the company must be the signer.

The DoC must include the QD ID, the product information, the feature set of the product, the Bluetooth specifications being referenced, and the version of the PRD and test specifications used, as well as the test case reference list used to determine the set of tests required and their categories.

Once this DoC has been signed by a member company, the product is covered under the royalty-free license, and the Bluetooth brand can used in association with this product.

All products that have been qualified must have the QD ID clearly marked on the device or in its documentation. This allows anybody to determine the set of features and, therefore, the requirements of the design.

15.9. Listing

Once a member company has a compliant product with a QD ID, a listing is next performed. This listing does not need to be made public immediately because this can have an impact on the marketing plan for the release of the product. But a product must be listed within three months. The product compliance is not considered to be effective until the product is listed; therefore, you must list your product before offering it for sale or distribution.

15.10. Combining Components

It is possible to combine multiple listings into a single product. For example, an end-product listing might use a controller chip from one manufacturer, a host stack from another, one service from another supplier, and another service from yet another. This product would, therefore, have five QD IDs: one for the controller, one for the host, two for the services, and one for all the QD IDs in combination.

It should also be noted that each of these qualified designs can themselves be a combination of multiple, qualified components. The host in the previous example might have a separate QD ID for each layer in the design: one for L2CAP, one for ATT, one for GATT, one for GAP, one for SM, and one for this combination of components as a host subsystem. This seemingly simple end product would therefore ultimately reference nine different QD IDs.

This tree of QD IDs is a very powerful concept; it allows packaged components or subsystems to be designed and qualified separately and then delivered as a complete design. These components or subsystems can be reused in many products, again and again, without paying any additional testing costs or listing fees. For devices that use many existing parts, the costs for creating a new product can be negligible.