CHAPTER 6: AGREE REVIEW CRITERIA AND TEAM

Once you’ve issued the RFI, you’ve then got a few weeks to plan how you will deal with the responses. In this period there are a few useful things you can do to prepare, such as getting the team ready, agreeing the criteria and drafting a scoresheet.

This chapter covers:

• agreeing the review team
• determining the review criteria
• options for reviewing, including weighting the criteria
• example scoresheets
• dealing with the suppliers.

Agreeing the review team

Generally, the review team will be the core selection team. However, you may want to include others if appropriate. For example, there may be a specialist element to the system/service, for which you need a person with the relevant expertise to be involved.

One option is to get everyone on the team to review the whole of every response. This can be a sensible choice if the selection team is fairly small and the longlist is not too long. However, as the numbers of people and responses increase, so does the time needed to do the reviews. You may recall an estimate of an hour per response for each person, so you can see how quickly the effort required will escalate.

One of the ways to make best use of the people and time available is to split up the criteria, so that each person is only reviewing a subset. This can be especially useful if your team come from different areas and so are interested in different parts of the system/service anyway. For example, if you were selecting a new supplier for your landlines, the telecoms person would be interested in the call options, the finance person would be interested in the billing and the admin person would be interested in how new lines are ordered.

Even if you are going to split up the criteria, at least one person should review each response in its entirety, to ensure continuity and to make sure nothing is missed. It’s important to note that you can’t split the responses up between the team (e.g. each person reviewing three); it would be very difficult to ensure a fair comparison between them.

A variant on this approach is for one person to review all the responses and summarise them for the rest of the team. There is an example ‘summary of responses’ document template provided in Chapter 7 which can be used for this purpose. It would be useful for a second person to check the summary to ensure that no bias has been introduced. The other reviewers can then read through the summary and just look at specific elements of interest in the responses themselves.

Whichever method you choose, there are three key points to remember:

• Every question you asked in the RFI needs to be reviewed by someone.
• Each person needs to review all the responses (whether wholly, in summary or for a criteria subset).
• Each person needs to use a fair and consistent method for review.

This last point is covered in more detail in the next two sections.

Determining the review criteria

The main aims for agreeing a set of review criteria are:

• to make sure that all important elements are reviewed
• to encourage consistency between reviewers.

How long and detailed the review criteria are depends very much on what it is that you are selecting. The main groups can follow how you grouped your questions in the RFI (like section six in the template in Chapter 4).

For the ‘Costs and timescales’ and ‘Company information’ sections, you could have one criterion for each question asked. You may wish to add an extra element to the cost section, to show a cost over three or five years (you’ll have to work this out if you’ve not included it as a specific question); this is especially useful if you are comparing hosted versus in-house systems, as the pricing models are quite different.

It’s worth considering the company’s financial summary; you can see how good a match they are in size for your organisation (you want to be big enough to matter to them, but not so large they will struggle to service you if there are peaks in demand). Is it a concern if you would be their largest customer? Or their smallest?

The ‘Product’s fit to requirements’ section can be tackled in a number of ways. You can have a criterion for each group of requirements (e.g. reporting, service level agreements) or you could have a far more detailed set, perhaps for some of the key (maybe the mandatory) requirements.

The next section should help you decide how detailed you want to be.

Options for reviewing

There are three main options for reviewing and writing up your assessments:

• Narrative assessment
  As implied, this means that you use narrative notes against each criterion without trying to give them an actual score.
• Basic scoring
  In this method, you give each criterion a score. You need to agree what range to use and roughly what each score means. For example, do you use ‘0’ or ‘1’ if it’s not covered at all? What if a criterion is only partially provided, but is done very well? The key thing is to make sure people apply scoring consistently across all their reviews.
  The advantage to this method is that you can add up the scores and compare results more easily.
• Weighted scoring
  Similar to basic scoring, but giving more weight to the most important elements. The easiest way to do this is to give those elements a larger range, for example, most criteria are scored out of five, while the most important ones are scored out of 10. The total results can then still be compared.

You can also use a mixture of methods. For example, if you have one person summarising the responses for the team, they could use a narrative method, then the rest of the team could use basic or weighted scoring.

Don’t be surprised if people come up with different total scores. The important thing is that each person scores consistently across their reviews. Some will undoubtedly be harder markers than others!

Example scoresheets

Narrative

This is an example for use if you are doing ‘narrative’ type reviewing. Put all your criteria into the ‘Description’ column; it may be helpful to group them as shown in the example. As you review each response, you add a short note or bullet list into each cell. It can be easier to use a spreadsheet for this, so that you can expand columns as necessary.

Table 3: Narrative criteria sheet

Basic scoring

This is not very different to the narrative sheet, except that each supplier has two columns, one for the score and one for any notes or comments.

You will also have a ‘Totals’ line at the bottom. Again, using a spreadsheet will help, as you can use a formula to add up the scoring columns.

This example shows the last section with the totals:

Table 4: Basic scoring sheet

Weighted scoring

As mentioned above, the easiest way to do this is simply to allow your priority items a higher maximum score. For example, if you don’t have much data to transfer into a new system, but you’re keen that ongoing support is good, you may score ‘Data migration approach’ out of five and ‘Levels of support’ out of 10.

The scoresheet can, therefore, look the same as the ‘Basic scoring’ one above.

You may want to add the total score allowable for each criteria item, perhaps in brackets at the end of the description, or maybe as an additional column between ‘Description’ and ‘Supplier 1’.

Dealing with the suppliers

During this period, you may also need to communicate with the longlist suppliers. They may contact you to ask questions, or you may need to chase them if they do not send a response.

So, how do you act if any of the suppliers ask you additional questions? Generally, it’s fine to simply answer the questions. However, sometimes you should send details of the answer to all the others on the longlist as well. This may be appropriate if a question makes you realise:

• you’ve missed something significant out of your RFI
• you’ve got something wrong
• your wording is unclear
• or anything else that could affect the way others would answer the RFI.

Generally, you should take it as a good sign when suppliers ask additional questions – it shows they are really interested! Keep a record of all the questions and answers as you may need to refer to them when you are reviewing the responses.

But what do you do if one of them asks for a meeting (face to face or phone)? At this stage, it’s not usually necessary to do this. Bear in mind that if you meet one supplier, you should really meet them all to keep things fair. Remind the supplier that this is just the first stage and, if they are taken forward, they will have opportunities for meetings at that point.

As the response deadline approaches, you will probably start to receive the responses. Always acknowledge receipt, and keep tabs on which ones have come in. It can be a dilemma whether to chase for any who don’t make the deadline. If you had quite a long list and have received a good number of responses, you may not feel it necessary to chase for the last few. However, if you’ve not received very many, or one that you felt was a very good prospect is missing, it can be worth chasing. You may find that a key person is off sick or the e-mail has gone astray (or maybe it was over your e-mail size limit so got bounced!).

As the responses are received, it’s worth putting them into a shared area on your network where the whole review team can access them. That way, if some arrive early people can spread their reviewing over a few more days. Try to avoid e-mailing them out to everyone; some responses can be very large!