Index
A
Accountability, 8, 16, 27
Accuracy, of data collection methods, 38
Acquisition costs, 68
Acting consciously, 107
Action plans, 43, 94–96, 182
Activities, 93
Activity-based approach, 27
Announcements, 87
Appendices, of detailed report, 80–81
Application and implementation (Level 3)
case study examples of, 128, 136–138
data, 17–18, 37
key questions for, 18
Application guide, 96
Application objectives, of technology-based learning programs, 32–33
Application tools, 94–97
B
Bad news, 83–84
Benefit-cost ratio (BCR), 16, 69, 157, 192
Best practice meetings, 76
Blended learning, 5, 92, 106
Bring your own devices, 6
Brochures, 77, 88
Business alignment
achieving of, 28–34
case study of, 110
process of, 30
program evaluation, 34
program objectives, 31–34
stakeholder needs, 28–29, 31, 162, 207–208
Business impact (Level 4)
case study examples of, 128, 138–140, 202
data, 18, 37
detailed report description of, 80
importance of, 50
isolating the effects of learning programs applied to, 50
key questions for, 19
Business results, 8–9, 49–50
BYOD. See Bring your own devices
C
Case studies
communicating the results through, 77, 94
description of, 22
e-learning, 149–159, 195–208
English-as-a-second language program, 161–175
financial services company, 149–159
Future-Tel, 125–147
mobile learning, 177–194
Performance Medica, 161–175
PolyWrighton, 105–123
publishing of, 82
Transoft Inc., 177–194
Client, communicating the results to, 76
cMOOCs, 7
Coaching role, 101
Collaborating, 107
Communication of results
announcements for, 87
audience considerations, 75–76
brochures used for, 77, 88
case studies used for, 77, 94
case study examples of, 120, 158, 174, 193
electronic media used for, 77
evaluation of, 83
formal reports used for, 77
importance of, 24, 73
internal publications used for, 77
media for, 76–77
meetings used for, 76
memos for, 88
reasons for, 74
reports used for. See Report(s)
workbooks for, 88
Comparison group analysis, 52
Conclusions section, of detailed report, 80
Contamination, of control groups, 52
Control group design, 51
Cost(s)
case study examples of, 191–192, 203–204
of data collection methods, 40
of e-learning, 154–156
fully loaded, 67–69, 80, 112, 140–141, 191, 205
of technology-based learning programs, 67–69
Cost of quality, as standard value, 64
Creative Commons, 7
Customer satisfaction, 61, 206
D
Data
application and implementation, 17–18
business impact, 18
failure to use, 84
hard, 60–61
intangible, 61–62
lack of, 11–12
learning, 17
Level 1. See Level 1 data
Level 2. See Level 2 data
Level 3. See Level 3 data
Level 4. See Level 4 data
from participants, 46, 55–59, 75
from participants’ managers, 46, 75–76
reaction and planned action, 17
reporting of, 24
soft, 60–61
sources of, 44–47
tangible, 61–62
Data analysis
conversion of data to monetary value. See Data conversion
description of, 23–24
Data collection
case study examples of, 130–133, 151–152, 165–166, 181, 183–185, 199–201
from databases, 47
description of, 23
from executives, 47
importance of, 48
Level 1, 41, 47, 113–114
Level 2, 41, 47, 113–114
Level 3, 41, 47, 113–114
Level 4, 41, 47, 60, 113–114
plan for, 35–37
response rates for, 44–45
from senior managers, 47
sources of, 44–47
timing of, 47–48
Data collection methods
accuracy of, 38
action plans, 43, 94–96, 182
cost considerations, 40
focus groups, 40, 43
improvement plans and guides, 96
interviews, 40, 42–43
overview of, 41
performance contracts, 43, 96–98
performance records, 43–44, 46
questionnaires, 40–42, 44
reliability of, 40
repeatability of, 40
time requirements, 40
utility of, 40–41
validity of, 40
Data conversion
case study examples of, 119, 167, 191
description of, 62
estimations for, 66
external databases used in, 65
external experts used in, 65
historical costs used in, 65
internal experts used in, 65
linking of measures for, 65
standard values used in, 62–64
steps involved in, 66–67
success factors, 62
Databases, 47, 65
Detailed reports, 78–81
Development of solutions costs, 68
Digital content, 4
Domain massive open online courses, 8
During-the-program activities, 101
E
E-learning
case study examples of, 149–159, 195–208
description of, 4
factors that affect, 5
Electronic media, for communication of results, 77
Employee engagement, 61
Employees’ time, as standard value, 64
English-as-a-second language program case study, 161–175
Estimations, for data conversion, 66
Evaluation
costs associated with, 68–69
planning of, 23, 150–156
Evaluation framework, of ROI Methodology, 17–19
Evaluation methodology section, of detailed report, 79
Evolving, 107–108
Executive summary, 82
Executives
description of, 47
expectations and, 91–92
technology-based learning and, 9–10
Expectations, 90–92
Expert estimation, for isolating the effects of technology-based learning programs, 55–59
External databases, 65
External experts, 65
F
Facilitator-led learning, 10
Financial services company case study, 149–159
Focus groups, 40, 43, 56
Forecasting
advantages of, 55
disadvantages of, 55
isolating the effects of learning programs through, 54–55
technology-based learning programs, 25
Formal reports, 77
Fully loaded costs, 67–69, 80, 112, 140–141, 191, 205
Future-Tel case study, 125–147
G
Game-based learning, 6
General audience reports, 82
H
Hard data, 60–61
Historical costs, in data conversion, 65
I
Impact objectives, of technology-based learning programs, 34, 93
Implementation, 22–23, 28, 68
Improvement plans and guides, 96
In-person interviews, 42–43
Initial analysis and assessment costs, 68
Intangible benefits, 24, 70–71, 80, 119–120, 142–143, 156, 192, 206
Intangible data, 61–62
Internal experts, 65
Internal publications, for communication of results, 77
Internet, 4, 65
Interviews, 40, 42–43
Isolating the effects of learning programs
case study examples of, 118, 189–190, 203–204
comparison group analysis for, 52
expert estimation for, 55–59
forecasting methods for, 54–55
importance of, 49–50
Level 4 application of, 50
selecting techniques for, 60
trend-line analysis for, 53–54
J
J4 approach, 92
Job aids, 97
K
“Knowledge broadcast” model, 4
L
Lean Six Sigma, 90
Learning
accountability for, 27
activity-based approach to, 27
blended, 5, 92, 106
facilitator-led, 10
flow of, 92
game-based, 6
mobile. See Mobile learning
results-based approach to, 27
ROI Methodology benefits for, 25
technology-based. See Technology-based learning
Learning (Level 2)
case study examples of, 128, 135–136
data, 17, 36
key questions for, 18
Learning and development team, 75
Learning Everywhere: How Mobile Content Strategies Are Transforming Training, 8
Learning management systems, 4–5, 150
Learning needs, 31
Learning objectives, of technology-based learning programs, 31–32
Learning program(s)
acquisition costs, 68
application costs, 68
business needs addressed in, 29, 31
costs of, 67–69
development of solutions costs, 68
evaluation costs, 68–69
forecasting of, 25
fully loaded costs of, 67–69, 80, 112
implementation costs, 68
initial analysis and assessment costs, 68
input needs addressed in, 31
intangible benefits of, 61–62, 70–71, 119–120, 156
justifying spending on, 24–25
learning needs addressed in, 31
maintenance and monitoring costs, 69
manager support for, 26, 75–76
objectives of, 28, 31–34
overhead costs, 69
payoff needs addressed in, 29
performance needs addressed in, 31
preference needs addressed in, 31
questions to ask before implementation of, 28–29
reporting costs, 68–69
stakeholder needs and, 28–29, 31, 162, 207–208
supervisor support for, 25
support costs, 69
support for, 25–26
tangible benefits of, 61–62
unsuccessful, 25
Learning program evaluation
business alignment and, 35
data collection plan for, 35–37
plan for, 34–38
ROI analysis plan, 35, 38
Learning technologies, 4
Learning transfer, 79
Level 1 data. See also Reaction and planned action
application of, 85
case study examples of, 110–111, 115, 128–129, 164, 199
collection of, 41, 47, 113–114
Level 2 data. See also Learning
application of, 85
case study examples of, 110–111, 115–116, 128–129, 164, 188, 199
collection of, 41, 47, 113–114
Level 3 data. See also Application and implementation (Level 3)
application of, 85
case study examples of, 110–111, 114, 116, 128–129, 165, 188, 202
collection of, 41, 47, 113–114
Level 4 data. See also Business impact (Level 4)
application of, 85
case study examples of, 110–111, 114, 116, 118–119, 128–129, 165, 202
collection of, 41, 47, 60, 114
Level 5. See ROI (Level 5)
Likert scale questions, 42
M
Macro-level reporting, 24
Macro-level scorecards, 82–83
Maintenance and monitoring costs, 69
Managers
coaching role of, 101
communicating the results to, 75–76
expectations and, 91
involvement of, 99–101
learning program support from, 75
participants’, data from, 46, 75–76
senior, 47
technology-based learning program support from, 26
Marketing programs, 49
Massive open online course
description of, 7–8
domain, 8
Measurement, 15–16
Measures
case study examples of, 164
conversion of, to monetary value, 62–66
identifying of, 90–91
linking of, 65
Meetings, for communication of results, 76
Memos, 88
Mobile learning
business benefits from, 9
case study of, 177–194
description of, 5–6, 92
Monetary value, converting data to, 62–66, 119, 167
MOOC. See Massive open online course
N
Need for the evaluation section, of detailed report, 78
Need for the program section, of detailed report, 78
O
Objectives, of technology-based learning programs, 28, 31–34, 93, 180
OER. See Open educational resources
Open Badges framework, 6
Open educational resources, 6–7
Operating standards, 20
Output to contribution, as standard value, 64
Overhead costs, 69
P
Participants
business measures defined by, 90
communicating the results to, 75
data from, 44–47, 55–59
peers of, 46
role of, 88–90
Participants’ managers
communicating the results to, 75
data from, 46
involvement of, 99–101
Payback period, 70
Payoff needs, 29
Payoff period, 70
Peers, data from, 46
Performance contracts, 43, 96–98
Performance Medica case study, 161–175
Performance needs, 31
Performance records, 43–44, 46
PolyWrighton case study, 105–123
Post-program activities, 101
Preference needs, 31
Preprogram activities, 100
Preprogram forecasts, 25
Priority setting, 25
Problem solving, 93–94
Process learning, 120–121
Q
Questionnaires
data collection use of, 40–42, 44
isolating the effects of learning programs as focus of, 57–59
R
Reaction and planned action (Level 1)
case study examples of, 128, 135
data, 17, 36
key questions for, 18
Reaction objectives, of technology-based learning programs, 31–32
Reflecting, 107
Reinforcement tools, 101
Reinforcement workshops, 101
Relevance-based design, 92
Reliability, 40
Report(s)
detailed, 78–81
executive summary, 82
general audience, 82
macro-level scorecards, 82–83
single-page, 82
Reporting of results
audience considerations, 75–76
brochures used for, 77, 88
case studies used for, 77, 94
costs associated with, 68–69
electronic media used for, 77
evaluation of, 83
importance of, 24, 73
internal publications used for, 77
media for, 76–77
meetings used for, 76
reasons for, 74
reports used for. See Report(s)
Response rates, for data collection, 44–45
Results
case study examples of, 169–176, 188–192, 205–206
communication of. See Communication of results
e-learning program, 156–158
technology-based learning program design for, 87–102
Results-based approach, 27
Results section, of detailed report, 79–80
Return on investment. See ROI
ROI
benefit-cost ratio versus, 69
calculation of, 24, 69–70, 141–142, 157, 192, 206
case study examples of, 179
definition of, 16–17
equation for, 16, 70, 141–142
key questions for, 18–19
rationale for, 179
0 percent, 33
ROI (Level 5), 110–111, 114, 119, 128–129, 165, 205
ROI analysis plan
case study examples of, 112, 134, 153, 186–187, 199
description of, 35
key areas of, 38
sample, 39
ROI Methodology
benefits of, 24–26
case applications and practice, 22, 109. See also Case studies
evaluation framework, 17–19
in executive summary, 82
guiding principles of, 20, 22
implementation of, 22–23
learning benefits of, 25
operating standards, 20, 22
process model of, 19, 21
reporting. See Reporting of results
spending on technology-based learning programs justified using, 24–25
summary of, 26
ROI objective, of technology-based learning programs, 33
ROI process
case study example of, 198–199
chain of impact in, 20
data analysis, 23–24
data collection, 23, 36–37. See also Data collection
evaluation planning, 23
model of, 19, 21
reporting, 24
ROI studies
communicating the results of. See Communication of results
costs of, 40
Future-Tel case study, 125–147
Role of participants, 88–90
S
Sales training programs, 140–141
Self-coaching skills, 107–108
Self-managing, 107
Senior managers, 47
Sequencing of materials, 92
Simulations, 93
Single-page reports, 82
Six Sigma, 90
Skill practices, 93
Skype, 5
Soft data, 60–61
Spending on technology-based learning, 24–25
Stakeholders, 28–29, 31, 162, 207–208
Standard values, 62–64
Structured interviews, 42
Succession planning, 162
Support costs, 69
T
Talent management, 5
Tangible data, 61–62
Technology, 3
Technology-based learning, 1–13
barriers to, 11–13
business results of, 8–9
concerns about, 8–13
executive view of, 9–10
lack of data about, reasons for, 11–12
programs for. See Learning program(s)
results of, 10–11, 11–12
Technology Learning Competency program, 197–198
360-feedback evaluation, 46
Time
for control group comparisons, 52
data collection methods, 40
Transfer tools, 97, 99
Transoft Inc. case study, 177–194
Trend-line analysis, 53–54, 112
U
Udell, Chad, 8
Unstructured interviews, 42
V
Validity, 40
Video streaming, 5
Virtual classrooms, 5
W
WebCT, 5
Women, 3
Work engagement, 106–107, 119
Workbooks, 88
X
xMOOCs, 7