7
KNOWLEDGE TRANSLATION AND INFORMATICS IN HEALTHCARE
Contents
Strategies for Improving Knowledge Translation in Healthcare
Areas of Health Professional Needs That Can Be Addressed by Health Informatics
Clinician and Patient Deficits in Knowledge and Skills
Introduction
Knowledge translation is the process whereby research findings, or any new and worthwhile knowledge and insights, are integrated into everyday practice. Louis Pasteur summed up this important aspect of research and development by saying: “To him who devotes his life to science, nothing can give more happiness than increasing the number of discoveries, but his cup of joy is full when the results of his studies immediately find practical applications.”
Pasteur did not, however, include in this statement how important the movement of research into practice is for those of us who stand to benefit from the research findings—or how difficult it is. In the healthcare world, estimates of the size of the gap between what we know about the best possible care from research and what we receive from health professionals is significant. Estimates of the size of the gap translate into the fact that on average Americans receive only 55% of their recommended healthcare services.1 This care gap comprises both care that is received that should not be given and care that should be administered and is not.
Woolf and Johnson make a strong statement about the magnitude of the effects of not using health research efficiently.2 They provide substantial evidence that suggests we need to put more money and effort into determining the most effective methods of implementing existing knowledge rather than funding additional basic and clinical research and other projects designed to discover new knowledge. Their estimate of the relative amount of research investment between basic science and other research and implementation is likely 100:1, favoring the discovery of new knowledge. Until we can ensure that new and important knowledge is integrated into our healthcare system in a timely manner lives will be lost, care will be substandard, funding will not be optimized, and the healthcare system will continue to be inefficient and possibly even dangerous.
The gap between best possible care and actual care, or between evidence and practice, occurs because of many factors:
• Ever increasing growth of new knowledge that leads to information overload
• Growth in the complexity of healthcare and the need to document this carefully. The potential for litigation is also a factor here
• Increasing demands on the time clinicians have to interact with patients (For this chapter the term clinician refers to a health professional who helps make clinical decisions for and with patients—usually physicians, nurses, and pharmacists, although others are often included.)
• Structural issues related to disincentives to improve care (e.g., financial penalties to provide advice by email rather than clinical visits)
• Organizational issues related to matters such as an inappropriate skill mix to address patient needs, lack of facilities, inadequate staffing, or outdated equipment
• Peer group pressures (e.g., local standards that tend toward less than optimal care)
• Individual deficits in health professional knowledge, skills, and attitudes
• Individual deficits in patients’ and families’ knowledge, skills, and attitudes
• Patient expectations that are not aligned with best care (e.g., direct to consumer advertising or expectations related to antibiotics)
• Basic human nature and our inclination toward resistance to change
Many people across disciplines, domains, countries, and time periods are interested in optimizing knowledge translation to close the gap between evidence and practice. Consequently multiple terms referring to knowledge translation exist (see http://whatiskt.wikispaces.com/) for a list of 100 terms related to the concepts of knowledge translation). For example, knowledge translation is a Canadian term. Other term examples include translational science/medicine in the United States, implementation science in the United Kingdom, research utilization in Australia, quality improvement/assurance in hospitals, and evidence-based practice/medicine/nursing in clinical care. For this chapter we use the term knowledge translation and we adhere to the term as defined by the Canadian Institutes of Health Research (CIHR):
Knowledge translation is a dynamic and iterative process that includes synthesis, dissemination, exchange and ethically sound application of knowledge to improve the health of Canadians, provide more effective health services and products and strengthen the healthcare system.
This process takes place within a complex system of interactions between researchers and knowledge users which may vary in intensity, complexity and level of engagement depending on the nature of the research and the findings as well as the needs of the particular knowledge user. (CIHR website: http://www.cihr-irsc.gc.ca/e/29418.html)
In its role to improve knowledge translation, CIHR has supported publication of a 2009 book on knowledge translation in the health sciences that can be used by those who want more information on the topic.3
Knowledge translation and its techniques are vitally important for researchers as well as developers and entrepreneurs in all knowledge domains. This chapter concentrates on knowledge translation and health information technology. Shekelle and colleagues summarized published evidence and showed that health information technology can be cost effective in improving the quality and efficiency of health-care. These authors also present challenges to the health informatics community.4 Their report is summarized as follows.
Health information technology (HIT) has the potential to enable a dramatic transformation in the delivery of healthcare, making it safer, more effective, and more efficient. Some organizations have already realized major gains through the implementation of multifunctional, interoperable HIT systems built around an EHR. However, widespread implementation of HIT has been limited by a lack of generalizable knowledge about what types of HIT and implementation methods will improve care and manage costs for specific health organizations.
The following sections of this chapter describe the interventions and actions that are most likely to improve and speed the flow of research evidence into practice and how informatics applications can enhance the likelihood that patients and their families obtain the best care possible.
The informatics world needs to understand how to optimize knowledge translation for two reasons. First we need to produce tools and services that will best serve the needs of health professionals to optimize their care process. We need to produce systems that can ensure quick, efficient, and accurate implementation of appropriate new advances and promote or block discontinuation of outdated care. Once we produce these tools and services we must work to insure that they are adopted, implemented, and sustained by the intended users, utilizing the most effective change strategies. Without this knowledge translation by informatics practitioners our tools and projects will not be used to their fullest potential.
Strategies for Improving Knowledge Translation in Healthcare
Which tools and techniques are effective for behavior change of clinicians and their institutions and organizations? Knowledge translation in the domain of healthcare deals with changing the behavior of individual clinicians and how clinics, hospitals, cities, municipalities, states and provinces, and even countries respond to the challenge of making health decisions based on evidence in a cyclical, ongoing manner so that care provided is the best currently available. Once we know what changes behavior, we can determine if and how the informatics applications we implement can enhance these interventions. Bero and colleagues summarize the evidence on interventions or actions that close the gap between research and practice.5 They summarize their findings of 18 separate review articles assessing interventions for health professionals into three categories: what works, what might work, and what does not work. Although the article by Bero et al. is somewhat dated, the general categories of what works and does not work remain applicable.
The first category is interventions that consistently change behavior: those that are supported by strong evidence that shows improved care across multiple studies and time periods.
• Educational outreach visits of an expert to discuss certain aspects of care, especially prescribing issues in North America. The drug companies use this technique quite effectively. Agnell cites data showing that 35% of the staff of pharmaceutical companies are employed in marketing departments, many of whom visit physicians, sponsor conferences, or provide educational sessions to emphasize drug adoption.6
• Computerized or manual reminders of care to be done or not done for a patient in a specific situation
• Multifaceted interventions (combinations of two or more of audit and feedback, reminders, local consensus processes, or marketing)
• Educational programs that are interactive—those that require discussion, role playing, or practice by health professionals in workshops or similar programs of active learning
The second group of interventions are those that show variable effectiveness. In some instances studies of these interventions have shown positive changes and some have been associated with no changes or detrimental outcomes.
• Audit and feedback of actual performance of the individual, often compared with group norms (e.g., surgical infection rates for a give individual that are 10% higher than those of the other surgeons in the hospital)
• Use of local opinion leaders (those in a group, such as a hospital or multiphysician clinic who are identified by their colleagues as being asked often for advice or guidance in care situations)
• Local consensus processes that involve practicing clinicians in a geographic area or group who work together to establish care norms or procedures for a specific situation or care issue. These documents have been called care maps or clinical practice guidelines. The can be for a small group such as a hospital ward or reflect the standard of care for national practice.
• Patient interventions whereby information is provided to patients, who in turn use the information to change clinician knowledge or behavior, for example, direct to consumer marketing
The third category is also important. These interventions have little or no effect on clinician behavior.
• Educational materials in paper or electronic format (not individualized)
• Didactic educational meetings, for example, lectures, and most standard conferences with featured speakers or researchers presenting papers to peers in lecture format
Areas of Health Professional Needs That Can Be Addressed by Health Informatics
The remainder of this chapter discusses challenges that clinicians face in improving their care of patients (ensuring the best possible care) and how various existing or new health information technologies can aid the knowledge translation process. The chapter ends with cautions for informaticians.
Information Overload
Information overload for clinicians is a real dilemma. The problem of too much information is evident on a number of fronts in the clinical world. Clinicians deal with approximately 11,000 different diseases and conditions, and most of these have multiple overlapping signs and symptoms. (For clinicians, symptoms are what patients report during a clinic visit and signs are what clinicians find when using their hands and tools such as thermometers, blood pressure cuffs, and stethoscopes.) Adding to this information that a clinician needs to synthesize and use for decision making is the information from the published literature. Medline, the major health database, contains approximately 17 million citations and publishes more than 12,000 articles per week. Of these, 300 are likely to have information relevant to clinical care.7 In addition, the web contains an estimated 25 billion pages as of July 2009. All of these factors provide tremendous pressure and tax an individual’s ability to cope. One way to alleviate this information overload is through the utilization of health informatics applications.
Point-of-care information systems are designed to address the flow of new information or can provide information backup when a clinician feels that he or she needs more information. These point-of-care systems provide access to e-resources using several communication means including hand-held devices. Prendiville et al. describe a study of Irish pediatricians using hand-held devices to answer questions that arose during hospital care.8 The summarized conclusions that follow depict the importance of these point-of-care informatics resources for this group of physicians.
The study received 156 completed questionnaires, a 66.1% response. 67% of pediatricians utilized the internet as their first “port of call” when looking to answer a medical question. 85% believe that web-based resources have improved medical practice, with 88% reporting web-based resources are essential for medical practice today. 93.5% of pediatricians believe attempting to answer clinical questions as they arise is an important component in practicing evidence-based medicine. 54% of all pediatricians have recommended websites to parents or patients. 75.5% of pediatricians report finding it difficult to keep up-to-date with new information relevant to their practice.
Another study of question answering using health information technology was conducted in Ottawa, ON. Hand-held devices were given to hospital physicians and physicians in training to provide information support. These devices were programmed to provide electronic communication to an information service provided by librarians.9 The librarian service was designed to provide fast and efficient answers and to determine if the twinning of the librarian with the mobile devices was more effective than with the devices with only access to online information resources. At the end of the study, more than 85% of the participants favored having direct access to a librarian to provide answers to clinical questions. Most of the physicians in the study felt that the service improved the care they provided. As another example of information support at the point of care, Cimino and colleagues have developed information systems embedded within electronic health records (EHR) systems. They provide strategically placed electronic “infobuttons” in systems in the EHR to indicate a probable information need. This might involve an issue such as an abnormal laboratory finding or some patient data suggesting a potential change in medication. The infobutton is actually a link to context specific information or another information resource that will likely address that abnormal finding or trend.10
EHRs also provide great value in helping clinicians with their information overload in other areas. A well-designed EHR will collect and integrate information from multiple sources such as hospitals, clinics, workplaces, and pharmacies for a given patient. A strong EHR will also enable important information to be collected just once and then make it available thereafter. For example, questions related to allergies, medications, home address, next of kin, and previous pregnancies are often asked and recorded multiple times during patient care encounters. This information should be collected once and then made available for all other recognized care providers. EHRs are also ideal tools for collecting and synthesizing sequential data and presenting it to depict time trends or abnormal patterns. Most chronic diseases such as heart disease, diabetes, and asthma rely on analyzing multiple data points and making decisions based on trends. The literature on EHRs is extensive: Häyrinen and colleagues have produced a systematic review of the definition, structure, content, use, and effects of EHRs.11
Clinical (or computerized) decision support systems (CDSSs) are also useful tools to help clinicians deal with information overload. CDSSs by definition are systems that integrate data from two separate sources. They store patient-specific information, entered either by the user or from EHRs. These patient-specific data are then integrated into a database of clinical knowledge rules or patterns (using an “inference engine”). This inference engine then produces suggestions for actions related to the diagnosis, treatment, or monitoring for that particular patient case—one of the main functions of CDSSs. The clinician can choose to act according to these recommendations or make other decisions. Open Clinical (an international organization that promotes the utilization of decision support and other knowledge-based technologies) provides a more detailed summary of the characteristics of CDSSs (http://www.openclinical.org/dss.html). Perreault and Metzger list the functions of CDSSs including the previously mentioned clinical support.12
Four key functions of [C]DSSs are outlined: (i) Administrative: Supporting clinical coding and documentation, authorization of procedures, and referrals. (ii) Managing clinical complexity and details: Keeping patients on research and chemotherapy protocols; tracking orders, referrals follow-up, and preventive care. (iii) Cost control: Monitoring medication orders; avoiding duplicate or unnecessary tests. (iv) Decision support: Supporting clinical diagnosis and treatment plan processes; and promoting use of best practices, condition-specific guidelines, and population-based management.
Haynes and his group have updated their review of the evidence of the effectiveness of CDSSs.13
In addition to the point-of-care information systems and CDSS, almost all informatics applications have components that assist clinicians with information overload. Many of the systems are designed to collect and analyze large quantities of evolving data across systems and from research studies or patients in general and their functionality continues to evolve. Areas of ongoing development for enhanced functionality for informatics include interoperability, intelligent information retrieval, automated or manual (people centered) mechanisms to keep information resources current with the standards of best practice, and better integration of research findings into EHRs.
Clinician and Patient Deficits in Knowledge and Skills
Clinicians, patients, and caregivers can have deficits in knowledge and skills that can impede obtaining, providing, or acting on the best possible care. For the general public, an example of the magnitude of these deficits is a review of the literature on public knowledge of the risks for and signs and symptoms of stroke. Strokes are common, carry considerable disability, and fast access to care lessens the potential suffering. In a study that addressed levels of knowledge in select communities regarding the risks of strokes, Nicol and Thrif found that between 20% and 30% of the people surveyed did not know a single risk factor for stroke despite considerable distribution of this information.14 Another example of clinician knowledge revealed that more than 75% of Australian family physicians felt that their knowledge of breastfeeding was inadequate.15 Informatics applications can identify gaps in knowledge and provide learning based on these gaps. For example, at the University of Pittsburgh residents are taught how to identify pathology abnormalities and conditions, as well as report writing, using intelligent tutoring programs. The system uses natural language processing and other informatics tools to identify when the residents show they have mastered a given set of content or if a student needs more practice in identifying certain diseases or content areas.16 Computer tutoring (individualized education) is on target to become even more highly used to educate undergraduate medical and nursing students and residents. Individualized educational programs that evaluate the learning achieved and direct further learning for patients have also been shown to be effective. For example, nursing researchers are building computer-medicated, Web-based, individualized, educational programs for women with ovarian cancer. Both format and content are based on the author’s analyses of data in 40 studies of computer education for women with cancer and chronic diseases.17
Fordis and colleagues showed that online continuing professional education is as valuable for providing education as in-person continuing education. Continuing education is vital to health professionals for two major reasons.18 First, because so much information is changing, health professionals must be lifelong learners to provide the best possible care. Second, health professionals must report formally on their learning to maintain their clinical certification. An added benefit of online courses and point-of-care learning systems over traditional learning is that the credential benefit (continuing education credits) of the course can be captured in a format suitable for submission to certifying bodies such as the American or Canadian Medical Associations. Some professional organizations are actively pursing automatic reporting of credit hours for certification via online systems. Keeping track of continuing education to support ongoing certification manually is time consuming and often inaccurate.
In addition to identifying knowledge gaps, health information technology can also detect errors. The U.S. Institute of Medicine has estimated that between 44,000 and 98,000 deaths occur in the United States annually because of errors in care.19 One of the most important areas in healthcare where errors occur, especially in hospital and long-term settings, is with medications. Errors can occur during ordering (for patients in hospitals) or prescribing (for patients meeting health professionals outside hospitals). Because children are given doses based on weights and ages, some people estimate that pediatric patients have prescribing errors rates three times as high as for adult patients.20 Ordering and prescribing can be improved using health information technology with technologies such as provider order entry systems (see later) to deal with poor handwriting and systems to block inappropriate dosing or prompt re-drug allergies. Dispensing can be improved by using bar code systems that notify the nurse giving the medicine that the bar code for the medication is not the same bar code that the patient has on his or her wrist. Monitoring of the patient to ensure that the drug is given in sufficient quantities to provide the proper effect and not cause adverse reactions is also available using information from EHRs. Agrawal provides more information on medication safety using health information technology.21 Another more specific example is a review by Eberts and colleagues who discuss the benefits of physician order entry on prescribing in pediatrics intensive care units.22 They found that the systems reduced prescribing errors but these errors did not reduce the rate of adverse drug reactions or mortality.
Time Pressures
Time is an important factor for current healthcare providers. Although clinicians perceive that EHRs, with or without personal health records (PHRs) components, consume more time than paper records they do offer some time-saving features.23 Examples where EHR or PHR systems or other devices such as tablets in waiting rooms can save time is if they enable patients to book their own appointments and provide data on current issues before clinic visits. The patients and caregivers can state the reason for the scheduled visit plus signs and symptoms, provide detailed lists of prescribed and over-the-counter medications, and update information on addresses and insurance coverage. The time that is available for the appointment can then be spent analyzing the data and formulating decisions and care plans. EHRs that integrate data from multiple caregivers and settings (e.g., primary care clinics, nursing homes, specialists, and hospitals) also save time during patient visits. Complete records can also alleviate the need for duplicate diagnostic or evaluation studies if all the patient data are successfully aggregated in a timely manner and available for all caregivers and all locations of care.
Computerized Physician (or Provider) Order Entry
Another informatics application designed to save time as well as improve information flow and reduce errors is computerized physician (or provider) order entry (CPOE) systems (also mentioned previously in the error section). Using a CPOE system, healthcare professionals place their orders for components such as medications, diagnostic tests, appointments with specialists or generalists, or discharge instructions online rather than verbally or on paper. These online orders are then quickly distributed without transcription and its inherent errors to those who can ensure that the tests are booked, carried out, and the results reported back to all who are involved. CPOE systems can not only speed care but also check and verify that the data are accurate, actions are appropriate, their booking is efficient, and communication is facilitated and recorded. A study by Wietholter et al. illustrates the time savings potential made possible by CPOE systems. The study compared processing time for prescriptions (time from the initial order by the physician or nurse until it is prepared and delivered by the hospital pharmacists) with and without automated systems. Processing time for prescriptions showed a mean time of 115 minutes before automation compared with 3 minutes to process the same drug order after the introduction of CPOE for prescribing.24
Workflow Applications
Workflow is vitally important for clinicians in all care settings and has often been overlooked or underappreciated by system developers. Workflow refers to the processes and their time sequencing when patients, healthcare professionals, and their system interact. Often physicians have different ways of working than do nurses. New systems, either online or not, must respect and adapt to these differences. For example, in a teaching hospital, the physicians in charge “round” each morning, bringing along the care team of medical students, residents, and sometimes pharmacists, social workers, or librarians. They meet as a team with each patient assigned to their care, ascertain his or her progress and needs, and plan for next steps. Each patient is completed and orders are given to address all of the needs—a patient-centered workflow. Nurses in hospital units deal with patients differently. For example, one nurse provides all of the medications for patients several times per day. The nurse wants the information in an EHR to be focused only on the needed medications and presented in patient order—needed medications for the patient in room 1 bed 1, for the patient in room 1 bed 2, and so on. Those who schedule diagnostic testing or assign operating room time to physicians or teams need to see the clinical information in another format as do the administrative team who maintain supplies and equipment. Informatics professionals must ascertain the flow of people, tasks, and information: the workflow and how it differs for different groups of people before planning for new systems.
Zahra Niazkhani of the Institute of Health Policy and Management in the Netherlands provides a summary of what is known about the effects of CPOE on workflow.25 Computerizing a system with existing poor workflow has been the downfall of many systems. Not recognizing and respecting the existing workflow as well as the culture of an organization has also proven costly and frustrating for system designers during implementation.
Computer Reminders
Manual or computerized reminder systems have been shown to be one of the most effective methods of improving clinical behavior, improving clinical practice or knowledge translation. Reminders originally were provided in the form of paper notes on paper charts for such things as missing childhood vaccinations, influenza shots, and mammographies. The next implementation came in the form of emails. These emails were often more general reminders such as notices of a new hand sanitizer solution system to be implemented into a hospital or to consider generic instead of brand name drugs. Paper and general email-based reminders are not as effective as those that are patient specific and delivered at the point of care from within an EHR. Shojania and colleagues produced a Cochrane Review of point-of-care reminders delivered by a computerized decision support system within an EHR. They reviewed 28 original studies that showed consistent increases in actions when the reminders were provided across a range of conditions, practitioners, and settings.26
Reminder systems within EHRs are important in many applications. The 28 studies in the Shojania review described the settings and content areas of the 32 comparisons in these studies:
Of the 32 comparisons that provided analyzable results for improvements in process adherence, 21 reported outcomes involving prescribing practices, six specifically targeted adherence to recommended vaccinations, 13 reported outcomes related to test ordering, three captured documentation, and seven reported adherence to miscellaneous other processes (for example composite compliance with a guideline).
Reminder systems can work for patients also. For example, Puccio and colleagues studied whether reminders via cell phones would improve adherence (taking their medications) with HIV and AIDS medication in adolescents and young adults.27
Producing New Knowledge from Data
Another opportunity for increasing efficiencies in healthcare is that data stored and generated by powerful integrated EHRs and PHRs, hospital information systems, and insurance collections can be mined via quantitative analytic techniques to extract information and produce new knowledge. This new knowledge can, and will, direct an increasing number of patient-specific decisions. Currently our limited data collection and analysis capabilities could create knowledge only on groups of people (i.e., populations) rather than individuals. These new data will likely have an impact on being able to predict prognosis for an individual (the likely path that a person’s disease or condition will take). This is very useful information for the patient, health professionals, and insurers. For example, data mining of information on 258 variables from 16,604 patients with heart–lung transplants showed associations between variables that could be used to predict survival in the patients where traditional statistical analyses of data sets with that many variables could not yield these results.28
In addition to data mining and knowledge discovery, advances in bioinformatics can link a person’s health and family history data with his or her genomic data for decision making. This information will allow for better targeting of treatments: some drugs work well for some people and not for others. Strong data collection and analyses will be able to sort this puzzle out. lhervet and colleagues provide a summary of the promise of tailoring drugs based on genomic data.29
The production of new data is not really a classical knowledge translation task but it is one that will become more important over time as our data collection methods become stronger and more genomic data are available. Our systems will continue to play an increasing role in the application of new and proven knowledge to maintain optimal healthcare, especially in the production and integration of new patient-specific information.
Challenges
Challenges exist in producing systems and projects designed to improve patient care through knowledge application. These challenges relate to the complexities of care, its ever changing and advancing knowledge base, and the size of the healthcare enterprise. We also are held back by funding limitations. Despite the significant resource allocation by governments into health information technology, there is still not ample funding to produce ideal systems that integrate across the multitude of people and organizations. We also must deal with multiple players; conflicting local, national, and international standards and regulations; and the fact that patients differ in their preferences, resources, and attitudes. Whenever clinicians, patients, or both seek individualized and tailored care, complexity creeps into our design specifications. Another challenge is that although our systems often change care processes and improve some intermediate outcomes such as knowledge or skills improvement, many of our projects are like those of Egberts et al.: The process changes do not always improve clinically important outcomes such as reducing mortality.22 Despite all of these challenges the opportunities are vast in the health informatics arena, where collaboration with our health professional peers will likely improve care.
Summary
Clinicians involved in healthcare struggle with changing practices, procedures, and the complexities of care in an ever changing world. Patients and their conditions, insurance providers, threats of litigation, and local organizational issues are factors that must be considered in managing resources in this dynamic environment. Clinicians need all the help that informatics have to offer in their quest to keep current with advances in care where effective implementation of new knowledge or knowledge translation is essential.
Enabling effective knowledge translation is an important and exciting challenge in the development in informatics. As we build and complete projects we need to think about the knowledge translation challenges our clinicians and other partners have in their quest to stay up to date and provide optimal care. We need to remember that some interventions such as timely, useful, and context-specific reminders; decision support systems and CPOE within EHRs; individualized and continuing audit and feedback; and combined interventions including patient training and educational interventions that allow interaction and reflection work well at improving care and bring care closer to the ideal.
This chapter was originally published in Healthcare Informatics, Improving Efficiency and Productivity, Taylor & Francis, New York, 2010.
References
1. McGlynn EA, Asch, J. Adams SM, Keesey J, Hicks J, DeCristofaro A, Kerr EA. 2003. The quality of health care delivered to adults in the United States. N Engl J Med 348(26):2635–45.
2. Woolf SH, Johnson RE. 2005. The break-even point: When medical advances are less important than improving the fidelity with which they are delivered. Ann Fam Med 3(6):545–52.
3. Straus SE, Graham ID, Tetroe J. 2009. Knowledge translation in health care: Moving from evidence to practice. Oxford: Wiley Blackwell and BMJ Publishing.
4. Shekelle PG, Morton SC, Keeler EB. 2006. Costs and benefits of health information technology. Evidence Report/Technology Assessment (Full Report). No 132 (pp. 1–71). Rockville, MD: Agency for Healthcare Research and Quality.
5. Bero LA, Grilli R, Grimshaw JM, Harvey E, Oxman AD, Thomson MA. 1998. Closing the gap between research and practice: An overview of systematic reviews of interventions to promote the implementation of research findings. The Cochrane Effective Practice and Organization of Care Review Group. BMJ 317(7156):4658.
6. Angell M. 2004. Excess in the pharmaceutical industry. CMAJ 171 (12):1451–53.
7. Glasziou PP. 2008. Information overload: What’s behind it, what’s beyond it? Med J Aust 189(2):84–85.
8. Prendiville TW, Saunders J, Fitzsimons J. 2009. The informationseeking behaviour of paediatricians accessing web-based resources. Arch Dis Child 94(8):633–35.
9. McGowan J, Hogg W, Campbell C, Rowan M. 2008. Just-in-time information improved decision-making in primary care: A randomized controlled trial. PLoS ONE 3(11e):e3785.
10. Del Fiol G, Haug PJ, Cimino JJ, Narus SP, Norlin C, Mitchell JA. 2008. Effectiveness of topic-specific infobuttons: A randomized controlled trial. J Am Med Inform Assoc 15(6):752–59.
11. Häyrinen K, Saranto K, Nykänen P. 2008. Definition, structure, content, use and impacts of electronic health records: A review of the research literature. Int J Med Inform 77(5):291–304.
12. Perreault L, Metzger J. 1999. A pragmatic framework for understanding clinical decision support. J Healthcare Inform Manage 13(2):5–21.
13. Implementation Science Series on Computerized Clinical Decision Support Systems. http://www.implementationscience.com/series/CCDSS (Accessed May 22, 2015).
14. Nicol MB, Thrift AG. 2005. Knowledge of risk factors and warning signs of stroke. Vasc Health Risk Manage 1(2):137–47.
15. Brodribb W, Fallon AB, Jackson C, Hegney D. 2009. Breastfeeding knowledge—The experiences of Australian general practice registrars. Aust Fam Physician 38(1–2):26–29
16. El Saadawi GM, Tseytlin E, Legowski E, Jukic D, Castine M, Fine J, Gormley R, Crowley RS. 2007. A natural language intelligent tutoring system for training pathologists: Implementation and evaluation. Adv Health Sci Educ Pract 13(5):709–22.
17. Dumrongpakapakorn P, Hopkins K, Sherwood P, Zorn K, Donovan H. 2009. Computer-mediated patient education: Opportunities and challenges for supporting women with ovarian cancer. Nurs Clin North Am 44(3):339–54.
18. Fordis M, King JE, Ballantyne CM, Jones PH, Schneider KH, Spann SJ, Greenberg SB, Greisinger AJ. 2005. Comparison of the instructional efficacy of Internet-based CME with live interactive CME workshops: A randomized controlled trial. JAMA 294(9):1043–51.
19. Kohn LT. 1999. To err is human: Building a safer health system. Washington, DC: National Academies Press.
20. Kaushal R, Bates DW, Landrigan C. 2001. Medication errors and adverse drug events in pediatric inpatients. JAMA 285(16):2114–20.
21. Agrawal A. 2009. Medication errors: Prevention using information technology systems. Br J Clin Pharmacol 67(6):681–86.
22. Egberts ACG, Bollen CW, van Rosse F, Maat B, Rademaker CMA, van Vught AJ. 2009. The effect of computerized physician order entry on medication prescription errors and clinical outcome in pediatric and intensive care: A systematic review. Pediatrics 123:1184–90.
23. Ash JS, Bates DW. 2005. Factors and forces affecting EHR system adoption: Report of a 2004 ACMI discussion. J Am Med Inform Assoc 12(1):8–12.
24. Wietholter J, Sitterson S, Allison S. 2009. Effects of computerized prescriber order entry on pharmacy order-processing time. Am J Health Syst Pharm 66(15):1394–98.
25. Niazkhani Z, Pirnejad H, Berg M, Aarts J. 2009. The impact of computerized provider order entry systems on inpatient clinical workflow: A literature review. J Am Med Inform Assoc 16(4):539–49.
26. Shojania KG, Jennings A, Mayhew A, Ramsay CR, Eccles MP, Grimshaw J. 2009. The effects of on-screen, point of care computer reminders on processes and outcomes of care. Cochrane Database Syst Rev (3):CD001096.
27. Puccio JA, Belzer M, Olson J, Martinez M, Salata C, Tucker D, Tanaka D. 2006. The use of cell phone reminder calls for assisting HIV-infected adolescents and young adults to adhere to highly active antiretroviral therapy: A pilot study. AIDS Patient Care STDS 20(6):438–44.
28. Oztekin Aa, Delen D, Kong ZJ. 2009. Predicting the graft survival for heart-lung transplantation patients: An integrated data mining methodology. Int J Med Inform 78(12):e84–96.
29. Thervet E, Anglicheau D, Legendre C, Beaune P. 2008. Role of pharmacogenetics of immunosuppressive drugs in organ transplantation. Ther Drug Monit 30(2):143–50.