Abstract
Background
A learning health system (LHS) is a health system in which patients and clinicians work together to choose care on the basis of best evidence and to drive discovery as a natural outgrowth of every clinical encounter to ensure the right care at the right time. An LHS for dentistry is now feasible, as an increased number of oral health care encounters are captured in electronic health records (EHRs).
Methods
The authors used EHRs data to track periodontal health outcomes at 3 large dental institutions. The 2 outcomes of interest were a new periodontitis case (for patients who had not received a diagnosis of periodontitis previously) and tooth loss due to progression of periodontal disease.
Results
The authors assessed a total of 494,272 examinations (new periodontitis outcome: n = 168,442; new tooth loss outcome: n = 325,830), representing a total of 194,984 patients. Dynamic dashboards displaying performance on both measures over time allow users to compare demographic and risk factors for patients. The incidence of new periodontitis and tooth loss was 4.3% and 1.2%, respectively.
Conclusions
Periodontal disease, diagnosis, prevention, and treatment are particularly well suited for an LHS model. The results showed the feasibility of automated extraction and interpretation of critical data elements from the EHRs. The 2 outcome measures are being implemented as part of a dental LHS. The authors are using this knowledge to target the main drivers of poorer periodontal outcomes in a specific patient population, and they continue to use clinical health data for the purpose of learning and improvement.
Practical Implications
Dental institutions of any size can conduct contemporaneous self-evaluation and immediately implement targeted strategies to improve oral health outcomes.
Key Words
Abbreviation Key:
EHR (Electronic health record), LHS (Learning health system), NA (Not applicable), Perio (Periodontitis), T0 (From 6 through 36 months before the reporting period), T1 (During the reporting period)
- Levit L.
- Balogh E.
- Nass S.
- Ganz P.A.
The National Academy of Medicine
has called for the development of a learning health system (LHS) in which patients and clinicians work together to choose care on the basis of best evidence
- Britto M.T.
- Fuller S.C.
- Kaplan H.C.
- et al.
and to drive discovery as a natural outgrowth of every clinical encounter to ensure innovation, quality, and value at the point of care. An LHS is a health system in which internal data and experience are systematically integrated with external evidence and that knowledge is put into practice. As a result, patients receive higher-quality, safer, and more efficient care.
This vision of an LHS has remained largely aspirational, especially in dentistry.
- Fontana M.
- Carrasco-Labra A.
- Spallek H.
- Eckert G.
- Katz B.
clinical decision support, and other knowledge management systems (knowledge to performance).
The findings from these improvement strategies are used to drive the next learning loop (performance to data). As such, the LHS requires a robust data infrastructure to provide real-time access to knowledge and digital capture of the care experience.
- Cresswell K.M.
- Smith P.
- Swainson C.
- Timoney A.
- Sheikh A.
This infrastructure requires comprehensive data sources, thoughtful data oversight, and appropriate data use
- Maddox T.M.
- Albert N.M.
- Borden W.B.
- et al.
The learning healthcare system and cardiovascular care: a scientific statement from the American Heart Association.
to ensure the trust of patients and providers.
- Eichler H.G.
- Bloechl-Daum B.
- Broich K.
- et al.
This inability of the health care system to learn from EHR data can lead to suboptimal health outcomes.
- Maddox T.M.
- Albert N.M.
- Borden W.B.
- et al.
The learning healthcare system and cardiovascular care: a scientific statement from the American Heart Association.
The EHR is a key data source for an LHS for a variety of reasons. First, although they lag behind medical practices and hospitals, dental practices in the United States are increasingly adopting EHRs.
- Acharya A.
- Schroeder D.
- Schwei K.
- Chyou P.H.
Second, EHR data have the potential to provide much more detail on patient-level encounters than administrative claims or other data sources.
Third, the immediate availability of data that is possible with EHRs allows for real-time use in clinical care. Bringing key information to the provider during the clinical encounter has the potential to improve clinical decision making. The timeliness of these data also allows for frequent assessment to identify patient-reported outcomes; to use machine learning algorithms to match suitable patients with clinical trials, observing their specific enrollment criteria; and to monitor practice trends for various patient populations.
- Maddox T.M.
- Albert N.M.
- Borden W.B.
- et al.
The learning healthcare system and cardiovascular care: a scientific statement from the American Heart Association.
The ways in which patient data are generated, stored, and used in the EHR are fundamental to the LHS.
- Eichler H.G.
- Bloechl-Daum B.
- Broich K.
- et al.
- Nazir M.
- Al-Ansari A.
- Al-Khalifa K.
- Alhareky M.
- Gaffar B.
- Almas K.
Second, periodontal disease is associated with other inflammatory and systemic conditions, such as cardiovascular disease and diabetes.
- Nazir M.
- Al-Ansari A.
- Al-Khalifa K.
- Alhareky M.
- Gaffar B.
- Almas K.
As such, it touches on multiple areas of health care delivery, including prevention, diagnostics, therapeutic procedures, and chronic disease management. This characteristic of periodontal disease care is especially important because clinicians may need to coordinate efforts with other providers in prevention and chronic disease management. As the vision of an LHS aspires to achieve effective care coordination, modulating periodontal care provision can affect those efforts. Third, there is a standardized approach to the collection, curation, and classification of clinical periodontal information. In 2017, the classification of periodontal disease was updated during the World Workshop on the Classification of Periodontal and Peri-implant Diseases.
- Tonetti M.S.
- Greenwell H.
- Kornman K.S.
Due to the multifactorial nature of periodontal disease and the considerable variation in its diagnoses, the development of EHR-based algorithms that determine periodontal disease outcomes has proven to be an arduous task.
- Mullins J.M.
- Even J.B.
- White J.M.
Together, these characteristics of periodontal care delivery make it an informative model in which to translate LHS concepts into action.
In this article, and as a first step with a focus on the data-to-knowledge and knowledge-to-performance parts of the learning loop, we report on the aggregation of relevant EHR clinical data elements to arrive at clinical periodontal diagnostic information, and then follow up the patients longitudinally for up to 3 years to measure their periodontal health outcomes. The outcomes of interest were new periodontitis diagnosis (for patients who had not received a diagnosis of periodontitis previously) and tooth loss due to progression of the disease in a patient who had received a diagnosis of periodontitis previously. We also introduce interactive dashboards to aid with the presentation of actionable clinical data to clinicians, patients, and administrators. In our study, we showcased the potential of EHR data and how we could start to use these available data to create a learning loop in dentistry on the basis of large data sets measuring periodontal health outcomes.
Methods
- Kalenderian E.
- Ramoni R.L.
- White J.M.
- et al.
Each of the participating institutions treats a diverse population, including private practices, specialty clinics, and teaching clinics. Institutional Review Board approval from the University of Texas Health Science Center at Houston was obtained to conduct our study. Strengthening the Reporting of Observational Studies in Epidemiology guidelines were followed.
- von Elm E.
- Altman D.G.
- Egger M.
- Pocock S.J.
- Gøtzsche P.C.
- Vandenbroucke J.P.
The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.
- Tonetti M.S.
- Greenwell H.
- Kornman K.S.
the critical data elements for periodontal diagnoses and new tooth loss due to periodontal disease were itemized. Next, an electronic script was developed to locate and extract each element from the appropriate section of the EHR. The measure summary and respective denominator and numerator specifications are presented in Table 1. Data were extracted from 3 calendar years (2017, 2018, 2019). Data included all qualifying visits in the year and the required data from the prior examination that may have occurred up to 3 years earlier.
Table 1Summary and specifications of the periodontal outcome measures.
Automated query implementation and validation in the EHR framework
Structured query language scripts were used to extract data in a standard format at all sites. The query generated a list of patients eligible to be included in both the denominator and the numerator. Each site tested the query before implementation. We compared the performance of the automated query with the results of a manual electronic chart review of more than 500 charts across the sites, which was considered the reference standard. Two calibrated, independent reviewers at each site, with experience in electronic patient chart reviews, conducted these reviews. To evaluate the concordance between the automated and manual queries, we calculated sensitivity, specificity, positive predictive values, and negative predictive values.
Analytic methods
Descriptive
- Callahan C.D.
- Griffen D.L.
and analysis of proportions
methods. The statistical process control methods can highlight whether the variation exhibited by means of a series of points occurred beyond random variation. The analysis of proportions methods adjust for denominator size when determining whether a particular point is a statistical outlier compared with the overall mean measure score. We used bar graphs to illustrate comparative new periodontitis and new tooth loss measures across the various patient dimensions.
Associational
To estimate multivariate associations, a multivariable logistic regression for repeated measures modeling the odds of new periodontitis and modeling the odds of new tooth loss was performed. We reported the odds of a new periodontitis diagnosis and new tooth loss as the measure of association, along with the corresponding estimates of precision and 95% CIs. Each model included the following covariates: sex, age, smoking status, plaque status, diabetes status, and time between consecutive visits. All tests were conducted at the standard significance level (P < .05). The interactive dashboard tool was created using the dashboard tool Tableau (Salesforce), and all analyses were performed with R statistical software (R Foundation for Statistical Computing).
Results
Table 2Concordance between the automated algorithm and manual chart review.

Figure 2Dashboard displaying performance on new periodontitis and tooth loss measures over time (according to quarter) using a control chart. Dashboard also indicates whether performance during a specific quarter was statistically different from the system mean. Users can compare demographic and risk factor differences for patients in the numerator. F: Female. M: Male. Perio: Periodontitis. Pt: Patient. T0: From 6 through 36 months before the reporting period.
Table 3The number of new cases and mean scores for periodontitis and tooth loss according to year.
New periodontitis
The new periodontitis section includes a plotted times series of new periodontitis cases according to quarter, which represent 12 time points (range, January 2017-December 2019). The chart shows that the first 2 quarters were high outliers, but that the rest of the quarters indicate a process in control, with only random variation. The 4 grouped bar charts below the line chart show the distribution of new periodontitis according to patient sex, age, smoking status, and diabetes status. The new periodontitis measure scores were higher among male patients (5.59%) than female patients (3.59%) and increased with increasing age. The measure scores for new periodontitis were higher among those who smoked (10.75%) than those who did not (4.04%). The measure scores for new tooth loss were higher among those who had received a diagnosis of diabetes (10.66%) than those who had not (4.06%).
New tooth loss
The new tooth loss section includes a plotted times series of new tooth loss cases according to quarter, which represent 12 time points (range, January 2017-December 2019). The chart shows that the quarterly measure scores were never significantly different from the system mean and were within the 95% CIs on all 12 consecutive occasions. The 4 grouped bar charts below the line chart show the distribution of new tooth loss according to patient sex, age, smoking status, and diabetes status. The new tooth loss measure scores were higher among male patients (1.5%) than female patients (0.9%) and rose with increasing age. The measure scores for new tooth loss were higher among those who smoked (3.67%) than those who did not (0.99%). The measure scores for new tooth loss were higher among those who had received a diagnosis of diabetes (2.80%) than those who had not (1.04%).
Adjusted analysis for periodontal disease and tooth loss with risk factors
The logistic regression confirmed the dashboard findings. Male sex was associated with increased odds of a new periodontitis diagnosis, adjusting for other covariates in the model (odds ratio [OR], 1.27; 95% CI, 1.21 to 1.34). Age categories 40 through 60 years (OR, 1.67; 95% CI, 1.58 to 1.77) and older than 60 years (OR, 1.42; 95% CI, 1.33 to 1.53) were significantly associated with increased odds of a new periodontitis diagnosis. Smoking (OR, 1.38; 95% CI, 1.27 to 1.50), high levels of plaque (OR, 1.95; 95% CI, 1.74 to 2.19), and diabetes (OR, 1.45; 95% CI, 1.16 to 1.81) were each significantly associated with increased odds of a new periodontitis diagnosis. Lastly, the increased length of time between consecutive dental visits was associated with increased odds of a periodontitis diagnosis (OR, 1.07; 95% CI, 1.02 to 1.13).
For the new tooth loss outcome, male sex was associated with increased odds of new tooth loss, adjusting for other covariates in the model (OR, 1.40; 95% CI, 1.31 to 1.50). Age categories 40 through 60 years (OR, 4.96; 95% CI, 4.40 to 5.58) and older than 60 years (OR, 9.19; 95% CI, 8.14 to 10.37) were significantly associated with increased odds of new tooth loss. Smoking (OR, 3.23; 95% CI, 2.95 to 3.54) and high levels of plaque (OR, 1.85; 95% CI, 1.58 to 2.17) were each significantly associated with increased odds of new tooth loss, and diabetes was not (OR, 1.29; 95% CI, 0.98 to 1.68). Lastly, increased length of time between consecutive dental visits was not associated with increased odds of new tooth loss due to periodontal diagnosis (OR, 1.02; 95% CI, 0.95 to 1.10).
Discussion
Quality Measurement in Dentistry: A Guidebook.
they can facilitate the tracking of key dental outcomes.
- Kenney G.M.
- Pelletier J.E.
,
- Herndon J.B.
- Tomar S.L.
- Catalanotto F.A.
- et al.
Increased adoption of EHRs has provided the tools to efficiently extract useful data for performance measures, assess the relationships between these measures and health outcomes, and benchmark population health.
- Bhardwaj A.
- Ramoni R.
- Kalenderian E.
- et al.
Population health is defined as “the health outcomes of a group of individuals, including the distribution of such outcomes within a group,”
and EHRs already provide some access to public health data to study the population for potential health improvements and act as a safety net for potential health threats.
- Diamond C.C.
- Mostashari F.
- Shirky C.
Until now, the clinical periodontology literature has mostly represented studies that use surrogate end points.
- Ramoni R.B.
- Etolue J.
- Tokede O.
- et al.
), with more than 95% use of structured diagnoses recorded across each site. Oral health care providers are also required to complete a standardized periodontal assessment form, in which periodontal indexes and risk factors are documented in a structured format. Previous work has assessed the quality of periodontal assessment documentation recorded within the EHR and its ability to identify patients who had received a diagnosis of periodontal disease, complete periodontal charting and periodontal risk factors.
- Mullins J.
- Yansane A.
- Kumar S.V.
- et al.
We built on that and reported the development of 2 periodontal outcome measures using a large EHR data set; our approach is scalable and transferable to other important disease markers and oral health outcome measures. The 2 outcome measures we presented can be implemented to assess success in the prevention and treatment of periodontal disease, and to facilitate learning and improvement. Team members have also created checklists, notifications, and other ways to integrate these data into clinicians’ workflow. The aim was to make this information accessible to improve best practices at the point of care. Our research team called this the “Rate [measuring and properly articulating the data], Communicate [presenting the data in a way that is understandable to the appropriate audience], Motivate [implementing reward systems for changed behavior and improved performance], and Iterate [to re-enter the cycle in order to foster continuous improvement]” model.
- Albino J.
- Dye B.A.
- Ricks T.
report affirmed that by 2035 there will be more older adults than youth in the United States. This aging US population is also more dentate. Consequently, there has never been a more important time to pay attention to periodontal health and to strategies that help us learn from clinical data to improve it. The structured data found within the EHR can help promote measure automation, which can ease the implementation process. It is easy to envision these types of practice-level quality measures aiding patients and providers alike.
Clinical information from EHRs can be excessive and is often scattered and hard for clinicians and policy makers to access. Using an interactive dashboard tool for the processing and presentation of large amount of information allows stakeholders to explore the data on different levels. In our study, we depicted a static representation of an interactive dashboard. In actual use, filters can be used to narrow down to a particular clinic in the institution or network. Clicking on a particular quarter or month can allow for other panels to be filtered to show the characteristics for that period only. Likewise, clicking on a risk factor can filter the time series to show comparative performance across the different dimensions. Additional dashboard actions can trigger patient-level detail across time.
- Bayley K.B.
- Belnap T.
- Savitz L.
- Masica A.L.
- Shah N.
- Fleming N.S.
These include missing data, erroneous data, uninterpretable data, and inconsistencies in the way data are recorded among providers and over time. Another issue is that patients often receive care from multiple providers using fragmented and often poorly integrated EHR systems, making it difficult to completely track patients across practices
- Kahn M.G.
- Brown J.S.
- Chun A.T.
- et al.
or systems. EHR information exchange, which allows health systems to access and share EHR data across organizational and geographic boundaries, needs continued enhancement and dissemination to increase the value of EHR data. In addition, critical clinical data are often recorded in unstructured, narrative text, complicating its use for learning and improvement. In our work, we relied on structured data. For dental clinics that do not use a standardized diagnosis terminology, innovations in methods such as natural language processing may be used to capture such unstructured data from the clinical narratives in the EHRs.
,
In addition, effective user interfaces can improve the ease and consistency of data entry, which simultaneously reduces user burden and decreases the amount of unstructured, and potentially uninterpretable, data in the EHR. Finally, there is a major need for rigorous EHR evaluation and data optimization to ensure valid and usable information.
- Arts D.G.
- De Keizer N.F.
- Scheffer G.-J.
,
- Botsis T.
- Hartvigsen G.
- Chen F.
- Weng C.
,
Conclusions
- Walji M.F.
- Kalenderian E.
- Stark P.C.
- et al.
is a consortium of 11 dental schools that share EHR data for research and quality improvement and are developing the components of an LHS. We also recognize that EHR data alone may not be inclusive enough to conduct meaningful learning. Rather, the value of EHR data might be realized when linked to other data sources, such as patient-reported behaviors, quantified self-data, and clinical trial data. As value-based care predominates, EHR data will occupy a central role in generating meaningful knowledge in support of LHS for improving oral health.
References
Patient-centered communication and shared decision making.
in: Levit L. Balogh E. Nass S. Ganz P.A. Delivering High-Quality Cancer Care: Charting a New Course for a System in Crisis. National Academies Press,
2013: 91-152Olsen L. Aisner D. McGinnis J.M. The Learning Healthcare System: Workshop Summary. National Academies Press,
2007Using a network organisational architecture to support the development of Learning Healthcare Systems.
BMJ Qual Saf. 2018; 27: 937-946
Learning healthcare systems: a perspective from the US.
Public Health Res Pract. 2019; 292931920
Our approach: getting in the loop. Learning Health Sciences, University of Michigan Medical School.
Improving caries risk prediction modeling: a call for action.
J Dent Res. 2020; 99: 1215-1220
Clinical Decision Support: The Road Ahead.
Elsevier,
2011Establishing data-intensive learning health systems: an interdisciplinary exploration of the planned introduction of hospital electronic prescribing and medicines administration systems in Scotland.
BMJ Health Care Inform. 2016; 23: 842
The learning healthcare system and cardiovascular care: a scientific statement from the American Heart Association.
Circulation. 2017; 135: e826-e857
Data rich, information poor: can we use electronic health records to create a learning healthcare system for pharmaceuticals?.
Clin Pharmacol Ther. 2019; 105: 912-922
Update on electronic dental record and clinical computing adoption among dental practices in the United States.
Clin Med Res. 2017; 15: 59-74
Toward reuse of clinical data for research and quality improvement: the end of the beginning?.
Ann Intern Med. 2009; 151: 359-360
Global prevalence of periodontal disease and lack of its surveillance.
ScientificWorldJournal. 2020; 20202146160
Staging and grading of periodontitis: framework and proposal of a new classification and case definition.
J Periodontol. 2018; 89: S159-S172
Periodontal management by risk assessment: a pragmatic approach.
J Evid Based Dent Pract. 2016; 16: 91-98
The development of a dental diagnostic terminology.
J Dent Educ. 2011; 75: 68-76
The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.
J Clin Epidemiol. 2008; 61: 344-349
Advanced statistics: applying statistical process control techniques to emergency medicine: a primer for providers.
Acad Emerg Med. 2003; 10: 883-890
Introducing analysis of means to medical statistics.
BMJ Qual Saf. 2012; 21: 529-532
Quality Measurement in Dentistry: A Guidebook.
American Dental Association,
2012Monitoring duration of coverage in Medicaid and CHIP to assess program performance and quality.
Acad Pediatr. 2011; 11: S34-S41
Measuring quality of dental care: caries prevention services for children.
JADA. 2015; 146: 581-591
Measuring up: implementing a dental quality measure in the electronic health record context.
JADA. 2016; 147: 35-40
What is population health?.
Am J Public Health. 2003; 93: 380-383
Collecting and sharing data for population health: a new paradigm.
Health Aff (Millwood). 2009; 28: 454-466
Endpoints of active periodontal therapy.
J Clin Periodontol. 2020; 47: 61-71
Adoption of dental innovations: the case of a standardized dental diagnostic terminology.
JADA. 2017; 148: 319-327
Assessing the completeness of periodontal disease documentation in the EHR: a first step in measuring the quality of care.
BMC Oral Health. 2021; 21: 1-8
Surgeon General’s Report: Oral Health in America: Advances and Challenges. National Institutes of Health.
So much data-so little information! Integrating and streamlining patient data for effective indicator tracking and outcome measurement.
CANNT J. 2020; 17: 43-44
Challenges in using electronic health record data for CER: experience of 4 learning organizations and solutions applied.
Med Care. 2013; 51: S80-S86
Transparent reporting of data quality in distributed data networks.
EGEMS (Wash DC). 2015; 3: 1052
Natural language processing and the promise of big data: small step forward, but many miles to go.
Circ Cardiovasc Qual Outcomes. 2015; 8: 463-465
The promise of electronic records: around the corner or down the road?.
JAMA. 2011; 306: 880-881
Defining and improving data quality in medical registries: a literature review, case study, and generic framework.
J Am Med Inform Assoc. 2002; 9: 600-611
Secondary use of EHR: data quality issues and informatics opportunities.
Summit Transl Bioinform. 2010; 2010: 1-5
Data quality assessment for comparative effectiveness research in distributed data networks.
Med Care. 2013; 51: S22
BigMouth: a multi-institutional dental data repository.
J Am Med Inform Assoc. 2014; 21: 1136-1140
Biography
Dr. Tokede is an associate professor, Department of Diagnostic and Biomedical Sciences, The University of Texas Health Science Center at Houston, Houston, TX.
Dr. Yansane is an associate professor, Preventative and Restorative Dental Sciences, School of Dentistry, University of California, San Francisco, San Francisco, CA.
Dr. White is a professor, Preventative and Restorative Dental Sciences, School of Dentistry, University of California, San Francisco, San Francisco, CA.
Dr. Bangar is a research associate, School of Dentistry, The University of Texas Health Science Center at Houston, Houston, TX.
Ms. Mullins is the vice president of operations, Willamette Dental Group, Hillsboro, OR.
Mr. Brandon is a consultant and a data scientist, Willamette Dental Group and Skourtes Institute, Hillsboro, OR.
Mr. Gantela is a research associate, School of Dentistry, The University of Texas Health Science Center at Houston, Houston, TX.
Mr. Kookal is a research associate, School of Dentistry, The University of Texas Health Science Center at Houston, Houston, TX.
Dr. Rindal is a senior investigator, HealthPartners Institute, Minneapolis, MN, and an associate dental director for research, HealthPartners Dental Group, Minneapolis, MN.
Dr. Lee is an associate professor, Department of Periodontics and Dental Hygiene, School of Dentistry, The University of Texas Health Science Center at Houston, Houston, TX.
Dr. Lin is a health sciences assistant clinical professor and the director of the postgraduate periodontics program, School of Dentistry, University of California, San Francisco, CA.
Dr. Spallek is a professor, the head of school, and the dean, The University of Sydney, Sydney, New South Wales, Australia.
Dr. Kalenderian is a professor, Department of Preventive and Restorative Dental Sciences, School of Dentistry, University of California, San Francisco, San Francisco, CA; a professor, Academic Centre for Dentistry, Amsterdam, The Netherlands; senior lecturer, Harvard School of Dental Medicine, Boston, MA; and an Extraordinary Professor, University of Pretoria School of Dentistry, Pretoria, South Africa.
Dr. Walji is a professor and an associate dean for technology services and informatics, Diagnostic and Biomedical Sciences Department, School of Dentistry, The University of Texas Health Science Center at Houston, Houston, TX.
Article Info
Publication History
Published online: August 12, 2022
Accepted:
June 14,
2022
Received in revised form:
June 7,
2022
Received:
March 14,
2022
Publication stage
In Press Corrected Proof
Footnotes
Disclosures. None of the authors reported any disclosures.
Drs. Tokede and Yansane contributed to this article equally as co-first authors.
Funding was provided by grant R01DE024166 from US Department of Health and Human Services , National Institutes of Health , and National Institute of Dental and Craniofacial Research .
Identification
DOI: https://doi.org/10.1016/j.adaj.2022.06.007
Copyright
© 2022 American Dental Association.
User License
Creative Commons Attribution – NonCommercial – NoDerivs (CC BY-NC-ND 4.0) |

Permitted
For non-commercial purposes:
- Read, print & download
- Redistribute or republish the final article
- Text & data mine
- Translate the article (private use only, not for distribution)
- Reuse portions or extracts from the article in other works
Not Permitted
- Sell or re-use for commercial purposes
- Distribute translations or adaptations of the article
Elsevier’s open access license policy
ScienceDirect
Access this article on ScienceDirect