Written Evidence Submitted by the FORRT (Framework for Open and Reproducible Research Training)
(RRE0080)
Authors
Meng Liu (University of Cambridge, UK)
Charlotte R. Pennington (Aston University, Birmingham, UK)
Madeleine Pownall (University of Leeds, UK)
Thomas Rhys Evans (University of Greenwich, UK)
Sam Parsons (University of Oxford, UK)
Mahmoud Elsherif (University of Birmingham, UK)
Leticia Micheli (Leibniz Universität Hannover)
David Moreau (University of Auckland, NZ)
FORRT (FORRT Community)
Towards a culture of open scholarship: The role of pedagogical communities
The Authors
The Framework of Open and Reproducible Research Training (FORRT) is a pedagogical community of more than 150 members from diverse disciplines and geographical locations building a pathway towards the integration of principled, open teaching and mentoring practices, whilst also generating a conversation about the ethics and social impact of higher-education pedagogy.
The Open scholarship movement promotes the idea that all knowledge should be openly shared, transparent, rigorous, reproducible, replicable, accumulative, and inclusive (1, 2). In the last decade, the accelerating adoption of open scholarship practices, such as preregistration and data sharing, have brought about well-needed behavioural change in how scientists conduct their research. However, while progress has been notable, it continues to vary widely across disciplines and career stages. As open scholarship practices become more prevalent, it is essential to update existing pedagogical tools, and develop new ones, to effectively teach and mentor undergraduate and graduate students towards reproducible and replicable research. Sadly, integrating these practices into higher education is far from commonplace. We argue that to improve reproducibility and research integrity, open scholarship must be integrated fully into teaching and mentoring practices as an academic norm.
Ever since the emergence of the “replication” or “reproducibility” crisis, many initiatives have been proposed to encourage greater openness and transparency, such as study preregistration and replication studies, open materials, data and code, and higher standards for quality of evidence (3). This push for reform and improvements has been more positively described as a “credibility revolution” (4). Fully acknowledging the significance of redressing perverse academic incentives and re-aligning research evaluations is critical for addressing deep rooted issues and achieving scientific utopia (5-7). However, there is an important component of research integrity that has been left unattended: how we teach, mentor, and supervise students through open scholarship in higher education. This includes the need for curricular reform, development of new methods of education, and addressing questions around how open scholarship practices relate to social justice and inclusive practice. In ignoring these pedagogical concerns, we are overlooking the opportunity to reshape the future generation of researchers and consumers of science towards a truly sustainable, bottom-up, and permanent transformation.
Neglect of pedagogy is not without reason, nor is it uncommon, particularly in research-focused institutions. The lack of top-down structure and institutional incentives transfers the responsibility to individuals who support the open scholarship movement and take on the onus of restructuring their teaching and mentoring accordingly. Faculty members and researchers often have a range of responsibilities, including teaching, research and administration; therefore, this widens the gap between support of open scholarship and its implementation into the teaching curriculum. Indeed, there is a substantive gap between researchers’ own stated endorsement of their open practices and actual rates of adoption (8), where many barriers, such as the lack of training and institutional support and infrastructure, currently inhibit uptake. As a result, even though the wave of scientific reform is influencing scientific practices and norms globally, the current model of higher education is largely outdated with respect to open scholarship.
How do we embed open scholarship practices that foster a culture of integrity without overburdening individuals or systems? Here, we propose that pedagogical communities can play a fundamental role in this process and should be supported by Research Stakeholders and government. Pedagogical communities are educationally-oriented ‘open science communities’ (9) that not only make open science knowledge accessible and facilitate communication between academia and policy, but also advocate for the integration of open scholarship into higher education and raise awareness of its pedagogical implications and associated challenges to better equip educators with the necessary didactic tools to incorporate open scholarship into curricula and educators’ teaching, mentoring, and research practices. We outline the benefits pedagogical communities can bring to the open scholarship movement and call for greater collaboration between such communities and all the stakeholders of research to minimise the demands of introducing open scholarship pedagogy and to improve, and make future-proof, research integrity.
Many students finish their degree without ever learning about the reproducibility crisis or open scholarship practices (10). This is likely because current academic norms reinforce and perpetuate problematic research practices, and these manifest throughout the curriculum. Whilst there are a number of current exceptions to this, including recently developed open science consortiums (e.g. 11), international replication projects designed for educational outcomes (e.g. Crowdsourced Replication Project; 12), and individuals creating resources and projects at a local level (e.g., Mass Replications & Extensions; 13), these are few and far between given the current climate of concerns regarding reproducibility and research integrity. We propose that teaching and involving students in the discussion of open scholarship, reproducibility, and the credibility revolution has at least three unique benefits:
1. Embedding open scholarship in pedagogy addresses concerns regarding research integrity, which is one of the root causes of the 'crisis' that many scientific disciplines currently face. We argue that one core issue is the displacement of academic norms and what constitutes their exception. It is one thing to first train students in a business-as-usual manner, where many “standard” practices are equivalent to questionable research practices underpinning the present crisis, and later on, call upon their conscience to unlearn what they’ve been taught and go against what used to be perceived as “norms”. But it is another thing entirely to foster core values of research integrity from the start, through which we remove the need for late “conversion”.
2. From the perspective of researchers, the integration of open and reproducible practices into teaching facilitates the alignment between research belief and research practice. We argue that open research is incomplete without open educational practices. Core values such as openness, transparency, and reproducibility are not exclusive to research alone and should be embedded and reflected in teaching. Training our future researchers through open scholarship allows open science practices to become the norm and to be passed on to the next generation, cumulatively consolidating the foundation for a sustainable future.
3. Integrating open scholarship into higher education advances social justice which, whilst being the most fundamental, is arguably one of the most overlooked tenets of contemporary scholarship (14). Indeed, the very notion of open scholarship, including open educational resources, is underpinned by the powerful idea that knowledge is a public good for all humanity (14-16). Current academic systems help perpetuate global inequalities with prescribed dogmas, reinforced hierarchies, and hidden curricula. There are still systematic barriers to accessing scientific knowledge, where barriers exist not only between and within institutions but also between academia and the public. We believe that integrating open scholarship practices into education through means of open educational resources plays an important role in promoting social justice and improving the accessibility of science.
As will be shown by our presented case studies, pedagogical communities exemplify a promising pathway towards a culture of open scholarship practices in research, education and training through empowering individual members of the research community. This includes not only those who conduct research on a day-to-day basis, but also students who constitute the future of our research community.
Bridging the gap: The role of pedagogical communities
Fostering a culture of open scholarship practices through communities can bring important benefits to the academic community. Despite the different missions and scope of initiatives, all are working towards integrating open scholarship principles into higher education while contributing to advancement of research transparency, reproducibility, rigor, and ethics through pedagogical reform. Chief among them, pedagogical communities facilitate the co-creation of open scholarship educational materials. Resources and didactics by educators for educators are crucial in facilitating the integration of open scholarship into higher education and reducing the burden placed on scholars. Pedagogical communities also offer a much-needed environment wherein scholars share individual experiences, identify common hurdles, and iteratively enhance their pedagogy and accompanying materials towards better addressing the unique challenges ensuing from curricular reform and novel educational methodology. Through these exchanges, pedagogical communities help create a culture of open scholarship, benefiting those within the community, and those that interact with it.
Pedagogical communities also offer a low-entry point into improved research and pedagogical practices. As pedagogical communities welcome scholars from all levels, including early career researchers, they are an accessible space for educators wishing to learn and practice open scholarship. By cutting across career stages, these communities, then, become essential to instilling the new and improved values and norms of open scholarship.
Further, pedagogical communities play an important role in offering a sense of community to those who would otherwise be deprived of such a learning opportunity when there are fewer top-down initiatives and infrastructure to encourage change. As such, these communities are essential to address recent concerns regarding the lack of diversity in the open scholarship movement (e.g. 17-21). Such communities break the boundaries of academic fields and geographical locations to advance social justice, making the movement more diverse and representative of the plural needs of academics.
The Framework of Open and Reproducible Research Training (FORRT) is one such pedagogical community aiming to build, together with educators and students, a pathway to the stepwise adoption of principled, open teaching and mentoring practices, whilst also generating a conversation about the ethics and social impact of higher-education pedagogy. It responds to calls for a wider interpretation of open scholarship as inclusive scholarship (e.g., 21-24) by involving those at all stages of learning. In this sense, FORRT’s mission seeks to empower teachers and their students, who may find it otherwise challenging, to not only develop strong competencies in this area but also incorporate open scholarship into their teaching and learning. Established in 2018, FORRT has already progressed 11 initiatives (14). Here, we focus on three initiatives that exemplify the role of the communities in co-creating pedagogical materials and in lowering the entry-level into open scholarship language and literature.
Lesson Plans. Developing educational resources is essential to facilitate engagement with, and adherence to, research integrity and transparency, replicability, openness, accessibility, and reproducibility. Despite the growing awareness of the benefits of training open scholarship methods within research practices, there remains a lack of systematic incorporation of open scholarship practices in taught courses across higher education. While many pedagogical resources are regularly developed for this purpose, they are not often openly and actively shared with the wider community. While the potential reasons for this are diverse, one reason may be the dearth of available ready-to-use educational resources. To support educators aiming to bridge that gap, the FORRT community set out to produce an open educational resource drawing from experts, interested parties, and stakeholders through a hackathon held at the 2021 Society for the Improvement of Psychological Science (SIPS) Annual Conference. We aimed to produce evidence-based, high-quality lesson plans and activities available to teaching faculty, thus reducing the labour required to develop and implement open scholarship content. We compiled nine lesson plans and almost sixty class activities that can be integrated into taught courses ‘out of the box’. Each lesson plan was enriched with—and categorized based on—theme, learning outcome, activity length, and method of delivery. All open educational resources are made publicly available at FORRT’s educational nexus (https://forrt.org/lesson-plans) and formally described in Pownall et al. (2). Further capturing FORRT’s ethos, like all educational resources, lesson plans are neither static nor rigid. FORRT welcomes contributors and feedback on existing materials, and plans to host hackathons and seminars to improve, update, and further expand these pedagogical resources.
Glossary. A central barrier to achieving the outcomes from such open scholarship developments is the lack of a coherent and consistent shared language. Many new terms have been recently established within the open scholarship communities and have been adapted or used interchangeably (e.g., creative destruction: 25, 26; paper mills: 27; reproducibility and replicability: 28, 29). For those unfamiliar, the new nomenclature can be a barrier to follow and join the discussions; for those familiar, potentially vague or competing definitions can cause confusion and misunderstandings. To reduce potential barriers to entry and understanding, FORRT’s community produced a crowd-sourced and consensus-based Glossary of over 250 terms relating to open scholarship, acknowledging the contributions of over 100 community members. Each term includes a concise definition (and when needed, any applicable alternative definitions), related terms, and supporting references. The glossary aimed to facilitate a shared perspective and language to benefit researchers and teachers alike, whether experienced or newcomers to open scholarship, whilst also highlighting important considerations for social justice by making a wide range of accessibility and inclusivity-related terms well-represented within our language. FORRT’s Glossary is made publicly available at FORRT’s educational nexus (https://forrt.org/glossary) and formally described in Parsons et al. (1). As with languages which are constantly in flux and evolving, FORRT’s Glossary has a system allowing for open review, improvements, and consensus, with the aim to derive a truly communal understanding of key terminology.
Summaries. As of yet, most open educational resources on open scholarship still consist of standalone resources or lists of readings. This lack of a pedagogical infrastructure (yet institutional emphasis on pedagogy and research-informed teaching) creates substantive strain on educators, thereby representing a significant barrier to the integration of open scholarship topics in higher education. To reduce the burden on educators aiming to integrate open and reproducible practices into their teaching and mentoring, and aid in the learning process of any person interested in staying up-to-date with the open scholarship literature, FORRT has prepared over 200 summaries of academic articles related to varied topics on open and reproducible practices. Every summary contains the article's main takeaways, abstract, direct quotes summarizing key points, and references to articles on similar topics. Further, a distinction is made between “Open and Reproducible Scholarship'' summaries and “Diversity, Equity Inclusion & Accessibility” summaries to highlight that the topics of social injustices and diversity, equity, inclusion and accessibility (DEIA) are often neglected in academia, and in open and reproducible scholarship. To ensure the quality of the summaries, a member of FORRT’s community drafts the summary, which is followed by at least two independent FORRT members reviewing and improving them. FORRT’s Summaries are made publicly available at FORRT’s educational nexus (https://forrt.org/summaries), where one can also learn about how to contribute to this resource.
Overall, we stress that it is critical to embed training in reproducibility and research integrity into higher education pedagogy to ensure long-term, future-proof, and sustainable change. Regrettably, to date, the responsibility for incorporating open scholarship principles into education and training has heavily relied on the initiative of early adopters of the scholarship movement. In this piece, we have identified key reasons for embedding open scholarship tenets into education and highlighted the important role of pedagogical communities in assisting scholars and students in realizing the benefits of this integration. In addition, we have discussed how open scholarship of this kind can promote inclusivity, diversity, equity, and inclusion within education and research. We present FORRT as a case study to exemplify the role of pedagogical communities in facilitating the transition towards a model of open education. FORRT has been hard at work to equip institutions and educators with the resources needed to develop, implement, and assess open and reproducible research training. The eleven initiatives (and counting) of FORRT aim to empower educators and citizens with the necessary tools and skill set to integrate the principles of open scholarship into higher education. FORRT invites all stakeholders of research, as well as other higher education educators, to engage with these community members and advocate for open scholarship in improving reproducibility and research integrity and effect long-term and sustainable change.
Pedagogical open scholarship communities—whether focusing on creating and developing new methods of education, addressing the new challenges of curricular reforms ensuing from new and improved research norms, or highlighting the importance of epistemic, cultural, and demographic diversity—are an essential component to the credibility revolution. Pedagogical communities go beyond educational and network purposes, working towards redefining the culture of open scholarship. Pedagogical communities provide an alternative to the current academic reality by creating and implementing fairer norms; building the foundations for an inclusive and safe environment welcoming to all people and perspectives; working towards crediting members for their work and helping them claim it; and creating pedagogical resources that unburden educators and unravel the hidden curriculum. Given their importance in leading the way to educational revolution, we advocate for greater recognition of pedagogical communities. We encourage all stakeholders in education and research to engage with these communities, and for researchers, educators and teachers to join initiatives towards creating a more accessible and widely disseminated open scholarship culture. We also see an important role of governance and funding agencies in strengthening pedagogical communities. Currently, pedagogical communities depend on the voluntary effort of its members. Funding can represent a crucial step towards the long-term sustainability of these communities by guaranteeing resources to employ community managers and covering the dissemination of communities’ work within the academic community. This may be of benefit to the lay public by enabling open access publications and the organization of (and participation in) open scholarship events. In addition, the flexibility granted by financial resources can be an important catalyst for the creation and development of new initiatives and pedagogical materials.
In conclusion, we call for greater collaboration with pedagogical communities, paving the way for a much-needed integration of top-down and grassroot open scholarship initiatives.
Authors' contributions: Conceptualization: F.A. Project administration: F.A. Writing - original draft: F.A., M.L., C.R.P., M.P., T.R.E., S.P., M.M.E., and L.M. Writing - review & editing: F.A., M.L., C.R.P., M.P., T.R.E., S.P., M.M.E., D.M., and L.M. Availability of data and materials: all resources discussed in this article are available at forrt.org.
(September 2021)