The origins of the “Open by Design” concept can be traced back to manufacturing methodologies developed during the post-World War II industrial expansion. Central among these methodologies is Design for Manufacturing (DFM), a framework introduced to proactively integrate manufacturing constraints, optimize production efficiencies, and systematically enhance product quality at the earliest design stages. DFM addresses manufacturing complexities by anticipating inefficiencies, reducing production costs, minimizing material waste, and ensuring superior reliability and product performance through early stage integrated planning processes (Boothroyd et al. 2011, Andreasen et al. 1987, Whitney 1988).

Further developments emerged in the form of Design for Assembly (DFA), a complementary approach specifically targeting simplified assembly operations. DFA methodologies prioritize reducing overall product complexity by minimizing the number of components, standardizing interfaces, utilizing modular assemblies, and facilitating straightforward, error-resistant assembly procedures. These design approaches have been instrumental in reducing production timelines, enhancing maintenance efficiency, and promoting the widespread adoption of enduring principles such as modularity, interoperability, and standardization. Baldwin and Clark 2014, through their analysis of modular and platform-based system architectures, further articulated these foundational concepts, emphasizing the benefits of modular designs in fostering technological adaptability and innovation.

Simultaneously, parallel advancements in manufacturing were realized through the adoption and implementation of Lean Manufacturing and Just-In-Time (JIT) production systems. Originating primarily within Japanese manufacturing practices, notably pioneered by Toyota, these methodologies institutionalized rigorous and disciplined production processes emphasizing workflow transparency, continuous incremental improvement (Kaizen), systematic elimination of waste (Muda), and enhanced inter-organizational collaboration. These principles cultivated an organizational culture characterized by openness, continuous iterative refinement, and collaborative transparency (Womack 2007), elements increasingly relevant in contemporary frameworks of open innovation and research.

The “Open by Design” concept extends these manufacturing practices by building upon Design Thinking principles. Design Thinking embodies a human-centred, iterative process that prioritizes empathy for users, rapid prototyping, and ongoing iterative testing. This methodological framework actively incorporates continuous feedback mechanisms, systematically refining products to ensure ongoing relevance, adaptability, and user satisfaction (Brown 2008). The iterative, participatory nature of Design Thinking aligns with open innovation frameworks (Chesbrough 2003), which emphasize openness, transparency, and collective stakeholder involvement as essential drivers of innovative capacity and sustained knowledge dissemination. Moreover, the broader influence of these manufacturing philosophies extends beyond mere technical improvements. They foster organizational behaviours conducive to cross-disciplinary collaboration, stakeholder engagement, strategy development and transparency in knowledge sharing (Platts and Gregory 1990).

While industrial design emerged from mass manufacturing, service design has emerged from digital transformation of many former in-person transactions, from banking to publishing. Emerging from product design, interaction design and cognitive psychology (Rytilahti et al. 2015), at its best, service design considers all relevant stakeholders and is centered on user needs, not the needs of the business - but also integrates a business case and service sustainability model. Designers engage users in developing narratives and co-designing the service, then prototype and test the service with users, before deploying a minimum viable service, which is then iterated upon and improved based on user feedback (Reason et al. 2015; IxDF 2023). This approach ensures that services are based on evidence and genuine comprehension of the purpose of the service as well as capabilities to deliver value to users.

More recently, the field of service design has evolved significantly from UI/UX work to become a driver for organizational transformation (Souheimo et al. 2025). Service design can fail when the designers do not have a holistic understanding of the systems in place in and around the product they are developing. The systemic service design approach ensures that broader social, economic, and political environments are considered when developing a service so that the design process incorporates multiple stakeholder perspectives, emphasizes relations and interconnections, and considers the ethical implications of different pathways.

Systemic service design is a useful construct for considering design and management of public goods. (Miettinen and Sarantou 2017). Services transcend design objects and become a means for supporting the emergence of a more collaborative, sustainable and creative society and economy. At the infrastructure scale, these designs can become commons – community-scale interventions engage staff, organization, and publics and ideally build capacity rather than dependency and can empower knowledge exchange and re-distribution of power (Bollier and Helfrich 2015, Sangiorgi 2011).

Together, the focus on product from manufacturing design and the engagement of communities in developing that product from service design incorporate the goals of open science: engaging communities in the scientific process and ensuring access to knowledge. The challenge in managing community products, or public goods, has been well-described by Ostram (2009) as a framework of interacting subsystems. Applied to the research, design, and development space (Fig. 1), this framework helps to articulate the context and high-level components that must be considered when designing research infrastructure. These include the resources (people, facilities, data, funding) that drive scientific processes; the overarching resource system at varying levels of scale from local to international; as well as the values, rules, policies, and ethics that govern how research is carried out; and the communities that use research. Infrastructure designers must endeavor to understand how these subsystems interact with each other to be able to develop effective services.

The adaptation of design principles into research infrastructures provides a robust, structured framework for embedding openness systematically across conceptual and practical dimensions. Like manufactured products, research infrastructures necessitate comprehensive strategic planning that incorporates sustainability, standardization, interoperability, and lifecycle management from initial design through continuous operational phases. As digital products, research infrastructure requires explicit community partnership from ideation through prototype through launch and subsequent iterative improvement (Fig. 2). Open source code and software are examples of community goods with a rich history of iteration regarding maintenance and governance (Rossi et al. 2012, Lin and Maruping 2021). These open tools and infrastructures serve the research community, after all, not the other way around.

Consequently, the “Open by Design” approach explicitly integrates principles of transparency, accessibility, modularity, and interoperability into early-stage decisions. This approach mirrors the proactive methodologies central to DFM and DFA, systematically embedding manufacturability and ease of assembly into product designs from inception. Research infrastructures inherently share characteristics with complex manufactured products, particularly regarding modular architecture and standardized interfaces.

Such structural considerations facilitate seamless updates, replacements, and integration with complementary systems, substantially enhancing infrastructure adaptability and long-term resilience. This foresight in the initial phases mitigates risks associated with technological obsolescence, vendor lock-in, and operational disruptions, ensuring enhanced sustainability and operational effectiveness (Mons et al. 2017).

Illustrative examples of successful research infrastructures incorporating these openness principles include platforms such as the WorldWide Protein Data Bank and PubMed. These infrastructures demonstrate tangible advantages gained from embedding systematic openness within their governance frameworks, operational protocols, and technical architectures. Consistent stakeholder engagement, transparent governance practices, and rigorous adherence to open standards substantially contribute to their enduring operational effectiveness and sustained relevance within the research community.

Additionally, akin to Lean Manufacturing’s commitment to continuous improvement and Systems Service Design’s commitment to ongoing community engagement, open research infrastructures significantly benefit from iterative developmental processes. The importance of iteration is well recognized in design research. Wynn and Eckert (2017) reviewed the efficacy of a variety of iteration methodologies and suggest functional categories to be selected based on design needs, from exploring and refining, to correcting, and also coordinating efforts within and across groups. The goal of iterative methodologies is to integrate robust feedback mechanisms from user communities, leveraging user insights to refine operational processes continuously.

Transparency in operations, systematic use of persistent identifiers, and clear, comprehensive metadata standards substantially enhance reproducibility, promoting cross-disciplinary interoperability and comprehensive data stewardship (Wilkinson et al. 2019). Furthermore, research infrastructures adopting these principles achieve greater stakeholder confidence through transparency and collaborative engagement, facilitating improved user trust and enhanced community participation. This openness significantly fosters greater innovation potential and adaptive capacity, enabling these infrastructures to evolve dynamically in response to emerging research needs and technological advancements.

Thus, applying open design principles to research infrastructures provides an integrated and pragmatic framework for embedding openness systematically throughout the infrastructure lifecycle. Outputs generated by open infrastructures are more likely to be ‘born with integrity’ as the originating systems are auditable and the formats more likely to conform to data standards. Consequently, infrastructures developed under this paradigm could potentially demonstrate robustness, enhanced accessibility, superior interoperability, and sustained long-term operational sustainability, aligning closely with contemporary standards of open innovation and built upon relationship and trust building in scholarly communication.

The concept of “open” as a scholarly value or principle emerged by the mid-1990s and gained prominence in the early 2000s, when it was invoked widely and variously in open letters, statements, declarations, and manifestos in academic and research communities (see FORCE11 Scholarly Commons Working Group 2017), particularly in the contexts of open access and open science (Skinner and Wipperman 2020). Since that time, it has been used to signal a range of mechanisms and actions that make research available freely online without financial or technical barriers. Openness, in this sense, can be embedded in or enacted by scholars, research objects, and research infrastructures including standards, protocols, and tools.

Applications of the term “open” extend across open science, open access, open source, and open data; they also extend across many different types of functions, including technical (e.g., interoperability, flexibility) and community engagement (e.g., participation, contribution, governance) (see Leonelli 2023). This highly variable use arguably made claiming and/or evaluating “openness” too subjective, and by 2016, research stakeholders were actively working to build and refine clearer, more consistent standards by which characteristics like “openness” could be measured (see CoARA 2025).

The resulting promulgation of frameworks between and beyond 2016-2022 (see Bilder et al. 2020, GO FAIR 2016, GIDA 2023, COAR 2024, SPARC 2019, DPSC 2018, AmeliCA 2019, Group 2025, UCOLASC 2018, Badolato 2019, OANA 2016, Lippincott and Skinner 2022,Agate et al. 2022,Schultes et al. 2020) provide those who engage in or manage research outputs, tools, standards, processes and/or protocols with ways of both articulating and evidencing a broad range of values and principles. These frameworks focus on a relatively similar core set of values and principles - including openness, transparency, accessibility, ethics, equity, sustainability, discoverability, reusability, and representation. They have been largely created and issued by associations and working groups with very specific remits, and although there is some level of cross-referencing between them, each stands largely on its own, used by a subset of the landscape. For example, FAIR and CARE are geared towards data collection and sharing (data scientists and analysts); Exemplarity Criteria are geared towards funders and recipients of grant funding; and POSI, It Takes a Village, and FOREST are geared toward those who build and/or offer digital services (e.g., publishing, preservation).

Focusing on the infrastructure builder use case, among the myriad of open research frameworks we have found the Principles of Open Scholarly Infrastructure (POSI) (Bilder et al. 2020), FAIR (Jacobsen et al. 2020), CARE (Carroll et al. 2021), and FOREST (Lippincott and Skinner 2022) translate well into practice. We describe each briefly in Table 1.

While there is ongoing work to develop implementation advice for each of these frameworks (e.g., see Jacobsen et al. 2020 and Carroll et al. 2021), our experience as advisors has clearly demonstrated that the people building research infrastructures – entrepreneurs and researchers – are often not able and/or unwilling to invest the time to translate the advice into their design process. As discussed below in Use Cases, a challenge with each of these frameworks is that their target audience and evaluation use case is post-hoc community evaluation. From a builder’s perspective, the frameworks are at best directional. From an Open by Design perspective, there are many different and at times apparently contradictory requirements, making it difficult to prioritize and build strategically. Further complicating matters, funders of research infrastructure have not provided clear and specific guidance or incentives to the builders to encourage adoption of these frameworks.

Given this context, we have developed builder guidance for implementing Open By Design principles from the inception of an infrastructure. With the technical goals of modular architecture, standardized interfaces, sustainability, and operational effectiveness, and the organizational behavior goals of cross-disciplinary collaboration, stakeholder engagement, strategy development, and transparency in knowledge dissemination, we translate the open research infrastructure frameworks into design questions that make these Open by Design principles immediately applicable (Table 2).

Part of the Open by Design framework is an evaluative component. One of the benefits of the Open by Design is the prototyping-iteration approach coupled with ongoing community engagement and user feedback. This provides builders with feedback on user experiences for the complement of services offered through the infrastructure, ideas for new features, as well as information on the value of the offering to the user community. What can we learn from existing evaluation frameworks?

Typically, infrastructure is evaluated based on “classical” technical criteria such as performance and security. While these are crucial metrics, Edwards et al. (2003) argue that we need to distinguish between technical workability and value for end-users. Ideally, infrastructures aim to support ongoing iterated patterns of relationships between human beings (Bijl-Brouwer 2017). Bollier and Helfrich (2015) call for evaluation to be oriented toward the vitality of the system measured through sustainability, resilience, and support of living things, rather than efficiency or profit maximization. This ideal is embedded in the United National Sustainable Development Goals (SDGs) (United Nations 2015), a global partnership that has developed a shared framework for peace and prosperity for people and the planet. SDGs were created through an open consensus process, are openly available, are internationally recognized across many sectors, and are imminently suitable for open evaluation purposes. In particular, SDG 9 calls for building resilient infrastructure, promoting inclusive and sustainable industrialization, and fostering innovation. This is the overarching frame in which all research infrastructure can be assessed.

The OECD decision support framework for investments in research infrastructures (OECD 2019), while based on large facilities, is well-suited also for assessing digital platforms. Both facilities and platforms are infrastructures purpose-built for research, and are expected to serve multiple community stakeholders over a long time period (decades) through provision of tooling, educational resources, and/or economic impact. The community-developed OECD framework includes 25 Core and additional standard Impact Indicators. The Core indicators are categorized by Scientific impact; Technological impact; Economic impact; Training and education impact; Social and societal impact. Subsequent community work extended these infrastructure indicators to include global-scale interoperability (National Academies of Sciences, Engineering, and Medicine 2018a). However, impact is difficult to assess because many outcomes are indirect and produced by users downstream rather than while using the infrastructure itself.

Starting in the 2020s, the Australian Research Data Commons (ARDC) has required infrastructures in their national research portfolio to integrate evaluation components, and encouraged the adoption of persistent identifiers as a strategy to “connect the dots” between researchers, data producers, data users, and the data itself and infrastructure(s) that house and deliver the data (ARDC 2020). Similar national-level policy efforts are underway in the UK, Canada, and Netherlands, and are captured in the Research Data Alliance (RDA) National PID Strategies Working Group (WG), which describes benefits of widespread and consistent PID adoption for open research infrastructure implementation frameworks (Brown et al. 2023).

As with the OECD frameworks, these national infrastructure evaluations have largely focused on downstream bibliometrics – publications and citations. While researchers use infrastructures and eventually write papers, the key value of infrastructure – research process support – as well as its openness is not captured using these metrics. Nor are infrastructures often designed using user experience principles to support generalist use (Osagie et al. 2017) or capture information that can be used for evaluative purposes (Seffah et al. 2006, Amorim et al. 2017, Haak et al. 2020).

Aligned with Open by Design principles, we propose an evaluation framework that is integrated into the design and build process. One that is based on the concept of research infrastructure as a public good as presented in Figure 1. Infrastructures typically operate outside the control of a single stakeholder, so it makes sense to relate the assessment of impact to its role in and fit in the environment it inhabits: Is the infrastructure adopted? Does it evolve? Does it scale? (Henfridsson and Bygstad 2013). Vora and Dolan (2022) suggest a framework that has both supply (design, implementation, and governance) as well as demand (scale, user experience, and outcomes/impacts) components.

The underlying question, then, is how does the infrastructure enrich the community it serves? Fig. 3 illustrates an Open by Design evaluation framework, which encompasses adoption, business sustainability, governance, community engagement and support, and relationship and trust building. We will explore how this framework incorporates Open Science principles and review its implementation in Section 4.

The ARDC has refined OECD impact indicators to create an Impact Reporting framework for Australian national-level data assets (ARDC 2020). The intent was to drive application of FAIR principles and persistent identifier (PID) strategies, and promote openness and transparency in data management. However, even with a funding supplement to promote implementation, the adoption barrier for those building the data assets was substantial. Infrastructure builders were focused on the specific use case they were solving for, and did not have the knowledge or capacity to learn about open principles and how to implement them.

Working in collaboration with the project teams in the Health Studies Australian National Data Asset portfolio and ten disciplinary National Data Asset projects, one of the authors (Haak) tested an Open by Design approach to translate the evaluation framework into practical implementation plans. These projects included field instrumentation and data processing and pipelines for ocean microbes, biomedical data discovery, geophysics, barley science, ocean wave dynamics, plant trait data, air pollution, species distribution data, electronic medical records, and a cancer registry for companion animals.

As a first step, teams reviewed webpages and initial funding proposals for each infrastructure project and extracted infrastructure services, beneficiaries, and impact goals for each project. The breadth of disciplines and use cases represented by the projects provided an opportunity to test for shared approaches and features while also identifying unique features and functions. Teams explored the question set described in Table 2 in a series of online workshops with project leaders from April to June 2023. Using whiteboarding, teams mapped out the value proposition and core technical features of each infrastructure. Then, teams focused on the behavioral aspects: stakeholders and primary and secondary beneficiaries; options for user and community engagement; evaluation strategies.

From findings elicited in these conversations, project feature maps were updated and then and then options were explored for implementing open principles. After highlighting structural congruences between infrastructures, existing modular architecture components were discussed, as well as standard interfaces and workflows, and schemas and persistent identifiers that can engage the intended infrastructure users while also providing information about usage patterns and longer term impacts on the research field or area of practice.

Based on this work, teams revised their infrastructure outcome statements to incorporate data collection strategies, available telemetry and instrumentation toolsets, and data processing pipelines to support integrated evaluation, as well as explicit alignment with UN Sustainable Development Goals (United Nations 2015).

Through this iterative consultative process, we found that the Open by Design approach provided a general design process as well as practical and flexible design elements that allowed builders to engage with open infrastructure principle frameworks across research disciplines, target users, and infrastructure types.

If asked, many stakeholders involved in building and managing research tools, standards, protocols, and services would readily say they believe in an open ethos. They may openly share (or be willing to share with anyone who asks) the content and software elements that they create and maintain. Some even use open licensing mechanisms (e.g., open source, open access) to indicate to others how their creations can be used. Still, the gap between professing openness and really practicing openness is wide.

Invest in Open Infrastructure (IOI), has been actively testing ways to improve investment in and adoption of “openness” (and other key scholarly values and principles) and to better understand what barriers stand in the way of its implementation for different stakeholders in the research and knowledge ecosystem. In 2022, IOI began inviting infrastructures to participate in Infra Finder (IOI 2024), an information tool that would showcase “open infrastructures” in research and scholarship, providing up-to-date and verified information about key attributes of these tools, protocols, and standards to help their prospective funders and adopters to quickly see what options are available and to gauge their fitness for purpose across a number of vectors. Infra Finder launched in April 2024 with an initial set of 57 infrastructures, and in its first year, it added more than 50 additional infrastructures through a rolling intake process.

The attributes Infra Finder focuses on and shares about these infrastructures are deeply informed by IOI’s work with values and principles frameworks, including POSI, FOREST, FAIR, CARE, and CHAOSS (CHAOSS 2025). Each infrastructure’s listing includes technical details (licensing, dependencies, road map) as well as information about its community engagement mechanisms, governance model, business form, and key policies.

To participate, the represented infrastructures submit information about their entity and its practices, including its launch date, maintenance status, technical documentation, licensing, programming languages, standards used (metadata, PIDs, etc), how and by whom it is hosted, integrations with other tools/standards, community norms and contribution pathways (e.g., code of conduct, contribution guidelines), governance, policies, organizational structure, and engagement with values frameworks (FAIR, POSI, etc). IOI vets the information for each entity at submission and again annually, ensuring clear evidence exists for each claim that is made. IOI also evaluates whether the infrastructure conforms to at least one of its key criteria. Along the way, there is usually some level of dialogue between IOI and the infrastructure to elaborate or clarify particular points about the infrastructure.

What IOI has learned in the first year of active hosting of this resource is just how much the process of engaging with IOI (and with the formal set of questions they answer in order to be included) has meant to its represented infrastructures. For these infrastructures, being represented publicly in this way has motivated and enabled them to clarify, both internally and externally, how they evidence their values.

Participating infrastructures have actively increased their values alignment, citing Infra Finder’s intake process as “kind of an x-ray of our operations and way of working” that “certainly revealed some areas where we could improve” (Collister 2024). The intake process has prompted infrastructures to create or make more readily available their policies and documentation. It has also helped some to question and pursue changes in their governance models, and to prioritize work they may not have considered before, including creating a “living will” or sunset plans describing where open assets will (and won’t) go in the case of a shut down or transition and budgeting an operational reserve to fund these activities if/when needed. Infrastructures that are not (yet) featured are also reporting their use of the framework to better understand and strategically build towards more of the mission-aligned characteristics that Infra Finder surfaces.

Much like the values and principles standards from which it has drawn (POSI, FOREST, etc), Infra Finder as a framework actively helps infrastructures to consider and evaluate their options for better values alignment; it also provides them with a crucial lens into the practices and behaviors modeled by their peers.

It also helps make others - the funders and adopters - more aware of differences in models deployed by various infrastructures and gives those funders and adopters ample details to help them structure a conversation with the infrastructures that interest them.

IOI created Infra Finder for a different, though related, purpose - to provide adopters and funders with additional, clear information about open tools, standards, and protocols that they could consider in hopes that it would increase their awareness and engagement with these infrastructures. In a 2025-2026 research study, “Measurement of Community Health Indicators,” funded by the Digital Infrastructure Insights Fund, IOI has been exploring how community health frameworks (CHAOSS, POSI, FOREST) are experienced by three key groups: infrastructures, adopters (institutions that use the software or standard, for example), and funders (e.g., philanthropies providing grant funding). Preliminary findings based on discovery interviews with academic libraries, research institutions, funders, technology creators, vendors, and collectives, include:

Many adopters/uses are primarily driven by practical needs like cost and fit-for-purpose. For this group, community health attributes are seen as value-added factors rather than decision drivers.

As reflected also in the ARDC example above, two key barriers to the use of open infrastructure principle frameworks are the time and attention that are required to understand, let alone explore use of them. Embedding the frameworks in a reference environment like Infra Finder incentivizes players to engage with the key values and principles in very practical ways, thinking explicitly about their value to the work they do rather than as a philosophical exercise or other time-consuming, nice-to-have but exhausting-to-undertake requirement.

In other words, part of what IOI is learning from running the Infra Finder service is that groups that resist spending time and energy on values and principles (either infrastructure providers evidencing them or funders and users finding evidence of them) may be highly motivated to provide clear signals of their open practices in an informational system that helps to spotlight these elements as design features (e.g., open licensing). The Infra Finder environment may help to encourage and incentivize “open by design” work without requiring separate evidence trails created for or provided by individual “principle” frameworks.

Library publishing, an early form of what is now referred to as diamond open access publishing, started in the early 2000s as a low cost way to put a range of content online, including journal and book content, as well as grey literature and digital humanities content. The California Digital Library (CDL), for example, launched the eScholarship Publishing program in 2002, offering free publishing solutions for the University of California (UC) faculty, students, and staff. eScholarship has become an exemplar for large scale diamond open access, providing publishing solutions for now over 90 scholar-led OA journals affiliated with the 10 campuses of the UC System. CDL supports publishing at this scale with a team of publishing professionals and technical staff, a level of dedicated resourcing that is not necessarily available to smaller libraries or other open access providers.

For smaller library publishing programs, externally hosted publishing platforms, including bepress have historically enabled them to serve their communities with diamond OA publication options. In 2017, when bepress, the main outsourced library publishing solution, was acquired by a commercial publisher, library publishers sought new options aligned with open, non profit, and community-led values. Many library publishers have also been looking for ways to expand their programs to provide a compelling alternative to commercial and society publishers in alignment with these values.

The Next Generation Library Publishing (NGLP) project launched in 2019, with the goal of knitting together open access components and filling in gaps in order to provide end-to-end infrastructures for submission, production, and distribution of journals at a level that would compete with proprietary commercial publishing options (NGLP 2022). With a grant from the Arcadia Fund, NGLP was formed by CDL, Strategies for Open Science (Stratos), and Educopia, and launched with a year of community engagement to gather requirements. The library publishing community indicated that, along with values-alignment, they wanted to build on existing infrastructures whenever possible and they would need hosting and production services since many could not manage hosting their own open source technologies in house.

Building on this needs assessment, then-NGLP Product Owner Sarah Lippincott and then co-Principal Investigator Katherine Skinner, created, vetted, and released the FOREST Framework (Lippincott and Skinner 2022), a new methodology and instrument for assessing alignment with community values. This resource codified and made explicit a set of open values that could inform the design process for the NGLP project. It provided guidance on how to embed and enact common values in open infrastructures, including Financial sustainability; Openness; Representative governance; Equity, accessibility, and anti-oppression; Sharing of knowledge, and Transparency (FOREST), at any stage of their development. For the NGLP team, FOREST served as a framework for both the project’s development and business-planning work, from determining what OS components to use and how to build and host the code base, to planning for its future governance and management as an open infrastructure.

NGLP has been designed to unite systems run by open infrastructures such as Open Journal Systems (OJS), Janeway, and DSpace, with a newly built publishing front end called Meru, built by Cast Iron Coding. Leveraging a modular framework, Meru enables aggregated display of publications managed in multiple back-end systems, supporting the growth of library publishing programs without requiring migration off of the often disparate tools they use. The NGLP project ran several pilots in 2021-2022 and received a lot of positive feedback from partners. The University of North Carolina (UNC) Press and Longleaf Services were able to turn this pilot into a successful business offering for local libraries and other customers, leading the way for a new set of mission-aligned services for diamond journal publishing.

Then, in 2022, NGLP received a grant from IMLS to explore scaling the Meru effort with partner University of Iowa Libraries, a member of the Big Ten Academic Alliance (BTAA) of libraries. Recently, NGLP announced a partnership with BTAA to bring scale to the effort, working with multiple member libraries to explore a shared solution for the consortium to aggregate and display its combined publishing portfolio. Despite a funding shortage from IMLS, which required recalibrating goals for moving the University of Iowa to a production-ready system, the BTAA pilot is underway, and will be another proof point that the modular, open source solution is flexible and can meet the needs of different diamond open access publishers. With current funding challenges in institutions in the US, it is unclear whether diamond open access will receive enough support, but having choices in open source solutions will reduce costs and enable more cross-institutional collaboration and scale.

The NGLP example is demonstrating that being intentional at the start of a large-scale infrastructure initiative can lead to alignment with community values and that separate and even competing services can share open infrastructures and assemble novel solutions that meet the needs of their user community.

We have taken learnings from each use case and here propose a flexible evaluation framework that combines public goods and open infrastructure frameworks (Fig. 1 and Table 1) with Open by Design product and process principles (Fig. 2 and Table 2) and the Open by Design evaluation framework (Fig. 3) organized by SDG, that can be used by any research infrastructure. Ideally, this framework would be consulted during the design phase, but it can also inform existing infrastructures with goals to align with open science principles.

The Open by Design framework aligns accountabilities and metrics for research infrastructures designed as research resource technology (SDG 9, Table 3), as training tools (SDG 4, Table 4), and as foundations for research and policy studies (People & Planet SDGs, Table 5).

Each infrastructure can ensure it meets the facets of open by design while also customizing its framework depending on its functional specifications and features, and also can use the framework to ensure that the product offering includes components of FAIR, CARE, POSI, and FOREST frameworks, as is pertinent to the product offering.

Once the design framework is established, the next step is the evaluation component. Given that each infrastructure has a specific constellation of services, activities, and beneficiaries, it is important that each customizes its community engagement and telemetry strategy. At the same time, there are similarities between infrastructures, and there is good reason to harmonise reporting when possible to enable effective internal and community oversight. To balance these needs, we have created a set of impact metrics for each Impact Area, aligned with the SDG pillars of Infrastructure, Education, and a grouping of SDGs around People-Planet. Within each, we ask a set of questions, and the infrastructure host can select which questions and measures are most relevant. The frameworks are shown below in Table 6, Table 7, and Table 8.

Measuring adoption and use of research infrastructures is complex and flexible, but do-able. As shown above, to capture impact we need to use a range of approaches, including user surveys, usage data, and outputs monitoring. A key component of any measurement strategy is a combination of pre-planning and telemetry (Gagnon and Swanson 2023), which can include the integration and use of persistent identifiers (PIDs) in data pipelines and workflows (Haendel et al. 2021, Haak et al. 2018, Brown et al. 2023) as well as in-workflow user surveys, monitoring of help desk inquiries, and community meetings.

For individuals and organizations that design, engineer, build user communities, and maintain research infrastructures:

Engage on international data standards

For organizations – including government sector Ministries and Agencies and philanthropic organizations — that develop policies, funding programs, and training related to research infrastructures:

Work with infrastructure hosts and builders at project initiation and regularly throughout the project to signal the importance of openness and transparency and perform regular assessments.