Evaluation of the Observation Systems, Planning and Preparatory Activities of the Space Utilization Program

Relevance

Satellites play an essential role in the lives of Canadians, and the SUP ensures that Canada takes full advantage of the potential offered by space for the benefit of its citizens. It supports the mandates of other entities that may have a positive impact on Canadians, for instance with respect to the environment, national security, and disaster management. Indeed, the SUP makes it possible for public, industrial and academic organizations within Canada to access space data, information and services in order to better understand the Earth, inform policy and decision making, and connect Canada and Canadians. The SUP's objectives and activities are aligned with the federal government's priorities and core responsibilities. The SUP is a pillar of the CSA, enabling it to support Canada's space ambitions. Furthermore, it is of strategic importance to many other federal departments in achieving their respective mandates. Canadian satellites, such as the RADARSAT Constellation Mission (RCM), provide considerable benefits for Canada, which justify the significant resources allocated to their design, development, and operation. Over the years, the SUP has accurately identified the needs of its target groups and it has designed and supported missions that meet the majority of the needs identified. However, the federal government's Earth observation data needs are increasingly numerous, and the SUP meets these needs within the limits of its human and financial capacities.

Effectiveness

Most of the data on the SUP's performance were available, reliable and useful for the purposes of the evaluation. In terms of effectiveness, the output and outcome targets were generally achieved or even surpassed. The number of agreements signed by the SUP has fluctuated over the years, but the number of images^{Footnote 1} obtained from synthetic aperture radar (SAR) satellites significantly increased between 2019 and 2022 due to the launch and commissioning of RCM in 2019 and the efforts undertaken by the SUP to maximize the availability of the data collected. The results of the last few years demonstrate the unparalleled effectiveness and capacity of RCM compared to its predecessor. As early as fiscal year 2019-20, the 53,648 images obtained through the CSA's SAR satellites included 32,062 images from RADARSAT-2, and 21,586 images that were already captured by RCM after only a few months of operation (not to mention its enhanced engineering capabilities). Nonetheless, RCM data accessibility proved to be more complicated than expected due to the high volume of acquisition requests, the sensitivity of some images captured, and important needs related to national defence.

Efficiency

The data collected indicate that OSPPA-related spending is appropriately managed and that actual expenditures are lower than anticipated in the financial forecasts, which include allocated funds for risk management. Over the years, the SUP has demonstrated its ability to work effectively with various internal and external stakeholders as part of an interdepartmental and international approach. The activities related to planning and developing satellite missions are harmonized both within the Program and the CSA, as well as across the Canadian space ecosystem and with international partners. However, unlike the CSA's other programs, the SUP operated with two Performance Information Profiles (PIP) from 2019 to 2022, i.e. until the second edition of CSA's PIPs in March 2022. The SUP strategic planning team has made significant efforts to improve performance measurement and create efficiencies, in collaboration with the team responsible for coordinating the Programs' performance measurement at the CSA. These efforts resulted in improvements, the most noteworthy being the consolidation of methodologies and indicators into a single PIP. In addition, international collaboration enables the Program and the CSA to engage in more opportunities and reduce costs, among other benefits.

The evaluation has brought to light challenges related to the planning and development of observation systems between 2017 and 2022. Overall, the prioritization of SUP's missions and activities was a challenge during the period covered by the evaluation, and the data collected show that this challenge results from various interrelated factors. Among other things, the Program branch responsible for strategic planning and stakeholder engagement is relatively new. In addition, the SUP's business model has evolved over the past few years, shifting from a model where almost all human and financial resources were geared towards a shared objective (RCM) to a model where several missions are progressing in parallel. This model has its advantages, but it also has an impact on efficiency because employees are working on developing various missions in a context where the number of employees is limited. Lastly, the CSA does not have a budget envelope to fund the development of specific initiatives from their inception, considering that satellite missions are mainly selected based on federal priorities, departmental objectives, the needs of target groups (e.g. federal departments), collaboration opportunities, as well as the risks and functional requirements of these initiatives; which require prior efforts and the right timing. However, it is important to mention that the efforts made between 2017 and 2022 made it possible to confirm future investments in observation systems. As such, the Program is now increasing its focus on developing the missions it has committed to delivering.

Furthermore, the design and development of observation systems are aligned with the CSA's governance structures and processes. However, the requirements related to the financial approval and detailed cost estimates required to pass Gate #2 of the CSA's Investment Governance and Monitoring Framework (IGMF) are considered challenging by many employees, who say it is difficult to estimate detailed costs for an observation system before even establishing the project needs and technical requirements. While this does not prevent the Program from successfully carrying out various missions, the lack of clarity on this element has caused a loss of efficiency, created tensions and complicated daily work. That said, the Treasury Board Secretariat replaced the Guide to Cost Estimating with the Guide to Costing in May 2023. It is therefore still too early to assess the repercussions of this new guide on the SUP activities. Moreover, the CSA's corporate teams are currently working to establish a standardized method and tools to streamline the costing process at the CSA.

Within the Program itself, the evaluation findings indicate that some elements could be clarified, documented, and centralized in order to optimize the decision-making process and to reduce staff workload in the short-term. First, there is currently no centralized management tool or process within the SUP to provide an overall portrait of the human resources assigned to each phase for all ongoing missions. This is particularly challenging at the outset of initiatives since the planning function is divided among the Program branches. Second, there is no centralized database within the Program to compile and organize all the needs identified over the years by employees of the different branches, which increases the number of consultations held by the Program and complicates the establishment of prioritization criteria related to the needs of the target groups. Finally, the evaluation found that roles and responsibilities should be clarified across three levels, i.e. within the Program (e.g. between branches participating in project teams), between the Program and the CSA's other directorates, and with other federal departments. The data collected also indicate that the work to develop a vision undertaken by the SUP over the last several years will only truly succeed as part of a concerted effort both within the CSA as a whole and among the wider federal government. On this last point, the onus is on stakeholders to determine the desired level of CSA involvement in the SUP areas of responsibility, and to identify the actions to be taken to achieve it.

Recommendations

In light of the key evaluation findings described above, the evaluation recommends the following actions to support the Space Utilization Program's decision making and efficiency:

1. Clarify the operational vision, roles and responsibilities, and develop a management framework that reflects the Program's direction.
2. Centralize, compile and organize information related to the needs of target groups.
3. Ensure centralized and holistic planning, management and monitoring of the Program's human resources related to observation systems, for each phase of all satellite missions undertaken by the Program.

Purpose and Scope

This report presents the findings of the evaluation of the Observations Systems, planning and preparatory Activities (OSPPA) of the Canadian Space Agency's (CSA) Space Utilization Program (SUP), in accordance with the commitment set out in the CSA's Departmental Evaluation Plan and the Treasury Board of Canada's Policy on Results (2016). The evaluation covers the main evaluation issues set out in the Directive on Results (2016), namely relevance, effectiveness and efficiency, and Gender-Based Analysis Plus (GBA Plus). It is part of a constructive approach and aims to provide a neutral assessment supported by evidence.

The evaluation was conducted by the CSA's Audit and Evaluation Directorate between January 2022 and September 2023, with the support of PRA Inc. It is the first evaluation to cover all SUP business lines. It covers planning, preparatory activities and the development of observation systems for which the SUP is responsible, for the period from April 1, 2017, to March 31, 2022. In response to specific information needs from SUP representatives, the evaluation also paid particular attention to decision making, priority setting, governance, and values and ethics considerations. While data collection was carried out in 2022, the evaluation considered all subsequent changes while drafting and publishing this report in order to ensure recommendations are relevant, and to present information in accordance with the most up to date information.

This evaluation is the first in a series of two separate but complementary evaluations of the CSA's SUP. It includes the "Planning and Preparatory Activities" and "Observation Systems" sub-programs, to which almost all SUP activities and resources are assigned. Data utilization and application development are excluded from the evaluation scope because they will be subject to a forthcoming evaluation.

Methodology

Case studies. Nine case studies were conducted as a part of this evaluation. They covered satellite missions of various types, scales and stages of development, namely the RADARSAT Constellation Mission (RCM), the Surface Water and Ocean Topography mission (SWOT), the WildFireSat mission, the High-altitude Aerosols, Water vapour and Clouds Mission (HAWC), the SCISAT mission, the Solar wind Magnetosphere Ionosphere Link Explorer mission (SMILE), the WaterSat mission, the Arctic Observing Mission (AOM), and the Canadian Hyperspectral Mission (CHM). The case studies are based on the analysis of data collected from the document review and interviews. Their purpose was to highlight practices related to the SUP's implementation, comparing various features of the satellite missions.

Document review. The evaluation is based, among other things, on the careful analysis of public and internal documents related to the SUP, and specialized books on project management. The objective of this method was to gain familiarization with the Program environment, identify best practices, and evaluate the Program's relevance, effectiveness and efficiency.

Internal data. Internal CSA data were used to evaluate the Program's effectiveness and efficiency, as well as to triangulate information and deepen the analysis. These data include financial data, human resources data and performance data, among others.

Literature review. A review of previous SUP evaluation reports, which comprised a wide breadth of SUP's activities, aided the evaluation team in their understanding of the Program.

Interviews with key informants. Semi-structured interviews were conducted to collect information on SUP's implementation and to expand on, corroborate and clarify the information obtained through other data sources. They contributed to documenting the planning process, facilitating factors, challenges encountered by stakeholders and efficiency considerations. A total of 76 individuals from different stakeholder groups were consulted through 55 individual and group interviews. More specifically, 33 interviews were conducted within the CSA among SUP teams^{Footnote 2}, CSA senior management and representatives from other CSA directorates^{Footnote 3}, and 22 interviews were conducted with external stakeholders from other federal departments^{Footnote 4} and universities^{Footnote 5}.

Observation. The evaluation team attended the 2nd edition of the National Forum on Earth Observation hosted by the CSA from October 4 to 6, 2022. They also attended a few meetings of the Space Utilization Steering Committee (SUSC) and the Space Utilization Management Committee (SUMC). The observation sessions helped provide an understanding of SUP's operations and corroborated the information collected through other data sources.

Consultations. An advisory group comprised of employees from both the Program and from across the CSA with a direct link to the Program was formed to give advice, feedback and guidance during the evaluation.

Comparative analysis. The evaluation considered how other organizations in charge of large-scale projects (including those outside of the space sector) carry out their strategic planning, in order to identify best practices and lessons learned. This analysis focused on two of CSA's partner space agencies (ESA and NASA) and on three large-scale Canadian projects, namely the National Shipbuilding Strategy, the construction of Canadian Forces Base Petawawa, and the construction of the Samuel-De-Champlain Bridge.

Limitations

Evaluation scope. The decision to evaluate the entire SUP in two separate but complementary evaluations was made with the intention of distributing efforts and more quickly responding to decision makers' information needs. It is prudent however to mention that not addressing the topic of data utilization as part of this evaluation does result in some limitations, since data utilization is the primary justification for the design and development of satellite missions, and because it is also an essential component of the SUP's expected outcomes.

Likewise, it is difficult to evaluate the planning, preparation and development of observation systems from April 2017 to March 2022 only, considering that the life cycle of satellite missions spans multiple years, if not decades. Indeed, some missions under review were planned and prepared well before the period covered by the evaluation (e.g. RCM) or will continue after this period (e.g. WildFireSat). It is therefore too early to evaluate the outcomes of certain practices and decisions.

To mitigate these limitations, the evaluation triangulated information using documents developed both before and after the evaluation period. Furthermore, it considered certain aspects of data utilization when directly relevant to the evaluation.

Financial data for fiscal year 2017-18. The financial data for fiscal year 2017-18 are difficult to compare with data from following years because of the different financial structure used by the CSA until April 1st, 2018. To mitigate this limitation and provide an indication of expenses and FTEs related to OSPPA in 2017-18, the evaluation and the Finance directorate attempted to align the amounts coded under the CSA's former Program Architecture with the CSA's current structure. It is important to note that there remains a potential margin of error.

Performance data for fiscal year 2017-18. The methodology employed to measure the performance of CSA programs for fiscal year 2017-18 is different from the methodology used to measure results from subsequent years. The CSA transitioned from Performance Measurement Strategies (PMS) to Performance Information Profiles (PIP) in 2019 in light of the Treasury Board's Policy on Results (2016). In 2017-18, the results of the SUP were essentially calculated in terms of areas of activity and subprograms under the CSA's former Program Architecture, whereas today's results are calculated per department and per service, making it nearly impossible to compare fiscal years, or the elements included and excluded from the evaluation.

General Overview

Program Logic

As set out in the PIP, the SUP's design is based on the assumption that the program's activities, i.e. partnerships, space missions, and support to users, will enable the program's target groups to use the space services and information provided (CSA, 2017a). Through the activities carried out and the outputs produced, the SUP expects to achieve a series of three immediate outcomes and four intermediate outcomes, which should ultimately allow the SUP to contribute to the CSA's final objectives, i.e. the Departmental Results Framework (DRF). The following figure presents the SUP's logic model that was in effect from December 2017 to March 2022, that is to say for almost the entire period covered by the evaluation. The logic model shows the causal links between the SUP activities, outputs and expected outcomes.

Figure 1. The SUP's Logic Model (2017-2022)

Activity #1 – Partnering (consultation/negotiation with stakeholders and users): This activity involves the engagement and collaboration with Canadian and international Program and stakeholders in order to:

1. Identify space solutions to meet needs on Earth;
2. Define, design and develop space missions;
3. Take part in missions led by other entities (e.g. NASA and ESA);
4. Access data (e.g. satellite scenes) and signals from missions that CSA is not involved in, but that benefit Canada;
5. Engage resources, including infrastructure to generate intermediate outcomes with these data and signals.

Activity #2 – Space missions (development, deployment and exploitation of space solutions for Earth): This activity includes the planning and implementation of satellite missions (or organized access to missions), in order to give users reliable access to space data and signals.

Activity #3 – Support to users (assessment/prioritization of science and user capability): This activity includes the support provided by the Program to users of space data and signals, to advance their capabilities to transform Earth observation data and space signals into final products. It is directly related to data utilization and application development.

Governance, Roles and Responsibilities

Several stakeholders and committees are responsible for the governance of activities related to OSPPA.

Program senior management: The Director General of the program assumes overall responsibility for activities carried out under the SUP. The Director General's duties are carried out in accordance with the applicable Treasury Board of Canada policies and the CSA's Investment Governance and Monitoring Framework (IGMF). Internal committees on space utilization (SUSC and SUMC) are both chaired by the Director General. The latter is supported by three directors who are responsible for the activities carried out within their respective branches.

• Space Exploitation: Responsible for satellite operations and operational missions, such as the RADARSAT Constellation Mission (RCM) and the mission that will follow it. Also typically responsible for mission management starting from IGMF Gate #2.
• Sun-Earth System Science: Responsible for scientific missions, designing and directing the development of new scientific missions, and co-funding (which is also discussed in the two other branches).
• Planning, Engagement and Applications: Relatively new branch that underwent a few changes over the evaluation period. Responsible for the SUP development, including strategic planning, performance measurement and partnership engagement.

Space Utilization Management Committee (SUMC): Responsible for the day-to-day management of the SUP, including human and financial resources for the SUP activities. Directors are responsible for applying measures resulting from decisions made by the SUMC chair in relation to their respective branches, and managers are responsible for directing their employees in the execution of those activities.

Space Utilization Steering Committee (SUSC): Establishes strategic direction and formulates directives regarding the SUP investments. The SUSC chair makes decisions related to investments after consideration of member recommendations (CSA, 2019a), for instance relating to the IGMF and comments from external partners (e.g. federal departments and scientific advisory committees), following consultations on needs for space data/signals.

Integrated Investment Review Board (IIRB): Central decision-making body of the CSA responsible for the proper management of investments, including the SUP's investments, to ensure their financial viability and that they produce the expected outcomes. It also works to coordinate and optimize CSA resources. It has decision-making authority regarding the approval and advancement of most SUP missions.

Interdepartmental governance committees: The CSA space missions carried out under its financial and project authorities are usually only subject to internal governance bodies (CSA, 2020b). However, depending on the type of investment, the complexity and the project risk, external governance committees may be solicited, such as the Deputy Minister Governance Committee for Space and the Director General Space Committee. RCM is an example of a Major Crown Project that required the decision making and oversight of interdepartmental governance committees (CSA, 2020b).

Federal and international stakeholders: Certain federal departments involved in space missions may have decisional authority on the furthering or discontinuing of space missions. Furthermore, when Canada takes part in satellite missions designed and developed by foreign agencies, they are the main authorities for those missions. As such, they decide the conditions and requirements to the missions (e.g. schedule, project continuity or stoppage).

Program Resources

Alignment between Program Objectives and Government Priorities

The Program aims to make sure that Canada takes full advantage of the potential offered by space for the benefit of its citizens, by ensuring that the Canadian public sector, private sector and academia have access to space data, as well as space-enabled information and services, to inform policy and decision making, connect Canada and Canadians, and understand Earth (CSA, 2017a). More specifically, the SUP aims to provide space-based solutions in order to:

• Assist the Government of Canada and broader public sector to provide services under their respective mandates with respect to national priorities;
• Enable the growth and competitiveness of Canadian industry (e.g. highly skilled jobs);
• Provide the scientific community with the necessary input required to perform research in areas related to the Sun-Earth System;
• Increase Canada's international influence (CSA, 2017a).

Needs of Target Groups

Identifying User Needs

Needs identification is mainly carried out through consultations with target groups, i.e. the public sector (especially federal departments), academia and industry. The consultations are intended to meet differing needs and they take various forms: ad-hoc consultations for the needs of a specific mission, working groups, interdepartmental and international committees, specific discussions during national forums, or even for the implementation of strategic documents. Thanks to these consultation mechanisms, the SUP was able to successfully identify target populations needs and have these needs reflected in its activities.

With respect to strategic planning, the program engaged in several efforts to identify the needs of target groups either to plan for future missions or to ensure the continued relevance of the systems currently in orbit. Of particular note, it organized the first National Forum on EO in November 2018 and led a series of consultations with various stakeholders from the public sector, industry and academia, which led to the development of the Engagement paper: Space-Based EO in June 2020, and the Canada's Strategy for Satellite EO in March 2022. The internal and external stakeholders interviewed on the subject stated that the Strategy was a significant step forward in identifying future EO needs in Canada, especially given that it provides the basis for a pan-governmental approach to using satellite EO data. Some external key informants pointed out the critical role that the SUP plays in this process, as well as noting the efforts it makes to successfully carry out this major consultation and coordination exercise.

Figure 4. Satellite Earth Observation (EO) Value Chain in Canada (CSA, 2020c)

"It's during Phase 0 or pre-Phase 0 that we identify needs, define the needs of users and missions. There may be changes to the needs or technical requirements later on in the process, but usually the concept studies are carried out with partners."

In the specific context of missions, needs are mainly identified during the planning and preparation phase of satellite missions, in accordance with the CSA's governance process under the IGMF. Among other things, mission managers must develop a business case and a GBA Plus for every mission in which the SUP is considering investing. Collaboration with stakeholders takes place at different stages of the value chain to identify needs and adapt missions as needed.

Meeting User Needs

The evaluation found that the missions designed or supported by the SUP in the last five years meet demonstrable needs in Canada's economic, scientific and social spheres. The program dedicated almost all of its efforts and resources to meet the EO needs of federal departments, since they are the biggest EO data users, and because EO was the most important domain with regards to the SUP's mandate. That said, the SUP's domains of activity aim to meet the needs identified across all relevant stakeholder categories. These three aforementioned categories of stakeholders (public sector, private sector and academia) create a mutually reinforcing ecosystem in which each contributes to providing for, and supporting, the other two.

EO has asserted its place as one of CSA's most important spheres of activity since the launch of the RADARSAT-1 mission in 1995, and this importance continued with RADARSAT-2, which was launched in 2007 and remains operational, exceeding its initial operational end date of 2015. In June 2019, the SUP enabled the CSA and Canada to make history by launching Canada's first satellite constellation mission, and the first mission owned and operated exclusively by Canada: the RADARSAT Constellation Mission (RCM). In 2017, the evaluation of the CSA's EO business line stated that the change from a privately owned satellite (RADARSAT-2) to a government-owned satellite (RCM) would help the CSA to more effectively meet the needs of government departments and organizations (CSA, 2017c). EO satellites make it possible to obtain data on oceans, ice, land masses and the atmosphere, and they are useful for monitoring, protecting and understanding the Earth, while helping in the management of resources and ensuring the safety of Canadians (CSA, 2023b). The currently active Canadian EO satellites are RCM, RADARSAT-2 and SCISAT, and Canada is contributing to international satellite missions such as SWOT. Furthermore, Canada's WildFireSat and HAWC missions (13 universities are involved) are currently being developed. Lastly, the Arctic Observing Mission (AOM) and the mission that is going to replace RCM are currently being prepared.

At the height of the RADARSAT Constellation Mission (RCM) project's development, almost all of the SUP's resources were assigned to the mission, and approximately 300 people were working on its construction in more than 50 Canadian companies. 125 suppliers in 7 provinces played a role in the project (CSA, 2019c). The main purpose of RCM's data is to support the services and needs of the federal government, but external stakeholders can also access most of the data, subject to certain national security restrictions.

The RCM is the SUP's biggest investment to date and one of the CSA's most significant investments ever made. It was designed and launched first and foremost for Government of Canada users, even though it also helps meet the needs of the scientific community for space-based information and contributes to the prosperity of the space sector. RCM was designed to ensure the continuity, growth and improvement of the capacity to use synthetic aperture radar (SAR) technology of the RADARSAT© missions, as well as to strengthen national sovereignty, ensure safety, protection and essential services to Canadians, and support the Canadian space sector. In order to meet these needs, RCM consists of three satellites which ensures better surveillance of Canada and the Arctic, and it is equipped with two payloads, a powerful SAR and an automatic identification system that is used for ship detection. The mission that is going to replace RCM is being prepared with an eye to the current and future user needs.

"I am preoccupied by the replacement of RCM and its lifespan."

In 2022, a report by the Office of the Auditor General of Canada titled Arctic Waters Surveillance, covering the period from April 1, 2021, to March 31, 2022, highlighted the importance of SAR satellites for Canada, along with their current limitations and related risks:

"Maritime surveillance in the Canadian Arctic relies largely on satellites that provide radar imagery and relay Automatic Identification System signals. […] For maritime surveillance, the federal government relies primarily on the Canadian RADARSAT‑2 and RADARSAT Constellation Mission satellites. […] The RADARSAT Constellation Mission satellites, already used at full capacity, cannot accommodate all the demands of federal organizations for radar imagery of Canada's territory. Every time a priority request is made, the imagery acquisition plan must be reviewed to determine which other needs will be delayed or not met. We also found that these radar imagery satellites are at or will reach the end of their expected service lives long before the planned launch dates of the replacement satellites. […] The government acknowledges that it will take another decade for the Canadian Space Agency to launch a successor to the RADARSAT Constellation Mission and that an interruption of satellite earth-observation services past 2026 is therefore a significant risk. Budget 2021 allocated $9.9 million over 2 years to the agency to plan the next generation of earth-observation satellites. At the time of our audit, the agency was developing options to replace the constellation's capabilities and to fill gaps in earth-observation services that could emerge after 2026. […] Should the RADARSAT‑2 or the RADARSAT Constellation Mission cease functioning or suffer performance limitations before new satellites become operational or other sources of satellite data are secured, National Defence and other federal organizations could be left with limited Canadian surveillance capabilities in the Arctic for years. This could significantly degrade Canada's ability to detect and track vessels in its Arctic waters and would likely increase Canada's reliance on its allies for surveillance information." (OAG, 2022)

The evaluation found that the federal government's EO data needs are numerous and constantly growing, and that the SUP is attempting to meet these needs, to the extent possible, within its human and financial capacity limitations. Target populations needs are well identified, documented and have been developed to the extent possible. That said, some non-CSA federal government representatives questioned the SUP's role and objectives, particularly because RCM plays such an essential role in national defence and its use is therefore not solely civilian in nature (see "Effectiveness" and "Efficiency" sections).

"We know the needs. The departments are clear. Regarding the chain, users do not pay for data. They want the data, they need it, they are dependent on it, but they are not the ones who pay for it."

"It takes time from the moment when a decision is made to when it can be implemented. Four months for approvals. And the time needed to coordinate with other government departments creates an additional challenge."

In addition to EO, the SUP's activities cover scientific satellite missions that are used in physics studies as well as in studying the effects of various phenomena, such as the impact of solar flares on the Earth's magnetic field (CSA 2023d). In the last few years, for example, the SUP has allocated resources to the operation of Canada's NEOSSAT satellite.

Finally, the SUP's activities also encompass satellite communications (SatCom), which enables the development of applications such as internet, television and telephone services. This field has become almost entirely commercialized, and CSA funding allocated to SatCom activities has significantly reduced over the past few years, now representing only a marginal portion of OSPPA activities. During the period covered by the evaluation, the SUP has nevertheless continued to oversee certain SatCom missions based on meeting the data requirements of user organizations, such as the Department of National Defence's data requirements.

Performance Information

In accordance with the requirements of the Directive on Results (2016), the evaluation found that the majority of SUP performance data was available, reliable and useful for evaluation purposes. A transition was made from the Performance Measurement Strategies (PMS) to the Performance Information Profile (PIP) in 2019, making it nearly impossible to compare performance data from 2017-18 with data from subsequent years, both between fiscal years and among elements included and excluded from the evaluation. The SUP results for fiscal year 2017-18 were calculated in terms of business lines and subprograms under the CSA's former Program Architecture, whereas results are now calculated by department and by service. Moreover, data on outputs are easily accessible, but incomplete, since there is no comparable for fiscal year 2017-18, and because reporting to central agencies regarding outputs was not a part of the CSA's PIP requirements for the 2017-2022 period. During this period, the SUP nevertheless undertook internal reporting on its outputs for its long PIP, in order to better report on its activities and results. However, considering that this data was collected on a voluntary basis, not all results were measured each year.

Moreover, the accuracy of the performance information for immediate outcome #2 (Space data/signals are obtained by users) is only partially validated because two different methods are used by the SUP to report on the official indicator (Number of scenes^{Footnote 12} acquired from the CSA SAR satellites). The CSA had two operational SAR satellite missions at the time of the evaluation, i.e. RADARSAT-2 and RCM. These satellites were the result of a collaboration between the federal government and industry, particularly prime contractor MacDonald, Dettwiler and Associates Ltd. (MDA). These systems have different characteristics in terms of both their engineering and property right specifications, which has a slight impact on the indicator's definition and on how the SUP can report on the indicator, which was originally created for RADARSAT-2. MDA operates and holds ownership of RADARSAT-2. The SUP's immediate outcome #2 is therefore measured based on the SAR scenes obtained from MDA, which is also made possible by the fact that RADARSAT-2 always has the same imaging parameters.

RCM, on the other hand, is owned by the federal government and is operated by the CSA. Furthermore, RCM's imaging parameters change according to the client's acquisition requests (height, definition, etc.), making it difficult to perform calculations based on the number of scenes obtained. For RCM, the SUP therefore measures this indicator based on acquisition requests and not on scenes obtained. This methodology did not restrict the SUP's ability to report on the indicator and outcome #2 for RCM, but it is worth noting that TBS had already stated that in order to effectively monitor the progress of a program, indicators should be valid, i.e. measure what they intend to measure (TBS 2010). As a result of several factors, including the fact that this indicator is linked to the SUP's DRF contribution, which will be subject to an upcoming review, the fact that the indicator is verging on data utilization and thus outside the scope of the evaluation, and that the OSPPA evaluation period happens to end just as SUP's new PIP and logic model come into force (March 31, 2022), it was deemed more appropriate to examine any potential changes to said indicator within the upcoming evaluation on data utilization and applications development. A key SUP informant also said that this methodology would be the subject of a forthcoming review in 2024, in collaboration with the CSA Governance and Performance (G&P) team, which is responsible for ensuring performance measurement coordination across CSA programs.

Similarly, the evaluation found that intermediate outcome #5 of the logic model (Space-enabled information informs policy and decision-making) was not officially measured from 2017 to 2022, because this outcome was not linked to any indicators and targets in the SUP's short PIP. Performance information for this outcome is available because the SUP measured it as part of its long PIP. However, a key informant indicated that the situation had been corrected in the second edition of the PIP.

Ultimately, the evaluation found that performance information was sufficiently available, reliable and useful to assess the SUP's effectiveness in achieving the OSPPA's intended outcomes. The evaluation also makes note of efforts made by the SUP and the CSA G&P team to ensure that the program's performance measurement is being continuously improved, which has resulted in significant accomplishments in the second edition of the PIP.

Program Activities

During the period covered by the evaluation, the Program carried out its activities in accordance with its PIP and logic model. Throughout the entire observation systems value chain, the SUP conducted a series of consultations with stakeholders and clients in order to identify their needs. It ensured partner engagement in order to define, design and develop satellite missions; that the resources needed to carry out these missions were allocated; and ensured that stakeholders had access to the satellite data and signals that they needed, through Canadian or foreign partner missions (Activity #1). By doing so, the SUP has also developed, deployed and exploited space solutions for Earth by planning and executing satellite missions (or organizing access to the missions) in order to provide clients with reliable space data and signals access (Activity #2). The SUP assessed and prioritized science and user capacity (Activity #3) in order to improve their ability to transform data and signals into products/applications. The figure below illustrates the logical connection between the three program activities identified in its PIP and its logic model (2017-2022).

Outputs and Outcomes Achieved

The SUP's mandate involves providing space data and signals to potential users. To fulfill this mandate, they must be involved in the SUP's various activities, such as needs definition. Indeed, developing and maintaining partnerships with the public, academic and private sectors is an essential part of the SUP's activities, whose role is to make sure that Canada takes full advantage of the potential offered by space for the benefit of its citizens. As previously mentioned, the SUP carried out a series of consultation and coordination activities with various stakeholders and users from the public, academic and private sectors over the past five years. These activities have led to a series of space mission partnership agreements that have been developed or supported. The SUP's leadership in coordinating efforts between national and international players was highlighted by nearly all internal and external respondents interviewed for the evaluation.

According to the SUP's internal data, it established 1,247 partnership arrangements^{Footnote 13} with stakeholders between 2018 and 2022 for all of its activities, i.e. 438 partnerships in 2018-19, 436 partnerships in 2019-20, and 203 partnerships in 2020-21. The results stem from grant and contribution partnerships made by the SUP, as well as contracts established starting in 2019. It should be noted that several partnerships can be established with the same partner. The number of agreements established is based on the number of partnerships listed in the CSA's UNITAS system, whose data are collected annually through the Performance Indicators Survey for all CSA programs. As expanded upon in the section on SUP resources, no funds were disbursed in the form of grants between April 2018 and March 2022, but $4,568,384 million in contributions were allocated for "Observation Systems" during that same timeframe. $300,160 in grants and $1,243,914 in contributions were also allocated in 2017-18.

For contracts specifically, the SUP established 11 partnerships with 12 different unique companies or universities in 2019-20, 27 partnerships with 30 different unique companies or universities in 2020-21, and 34 partnerships with 38 different unique companies or universities in 2021-22. It should be noted that the increase over the years is due to the inclusion of more types of contracts in the Performance Indicator Survey (R&D contracts) and not due to a real increase in the number of contracts. The results are related to development contracts (including subcontractors and international companies) between Gate #2 and Gate #4 of the IGMF, because pre-project contracts have not been processed uniformly in recent years. Furthermore, the results do not include contracts dedicated to RCM operations. However, it is important to mention that this situation goes beyond SUP's control, and that efforts are currently being made across the CSA to address these issues.

Through the partnership arrangements established, the SUP achieved its immediate outcome #1, as defined in its logic model 2017-2022, i.e. stakeholders and users collaborated and were engaged. The SUP created or sustained partnership arrangements with an annual average of 166 organizations between 2018 and 2022 across all SUP activities. The number of organizations with which partnerships were established and maintained by the SUP fluctuated during the evaluation period, but targets were met and surpassed for all fiscal years except 2021-22. According to a key program informant, the gap was mainly caused by the Government Related Initiative Program (GRIP) coming to an end in 2020-21, which reduced the number of partnerships made with the public sector captured through the Performance Indicator Survey, and also because of delays caused by procurement-related issues that prevented the implementation of SmartEarth initiative projects, which had the potential to generate many partnerships. The upcoming evaluation on data utilization and application development may further explore this fluctuation. While these two elements had a downward impact on the outcome, the number of partnerships has nevertheless been relatively consistent over the past four years.

Moreover, considering that the SUP's mandate is to provide space data and signals to users, it must also ensure that these data/signals are accessible to those who wants to use them. Space data and signals made available through SUP's support are obtained either from the development of a Canadian mission or by the SUP's participation in a foreign mission, and the output is calculated by the Program with each initiative counting as one dataset. According to internal SUP data, the latter made 20 datasets available per year between 2018 and 2020. Data and signals from many missions supported by the CSA are available to users, but the fact that the CSA makes data available does not necessarily mean that the target audience is using them. Therefore, the SUP measures the amount of data downloaded by users.

The CSA does not have access to user information or statistics for every CSA supported mission (e.g. ESA missions). Only CSA SAR satellites are used to measure immediate outcome #2 (RADARSAT-2 and RCM) considering that the CSA has access to information and statistics on users for these satellites. Once data have been made available and collected, they can then be used by external stakeholders to produce intermediate and ultimate outcomes. These results are going to be evaluated soon in the data utilization and application development evaluation, because the OSPPA evaluation is not intended to assess how users are using these data and signals.

702,253 images^{Footnote 14} were obtained from SAR satellites between 2018 and 2022. More specifically, 34,529 images were obtained from the RADARSAT-2 satellite for fiscal year 2018-19, and the significant increase observed from 2018 to 2022 can be explained by RCM's launch (in June 2019) and commissioning (in November 2019). The results of the last few years, particularly fiscal year 2021-22, highlight the unparalleled effectiveness and capacity of RCM when compared with its predecessor. As early as fiscal year 2019-20, the 53,648 images obtained through the CSA's SAR satellites included 32,062 images from RADARSAT-2, and 21,586 images from RCM after only four months of operation. In addition, RCM has enhanced engineering capabilities (e.g. improved quality of images captured).

The marked growth in data obtained for 2020-21 and the deviation from the target can be explained by the fact that RCM was already operating at full capacity (250,000 images) in its first full year of operation. Meanwhile, the increase in images captured in 2021-22 all of the way up to 350,270 images is attributed to Program efficiencies in improving request coordination thereby maximizing the availability of the data collected and more effectively responding to user needs.

"We went from 5,000 images per year with RADARSAT-2 in 1995 for other federal departments, to 250,000 images used per year now. It means that we have 50x more images, more hardware, more storage, more automation. Every time we build a new satellite, there are new features, it is way more performing. The key word here is 'transition.' When we left RADARSAT-2 utilization to integrate RCM, departments were ready. Without CSA support, departments would have had more difficulty preparing to absorb this amount of data. From the very first year, we saw that the utilization growth was optimized. Usually, when you start using a new system, you start by slowly using it to do tests. However, in this case, departments were prepared, so we were already almost at full capacity when RCM became operational. I think RCM is a huge success, because the investment has been used at 110%. Nevertheless, we placed considerable emphasis on other government departments, while we let industry and academia fall by the wayside a bit when it came to access or reuse of the data. We have been somewhat limited by the data distribution policy."

Unexpected Outcomes

Information gathered from internal and external key informants indicates that RCM data accessibility has proven to be more complicated than expected, because of a large number of acquisition requests and the sensitivity of some images captured. In December 2019, the Department of National Defence (DND) noticed that RCM was taking photos of sensitive areas. Although federal departments quickly gained access to RCM data, external users, such as academia and Canada's partner space agencies, gained access more slowly due to DND and Global Affairs Canada requirements to develop an RCM data access policy, for national security purposes.

Similarly, some external respondents also noted that DND and the Canadian Armed Forces' (CAF) significant need for RCM data, in order to ensure the safety and protection of Canada and Canadians, had the unexpected outcome of creating an imbalance and limited RCM's capacity to meet all user needs, including users within the federal government. This concern is heightened by the fact that the CSA's mandate is first and foremost linked to the civilian use of space (see "efficiency" section).

Finally, internal interviews also indicate that RCM has strengthened Canada's global reputation in the field of SAR data, which had the effect of maintaining and nurturing important international collaborations to increase the SUP opportunities of collaboration in the context of future satellite missions, but also to have a strong presence of Canada on the international stage.

Alignment of Activities, Roles and Responsibilities

Resource Utilization

Reference materials

• GENEST, Bernard-André, NGUYEN, Tho-Hau, et al. (2015). Principes et techniques de la gestion de projets, 5th edition, Les Éditions Sigma Delta, Québec, Canada, 555 pages.
• PROJECT MANAGEMENT INSTITUTE (2017). A Guide to the Project Management Body of Knowledge (PMBOK® Guide), 6th edition, Project Management Institute, Pennsylvania, United States. 756 pages.

Legislation

• Canadian Space Agency Act, S.C. 1990, c. 13, [Online], [https://laws-lois.justice.gc.ca/eng/acts/c-23.2/page-1.html].
• Constitution Act, 1867, 30 & 31 Victoria, c. 3. [Online], [https://canlii.ca/t/ldsw].

CSA internal documentation

• Canadian Space Agency (CSA, 2022b). Investment Governance and Monitoring Framework (IGMF), Revision E, March 2022.
• Canadian Space Agency (CSA, 2020). Space Utilization: 2020 Organizational structure.
• Canadian Space Agency (CSA, 2019a). Performance Information Profile of the Space Utilization Program, long version.
• Canadian Space Agency (CSA, 2017a). Performance Information Profile of the Space Utilization Program, 1st version, official (short) version.
• Canadian Space Agency (CSA, 2017b). Policy on Gender-Based Analysis Plus.

Public documents

• Canadian Space Agency (CSA, 2023a). Grants and Contributions, [Online], [https://www.asc-csa.gc.ca/eng/funding-programs/grants-and-contributions-snapshot.asp].
• Canadian Space Agency (CSA, 2023b). Earth observation satellites, [Online], [https://www.asc-csa.gc.ca/eng/satellites/earth-observation/].
• Canadian Space Agency (CSA, 2023c). RCM – Thermal vacuum testing, [Online], [https://www.asc-csa.gc.ca/eng/multimedia/search/image/7793].
• Canadian Space Agency (CSA, 2023d). Science satellites, [Online], [https://www.asc-csa.gc.ca/eng/satellites/science/].
• Canadian Space Agency (CSA, 2022a). Evaluation of the Class Grant and Contribution Program to Support Research, Awareness and Learning in Space Science and Technology, [Online], [https://www.asc-csa.gc.ca/eng/publications/er-2021-0201.asp].
• Canadian Space Agency (CSA, 2021). Evaluation of the implementation of Gender-Based Analysis Plus at the Canadian Space Agency, [Online], [https://www.asc-csa.gc.ca/eng/publications/er-1920-0201.asp].
• Canadian Space Agency (CSA, 2020a). Audit of the Class Grant and Contribution Program to Support Research, Awareness and Learning in Space Science and Technology, [Online], [https://www.asc-csa.gc.ca/eng/publications/ar-1819-0104.asp#1.1].
• Canadian Space Agency (CSA, 2020b). Book 2 – Corporate Reporting and Governance, [Online], [https://www.asc-csa.gc.ca/eng/transparency/briefing-materials/2020-book-2-corporate-reporting-and-governance.asp].
• Canadian Space Agency (CSA, 2020c). Engagement paper: Space-Based Earth Observation, [Online], [https://www.asc-csa.gc.ca/eng/publications/sbeo/2020-engagement-paper-space-based-earth-observation.asp].
• Canadian Space Agency (CSA, 2019b). Exploration, Imagination, Innovation: A New Space Strategy for Canada, [Online], [https://www.asc-csa.gc.ca/eng/publications/space-strategy-for-canada/].
• Canadian Space Agency (CSA, 2019c). What do you know about the RADARSAT Constellation Mission?, [Online], [https://www.asc-csa.gc.ca/eng/quiz/radarsat-constellation-mission/].
• Canadian Space Agency (CSA, 2018). Alouette I and II, [Online], [https://www.asc-csa.gc.ca/eng/satellites/alouette.asp].
• Canadian Space Agency (CSA, 2017c). Evaluation of the Earth Observation Business Line of the Canadian Space Agency, [Online], [https://www.asc-csa.gc.ca/eng/publications/er-1516-0202.asp].
• Government of Canada (GC, 2021). Budget 2021: A Recovery Plan for Jobs, Growth, and Resilience, [Online], [https://www.budget.canada.ca/2021/home-accueil-en.html]. Innovation, Science and Economic Development Canada. (TBS, 2017).
• Government of Canada (GC, 2014). Canada's Space Policy Framework, [Online], [https://www.asc-csa.gc.ca/eng/publications/space-policy/].
• Office of the Auditor General of Canada (AGC, 2022). Report 6 - Arctic Waters Surveillance, Reports 5 to 8 of the Auditor General of Canada to the Parliament of Canada, [Online], [https://www.oag-bvg.gc.ca/internet/English/parl_oag_202211_06_e_44152.html].
• Treasury Board of Canada Secretariat (TBS, 2016). Directive on Results [Online] [https://www.tbs-sct.canada.ca/pol/doc-eng.aspx?id=31306].
• Treasury Board of Canada Secretariat (TBS, 2010). Supporting Effective Evaluations: A Guide to Developing Performance Measurement Strategies, [Online], [https://www.canada.ca/en/treasury-board-secretariat/services/audit-evaluation/guide-developing-performance-measurement-strategies.html].