Evaluation of the Canadian Space Agency Space Expertise and Proficiency Sub-Program

2.1.1 The development of HQP

The report explores a number of dimensions related to the development of HQP and, consequently, it is helpful to clarify how HQP itself is to be understood in the specific context of this evaluation.

While frequently used, particularly in science, technology and innovation, the notion of HQP is not limited to a single generally accepted definition. It is context-driven, and typically serves to identify a sub-group of individuals who must possess more advanced qualifications to undertake certain tasks. For instance, Statistics Canada defines HQP "as individuals with university degrees at the bachelors' level and above" (Statistics Canada, 2008), while the CSA's definition of HQP focusses on engineers, scientists, and technicians (Canadian Space Agency, 2016f, p. 32). Not surprisingly, the workforce within the space sector includes a large proportion of individuals who fall within these broad definitions of HQP.

A more targeted definition of HQP focusses more specifically on post-secondary students and young researchers. At that stage, these individuals are essentially transitioning from having acquired advanced knowledge and competencies, toward positions that allow them to apply this knowledge as part of a time-bound project or by joining the space sector in the private sector, in government, or in academia. As an illustration, the 2015 Announcement of Opportunities for Flights and Fieldwork for the Advancement of Science and Technology (FAST) released by the CSA specifically defined HQP as including undergraduate students, graduate students (master's and PhD levels), and post-doctoral fellows (Canadian Space Agency, 2015b).

In the context of this evaluation, and considering the range of issues addressed, this more targeted definition of HQP is particularly helpful and, unless otherwise stated, it is the one that should be applied.

A final consideration in defining HQP relates to the field of work or study in which these individuals are involved or are in the process of getting involved. Just as it is the case with HQP, there is not a single and universally accepted definition of the space sector (OECD, 2012, p. 8). The CSA has opted for the following definition: "The Canadian space sector is defined as organizations (private, public and academic) whose activities include the development and use of space assets and/or space data" (Canadian Space Agency, 2016f, p. 9). This broad definition is meant to include both the upstream segment (research, engineering, testing and consulting, and manufacturing) and the downstream segment (satellite operations, products and applications, and services) of the space sector (Canadian Space Agency, 2016f, p. 10).

As a result, the pool of space-related HQP includes both individuals whose qualifications directly relate to space science (e.g., astrophysics, planetary science, space-related life science), as well as individuals whose qualifications are not strictly limited to space, but find application in that field (e.g., computer engineering, electrical engineering, software developers, technicians).

2.1.2 Employment in the Canadian space sector

Considering how fluid the definitions of HQP, of the space sector, and of the space economy are, it will come as no surprise that some limitations apply when it comes to measuring the size of the space workforce. Moreover, as there are no specific North American Industry Classification System (NAICS) or National Occupational Classification (NOC) codes that may be used to isolate space-specific employment, we must rely on surveys conducted by the CSA or other organizations to provide an overall estimate of that workforce (Hickling Arthurs Low, 2015, p. 5).

On that basis, and considered in its broader sense (upstream and downstream segments, HQP and non-HQP), the Canadian space workforce is estimated to include approximately 11,000 FTEs:

• Over 10,000 FTEs are employed in private-sector enterprises or in universities and research centres involved in the space economy (Canadian Space Agency, 2016f, p. 30).
• Roughly 1,000 FTEs are involved in space-related employment with the CSA, the Canadian Ice Service, the Canadian Meteorological Service, and the Centre for Remote Sensing (Hickling Arthurs Low, 2015, p. 44). On its own, the CSA engages close to 620 FTEs (including both HQP and non-HQP) (Canadian Space Agency, 2016a, p. 6).

The space workforce includes a variety of employment categories. Among them, engineers, scientists, and technicians^{Footnote 1} are considered as HQP as per the CSA's definition (Canadian Space Agency, 2016f, p. 32). As illustrated in Figure 1, these categories represented 42% (or 4,226 FTEs) of the space workforce (excluding government employees) in 2014.

In academia, there are approximately 50 research institutions and universities that are involved in space-related studies in Canada (Euroconsult, 2015, p. 57). As of 2014, they were employing roughly 1,930 FTEs, of which 71% were within the HQP category when using the CSA's definition, which includes engineers, scientists, and technicians (Canadian Space Agency, 2016f, p. 13).

2.2.1 Overview

The Sub-Program's fundamental purpose is to contribute to building and sustaining Canada's space capacity. More specifically, it "supports research in private or public organizations and sustains the development of highly qualified personnel (HQP) in science and engineering" (Canadian Space Agency, 2016e, p. 11).

Scientists and engineers who benefit from the support provided by the Sub-Program are expected to undertake relevant developmental activities related to SST. This is achieved mainly through direct support to research activities, along with access to infrastructure devoted to space-related research, including flight opportunities through fast execution and small-size missions.

The Sub-Program also pursues a broader goal of raising awareness and encouraging learning among Canadian youth and students in areas related to the space sector. While still active to some degree (e.g. by providing funding support to graduate students to help them participate in space conferences or similar events), it should be noted that most of these activities have been suspended during the period covered by this evaluation, following the CSA's 2012 program review.

2.2.2 Components

Five components are included under the Sub-Program and are described in the following sub-sections.

2.2.3 Resources and management

2.2.3.3 Roles and responsibilities

The CSA includes three programmatic branches, namely, Space Utilization (SU), Space Exploration (SE), and SST. The Director General of the SST branch is responsible for the overall management of all components of the Sub-Program. He is supported by the director of the Engineering Development Directorate and the director of the Technology Development Management Directorate, who are responsible for various components, as indicated in Figure 2.

Figure 2: Sub-Program Management Structure

To facilitate the ongoing management of the Sub-Program, the CSA has established a number of tools, including an updated Project Management Policy, which provides guidance related to the planning, approval, and development of new projects, as well as a policy on investment planning related to investments in assets and services (Canadian Space Agency, 2016e).

As previously noted, the CSA also uses its Class G&C Program to support the selection and management of projects for which grants or contributions are allocated.

2.2.4 Program logic

The logic of the Sub-Program is described in its Performance Measurement Strategy (Canadian Space Agency, 2016e).

Figure 3: Logic Model – Space Expertise & Proficiency (1.3.1)

3.2.1.1 Document and data review

This review was initiated by an analysis of documents and information pertaining to each component of the Sub-Program. This included relevant CSA policies and strategies, planning documents, program descriptions, and other departmental documents (including performance reports, presentations, or background documents).

The second step consisted of reviewing publicly available documents and publications covering more broadly issues such as space workforce and employment, participation in and promotion of STEM, particularly among youth, HQP development, and gender representation among HQP and the space workforce. This review covered the Canadian experience, as well as the experience of other nations involved in space activities.

The third step consisted of a review of relevant administrative data related to each Sub-Program component, as applicable. In particular, the information through the CSA UNITAS database provided detailed insights regarding the range of projects funded through the Sub-Program, including an assessment of project results. The data review also included, to the extent possible, internal activities such as those associated with the engineering development support (ED1), STRATOS, or the Junior Engineers Training Program.

Finally, the document and data review was used to inform each of the case studies.

3.2.1.2 Key informant interviews

Key informant interviews contributed to the in-depth understanding of the Sub-Program, including results achieved and challenges faced by each of its components. These interviews also corroborated, explained, or further elaborated on findings from other data sources and provided important input into whether outcomes have or have not been achieved, and why they have or have not been achieved.

A total of 52 interviews of approximately one hour each were conducted, which involved 61 individuals (some were group interviews) from seven different stakeholder groups. The distribution of these interviews is included in Table 8.

3.2.1.3 Case studies

A total of six case studies were completed as part of this evaluation. The list of these case studies includes both activities contained within the Sub-Program as well as other activities that fall beyond the Sub-Program, but whose relevance and expected results directly relate to the goals that the Sub-Program pursues. More specifically, the list of case studies includes:

• Mars Sample Return Simulations;
• Tomatosphere;
• the Canadian Satellite Design Challenge;
• the Junior Engineers Training Program;
• the "End-to-End Airborne Quantum Key Distribution (QKD) Demonstration" project funded under SAD; and
• the "Satellite prototype development of the Aerosol Limb Imager (ALI) instrument" project supported through STRATOS and SAD.

Each case study included up to four interviews with stakeholders associated with the project, program, or activity, including external stakeholders (recipients, participants, industry representatives, etc.) and internal CSA stakeholders. In total, 16 interviews were conducted involving 29 individuals.

Case studies also involved a review of project, program, or activity-level documents and data maintained by the CSA, as well as publicly available information (e.g., websites, online documents). Short case study reports were prepared using a standard case study report template.

4.1.1 Addressing a demonstrable need

Developing HQP and promoting STEM are fairly broad goals and, as such, it is helpful to further clarify how the need for such activities has been documented and how it is perceived by key stakeholders. The following sub-sections summarize applicable evaluation findings before turning to each component of the Sub-Program and to other contributors that must be considered.

4.1.2 Alignment with CSA and government priorities

Federal priorities

At the time of this evaluation, the goals of developing HQP, advancing space science knowledge, and promoting STEM among Canadian youth were all directly aligned with the priorities of the federal government. As noted during interviews, this constitutes a significant shift when considering the entire period covered by the evaluation. The cost-reduction measures implemented as part of the 2012 program review greatly reduced the ability of the CSA to engage in achieving all three outcomes, but this policy landscape has clearly evolved.

In November 2015, in the mandate letter he provided to the Minister of Science, the Prime Minister set the tone: "We are a government that believes in science – and a government that believes that good scientific knowledge should inform decision-making" (Office of the Prime Minister, 2015b). The letter emphasized that one of the federal objectives when it comes to science is finding "an appropriate balance between fundamental research to support new discoveries and the commercialisation of ideas."

The recognition that innovation is a fundamental pillar of economic growth was also echoed in the mandate letter provided to the Minister of Innovation, Science and Economic Development, whose portfolio includes the CSA. Among other things, it tasks the Minister to work with a range of stakeholders, including post-secondary institutions, "to improve the quality and impact of our programs that support innovation, scientific research and entrepreneurship" (Office of the Prime Minister, 2015a).

Shortly after, in June 2016, the federal government released its Innovation and Skills Plan, which rests on six pillars, including the development of a highly skilled workforce (Entrepreneurial and Creative Society), the enhancement of Canadian science capabilities (Global Science Excellence), and the establishment of clusters involving businesses, research institutions, and government (World-Leading Clusters and Partnerships) (Government of Canada, 2016, pp. 4–5). In support of this, the federal government also unveiled Canada's new Global Skills Strategy, which aims to facilitate "faster access to top global talent for companies doing business in Canada that are committing to bring new skills to Canada and create more Canadian jobs" (Innovation, Science and Economic Development Canada, 2017c).

In the fall of 2016, as part of the Canadian Aerospace Summit, the Minister of Innovation, Science and Economic Development emphasized the significant contribution that the aerospace and space sectors are making to innovation, and confirmed that the Space Advisory Board would be revitalized and asked to support the development of long-term priorities for the space sector in Canada, including a new space strategy. Among other things, the new space strategy is expected to support "talent, research and entrepreneurship within the industry" (Innovation, Science and Economic Development Canada, 2016).

More recently, in February 2017, the federal government launched its Choose Science initiative that supports the promotion of STEM among Canadian youth, with a particular emphasis on young women (Innovation, Science and Economic Development Canada, 2017a). The initiative focusses on social media outreach, along with provision of education material for parents, teachers, and mentors (Innovation, Science and Economic Development Canada, 2017b).

These various commitments were further advanced when the federal government tabled its budget in March 2017. Recognizing that "big challenges demand new and innovative solutions," the budget unveiled measures to strengthen science in the federal government. In addition to establishing a Chief Science Advisor, the federal government will be developing a new federal science infrastructure strategy, which will include a review of federal investments "in federal science infrastructure, including federal laboratories and testing facilities" (Government of Canada, 2017, p. 88).

The federal budget also included new investments related to space that will contribute to advancing SST and developing HQP:

Canada has a long and proud history as a space-faring nation. As our international partners prepare to chart new missions, Budget 2017 proposes investments that will underscore Canada's commitment to innovation and leadership in space. Budget 2017 proposes to provide $80.9 million on a cash basis over five years, starting in 2017–18, for new projects through the Canadian Space Agency that will demonstrate and utilize Canadian innovations in space, including in the field of quantum technology as well as for Mars surface observation. The latter project will enable Canada to join the National Aeronautics and Space Administration's (NASA's) next Mars Orbiter Mission. (Government of Canada, 2017, p. 90)

On the promotion of STEM among Canadian youth, the federal budget unveiled two new measures: an increase in the budget allocated to NSERC's PromoScience Program to expand the reach of activities delivered by its beneficiary organizations, including activities targeting underrepresented groups in STEM. The budget also included a broadening of the Prime Minister's Awards for Teaching Excellence to include 17 new STEM-themed awards (Government of Canada, 2017, p. 74).

CSA priorities

As it relates more specifically to CSA's priorities, the period covered by this evaluation began with the review of the aerospace and space sectors announced in the 2011 federal budget (Government of Canada, 2011, p. 86). What came to be known as the Emerson report included a number of recommendations, including one on the need for the CSA to establish a clearer vision of Canada's goal as it relates to space: "in a sector whose undertakings are, by definition, long-term, expensive, and complex, it is especially important to have concrete goals, predictable funding, and orderly implementation" (Aerospace Review, 2012, p. 26).

Partly in response to this review, in February 2014 the CSA unveiled its Space Policy Framework. The vision articulated in this framework rests on five core principles, including the need to enhance key capabilities (e.g., telecommunications, remote sensing, robotics), and to inspire Canadians to pursue careers in space (Canadian Space Agency, 2014b, p. 10). The framework includes four avenues of strategic actions, including a commitment to work with industry and the space research community to support further opportunities in research and development, and innovation (Canadian Space Agency, 2014b, p. 11).

These findings confirm that the Sub-Program is in a position to support federal priorities related to innovation, including the space priorities that the CSA has identified.

4.1.3 Distribution of roles and responsibilities

Evaluation findings described in the preceding sub-sections provide ample evidence as to the range of stakeholders that are actively engaged in developing HQP, advancing space knowledge, and promoting STEM among Canadian youth. Reaching the most efficient equilibrium in the distribution of roles and responsibilities among all these organizations is bound to be an ongoing process that reflects evolving mandates, activities, and opportunities.

Exploring more specifically the role that the CSA is expected to play, it is helpful to recall the fundamental mandate that Parliament has assigned to the CSA, as reflected in the Canadian Space Agency Act: "The objects of the Agency are to promote the peaceful use and development of space, to advance the knowledge of space through science and to ensure that space science and technology provide social and economic benefits for Canadians" (Statutes of Canada, 1990, sec. 4).

As already noted in sub-section 4.1.1.1, the expected results of the Sub-Program are directly linked to the notions of advancing "the knowledge of space through science" and ensuring that space science provides "social and economic benefits to Canadians." In turn, given the complexity of space science, it logically requires measures to build, maintain, and enhance the pool of HQP involved in space, from the moment a child engages with his or her broader environment, up to his or her enrolment in highly specialized graduate studies in one of the many fields of study associated with space.

On the specific issue of promoting STEM among Canadian youth, there is already a fairly broad and active network of organizations that are engaging youth in STEM and playing complementary roles. Ministries of education, school boards, and schools expose Canadian youth to the nature and applications of STEM fields. Other organizations, including a number of non-profit organizations that have a particular focus on space, are called upon to expand these STEM opportunities. As for the CSA, it stands as the primary centre of space expertise in Canada. In this context, the ongoing contribution of the CSA must be targeted and complementary, in addition to falling within its legislated mandate. In doing so, the CSA can capitalize on the fact that its assets have no equivalent, and that the organization holds, or has access to, a level of space expertise that is unmatched in Canada. Building on the range of communication activities that it already undertakes, the CSA can collaborate to more systematically promote STEM through space, and space through STEM. The tailored and targeted approach used by other agencies in pursuing this goal (as noted in sub-section 4.1.1.2 of this report) is particularly appealing in the Canadian context.

4.2.1 Design and delivery structure

SAD component

Evaluation findings on the delivery of SAD activities essentially focus on the AOs issued under FAST, as key informants were not in a position to offer insights on previous AOs that were issued under other components of SAD in the early part of the period covered by the evaluation.

Overall, funding recipients perceived the management of FAST AOs positively. They systematically emphasized the professionalism of CSA representatives and their willingness to offer assistance as needed. Interview findings also indicate that funding recipients were generally familiar with the range of activities covered by FAST AOs and were in a position to prepare their submission accordingly.

Interview findings also pointed to some challenges related to the FAST AO process. On several occasions, internal and external key informants encouraged the CSA to predominantly use AOs to provide funding support (as opposed to unsolicited proposals), and to issue its FAST AOs on a more predictable basis. This finding is directly in line with the findings of the evaluation of the Class G&C Program discussed in sub-section 2.3. The other main challenge raised was around the burden associated with the proposal and reporting requirements. Some key informants were of the opinion that the amount of information required in proposals was too onerous, particularly when considering the level of funding provided. There was also a general perception among funding recipients interviewed that the reporting requirements were particularly burdensome, and they were unsure as to how the CSA was using this information.

Somewhat linked to the perceived unpredictability of AOs, interview findings indicate that coordinating activities between FAST and STRATOS, or funding between FAST and other granting organizations, remains a challenge for many recipients. A number of key informants suggested the CSA work more closely with other granting organizations such as NSERC or the Canadian Foundation for Innovation, and other federal departments as applicable, to enhance the complementarity of their respective funding programs and, possibly, their funding calls and reporting requirements.

On the timeliness of the allocation process, interview findings indicate that one area of concern is the delay experienced between the time the results of an AO are announced and the establishment of agreements, and the associated flow of financial resources. Evaluation findings indicate that several weeks may be required to complete this final step, which restricts the ability to promptly undertake the funded projects.

As for the CSA's internal capacity to deliver the SAD component, evaluation findings indicate that the SAD group is already struggling to monitor all current projects, offer the required support to funding recipients, manage unsolicited proposals, and appropriately plan for future AOs. In this context, it would not appear reasonable to expect that the CSA could provide, if such was its desire, more predictable AOs as recommended by many of the participating stakeholders.

STRATOS

The design and delivery structure of STRATOS was described positively throughout the interviews conducted as part of this evaluation. STRATOS was regularly described as a being an exemplary program when it comes to supporting HQP development and SST knowledge growth.

Among the factors contributing to this success is the partnership between the CSA and the CNES. Representatives from both agencies highly value this working relationship and the complementarity of experience and knowledge that derives from it. In fact, representatives of the CNES do not only wish to see the partnership continue, but they would welcome opportunities to expand it. An example provided was the possibility of having joint projects between France and Canada, as opposed to only sharing available space on stratospheric balloons.

Both professors and students who had participated in a STRATOS campaign, and who were consulted as part of this evaluation, emphasized the professionalism and commitment of CSA representatives in providing the required support to ensure that their payload could be placed on a stratospheric balloon and that their mission could be successfully carried out. Even when unexpected difficulties arose (such as delays resulting from adverse weather conditions), the STRATOS team worked closely with participating teams to manage the impact of these delays on their experiments.

When asked if STRATOS could better meet their needs, some key informants noted that the current parameters of the program in terms of the weight of eligible payloads and the duration of flights limit the range of experiments that can be conducted. As a result, broadening the capacity of these stratospheric balloons would, in turn, open up new opportunities that, at the time of the evaluation, could only be sought in other jurisdictions, such as NASA.

Internal activities

Both the ED1 and the Junior Engineers Training Program are managed internally. As such, the evaluation gathered some findings related to their delivery, but the scope of this information remains limited.

As previously noted, the delivery of ED1 activities varies in accordance with internal requirements and external opportunities. While some tracking of time has been done to better understand how ED1 activities were evolving, it only provided a partial picture of how these activities were being planned and managed. Evaluation findings indicate that this working framework is in the process of changing to become more systematically structured. As such, the Engineering Development Directorate is implementing new processes that will ensure that all activities undertaken as part of the ED1 component of the Sub-Program are accounted for and linked to an established priority of the CSA.

As for the Junior Engineers Training Program, only two rounds of recruitment had been completed at the time of the evaluation, the first one involving eight participants and the second one involving four participants. In this context, the program is still evolving. A key difference between the two rounds of recruiting is the fact that the first one advertised for indeterminate positions, while the second one advertised for determinate positions.^{Footnote 10} As noted during interviews, this could explain, at least in part, the fact that over 1,000 applications were received during the first round, while 400 applications were received during the second round. Participants in the program who were consulted as part of this evaluation did emphasize the quality of the recruitment process and of the ongoing support provided throughout their various assignments. As noted in the case study report, one suggestion for improving the delivery of the program would be to complete the selection process in a timelier manner, to allow candidates to pursue other avenues in case they are not selected.

4.2.2 Performance measurement strategy

During the period covered by the evaluation, there was no performance measurement strategy in place in relation to the Sub-Program. However, the CSA developed a strategy that became operational as of April 1st, 2016 (Canadian Space Agency, 2016e). This tool includes all key components required by such a strategy, including a list of indicators that will allow for the activities and some of the results achieved under the Sub-Program to be documented in a systematic and ongoing manner.

As noted in the Performance Measurement Strategy, this evaluation of the Sub-Program includes a series of findings and insights that could provide assistance in refining the strategy, clarifying what can be expected to be measured as part of the strategy, what is expected to be covered by a formal evaluation process, and how the two can act in a complementary manner.

Also noted in the Performance Measurement Strategy is the fact that resources required to fully implement the strategy have yet to be identified. Along with addressing this issue, it would be beneficial for the CSA to also confirm how performance data will be collected, analyzed, stored, and shared, both as part of the ongoing management of the Sub-Program and also during the next evaluation of the Sub-Program. For instance, creating annual performance reports that centralize all the performance information in one document is an approach used by other federal departments that could be applied in relation to the Sub-Program.

4.2.3 Gender-Based Analysis Plus

Just as this evaluation got underway, in July 2016, the federal government released its new Policy on Results, along with its associated Directive on Results, which replaced the 2009 Policy on Evaluation. This new framework clarifies expectations related to gender-based analysis. First, it confirms that in establishing their performance measurement strategy, program managers must include, where relevant, a gender-based analysis (Treasury Board of Canada Secretariat, 2016, para. A.2.5.10). It also identifies, as a mandatory procedure, that all evaluations be planned to take into account, where relevant, gender-based analysis (Treasury Board of Canada Secretariat, 2016, para. C.2.2.1.6).

These new requirements reflect a desire on the part of the federal government to achieve greater results on the issue of gender consideration. Back in 1995, the federal government made a formal commitment to integrate gender-based analysis in the development of its policies, programs, and legislation (Status of Women Canada, 2016b). However, significant shortcomings in implementing this commitment continued to be identified, including those documented in the 2015 report from the Auditor General that focussed specifically on this issue (Office of the Auditor General of Canada, 2015). In response to this audit, the federal government released its Action Plan on Gender-based Analysis (2016–20), which includes, among other items, a commitment by the Treasury Board of Canada Secretariat to provide greater assistance to departmental evaluation functions to take into consideration Gender-Based Analysis Plus (GBA Plus) when evaluating federal programs (Status of Women Canada, 2016a).

During the period covered by this evaluation, the CSA did not have a formal gender-based analysis framework in place. As such, the goal of the evaluation is not to assess the extent to which proper gender-based analysis had been performed in relation to the Sub-Program. Instead, the objective is to assess the extent to which the Sub-Program will be in a position going forward to undertake such an analysis, as required.

Two main evaluation findings emerged on the issue of gender considerations. First, as documented in this report, the engagement of young girls in STEM-related studies, and of women in the space sector more generally, remains limited, which has triggered a number of initiatives in Canada (as well as in other countries) that specifically aim to address these shortcomings. This is the broader policy context in which the Sub-Program will continue to evolve, and which confirms that gender considerations do apply to the Sub-Program, particularly to its SAD component, and the outreach activities targeting Canadian youth.

Secondly, during the time this evaluation was taking place, the CSA was developing its own policy and procedures governing GBA Plus, which were approved in March 2017. The set of tools developed includes an implementation guide, along with a questionnaire that will help program managers to be in a position to undertake this analysis and to ensure that proper gender considerations are applied in the design and management of their programs and activities.

The ongoing performance measurement activities, along with the next formal evaluation of the Sub-Program, will provide opportunities to assess the extent to which the Sub-Program has succeeded in implementing this requirement.

4.3.1 SAD component

Funded projects

In terms of solicited projects, during the period covered by the evaluation, the CSA issued four AOs related to the SAD component of the Sub-Program. As indicated in Table 9, a total of 52 projects were selected through these processes, for a total of $19.1 million in funding support.

Source: CSA, Unitas data

In addition, a total of 11 unsolicited projects were approved between 2011 and 2015, representing a total investment of $3,044,317.

In both cases (solicited and unsolicited projects), some projects approved prior to 2011 were still active during the period covered by the evaluation.

Project results and impacts

Evaluation findings provided a number of insights on the results achieved by the projects funded under the SAD component of the Sub-Program. Information and data from reports provided by funding recipients from the interviews conducted as part of this evaluation and from the case study on SAD are summarized in the following paragraphs.

While it must be appropriately contextualized and used with some caution, the information presented in Figure 6 provides an illustration of the reach that these funded projects had in 2015–16. The need to be somewhat careful in using this data comes from the fact that only one year is presented in this figure (for illustrative purposes), that each reported individual may have had various levels of involvement in these projects, and that it is possible for one individual to have been involved in more than one project. With this in mind, the data indicate that a fair balance is achieved between the involvement of university professors and students. Also, the data show that undergraduate and graduate students are equally engaged in these funded projects. Participants were associated with universities (in Canada and abroad), research centres, departments and agencies, and private-sector industries.

Of particular importance for the purpose of this evaluation is the perceived impact that these projects have had on advancing SST knowledge and allowing space partners to collaborate. As illustrated in Figure 7, the vast majority of projects reported a direct contribution to the establishment or maintenance of research partnerships, and to the ability of funded recipients to carry out outreach activities that enhanced general scientific awareness. The majority of funded projects have also led to technological or scientific breakthroughs and the development of new ideas that might be integrated in future space missions. Other significant contributions include the ability to leverage funds and to engage new players in space research. More than half of the projects funded contributed to the development of applications and algorithms and have used satellite data. In a minority of cases, the projects directly contributed to advancing the TRL of the associated technology. Finally, Figure 7 shows that only a small portion of funded projects lead to commercial successes.

These trends were also reflected in interview findings gathered as part of this evaluation. The establishment of partnerships is seen as an important contribution of the SAD projects. As one key informant put it, "FAST and STRATOS have both been very helpful for developing partnerships. FAST has really been useful for me to partner with researchers in Canada as well as internationally." And these partnerships extend beyond Canada. As another key informant noted, "The FAST grant is allowing us to collaborate with a team at NASA and some US universities. We are taking advantage of their heavily operational study and getting science out of it."

It should also be noted that over 90% of funding recipients indicated in their final reports that they expect the partnerships they established as a result of the CSA funding to continue when their project has been completed, and that other follow-up projects will be generated as a result of this funding.

The case study report on SAD illustrates how FAST funding can lead to technological advancements. As previously noted, in this particular project relating to airborne quantum key distribution, the research team succeeded in increasing the TRL of key-related technologies. It also established partnerships with research institutions and private-sector industries, and leveraged in-kind and financial supports from five organizations. Finally, the project provides a good illustration of the range of students that these projects may engage. In this case, four post-doctoral fellows, two PhD students, two master's students, and four undergraduate students participated in the project.

On the specific impact that these projects have on students, evaluation findings indicate that both soft and hard skills are gained through SAD-funded projects. As illustrated in Figure 8, communication skills, team work, increased self-confidence, leadership abilities, and learning to adapt to change are among the skills that are reported more systematically by funding recipients. Moreover, the majority of projects allow participating students to enhance their knowledge of science, including practical experience with space science and technology. Only a minority of projects have reported a significant contribution in building system engineering experience. Overall, funding recipients were of the opinion that the SAD projects in which students were engaged will contribute in preparing them for careers in the space sector as HQP.

Administrative data indicate that results similar to those presented in Figure 8 are also reported for projects engaging students who are financially supported by the Space Exploration and Space Utilization branches, such as the initiatives described in sub-section 4.1.1.4.

One prime indicator of the impact of funded projects on advancing space science is the level of activities in publications and other communication activities supporting a sharing of knowledge. As illustrated in Figure 9, peer-reviewed publications are the prime targets of funded recipients. Recipients have also engaged in non-peered reviewed publications and other technical publications.

In addition to publishing, funding recipients also carry out various presentations, including workshops, general public presentations, and media activities. During the period covered by the evaluation, more than 2,600 of such presentations were reported by funding recipients.

The SAD case study provides a good illustration of the range of communication activities in which project recipients engage. As a result of the technological achievements resulting from their funded projects, participants were involved in a total of eight publications, including three in peer-reviewed journals, along with 17 presentations. Along the same line, the team engaged in outreach activities such as a laboratory tour open to the public at the Institute of Quantum Computing at Waterloo University, where they were given an opportunity to present their projects.

Evaluation findings also provided an opportunity to better understand the range of challenges that funding recipients face when they implement their projects. In doing so, and as noted during interviews, it must first be emphasized that space science and technology is a particularly complex field, where demonstration activities are bound to experience some level of challenge and technical difficulty. This is a logical condition of undertaking activities in a pre-space or space environment.

Keeping this in mind, Figure 10 confirms that some technical or scientific issues have been faced by half of the project recipients. Slightly more than one-third of them also experienced equipment or facility problems.

What may come as more of an unexpected issue is the fact that half of the funding recipients also experienced staffing issues, and a quarter of them reported funding difficulties. While the available data do not provide details on the range of staffing problems encountered, interviews conducted as part of this evaluation provide relevant insights. Since there is typically a gap of at least two years or more between AOs (depending on the space area one focusses on), interview findings indicate that it may be difficult to align these projects with the cycle of postgraduate studies. This means that students may not remain involved during the entire project period.

The same largely applies to funding. As projects are normally funded through multiple sources, interview findings indicate that it may prove difficult to align the cycle funding with the project requirements.

Overall, funding recipients were particularly appreciative of the support provided by the SAD component of the Sub-Program. As one key informant summed it up, "We couldn't have done any of our projects without CSA funding." The support provided is seen as providing a unique opportunity for space scientists: "The money to hire the people, the money to develop the instrumentation, and the money to execute the project are all within the FAST project. There's nothing else like it in the country."

4.3.2 STRATOS component

Each STRATOS campaign involves between 4 to 10 flights, each carrying a number of payloads, including, but not limited to, Canadian payloads.

Source: CSA, 2013b and administrative data.

As indicated in Table 10, during the period covered by the evaluation, three STRATOS campaigns were successfully completed, involving 15 Canadian payloads. Of note is the fact that the 2013 campaign was a qualification one, the first campaign to be launched from the newly established CSA Timmins Stratospheric Balloon Base. The following two campaigns were also launched from Timmins. During the post-evaluation period, in 2016 and 2017, a total of nine Canadian payloads were also expected to participate in STRATOS campaigns, including one campaign held at the Esrange Space Centre in Sweden, and one held in Alice Spring, Australia.

While STRATOS campaigns predominantly involve teams of academics and students, program data indicate that they also involve space industries.

When exploring the impact of STRATOS, it must first be emphasized that the program operates as an enabler, as other mechanisms or funding sources must be used to develop the technology that will be eventually integrated in a stratospheric balloon flight, and to analyze the data gathered as part of these flights. As previously noted, FAST has been a predominant source of such funding. As a result, the impact of STRATOS is always linked to the impact of other activities, such as FAST.

On training HQP, evaluation findings indicate that this enabling contribution of STRATOS has a significant impact on the range of skills that funding participants, including students, can gain through this hands-on experience. The case study on STRATOS provided relevant insights in that regard. In this project, involving the ALI instrument, a total of five graduate students and eight undergraduate students participated in one of the 2014 stratospheric balloon flights. They developed, designed, and, with the close collaboration and support of the CSA's STRATOS team, tested the instrument. On two occasions, CSA engineers visited the students to review and discuss their project. The student also collaborated with an industrial partner. Findings from this project have already been used in two peer-reviewed journals, and three theses. The team also had the opportunity to present its findings at the ESA Rocket and Balloon Workshop in Norway.

This added dimension that STRATOS can bring was also acknowledged during interviews conducted as part of this evaluation. As one key informant put it:

"Our project would not have happened without CSA support. Some of the work would have been carried out, but it would have been kept at a fairly low level and we probably wouldn't have achieved the flight opportunities that we had through STRATOS. We might still have developed some instrumentation but flying the technologies almost certainly wouldn't have happened without CSA support."

Evaluation findings also indicate that STRATOS activities directly support the advance of space technologies. The project examined as part of the STRATOS case study became the first suborbital demonstration of an imaging aperture-acousto tunable filter (AOTF) for atmospheric remote sensing. It also contributed to increasing the TRL of key associated technologies. The 2014 campaign also provided an opportunity for Xiphos Systems Corporation to test and qualify its Q6 processor, which has since flown on a number of missions, including one involving the International Space Station (Xiphos Systems, 2015).

4.3.3 ED1 component

As previously noted, the ED1 component of the Sub-Program provides a range of support to internal and external stakeholders, including those involved in other components of the Sub-Program, such as FAST, STRATOS, and the Junior Engineers Training Program. Sub-section 4.1.1.3 of this report has already noted the relevance of the subject-matter expertise that this group provides.

As a result of the nature of this activity, evaluation findings cannot offer an illustration of all the various contributions that ED1 has made during the period covered by the evaluation. However, one such illustration is provided by the case study report on the Canadian Satellite Design Challenge. Over the course of the period covered by the evaluation, two competition rounds were held (2012 and 2014), each lasting two years, and involving 8 to 12 interdisciplinary teams of undergraduate and graduate students. ED1 team members served as subject-matter experts and judges during these competitions, in addition to facilitating the access to the David Florida Laboratory (DFL) testing facilities.

The contribution of ED1 to HQP development is also illustrated through its support to the master's degree program offered by the Comité sectoriel de main-d'œuvre en aérospatiale du Québec (CAMAQ), which involves five universities in Québec (McGill, École de technologie supérieure Montréal, the Polytechnique, Concordia, and Sherbrooke). Every second year, members of the ED1 group deliver one of the eligible courses (Aerospace Case Studies), which covers the fundamentals in space mission and subsystem design.

4.3.4 Junior Engineers Training Program

Evaluation findings confirm the positive impact of the Junior Engineers Training Program on its participants. As noted in the discussion on the relevance of the component (see sub-section 4.1.1.3 of this report), young engineers who were provided with the opportunity to join the program considered it an exceptional opportunity.

The case study focussing on this program also provides additional insights on the results achieved. Of note is the fact that, of the eight individuals who took part in the first recruitment campaign, seven have successfully completed the program, including six who have now joined the CSA as engineers. Those recruited during the 2014–15 campaign were still completing the program at the time of the evaluation. Program participants are also provided with the opportunity to engage in outreach activities targeting Canadian youth, and to teach courses at the graduate level.

4.4.1 Resource allocations

The Sub-Program experienced substantial shifts in human and financial resources throughout the period covered by the evaluation, and a number of these changes occurred as a result of the implementation of the 2012 program review. This is important contextual information when it comes to analyzing the efficiency of the resource allocation process.

The largest portion of the Sub-Program's financial resources was allocated to the SAD component. As previously indicated in Table 1 (page 6), the actual expenditures of this Sub-Component went from $11.8 million in 2011–12 down to $3.4 million in 2015–16. In total, the expenditures for this component were $1.4 million less than initially planned by the CSA.

The second largest investment of the Sub-Program was directed to ED1 activities. In this case, and as previously indicated in Table 2 (page 7), the total budget went from $4 million in 2011–12, down to $1 million in 2015–16. In total, the expenditures for this component were $3.4 million less than initially planned.

In both cases, and as illustrated in Figure 11, the number of FTEs assigned dropped significantly during the evaluation period, as a result of restructuring (including the dissolution of the Space Science directorate) and reallocation of resources from ED1 to STRATOS.

As for the STRATOS program, close to two-thirds of the actual expenditures incurred during the period covered by the evaluation were assigned to operations and maintenance, which were essentially used to establish the Stratospheric Balloon Base in Timmins. Out of a total of $9.1 million in actual expenditures over five years, $5.8 million was allocated for this purpose.

As for the Junior Engineers Training Program, resource allocations were linked to actual recruitment campaigns and were mostly allocated to cover the salaries of program participants. In the case of outreach activities targeting Canadian youth (Space Learning, as well as Awareness and Learning), their associated expenditures essentially ceased in 2012–13.

Interview findings indicate that this environment has been particularly challenging for those involved in the delivery of the Sub-Program activities. The uncertainty and the decreasing capacity to carry out all expected activities covered by the Sub-Program have raised concerns as to the ongoing sustainability of the Sub-Program to achieve its expected results. Evaluation findings indicate that the SAD group is experiencing challenges with regards to its capacity to monitor all current projects, offer the required support to funding recipients, manage unsolicited proposals, and appropriately plan for future AOs. In this context, it would not appear reasonable to expect that the CSA provides — if such was its desire — more predictable AOs as recommended by many of the participating stakeholders. Along the same line, and as already noted, the current capacity level of the ED1 group is significantly narrower, and it remains unclear how the expertise currently provided will be sustainable in the long term.

4.4.2 Processes to enhance efficiencies

At the time of the evaluation, a number of processes were in place to enhance the efficiency of the Sub-Program, including the following components:

• Through the establishment of the Science, Technology and Expertise Development in Academia (STEDIA), which replaces the SAD, the CSA wishes to enhance its ability to coordinate and integrate all activities related to the development of HQP and the growth of SST knowledge.
• The CSA was also in the process of centralizing and upgrading all of its support for demonstration activities, ensuring that STRATOS and other demonstration avenues were being implemented in a coordinated manner. The CSA has set aside a total of $18.4 million over five years (2017–18 to 2021–22) for capacity demonstration activities.
• The CSA has also implemented processes to enhance the planning, monitoring, and implementation of activities falling within the ED1 component. While this component remains committed to operate on a matrix basis, and being sufficiently flexible to respond to the various needs of internal and external stakeholders, the goal is to ensure that these activities remain aligned with priorities of the agency.

These measures are expected to support the CSA as it engages in the various commitments contained in its 2017–18 Departmental Plan that relate to the development of space expertise and space capability demonstration (Canadian Space Agency, 2017a, p. 21).

Otherwise, evaluation findings have not identified alternative mechanisms that could be used to more effectively deliver the support provided through the Sub-Program.

SAD

The evaluation indicates that the SAD component of the Sub-Program responds to the need of Canadian research institutions to engage in hands-on space research projects. SAD activities represent a significant means by which the CSA supports SST knowledge growth, including the development of HQP in Canada. Research institutions require opportunities to engage students in SST projects in order to provide adequate training. In turn, projects advancing SST require the participation of students to support their implementation. The SAD component of the Sub-Program provides such opportunities and while other granting organizations play a critical and complementary role, what the SAD component offers has no equivalent in Canada.

During the five years covered by the evaluation (2011 to 2016), the SAD component of the Sub-Program was largely focussed on FAST funding, which responds principally to the need for end-to-end projects. Only limited activities were undertaken in other priority areas identified by research institutions such as research clusters, industrial chairs, and other studies on space science instruments or data analysis. Responding to the needs of research institutions to further undertake such research activities thus represents an opportunity for improving the SAD component of the Sub-Program.

STRATOS

The evaluation demonstrates that having access to stratospheric balloons in Canada represents a well-suited tool to support SST and HQP development. The STRATOS component is particularly well-suited for supporting end-to-end research projects, which is the focus of FAST funding. In that sense, these two types of activities are highly complementary. The relevance of the stratospheric balloon activities is also illustrated by the fact that it is a fairly flexible, accessible, and efficient demonstration option.

Broadening the range of suborbital platform and stratospheric balloon activities (in terms of payload weight and flight duration), and integrating in a more systematic fashion the support provided for other types of demonstration activities would further enhance the capacity of research institutions to engage in SST and HQP development.

ED1

The group of experts under the ED1 component provides valuable expertise to both internal and external stakeholders. In addition to providing needed matrix support for internal CSA funding initiatives, including SAD and STRATOS, ED1 activities contribute to the CSA's participation in a number of space-related international fora.

ED1 members also support HQP development by, among other things, acting as external examiners or co-supervisors of theses, and by supporting engineering capstone projects. While many benefits can be derived from having CSA engineering experts involved with students activities, it would be beneficial to clarify the rationale for doing so, and the specific niche that the CSA wishes to occupy in that regard.

Junior Engineers Training Program

During the period covered by the evaluation, a number of emerging engineers have joined the CSA through the Junior Engineers Training Program, exposing them to the full range of activities undertaken by the Agency's various directorates. As such, it represents a remarkable opportunity to integrate new HQP to the space sector. In turn, the Junior Engineers Training Program provides a valued opportunity to join the ranks of the CSA, while the program is also contributing, to some extent, to renewing CSA's internal engineering expertise.

Awareness and Learning and Space Learning

The evaluation describes the wide range of activities undertaken by the CSA, as well as in Canada and around the world to engage youth (elementary and secondary students) and post-secondary students (college and undergraduate students) in STEM. There is a strong rationale to pursue these activities in order to reverse the documented decline in STEM interest, and the lower level of STEM engagement found among young women.

As the evaluation demonstrates, the CSA possesses a unique attraction pole to engage young Canadians in STEM fields of studies to which the Sub-Program contributes. The CSA holds unique assets that it can lend to this important endeavour, through the involvement of the Sub-Program and its other directorates. At the time of the evaluation, the Communications and Public Affairs Directorate was carrying out a wide range of outreach activities targeting Canadian youth, and the Sub-Program's contribution was more focussed on supporting specific activities for university students (such as attending space-related conferences or through projects funded by the SAD component). The CSA is further structuring these activities through a STEM plan that is expected to cover activities targeting both Canadian youth and students, and through revisions to the component of its Class G&C Program relating to Awareness and Learning.

Immediate outcome 1: Partnerships among research organizations established or maintained

During the period covered by the evaluation, the Sub-Program contributed to the establishment of a wide range of valuable partnerships, including those involving universities and research centres, those that enabled collaborations between universities and space industries, and those collaborations reaching out to other space agencies or space organizations. Activities under FAST, STRATOS, or ED1 have all supported various forms of partnerships, with a predominant number relating to end-to-end projects. In a number of cases, these partnerships have benefitted from funding support provided by other granting organizations, such as NSERC.

Immediate outcome 2: Increased or maintained number of students and HQP in SST

The evaluation demonstrates that the Sub-Program has allowed students and HQP to be involved in SST projects. Sub-Program activities have facilitated the involvement of students, both undergraduate and graduate students, in a wide range of activities, with a particular focus on hands-on projects such as those under FAST and STRATOS. These projects add a new and important dimension to the education students receive through their study programs.

The Junior Engineers Training Program has also provided a unique opportunity for those selected participants to successfully engage in SST, by joining the CSA workforce.

Immediate outcome 3: Increased or maintained number of research projects performed in SST

The Sub-Program has proven to be a key contributor, along with other granting organizations in Canada, to research projects performed in SST. These projects have supported the growth of SST knowledge in a number of areas targeted by the CSA, particularly through its FAST AOs. This has, in turn, contributed directly to the fundamental and legislative object of the CSA to "advance the knowledge of space through science."

Immediate outcome 4: Increased or maintained possibilities of accessing pre-space and space

The Sub-Program's direct contribution to accessing pre-space and space environments has been focussed on stratospheric balloon launches. In collaboration with the CNES, the Sub-Program has successfully launched stratospheric balloon campaigns throughout the period covered by the evaluation, including campaign launched from the newly established CSA Timmins Stratospheric Balloon Base. These campaigns have predominantly engaged students and academics, but also the space industry. Funding provided by components such as FAST AOs also provided access to other demonstration opportunities, such as analog sites and parabolic flights.

Notwithstanding the achievement by the Sub-Program of its expected outcomes, the evaluation indicates that transitioning space-focussed students and HQP into the labour market remains a concern. While some of the students benefitting from the Sub-Program's activities will be able to pursue a space career in Canada in their field of expertise, evaluation findings indicate that others will need to apply generic skill sets in other space-related fields or apply these skills in other STEM-related fields that are not directly related to space. In this context, the implementation of the Sub-Program activities should be done in such a way as to support, when feasible, the transition of the trained HQP in various space-related fields, including space science and technology, space exploration and space utilization, and in non-space fields.

In order to address the concern raised above, the evaluation recommends the following:

The Space Science and Technology Directorate ensures that its objectives and activities are aligned not only with current but also emerging needs of Canadian universities and research institutions in advancing space science and technology knowledge, while also facilitating the work transition of the developed HQP toward sectors, including but not limited to the Canadian space sector, where their advanced technical skills and behavioural competencies (soft skills) can be best applied.

This recommendation is based on the premise that the range of skills developed through the projects funded by the Sub-Program will directly match some of the HQP requirements of the space sector, including the space industry, while acknowledging that the challenge related to transitioning toward the labour market far exceeds the strict parameters of the Sub-Program.

Finally, the evaluation provided an opportunity to document how other activities in which the CSA is engaged contribute to the development of space-related HQP and SST. The goal was not to formally evaluate these activities, but rather to explore their relationship with the Sub-Program, and their potential contribution to the achievement of the Sub-Program's expected outcomes. Activities undertaken by the CSA beyond the Sub-Program also contribute to these expected outcomes. Both the Space Exploration and Space Utilization branches engage universities and research institutions in a range of projects that, in a complementary manner, also further SST knowledge and train HQP.

Along the same lines, the evaluation provided an opportunity to better situate the work of the CSA in promoting STEM studies among Canadian youth, by documenting not only what is already done by the CSA (within and outside of the Sub-Program), but also how a wide range of other stakeholders contribute to this goal.

This analytical approach has provided a wider context that has supported a better and more integrated understanding of the Sub-Program activities and achievements.