Audit of the Management Framework for the Enabling Technology Development Program

Criterion 1

Program activities are planned appropriately.

Condition

Conclusion about the criterion

The audit showed that program activities are planned appropriately. We nevertheless identified one opportunity for improvement that involved integrating the planning of the program's R&D activities with CSA organizational planning.

Investment priorities

We found that stakeholder needs are identified on a regular basis, and the selected areas for investment reflect the identified needs and priorities. Investment plans are prepared, approved and reviewed on a regular basis.

Strategic planning roles and responsibilities

The STDP is under the authority of the CSA Space Science and Technology (SST) Directorate. Strategic planning for this directorate is carried out by the Innovation Planning and Commercialization Group (IPCG). This group uses an integrated stakeholder approach. The key STDP stakeholders are Canadian space-industry players (businesses and universities) and CSA program sectors, namely Space Exploration (SE), Space Utilization (SU) and SST. Planning begins by identifying stakeholder needs in order to determine the program's investment priorities.

The CSA's internal Technology Planning Working Group (TPWG) is made up of representatives from the three CSA program sectors and the Policy and Programs and Integrated Planning Directorates. The TPWG is the main CSA committee for space technology development. Investments are endorsed by the SST Steering Committee and approved by the Integrated Investment Review Board (IIRB).

Needs identification

Consultations are occasionally held with Canadian space-industry representatives to discuss their needs. These consultations are carried out according to different streams. One consultation stream was with the Aerospace Industries Association Canada (AIAC). A joint CSA/AIAC working group has been created. The working group has been tasked with developing and suggesting changes to the CSA STDP in order to make it more responsive to the current commercial environment in the space sector. Another stream was with the Canadian space industry and universities. Its goal was to present a draft of the new STDP framework for feedback. In November 2016, 90 of the 200 invited organizations and universities took part in webinars to endorse the joint recommendations of the CSA/AIAC working group. A third stream consisted of meeting with program sectors at the CSA and other key departments to identify challenges and discuss potential solutions. The recommendations emerging from these consultations were implemented, to the extent possible, and led to a number of changes in the STDP. For example, specific funds now target small and medium-sized enterprises. Moreover, in November 2017, Euroconsult submitted a report to the CSA entitled Technology Trend Assessment of the Canadian Space Sector. Euroconsult assessed technological trends in order to identify technologies most likely to have a major impact on the space sector over the next 5 to 10 years, and to determine their relevance for the Canadian space sector. These trends will help to inform CSA investment priorities.

The investment priorities of CSA program sectors, in line with their respective objectives, are established following their own consultations with the industry. These needs are submitted to the STDP through the internal consultation exercise, along with an estimated value of the work. Before an RFP is officially launched, the industry is informed of the CSA's intention to launch this AO, the details of the expected technological development (as well as the allocated budget). Industry is also asked to validate industry interest, the content of the work and the budget allocation. Since the RFP 16 (2017/2018), business needs must be validated by the IIRB before being submitted to the TPWG and the RFPs be posted.

Prioritization of areas for exploration and approval of investment plans

Announcement of Opportunity (Contributions)

In order to meet the priorities of Canadian space stakeholders, the CSA helps to position the sector through contributions to technology development for potential commercialization opportunities. A broad range of projects is funded through this type of financing vehicle. Specifically funded projects are based on industry needs.

The contributions awarded through AO 4 in 2017 targeted three project categories: basic space R&D for projects that are expected to produce economic benefits in the short and medium term (2 to 5 years); basic space R&D for projects from small businesses (50 or fewer employees) that are expected to produce economic benefits in the medium and long term (5 to 10 years); and, basic space R&D for feasibility studies for projects that are expected to produce economic benefits in the medium and long term (5 to 10 years).

A selection process is used to review the proposals received in response to an advertisement. Proposals that meet the mandatory criteria listed in the AO are screened in for assessment. Once this assessment is completed, the assessed proposals are listed in order of their scores and submitted to the SST Steering Committee for review and endorsement. Funding is then allocated to the best-rated proposals until the budget is fully allocated. The final list is approved by the IIRB. The approvals are recorded in logs to this effect.

Request for proposals (contracts)

This funding vehicle addresses the technological uncertainties related to future CSA missions. The projects funded by the program are based on needs identified by CSA program sectors for the development of potential projects. The CSA is the client for R&D activities carried out through the contracts awarded.

A process for prioritizing the technologies to be developed is in place and consists of a structured method for identification, prioritization and selection by the TPWG. Technology identification begins with the presentation, by each CSA program sector, of their needs in relation to their mission plans, roadmaps and according to the budget amounts allocated to them. Every technological need is associated with a formal Technology Development Request (TDR).

The TDRs are sent to the TPWG, which includes representatives from the three CSA program sectors, the Policy Directorate and the Programs and Integrated Planning Directorate. At this stage, the TPWG's work consists of validating the process used by the sectors and aligning the selection criteria. A list of priority technologies prepared by the TPWG is recommended to the SST Steering Committee and sent to the STDP for presentation to and approval by the IIRB (see process at Appendix B). In recent years, the STDP's budget for the RFP has been equally divided between the sectors SU and SE.

Following the presentation to the IIRB, the STDP manager defines the actions that will be taken to fund the development of prioritized technologies. A statement of work is created for each technology, and a development value is assigned to each statement of work. A letter of interest is then posted by Public Services and Procurement Canada (PSPC) to obtain feedback from interested contractors on the direction taken by the CSA and the estimated amounts. This feedback is used to refine the statement of work, among other things. The IIRB approves all contracts over $75K. The approvals are recorded in logs to this effect.

Although many consultations are held with stakeholders and a rigorous approval process is used, it is unclear how R&D investments through the awarding of contracts by the STDP align with organizational planning. The initial prioritization is currently carried out by the sectors, and the IIRB is consulted fairly late in the process. We feel that prior consultation with the EC would better integrate organizational priorities and improve the effectiveness of the planning process.

Financial planning

We found that an appropriate planning of financial resources is carried out. The roles and responsibilities for planning financial resources are well defined and applied. The program has no capital expenditures. Specific budgets are established for contribution agreements and for contracts.

The budgets are continuously updated. Management monitors financial planning on a monthly basis. In 2017-2018, 82% of the original budget for contributions was used. The available amounts were reallocated to other CSA activities throughout the year.

With respect to contracts, a net amount of $4.3M was added to the initial program budget for the 2017-2018 fiscal year. This amount was mainly used to fund additions to existing projects. These funds were allocated to the program during the year.

Organization

Programs and integrated Planning Directorate

Function

Executive Director, Programs and Integrated Planning Directorate

Action plan details

The Programs and Integrated Planning Directorate, in collaboration with the Finance, Space Utilization and Space Exploration sectors and the Science Advisor, will be responsible for defining and updating strategic priorities (including the list of priority missions), while taking into account various science and mission roadmaps and submitting them to EC for approval. Once the priorities are approved, the STDP investment planning process can begin. The Program and Integrated Planning Directorate and Finance will update the processes to maintain the list of priorities on an ongoing and timely basis.

Deadline

• Priority list: September 2019
• Process update: March 2020

Criterion 2

Program activities are monitored appropriately.

Condition

Conclusion about the criterion

The audit showed that program activities are monitored appropriately. Roles and responsibilities are clearly defined. Day-to-day activities are implemented, supervised, measured and compared with forecasts. Activities are reported and approved in accordance with internal CSA policies and directives. Standard, structured processes and procedures are applied. Furthermore, the preparation and publication of AOs and RFPs, and the intake, analysis, assessment and selection of proposals follow established processes and procedures. These are applied in accordance with program terms and conditions, and in compliance with the applicable legislation, policies, regulations and guidelines.

Roles and responsibilities are clearly defined

We found that roles and responsibilities are clearly defined and communicated. The senior STDP administrator and two managers are responsible for day-to-day activities. These three people lead activities related to the planning, drafting and posting of AOs/RFPs, as well as the process for evaluating proposals and awarding funding. The contributions process is supported by the CSA Grants and Contributions Centre of Expertise (G&C COE). As for contracts, only PSPC has the contracting delegation to award R&D contracts for the federal government. The program has signed a memorandum of understanding with PSPC to provide procurement and support services for the STDP. A single operational point of contact at PSPC has been assigned for the entire STDP.

During the assessment process, evaluators in the subject areas of each proposal are recruited inside and outside the CSA, according to the expertise sought. One person is put in charge of each proposal. This person is responsible for leading the group of evaluators and drafting the final comments for each proposal assessed by the group.

After the agreements and contracts are awarded, a project manager is assigned to each file. This person is responsible for following up with organizations and reviewing claims for payment, among other things. The level of follow-up varies according to the nature of the agreement and the organization's risk level.

Day-to-day activities are supervised, measured, compared with forecasts, reported and approved in accordance with internal CSA policies and directives. Furthermore, the preparation and publication of AOs and RFPs, and the intake, analysis, assessment and selection of proposals follow established processes and procedures that comply with program terms and conditions, as well as relevant legislation, policies, regulations and guidelines

We found that adequate control measures are in place with respect to the implementation, oversight, approval and measurement of activities. The preparation and publication of AOs and RFPs, and the intake, analysis, assessment and selection of proposals follow rigorous standard processes and procedures. Their application complies with program terms and conditions, as well as the relevant legislation, policies, regulations and guidelines.

Assessment of proposals

We reviewed the processes used to assess AOs and RFPs. We found that the expected procedures and controls are applied. The evaluation work was well structured and supervised. The necessary approvals were obtained, and the final assessment reports were adequate.

The STDP group is responsible for the process used to assess bids. The actual assessment work is carried out with the support of appropriate technical or scientific authorities. Assessment plans have been prepared and applied. These plans include a description of the assessment process and schedule, as well as a list of the evaluation committees involved and the methodology used. Proposals are assessed using the UNITAS application. This application is well structured, easy to use and includes links to documents that assessors need.

Evaluators with expertise in the subject areas of the proposals were recruited inside and/or outside the CSA. There are at least two "technical" evaluators and two "commercial" evaluators for each proposal. Preliminary meetings were held to inform assessors of the assessment process and schedule.

Once the proposals were assessed, review meetings were held to compare each evaluator's results and reach a consensus. The final comments and scores are sent to all successful and unsuccessful bidders. The substantive manager reviewed the results of each proposal, and initial assessment reports were produced, containing the ranking of the provisional winning proposals.

The appropriate CSA authorities were consulted, and the necessary recommendations, endorsements and/or approvals were obtained. The final list of proposals recommended for contribution funding were submitted to the IIRB for final approval.

When necessary, risk assessments were completed as stipulated in the Recipient Risk Assessment Framework for CSA Grants and Contributions Program. When required, final assessment reports were submitted to the G&C COE for endorsement and, lastly, to the DG, SST sector, for approval.

Awarding of agreements/contracts

The STDP worked with the G&C COE to establish and manage contribution agreements. The G&C COE ensured compliance with CSA contribution-agreement rules and provided organizational support during the various phases of the contribution process. For contracts, a dedicated PSPC officer is assigned exclusively to the STDP to act as the contracting authority.

Oversight

We found that contribution agreements and contracts receive appropriate oversight. Every contribution agreement and contract is assigned to a project manager in order to ensure monitoring. This person is responsible for day-to-day management until completion.

The claims are verified. A due-diligence process is used, supporting documents are requested, activity reports are sought according to the recipient's risk level and project advancement, and budgets are monitored. For contracts, the participation of technical authorities is also required for monitoring.

No recipient audits have been conducted since the program was introduced. The Treasury Board Policy on Transfer Payments proposes a risk-based approach for recipient audits. At the present time, the audit framework for CSA contribution recipients does not provide for audits.

Spending authorizations are controlled, expenditures are approved and consistent with legislation and regulations, and transactions are recorded appropriately

To assess this criterion, we selected five contribution agreements and five contracts. Our review determined that spending authority is exercised in accordance with delegated authorities (FAA section 32); that expenditures are made in accordance with contractual agreements, legislation, policies and regulations; that payments are based on statements certifying that the goods have been delivered, the services rendered and the work performed (FAA section 34); that the payments are made in accordance with the agreement and approved (FAA section 33); and that expenditures are reported appropriately.

Initial agreements

Each of the five contribution agreements and five contracts examined were certified pursuant to section 32 of the FAA, and approved and signed by authorized incumbents. Needs, including deliverables, were clearly defined in a quote or statement of work. No work was authorized/performed before the agreement was officially approved. A copy of the agreement signed by both parties was on file.

Amendments

One element noted with respect to contracts issued under the program is that many amendments were made during the year. From the time of their original grant date until the date of our review, 16 amendments had been made to the five contribution agreements and five contracts in our initial sample. Four amendments included an increase in the funding value of the contract following the addition of deliverables.

After reviewing the initial sample, we examined all financial amendments for 2017-2018, and identified 30. Of these, we selected five for more in-depth review.

The following is a summary of the financial amendments for 2017-2018:

Almost all of the amendments were funded from the budget increase allocated to STDP during the year.

We found that the amendments are made in accordance with the relevant legislation, policies, regulations and guidelines issued by the CSA and central agencies. We also noted that many controls are used when implementing amendments. The proposals received from contractors are reviewed by the sectors, project authorities and technical authorities. The process for approving selected investments requires the signatures of the Director General, SST, the Chief Financial Officer and the Vice-President. Investments exceeding $75K must be approved by the IIRB. Regardless of size, proposed STDP R&D amendments are reviewed, questioned and approved by the PSPC contracting authority. A process to this effect is well defined. The information requested by PSPC includes a description of the additional work, the reason why this work was not included in the original contract, the reason why an amendment to the existing contract was being requested instead of creating a new posting and a cost estimate.

As soon as requests are received, the PSPC contracting authority analyses them, assesses the relevant risks, compares the advantages and disadvantages of an amendment or new posting and determines the level of approval required on their part. PSPC confirmed that no contract had been increased in order to allow a contractor to complete the work in an existing contract. Amendments always involve new deliverables.

With respect to non-financial contract amendments, the initial sample indicated that only three of the seven contract amendments that did not increase the value of the agreement were supported by an amendment request certified by an authorized incumbent, pursuant to section 32 of the FAA. The other four amendments were supported by correspondence from the project authority or technical authority. The Requisition Guide for the CSA Procurement and Contract Administration (PCA) division stipulates that an amendment request is required when the cost, delivery date, description or any other change must be made to a contract. The Guide also indicates that the amendment should always be made on the same requisition used for the original contract. Although the four contract amendments supported only by correspondence did not include an increase in the value of the agreement, we feel that an amended requisition should be prepared in order to comply with the PCA Requisition Guide and maintain a synchronized number of amendments between the requisitions and contracts prepared by PSPC. If the CSA is of the opinion that an exchange of correspondence between the program and PCA/PSPC is sufficient, and the preparation of an amended requisition is not necessary, we feel that the Requisition Guide should be updated accordingly.

Payments

With respect to payments, we selected the last invoice submitted for each of the five contribution agreements and five contracts in our sample. We found that the amounts invoiced matched the basis of payment in the agreement, that any gaps between the amounts claimed and the terms and conditions of the agreements/contracts were justified, that the goods and services were received within the timeframes specified in the agreements/contracts, and that the necessary supporting documents have been provided. Furthermore, the invoices were certified by an authorized incumbent pursuant to section 34 of the FAA, the claims made against contracts awarded by PSPC were authorized by these people before payment, and the payments were certified by an authorized incumbent pursuant to section 33 of the FAA. Lastly, the expenses matched the amounts entered in the CSA financial systems (SAP) and the expenditures were recorded appropriately (expenditures, assets, general ledger, etc.) in CSA logs.

Organization

N/A

Function

N/A

Action plan details

N/A

Deadline

N/A

Criterion 3

Program activities are subject to accountability.

Condition

Conclusion about the criterion

The audit found that there is accountability for the program's activities. However, we identified one opportunity for improvement with respect to the information available on long-term outcomes for technology funded through contracts.

Program activities are subject to accountability

Accountability at the CSA is achieved through a number of processes. First, there is the SST business plan, which articulates sector objectives for the fiscal year. The program is required to submit quarterly reports on the progress of main tasks that must be completed during the year. This document is used to prepare the overall CSA business plan.

Second, the program reports on its activities while preparing the annual SST performance report. The annual performance report provides the SST directors and director general with data and analyses on the results achieved during the previous fiscal year. This report is also used to complete Electronic Work Plans (eWPs).

Contribution agreements

Short-term contribution results

Contribution agreements provide for the disclosure of results achieved following project implementation. Recipients must demonstrate the R&D work that was accomplished. The CSA requires recipients to produce a report demonstrating how the contribution was used. Results and data on indicators must be shared with the CSA.

The CSA compiles these results at the end of the agreements. We reviewed a performance-data compilation report for the AO 2. The program has used data obtained from recipients to produce this report, which was dated September 2017. The data are provided according to the following five indicators set out in the contribution agreement:

• Technology readiness levels (TRLs) and know-how;
• Intellectual properties generated;
• Basic industrial skills and products developed;
• Nature of the infrastructure developed;
• Highly qualified personnel involved.

The report provides an overview of the impact of the AO 2 on the Canadian space industry. Some of the data collected for this AO were not sufficiently well defined to draw accurate conclusions on some issues, and enhanced directives were expected to be issued for subsequent AOs.

Long-term outcomes and feedback on technological developments for contributions

For the first time, in 2018, STDP contribution recipients were asked to complete a survey on long-term outcomes for the AO 2. The agreements had ended a few years before. The key elements assessed during this survey focussed on the development of industrial capacity and the commercial benefits generated with the help of the developed technology. Since this exercise had yet to be completed at the time of this report, the results were not reviewed by the audit team.

This monitoring of outcomes over the long-term is a best practice, as it makes it possible to determine the program's effectiveness with respect to industrial capacity building. The program plans to continue this activity.

Contracts

Short-term outcomes

With respect to contracts, the deliverables are set out in the contracts. There are specific performance requirements. The contractors must therefore achieve the expected outcomes. If these outcomes are not met, the contractor's remuneration is withheld until the CSA is fully satisfied. The last phase of the projects includes a final review by the CSA with the supplier. The sectors affected by the work are invited to take part in order to express their level of satisfaction.

Long-term outcomes and feedback on funded technology

Program management has started conducting internal surveys of CSA sectors to obtain feedback on technological developments funded through contracts (RFP) in order to identify the impact and benefits observed in the longer term.

We reviewed some of the completed surveys. They indicate, for instance, that some contracts had helped to eliminate many technological uncertainties, finalize estimates for mission aspects (e.g., cost, level of effort, schedule) and reduce major technological risks for a potential mission. They also indicate that some technological developments sought for specific missions have been approved while others were pending. Some technological developments should help to inform decision making regarding the approval of Canada's participation. It is expected that new internal surveys will be carried out shortly, that the results will be analysed and that a report will be submitted to management. For example, it would be relevant to know how many technological developments funded have been used in a mission when assessing the program's effectiveness.

We feel that more detailed accountability and statistics could support planning and decision making for future investments.

External accountability

External accountability requirements are met. The program's main external accountability requirements come from the following directives: proactive disclosure of contracts; disclosure of grant and contribution awards; disclosure of results; and disclosure of travel, hospitality, conference and event expenditures.

All contracts awarded under the STDP are issued by PSPC. PSPC is responsible for disclosing them in accordance with applicable regulations. Moreover, STDP agreements are posted on the CSA website in the weeks prior to the contracts being awarded. The same is true for contribution agreements awarded by the CSA.

The disclosure of travel, hospitality, conference and event expenditures is done by the CSA Finance Directorate.

Organization

Space Science and Technology Directorate

Function

Director General of the Space Science and Technology Directorate

Action plan details

The Space Science and Technology branch will continue developing the mechanism established in 2018 under the STDP's contribution element. We will therefore survey, on a annual basis, organizations that have conducted research and development projects to determine the impact of the research results on those organizations.

A similar mechanism will be developed and implemented under the STDP's procurement element. The internal sectors that initiated technology development requests will be surveyed at the end of each contract to determine the extent to which the research results are relevant and useful. Companies that have developed technology will also be surveyed to determine the benefits of their contracts.

• The Space Science and Technology branch will be responsible for periodically following up on and disseminating the results as an internal report to support planning and decision-making for future investments; and
• The internal sectors in question will be responsible for providing the information requested through a survey.

Deadline

September 2019