Assemblée législative de l'Ontario

The committee met at 0900 in room 151.

Committee business

The Chair (Ms. Cheri DiNovo): Good morning, everyone. The minister is on her way. Until she arrives, are there any questions? Yes, Ms. Kiwala.

Ms. Sophie Kiwala: I would like to bring forward a request for unanimous consent. Given that we have a considerable amount of time left for Treasury Board, and having a look at the schedule and how things are going to play out this afternoon, I would like to request that we have unanimous consent to bring forward energy tomorrow and start fresh with energy tomorrow, rather than—I think we’ve got about 45 minutes or so left—just so that we can make the committee run a little more efficiently.

The Chair (Ms. Cheri DiNovo): Okay. Does everybody understand that? Ms. Kiwala has moved that we move energy to tomorrow and finish with Treasury Board today. Is there unanimous consent—a question?

Mr. Todd Smith: Yes, just a question: We don’t lose any of our committee time with the various ministries by pushing the time ahead, correct?

The Chair (Ms. Cheri DiNovo): The Ministry of Energy will still have seven and a half hours.

Mr. Todd Smith: Do we lose at the back end, though, with future ministries?

The Chair (Ms. Cheri DiNovo): We’ll lose a little bit.

Interjection.

Mr. Todd Smith: They’re your ministries, probably, that—

Mr. John Vanthof: Not if we’re going to lose time. I’m not in favour of unanimous consent if we’re going to lose time.

Mr. Todd Smith: I’m not either. I believe that we should maximize the time that we have.

The Chair (Ms. Cheri DiNovo): Okay. Ms. Kiwala, you’ve heard that. Shall we recess for a few minutes? Would you like to draft a motion?

Ms. Sophie Kiwala: I think we would like to draft a motion, yes.

The Chair (Ms. Cheri DiNovo): Okay, we’ll recess just for a few minutes. That gives the minister time to get here, and we’ll look forward to your motion. We will reconvene in five minutes.

The committee recessed from 0901 to 0907.

The Chair (Ms. Cheri DiNovo): Okay, I think we’re ready to begin. Ms. Kiwala, do you want to read this motion?

Ms. Sophie Kiwala: I move that the committee begin its consideration of the 2017-18 estimates of the Ministry of Energy following routine proceedings on Wednesday, October 4, 2017.

The Chair (Ms. Cheri DiNovo): Any debate on that motion? Yes, Mr. Vanthof?

Mr. John Vanthof: We are not in favour basically because, at the end, there will be time lost by this committee.

The Chair (Ms. Cheri DiNovo): Thank you. Ms. MacLeod?

Ms. Lisa MacLeod: We in the official opposition are in concurrence. Estimates is an opportunity for the opposition to question ministers and hold the government accountable. I think this motion flies in the spirit of that.

The Chair (Ms. Cheri DiNovo): Any further debate? Ms. Kiwala.

Ms. Sophie Kiwala: Just one final comment, and something that I’ve brought up in the past: Had the opposition members chosen the ministry estimates for consideration and not delayed the choices for several months, we could have done this in the last session and would have been able to better plan for these hearings.

The Chair (Ms. Cheri DiNovo): Any other debate?

Ms. Sophie Kiwala: I move that we proceed to a vote.

The Chair (Ms. Cheri DiNovo): We will then proceed to the vote. All those in favour of Ms. Kiwala’s motion, please raise your hands. All those opposed to Ms. Kiwala’s motion, please raise your hands. I declare the motion carried.

Treasury Board Secretariat

The Chair (Ms. Cheri DiNovo): We’re going to resume consideration of vote 3401 of the estimates of the Treasury Board Secretariat. There is a total of one hour and 48 minutes remaining.

Before we resume consideration of the estimates, if there are any inquiries from the previous meetings that the President of the Treasury Board has responses to, perhaps the information can be distributed by the Clerk. Are there any items, President?

Hon. Liz Sandals: I do not have any information at this time.

The Chair (Ms. Cheri DiNovo): Thank you.

When the committee last adjourned, the government caucus had 10 minutes left in their round of questions. Ms. Hoggarth.

Ms. Ann Hoggarth: Good morning. The Ontario public service has been recognized many times as a top employer and has also received awards for the support of a diverse workplace. In fact, the OPS was recognized as one of Canada’s top 100 employers in 2017, one of greater Toronto’s top employers in 2017, one of Canada’s top employers for young people in 2017 and one of Canada’s best diversity employers in 2017. How is the Treasury Board Secretariat ensuring that the Ontario public service continues to be an employer of choice?

Hon. Liz Sandals: Thank you. That’s a lovely question. I’ll pass this on to the deputy, but it does give me an opportunity just to comment that one of the things that we are blessed with in Ontario is a wonderful public service. Governments come and go. All of our parties here have been government at one time or another, but the public service is the constant, and if you are going to have good public service, then you need good public servants. We are particularly fortunate at Treasury Board, where I have an amazing group of public servants to work with all the time. So thank you very much, everybody.

Over to you, Deputy.

Ms. Helen Angus: Sure, delighted. The OPS has a long—

The Chair (Ms. Cheri DiNovo): Can you introduce yourself just again? New day. Thank you.

Ms. Helen Angus: Yes, absolutely. Helen Angus. I’m the Deputy Minister for Treasury Board Secretariat.

The OPS has a long tradition of transforming and adapting to changing conditions and delivering public services in new and innovative ways. Obviously a key component of the strategy is people—not just in Treasury Board but across the various ministries that comprise the government. In order to attract and retain the talented leadership, we need to deliver on the priorities. Treasury Board is leading or contributing to a number of enterprise-wide initiatives. We also provide, as you’ve heard over the last couple of days, policies and programs that underpin a strong workplace culture.

Janet O’Grady is going to get up and give you a little bit more information. I’ll just give you some highlights as she gets up to the chair. She’s going to talk a little bit about some of the amazing work we do in leadership development, both at the manager and executive levels; a little bit about the recognition and awards that we’ve received; how we’re actually looking at public sector renewal and how we attract top talent, and that will build on some of the remarks that Diane McArthur made last week; what we’re doing around the respectful workplace; and, again, how we’re trying to deepen our understanding of employee engagement and what measures we’re doing to improve that.

With that, I’ll hand it over to Janet, who—you may introduce yourself as well, but Janet is the director of human resources policy and planning within the Centre for Leadership and Learning.

Ms. Janet O’Grady: I am Janet O’Grady, director of HR policy and planning in Treasury Board Secretariat.

To ensure that the OPS remains an employer of choice, as the deputy said, TBS leads the development of HR policies and strategies to enable government outcomes. TBS works closely with MGCS, the Ministry of Government and Consumer Services, in implementing HR policies. They develop and deliver the related HR programs and services to support the implementation.

Among the enterprise-wide initiatives that TBS leads are, first of all, the OPS HR plan. That is a strategy that’s under way. It’s a five-year plan that addresses the key priorities of workplace culture, leadership and HR practices. We’re also leading a public service renewal initiative. That is going to build on the HR plan that is currently under way and other transformational strategies, including digital, open government and so on, to ensure that we have the diversity, skills and leadership capacity to support innovation and enhance resilience within the Ontario public service.

If I start with the OPS HR plan, which is in place from 2015 to 2020, it is a priority-based plan, and so I’ll share some information and some actions related to each of those priorities.

The first is enhancing a positive and inclusive workplace culture. TBS is working with MGCS on safe and healthy workplace initiatives. We have developed an inclusive policy framework to enable employees to fully participate in the workplace without barriers and without fear of harassment or discrimination.

In 2016, we launched the respectful workplace policy. I think Diane McArthur mentioned it briefly. It establishes a framework for preventing, identifying and effectively responding to workplace harassment and discrimination within the OPS. It focuses on prevention where possible, and effectively responding to and addressing problems when they arise.

TBS holds the OPS accountable for the health of its workplace through these policies, and it reinforces them through ongoing development of resources and communications to reinforce those messages.

Continuing with culture, TBS is using employee engagement as a means of fostering a meaningful relationship with OPS staff. Employee engagement is an important determinant of an organization’s success. High engagement benefits both the OPS and the public that we serve. It helps maximize productivity and encourages commitment, which helps to retain valuable knowledge and experience. Our engagement score improved from 67.3 in 2014 to 69.9 in 2017. The OPS is committed to being a top employer and continues to develop workplace policies and strategies incorporating feedback identified in the employee survey.

By leveraging the survey to create a two-way dialogue with employees, TBS is ensuring that we maintain the workplace culture that has made the OPS an employer of choice. There is evidence that we are on the right path. As you noted, since 2009 the OPS has been honoured with awards that recognize the organization’s operations and HR practices across a wide range of factors. That’s everything from physical workplace to work atmosphere, health, financial and family benefits, diversity, charitable efforts and so on. As a result, as you noted, the OPS has been recognized as one of Canada’s top 100 employers in seven of the last 11 years. In 2017, as you noted, we have received a number of awards: Canada’s Top 100 Employers, Greater Toronto’s Top Employers, Canada’s Top Employers for Young People, Canada’s Best Diversity Employers and Canada’s Greenest Employers. We are the only government to receive Canada’s Best Diversity award for 10 consecutive years.

This decade-long track record affirms the OPS as a leader in positive workspace and culture, in both public and private sectors, municipally, provincially and federally.

To move to the second priority under the HR plan, it focuses on leadership. TBS is committed to investing in the development of the future leaders of the OPS, focusing on enhancing business-critical skills such as change management, transformation, communications and strategic management. Our leadership programs provide support and guidance through all stages of an OPS career, from aspiring managers to senior leaders. Some examples of those programs include Advancing Into Management, which is designed to prepare high-potential individual contributors for transition into management roles. We also have the Senior Leadership Potential Assessment, which is a best-practice framework through which senior leader potential is assessed in the context of ability, aspiration and engagement.

The Chair (Ms. Cheri DiNovo): Just over a minute left.

Ms. Janet O’Grady: Okay. I’ll move on, then, to talking about the third area, which is effective and fair HR within the OPS.

The OPS hires all individuals based on the principles of merit, inclusion and accessibility. We have a number of specialized programs where we target outreach, including the Ontario Internship Program for youth, the OPS Internship Program for Internationally Trained Individuals, and the OPS Internship Programs for Internationally Trained Engineers. We also have a specialized program called OPS Learn and Work, which is a leading example of how the OPS as an employer models corporate leadership. It is a specialized co-op program designed to re-engage high school students from priority communities in Hamilton, Thunder Bay, Toronto and Windsor. The OPS Learn and Work Program provides youth with the opportunities to earn credits toward their high school diplomas—

The Chair (Ms. Cheri DiNovo): And I am afraid that is it. Thank you.

We now move to the official opposition. Mr. Smith.

0920

Mr. Todd Smith: Back to the minister, the President of the Treasury Board—good morning.

Hon. Liz Sandals: Good morning.

Mr. Todd Smith: I know, when we started off estimates a couple of weeks ago—or it seems like it was a couple of weeks ago; I guess it was just last week—my colleague from the NDP was asking about the fair hydro plan ads and had a lot of questions about that. I know they had a freedom-of-information request that came back saying there was $5.5 million spent on ads for the fair hydro plan. I was just wondering who was contracted to create those ads. Do you have that information?

Hon. Liz Sandals: I don’t believe I do. In fact, maybe I’ll talk a little bit about the approval process for advertising. I don’t know if the FOI was from our ministry, but what I can tell you is that I don’t believe that we have processed the receipts yet. So I think what you are quoting might be a budget allocation amount as opposed to an actual spend amount. If I talk a little bit about the approval process, then you’ll see why the information from one ministry would be, possibly, different from the ultimate spend, which is where we get involved.

Mr. Todd Smith: Just before you do that, would it be possible for you to enlighten us as to who—at a future date, to provide the committee with the information as to who—

Hon. Liz Sandals: Well, number 1, you would need to ask the Ministry of Energy at this point, because to the best of my knowledge—okay? I’m just saying that, to the best of my knowledge, we have not received the invoices. So I can already tell—is that correct?

Interjection: It’s true. You’re on firm ground.

Hon. Liz Sandals: Okay.

Mr. Todd Smith: There’s a lot of nodding heads back there.

Hon. Liz Sandals: Nodding heads are behind me, so let me answer that more definitely: I cannot produce what we do not have, and we do not have the actual invoices.

Mr. Todd Smith: Okay. That’s fair enough.

Hon. Liz Sandals: So I can tell you that, no, I can’t give you that because I don’t have it.

Mr. Todd Smith: We’ll ask the Minister of Energy tomorrow.

Hon. Liz Sandals: So let me tell you, though, a bit about the process, because I think there’s a fair bit of misunderstanding. It is certainly true that there is an account, which obviously shows up in the estimates, called the bulk media buy fund. That shows up in the Treasury Board estimates. However, Cabinet Office communications is actually responsible for the overall coordination of the bulk media buy fund, planning against the allocated budget—so you can see the allocated budget in our accounts—and supporting ministries with marketing expertise. Cabinet Office ensures campaigns align with industry best practices, with French and multicultural requirements, with the AODA—that’s accessibility compliance—and adherence to the standards of the Government Advertising Act.

TBS, as I said, does hold the approved funding for the bulk media buy advertising and liaises with ministries to guide them through the submission requirements as set out in the Financial Administration Act, which is the submitting of invoices to us, and the administrative requirements for submissions to Treasury Board and Management Board, which are required at one point in the process. But associated expenses for each ministry-sponsored advertising campaign are reported in their respective ministry public accounts. So ultimately, the report on the spend will go to the individual ministry.

To start off, to obtain that funding, ministries prepare a request for a Treasury Board order to transfer the required amount, consistent with the actual invoices, to their budget from the bulk media buy fund.

However, let’s back up a little bit. I can talk more about what happens when we get the invoices, if you’re interested.

Mr. Todd Smith: Chair, can I just ask one question about that before you continue? I’m just wondering—in the case of the fair hydro plan advertising that we heard all summer, it would be the Ministry of Energy that, I guess, seeks out the organization or the company that’s going to provide that creative and—

Hon. Liz Sandals: Okay, let me explain a little bit. If a ministry wants to use funding from the bulk media buy—and there are some ministries that would have some of their advertising fund at the ministry. For example, something that goes on every year, something like the Foodland Ontario ads—they happen every year, and while they still have to go through the approval process, the budget for that actually just permanently sits at OMAFRA.

Some of the Ministry of Health funding is in the bulk media buy. But there is also some of the public health. Flu season happens every year and they always remind people, so some of that public health budget actually sits at the Ministry of Health.

MNR has things that they routinely advertise about every year, maybe how to do online campsite reservations, something like that. That’s a routine and it would sit at MNR.

The bulk media buy is, if I can say, maybe the more flexible global amount that is going to be different each year because each year it will be addressed at different issues depending on what’s going on in government. That process actually begins with the ministry going to Cabinet Office with a proposal for advertising and getting approval from Cabinet Office communications. What they would be given by Cabinet Office would be a maximum allocation that they could spend.

The process then goes on to both the creative and the actual procurement. The procurement at that point must be done through the vendor-of-record process. It’s actually the Ministry of Government and Consumer Services that is responsible for creating the vendor-of-record list. That’s something that happens periodically: Every four or five years, there’s an opportunity to get on the vendor-of-record list.

Once there is an approval to have an ad campaign, then the ministry—not Treasury Board but the ministry that has the approval for the campaign—would work with the Advertising Review Board, which is an agency or a branch at the Ministry of Government and Consumer Services, to actually do a competitive procurement amongst people who are on the vendor-of-record list.

Mr. Todd Smith: And there’s no creative direction that comes from your—

Hon. Liz Sandals: At this point, Treasury Board has nothing to do with anything. We’re sitting with this consolidated fund, but it has been at cabinet, or at least Cabinet Office, it’s been at the line ministry initiating, and then the line ministry is working with the vendor-of-record list and the Advertising Review Board to do the procurement.

Early on in the process, the ministry would take the proposed ad to the Auditor General and get the Auditor General’s approval for the advertising, because there’s no point in fleshing out the creative work if it is going to be rejected.

Once they’ve gone through that formal competitive procurement process and got the preliminary approval from the Auditor General for the preliminary proposal, they would then do the creative work. They work with Cabinet Office again and then the Advertising Review Board in terms of actually procuring the media buy. Note that this could be—I think we all jump to the conclusion that it’s television, but it might be television, it might be digital, it might be radio, it might be newspaper. You can go on and on and on.

0930

Mr. Todd Smith: So is it safe to say, because—

Hon. Liz Sandals: And we see it at the point where there are actual invoices.

Mr. Todd Smith: Right. Okay. So you’re not involved in the creative part. You don’t have focus groups at Treasury Board that are telling you what to advertise.

Hon. Liz Sandals: No. We have nothing to do with the content. Our role is that eventually the invoices come to Treasury Board and, based on the invoices then approved by Treasury Board, there will be a Treasury Board order to transfer the money to the ministry.

Mr. Todd Smith: Sure. Okay.

Hon. Liz Sandals: Which is why you see this pot of money at the beginning of the year in the estimates. By later in the year, you would see that much of that money has been transferred to various ministries.

Mr. Todd Smith: And that pot of money is now much larger than it has been in the past, correct?

Hon. Liz Sandals: Yes, but that’s—

Mr. Todd Smith: Is that because there are more ads running? Is that because of the cost of producing ads is more?

Hon. Liz Sandals: No, it actually isn’t. I’m just going to look for some detail.

Ms. Helen Angus: Mel can talk about—

Hon. Liz Sandals: Okay. While Mel’s coming up—

Mr. Todd Smith: Mel has been doing most of the nodding back there.

Hon. Liz Sandals: Yes. She’s the person in charge of nodding and a whole lot of other things.

What I was going to say is, the reason that you think you’re seeing a dramatic increase is that last year, 2016-17, at estimates time, there was $25 million and change in the bulk media buy fund. Shortly after the estimates in 2016-17, there was a decision to consolidate some of the flex money from a variety of ministries. I have got the numbers, but I’ll just read the names. The Ministries of Children and Youth Services; Citizenship and Immigration; Community Safety and Correctional Services; Culture, Tourism and Sport; Economic Development and Growth; Education; Energy; Government and Consumer Services; Health and Long-Term Care; International Trade; Labour; Status of Women: They all transferred money into the bulk media buy fund. That then set up a much bigger fund, so that in 2016-17, after that consolidation, the total bulk media buy fund was $50,465,000. It is true that this year the bulk media buy fund, with the consolidation from other ministries, is $56,681,500. It went up about $6 million. But that’s got more—remember I said they were in charge with AODA compliance and all that stuff?

Mr. Todd Smith: Sure.

Hon. Liz Sandals: It’s actually more that, and the fact that a lot of digital stuff used to be free. Now a lot of it is not just that you have to pay for it; it’s getting more expensive.

Mr. Todd Smith: I’m aware of that.

Hon. Liz Sandals: So it isn’t more advertising. If you actually compare apples to apples, the increase has really more to do with the expense—

Interjection.

Mr. Todd Smith: Thanks, Minister. And no offence, Ms. Fraser, but maybe we can talk after. I know my colleague Ms. MacLeod has some questions she wants to get in in the remaining few minutes.

Ms. Lisa MacLeod: Just to follow what my colleague was talking about, you’ve jumped 126% from your estimates to actual, according to page 117. How could you be so far off, and is there any other advertising that is done by government that’s not included in this bulk media buy, and what would that be?

Hon. Liz Sandals: Oh, yes. I just actually explained that. There are a number of ministries which retain advertising funds.

Ms. Lisa MacLeod: I understand that. What would that total number be, then?

Hon. Liz Sandals: I’m sorry. I don’t have that in front of me, because it would be in the estimates of a variety—

Ms. Lisa MacLeod: Could you provide that to the committee, Minister?

Hon. Liz Sandals: It isn’t in our estimates.

Ms. Lisa MacLeod: Okay.

Hon. Liz Sandals: It’s in a variety of other ministries, actually. It’s not in our estimates.

Ms. Lisa MacLeod: Was your ministry responsible for the ORPP ads?

Interjection: No.

Ms. Lisa MacLeod: No?

Hon. Liz Sandals: Ultimately, as I say, it’s a Cabinet Office decision.

But I want to take issue with the number you were quoting there because, as I say—

Ms. Lisa MacLeod: Which number: the $5.7 million?

Hon. Liz Sandals: If you look at the—

Ms. Lisa MacLeod: Which number?

Hon. Liz Sandals: The comparison that you were making from one year to the next year.

Ms. Lisa MacLeod: Oh, in terms of the estimate and the actual?

Hon. Liz Sandals: If you look at the at the estimates in 2016-17, it is true that they started off at, I think, $25 million—that is true—but as I say, the decision was made shortly after the estimates were published to create a larger consolidated fund by transferring money from other ministries.

Once you had transferred money from the other ministries into the consolidated bulk media buy fund, the number then—and I agree, it wasn’t what was printed in the estimates, but we transferred money into the bulk media buy fund and the number was $50,465,000. So if you compare apples to apples, the increase is about $6 million and, as I was just explaining, that isn’t about buying more advertising; it’s about compliance, it’s about translation and it’s about the increase in the cost of digital.

Ms. Lisa MacLeod: I know I don’t have very much time. The fair hydro plan, that was through the Ministry of Energy or through you?

Hon. Liz Sandals: That’s what we were talking about initially. The number you would have received from energy was the maximum allocation that they were allowed for the fair hydro.

What I cannot tell you, because we literally have not received the invoices, is what the actual spend is because they will come to us for a Treasury Board order to transfer the money to them to cover the actual costs.

Ms. Lisa MacLeod: Would it be fair, then, to say that—the leader of the third party has been raising this issue of $5.5 million for the fair hydro plan—it could actually be higher than that, the advertisements for the fair hydro plan?

Hon. Liz Sandals: No. As I say, that is information that came from another ministry. We have nothing to do with the Cabinet Office notional allocation of a maximum amount to the Ministry of Energy. All we see is when the invoice comes to us, and we have not yet received the invoices.

The invoices will absolutely be lower than the maximum that was allowed. They cannot exceed the maximum that was allowed, so whatever the actuals are, they will be lower than that amount. That’s all I can tell you now because we haven’t got the invoices.

Ms. Lisa MacLeod: Because I only have two minutes, quickly, opioids are a huge issue. I’ve been calling for more education and awareness and advertising. I’m wondering if there’s a line item in the bulk media buy for government advertising with public health in mind with respect to the opioid crisis. If so, how much is it and could it equate to the $5.5 million that is for the fair hydro plan and the $5.7 million that the ORPP—

Hon. Liz Sandals: So once again, that request would not come to us. If a ministry wanted to advertise about opioids—obviously, the Ministry of Health. If it wanted to advertise about opioids, probably if it wanted to advertise about shingles vaccines—there’s a whole host of things and, in fact, the Ministry of Health is typically the largest advertiser, given that it’s the largest ministry.

Ms. Lisa MacLeod: Yes. What was it, about $9 million last year?

Hon. Liz Sandals: Again, the actual spend will ultimately be on the books of somebody else—on the ministry’s books.

The process, regardless of what the issue is, is the ministry goes to Cabinet Office and Cabinet Office controls the allocation from the bulk media buy fund to individual advertising campaigns at individual ministries. The allocation that Cabinet Office makes will be the maximum amount that that ministry can spend on that specific campaign, and that is also some money that is retained for routine public health.

0940

The Chair (Ms. Cheri DiNovo): I’m afraid, with that, your time is up. We now move to the third party: Mr. Vanthof.

Mr. John Vanthof: Good morning, everyone. I’ve been listening to all the talk about media buy. First, not a question, a comment: It has occurred in many parts of rural Ontario—and I know each individual ministry has responsibility for this, but you control the overall budget. Perhaps you can pass it down that more and more the government focuses on digital advertising, and in many parts of certainly the north and rural Ontario, community newspapers—there are a couple in my riding—are still the primary way that the public gets information. In many cases, the government no longer advertises the issues that are important. The things that government should be advertising to the general population are no longer being done, in many cases, in community newspapers. In places where there is no Internet, outside of a town—unless you pay $150 a month, which most people don’t pay—you’re not reaching, in many cases, the population that you need to reach. I need to put that on the record because it is a huge issue. I’d just like to put that on the record.

At the last estimates meeting, we had a great presentation from the behavioural analysis group. It got me thinking a lot about behavioural analysis. I drive a lot on the weekend to go between events, and there are some pretty good podcasts and stuff on behavioural analysis.

Hon. Liz Sandals: A new area of expertise.

Mr. John Vanthof: In marketing, it’s called “nudging.”

Hon. Liz Sandals: Exactly.

Mr. John Vanthof: Oh, yes. I can give you an example of nudging. The British government had an energy conservation program to insulate people’s attics, and there was no uptake. Through behavioural analysis, they figured out that people didn’t take the program up because nobody felt like cleaning the junk out of their attic.

Hon. Liz Sandals: That actually makes sense to me, except in my case it’s the basement.

Mr. John Vanthof: So when the program included cleaning out your attic: big boom in uptake. Behavioural analysis is an incredibly powerful tool.

My line of questioning is: When and who decides and how is it controlled within the ministry—how do you pick or who picks what behaviours you’re going to analyze? Because this is an incredibly powerful tool. It also is very important how it’s directed.

Hon. Liz Sandals: Do you know what? The deputy might have a few words about this, but we do have with us today the head of the unit.

Mr. John Vanthof: Oh, cool. Cool.

Ms. Helen Angus: The behavioural insights unit: You met with one of the staff, Dr. House, last week. Didem is actually the director for the Centre of Excellence for Evidence-Based Decision Making, which houses—no pun intended—the behavioural insights unit.

You can talk a little bit about when you have a new unit in government, sometimes you have to till the soil and beat the bushes and find clients. We have been pretty successful in getting more ministries to come to us with their projects, where they’re trying to put their programs in on the ground or they’re to make improvements to the way that they deliver services and see that behavioural insights could actually help them be more successful more quickly.

Maybe you can talk a little bit about the nudge challenge, which might be of interest. Since you’ve used the word “nudge,” the nudge challenge might be an interesting thing to explore.

Ms. Didem Proulx: Absolutely. I’m Didem Proulx. I’m the director of the Centre of Excellence for Evidence-Based Decision Making, and the behavioural insights unit, as the deputy mentioned, is part of our unit. As the deputy mentioned, originally we had to go to the ministries and, based on some of the interventions that worked in other jurisdictions, made pitches and started developing our project base with respect to that.

But we also, as part of the capacity building and engaging OPS at all levels—did initiate what we call the OPS nudge challenge, in which we reached out to all the ministries to solicit ideas from the staff who run programs and who actually are involved in the delivery of services on a day-to-day basis, where they saw the opportunity for improvement where the behavioural insights can help.

There was tremendous interest. We received almost 100 applications, and we only had so many spaces so we chose the best project and group. We brought a group of almost 40 or 50 OPSers together and went through a rigorous process in which we had developed some of their own ideas about the programs and policies that they deliver on a day-in, day-out basis. Now that those are fully developed, those are back with their ministries to implement.

As the word gets around, there’s a difference that behavioural insights can make into the effectiveness of programs and outcomes and how clients experience the services and the programs that they partake in. We’re getting a lot more interest and a lot more ministries are coming forward to explore, as the deputy said, if behavioural insights can help with their particular programs.

Mr. John Vanthof: So this behavioural analysis insight program is at the beginning of its—there’s no official protocol of how to pick a project?

Ms. Didem Proulx: I think it’s a kind of combination of ministries reaching out to us with respect to the programs that they want to examine and us working through them to see, because not all programs have that behavioural touchpoint where you can actually make a difference. So we sit down and work with them to see if it is something the behavioural insights unit can help with. If it is something, we absolutely roll up our sleeves and work in tandem with them and develop a number of possibilities and test them out.

This is part of the evidence creation space as well, where we test out different interventions and see which one is most effective, rather than just assuming a particular one will be the most effective option. When the results come in, when we figure out what is most effective, then we work with the ministry. We actually hand the results back to the ministry and they scale it up across their entire operations.

But we take on projects through a number of different ways, so there is no one standard process now. We are trying to be nimble and innovative and be as responsive to our line ministry colleagues and partners as we possibly can.

Mr. John Vanthof: I fully appreciate the benefits that this could have, and I’ve never used this word in the Legislature before and I probably never will again, but behavioural analysis, if you do a lot of reading on it, it’s an incredibly powerful tool. If you look at marketing campaigns, it borders on Orwellian, how powerful it can be.

So when I hear government—and government involves politics—talking about behavioural analysis, I want to know what controls are going to be in place. How do you define—this isn’t an OPS question; this is a political question. Who is the gatekeeper? What behaviour are you going to try and influence? Because it can be—I don’t have the book in front of me with some of the examples, but some of them are truly chilling.

Hon. Liz Sandals: I’ll go back to Didem, but if I could, I think maybe the question you’re asking is, do ministers’ offices sit around thinking up—

Mr. John Vanthof: I don’t think you’re that open about it.

Hon. Liz Sandals: —you know, things that we want to do to nudge the people into political behaviours, or how do projects get chosen? I think the bottom line is that the projects that the unit would be working with are things where it’s already a government process. We gave the example of organ donors. That’s something we already do.

0950

Mr. John Vanthof: Yes.

Hon. Liz Sandals: How do we get people to go online to renew licence plates? Something we already do. So it’s not that the project of this government service starts with the unit; it’s that this is already an approved government program that is less effective than we would like it to be.

So, back over to you, but I think that’s maybe what you’re trying to figure out?

Mr. John Vanthof: If I could, an example, going back to what our party brought up and the official opposition also brought up: advertising for the fair energy plan. That’s something the government has decided to do. We obviously disagree politically on whether that’s a good idea or not, but it’s something the government already does. So, by your definition, the government could decide to apply more behavioural analysis to that.

Hon. Liz Sandals: Except I don’t think you—there’s no intersection just because the centre of excellence is at Treasury Board and the bulk media buy fund are at Treasury Board. There actually isn’t an intersection there, really. The advertising is a different stream than the programs you’re working with.

Didem can maybe give you some more examples of some of the programs that we work with.

Ms. Didem Proulx: Absolutely. Just to be clear, we don’t work on anything that is related to marketing or communications. Our work is strictly with respect to policies and programs that are already in existence and how do we make it easier for people to access to increase the effectiveness of them.

I believe last week the organ donation, the online registration for vehicle licences—a number of the completed projects have been discussed. Of the projects that we are currently doing, we are working with ministries on how to increase more student engagement: How do we find the non-traditional ways of getting at students or parents? We are working on recycling behaviour. I think all of us intend to walk towards the garbage bins and put the right things in the right buckets because the labels are different and they say different things. The choice doesn’t always seem obvious, so we have been working with the Ministry of the Environment to see if the labelling and how we tell people what to do can make a difference.

There are many examples where behavioural insights are used. Actually, I always turn to my wrist and Fitbit—how do you make something more tangible so that people do the right thing, that they always know that they need to do, such as getting more exercise? We are not currently working on this, but these are the kinds of examples of what is possible. Public health-related issues: how to make people more mindful of their antibiotics or addictions issues. We have a lot of projects currently under way with partner ministries. The cervical cancer screening program: It is a good test to have and it saves lives, but many women who are eligible for the test are actually not taking advantage of what could possibly be a life-saving test and opportunity. How do we let people know that they’re overdue for their test, and do it in a meaningful way so that it doesn’t become a letter that gets chucked in the garbage, but something that they pay attention to and take action on.

We are working on a number of existing programs that way. They’re all programs that I think have been around for a long time and have cleared the public-good test, so we are entirely in that space.

Mr. John Vanthof: Again, I think that’s a good idea. The only thing I really want to put on the record is that it’s when we start with digital. Digital computers have all kinds of great things, but now we’re seeing the problems we’re running into with digital. A lot of those problems are because we didn’t think about the problems we were going to have or could have at the start of the process. When I listened to the original presentation about the behavioural analysis and how we were leading the pack along with the OECD and a few others—we’re leading the pack? Then we also need to lead the pack to make sure that the checks and balances are in place.

I’m not trying to accuse; I’m not trying to be partisan or political here. But whenever you talk about controlling people’s actions, you’ve got to be cognizant that in a political atmosphere, there has to be some kind of checklist. The example of the behavioural analysis unit is not attached to the media buy, but I’ve been here for six years and until I sat here, I didn’t know any of this stuff about how the digital is controlled by the Treasury Board Secretariat.

Unless you have some kind of actual controls, eventually we’re going to run into troubles like we’re running into troubles with digital.

Hon. Liz Sandals: Can I just make an overall comment? In some ways, I think that often is the role of Treasury Board Secretariat. In a whole bunch of areas, we hold the policy. Whether it’s HR policies—and that includes the obvious labour relations, human resources sorts of things, but it also has to do with diversity and harassment and a variety of areas where we hold the policy. If you look at the whole area of government procurement, we hold the policy.

That’s something that has evolved over time. That actually is a role that Treasury Board Secretariat plays. I think it would also be, say—we were talking last week about how we’ve consolidated a lot of the I&IT, which means that we actually hold a lot of the policy around I&IT.

I think the behavioural unit and the centre of excellence is like a whole lot of other things in that this is an area where it’s only a few years old. As we get that experience and think about what services we can best provide, what services we need to provide and what services we don’t want to be involved in, that’s an area where policy will evolve as we look at the various initial things.

In fact, part of the role of Treasury Board Secretariat is that evolving policy role because policy is something that does evolve; it’s not static. While being a policy wonk is sort of, “Oh, that’s those weird people over there”—we were laughing this morning while we were waiting for my binder to arrive that in some ways, we’re the centre for nerds and geeks because it is a lot of policy wonks. It is people who are very technical; it is people who are concerned about the detail of policy and the evolution of policy. We actually have a staff that probably is one of the staffs that comes from a more technical qualification piece, and they actually are aware of the need to have that evolving policy role. You’ve actually identified something that really is an important function of Treasury Board Secretariat.

The Chair (Ms. Cheri DiNovo): You have just a few seconds left, Mr. Vanthof.

Mr. John Vanthof: Thank you.

The Chair (Ms. Cheri DiNovo): Okay. We now move to the government side: Ms. Des Rosiers.

M^me Nathalie Des Rosiers: I think we may want to have you continue a little bit on explaining to us how Treasury Board supports evidence-based decision-making. We all want evidence-based decision-making, and I’m interested in hearing a little bit more about not only the behavioural unit, but maybe overall what’s the thinking about how you strengthen evidence-based decision-making within government.

1000

Hon. Liz Sandals: So a bit more about the overall, broader centre of excellence writ large? Deputy, did you have some remarks, or over to Didem?

Ms. Helen Angus: I can give a little bit of an overview about what Didem’s going to talk about.

Last week, you heard from Dr. House. As we mentioned this morning, he’s part of a team at the Centre of Excellence for Evidence-Based Decision Making. I think this is a good chance to talk a little bit more about how we set and uphold expectations around rigorous decision-making and the use of evidence in that way.

We have a clear and explicit focus at Treasury Board on the use of evidence. We have a number of tools, and Didem’s going to go into some of those in a little bit more detail. We actually have a framework for evidence-based decision-making. You heard a little bit from Gary last week about an enterprise risk management framework and also around IT governance. Those are mechanisms that we use, actually, to make sure that we’re using evidence and we’re being as disciplined as possible in the recommendations that we bring forward.

We also have a systematic and structured decision-making process. So we have tools and processes. The process that we’ve also talked about here over the last few hours is Program Review, Renewal and Transformation, short form PRRT, that really brings a strong foundation of evidence about government’s programs through key performance indicators and program inventories that we ask ministries to refresh every year.

We also help ministries build a better understanding about how their programs are performing by using evidence to inform choices, not only about policies but also about continuous improvement. Obviously, behavioural insights are a part of that, but the search for continuous improvement is, I think, ongoing and is a very big focus of the staff across Treasury Board.

We really support the government in its overall PRRT approach to look at progress on strategic objectives, looking at those initiatives not only within ministries, but horizontally across ministries, how they line up to produce outcomes and the value that they create. We’re a partner in monitoring ministries’ key performance indicators and how they’re actually progressing on their mandates.

We also work to pilot and test new approaches—Didem will talk a little bit about that—and then are partners in the scaling up of best practices, where evidence shows that they improve services and enhance outcomes.

Didem is going to talk a little bit more about the evidence-based decision-making framework that I referenced and how we’re building capacity not only within Treasury Board but across the ops, to be able to use evidence in better and more disciplined ways to improve programs. She’s going to elaborate a little bit more on the role of measurement and continuous improvement as well. It’s been said in many places: If you can’t measure it, you can’t manage it. We take that pretty seriously, so that’s a big part of the work that we do.

With that, Didem, I don’t know if you need to introduce yourself again, but there we go.

Ms. Didem Proulx: Thank you so much, Deputy. Thank you so much to the committee members, as well, for giving me the opportunity to speak a little bit about the exciting work that we’re doing at the centre and at TBS writ large to really focus on outcomes, client service delivery and supporting continuous improvement for all the services that are delivered across many ministries in government.

Evidence-based decision-making, as you know, is about the use of the best available analytics and evidence to identify what works and what may be lacking and could use some improvement, allowing decision-makers to take that information and use it to support their decisions and to kind of move the yardstick forward.

In a lot of ways, evidence-based decision-making is not new. I think all governments, going generations back, have wanted to improve things and wanted to use evidence that they heard. But what has changed in the last number of years, thanks largely to IT, is that governments have really rich administrative datasets now, and IT is allowing us to synthesize and analyze that data to bring forth new forms of evidence beyond the traditional forms like consultations or other means, which are just as equally valid in the policy and program development process, but just to supplement it with research and analytics. That’s the part that we are very excited about and focused on.

As the minister mentioned, in our RAD unit, we are very technically focused. That is the same love for analytics, evaluation and the rigour that we’d like to share with our colleagues and work with them to implement across the ministries.

We are not new in this. We are not the only player in this endeavour. Other governments around the world, like the UK, Australia, some US states—everybody calls it something slightly different, but are actually embarking on this journey. The work that TBS is doing is actually pretty cutting-edge and we are very proud of it.

The deputy mentioned in our mandate that we are a relatively new part of TBS. We were created about two and a half years ago. There are three components to our mandate: developing and implementing the OPS enterprise evidence-based decision-making framework and developing the tools to really support that; building capacity across the OPS; and also, supporting the decision-making process.

With your indulgence, I’ll take a moment to highlight the work in all of those regards.

The evidence-based decision-making framework was released in 2016 to set clear expectations and standards for evidence which should be used to guide the planning and administration of public services. I use that broadly because it’s meant to encompass policy and programs all together. Like the deputy mentioned, the framework builds on the expectations of other frameworks, like enterprise risk management and open government, which were discussed last week.

I call it the three pillars. Why this framework? It’s to really focus on strategic alignment and are the right things being done to meet the needs of the clients and Ontarians? Then to really measure the impact of those programs and to say, “Are the policies and programs achieving their intended outcomes and impacts?” If we are delivering one program, it clearly is another form of that program we’re not delivering, so how do we do the cost-benefit analysis of that? How do we support continuous improvement, very much to focus on outcomes and continuous improvement? The third pillar, which I think is the most important, is always learning. How can we be doing better? Having had the opportunity to serve in both policy roles and in operational roles, on-the-ground reality is an incredibly valuable perspective in being able to refine the policies and programs of government. How do we encourage and foster that dialogue so on-the-ground experience can be reflected to make the programs better over time?

The framework is intended to bring evidence to bear at every stage of the policy and program development process, starting with the “problem” definition—what is the problem that we are trying to solve—and to bring rigour to the identification and the magnitude of that problem.

The other thing that I think is really interesting and goes with the horizontal aspect of our discussions and the PRRT process is, how does this problem fit with the spectrum of the suite of programs that are currently funded by the government—provincially funded programs—and then putting the outcomes right at the beginning of the process? What are the outcomes we want to achieve? What does success look like?

That is an incredibly important aspect, because in the more traditional forms of policy-making, evaluation comes at the very end. Unless there is a very concerted effort to define the outcomes we want at the beginning and put the measurement places along the way, it is very difficult to do a thorough evaluation of a program at the very end. We are actually trying to put the focus on outcomes all the way through and embed pieces along the journey so that, after the program is implemented, we actually have a way of evaluating it and supporting continuous improvement and options assessment—the desire to give the decision-makers a set of viable options in terms of how to implement it—and really managing risk and performance, because implementation is very, very important.

I know that my colleague Gary has been talking about their risk management framework, so I won’t get into too much detail. But the idea is that the two should reinforce each other and that there are risks to policies and programs before implementation and post-implementation—and how do we manage that in an effective way using data and analytics and also, as I said, reflecting the on-the-ground reality to support continuous improvement? And very, very importantly, now that we have laid all the groundwork, how do we do the evaluation at the end: Is this program achieving its goals? If not, how should it be tweaked? Or should we be torqueing to an entirely different mode of delivery?

1010

So by striving for higher standards—

M^me Nathalie Des Rosiers: Can I just interrupt you—

Ms. Didem Proulx: Sure.

M^me Nathalie Des Rosiers: —and ask you to give an example, if you have one, of how you would have decided early on what the outcomes were that you wanted to achieve, the data that you needed to collect, the ongoing process, and then, if possible, whether that evaluation was conducted and if it was satisfied by the program?

Ms. Didem Proulx: We have been working collaboratively with—because we in Treasury Board don’t hold any programs of our own—line ministries on their policies and programs with the goal of increasing employment for people with disabilities. There’s a new strategy—

M^me Nathalie Des Rosiers: That’s interesting. I was just thinking about that, so I’m glad that this is the one you chose.

We want to have more people with disabilities being able to find fulfilling work. So how do we go about thinking about this program?

Ms. Didem Proulx: Absolutely. The outcome statement is easy enough, but there are all sorts of definitions around, is it just any employment or is it sustainable full-time/part-time employment? How do we tackle issues around how we define disability—because it varies quite significantly with respect to its impact on somebody’s capacity to work. One can come up with a perfect evaluation plan, but if that perfect data doesn’t exist—how do you use proxies? We have worked with ADO to come up with what the existing datasets are, what the other pieces of information are that we need to collect as we roll out these initiatives.

Similarly, we have been working with a lot of different ministries as they embark on journeys—and the learning ministries that are data-rich. What are some of the gaps? We have done some work with—it was municipal affairs and housing at the time, but as they were rolling out the housing strategy. How do we measure need for some of the things that are not quantified now?

The evaluation frameworks are something that we actively work with ministries to support. The framework is great, but of course OPS is big and there are lots of programs. So how do you engage people in line ministries? We had been focused extensively on capacity-building and training. I’m very proud to say that, be it a focus on program evaluation or data analytics or behavioural insights, we have reached approximately 10,000 people since the centre was created and the framework was released in 2016. We’re a small group, and we can’t do this work for all files, so we need to engage and energize people in the line ministries to be the flag-bearers.

Last but not least in the decision-making process: The deputy mentioned PRRT is so important, because that is where we embed the evidence-based decision-making principles in terms of the submissions we require from the ministries. We are building a detailed inventory of government programs and identifying—speaking of strategic alignment—ministry-level strategic KPIs and metrics at the program level, so that we can create the full cascade that will make continuous improvement possible for all programs.

M^me Nathalie Des Rosiers: Thank you.

The Chair (Ms. Cheri DiNovo): Thank you all. This committee stands recessed until 3:45.

The committee recessed from 1015 to 1557.

The Chair (Ms. Cheri DiNovo): Good afternoon. We are now going to resume consideration of vote 3401 of the estimates of the Treasury Board Secretariat. There is a total of 43 minutes remaining. When the committee recessed this morning, the government caucus had five minutes left in their round of questions.

Are there any questions?

Hon. Liz Sandals: If I may, Chair, I think we were partway through—MPP Des Rosiers had asked about some other examples, and we’re prepared to carry on with just finishing with the examples, if we may.

The Chair (Ms. Cheri DiNovo): That sounds good. The floor is yours. Could you introduce yourself again?

Ms. Didem Proulx: Certainly. Didem Proulx. I’m the director of the Centre of Excellence for Evidence-Based Decision Making Support at the Treasury Board Secretariat.

I would like to talk about a project that we have done with the Ministry of Advanced Education and Skills Development and on the Youth Job Connection project.

Of course, MAESD has a full suite of programs around training, apprenticeship and creating employment opportunities for adults and youth. But, as we analyzed the unemployment data, it was very clear that youth facing multiple barriers—be it due to a state of homelessness or background or learning disabilities—were employed at a significantly lower rate.

So the government introduced the Youth Job Connection program and, as part of the design of the program, because there was uncertainty as to the exact mix of clients that would be served and what would be the most effective programs to deliver for them, rather than making assumptions about what would be effective, the program was originally designed to be flexible, where different clients could take supports on the basis of their need. Then we would track them at three months, six months, 12 months, 18 months and 30 months to assess both the outcomes for the individual and, in turn, using that to assess the effectiveness of programs over time, examining what level of intensity delivers the best results for the clients or what sort of group of programs are yielding the best results for different client groups that are served.

Over time, as we observe what the most effective interventions and supports are, those are the programs that are going to be funded on a go-forward basis. Because we are actually tracking the clients who are served through the program, we can also answer questions around the results for these groups of students or youth who are part of the Youth Job Connection program, how those outcomes compare to the other programs that we offer, because if the outcomes are more or less the same, it’s okay to have one big program, whereas—does a differentiated level of support and differentiated programs make a difference? The program was designed very much keeping the principles of flexible program design, constant evaluation and continuous improvement in program design to make it as effective as it possibly can be.

If I may also go back to one question that was raised earlier today, because it’s applicable to all the work that we do, to Mr. Vanthof’s question this morning about the ethics of the projects and how we select them, I should have mentioned this in the morning, but we do abide by the ethics frameworks that are usually used in research institutions. As you know, as part of the team we have people who have experience, researchers, and we make an effort. We are not always up to date but we eventually catch up to actually document all the files and all the cases that we have worked on on our website, both as a measure of—like letting it be a reference document to OPSers, as well as being a measure of public transparency, because the work that the behavioural insights unit has been doing has been getting a lot of attention. It’s even published in some journals and outside publications.

As part of that, it gets reviewed quite considerably by both our peers within the government and beyond the borders of the Ontario public service, so we make sure, both by the principle of doing it as well as its publication, that we hold ourselves to the highest ethical standards, both with respect to the projects and how they are conducted, how the results are analyzed and all of that information. I just wanted to loop that back and to provide that reassurance to you.

The Chair (Ms. Cheri DiNovo): You have just under a minute left, if you want to wrap up.

Ms. Didem Proulx: I just wanted to say thank you for the opportunity again. We work on a large number of projects which I clearly won’t have the opportunity to highlight, but as I said, on our website we have lots of case studies, and if there are any further questions, we would be more than happy to answer.

The Chair (Ms. Cheri DiNovo): Thank you. We’ll now move to the official opposition. You have 12 minutes and 40 seconds.

Ms. Lisa MacLeod: Thanks very much, Chair. Thanks very much. It’s been a slice. We had a few more minutes left to go.

My colleague Sylvia Jones, the member from Dufferin–Caledon, today talked about the Carrot Rewards wellness app. It’s a $1.5-million smart phone app used by a couple of other provinces—British Columbia, Newfoundland and Labrador—and, of course, the federal government. It’s got about 200,000 active users in the province, but the data was actually just breached with Equifax. I’m wondering if we could have a conversation about this because last week we talked about safety and security.

You talked about how we keep our data safe in the province of Ontario and that we had ethical hackers, that sort of thing. I’m wondering, with respect to this particular breach, how the OPS handled that.

Hon. Liz Sandals: I think the first question would be determining whether or not that’s an app that we’re actually involved in.

Deputy, can you—

Ms. Helen Angus: Yes. I’m familiar with the app. I don’t know what our relationship is to it. Maybe, Mohammad, do you—

Mr. Mohammad Qureshi: I don’t know enough about the app.

The Chair (Ms. Cheri DiNovo): Could you introduce yourself—

Mr. Mohammad Qureshi: Sorry. Mohammad Qureshi, head of cybersecurity with Treasury Board Secretariat. I don’t know enough about the app, but I do know that it was a rewards app for healthier living and it was tied back to Trillium, which is a separate Ministry of Health sort of agency.

Ms. Helen Angus: Is it the Trillium Foundation or the Trillium—

Mr. Mohammad Qureshi: That I would have to go back and—

Ms. Lisa MacLeod: So if you could get me more information on that. I don’t want to put you on the spot, if you don’t have it.

Mr. Mohammad Qureshi: Sure.

Ms. Lisa MacLeod: My understanding from it—and I’m just learning a little bit more about it, as my colleague raised it earlier today—is that this was part of the Ministry of Sport, with the intention to have people eating better, but that you would still, as a result of looking after the security of Ontarians’ data when it comes to your apps or your services—that that would be part of it.

Hon. Liz Sandals: I think that all we can say at this point is that this is obviously not a project on which Treasury Board has had any carriage. So we really have no way of commenting on the questions that you’re asking.

Ms. Lisa MacLeod: Okay. Can we talk a little more about the advertising again, if you don’t mind? I know that the Auditor General has been critical of some of the ads, saying that they have been—I guess one of the terms was “patting on the back.”

The other thing I think she has been critical of is—you referenced this morning that there is a protocol or a law in place for her to reject ads. My understanding is that she’s not happy with some of the changes that have impeded her ability to do that, and that that was changed.

I’m just wondering, with respect to those significant changes to the Government Advertising Act, and the auditor calling for more powers, or to have her powers restored, if the minister could comment on that.

Hon. Liz Sandals: First of all, what I would say is that we continue to have the only government advertising legislation, not just amongst the provinces and territories but literally in the country. So we actually do have quite strong rules in terms of making sure that the ads have reached approval.

As I think I alluded to this morning, one of the issues that we had run into with the earlier version of the legislation was that it didn’t take the campaign to the auditor until the production had been done. In a case where the auditor rejected the campaign or asked for changes in the campaign, it became quite expensive and/or it was just simply money down the drain.

Ms. Lisa MacLeod: So it was too expensive to go through the—

Hon. Liz Sandals: That’s why we changed the protocol, so that there are two check-ins with the auditor instead of one. There would be one check-in with the auditor with the preliminary plan for the campaign, the preliminary content, so that if she is going to veto the ad or ask for a major change in the direction that the campaign is taking, she can do it at an early stage. But there is a second check-in with the auditor when there is a finished product.

So, in fact, with the new legislation, there are two check-ins with the auditor and in a way which doesn’t have the spending of public monies on something that is ultimately going to be rejected.

Ms. Lisa MacLeod: So this was about saving money, then.

Hon. Liz Sandals: No. It’s certainly part of the process, but it was about getting a preliminary opinion from the auditor before people had done all the work and engaged various firms to do the detailed creative work.

I remember the one ad, the Foodland Ontario ad. As I mentioned, Foodland Ontario advertising is not new. Foodland Ontario is an advertising campaign that has been going on for years.

Mr. John Vanthof: Big fan.

Hon. Liz Sandals: John, as a farmer, would recognize that Foodland Ontario had been going on for years. John might remember the rather odd Foodland Ontario—I think it was local farmers’ markets that were being promoted, and it showed up in black and white. The reason it showed up in black and white was because the auditor had ruled that the tomatoes were too red, the strawberries were too red and the apples were too red. Given that at that point the content had already been produced, which was shots of fruit and veggie stands in farmers’ markets, the only alternative was to run the ad in black and white. That’s why you—

Ms. Lisa MacLeod: Thanks for that. I appreciate it. I’ve got a couple of more questions, with about five minutes left. Thanks very much.

1610

Mr. Mike Colle: They should have used green tomatoes.

Ms. Lisa MacLeod: Obnoxious.

Hon. Liz Sandals: By the time they get to the farmers’ market, you hope they’re red and ripe.

Ms. Lisa MacLeod: “Government-Friendly Advertising by Crown Corporations”—she referenced that report on page 759. In the $56 million—

Hon. Liz Sandals: You’re not reading from the estimates.

Ms. Lisa MacLeod: No. Don’t worry about it. I was reading from the auditor’s report. I want to go back to it because she talked about crown corporations and their advertising. I’m wondering, is the crown corporation advertising part of the $56 million in the estimates this year?

Ms. Helen Angus: Some of the agencies are not included. I think we’ve talked about Foodland Ontario. LCBO and Metrolinx would be examples that would be outside of the bulk media buy. So I don’t know if there’s—

Hon. Liz Sandals: Is your question about whether they’re outside of the bulk media buy?

Ms. Lisa MacLeod: Yes.

Hon. Liz Sandals: Okay. Yes.

Ms. Lisa MacLeod: Do you have a sense of how much Metrolinx, Foodland Ontario and LCBO spend on government advertising?

Hon. Liz Sandals: We would not have that because it’s not in the Treasury Board books. Because it’s not in the Treasury Board books, we would not be getting the individual campaign invoices. It ends up as part of the—

Ms. Lisa MacLeod: Public accounts?

Hon. Liz Sandals: —public accounts, however the ministry shows it or the agency shows it would end up.

Mel, do you want to talk about how that works a little bit more?

Ms. Melanie Fraser: Sure.

Melanie Fraser. I am the chief administrative officer and assistant deputy minister with corporate services at Treasury Board Secretariat.

The minister is correct to say that the agencies are not funded through the bulk media buy. Ontario Lottery and Gaming Corp, Liquor Control Board of Ontario, Metrolinx—I think those are some of the publicly largest examples where you might see advertising. That wouldn’t be funded from the bulk media buy.

Where you would see spending would be for agencies where their accounts are consolidated onto the government books. But again, we don’t have a scenario in Treasury Board where we would be consolidating the spend for any agencies onto our books, so we couldn’t speak to any advertising.

Ms. Lisa MacLeod: We talked over the past couple of days about some digital advertising. The new media buy, the $56.6 million, does that now include social media websites like Facebook, Twitter, Instagram, Google AdWords, all that type of thing?

Ms. Helen Angus: Yes. First, the Auditor General does have oversight over digital advertising, so her powers now cover more media than they did before.

Digital advertising is defined in that act, in the regulation, as paid advertising—I’ll just read this to you—“consisting of video, text, images or any combination of these ... displayed on a website, other than a social media....” So some of the faster-moving social media—Twitter—is not covered by this, for example.

Ms. Lisa MacLeod: The $56 million would include that type of advertising, though?

Hon. Liz Sandals: The digital.

Ms. Helen Angus: Yes.

Ms. Melanie Fraser: It would include creative services, production costs, paid search—that’s Google, Bing, those sorts of things. And social media costs would qualify as expenditures under the bulk media buy.

Ms. Lisa MacLeod: Chair, I’m finished. I want to say thank you to the minister, all of her staff—from the bureaucracy, as well as political—who spent the last three days, for what seemed like an eternity, with us here in this committee. It was very informative.

I very much appreciated the insight into your department. So thank you.

Ms. Helen Angus: Thank you.

The Chair (Ms. Cheri DiNovo): We now move to the third party. Mr. Vanthof, you have 12 minutes and 40 seconds.

Mr. John Vanthof: I’m just going to go on on advertising. As I stated before, many parts of the province aren’t really well covered by digital advertising. Is there any way that we could look at if the amount spent has changed? I don’t expect you to have that answer today. But what has traditionally been spent on newspapers? I would assume that the amount spent on digital advertising would be more because, obviously, for a huge segment of the population, that’s their main source of information. Is there any way that we could come up with a comparison of how those figures have changed in perhaps different parts of the province? Is that possible?

Hon. Liz Sandals: I think all we could do is attempt to see whether or not that information is available. Because what we get at Treasury Board are invoices, the invoice wouldn’t necessarily show what the markets were. If I think of radio ads, because there are obviously radio stations all over the province, we might be able to say there was an invoice for radio ads, but I don’t think we would have the detail of information that would let us determine what station is located where, if I may.

Mr. John Vanthof: Fair enough.

Hon. Liz Sandals: This is me speaking as a former Minister of Education. When we were dealing with child care, and we wanted to get the word out to parents that there’s a difference between licensed and unlicensed home care, if you’re going to look at child care, there now is a government website where you can go and check to see if there are any complaints about the particular provider.

We made a conscious decision, when we were looking at that campaign, that we would just simply do digital, which obviously is the issue that you’re identifying. We made that decision because we thought, where are we most likely to reach the people we want to reach, who are young parents who are looking for child care? You’re probably going to meet them online, because they’re probably online searching for child care. So in that particular case, the entire focus of the campaign was a digital campaign, because we anticipated that that was where we would find the most young parents.

Mr. John Vanthof: Okay. I think that’s very reasonable. I’m not trying to reinvent the wheel and go back 100 years.

Hon. Liz Sandals: But I do understand what you’re saying about northern Ontario. Whether or not you can get it depends on whether you’re at the top of the hill or the bottom of the valley, and who decided to put a tower where.

Mr. John Vanthof: If we start on education—obviously, I like to tell stories—I’ve got a community in my riding called River Valley, and they have a school. The town does everything they can to keep that school. They have smart boards provided by the school board but dial-up Internet connection that makes—they might as well have blackboards. Those are the issues.

Every year, I go to that community, to their Christmas dinner. They raise money to keep that school open. I was a bit early, so I stopped at the local general store, and they have no cellphone service there either.

I talked to the lady behind the counter. I said, “You don’t have much cell.” She said, “Oh, no, no. If you go back three kilometres, on the hill, on Drew Road, you have service on the railroad track.” Then I said, “What about your land lines?” She said, “The land lines are fine.” I said, “Well, that’s funny. I was in St. Charles last week at the Christmas parade, and their phones don’t work when it rains.” The lady at the counter in the general store said, “Sir, everyone knows that phones don’t work when it rains.”

That’s Ontario. That’s our Ontario, and that’s six hours from here. We’re not talking inexplicable distances, and when people say that there is no rural-urban divide, just remember that. Everyone knows that phones don’t work when it rains.

1620

Hon. Liz Sandals: And you don’t have to go six hours away. I actually had one at the cottage for years, where my line didn’t work when it rained.

Mr. John Vanthof: Yes. But this is a whole community that has smart boards and dial-up Internet with phones that don’t work.

Hon. Liz Sandals: Can I just make a comment? Going back to putting my Treasury Board hat on instead of my former Minister of Education or cottager hat, to be serious about the issue that you’re talking about, there actually was funding in this year’s PRRT to work on digital access for all the school boards in Ontario.

Mr. John Vanthof: They’re working on it.

Hon. Liz Sandals: They’re working on it. Is that something—

Mr. David Nicholl: Yes, absolutely.

Ms. Helen Angus: We just had a meeting on it yesterday where we talked about broadband.

Hon. Liz Sandals: So let’s get—

Ms. Helen Angus: So let’s get David up here.

Mr. David Nicholl: David Nicholl, corporate CIO, Treasury Board Secretariat.

Yes, it’s a program that is out of the Ministry of Education: to actually obtain at least a megabit of data for every student in school. That is what the program is really destined to do. It’s a multi-phased approach. They chose, in phase one, 170, 180 schools spread around the province. The real problem was the fact that schools were all attached to their school board, and the school board was then attached to the Internet. There was dial-up between the school and the school board, and the school board had fibre typically back into the Internet. What they’ve done now is they’ve put technology in to allow the schools to actually get direct connect through an ISP, a local ISP, to get Internet service into the schools.

It’s a long-term strategy. It’s going to take us a while to get all schools covered, but the Ministry of Education has a very focused outcome target on the one megabit per second per student. It’s really because of the shift in the way kids are learning. It used to be, in the old days, it was for the teachers, for the administration of the school. It’s not that any longer. The kids are bringing their own stuff into schools. They want to learn online. Of course, the next challenge is going to be how to get that into the homes as well. But, yes, it’s a great program.

Mr. John Vanthof: I was talking to the director of education from that school board and they are working with that program. They are not there yet.

Mr. David Nicholl: It’s a really good example of where the school board association and the Ministry of Education are working together with the schools to try to get this solved, because it is a real problem. It’s a great program, though.

Mr. John Vanthof: On the I&IT—and I’m the last person who should be talking about I&IT—in our last session, we were talking about the greatest risk being human error. Are there other risks that you’ve identified that we have to be cognizant of in the future?

Mr. David Nicholl: As far as cyber is concerned?

Mr. John Vanthof: Yes.

Mr. David Nicholl: Yes, there are a lot of risks. I think the ability for bad people to get through to all of us through any kind of spamming attempt to your email is still probably the highest risk—not even so much at work, but actually at home. Typically, most bad things happen, actually, on your home PC, and then you quite often will transmit that. So I think spam is still a huge, huge issue. Recognizing when an email is not what it seems to be is still the number one thing we want to teach people, because that’s typically where Mohammad’s ethical hackers—they will go after you to see if they can actually encourage you to respond to an email when you shouldn’t respond to it. But absolutely—

Mr. John Vanthof: Warn me first.

Mr. David Nicholl: No, no, no. We don’t warn you first. That’s the whole point: We don’t warn you first.

But of course, there are all kinds of other, more technology-related threats, where Mohammad deals with millions and millions of attempted intrusions every day in our networks. A lot of those are not targeted to us; they’re just a general cast-out to see who they can get hold of and to see what they can learn. I think we gave the stats: We’re looking at hundreds of true-blue attempts to get into the Ontario government on an annual basis, which Mohammad’s team, through his 7/24 CSOC operation—his security operation centre—actually are rebuffing.

It’s a war. We’re in a war right now, and everybody’s in that war.

Mr. John Vanthof: I would like to thank the minister and I’d like to thank the staff. I’ve learned a lot. I have a bigger appreciation for the TBS than when I started the process.

The Chair (Ms. Cheri DiNovo): We now move to the government side. You have 12 minutes and 40 seconds, Mr. Potts.

Mr. Arthur Potts: Maybe Mr. Nicholl does want to run away, but we’ll see; maybe the deputy minister can manage this.

I have two areas I want to go into in the 12 minutes left to us. The first area goes back to the “I” in the IT sector. What strikes me is how many servers we manage in the province of Ontario. You must have an incredibly dedicated team. When you look at the data that’s on there—we’ve talked a bit about security issues, but the data that goes on a driver’s licence, and all the health data: There are a lot of privacy concerns. You must have an incredibly dedicated team of people working on it.

I thought maybe you could talk a bit about the strategy of how we manage all these servers, and maybe some of the extraordinary individuals who have worked in the past with the ministry to put this together.

Hon. Liz Sandals: Dave no longer needs any introduction.

Mr. David Nicholl: I just want to take an opportunity, if I could, with your indulgence: We lost our first corporate CIO, actually, on Friday: Scott Campbell. Scott had been a long-long-term OPSer. He retired in 2001.

Scott was very much the architect in 1998 of, frankly, the strategy that we’ve been following over the last almost 20 years. He was visionary. He was aspirational. He set enormous targets for both the OPS and the I&IT organization. He invented clusters. He created shared services—something that the federal government is still trying to sort out.

I just want to take a second to recognize Scott Campbell’s contribution. It is truly enormous, and we’re going to miss him. Even though he’d retired in 2001, he was on the board of Cancer Care Ontario. He was on many boards. He was still involved. He came in to see me at least three or four times a year. He always gave me lots of advice, and never stopped doing that. I just wanted to take an opportunity—thank you for giving me that shot at recognizing Scott.

Referring to that, we really have been guided since 1998 by a series of strategies with different focuses. Going back to that 1998 strategy, it was very much about, how do we consolidate so we can manage the complexity that you were referring to? When you spread yourself thin and you’re trying to do that in 10 different places, obviously the opportunities for not getting efficiencies, or actually opening yourself up to risks, become much greater. Certainly we’ve found since 1998 that by focusing on our consolidation efforts, we’ve made ourselves a lot more efficient.

We’ve talked a little bit about some of the savings we’ve had. The savings have really been about focusing on transferring that cost of commodity-type services into true business solutions, so really helping to deliver better services for Ontarians and the business of Ontario, as opposed to running lots of servers and running lots of boxes.

In fact, to your point specifically around servers, our whole point is to reduce that number dramatically—year in, year out—to get ourselves down to the point where we have fewer servers to manage.

Mr. Arthur Potts: Have you set targets?

Mr. David Nicholl: We’ve set targets right back to eOntario, when we started in 2006. If my memory serves me, I think we were at probably around 6,000 physical servers at that time. I think we’re at well down below 2,000 now. Technology has helped a lot for us to be able to do that.

Also, we’re now taking much more advantage of opportunities with the cloud, for instance, looking at things like Microsoft 365 for Office and email, where it will all be run in Microsoft Cloud, as opposed to actually running it dedicated within our data centre.

Looking at things for maximum efficiency and scaling up, things like the cloud are really going to allow us to take better advantage of some of the solutions that are available and to focus less on running infrastructure and wires. I think that’s a really important point.

Looking ahead, clearly we’re very focused on digital services: How can we further the supply of digital services to Ontarians and businesses—how to make it simpler and easier and more straightforward for people to actually transact with government? That’s what digital really is all about.

1630

We’re also very focused on—I think we’ve talked about it before, but we’re sitting on this incredible pool of data, a rich pool of data. Allowing people in our ministries as well as in our university communities and in our research communities to start taking a lot more advantage of that data in a very safe and a very privacy-by-design method forms a key part of what we’re focused on, from a strategic perspective.

Our other real leg that we’re focused on is around, how do we encourage people to do things in a more innovative or a different kind of way by demonstrating the power of technology to them? It’s bringing ideas forward. It’s allowing more experimentation. It’s allowing people safe places where we’re not concerned about breaches, and where we can secure things in a much more sandbox kind of way and allow people to experiment in technology and how it could allow them to do things very, very differently.

Mr. Arthur Potts: I appreciate that very much, particularly your very kind words for Scott, who was obviously a very close friend of yours. I’m delighted for his service, and we’re delighted for the fact that you had a chance to memorialize him here. I’m sorry for your loss and his family’s loss.

Mr. David Nicholl: Thank you.

Mr. Arthur Potts: The next piece I wanted to go into is a little different from IT, but it speaks to the balanced budget process. Part of the work that has been done in IT, obviously, has helped us get there with the savings we’ve seen across the board. More specifically, that was a key election promise, and it has been a key deliverable.

I’ve noticed that, over the years, with the projections of our deficits, we keep coming in well under them, year in and year out, and it has brought us to the place where we’re going to be by the end of this year in a balanced place. We’re allowed to invest a lot of those monies into new programs: pharma plus, free tuition and all these other areas—great programs that help. Hopefully, there will be more money and we can get into basic dental on OHIP and other fantastic, progressive measures.

Specifically, we’re projecting that we’re going to be balanced for at least the next three years. How are we doing that? What are the measures we’re taking in Treasury Board to ensure that? We flatlined all salaries over the last years. Are we continuing that process, growing the economy, investing in the economy and flatlining, or are we actually starting to invest that money back into programs and services?

Hon. Liz Sandals: We may want Karen to come up for a few minutes to talk about the PRRT process.

More globally, I think your observation is correct. As we have gotten closer to balance, it does give us the opportunity to be able to start to reinvest. The one that I think is really exciting, other than OSAP, which we’ve talked about before as an example of transformation, is that if you look at OHIP+, which is the investment in pharmacare for people under 25, that’s just plain a new investment which, by careful management of money, we’ve been able to achieve.

In some ways, there really isn’t any magic formula. It has been the great work that my staff do, working with ministries on their annual allocation. Certainly, what we do, and are in the process of doing already for next year, is that each ministry gets an annual allocation, and then the staff at Treasury Board work with each ministry to think about what decisions you need to make to ensure that each of the ministries is working within their allocation.

I think it has been, in part, just simply the constant attention to the detail, to make sure that people are presenting budget plans that fit within their allocations for the year. Then the hard work that goes on at Treasury Board over the course of the year is trying to ensure that people live within their allocations once they’ve received them.

As we’ve gotten into in a few areas here, just because it says “exactly this” in the budget, it isn’t necessarily exactly what happens. There are things that happen during the year. Certainly, climate change leads us to more flooding events. Those aren’t budgeted for, particularly. With MNR, firefighting is something that is quite variable from year to year, depending on what the summer weather happened to be like, particularly in northern Ontario. That varies from year to year.

So some of these things we anticipate will come from a contingency fund. We do set up a contingency fund each year at the beginning of the year so that we do have that space for either urgent events like climate-related events or where things happen in a year where you actually do need to adjust the allocations that a ministry has. That’s also part of our job at Treasury Board, that ministries do come forward during the course of the year and ask for more money. But if they’re going to get more money then, again, my analysts are going to look at that very carefully and make sure that there’s a very good business case for drawing on that contingency fund.

Mr. Arthur Potts: How much time do we have?

The Chair (Ms. Cheri DiNovo): You have two minutes left.

Mr. Arthur Potts: Great. Part of the conversation is about us exceeding targets, and by holding levels of expenses and getting more revenue in as a result of GDP growth—which we’re leading in North America—getting those additional revenue sources while we hold the line on expenses, that’s what’s allowed us to do it without slashing expenses. We’re very appreciative of that.

Will we be at a point where we would be in surpluses and using those surpluses to pay down debt, or to pay down, particularly, stranded debt? Maybe people would appreciate the distinction between our debt, which is backed by assets, and that which is stranded, which is not backed by assets.

Ms. Helen Angus: I think, to your point, to get to a balanced budget and then exceed it requires both the Treasury Board to do the job that the minister just described, but also our colleagues at the Ministry of Finance. It would be remiss if we didn’t give them some credit for the work that they’ve done, as you suggest, on the revenue side, but also on interest on debt, where they’ve exceeded their targets year over year as well. I think the Ontario Financing Authority—I don’t want to speak for them, but they certainly have done an excellent job managing the provincial borrowing program, and have been a contributor to the balanced budget as well. We work very closely with our colleagues at the Ministry of Finance throughout the year—

Mr. Arthur Potts: And they’ve taken a long-term perspective where they are securing great rates into the long term so we have predictability. So for crazy events, we’re insulated from that.

Ms. Helen Angus: Yes.

Hon. Liz Sandals: I think that’s perhaps one of the learnings coming out of the recession—again, we’re bleeding into the area that is Finance—that it isn’t just expenditure planning. It’s also debt planning and the management of that debt so that you actually have done the borrowing at a reasonable cost when the opportunity is available.

The Chair (Ms. Cheri DiNovo): I’m afraid that concludes the government’s time.

This concludes the committee’s consideration of the estimates of the Treasury Board Secretariat. Standing order 66(b) requires that the Chair put, without further amendment or debate, every question necessary to dispose of the estimates. Are the members ready to vote?

Mr. Arthur Potts: No, actually. I’d like to ask for a 20-minute recess, if I could, before we vote. Also, I’d like to make sure we record all the votes.

The Chair (Ms. Cheri DiNovo): Okay. We will recess for 20 minutes and we’ll be back here at 5 o’clock.

The committee recessed from 1640 to 1700.

The Chair (Ms. Cheri DiNovo): Okay. We are going to proceed right to the vote. Are the members ready?

Mr. Arthur Potts: Recorded.

The Chair (Ms. Cheri DiNovo): Of course, it will be recorded, as requested.

First of all, shall vote 3401, Ministry Administration Program, carry?

Ayes

Colle, Hoggarth, Kiwala, Potts, Rinaldi.

Nays

Harris, MacLeod, Vanthof.

The Chair (Ms. Cheri DiNovo): Carried.

Shall vote 3402, labour relations and compensation, carry?

Ms. Lisa MacLeod: Same vote.

The Chair (Ms. Cheri DiNovo): Okay. Carried.

Interjection.

The Chair (Ms. Cheri DiNovo): Actually, you asked for a recorded vote, so we should actually—

Mr. Arthur Potts: Same vote—same recorded vote.

The Chair (Ms. Cheri DiNovo): Can we do that?

Ms. Lisa MacLeod: House rules.

The Chair (Ms. Cheri DiNovo): We think, so there we go.

Mr. Michael Harris: How many are there?

The Chair (Ms. Cheri DiNovo): We have 12.

Interjections: Same vote.

The Chair (Ms. Cheri DiNovo): Okay. Let’s ask this: Does the committee agree that we’ll do same vote?

Interjections: Agreed.

The Chair (Ms. Cheri DiNovo): Okay, and recorded as such.

Shall vote 3402 carry? Carried.

Shall vote 3403, Employee and Pensioner Benefits (Employer Share) Program, carry? Carried.

Shall vote 3404, Treasury Board Support Program, carry? Carried.

Shall vote 3405, Governance, Agency Oversight and Centre for Leadership and Learning Program, carry? Carried.

Shall vote 3406, Audit Program, carry? Carried.

Shall vote 3408, Enterprise Information Technology Services Program, carry? Carried.

Shall vote 3409, Central Agencies Cluster Program, carry? Carried.

Shall vote 3410, Agencies, Boards and Commissions Program, carry? Carried.

Shall vote 3411, Bulk Media Buy Program, carry? Carried.

Shall the 2017-18 estimates of the Treasury Board Secretariat carry? Carried.

Shall I report the 2017-18 estimates of the Treasury Board Secretariat to the House? Carried.

Having carried all of those votes, I will report the estimates of the Treasury Board Secretariat to the House.

I want to thank the minister, all of those from the ministry and all members for attending.

We are adjourned until tomorrow at 3:45, at which point the Ministry of Energy will be before us.

The committee adjourned at 1703.

STANDING COMMITTEE ON ESTIMATES

Chair / Présidente

Ms. Cheri DiNovo (Parkdale–High Park ND)

Vice-Chair / Vice-Président

Mr. Michael Mantha (Algoma–Manitoulin ND)

Mr. Mike Colle (Eglinton–Lawrence L)

M^me Nathalie Des Rosiers (Ottawa–Vanier L)

Ms. Cheri DiNovo (Parkdale–High Park ND)

Mr. Michael Harris (Kitchener–Conestoga PC)

Ms. Ann Hoggarth (Barrie L)

Ms. Sophie Kiwala (Kingston and the Islands / Kingston et les Îles L)

Mr. Michael Mantha (Algoma–Manitoulin ND)

Mr. Arthur Potts (Beaches–East York L)

Mr. Todd Smith (Prince Edward–Hastings PC)

Substitutions / Membres remplaçants

Ms. Lisa MacLeod (Nepean–Carleton PC)

Mr. Lou Rinaldi (Northumberland–Quinte West L)

Mr. John Vanthof (Timiskaming–Cochrane ND)

Clerk / Greffier

Mr. Eric Rennie

Staff / Personnel

Mr. Nick Ruderman, research officer,
Research Services