STANDING COMMITTEE ON PUBLICÂ ACCOUNTS
COMITÉ PERMANENT DES COMPTESÂ PUBLICS
Thursday 19 April 2007 Jeudi 19 avril 2007
2005 ANNUAL REPORT,
AUDITOR GENERAL
MINISTRY OF CHILDREN
AND YOUTH SERVICES
| Â |
| Â | Â | ||||
The committee met at 0958 in committee room 1, following a closed session.
2005 ANNUAL REPORT,
AUDITOR GENERAL
MINISTRY OF CHILDREN
AND YOUTH SERVICES
Consideration of section 4.02, children's mental health services.
The Chair (Mr. Norman W. Sterling): Good morning. My name's Norm Sterling. I'm the Chair of the standing committee on public accounts. Thank you very much for coming, Deputy Minister Wright and your people with you. We appreciate it very much. We have the Deputy Auditor General with us today, Mr. Peall, who's filling in for our Auditor General, who is away for a couple of weeks.
You've handed out a statement, Ms. Wright. I presume that you will want some leading-off remarks, so I welcome you to give those at this time and then we'll ask some questions after.
Ms. Judith Wright: Thank you very much, Chair and members of the committee. It's a pleasure to be here on behalf of the Minister of Children and Youth Services to report back on the two outstanding sub-recommendations of the committee's report. Specifically, we are here to talk about what the ministry has done in terms of improving outcome measures, data reliability and information systems within the ministry.
I would like to begin, first of all, by apologizing for not having submitted information to you prior to this committee. I have actually been out of the country for a month and so we did not manage to do that. So I beg your forgiveness for that. We will ensure it doesn't happen again.
The Chair: As long as it wasn't south.
Ms. Wright: It wasn't.
We did report back to the committee, as you know, in February 2006, and we have provided written responses to all of the committee's recommendations and what we have done to address the issues raised by the auditor.
As I said, today we really do want to briefly outline in this statement what we have done for the two outstanding sub-recommendations.
I'd like to introduce the members of the ministry who are at the table with me today. To my immediate left is Alex Bezzina, who is the assistant deputy minister for the program management division for the Ministry of Children and Youth Services and the Ministry of Community and Social Services. Alex's division is responsible for the operational oversight of transfer payment agencies, including funding for child and youth mental health agencies.
In addition, we've invited Jeff Wright, who is no relation to me, who is the director of the research and outcome measurement branch in the ministry's strategic policy and planning division. Jeff's branch is accountable for developing and monitoring the outcome measures for all of the sectors within the children and youth ministry but also specifically for children's mental health.
As I did last time, I'd like to thank the committee once again for its commitment to following up on the auditor's recommendations and its own concerns. Your input and direction have greatly assisted us in moving forward on improvements to mental health services for children and youth in this province.
I would like to take the liberty of beginning my remarks by talking very briefly about the policy framework. I know we were not asked to report back on the policy framework as part of this, but we think it's an important step to provide a context for ensuring that we improve our outcome measurements.
As you know, we released this report in November 2006. It's entitled A Shared Responsibility. It is an important step in helping us and working with the sector to integrate services for Ontario's children and youth.
The overall purpose of this framework is to foster collaboration between and among everyone who shares responsibility for the healthy development of Ontario's children and youth. It provides a road map that will set out, in conjunction with the sector, the long-term direction for the delivery of child and youth mental health services and supports in this province.
I would be remiss if I did not acknowledge, with gratitude, the efforts of the many stakeholders who provided, and continue to provide, widespread cross-sectoral input on this policy framework. It was truly a joint effort with Children's Mental Health Ontario, clinical experts, agencies, families and youth across the province.
In implementing the policy framework now that we've released it, communities will work through a collaborative community planning process to map locally the available services against a continuum of services that have been set out in the framework. This will enable all service providers to consistently look at available services, service needs, service coordination and funding allocations.
The next step, once the community has mapped those services, will be to help identify where strategic realignment of services may need to take place and identify priority areas for funding, in order to meet the goals set out in the framework and better serve the children, youth and families experiencing mental health issues.
Specifically with reference to recommendation 2.1.2, the ministry was requested to report to the committee on its established outcome measures and the reliability of baseline data for both the newly funded programs and core services identified by its new policy framework.
When we last appeared before this committee in February, we had not yet finalized the policy framework. At that time, we were considering–and I think we talked about this at the committee—defining core services as a key component of the framework. Based on consultation with the sector and our own research, and as we have highlighted in the November 2006 written report to the committee, we've shifted from defining core services to looking at this continuum of needs-based services and supports. This approach, we believe, will enable communities to be flexible in determining the types of services that will meet the unique and changing needs of children, youth and families and their communities.
In outlining the ministry's activities in addressing the recommendations, I'd like to touch on our initiatives related to program evaluation, measurement instruments and data collection.
As we've noted before, the ministry undertook an evaluation of programs funded through the 2004 budget enhancement. This evaluation represents the first broad-based look at outcomes by the ministry in the child and youth mental health sector.
We have completed phase 1 of this comprehensive evaluation. Between April and September 2006, we monitored common outcomes from 79 services that had received this funding. The evaluation shows that there has been substantial improvement in coordination and collaboration within and across programs.
We have also begun to see favourable results in terms of access to services and improved outcomes to child and family functioning. For example, we know that 92% of children who were referred have commenced service and only 9% failed to complete service. We also now know that 75% of children and youth within this cohort that we studied showed functional improvement at service completion, of which 63% showed at least moderate improvement and 30% required no further services.
We are proceeding with phase 2 of the evaluation. The focus of phase 2 will be on evaluating an established evidence-based program that can be generalized across the sector as a best practice. The centre of excellence at CHEO, the Children's Hospital of Eastern Ontario, is leading the phase 2 evaluation and is currently finalizing the methodological design of the evaluation with a view to procuring an evaluation team within the next two months.
With respect to outcome measurement and its application, which we've discussed at length at this committee previously, we have continued to work with agencies to improve the utilization of the brief child and family phone interview, or BCFPI, and the child and adolescent functional assessment scale, or CAFAS. We are also continuing to work with agencies so that they have a greater capacity to use the data and the outcomes that we are getting from those two standardized instruments.
In 2000, Ontario introduced these two standardized instruments to monitor outcomes in the sector. We began working with service providers and other stakeholders in 2004 to disseminate the findings of these instruments and thereby expand and improve outcome measurement in Ontario.
Over the past three years, we increased training and communities of practice in the use of the two instruments.
We've adopted consistent monitoring and sector-wide dissemination of quarterly and annual reports based on data from the BCFPI and CAFAS.
We've incorporated data collection and reporting in our transfer payment contracts. I think it was one of the issues raised by the auditor.
We've collaborated with and supported the Ontario Psychological Association so it could showcase these instruments to other professionals and service providers.
In our written submission to the committee in November 2006, we identified 2005 data from CAFAS and BCFPI. Since then, we have seen a steady improvement in the use of these instruments.
We have recently received the final versions of both the BCFPI and CAFAS 2006 annual reports. These reports will now be shared with our regional offices and participating service providers. In addition, for the first time, each service provider will receive its own agency level results in order to benchmark its performance relative to regional and provincial comparators. This is a very significant step in the use of both the data and an ability to begin to compare outcomes. Where anomalies between agency results and system comparators exist, the ministry's regional offices will work with service providers to understand the anomalies and to work on taking corrective action.
To complement these activities that we're doing with the sector, we have within the ministry initiated a ministry-wide project to strengthen our capacity as well as transfer payment agencies' capacity to use data and information for decision-making and planning purposes.
Led by Jeff's branch—the research and outcome measurement branch—working groups have been established to address five areas that we as a ministry have identified as key to ensuring timely, reliable and safeguarded information.
These areas include training for service providers and ministry staff to ensure that we get better data and that they understand why this is important and how to use it; alignment of data collection with key policy and program questions so that we are clear on the year-over-year data that we think is important to collect to address policy and program issues; and enhancing our ability to access and link up the data sets in a secure and appropriate way.
1010
Finally, perhaps of most interest to this committee is that we will begin to examine the best way to address the issue of the unique identifier for children and youth in Ontario. At our previous appearance at this committee we talked at some length of the importance of having a unique identifier in order to be able to track and monitor the outcomes for kids in a more meaningful way. This project team will begin to do that, and we are committed to meeting the requirements of the Ontario privacy commissioner before we finalize our system of data linkage.
In relation to recommendation 2.1.7, we were requested to report back to you on an information systems update, and I'm pleased to do that. I think, as we've said before, more than 800 staff throughout the Ministry of Children and Youth Services and the Ministry of Community and Social Services use our service management information system, or SMIS. This data application manages child and youth mental health and other transfer payment contracts and licensing operations for both ministries.
We have made a number of improvements to address the auditor's concerns about the usability of SMIS data. Last fall, we improved the electronic upload of agency budget submissions and quarterly reports. Instead of having to manually enter the data we receive from agencies, we now automatically transfer the data that we receive from agencies directly into SMIS, thus decreasing the error rate.
To ensure that our staff can identify and promptly address data quality issues, we have held 42 regional and corporate sessions across the province to train staff on how to use SMIS more effectively and efficiently.
In January 2006, we established a cross-ministry user group comprised of our key SMIS users. They meet regularly to identify problems and solve those problems.
In addition, the ministry improved the reporting of SMIS data in two areas. In 2006-07, the ministry implemented a web-based business intelligence tool called Cognos ReportNet. This reporting tool provides users with access to the SMIS data that they need and ensures that business-critical reports are sent directly to users in a format and timing that they require. The ministry's Cognos ReportNet website now makes accessible more than 78 self-service reports and over 40 scheduled customized reports.
Effective March 2007, an update to the SMIS service model provides enhanced reporting of French language service designated agencies. This enhancement also includes an improved business process to update and maintain the designation information between the performance management branch and the French language services unit in corporate policy.
All of these data collection and management process changes will improve our ability to monitor transfer payment expenditures and help us ensure that agency funding is appropriate and based on meeting the needs of children and youth in their communities.
Before I conclude, I'd just like to flag a couple of the challenges that the ministry and the sector face in moving towards developing a robust evidenced-based culture around measuring outcomes.
This is a sector, as we've discussed at this table before, that historically has lacked appropriate system integration and outcome measures. Issues such as inconsistency of individual client diagnostic criteria and classifications—which vary across the province, across the country and across the world—make it difficult to produce reliable and comparable data. Professional judgment is important, but it does create challenges in accepting one standard measure for the assessment of a clinical order. As we've discussed here previously, the sector is made up of a variety and diverse set of agencies, from small to large, some of which have sophisticated information technologies and some of which are just beginning to have them.
I believe we are moving forward to address these challenges. In particular, the progress we've made in embedding BCFPI and CAFAS is a significant step forward. The release of the policy framework sets a long-term direction for change in child and youth mental health systems. Finally, we are moving to address what we think is the most significant issue of the unique identifier.
I would like to conclude by taking this opportunity once again to recognize the many dedicated individuals who work hard every day to support children and youth with mental health issues. We will continue to work across all regions, and with our many dedicated partners, to build an Ontario in which child and youth mental health is recognized as a key determinant of overall health and well-being.
The Chair: Thank you. Do we have some questions?
Mr. Ernie Hardeman (Oxford): I just have a couple. It's more about the generalities of how we provide the services and how we decide to deal with those people that we have the capacity to serve when the need arises, and those families who come in with children who need help, who are not considered critical—the budget is short, and so we put them on a waiting list. The waiting list never disappears, but the people on it do.
How do we deal with our success or failure with those who are, in essence, denied service because their need was assessed at the start not to be critical enough? Is there a measurement of that, to make sure that we're not leaving some behind?
Ms. Wright: I'll have Alex talk more specifically to the process of that. The system itself is built on clinical judgment around triaging. So you are correct, sir, that the most acute cases get serviced before the less acute cases, and that's a professional judgment that gets made at that local level.
One thing that the new instruments—not so new anymore—BCFPI and CAFAS, let us track is exactly the data I quoted in my remarks, which is, we have a better sense now of how long it's taking for different types of services to be accessed by children and youth, and that, in turn, will enable agencies to have a better tool, and a more robust tool, to actually help triage in the most effective way.
Alex, did you want to add anything?
Mr. Alex Bezzina: Just a few things, and thank you for this opportunity. The BCFPI and CAFAS tools—in particular, the BCFPI tool is a useful tool, as the deputy indicated, to look at wait times and outcomes, but it's also a very important tool at the agency level.
To begin to analyze those people who are waiting for longer periods of time, and who may not get the service in as timely a fashion as one would like, it allows them to look at commonalities amongst those people who are waiting and devise ways of reorganizing their resources so that they may be able to meet those needs in a more timely fashion.
For example, one agency that we are aware of has very recently taken a look at this very issue that you're describing—those families that are on the waiting list for longer periods of time—and have introduced a modality of treatment that is short-term and solution-focused, and it's evidence-based.
Evidence increasingly shows that when you're doing counselling with individuals and with families, it is in the first three or four sessions that you have the greatest uptake of clinical change. So they've introduced a program by which they have short interventions, six counselling sessions. They've determined, on their waiting list, those families that are willing and open to receiving this short-term, and in fact, we are seeing improvement in the lives of these children and families as a result of this intervention.
So it's a way that BCFPI data can be used by agencies to say, "Okay, we do have these people on wait-lists. How, then, do we use our resources in a bit of a different way to address these situations?"
Mr. Hardeman: Relating to the same thing, obviously, in rural and small-town Ontario, we have a challenge with having enough service providers. It's one thing to fund them, but it's another thing to have enough.
In my community, we have one very active Oxford Child and Youth Centre. They provide a lot of services to a lot of young people. The problem with the funding is that they get funded based on the area that they were serving when they started. They've gone into opportunities well beyond that jurisdiction, but their funds keep being stretched further and further.
Is there, within the ministry, a way of dealing with the expansion of service providers into areas that presently don't have the service so that they wouldn't have to do that using the dollars that they've been allocated in the past, so they can actually provide service somewhere else and get extra funding to do that?
1020
Mr. Bezzina: Just a couple of points on that one. It is true that there are parts of the province where organizations such as the one that you're describing have broadened their mandate in order to try to address service needs that up until now have not been addressed. The ability that we had a couple of years ago, when we provided some additional dollars to the mental health system, was to prioritize those areas where dollars were being stretched in order to meet areas where service was not provided. So three years ago, when we introduced additional dollars into children's mental health—and it was for new service—through the community planning tables, the priorities were established. So there was the ability for the service providers to say, "Look, we're getting a lot of referrals from these particular areas. We're not able to meet the demand," or, "We are trying to meet the demand, but our resources are being stretched to the limit." That was an opportunity, through community planning tables, for the ministry to identify priorities for funding.
We will be going through a similar exercise now, over the next several months, because through the most recent child and youth mental health funding announcement, an additional $4.5 million is being made available to address regional and community priorities. Again, we don't make those decisions from Queen's Park. We do work very closely at the regional level and with our community partners to say, "Where are we being stretched? What kinds of issues are we dealing with?" I think it's really important not to characterize child and youth mental health as a single type of service. There are many types of services within child and youth mental health: residential and non-residential, behavioural interventions, family-based counselling and individual counselling, case management. So what of those types of things do we need? We will be going through that community planning process again in that regard.
Mr. Hardeman: Finally, and obviously not saying that there are no pockets in the province that are short on services, is it fair to say, forgetting about just my riding but province-wide, that we're providing fair and equitable child and youth services across the province to all people?
Mr. Bezzina: The issue that you raised earlier, which has to do with the availability of professionals, is an important issue. There is that particular issue that we constantly struggle with, working with our partner agencies. We have increasingly looked at ways of ensuring that when we are allocating funding, we are not simply doing it on a population base. We have to look at a needs base; we have to look at underserviced areas; we have to look at areas where there is significant population growth. So we need to look at addressing those parts of the province, frankly speaking, where we have had traditional programs but population growth is significant—we have that issue—and then we have the issue that you are describing. So we're constantly monitoring both wait time data and other kinds of information that we get through our community consultation tables to identify areas of priority for funding. The demand is great.
Ms. Wright: Just to add two points to that, we have made an investment in telepsychiatry, which I think is a really significant contribution to being able to address the shortage of access to professionals in rural and northern communities, and we can talk more about that, but that has turned out, I think, to be a very significant step in moving in that direction.
Secondly, the next step in the framework, as I indicated, was to map community services. That actually will happen at the planning table that Alex mentioned as well, but that will enable the agencies also, within the framework of the continuum of services that should exist, to have a conversation at the local level about where the overlaps are, where we have too much money in acute and not enough in prevention, or too much in prevention and not enough in acute. I think just creating that forum will be an important step to have the community begin to address the very important issues you've raised.
Mr. Hardeman: When you get through with the benchmarking information, is it reasonable to assume that it will also tell you the areas where the service is not being provided as opposed to just areas and how well they're doing with the services they are providing?
Ms. Wright: The mapping process will say we aren't providing this service in this area and we need to realign resources or we need additional resources to identify that but, most importantly, I think it enables the community to say, "This is where we think the important services should be for our community."
Mr. Hardeman: Thank you.
The Chair: On page 9, you mentioned, "These reports will now be shared with our regional offices and participating service providers. In addition...." etc. Are those public reports?
Ms. Wright: Yes.
The Chair: So this is going to be a public process. I will be able to say that I'd like to know in my community—for instance, the people I represent—how well children's mental health services are being provided to my people, so that I can come back down here and ask the minister for help if I need help in a certain area. I would like them to be public, and I would like comparators to be public as well.
Ms. Wright: I answered too quickly, and I apologize, Chair. The BCFPI annual reports are public. I will ask Jeff to speak more specifically on how we're disseminating the agency-specific results and how that's going to work, because I think that's an important piece of information. I think you've raised an important issue.
Jeff, can you speak to that? Thanks.
Mr. Jeff Wright: Sure. We are, and we have been for the last couple of years, distributing the aggregate provincial reports with regional reports as well, so that regional offices and agencies could understand what provincial trends were and understand what regional trends were relative to the provincial trends. Something that we introduced this year, which I think is really important for agencies, is their own individual report. On all of the dimensions that we're measuring provincially and regionally, we're actually providing those same comparable dimensions to compare to the provincial and regional reports.
The agency report currently goes to just that agency, not to other agencies, and we've been told by the sector that that's probably the best way to ensure that they are continuously improving, that making public their results relative to others is a bit of a sticky wicket and may create some challenges in terms of bringing them along in their continuous improvement. What they've said to us is more valuable is that they actually have their own results to compare against the regions and provinces and that they can work with their regional office to actually make changes that are required, should something be kind of anomalous between their experience and that of their region or province, the like comparators.
On those local reports, we haven't organizationally come to a clear decision on how public those would go beyond the agency level, for the reasons I described. The aggregate—
The Chair: They would be FOI-able, though, would they not? They're public information.
Ms. Wright: Yes, they would be FOI-able, Chair. I think the point that's important here is the balance between getting the agencies and sectors to embrace and use this kind of data. This sector has moved a long way in the last five years, but it has a long way to go. I think the point we're trying to make is, yes, of course they should be public in the way that you've just said, but for the agency I think it's important for them to, first and foremost, understand and own the responsibility for what the data says about their service.
The Chair: Questions? Shelley?
Ms. Shelley Martel (Nickel Belt): Thank you, deputy and staff, for being here today and for providing the information that you did. I wasn't going to start at this point, but because the Chair has asked some questions, let me just go back to the reports that you're talking about.
I'm not clear what comparators are being used, so I'd like to get an idea of what those are.
1030
Mr. Wright: I'll speak to the two measures separately. The BCFPI is really a tool that helps us understand the profile of children. That's sort of the numbers of children who were seen who are experiencing a mental health issue, the types of children we're seeing and the gender breakdown of the children that we're seeing. We can even fine-grain to things like, what is the family composition of the families that we're seeing? For instance, what percentage might be single-parent-headed, relative to two-parent-headed?
This information is enhanced by our ability, through the BCFPI, to also collect wait-time information. So the agency reports—when we're talking about comparisons, we would have an opportunity for the agencies to actually compare on those separate dimensions I just described against their regional and their provincial report to see what kind of variance they have. So for their own planning, is there real variance, for instance, at agency X in terms of the gender of the children they're seeing, the types of families they're seeing and the types of presenting problems they're having to manage, relative to others, and the acuity of the problems?
The great value of the CAFAS is that it provides us information about pre-treatment and post-treatment change. We can then compare at the agency level, or the agency level can compare to the regional report and to the provincial report, on that dimension. If there's real variance, I think that creates a very important discussion between themselves and the regional office, in terms of how they plan for these things.
It's very important though too to recognize that these measures are good only insofar as they're used most effectively, and I think the most effective use of these tools is to do repeated measures. At the agency level you would want to be looking at your results each year, relative to the previous year and, for that matter, the former year. Likewise, at the provincial and regional levels, we want to be doing the same and looking for things that seem to be stable on those dimensions I described and looking for things that seem to change and, particularly, not just change in the period of one year of time elapsing between two periods of measurement, but two years. If we start to see those sorts of trends, then I think we really need to probably look at them very closely, understand them and make changes if necessary, either at the system level or at the agency level.
Ms. Martel: So am I correct to assume that if there's corrective action, it really has to do with information that comes from CAFAS? Because the first one is essentially a profile. There's not much an agency can do about its geographic location, how many single-parent-led families there are, and gender etc.
Mr. Wright: I would say, though, I think there's something very rich there on the profile piece. For instance, if you were at an agency level and you were providing a standardized, evidence-based program that was perhaps better for boys, or recognized through the evidence to be better for boys, but you had year-over-year information that's telling you that you were actually seeing more girls, you would want to reflect, I think, on the choice of that particular evidence-based program.
The other thing in BCFPI that is really important is the wait time. I would say that that is a very important piece, as well as improvement in child and family functioning.
Ms. Martel: Now, are all agencies using both systems?
Mr. Wright: We have 120 licences for each that we pay for each year; 115 of the agencies that participate use both instruments. So we have a difference of the five where it's either one or the other. The reason that it's one or the other is because in those cases where it is one or the other, the CAFAS is being employed almost independently where it's more of a treatment environment. They've already had a client referred to them who has been formally assessed and they are wrapping around an intervention.
In those few cases where we see primary use of the BCFPI, they're more the cases of the centralized intake that has evolved in different regions to provide that front-end assessment and referral to the places that treat.
Ms. Martel: Can you explain to me the difference between your agencies with licensed agreements and your others? The auditor looked at 250 agencies. Some have licences; some don't. I don't understand the distinction.
My next question would be, if you've got other agencies not using those two tools, how are you monitoring them?
Mr. Wright: The 120 agencies began with a decision taken many years ago when this was being introduced into the system, in 2000. The criteria that were introduced, which really haven't changed largely, are—of those 200-some agencies that you described, there was kind of a culling out of the ones that were the largest that were also having, as their main, staple service, children's mental health. We have a number of other agencies in the children's mental health line that would fall into that 200-plus group, but when we do our analysis, we find that they're providing some level of children's mental health but, by and large, it could be child welfare services; it could be youth justice services. So the focus of the 120 licences is really on those that are providing core children's mental health services.
Ms. Martel: That's helpful. Thank you.
Let me go back to your evaluation, then, on page 6, when you said you'd just completed phase 1 from 79 services. Originally, I thought that you were going to be applying that to the 113 new services that had been funded. Are the balance of the 113 minus the 79 going to be examined as well? That's the first thing. Secondly, because that was only new services, there were some 96 other existing services. Are they being evaluated at all, and how?
Mr. Wright: There is a mix of new services of the 113. Part of the mix is also the 96 that were enhanced services. The reason that we show 79 services that are actually being evaluated through this is, we took a decision early on, in evaluating these programs—I think one thing that often happens with evaluation is, you commit the people who receive the funding to evaluate the program, which we did. When we started to see the regional plans, there was a clear lack of sophistication around what they intended to do. Particularly from a regional and system planning perspective, we were not going to get information on common outcomes that was going to be very useful to plan. The amount of money that each agency would have of the aggregate pot to actually examine their own investment would not be as good as us applying a common outcomes approach to as much of the initiative as we could. That was a decision that we took, and it was aligned with the federal indicators project. The 79 programs, then—part of the decision-making was, we decided to look at those programs that received $50,000 or more. The ones that were getting less—it was very difficult to involve them in the evaluation because they didn't have a lot of resources on the ground to even participate.
So, rightly or wrongly, we took a decision that it would be those that received minimally $50,000 that would be part of this evaluation against the common outcome indicators, which we could then analyze at a provincial level, a regional level and an agency level.
Ms. Martel: And when you say you monitored common outcomes, can you tell me what that means?
Mr. Wright: We were interested in how accessible services were to children and families. We were interested in understanding how children and families improved after receiving an intervention. We were very interested in understanding how the investment was leading to better collaboration amongst service providers and coordination in the sector.
1040
Ms. Martel: So you've got the 79. It's not your intention to do more at this point. As a result of that evaluation, which is complete, how is that information now being used?
Mr. Wright: We just completed our first report. As we are with all of our reports and sectors, we're moving towards a real, strategic dissemination approach. We're of the mind to try and share as much as we can with those people who are working with us so they can continuously improve. This report, unlike the CAFAS and the BCFPI, is one that we would share corporately, that we would share with the regions and that we would share with the varying participating agencies so that they could have those sorts of similar discussions that I described around continuous improvement with both CAFAS and BCFPI.
Ms. Wright: This evaluation, Ms. Martel, was not meant to be an ongoing evaluation. It really was meant to be a—
Ms. Martel: Snapshot?
Ms. Wright: A snapshot, thank you. I think we would say that one of the unintended side effects of this evaluation has been an opportunity for the agencies that were involved in it to have a good look at their own data and to look at their data from the perspective of common outcomes and whether they had the capacity to measure it or not. I would say that we will feed this information back to the sector, as we would normally do, but we also have had an opportunity to engage with these agencies on this broader issue, which the auditor has raised, on how you actually use your own information and data to measure outcomes.
Ms. Martel: How does that data fit into the community planning tables?
Ms. Wright: That information would go back to the agencies that had been involved that could come to the community planning table, if the community planning table chose to do it. Because it isn't every agency that's at the table, it's a more contextual piece, I would submit to you, than an agency-by-agency piece. The CAFAS and BCFPI data would be of more interest to the planning table.
The last time the planning table—it has two functions. One is to look at allocating the new money. They had experience doing that the previous time that we gave them new money. I think the planning tables themselves develop their own way of doing that. On top of it, we will now be asking them to do the mapping for the framework. That will help them put a certain amount of rigour into the conversation that they're going to be having on the allocation of the new money.
Ms. Martel: But they have access to all of this data?
Ms. Wright: They have access to the BCFPI/CAFAS data.
Mr. Bezzina: A few things: The community planning tables are much broader than just the mental health organizations. What would be very important to them is going back to the conversation that we had about BCFPI and CAFAS regional reports. Those would be of more use to the planning tables than this particular data set that we're talking about. Certainly in order to do the planning, we would make that regional report available to them. But I want to emphasize that the planning tables are much broader than our funded agencies.
Ms. Martel: Right, because they would just have paediatricians and health care professionals who are involved.
Mr. Bezzina: School boards, yes.
Ms. Martel: Explain to me, then, the difference between phase 1, which you've just completed, and phase 2, which you are just starting, and which programs are being targeted.
Mr. Wright: Phase 1 is an outcome monitoring approach which we took. That's applying common indicators to the 79 programs and having them provide data against those. That's really important from a planning perspective because it gives you a dashboard understanding of what's going on at several levels. What it doesn't do from a research perspective is allow you to understand what it is underneath the results you're seeing that may be causing those results. As you apply a repeated measure, for instance, on those same indicators, it really doesn't tell you why you might have a change for the better or a change for the worse.
Phase 2: We thought it also important not just to have a dashboard kind of understanding of things that would lead to a constructive conversation between those who are funded and those in the regions and so on to try to understand what was underneath and make continuous improvement, but it was important to also invest in a more rigorous evaluation, a more rigorous experimental evaluation where we could quantitatively understand what it was about an evidence-based program, specific program components and the results, favourable or unfavourable, that the program would have.
So phase 2 was about looking at all the various programs that received investments and applying criteria. We had people from the centre of excellence, some of their scientists, and some of my researchers working together to cull through the various submissions that were made from the agencies in the regions about the programs they were to provide and look at those ones that looked on face to have evidence behind them and that also looked like they were reasonably mature, sustainable and evaluable. We then had that group come back together and refine down to 12 programs that seemed to meet those criteria on face, and the staff from my branch and from the centre of excellence then went out and did fieldwork and did key informant interviews with the various agency personnel, observed some of the interventions that they were providing and came away with a clearer sense and a next cut of what was probably a premier kind of program that we would want to really evaluate rigorously with a view toward generalizing across the system.
The one that we landed for the time being of the three was chosen because it was most mature and it was ready to be evaluated. The other two programs—I think there's still a way to go in terms of their implementation, but we would still look at them over time to see if they may on face turn into a potential best practice, which we would then also want to rigorously evaluate.
Ms. Martel: If you're down to one, in terms of establishing best practices, I assume you want more than one, or—
Ms. Wright: Well, the purpose of phase 2 evaluation really is—to simplify this a bit—to look at an agency that has good data, has good outcomes, is evaluable in the classic program evaluation way, and then to do that and say, "Here's what the evidence tells us about this program, about what works and doesn't work"—I would say this is an example of how to do that—and then to share those best practices and evidence across the province. I think what Jeff is saying is that in the group we were looking at, this one was strong. We'd like to do it and see how that happens as opposed to—it's not meant to be a comparable evaluation. It's actually meant to say, "Here's the evidence that this program works and therefore actually other agencies could use this program." Is that plain language?
Ms. Martel: When will that be done?
Ms. Wright: CHEO is just beginning to put the evaluation team together. How long is it going to take, Jeff?
Mr. Wright: I think we're looking for interim results within a year. Typically with these sorts of things, we have about a two-year commitment and follow-on.
The Chair: We'll come back. Do you have any idea of the total budget, what that would represent? I don't know whether you can, because of the way you fund—
Ms. Wright: We don't have it with us.
The Chair: I mean approximately. Is it 60%, is it 90%?
Ms. Wright: We'll get back to you with that data. That's a good question.
The Chair: Okay. Mrs. Van Bommel.
Mrs. Maria Van Bommel (Lambton—Kent—Middlesex): Deputy Minister Wright, you talked a tele-psychiatry program. As an MPP for a rural riding, it sounds pretty innovative for access. But how would families access? How do they get to the program? How does it work?
1050
Ms. Wright: I will ask Alex to give a more detailed explanation of the program. As I said, its purpose is actually to hook up professionals with service agencies that may not be able to have access to the particular professionals. We have invested in it over a number of years now, but I will ask Alex to explain more precisely how the program works.
Mr. Bezzina: The telepsychiatry program is, as the deputy indicated, an ability to link up areas of the province where psychiatric expertise is not available or is available only in a limited way. So in a number of rural, underserviced northern areas, there simply are not child psychiatrists available.
Through the initial telepsychiatry program, we had 14 sites across the province that were linked by technology to the Hospital for Sick Kids here in Toronto. At the request of the agency that needed a consult about either a particular family or a program that they were trying to establish or to take a look at or about a particular child who did not have a clinical assessment as yet, they were able to link up through video teleconferencing with a psychiatrist that the Hospital for Sick Kids was able to make available. An appointment would be set up, the family would come to the video teleconferencing site, there would be an assessment done over the—it would be like doing a clinical assessment in a doctor's office except we're using video teleconferencing. An initial diagnosis can be made, a medication regime determined, and a plan for care start to get established in consultation with the professionals, who would also be on the video teleconferencing site. That's one kind of consultation that can take place.
Another kind of consultation is program consultation. There is a particular issue in the community, for example, through BCFPI. They are determining that they're seeing increasing numbers of kids, for example, who might be prone to a particular type of behaviour. I'll just use fire-setting as an example. The organization may not have the expertise or the background to say, "How do I develop a program for these kinds of kids who are exhibiting this type of behaviour?" So that program type of consultation can take place.
The other kind of consultation that can take place is if a professional is experiencing some issues or difficulties in terms of treating a particular family or client. So a professional can also do a professional-to-professional consultation to say, "Here's what we've tried. These are the outcomes that we're achieving. We seem to not be doing as well as we could. Do you have any other suggestions for intervention with that family?" That's the third kind of consultation that can take place, the professional-to-professional consultation.
The other thing that the video teleconferencing link-up provides for is fairly inexpensive educational seminars to take place, where you can link up a number of different sites on a particular topic. We provide, through the Hospital for Sick Kids at this point in time, a roster of clinical seminars that people can dial into and participate in. It's a cheap way so that you don't have to fly people in to a major site like Toronto or Ottawa or London, where the clinical expertise exists, and we can do seminars in that particular way. It provides for capacity-building for organizations and for the professionals in those organizations.
As I mentioned, we started off with 14, and in the fall we announced the expansion of the telepsychiatry program. We had been meeting on a regular basis with the lead of the telepsychiatry program at the Hospital for Sick Kids, who was indicating that they were getting increasing requests from other communities, and the capacity for dealing with the requests was overextending their budget. So we began to look at, how do we stabilize and expand the telepsychiatry program? As a result of the expansion, we will be going into 10 new communities and establishing two additional hubs. The hub is the function that the Hospital for Sick Kids has played up until now. We will be establishing a hub in the western part of the province through London and a conglomeration, a collaboration, of agencies in London that will be providing that clinical expertise. In the eastern part of the province, which will also add to our French-language capacity in this regard, we will be working through CHEO, the Children's Hospital of Eastern Ontario, to establish a hub. They can continue to support not only the 10 new sites but the 14 original sites.
Mrs. Van Bommel: How do you measure outcomes in that respect, when you're working through teleconferencing?
Mr. Bezzina: The outcomes would be measured by the agency that's actually delivering the service. The clinical outcomes are going to be measured using CAFAS, the child and adolescent functional assessment survey—
Mr. Wright: Assessment scale.
Mr. Bezzina: Sorry; I don't have the right terminology here. But we're using CAFAS to measure how kids are doing.
What we have with the Hospital for Sick Kids and which we will be using for the other two hubs that I described are a number of indicators in terms of the kinds of requests that we're getting, the kinds of consultations that are going on. We can analyze trends of the kinds of clinical consultation questions that are coming up or professional-to-professional consultations that are coming up so that we can then devise the educational roster. So if you're getting regular questions about a particular type of programmatic intervention or a particular clinical symptom that professionals are struggling with, you can develop some educational sessions.
We collect data at the hub level to ensure that the telepsychiatry program is meeting community needs, but in terms of the outcomes of the kids, those are measured by the agency that is delivering the service.
Mrs. Van Bommel: Thank you.
The Chair: Mr. Milloy?
Mr. John Milloy (Kitchener Centre): I want to thank the presenters for coming before us today.
I had a general question about benchmarking and measurement and things like that. Obviously, like many people around the table, I deal with constituents who have children in various programs, not just children's mental health but related programs, and, of course, I deal with the agencies themselves. Sometimes there seem to be three propositions that—I won't call them contradictory, but I guess I'm just wondering how we deal with it in terms of some of the measurements.
The first is that we want to have a system where the children with the most serious, pressing problems are treated immediately. I don't know if it's fair to make an analogy to an emergency room, but the person with the heart attack gets to the front of the line. Second, you want to have a system that has services available to everyone, so even something which is not as serious or pressing, they have every right and opportunity to those services on a timely basis. Then you have the parent who says, "Look, my son" or daughter "may have a concern or something that needs treatment which is relatively minor right now, and no one is paying a lot of attention, but the fact of the matter is that six or eight months from now, it's going to become a lot more serious."
I'm just wondering, in the benchmark—I know this is probably not a very easy question. It's probably something every agency or government that's overseeing is trying to grapple with. I just wonder how you deal with those three so that we know that children in the most serious cases are going to get to the front of the line but the other ones aren't forgotten.
To be honest with you, sometimes I don't find the wait-time data very satisfying, because if it's percentages or even numbers, I want to know, hey, does the kid at the front of the line have the equivalent of—if you'll excuse the analogy—the heart attack in the waiting room, or does the child at the front of the line have something that, if they're seeing a counsellor or someone giving them treatment in three weeks or even a month or two months, it's not really going to make a big difference? I just wonder how you grapple there. An easy question.
Mr. Wright: I think you're right: The wait-time data alone is not always particularly helpful. That's why we use the CAFAS. The CAFAS is a really strong instrument for helping clinicians, when they're meeting with a child and a family, assess how acute a case is. So it's very valid and it has been validated reasonably extensively to ensure that how a child would score after that assessment occurs—that score is actually related to different levels of intervention that are required based on acuity of the presenting problem. So that level of sensitivity in the measure is really important. It's a very valid measure for determining who those kids are—to use your notion of the heart attack, the kids that would fall into that more serious range. The CAFAS actually directs, based on the results of the assessment and the score that the child would receive, a very specific kind of intervention. There's a range from having a low score, which puts you in the "I should go see my GP with some regularity every six months just to make sure that this doesn't evolve into something that's more serious," through to a secure type of in-patient treatment. About 6.7% of our kids across the province fall into that category once assessed by the CAFAS. Most of our kids fall more toward the middle, and many fall toward the lesser end, but the value of the CAFAS is in determining where they fall and also pinpointing the level of service that is required, if any, for the case.
1100
The other thing that's really interesting too is that the CAFAS really shows us what the presenting problems are for kids who are pre-adolescent versus those who are adolescent. As you could expect, once kids become adolescents, if they haven't been diagnosed and/or helped earlier, their situation can be more debilitating. So typically what we see through the CAFAS is children, before they become adolescents—those more serious problems in adolescents are things like psychosis, drug abuse, self-harm. At the child level, through the CAFAS, we typically pick up things more like school problems, problems at home and problems relating to others. Many of these sorts of things don't require a really formal intervention so much as direction to parents about how to involve the school more to help their child, to take preventive steps so that they don't become more acute.
Mr. Milloy: Will the CAFAS pinpoint—I guess you've half answered the question. Does it pinpoint with a bit more specificity the kids that, "Hey, we need some intervention now," where there were some warning signs that perhaps in the past might have been overlooked or were sort of shuffled back?
Mr. Wright: Yes, all the items that are included, the things that get measured against, are all warning signs. It is sensitive to pick up any and all of those warning signs that are in the literature. What makes a case more acute, though, is typically having more than one of those things concurrently. To your point about the more at-risk cases and if they get to the front of the line through the assessment, yes, because they are typically having multiple problems that are picked up through this sensitive measure, and then they are having a service wrapped around that is indicated in that situation. But for those kids for whom we need to employ prevention, prevention early, who are starting to show a sign, yes, it will be responsive to that as well. Not only will it be responsive to picking it up, but it will tell you specifically what the area is so that you can focus your preventive intervention around that. So as I say, for instance, problems in school, it will pick that up, and then you know to direct your attention toward managing the specific difficulties this child or youth is having in school.
Mr. Milloy: So I take it the good news is that, in the past, just the general assessments would pick up the very obvious cases, that CAFAS in fact is picking up the one where the parents say, "Hey, you know, we can head it off at the pass now," and there's an opportunity for it.
Is the wait-time data going to become a bit more sophisticated, then, in looking at what level of acuity is where in the list?
Mr. Wright: It'll become more sophisticated once we're able, in Ontario, to move toward a unique identifier, so that we can then tag the results from the measures against individual case wait-time information and then aggregate that up.
Mr. Milloy: And can you tell me a little bit more about the unique identifier? Would that be a level? What is it going to look like? Like you're a "level 1" or a "level 1B" or a level—
Mr. Wright: No. A unique identifier—what we're referring to there is, very often in research you work with a sample or a group of people whom you're trying to understand through measurement. And then you want to introduce other data from another system they go to. So for instance, if they're in the child welfare system but you also want to understand education information about those individuals, what you would like to do is to be able to have some identification like a number that is applied across all these various sectors, like child welfare, education, children's mental health and so on, so that you could actually follow a case between them without identifying the individual but by identifying them through a common number, like a health number, for instance. So that's what we're talking about in terms of a unique identifier.
We, through our long-term-data strategy, are moving towards trying to introduce that unique identification process in Ontario. We are learning through the experience of a couple of provinces that are much further along in this regard, who've actually either introduced an outright unique identifier number, and they've had that support from their privacy people, or they've introduced something that's nearly as strong, which is a de-scrambling technology to follow one case across those sectors. So they introduced a way that a business intelligence tool can go into the various data sets from the different sectors and pull out an individual based on, say, four demographic pieces of information that are contained in all the sets.
That's what I was referring to in terms of unique identification. Until we can actually move formally and have support within Ontario—which we're working toward with the privacy commissioner—to having unique identification, it will make what you're wanting tricky for the time being.
Ms. Wright: We've talked about this before in the sense of, without some way of tracking children and youth on an individual basis, even within the children and youth mental health system, we actually can't tell you really with any confidence how many services that child or youth is in need of or accessing. If we could get to a clear unique identifier that cut across systems so that we could actually then know if a child who had mental health problems succeeded or didn't succeed in school, we could actually have a really interesting database that would enable us to have stronger evidence on what works and doesn't work for whatever the outcomes are that we identify as being significant for children and youth.
There are significant privacy issues on having a unique identifier, which is why we keep mentioning it. Many provinces have struggled with that. The UK has recently introduced an index that enables anybody who is helping a child to go into a common database to see what service that child has had. It has been highly controversial in the UK, but it is an attempt to ensure that all people who are serving or helping children and youth understand what other services that child or youth is getting.
Mr. Milloy:Sorry, Mr. Chair; I'm using up lots of time here. But then will there be a way to—I mean, obviously there is, but are we going to be developing an acuity level, if that's the term, where you can start to compare kids, as I say, whether it's a child who's in desperate need of services—where I'm coming from as an MPP, and I guess everyone around the table would identify with this, is the number of parents who come to see you, the media reports. The general thesis is, "My child is stuck in a line and desperately needs to be at the front of the line."
1110
The reality, and I have tremendous sympathy for them, is that all of us as legislators would say, "Hey, the resources have to be meted out in a way so that the most serious cases are being dealt with first." I always find that in this kind of thing to describe what is the most serious is really, really tricky, especially when you're dealing with unique—literally, by its title—each individual child. Are there going to be ways to get some handle on who's falling in a category 1 or a red category or a blue category? Is that the final step?
Ms. Wright: As Jeff says, CAFAS does do a version of that. I think a longer-term strategy about this, though, is to strengthen the ability of the agencies to use evidence to do triage as effectively as possible so that there is a clear and public understanding of why child X got ahead of child Y.
Secondly, in a system where demand outstrips supply, we also have to be very careful that all of the resources don't go into acute so that we don't deal appropriately with the prevention side or that group of kids—I can't remember your category—who aren't going to tip into crisis but if we don't intervene now may tip into crisis. I think CAFAS does provide the agencies with some capacity to make those judgments, and secondly, it is a professional judgment that is being made by the clinician.
The unique identifier is really about measuring outcomes more than it is about answering your question, if I can put it that way. It really isn't a capacity to be able to say, "Here are the kids being served and here's what's working and not working."
The Chair: Thank you. You used up 20 or 22 minutes of your rotation. Ms. Sandals, how long do you anticipate your question will be, because I should go to—
Mrs. Liz Sandals (Guelph—Wellington): If Ms. Martel wants to go, and then you can come back and I'll know—
The Chair: Sure. Ms. Martel.
Ms. Martel: I wanted to ask some questions about your SMIS. I had forgotten this, but the deputy auditor reminded us that he thinks at our last meeting there was some discussion that the ministry was going to be looking at systems in other jurisdictions to see what might be useful to pull from other jurisdictions. I'm wondering if that was correct and if that actually happened.
Ms. Wright: I'm sorry, Ms. Martel, I actually don't remember this commitment. We do, on an ongoing basis, look at our own information systems in terms of what we can learn from other jurisdictions. I'll have to go back and check if we actually did do a comparative analysis of the SMIS system. I apologize; I don't remember.
Ms. Martel: No, and it's not my intention to put you on the spot. I'm sorry about that. That was raised earlier this morning, so I thought I would see. So if you can let us know, that would be very useful.
The Chair: Ms. Martel, maybe the deputy auditor can clarify that.
Mr. Gary Peall: Maybe I'll just try and clarify, and then you can jump in. I think in November 2005 you talked about a major review of SMIS, but in particular looking at the transfer payment business practices in other jurisdictions, which you were hoping was going to be finished around March 2006. I think that was the review you were talking about. It may have had implications for SMIS, but it was also the business processes themselves for how you manage the relationship and the funding between your agencies and you.
Ms. Wright: We have done some work, less on a systems basis and more on governance and accountability around transfer payment agencies. We've been doing some background research and some papers on looking at the accountability side of it. As I outlined in my presentation, we made some very specific improvements to SMIS itself, and I think the next stage of this would probably be to put the governance work and accountability work together with the systems work.
Ms. Martel: Do the upgrades that you've made—I think that's the best term to describe them—make you feel confident that the information that the ministry is receiving now is up to date, is correct, from the agencies sending it in and then what the regions are looking at and then when it gets sent to corporate office?
Ms. Wright: I'll have Alex speak more directly to this, but I think the changes that we've made have removed a lot of the potential error rates and transposition rate of mistakes. I also think we can access that data in a way that's much more real-time access than we've been able to do in the past.
This is a bit of a cliché, but I think we need to have a continual improvement view of the financial accountability and service information that we get from transfer payment agencies.
We are much better at bringing the service data together with the financial data, which I think has been a regular recommendation from the auditor over the years, and Alex can speak more specifically to that, but I would say that we can still continue to make improvements.
As you know, we fund a lot of agencies. It is a big system when you look at all the agencies, not just children and mental health. SMIS is for all the agencies, including community and social services.
I don't know, Alex, if you want to add anything.
Mr. Bezzina: You asked if the changes make us confident. They make us more confident. As the deputy indicated, this is a continuous improvement exercise.
One thing that has made me feel more confident is the fact that I require director-level sign-off now on the data, so my regional directors have to sign off on the data. It requires, therefore, that the program supervisors in the regional office are looking at the data and ensuring that it makes sense to them. In order to get there, we've started and we need to continue to work with our program supervisors and business staff in the regional office to get them to understand the SMIS data: to interpret it correctly, to understand when there's an anomaly and, if there's a variance in what we think we should be getting from the agency and what we actually see, to ask the right questions. It's about interpretation of the anomaly. So we do have more work to do in that regard, and we will continue to do that.
The other thing that the deputy did refer to is the fact that we now have automatic upload of the Excel spreadsheets, both in relation to the initial budget submission and in relation to the quarterly reports—again, this is both financial and service data that come in the Excel spreadsheets. They're automatically uploaded into SMIS, as opposed to doing the manual inputting that led to high rates of transpositional errors. That, again, helps to increase the confidence level in SMIS data and also in the usability of SMIS data, so that when I ask for a report on whatever, I feel confident that the data has the integrity that it needs for me to either make decisions or to say, "Oh, I've got a problem here and I need a fix."
So the reporting tool that we introduced this year allows me to get regular reporting on specific items and actually take some action associated with it.
Ms. Martel: Are there other proposed changes that would be required that would also require a capital investment?
Mr. Bezzina: I'm not sure if you're talking about an IT infrastructure investment.
There are always things that we can do to improve. We are now looking at a multi-year plan associated with the improvement of SMIS and/or whatever the data system ends up being, and we need to consider appropriate resources in order to do that. We are now beginning to sit down—and this is, in part, in collaboration with Jeff's area—to say, "Okay, SMIS data should not just be capturing the kind of data that we're currently capturing; what are the data sets that we really need to be capturing in there?" So there's a need to bring together the required data elements to make sure that there are clearly defined data definitions, so that everybody understands that this is what this particular datum means when we're entering it or when we're interpreting it, and then the ability to ensure that we have more timely and accurate analysis tools.
1120
Part of it is that we believe we can continue to do what we're doing right now. Should we move to another platform down the road? We need to take a look at that. My main concern right now is asking, along with Jeff's area: What are we asking for and how do we start introducing additional data sets into the system—and not just data sets for the sake of data, but data that actually will tell us something? So that's where we're currently at. We have to sit down and do that work, and that work is under way.
Ms. Martel: Okay. I just wanted to ask you some other questions that you didn't address in here but were part of some of the appearances you've had before us. When you talk about data, let me ask: What kind of data are you getting now from the regional providers for autism? I'm interested specifically in waiting lists for assessments; waiting lists for children who have qualified but are waiting for service; if the data is sophisticated enough that it's showing the age levels of children being serviced and those who are waiting—I'm sorry; I've got it written down here. I would like to know if you're capturing some of this information: numbers of kids waiting to be assessed; secondly, the number of kids who have been assessed and are qualified but are waiting for service; the ages of those receiving service and the ages of those who have qualified but are still not receiving service; and also, agency by agency, any record of lost service hours.
Ms. Wright: And service provider by provider?
Ms. Martel: Yes.
Mr. Bezzina: When we started a few years ago to look at the issue of data and needing to have regular reporting on some of the issues that you're talking about—waiting for eligibility assessments and then waiting for service—we began to do, as you probably are aware, an exercise, along with the regional provider, on developing a quarterly report. That quarterly report is done by our regional offices sitting down with the regional providers and uploading the data. It's specific to a few things: number of kids waiting, number of kids for eligibility, number of kids waiting for service.
When we begin to ask more detailed questions, like age of kids, we begin to move to another project that we've undertaken and is still in process, and that's the ISCIS project. In this particular committee we've talked about the Autism Program Information System, or APIS. ISCIS is the Integrated Services for Children Information System. We're using it on a few fronts. Preschool speech and language is an example of that. We have introduced it into autism. All the regional providers are currently entering data into this ISCIS database.
We have contracted with Smart Systems for Health Agency to collect the data and develop the data and begin to get us some regularized reports, and we can also get ad hoc reports. So to the question that you asked earlier about age, we can get that information from ISCIS. We did actually ask that very question in the fall, I believe: How many kids in service are—and we asked a very blunt question—age six and above and age six and under, and how many kids waiting for service fall into those categories? We were able to determine, from ISCIS data, those numbers.
It is beginning to tell us a richer story through ISCIS, but we still have not gotten to the point of maturity with ISCIS, where we're able to mine the richness of the data that's in there. We're working with Smart Systems for Health to get us there.
Ms. Martel: Okay. So I should assume that the question that was asked of the agencies was a special request, and it would have been a special effort to retrieve that, but it's not being retrieved on a quarterly basis at this point.
Mr. Bezzina: Right. Because the regional providers are now entering into ISCIS, when we had that ad hoc question we needed to answer, we didn't have to go to the regional providers; we were going to ask it of ISCIS. Smart Systems for Health was able to get us that information based on the data that they have.
Ms. Wright: You are correct, Ms. Martel: At some point we have to take the quarterly report request and integrate it into ISCIS to make it one process. We are a little bit parallel right now.
Mr. Bezzina: We're parallel-tracking.
Ms. Martel: I'm just not clear on why you'd need Smart Systems for Health to do that. What are the problems or the complexities here that make it not possible at this point to actually retrieve that information?
Mr. Bezzina: It's a capacity question. Just in terms of the database itself, we could have retained it within the ministry. We instead decided to contract out the collection of the data for us—the analysis of the data with the development of the reports.
Ms. Wright: We could have contracted it someplace else. We chose Smart Systems.
Ms. Martel: Okay. So they're giving that back to you on a quarterly basis. Are you asking now, as a matter of policy, from Smart Systems—maybe this is a better way to do it—some of those questions that I've just put in terms of information?
Mr. Bezzina: There will be a series of reports or a series of data sets that we will be able to get from Smart Systems for Health. We're not there yet in terms of getting quarterly data from ISCIS and Smart Systems for Health. That's why we continue to do the quarterly ask-outs to the regional providers.
When it is a mature system—and we're working with Smart Systems for Health—we will be able to simply get those quarterly reports from SSHA and dispense with the more labour-intensive approach of getting the materials from the regional provider.
Ms. Martel: Is it possible—and I honestly don't know this—to give this committee the set of questions that you're pulling right now from the regional provider information? Is that public knowledge?
Ms. Wright: Through the quarterly reports, you mean?
Ms. Martel: Yes.
Mr. Bezzina: Through the quarterly reports, currently we collect information on the number of children waiting for eligibility assessments. We collect data regarding those kids who are waiting for service, once deemed eligible, and children in service. We can also collect discharge information. We can track it back as far as 2004, and we can look at trends associated with that. Finally, we can compare region to region, which is a very important systems management tool that we have.
We also can track some data elements, which I'm not as familiar with off the top of my head—I'd have to look it up in here—on the school support program. We're able to analyze trends in terms of region to region, DFO versus DSO as well, where we see changes in those percentages and where we see growth or lack thereof in the DFO rates, for example—not the payment rates, but the number of families that are accessing the DSO.
We can begin to also look at the issue of discharge, as I mentioned earlier—are we seeing discharge rates higher in some areas and not in other areas?—and begin to ask questions about why we see a difference here versus there. It provides us an ability to ask some questions, region to region.
In terms of some of the investments that we do, we can track them specifically. There were some investments over the last year. We can track the investments against the targets very specifically through this mechanism.
1130
Ms. Martel: Is that information shared between the regions; particularly, between the providers?
Ms. Wright: Which information?
Ms. Martel: The quarterly reports.
Mr. Bezzina: The quarterly reports are shared.
Ms. Martel: Between all the regions, so they can see how others are doing and also ask some questions?
Mr. Bezzina: Yes.
Ms. Martel: Usually I get that information when I do an FOI. Is this information confidential?
Ms. Wright: What do you mean by "confidential"?
Ms. Martel: Is it posted somewhere, and could I get it, or do I need to keep using the mechanism of FOI to do that?
Ms. Wright: I do believe that we've sent it to you before without FOI.
Ms. Martel: It came to us through a committee request before.
Ms. Wright: We would be happy to submit it to the committee again.
Ms. Martel: I would be interested just to see the areas that are now being tracked.
Mr. Bezzina: The quarterly report that you will see aggregates the data provincially, though. What I see is a little bit more granular. For the purpose of system management, I dig another level deeper.
Ms. Martel: So what you're saying is that I wouldn't see the discrepancies and differences, region by region, in the quarterly report.
Mr. Bezzina: No, not in the quarterly report.
Ms. Martel: How do you see that, then, if it's not in the quarterly report?
Mr. Bezzina: The quarterly report, just as a report itself, is the aggregated data that we provide at request and that allows us to also report to the deputy and to the minister in this regard. My stats provide the ability, though, to analyze it region by region. So I ask questions about, for example, discharge; I ask questions about DFO versus DSO; I ask questions about where we see eligibility wait lists growing, or not, as the case may be.
For example, when we were looking at allocating some of the resources in this past year, it allowed us to look at wait-list information and to say that just taking the pot of money and sending it out by base population doesn't make sense, because we can allocate the money region by region based on wait lists, and we can actually, then, provide targets to the regional providers, in terms of meeting those, and when they're not meeting them, we can follow up.
Ms. Wright: I was just going to add the comment that we have set pretty aggressive targets, as you're probably aware, for the regional providers, and I think that's one of the reasons why Alex digs into the data a little bit deeper: to ensure that we're setting targets that are appropriate but also just to ensure that providers are meeting those targets. We need a more granular set of discussions and data at the regional level.
Ms. Martel: So if we wanted some information about region by region, we should do an FOI.
Ms. Wright: Yes.
Ms. Martel: I have one final question; it has to do with the lost-service hours. This has been a topic of discussion a couple of times. I have certainly made the argument that where that happens and where that's the fault of the agency, the agency, which has already been paid for those hours, should be responsible for making those up. I don't know what, if any, progress you've made on trying to put that in place or whether or not the ministry is really going to try to put something like that in place.
Ms. Wright: There have been a number of discussions with the regional providers on this. I think it's fair to say we haven't found a rule that we need to impose on them, so much as we would like to encourage them to minimize the number of lost hours.
As we talked about at the committee the last time, this issue has slightly less urgency, given that the age limit has been removed. I think the existence of the age limit really did aggravate that situation quite a bit.
We have had a number of discussions with them about what they're doing to track sickness with their workers and those sorts of things, and we'll continue to have that ongoing conversation with the regional service providers to ensure that we can both monitor this and continue to minimize the number of lost hours.
Mr. Bezzina: Just a couple of additional things on that: The lost hours from an agency perspective often have to do with illness; it can also deal with turnover. The issue of turnover is real, and we need to recognize that unfortunately this is a high-intensity intervention and workers don't spend their whole career in this particular intervention. In terms of illness, and this is true of any group of people who are working with young children—you can look at illness rates of kindergarten and grade 1 teachers as an example of this—they just get sicker because the kids don't blow their noses and all the things that kids don't do when they are ill. So there is a higher level of illness, unfortunately.
We have a couple of things to ask the regional providers to look at: their backup systems in terms of when their staff are off ill; mechanisms for reducing the possibility of picking up those cold and flu symptoms within the organization; other health and safety issues within the organization; and finally their EAP provisions—if this is a high burnout kind of venture, what kinds of supports are staff being provided with?
The Vice-Chair (Mr. Ernie Hardeman): Thank you very much. Mrs. Sandals?
Mrs. Sandals: I'll see how long my voice lasts.
If we could go back to the BCFPI and the CAFAS reports, I take it that when you look at those two reports you would be able to determine things like: For this particular profile of children and this acuity, it's those sorts of kids in an agency have one wait time, whereas a different group of kids who need a somewhat different treatment, maybe even with a lower acuity, may have a different wait time. Can you get at that within agencies, depending on the profile of the children, what the wait times are for different treatments that are required?
Mr. Wright: I think you can do that level of analysis at the agency level manually. You are able to determine what the proportion of acuity of problems is at your agency. You're also able to—sorry. So the one was—
Mrs. Sandals: What I'm wondering is, is the system capable of separating out, "Here's one profile of kids with a set of problems, and this is how they're being responded to"—because not all kids have the same problem—"and here's another group of kids with a different profile and with a different problem, and this is how they're being responded to," because within an agency you can have more than one treatment stream.
Mr. Wright: Right. You can do that manually. There was a suggestion that was made to Kay Hodges, who owns the CAFAS and who we have as our vendor that we use each year. One of the suggestions that was made so that you don't have to do that manually is actually to build that into the report itself, the electronic report. So that is a customization that I think is being looked at quite closely.
So I guess the answers are yes and yes, but—
Mrs. Sandals: At the moment, you would have to get it manually, so we'd have to really ask some specific questions and have somebody with a little bit of expertise with numbers sit down and pull it out.
Mr. Wright: Right, yes.
Mrs. Sandals: Then probably my next question is the same thing, which is, can you look on an agency-by-agency basis—because you said that the CAFAS was doing pre-treatment and post-treatment assessments—at kids with different issues to see that, you know, "On this one, they're getting pretty good results; on this one, we don't seem to be getting a whole lot of effectiveness in what's happening"?
1140
Mr. Wright: We could. That isn't what we're inclined to do right now, but it is something that we may consider moving to in future. We are conscious of this being a relatively new way of doing business, and I think if we start drawing those comparisons between the agencies too early, it can compromise their uptake of this. So for the time being, we really have them looking at their own agency results against regional and provincial aggregate comparisons. But in time, as with many measurement systems, I think you would want to evolve there, once you had brought the culture sufficiently along.
Mrs. Sandals: I know you said before that that agency-by-agency data isn't really public, that it belongs to the agency. But it seems to me that when you get to the planning table, and I presume the planning table—I think you mentioned the school boards are there and children's aid. Hopefully everybody who's involved with kids is sitting at that planning table. If you're doing that on a county basis or whatever the basis is for that planning catchment area, it seems to me it would be very useful to be able to look at which agencies are having success with which issues and where the gaps are, which is, from my point of view, the purpose of the planning table: to see what we're doing well and where the gaps are. There's a potential to draw a lot of that data, if it was just analyzed properly, with what you've already got.
Mr. Wright: Right, yes.
Mrs. Sandals: What concerns me is that when you have a lot of different agencies providing various services, there's a certain degree of inertia that the treatment provided is related to the people who happen to be on staff, as opposed to the actual profile of the kids who may present. But, even recognizing the inertia in the system, you might be able to make better use of the available expertise if you were able to say, "This agency is doing really well with this set of problems, so let's direct everybody with this set of problems to this agency, and let's send people with a different set of problems to another agency." You'd at least have people dealing with the thing they do best. Are we looking at moving into that capacity in any way, so that we can support the planning with this data?
Mr. Wright: At the system level, we concurrently look at doing those things and actually do that kind of analysis. But in terms of introducing it into a planning cycle, we would have to consider how that would be done meaningfully. Another challenge, though—through the planning you can say that a certain agency does something better, but then there's another argument to say, "Well, then, why don't we train up another agency that's in closer proximity to that child or family so that they have the capacity?"
Mrs. Sandals: But either way, you would be using the data to provide the service, whether it's focusing the clients on a service with expertise or whether it's being used to do the sort of best practice and modelling other agencies on best practice.
Mr. Wright: I think the important thing is that we collect the right data elements so that we can choose to do those forms of analysis should they be relevant to planning.
Ms. Wright: I think much of this will come out of the mapping exercise we will be doing in implementing the framework, but I would hesitate to say that the purpose of that is a kind of specialization of agencies, which is sort of what I heard you say, Mrs. Sandals. It is more looking at what are the needs of children and youth and how they are being met, because—it's a little bit the rationale for having not done core services. We went away from saying, "These are the core services," to more, "This is what the community and the children and youth in those communities need. What are the functions and what's the best way to deliver them?" rather than, "What's the best core service?"
Mrs. Sandals: Yes, I understand that. You're looking at the best way of delivering the required service. It just seems to me that there's a wealth of data there, if you could get at it.
The mapping exercise: What sort of geographic territory is that being done on?
Mr. Bezzina: It really depends on the region, obviously. In larger regions of the province, we would have to break it down by smaller communities. In Toronto, we would have a single planning table.
Planning tables were developed back in 2004. We will continue with those same—in fact, those planning tables, in many cases, continue to meet, continue to discuss issues. It was a forum for looking at children's services, so it's excellent that they continue to meet, and now we can just continue to build on that.
From a mapping perspective, just to add to what the deputy said, there is an ability for people—although they may not get very specific about, "We're really good at treating this particular kind of diagnosis or this particular kind of symptom"—to maybe say, "In our system, we have groups of organizations or we have an organization that's really good with the specialized care." These are where kids have very, very serious mental health problems that are expected to occur over a period of time. They have the residential capacity to deal with it. They've got highly specialized psychiatrists to look at medication regimes etc. You may have other agencies that are really good at family or individual counselling for dealing with more functional types of issues, whether it's within the family or in the community or at the school level. Then you may have other organizations that know how to do self-help, prevention and other kinds of health promotion activities.
So the mapping will help us to get at that.
Mrs. Sandals: The local planning table is where you're going to get the service coordination and rationalize the services in a way that makes sense locally, but the better the data available to support that, the better the service coordination model that you're likely to come up with.
The other issue I'd like to go back to is the unique identifier and the degree to which you're constrained by privacy considerations, because there are obviously two other unique identifiers sitting out there: the OHIP number—because of the Canada Health Act, children's mental health services aren't a Canada Health Act service, so it's not really an OHIP service, but I'm sure all these kids are being seen by family doctors and all have OHIP numbers, or could have—and, for any who are school-age, they'll also be having an Ontario student number. There has been a lot, as you know, of effort going into developing an Ontario student number so we can follow kids through the school system. Is there any possibility of building on the work in either of those areas that has already been done, or do the privacy considerations force you to start all over again, which isn't necessarily—it's certainly not efficient and may not be very effective.
Ms. Wright: We haven't had a direct conversation with the privacy commissioner about this issue, to be able to answer your question very directly. Within our own discussions, we think somehow linking to the education number makes lots and lots of sense. But the privacy commissioner will have her own views on that that need to be respected, of course.
There are other models for—a unique identifier that is one number per kid is the ideal system. If we can't get that, there are other ways of being able to track kids in which maybe your data isn't 100% accurate—correct me, Jeff, on this—but it's good enough to be able to make some judgments about outcomes. We've had no conversations with our colleagues in the Ministry of Education about using the education number, but I think it's very much on the table for discussion, subject to what the other players would say. We all know that education outcomes are the number one indicator of success of kids, so if we care about those outcomes of success of kids, we should be linking up with that number as much as we can. But we're in a very early stage.
Mrs. Sandals: I know you certainly get things like kids who are the wards of the children's aid in Toronto who end up in some rural community in a group home or being fostered. If you could use the education number, you would, at least on the school-age ones, get the tracking of where they've been and figure out the whole story, which would actually be to the benefit of the kid in terms of tracking and providing the best service if you understand the background.
Ms. Wright: As I said, we just started this data strategy in January in the ministry. We have teams of people together who are basically scoping what I would say are the implementation and policy challenges, and then we'll move forward on those.
Mrs. Sandals: I hope you can build on the work somebody else has already done. Thank you.
The Chair: Thank you very much for coming, and I might say I am impressed that you're really trying to get some evaluation in this area. I think your work is really good, so far.
Ms. Wright: We know we can do better.
The Chair: Thanks for coming.
Ms. Wright: Thank you.
The Chair: I don't think we have enough time before private members' to consider the reports, so we'll postpone that for two weeks. Next week we have school boards and then we'll plan to do the reports after that. Agreed? Okay.
We're adjourned.
The committee adjourned at 1152.
CONTENTS
Thursday 19 April 2007
2005 Annual Report, Auditor General: Section 4.02, children's mental health services P-293
Ministry of Children and Youth Services P-293
Ms. Judith Wright, deputy minister
Mr. Alex Bezzina, assistant deputy minister, program management division
Mr. Jeff Wright, director, research and outcome measurement branch,
strategic policy and planning division
STANDING COMMITTEE ON PUBLIC ACCOUNTS
Chair / Président
Mr. Norman W. Sterling (Lanark—Carleton PC)
Vice-Chair / Vice-Président
Mr. Ernie Hardeman (Oxford PC)
Mr. Ernie Hardeman (Oxford PC)
Mr. Jean-Marc Lalonde (Glengarry—Prescott—Russell L)
Ms. Lisa MacLeod (Nepean—Carleton PC)
Ms. Shelley Martel (Nickel Belt ND)
Mr. John Milloy (Kitchener Centre / Kitchener-Centre L)
Mr. Richard Patten (Ottawa Centre / Ottawa-Centre L)
Mrs. Liz Sandals (Guelph—Wellington L)
Ms. Monique M. Smith (Nipissing L)
Mr. Norman W. Sterling (Lanark—Carleton PC)
Substitutions / Membres remplaçants
Mrs. Maria Van Bommel (Lambton—Kent—Middlesex L)
Also taking part / Autres participants et participantes
Mr. Gary Peall, Deputy Auditor General
Clerk / Greffier
Mr. Katch Koch
Staff / Personnel
Ms. Elaine Campbell, research officer,
Research and Information Services