Assemblée législative de l'Ontario

The committee met at 1228 in committee room 1, following a closed session.

2009 ANNUAL REPORT, AUDITOR GENERAL

MINISTRY OF EDUCATION

Consideration of section 3.04, Education Quality and Accountability Office.

The Chair (Mr. Norman W. Sterling): We’ll call the meeting to order. Today the public accounts committee is considering section 3.04 of the auditor’s report dealing with the Education Quality and Accountability Office, which reports to the Ministry of Education.

I believe we have the deputy minister, Mr. Kevin Costante; Ms. Mary Jean Gallagher, assistant deputy minister, student achievement division; Kevin Dove, the manager of the issues management, communications branch; as well as three other individuals before us.

Perhaps I will turn it over to the deputy minister at this time, and you can make the appropriate introductions of the other people sitting with you. I believe you have some opening remarks, and then the committee will put forward questions.

Mr. Kevin Costante: Thank you, Mr. Chair. It’s a pleasure to be here. I’m the Deputy Minister of Education, and I’m pleased to be here with my colleagues from the EQAO. I’ll be sharing the time on the introductory comments with Brian Desbiens, the chair of EQAO; and with the CEO, Marguerite Jackson. Marguerite will introduce the other individuals at the table with us. I think you’ll hear from them in greater detail about the particulars of this audit and about EQAO, generally.

I want to begin by thanking the Auditor General for his valuable recommendations on how to improve and enhance the EQAO processes. We are pleased with the results of the audit and that they attest to the success of the EQAO assessment programs. I know that EQAO has taken the auditor’s recommendations very seriously to make the program even more effective than it is now.

As the committee knows, EQAO is an independent agency of the government of Ontario. It was created as an arm’s-length agency in order to conduct assessment and gather objective information from our schools. EQAO works closely with our ministry’s student achievement division, and Mary Jean Gallagher, who’s with me, is the assistant deputy minister of that division. They also work closely with schools and school boards to help work with us on improving student achievement.

As you know, education remains a priority of the government and the ministry, and we are continuing to focus on our three core priorities: increasing student achievement, closing the gaps in student achievement, and increasing confidence in public education.

We believe we’ve been making significant gains. As was recently announced, more students are graduating from high school each year. In the past five years, the graduation rate has risen from 68% to 79%. Since 2003, 52,500 more Ontario high school students have graduated. Much of this success is due to many strategies, including some programs that have been introduced into our high schools like high-skills majors and dual credits with colleges.

EQAO provincial assessments provide us with a snapshot of student achievement at a particular point in time to ensure our approach has remained focused and relevant. They allow us to track the progress of our students at three key points of their school careers: grades 3, 6 and 9. This allows us to pinpoint any roadblocks to success or declines in achievement. These checkpoints also allow us to make necessary adjustments to our approach to student success and perspectives on student achievement.

Literacy and numeracy skills have steadily improved. We know this thanks to the province-wide testing that’s done. The EQAO results show that in 2003-04, 54% of students were achieving at or above the provincial standard. It’s worth nothing that the provincial standard is a level 3, which corresponds to roughly an average of B. In 2008-09, 67% of grade 3 and grade 6 students were achieving at that standard. That’s a gain of 13 percentage points since the earlier date.

The EQAO tests measure students’ performance in reading, writing and mathematics in relation to the common provincial standard. The assessments are developed and scored in a way that ensures that the results can be compared from one year to the next.

We stand behind EQAO, but acknowledge to our stakeholders that standardized tests provide only one of several measures of student achievement and should not be used to rank school performance. EQAO assessments provide valuable information for educators. The results both inform and support decisions made at provincial, board and school levels regarding required resources, capacity-building needs and specific instructional strategies.

Principals and teachers indicate that results help them plan strategies to enhance classroom learning and provide feedback on how well they conform to the curriculum. For parents and the ministries, teachers’ appraisals of student work will always be an integral part of our education system; however, parents also want to know how their children are doing based on objective measures of achievement in relation to the Ontario curriculum. It is therefore important to analyze the results of provincial assessments along with those evaluations that teachers administer on a regular basis to help understand and support student achievement in Ontario.

Province-wide assessments continue to contribute to greater quality and accountability in the publicly funded school system. EQAO releases an annual report of provincial results and makes that accessible to the public on their website. In December, the Auditor General found that the province-wide tests reflect the provincial curriculum expectations fairly and accurately. They also found them to be consistent in difficulty from one year to the next and that they are administered and marked in a way to ensure that the results are valid, consistent and reliable indicators of student achievement.

Our primary goal is providing students with a strong, well-rounded education that prepares them for the future, and the EQAO data is foundational in our efforts. These test scores are key indicators of the health of our system. They show us when we’re on the right track. They help us identify what’s working and what is not, and they provide us with useful data for school boards and schools to take action. EQAO data are also helping us make real improvements in student outcomes.

I’d now like to turn the floor over to Brian Desbiens, the chair of EQAO.

Mr. Brian Desbiens: Thank you very much, Deputy. Mr. Chair and members of the committee, thank you very much for the opportunity to come and speak with you.

I’m pleased to be here as the chair of the board of directors of the Education Quality and Accountability Office. I’ve served on the board for five years, and I’m in my third year as chair. I’ll be sharing my time with our chief executive officer of EQAO, Marguerite Jackson.

EQAO, as has been indicated, was established based on the recommendations of the Ontario Royal Commission on Learning in 1995. The commission consulted extensively with educators, parents, students and taxpayers and concluded that.the province-wide assessment would help respond to public demands for greater equality and accountability for the publicly funded school system.

1240

EQAO is an independent operational service agency that provides a measure of the quality and accountability of Ontario’s education system. EQAO delivers assessment programs that provide objective and reliable information about student achievement of the curriculum expectations. The information is used by educators and parents to improve learning. Our focus is on improving learning for all students. EQAO publishes school and school board reports to parents, educators, policy-makers and members of the public, who can use them to monitor the effectiveness of the educational system over time.

The agency is governed by a board of directors appointed by the Lieutenant Governor in Council. EQAO has been in place for some 14 years now under three different governing parties and has always maintained its arm’s-length status, which is essential for its credibility as an independent measure of performance.

The board of directors is accountable to the people of Ontario for maintaining this arm’s-length relationship, for setting strategic policy directions, for achieving its mandate, for setting operational priorities, for achieving excellence in the delivery of Ontario’s large-scale assessment program and for communicating with the public and the educational community regarding student achievement.

To guide its work, we’ve established strategic priorities. In your supplementary handout document, there is detail behind each of these priority areas. I will not be going into the detail. Perhaps you may wish to question at a later date. I do wish, though, to hit on those five priorities.

The first is that we wish to be the best of class assessment, not just in Ontario or Canada, but clearly in the world. We’re committed to providing credible evidence of student learning based on the Ontario curriculum, and that’s a particularly unique focus in large-scale assessment—the fact that it’s based upon our curriculum.

Second, we want to support student progress. We have a commitment to be the integrator of a broad range of evidence that identifies areas where interventions can be made to ensure that all students have the opportunity to reach their highest possible level of achievement. We see our assessment as only part of a fuller assessment and evaluation model that can assist. As the deputy has said, we believe that the most significant assessment is the teachers, but it needs to be complemented, and we provide the normative and comparative data that can assist them to know and guide them in their learning adaptation.

The third priority is to build capacity. We are committed to building capacity for the use of data through services to educators, parents, government and the public, and we’re proud to say today that there’s this culture of evidence that truly is out there in our school system that is leading to debate and discourse that simply wasn’t there because it didn’t have the foundation on which to have that discourse a decade and a half ago.

Fourth, we value research data. We’re committed to demonstrating that all our practices at the agency are grounded in research-based knowledge. You’ll see in the detail that our research is not only just about how to have the appropriate assessment tools, but it’s also focused on how to foster learning, because the real result of a proper, large-scale assessment is in fact the improvement of learning in our province. We are interested in identifying these gaps, working with the ministry and the educational community and adapting our teaching methodologies in order to foster greater learning.

The fifth priority is effective business practices. We try to strategically align our business practices to enhance the agency’s ability to focus on and deliver key priorities and commitments, and you can see the detail of that in our report.

The board’s connection to EQAO is through the chief executive officer, Marguerite Jackson.

Today, we are before you to present an update to the responses to the recommendations from the Auditor General’s 2009 value-for-money audit report. We want to begin by thanking the Auditor General for the professional quality of his in-depth, seven-month audit of our organization; for his observations, affirmations and recommendations that he has provided to us. The review is yet another strong endorsement of the quality and value of EQAO’s provincial testing program. I am proud to be associated with the agency whose work has a direct impact on the work of our schools, our boards and the support of students across Ontario.

If I might just be allowed an analogy, it seems to me that the government has placed great trust in the Auditor General to do assessments of different programs and services—this year EQAO—and to provide feedback to those organizations in order to improve. Quite frankly, that’s a parallel to the role of EQAO. We provide assessment and we provide data back so that the schools, teachers and students can improve on their performance. I think there’s a direct parallel in your trust in the Auditor General, who has challenged us to be the best that we can be.

We value significantly the process of assessment, the process of receiving this information, and have taken seriously those suggestions that have come before us because we believe it is going to help us improve our performance and, in turn, help us assist teachers, educators, parents and students in the province of Ontario.

At this time I’m pleased to introduce our CEO, Marguerite Jackson, to go into the detail of our response to the Auditor General’s report.

Ms. Marguerite Jackson: Thank you, Brian, and good afternoon, Chair and members of the committee. Thank you for the opportunity to speak with you about EQAO and about how we are responding to the auditor’s report about our agency.

To begin, I want to thank the auditor for the team that was assigned to our work. The aspects of designing and delivering a large-scale assessment program are complex and, in some cases, intricate. We very much appreciated the thorough and the professional approach that the audit team took in exploring every aspect of how we do our work. I must say, though, that after seven months we were wondering if they were ever going to finish.

We appreciate that in the report we did receive recognition for some of the things that we do well and we value the recommendations that have been given to us.

I’m joined today by three of my colleagues who really were the people who helped the audit team to understand our work and will assist me if you have questions that need some detailed answers: Robin Dafoe, who is sitting behind us, Richard Jones to my far right, and Michael Kozlow to the right of the chair. We’re very proud to represent this agency and we’re very proud of the work we do on behalf of the public, young people and educators.

This agency has existed for just a little over a decade. At its midpoint, about five years ago, we undertook a major initiative to modernize all of its processes and assessments. What the auditor saw is a result of that transformation. Then, as now, we benefited from the eyes of external advisers. We were determined to be best of class then and we will apply that same determination to the auditor’s recommendations.

We’re committed to delivering assessments that test the curriculum that teachers are helping young people to acquire. Therefore, we were very pleased that the auditor confirmed that our assessments fairly do that job. We’re also committed to providing assessments that are comparable in difficulty from one year to the next and are administered and marked in such a way that the results are reliable and valid indicators of student achievement in this province. This allows us to track improvement over time. We again were pleased that the auditor recognized that we’re methodical in our processes that we assign to making that happen, and further, that the stakeholders with whom the auditor spoke indicated that they agree that the test development processes we have in place do that job.

We appreciate that the auditor recognized our solid financial practices and also acknowledged that the agency has reduced its annual expenditures by over 20% in the last five years while delivering substantially the same services.

We see this report as a strong validation of EQAO’s provincial assessment program and our business practices. We see it as a validation that our practices provide an accurate measure of curriculum expectations, comparable year-to-year results, a reliable indication of student achievement, and valuable information for the government, parents, educators and the public.

1250

We also welcome the Auditor General’s recommendations and have closely considered how we might address each recommendation to strengthen Ontario’s assessment program even further.

You have before you a status report that describes EQAO’s completed and planned undertakings with regard to the auditor’s recommendations. I trust that this report provides confirmation that we have given thoughtful attention to the auditor’s recommendations.

We’ve taken steps to address all the recommendations, and in some cases we’ve already implemented the actions from the recommendations. For example, as suggested in the report, we’ve already added additional steps to the selection of schools to be visited by external monitors to ensure that all school boards and types of schools are periodically visited.

Secondly, we’ve already updated all of our administration guides to clearly highlight any significant changes in procedures from the past year and to clarify the potential implication of breaches to those guidelines.

Thirdly, we’ve introduced survey questions to the current grade 9 math test to determine the extent to which teachers use the EQAO tests as a part of their final term mark to students, with the notion that we might be able to determine whether doing so influences how students do on the EQAO test, particularly for those students in grade 9 applied math.

We value the Auditor General’s report, we’re pleased with the commendations, and we’ve learned from the recommendations.

We’re pleased that the auditor found general consensus among the stakeholders, including principals and teachers, that the tests are an accurate reflection of students’ achievement of the curriculum expectations.

The results on our tests provide evidence of how Ontario’s students are achieving the foundational skills of literacy and numeracy, skills that are essential to thriving in any other aspect of the curriculum.

I’ve provided you with a package of supplementary material, with a cover page that looks like this. Some of the slides in that package give you samples of the evidence we’re able to present to the government, to the public, to parents and to educators about student achievement.

The evidence we have about Ontario’s students gives us cause to celebrate. Yet when you look at it, it also compels you to ask questions, to challenge practices and to monitor even more closely individual students so that every young person may leave the school door at the end of the day with the exhilaration of confidence of having known that they are learning and growing and able to do what their friends are able to do.

EQAO is committed to contributing to a culture of conversation, decision-making and action based on evidence of achievement.

In conclusion, I’d like to emphasize three themes which I believe came through in the Auditor General’s report regarding the work of this agency: first of all, that our assessments are good indicators of student achievement in the areas where we assess; secondly, that the participation of teachers in every aspect of our test development process adds to the credibility of the work that we do; and thirdly, that the program we have put in place with these assessments, which focuses on measuring every student, has been a catalyst for improvement for thousands of students across this province. Indeed, if you look just at a period of five years, there are 20,000 more 12-year-olds who left grade 6 last year with solid literacy skills than five years ago.

Thank you, Mr. Chairman and members of committee, for this opportunity to speak to you about the agency and about the audit.

The Chair (Mr. Norman W. Sterling): Thank you very much. Before we go to questions, I would just like to invite the chair—perhaps if he would indicate to the committee whether he agrees with all of the recommendations made by the Auditor General and, if not, which ones he would differ with.

Mr. Brian Desbiens: You have a report before you. We’ve received and we’ve reviewed all of these. They seem to be reasonable recommendations that we have taken under consideration. There are a couple of items in there that we’ve presented our reactions to. I suspect we will get into those a little bit later. But in general, almost every single one of those, I think, are of value to us. There are a couple in the policy area where we might have some differences of opinion.

The Chair (Mr. Norman W. Sterling): Perhaps you can outline those during the question period—where those differences might lie.

Ms. Sandals?

Mrs. Liz Sandals: Let me dig into one where you might at least be able to expand, because there was the issue that was raised by some of the schools and this whole issue of, are special-education students and special-needs students exempted? Then, if they are exempted, how do you do that accounting when you’re reporting back on the schools’ results? That’s an area where I sense that you might have some comments. So let’s talk first of all about who would actually be exempted, because I understand that the schools are given some direction as to who should be and who shouldn’t be exempted.

Ms. Marguerite Jackson: A great deal of our processes rely on professional respect and mutual trust. The determination about whether a child can step up to a test or not at a particular point in time is really best made by the people who are closest to that child.

For exemption purposes, generally speaking, it’s understood that it might be a child who has a special- education designation. It could be a child who just recently arrived in the school and hasn’t had the opportunity to get into the flow. That being said, for the most part, special-education students should be participating in these assessments, because we offer programs of support for them. Indeed, we allow all of the modifications they would have in the course of the day to be provided for them when they’re taking the assessment. Basically, this operates on a basic principle: We believe that public schools should be accountable for every child. We don’t believe you should be able to set a child aside because you don’t want to talk about what you’re doing for them. There are very unique circumstances where children have specific and limiting disabilities that we recognize, from the dialogue we’ve had with the auditor, that we need to do more dialogue with the education community and find a way of reporting so that they see that as fair.

In summary, as a general principle, I would say that we see that we’re accountable for all children, and we should be reporting on how all children are doing. The reality may be that they may not achieve the standard, but they may achieve level 2 or 1, and you know that. So you’ve got some information to work with. The other reality is that we’ve tracked children from grade 3 to grade 6 to grade 9 in OSSLT. Children who are removed from the process don’t catch up. So this is my point: We have to pay attention to individual children, and we have to have evidence with which to do it.

Mrs. Liz Sandals: Just from conversations I’ve had over the years, my sense would be that MACSE, the Minister’s Advisory Council on Special Education, and particularly the Learning Disabilities Association of Ontario have been quite strong in advocating that special- needs students be included in the process, because of the fact that the parents want to know how their students are doing relative to the curriculum. I’m assuming that it would be pretty standard, however, that if you have a very high-needs child who’s in a self-contained class who isn’t participating in the regular curriculum, those children would be exempted.

1300

Ms. Marguerite Jackson: That would be understandable. The general guideline that I would use in conversations with the principal would be, “If the child is following the Ontario curriculum, then you should be very thoughtful about why you would exempt the child from the assessment,” because these assessments are based on the Ontario curriculum.

Mrs. Liz Sandals: Okay. I think what I’m hearing from you is that you’re satisfied with the policy which encourages people to include as many students as possible in the process and that the exemption process, if you were to pull those out of the school results, might be inclined to encourage people not to include all students in the process.

Ms. Marguerite Jackson: We also believe very much that accountability begins at the source, and we want the principal to talk about the school’s results. I think it’s quite acceptable that a principal would speak about it. For example, when I was principal of a school, I had some classes that were children who were not participating in the Ontario curriculum. It always was quite delightful to me when I talked to the parents that they were surprised we had those children in the school. But the principal and the teacher know the children, and that’s where that personal conversation should be. In no case should a child be exempted without the parent being party to the discussion.

Mrs. Liz Sandals: Another one of the issues that the auditor raised, and which you mentioned in your opening remarks, is this whole issue around grade 9 applied math, where there’s some evidence that students don’t necessarily take the EQAO test totally seriously. There’s a possibility of including it in the final marks, and students tend to take things that count more seriously than things that don’t count. So what I think I heard you saying was that you’re actually going to do a little bit more of a detailed analysis so you can get some valid analysis of what’s going on there. Could you tell us a little bit more about the details of that?

Ms. Marguerite Jackson: Yes. Ministry policy allows for an end-of-term mark to include other things than the tests that the teacher gives. We know, from the time the grade 9 test has been administered, that some teachers have used parts of the EQAO test as contributing to that final term mark. The policy, as it’s written, is either a decision of the school or, in some cases, school boards have made a decision. We know of school boards where they’ve said, “You will use a part of the EQAO assessment as part of the final mark.” But that decision is a local decision; it’s not our decision. However, as a result of the dialogue we’ve had with the auditor and the concern there is about how young people in the applied math program are doing, we have introduced this question on our survey to determine if, when the teacher does include it as part of the term mark, it makes any difference and contributes to how the child does on the EQAO test as well.

Mrs. Liz Sandals: Will you have any way of getting information about student grades or average student grades from the teacher in the questions you’re asking so that you can actually do a formal sort of correlation between inclusion and—

Ms. Marguerite Jackson: We didn’t ask about the teacher’s grade; we simply asked the question, “Are you using any part of this assessment as part of your term grade?” We do have, through our relationship and collaboration with the ministry, access to report card grades, but not right at the point when we’re doing our initial analysis. It’s a bit delayed in terms of doing that.

Mrs. Liz Sandals: Okay. It’s a really interesting question.

One of the things that comes up sometimes—and I don’t think this was so much an issue that the auditor raised; he maybe just noted it—is the whole issue around mandatory every-child testing versus random testing. Could you give us some of the rationale for the every-child testing and why that policy is in place?

Ms. Marguerite Jackson: Well, the rationale gets its genesis in the royal commission. It was very clearly a call from that commission. It seemed to them—in fact, I think I included the quote at the back of this—that it was reasonable to have a check on student learning at a few critical transition points and that parents should know how their child is doing relative to a provincial standard. So that’s the genesis of it.

Secondly, as we have worked with this data and as I’ve said in my summary comments, if you look only at the aggregate, you’d have lots of things to celebrate. It’s when you disaggregate and start to ask, “Who are the 30% of children who are not achieving?” that you really make a difference. If you don’t have every student’s data, you can’t do that.

The third thing is, across this province, if you sample, you have a very expensive program if you want to be able to give information back to every school. Generally speaking, if you sample, you do not give information back to schools. In fact, with our national and international assessments, we’re not even in a position to give information back to school boards; all we can do is talk about where Ontario stands relatively. But generally speaking, even though we ask the schools to participate, they’re not in a position to do anything with it other than to say, “Well, we helped to demonstrate that Ontario is doing very well.”

So from our perspective, the benefit of this program is all about learning: about children learning, about children mastering, about children being able to go on to the next stages. Having the kind of information that we’re able to present to schools gives them a point of conversation to change their practice, to focus in, to intervene and to make a difference for young people.

Mrs. Liz Sandals: Following up on that, then, and talking about the value of disaggregation, can you describe to us the information that’s going back to the local school level? Because what we see, especially if all you do is look at the median and don’t actually look at the EQAO website, is scores for schools and people making futile attempts to rank schools and doing all sorts of bad things with the data. What sorts of things do the actual schools get back that help them to influence either individual children or curricula?

Ms. Marguerite Jackson: The individual schools get a very detailed report that provides the aggregate information over time for their children. In relation to your question about including exempted children, they also get a report for the children who participated, so they know how they did on aggregate. They get sub-reports on gender—how boys and girls did; they get sub-reports on how children who are in ESL did; they get sub-reports on how children in special education did; they get survey results back that were completed by their students; and they get what we call an item information report, which shows every curriculum aspect that we were looking at and not only how the individual child did, but as a general how the class did on that curriculum expectation.

Mrs. Liz Sandals: So the principal and the teacher could get down to the point where you’re saying, “This class did well in these three strands of the math curriculum, but they’re rather weak in these other strands”? You can actually get down to that level with the individual child?

Ms. Marguerite Jackson: That’s right.

Mrs. Liz Sandals: So even if the child’s gone on to the next grade, the teacher would still be able to look back and see that for this child, this is where they’re okay in math; this is the area where their concepts are weak. I mean, that’s going to carry on as you move from grade to grade.

Ms. Marguerite Jackson: And generally beyond that, it’s our intention that this would be a school team talking about, “How is our school doing?” so that the primary division would look at how the children did at the end of grade 3 and say, “So in this area of the curriculum, if our children are not doing well, what should we be doing in grade 2 and grade 1?” It’s about the progression through school and, likewise, in the junior division.

Brian is just nodding to me to speak about a product that we provide as well.

1310

Mr. Brian Desbiens: There is a series of reports that we also provide to teachers and to the schools. This one I have is on the primary junior; there’s one on the literacy and there’s one on mathematics. They present the data, as Marguerite has just talked about. I was just teaching a grad class, and I used this as a teaching tool to a curriculum class.

The other thing that we’ve done, because we have an extensive outreach where we go and consult with teachers—included in these documents are also the observations about learning and then pointing to learning strategies, very specific learning strategies. These are significant tools that are available to teachers. If the method they’re using now isn’t quite working on that particular curriculum item, they are able to receive assistance from this.

This is the role that we have in terms of working with the profession, not just to use the data. The data triggers a discussion, which then triggers another discussion about where we can go then to adjust our teaching and teaching methodologies. We gather that from visiting the schools, dialoguing with them and pointing out the areas that need improvement.

Mrs. Liz Sandals: Thank you very much.

The Vice-Chair (Mr. Peter Shurman): Thank you. To the NDP. Mr. Marchese?

Mr. Rosario Marchese: Yes. I was thinking.

The Vice-Chair (Mr. Peter Shurman): That’s okay. I do it sometimes too.

Mr. Rosario Marchese: Liz Sandals knew the question I was going to ask. You helped a little bit. My question had to do with how teachers use the results. I’ve heard the minister use the word “diagnostic.” I’ve heard other people use the word “diagnostic,” and I don’t see it.

Then you described, Marguerite, how data is used, so they get information in the aggregate: students, special education needs and whatever information you provide about that—I don’t know how that helps diagnostically—discussions about gender reports, how girls and boys do, presumably, in a general way, in the aggregate, who participated and who did not, FSL.

Then Liz tried to sort of get you to the point of saying how teachers actually use it, and I still don’t see that. I don’t quite get how teachers actually get the results and then use that information to help that student who did the test subsequently. So I’m not quite sure whether that information is used diagnostically. Maybe you could define what the minister means by it. How is it used to help the student who has gone through the test, and how do you then use it diagnostically with the students who are about to take the test?

Ms. Marguerite Jackson: I’ll start and then, Rick, I’ll ask you to help me out.

It’s used diagnostically in at least two ways. One, in terms of the school learning community, what the teachers tell us—and we ask the teachers who administer the tests; these are the grade 3 teachers, and we had 8,600 of them respond last year. We asked them if they used this to identify areas of strength and areas for improvement in the grade 3 reading, writing and math program. Seventy-nine per cent of them told us they used it that way. They told us that they use it to inform how they do their program planning; 66% told us they do that.

The principals tell us that they use it to engage their teachers in discussion. So if you’re a principal of a school and you have almost all of your children leaving grade 3 or grade 6 confident in how to read, write and do the numeracy that we wish them to do, you’ll have a different discussion with your teachers than you will if you’re from a school that is in this supplementary package that I’ve shown you, where year over year you have less than half of your children finishing the grade able to do the work. If that’s the situation I have as a principal, I want to say, “What is it that our children are not able to step up to?” We provide them with a breakdown directly linked to the curriculum, which I’ll ask Rick to speak about in a moment, that says, “This was a curriculum expectation. This is how many children, and these are the children, who were able to do it or not able to do it.”

In terms of the individual child, which you asked about in terms of diagnostic, what we know from tracking children from grades 3 to 6 is what the teachers are telling us is indeed happening, because we are now able to tell you, if a child didn’t achieve the standard in grade 3, how many of them did when they got to grade 6. I think this is tremendously rewarding, for teachers and principals of schools to say, “We looked at the results when we got them in grade 3. Our children weren’t at the standard. We’ve done this, this and this, and look: They’ve left us at the end of grade 6”—

Mr. Rosario Marchese: So we’re trying to track that child—Johnny, let us say—in grade 3, and individually, we know we all are different. It’s not an “in the aggregate” kind of discussion; it’s how Johnny’s doing. So the teacher the next year in September should be looking at something that says, “Here are the problems. Here is the learning problem,” because it’s not so generic as particular. Does that teacher in September have the tools to know what to do?

Ms. Marguerite Jackson: They get it from us. I’ll ask Dr. Jones to speak about what they get from us.

Dr. Richard Jones: Thanks very much, Marguerite.

Marguerite talked earlier about the link of Ontario’s provincial assessments to the curriculum. I do have copies of the curricula here, and we can circulate them if you’d like to have a look at them. I’ve highlighted the areas that indicate for the various areas in reading, writing and mathematics where the overall expectations for learning are presented, as well as the specific expectations. We do develop our assessments with regard to those specific expectations. We can talk more about that later if you wish. I can certainly circulate this if people would like to look at the document.

Mr. Rosario Marchese: It will be difficult to have the time.

Dr. Richard Jones: Just in case.

So, directly linked to the curriculum, Marguerite also mentioned that the vast majority of teachers do indicate to us through the questionnaires, through our surveys, that they use EQAO data to look at areas of strength and where there are areas of concern—

Mr. Rosario Marchese: Sorry, sorry. The individual teacher gets the data, and the data that she or he gets is what Marguerite was talking about in terms of “in the aggregate” information; every teacher gets that, not a number, but—

Dr. Richard Jones: In many forms and at various levels. We do have, for example—

Mr. Rosario Marchese: Do you have a copy of what you give individual teachers?

Ms. Marguerite Jackson: Does anyone have an IIR with them? Do you have an IIR with you?

An IIR is an item information report for a class. So for every child, it tells how the child did on every item in the assessment.

Mr. Rosario Marchese: So it basically says number 3 or 4 or 2. Is that—

Ms. Marguerite Jackson: Well, we give them the test—the test is released so they can see the question that we were asking—we tell them what the curriculum expectation was, and we say, “On this question, which was testing this curriculum expectation, Rosario did this well, Michael did this well, Marguerite did this well,” and we tell them how the class as a whole generally did. We send that to the school by early September so that it’s there—

The Vice-Chair (Mr. Peter Shurman): Excuse me. Could I just ask that you send a copy to the clerk of the committee so it can be distributed to everybody?

Mr. Rosario Marchese: Absolutely. So you send it to the principal, who distributes it to each individual teacher. Is that the way it works?

Ms. Marguerite Jackson: Our linkage to schools is through the principal.

Mr. Rosario Marchese: Through the principal. And we assume that the principal, then, gives all that information to every teacher. Perhaps they have meetings to talk about this. Is that—

Ms. Marguerite Jackson: We have good information that they do, because we do regional workshops where we bring principals and teachers as teams—we just finished one yesterday—where we bring a principal and the teachers to the discussion and we work with their data—

Mr. Rosario Marchese: And do they get—sorry for interrupting you, but I’m very specific about what I’m looking for. Do they get strategies from you in terms of, “Here are the weaknesses”—because in the subjective portion of your test, I don’t know what strategies you provide to teachers. All I get is a number. How do I help Johnny—because I’m still focusing on him—to be able to improve that skill? What strategies do you recommend to improve that skill?

Ms. Marguerite Jackson: We provide them with a summary of results and suggested strategies that will help children step up to competency. Our role is not to do that type of work in schools; Mary Jean has a team of people who help teachers. But because we know the assessment and we know what the question is checking, we know what should have happened in terms of the child being able to do the question. So we do provide that. Then Mary Jean’s team picks it up and does in-depth work, as do other people in school boards.

1320

Mr. Rosario Marchese: Okay. Before you both get to it, Professor David Johnson argues, in his study An Update to Signposts of Success, that 40% to 50% of the variation in schools’ average test scores can be explained by schools’ socio-economic environments—50% is socio-economics. Then he says that the remaining variation reflects factors at the schools themselves, which are the principals, the teachers and other staff. So, 50% is already socio-economic; the tests are not going to help us very much with that.

We know that the other half are having difficulties of sorts. You’ve got mental illness issues, which are getting bigger in our educational system and getting bigger in society in general. You’ve got kids who have fetal alcohol spectrum disorder, which most teachers can’t identify because they don’t understand it; most doctors don’t know it. There are about 100,000 to 200,000 people suffering that; I’m not sure how many of those are students. You’ve got kids who are excessively poor, who bring problems from home into the classroom. Some have substance abuse, because some people drink and others take drugs, and that affects the learning of those kids.

So we’ve got some serious problems. How does the test, or the system, deal with these issues?

Ms. Marguerite Jackson: Let me just say that we recognize that all children don’t have the same supports and advantages. But if public education exists for any purpose, it’s to level the playing field. It would seem to me that the supports we put into our public school system are designated to offset those disadvantages and help to raise the bar as well as level the bar.

That being said, some teachers have a much more challenging job to do, and they deserve to be highly recognized and regarded for the successes. We have plenty of evidence that there are situations—because we have the evidence—where those factors exist and children are thriving. We have other evidence where the factors are the opposite of that and children are gliding but they’re not striding.

That’s what evidence does for you. It challenges you to say: If a child’s in a school where they’re coming from circumstances that are what all of us would desire, why aren’t all the children leaving grade 6 with the confidence that they can read as well as we expect? This is not university level; this is basic reading and writing that they need to go on through their journey of school.

Mr. Rosario Marchese: Let me ask you, Marguerite: The test scores are going up every year and now graduation rates are going up every year. Yet we have professors who are saying, “The product that we get is not very good.” The results are better, because these are standardized tests. They obviously evaluate writing, multiple-choice questions, all sorts of different issues. So one would expect that the overall improvement, including graduation rates, should be reflected in the quality that you get once you go to university. Yet professors are complaining that nothing has changed.

Ms. Marguerite Jackson: I can’t answer for them, but I would say to you that there was a study out a couple of weeks ago about young people coming in with mathematics. I read it with interest, because they were saying that about 58% of the young people coming into university had the kind of mastery that they expect in mathematics, and I thought, “That’s interesting,” because the kids who are leaving grade 6 with mathematics are around that in terms of those who have high competency.

A kid can graduate from school without having the level of competency we’re asking for, because you can graduate with a C. But a C doesn’t take you into life in the way that an A or a B does. We believe, for 12-year-olds in particular, that nothing less than an A is acceptable in terms of being able to read, write and do the numeracy we want. As they get into adolescence and high school and start to diversify, it’s a different discussion.

Mr. Brian Desbiens: If I might respond: As a former college president who received those students, I think we have had historically a challenge that students have been moved through the system without having the competencies to be able to perform. I still think we have a challenge if we’re trying to get to that 70% level. But the chances of the ability to be remediated are significantly improved through the identification of the competency gaps if we identify them earlier. What’s terrific about this—and there’s a whole conversation—is that we’re starting now with the early years to recognize that there are people coming to us with differentiated skills before they even get to school, and so we’re doing the early years piece. We know, when they’re going to secondary school, that there’s differentiation. But we’re doing it specifically. Now we know what the math competency issues are and the literacy and reading problems are. Before, we were testing students, quite frankly, who were reading at the grade 8 level coming into post-secondary institutions. We have a much better capability of being able to provide interventions in order to help student success in secondary schools because of this information being available. I think we’ll see the results of this.

I’d like to come back. One of the things that’s really exciting about this is that we do track the demographic piece that you’re talking about. You presented a tremendous argument that supports our belief that you can’t compare schools because they differ substantially. But what we have seen in here are schools that we visited through our outreach to learn from the schools. There are schools that are making a difference where special-needs students’ performance is significantly improved because of interventions that have been targeted. We know that new Canadians have improved substantially in their performance over time. We’re starting to see significant improvements in other areas. So it’s through having the evidence to do the interventions—I guess I’m a little more optimistic that teachers make a difference. There are those challenges there, but what we have now is the evidence that truly, if we can identify a school that’s in trouble through the different interventions that take place, we can make a difference early on and they will have a better chance of being successful and graduating from secondary school and then being successful in post-secondary.

The Chair (Mr. Norman W. Sterling): You have about three minutes left.

Mr. Rosario Marchese: He wanted to answer.

Dr. Richard Jones: I just wanted to add one thing with regard to something that Marguerite mentioned in her opening remarks, and that was the importance of looking at achievement over time and being confident in the comparability over time. Certainly, we can say with confidence that the standards are the same from year to year. There’s a whole process that’s in place to initially establish the standards and we have very stringent processes that we use year to year to be sure that we can do that equating from one year to the next, so that when we talk about increases from one year to the next, we’re talking about real increases. We’re quite confident in our ability to make those kinds of statements.

Mr. Rosario Marchese: Brian, you were talking about interventions in special education. What do we get out of the test that we glean from to be able to bring about different kinds of interventions? What kinds of interventions are we introducing that are either helping special-education kids learn better or creating a different environment for them to learn better or are giving techniques or strategies to teachers to teach better? What kinds of interventions are we talking about?

Mr. Brian Desbiens: There are a lot more experts than me here. Perhaps we should go over here.

Ms. Mary Jean Gallagher: If I may, the work that the student achievement division does with schools across the province is in fact focused on exactly what those interventions are for a whole host of young children. I would point out that two years ago the data across the province showed that there was a significant improvement in writing skills at both grade 3 and grade 6. If you looked at the data and disaggregated it, the group that had the greatest spike in their achievement that year were students with special needs. When we went back and looked at it, schools, through their school improvement planning process and boards, had set writing as a focus for their activities in terms of building the capacities of teachers. A number of boards really began to focus on much more specific work with students and much more specific tasks with students around writing skills. They started to be more specific in the teaching of different genres of writing, to be much more structured in expectations and tasks given to children in their schools around writing and understanding that you can write in different persons’ voices and a focus on that kind of much more detailed, structured program.

What we learned from that as we watched boards implement those teaching strategies was, in fact, that it was good for all students but it was absolutely necessary particularly for the students with special needs. That’s just one example of where the data and the analysis of the IIR telling boards where the gaps were leading them to say, “All right; this year, our school”—or board, whatever—“is going to really push the limit on what we can do about writing and how we can change it.”

1330

Quite honestly, the next step of our process around that was going back into schools and areas where they had, in fact, achieved those gains and working with those teachers and schools to identify what they were doing that was so successful so that we could then take that information and share it. We’re using those same processes in all the areas.

Mr. Brian Desbiens: If I might give one concrete example from my own experience: As mentioned earlier, there’s a gender gap that has been identified. We identified a school in northern Ontario—I really like getting out and visiting, so we visited this school because for their special-needs students, the gender gap wasn’t as great, so we wanted to learn from them. We identified it through the data and we went and visited the school.

It happened to be a laptop elementary school. What they were using was the computer and helping little boys learn how to write through the computer, which seemed to be much more amenable to boys learning how to write. They had used the technology to act as an intervention, both for the gender issue as well as the special need.

It’s using data to identify who’s doing right things—not just who’s not doing things but who’s doing right things—and then going and learning from them what kinds of interventions can make a difference.

The Chair (Mr. Norman W. Sterling): Thank you very much, Mr. Marchese. You’ll have another opportunity. Mr. Ouellette?

Mr. Jerry J. Ouellette: Thank you for your presentation. Brian, you had mentioned that you were, I think, president of a college. Were you president at the time prior to and during the introduction of the EQAO?

Mr. Brian Desbiens: Yes, I was: when it came in.

Mr. Jerry J. Ouellette: I was on the board of governors for Durham College prior to this. One of the standard questions that came up on a regular basis at the meetings was entrance exams for students coming into the college sector. The main reason for that was because the level of graduating students was not consistent. You could take somebody with grade 12 or 13 from one school which would not equate in any way, shape or form with another school. Do you find more consistency with the graduating students and their abilities being more consistent as they graduate with this?

Mr. Brian Desbiens: First of all, I’ve been retired for six years, so I can’t speak to today. I don’t know, would be my honest, real answer from a post-secondary perspective, because I’m not interacting with the people who are doing the assessment.

I would expect that there would be. I would expect that there would be a clearer understanding. Remember: What this does also is provide a basis on which to have some conversation. For example, mathematics is one of the huge areas, especially with Durham, which wants into technologies and things. Our math teachers can have conversations with the secondary school teachers, and they have a basis on which to have that conversation, based on some of the results. Math has changed dramatically in the secondary and elementary schools.

The value of EQAO is that it gives a common basis for a common language. We do not have exit examinations in Ontario. That’s not part of the model, unlike other jurisdictions, such as in the US. But we do know that we have our grade 9 math, which gives us a chance for conversations in the secondary school, and we do have the literacy.

Personally, I think that the literacy tests are very important for mathematics. You cannot do math problem-solving without being able to read with comprehension. I think there will be improvement; I just don’t have the data to be able to give it to you right now.

Mr. Jerry J. Ouellette: Right. Brian, you also mentioned the gender gap, which was particularly discussed earlier on. When I speak to teachers, there is an expression of concern that the grade 3 testing is gender-biased in that it’s extremely difficult for males to comply with the test. Are you finding that? Are you finding results that would comply with that?

Mr. Brian Desbiens: Actually, I might turn that over. We’ve just done some research in the gender bias area.

Ms. Marguerite Jackson: I think maybe we could ask Richard Jones to speak about how we attempt to make the test questions gender-free.

Dr. Richard Jones: One of the really important components of our test development process is the use of various educator committees. We bring together individuals who have content expertise and expertise with regard to sensitivity issues, cultural bias, gender bias and so forth who review all of our reading selections, for example, for that sort of issue, and all the other issues as well, and provide us with feedback on those reading selections as well as the actual test questions themselves. As part of the development process, we try to weed out those particular items before they get too far down the road and before they become an actual component of the assessment. That goes an awfully long way in that particular direction.

We also do some statistical work on our field test questions—Michael might be able to speak a little bit about that—that also has a look at bias-related issues. We can potentially see them through statistics, again, before we get test questions that find their way onto an operational test, one that actually counts for students’ scores.

Mr. Jerry J. Ouellette: Okay. One of the areas—I’m not sure; would they have a copy of that or not?

Mr. Jim McCarter: Do you have a copy of the auditor’s report?

Mr. Jerry J. Ouellette: The auditor’s report, where it shows the statistics. Figure 2: Grades 3 and 6—Percentage of Students Achieving Provincial Standard. I don’t know if you’ve seen what I’m referring to.

Mr. Jim McCarter: It’s on page 132.

Mr. Jerry J. Ouellette: Page 132. When you look at the statistics there, if you compare the grade 3 math averages with those same students moving to grade 6—for example, the 1999-2000 grade 3 students would be in grade 6 in 2002-03—the math average has moved from 57% to 53%. In the following year, it’s the same situation; it moves from 61% to 57%. In the following year, from 58% to 60% and from 57% to 61%. If you look, however, at the French comparison, in the first year it’s from 41% to 66%, and then they go from 40% to 70% and from 47% to 74%. There’s quite a dramatic change in those gradings.

Is there some explanation or understanding that we can gain from that, or is there something that you’ve learned from that that you’ve been able to implement through the school system?

Ms. Marguerite Jackson: Well, to begin to speak about the English cohort, we do now track how the children do in grade 3 and grade 6 because we’re able to track them child by child. This is an issue we have identified, that the cohort is not moving as high as we would want it to in grade 6 math. Understandably, the math concepts are more challenging as you get to grade 6, so—

Mr. Jerry J. Ouellette: But the English compared to the French is a dramatic change. When you look at the changes in the French comparison for the first year, they go from 41% to 66%. In the second year, they go from 40% to 70%. In the third year, they go from 47% to 74%, and it continues on where there is a dramatic difference between the French and the English. I just wondered if there’s some explanation for that.

Ms. Marguerite Jackson: You may appreciate that the French curriculum is a curriculum for the francophone community. We have had lots of conversations about this. We know that one of the contributions we’ve made to improve learning is that teachers are paying closer attention to the curriculum, and this is the case in both the English and the French cohort.

In the French community, although it is very diverse in terms of being spread across the province, it’s a smaller cohort of students and a smaller cohort of teachers. Therefore, you have more focused opportunities, perhaps, to bring all of your teachers to the same discussion, the same understandings about approaches, the same implementation of practices that will make a difference.

Certainly we know that the francophone community, as the anglophone community, has been very focused on helping kids to do the best they possibly can, but they may have a bit of an edge in terms of getting their teachers together and getting all of their teachers on the same page at the same time.

Mr. Jerry J. Ouellette: Okay.

The Chair (Mr. Norman W. Sterling): Could I just ask for a point of clarification? You talked about the francophone—

Mr. David Ramsay: Curriculum.

The Chair (Mr. Norman W. Sterling): Curriculum. Isn’t the curriculum for math in the francophone schools the same as it is in the English schools, and are the tests the same?

1340

Ms. Marguerite Jackson: The core principles would be the same. From here I’ll pass it to Mr. Jones to speak about the specifics of how the curriculum expectations are written.

Dr. Richard Jones: I must admit, I really couldn’t comment on the specific details except to say that yes, I think the underlying concepts would be the same, but perhaps there would be some differences in sequencing, for example. There may be certain parts of the curriculum that are given more emphasis at certain times than others. The end result would likely be the same at a certain point, but the sequencing would be different at different stages.

The Chair (Mr. Norman W. Sterling): So you don’t know if the test is the same test?

Dr. Richard Jones: They are different tests. They’re developed totally separately. We have French-language development and English-language development. We go to great lengths not to compare English to French for that reason, because there is the curriculum difference.

The Chair (Mr. Norman W. Sterling): So they’re not the same tests?

Dr. Richard Jones: They’re not the same tests, no. In many other jurisdictions, the French-language, the Spanish-language one or the second language one is a translation of English, but in the case of Ontario, they’re developed totally separately.

Mr. Brian Desbiens: Two different curricula, two different tests.

Mr. Rosario Marchese: I didn’t know we had two sets of curricula.

The Chair (Mr. Norman W. Sterling): The other question is, in the francophone system or the French school system, do they have individual math teachers? Or is that taught the same way as it is in the English system, where math is taught by the same teacher who teaches English?

Ms. Marguerite Jackson: It depends on the grade level, I think. But maybe I could ask François Lavictoire to speak about this, because he is our French-language coordinator.

Mr. François Lavictoire: François Lavictoire, the French-language coordinator at EQAO. French-language schools are structured the same way as the English-language schools: The grade 3 teacher teaches English and math. In the secondary schools, then they specialize. The French secondary schools are very small, and often, to complete a schedule, a teacher will teach outside their specialty. But they try, for languages and math, to keep those people in their specialties to make sure that they do get the best programs possible.

Mr. Rosario Marchese: What are the curriculum differences?

Mr. François Lavictoire: For example, the French-language population, working with the ministry, has decided on introducing the poem in grade 4 as opposed to grade 3. So when we look at the assessment of the grade 3 level in English, we do have a section that deals with poetry but not on the French. That would be picked up in grade 6, for example, because the program covering poetry starts in grade 4 as opposed to grade 3. By the end of grade 8, they will have basically covered the same things except it’s covered at different times.

Mr. Rosario Marchese: But what’s the rationale in terms of why we do it in different grades? Is there such a rationale? Or is it that they just do it differently?

Mr. François Lavictoire: It’s done differently. A lot of it is cultural also. The French-language schools have a mission to really make sure that the culture is developed. There is not necessarily the same requirement in English schools. In elementary schools, for example, there is a lot of concentration on developing these speaking skills, because not all students who come into the program are necessarily fluent in the language, so before you can actually get into the—

Mr. Rosario Marchese: By the way, that applies to English.

Mr. François Lavictoire: But they do have ESL programs that are structured differently than the French-language program in elementary schools.

The Chair (Mr. Norman W. Sterling): I think we can understand language; we can understand that part, but what about mathematics? Why would there be any difference between a grade 6 program for an English student and for a French student?

Mr. François Lavictoire: Again, the core, it’s the approach that is different. Math is tied a lot to language when we ask questions also, so it could be tied to language. The ministry—

The Chair (Mr. Norman W. Sterling): We’re trying to figure out what’s the difference in these things. Now we find out they’re different tests. That’s quite an explanation. I’m astounded that we have different tests because I would like to have some kind of idea as to how the general populace is working and whether or not the francophone system has a better system for mathematics than the English system.

Mr. Rosario Marchese: How do you compare them?

The Chair (Mr. Norman W. Sterling): Yes.

Mr. Rosario Marchese: I don’t mind the differences, but how do you compare them?

The Chair (Mr. Norman W. Sterling): I’m sorry, I butted in on Mr. Ouellette. Mr. Ouellette?

Mr. Jerry J. Ouellette: Any questions?

Frankly, when we’re dealing with this issue of parents making decisions on where their kids would go to school, if their focus is to deal with mathematics and they look at the mathematics marks for the grade 6 students, it may influence where that is, because everybody believes that it’s a standard throughout the entire system. What we’re finding out is that it’s not, which is good to understand.

A question came up earlier on, I believe from our AG, regarding potential tracking for teachers or principals. Is there any information that would indicate whether one principal, with one school, would have that school’s marks—because there’s a rotation of principals, as I’m sure you’re aware—and the increase or the changes in marks seem to follow those individuals because they’re implementing different programs? Do you keep statistics like that or use that?

Ms. Marguerite Jackson: That’s not in our jurisdiction; that is the jurisdiction of a school board to manage.

Mr. Jerry J. Ouellette: Okay. Would it be the same, then, if it’s an identified school that has consistently low marks? Would there be allocations that would come into play to ensure that that school would be brought up to a standard that was found in that area?

Ms. Mary Jean Gallagher: As a matter of fact, we’ve had a very specific program operating for a number of years at the elementary level and have initiated one at the secondary level that focuses on assisting those schools that do have very low achievement. At the elementary level it’s called the Ontario-focused intervention partnership, and in 2004-05, when it began, we identified schools where fewer than a third of the students were achieving at a provincial standard on EQAO indicators. That, at that point in time, represented 19% of the schools in the province of Ontario. The number of schools where students have achieved at that low of a level has now been reduced to 6%, I think, in the last round.

There is a very specific intervention program in which we’ve engaged. Some of our OFIP support goes to all school boards, because virtually all school boards have some low-achieving schools, but then we have very specific intervention programs for those schools that are low-achieving. In fact, we’ve now upped the ante. We consider low-achieving schools to be those where fewer than 50% of the children are making the provincial expectation, because that gives us a larger number of schools to work with. We’ve eliminated many of them.

For a number of years, our student achievement officers have been working with those schools. We’ve provided resources and supports in terms of additional book materials etc. to focus on the reading and the literacy. We’ve had student achievement officers working with the staff in the school to look at how you would engage in those strategies that actually do make a difference.

I would point out as well that our OFIP schools are a great success story from the point of view of demonstrating that demographics need not be destiny. Many of our OFIP schools are the schools that have higher levels of challenge in terms of poverty or English-language learning or higher numbers of students with special needs. We have evidence in the success stories of large numbers of those schools that staff are becoming increasingly successful at ameliorating those challenges.

Mr. Jerry J. Ouellette: Good. One last question that I have is, are you noticing, particularly for grade 9 students, where the EQAO testing is a component of their final mark, a difference in the commitment? As the kids grow older—and quite frankly, I’m going to rush out of here and go do my parent-teacher interview with my grade 9—are you finding that there’s a commitment? Because once they hit high school, they realize that this doesn’t mean anything, so why should I try? However, if it means something to them, are you seeing a difference?

Ms. Marguerite Jackson: We don’t know for sure; that decision is made at the local level. But we have introduced a question this year of teachers and we will be using the answers to that question—“Have you used any portion of EQAO as a part of the final mark?”—to see if there’s a correlation between a teacher doing that and the result that a child gets. I would suspect there may well be.

Mr. Jerry J. Ouellette: Thank you.

1350

The Chair (Mr. Norman W. Sterling): I’m going to allow Mr. Shurman about five to seven minutes, because of Mr. Marchese’s intervention.

Mr. Peter Shurman: Thank you, Chair.

I’d like to go to a different area. A couple of months ago we did a pre-budget tour around the province. We get presentations, as you well know, from an awful lot of interest groups, both public and private. I just keep a running narrative on everything that I hear, and something struck me, so I consulted it. What I noticed was a bullet point under one of the teachers’ federations that says, “Don’t cut our members. Cut EQAO and numeracy/literacy for cost savings.” Then there’s a little editorial comment—I won’t read it into the record—that I put in there.

Having said that, I’m interested in whether the office seeks to address what is apparent antipathy, or at least some push and shove, between some of the teachers’ federations toward the EQAO mandate. Is that something that you observe, and, if so, how are you addressing it? You’ve talked in some pretty glowing terms here over the last little while—and, let me hasten to add, I think quite well and very professionally—about the great relationships you have with teachers. By the way, I don’t question that either, but I suspect they are more one to one than in an umbrella way with groups. Can we hear some comments on that?

Ms. Marguerite Jackson: Actually, I would say to you we have great relationships with the teachers’ federations as well. We just happen to have a different point of view.

Mr. Peter Shurman: Have you thought about running for office?

Ms. Marguerite Jackson: I met with Sam Hammond earlier this year, in January, prior to the bulletin that you’re speaking about. He advised me that he had a motion that had been passed at his general assembly that was going to result in a magazine that was going to speak about assessment from the perspective that his members had given him. I shared with him the kind of information I’ve shared with you in terms of what teachers say to us. We always invite them to any of our symposia where we’re doing training sessions about data. We also have them sit on an advisory council that meets at least twice a year to look at all of our processes and to advise us on what they see as acceptable or not.

In particular, I’d mention to you that about five years ago we undertook an initiative to really listen to people, because, as you may be aware, early in the agency’s history there was a lot of dissonance about children being under stress, interfering with what a teacher was trying to do, teachers teaching to the tests. So we did a lot of listening to people and we implemented some changes as a result of what we heard. For example, we made the tests shorter. We didn’t think that an eight-year-old needed to spend 12 hours for us to decide whether they could read and write as we would expect an eight-year-old to do. So we reduced the tests substantially.

It was at that point too that we said to the teachers, “Our commitment to you is that the tests will come directly from what you’re trying to do. They will link directly to the curriculum. There will be no surprises here. These are the expectations of the curriculum. We’re going to give children challenges to see if they can demonstrate that they’ve learned this.”

So I understand that everybody is not with us, but I also understand that the schools don’t just belong to those of us who are teachers. Others have the right to have confidence in what’s happening for young people. In particular, parents have asked to know, “Is there a reference point that I can have confidence that no matter where my child goes to school in this province, completing this, I know that my child is getting the best shake out of this school system?”

Mr. Peter Shurman: Well, you know, I don’t think you’d get any argument from me on that, and I suspect you wouldn’t get any argument in this room on that. I think the argument, if there is one—

Mr. Rosario Marchese: He’s a friend.

Ms. Marguerite Jackson: Okay.

Mr. Peter Shurman: Yes, that’s true. Where it gets down to a bit of push and shove is if you’ve got some antipathy on the part of teachers’ unions, federations, whatever, and you’ve got some teachers who are very supportive, as you’ve pointed out in various of your presentations and in some of the answers to your questions, and you’ve got some teachers who are on the other side of the fence, saying, “This is somebody breathing down my neck,” for want of a better term. Then what is necessarily there is, as you’ve pointed out, some positive, some negative and the real chance for inconsistencies, which in some degree have been pointed out in the course of this report and in some degree you have taken at face value, and you’ve tried your best, obviously, to even the thing out.

Give us a more expanded view, based on that foundation I’ve laid, for what steps you’re taking to ensure that there is consistency in how teachers view what it is you have to do, so that in turn they apply it in an even-handed way, because that obviously affects results.

Ms. Marguerite Jackson: In my experience, life is mostly about relationships. What we have done is try to put a face to the agency that goes beyond a piece of paper.

We have a very small group of five people who spend their lives travelling the province, meeting with principals, teachers and parents, helping them to see the evidence that we have, talking with them about the way in which this evidence can be helpful to them, and listening to them, indeed, and learning from them.

We’re doing the same with the federations. The people who lead the federations have been teachers at one point in time. They have the same kind of determination that children will get the best out of public education as they can. They’re very concerned about using just a small piece of what education is about to say, “This is what a school is like.”

We’re concerned about that too. We know that schooling is much more than reading and writing, and we don’t believe these results should be used to say, “This school’s better than that school.” We’re working with them on that. I think we can find ways of saying people have a right to know what this school is about—

Mr. Peter Shurman: Well, sure, but how far can you get if what you’ve got is, at the first sign of financial trouble, the instantaneous response, “We’ll get rid of EQAO”? Because that’s what we’re talking about. That’s not me paraphrasing; that’s a quote.

Ms. Marguerite Jackson: I don’t think this is the first time that has been said. We put a lot of dollars to education. I don’t think that $15 per student out of a $20-billion budget is very much money to have information that helps us know how to be sure that every child is getting the best shake out of it.

Mr. Peter Shurman: Okay.

Mr. Brian Desbiens: If I might just add: I’ve been extraordinarily impressed at EQAO at the engagement of teachers in the process. There are thousands of teachers who are involved in test item development, administration, assessments, expert panels—at all levels.

I suspect, in your own community, that not everybody loves you as a politician and that you probably have a few detractors—

Mr. Peter Shurman: Oh, you’ve been to Thornhill.

Interjection.

Mr. Peter Shurman: I know, Yasir, they all love you.

Mr. Brian Desbiens: What’s important is to have the level of conversation. Last year we went and visited, as chair and executive director, with the unions, to listen to them and hear them. But the reality is there’s a rhetoric at a particular level, quite frankly, that I don’t think is going to necessarily go away.

The conversations are still at that level with them, the conversations with teachers, school boards and principals, but what we’re really focusing on as a board, quite frankly, is parents and students, not just educators. We really have set out a priority to talk with parents, to have a better understanding of the value, because they’re the ones we’re accountable back to. They’re the ones who have asked for this accountability, and I think we have a ways to go there.

I absolutely respect the union and their role, but I think the value here is substantially greater.

The Chair (Mr. Norman W. Sterling): Okay. Ms. Sandals and then Ms. Van Bommel.

Mrs. Liz Sandals: I just couldn’t resist throwing in a comment, as the person who has done the education stakeholder consultation on the grants for the last several years, that the comment that the first thing to do would be to cut EQAO is actually an annual comment. It isn’t actually related to the economy. And you can tell that the government actually didn’t listen to it.

Mr. Peter Shurman: The government didn’t listen to any of those, but that’s normal too.

1400

The Chair (Mr. Norman W. Sterling): Mrs. Van Bommel.

Mrs. Maria Van Bommel: This has been a very interesting discussion.

Earlier you mentioned something about how boys’ writing skills seemed to have improved when they were given laptops; I know boys like technology and that. But it took me back to a time when my children were in school, which is before EQAO, and there was a fairly broad discussion around the fact that girls were not strong in maths and science.

Mr. Jones, you said that your questions are now trying to take out a gender bias. That being the case, can we tell if the girls are getting stronger in their math skills? Or have you kind of taken the questions and taken that out so that you can’t tell whether the girls are actually getting stronger in the maths and then, hopefully, the sciences? I guess that’s the first part of the question.

Ms. Marguerite Jackson: Actually, the girls, all the way through, are head-to-head with the boys, or the boys are head-to-head with girls in mathematics.

Mr. Rosario Marchese: Until grade 10, when streaming happens. That’s where some of the separation happens.

Ms. Marguerite Jackson: We don’t have mathematics at grade 10, but there isn’t any reason why the girls wouldn’t choose mathematics based on the information we have of their foundational mathematics skills.

I think those initiatives that you mentioned from 15 or 20 years ago have definitely made a difference. There are some of us who think that maybe similar initiatives would work for boys with reading and writing.

Mrs. Maria Van Bommel: You are able to tell definitely that girls are getting stronger in the maths?

Ms. Marguerite Jackson: Well, we’re able to tell that they’re as strong as boys. We’re not satisfied with where all girls are or with where all boys are. For example, in the last year’s grade 3 math test, 69% of the girls met the standard, and 67% of the boys did. This is across the province.

Mrs. Maria Van Bommel: You are able to gauge if there is, say, to be a drop in one side?

Ms. Marguerite Jackson: Yes.

Mrs. Maria Van Bommel: When we talk about taking out the gender bias, you’re still able to gauge if something is going wrong or changing, if there’s a shift? Okay. What would you do if there were to be a shift? How do you address that, then?

Ms. Marguerite Jackson: If there were to be a shift in terms of—

Mrs. Maria Van Bommel: If the boys started to drop in their writing skills, or if the girls started to drop in their math skills, what would you do then? What does that tell you?

Ms. Marguerite Jackson: It tells us we need to pay attention.

Mrs. Maria Van Bommel: I guess so.

Ms. Mary Jean Gallagher: In fact, that’s exactly the kind of analysis that we do regularly in the student achievement division. There are a number of responses based on what it is that we think is the cause for students not moving up as quickly as we would like them to. In some cases, we gather people together to say, “Where are the boards that, in fact, are achieving continued gains, and how are they doing that?” and distill that information from it. We’re also involved quite deeply in a number of schools right now.

If you look at the achievement levels of our students, we have successfully reduced the number of exemptions over the last number of years from the tests. We’ve successfully reduced significantly the number of students who would be performing below level 1 in the work on their EQAO test. We’ve significantly reduced the students who are performing at level 1. In fact, the evidence in the data is that we’ve moved most of those students up into level 2. We are now at the point where we’re looking very closely at those students whose work is being judged to be at level 2, working with their teachers and becoming very precise about examining what it is that the teacher does and needs to do in order to help that student make that journey from performing at level 2 to being able to perform at level 3, because that’s where we have about 30% of the students in the province.

Interestingly enough, of that group who are at level 2, the majority of them are at the very high end of level 2 now, so we actually put additional resources in the field. We have 50 teachers who are working every day in classrooms with teachers looking at what it is that happens to successfully support that journey. That’s not only a commitment to higher levels of achievement, but it, in fact, is part of our closing-the-gap strategy. We know that among those students in level 2, there is, at least in reading and writing, a higher proportion of boys, a higher number of students than the average of children with special needs, and so on. So we’re really becoming increasingly precise, both in what tasks will lead to success and which tasks match which sets of students’ needs. We engage teachers in schools all across the province in professional learning dialogue, where they share with each other, in schools and across schools, what the strategies are that are going to address those things.

Mrs. Maria Van Bommel: You said that boys and girls are essentially coming to an equal status on the math side. When we started this conversation, we talked about the boys and the laptops and the writing skills. Is there still a gap among the boys in terms of writing and reading—

Ms. Marguerite Jackson: There is.

Mrs. Maria Van Bommel: —if they don’t happen to have a laptop?

Mr. Brian Desbiens: We actually identified this. Part of our role is to take the data and identify what the gaps are. Quite frankly, as chair, briefing the minister—every time we met, we talked about that. Of course, the ministry has initiated some good interventions. We also initiated a research study, because we wanted to know whether this is a problem only in Ontario or is it unique to our curriculum, or whatever. The data definitely shows that this gap is a worldwide phenomenon.

We know that there are some boys who grow and develop a little differently than girls. The question then becomes, how do you differentiate the instruction in order to be able to get at it? In our research, which will be posted on our website for all educators, we’re trying to identify some of the best practices that people are trying to do in this, and also to try to understand it.

One of the things is—we haven’t talked a lot about it—there is a demographic profile in here. In addition to the test, the students also fill out a demographic. One of the things that jumped out at me when I was looking at the demographic was that they asked girls whether they write at home—notes or whatever—and asked boys whether they write, and there was a substantial difference between them. Little girls write notes, they do diaries, they write notes to their girlfriends; boys throw rocks. They just don’t write very much. There are real differences, developmental differences. So we have to look at how we can do that over time.

But the optimism is that we’ve made a difference in science with girls. We had to figure out how to do it. We figured out that girls are doing really well in sciences now—as well as boys and better, in many cases.

I think that if we put our heads to it with the right data and work with teachers and educators, we’ll be able to figure out how to do it. I think technology is one of those ways. Little boys love gadgets; we’ve just got to figure out how to get the gadgets to help them write a little bit more and read a little bit more.

Ms. Mary Jean Gallagher: There’s another reason as well for some optimism. Most recently, if you look at the difference between a previous administration and the most recent administration of the PIRLS and TIMSS tests, Ontario actually has demonstrated a reduction in the performance gap in language arts, or in language skills, between the boys and the girls. In either the PIRLS and the TIMSS, we went from a 20% gap in the average achievement of boys and girls to 13%. We’ve seen the same thing in another one of the international tests, where we’ve taken that gap from a 10% gap to a 5% gap.

There is some reason for optimism there. It begs the question: Why aren’t we seeing that in our own EQAO tests? Certainly, I have some research staff who are working on exactly that question, but we are focusing on what the successful strategies are. The student achievement division of the literacy and numeracy secretariat just sent out to schools, in our most recent mailing in the fall, one of our DVDs, which is on educating boys and boys’ literacy. It is a very rich DVD that is used in schools that delineates and identifies schools and practices where the gender gap has been minimized. They, in fact, have found ways to engage boys differently. We share that with all the schools in the province.

Ms. Marguerite Jackson: I would say that our data says that boys have to read to learn to read, and the gap in achievement is about the same as the gap in the percentage of girls who say they read and boys who say they read. This is not about playing with DSs or doing whatever; to learn a skill, you have to work at it, and the boys are going to have to read. We have to find the materials and we have to structure the circumstances so that they’re reading, and reading the types of materials that are drawing upon their thinking, their interpretations and their extrapolations.

1410

Sometimes it’s so evident to us: 73% of the girls say they read at home, outside of school, and it’s 61% of the boys. That about the same gap in terms of who’s meeting the standard and who’s not.

The Chair (Mr. Norman W. Sterling): Mr. Naqvi.

Mr. Yasir Naqvi: Let me just first start by thanking all of you for your strong contribution to making a strong education system. I find the whole discussion very fascinating.

I just have a few questions to ask, and they’re sort of bits and pieces picking up from what other colleagues of mine have asked. My first question is a very simple one, because I can never figure this out. The correlation between what grade students are in and what age they are: Is there a simple formula I can remember? How old are the grade 3s, 6s, 9s and 10s?

Ms. Marguerite Jackson: Grade 3 children, when they finish grade 3, will become eight in that calendar year. So children who finish grade 3 in 2010 might have become eight in January, they might become eight in December. This is an interesting factor too, if you start the school almost a year ahead. So in grade 3, they’re eight; at the end of grade 6, they’re 11; and in grade 9, I guess they’d be what? Fourteen?

Mr. Yasir Naqvi: And grade 10s would be 15 years old. Okay, great. So there’s no formula I can memorize? I just always have a hard time.

Ms. Marguerite Jackson: If you start with the year they enter grade 1, they would be six. You can sort of calculate it out from there.

Mr. Yasir Naqvi: Mr. Marchese earlier was trying to make a link between how the test scores are going up, but perhaps the quality of students, when they go to university or college, might not be that great, this common complaint that’s always out there. I’m just trying to figure out why the tests might be going up besides all the good work that’s being done by the achievement office.

One of the claims that has been made is that teachers now teach to the test and they prepare students for that test. Is there validity to that claim? Is that part of the reason we’re seeing the scores going up, because kids are being taught according to the test?

Ms. Marguerite Jackson: To this extent: If they teach the curriculum, the children will do well on the test.

Mr. Yasir Naqvi: So if you follow the curriculum, the curriculum is designed to meet the factors that are outlined in the test.

Ms. Marguerite Jackson: That’s right.

The other point, I think, particularly when you look over the years, is when you look at the early results on these tests, in the very first few years, they were quite low. The curriculum was also new at that point, and indeed, the alignment of the curriculum and the test happened as the curriculum was introduced. Teachers are much more familiar with the curriculum, they’re looking at it—we have a very fine curriculum in Ontario, by the way. It’s certainly the best that we’ve had in my career as a teacher. It’s clear about what we’re trying to achieve at various stages along the journey, it gives examples of how to do it and, of course, now they have evidence of whether kids are learning it or not.

Mr. Yasir Naqvi: Is the test designed to the curriculum or the curriculum designed to the test?

Ms. Marguerite Jackson: The test is designed to the curriculum.

Mr. Yasir Naqvi: So that it tries to then measure what’s being taught to the children.

There was talk of remedial steps. There was talk about—again, it was Mr. Marchese who was asking the questions—those students, be it for certain abilities, who are not doing well, and I think someone mentioned that there are remedial steps that are taken. Are we talking about really fine-tuning to a particular student and then working on him or her to see if he can elevate them? Is that what we’re talking about with remedial steps? If you can sort of guide us as to what those steps could be in a classroom.

Ms. Mary Jean Gallagher: The answer is yes, some of it is particular to individual students and supports that are provided to those students, but the data that comes back also leads the teacher to examine where the gaps that larger numbers of students in the class might have are. So it’s both targeted to the class and targeted to the student. Our resources and supports to schools, say in the OFIP program, are actually targeted to schools where there are larger numbers of children who have challenges of various sorts.

The kinds of interventions that we would be talking about range, particularly when you get down to individual students. We have resources that we provide to schools and boards that provide two forms of tutoring, for instance, in some schools. One is tutoring under an OFIP program where teachers are asked to provide additional tutoring supports for students who would have needs after school, and the other is another form of tutoring where we hire university students etc. to assist in classrooms with tutoring during the school day. In fact, we differentiate some of our funds to school boards and schools to provide summer literacy programs for students who may need additional assistance over the summer in order to be assured they can move forward with those literacy and numeracy skills that are foundational to their success later in school.

There’s a whole raft of interventions that would take place in a school, whether they’re designed to address a child who may have special needs or special education needs, such as the intervention of a teaching assistant in the classroom with those children to help them access the curriculum to be able to perform better. We have other students with special needs for whom there may be technical equipment provided. For a child who may not have the motor skills to be able to print or write properly, we would provide them with computers. Blind children are provided with assistance and mobility coaches. It ranges, and it’s very much focused on what needs the students have.

Mr. Yasir Naqvi: Thank you. To what extent, in that type of intervention, are social factors being considered: language differentials—and I’m not just picking English or French; other languages that might be spoken—or the socio-economic status of a family, which might be impediments for a child to learn? Are those factors being taken into account when a plan is put in place to help that particular student?

Ms. Mary Jean Gallagher: Absolutely. And in fact, when you look at the data that tells us how our children are performing, one of the stories of the last five or six years on our EQAO tests is that the gap for English-language learners, for instance, in our province has significantly narrowed. That tells us that the interventions for English-as-a-second-language students coming to our province are in fact working much more successfully than they were a few years ago.

As well, in terms of the socio-economic challenges, we have seen instances through our OFIP program where schools which have faced a range of those kinds of challenges have, in fact, made a huge difference in children’s outcomes. I was a director of a school board in southwestern Ontario, and I can think of one of my schools as an example. It was in an area of the city where probably 65% or more of the children lived in poverty, often in single-parent families in public housing. In excess of 45% of the children in the school were recent immigrants to the community. So this was a school that had all of those challenges in its demographics.

At the beginning of the process, to our great embarrassment—and with teachers and a principal who cared desperately about those kids—that school was posting fewer than 20% of its children meeting provincial expectations and the provincial benchmark. After going through the turnaround and OFIP process over a period of two or three years, with that intervention and assistance, that same school raised its results to almost 60% of the children achieving at provincial standard. That was a direct result of hard work on the part of those same teachers and principal, but also learning through the intervention and support of the programs that the Ministry of Education provides, actually discovering that there were ways to approach the problem that can make a huge difference for those very children.

So have we levelled the playing field completely for all of those kids? No, we have not. The gap still exists. But Ontario has made some significant gains in that area and the core of that is essentially the basis of the work that my staff does every day.

Mr. Kevin Costante: If I could also add: Through the funding formula, we have a learning opportunities grant that is specifically focused on providing additional resources for boards where there are issues of poverty and low income. I believe that amount is around $400 million a year.

Mr. Yasir Naqvi: Is my time done, Chair?

The Chair (Mr. Norman W. Sterling): Yes, it is.

Mr. Yasir Naqvi: Okay, thank you.

1420

The Chair (Mr. Norman W. Sterling): You’ll have another opportunity in another round.

Mr. Rosario Marchese: I just wanted to make a few comments based on what Maria had raised, because I think boys—it’s a matter of genetic stupidity, really. I think we have a defect, as boys, and I don’t know how we fix that. Girls do better in everything, on the whole. In the studies I read many years ago, girls did better in math and the sciences up until grade 10, until they were culturally streamed out of those courses. It’s changing now because we have higher participation levels of women in university than we do men. It has become a concern for a whole lot of folks in society, because boys are not going to university as much, generally speaking. But we do have this problem. Mercifully, boys tend to recover at age 23 or 24—not all, but they do—in terms of achieving some level of equality or maturity. But we do have a lot of work to do, and I’m not sure the test can solve that. We do have problems. There are a lot of great teachers who solve some of these gender problems.

We know that we can learn from great teachers; we just don’t know how to apply it board-wide. That, for me, is the difficulty. We have to solve this boys problem. It continues to be an issue, and we haven’t yet found generic ways of dealing with it.

Peter Shurman raises some questions about what strategies you have to deal with the federations, and I was thinking, “What an interesting question.” You know Marchese has made some comments about your office over the years because you read Hansard, I’m assuming.

Ms. Marguerite Jackson: I’ve noticed them.

Mr. Rosario Marchese: So I thought, “Hmm, do they have a strategy to deal with Mr. Marchese?” But you don’t have to answer that.

There are some issues that the Auditor General raised—

The Chair (Mr. Norman W. Sterling): They work in the realm of possibilities.

Mr. Rosario Marchese: It will take some work.

The Auditor General talks about—and you’ve answered it in part, Marguerite. Maybe Brian has touched on it too. There was one board that used the—I’ll just read it from the report: In the 2008-09 year, one of the school boards visited by the audit team “decided that EQAO results would count for 15% of the student’s final Grade 9 mark.”

I'm assuming that there’s only one board that you’re aware of, but there could be more. Is that correct?

Mr. Jim McCarter: It could be a range. It could be from 0% to—I think the highest was a solid 15%.

Mr. Rosario Marchese: So different boards use different marks: “The degree to which these test results form part of the final mark is inconsistent province-wide, ranging from zero to 15%.” He raises a good point, which speaks to inconsistencies, obviously, and I don’t know what your position is. Do you have a position, or does the ministry have a position on how to deal with that? If some boards are using that, that could show a higher mark than others based on how some students do in some boards. Therefore, they look good by applying a higher number to that, I’m assuming, but it does present some problems. I don’t know whether either of you have a position on this in terms of how you deal with the inconsistency.

Ms. Marguerite Jackson: Well, we’ve accomplished much if not all of our work by building collegial partnerships. I think there is an opportunity for us here, if we can demonstrate that it makes a difference when a teacher takes a part of the EQAO test and puts it on the final mark, to work with teachers to say, “Look, this makes a difference for kids and for your schools, and we strongly encourage you to do it.” As it is now, the curriculum guideline allows the teacher—and I think rightly so—to determine how a child will get his or her final mark, and they may determine to use a number of things to contribute to that final mark. That’s their call.

Mr. Kevin Costante: As I understand it, EQAO is going to be looking at whether there’s evidence in this upcoming test year around including in the final mark some of the EQAO testing results. I think the ministry will be looking at results coming out of this year, and we’ll see whether we need to develop a policy or more consistency province-wide. I think we want to see the evidence first.

Mr. Rosario Marchese: Part of the worry—and I think it connects to a question I was going to ask about the survey the OSSTF did a couple of years ago, that you’re aware of, where 40% of the respondents claim that they are under tremendous pressure to increase numbers. You’ve heard about that, obviously.

That presents a problem. If teachers are under pressure, presumably from the principal—and principals presumably are under pressure from somebody else—to deal with that so that they’re more lenient about the kinds of marks they give, does that affect in any way this issue I just raised? How do you deal with it in general?

Ms. Marguerite Jackson: I would hope that the pressure teachers would feel would be if a young person were leaving their class at the end of a term without having achieved what you wanted them to achieve, the sense of, “I didn’t do the best I could by that kid.” That’s the kind of pressure I think would be there. I think it has been true. In the educational community in recent years, there is much more of a culture of expectation that every child should be able to achieve what we’re expecting at various stages along the way.

Mr. Rosario Marchese: Sure. Of course.

Ms. Marguerite Jackson: That, I think, is the pressure, and that’s quite different from streaming kids on a bell curve.

Mr. Rosario Marchese: Yes. The pressure I’m talking about is where a student has a 40% mark and the teacher is pressured by the principal to give a 50. The pressure I’m talking about is where a paper has been plagiarized and there’s pressure to not condemn it too much, or not to lose the marks or as many marks. You must have seen that study by OSSTF. That’s what I’m talking about.

Ms. Marguerite Jackson: I guess what I would say is, should that happen, that’s unethical behaviour. I would hope it’s the teacher’s interpretation as opposed to a direction. But we have had occasions where we’ve had to sanction people for improprieties in terms of directions that they have given to teachers.

Mr. Rosario Marchese: How many times has that happened, where you sanctioned someone?

Ms. Marguerite Jackson: In the last five years—

Interjection.

Ms. Marguerite Jackson: Four or five times, and the college of teachers becomes involved in those cases, because it is a matter of professional ethics and there can be some very serious consequences, and there have been some very serious consequences for teachers. We all live and abide by a professional code of ethics, and then we live and abide by a moral code.

Mr. Rosario Marchese: There is tremendous pressure on the system to deliver the results that some people are expecting. That’s why I think this is a very real fear and a reality, because I hear it from a lot of teachers. This isn’t just based on the study. People email us and talk to us about these things. I just wanted to raise it.

The Auditor General raises another issue about fluctuations in the marks in different boards. Sometimes they’re very wild. He alleges—you don’t allege that, but you say that—that you don’t undertake a formal analysis or investigation to determine the cause of sudden or significant changes. Why is that?

Ms. Marguerite Jackson: I think the Auditor General would confirm that for the most part, what the team found when they came to look at our work was that we have our processes very clearly documented in writing. In this case, we didn’t have a documented process. It is a fair observation: The process we had used in the past was one where Michael would bring the reports to me and I would look at them with him, board by board. Typically what I would say to him, as we were getting ready to report, was, “Okay, we’re ready to report. What are the results, first of all?” Then secondly, I would say to him, “Of the 72 school boards, is there anything unusual about any of these?” If he told me a school board went up 10%, I would say to him, “Let me look at the schools there.” We would look at the schools to see who contributed to that 10%. It is true, I didn’t have that all documented in terms of, “This is what I do on this date and that date,” but I will have it for the next time, because we do do that.

If the school board’s results seem questionable to me, just from the perspective of, “Could that be reasonable?” I call the director of education, and have done that in past years.

1430

Either way, incidentally: In some cases boards drop, and I call the director to say, “Would you be surprised to know that your board is dropping by 8%?” That, of course, is particularly significant if you’ve got a large school board, with 15,000 kids—

Mr. Rosario Marchese: So you do that in individual cases, but is that the formal process that you use when you see these erratic changes or wide fluctuations? You might call or you would call? Is that the formal procedure?

Ms. Marguerite Jackson: That’s part of it, and then if we have a concern, we would require an investigation of what contributed to the fluctuation, particularly if the fluctuation was dramatically up. In that case, the superintendent is required to become involved, and in the future, the superintendent will be required to submit a written report to us of what the investigation was and why he or she has confidence that the results are valid.

Mr. Rosario Marchese: So, Jim, does that constitute a formal kind of process, or are we looking at something different?

Mr. Jim McCarter: When we asked the question, we said, “Have you looked into it?” They said, “Yes, we have.” We said, “Well, can you show us what you found?” There wasn’t documentation in a lot of cases, so we’d like to see the documentation. The second step would be, we think it’s great to go back to the school and actually ask for something in writing: “Can you explain the fluctuation?” That’s the sort of thing we’d be looking for, so that’s good to hear.

Mr. Rosario Marchese: Okay.

I wanted to ask you a question about the confidentiality agreement you have with the markers. Could any one of you explain why you have it? What is your worry about markers not being able to talk about how they mark or their experience? Why do they sign confidentiality agreements?

Ms. Marguerite Jackson: Rick, do you want to speak about the confidentiality agreements?

Dr. Richard Jones: Sure. We expect scorers to sign confidentiality agreements because, on the one hand, we actually encourage teachers to talk in general about the kind of processes that we go through in scoring. They’re very systematic, very consistent processes. The training is exceptionally strong in terms of how we go about training our scorers and so forth. We really like the idea of messages like that getting back to the field around the rigour that’s a part of the scoring process. But you have to remember that we’re scoring operational—those are the actual items that are being scored that count toward a student’s mark or score—but we’re scoring field test items as well. We want to be sure that there’s no information, particularly around those items that are going to be used in the future on future assessments, that could get out to the field.

Mr. Rosario Marchese: Okay. So if a marker wants to talk to a journalist, just to talk about what their experience is, could they do that now?

Dr. Richard Jones: About the general experience and the general process? I wouldn’t have a problem with that at all. But to get into the specifics of the particular items, particularly when you’re talking about items that would be field test questions, is something that we would be concerned with.

Ms. Marguerite Jackson: What we wouldn’t consider acceptable is their speaking on behalf of EQAO—

Mr. Rosario Marchese: Absolutely; I’m assuming they wouldn’t want to do that.

Ms. Marguerite Jackson: If they speak as an individual citizen or teacher, that’s up to them, but to speak about our processes on our behalf, no.

Mr. Rosario Marchese: So the confidentiality agreement doesn’t prevent them from talking to journalists if they wanted, in a general sort of way?

Ms. Marguerite Jackson: Many of them probably have over the years.

Mr. Rosario Marchese: Actually, they haven’t.

Ms. Marguerite Jackson: They haven’t?

Mr. Rosario Marchese: No. They’re quite afraid, in fact.

Ms. Marguerite Jackson: Are they?

Mr. Rosario Marchese: Yes, and that’s why I ask you, because I would love for them to talk publicly.

Ms. Marguerite Jackson: What would they want to talk about that they don’t feel comfortable talking about?

Mr. Rosario Marchese: Some of the questions you get from the subjective questions are that some of them are encouraged to mark up, not down, usually.

Ms. Marguerite Jackson: Really?

Mr. Rosario Marchese: We get that. And often they talk about how, in the subjective questions, as long as they touch on various parts of the answer—it doesn’t have to be eloquent, even—however erratic, they get a mark or they get the marks that they need; that kind of thing.

Ms. Marguerite Jackson: Well, if that were the kind of thing they were going to talk to a journalist about, it would not be acceptable, because they’ve misinterpreted our directions. Rick would want to speak about the scoring rubrics and the training we give to the scorers.

Mr. Rosario Marchese: Right. So can I ask you—I understand that your techniques or whatever you do is very rigorous. That’s what, I think, Jim has said, and you would admit to that. So one assumes, and I forget the language—it’s on page 9, about how thoroughly you do this. “At the beginning of the marking process, all markers are provided training to develop a common understanding for interpreting and applying the requirements.” So even though it’s thorough and there’s a common understanding, some people possibly just didn’t quite get it. Is that what you’re thinking happens?

Ms. Marguerite Jackson: It does. We’re all human. We do have a process, though, if we sit down to mark the same piece of paper, to get as close to the likelihood that each of us would give the same score as possible. Either Michael or Rick could speak about that process.

Mr. Rosario Marchese: But none of them are under any pressure to mark up—none of them?

Ms. Marguerite Jackson: No. But we do do random checks. If you were a marker and you mark for us, we do random checks of your mark against expert papers to ensure that you’re following the expected scoring rubrics and we retrain, if necessary.

Dr. Richard Jones: Before anybody can score at all, they have to pass a qualifying test—before they can even begin. And if there’s any retraining that’s required, we go through that process with them.

Mr. Rosario Marchese: But if somebody wanted to be interviewed—sorry.

Mr. Michael Kozlow: It’s also important to note that our scoring rubrics and samples of student work that are assigned particular score points are on our public website, so that if you have a particular question in mathematics or a reading item, we clearly describe what is required for a student to demonstrate this score-code point, and here’s an example of a student’s response that does that. The criteria or the main direction that we give to scorers is to try to match that. So you look at a student’s work and you say, “It doesn’t match this student’s response,” they’re not told, “Score high or score low.” They are told, “Match to that evidence of that student’s work to decide which score to give.”

Mr. Rosario Marchese: I understand—

The Chair (Mr. Norman W. Sterling): Mr. Marchese, you may get a chance at the end, depending on the other parties, but—

Mr. Rosario Marchese: Thank you very much.

The Chair (Mr. Norman W. Sterling): Mr. Shurman.

Ms. Marguerite Jackson: We’re quite pleased to meet with you at any time, just separate from this, to show you our processes so you can see how we do this work.

Mr. Peter Shurman: You might want a day.

Laughter.

Interjection: It took the auditor seven months.

Mr. Peter Shurman: I just have a couple of final points that I’d like to make with you. I like to bring things down to a bottom line. When I compare you to some of the people who we sit and talk to in this room, there’s no question of professionalism. Sometimes that’s not the case. What we’re looking at is a professional group that tries to administer a test or several tests even-handedly, and I recognize that.

However, if I did bring this down to one word, I would say the word that’s being questioned or discussed by all of us and in the Auditor General’s report is “consistency.” It’s a hard thing to achieve for you; it’s a hard thing for us to get our arms around. If we can focus on consistency for a second, there are a couple of questions that come to mind. Consistency would enter into the issue of how individual teachers behave, how individual principals drive the process, how individual schools—what the makeup of those schools is, and areas—we’ve heard about linguistics today.

Let’s talk about the makeup of the test. The EQAO scores, as you have, with some pride, pointed out, have been going up year after year. Does your office have the capability of analyzing past scores on an isolated basis in just the area of multiple choice, separating it out, then, from what we could describe as the subjective portion of the tests? Because I think that might be a more apples-to-apples way of looking at things.

Ms. Marguerite Jackson: Michael?

Mr. Michael Kozlow: Yes, certainly we do. We use a very well established statistical procedure for equating from year to year, and in that equating, we do include both multiple-choice and open-response items. That’s important because student skills can change differently. For example, if teachers take a heavy focus on helping students to respond to open-response items, to identify inferences, to make connections, the performance on those items might improve more than on multiple-choice, so we ensure that both are included.

The purpose of the equating is to place the students in 2009 and those students in 2008 on a common scale so that we have high confidence when we say, “This group of students in 2009 received a level 4, and a different group of students received a level 4 in 2008, but those groups of students demonstrated the same level of knowledge and skill in order to be assigned that level.”

We do sometimes examine what happens if we do equating using only multiple-choice items, only open-response items. For example, in a particular test, we might have more than half of the test on open-response items and less than half on multiple choice.

1440

So we can look at that, and we can look at it over the years. We haven’t done that in a systematic way but periodically we do look at that. We see some minor differences. You don’t usually see large differences if you just factor in only multiple-choice items as opposed to the open-response items—

Mr. Peter Shurman: If I asked you, though, to do the same yardstick type of measurement using multiple choice only and provide us with a read, could you make the same statement? Could you say, based on multiple choice only, which is far less subjective, obviously, that there has been a steady rise in EQAO scores?

Mr. Michael Kozlow: I would expect, yes, we could do that analysis to get the formal answer, but each year when we do the equating, we examine all the items very carefully on the test. In order to do the equating, some students in the previous year write these items—what are called matrix items, so they’re imbedded in the test. The students don’t know which ones they are, so we can compare students in 2008 writing the exact same items as the students in 2009.

We look at differences in multiple-choice questions—differences in performance from year to year in multiple choice and open response. We see changes in both, so my expectation is we would—it may not be exactly the same magnitude because the open-response items measure different things. It might be a greater magnitude in some years, depending on what is happening in schools. But I definitely am confident that we would see a similar pattern across the years if we evaluate only multiple-choice items.

Mr. Peter Shurman: How hard would it be to do, if I asked you to do it?

Mr. Michael Kozlow: It would be feasible.

Mr. Peter Shurman: I don’t know if I’m asking you to do it; I’m thinking about it. Let me ask you another question, though, that is involved with the very same thing. Which tests, by grade, have the multiple choice included, or do they all?

Mr. Michael Kozlow: All have some multiple choice.

Mr. Peter Shurman: All have some, but varying amounts?

Mr. Michael Kozlow: Yes. The one that has the least proportion of the assessment in multiple choice is the writing. It’s logical.

Mr. Peter Shurman: Of course. Is there, by grade, any one that has the least amount of multiple choice or are they all about the same percentage?

Mr. Michael Kozlow: For the mathematics assessment it’s a fairly similar pattern across the grades, and likewise for the reading assessment. It’s a similar pattern.

Mr. Peter Shurman: Thank you very much.

The Chair (Mr. Norman W. Sterling): Just a supplementary on that: Is there any multiple choice in the writing at all? I don’t know how you would do that.

Mr. Michael Kozlow: We do have a small number of items. For example, on the grades 3 and 6 assessments there are eight multiple-choice items. They measure things like punctuation. For example, which of the following sentences has correct punctuation? They can measure word usage: Which of the following is using the verb correctly in the sentence? Verb-subject agreement: Which of the following has a correct supplementary clause used in the proper way? Those kinds of things.

The Chair (Mr. Norman W. Sterling): Those multiple choice, are they marked by automation?

Mr. Michael Kozlow: Yes. The student’s response is recorded in the computer. There’s one correct response and if the student chose that response, it’s right. If they didn’t choose the response, it’s scored wrong.

The Chair (Mr. Norman W. Sterling): And how long would it take you to do this analysis for the past 10 years for the committee?

Mr. Michael Kozlow: We wouldn’t be able to do it for 10 years because the structures of the tests have changed a little bit. Certainly, it would be quite feasible to do it back to 2006, because that’s when the structures of the tests had been very common; 2004 was when we introduced, after the external review, a change in the structure of the tests. Although, again, you can change the structure of the test but still keep the results comparable by having, as I said, common items that you can compare. I don’t know exactly how long it would take. It would probably take several days of someone’s time to do all that analysis.

The Chair (Mr. Norman W. Sterling): Thank you very much. Mr. Naqvi.

Mr. Yasir Naqvi: As we were having this conversation, I was thinking, has anybody seen these tests we’re talking about? I’ve never seen a test—not that I want to take it. Do you think we can get a copy, Mr. Chair, of the test so we can at least see what they look like?

The Chair (Mr. Norman W. Sterling): I’m not sure. Are they public documents?

Ms. Marguerite Jackson: Actually, I would invite you to go to our website; a number of the tests are there. We have, on occasion, published the tests, and they’ve been in the newspaper, particularly the grade 10 literacy test. Yes, there are samples of all of our tests there. We’ve put them there, again, so the teachers can learn from the feedback we give them. They can look at the type of question we were asking, know what we were trying to get and see whether or not—we also put there samples of what good answers look like, particularly in terms of those answers that are written out. The teachers use this a great deal. It’s there for the public as well.

Mr. Yasir Naqvi: What’s the website?

Ms. Marguerite Jackson: It’s eqao.com.

Mr. Yasir Naqvi: Thank you. I know what I’m doing tonight.

I just wanted to bring this conversation around to the full-day learning initiative, which is being rolled out in the province of Ontario, and hear from you, maybe from the deputy minister, perhaps, or the ADM, as to what kind of impact you are hoping or expecting to see on these tests starting in grade 3 because of the full-day learning initiative.

Mr. Kevin Costante: I think what we are expecting to see is perhaps similar to what we’re seeing in the French-language boards. You’ll note that their reading and writing scores in grade 3 are higher than the English-language boards’. The French-language boards have had full-day learning for about 10 years now, for the most part. We are expecting to see, over time, an increase in those scores.

As well, I think there has been a number of studies that show that early learning does pay off in terms of kids being ready to learn in grade 1, so we are obviously implementing early learning based on all that evidence that has grown up over the years that this is a good investment in the future of our children.

Mr. Yasir Naqvi: That raises an interesting point. We were talking about the differences between the tests for English and French students. Part of the reason, perhaps, the French test is different is because of the full-day learning already in place. Are there any special parameters or barometers you use or are thinking of using, once full-day learning is fully implemented, because you think that there will be changes in the learning behaviour of those children, as was outlined by the deputy minister, so you need to test in a different way or maybe in a stricter fashion? Is there any thought put into place on that?

Ms. Marguerite Jackson: The test is different because the French-language community has its own curriculum. As I said, generally the principles that are in the curriculum are similar. As François said, there are cultural interests in terms of the francophone community. But just as we have a book here that describes the expectations in English, there is a book in French for francophones. We have a francophone team on our staff which develops the French tests. We work with francophone teachers who teach those children to help us build the stories, the test items etc. That’s the difference in them.

Do they still learn to count? Do they still learn to read? Do they still learn to draw inferences? Yes, the same kind of specific principles, but there may be some nuances of uniqueness, as François pointed out.

For example, when we test the grade 3 children, one of the genres they are to have learned and worked with by the end of grade 3 in English is poetry. That is not the case in French; it’s not until grade 4. So we don’t test that genre until the grade 6 test for the francophones.

The threads that run through the curriculum would be very, very similar.

Mr. Kevin Costante: One of the things that we are going to look at in early learning, and we’re doing that research now, is right now—I believe it’s in senior kindergarten—many schools use an EDI test, educational development index, which tests kids on five different components of their knowledge and preparedness.

I think the percentage of kids coming into grade 1 who are lacking in one of those five competencies is somewhere in the high 20s. So one of the things that we’re considering is whether we use this more broadly. Over time, then, we can see whether early learning is actually reducing that and make sure that we have evidence to show that kids coming into grade 1 are better prepared. The real test of early learning is going to be a continued reduction of the number of kids coming in who are struggling with one of those five strengths.

Mr. Yasir Naqvi: Brian, do you want to—

Mr. Brian Desbiens: Yes. I think the deputy has pointed out that perhaps EQAO might be helpful in using some of our processes of gathering information, so that we can be helpful in terms of tracking that and looking at what makes a difference.

When I have this conversation out there, parents do not want their kids being tested more. We’re not talking about more pencil-and-paper tests and stuff like that; we’re talking about instruments that are already being used that are really more observational. But the importance here is to get a standardized approach across the whole province so that we can get a proper baseline, and then do the kinds of intervention—some of the processes and analysis that EQAO has been party to could maybe be helpful. It becomes a beginning point that the educational system could use for decision-making.

Mr. Yasir Naqvi: Great, thank you. I have no further questions.

The Chair (Mr. Norman W. Sterling): Thank you very much. This draws our hearing to a close. I’d like to thank everyone for coming today. I think you got from the general tenor of all members of the Legislature who are here today that there’s a lot of support for the work that you’re doing. We would like to help improve that process, and that’s how the public accounts committee considers its role: trying to help people who are charged with implementing government programs and doing what we can to encourage senior people to take those steps to improve whatever we’re doing. I’m sure our report will be in that tenor.

I would ask members of the committee to stay for a few minutes after you leave. We’ll have the opportunity to give our researcher some ideas that we would like included in the report.

Thank you again for coming. Have a safe trip home.

The committee continued in closed session at 1450.

CONTENTS

Wednesday 31 March 2010

2009 Annual Report, Auditor General P-21

Ministry of Education P-21

Mr. Kevin Costante

Mr. Brian Desbiens

Ms. Marguerite Jackson

Dr. Richard Jones

Ms. Mary Jean Gallagher

Mr. François Lavictoire

Mr. Michael Kozlow

STANDING COMMITTEE ON PUBLIC ACCOUNTS

Chair / Président

Mr. Norman W. Sterling (Carleton–Mississippi Mills PC)

Vice-Chair / Vice-Président

Mr. Peter Shurman (Thornhill PC)

Ms. M. Aileen Carroll (Barrie L)

M^me France Gélinas (Nickel Belt ND)

Mr. Jerry J. Ouellette (Oshawa PC)

Mr. David Ramsay (Timiskaming–Cochrane L)

Mrs. Liz Sandals (Guelph L)

Mr. Peter Shurman (Thornhill PC)

Mr. Norman W. Sterling (Carleton–Mississippi Mills PC)

Mrs. Maria Van Bommel (Lambton–Kent–Middlesex L)

Mr. David Zimmer (Willowdale L)

Substitutions / Membres remplaçants

Mr. Rosario Marchese (Trinity–Spadina ND)

Mr. Yasir Naqvi (Ottawa Centre / Ottawa-Centre L)

Also taking part / Autres participants et participantes

Mr. Jim McCarter, Auditor General

Clerk / Greffier

Mr. Katch Koch

Staff / Personnel

Ms. Lorraine Luski, research officer,
Legislative Research Service