I’m Richard Hill and I’m with the national center for the improvement of educational assessment, I’m the executive director.

The things I’m most interested in uh were the bias that I bring is understanding the distinction between external and internal assessments and that the responsibility of the teacher is internal assessments and it’s internal assessments that drive the real warning that takes place in --- and I have a strong bias on about those internal assessments should be done. So that’s the that’s the basic.

Well why is there a concern for improvement for educational assessment is uh because there are uh a lot of practices taking place in the country today that um at best or not aiding to educational achievement. Uh I’m a strong believer that there are major improvements that we can uh see in uh in in education uh educational achievement. I’ve done a lot of work in standard setting across country and the uh thing that always surprises people is that when they look at student work and see what they do the very small percentage of students who are able to do the sorts of things that people think just about everybody can do when they finish high school. We did a study in Arkansas for example and uh we brought in business people and educators and we had a very uh very thoughtful and and middle of the road and very deliberate group uh we spent a couple of days with them setting standards and they pick out work that they thought was appropriate for a high school student to be able to uh complete. Um it turned out that three percent of the students in Arkansas were able to produce work at that level of uh of quality and that was suppose to be a minimum requirement for high school graduation diploma. So you can, so that that fell apart very quickly. But I’ve taken those examples of work all over the country, I presented it to the Pennsylvania state board of education for example and uh they took a look at it and thought that that what the the level of achievement that the group had decided and I’ve looked pretty good but that was Arkansas and maybe Pennsylvania should be a little higher and I through national assessment I had enough information to tell them at best twenty percent of the students in Pennsylvania would be able to perform at that level. That was a tremendous surprise to them. Uh one of the real problems we’ve had in helping states implement accountability systems that work is they often presume that when they set high standards a high portion of students will be able to meet those standards and they paint themselves into an awful corner uh and then have to make a decision that no they really did meet the high standards or no they really didn’t mean that people really have to have this. Um I I could actually I could go on quite a while about that. But there there are there are many things about both assessment practices and accountability practices across the country that are not being done as well as as they ought to be and so the mission of the organization is to see whether or not we can get people to improve the kinds of things they are doing.

One point I’d like to make first is the importance of distinguishing between assessment and accountability. Um they’re uh they really are very different things. Uh assessment is the acquisition of the numbers we have about uh what students know and can do and uh accountability is is the is the use of those data uh to make decisions and and establish uh uh uh the consequences. You can’t have good accountability with out good assessment that’s a basic building block of good accountability but you can have very good assessment and still have bad accountability. Uh so that’s that’s certainly one distinction to make. But there are also is a distinction that needs to be made between external and internal assessment. External assessment is the information that comes from out side the school uh the information that tells the school how the students are doing. But I have a strong bias that if we’re really going to improve educational achievement in schools uh it’s the internal assessment that is going to drive that change and internal assessment is the process by which the school can tell the state how the students are doing rather than the external assessment in which the state tells the students eh tells the school how the students are doing. So um internal assessment is a very critical thing.

Well when I’m talking about internal assessment what when I am really functioning on is the information that a teacher uses to know what it is students can do and can’t do and what the next logical step in their educational attainment will be. Um I have a very strong bias that the key to improved uh educational achievement is feedback. Focused feedback and getting information to people in a way that tells you what they know and what they don’t know and what they need to learn in order to move to the next step. Uh so that when I’m talking about internal assessment I’m really talking about a teachers effective use of the information that students ought to be providing to them on a continual basis about what they’re doing. So the way that I see an ideal classroom is operating is uh very different from the way a traditional classroom typically looks. Students should be uh almost continually actively engaged in in in uh work that draws on the skills that they have and makes it clear what skills they don’t have and what the stumbling blocks are to the next level of uh achievement. Um I gave a paper here this year on uh the writing portfolio con—experience in Kentucky and uh how much uh that really um demonstrated the what what we’re talking about. Uh to be a good teacher you really have to have a very firm grasp of your of your your content area. Because it does two critical things if you really understand your content. Number one it gives you a clear understanding of what skills it takes in order to acquire that and until you’ve taken that path yourself it you really can’t bring other people down that path. The other thing it helps you distinguish between what’s important and what isn’t important. Um one of the problems with uh a lot of efforts in educational reform and that has come out many times at this conference over the last couple of days, is how we’ve uh uh tried to tackle the entire front all at the same time. And uh that’s that’s a symptom of true lack of understanding because if you do truly understand a a ah a content area you do understand that there are sequences of things and you do understand that some things are more important than others and certain skills are critical to ultimate uh uh achievement of that skill. So uh step number one or rule number one if you’re going to be a good teacher, know your content area and know it extremely well. But the second part then is that uh once once you know once you know your content area and you can pri—prioritize things then you start a process of actively engaging people to see whether or not they how far along down the road to those critical skills uh they they are and what the stumbling blocks are. By having people actively engaged in activities in the classroom you can constantly be observing what they know and what they don’t know and what the next steps are. The amount of feed back then that you provide increases dramatically and then I believe the amount of educational output that you get increases uh very very rapidly as a result. So this is this this is really the critical element I was saying earlier that that that I talked earlier in this conference about our experience in Kentucky. And we found out that when we started the writing portfolio process in Kentucky that that um portfolios weren’t written particularly well um by by students but we also found that teachers didn’t score them very well. Um and it turned out that the two pieces of information were very directly related to each other. That once we provided feedback to teachers on what they were doing wrong in scoring they very quickly understood what it was they were suppose to do correctly very rapidly became much better scorers of portfolios then they had been. And then as a consequence were able to teach their students what it was they were able to do. So I talked about an experience we had in Kentucky where we had we we went out and pick a series of schools where is was pretty clear that uh that portfolios were being mis-scored by uh a wide margin. We collected portfolios from the schools and we scored them and found that they they were mis-scored on on the scale that we use in Kentucky by uh more of of of of forty points. Maybe I ought to say that again. By more than forty points um we (interviewer asks to start again)

We had a very interesting experience in Kentucky that really demonstrates the point that I’m trying to talk about. Um when we started the process in Kentucky we found that uh that uh portfolios weren’t written particularly well but from our point as being responsible for the accountability system uh initially we were very concerned that they weren’t being scored very well and we did not connect those two pieces. So we went out to a series of schools that had clearly mis-scored their portfolios by a wide margin and we collected the portfolios from those schools and brought them in we scored them and found out that the average score that they had assigned to the portfolios was sixty something but in fact the right score if they’d correctly scored the portfolios was fifteen something’s so they had mis-scored the portfolios by a tremendous margin. Um we held a series of one day meetings with the teachers in those schools to uh review with them what had happened and of course what what was going on then is we were providing feedback to them for the first – we had given them something to do, something active, they had to score portfolios. We saw what they did right we saw what they wrong and we provided feedback to them and uh and uh in a very short period of time in just this one day, in part because that they had received lots of training in the past but it had really never taken hold because they had never been asked to do anything with it reinforcing my bias. Uh they uh were very quickly able to re-score. When we collected the portfolios from those same schools the next year we found that that the real scores of the students portfolios had gone from fifteen something to thirty some thing, a tremendous improvement in one year. But more importantly uh well actually let me let me start that over and repeat it. Uh when we collected the portfolios from those schools we found that the scoring error had had dropped from forty something points down to three points. But more importantly we found that the the quality of the student writing had really gone from fifteen something up to thirty something. So the quality of the writing had improved tremendously in one year. And we don’t think that is was a coincidence by any stretch of the imagination that by providing the feedback to the teachers they became immediately much more effective teachers themselves since they knew what was needed they were able to pass that information on to their students. But for us the key lesson was was that we gave them something to do. We observed it. We were intelligent reviewers and critics of that use that to target our next set of instruction back to them which became very targeted then to what they knew how to do and what they didn’t know how to do. And they’ve become very rapid learners and they were able to pass that on to their students. So that experience really reinforced tremendously our belief that having students be active, having them uh having the teacher be a skilled observer of what they’re doing is re- is really the key to breaking out of the mold we’re in where we get these small incremental improvements in achievement by focusing on curriculum alignment and better item test writing and all, to me those are pecking around the edges of the really critical piece which is to get teachers to change the way that they approach their their their classroom practice. So for us the external assessment helps to monitor what’s going on and is is is is an important piece but the vital part because it’s not collecting assessment information it’s changing around the way teachers teach and and I believe greatly increasing the amount of of uh educational output that they get the amount of uh achievement that their students have is this internal piece. So it isn’t a question of who’s responsible for it and what the principle’s role is and what the teachers role is it’s it’s it’s it’s fundamentally rethinking what do you mean by a test or more importantly what do you mean by collecting information about a student, when does it happen. Doesn’t happen after the learning takes place, it happens while the learning is taking place. The reason why uh education is so inefficient is that usually one person is standing in front of a room and passing out information without any feedback to the teacher about what the student’s acquiring and what the students hasn’t acquired and because the student is an asked to demonstrate it there really isn’t any reinforcement of all those skills. So I’m not saying that a hundred percent of educational time would be spent doing activities uh there certainly is a role for communicating information out to get people started and get get an entire class getting a certain amount of information. But we usually stop there with our educational process and if I could say one thing to your your uh people who are going out and be teaching in the field is is integrate your assessment into your teaching activities. If it’s separate from what you’re doing as teaching you’re not going to be teaching as efficiently and as effectively as you could and should.

Well the story about how uh assessment can impact instruction is really the Kentucky experience that we have but uh our staff from uh the Kentucky office advanced systems uh molded a lot of this thinking and had worked with a lot of districts uh across the state. We find that uh it seems obvious how to do it and it really doesn’t work very well. One of the things uh often, uh it it takes a lot of work to get people to do it right, maybe that’s the better way of of phrasing that. One of the things that we’ve found is that people mistake any activity for an effective activity. So one of the reasons why active learning perhaps hasn’t taken off as much as it should is because uh it’s been misinterpreted in a lot of cases you say well if you’ve got the kids busy and doing something that’s good. You really do have to very carefully take a look at what you’re trying to accomplish be it ste--- uh uh um the the standards of frame works that the state has published or something that your school district has published or even your own sense of what you want to accomplish by the end of the year. And for every activity that you have students do you have to ask the question before you start that activity how does that contribute to the big picture to the learning that you’re trying to have them do. So just having students busy and active doesn’t mean that you’re uh doing the kinds of things that we think you ought to be doing. To have them actively engaged in in activities whose performance or lack there of will inform you about the students ability to accomplish the standards that you’re looking for is the kind of activity that we’re talking about. So just any activity is a waste of time. A targeted activity to the standards that you’re looking at is really what we’re talking about. But if teacher do that in in in the way I would the way I would determine an effective classroom, if I were to walk in and observe, is I would see a teacher who was uh leading the educational process but not but does not view herself or himself as the sole source of information for students. Uh the process becomes very efficient when students learn number one to be self-evaluators. One of the one of the big lessons that we’ve learned all the way along in writing instruction is that it takes off tremendously when teachers truly understand what the scoring guides are. That was the lesson we learned in Kentucky. We’ve learned that in other states as well. But uh we did we did a lot of work up in Maine where we found that it takes off yet again uh another uh multiply increase in learning when the students have acquired those standards. Um I did a uh session uh at the CCSSO conference in 1999 which I tried to demonstrate that in a in a very practical and fun way. Uh one of the things you try to try to do uh when you subscribe to this idea is to always be thinking about you can demonstrate this when you’re trying to get people to come along and not lecture like I’m doing today. So I came to the uh CCSSO conference in 1999 with uh with a slot of time and uh my purpose was to demonstrate to people this very point that we’re talking about and what I did was I brought a series of uh Velcro targets and ping pong balls with the Velcro on them. And uh just to demonstrate uh many of the principles that we were talking about in a in a very interesting engaging way. So we had people in for about an hour and what we had them do is uh we started and we had uh them blind folded and as a teacher we would teach them the kinds of things that they should do and then we would have them toss the balls at the target. So the the first step was I lectured them on uh what proper technique would be. And we had them get up to a line and we had the target out there and we handed them three balls and we had them toss the balls at the target with no feedback at all and sometime they hit the target and sometime they didn’t. Then uh the next stage is is with a student blind folded we had them stand on the line and we had a teacher them observe what they were doing and give them advice and feedback. And uh they improved quite a bit obviously I mean the it’s it’s so obvious once you think about it but but being able to do it and get feed back as opposed to somebody telling you what to do and having you just try to understand that and apply it without any practice certainly made sense. But then the third thing we did is that we took off the blind folds and let people practice for themselves. Now the test was still the same you had to put the blindfold on but of course by that point people had really you know you know learn to put a chair along side themselves to to get things lined up and and and they saw how skillful they became. That’s a very obvious example but it was a real fun way of having people go through the process and and and see that kind of thing. Now because it was a physical skill it became really obvious why physical practice and feedback was worth while but intellectual skills work the same way um in we’re working on the design for the system that will do that. We learned a few things out of it and that’s that’s that’s always the interesting thing when you do something like that you would do it better next time. It became clear to me how skillful the teacher needed to be. That we didn’t really understand enough of the physics if it will of the the the the physiological elements of tossing a ping pong ball to be really good teachers and so if I were to do this again I think the first thing I would do would be to go to a phys-ed teacher or somebody who is skillful in that and ask them to train me first as the teacher so that when I went out and did it I really would have better teaching tools available to me then I then I did. So it reinforced the idea that that uh you just there’s there’s no there’s no uh simple cook book method of going out and doing it to be a skillful teacher you truly do have to understand the content area yourself. You have to be a skillful person in this area in order to bring other people along. But once you do have those skills you can you can apply them so much more easily and effectively if you’re observing what it is people are doing well and not doing well then just delivering the information that you have in your mind and hoping that they pick it up.

Well assessment literacy for me is probably going to be different than it is for most measurement people as it has been clear from uh answers I’ve been giving to questions so far I have a strong uh belief in the power of the assessment that teachers do in classroom while instruction is going on and and while I like to see other kinds of sumitive(?) assessments done well uh I truly do not believe that they have the same impact and so what I’m look for teachers to become uh literate in assessment is very different from what it was when I was an instructor as Syracuse University in teaching the masters level course in uh that uh every teacher was required to take in uh New York state uh for certification. There all sort of things that you could learn about how to write quality test items um how to um develop ---(?) all of those things. Uh I and I use to have uh much more uh belief in their importance that that the classroom teacher would know how to how to how to how to do that stuff. But as I have evidence by bias is is that I I use the word assessment much more broadly so I’m not thinking of paper and pencil test. I’m not thinking of I give you instruction and now when it’s all over I’m asking what is it you’ve learned. To me the kind of assessment skill that I would want a teacher to have is the kind of skill that a good a detective has. I want you to do I want you to do put yourself in a position where you collect a lot of evidence about what it is students know and what they don’t know and use that evidence in skillful way to to uh make decisions about were each student is on the continuum of achievement and and use the evidence to make decisions about what it is they need to know next and target your instruction so that the student is learning the next the next set of skills that they need. Once you start applying it it’s so obvious. I had a very frustrating experience in Kentucky, I had a a principle call up, we had delivered the uh portfolio results back to the schools. And he said, he called me up to say that the portfolio results were not as high and he was very disappointed that there was no diagnostic information in the results so that he could figure out what he was going to be doing next with his teachers. The portfolio information had come from his teachers. They had classified this student as apprentice and this student as proficient uh for a reason. This student was apprentice because he or she hadn’t acquired all the skills necessary to become proficient. Well what were the skills that that uh that were preventing the student from producing proficient work. The teacher knew that. The teacher didn’t need to call the contractor six months after the results were turned in to find out that the principle needed to go to the teacher and say what what you know what’s going on what can I do to support you so that so that you now have the skills to help to help this student who was only doing apprentice work that you classified as apprentice because it had these weaknesses what were the weaknesses and what resources do you need to get this student to to to strengthen those points and move on to the next level. So that’s my bias about what what it what it what assessment should be. So learning how to write test questions and learning how to develop scoring ---(?) and all of that I I don’t see if I had to prioritize that’s not where I would start with a classroom teacher. So if that’s the kind of instruction you’re getting boys and girls out there uh tell your teacher that you’re instructed that you need something else.

The questions that that that I would be asking myself if I were in a classroom are do am I collecting sufficient evidence that I could really classify every one of my students. One of the principles that we had in our Kentucky assessment was that the information about how students were doing was suppose to come from the teachers. That that if the system was working to way it was suppose to every teacher should know, every day where every student is. And let’s take writing portfolios for example. Uh cause writing is clearly is the easiest area in in in which to have success and I would encourage people getting into this to do something. Don’t don’t try to solve all the problems the first day you are in the classroom do something that you’re pretty sure you’re going to have success with. Learn the lessons from that and then build off that. So run a fairly traditional classroom if you will in certain ways. But experiment with uh with uh things that you might do and writing is usually the easiest. So if you are doing something like this, I mean principle number one is your students should be writing all of the time and you should be looking at what they’re writing all the time. Not to give it a grade, not to call it A B C or D but to identify what the strengths are and what the weaknesses are relative to the standards that either you or your district or the state has established as what constitutes good writing. Uh one one of one of the reasons why writing is a good area to start in is there’s pretty universal agreement at this point about what good writing looks like. There are certain elements of grammar and style that should be attended to but but there are more fundamental elements of good writing of the ability to organize the ability to communicate the ideas that you have, to write with voice, uh all of those are more sophisticated elements of writing that are that are the parts that that that uh people need to know. So my advice would be to number one have a clear picture in your mind of what good writing looks like. Either from your own experiences or from what other people tell you and not just is it prettier but but but what really are the fundamental elements of quality writing. And then have what ever your students write look at your idea model verses what they’re producing and ask yourself what the gap is between those two and why that gap exists. Then target your next instruction for and and the answers going to be different for every student or certainly there are going to be many different answers with in your classroom. Sometimes you can cluster them. That’s efficient and worth while doing if you can. But ask yourself the question of where the gaps exist and then and then think through from the way you learned how to write is a good starting point. How would you close that gap? How would you take the the the the lack that is in this piece of writing that is in this piece of writing over here that is a quality piece of writing and ----(?) what ever the weakness in the student that’s going to get them to move down the road towards writing the way this writing over here looks like. It’s a very powerful technique and it works in a very short period of time. And again the idea of that actually one of the things that I’ve just very recently picked up is is uh the importance of not trying to uh be successful in everything you do the first time out. Pick something that you think you’re going to be successful at, work at that, get the lessons from that, and then apply that to other areas, build.

Uh accountability um in the current viewpoint I think is externalized. I I think there are usually five dimensions in it. One is the purpose uh so accountability um is usually a tool to achieve some some goals um the ones that were most associated with are improving student learning. But here are other types of purposes um there there may be a limited number of resources and you’re trying to select those who are most um qualified for for those uh and so if you go outside of education or sometimes in in education people will be looking for ways to um allocate resources. A second part of accountability I think is -----(?) um who is doing what to whom. And that’s uh uh accountability systems that are coming in to states are seen a particularly external to the classroom. I think that’s a tremendous uh issue but it doesn’t have to to uh be that way. Um I think a third part of accountability is the evidence that presented. What is it that you’re basing whatever decisions or judgments on and um we work a lot on developing evidential systems for making accountability decisions. I think that there are two other parts of accountability that are really important, one is um the incentives or consequences usually the difference between a reporting system uh and an accountability system is there’s some consequence about people didn’t care about the information. There’d be there there would be very little accountability. I think the fourth is that um in education we’re very concerned about how accountability fits within a larger system. Um in particular saying who’s bad and who’s good with out providing resources or some intention to change that or to improve it doesn’t really fit very well.

I think that there there are five dimensions of accountability that that uh I I’ve been particularly interested in. Uh one is the uh purpose, the second is the um governess (?), the third is the evidence, the fourth is the incentives or consequences and fifth is how it fits into a larger system, particularly how accountability um fits in with improvement.

There are about um over thirty-five states now I think that are have some type of large-scale accountability system. Either in place or or developing or announced as a legislative intent. Uh almost all of these are are response to a legislative mandate. Very few of them have been a voluntary, this is what we think is the the uh best thing for education. Um the accountability systems usually have a couple of rationales behind them. One rational is that um comes from the business world um and often is tied to the history of large-scale accountability where there are funding increases often because of a court case about equity. Uh and the funding for education goes up dramatically then the business community and legislatures often say we want to make sure we’re getting our moneys worth. So if we but more money in show us that there is some results. I think a second uh very different rational is that uh people are saying we need to have more commonality. Uh we live in a community that is larger than our local community and whether that is because there are disparities in achievement or opportunity or we think that kids are moving around more in in the smaller society and and uh we need to have some common standards and and so on. That’s a second rational. I think a third one is um that large-scale accountability is seen as providing feedback for for uh different groups in a way that will provide some validation of what is being done locally. So there are very different reasons for uh accountability systems and sometimes the characteristics of those reflect those different purposes. Uh often the confusion comes because people say that they’re in one purpose but the accountability that’s ---(?) design better for one of those other purposes.

I think that one of the most um important perspectives that teachers can have is is that um is a shift from an external to an internal. What is it that uh that is good about these? Usually there not done in uh there not intended to be perverse or punitive although a lot of times we sometimes feel that they uh they are. Uh for example it’s um do teachers uh I wish uh as a teacher I might ask myself how much do I value this notion of commonality of common standards? Um often we see that teachers uh the great thing about American education is that it fosters individuality and entrepreneurship. When I shut the door I can do what I think is best. Um that is intentional with this idea that I’m the kids from my class are going to be going on to another teacher’s class next year and kid’s from two different classes or multiple classes may be mixed there. What does that teacher do if we are doing if the the preceding teachers are doing very different things. So this question about commonality I think is a very important one and uh the acc—the large scale accountability pushes that notion and it says there are some things that are very important for us to share. I think teachers need to uh answer that for themselves how much do I believe that and and uh I think the second one is um what information can I get out of of this that would be useful. I think most people agree that uh large scale assessments and accountability systems are not intended to be classroom diagnostic things and once we get past that teachers say what value is this to me. Is it is it doing what um I would like to have on a daily basis or the answers no and and they just uh we just need to look for different types of value from it. Usually those values are in um longer-range curricular or programmatic improvement.

I I was involved in a study once where we were watching uh teachers in classrooms and trying to code when they were giving feedback and uh turns out teachers um give a lot of feedback um and these were in elementary classrooms it’s different as you get higher because the interaction is different. But uh this teacher was giving an enormous amount of nonverbal feedback. For for example uh she’d ask a question and a student would answer and just by pausing the stu—it’s a type of feedback the students says uh the teacher says and the student says oh I’d better think about that. Uh or uh there’s a lot of disciplinary feedback going on. She may look at someone or or she may raise her finger and and there’s an enormous amount of feedback going on in classroom management and and in learning. Um I think that the the um important part of the on-going classroom assessment is how does it match with instruction uh and and assessment is a two way street. You’re getting information. Uh if a teacher gets a lot of information but has a limited instructional repertoire it doesn’t do much good I I I found out that that my twenty-five students are doing twenty-five different things but I can only teach two different ways. It it you need to have instructionally sensitive assessment but you also need to have assessments uh sensitive instruction.

I think teachers assess all the time. Uh and um I I think that part of our professional responsibility and to be what we want to be as teachers. We want to make a difference. Uh that assessment needs to run a a wide range and and so we’re we do a lot of informal assessment um sometimes we can develop that more either by knowing more about the content or different styles of learning and and trying to move that in. But I I think that uh there’s an opportunity to get a lot of information from these other assessments that are coming and so I I think one of the greatest needs is to change our role so that we aren’t resistant to these different types of information but to say to internalize these and how can I become good at using these different types of information that are now becoming available.