Wednesday, September 11, 2013

Educators slam flaws in Value Added Model



Editor's note: The letter below was sent from the administration and faculty of  the W.W. Lewis Middle School in Calcasieu Parish to members of the Board of Elementary and Secondary Education and officials in the Department of Education:





This is being sent with endorsement from Wayne Savoy, Superintendent of Education for Calcasieu Parish Schools. 

Attention BESE Board and LADOE Officials:

     We wish to address three major concerns that could have a negative effect on VAM scores for many 8th grade teachers as well as student scores across the state.  The first issue effects 8th grade teachers and the other two affect all teachers. 

     The first concern is with the construction of 8th grade LEAP tests and how the current test affects teachers’ VAM scores.       
In middle school, Sciences and Social Studies are taught in the following grades:
                        Science                      Social Studies
        6th grade    Physical Science          World History
7th grade    Life Science                        American History
8th grade    Earth Science             Louisiana History

6th and 7th grade students take an iLEAP test that covers only the material they were taught during that school year.  However, in 8th grade the students take a test that is comprehensive for all three middle school years.  The Science LEAP test covers not only the Earth Science that 8th grade students were taught, but Physical and Life Sciences from 6th and 7th grades as well.  We understand the need for a comprehensive exam, but to have one teacher’s VAM score tied to what other teachers have or have not taught makes the score invalid. 
     The Social Studies LEAP covers not only the Louisiana History that 8th grade students were taught, but also World and American History from 6th and 7th grade.  On the LEAP test, students are then required to remember or reproduce information that they were taught 1-2 years earlier.  8th grade teachers have so much material to cover from their own curriculum that they cannot possibly address 6th and 7th grade GLEs as well, yet the students are tested on that material. 

8th grade Social Studies LEAP test:
Strands/ Categories

Items

% of
Total
points






GEOGRAPHY





  A. The World in Spatial Terms





  B. Places and Regions

15

25

  C. Physical and Human Systems





  D. Environment and Society





CIVICS





  A. Structure and Purposes of Government




  B. Foundations of the American Political System
12

20

  C. International Relationships





  D. Roles of the Citizen





ECONOMICS





  A. Fundamental Economic Concepts




  B. Individuals, Households, Businesses
9

15

      and Governments





  C. The Economy as a Whole





HISTORY





  A. Historical Thinking Skills





  B. United States History

24

40

  C. World History





  D. Louisiana History











TOTAL

60

100


A large percentage of this test comes from material the students were to have learned in grades 6-7, so the 8th grade teacher gets a VAM score based on what two other teachers have done or not done in grades 6-7.  The VAM score is therefore invalid.  

     For Social Studies, even though much of the material for all three grades comes from the same strand and/or benchmark, the teaching would be specific for the area of history being studied.  For example; all three grade levels teach Foundations of the American Political System from the Civics strand.  However, it is from three different perspectives- World, American, and Louisiana.  While this is a key concept and must be explored by students from all three perspectives and in all three grade levels, an 8th grade teacher whose focus is on Louisiana should not be held accountable for the same concept from a World view point that was taught two years earlier.

8th grade Science LEAP test:

Strand
Multiple
Short
ECR
Score

Choice
Answer
Items
Points

(1 point)
(2 points)
(4 points)

1. Science as Inquiry
8
0

8
2. Physical Science
8
1

10
3. Life Science
8
1

10
4. Earth and Space Science
8
1

10
5. Science and the Environment
8
1

10





Comprehensive Science Task




1. Science as Inquiry

1

2
   Dimension 1 (Questioning,




   Planning, Doing and




   Recording)




1. Science as Inquiry

2

4
   Dimension 2 (Interpreting and Communicating)




2. Physical Science


1

3. Life Science


(in each of
4
4. Earth and Space Science


the four

5. Science and the Environment


strands)











Total Score Points
40
14
4
58

The LEAP test is approximately 24% Physical science (taught in 6th grade), 17% Earth Science (taught in 8th grade), 17% Life Science (taught in 7th grade), 17% Environmental Science (taught in all grades) and 24% Inquiry.  This shows that at least 41% of the 8th grade test is based on 6th and 7th grade material.  The 6th and 7th grade tests do not include Science task questions in which students are required to set up graphs and draw conclusions based on a given scenario.  This could be an issue if the information needed to complete the task came from another grade level’s material.  Neither of these graphs reflects the fact that a constructive response item could come from another grade level’s material.  

     It is difficult enough to expect an 8th grade student to write on a topic that he or she was taught 2 years earlier, and invalid to have the score reflect on an 8th grade teacher.  The student has already taken an iLEAP test on that material.  Shouldn’t that test be enough to determine Proficiency without it reflecting on an 8th grade teacher?   

     The 8th grade math and ELA LEAP tests are set up in a similar manner (as comprehensive middle school exams) but the results are not as potentially harmful to the teacher.  Both of those subjects have concepts that are built upon, unlike Science and Social Studies who have so many isolated GLEs.  The problem then is if 6-7th grade teachers have not been successful in teaching the concepts, the 8th grade teacher’s VAM score suffers and is therefore invalid.    

     In schools with a high student turnover rate, an 8th grade teacher’s score could be affected by students who studied 6th or 7th grade Science or SS at another school, multiple schools, or even in another state.  

     Shouldn’t a teacher’s VAM score be based solely on what that teacher has done with a student?  How can the state rate a teacher “Ineffective” based on another teacher’s effectiveness in teaching 1-2 years earlier?

     One possible solution would be to break a student’s 8th grade score into two categories- one for a comprehensive Science and Social Studies score and one score which reflects only what a student scored on material from his or her 8th grade subject.  By doing this, the teacher’s VAM score could be a valid representation of only what that 8th grade teacher was responsible for teaching.   


     Our second concern is in the form of a question.  Our current understanding is that a student’s test score history for several years is used to predict an iLEAP or LEAP score for that student.  Following this premise, if a student scored Unsat for 2-3 years in a subject, then his predicted score would be Unsat as well.  Our question is- If the student does score Unsat, as predicted, would that score still count against the teacher because he was not Proficient even though Unsat is the predicted score?  If so, how is that valid for a teacher’s VAM score?   


     Part A of our third concern is about the attendance factor for SLTs.  We were told to use 80% attendance as the factor by which we would determine which students would and would not be included in our SLT.  No one questioned that, and we just added it to our SLTs as directed.  Recently we realized what 80% attendance really means.
  • 80% attendance is a student missing school one day a week, every week
  • Could a teacher be effective if a student missed one day a week?
  • Think about the repercussions on student learning if teachers missed one day a week every week
  • 80% means a student could miss 36 days of school a year and still count towards their SLT
  • 10 absences a year is the maximum- by the state's own standard
  • 80% attendance means we have basically not excluded anyone from our SLT count
  • 80% means we are still accountable for those who do not value education enough to attend school

Even if we increase the attendance factor to 90%, a student could potentially miss 18 days a year and still count against a teacher’s SLT and VAM score.  We do not believe students who miss 18-36 days a year can grasp the numerous concepts taught in six or more subjects.

Looking at it from this perspective, if teachers verify their CVR roster from 1 Oct until testing, that is about 175 actual school days minus about 30 days from the start of school until 1 Oct and about 20 days after testing.  That leaves roughly a 125 day window.  Using 125 days, 80% attendance still means a student could be absent 25 days and count against a teacher on their SLT.  Increasing the attendance rate to 90% and using the CVR window means a student could be absent 12.5 days.  This seems a little more realistic for SLTs and VAM scores, although still above the state’s maximum 10 day absentee policy.  
 
     Part B of the third concern is that we have been told that an alternate formula will be used to calculate predicted test scores for students with attendance problems.  This leads us to believe that the same 80% attendance variable will be used by the state for VAM teachers to determine which students’ scores are calculated with which formula.  The numbers in the above paragraphs prove that 80% attendance is not a valid number.  It removes few if any students from the equation used to determine teacher effectiveness in VAM scores or SLTs. 

     We respectfully request that the BESE Board and State Department of Education work to resolve these issues as soon as possible.  No one would want teachers to be rated ineffective because of invalid data.  With tenure and possible teacher compensation relying on teacher effectiveness and test scores, the consequences could be devastating to careers and school systems. 


No comments: