Measurement Project Requirements

Your Measurement Project may be initiated as soon as you have successfully completed the necessary coursework. For option A, IP&T 652 is the only prerequisite. For options B, C, or D, IP&T 653 is also required.

The measurement project is designed to help the student apply information acquired through coursework and reading to actual measurement problems and settings. Additionally, the project allows the student to assume at least some of the responsibility for conduct of an actual measurement project. The project report and comprehensive exam should be written and submitted together as one document. Procedural instruction for meeting the Measurement and Comprehensive Exam requirements are described below.

STEP 1: Faculty Sponsor
STEP 2: Enroll for IP&T 657R
STEP 3: Prepare Your Project
STEP 4: Complete the Project
STEP 5: Project Options
STEP 6: The Written Report
STEP 7: Faculty Sponsor Evaluation and Final Grade
STEP 8: Comprehensive Exam
Prospectus Approval Form

A student desiring to complete a Measurement project must identify a suitable project in consultation with a faculty member and obtain the consent of the faculty member to serve as their sponsor for the project. In most instances, a student will co-author the report with the faculty sponsor. The sponsor may or may not be a member of the student's committee.

Enroll for three hours of IP&T 657R credit the semester the work is to be completed.

1. Prepare a prospectus - Prepare a prospectus describing the proposed measurement project and submit a copy to the Department Chair, along with the Prospectus Approval Form signed by the faculty sponsor (see page 6). The Prospectus and Approval Form must be submitted to the Department Chair before the work is begun, and no later than the second week of the semester in which the work is to be completed.
   
   NOTE: It is desirable that students define the primary purpose or focal issue of their Measurement project during the latter part of the semester prior to the semester in which they will work on the project.
   
   Students should retain a copy of the Prospectus Approval Form in their own files.
   
   
2. Must be a defensible effort - In some instances a project will be the outgrowth of one of the two measurement courses (IP&T 652 or 653), or an Internship. When this is the case, steps should be taken to ensure that the effort is defensible in terms of the three hours credit awarded for the project. This is accomplished by establishing, in writing, that the two activities are going to be combined and by having the joint effort approved by the faculty sponsor and the Department chair in advance.
   
   It is not permissible to be awarded course or Internship credit and project credit for the same activity unless the total time devoted to the activity is commensurate with the total amount of credit being awarded and approved by the faculty sponsor. Double credit is not permissible.
   
   The measurement and Comprehensive Exam requirement may be challenged by making a formal request in writing to the Department Chair and by submitting a report of a suitable measurement project written prior to entering the Instructional Psychology & Technology program, along with supporting documentation as described in Step 5. Such reports will undergo the same faculty review as those which received advance approval.
   
   
3. Funding - The Instructional Psychology & Technology Department does not provide direct financial support for projects. However, the student's time and expenses related to a project may be funded by a sponsoring faculty member or agency. If a student is unable to find a sponsor who is able to fund the project, the costs must be borne by the student.
   
   
4. Schedule - Successful completion of a Measurement or Evaluation Project is a necessary step in qualifying for doctoral candidacy; therefore, students cannot begin formal work on a dissertation before the requirement is met. However, informal work (reading, assembling related research, pilot studies, etc.) is encouraged as early as possible in the program.

After obtaining approval of the prospectus, the student may proceed with the measurement project under the supervision of the faculty sponsor. The student will submit those documents required by the faculty sponsor. The faculty sponsor will provide the criteria for each required document.

When a member of the faculty agrees to serve as a sponsor for a measurement project, that faculty member agrees to provide the student individual consultation regarding the project on a regular basis and to review the student's work systematically. How the measurement project is accomplished is left up to the discretion of the faculty sponsor. The only stipulation is that the faculty sponsor agrees to review whatever the student writes in connection with the project. At the outset of the project, the faculty member and the student should determine specifically what the sub-tasks are going to be and agree on deadlines for completing each sub-task. The schedule in the prospectus should reflect these deadlines.

The requirements for the Measurement Project can be fulfilled by successfully completing one of the four options described below.

Option A. Construct a new assessment instrument or procedure

1. The instrument or procedure to be developed can be selected from any of the following categories:
1. An achievement test (A test intended for one-time use in a classroom setting is not acceptable. To be acceptable, the test must be good enough to warrant repeated use in assessing a reasonably stable domain that includes intended outcomes beyond the recall level.)
2. A performance test for assessing learners’ ability to do some complex task that is valued in the workaday world outside of school contexts (The resulting test should include all displays, materials, equipment, and instructions necessary for actually administering the test.)
3. A diagnostic test of learners’ understanding/misunderstanding of the concepts, principles, rules, or procedures included in some domain that is a prerequisite for successful performance or further study in the subject.
4. A scale, inventory, or other instrument assessing one or more affective characteristics (attitudes, values, interests, anxieties, locus of control, academic self-esteem, etc.) related to some instructional experience or educational issue.
2. In addition to the instrument or procedures developed, the following ancillary products must be included to fulfill this option:
1. The set of written specifications used to guide the development of the instrument or procedure.
2. Evidence of an adequate review of relevant research, theory, or subject matter content.
3. Evidence of formative tryout and revision including item analysis data, reliability estimates, and at least content evidence of validity.
4. A User’s Manual including procedures for administering the instrument or procedure, scoring procedures and criteria, and guidelines for interpreting the resulting scores.

Option B. Conduct a validation study of an existing assessment instrument

1. Prepare a written summary of any previously conducted reliability and validation studies. Include an assessment of the strengths and weaknesses of these studies, and identify validity issues which have not been adequately addressed.
2. Design and conduct a study to collect additional evidence of the degree to which scores obtained from the instrument in question possess or lack validity. (Remember that establishing validity is a cumulative, ongoing process and that no single study is likely to provide conclusive answers to all validity issues. The goal here is to make substantive contribution to this ongoing process.)

Option C. Conduct a study focused on a practical measurement need or problem faced by a test-user

1. The study could include one of the following:
1. A standard setting study to empirically determine what cut-off scores should be used to make mastery-nonmastery, pass-fail, or admit-deny kinds of classification decisions based on test scores.
2. A study to detect test items that may be biased (differential item functioning) against some definable subgroup of the examinee population.
3. An equating study designed to equate scores from one test with scores from a different, but similar measure (e.g. as the ACT and the SAT).
4. A norming study that will provide relevant and representative norms for interpreting scores from a newly-created instrument or updated norms for an existing instrument.
5. Estimate the relative magnitude of different sources of error in supervising teachers’ ratings of student teachers.
6. Some other similar problem or issue approved by your project sponsor.
2. Prepare a written report for the client which describes the procedures used, presents the results, discusses the limitations of the study, and offers specific recommendations based on the results.

Option D. Conduct research on an unresolved methodological issue

1. Select some unresolved methodological issue related to educational measurement and conduct a research study designed to help resolve this issue. Some possible problems or issues include--
1. Factors influencing the reliability of scores obtained from performance tests
2. The cost-effectiveness and feasibility of using “authentic” tests
3. Problems associated with the use of partial-credit scoring
4. Problems of using formula scoring with multiple-choice tests
5. Problems of using complex multiple-choice (Type K) items
6. Effects of using different approaches to standard setting
7. Effects of violating unidimensionality in applications of item response theory
8. Effects of ‘fat’ versus ‘slim’ matching in Mantel-Haenszel studies of differential item functioning
9. Problems associated with the use of technology in assessing student knowledge or performance
2. Prepare a report that defines the purpose of the study, summarizes relevant previous research, describes the method used, presents the results, and interprets the findings in light of previous research and present practice and theory. The report should be suitable for submission to one of the following journals or an equivalent:
1. Journal of Educational Measurement
2. Applied Measurement in Education
3. Applied Psychological Measurement
4. Educational and Psychological Measurement

Regardless of which option is selected, a written report must be prepared. This document should include at least the following information:

1. The purpose of the project and the problem, questions, hypotheses, or issues it was designed to address.
2. A rationale or justification explaining the need the study was intended to fill or the reasons why it needed to be done.
3. A critical, integrative review of relevant research literature.
4. The method used to collect data including a description of the design, sample, instrumentation, and procedures used.
5. The results of the study including any products created and statistical findings.
6. Conclusions or recommendations accompanied by a discussion of the implications of the findings for instructional or measurement theory and practice.
7. A critique specifying the strengths and weaknesses of the study in light of existing theory in the field and describing what you learned from the experience and how you would improve the project if you were to do it again.
8. The proposed schedule submitted with the prospectus, an actual schedule describing how the study was actually conducted, and an explanation for any discrepancies between the proposed and actual schedule.
9. A project budget comparing expenditures and actual expenditures plus an explanation of discrepancies.
   

If you choose option A, a copy of all instruments and ancillary products should be included as appendices to your report. Data summarizing the results of tryouts and revisions should be included either as tables in the body of the report or in the appendices.

If you choose option C or D, the report you prepared in Step 5 will substitute for items 1-6 in the above list, but must be accompanied by a written critique incorporating items 7-9 from the above list.

In preparing your measurement project report, you should carefully follow the style guidelines outlined in the Publication Manual of the American Psychological Association. You should consult the APA Guidelines web site prepared by the IP&T Department that outlines some of the ways to avoid the most common errors found in student papers and enumerates some variations from the guidelines recommended for dissertations and other student papers. You should also access the Microsoft Word Heading Styles Tutorial on how to use styles in Microsoft Word to facilitate the implementation of the APA heading guidelines. This tutorial includes a link to a Microsoft Word template document that includes all the necessary styles already defined.

When the faculty sponsor is satisfied the student has demonstrated sufficient originality and quality in the report, he will sign the attached evaluation report (see page 7) verifying the student has satisfied the Measurement project requirements.

The student will then submit two copies of the final project, along with the faculty sponsor evaluation report to the Department Secretary. The Department Secretary will then send the project out to two additional readers for review (see Step 8). The student will receive a final grade for the project after the project passes these two readers. The sponsor and readers will each grade the project, the final grade will be an average of the three grades received. The grade will not be submitted to the Records Office until this Comprehensive Exam has been passed.

The completed project documents will be submitted by the Department Secretary to two additional IP&T faculty members for their evaluation. The reviewers should judge the quality of the project and Comprehensive Exam in terms of the criteria listed on the Faculty Evaluation Report form (see page 7). (Note that these reviewers are not required to judge the originality of the work, or determine if the project requirements have been met. This is the responsibility of the sponsor. However, they are required to review the quality of all aspects of the project as outlined on the Faculty Evaluation form.) The results of these evaluations will be submitted on the Evaluation of Examination Form (56) to the Department Chair.

The Measurement Project Prospectus Approval Form can be downloaded here:

Word Format (.doc)