UEA 1960's Images - Pete Goodrum 23/08/13

Moderation of Assessment

UEA Policy on the Moderation of Assessment - 2022/23

1. Preamble

1.1 Chapter B6: Assessment of Students and the Recognition of Prior Learning (2013) of the UK Quality Code for Higher Education states that assessment is not a linear process but “an ongoing cycle through which staff design, set, mark, engage in dialogue about performance, review and develop assessments.” The Quality Code’s Expectation about student assessment is that an institution’s policies, regulations and processes result in “equitable, valid and reliable processes of assessment ... which enable every student to demonstrate the extent to which they have achieved the intended learning outcomes for the credit or qualification being sought”.

1.2 Indicator 4 of Chapter B6 states that all staff involved in assessment processes must possess the necessary knowledge and skills to undertake their roles including appropriate training or development opportunities. The result should be a shared understanding (a “Community of Practice”) amongst Academic and Teaching Teams¹ of appropriate assessment practices within a particular academic discipline and standards (including the types of evidence that demonstrate differing qualities of student performance).

1.3 The means by which much of the shared understanding of assessment develops is through the framework of expectations laid out in the University’s Policy on the Moderation of Assessment. The Policy applies to all taught programmes of study leading to an award by the University or an award of credit.

1.4 “Moderation” refers to a holistic process designed to ensure the appropriate quality of summative assessment tasks² with respect to the stated learning outcomes of the course or module, including the valid and consistent judgement of student performance on these tasks. This is intended to assure the quality and integrity of the University’s certification of student achievement. In line with Indicator 10 of Chapter B6 the process of moderation is intended to ensure that all students for all assessments have equivalent opportunities to equitably demonstrate the learning outcomes related to a particular assessment and to have these demonstrations justly recognised.³

1.5 Aside from quality assurance, the process of moderation crucially contributes to the continuous enhancement of assessment practices in order to improve the quality of student learning opportunities within the University.

1.6 In light of the cycle of assessment described in 1.1 the Policy assumes a moderation process consisting of a co-ordinated set of practices directed at the following three broad stages of the assessment cycle:

Assessment Design and Development (“Assessment-Moderation”): occurring before the assessment is released to students;
Marking and Grading (“Mark(ing)-Moderation”): occurring before the return of the marks to students;
Review and Evaluation (“Review-Moderation”): occurring after the marks have been returned to students.

The outputs of Review-Moderation will subsequently inform the assessment design and development phase in iterations of the cycle in future years.

1.7 It should be recognised that the process of moderation does not necessarily provide an absolute assurance of the veracity of a particular assessment task or the marks awarded but is designed to deliver as high a level of confidence as possible for all stakeholders (students, staff, the University, employers and the wider public) in the high quality of the assessment practices of the University.

1.7 The Policy permits a degree of discretion amongst Module Organisers and Academic Teams to determine the forms of moderation most suitable to particular assessments within their discipline. Where a Professional, Statutory or Regulatory body, however, requires more stringent conditions than those specified by this Policy, the former take precedence.

1.8 The Module Organiser is responsible for ensuring that Assessment-Moderation and Marking-Moderation is properly carried out.⁴ Review-Moderation forms part of Internal Quality Assurance/Quality Review Framework.

FOOTNOTES

[1) “Teaching Team” is used to refer to those directly involved in the delivery of a particular module. “Academic Team” is used to refer to the Teaching team as well as the broader constituency of Course Directors, Teaching Director, Moderators and other individuals with an interest in the particular modules. External Examiners may also be considered part of the Academic team (see 2.6).

[2] Although the focus of the Policy is on summative assessment tasks, teaching teams and Module Organisers should also reflect on whether formative tasks are consistent with the precepts of the Policy.

[3] Within the Policy:

“assessment” is used to refer to the general process whereby academic staff form judgements as to the extent that students have achieved the learning outcomes of a course, a module or piece of work in a module undertaken by a student;
“assessment task” refers to any item of assessment (whether examination, course test, coursework, presentation, etc.); on occasion an assessment task, where the context is clear, may be referred to simply as an “assessment” for stylistic variation;
“assignment” refers to assessment tasks usually undertaken by the student(s) in their own time with an associated submission deadline.

[4] In some circumstances a suitable alternative person shall be responsible (for example, in the Norwich Medical School this may be the assessment lead). Depending on circumstances, the Module Organiser or other responsible person may not be directly involved in the sampling or moderation process, but shall still be responsible for ensuring that it is done.

2. Assessment Design and Development Moderation

2.1The Assessment Design and Development stage of moderation (“Assessment-Moderation”) focuses on the nature and description of all summative assessment tasks prior to their release to students. The central goal of Assessment-Moderation is to ensure that all module assessment tasks are transparent, valid, rigorous and as equitable as possible for all students.

2.2The degree of scrutiny required for Assessment-Moderation is left at the discretion of the Teaching and Academic teams. Factors such as the inexperience of the Module Organiser, whether the assessment task is new or where similar tasks have received negative feedback in the previous years’ Review-Moderation (including comments by External Examiners) are likely to require more rigorous moderation than longer-standing assessment tasks where there are only minor modifications year-on-year.

2.3The moderator(s) involved in Assessment-Moderation, wherever possible, should not be part of the Teaching team in order to provide suitable external scrutiny.

2.4The moderator(s) involved in Assessment-Moderation should possess sufficient discipline knowledge and experience to be able to offer appropriate and constructive comments on the proposed assessment tasks.

2.5It is recognised that in some cases where there is insufficient subject knowledge outside of the Teaching Team, it may prove difficult for the School to provide moderators who are able to properly judge the subject content of the assessment task. The moderator(s) should, however, be of sufficient experience to be able to meaningfully explore with the Teaching Team broad content issues if not all specifics; in some cases it may be possible to ask the relevant External Examiner for their comments (see §2.6). It should be noted that, as outlined in §2.7, Assessment-Moderation is intended to scrutinise more than subject discipline content.

2.6Assessment-Moderation should be guided by the precepts of the University’s Policy on Assessment Design.

2.7Once Assessment-Moderation has been satisfactorily completed, the assessment task(s) may be made available to the students.

3. Assignment Briefs

3.1In order that Assessment-Moderation can proceed appropriately, the Module Organiser will need to provide a written description of the assignment task. Such descriptions will be referred to as “assignment briefs”. The assignment brief will eventually be released to the students as the core information point specifying the assessment-task.

3.2 An assignment brief simultaneously fulfils a number of different functions:

it is the primary document for Assessment-Moderation;
it provides an important aid for Marking-Moderation by outlining the precise nature of the assignment task and the assessment criteria;
it provides a useful brief for Academic Advisers when discussing assessment performance and feedback with their Advisees;
it is the core document provided to students specifying the details of the assignment’s requirements and expectations, learning outcomes and marking criteria.

3.3It is recognised that assignment briefs will take different forms depending upon the particular assessment task and the nature of the subject discipline.

3.4It is a crucial part of Assessment-Moderation that the assignment brief is scrutinised from the perspective of the student regarding its content and communicative clarity.

3.5The Module Organiser will ensure that students are provided with an assignment brief for all elements of summative assessment associated with the module. This should normally be provided at the earliest opportunity in advance of the submission date.

4. Marking Moderation

4.1 All summative assessments (including delayed first sits and reassessments), irrespective of the mode of assessment, from level 3 (Foundation Year) to 7 (Masters) will be Mark-Moderated irrespective of the percentage of their contribution to the overall module mark.

4.2 The mark or grade awarded to any assessment task must reflect how well a student has demonstrated achieving the learning outcomes and other requirements of the task in accordance with the marking criteria as specified in the assignment brief. The returned work must also provide clear feedback to the student indicating how the mark was derived against the learning outcomes and marking descriptor as well as indicating how the student may improve on future assessments (“feedforward”).

4.3 Marking-Moderation is designed to ensure that the assessment outcome (e.g. a mark or a pass/fail grade) is equitable, consistent with the assessment criteria and reliable (i.e. comparable judgments are made across the cohort) and with any differences in academic judgement between markers both formally acknowledged and resolved.⁵ Moderation of the feedback to the students is a crucial component of the moderation process which involves questions of quality assurance and, possibly, enhancement.

4.4 It is the role of the Mark-Moderator to ensure that the requirements of §4.2 and §4.3 are consistent with the information contained in the assignment brief (learning outcomes, mark descriptors, marking schema, etc.). Further consideration may also relate to generic feedback, model answers, etc. Mark-moderation should also include reflection on the overall distribution of marks for the assessment. Other statistical measures may be used where thought relevant and appropriate in assessing the students’ performance on the assessment task.

4.5 The Mark-Moderator will complete and submit a report detailing the nature and outcomes of the moderation process.

4.6 Marking-Moderation for all assessments submitted by the appropriate deadline must be completed before the work is returned to LTS and, subsequently, the students.

4.7 Mark-Moderation of late submitted work is not required if this will delay return of the marks to students beyond the University’s standard turnaround period. The current moderation policy does not require moderation of late submissions, however, schools (via Course Director or delegate) should consider moderating a sample of late submissions if they do not consider that the on-time submissions provide an adequate range of marks or number of assessments for thorough moderation. This should be factored into the estimated date of feedback and marks of late submissions to students.

4.8 Wherever possible Mark-Moderators should not have been involved in the first marking of the assessment task. They should also possess appropriate subject expertise and Higher Education experience (at least two years) in marking.

4.9 Although a Module Organiser may wish to keep a check on members of a team of markers during the course of marking, this does not constitute “moderation” in the sense of Marking-Moderation in this Policy.

4.10 Where specialised expertise is not available, the assigned Marking-Moderator should still be able to make judgements about the consistency of marking and the quality of the feedback but will need to be guided by the first marker’s individual marks and subject-specific content in the feedback.

In some cases it may be decided that external moderation may be needed to ensure quality assurance. If so, an application to the relevant Faculty Associate Dean for Teaching and Learning Quality should be made outlining the reasons. On approval, the School will appoint an external moderator with the required expertise. The marked assessment will be returned to LTS/students within the normal deadlines but the marks will be deemed to be provisional until scrutinised by the external moderator. Copies of the work and the distribution of marks will be sent to the external moderator for review.

4.11 Pre-marking moderation (or standardisation) is considered best practice when more than a single first marker is involved with an assessment task in order to ensure that there is a shared understanding of how to apply the marking criteria to achieve consistency of marking across the team. Pre-marking moderation should also reach an agreement as to the quantity, style and focus of feedback provided to the students.

If pre-marking moderation is not practical, the Module Organiser may need to check consistency across the marking-team during marking.

4.12 Post-marking moderation, the main focus of this Policy, happens once the assessment task has been marked by the first marker(s) and is designed to ensure that the goals of equability, consistency and reliability in academic judgement have been duly met and to assess the appropriateness and quality of the feedback to the student.

4.13 Marking-Moderation may be classified along two dimensions:

(a) Whether the moderator is aware of original marker’s marks, comments and feedback or not:

Blind double marking: the two markers mark the work unaware of the other marker’s mark, comments and student feedback. Through discussion the two agree a single mark and appropriate feedback.⁶
Second marking (non-blind moderation): the moderator has access to the first marker’s mark, comments and student feedback and verifies the equability, consistency and reliability of the academic judgement along with the quality and appropriateness of the feedback.

(b) The proportion of submitted assessments scrutinised:

All submitted assessments moderated.
A sample of assessments moderated.

4.14In principle both dimensions of §4.11 can be combined so that either all or a sample of assessments may be blind double or second marked.

4.15Where a subset of the overall submission is used for Mark-Moderation, the minimum sample size will be √ n(where n is the number of assessments submitted by the published deadline and √ n is rounded up to the nearest integer) in addition to all failing assessments. For example, given 10 submissions, √10 = 3.2 which rounds up to a sample size of 4 (plus any failing scripts). Aside from the fails, the sample should include examples from the full range of marks.

4.16Where marking is conducted by a team of markers, the sample shall be √n for each member of the team.

4.17 Since Marking-Moderation cannot provide an absolute assurance of the veracity of academic judgements and since moderation comes with significant resource implications, Teaching and Academic Teams during Assessment-Moderation are advised to consider a “cost/benefit” calculation regarding the type of marking moderation which delivers the most reasonable level of confidence in the quality of the marking process given the required resource.

4.18 Special consideration needs to be taken over the moderation of assessment tasks which carry particular risks of being inequitable or inconsistent. In general it is for the Assessment-Moderation process to identify which assessments may be at risk and to determine the method of marking best suited to mitigate the issues.

4.19 For oral examinations and presentations, the first marker(s) and Mark-Moderator may each be in attendance at the assessment event. In such cases, marks will be awarded without consultation before discussion results in an agreed mark. If the moderator is not in attendance, a video or sound recording⁷ of suitable quality made for post-event moderation will be required.

4.20 Where a station/item for an assessment by an Objective Structured Clinical Examination (OSCE) or Objective Structured Practical Examination (OSPE) is double marked (i.e. two assessors are present at the station), further moderation is not required. Where an OSCE or OSPE is assessed by a single marker, a suitable moderation process should be in place; for example

identified individuals appointed to oversee and observe the assessment practice across a sample of stations and assessors;
video or sound recording⁸ of a sample of stations for later review;
post-assessment analysis of assessor behaviour using relevant data.

4.21 Where an External Examiner is routinely scrutinising a module, the sample of scripts sent should normally include, at least, the moderated sample (or part thereof) along with the Mark-Moderator’s report.

4.22 On occasion, the Mark-Moderator may find their judgements differ significantly from those of the first marker(s). Such discrepancies need to be carefully resolved through discussion between the marker(s) and Mark-Moderator in order to agree a final mark.

4.22.1 Individual marks may only be changed if the individual submission has been Blind double marked or the complete set of submissions has been moderated.

4.22.2 With respect to second marking of a sample where the moderator identifies significant differences with their own judgements (for example where there is a 10% or more variation between the original mark and that of the moderator in, at least, half of the moderated submissions or the averaged marks across the sample for the two markers is greater than 5% suggesting a consistent pattern of disparity) or an anomalous distribution of marks, the moderator can suggest remarking or a suitable adjustment of marks (scaling):

for all submissions irrespective of they have been moderated or not, i.e. not for individual submissions (except where a pass is felt by the moderator to be a fail and there are implications for a student’s fitness to practise);
for all of the work by individual markers irrespective of they have been moderated or not (but not individual submissions);
for sections within a submission for all submissions (e.g. where a problem is identified relating to one question on an examination paper).

4.22.3 If the markers cannot satisfactorily resolve their differences, an appropriate third party will adjudicate. The adjudicator should be an appropriate office holder in the School such as:

Chair of the Board of Examiners;
School Director of Learning and Teaching;
Assessment Coordinator/Lead or equivalent in those Schools which possess such roles.

4.22.4 The adjudicator is required to arbitrate on the nature of the disagreement between the markers rather than re-mark the assessment itself. Accordingly, the adjudicator will mainly consider the marks and comments of the two markers, including the Moderator’s report. In some cases the adjudicator may wish take advice from a third party, including the External Examiner(s).

4.22.5 The adjudicator shall be responsible for making the final judgement. The awarded mark should be within the range bounded by the first marker’s and moderator’s marks nor the two markers in the case of Blind double marking.

4.22.6 Before the return of work where disagreements in judgment have been identified and discussed, the feedback must be checked for consistency with the final agreed mark.

4.23A clear and transparent audit trail is required detailing Mark-Moderation and recorded through the Moderator’s report. The report should include the form of moderation used (blind double or second marking moderation) and the particulars of the sample. Where marking discrepancies have been discussed, the report should provide details of the nature of the differences (including the raw marks) and the method and rationale by which the agreed mark(s) was/were arrived at (including any adjudication).

FOOTNOTES

[5] In the case of more objectively marked tasks, Marking-Moderation may only require an assurance of procedural regularity. Examples of such cases would include Multiple Choice Question tests or examinations or assessments where there is a clearly defined marks schema (for example in some forms of Mathematics assessment).

[6] With Blind Double Marking the notions of “first marker” and “moderator” do not apply although the overall process is one of Marking-Moderation.

[7] NB students must consent to any recording.

[8] NB students must consent to any recording.

External Examiners and External Moderation

5.1 As part of their duties, External Examiners are expected, not only to ensure that the academic standards for UEA awards are at the appropriate level in comparison with other Higher Education Institutions, but also to confirm that assessment processes appropriately measure student achievement against learning outcomes in ways which are transparent, valid, rigorous and as equitable as possible for all students.

5.2 External Examiners should be encouraged to be involved at all three levels of moderation, including Assessment-Moderation, to allow their expertise to “inform [the] Institution’s practice as it occurs, rather than providing an exclusively retrospective comment on past practice” (UK Quality Code for HE, Chapter B7: External Examining). Any sample sent to the External Examiner for scrutiny should, at a minimum, include those scripts internally moderated plus an additional 10% or 3, whichever is the larger. The External Examiner may request a larger sample on request.

5.3 The particular duties of an External Examiner are laid out in the University’s Code of Practice for the External Examiner System (login required).

Review and Evaluation Moderation

The details of this section are currently under review as part of the implementation of the Quality Review Framework.