Overview
As far as the ‘universal evaluation tool’ does not exist, each trainer must develop his own evaluation tools. Nevertheless it is not necessary to invent the wheel again for each taining scenario! The eMEL experiments determined a set of concrete bases allowing each trainer doing it in the most efficient way following simple guidelines. This tutorial gives you general and specific guidelines to conceive, realize and interpret the results of your own evaluation test, specifically designed for your own use of the eMEL training scenarios in your specific context.
In this purpose, you’ll find six different models (we called ‘prototype’). Each prototype is focused on a specific range of competences (analysis competences, production competences, didactic analysis competences, didactic production competences) and show you several manners to evaluate it, with possible variants when it’s relevant. For each prototype, hyperlinks will also guide you to eMEL experimentations that used this specific prototype as model for their competence evaluation test. You can use it as models.
Each prototype is based on the same method: a single task repeated twice: before the training scenario (pre-test) and after the training scenario (post-test). Each test give information about the trainees’ level of competences at this specific moment (before and after the training). Comparison between the results of the two tests allows to highlight the trainees’ evolution during the training.
For each prototype, you’ll find:
- The set of competences the prototype is designed to evaluate;
- The main task the trainees will have to realize during the evaluation;
- The possible variants in the task or in the way presenting it to the trainees;
- The general data processing strategy and the evaluation scale and its signification;
- Eventual extra needed information;
- Examples of application in eMEL experiments.
You will find also a synthesis table of the 6 competences evaluation prototypes.
How to use it?
It’s quite easy. Follow these steps:
- Identify the competences you want to evaluate
- Find the prototype corresponding to these competences
- Choose a method (within the prototype description) in regard of your aims and/or your own context
- With the help of the prototype description, identify indicators specific to the (sub)competences you want to evaluate
- Define the exact task: how the generic task described in the prototype become a real concrete task for trainees? You have to define: the exact requirements, the documents to comment/analyse, possible extra precision (like a thema, audience, etc.). You should produce 2 questionnaires with this task: (i) the pre-test and (ii) the post-test questionnaires.
- Pass the pre-test questionnaire before the training scenario (provide enough time for it).
- Pass the post-test questionnaire after the training scenario (provide enough time for it).
- Correct the questionnaires with the help of the prototype interpretation table and the specific indicator you defined at step 4.
- You may make comparison between pre- and post-tests to assess the trainees evolution during the training scenario.
Evaluation prototypes list
- Prototype 1 – Analysing competences – Media Literacy by systematic analysis
- Prototype 2 – Analysing competences – Media Literacy by comparison
- Prototype 3 – Analysing competences – Media Literacy by classification
- Prototype 4 – Producing competences – Didactic axis
- Prototype 5 – Analysing competences – Didactic axis
- Prototype 6 – Media Literacy Production compétences
| # | Competence dimension | Method | Variants | Scoring method | Scoring indicators | Scoring levels | Conditions |
| 1 | Analysing competences – Media Literacy | Analyse media one by one | -Respond to observation questions -Analyse its practices |
A. Key words B. Content analysis (CA) |
A. Number of keywords B. CA based on the relevance, the accuracy, the coherence and the exhaustiveness |
4-levels scale build on the evaluated competence | Same evaluator for pre- and post-tests |
| 2 | Analysing competences – Media Literacy | Compare 2 (or more) media objects | -Free comparison -Comparison based on/guided by detailed questions -Mindmap: trainees have to draw a mindmap reflecting their analysis of the objects/of the question |
A. Key words B. Content analysis (CA) C. Coherence of the mindmap |
A. Number of keywords B. CA based on the relevance, the accuracy, the coherence and the exhaustiveness C. Coherence of the mindmap |
4-levels scale build on the evaluated competence | Same evaluator for pre- and post-tests |
| 3 | Analysing competences – Searching, selecting and classifying competences | Classifying media objects following differents criteria, arguing criteria | -Objects are given -Objects have to be searched -Possibility to ask for differents classifications following different criteria or for a specific task (i.e. “objects suitable for…”) |
A. Number of right/relevant criteria B. Justification of the criteria C. Ability to classify |
A. Number of right/relevant criteria B. Relevancy of the criteria (regarding the task) C. “Richness” of the classification (number of criteria in the classification) |
4-levels scale build on the evaluated competence | Evaluator will focus on:
A. Media objects relevancy |
| 4 | Producing competences – Didactic axis | Concieving (describing) a pedagogical situation | -Free page for responding -Formatted questions -Mindmap: trainees have to draw a mindmap reflecting their analysis of the objects/of the question |
A. Key words B. Content analysis (CA) C. Coherence of the mindmap |
A. Number of keywords B. CA based on the relevance, the accuracy, the coherence and the exhaustiveness C. Coherence of the mindmap |
4-levels scale build on the evaluated competence | Same evaluator for pre- and post-tests |
| 5 | Analysing Competences – Didactic axis | Analysing and criticizing an existing pedagogical resource | -Analysing & criticizing -Analysing, criticizing and adaptating to a specific (new) situation -Summarizing and commenting for a colleague |
A. Key words B. Content analysis (CA) |
A. Number of keywords B. CA based on the relevance, the accuracy, the coherence and the exhaustiveness |
4-levels scale build on the evaluated competence | Same evaluator for pre- and post-tests |
| 6 | ML Production | Producing a scenario presenting a media about [subject] to [audience] | -Free page to describe scenario -In two steps: (i) collect and analyse existing presentations; (ii) “what will you do?” |
A. Key words B. Content analysis (CA) |
A. Number of keywords B. CA based on the relevance, the accuracy, the coherence and the exhaustiveness |
4-levels scale build on the evaluated competence | Dependant of TS
Same evaluator for pre- and post-tests |
Download the full PDF of the Competences Evaluation Tutorial of e-MEL