Background

Computer-assisted marking formulae can greatly assist, or even totally eliminate, the manual marking process for assessors.

The formulae instruct the system to identify keywords and phrases that can be ‘scored’.  When these keywords and phrases are identified, they are underlined in green on the marking screen.  A passing score can be set for the question.  When the required score is reached the system will display a green ‘thumbs-up’ symbol and the passing score.  This allows the assessor to confidently mark the submission as correct.


                                                                    Fig. 1.



The formulae can be set to highlight only, and not score.  Just to help assessors note when keywords and phrases have been used.  In these cases, the underlining will show in blue.

The system can be set, on a question by question basis, so that any passing free text submission is passed automatically and does not need to be passed by an assessor. 


Creating the C.A.M. formulae

  1. Navigate to the free text (essay) question you want to add CAM to in the Quiz Builder module and click on the question to open the question settings.

                                            Fig.2.

  2. Scroll to the bottom of the page to ‘Automated Scoring Formula” and click on the ‘(Edit)’ link.

                                            Fig.3.

  3. You can build the formulae in the scratchpad, send system queries and download actual results.  Your final formulae can be then copied and pasted into the Automated Scoring Formula text box.

  4. Automated Scoring Formula text box:
    1. The first line is where you add in your passing score for the question.  In the pictured example in fig.4, the passing score for the question is set at 100.
      Meaning of True, x, y is…
      Init(Enabled, PassScore, FailScore)
      Init(Enabled
      the Enabled means "allow full auto marking to occur".
      If enabled is true and if the pass/fail scores are provided, and the score is a pass or fail, then the computer will immediately auto-mark. This is "essay auto marking".
      If enabled is false, then the computer will never auto-mark, but the trainer will see the score and thumbs up/down advice as they manually mark. We call this "assisted manual marking".

      Init(True, passing score, failing score) +

    2. It is a good practice to have the next line of code provide the details of when the formula was last updated.  You can see in the example in fig.4. that the pictured formula was last updated on January 31, 2017, at 1:30 pm by MR.

      This uses the ‘AddTerm’ function with the term being ‘lastupdated’.  When coming back to this formula and editing, you just need to change the date and time and initials part of the line.

      AddTerm(“#lastupdated”, “yyyymmdd_hh:mm_initials”) +

       
    3. The next thing to do is to set up your terms to assist to identify keywords.
      AddTerm(“#yourword”, “yourword|every_version_of_your_word|synonomyn|synonomyns|addplurals|etc”) + 

      When your term is made up of more than one word you need to have an underscore between each word so it will be one complete term.
      The hashtag is essential here and in the search criteria.

    4. Search strings.  These are the instructions to the system to search out your keywords, or terms, or combinations of words and terms.
      Look at the examples in Fig.4. below to see how these are put together.
      Every part of the string means something so don’t miss a comma or a quotation mark or your search string won’t work.
      Note:  the tilde symbol [~] placed before a word in a search string means that word is optional.
      At the end of the string, you add the score.
      If you just want to highlight the phrase and not score it, don’t add a number here, or make the number zero.
      if the score is positive it is green underlining, negative is red and 0 is blue.
      So, you can have the system also check for incorrect words or phrases in the response.  Just have the score a negative number.

      Search(“this is where you enter ~your #keywords|#AddTerm and phrases”, 100) +

  5. Hit the ‘Check’ button periodically, or after every new search string is added, to check it is working and you haven’t forgotten to add a comma, bracket or quotation mark.

  6. Hit the ‘Test’ button to test your formula on real quiz responses.  The aim here is to get the ‘Formula Pending’ percentage score as low as possible.  See Fig. 5.

  7. The ‘Download’ button will download an xls spreadsheet which provides you with actual quiz responses and compares how the human marker has decided to mark against the formula.  This is a good source to find other possible search terms that have been used by students.

  8. Save – save the formula.  Don’t exit the page without saving or you will lose the formula completely.


V-20180914