Standardized Online Testing in Education and Recruitment

From New Media Business Blog

Jump to: navigation, search

Contents

Standardized Online Testing in Education

Online testing refers to any test given online, accessible anywhere by anyone on any device. Standardized testing is defined as a test in which one must answer similar or the same questions as other test takers, that are then marked in a standardized manner in order to compare relative performance. These methods of testing are becoming more widely used in conjunction by exam awarding bodies. [1]

Three Levels of Online Examination

  • Proctored exams in person: This occurs in online programs where students live near the school's physical campus, the solution can be as simple as designating a testing centre and setting aside a specific time frame for when students need to attend. For programs with more remote students, sometimes arrangements can be made to have officials at another school or library act as proctors.
  • Online exams that are monitored by humans: This is a middle ground where students are still watched by a human via webcams or special software but not by someone who is actually in the same room. Since having real people involved limits capacity, users need to make an appointment even though they will not be taking up space in a physical test center.
  • Online exams that are monitored by computers: This is the newest arrangement, and the one that offers the most flexibility. Services take advantage of technology similar to what allows a mobile phone to keep track of whether its owner is looking at the screen or not. In this case, though, the software monitors what is happening on the screen, on the webcam and through the microphone to look for any behaviors that might indicate an attempt to cheat.

[2]

Example Exam Formats

  • Diagnostic test: Testing how much students already know about a given subject or topic. The results of this test can be used to schedule classes, Keeping in mind which subjects need some extra attention.
  • Placement test: Testing students to place in appropriate class level or subject matter level.
  • Progress or Achievement tests: Testing used to measure progress in a given subject. Usually best to do them frequently and consistently to get a better picture of class performance.
  • Objective test: Testing that has clear right or wrong answers. All multiple-choice tests fall into this group. The students get a set of possible answers to choose the correct answer from.
  • Subjective test: Testing in which the maker of the exam has to pass judgement on the answers of the student. Mostly this in the form of free text questions or essays.


The Evolution of Online Testing in Education

Some schools all over the world have moved their testing online as well as entire states in the US. It is predicted that by 2035 most school tests will be conducted online. [3]

In 2007, approximately 3 million students were taking an online course, by 2017 that number doubled, with 1 in 4 students enrolled in an online course in the US. [4]

A report by google suggests the market for online education will hit 1.96 billion by 2021, and a significant portion of that market consists of online examinations. [5]

Businesses Creating Testing Applications

Online Remote Proctoring

Mettl - Mercer:

General online assessment tool, has an online certification program for certifications. They have an AI powered tool to prevent cheating - Tools used to prevent cheating include facial recognition, device detection, multiple people detection and distraction level detection.[6]

  • They offer three modes of test taking:
    • Auto Proctoring: AI powered to allow large scale examination
    • Live Proctoring: Students take an exam at home and are monitored via webcam and voice. A proctor watches from a different screen and monitors multiple students at once to ensure no cheating.
    • Record and Review Proctoring: Same as above but instead of watching them live, the recorded video and audio is reviewed at a later time.


ProProfs (Online Assessment):

ProProfs Quiz Maker is a web-based quiz authoring application that you can use to create and share interactive tests. It is ideal for educational institutions and educators.You can create and conduct secure online tests and include features and types such as essay and question, short answer, matching, multiple choices, randomized questions, public and private test access, and time limits. Connects simply using an API to current LMS/CMS system. [7]

Cheating - Risks and Mitigation

Risks of Cheating

Cheating becomes easier when you are not physically being monitored, and as students get older, they are more likely to cheat. This is shown by a college survey in which 75% of students surveyed admitted to having cheated on a college exam, and 95% of these students had not been caught.[8]


Tools for Prevention

Online standardized testing is becoming more mainstream, as a result, a lot of companies have found creative ways to catch cheating. Online test applications now incorporate facial recognition and detection technology that not only verifies student identities, but can detect when test takers leave the area or turn their head for a certain amount of time, flagging potential concerns for faculty. Some examples of anti-cheating measures are:

  • Keystroke Analysis: Applications now analyse key strokes to track whether a student is answering a question, searching up a question or even trying to screenshot.
  • Webcam Biometrics: Can detect multiple suspicious activities. For example: A student attempted to cheat by writing notes on the inside of a water bottle label, the biometric system flagged the suspicious activity of him looking at his bottle too often, and alerted the examiners, which reviewed and caught the student. [9]
  • Lockdown Browser: Students use an application for online testing, these applications often have a lockdown browser feature built in. Once students have accessed an exam, they are unable to print, copy, go to another URL, or access other applications. Once an assessment is started, students are locked into it until it is submitted for grading. For example: Respondus Lockdown Browser is integrated into current college systems like Canvas and blackborn so any online test can easily be required to be done through the lockdown browser. This browser also records video and audio of the students during examination and flags suspicious activity. [10]
  • Other methods:
    • Select questions randomly from a question bank for each student in order to reduce the effect of students sharing questions and answers
    • Reduce the average time to answer each question in order to reduce the time students have to look up or ask for answers
    • Allow only one attempt to take test
    • Require completion of test once started
    • Present one question at a time
    • Randomize questions and answer choices for each student
    • Use a lockdown browser
    • In objective test use more multiple choice understanding questions and fewer remember type questions
    • Proctor exams with camera
    • Proctor exams with human proctor
    • Develop new questions each semester
    • Use essay questions

[11]

Artificial Intelligence

Artificial Intelligence (AI) enables machines to learn from experience, adjust to the environment, and thereby perform human-like activities. Using deep learning algorithms and neural networks, machines can be trained to accomplish specific tasks by processing large amounts of data and recognizing patterns in it.

The availability of proctors for live proctoring has become a limitation in testing, not to forget the additional costs involved with it. However, with AI coming into the picture, universities and other educational institutions are increasingly embracing auto proctored tests since they are the cheapest solution while maintaining integrity at the same time. [12]

AI is equipped with advanced facial recognition features. To ensure that students take no unfair means to ace an exam, universities use automated proctoring units that rely on face recognition. The accuracy and precision of detection are directly proportional to the volume of images being fed to the system. The cost is reduced by one third when an auto invigilator is used instead of a live proctor. [13]

Paper vs. Online Testing

Advantages

  • Security and Confidentiality
    Prepared exams need to be securely kept. Any leakage will definitely compromise the standard of the exam.. In an online exam system, the questions are kept in a secure database, access to the database is only possible with an authorized personnel. Questions can easily be randomized so that no participants see the same questions in the same order. Questions can easily be mixed as each new question is added to the system’s database. The questions can then be randomly drawn from the database. This is why most examination bodies have gradually adopted the online platform. [check citation]
  • Accessibility and Flexibility
    Exams can be conducted anywhere. All a student needs is a personal computer with internet connection. This also means that thousands of students can take the same exams over a wide spread of locations. Examiners also benefit from this. Examiners do not have to bother with the laborious task of marking exams as this is well taken care of by the system, depending on the test type. The system actually marks each exam and presents the result to the student at the end of the exam. Examiners are also afforded the opportunity to create exam online through an online examination system that can present examinations in multiple languages. Multiple exams on multiple subjects for multiple courses can also be set. Exams can also be configured for 24/7 availability. This allows candidates to take exams at their own convenience. [check citation]
  • Cost Savings
    There are significant cost reductions when compared with running face-to-face exams. In-person exams can be very expensive as there are substantial fees associated with factors such as hiring a test hall, supplying or hiring invigilators, securely couriering exam papers and answer sheets, printing exam papers, scanning and correcting solutions, securely storing and disposing of exam papers and scripts, staff overheads to manage the process and logistics, and dealing with re-sits.
  • Scaleability
    Unlike exam halls which are limited by the number of available seats and test administrators, online exams can be done remotely and using AI. [check citation]


Disadvantages

An Example: CPA Common Final Exam 2019[14]

  • What happened?
      Students were running into connectivity issues and exams were not loading, exam times were delayed by up to 5 hours and those who selected to continue the exam, did so without access to the Handbook. Some students experienced anxiety and mental health episodes
  • What caused the technical issue?
      Technology problems appeared to be centred around internet access and the failure of Surpass (SecureClient) software to connect. According to CPAWSB, “Surpass (SecureClient) is an application that locks down your computers so that no other programs or files can be accessed during an examination.” The software was first introduced in September 2018 for CPA preparatory courses. Ultimately, those students plagued with technical issues were told to write the exam outside the secure software and without the reference material (handbook). There is also no current news from the organization on whether this years exam will count or not. [15]


The Impact of Online Testing on Test Scores

A study compared test scores between paper exams and online exams, the results showed a clear disadvantage for online test takers. A 0.10 standard deviations in math and 0.25 standard deviations in English Language Arts existed between the 2 modes of tests. That amounts to up to 5.4 months of learning in math and 11 months of learning in ELA in a single year. [16]

  • Could this be a case of historically low performing districts using online tools?
      No, districts that switched to online testing had the higher prior achievement. Back when everyone was taking tests on paper, these districts scored consistently higher year after year. Then once those districts switched to online testing, their achievement fell. [17]
  • Theories about the difference?
      Students are more use to paper tests where they can flip back and forth on pages instead of manipulating a scroll wheel. The online exam also only showed 1 question at a time. Another possibility is that students were just not used to the tests online. Some evidence supports this claim, the 2nd year of this study showed a narrowing of the gap between online and paper scores, a still not insignificant margin still existed. Lack of fluency with a computer could also affect the scores.[18]


Cost

Both companies we contacted have a similar price structure. They charge a tiered monthly subscription, a small plan, up to 50 students, is $149 USD. The medium plan is for $279 USD and covers up to 150 students. The large plan is for $499 USD, and covers up to 500 students. The enterprise plan is priced on a case by case basis. The pricing also varies based on which proctoring type is chosen. In general, the record proctoring is the most economical, because the proctoring is done by the customer. The live proctoring by the providing company's trained staff is the most expensive, and AI proctoring is priced just below that. [19] [20]

Standardized Online Testing in Recruitment

The Importance of Online Testing for Businesses

It is important for companies to make alterations to their recruitment techniques as times change. It is almost impossible to hire candidates based solely on resumes and online applications. These methods do not allow employers to assess both the candidates hard and soft skills and make it difficult to determine if the candidate is a cultural fit.

Further, face-to-face interviews with every candidate are difficult to conduct; as time and budget constraints make it nearly impossible to interview hundreds of candidates. Web-based employment application software eliminates the need for paper-based job applications without requiring an installation of specific software. [21]

The Purpose of Online Recruitment Tools

Pre-employment testing is an efficient way of gaining insights into the candidates qualifications. They allow for a standardized way to assess a candidate's qualifications; providing relevant information regarding the candidate's skillset. Because the rise of resume spammers has lead to, on average, 250 resumes submitted per every job opening, pre-employment tests allow for filtering large applicant pools. In a situation with hundreds and thousands of applications, pre-employment tests add value to the hiring process by making the process efficient and less time consuming. [22]

What Standardized Tests Measure

The standard categories pre-employment tests cover are aptitude, personality, and skills tests.

Aptitude test

Aptitude tests measure critical thinking, problem solving, and the ability to learn, digest and apply new information. According to a study conducted by LinkedIn, two important qualities hiring managers see are problem solving skills and the ability to learn new concepts. Because these qualities are difficult during interviews, aptitude tests allow for these skills to be tested during pre-employment testing. [23]

Personality test

Personality tests assist with determining whether a candidate will be comfortable in their new position. These tests include behavioural-based questions which will allow for hiring managers to assess whether the candidate's behaviours will fit well with the position and company. The tests measure the “Big Five” personality traits: agreeableness, conscientiousness, extraversion, openness (to experience), and stress tolerance. [24]

Skills test

Skills test can measure either broad competences such as verbal, math, and communication; or specific ones like typing and computer skills. Candidates have usually acquired these skills throughout their education or career and are based on previous experience. To maximize pre-employment testing, companies tend to make use of a combination of the different tests. This allows the employer to assess the candidate in more than one category; providing reliable data and assisting with the hiring decision by providing objective data.[25]

Future of Pre-employment Testing

Because of the increase in applications applying online, the use of pre-employment testing has increased over the years. Recruiters and hiring managers rely on these sources to decrease the workload related to finding the perfect fit for their position. According to a survey conducted by the American Management Association (AMA), the use of pre-employment testing has been growing steadily in the past fifteen years. The AMA's data revealed that:

  • 70% of employers did some sort of job skill testing
  • 46% of employers use personality and/or psychological tests on applicants or current employees
  • 41% of employers test applicants for basic literacy and math skills [26]

Testing Tools - Examples

Hundred5

  • Hundred5 is a skill-based hiring tool allows companies to make candidate selection easier and faster. Instead of analyzing every incoming resumes, hiring managers create a customized assessment that aligns with the position and company culture using Hundred5. How the system works is the candidate take a quick 10-minute skills challenge and once completed, the system identifies top performers and gives the hiring managers the list. The system comes with a growing pool of templates for companies to follow created by industry experts, allowing the system to be used across different industries. In terms of costs, Hundred5 charges $290 for one role per month. For a medium-large sized company, this investment will save time required for paper-based testing, as well as money in terms of hours spent by hiring managers and recruiters finding the right candidate for multiple positions. [27]

Codility

  • Although pre-employment tests allow managers to gain insights on candidates soft skills, it is also important to have reliable data regarding candidates technical skills. Codility generates reports about each candidate’s coding abilities, allowing for only qualified candidates to make it to the interview round. Hiring managers are able to create their own customized tests and questions. [28]

HackerRank

  • HackerRank is the leading end-to-end technical recruiting platform for hiring developers[29]. HackerRank allows for hiring managers and recruiters to objectively assess developer skills. The system has helped over 1500 companies around the world by inviting candidates to take tests for specific vacant positions. Further, candidates are able to take the tests in 35 programming languages, allowing companies to take advantage of talent available outside the country. In terms of costs, there are three plans employers can choose from; depending on the size of the company. Startups can choose between $249/month paid annually or $299/month paid monthly. The team account gives access to 7 users and engages up to 100 candidates a month for a price of $2,996 for four months or $7,188 for one year. For large sized companies, employers can choose the Enterprise account which offers customized pricing. [30]
  • Products 1: HackerRank CodeScreen
    • Codescreen enables hiring managers to assess candidates strengths and weaknesses and evaluate candidates’ coding skills. Hiring managers create a role specific technical assessment test and invite candidates to take the test. Once the test is completed, HackerRank automatically reviews and analyzes the scores and ranks the candidates and identifies the top performers.[31]

Web-based Applications

Advantages

Web-based software applications store data in one place that is accessible from a central computer. Employers receive a list of the best applications without having to manually look through paper based applications. Further, templates are also available so that businesses that do not have the manpower or time to create personalized test have a template to work with. [32]

Web-based employment application software makes it easy for potential employees to apply for a job from across the globe at any time as long as they have access to a computer and the Internet. [33]

Web-based employment application software offers savings to both businesses and job seekers. Businesses save on the cost of paper applications while potential employees do not need to pay to print out resumes or mail in the application. Although businesses may have to pay for the application system, for most it is still worthwhile. Businesses that go with recruiters instead could lose up to 20 percent of first year salary for the service. [34]

Disadvantages

Web-based employment application software presents some disadvantages. There is no way to ensure a candidate is completing the online application or assessments without the help of other resources or other individuals. Also, if a program searches for keywords to weed out applicants, highly qualified applicants can go undetected. Another issue is that websites could experience problems, preventing applicants from entering information. If a website malfunction occurs, information could be lost which could present privacy concerns.[35]

Case Example: Amazon Scraps Secret AI Recruiting Tool that Showed Bias Against Women

Since 2014, Amazon was in the process of building a system to review job applicants’ with the goal to identify top candidates. The company’s potential hiring tool used artificial intelligence to rate candidates out of five stars and identify the most qualified to least depending on their resumes. [36]

By 2015, a problem was detected that the software was not rating candidates in a gender-neutral manner. The system was trained to rate candidates by observing patterns in resumes submitted to the company over a 10-year period. Most of the resumes analyzed came from men, a reflection of male dominance across the tech industry (as noted by the figure on right).[37]

For example, applicants that used the word “women” on the resumes were penalized and not selected for the job. Another cause was masculine language, through which men were more likely to use words like "executed" and "captured" which resulted in further bias in the system. Aside from the gender neutrality issues, the software also recommended candidates that were unqualified for roles.[38]

Amazon first made changes to the system, but eventually terminated the system because it was difficult to ensure the changes made were executed by the system due to other subliminal biases. The company's experiment speaks to a greater struggle with the limitations of machine learning, although other companies are continuing to attempt to harness AI for recruitment. An opinion piece delved further into the idea that data cannot be inherently biased, and the motivations, objectives and resources of the developers of the software must be analyzed in order to gain an understanding of what went so wrong with the Amazon experiment. [39]

Authors

Muddassir Mohammed Arshveer Rai Robin Sandhu
Beedie School of Business
Simon Fraser University
Burnaby, BC, Canada
Beedie School of Business
Simon Fraser University
Burnaby, BC, Canada
Beedie School of Business
Simon Fraser University
Burnaby, BC, Canada

Sources

  1. https://www.edglossary.org/standardized-test/
  2. https://www.usnews.com/education/online-learning-lessons/articles/2016-05-20/3-ways-online-students-might-take-exams
  3. https://www.edtechstrategies.com/research-and-writing/usk-8-testing/
  4. https://www.onlineexambuilder.com/knowledge-center/exam-knowledge-center/types-of-exams-and-tests/item10246
  5. https://assets.kpmg/content/dam/kpmg/in/pdf/2017/05/Online-Education-in-India-2021.pdf
  6. https://mettl.com/en/
  7. https://www.proprofs.com/
  8. https://www.insidehighered.com/digital-learning/views/2017/09/20/creative-ways-students-try-cheat-online-exams
  9. https://www.insidehighered.com/digital-learning/article/2017/05/10/online-exam-proctoring-catches-cheaters-raises-concerns
  10. http://www.respondus.com/products/10reasons-LDB.pdf
  11. https://pdfs.semanticscholar.org/dbe0/d138a192b5134bcb87b858fd90df586085b8.pdf
  12. http://faculty.nps.edu/ncrowe/dlcheat.htm
  13. https://blog.mettl.com/assessment-technology/how-ai-is-helping-universities-scale-online-examination
  14. http://www.canadian-accountant.com/content/business/accounting-students-furious-over-cpa-cfe-tech-failures
  15. https://business.financialpost.com/news/fp-street/fyre-festival-for-accountants-cpa-canada-takes-heat-after-national-exams-plagued-by-glitches
  16. https://www.future-ed.org/work/paper-vs-online-testing-whats-the-impact-on-test-scores/
  17. https://www.future-ed.org/work/paper-vs-online-testing-whats-the-impact-on-test-scores/
  18. https://www.future-ed.org/work/paper-vs-online-testing-whats-the-impact-on-test-scores/
  19. https://mettl.com/en/?utm_medium=ppc&utm_source=adwords&utm_campaign=2075427832&ads_adid=74634873337&ads_targetid=kwd-372040230718&ads_network=g&ads_creative=372669527901&ads_adposition=1t1&ads_kw_term=mettl&gclid=CjwKCAiA5o3vBRBUEiwA9PVzajKvLFq-RCtfVyKoZnoFOeSAIBQBVUeE8hsaCQI6pUvIIdQJKgM_5BoCPPUQAvD_BwE&
  20. https://www.proctoru.com/
  21. https://hundred5.com/blog/the-ultimate-list-of-pre-employment-testing-tools-vol-1
  22. https://www.criteriacorp.com/resources/definitive_guide_what_are_preemployment_tests.php
  23. https://www.criteriacorp.com/resources/definitive_guide_what_are_preemployment_tests.php
  24. https://www.criteriacorp.com/resources/definitive_guide_what_are_preemployment_tests.php
  25. https://www.criteriacorp.com/resources/definitive_guide_what_are_preemployment_tests.php
  26. https://www.criteriacorp.com/resources/definitive_guide_what_are_preemployment_tests.php
  27. https://hundred5.com/
  28. https://www.codility.com/
  29. https://www.hackerrank.com/?utm_expid=.2u09ecQTSny1HV02SEVoCg.0&utm_referrer=
  30. https://www.hackerrank.com/?utm_expid=.2u09ecQTSny1HV02SEVoCg.0&utm_referrer=
  31. https://www.hackerrank.com/?utm_expid=.2u09ecQTSny1HV02SEVoCg.0&utm_referrer=
  32. https://smallbusiness.chron.com/pros-cons-using-webbased-employment-application-software-45611.html
  33. https://smallbusiness.chron.com/pros-cons-using-webbased-employment-application-software-45611.html
  34. https://smallbusiness.chron.com/pros-cons-using-webbased-employment-application-software-45611.html
  35. https://smallbusiness.chron.com/pros-cons-using-webbased-employment-application-software-45611.html
  36. https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G
  37. https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G
  38. https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G
  39. https://becominghuman.ai/amazons-sexist-ai-recruiting-tool-how-did-it-go-so-wrong-e3d14816d98e
Personal tools