Evaluation of subjective questions

This article explains the various evaluation methods by which subjective questions can be evaluated.

HackerEarth offers two response options for subjective questions to the candidates: text and video. Each option has different evaluation methods:

Evaluation methods

Subjective questions provide you with three methods of evaluation, which are as follows:

1. AI evaluation
2. Keywords evaluation
3. Manual evaluation

1. AI evaluation

AI evaluation uses Hackerearth's proprietary AI model to evaluate text answers automatically. It will compare the base answer you provided to the candidate's submission. You can use the AI evaluation method to answer essay questions or questions with long text answers. 

You also have the option to view the difference between the actual answer and the candidate's submission. For example, you are comparing the two answers after enabling View difference.

You can also provide a solution template for candidates, which can work as a starting point for them to write the answer.

Note: Turn this feature on only for text-based answers and not for numerical answers or answers where the language is not English or video answers. You can use the Keyword evaluation method to evaluate such answers.

2. Keyword evaluation

Subjective questions with text answers can also be evaluated using the keywords method.  A Keyword can be any of the following:

1. Text string: This can include spaces, special characters, and alphanumeric characters. It must have at least one non-whitespace character.
2. Number: This can include an integer or decimal.

Note: You can only use Keyword evaluation on subjective questions with text answers.

You can use keyword evaluation to evaluate questions related to data analytics (MS Excel), mathematical numerical, or fill-in-the-blank questions.

Keywords will be trimmed of leading and trailing whitespaces once an admin adds them. You can use Keywords with a maximum length of 30 characters. At least 1 Keyword is mandatory for the Keyword evaluation method. You can add more than one Keyword up to a maximum of 15 keywords. Each option mandatorily has a corresponding score. At least one keyword option score must be equal to the maximum score of the question.

For every keyword option, the subsequent condition needs to be fulfilled:

1. Organize Keyword Options in descending order based on their scores.
2. Verify whether the keyword is present in the candidate's response at least once. This verification is case-insensitive.
3. If the keyword is found, allocate the associated keyword score as the question's score.

If the keyword is not found, the steps get repeated with the next highest-scoring keyword. 

Example: Text string as a keyword

Here is an example where the added keyword is a text string. 

                                                 Example problem statement

Once the question is evaluated, and if the answer contains the exact keyword, the candidate will receive full marks assigned to that keyword.

Keyword evaluation 

Example: Number as a keyword

Here is an example where the added keyword is a number. 

If the answer contains the exact keyword, the score assigned to the keyword is allocated to the candidate’s answer.

Note: Partial matches will not be given a score.

For example, only the entire keyword will be the right answer to the given question.

The score will not be provided as the answer is 12, but the submission is 1. Similarly, with string answers, only the answer with the entire keyword will be the right submission. For example, if the expected keyword is mango, and the candidate gave the submission as man, the candidate will not receive any score.

3. Manual evaluation

Subjective questions can also be evaluated manually, which means you can give your own answer in the base answer, but it is not mandatory to write the base answer in this type of evaluation. You can manually check the candidate’s submission and use the base answer as a reference. The base answer will also be present in the candidate’s report to make the comparison easy.

Important: Subjective questions with a video response type can be evaluated manually only. You will need to review the candidate’s video submission directly.

These three evaluation methods can be used to evaluate a subjective question.