Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature][Core] Implement a generalised evaluation script #89

Closed
Udayraj123 opened this issue Oct 9, 2022 · 5 comments · Fixed by #118
Closed

[Feature][Core] Implement a generalised evaluation script #89

Udayraj123 opened this issue Oct 9, 2022 · 5 comments · Fixed by #118
Assignees
@Udayraj123 @Rohan-G
Labels
advanced enhancement New feature or request

Comments

@Udayraj123
Copy link
Owner

Udayraj123 commented Oct 9, 2022 •
edited

Is your feature request related to a problem? Please describe.
Need support for a customisable marking scheme with negative marking, section-wise marking, bonus questions, etc. Answer keys may exist for a test booklet code. We can also support custom logic via eval scripts for handling any other cases.

Describe the solution you'd like
Basic evaluation scheme can be stored in template.json file

Criteria for Answer Schema:
- It should allow simplest way to grade +1 -0 case
- Also for +4 -1 case, it should be of minimal effort
- In all other cases we want users to modify 3-4 keys

Describe alternatives you've considered
Right now we can solve this by applying Excel formulae(link to sample sheet) over the output results file. It works fine for simple cases which still requires knowledge about writing formulae in excel.

Additional context
NA
@Udayraj123
Copy link
Owner Author

Some links:

@Rohan-G
Copy link
Contributor

Rohan-G commented Oct 22, 2022 •
edited by Udayraj123

Current schema for evaluation:

In the tempalte.json, we include a Solution section, with the data for each question ([qNo] : [[Answers,mark1],[Answers,mark2],['DEFAULT',Default_marks]]) as a general set of rules with more details as given below:

Schema flexibility:

Double digit integer                                       - { [qNo] : ['4', '04'] }  
Double digit integer(X is provided in "emptyVal")          - { [qNo] : ['4', 'X4'] }  
Multiple choice                                            - { [qNo] : [ ['A', 4] ] }
Strictly Multiple correct                                  - { [qNo] : [ ['AB', 2] ] } 
Partial Marking                                            - { [qNo] : [ ['A', 4], ['D', 1] ] } 
Ambiguous correct                                          - { [qNo] : [ ['A', 4], ['B', 4] ] }
Ambiguous correct(Cumulative score 2 options)              - { [qNo] : [ ['A', 4], ['B', 4], ['AB', 8] ] }
Ambiguous correct(Cumulative score 3 options)              - { [qNo] : [ ['A', 4], ['B', 4], ['C', 2], ['AB', 8], ['BC', 6], ['AC', 6] ] }

Negative (+4 -1)                                           - { [qNo] : [ ['A', 4], ['B', -1], ['C', -1], ['D', -1] ] }
Negative (+4 -1)                                           - { [qNo] : [ ['A', 4], ['*DEFAULT*', -1] ] }
Negative (+1 -0.25)                                        - { [qNo] : [ ['A', 1], ['*DEFAULT*', -0.25] ] }
Bonus (+4 for all)                                         - { [qNo] : [ ['*DEFAULT*', 4] ] }

Note: order of concatenations is governed by "vals" array from template.json

Edit: many changes in this schema. Follow evaluation_schema.py file for latest schema

@Udayraj123
Copy link
Owner Author

@Rohan-G are you available to work on this further?

@Rohan-G
Copy link
Contributor

Rohan-G commented Dec 4, 2022

@Rohan-G are you available to work on this further?

Yep I can continue working on this now.

@Udayraj123 Udayraj123 mentioned this issue Dec 28, 2022
Closed
@Udayraj123 Udayraj123 mentioned this issue Jan 3, 2023
Merged
@Udayraj123
Copy link
Owner Author

Note: there are many changes in the above schema. I'll be populating many examples in the existing samples as a demonstration

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Udayraj123 Udayraj123

@Rohan-G Rohan-G

Labels
advanced enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Evaluation Schema + a necessary revamp Udayraj123/OMRChecker
2 participants
@Udayraj123 @Rohan-G