The Story behind OMRChecker

Hello readers! This is the second post of a retrospective series 'Open Source Experience with OMRChecker'. Previous post: [The Exposure]

I want to reflect upon how beautiful, exciting and rewarding this journey of working on OMRChecker has been. In over 6 years, it has gone from being just a simple script to seamlessly scanning OMRs at large scale and much more.

I wish to cover this story in a timeline. Starting with the chapter of how the idea was born -

July 2016

For the first time in my own city I conducted Technothlon, an international examination organised by the students of IIT Guwahati.

While roaming around in the exam halls, I had observed students filling bubbles in different ways. It made me wonder how a machine must read such OMR sheets accurately. 

While the students were solving the mind-boggling questions in Technothlon, my mind was boggled by something else - "How exactly are OMR sheets evaluated by a machine?"

I then started actively observing the process in the organisation. Once we returned to the IITG Campus, our team organised an OMR Collection Drive where each student organiser brought their OMR sheets/couriered them to the Technothlon heads. 

In the first week of August - began the Scanning Phase. We had just a single office scanner, each and every OMR sheet was scanned in small batches. Here is a glimpse of how the scanning used to take place -

This was the most time consuming part of results generation process. We could allot only 4 team members at a time to scan in day and night shifts. "The scanning must not stop" was our motto that time. We had no way to get the whole team to work in parallel. I wondered why we couldn't get better scanners and realized the reason was very high cost for large scale scanners. We didn't accommodate that much of an investment.

A thought shaped up in my mind: "we need to eliminate the need for a scanner!"

But in the Results Phase next week - I realized the scanner problem wasn't the biggest pain. The paid software we used could not even handle slight shifts in scanned images, and we had to manually fix the template layout for every new batch of scans. It was one of the most frustrating software I've ever used.

Furthermore, majority of the OMRs couriered by the City reps were xeroxed, thin paper copies. They had also suffered bending due to the monsoon moisture.

Our team had to manually check for errors and verify all such OMRs carefully, ALL of the OMRs that we collected. Adding hours and hours to our work.

"This is slowing us down from declaring results, a big time! I want to fix this."

The problem scope now had widened - we felt the need for a fast, very robust, yet flexible OMR checking software to support a variety of conditions of OMR sheets.

My interest in the whole process kept me working dedicatedly as a Technothlon Team member. I became one of the six regional heads and the online head next year!

Mar 2017

Technothlon '17 Head

As a Technothlon head, I felt the responsibility to dedicate myself to solve this very interesting problem. Here's a snippet from an interview article from Mar 2017 -

In the meantime, being a curious learner, I had started learning OpenCV to make a color detection module for Rubik's cube solver robot as part of IITG Robotics Club. 

May 2017

Even before summer vacations started, I was sitting in the computer lab, trying to solve the problem that had been hanging in my head for so long. First step was to check if I was reinventing the wheel - I did not find any reliable open source/free software for checking OMRs.

So I decided to write one myself.

OMRChecker started as a basic python script that does the job. I had very few images to test with back then.

June 2017

Gaining authority as the online head enabled me to improve the OMR process from the roots.  For Technothlon 2017, I designed a new OMR template for Technothlon with a new marker.

This was done by keeping in mind that I could test it using my mobile phone from any orientation.

I wasn't sure about its effectiveness as I didn't have any previous reference images, yet I took a leap based on a strong intuition.

August 2017

At this point, OMRChecker script had most of its tuning hardcoded. I had set it according to the 10-15 different bad quality images that I had manually printed and filled for testing.

It was time. Time for the results phase again! I was excited, thrilled and nervous at the same time for the first large scale use in Technothlon!

But I was soon hit with a reality shock, real-life images were really worse than the samples. The script was not very optimized as well. We had to re-tune and re-run OMRChecker multiple times, each run taking about 4 hours.

But I was stubborn to get it working, finally after a 12 hour stretch I got the best tuning possible working for OMRChecker. 

We had to delay the results by one day.

We had also used the paid software alongside and as it turns out, it performed poorly on the same images. Perhaps, OMRChecker wasn't bad after all. The manual work had also slightly reduced this time.

I did not deliver on the promise I made during my Technothlon interview. Yet my mentors, my co-heads were actually glad, because they saw our own creation take shape and show potential. They motivated me to keep working on it.

I had a very vivid idea on how OMRChecker can do better. I collected about 30+ reference images and formulated next problem statement to solve.  

Just after the results, I was still thinking and trying new ideas, testing on those reference images, improving the script bit by bit. This is where OMRChecker became close to my heart, I couldn't stop myself from making it better.

Mar-May 2018

Half a year goes by, The script had taken shape into a project.

The OMRChecker github repository was created in March 2018. Adhering to values of Technothlon, a non-profit organisation, I decided to keep the code repository public.

There were no expectations about whether someone would find it interesting to contribute, or benefit from OMRChecker then, but I released it with hope that someone just might.

I was fixing issues with the new reference images, one by one. Removing many hardcoded configurations and making OMRChecker more robust.

July - Aug 2018

This time OMRChecker was ready, we replaced the existing paid software completely. We chose to use only scanned images as mobile image accuracy was still not that great. The scanning went on for days again.

We were drained of sleep, all of us heads would sleep in the Techniche Room itself after the continuous shifts of OMR scanning:

Each run of OMRChecker would still take hours. But this time the script showed really good results and we needed very few fine tunings.

It was still a really painful, but a very successful process - We were prepared with our results ONE DAY before the official date! 

We got extra time to verify everything and declare results with the best confidence. (Further we moved to mobile-based scanning, making the whole process seamless! But that story is best saved for the next post.)

In the following year, for Technothlon '19, I would be a mentor for the new heads, while working on my first job. But still I couldn't keep myself away from doing things for Technothlon - the closest thing to my heart in college.

The Journey Never Ends

The journey of OMRChecker is unforgettable, so eventful! Working on challenges like these is what I desired the most. I would forget eating and sleeping as it was more enjoyable to solve one more OMR sample!

I kept solving many more challenges, wrote a draft research paper as part of my Bachelor Thesis on OMRChecker, even making an Android App to scan with a mini drafter stand! OMRChecker also improved search traffic organically, received global exposure, participated in hacktoberfest, now reaching out to more people. I also realized that grading exams isn't the only scope of OMRs. The journey never ends...

Upcoming

In the next post I wish to cover happenings since Sep 2018. Especially how beautifully the open source community helped OMRChecker grow. I intend to cover more on the technical side.

I also wish to cover how we achieved converting the stressful 72 hour scanning in day and nights shifts into an enjoyable process of just 8 hours scanning with the whole team!

Until next time, cheers!