Testing scenarios for Gru

  1. User takes test for first time
  • If client gives wrong token, display error.
  • If token is correct, display demo qns first and then display test questions.
    Things to check for
    • Score and time are reset properly when test is started after demo.
    • All interactions are logged.
  1. Client crashes in the middle of demo and restarts.

  2. Client crashes at end of demo and restarts.

  3. Client crashes in the middle of test and restarts.

Server crashes.

@ashwin95r lets make this an exhaustive list by thinking out all scenarios. Every bug should go into one of the scenarios.

When the client crashes a few times (say 3) and restarts, it gets the error

Could not get question.Got err: rpc error: code = 2 desc = Didn't find qn with label: test, for candidate: abcd1234

on attempting some questions after the restart

So, I played with it today, and it does feel pretty solid. Good job!

Small language changes, I just made them directly to the client. Here’s the rest of the feedback:

  • The duration of the test is 1h0m0s: Show the time in minutes always.
  • For a multiple choice question, we should allow partial score if they choose some options correctly. I think we shouldn’t do negative scoring there.
  • On the right, show the right answer, wrong answer, and “Skip question => 0.0”.
  • If the server dies when the client is working, a weird log shows up on the screen. That shouldn’t happen. In fact, the client should continue to retry the connection.
  • The same problem above can happen if the connection is broken for some time. We should show that connection is broken, but the client shouldn’t crash. It currently crashes.
  • On a server restart, the client gets shown the same questions. We should be reading a log file at the server side to avoid asking the same questions.
  • And/Or invalidate the token, because log shows that the token has already taken test and if it’s over the time.

Other feedback, which might be more of a problem with Terminator:

  • On a resize, the screen just gets lost. Maybe, it’s due to my window system.
2 Likes

Maybe for multiple choice questions we don’t give negative score and give positive score only when all options are correct.

Or if we allow partial scoring we would have to do negative scoring too to disincentivize guessing.

These two should already be happening. There was a bug which caused questions to be repeated but fix for it was merged yesterday.

Good point! We could give a partial score with negative scoring. So, if they choose one right and one wrong answer, the final result would be zero. But, if they choose one right answer only, then they’d get say half of the positive score (if 2 are correct).

Nope. A server restart caused the same question to be shown again. Including the same demo. It didn’t realize that I’d already done the demo and completed those questions.

Regarding the key mappings, I was thinking if we could have 1, 2, 3, 4 instead of a, b, c, d. This would make the selection easier and add a natural order to the way keys are arranged in the keyboard and the way they appear on the screen. Also, we can have esc to cancel any selection that has been made instead of pressing any key. How does this sound?

It would make easier design wise if we just used Esc to cancel a selection. What do you think about it @mrjn?

Sounds good regarding ESC. But, then other keys shouldn’t do anything really. And on selection screen, you should show that Press ESC to go back.

I’m not sure using numbers is such a good idea. abcd is pretty standard for MCQs, numbers not so much.

Yeah, so for multiple choice questions our scoring can represent score for a correct answer and wrong answer. And maybe we can have the same score for that. So then your final score could be
c * sc - w * sw
where c is the number of correct answers
w is the number of wrong answers
sc is the score for a single correct answer
sw, score for a single wrong answer

Note that sw can’t be lower than sc, otherwise, choosing all 4 questions would yield a positive score. In this case, I think sw >= sc, and we should enforce that when doing the precondition checking for the quiz.

1 Like

For reading updates to candidate file without restarting the server, how does this https://github.com/hpcloud/tail look?

We can read the candidate file and whenever some new candidate is added to it we can create an instance of Candidate and add it to our map.

I think we’ll be doing github pushes for the config file. So, we can have a cron which does git pull, and just have our server reload the file every X mins. Reloading the map from scratch every time, under a mutex lock. That should be sufficient. We don’t need to do anything more complicated than that.

2 Likes

We could also look into webhooks to do this

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.