Screening Quiz commandline tool and server written in Go

This topic is to discuss the features for the Go quiz tool that we are developing.

Please add your ideas here.

Okay, so we had a long chat about this already. Maybe someone can summarize it here.

Meanwhile, here’s the name I’m thinking of: Gru from Despicable Me

Why Gru? I think instead of calling ourselves staff, we should call ourselves minions. They’re fun, excited, positive. They make mistakes, but they resolve them within themselves. They’re great team players, who work without ego. They’re respectful but also willing to make fun of each other. I think minions represent our team culture pretty accurately :smiley:.

Gru is the one who hires minions. And hence, our super awesome quiz is the evil but kind hearted Gru to do the hard job of finding the right minion for the team.

5 Likes

Okay, think I’ll summarize what we discussed:


Client & Server

  • The format of the test would be like GMAT, where we show one question at one time.
  • The candidate won’t be allowed to go back to an earlier question. Either they answer or they skip.
  • Any answer entered by the candidate would be sent to server. Client wouldn’t store which answer is correct.
  • Depending upon the answer, the server would either give next question, or allow a re-try with remaining options.
  • The client would show the live score achieved so far, and the remaining time.
  • The client would also run a goroutine to ping the server every 5 seconds, including some status information like which question the candidate is currently viewing, so we can figure out how long did the candidate spend on any given question.
  • The server should tell the client if only one answer is correct, or multiple answers can be correct. The client interface should clearly mention that.

Questions

  • Each question would have a unique string id.
  • Each answer would have a unique string id, where the answer id might be prefixed with the question id.
  • The idea behind these ids is that when looking at an id, the test setter should be able to immediately tell which question and answer it is.

Marking

  • Each question would have it’s own score (say, S).
  • Skipping would be equal to zero marks.
  • Answering wrong could have negative marking (-0.5S). Each question may or may not have a negative marking.
  • In a single choice question, a retry may be provided to jump back from the wrong answer. Each question may or may not provide such an option.
  • In a case where jump back is provided:
    • correct answer on second try would bring it up to zero (from negative -0.5S).
    • another wrong answer from remaining 3 options would have even more negative impact (total: -1.2S)
  • I think no live feedback should be provided if multiple answers are correct. I can’t think of a good way to provide live feedback without making it easier to guess the answer.

Security

  • The client would run a hash and send it to server so it can ensure that the client hasn’t been modified. Have to figure out the best way to identify authenticity of client.
  • All communication will happen over TLS over TCP.
  • Using protocol buffers for communication would also help us ensure that the latest version of the client is being run.
  • A token would need to be set in all the communication so we can narrow it down to the right candidate.
  • The first communication with the server would also provide a unique session id which needs to be kept in memory of the client and added to each request to the server.
  • The session id would ensure that the candidate doesn’t run two processes of the client simultaneously.
  • Once the test is started, the token will only remain valid for just over the test duration, which is right now an hour.
  • This would be true, despite any client crashes or intermittent network failures.
2 Likes

Here’s a repo I created for the project:

We can change the name if someone has a better idea, but I like this so far :slight_smile:.

1 Like

Thinking about the syntax of the question document. How about something inspired by apiblueprint.org

# Question (qid, complexity, positive score, negative score)
Yada Yada Yada
# End Question

; This is a comment.
; All comment lines start with semicolon.
; Indentation is not mandatory in this system.

+ Answer (aid0, wrong)
Something
; Notice that each answer would end with a blank newline

; W stands for the wrong answer
+ Answer (aid1, wrong)
Something else

; This is the right answer
+ Answer (aid2, right)
The right answer

+ Answer (aid3, wrong)
somewhat different


; Let's add another question here.

# Question (qid2, easy, 3, -1.5)
...
+ Answer
...

; Now iterate all the questions which have been removed from the file.
; So it makes it easy to do collision detection.
# Deprecated (qid0)
# Deprecated (qid1)
# Deprecated (qid2)

This is easy to implement with a scanner, and is simple enough for a human to read and write.

1 Like

Sounds good. My only concern was the manual generation of UUIDs. We could do that programmatically too. Have some placeholder for the IDs initially, and replace them with a UUID when scanning the file.

Note that these are not UUIDs. In fact, they shouldn’t be.

That’s the underlying idea. And the parser in the server should log FATAL if it finds non-unique ids. That would be very easy to check.

Ok, I was thinking if we could automate the generation of these id’s too just so that they are auto incrementing.

We could automate it. But, then would this hold true, @pawan?

The idea behind these ids is that when looking at an id, the test setter should be able to immediately tell which question and answer it is.

Not if it generates a new id on every run. Though, say if we were to assign dummy placeholders for new questions which were added.

Q for question
A1,A2,A3 … for Answers

If our scanner finds these it replaces them with new id’s and updates the file. Now when we encounter this question again, we would see it has id’s which are different from placeholders, so we would just use that.

Hmm… I think the idea is very clear to me, but I doubt it’s coming across correctly. So, let’s take this situation, where either a numeric id is automatically assigned to questions, or we provide a list of unused ids, which then get assigned to the question. Both are similar because the id itself holds no relevance to the question.

Question: Id

Linked List: 0x0100, Answers: 0x0101, 0x0102, 0x0103, 0x0104
Distributed Systems: 0x0200. Answers: 0x0201, 0x0202, 0x0203, 0x0204.
Time complexity: 0x0300. Answers: .. and so on ..
Synchronization: 0x0400

Over time, we remove Synchronization question and add a new one. In both the systems we’ll have to consult with a database, and ensure that the new id generated or assigned is still unique. Otherwise, our previous records in the data would be pointing to the wrong question.

Linked List: 0x0100
Distributed Systems: 0x0200
Time complexity: 0x0300
Data Structure: 0x0500. Answers: 0x0501, 0x0502, 0x0503, 0x0504

Now, an engineer now is looking at the logs of the events. And sees, that the candidate spent a lot of time on 0x0300. What does it tell you about the question? Absolutely nothing. It’s a unique id, which was randomly assigned to the question.

Sure, you argue that the engineer can then look at the current version of the questions list, and figure out what 0x0300 corresponds to. Sure, they could do that. But, that’s a level of indirection, that has to be resolved manually for every question that the engineer’s memory doesn’t remember.

Then you argue that this doesn’t have to be done manually. You could write a tool to tackle this indirection. A tool which can look at the unique id and show you the relevant question. It should be a simple tool, just parsing a file. You could use grep, hey! But, is it?

What if you see 0x0400 id, which is no longer present in the questions file? Now you have no idea what question is that, and the tool would have to dig into the file history to tell you what it is. So, the tool isn’t so simple anymore. It would be at the very least a complicated bash script, or even a hand written program.

On the other hand, say we manually assign these ids, like so:

Linked List: linked-list. Answers: linked-list-head, linked-list-tail, linked-list-middle, linked-list-root
Time complexity: time. Answers: time-nsq, time-nlogn, time-logn, time-n
Distributed Systems: dist. Answers: .. and so on ..
Synchronization: sync
Data Structure: struct

The same engineer can be looking at the logs of the database, and with the unique ids being related to both the question and the answer, they can immediately relate that it means.

When Synchronization question is removed, and the engineer happens to look at an old log, they would still relate that the sync question is no longer there. You save a level of indirection, for every log that they have to read. These IDs would in fact become the lingua-franca to talk about the questions.


If this seems unique and unnecessary, think about why Slack doesn’t just assign you an incrementally increasing or randomly chosen UUID? Why does it allow you to set a username, which you then choose yourself? Because manually chosen uids tend to be relatable. By looking at these uids one can recognize @koppula from U13NA5ALX, which if you didn’t recognize is the UID assigned to Prashanth. Would you actually remember the latter id? Most likely not.

And that’s the reason why I think we should have relatable manually written unique ids relevant to the questions and their answers.

As Rob Pike says, “Simplicity is Complicated.” :slight_smile:

2 Likes

Okay, I get the reasoning behind the manual UIDS now.

1 Like

I had come up with this format :

1 test:
  2 - qid: A
  3   question: Which of the following is true about HTTP?
  4   correct: [A1]
  5   opt:
  6   - uid: A1
  7     str: O(N * log(NK)), O(N^3)
  8   - uid: B1
  9     str: O(K * N^2 * log(N)), O(K * N^2)
 10   score: 5
 11   tag: Easy
 12 
 13 - qid: B
 14   question: |
 15    Which of
 16    the following is true
 17    about "HTTP"?
 18   correct: [A2, B2]
 19   opt:
 20   - uid: A2
 21     str: O(N * log("N*K")), O(N^3)
 22   - uid: B2
 23     str: O(K * N^2 * log(N)), O(K * N^2)
 24   score: 5
 25   tag: Easy

It is easy to read as there is indendation, the keys make it easy to identify each line ( eg question, qid options).

We could remove the correct field and make it part of the options if required. Also, we could add a comment field that could describe these questions.

@staff What do you think about this format?

1 Like

This is good too. The problem I have seen with indentations (languages like Python) is that spaces != tab. Someone’s editor will use spaces; someone else’s will use tabs. And it then starts to look messy.

The great thing about gofmt is that it can fix all that for you. But, we’d then have to write a tool to do that. Better is to avoid indentation altogether.

I think I’ve also worked with a few YAML formatted docs, and my experience hasn’t been great, honestly.

YAML doesn’t allow tabs, so that would be reported by the compiler. And, there is going to be only 2 levels of indentation in our case. So, I don’t think it will cause a lot of problems like the files with a lot of them.

1 Like

Regarding scoring: typically adaptive tests don’t show the score during the test. They reveal the score only at the end. I think the reason for this may be to have the test taker focus on answering the question on hand and not worrying about why they answered the last question incorrectly. This may be worth considering. What do you think @staff?

Can you also include the deprecated part?

Maybe in the tags, where we say easy, medium, hard, we could say deprecated. And that’d be it. But, then our file will just keep on increasing. So, better maybe to only have one line per deprecated question?

OTOH, we can just move all the deprecated questions to the bottom of the file. That should serve the same purpose. Plus that way, we don’t loose the questions.

Yeah, maybe the deprecated tag is sufficient.

1 Like

My experience with YAML has also not been very pleasant. Also Ashwin, you mentioned that YAML to Go struct conversion was not straight forward. Maybe the format Manish mentioned would work better?

Yeah, it took a bit of learning to understand go and yaml interaction. But I’m comfortable now. That shouldn’t be a problem for the user though. The question setter wouldn’t have to worry about it. So, the best way to decide would be which format is easier for the user, readability and maintenbility of the questions file. What do you guys think about the ease of use of the two formats @staff ? I feel we should use a standard format and reuse the libraries than spending time on writing our parser if there is no major issue that concerns us

1 Like

I think the YAML structure that you proposed looks good as well. If you take one real life example and fully flesh it out and show it, that’d be a good way to judge. Containing text, code and the 4 options. You can probably find a question online. Don’t use any from our existing library.

1 Like