Made using Adobe Flex, the NMAT 2011 demo combined a tutorial followed by a 15-question demo test, that is supposed to be similar in look and feel to the actual NMAT test interface. However, the demo test does not implement all the features that the tutorial claims the test would have.

The demo is accompanied by a Pearson VUE disclaimer, “Please note that the functionalities displayed in this video are representative and may not include all the functionalities of the actual NMAT computer-based test.” The combined message of releasing a demo test and not implementing all the test features in it could be confusing to those unfamiliar with computers and to an extent defeats the purpose of a demo test.

To begin with, we must say that the NMAT demo test has been created very sloppily. The quality of screen shots used are poor and the resolution used is distorted against the vertical axis, making the fonts look wide and flattened. The Review Screen does not show flagged questions. The demo is also poorly coded under-the-hood and on slow connections, it loads incomplete screens.

“The demo does not load properly until you refresh it a couple of times. We tested this on three different computers at different locations and internet connections.“

What we liked

1. You can hide the clock on the top-right of the test-screen that shows the ‘Time remaining’ – this is a thoughtful feature for those type of test-takers who get freaked out by a constant timer staring at them. You can always bring back the timer by clicking on the ‘clock’ symbol again.

However, this is one of the features that is claimed in the tutorial but when one clicks on the ‘clock’ symbol in the demo test, nothing happens.

2. Review screen shows questions segregated by sections.

3. Review screen shows the number of unanswered questions in each section. This is a useful feature many have been demanding in Prometric’s CAT test interface, where the task of counting the number of attempts from the Review snapshot is left to the user. NMAT counts this for you, helping you take decisions faster.

4. Questions are segregated by sections – something that the CAT does not do and should do too.

What we didn’t like

1. No access to Review Screen from questions – you cannot jump to the review screen from any question in the test. You have to go through each question from beginning to end in order to reach the Review Screen. In that sense, the creators of NMAT have nipped some very common test-taking strategies in the bud. For example, if you are not comfortable with the section that appears first in the exam, there is no way you can jump to the one that you are strongest at without pressing the ‘Next’ button repeatedly. Or, you cannot decide the sequence of sections you want to answer the paper in without going through an inefficient routine of clicking buttons. This is a crippling feature.

Such an implementation is okay  in an adaptive test, where questions are generated and answered sequentially. But in a test with discrete questions, it is unclear why random access to all questions is not provided throughout the test.

To this extent, the NMAT test interface restricts you from having the freedom of deciding how you must take the test. We wish Pearson VUE allowed access to the Review screen from anywhere in the test.

2. The visual design of the question flagging system is poor – what ‘Marked questions’ are to the CAT, ‘Flagged questions’ are to NMAT. You can flag questions by clicking on a toggle button on the top-right of the screen. The visual cues that indicate the flagged state are too small and subtle and do not aid the eye efficiently. In the questions, a flagged state is indicated by a yellow-colored symbol, while in the Review screen the flagged state is denoted by a blue-colored symbol. Interaction Designers in good software product companies get fired for such thoughtless design. For computer-savvy users, this may not be a big deal. But for those unfamiliar with computers, such conflicting visual design might create confusion. We wish Pearson VUE’s designers worked on a more consistent iconography.

3. ‘End Review’ button ends the exam, but it does not tell you so – After you click the ‘End Review’ button on the Review Screen, the interface shows you two warnings asking you to be sure that you really want to end the review. But it does not tell you that ending the review also ends the test. Many will misinterpret ‘End Review’ as just that — ending of the review and resuming of questions — and the warnings not mentioning that it will end the test does not help. Yes, the tutorial did mention that ending review will end the test, but this is bizarre design which expects a user to rely on a manual to avoid the obvious pitfalls of a badly designed feature. By designing the feature better, the need to explain it in the tutorial wouldn’t have been there in the first place. This too could be a crippling feature to the non-savvy test-taker. We wish Pearson VUE added a more descriptive warning for the End Review button.

“If ‘Ending the Review’ is going to end the test, this should be clearly mentioned in the warnings”

4. No on-demand help – we found such a feature lacking in the CAT test interface too. For all the stress Pearson VUE and Prometric are placing on watching demo videos and practice tests, once you start taking either CAT or NMAT there is no handy help to understand the test interface features. Both tests could have had a page explaining the layout of the test interface which could be invoked anytime during the test.