Computer Laboratory

Malcolm Scott

Maemo 5 Word Completion Experiment Report

Note: the canonical location of this document is here. There is also a PDF version.

[screenshot of Maemo 5 text entry]

This experiment attempts to investigate the utility of Nokia's predictive text entry system in their Maemo 5 phone operating system, by measuring the time taken to enter English text with and without this feature enabled.

Background

Maemo 5

Maemo is a touchscreen-oriented operating system for smartphones; version 5 is the latest and has a significantly different UI to previous versions. It is designed to work without a physical keypad, so it includes an on-screen virtual QWERTY keyboard for text entry. By default, this offers a suggestion for the word the user is currently typing, with the option to tap that suggestion in order to complete the word instead of continuing to type it one letter at a time. This predictive suggestion feature, termed "word completion", can be turned off.

Nokia N900

The Nokia N900 is a smartphone running Maemo 5, on which this experiment will be carried out. At the time of writing it is the only Maemo 5 phone publically available. Relevant features are:

  • Optimised for landscape orientation (which makes for a larger keyboard compared with phones such as the iPhone which are designed for mainly portrait orientation)
  • Resistive touch input (which is more accurate than most capacitative displays such as that found on the iPhone, but with no multi-touch support; also may require slightly more pressure on the screen in order to register a touch compared with a capacitative display)
  • Support for either finger-based or stylus-based touch operation, but for this experiment the stylus will not be offered as many phones do not have this facility
  • Slide-out physical keyboard, but this will not be used for this experiment

Aim of this Experiment

This experiment seeks to determine how useful (or otherwise) the word completion feature is—specifically Nokia's implementation thereof, but in a manner which could perhaps be extrapolated to other phones. Participants will be timed whilst entering various English sentences, with word completion turned either on or off, and with all other options unchanged.

The aim is for accurate text input, and participants must enter the text with few or no errors. For the purpose of this experiment, that means that the words are typed in the correct order and spelled correctly in the majority of cases. Words must be separated by at least one space, but beyond that, punctuation and capitalisation will not be considered.

Method

Overview

The experiment involves asking participants who have had a short time to familiarise themselves with the input method to input four short pieces of English text in a predefined order whilst being timed. Of these, two (randomly selected) will be input with the word completion feature turned on and two with it turned off. The participants will also be asked to fill in a questionnaire to determine their level of experience with this type of input method, and to determine whether they thought the completion feature was useful.

Participants

Volunteer participants will be recruited from amongst the experimenter's friends, colleagues and family and are expected to mainly be aged between 20 and 35. Unfortunately, due to the timescale of this experiment, the sample size will necessarily be small, however some effort will be made to ensure a balanced sample.

Only fluent English speakers will be accepted as participants.

Ethical Considerations

No participants will be invited who are minors or otherwise vulnerable.

All data will be anonymised: each participant will be given a numeric ID and the data and questionnaire responses will be indexed only using that ID. The mapping of participant name to ID will be kept separately and destroyed after the experiment has been concluded (12:00 noon on Monday 6th December 2010).

Participants will be informed that they can withdraw from the experiment at any time without penalty, and that if they do so all data concerning their participation will be discarded.

Participants will be offered no compensation.

Participants will be asked to read and sign a full explanation of what will be involved in the experiment and the way in which the data will be used before the experiment takes place, as follows:

Thanks for volunteering to participate in this experiment. This will involve being timed whilst you enter four pieces of English text using a mobile phone. It should take no more than 15 minutes of your time.

Please read and sign the following declaration to indicate that you are aware of the way in which the experiment will be carried out, and of your rights.

  • I am aware that I have the right to withdraw from the experiment at any time without penalty, and that if I do so, all data concerning my participation will be discarded.
  • I consent to the storing and processing of data gathered from my participation in an anonymised form. In particular, I am happy that the raw data gathered, as well as an analysis thereof, will remain publically visible on the Internet indefinitely. However, no personally-identifiable information will be made public, and the list of participants will be destroyed before Tuesday 7th December 2010.
  • I declare that I am over 18 years of age, and a fluent speaker of English.
  • I understand that this experiment requires accurate reading and entry of text; if I have any concerns about this—for example, eyesight issues or dyslexia—I will discuss them with the experimenter before continuing, who will take a note of them along with my anonymous ID (and no personally-identifiable information) as they may have an influence on the results.

Name:

Signed:

Date:

Anonymous participant ID (assigned by experimenter):

As this experiment will be carried out in accordance with pre-approved requirements, no application to the Ethics Committee is required.

Apparatus

The experiment will be carried out using a single Nokia N900 (see above). Times will be measured on a stopwatch, started and stopped by the experimenter when the participant enters the first and last characters of text respectively.

[screenshot of settings]The N900 offers some options concerning the behaviour of the virtual keyboard (via the Settings application, see screenshot):

  • "Use virtual keyboard"—turning this off means that text entry is impossible without a hardware keyboard. This will be turned on throughout the experiment.
  • "Word completion"—turning this off means that word suggestions are not offered. This is the subject of investigation and data will be gathered with this setting turned on and off.
  • "Auto-capitalisation"—this will automatically capitalise the first word in each sentence. This will be turned on throughout the experiment (although the participants will not be judged on correct case).
  • "Insert space after word"—this causes a space to be added after the user selects a word suggestion (any subsequently-typed punctuation will however be placed before the space). This will be turned on throughout the experiment; it has no effect when word completion is turned off.
  • Language(s) and dictionary for word completion, which will be set to English (United Kingdom) throughout the experiment.

Procedure

  1. Each participant will be allocated a numeric ID (assigned sequentially) and asked to read and sign the declaration.
  2. Participants will be given the background information section (above) to read, in order to ensure that they understand the context of this experiment. They will additionally be told: "If you have used a touch-screen phone before, you may find that you have to press slightly harder on the screen of this one than you are used to", as initial investigation showed that this is the most likely problem a new user of this phone will face, especially if the user is used to the iPhone.
  3. Participants will be allowed to practice entering text on the N900 virtual keyboard, with word completion turned on and all other settings as described above, for two minutes. They will then be allowed one minute of further practice with word completion turned off.
  4. Participants will be shown each piece of text (A, B, C and D) in order, with word completion turned either on or off depending on the table shown below. They will immediately be asked to type a copy of this text on the phone, with the words spelled accurately and in the right order, but without paying undue attention to correct punctuation or capitalisation. A stopwatch will be started when the participant enters the first letter and stopped when they correctly enter the last; the time will be recorded to a resolution of 0.1 second against the participant ID and the text ID.
  5. Finally, participants will be asked to fill in a questionnaire to gauge their level of prior experience of on-screen keyboard text entry, on this or other phone operating system. The questionnaire follows:

Note: please no not write your name or other personally-identifiable information on this sheet.

Participant ID:

Have you used this specific model of phone (Nokia N900) before? (Y / N / don't know)

Have you used a different type of touch-screen phone with an on-screen keyboard before? (Y / N / don't know)

If so, what sort? (e.g. iPhone / Android / Nokia / don't know)

Your age, in years (optional):

Your gender (optional):

You have reached the end of the experiment. Thank you very much for your time!

The text samples are as follows (all excerpted with slight modification from Wikipedia, used and made available under the Creative Commons Attribution-ShareAlike License):

  1. "Albatrosses are among the largest flying birds, and the great albatrosses have the largest wingspans of any extant birds." (source)
  2. "Killer Whales have been known to prey on moose when swimming between islands out of North America's Northwest Coast." (source)
  3. "A local myth persists that the road had to be split because the owners refused to sell the land during its construction." (source)
  4. "Although the texture of white chocolate is similar to milk and dark chocolate, it does not contain any cocoa solids." (source)

Design

The intention is to have an approximately equal number of data points for each text sample which were timed with and without word completion enabled. In order to avoid any effects caused by the order in which the "word completion on" and "word completion off" samples were taken, for example due to gradually-increasing confidence with the input method or due to increasing frustration, the order of "on" and two "off" samples will vary between participants in an approximately balanced manner. In order to achieve this, the order of the text samples remains constant, as follows:

ParticipantSample 1Sample 2Sample 3Sample 4
(text A)(text B)(text C)(text D)
1ononoffoff
2offoffonon
3onoffonoff
4offonoffon
5onoffoffon
6offononoff

This lists every possible combination with two "on" and two "off" samples. Subsequent participants repeat the sequence from 1.

In order to avoid unexpected confounding effects, as much as possible will remain constant. For example, the initial contents of the phone's display will be the same in each case, the text input buffer will be erased before each sample, and the script above will be followed as precisely as possible in every case.

Results

Data from four participants was gathered for each of the four text samples. For each text sample, two times were measured with word completion turned on, and two with it turned off. This is too few samples to analyse each text sample independently, but the datasets cannot simply be concatenated as some samples consistently took longer than others due to slight differences in length or complexity.

Thus the samples were normalised to form a new metric: time relative to mean Ti = timei / mean(all times for same text sample as i)

Hence all samples could be aggregated into two datasets: samples with word completion, and samples without word completion. Histograms of these datasets, with overlayed normal distributions, follow:

Without word completion:
[histogram: without completion]

With word completion:
[histogram: with completion]

Box plot comparing the distributions:
[box plot]

It is noteworthy that if word completion has any effect, it is to increase the time taken to enter text. However an independent-samples t test showed no significant effect of word completion on text entry time (t(13) = -1.22, p > 0.05).

Summary of Demographics and Survey Results

None of the participants had experience of the Nokia N900 on-screen keyboard. (One reported that he had used a N900 before, but not the on-screen keyboard; the others had never used a N900.) All but one had some experience of the Android on-screen keyboard, with one also having experience of the iPhone.

One participant noted that during the experiment with word completion turned on, he felt that there was higher latency between pressing a key and the letter appearing.

Of the four participants, three were male and one female. Ages were 22, 23, 26, 28.

Discussion

First and foremost it is clear that the word completion feature on the Nokia N900 is not beneficial to the demographic tested. This is an interesting result and could merit a further experiment to determine whether this is also the case for a broader demographic.

It may be the case that word completion is actually detrimental, i.e. more of a hindrance than a help. A possible reason for this could be that the user must keep looking up from the keyboard to the area above in which the suggestions are displayed, make a decision as to whether to accept the current suggestion, and move his or her finger to the suggestion to tap it if so, when it may have been quicker to just complete the word manually. However this cannot be said conclusively given the data gathered.

Appendix: Raw data

In the table below, "A,0" refers to "text sample A, without word completion".

Times (minutes:seconds)
ParticipantA,0A,1B,0B,1C,0C,1D,0D,1
11:51.52:01.51:25.61:48.1
21:36.01:20.51:34.51:41.8
31:36.91:41.91:26.11:04.4
41:08.61:22.31:23.61:10.3

Appendix: Transcript of data analysis