Type Anywhere You Want: An Introduction to Invisible Mobile Keyboard

1 Introduction

Text-entry plays a crucial role in Human-Computer Interaction (HCI) applications. It offers an effective and efficient way for humans to deliver messages to computers. In the early stage of text-entry research, physical keyboard-based text-entry methods were dominant. As mobile technology integrates into daily interactions in people's lives, the need for a text entry system that suits contemporary mobile devices has emerged. In the process, soft keyboards have become the gold standard of text-entry methods for mobile devices. Soft keyboards do not require additional hardware, which improves mobility and provides an intuitive interface that guarantees usability.

However, current soft keyboards reveal some limitations: high rate of typos, lack of tactile feedback, inconsideration of users' different keyboard mental model, and large screen occupation. Since there are no clear boundaries between keys, typos commonly occur in soft keyboards by pressing an adjacent key by accident. Lack of tactile feedback requires the user's constant monitoring which could result in physical strain. Furthermore, all users are forced to type according to the given standard keyboard without any customization, although each person has a different keyboard mental model depending on their physical structure or typing habits. Lastly, with the development of Information and Communication Technology (ICT), the amount of information that smartphones can provide to humans has exploded, but soft keyboards ironically cover about half of the screens, obstructing the graphical user interface rather than delivering more information compactly.

∗Equal Contribution

To overcome the limitations mentioned above, we propose Invisible Mobile Keyboard (IMK) with our baseline solution, Self-Attention Neural Character Decoder (SA-NCD), and collect the most extensive dataset of text-entry studies to train the proposed decoder. First of all, the proposed IMK is invisible; the entire screen can be used for displaying content which enhances the interaction between human and mobile devices; it lets users freely type text anywhere on the screen according to their typing habits. By fulfilling the IMK decoding task, users can start typing on any position at any angle as they wish.

The proposed decoder for the IMK task, SA-NCD, actualizes the full customization of soft keyboards. SA-NCD first decodes user inputs through a geometric decoder. The decoded result at this step could contain typos since users' mental models would not precisely match the conventional keyboard layout. Then, SA-NCD semantically analyzes the user inputs (the decoded result of the geometric decoder) and corrects the typos using a language model.

In summary, the main contributions of our work are as follows:

• We have collected a large-scale, richly annotated IMK dataset which contains approximately 2M pairs of touch points and text.
• We analyze the characteristics of user behavior on the new proposed task in-depth.
• We propose a novel deep neural architecture, SelfAttention Neural Character Decoder (SA-NCD) as a baseline decoder for IMK systems.
• We conduct a user study with the IMK prototype using our SA-NCD as a proof-of-concept to identify users' demand for research in this area.
• To contribute to the corresponding research community, we make the source code of SA-NCD and the collected data public. The source code and data are available at https://github.com/sahngmin/InvisibleMobileKeyboard.

2 RelatedWork

Text entry systems of mobile devices thus far use various decoding methods, where the methods can be largely divided into two categories: statistics-based and deep-learning-based methods. For statistical decoding methods, generally a probabilistic model is adopted. For instance, in [Findlater andWobbrock, 2012], a Na¨ıve Bayes classifier was implemented to decode user input from an adaptive keyboard where the layout of the keyboard is altered to personalize to users. In [Vertanen et al., 2015], a sentence-based decoding method was used to automatically correct typos after the entire sentence was entered. They adopted a probabilistic model for their decoder, computing the likelihood of each character using the Gaussian distribution of each key. In [Lu et al., 2017], the absolute algorithm and the relative algorithm were proposed to enable eyes-free typing, where the absolute algorithm decodes user input based on the absolute position of touch endpoints, while the relative algorithm decodes based on the vectors between successive touch endpoints. Later, [Shi et al., 2018] enabled eyes-free typing on a touch screen by applying a MarkovBayesian algorithm for input prediction, which considers the relative location between successive touch points within each hand respectively. To allow typing on an invisible keyboard on mobile devices, [Zhu et al., 2018] proposed a decoding mechanism which consists of a spatial model and a decoder. The spatial model links a touch point to a key, while the decoder predicts the next word given the history of previous words.

Recently, with the rise of deep learning, [Ghosh and Kristensson, 2017] developed a sequence-to-sequence neural attention network model for automatic text correction and completion. This was achieved by a combination of a characterlevel CNN and a gated recurrent unit (GRU) [Chung et al., 2014] encoder along with and a word-level GRU attention decoder. In addition, I-Keyboard [Kim et al., 2019], another deep learning-based text entry method, was proposed. For I-Keyboard, Deep Neural Decoder (DND) was proposed as a decoding algorithm which consists of two bidirectional GRUs: one for a statistical decoding layer and the other for a language model. The decoding algorithm we propose in this work is deep-learning-based and employs not only a statistical model with masking process, but also a semantic model to understand the relationship between characters.

3 Dataset

3.1 Dataset Construction

To collect data, we first implemented a simple web user interface which can be accessed from any mobile device. Participants used their own mobile devices to type given phrases on the interface. The details of the data collection procedure are described in the supplementary material. Our dataset contains user initial, age, type of mobile devices, size of the screen, time taken for typing each phrase, and annotation of typed phrases with coordinate values of the typed position (x and y points). The dataset we collected is the first and only dataset for a novel IMK decoding task. Further, we have collected significantly more data compared to the datasets used in the existing text-entry research (see Table 1).

3.2 Dataset Analysis

We analyzed the user behavior when typing on IMK and identify the characteristics of the collected data.

Variance

We calculated the standard score (z-normalization) of x and y coordinates by averaging the entire participants and characters over typing index t. The standard score of x and y positions, zx(t) and zy(t), are calculated as follows:

where v is the vocabulary size, N is the number of participants, xti,j is the x coordinate when i-th participant typing jth character for t-th time, xi,j is the mean value of xti,j taken for all t, and ni,j is the maximum number of times the j-th character appears for the i-th participant. zy(t) is calculated in the same way as zx(t), but with y coordinates. Both zx(t) and zy(t) fluctuate and the fluctuations get larger over time in both vertical and horizontal directions (see Figure 3).

Physical Dimensions

We examined the physical dimensions of user mental models to analyze how widely and actively users moved their fingers to type on IMK. First, we analyzed the positions of 'q' and 'p' since they are the farthest keys from the space bar. The average positions of 'q' and 'p' for each user were inconsistent and varied significantly (see Figure 2). Some users typed 'q' and'p' at an average of only 50 pixels apart on the x-axis, while others typed at a distance of 300 pixels.

In addition, we calculated statistics of the physical dimensions through the average distance between 'space' and 'p' in each sentence. Since 'p' appeared 23.9 times more than 'q' in our dataset, 'p' was adopted for meaningful statistical analysis. We omitted the sentences that did not include 'p' for the analysis. Moreover, we use the average positions of 'space' and 'p' for the sentences where 'space' and 'p' appeared more than once. Table 2 summarizes the mean, standard deviation(std. dev.), min and max values of the scale, and offset of user mental models.

The first distance between 'space' and 'p' was defined as scale 1, and the offset was calculated by taking the difference between the average position of the first 10 inputs and the last 10 inputs. From the statistics, it can be seen that the users type on IMK with their own mental keyboards of inconsistent scales, and the offset over time is large. The fact that the minimum of the y-axis scale is negative means that data also includes some typos such that 'p' key is entered below the space bar.

As the analysis suggests, decoding our IMK dataset is a challenging task. Additionally, as Figure 1 shows, characters are not clustered neatly and typing patterns differ from user to user. Plus, the typing patterns of the right and left hands were distinguishable. Lastly, the distribution of the data consistently changed over time (see Figure 3) and the scale of the keyboard size was undefinable (see Figure 2).

4 A Baseline Approach for IMKDecoder

4.1 ProblemFormulation

We propose a novel baseline approach for the IMK decoding task. Specifically, taking a sequence of coordinate values as input, we aim to decode a phrase which users intended to type on an invisible layout. Let ˆC = { ˆ c1, ..., ˆ cn} denote a predicted character sequence by the decoding algorithm. The decoder aims to find the character sequence, ˆC, with the highest conditional probability given a typing input,

4.2 Network Architecture

Our baseline approach, Self-Attention Neural Character Decoder (SA-NCD), consists of two decoder modules: 1) a geometric decoder (G) with its parameters φG and 2) a semantic decoder (S) with its parameters φS. For an input sequence X = {x1, x2, ..., xn}, let G(X; φG) and S(X; φS) denote the output from the geometric decoder and the semantic decoder, respectively. The geometric decoder takes in a touch input sequence and converts it into a character sequence by using the touch locations in the invisible keyboard layout. Then, the semantic decoder corrects decoding errors in the character sequence estimated by the geometric decoder while considering semantic meanings. Here, the confidence masking process determines the input of the semantic decoder by finding locations of the errors produced by the geometric decoder. The errors could be simply a case of incorrect predictions by the geometric decoder, or a case of absolute typos made by a user. One thing to note is that SA-NCD decodes the entire sequence input up to that point (not merely the most recent touch input). Therefore, the characters decoded in the previous steps are not fixed but can be changed by reflecting the semantic dependence.

Geometric Decoder

We use Bidirectional GRU (BiGRU) for the geometric decoder. To overcome the vanishing gradient problem, we adopt GRU as a recurrent neural network. In our task, even if a user enters the same character index, the input (the touch points) is different every time. Since the inputs are not a definite character index, the recurrent network suffers from the instability accumulation if it only proceeds in uni-direction. Thus, we take the advantage of forward and backward passes of BiGRU to overcome this issue.

Confidence Masking

The length of G(X; φG) = {o1, o2, ..., on} is the same as the input length, and each element has the dimension size of the vocabulary size (v). When considering the i-th input (1 ≤ i ≤ n) from a sequence, the probability (confidence) of the k-th character in the vocabulary, pki , can be calculated by the softmax function as follows:

with

where oji is the value of j-th element of oi in G(X; φG) and v is the vocabulary size.

At this step, the touch input detected at an ambiguous location that is difficult to decode with only geometric information results in low confidence. We replace the positions with a confidence lower than τ with mask tokens to provide the masked character sequence as input to the semantic decoder (confidence masking process). The following CM and Mask refer to the confidence masking process and the individual masking function as follows:

with

where xi = (p1i , p2i , ..., pvi ) and j[mask] is the index of mask token. Embed is an embedding layer which encodes index to an embedded vector.

Semantic Decoder

We use the Transformer [Vaswani et al., 2017] encoder architecture as a semantic decoder. The embedded hidden state that has passed the geometric decoder followed by confidence masking is processed by the semantic decoder with the self-attention mechanism of Transformer. Here, the semantic decoder acts as a character language model, replacing the masked characters with appropriate characters. Through the proposed network architecture (see Figure 4), the decoded character sequence ˆC in Equation (2) can be expressed as follows:

Training Scheme

We pre-train both the geometric decoder and the semantic decoder to enhance their respective roles. In particular, we pre-process Billion Word Benchmark (1BW) [Chelba et al., 2014] to train the semantic decoder as a masked character language model by following the same pre-training scheme of BERT [Devlin et al., 2019]. Then, we fine-tune the two decoder modules together by first freezing φG and only training S(X; φS), and then training G(X; φG) while freezing φS repeatedly.

5 Results

5.1 Experiment Settings

Data Split

Of the 88 participants of our study, 78 participants collected data using 300 sentences of 1BW, and each five of remaining participants collected data using 100 phrases of MacKenzie set (MacK) [MacKenzie and Soukoreff, 2003] and 1,000 most commonly used English phrases set (Common) [EnglishSpeak, 2019], respectively. Of these, we only utilized the data collected from 1BW data to train our decoding algorithm. We used the remaining MacK and Common sets for evaluation purposes to assess whether the proposed decoder performs adequately on sentences it has never seen before.

We divided the 1BW data into a train (from 70 participants), validation (from 3 participants), and test (from 5 participants) sets. We divided the dataset by person rather than combining all the datasets and dividing it by ratio. This split method would verify if SA-NCD can decode sentences even from a new person with an unseen mental model.

Metrics

We used CER (Character Error Rate) and WER (Word Error Rate) as evaluation metrics defined as follows:

where MCD(ˆC, P) is the minimum character distance between the decoded phrase ˆC and the ground-truth phrase P

and lengthc(P) is the number of characters in P. Similarly, WER is defined as

where MWD( ˆC, P ) is the minimum word distance between ˆC and P and lengthw(P) is the number of words in P . CER and WER count the number of insertions, deletions and substitutions of characters or words to transform ˆC into P .

Compared Models

We verified the performance of the proposed baseline SANCD compared to several sequential neural models. We implemented representative deep neural networks designed for sequential data: GRU, LSTM, Transformer, and I-keyboard which is the state-of-the-art method of the invisible keyboard task.

We measured the performance of various networks by varying the sizes which is denoted as (L, dh), where L is the number of stacked layers and dh is the size of the hidden state. In order to thoroughly check the performance according to shallow to deep, narrow to wide networks, we adopted 4 and 12 for L, and 256 and 512 for dh.

Implementation Details

We optimized φG with SGD update, with the learning rate of 3.0 and gradient clipping at 0.5. For φS, we used the Adam optimizer with learning rate 1e-4. Threshold τ for confidence masking was 0.45 and the vocabulary size v was 31. We minimized Cross-entropy loss to optimize the parameters. In addition, to quickly propagate information within the network, we employed the auxiliary losses for the character language model [Al-Rfou et al., 2019]. Moreover, we altered the position of layer normalization prior to self-attention to obtain the stability of the gradient and induce rapid convergence[Xiong et al., 2020].

We implemented deep neural network models with the open-source library PyTorch 1.4.0 version and used CUDA version 10.0 for the one GPU (GTX 1080ti) computation on a Ubuntu 16.04 workstation. We augmented our data by randomly shifting the x and y positions of each touch point by 3 or fewer pixels with a 50% probability. The validation accuracy was measured by using the validation set at every epoch, and if the maximum accuracy was not updated for more than 3 epochs, the training was terminated by early stopping.

5.2 Results and Analysis

Decoding performance

Table 3 shows the decoding performance (CER and WER) of our proposed baseline and the compared models with various network sizes. GRU showed better performance than LSTM in our task. Dealing with the gates of GRU is more advantageous rather than filtering the previous information with the cell state in the LSTM for the sequence data with geometric information. Since I-keyboard stacked two modules of bidirectional GRU, it performed better than a single GRU. Yet, Ikeyboard also suffered from overfitting when stacked deeply, similar to GRU and LSTM. On the other hand, Transformer and SA-NCD could take advantage of greater complexity and lead to higher performance without any overfitting when the model becomes deeper. This was because they utilized efficient multi-head self-attention. However, we found that the Transformer was incapable of extracting meaningful information in the IMK task. On the other hand, by separating the geometric decoder and the semantic decoder, SA-NCD showed the best performance. In particular, the proposed SA-NCD showed 5.51 and 7.71 CER on the MacK set and Common set, which consisted of unseen phrases from unknown users.

Figure 5: Pixel-wise prediction for current input tn when typing each phrase. The letter inside ' ' indicates the label of the current input, cn. Each bold character is marked at the average position.

Ablation study

For in-depth analysis, we conducted an ablation study of our SA-NCD network. Based on the best performing (L, dh) = (12, 512) configuration of SA-NCD, we explored how SANCD is affected without pre-training geometric decoder (G), and pre-training semantic decoder (S) (see Table 4).

SA-NCD w/o pre-training G suffered significantly compared to SA-NCD w/o pre-training S. SA-NCD w/o pretraining G experienced unstable learning due to the characteristics of our input. This suggests that pre-training geometric decoder plays a key role in our training strategy because input T itself contains only geometric information. Therefore the semantic decoder converged well because a stable input G(X; φG) with a fixed character index was provided by the geometric decoder. On the other hand, the decoding performance of SA-NCD improved slightly by pre-training S.

Qualitative Visualization Results

We visualized the outcome of the decoder at a pixel level. When the current touch input tn comes in given the previously touched input sequence {t1, ..., tn-1}, the decoder outputs the character sequence ˆ c1, ..., ˆ cn. We plotted every pair of (tn, ˆ cn) to see how each predicted character by decoder is distributed over the typing area (see Figure 5). The area corresponding to the label is located widely by the decoder.

6 User Study

In order to examine the usability of IMK in real-life settings, we conducted a verification experiment. The details of the user study are described in the supplementary material, including the example video.

We measured WPM (words per minute) as a metric of the typing speed of each participant, where WPM is defined as follows:

where M is the time elapsed from the first keystroke to the last keystroke for ˆC in minutes and nc is the average number of characters in a word.

The users were able to type 157.5% faster by using IMK than the third-party soft keyboard on their own smartphones. We can infer that typing without any strict key layouts in combination with auto typo correction by SA-NCD allowed substantial enhancement in typing speed. We further compared the speed of IMK with other types of text entry methods such as gesture-based, touch-based, and ten-finger text entry (see Table 5). IMK surpassed all of the text entry methods, even I-keyboard which is a text entry system using ten fingers.

We conducted a survey after the verification experiment to evaluate the satisfaction rate of IMK. As Figure 6 shows users were able to type quickly while not experiencing any physical or mental fatigue when using IMK. Overall, the users showed high satisfaction with our system. Additional usability evaluation results are described in the supplementary material.

7 Conclusion

In this work, we proposed a novel Invisible Mobile Keyboard (IMK) decoding task that eliminated the limitations of soft keyboards while maximizing the utility of mobile devices. We collected a suitable benchmark dataset for the task and designed a baseline approach, Self-Attention Neural Character Decoder (SA-NCD), with a new deep neural network structure and training scheme. SA-NCD led to significantly lower decoding error rates than the state-of-the-art methods of other sequential tasks. Besides, through a user study, we witness a typing speed of 51.6 WPM when using our proposed decoding algorithm, which surpassed all of the existing text-entry methods. Simultaneously, the users felt low physical, mental, and temporal demand when using our IMK with high satisfaction. The future works shall be on a front-end design for the practical implementation of IMK.

Acknowledgements

This work was supported by Institute of Information communications Technology Planning & Evaluation(IITP) grant funded by Korea government(MSIT) (No.2016-0-00563, Research on Adaptive Machine Learning Technology Development for Intelligent Autonomous Digital Companion and No.2020-0-00440, Development of Artificial Intelligence Technology that Continuously Improves Itself as the Situation Changes in the Real World).