All fingers vs. all thumbs

---

1 of 10 

Unbuttoning the Interface
Towards tactile interfaces without discrete controls 

With the exception of gestures, the tactile user interfaces of conventional handheld electronic devices confine operative contacts to predetermined discrete locations configured with controls, to which the functions of the device are mapped. To execute a function the user actuates the control at the location assigned to the function with a finger. The control can be a physical component, like a pushbutton, or virtual, that is, a discrete spot on the interface configured to emulate a physical component, when the user touches it. 

To engage its interface, the user holds a device in a handshake-like grip with the interface facing him; the interface is not fixed in place like a control panel. The fingers holding the device can not be used to engage its interface. When holding and operating the device with the same hand, only the thumb is available. The fact that it is not particularly dextrous, is therefore a liability. A further drawback is that holding the device locks the user’s hand in place, restricting the freedom of movement of the fingers, the thumb in particular. Moreover, the small size and crowded layout of the interface impair the user’s ability pick out individual controls with a blunt instrument like the thumb. 

The following presentation assesses the issues that arise from the fact that conventional interfaces are based on topical controls, that is on discrete controls in fixed locations. It then goes on to explore a way to overcome these issues by means of an innovative tactile interface without buttons or any other kind of topical control. The presentation uses the term ‘interface’ with a systematic ambiguity to refer to the surface configured with controls or sensors that the user’s hand enters into contact with or to the logic the device is equipped with to regulate its operations based on the output from these controls or sensors. What the term means is clear from the context. 

By and large topical user interfaces treat the physical placement of controls as a matter of discretion. Interface layouts typically reflect a desire to ensure a structured, logical presentation, e.g. the numbers in the form of a keypad or itemised lists of choices etc. At times, the layouts are purely conventional, like the slanted columns of the keyboard or the QWERTY layout of its keys, which perpetuate outdated engineering constraints related to mechanical typewriters. Except for interfaces that can be adapted for left or right hand use, little attention is paid to the motor capabilities and limitations of the user’s hand. 

2 of 10 

The size of handheld devices puts space at a premium. It limits the size of their interfaces and in turn the size of their controls and displays. For the most part, interfaces are on the user-facing side of these devices, which they take up more or less completely. The controls and the display, on devices that have one, compete for the same limited space. There are several approaches to optimising the use of the available space, none of them entirely satisfactory. The touchscreen is one. Instead of partitioning the available space into a part for the controls and a part for the display, both share the combined space. This, however, means the fingers operating the device can and in fact often do get in the way of the user’s view of the interface, sometimes masking important landmarks. Another approach has been to increase the size of the device to make room for a larger interface. The gain in interface space, however, is offset by the blown up interface layout which puts some of its controls out of reach. To counter this effect and bring the controls within range, the interfaces of the devices in question can usually be scaled back, allowing the user to toggle between full size for viewing and a reduced configuration to facilitate operatively engaging the controls. Perversely, it is precisely here that a larger interface would be useful. 

The user interfaces of handheld electronic devices are on the front of the device, if only for the user to be able to see their controls, in particular the labels indicating what function they perform. With virtual controls this is a necessity. While the user can detect controls consisting of physical components by their profile, even without being able to see them, virtual controls are dematerialised. They have no parts and no profile or other characteristic feature to set them apart from the rest of the interface surface. They do not even occupy the same spot every time they appear, so their location is no help in their identification. The only effective cues to where they are, what they are and the area they cover are visual. To engage a virtual control, the user has to be able to spot it. This makes them difficult, if not impossible to use with impaired vision, a matter to which we return below. 

To fit the interface, the controls generally have to be small and/or tightly packed. The fingers, on the other hand, are disproportionately large. With their blunt endings they are not suited to pinpointing small controls. Also, the finger hides its effective point of contact, which is much smaller than the fingertip and not where the user visualises it, but slightly set back from the tip of the finger. Consequently, the user’s aim is systematically off and his precision is inherently compromised. Moreover, as a finger closes in on a control, it often masks it, which is not conducive to accuracy either. To make up for this, the domain of a control, i.e.the area the effective point of contact must touch, has to be relatively large to ensure the user can hit it reliably most of the time, notwithstanding the uncertainty of his aim. In addition, closely spaced arrays of controls like keyboards or keypads increase the risk that, in aiming for one, the fingers unintentionally engage an adjacent one. A good deal of engineering effort has gone into making 

3 of 10 

engaging controls reasonably reliable, including the development of algorithms to correct for stray contacts. 

The normal way to hold a device to operate it is the way you hold a deck of cards when dealing a hand, with the thumb poised to shove cards off the top of the deck. However, to engage the controls of a conventional, topical interface the thumb has to perform more complex movements, some of which are outside its natural range, and do so with a degree of precision that is difficult to attain with the thumb. To locate a control the thumb sweeps the interface in the manner of a windshield wiper with a fixed pivot point where its base touches the edge of the device. At full extension the thumb is more or less parallel to the interface surface, its angle of attack practically flat, which makes it difficult to pinpoint a control and degrades the user’s accuracy. At the other extreme, the thumb can not reach points close to the pivot because its two phalanges can not double back on themselves completely. Reaching points close to the pivot requires contortions that are quite unnatural. Controls outside the area swept can not be reached at all unless the user changes his grip on the device. No one would normally use the thumb to push a door bell or an elevator button, but conventional interfaces rely on it for far more complex manipulations. These would be easier to perform with any other finger, but the other fingers are tied up with holding the device. Evidently, the modalities of conventional interfaces for holding and operating handheld devices in one hand are not aligned with/do not make good use of the ergonomic possibilities of the hand. 

Even though handheld electronic devices may have been intended for one-handed use, in view of the drawbacks of holding and operating them with the same hand, users resort to other practices with considerable regularity. One of these stands out. It is to operate the device with one hand while holding it with the other or placing it on a support surface instead (often holding it in place with the other hand). In principle, this makes the more dextrous fingers of the hand available to engage the interface, employing them for tasks that make better use of them than just holding the device. It also affords them a better angle of attack. In reality, though, the limited size of the interfaces and their often compact layout makes using more than one finger impractical, so that most users content themselves with using the index finger, a slight improvement over the thumb. Another popular alternative, for typing in particular, does make use of two fingers, albeit two thumbs. With this method the user cradles the device in both hands, one from each side, and engages the keyboard with both thumbs, the least dextrous of the fingers as has been pointed out before. This method uses the tip of the thumbs to tap the keys, which is rather unusual; nevertheless it makes it possible to achieve higher speeds than typing with one finger alone. Apart from bringing the thumbs into position, holding the device with two hands has no advantage. Overall, the ergonomic benefit of these measures is limited. Besides, employing both hands to 

4 of 10 

operate the device deprives the user of the ability to make use of the spare hand for something else like carrying a briefcase or holding on to a handrail or holding an umbrella, all of which can be useful at times. 

This then illustrates some of the unfortunate consequences of the topical interface paradigm, which users and device designers alike take for granted, on the whole. The core concept it is built on is the control panel. Control panels are assemblies of the controls and readouts required to operate a system or device arranged in a more or less orderly fashion. Their layout is fixed. Standard control panels are mounted in a place conveniently accessible to the user, who can engage their controls with his hands without restriction. The control panels of handheld electronic devices, on the other hand, are part of a portable device the user holds when operating it, not fixed in place where they are ready to use. This changes the game. The topical interface paradigm treats holding the device and operating it as separate tasks. This may seem normal, even logical, but has unfortunate repercussions. In effect it relegates the most dextrous fingers of the hand to holding the device, leaving its residual motor capacity, usually just one finger, the thumb, to operate it. This underutilises the former and misapplies the latter, with the adverse consequences for the usability of the interface outlined above. In fact, it is not necessary to keep holding and operating a device apart. Just think of the way you play a woodwind instrument like a recorder, to see how the two tasks can be combined: the fingers that cover its holes to play a tune serve to hold the instrument at the same time. We will return to this point below. 

On top of the suboptimal use of the fingers, the reduced size of the control panels of handheld electronic devices also has material consequences for their usability and gives rise to a number of issues, many of which are outlined above. These drawbacks are nevertheless accepted as normal or unavoidable, as if there were no other possibility for tactile user interfaces. It is within these confines that interface design evolves. The issues preoccupying research and development are the consequence of the underlying topical interface paradigm. In view of the lack of real progress in resolving them, considerable effort goes to sidestepping them altogether by developing alternative interface modalities such as voice control1 or eye tracking. 

And yet, have the possibilities for tactile interfaces really been exhausted? Is there really no other way to approach the question? Does the tactile user interface really have to be on the user facing side of the device? Is associating functions with distinct locations on the interface the only way to present a set of options to the user? Does the user have to negotiate the space between his 

1 It is unlikely that voice recognition will make typing obsolete. Few people have the capacity to dictate a coherent piece of text of any length, not to mention the ability to edit a text in spoken form by means of spoken commands. This factor is probably more important than the circumstances where dictating aloud would be inappropriate or undesirable, often adduced in support of this point. 

5 of 10 

fingers and the controls on the interface or could the interface put the controls at the user’s fingertips. And could the fingers hold the device and engage the interface at the same time? Does touching the interface with one finger have to have the same effect as touching it with any other? What if individual fingers had distinct values, as they do in certain forms of finger play, finger counting or sign language? Could the user engage the interface in the manner of a secret handshake? 

In fact, there is ample scope for a different approach to the design of tactile interfaces for handheld devices, before turning to developing something different. 

Imagine an interface that can tell the fingers apart, so that, no matter where a finger touches it, the interface recognises which finger it is. This makes it possible to associate functions with fingers, so that the finger determines the function it activates when it touches the interface and not the spot it touches. Accordingly each finger could actuate a function of its own. This is the opposite of a conventional interface, where a given function can be activated by any finger (or even a stylus) as long as it engages the interface at the location associated with the function. With such an interface each finger is capable of activating a specific function, no matter where it touches the interface. It is as though each finger came with its own button, which the user can activate by tapping an undifferentiated interface surface anywhere with the finger. 

As a first approximation this may be good enough to illustrate the new interface paradigm put forward in this paper, in which the finger determines which function it activates and not the spot it touches. However, in reality the interface recognises each finger as it is and not by means of some sort of token attaching to it, in particular not its fingerprint. A simple algorithm tells the fingers apart on the basis of the relative position of the contact patches they (and the base of the thumb) make on the interface surface. The algorithm, which is the subject of a patent application2, is defined for all hands regardless of their size and morphology and no matter whether they are right or left hands. This means that as long as a user can hold a device in a natural, handshake-like grip, the interface can tell the fingers apart and thus determine the functions assigned to each of them. This lets the user operate the device as is and with either hand. No need to configure the interface for a particular user profile or for right or left hand use or to scale it to bring its controls within reach of the user’s fingers. One can think of it a one size fits all solution. 

Mapping functions to fingers lets the user engage the interface without being concerned with the placement or size of controls. There is no distance between fingers and controls that the user has to negotiate. In fact there are no controls. This avoids the issues besetting handheld devices with 

2 See https://patentscope.wipo.int/search/en/detail.jsf?docId=WO2017221141 

6 of 10 

topical interfaces. The issues related to hitting small, often tightly spaced targets with a stubby instrument like a finger simply do not arise. There are no predetermined targets big or small, for the user to aim for. This avoids the issue of lining up the effective point of contact of the user’s fingers and the domain of a control as well as the risk of overlapping with adjacent controls. For the same reason, there is no issue with regard to targets that are difficult or impossible to hit. There is no need for the user to reach for anything that is not at his fingertips, whether it would be a stretch or not. The user simply engages the interface where his fingers come into contact with it when holding the device in a natural manner. The operative contact takes place wherever the fingers touch the interface, when the user is holding the device in a natural, handshake-like grip. The interface is atopical. 

Moreover, when holding a device naturally the user’s fingers do not get in the way of the user- 

facing side. When holding a device in a handshake-like grip, the fingers and the base of the thumb are occupied with securing it by its edges. They do not obscure the display, when there is one. 

To let the user know what actions are available at a given step, the display can be configured to show call-outs pointing to the fingers at the edges of the device and spelling out the function a finger triggers and how to actuate it. (There are several ways in which a finger engages the interface, a matter to which we return below.) This resembles the soft keys featured on devices like ATMs at the edges of a display. Each of these can be programmed dynamically to invoke a function called out in the display, which may change in the course of a session. Unlike soft keys, though, the position of the fingers varies from user to user or if a user changes his grip on the device and there are no buttons for the fingers to press. 

The fact that operative contacts with atopical interfaces are not confined to predetermined locations has unique advantages for visually impaired users, because it does not require the user to engage the interface at predetermined contact points that a visually impaired user can not see; they permit the user to engage the interface where his fingers happen to enter into contact with it. All it takes to render a device usable with impaired vision is for the call-outs to be made audible or haptic. This would allow the user to determine the function associated with a finger by invoking its call-out, say, by pressing the finger and either rejecting the function by letting go or triggering it by keeping the finger pressed. 

Holding a device and operating it are not separable functions within the atopical interface paradigm. All fingers perform both. However, the fingers are already in contact with the interface as a consequence of holding the device. This produces a baseline contact that the user modulates to engage the interface operatively. The presence or absence of a baseline contact can, 

7 of 10 

incidentally, serve to activate the device or put it on standby respectively, obviating the need for a switch. This is not just a technical gimmick; it makes the use of the device intuitive in that it turns on automatically as soon as the user picks it up and holds it to use it. 

To engage a tactile interface of any type the user manipulates it with his fingers. There are three basic ways to do this. The user can exert pressure with a finger, displace its point of contact with the interface or release the pressure. Each of these manipulations changes particular physical attributes of the contact patch the finger makes on the interface surface, in particular its size, pressure and location (specifically the location of its centroid). The interface is equipped with sensors that measure these attributes and pass their output to the interface logic for processing. 

The user can form distinct inputs by combining or otherwise structuring the three basic finger movements to impart a pattern, a kind of coding, to the composite movement, often referred to as a gesture. For example, a finger can perform a movement akin to pressing a button, by momentarily exerting pressure on the interface and then releasing it. Any finger can perform this movement on top of its contribution to holding a device. Since atopical interfaces can tell the fingers apart, pressing yields five distinct inputs. (Topical interfaces can only tell button presses apart if they occur at different predetermined locations.) 

In practical terms the use of a device with an atopical interface is not as different from the use of a device with a conventional topical interface as the conceptual difference between the two interface paradigms might suggest. The finger movements (gestures) a user makes to engage the interface are the same he would make on a topical user interface. Atopical interfaces do not call for any movements the user is not already familiar with. What is more, they only make use of natural finger movements; they avoid any form of unusual finger gymnastics, in particular the contortions conventional handheld devices with topical interfaces require the user to perform at times, with the thumb in particular. There is, of course, no need to acquaint the user with all the features of the new interface up front. Letting him know the effect a finger movement (gesture) has in a particular situational context, allows the user to become proficient with the new interface progressively. Furthermore, call-outs that are visually intuitive, i.e. call-outs that provide visual rather than verbal hints, particularly animated visual hints convey instructions to the user in an intuitive manner, without verbal instructions. A system of call-outs designed along these lines will make using the new interface rather straightforward, easily within the capabilities of the untutored user. [Examples: https://1stwebdesigner.com/wp-content/uploads/2015/10/gif_animation_of_template_gallery.gif and https://netdna.webdesignerdepot.com/uploads/2017/04/3-1.gif and  https://maniacdev.com/wp-content/uploads/2016/04/Pulsator.gif ] 

8 of 10 

Incidentally, the areas taken up by topical and atopical interfaces are largely complementary. It is therefore possible to configure a device with both, which would no doubt facilitate transitioning from one Interface paradigm the other. 

Often the user has no more than five functions at a time to chose between. Take a music player for instance. Once the user has selected a piece, the choices he has are to start, stop, pause, fast forward or rewind the recording. Each of these functions can be allocated to a specific finger. That is, the interface can be programmed so that pressing a finger against the interface surface activates the function mapped to it, anywhere the finger happens to be in contact with the interface when holding the device. 

When there are more than five concurrent choices, there are other ways for a user to select one. For instance, the interface can be configured with a touchpad on the back of the device, which the user engages with his index finger in the usual way. This enables the user to place a curser on any particular item in the display by controlling its movement with the touchpad and to select it by tapping. The user can employ the same manoeuvre to input multi-dimensional or continuous data, to select a location on a map, for example, or to designate the point the user wants a camera to focus on etc. The location of the touchpad on the back of the device where it is out of sight of the user might, at first blush, seem unusual. However, to position a cursor the user tracks the movements of the cursor3, not the movements of his finger on the touchpad or mutatis mutandis the movement of his hand with a mouse, the same as with any pointing device. 

Presenting the choices in the form of a matrix as on the typical home screen of a smartphone, gives the user another way to select an item, valid for six by six matrices (or smaller). The user first selects the row the desired element is on and then, in a second step, the desired element (after the interface has transposed the row selected, allocating each of its elements to a row of its own). Five of the rows are selected by pressing the finger they are mapped to (ideally the proximate finger), with the sixth row selectable by pressing all fingers together. This method lends itself to typing. With the matrix populated with the letters of the alphabet and some special characters, the user could type any letter (i.e. select it from the matrix) in two steps (i.e. 

3 For visually impaired users the interface could be configured to let the user know what choice the cursor is on by means of audible or haptic feedback. 

9 of 10 

keystrokes). This compares favourably with typing using a keypad4 as well as with picking letters on a miniature keyboard. It obviously does not compete with a full-size keyboard that allows the user to type with both hands. This is not very significant, though, given the way users actually employ handheld devices. 

The use of a combination of fingers in this example touches on the concept of chorded keyboards, which could easily be implemented within the interface paradigm put forward here. The fact that this concept has not had much acceptance, in spite of its very long history, would indicate that this approach is not worth pursuing. 

The interface logic is able to distinguish between different composite movements, for instance between pressing (exert pressure on the interface and then release it), pointing (exert pressure, displace point of contact inane direction, eventually repeatedly) and scrolling (exert pressure, displace point of contact in a single direction and release pressure in that order). Since the interface can tell the fingers apart it can treat a given composite movement, scrolling for instance, made with different fingers as distinct inputs, the same as with button presses. A topical interface can not. A given composite movement (gesture) has the same input value no matter the finger or fingers it is made with. When holding a device of typical shape, the user can perform a scrolling movement with two fingers, the thumb and the index finger. The thumb is normally in a vertical orientation with respect to the device and the index finger in a horizontal orientation. In view of the natural orientation of the two fingers, it is more intuitive (but not necessary) to associate an up and down movement in the display, to scroll through a list of choices say, with the thumb and a lateral movement, like panning, with the index finger. 

While any finger can perform a button press in conjunction with holding a device, the weak fingers, i.e. the ring finger and the little finger, can not perform a scrolling movement while holding a device, at least not easily. Limitations of this sort are due to a variety of factors. The effect of holding the device has already been mentioned. Then there are the involuntary linkages between the fingers known as enslavement effects. Last but not least, not all fingers are equally dexterous. As already 

4 Anyone who has learned to type a message on a keypad, will certainly recall that the process is complex. The number of keystrokes per character varies (between one and four) and the flow of keystrokes is syncopated. Furthermore, to type the same character twice in a row the user has to interrupt the already syncopated rhythm but only for specific character sequences. For two spaces in a row, to give an example, the user just hits the space key twice without pausing. Not so for two Ts. Typing two Ts in a row without pausing produces a U. This is because U is on the same key as T. U requires two keystrokes and T one. Pressing the key twice without pausing (for the T to register) produces a U. These heteroclitic conditions are an impediment to fluid typing and add complexity to an already complicated system. 

10 of 10 

mentioned, atopical interfaces do not call for finger movements that involve contortions or are otherwise unnatural. 

As long as a user can hold a device in a natural, handshake-like grip, the interface is usable, as mentioned above. As a corollary to this, as long as the size and shape of a device permit the user to grasp it in this manner, it is possible to configure the device with an atopical interface. Accordingly, atopical interfaces can accommodate a variety of shapes, e.g. devices in the form of a cylinder or a disk (which do not have distinct lateral edges) or a steering wheel, a control stick in an aircraft, the armrest of a wheelchair and more. 

While the structural parts of handheld electronic devices are typically made of a rigid material such as metal, glass or plastic, this is no requirement of the atopical interface paradigm. For the purposes of the interface paradigm the device could be compressible. A compressible outer layer would, indeed, provide a degree of tactile feedback to the user. Moreover, as suggested above, the interface surface could be equipped to give tactile feedback, in the manner of a refreshable Braille display, perhaps, or by momentarily heating a contact patch. 

What sets atopical interfaces apart from conventional topical interfaces is the fact that the operative contacts with the interface are not confined to particular predetermined locations, each associated with a specific function that the user engages by bringing a finger into contact with it. Instead, the operative contacts take place at the contact patches the fingers spontaneously make when holding the device in a natural manner. The interface dynamically maps functions to contact patches and processes the sensor output from them in accordance with the function currently associated with the finger whose contact patch it is. It is the fingers that determine where the operative contacts take place. The interface dynamically aligns with the contact patches of the fingers of any user, rather than the users having to align their fingers with controls at predetermined spots on the interface. There are no buttons or other discrete controls to determine where the interaction must take place.