By Kirsty Beilharz (1), Andrew Vande Moere (2), Barbara Stiel (3), Claudia Calo (1), Martin Tomitsch (2), Adrian Lombard (2): 1. UTS DAB Sense-Aware Lab CB06.50.48, NSW 2007 Australia; 2. Faculty of Architecture, Design and Planning, The University of Sydney, NSW 2006 Australia; 3. Research Group for Industrial Software (INSO), Vienna University of Technology,1040 Vienna, Austria
 |
| Figure 1. Folds and contours in stone revealing historic, formative processes. Photo by Andrew Vande Moere |
'Blake' is a wearable sonification and visualisation display that uses physical analogue visualisation and digital sonification to convey feedback about the wearer's activity and environment. Intended to bridge a gap between art aesthetics, fashionable technologies and informative physical computing, the user experience evaluation reveals the wearers' responses and understanding of a novel medium for wearable expression. The study reveals useful insights for wearable device design in general and future iterations of this sonification and visualisation display. Our motivation includes the design of a wearable sonification and visualisation device conceived in a physical computing paradigm suitable for wearing on the body - distinguishing it from a miniaturised screen-based display paradigm that draws on desktop computing; a discussion of abstract vs. literal information representation, public vs. private information revelation; and a brief technical explanation of the information captured, processed and represented to the user.
 |
Figure 2. Painted drapery and folds in cloth hint about people, actions and interactions. Two draped open coat paintings by Australian artist, Amanda Robins [10]. |
Background
The idea of the folding metaphor to represent data visually casts back to mythological origins of giving voice to fabrics and the tradition in drapery and arts of creating folds to signify embedded meanings. Our project explores the inter-textuality and inter-modality of drapery (found originally in painting) as the context for sonic drapery or sonification of material in motion and a novel wearable visualisation able to sense and externalise environmental data about the wearer using a deliberately subtle and ambiguous representation metaphor. Folds in fabric, clothes, materials that were historically painted and drawn, as well as folds in the house (sheets, indentations in chairs, cushions), folds in the geological landscape (Figure 1) and layering of contoured stone each convey traces, remnants, memories or an embedded history of events or people that have led to the present state (Figure 2).
The representation or physical state of the folds captures the immediate past and the present and is less ephemeral than a non-cumulative real-time display only capable of representing transient information at the moment it occurs. The physical representation captures a blend of immediate real-time response and an analysis of cumulative historic transactions sampled over the previous ten-minute interval. For example, a display that responds to the level of motion activity in the moment does not reveal anything that is not immediately obvious to the user but it may be interesting to reflect on levels of activity, social interaction, noise levels over a period of time and in comparison with the sonification of the previous interval, observing change. These kinds of sensed data correlate to more subjective, socially relevant experiences, such as: being in a noisy place, meeting friends, sleeping, relaxing, playing sport, etc.
The first iteration of the folding wearable expression took the form of felt pouches that revealed coloured layers as slits were stretched open by muscle wire (Figure 3) in response to social interactions. Some technical modifications led to a second iteration that added sonification capability in the sensing microphone, computation and rendering through a small on-board synthesizer and speaker as well as reduced heat emission of the muscle wires and power source. A significantly smaller battery and more efficient circuit design was implemented to overcome weight and temperature concerns that affected the user's comfort and mobility. The power source and visualisation, sonification folding display were integrated into a single encapsulation material design, worn over the shoulder, with no external wires or controls (Figure 4).
 |
Figure 3. Iteration 1 - felt pouch revealing coloured slits in response to motion. Photo by Monika Hoinkis. |
The first iteration of the folding wearable expression took the form of felt pouches that revealed coloured layers as slits were stretched open by muscle wire (Figure 3) in response to social interactions. Some technical modifications led to a second iteration that added sonification capability in the sensing microphone, computation and rendering through a small on-board synthesizer and speaker as well as reduced heat emission of the muscle wires and power source. A significantly smaller battery and more efficient circuit design was implemented to overcome weight and temperature concerns that affected the user's comfort and mobility. The power source and visualisation, sonification folding display were integrated into a single encapsulation material design, worn over the shoulder, with no external wires or controls (Figure 4).
Physical Visualisation and the Folding Metaphor
The wearable visualisation and sonification display represents live sensor-based information (processed against cumulative time-based calculations) to the wearer and people in her/his immediate vicinity, with the aim of augmenting social interaction and interplay. Introducing the concept of analogue representation into the digital mobile/wearable computing domain, we attempt to preserve the affordance of the physical medium by avoiding the anti-ubiquitous (clichéd, inartistic) communication crutch of an LCD screen. Instead of the common integration of the desktop metaphor in many miniaturised devices of today, we found it important to reinterpret the portable display medium and to try to return some of the tangible, emotive, yet subtle and semantic, expressive attributes of analogue media in order to gain the power of self-representation, expression, reflection and control of privacy. We hypothesised that users may attribute greater importance to appearance, sound, aesthetic materiality over utility and familiar technologies in a wearable display (see User Experience Evaluation following in section 4).
 |
Figure 4. Iteration 2 - integrated power, sensors and analogue visualisation. Photo by Kirsty Beilharz. |
For example, Kodama and Takeno’s Protrude, Flow [6] and Frey’s Snoil [4] are physical displays that use the same medium, ferro-fluid, a liquid that becomes strongly polarised in the presence of a magnetic field. Whereas one ferro-fluid “display” is based on simulating an underlying 10x10 pixel-matrix, the other is specifically exploring the material’s physicality and behavior, to create abstract, unpredictable shapes that require intense and emotional involvement of the viewer. The resulting pixilated and extruded outcomes utilise very different techniques expressed through an identical presentation medium. We try to elicit an emotional response and imply expressive qualities through folded fabrics and dynamic textile design.
In the context of self-expression, an open interpretation invites onlookers to imagine their personal thoughts and feelings, potentially going beyond what the wearer, or even the designer, had originally anticipated. Such ambiguous systems may even be safer than those implying a single preferred meaning, since they highlight the user’s own responsibility in deriving a suitable interpretation [13]. Our display method is not meant to convey precise information to the public, but rather aims to simulate a subtle form of visual expression that is not intruding the wearer’s physical or emotional presence.
Sonification
Sonification uses non-speech sound to represent data, i.e. it is auditory encoding or auditory graphing, primarily for informative purposes, such as scientific monitoring. We consider here the re-contextualising of abstract sonification for wearable computing. The research also queries the interaction between auditory and visual display in this bi-modal scenario, the abilities of sound to reinforce, redundantly or otherwise, the information being shown in the visualisation and the potential for visualisation to capture a “permanent state” or lasting memory condition of system status that triggers a momentary audio event. The sonic representations of behavioural data are emitted only periodically every 10 minutes in a conflated burst whose attributes represent a concatenation of events and time/activity analytical calculations in the preceding interval of wear. To use an on-board integrated miniature synthesiser and speaker, constrained the sound production to a tiny Soundgin device whose sound qualities resemble those people hear from mobile phones. The perceived loudness of the device itself is affected by the wearer's context. The sonification affords a new perspective because it can be peripherally absorbed without requiring the user to view the device directly and thereby operates in a partially ambient way.
Aesthetics for Wearable Expression
The wearable display acts as an interpretation of the nature of one’s individuality by interpreting environmental features captured through parallel, real-time sensor readings. The wearable display is capable of revealing an historical view: it travelled with users to work, social encounters, outdoors, in individual home contexts and highly interactive crowded spaces, capturing through proximity, motion and sound-sensing attributes of both the wearer and her/his environment. We define wearable visualisation as a research field that conceptually relates to “wearable computing”, “smartwear” and electronic fashion technology (e-fashion). E-fashion typically addresses issues ranging from embedded system hardware and software, to high-tech reactive electronic fabrics. However, instead of focusing on sensor and signal analysis, real-time context recognition or hardware development and miniaturisation, wearable visualisation is specifically concerned with the visual and auditory communication of information to the wearer, or any people present in the wearer’s physical vicinity, in a truly pervasive way.
Some research projects have already implemented simple forms of wearable displays interpreted as notification interfaces [7]. Other creative initiatives have explored more artistic application purposes, while investigating radically alternative technical means to convey sensor data publicly on or around the human body. These approaches tend to experiment beyond the use of simple LCD or pixel-based displays, for instance by utilising subtle forms of LED lights [2], so-called ‘e-textiles’ [8], electroluminescent wires (e.g. [11]), thermo-chromatic inks, shape-changing materials such as shape memory alloys (e.g. [3]), or even air inflatables (e.g. [1]). Conceptually, a wearable visualisation can be interpreted as a fashionable accessory that augments self-representation, similar to jewellery or high-tech gadgetry. Such objects are often deliberately chosen to reveal, or hide, personal aspects, such as one’s mood or social status, to the outside world.
Abstract vs. Explicit Information Expression
The subjective interpretation of a “wearable” display, by the wearer on any person in the vicinity, is inherently influenced by the ever-changing context it is perceived in, such as the wearer’s physical presence, in time (e.g. morning versus late-night) as well as by its environmental location (e.g. at home versus in a bar). As such, the design of a wearable display is obliged to take into account several ethical considerations, such as accuracy (does the display accurately measure and reflect the actual situation?) and privacy (does the wearer wish to show private data, or alternatively, are onlookers willing to be exposed to it?). The representation metaphor thus requires a degree of “deliberate ambiguity”, depending on the context it is observed in. The display attempts to avoid creating any barrier for social communication in face-to-face interactions. The wearable visualization should become a natural extension of the person herself: neither the wearer nor the people she interacts with should be (negatively) disturbed by it. Therefore, the display is designed to be relatively style-less, so it can “blend in” with everyday accessories that people use for functional decoration (e.g. scarves, gloves, fabric brooches, tie, etc.). By using the fabric as the main communication medium, the display should not be perceived as a technological or aesthetic intruder. More specifically, the display method experiments with the folding (or bending) of fabric to convey meaning. Conceptually, folds are visual and tangible manifestations of archived actions, they are physical artefacts, reminiscent and irrefutable proof of what happened in the past.
 |
Figure 5. 'Internals' of folding display control boards, sensors, and muscle wire activation. Photo by Kirsty Beilharz. |
The sensing part of the wearable expression captures ambient noise levels and executes a time-based concatenation using a small microphone; a light sensor whose purpose is to distinguish between an indoor or outdoor environment; and multi-dimensional gyroscopic sensor to capture movement information. The programming side was used to examine this data as a factor of time, e.g. amount of activity and whether that represents an increase or decrease, whether sustained or momentary, and comparative ambient noise levels. The Soundgin synthesiser and small 5V sensors were used in conjunction with an Arduino microprocessor board and Processing programming environment, together with a custom board (Figure 5) that operated the controllers and output. Output or physical display consisted of the sound speaker and moving fabric slits revealing different mounts of underlying colour. Muscle wire (extremely fine conductive thread that contracts when charged with electrical current) was used as the mechanism to pull the material, counter-released by elastic threads for returning openings to their original state (because there is no "undo" operation for muscle wire).
User Experience
User Experience
Designers always have to conceive of something intuitive (which is easy to learn and to use), innovative, useful and desirable. An intuitive and user-centered device is a system, which makes technology disappear during the user’s engagement with it. By involving potential users in the design process and employing their knowledge and intuition, it is possible to improve designs towards this goal. The methodology we pursued aimed to capture the hidden and underlying needs of users in order to develop a product that will best meet the consumers’ expectations and desires.
Participants and Methodology
 |
Figure 6. Percentage of males and females involved in the evaluation process. |
Five participants took part in the evaluation process, including: four students (female: two; male: two; age range 26-40) and one employee (female, age 28), (see Figure 6).
The evaluation participants were asked to wear the foldable display for three days during their normal daily activities. They were also asked to complete a diary (see Figure 7) in which they wrote their responses: what they understood about the visualisation and sonification in the physical computing display; and the reactions the device provoked from onlookers (e.g. family, friends and strangers) in order to keep track of their interaction with the wearable display.
Users also participated in a brief daily one-to-one interview, useful to review their experience with the wearable device and completed a questionnaire in which they were asked to describe the device in their own words and to offer tips and suggestions for future iterations. All subjects were fluent using technologies. They were not aware of the device’s purpose and they did not receive any incentive for their contribution.
 |
Figure 7. A page of the diary given to the evaluation participants. |
By gathering comments and thoughts from the recruited users, it was hoped to detect critical issues in the system and increase its usability and ergonomics. During the evaluation study, different kinds of qualitative data were collected.
First impressions were collected that bridge the associative gap between wearable devices, fashion and consumers. It is not surprising that some users suggested transforming the wearable system into a fashion accessory. In effect, subjects indicated that merging wearable systems with fashion design (e-fashion) would be strongly effective for both gathering data and revealing information about wearers during their daily activities. Doing that, users could wear the device with greater naturalness and without feeling different from other people. Ironically, this turned out to be quite the opposite view to the designers' hypothesis that an attention-seeking, conspicuous approach might be novel, interesting, evoking curiosity and expressive.
Some participants proposed the integration of flashing lights and voice in the wearable system. This response could be the consequence of the gulf of evaluation. According to Donald A. Norman, "The gulf is small when the system provides information about its state in a form that is easy to get, is easy to interpret, and matches the way the person thinks of the system" [9]. Hence, the gulf of evaluation is the difficulty of assessing the state of the system and how well the device supports the discovery and interpretation of that state. In this case, wearers did not seem to accurately perceive and interpret the meaning communicated by the flexible display in sound or visual display. Some participants found the wearable display un- ergonomic. Ergonomics refers to making artefacts comfortable and efficient, playing to the strengths, not stressing the body part interfacing with them. The most perfect ergonomic wearable device is something that the user does not notice on her/his body. The body is the primary input device for any wearable interface and kinesiology helps designers understand the ergonomics of motion and the general capabilities of the average human [12].
 |
Figure 8. "Off the leash" by W. B. Park [9]. |
Future Directions
On the whole, the obtained data indicates that the foldable display requires further design improvements that draw from e-fashion, ergonomics and customisation, in order to communicate a stronger affordance to users and limit the gulf of evaluation (Figure 8). There remains a significant gap between the expectations/aspirations of the designers and relative conservatism of expectations of users who generally preferred to liken the device to familiar display methods and explicit representations. Analogue, physical and subtle display aesthetics are not intuitive or easily semantic and remain more difficult to appreciate. It seems that the body-worn aspect influences users strongly to desire a fashionable, customisable, changeable and controllable device. Greater contextual awareness or intelligence on the part of the device, able to respond to different noise levels and social contexts would be a major improvement. The differences between designers' intentions and users' expectations or satisfaction corroborates the value in multiple iterative user evaluation studies throughout the prototyping development, especially as this research is trying to break into a new medium of expression converging informative and sociable, aesthetic and fashion principles in a single device. As wearable expression is clearly hugely subjective, perhaps the best approach is to emphasise customisability, configurability and greater user intervention in the sonification and visualisation wearable device.
Acknowledgements
Acknowledgements
This research was supported by a University of Sydney R&D New Staff Grant (Author 2 2006) for wearable visualisation and was subsequently developed with the support of Australian Research Council Discovery Project DP0773107 (Kirsty Beilharz 2007-2010) integrating aesthetic gesture sonification. Special thanks to Monika Hoinkis, Justin Clayden, Adrian Lombard and Andrew Wabnitz for contributing to the conceptual design and technical implementation; and to Author 3 and her supervisor, Martin Tomitsch, for assistance in the user experience evaluation and analysis. This work is supported by University of Technology, Sydney Sense-Aware Lab. The user experience evaluation is approved by the University of Sydney Human Research Ethics Committee (Project 02- 2006/8811).
References
References
- Almeida, T. Modes for Urban Moods. Thesis, New York University Interactive Telecommunications Program, NY, 2005.
- Berzowska, J. Memory Rich Clothing: Second Skins that Communicate Physical Memory. In Proceedings of Conference on Creativity & Cognition, London, ACM Press (April 2005), 32-40.
- Berzowska, J. and Coelho, M. (2005) ‘Kukkia and Vilkas: Kinetic Electronic Garments. In Proceedings of Symposium on Wearable Computers ISWC’05, Osaka, IEEE (October), 82-85.
- Frey, M. Snoil - A display based on Ferrofluid, design developed at The Berlin University of the Arts, 2005.
- Gaver, W. W., Beaver, J. and Benford, S. Ambiguity as a Resource for Design’. In Proceedings of CHI’03, Fort Lauderdale FL, ACM (April 2003), 233-240.
- Kodama, S. and Takeno, M. Sound-Responsive Magnetic Fluid Display. In Proceedings of INTERACT’01, Waseda Japan, IOS Press (July 2003), 737-738.
- Kostov, Vlaho, Jun Ozawa, and Satoshi Matsuura Analysis of Wearable Interface Factors for Appropriate Information Notification. In Proceedings of IEEE International Symposium on Wearable Computers (ISWC'04), Washington, October-November, 2004.
- Martin, T., Jones, M., Edmison, J. and Shenoy, R. Towards a Design Framework for Wearable Electronic Textiles. In Proccedings of IEEE International Symposium on Wearable Computers (ISWC’03), IEEE White Plains NY, (October 2003), 190 -200.
- Norman, A. D. The Design of Everyday Things. Basic Books, June 12, 1988.
- Robins, A. Slow Art: Painting and Drawing as a Meditative Process. Ph.D Thesis, University of NSW, 2006.
- Rosella, F. and Genz, R. Cute Circuit Kinetic Dress. SIGGRAPH Cyber Fashion Show. Los Angeles, July, 2005.
- Saffer, D. Designing Gestural interfaces. O’Reilly Media, Inc., November, 2008.
- Sengers, P. and Gaver, B. Staying Open to Interpretation: Engaging Multiple Meanings in Design and Evaluation. In Proceedings of ACM Conference on Design Interactive Systems (DIS’06), Pennsylvania, ACM, June 2006, 99-108.
