PROJECTS

Design Strategies for Concurrent Sonification-Visualisation of Geodata

• PI :: PerMagnus Lindborg, PhD, Associate Professor, School of Creative Media, City University of Hong Kong
• Co-I :: Sara Lenzi, PhD, Ikerbasque Research Fellow, University of Deusto, Spain
• Co-I :: Paolo Ciuccarelli, Professor of Design, Northeastern University, Boston, USA

About

Sonification is the translation of data into sound. Inherently interdisciplinary, the field has seen tremendous development characterised by 1) expanding the definition to embrace aesthetics, via electroacoustic music composition; 2) professionalisation of terminology, techniques, and community-building; and 3) increased attention to visualisation. Time is ripe to focus efforts on the third point. We employ knowledge from dynamic data visualisation to improve on sonification techniques, to generate a cross-modal perception informed theoretical framework, and to determine practicable strategies for concurrent sonification-visualisation design. Project targets are: 1) a set of design guidelines, and 2) a proof-of-concept software system applied to geodata with real-life importance, such as rain and wind, pollution and traffic, forest fires and landslides. People seek to understand their physical environment. Accurate and engaging information design helps both in everyday activities and in making life-choices. Laying the research groundwork for a concurrent sonification-visualisation system for communicating environmental geodata has the potential for real-life applications with broad public appeal and societal impact. Ultimately, the goal of the project is to contribute to the digital fabric of society and improve people’s quality of life.

Funding

Stragegic Research Grant (SRG-Fd), City University of Hong Kong (2023/09–2025/02)

Links

• PerMagnus Lindborg, https://www.scm.cityu.edu.hk/people/lindborg-permagnus
• Sara Lenzi, https://www.saralenzi.com/
• Paolo Ciuccarelli, https://camd.northeastern.edu/faculty/paolo-ciuccarelli/
• Data Sonification Archive, http://sonification.design
• Repository on GitHub, https://github.com/SonoSofisms/ears-eyes

Publications by the team Lindborg

• PM, Caiola V, Chen M, Ciuccarelli P, Lenzi S (2023/09, in review). “A Meta-Analysis of Project Classifications in the Data Sonification Archive ”. J Audio Engineering Society.
• Lenzi S, Lindborg PM, Han NZ, Spagnol S, Kamphuis D, Özcan E (2023/09). “Disturbed Sleep: Estimating Night-time Sound Annoyance at a Hospital Ward”. Proc European Acoustics Association.
• Lindborg, PM, Lenzi S & Chen M (2023/01). “Climate Data Sonification and Visualisation: An Analysis of Aesthetics, Characteristics, and Topics in 32 Recent Projects”. Frontiers Psych 13.
• Lenzi S, Sádaba J, and Lindborg PM (2021). “Soundscape in Times of Change: Case Study of a City Neighbourhood During the COVID-19 Lockdown.” Frontiers Psych 12:412.
• Lenzi S & Ciuccarelli P (2020). “Intentionality and design in the data sonification of social issues.” Big Data and Society.

GRF #11605622, 2022/23

PI :: PerMagnus Lindborg (City University of Hong Kong)

Co-Is :: Francesco Aletta (University College London, UK), Kongmeng Liew (Nara Institute of Science and Technology, Japan), Yudai Matsuda (CityU, HK), Jieling Xiao (City University of Birmingham, UK).

Project website :: http://soundlab.scm.cityu.edu.hk/mmhk/

Abstract

The sensory cultural heritage, combining tangible and intangible heritage, creates identity and cohesion in a community. In urban research, analysis of everyday-ish and informal customs typically rely on visual images, texts, and archival materials, to describe the multifarious aspects of culturally significant places and practices. By contrast, the acoustic environment is often not part of the narrative, and very rarely is the olfactory environment recorded. Given the contemporary context of rapid and profound transformation in Hong Kong, essential threads of the city fabric risk being neglected, and might even disappear before they can be documented. Can we really claim to know urban places without thoroughly considering, and documenting, the sensory cultural heritage represented by sounds and smells?

Our project seeks to preserve the threatened environment of some of Hong Kong’s signature sites and create a more accurate and richer understanding of culturally important places, rituals, and social practices, allowing greater appreciation of the heritage. In the meantime, this project aims to shed light on the crossmodal relationships between urban landscape, soundscape, and smellscape.

We propose a multimodal research approach that takes sound and smell as core components of the immersive urban experience. The project will document a large sample of characteristic sites in Hong Kong, focusing on places for Street food (街頭小食), Chinese Temples (寺廟 [佛祖, 天后…]), and Wet markets (傳統市場). The database will be open-access via a dedicated project website, connecting with recently initiated international soundscape–smellscape projects. It will contribute to the current need for detailed documentation of the local cultural heritage; support interdisciplinary collaborations; be a significant resource for future longitudinal studies of urbanism in Hong Kong; and a reference point for cross-cultural studies with other cities.

Methodology-wise, we will develop a capacity to systematically collect and analyse data from complex physical environments, integrating sonic and olfactory measurements with video capture and narratives. Field data will be both objective and subjective, to include 360 ̊ video, 3D audio (Ambisonics), and ‘smellprints’ (gas chromatography-mass spectrometry analysis of air samples), as well as sensory walks with observers making structured annotations of the perceived visual, auditive, and olfactory environment, and interviews with stakeholders. The database generated in the project will serve further research in environmental psychology, multimodal perception, and sensory integration. It will also prepare the ground for future multisensorial applications in virtual tourism, art, games, film, and spatial design at museums, galleries, and commercial venues.