ITA: Interactive Technologies for Accessibility
ITA: Interactive Technologies for Accessibility is an ACIM Research Cluster spearheaded by Dr. Kening Zhu, Prof. Hongbo Fu, and Dr. Miu Ling Lam. The team aims to create new interactive techniques to improve accessibility. The three directors are experts in the fields of computer graphics, human-computer interaction, robotics, computational imaging, and tangible user interfaces.
The research objectives are:
- To develop innovative interactive technologies to help people with various accessibility problems (e.g., children, elderly, people with disabilities, etc.);
- To augment the capability of non-experienced users on professional computational tasks (e.g., rapid hardware prototyping, 3D modeling and animation) through interactive technologies
- To evaluate the task effectiveness and the user experience of these technologies
- To promote social empathy towards in-need communities through art-tech education
This project (ITA 3.0) is to expand and deepen our previous ACIM projects: ITA and ITA 2.0. During the project period (2017 - 2022), we have developed a series of technologies and prototypes to enhance people’s accessibility to digital information, including AR technologies for 3D content creation, smart ring, VR gloves, and white cane with thermotactile feedbacks, tangible programming blocks for visually-impaired children, text-entry technique on the fingertip for smart devices, bezel-initiated swipe interaction for round smartwatches, and so on. With these results, we received external grants with a total amount of over HKD 20 million (e.g., GRF, NSFC, Jockey Club donation, and so on), published over 70 papers, initiated more than 10 patent applications, and presented several art exhibitions.
For ITA 3.0, we propose to promote the social understanding of accessibility and inclusion through art and technology by developing novel art-tech toolkits and deploying art-tech educational workshops to the youth. In addition, we continue to leverage the smart technologies, such as machine/deep learning and smart devices (e.g., smartphones, watches, wearables, sensors, and so on), to improve the life for people with various accessibility problems. We aim to improve the accessibility of digital content through multimodal and embodied interaction, such as hand gestures, body postures, voices, and haptic feedback.
PROJECTS
Impression Machine is a media art project based on robotics and kinetic photography. In this project, we have developed a new system and used a highly experimental approach to create a new visual experience for the audience. The work is embodied as a "performative" installation comprising a 6-axis robot arm, a digital camera, a computer and two screens (a 60" TV and a 3m wide projection). The system/machine/installation is performing while the audience is invited to observe a sequence of events, then comprehend the reasons and meanings behind the abstract visuals presented.
The camera is mounted on the robot arm, moving in the space while taking long exposure photographs. On the TV, a set of contour lines are displayed and captured by the camera. As the camera motion and the displayed contour lines are tightly synchronized, the light contours appear to be stacked in an aligned manner in the long exposures, constituting a number of 3D geometric shapes conceived by Leonardo da Vinci. This asks us to rethink how "perspective" is created in a 2D representation (such as a painting or a photograph), and directly responds to Leonardo's study of perspective. Each resultant long exposure photo is displayed on the projection wall. All light contours are real-time rendered and all long exposures are real-time captured on site. This work has developed a technological innovation for new media art embodiment and facilitated a new artistic experience for the audience.
Impression Machine has been shown in "Leonardo da Vinci: Art & Science. Then & Now” Exhibition" during September to December 2019 in CityU Indra and Harry Banga Gallery. 12 contemporary artworks, including Impression Machine, alongside 12 original drawings of Leonardo da Vinci are displayed in the exhibition, celebrating the 500th anniversary of his death. The exhibition has attracted over 11,000 visitors.
The project was partially supported by ACIM Research Fellowship and HKADC Project Grant.
Website: https://www.cityu.edu.hk/bg/exhibitions/leonardo-da-vinci
ThermAirGlove (Kening Zhu) is a pneumatic glove that provides thermal feedback to support the haptic experience of grabbing objects of different temperatures and materials in VR. The system consists of a glove with five inflatable airbags on the fingers and the palm, two temperature chambers (one hot and one cold), and the closed-loop pneumatic thermal control system. User studies on VR experience showed that using TAGlove in immersive VR could significantly improve users’ experience of presence compared to the situations without any temperature or material simulation.
Eyes-free Smartwatches (Kening Zhu) expands the interaction space of touch-screen devices (e.g., smartphones and smartwatches) via bezel gestures. While existing works have focused on bezel-initiated swipe on square screens, we investigate the usability of BIS on round smartwatches via six different circular bezel layouts. We evaluated the user performance of BIS on these layouts in an eyes-free situation and found that the performance of BIS is highly orientation dependent, and varies significantly among users. We then compare the performance of personal and general ML models, and find that personal models significantly improve the accuracy for a range of layouts. Lastly, we discuss the potential applications enabled by the BIS.
TipText (Kening Zhu) investigates new text entry techniques using micro thumb-tip gestures, specifically using a miniature QWERTY keyboard residing invisibly on the first segment of the index finger. Text entry can be carried out using the thumb-tip to tap the tip of the index finger. The keyboard layout is optimized for eyes-free input by utilizing a spatial model reflecting the users' natural spatial awareness of key locations on the index finger. Our user evaluation showed that participants achieved an average text entry speed of 11.9 WPM and were able to type as fast as 13.3 WPM towards the end of the experiment. Winner: Best Paper Award, ACM UIST 2019
PI: Kening Zhu
Chen, Taizhou, Lantian Xu, Xianshan Xu, and Kening Zhu (*). "GestOnHMD: Enabling Gesture-based Interaction on the Surface of Low-cost VR Head-Mounted Display". IEEE Transactions on Visualization and Computer Graphics, vol. 27, no. 5, pp. 2597-2607, May 2021, doi: 10.1109/TVCG.2021.3067689.
Low-cost virtual-reality (VR) head-mounted displays (HMDs) with the integration of smartphones have brought the immersive VR to the masses, and increased the ubiquity of VR. However, these systems are often limited by their poor interactivity. In this paper, we present GestOnHMD, a gesture-based interaction technique and a gesture-classification pipeline that leverages the stereo microphones in a commodity smartphone to detect the tapping and the scratching gestures on the front, the left, and the right surfaces on a mobile VR headset. Taking the Google Cardboard as our focused headset, we first conducted a gesture-elicitation study to generate 150 user-defined gestures with 50 on each surface. We then selected 15, 9, and 9 gestures for the front, the left, and the right surfaces respectively based on user preferences and signal detectability. We constructed a data set containing the acoustic signals of 18 users performing these on-surface gestures, and trained the deep-learning classification models for gesture detection and recognition. The three-step pipeline of GestOnHMD achieved an overall accuracy of 98.2% for gesture detection, 98.2% for surface recognition, and 97.7% for gesture recognition. Lastly, with the real-time demonstration of GestOnHMD, we conducted a series of online participatory-design sessions to collect a set of user-defined gesture-referent mappings that could potentially benefit from GestOnHMD.
For more information about the project, please visit https://meilab-hk.github.io/projectpages/gestonhmd.html
PI: Miu Ling Lam
AIFNet is a deep neural network for removing spatially-varying defocus blur from a single defocused image. We leverage light field synthetic aperture and refocusing techniques to generate a large set of realistic defocused and all-in-focus image pairs depicting a variety of natural scenes for network training. AIFNet consists of three modules: defocus map estimation, deblurring and domain adaptation. The effects and performance of various network components are extensively evaluated. We also compare our method with existing solutions using several publicly available datasets. Quantitative and qualitative evaluations demonstrate that AIFNet shows the state-of-the-art performance.
For more information about the project, please visit:
https://sweb.cityu.edu.hk/miullam/AIFNET/
People
