Dissertation Defense Announcement

Faruk Ahmed

October 22, 2019 at 02:30 PM in Engineering Science Building

Advisor: Mohammed Yeasin

Ambient awareness on a sidewalk for visually impaired

ABSTRACT: Safe navigation by avoiding obstacles is vital for visually impaired while walking on a sidewalk. There are both static and dynamic obstacles to avoid. Detection, monitoring, and estimating the threat posed by obstacles remain challenging. Also, it is imperative that the design of the system must be energy efficient and low cost. An additional challenge in designing an interactive system capable of providing useful feedback is to minimize users' cognitive load. We started the development of the prototype system through classifying obstacles and providing feedback. To overcome the limitations of the classification based system, we adopted the image annotation framework in describing the scene, which may or may not include the obstacles. Both solutions partially solved the safe navigation but were found to be ineffective in providing meaningful feedback and issues with the diurnal cycle. To address such limitations, we introduce the notion of free-path and threat level imposed by the static or dynamic obstacles. This solution reduced the overhead of obstacle detection and helped in designing meaningful feedback. Affording users a natural conversation through interactive dialog enabled interface was found to be effective in safer navigation. In this dissertation, we modeled the free-path and threat level using a reinforcement learning (RL) framework. We built the RL model in the Gazebo robot simulation environment and implanted that in a handheld device. Feeding the Wizard of OZ conversational data materialized the natural conversation model through the RASA framework. The RL model and conversational agent model together resulted in the handheld assistive device called Augmented Guiding Torch (AGT). The AGT provides improved mobility over white cane by providing ambient awareness through natural conversation. It can inform visually impaired about the obstacles which are helpful to be warned about ahead of time, e.g., construction site, scooter, crowd, car, bike, or big hole. Using the RL framework, the robot avoided over 95% obstacles. The visually impaired avoided over 85% obstacles with the help of AGT in a 500 feet U-shape sidewalk.