Researchers from Texas University, based in Austin, have recently unveiled a groundbreaking AI model that has the ability to convert sounds into visual images of streets. The study, published in a research article, highlights how machines can mimic the connection between auditory stimuli and visual perception, similar to human beings.
The AI model was trained using a vast dataset that included audio and visual materials from various city and rural landscapes. Following the training, the AI successfully generated images based on audio recordings. The researchers noted that the acoustic characteristics of different locations contain valuable visual cues that help in creating recognizable images.
During the training process, 10-second audio clips and corresponding images from YouTube videos recorded in cities across North America, Asia, and Europe were utilized. Post-training, the researchers conducted tests to compare the generated images with real photos. The evaluation involved analyzing the proportions of greenery, buildings, and sky, as well as human participation to select the correct image matching the sound sample, resulting in an 80% accuracy rate.
The AI model demonstrated a strong correlation between the proportions of sky and greenery in the generated images and real photos, effectively capturing architectural styles and spatial distances. It also accounted for lighting variations, enabling the distinction between day and night sounds associated with transportation noise or nocturnal nature sounds.
The researchers highlighted that, historically, the ability to visualize scenes based on sounds was a uniquely human trait, but advancements in technology have enabled AI to replicate this capability. They believe that such developments could enhance our understanding of human-environment interactions.
Besides sound conversion, the scientists are exploring the potential of AI in studying the distinctive characteristics of cities. This research underscores the significance of multisensory elements in spatial perception and introduces new possibilities for utilizing geospatial AI in urban planning and sociology.