At the recent Google I/O developer conference, Google DeepMind unveiled an ambitious integration of its Project Genie with Street View, tapping into over 280 billion images collected from across 110 countries. This advancement allows users to interact with and simulate real street environments in new ways.
The aim of this integration is to enhance familiar sights with dynamic elements. Picture adjusting the weather in these simulations or visualizing a snowy New York City block from miles away. Jack Parker-Holder, a research scientist on DeepMind’s open-endedness team, highlighted the potential of this technology for both robotics and human users, stating, “It’s really powerful for both the agent [and robotics] use case and for humans to play with.”
One practical example Parker-Holder shared involved a robot deployed in London, a city that often lacks sunlight. With Genie, the robot could be trained to recognize those rare sunny days by simulating sunlight glinting off Victorian homes, easing the transition to such conditions in reality.
This integration builds on a solid foundation of Street View data, which Google has been gathering for two decades through camera-equipped vehicles and individuals using tracker backpacks. This extensive imagery provides a rich backdrop for interactive simulations, merging real-world data with the creative capabilities of Project Genie.
Genie was first released in August 2022 as a research preview and became accessible to Google AI Ultra subscribers in January. The tool has already shown its utility in various fields, including gaming, education, and robotics training. Notably, it is currently powering simulations for Waymo’s self-driving cars, preparing them for rare events like tornadoes or unexpected wildlife encounters.
Waymo's existing simulator has already facilitated its expansion to 11 cities across the U.S., but adding Street View data could significantly enhance its training capabilities. Unlike traditional simulations that focus solely on a vehicle's perspective, Genie allows for a more versatile viewpoint, accommodating the experiences of different agents, whether they be vehicles, humans, or robots.
The implications of this integration go beyond mere visualization. It opens doors for new educational experiences and gaming environments that are uniquely grounded in reality. As Google continues to innovate with its AI infrastructure, merging Street View with Project Genie marks a significant step toward more immersive and interactive virtual worlds.
