Gemini's New Ability to Read Your Screen Heads to More Android Users

Gemini's on-screen awareness feature hits more users

While Gemini on the Pixel 9 and Pixel 9 Pro (review) comes with the "Ask about this screen" feature, this has only started rolling out to more Android users recently. It is spotted (via AndroidPolice) that the latest Google app beta enables a screen-aware feature in Gemini, letting users get insights on their phone's screen including YouTube videos via "Ask about this video."

"Ask about this screen" appears as a button above the Gemini floating window. Tapping it will have the Gemini assistant take a screenshot and analyze what's on your current screen. Subsequently, you can fine-tune your queries with text or voice prompts. It should also be noted that the screenshots generated are only temporary and not saved to your gallery.

This supports texts, images, and videos (YouTube) and can give you summaries, descriptions, and answers. When you're on YouTube, the function is generally the same but the button and label will change to "Ask about this video," which can be launched similarly to the text version.

Our colleague Antoine already tested the feature on the Pixel 9 Pro XL, which has shown to be capable of summarizing videos as long as these have captions or transcripts on them. Additionally, Gemini can also give you recipes based on the pictured ingredients.

It's unclear as to which devices are supported by this feature during the initial release, but it should be available to Google One AI subscribers.

Gemini can now better filter your emails

Separately, Google has also rolled out Gmail Q&A, which is a Gemini tool for the Gmail app for Android. It's available to users with any of the Google One AI premium subscriptions. It is planned to be added to the iOS version of the app soon, although the company didn't specify when.

As for Gemini Q&A, it comes as a new button positioned beside the main search bar at the top. By launching it, you can ask for specific actions like searching for specific items in your emails, summarizing emails, showing unread items, and more.

However as noted by Google, the initial release has only the search function present, though the company said they are going to add more capabilities in the coming weeks.

Remember that this is a different feature when composing messages or emails using Gemini, which is also another premium tool under the Google One AI plans.

How's your experience with Gemini so far? Which AI features have you been using the most? Let us know in the comments.

Via: Android Police Source: Google Workspace