Posted by Florence Ion
https://lifehacker.com/tech/geminis-nest-camera-reports-arent-helpful-yet?utm_medium=RSS
I may be home all the time, but I'm enjoying Nest's Gemini-aided summaries, even if they're not always on the mark. The new summaries let me easily glance at a notification to see if it's worth getting up for. They've become a metronome of my day, as I see that the local feral cat is on its rounds of mouse hunting in my backyard. And the daily Home Brief is good at summarizing the relative chaos coming in and out of my house.
I'm not even using the latest Nest hardware. I have a mix of first- and second-generation cameras, including two 2012-era indoor Nest cameras and a blend of 2021-era releases. It's been interesting to see how the Gemini infusion has revived the aging hardware, though it's still far from being the future-facing smart home tech it's been billed it to be—especially at this price. There's still work to be done before it becomes the trustworthy, contextual assistant Google wants it to be. Here's what to expect as Gemini becomes a core part of the Nest camera experience, and what you can do to make it work better for you.
Where Gemini excels with Nest
I've been actively testing Gemini in the new Google Home app since it launched last month. It's available after a significant, years-long overhaul of the app. However, most of the features discussed in this piece are only available with the Advanced tier of the Google Home Premium subscription, which costs $200 per year. That tier unlocks Gemini's Home Brief, the Ask Home video history search, detailed event descriptions and notifications, and 60 days of scrubbable event history. You can choose to pay only for the Google Home Premium Standard plan, which includes 30 days of event history and costs $100 per year, but it does not include the Home Brief and other AI-led features mentioned here.
Gemini's smart home features are an additional cost on top of using the regular Gemini chatbot, unless you pay for the Google One AI Pro tier (so many subscriptions). The summary and search capabilities have been time-savers when they work, though they've yet to justify their cost. Right now, the price is only worth it if you're after the colorful, detailed AI-generated notifications and 60-day event history.
You can ask Gemini to tell you more about the Home Brief and bring up specific events.
Credit: Florence Ion/Lifehacker
On both Android and iOS devices, the app has been consolidated into three main tabs: Home, Activity, and Automations. The Gemini summaries live in the Activity tab and are refreshed every morning with the lowdown of the day's prior events. You can filter notifications by device, if you're more interested in what the doorbell camera caught than in the one facing the backyard, or read all the summaries at once.
Summaries are generally a variation of the same thing each time. They're simple and succinct, often reading like dispatches of what's happening rather than alerts that a device has detected motion. Reports read like, "a person and a child walked by the front, followed by another person approaching the door." Package notifications will include whether FedEx or UPS is specifically making the drop, and if it can't figure it out, it simplifies it to "delivery person." In most cases, if it's properly labeled, Gemini will mention the camera that caught the action. "Two different people walked by the Front Door, and a cat was seen walking by the Side Door camera." If it identifies people it knows through Familiar Faces, the Home Brief's summaries can be even more dynamic, bordering on narrative. "Flo was seen interacting with a child, lifting them, and later sitting with them on the couch."
Gemini summaries can be very plain or they can be extremely colorful, like this one.
Credit: Florence Ion/Lifehacker
While it can get repetitive reading the same thing over and over again, that familiarity makes it possible to scroll through and find a standout event. Typically, I check the summary before I delve further. If there's something of note—"an unrecognized person approached the porch, looked at the camera, and shined a flashlight before leaving"—I tap on the Gemini icon to start a chat about the day's events. It'll pull up the camera clips associated with the devices I selected to review, along with events from that day, and then I can type or dictate my concerns.
A detailed, Gemini-led notification on the Pixel Watch 4.
Credit: Florence Ion/Lifehacker
There's no granular control over the Home Brief in its current implementation and you can't decide whether Gemini flourishes the narrative or keeps it simple. You can ask Gemini to focus on "certain things," broadly speaking, like whether you want it to ignore vehicles or animals. You cannot get more specific than that, like asking for it to hone in on cats over opossums. Push notifications are managed the standard route, through the Google Home app and each individual camera.
Customization for the Home Brief is currently very limited.
Credit: Florence Ion/Lifehacker
Gemini has a people problem
While Gemini excels at differentiating an opossum from a cat, and a dog from a raccoon, it's still tripping up on arguably one of the most important recognition tasks and a core part of the paid Google Home Premium features: identifying the humans who actually live here. Gemini isn't always aware of who is at the door or inside the house. It's only named my husband and me twice since I started testing everything. It's been pretty good at pointing out that my kid is a child, though a few times it's identified her as "children," plural, which creeped me out the first time I saw the notification. It's never referred to her by her name, though she's a registered Familiar Face. Even when Gemini does provide detail—"a person departing, followed shortly by a person and a child"—it fails to note that it's the same person walking in and out of the house in succession.
I've read through Reddit threads in the last month complaining of similar woes. A quick look at Google's support pages shows Familiar Faces wasn't always consistent, even before Gemini. We know that AI generally has a history of hallucinations, and there are even instances of Gemini's Nest camera summaries making stuff up. But it's even more obvious now that it's part of a daily summary. And while I appreciate the responsible way the AI generally refers to unknown people as just "persons," Gemini's failure to signal that it's the same face, one after the other, makes the Nest cameras feel more like overactive motion detectors.
The other problem is that the Gemini summaries' descriptions are not always correct. The Google Home activity summary brought up a person with a flashlight looking into the front doorbell camera. Naturally, I was struck by the description when I read it. After some scrubbing in the timeline and talking to my husband, it turns out we'd had a nighttime Amazon delivery—common now that daylight savings time is over where I live—and the person leaving the package was using a flashlight to ensure the address matched the house. While I don't expect Gemini to handle that level of detail quite yet, leaving out the delivery person's role only added to the confusion.
Gemini noticed a person approaching porch, but failed to mention it was a delivery person. And so, my heart skipped a beat that I'd missed something.
Credit: Florence Ion/Lifehacker
Perhaps most egregious of all, Gemini missed someone stealing all our candy on Halloween night. It didn't even alert me to the fact that people were in the frame. A little less than an hour before the incident, the first-gen Nest doorbell picked up my family and me returning from trick-or-treating, down to the colors of our costumes. But when two adults and a kid walked up to the edge of the walkway outside my front door, it didn't even register as an event. The Google Home app had only denoted it as a "sound."
I scanned the timeline many times before I saw the moment the candy theft occurred. The lack of an incident report tied to a clip made it hard to pinpoint exactly when it happened. Eventually, I took a screen recording of a several-minute video stream around the time the "sound" had been detected. That's when I saw one of the adults in the group stand in front of the candy bowl, I assume to hide the dumping of its entire contents into the bag. Were it not for the physical blockade, the first-generation Nest doorbell camera might have caught the whole thing. Still, it failed to tell me there were people outside my door for a prolonged period, which is the essential information that should have been summarized.
How to set up a Nest camera for success
While Gemini still requires significant refinement on Google's part, you can do a couple of things on your end to optimize the Nest cameras so that the AI summaries aren't consistently off base. The Gemini summaries rely on the quality of data the cameras collect, so refining the Familiar Faces library can help immensely. This helps improve Gemini's "confidence," so to speak, so it's not simply defaulting to "a person."
When I went in to curate Familiar Faces, I noticed that Nest has been bundling my face in with our sitter's, and my husband's face with our daughter's. It's neat that the AI can pick up on subtle similarities here, but it's not super helpful for revealing exactly who is at the door. If the camera mistakenly included the wrong picture, or even a blurry photo, you can delete it from the Familiar Faces library. While you're here, check if the camera has created multiple profiles for the same person. You can merge them to make the data more comprehensive and reduce Gemini's risk of faltering.
The best way to fix Gemini's faux pas is to edit you Familiar Faces library.
Credit: Florence Ion/Lifehacker
If you've got a Google Home full of aging Nest devices, ensuring the camera lenses are clean and the hardware itself is properly mounted can do wonders. Google support says doorbell cameras should be about four feet off the ground, while general cameras can be mounted six to eight feet. Most people are captured within 10 feet of the camera. You'll also want to pay attention to light and shadows. Pop in to learn what the cameras see at varying times of day. You're supposed to avoid placing the camera where the sun or bright external lights can backlight a face, but unfortunately, my front door faces west, and the sun loves to hover on that side for the better part of the afternoon.
Activity zones are also crucial to getting Gemini to tell you what's going on. If a camera faces a wide area with lots of irrelevant motion, like trees swaying in the wind, use Activity Zones to specify exactly where packages are dropped in the camera preview. The AI will skip over plants and focus more on the area highlighted. If I had this engaged to look off in the distance, maybe the doorbell camera would have caught the Halloween candy heist.
Gemini is still learning
While aging Nest hardware is getting a tiny boost in utility now that Gemini summaries are a core part of the experience, it still struggles with context. The Nest cameras are good at general motion detection, but Gemini struggles to distinguish people and determine when a human-driven event needs to be addressed.
I reached out to Google with the very specific case of my Halloween candy escapade to figure out what criteria were needed to get the AI to catch on. I received a response with tips on what I could do on my end to improve Gemini's chances of getting summaries. Unsurprisingly, I didn't get a more specific answer than that. The new Gemini infusion is in its nascent stages, which means it's still just the beginning of the road for it. Google wants you to send feedback so it can learn what to tweak to improve the integrated Gemini experience over time.
The Gemini-led Nest summaries in the Google Home app are a valuable tool for reducing notification fatigue and checking quickly on what's happening at the door. But until AI can reliably distinguish between a family member, a delivery person with the right intentions, and an unusual human action as a high-priority event, you'll need to continue to perform your own due diligence. That's not to say the foundation isn't there. But for now, Gemini is still learning.
https://lifehacker.com/tech/geminis-nest-camera-reports-arent-helpful-yet?utm_medium=RSS