cross-posted from: https://lemmit.online/post/225981

This is an automated archive made by the Lemmit Bot.

The original was posted on /r/homeassistant by /u/janostrowka on 2023-07-19 12:49:02.

Hopefully this will come in handy for our Year of the Voice.

TL;DR: Justin Alvey replaces Google Nest Mini PCB with ESP32 custom PCB which heā€™s open-sourcing. Shows demo of running LLM voice assistant paired with Beeper to send and receive messages.

Tweet text thread (I would also highly recommend checking out the video demos on Twitter):

I ā€œjailbrokeā€ a Google Nest Mini so that you can run your own LLMā€™s, agents and voice models. Hereā€™s a demo using it to manage all my messages (with help from @onbeeper) šŸ“· on, and wait for surprise guest! I thought hard about how to best tackle this and why

After looking into jailbreaking options, I opted to completely replace the PCB. This letā€™s you use a cheap ($2) but powerful & developer friendly WiFi chip with a highly capable audio framework. This allows a paradigm of multiple cheap edge devices for audio & voice detectionā€¦

& offloading large models to a more powerful local device (whether your M2 Mac, PC server w/ GPU or even ā€œtinyboxā€!) In most cases this device is already trusted with your credentials and data so you donā€™t have to hand these off to some cloud & data need never leave your home

The custom PCB uses @EspressifSystemā€™s ESP32-S3 I went through 2 revisions from a module to a SoC package with extra flash, simplifying to single-sided SMT (< $10 BOM) All features such as LEDā€™s, capacitive touch, mute switch are working, & even programmable from Arduino (/IDF)

For this demo I used a custom ā€œMaubotā€ with my @onbeeper credentials (a messaging app which securely bridges your messaging clients using the Matrix protocol & e2e encryption) which runs locally serving an API

Iā€™m then using GPT3.5 (for speed) with function calling to query this

Fro the prompt I added details such as family & friends, current date, notification preferences & a list additional character voices that GPT can respond in. The response is then parsed and sent to @elevenlabsio

Iā€™ve been experimenting with multiple of these, announcing important messages as they come in, morning briefings, noting down ideas and memos, and browsing agents. I couldnā€™t resist - hereā€™s a playful (unscripted!) video of two talking to each other prompted to be AIā€™s from "Her

Iā€™m working on open sourcing the PCB design, build instructions, firmware, bot & server code - expect something in the next week or so. If you donā€™t want to source Nest Miniā€™s (or shells from AliExpress) itā€™s still a great dev platform for developing an assistant! Stay tuned!

  • phx@lemmy.ca
    link
    fedilink
    English
    arrow-up
    24
    arrow-down
    1
    Ā·
    1 year ago

    Itā€™s not a jailbreak itā€™s a chip/board swap.A jailbreak would let you run your own code in the original firmware/os at an administrative privilege level.

    Still cool but Iā€™d be even happier with a proper jailbreak or firmware overwrite that lets you use the original guts with your own software.

    • Deez@lemm.eeOP
      link
      fedilink
      English
      arrow-up
      2
      Ā·
      1 year ago

      Agreed, I put it in quotes, and listed the PCB swap in the title for that reason. I probably should have used single quotes.

    • Deez@lemm.eeOP
      link
      fedilink
      English
      arrow-up
      3
      Ā·
      1 year ago

      Yeah Iā€™m super keen to gut some old minis. Hopefully it works with the Google Home Minis as well.

    • lhamil64@programming.dev
      link
      fedilink
      English
      arrow-up
      1
      Ā·
      1 year ago

      Yeah, I have a Nest Mini that randomly died and is now just sitting in a box. This might be the perfect project, assuming whateverā€™s broken would get swapped out.

  • iMeddles@infosec.pub
    link
    fedilink
    English
    arrow-up
    5
    Ā·
    1 year ago

    My aim for the year of voice is to replace my google minis with something that works locally with ha, if this gets integrated that way its gonna save me reasonable amounts of money on speakers :D

    • RegalPotoo@lemmy.world
      link
      fedilink
      English
      arrow-up
      6
      Ā·
      1 year ago

      Same, but lack of an open source Cast receiver is going to make that a hard sell for me. I hate that itā€™s anticompetitive proprietary bullshit, but it works, and works really well.

  • musictechgeek@lemdit.com
    link
    fedilink
    English
    arrow-up
    4
    Ā·
    1 year ago

    Iā€™m happy(ish) with my current setup, but Iā€™d really love eventually having the option to replace my Echo and Home devicesā€™ innards with something local and open-source.

  • Jeearr@lemmy.world
    link
    fedilink
    English
    arrow-up
    1
    Ā·
    1 year ago

    Is it just me or is Google Home devices getting worse and worse? The voice recognition, the responses, etc etc.

    Someone should definitely make a video on this. Would love to eye it out for confidence.

    • Deez@lemm.eeOP
      link
      fedilink
      English
      arrow-up
      1
      Ā·
      1 year ago

      When I first bought the speakers/displays they seemed to be releasing software updates at a quick pace, but it seems like they have slowed to a crawl. Classic Google to abandon products to move onto the next shiny thing.

      Over the last year or so there was big cuts to the Google Assistant teams, and then they merged the Assistant and Bard teams together.

      Personally, Iā€™ve recently had issues with one of mine not turning off the display at night. I was also really disappointed when they stopped integrating with third party shopping list apps. My devices are definitely less useful now.