llama-conductor
codeberg.org
Route workflows, not models. Glass-box, not black-box. Squash LLM nonsense.

Hello!

As a handsome local AI enjoyer™ you’ve probably noticed one of the big flaws with LLMs:

It lies. Confidently. ALL THE TIME.

(Technically, it “bullshits” - https://link.springer.com/article/10.1007/s10676-024-09775-5

I’m autistic and extremely allergic to vibes-based tooling, so … I built a thing. Maybe it’s useful to you too.

The thing: llama-conductor

llama-conductor is a router that sits between your frontend (OWUI / SillyTavern / LibreChat / etc) and your backend (llama.cpp + llama-swap, or any OpenAI-compatible endpoint). Local-first (because fuck big AI), but it should talk to anything OpenAI-compatible if you point it there (note: experimental so YMMV).

Not a model, not a UI, not magic voodoo.

A glass-box that makes the stack behave like a deterministic system, instead of a drunk telling a story about the fish that got away.

TL;DR: “In God we trust. All others must bring data.”

Three examples:

1) KB mechanics that don’t suck (1990s engineering: markdown, JSON, checksums)

You keep “knowledge” as dumb folders on disk. Drop docs (.txt, .md, .pdf) in them. Then:

  • >>attach <kb> — attaches a KB folder
  • >>summ new — generates SUMM_*.md files with SHA-256 provenance baked in
  • `>> moves the original to a sub-folder

Now, when you ask something like:

“yo, what did the Commodore C64 retail for in 1982?”

…it answers from the attached KBs only. If the fact isn’t there, it tells you - explicitly - instead of winging it. Eg:

The provided facts state the Commodore 64 launched at $595 and was reduced to $250, but do not specify a 1982 retail price. The Amiga’s pricing and timeline are also not detailed in the given facts.

Missing information includes the exact 1982 retail price for Commodore’s product line and which specific model(s) were sold then. The answer assumes the C64 is the intended product but cannot confirm this from the facts.

Confidence: medium | Source: Mixed

No vibes. No “well probably…”. Just: here’s what’s in your docs, here’s what’s missing, don’t GIGO yourself into stupid.

And when you’re happy with your summaries, you can:

  • >>move to vault — promote those SUMMs into Qdrant for the heavy mode.

2) Mentats: proof-or-refusal mode (Vault-only)

Mentats is the “deep think” pipeline against your curated sources. It’s enforced isolation:

  • no chat history
  • no filesystem KBs
  • no Vodka
  • Vault-only grounding (Qdrant)

It runs triple-pass (thinker → critic → thinker). It’s slow on purpose. You can audit it. And if the Vault has nothing relevant? It refuses and tells you to go pound sand:

FINAL_ANSWER:
The provided facts do not contain information about the Acorn computer or its 1995 sale price.

Sources: Vault
FACTS_USED: NONE
[ZARDOZ HATH SPOKEN]

Also yes, it writes a mentats_debug.log, because of course it does. Go look at it any time you want.

The flow is basically: Attach KBs → SUMM → Move to Vault → Mentats. No mystery meat. No “trust me bro, embeddings.”

3) Vodka: deterministic memory on a potato budget

Local LLMs have two classic problems: goldfish memory + context bloat that murders your VRAM.

Vodka fixes both without extra model compute. (Yes, I used the power of JSON files to hack the planet instead of buying more VRAM from NVIDIA).

  • !! stores facts verbatim (JSON on disk)
  • ?? recalls them verbatim (TTL + touch limits so memory doesn’t become landfill)
  • CTC (Cut The Crap) hard-caps context (last N messages + char cap) so you don’t get VRAM spikes after 400 messages

So instead of:

“Remember my server is 203.0.113.42” → “Got it!” → [100 msgs later] → “127.0.0.1 🥰”

you get:

!! my server is 203.0.113.42 ?? server ip → 203.0.113.42 (with TTL/touch metadata)

And because context stays bounded: stable KV cache, stable speed, your potato PC stops crying.

There’s more (a lot more) in the README, but I’ve already over-autism’ed this post.

TL;DR:

If you want your local LLM to shut up when it doesn’t know and show receipts when it does, come poke it:

PS: Sorry about the AI slop image. I can’t draw for shit.

PPS: A human with ASD wrote this using Notepad++. If it the formatting is weird, now you know why.

  • SuspciousCarrot78@lemmy.worldOP
    21·
    4 hours ago

    Fuck, I’m an all seeing, prescient wizard! Neat!

    Uh, I mean… “yes of course, I intended to have Ai Horde piping baked in from the get go”.

    Do let me know if my shit works with their shit to make…not shit.

    #####################

    EDIT: For fun, I asked my router to ask Qwen3-4B 2507 WTF Ai Horde was. I’m showing this for interest sake, because 1) shows you how the thing works IRL 2) shows you how and when it’s WRONG in a LOUD way, if you know how to read it.

    ######################

    This is in default mode, no KBs attached. Just straight YOLOing the question at it, cold, and asking it to argue from it’s own internal priors.

    AI-horde is a decentralized platform where users rent or sell computational power (GPU time) to run AI models. It connects AI model developers who need processing with individuals or organizations who have idle GPU hardware. The platform uses blockchain-like tokens to manage payments and incentives. It’s not a single AI system but a marketplace for GPU resources.

    Missing details: Specific token mechanics, current model library, or user onboarding flow could affect accuracy. For example, if token economics changed recently, the explanation might not reflect current conditions.

    Confidence: Medium | Source: Model

    #####################################

    That’s the routers way of saying “hey, best guess. Sorry”. And from what I can see, it sort of right, but specifics are wrong. So, ok for a cheap YOLO, not for “I bet my money on this” Let’s call this a 4/10

    ####################################

    Now, I gave it web access privileges. It did this. Let’s call this 7/10

    An AI-horde is a free, open-source community service that allows anyone to contribute computing power to generate AI-generated images and text. It functions as a crowdsourced distributed cluster of workers focused on image and text generation tasks [1]. This model resembles collaborative projects like Folding@home, where users share computational resources for a common goal [2].

    ####################################

    Finally, here’s what it does in >>KB mode. I scraped the FAQ and then did >>summ new. Eyeballing it against the FAQ, it’s call this 8.5/10. YMMV

    AI-horde is a decentralized platform for distributed AI computing. It allows users to rent out their idle GPU hardware to run AI models, connecting developers needing processing power with volunteers who provide it through open APIs. The system uses a non-cryptocurrency-based incentive called Kudos to prioritize contributions, where workers earn Kudos for completing tasks, and these are not tradable assets but rather internal metrics of service quality and project loyalty.

    Access to AI-horde is managed via API keys; anonymous usage is allowed but receives the lowest priority. The project emphasizes volunteerism and discourages commercialization of Kudos, treating its sale or exchange for money or cryptocurrency as contrary to its ethos. Participation as a worker involves installing the AI-horde client, selecting task types (e.g., image or text generation), configuring supported models, and running the worker to begin earning Kudos.

    Confidence: high | Source: Docs

    ################################

    FINALLY, for “fun”, I >>kb attached all and asked it to reason over the above three responses, in context of what it has in the KB store, and rank them out of 10.

    [3] is most correct, scoring 9.5/10. It accurately reflects the core model: decentralized GPU renting via open APIs, Kudos as a non-crypto, non-tradable incentive tied to volunteerism, and the crowdsourced nature of contributions. It also correctly notes access via API keys and lower priority for anonymous use.

    [2] is accurate but less complete, scoring 7.5/10. It correctly identifies the crowdsourced, open-source, volunteer-based nature and analogy to Folding@home. However, it omits key specifics like Kudos, API key access, and the explicit non-commercialization stance.

    [1] is partially correct but misleading, scoring 6/10. It correctly describes GPU

    Confidence: medium | Source: Docs

    TL;DR

    • KB mode: source-based info and when queried against, adjudication accuracy = v.good
    • Clean, reproducible demo. Try it yourself
    • db0@lemmy.dbzer0.com
      1·
      4 hours ago

      Very impressive. The only mistake on the third one is that the kudos are actually transferrable (i.e. “tradable”), but we forbid exchanges for monetary rewards.

      Disclaimer: I’m the lead developer for the AI Horde. I also like you’ve achieved here and would be interesting if we can promote this usage via the AI Horde in some way. If you can think of some integration or collaboration we could do, hit me up!

      PS: While the OpenAI API is technically working, we still prefer people to use our own API as it’s much more powerful (allowing people to use multiple models, filter workers, tweak more vars) and so on. If you would support our native API, I’d be happy to add a link to your software in our frontpage in the integrations area for LLMs.

      • SuspciousCarrot78@lemmy.worldOP
        1·
        4 hours ago

        Oh shit! Uh…thank you! Umm. Yes. That was unexpected :)

        Re: collab. I’m away for a bit with work, but let me think on it for a bit? There’s got to be a way to make this useful to more peeps.

        Believe it or not, I am not a CS guy at ALL (I work in health-care) and I made this for fun, in a cave, with a box of scraps.

        I’m not good at CS. I just have a … “very special” brain. As in, I designed this thing from first principles using invariants, which I understand now is not typical CS practice.