Building a Family‑First Memory System for AI Agents: Lessons
Key takeaways
- Mwe‑MCP introduces per‑paragraph ACLs, allowing fine‑grained privacy control in AI memory.
- Storing knowledge as wikis with linked namespaces enables both personal and shared contexts.
- An agent‑agnostic memory server simplifies integration of new language models and ensures consistency across assistants.
- Deploying Mwe‑MCP is straightforward with Docker, and a tiny HTTP API keeps the system lightweight.
- Auditing and versioning are critical for maintaining trust and correcting mistakes in a household memory system.
When I first started experimenting with AI agents for my smart home, I quickly realized that the existing memory solutions were built for personal assistants—think “my calendar, my contacts, my preferences.” What I wanted was something that could act like a family member, respecting the privacy boundaries of each person in the house while still sharing the information that truly belongs to everyone.
The first attempts were disappointing. OpenClaw’s built‑in memory felt like a monolithic notebook that anyone could read. I tried dozens of third‑party plugins, but each one either exposed everything to every agent or forced me to write custom filters that quickly became a nightmare to maintain.
So, like any good maker, I decided to build my own.
---
From a Simple Plugin to an Agent‑Agnostic MCP Server
The project began as a plugin for OpenClaw, a popular open‑source AI framework. The goal was modest: attach a lightweight store to the assistant that could keep track of household facts—who bought milk, which movies the kids have already watched, the Wi‑Fi password, etc.
Over the course of a year, that plugin grew into Mwe‑MCP (Memory Control Plane), a standalone server that any AI agent can query via a tiny HTTP API. It now powers a home of four people and three different agents, all reading and writing to the same knowledge base without stepping on each other’s toes.
---
The Core Innovation: Inline ACLs on Wiki Prose
The breakthrough that separates Mwe‑MCP from other memory stores is the inline Access Control List (ACL) attached to every paragraph, list item, or even a single sentence. Inspired by the redacted documents that surfaced during the Epstein investigations, I realized that privacy can be enforced at the granular level of the text itself.
How It Works 1. **Capture** – When a user speaks to an agent, the utterance is immediately parsed into a small wiki page. Each logical chunk receives a default ACL based on the speaker’s identity. 2. **Ingest** – Documents, images, or PDFs uploaded to the system are automatically split into sections, each inheriting an ACL derived from the uploader or from metadata embedded in the file. 3. **Serve** – Before any agent reads a page, the server **redacts** everything the requesting identity is not permitted to see. The agent never even sees the hidden text, eliminating the risk of accidental leakage.
Real‑World Example - **Parent says:** “We’re out of almond milk, can you add it to the shopping list?” - ACL: `family` (visible to all household members and agents). - **Child asks:** “What did Mom say about the weekend?” - The child’s request is filtered; the response only includes items with the `family` ACL, omitting any private parental conversations.
---
Wiki‑Style Organization for Everyone
Memory in Mwe‑MCP is stored as wikis. Each user, group, and agent gets its own namespace, but wikis can link to each other. This design mirrors how families actually share knowledge:
- Personal wikis – Private journals, health data, or personal to‑do lists. - Group wikis – The family shopping list, shared calendar, or home maintenance log. - Agent wikis – Context that a specific assistant needs, such as device status or conversation history.
Because the ACL is embedded in the prose, the same page can be rendered differently for each viewer. A parent sees the full recipe with notes about allergies, while a child sees only the steps that are safe for them.
---
Benefits of an Agent‑Agnostic Architecture
1. Future‑Proofing – As new models (GPT‑5, Claude‑3, etc.) become available, you can plug them in without rewriting memory logic. 2. Consistency – All agents read from the same source, so there’s no “I know X but you don’t” confusion. 3. Security – Centralized ACL enforcement means you only need to audit one place.
---
Lessons Learned and Best Practices
| Lesson | Recommendation |
|--------|----------------|
| Granular ACLs are essential | Do not rely on coarse‑grained permissions; embed ACLs at the sentence level.
| Keep the API tiny | A simple GET /page, POST /page, PATCH /acl interface reduces bugs and makes it easy to support any language model.
| Version your wikis | Household knowledge evolves; versioning lets you roll back when a mistake is made.
| Audit logs matter | Store who read or wrote each fragment; this is invaluable for debugging privacy incidents.
| Test with real users | Simulate family conversations early—what feels private to a parent may be obvious to a teenager.
---
Getting Started with Mwe‑MCP
1. Deploy – The project ships as a Docker image. Run it on a modest Raspberry Pi or a home server.
2. Configure ACLs – Define users (alice, bob, charlie) and groups (family, guests).
3. Connect an Agent – Use the provided Python client to point your OpenClaw or LangChain bot at the MCP endpoint.
4. Populate – Start speaking! The system will auto‑create pages for each interaction.
5. Monitor – The web UI shows redacted views per user, helping you verify that privacy rules work as intended.
---
Looking Ahead
Mwe‑MCP is still a hobby project, but the concepts it demonstrates are ready for broader adoption. Imagine a community‑wide memory network where neighborhoods share resources (tool libraries, local events) while still protecting personal data through inline ACLs. The same architecture could power corporate knowledge bases, educational platforms, or even decentralized social media.
If you’re building AI agents that need to live alongside humans, consider moving beyond “personal assistant” and toward family‑first memory—a system that knows who may know what and respects those boundaries automatically.
---
Happy building, and may your agents always remember the right things for the right people.
Sources: https://github.com/Fr4nZ82/mwe-mcp