// cd ~
// 2026-05-29

The real moat isn't software

Josh Adler — The Real Moat Isn't Software

AI has a three-layer stack that almost nobody is thinking about correctly: Observation, Memory, Reasoning. Billions flow into Reasoning (Anthropic, OpenAI, Google, Meta, all competing to build smarter models). A growing wave of startups target Memory (vector databases, RAG pipelines, context managers). Almost zero investment goes into Observation, which is the layer that determines whether the other two have anything real to work with.

This is the investment gap that will define the next decade of AI. Not who builds the best model. Not who builds the best wrapper. Who gets AI out of the chat window and into the physical world.

The three-layer stack

Layer 1: Observation. Cameras, microphones, motion sensors, wearables, environmental capture. Raw data from the physical world, converted into formats that downstream systems can process.

Layer 2: Memory. Persistent, cross-session, intelligent storage and retrieval. Not a database with a search bar. An encoding system that decides what matters, lets stale data decay, and surfaces relevant context when the reasoning layer needs it.

Layer 3: Reasoning. The LLM. Claude, GPT, Gemini, whatever ships next quarter. The part that thinks.

Right now the reasoning layer is extraordinary. Claude can hold 1M tokens in context. GPT-5.5 can do multi-step planning. These models are genuinely good at thinking. But they're thinking about almost nothing, because the input layer feeding them is a text box. Your AI knows what you type into it and nothing else. Every preference, every correction, every piece of behavioral context came through a chat interface during a conversation you chose to have about a topic you remembered to bring up.

That input method captures maybe 1% of the information that would make AI actually useful to a specific person. The other 99% is sitting in the physical world where no AI system has ever been.

What the observation layer actually requires

Building observation infrastructure is not a software problem and that's precisely why most AI companies avoid it. Software can be replicated in a weekend. Someone reads your paper, understands the approach, ships their own version. I've watched it happen with RAG improvements and encoding strategies and retrieval architectures. Code is not a moat.

Hardware deployment is a different animal. Here's what the observation layer looks like in practice:

Sensor selection. Camera modules vary wildly in power draw, thermal behavior, driver stability, and image quality. You have to pick sensors that actually survive continuous operation on low-power compute boards, which eliminates most of the impressive-looking options immediately.

Compute platform. Needs to be cheap enough to deploy at scale (one per room minimum), small enough to mount unobtrusively, powerful enough to run local motion detection and audio triggering without shipping every frame to a central server.

Data pipeline. Raw frames to motion-triggered recording to vision model inference to structured text to memory encoding to retrieval. Six steps, each introducing noise and latency. The pipeline between "camera sees something" and "LLM can use that information" is where most of the engineering complexity lives and it's the part nobody talks about.

Network and storage. Continuous multi-camera recording generates enormous data volumes even with motion triggering. You need local NAS infrastructure, cleanup pipelines, retention policies. This is infrastructure work, not application development.

Inference. The captured visual and audio data needs to be processed through vision models and speech-to-text before it becomes LLM-usable context. That requires GPU compute, which means either cloud costs per frame or local GPU hardware.

Each of these components introduces failure modes that software engineers never encounter building chat wrappers. Loose ribbon cables. Kernel version mismatches between identical SD card images. Power supplies that can't sustain current draw under camera load. Driver conflicts that only manifest after 20 minutes of continuous operation. The debugging process is physical, iterative, slow, and it teaches you things no amount of reading documentation will prepare you for.

What I built

I deployed a five-node sensor network called Paradox across my apartment. The hardware per node:

Total hardware cost: under $500 for the full deployment.

Each node runs a custom Python daemon that handles motion-triggered and audio-triggered recording. Primary stream captures at 1280x720 at 15fps. A separate low-resolution 320x240 stream runs continuous motion detection without taxing the main capture pipeline. When motion or audio crosses a threshold, the daemon switches from idle monitoring to full recording in MJPEG/H.264 for video and WAV for audio.

Storage ships to a NAS on the local network. Inference runs on an RTX 5090 with 32GB VRAM, processing captured footage through vision models and generating structured behavioral observations.

That description makes it sound clean. It was not clean.

The honest engineering constraints

Thermal throttling. The Pi Zero 2W draws about 1.5W idle but spikes to nearly 4W under camera load. In an enclosed case, the SoC hits 80C within twenty minutes and starts thermal throttling, which means dropped frames, recording gaps, and eventual lockups. You either run the boards without cases (ugly, fragile, collects dust) or design cases with active ventilation (adds cost, complexity, and noise). There's no good answer. I tried heat sinks, copper shims, custom 3D-printed enclosures with ventilation slots. Every solution involves a tradeoff.

Driver instability. I started with OwlSight 64MP sensors running the ov64a40 driver. On paper they looked incredible. In practice the driver required a custom dtoverlay configuration with a specific link-frequency parameter that behaved differently across kernel versions. I spent entire nights staring at dmesg output trying to figure out why a camera would initialize on one node but not another running the identical SD card image. The answer was always something mundane: a ribbon cable seated 0.5mm off, a kernel micro-version difference, a power supply marginally below spec.

I eventually ripped all five ov64a40 sensors out and migrated to the IMX708. Less impressive on paper (12MP versus 64MP), dramatically more stable in continuous operation. The dtoverlay went from a multi-parameter nightmare to a single imx708 declaration. That migration took a week, including re-running all five nodes, re-calibrating motion detection thresholds, and re-validating the recording pipeline end to end.

Data volume. Five cameras at 15fps with motion triggering still generates a genuinely absurd amount of data. The NAS fills faster than you'd expect. I built a cleanup pipeline with configurable retention windows just to keep storage from overflowing, and even so it fills up fast if I don't tune the motion sensitivity thresholds to avoid false triggers from lighting changes and shadows.

The inference chain. Raw video frames are useless to an LLM. The pipeline from captured frame to LLM-usable context requires: motion detection (local, on the Pi), frame selection (choosing which frames actually contain meaningful activity), vision model inference (running selected frames through a model that can describe what's happening), text structuring (converting free-form descriptions into consistent structured data), and memory encoding (deciding what's worth storing long-term versus what's noise). Each step adds latency and introduces potential errors. A shadow triggers motion detection, a blurry frame gets selected, the vision model misidentifies an object, the structured output contains a hallucinated detail. These errors compound through the pipeline.

Social constraints. My girlfriend didn't speak to me for two days after I installed cameras in the apartment. "I'm building an observation layer for my AI memory system" is not a sentence that makes someone comfortable in their own home. We worked it out with zones and schedules, rooms where the system doesn't run and times when it goes dark. But social acceptability is a constraint as hard as any thermal limit, and you can't engineer around it the way you can engineer around a driver conflict.

These are not theoretical problems. They are the daily reality of building observation infrastructure and they are why almost nobody is doing it.

Why hardware moats beat software moats

Software moats are measured in weeks. Someone reverse-engineers your approach, implements it differently, ships. Open-source accelerates this. A novel retrieval algorithm published in a paper becomes six competing implementations within a month.

Hardware moats are measured in months to years. The physical deployment, the sensor calibration, the iterative debugging of failure modes that only appear in production, the institutional knowledge of which components actually survive continuous operation versus which ones look good in spec sheets. You can't replicate that by reading a GitHub repo.

More importantly, observation data is unique. My five-node deployment generates behavioral data that literally does not exist anywhere else. No other system has watched me work, measured my actual sleep patterns (lights off to lights on, not what I told a sleep tracker app), tracked how long I actually sit at my desk versus how long I think I sit there. That data, combined with intelligent memory, creates a feedback loop that pure software systems cannot replicate because they don't have the input.

The gap between self-reported behavior and observed behavior is enormous. People say they work out four times a week when they go twice. They describe themselves as morning people while consistently opening their laptops at noon. Chat-based AI only gets the self-reported version. Observation infrastructure gets the truth, or at least something much closer to it.

Memory as the connecting layer

Observation without intelligent memory is just surveillance footage. Terabytes of video collecting dust on a NAS. The observation layer generates raw data. Something needs to decide what matters, encode it persistently, let irrelevant details decay, and surface the right context when reasoning needs it.

TrueMemory is the system I built to solve this. The architecture implements a computational encoding gate modeled on biological memory: incoming data gets scored across novelty, salience, and prediction error, and only observations that pass the threshold get encoded. Low-scoring observations get discarded, not archived. Because storing everything makes retrieval worse, not better. Every stored memory competes with every other stored memory during search, and noise drowns signal fast.

The full architecture is described in my arXiv paper. The key insight is that memory is not a storage problem. It's an attention problem. What you choose not to remember is as important as what you keep.

TrueMemory sits between the observation layer and the reasoning layer, turning raw behavioral data and conversational context into persistent, retrievable, intelligent memory. Without it, the observation data is just files. With it, the reasoning layer has actual ground truth about a specific person instead of whatever they remembered to type into a prompt.

Where this matters

Consider what happens when all three layers connect. The observation layer notices you haven't left your desk in six hours. The memory layer cross-references this with your calendar showing three back-to-back meetings and recalls that this pattern preceded you getting sick last month. The reasoning layer synthesizes this into a proactive suggestion, unprompted, because the system was in the room with you and remembered what happened last time.

That's not a chatbot anymore. That's what AI assistance was always supposed to look like, and it requires all three layers working together.

Nobody is going to win this by building a better chat interface. The chat interface is a temporary artifact of not having figured out how to get AI into the room. The companies that solve unobtrusive observation and connect it through intelligent memory to powerful reasoning will define the category. Everyone else will be fighting over which wrapper is 3% better at summarizing chat transcripts.

The hard part was never the software. The hard part is getting AI into the physical world and building the memory to make that data useful. That's a hardware problem, a pipeline problem, and a social problem all at once, and it's why I'm debugging thermal throttling at 2am instead of writing another RAG tutorial.

Josh Adler is a researcher at TrueMemory, a Sauron company. Research: arXiv:2605.04897. More at joshadler.com.