perception · Seeing and hearing in the loop

The perception layer is the first interface between the entity and the real physical space. Without it, the entire system can only "passively wait for people to poke it" - the user must actively type or click buttons for interaction. With perception, the entity has the ability to "actively realize your presence" for the first time: you sit in front of the camera, it knows; you speak, it knows; you walk away, it also knows.

it runs on a Local resident daemon (agent runtime) The presence module consists of three sub-modules — face (face detection and following),vad (voice activity detection),probe (Presence detection and event aggregation). The upstream is the resident process itself, which continuously supplies camera frames and microphone streams; the downstream is the Cognition cognitive layer - when the perception module determines that "someone is present and starts to speak", it throws this event to the agent for decision-making, rather than brainless wake-up.

Therefore, the core value of this layer is not "how accurate the identification is", but "turning the presence signals of the physical world into events that can be consumed by the system with low enough disturbance and stable enough judgment".

In the vision of lasting digital life, entities must "first sense, then respond", rather than "exist after being activated." The paradigm of mainstream voice assistants is wake-word triggering: you must shout a wake-word before it opens its eyes. This paradigm is misplaced for "a being that is always present" - the thing that is truly present should be aware of it as soon as you enter the room, without the need for you to announce its name first.

What perception wants to solve is to use local visual + voice dual channels to turn the three things "Is there anyone/is it looking at me/is it talking" into signals that are continuous, low-latency, and can be subscribed by the agent. face Use OpenCV for face detection and let the line of sight/camera follow the movement of the face; vad Perform voice activity detection to distinguish between "environmental noise" and "really talking to me", only the latter triggers interaction; probe Aggregate multiple signals into a "presence state" for scheduling by the resident daemon.

If this layer is missing, the entity will always be a "tool waiting for instructions" rather than "notice your presence". This is a key step in pushing virtual humans from chat surface to ambient presence.

The entire layer is deliberately made to be "local first, lightweight first": perception is high-frequency and continuously running. Once each frame is transmitted to the cloud, there are not only privacy issues, but also latency and cost issues. Therefore, all visual and voice judgments are completed locally, and only the "refined events" are thrown upward.

Status: building. The skeletons of the three sub-modules are already running in the resident daemon, but they are still being polished from "can be detected" to "determined to be stable enough and not disturbing enough", and have not yet reached the level of daily dependence.