TorresVault Documentation

You've loaded an old revision of the document! If you save it, you will create a new version with this data.

===== Jarvis AI System (TorresVault 2.0) =====

The **Jarvis AI System** is the next major evolution of the TorresVault ecosystem — a fully local, private, high-performance AI platform built on modern GPU hardware, integrated deeply with Home Assistant, the Proxmox cluster, the UniFi network, and smart-home automations.

This page outlines the **theoretical architecture** for Jarvis under the TorresVault v2 upgrade plan.

----

==== 🏗️ System Architecture Overview ====

Jarvis is designed as a **local-first AI compute node** with the following goals:

* Zero dependency on cloud services  
  * Near-instant local processing  
  * High availability as part of the wider TorresVault ecosystem  
  * Tight integration with HA Voice, Nabu Casa, BLE tracking, and dashboards  
  * Scalable model hosting for multiple LLMs and vision models

----

=== Hardware Overview ===
Jarvis AI Box (theoretical build):

* **CPU:** AMD Ryzen (preferred for efficiency + PCIe lanes)  
  * **GPU:** NVIDIA RTX 3060 (12GB VRAM)  
  * **RAM:** 32–64GB  
  * **Storage:** NVMe SSD (OS + models), optional SATA SSD for app data  
  * **Networking:** Dual 2.5Gb or 10Gb depending on minisforum/mini-PC  
  * **OS:** Ubuntu Server or Debian  
  * **Containerization:** Docker or Podman  
  * **Base Services:** vLLM or Ollama, Whisper, Piper, FastAPI endpoints

Jarvis exists as its own node separate from PVE1/PVE2 to maintain **failure domain isolation**, consistent with your lab philosophy.

----

=== Architecture Diagram ===

┌──────────────────────────┐
           │     Jarvis AI Server     │
           │  (RTX 3060 • AMD Ryzen)  │
           └───────────┬──────────────┘
                       │
    ┌──────────────────┼──────────────────┐
    │                  │                  │

┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ vLLM / │ │ Whisper STT │ │ Piper TTS │
│ Ollama API │ │ (Speech→Txt)│ │ (Txt→Speech) │
└──────┬────────┘ └──────┬───────┘ └──────┬────────┘
│ │ │
└──────────────┬────┴─────┬────────────┘
│ │
┌───────────▼───┐ ┌──▼──────────────┐
│ Home Assistant │ │ n8n │
│ (Voice / BLE │ │ Automations Hub │
│ Automations │ └─────────────────┘
└───────┬────────┘
│
┌────────────┼──────────────────┐
│ │ │
┌─────────┐ ┌──────────────┐ ┌──────────────┐
│ UniFi │ │ Proxmox PVE1 │ │ Proxmox PVE2 │
│Network │ │ VM signals │ │ VM signals │
└─────────┘ └──────────────┘ └──────────────┘

----

==== 🔊 Speech Pipeline (STT/TTS) ====

Jarvis handles speech the following way:

=== 1. **Speech Input (STT) – Whisper** ===
All voice commands enter through:

* **Home Assistant Voice Assistants** (1 per room eventually)
  * The **car BLE + TTS node** for mobile interactions
  * n8n request endpoints  
  * Optional microphone endpoints on Pi-based sensors

Whisper STT pipeline:

* Local Whisper server running large-v3 or medium-int8  
  * Audio cleaned → transcribed → command classification  
  * Routed to HA or the LLM depending on context

=== 2. **Speech Output (TTS) – Piper** ===
Home Assistant Voice-ready.

Used for:

* Spoken responses from Jarvis
  * Car TTS notifications (weekday/time logic)
  * Room announcements (office, kitchen, hallway)

Languages/voices:

* en-us (rhasspy onnx baseline)
  * Optional emotional / expressive voices later

----

==== ⚙️ n8n Automations Layer ====

n8n acts as the **automation router**:

* Receives events from HA (motion, BLE, network events)
  * Sends structured workflows to Jarvis (LLM queries, text cleaning)
  * Sends TTS announcements back via HA/Piper
  * Handles long-running processes that HA shouldn’t manage

Example workflows:

* “Item Left Behind” (BLE + car tracker → TTS + WLED warning)
  * “Client Count Drop” → notify Jarvis → escalate warning
  * “Network health degradation” → log into Grafana via webhook
  * “Daily home-health summary”

----

==== 🧩 Integration With Home Assistant ====

Jarvis integrates deeply into HA:

=== Inputs from HA → Jarvis ===
  * microphone audio → Whisper  
  * room presence (BLE/EP1/Atom Lite/Pi Zero)  
  * sensor events (motion, lights, AP offline, doors)  
  * network events (UniFi client counts, WAN latency)  
  * FPP show status  
  * System notifications (VM backups, node failures)

=== Outputs from Jarvis → HA ===
  * TTS responses (Piper)  
  * Script execution  
  * Automations (scene changes, WLED preset triggers)  
  * Dashboard chips/cards (dynamic status indicators)
  * FPP commands (start playlist, stop, status polling)

----

==== 🌐 Integration With Proxmox & Servers ====

Jarvis is not a hypervisor — it *observes and automates*.

Planned integrations:

* Monitoring PVE node status via API  
  * Reading VM load / uptime / alerts  
  * Suggesting workload migrations  
  * Triggering safe shutdown ops if NUT signals UPS failure  
  * Reading Prometheus/Grafana alerts  
  * Providing natural-language summaries of cluster health

Example:

* “Jarvis, is my storage node overloaded?”  
  * “Jarvis, summarize the last 24h of Proxmox warnings.”

----

==== 📡 Integration with the Network (UniFi + Pi-hole) ====

Jarvis can interface with:

=== UniFi ===
  * Client analytics (AP client drops)
  * Device heartbeat checks
  * WAN latency monitoring
  * Firewall rule toggles (matrix, mega-tree, VPN-to-TX)
  * SSID toggles

=== Pi-hole ===
  * DNS lookup logs  
  * Query summaries  
  * Blocklist additions  
  * Warn if VIP 192.168.1.5 becomes unreachable

This creates a unified “AI-aware network”.

----

==== 🚗 Car Integration ====

Using:

* M5Stack Atom Echo (`car_ble`)
  * BLE detection for keys, wallet, badge, laptop
  * Piper TTS for “item left behind”
  * WLED (`wled_car_warning`) for green/red status visuals

Morning startup workflow:

* Car BLE powers on  
  * BLE scan → verify all 4 tracked items  
  * Piper TTS: “All items present — have a great day Nathan!”  
  * WLED animation: green  
  * OR if missing: “Your work badge is not detected.” (red animation)

----

==== 🧱 Future Modules (TorresVault 2.0 Timeline) ====

* On-device embeddings for faster semantic search  
  * Camera inference (face / person detection) on the GPU  
  * Local RAG system using your wiki + docs  
  * Full-room voice assistants replacing all Echo devices  
  * Kubernetes migration (GPU node + HA control plane)  
  * Distributed Pi Zero fleet for BLE triangulation  
  * Local-first AI dashboards on tablets

----

==== 📌 Status ====
This page represents the **design phase** of Jarvis for TorresVault v2.  
Implementation will begin after the Promove → TorresVault v2 migration is stable.

Edit summary