Consolidation & Decay

Source files

Key implementation files:

• consolidation.py#L111-L770 — MemoryConsolidator and ConsolidationScheduler classes
• consolidation.py#L178-L230 — Relevance score calculation (decay task)
• consolidation.py#L232-L330 — Creative association discovery
• consolidation.py#L332-L430 — Clustering algorithm
• consolidation.py#L433-L545 — Forgetting/archiving
• consolidation.py#L773-L873 — Scheduler
• app.py#L80-L102 — Scheduler initialization
• app.py#L1866-L1900 — Background thread and manual endpoint
• tests/test_consolidation_engine.py — Test coverage

The Consolidation Engine maintains and optimizes the memory graph through scheduled background processing inspired by biological memory consolidation. It applies exponential decay, discovers non-obvious associations, clusters similar memories, and implements controlled forgetting to prevent unbounded memory growth.

This page covers the MemoryConsolidator and ConsolidationScheduler classes and their integration with the Flask API. For the enrichment pipeline that processes new memories, see Enrichment Pipeline. For the overall background processing architecture, see Background Processing.

---

System Overview

The consolidation engine runs independently from client requests on configurable schedules. Unlike enrichment (which processes individual memories) or embedding generation (which handles batches of new memories), consolidation operates on the entire memory graph to maintain its health over time.

Consolidation in the Background Processing System

graph TB
    API["Flask API<br/>(app.py)"]

    subgraph "Background Workers"
        EnrichQ["Enrichment Queue<br/>Per-memory processing"]
        EmbedQ["Embedding Queue<br/>Batch generation"]
        Sched["ConsolidationScheduler<br/>(consolidation.py)"]
    end

    subgraph "Consolidation Tasks"
        Decay["Decay Task<br/>calculate_relevance_score()"]
        Creative["Creative Task<br/>discover_creative_associations()"]
        Cluster["Cluster Task<br/>cluster_similar_memories()"]
        Forget["Forget Task<br/>apply_controlled_forgetting()"]
    end

    Falkor[("FalkorDB<br/>memories graph")]
    Qdrant[("Qdrant<br/>vectors")]

    API -->|"POST /memory"| EnrichQ
    API -->|"POST /memory"| EmbedQ
    API -->|"Starts on init"| Sched

    Sched -->|"Hourly"| Decay
    Sched -->|"Hourly"| Creative
    Sched -->|"6 hours"| Cluster
    Sched -->|"Daily"| Forget

    Decay -->|"Update scores"| Falkor
    Creative -->|"Create DISCOVERED edges"| Falkor
    Cluster -->|"Create MetaMemory nodes"| Falkor
    Forget -->|"Archive/delete"| Falkor
    Forget -->|"Delete vectors"| Qdrant

---

Core Components

MemoryConsolidator Class

The MemoryConsolidator class implements the four consolidation tasks. It depends on a graph store (FalkorDB) and optionally a vector store (Qdrant) for deletions during forgetting.

Key Parameters:

Parameter	Default	Purpose
base_decay_rate	0.1	Daily exponential decay rate
reinforcement_bonus	0.2	Strength added when memory accessed
relationship_preservation	0.3	Extra weight for connected memories
min_cluster_size	3	Minimum memories per cluster
similarity_threshold	0.75	Cosine similarity for clustering
archive_threshold	0.2	Archive below this relevance
delete_threshold	0.05	Delete below this relevance

ConsolidationScheduler Class

The ConsolidationScheduler manages when each consolidation task runs based on configured intervals.

Default Schedule Configuration:

Task	Interval (Environment Variable)	Default Value
decay	CONSOLIDATION_DECAY_INTERVAL_SECONDS	3600 (1 hour)
creative	CONSOLIDATION_CREATIVE_INTERVAL_SECONDS	3600 (1 hour)
cluster	CONSOLIDATION_CLUSTER_INTERVAL_SECONDS	21600 (6 hours)
forget	CONSOLIDATION_FORGET_INTERVAL_SECONDS	86400 (24 hours)

---

Decay Task: Relevance Score Calculation

The decay task updates relevance_score for all memories using exponential decay based on age, access patterns, relationships, importance, and confidence.

Relevance Score Algorithm

graph LR
    Memory["Memory Node<br/>(FalkorDB)"]

    subgraph "Factors"
        Age["Age Factor<br/>exp(-0.1 * days)"]
        Access["Access Factor<br/>exp(-0.05 * days_since_access)"]
        Rel["Relationship Factor<br/>1 + 0.3 * log(1 + rel_count)"]
        Imp["Importance<br/>0.5 + user_importance"]
        Conf["Confidence<br/>0.7 + 0.3 * confidence"]
    end

    Memory --> Age
    Memory --> Access
    Memory --> Rel
    Memory --> Imp
    Memory --> Conf

    Age --> Score["Relevance Score<br/>= decay * access * rel * imp * conf"]
    Access --> Score
    Rel --> Score
    Imp --> Score
    Conf --> Score

    Score --> Update["SET m.relevance_score"]

Calculation Steps:

1. Age-based decay: exp(-0.1 * age_days) — Exponential decay from creation timestamp
2. Access reinforcement: 1.0 if accessed within 24 hours, else exp(-0.05 * days_since_access)
3. Relationship preservation: 1 + 0.3 * log(1 + relationship_count) — Connected memories decay slower
4. Importance scaling: 0.5 + importance (scales from 0.5 to 1.5)
5. Confidence bonus: 0.7 + 0.3 * confidence (adds up to 30%)
6. Combined score: Product of all factors, capped at 1.0

Cypher Query Pattern:

MATCH (m:Memory)
WHERE (m.archived IS NULL OR m.archived = false)
  AND m.importance >= $importance_threshold  -- Optional filter
RETURN m.id, m.timestamp, m.importance, m.last_accessed, m.relevance_score

The consolidator then updates each memory with:

MATCH (m:Memory {id: $id})
SET m.relevance_score = $score

Relationship Count Caching Optimization

To avoid O(N) graph queries during decay, relationship counts are cached with hourly invalidation:

The hour_key parameter causes automatic cache invalidation every hour, balancing freshness with an 80% query reduction.

Note

The relationship count cache is an LRU cache keyed by (memory_id, hour_key). Each new hour, cached counts are considered stale and re-queried from FalkorDB. This design intentionally accepts slightly stale counts to avoid N+1 graph queries during every decay cycle.

---

Creative Task: Association Discovery

The creative task discovers non-obvious connections between memories by randomly sampling the graph and analyzing semantic similarity and temporal patterns.

Creative Association Algorithm

graph TB
    Sample["Sample Memories<br/>WHERE relevance_score > 0.3<br/>ORDER BY rand()<br/>LIMIT 20"]

    subgraph "Pairwise Analysis"
        Check["Check Existing Edges<br/>MATCH (m1)-[r]-(m2)"]
        Similarity["Calculate Cosine Similarity<br/>_cosine_similarity()"]
        Type["Determine Connection Type"]
    end

    subgraph "Connection Rules"
        R1["Decision + Decision<br/>similarity < 0.3<br/>→ CONTRASTS_WITH"]
        R2["Insight + Pattern<br/>similarity > 0.5<br/>→ EXPLAINS"]
        R3["Different types<br/>similarity > 0.7<br/>→ SHARES_THEME"]
        R4["Same week<br/>similarity < 0.4<br/>→ PARALLEL_CONTEXT"]
    end

    Sample --> Check
    Check -->|"No edge exists"| Similarity
    Similarity --> Type
    Type --> R1
    Type --> R2
    Type --> R3
    Type --> R4

    R1 --> Create["CREATE (m1)-[r:DISCOVERED]->(m2)"]
    R2 --> Create
    R3 --> Create
    R4 --> Create

Connection Types Discovered:

Type	Conditions	Confidence	Example
CONTRASTS_WITH	Both Decision, similarity < 0.3	0.6	Opposing architectural choices
EXPLAINS	Insight + Pattern, similarity > 0.5	0.7	Insight explains pattern
SHARES_THEME	Different types, similarity > 0.7	similarity	Cross-domain patterns
PARALLEL_CONTEXT	Same week, similarity < 0.4	0.5	Unrelated concurrent work

Edge Properties:

All discovered edges are labeled with DISCOVERED in FalkorDB and carry metadata about the connection type and confidence:

{
  "type": "CONTRASTS_WITH",
  "confidence": 0.6,
  "discovered_at": "2025-01-15T10:00:00Z",
  "algorithm": "creative_association"
}

Tip

The creative task mimics REM sleep — it randomly samples from the graph rather than exhaustively comparing all pairs. This keeps computation bounded (O(N²) on sample size, not total memories) while still discovering meaningful connections.

---

Cluster Task: Semantic Grouping

The cluster task groups semantically similar memories using a graph-based clustering algorithm (similar to DBSCAN) and creates MetaMemory nodes to represent patterns.

Clustering Algorithm

graph TB
    Load["Load Memories with Embeddings<br/>WHERE m.embeddings IS NOT NULL<br/>AND m.relevance_score > 0.3"]

    Build["Build Adjacency Graph<br/>For each pair:<br/>if cosine_similarity >= 0.75:<br/>  connect(i, j)"]

    DFS["Find Connected Components<br/>DFS traversal to find clusters"]

    Filter["Filter by Size<br/>Keep clusters >= 3 memories"]

    subgraph "Meta-Memory Creation"
        Theme["Identify Dominant Type<br/>max(set(types), key=count)"]
        Span["Calculate Temporal Span<br/>max(timestamps) - min(timestamps)"]
        Create["CREATE MetaMemory Node<br/>type: 'MetaPattern'<br/>cluster_size: N"]
        Link["CREATE (meta)-[:SUMMARIZES]->(m)"]
    end

    Load --> Build
    Build --> DFS
    DFS --> Filter
    Filter --> Theme
    Theme --> Span
    Span --> Create
    Create --> Link

Clustering Parameters:

• Similarity threshold: 0.75 (configurable via self.similarity_threshold)
• Minimum cluster size: 3 memories (configurable via self.min_cluster_size)
• Relevance filter: Only clusters memories with relevance_score > 0.3

MetaMemory Node Properties:

Property	Type	Description
label	string	"MetaPattern"
dominant_type	string	Most common memory type in the cluster
cluster_size	integer	Number of memories in the cluster
temporal_span_days	float	Days between oldest and newest memory
created_at	ISO datetime	When this meta-memory was created
content	string	Auto-generated cluster summary

MetaMemory nodes are connected to their member memories via SUMMARIZES relationships:

MATCH (meta:MetaPattern), (m:Memory {id: $member_id})
CREATE (meta)-[:SUMMARIZES]->(m)

---

Forget Task: Controlled Memory Pruning

The forget task archives low-relevance memories and permanently deletes very low-relevance memories, preventing unbounded graph growth.

Forgetting Thresholds

graph LR
    Memory["Memory<br/>relevance_score"]

    High["relevance >= 0.2<br/>PRESERVE<br/>Update score only"]
    Archive["0.05 <= relevance < 0.2<br/>ARCHIVE<br/>SET archived = true"]
    Delete["relevance < 0.05<br/>DELETE<br/>DETACH DELETE + vector delete"]

    Memory -->|"Calculate current relevance"| High
    Memory --> Archive
    Memory --> Delete

    Archive --> Graph["Mark in FalkorDB<br/>SET m.archived = true<br/>SET m.archived_at = now()"]
    Delete --> GraphDel["Delete from FalkorDB<br/>DETACH DELETE m"]
    Delete --> VectorDel["Delete from Qdrant<br/>vector_store.delete()"]

Forgetting Lifecycle:

1. Fresh memories (relevance > 0.2): Updated but preserved
2. Low-relevance memories (0.05-0.2): Archived (kept in graph, marked archived = true)
3. Very low-relevance memories (< 0.05): Permanently deleted from both stores

Archive Query:

MATCH (m:Memory)
WHERE (m.archived IS NULL OR m.archived = false)
  AND m.relevance_score < 0.2
  AND m.relevance_score >= 0.05
SET m.archived = true,
    m.archived_at = $now

Delete Queries:

-- FalkorDB deletion
MATCH (m:Memory)
WHERE m.relevance_score < 0.05
  AND (m.archived IS NULL OR m.archived = false)
DETACH DELETE m

-- Qdrant deletion (for each deleted memory ID)
vector_store.delete(memory_id)

Caution

Deletion from both stores is permanent and irreversible. Memories are only deleted after reaching relevance_score < 0.05, which requires sustained low access and low importance over a significant period. High-importance memories (importance = 1.0) receive a 1.5× importance scaling, making them significantly harder to forget.

---

Scheduling and Execution

Background Thread Integration

Consolidation runs in a background thread started at Flask application initialization:

# app.py — startup code
scheduler = ConsolidationScheduler(
    consolidator=consolidator,
    decay_interval=int(os.getenv("CONSOLIDATION_DECAY_INTERVAL_SECONDS", 3600)),
    creative_interval=int(os.getenv("CONSOLIDATION_CREATIVE_INTERVAL_SECONDS", 3600)),
    cluster_interval=int(os.getenv("CONSOLIDATION_CLUSTER_INTERVAL_SECONDS", 21600)),
    forget_interval=int(os.getenv("CONSOLIDATION_FORGET_INTERVAL_SECONDS", 86400)),
    tick_seconds=int(os.getenv("CONSOLIDATION_TICK_SECONDS", 60)),
)
thread = threading.Thread(target=scheduler.run, daemon=True)
thread.start()

The scheduler checks every CONSOLIDATION_TICK_SECONDS (default: 60) whether any task is due for execution.

Manual Consolidation Endpoint

Administrators can manually trigger consolidation via the /consolidate endpoint:

POST /consolidate
Authorization: Bearer <admin_token>

{
  "task": "decay"   // Optional: run specific task only
}

If no task is specified, all four tasks run sequentially.

Consolidation Control Node

Consolidation state is persisted in a ConsolidationControl node in FalkorDB:

Node Creation:

MERGE (c:ConsolidationControl {id: 'singleton'})
ON CREATE SET
    c.last_decay = null,
    c.last_creative = null,
    c.last_cluster = null,
    c.last_forget = null,
    c.history = '[]'

Update After Task:

MATCH (c:ConsolidationControl {id: 'singleton'})
SET c.last_decay = $now,
    c.history = $updated_history

This allows the /consolidate/status endpoint to report when each task last ran and its result.

---

Configuration Reference

Environment Variables

Variable	Default	Description
CONSOLIDATION_TICK_SECONDS	60	How often scheduler checks if tasks are due (seconds)
CONSOLIDATION_DECAY_INTERVAL_SECONDS	3600	How often decay task runs (1 hour)
CONSOLIDATION_CREATIVE_INTERVAL_SECONDS	3600	How often creative task runs (1 hour)
CONSOLIDATION_CLUSTER_INTERVAL_SECONDS	21600	How often cluster task runs (6 hours)
CONSOLIDATION_FORGET_INTERVAL_SECONDS	86400	How often forget task runs (24 hours)
CONSOLIDATION_DECAY_IMPORTANCE_THRESHOLD	None	Optional: Only decay memories with importance >= threshold
CONSOLIDATION_HISTORY_LIMIT	20	Max consolidation runs to keep in history

Tuning Recommendations

High-Traffic Production (>10k memories/day):

CONSOLIDATION_DECAY_INTERVAL_SECONDS=1800      # Every 30 minutes
CONSOLIDATION_CREATIVE_INTERVAL_SECONDS=7200   # Every 2 hours
CONSOLIDATION_CLUSTER_INTERVAL_SECONDS=43200   # Every 12 hours
CONSOLIDATION_FORGET_INTERVAL_SECONDS=86400    # Daily (keep at 24 hours)

Low-Traffic Development:

CONSOLIDATION_DECAY_INTERVAL_SECONDS=3600      # Every hour
CONSOLIDATION_CREATIVE_INTERVAL_SECONDS=3600   # Every hour
CONSOLIDATION_CLUSTER_INTERVAL_SECONDS=86400   # Daily
CONSOLIDATION_FORGET_INTERVAL_SECONDS=604800   # Weekly (longer retention)

Memory-Constrained Environments:

CONSOLIDATION_FORGET_INTERVAL_SECONDS=43200    # Every 12 hours (aggressive pruning)
CONSOLIDATION_DECAY_IMPORTANCE_THRESHOLD=0.3   # Only decay lower-importance memories

---

Performance Characteristics

Execution Time Estimates

Task	Complexity	1k Memories	10k Memories	100k Memories
Decay	O(N)	<1s	~1s	~10s
Creative	O(N²) sample	<1s	<2s	<5s
Cluster	O(N²) edges	2-3s	10-15s	60-90s
Forget	O(N)	<1s	~2s	~15s

Notes:

• Decay benefits from 80% cache hit rate via relationship count caching
• Creative samples only 20-30 memories, so scales with sample size not total memories
• Cluster complexity depends on embedding similarity distribution
• Forget includes vector store deletions which add latency

Memory Usage

Typical Memory Footprint:

• Baseline: ~5MB (LRU cache + worker overhead)
• During decay: +1-2MB (result dictionaries)
• During creative: +2-3MB (sample embeddings)
• During cluster: +10-50MB (all embeddings + adjacency graph for 10k memories)

Note

The cluster task loads all memory embeddings into memory simultaneously to build the adjacency graph. For very large memory collections (>100k), consider running cluster less frequently or increasing available RAM.

---

Testing and Observability

Test Coverage

The consolidation engine has comprehensive test coverage in tests/test_consolidation_engine.py:

Key Test Fixtures:

• FakeGraph: Simulates FalkorDB with configurable responses
• FakeVectorStore: Simulates Qdrant for deletion tracking
• freeze_time: Freezes datetime for deterministic decay calculations

Tests cover all four tasks, edge cases (empty graph, single memory, max thresholds), and the scheduler timing logic.

Monitoring Endpoints

GET /consolidate/status (Admin token required):

{
  "last_decay": "2025-01-15T10:00:00Z",
  "last_creative": "2025-01-15T10:00:00Z",
  "last_cluster": "2025-01-15T06:00:00Z",
  "last_forget": "2025-01-15T00:00:00Z",
  "history": [
    {
      "task": "decay",
      "ran_at": "2025-01-15T10:00:00Z",
      "memories_processed": 1542,
      "duration_seconds": 0.8
    }
  ]
}

Logging

Consolidation operations emit structured logs:

[consolidation] Starting decay task (1542 memories)
[consolidation] Decay complete: 1542 processed, 0.8s elapsed
[consolidation] Creative task: discovered 3 new associations
[consolidation] Cluster task: created 2 MetaMemory nodes from 8 clusters
[consolidation] Forget task: archived 12, deleted 3 memories

---

Biological Inspiration

The consolidation engine implements memory principles from neuroscience research:

Task	Biological Analog	Implementation
Decay	Synaptic weakening over time	Exponential relevance decay based on age and access
Creative	REM sleep association formation	Random memory sampling + similarity analysis
Cluster	Memory compression during sleep	Semantic grouping + meta-pattern creation
Forget	Controlled forgetting during consolidation	Archival before deletion, importance-weighted

Key Research Connections:

• Exponential decay mirrors forgetting curves (Ebbinghaus, 1885)
• REM-like processing inspired by memory replay during sleep
• Clustering implements principles from MELODI (DeepMind, 2024) for memory compression
• Controlled forgetting follows active forgetting research showing forgetting aids learning

For related API operations (triggering consolidation manually, viewing status), see Consolidation Operations.