Harvester

System Specification // Status: Operational

1. Executive Summary

Harvester is a deterministic, open-source experimental framework designed to act as a stateless pre-processing node for Open Source Intelligence (OSINT) pipelines. By standardizing ingestion protocols and deterministic filtering, this system quantifies and isolates high-value signal from internet noise prior to any semantic evaluation by Large Language Models (LLMs).

This document serves as the public, institutional specification for the system. The source code is publicly accessible via our GitHub repository.

2. System Boundaries (Scope)

To ensure absolute engineering rigor and pipeline integrity, this system operates under strict boundaries:

• No Semantic Processing: Harvester does not "understand" text. It is a purely deterministic engine relying on high-performance string matching (Regex).
• Domain Isolation: We measure the validity of data at the edge. The core domain never interacts with raw external structures.
• State Sovereignty: Memory is handled exclusively via embedded Local-First ledgers (SQLite) to restrict redundant I/O operations and guarantee node autonomy.

3. Core Architecture (The V5 Standard)

The system architecture relies on the implementation of Sovereign Data Transfer Objects (DTOs). Connectors mapping external payloads (e.g., RSS, standard API responses) must hydrate immutable Python dataclasses before interacting with the core filtering logic. This establishes an anti-corruption layer that immunizes the central pipeline against upstream API depreciation.

The filtering engine implements an implicit OR block across parameter arrays. This mechanism natively supports bilingual or multi-conditional evaluations without requiring complex Boolean string parsing.

4. Parametric Control (SCOUT_LENS)

Behavioral parameters are fully decoupled from the system's codebase. Harvester relies on a portable schema file (SCOUT_LENS.json) which defines the execution contract:

project: "System Name"
target_sources: { "rss": ["..."] }
deep_keywords: ["\"Condition A\" AND \"Condition B\""]
bullshit_filter: ["noise_term"]

5. Telemetry & Output

Upon successful execution, data structures that survive the deterministic filtering are serialized into strict Markdown manifests, embedded with the overarching thesis defined in the Lens.

Standard Execution Log:

[INFO] Output directory: $HARVESTER_DROP_PATH
[EXEC] Duplicates skipped: 5
[EXEC] Filtered (No Match): 374
[INFO] Process complete. Yield: 0.

6. Collaboration & Reproducibility

We actively invite collaboration from data engineers, OSINT researchers, and systems architects. To protect our research from uncredited commercial exploitation, the source code and methodologies are published under the Elastic License 2.0 (ELv2).