TACITUS

Research and evaluation

Test what helps.
Keep the evidence.

Research informs the structures, context selection and review methods. It does not establish that attaching a Tacitus tool improves every model or task.

Determinism / concepts / provenance

What can be repeatable?

Versioned records, explicit relations, declared evidence requirements and rule evaluation. Interpretation and generated prose remain subject to review.

TrustGraph

Its ontology-grounded retrieval and reusable context cores inform our architecture. Its retrieval documentation also describes model-based concept extraction and scoring. We distinguish those steps from deterministic record checks.

Retrieval architecture ↗ · Context cores ↗

W3C & rule standards

RDF and OWL express terms and relations; SHACL checks declared graph constraints; PROV-O describes provenance. LegalRuleML informs the separation of normative statements, conditions and exceptions.

SHACL ↗ · PROV-O ↗ · LegalRuleML ↗
Compiler regression: evidence retained at the same limit

On frozen v10 inputs at 24 KB, the old compiler retained zero exact quotations and omitted the artifact. The revised compiler retains three quotations and the artifact, using 4,793 rather than 5,126 o200k_base tokens. Current release objects add a fourth quotation and four concept records and use 5,617 tokens. These are assembly measurements; no model-quality or analyst-time gain is established.

Inspect the reproducible report ↗

These are design references. Tacitus is not integrated with or endorsed by TrustGraph. A checked structure does not certify a political claim.

Read the implementation and research audit ↗

Benchmarking

Measure the answer.
Count the whole cost.

Structure is useful when it preserves meaning and reduces work on a real task. A smaller prompt or a larger graph alone does not establish that benefit.

A measured assembly fix

The old compiler could keep instructions while dropping exact evidence. On the same frozen inputs and 24 KB limit, the revised compiler retains the quotations and artifact.

Frozen inputsExact quotesArtifactTokens
Before0Omitted5,126
Revised3Retained4,793

o200k_base encoding. Current examples add a fourth quotation and four concepts and use 5,617 tokens. This is assembly evidence, not proof of better model answers or total-cost savings.

Inspect the reproducible report ↗

What a pilot must establish

  1. Use held-out questions and the same source pool across task-only, retrieval and structured-context inputs.
  2. Blind the outputs for domain review: identity, attribution, conditions, supported claims and useful uncertainty.
  3. Record preparation, retrieval, model tokens, latency, failures and reviewer time.

Our earlier 48-response model evaluation was mixed. Consistent generation-quality and efficiency gains remain unproven.

FActScore ↗RAGAS ↗Evaluation questions ↗

01 / Saliency

Keep what matters
to the question.

Relevant evidence includes its dates, exceptions and uncertainty. Test whether the answer preserves the distinctions that change the conclusion.

02 / Efficiency

Measure the whole
process.

Compare useful outputs at comparable budgets. Count retrieval, model input and output, review effort and failures. A shorter prompt can still produce worse work.

Sixteen papers.
Specific design questions.

Quotes below are short excerpts from official abstracts. Open a citation for the design choice, evaluation question and limits.

Choose and limit context

01Lost in the Middle: How Language Models Use Long ContextsNelson F. Liu et al. · 2023
“performance can degrade significantly when changing the position of relevant information”

Abstract · reviewed v3

Design question
Put task-critical evidence and qualifiers in a deliberate order; test position changes instead of assuming longer context is better.
What to test
Does moving a decisive qualifier change the answer or its citation?
Limits
The paper evaluates particular models and retrieval tasks. It does not establish the behavior of every current model or validate Tacitus capsules.

Read the reviewed paper ↗arXiv:2307.03172 · 2023-11-20

02REPLUG: Retrieval-Augmented Black-Box Language ModelsWeijia Shi et al. · 2023
“REPLUG simply prepends retrieved documents to the input for the frozen black-box LM.”

Abstract · reviewed v4

Design question
Keep a canonical evidence store separate from the generation model; adapt retrieval and prompt assembly through a replaceable reader interface.
What to test
Does selected evidence outperform the same-budget untuned retrieval baseline?
Limits
Its gains depend on its trained retriever and studied tasks. Plainly prepending hand-designed context does not inherit those results.

Read the reviewed paper ↗arXiv:2301.12652 · 2023-05-24

03LLMLingua: Compressing Prompts for Accelerated Inference of Large Language ModelsHuiqiang Jiang et al. · 2023
“a budget controller to maintain semantic integrity under high compression ratios”

Abstract · reviewed v2

Design question
Use an explicit budget and assess what was lost; retain legal exceptions, dates, attribution and uncertainty as protected units.
What to test
At a matched answer-quality threshold, how much context can be removed without losing a decisive condition?
Limits
Token compression is a learned algorithm evaluated on particular datasets. A byte-bounded AI input is not this compressor and cannot claim its measured speedups.

Read the reviewed paper ↗arXiv:2310.05736 · 2023-12-06

04LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt CompressionHuiqiang Jiang et al. · 2023
“LLM performance hinges on the density and position of key information in the input prompt.”

Abstract · reviewed v2

Design question
Condition selection on the actual question and test the ordering of relevant evidence; record both compression overhead and generation cost.
What to test
Does task-aware selection preserve more useful evidence than dropping later sections to meet a budget?
Limits
Task-aware compression results are specific to its method, model, benchmarks and hardware. Density alone does not establish useful policy judgment.

Read the reviewed paper ↗arXiv:2310.06839 · 2024-08-12

05LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt CompressionZhuoshi Pan et al. · 2024
“This paper focuses on task-agnostic prompt compression for better generalizability and efficiency.”

Abstract · reviewed v2

Design question
A reusable compressor can be separated from the downstream reader; evaluate transfer and preserve a route back to complete original evidence.
What to test
Which exceptions, names, negations and minority-language spans survive compression across two model families?
Limits
The paper studies learned token classification on specific datasets and target readers. Its HTML reports a gap against task-aware methods on LongBench. It does not prove universal faithfulness for legal or conflict material.

Read the reviewed paper ↗arXiv:2403.12968 · 2024-08-12

Connect knowledge

06Knowledge GraphsAidan Hogan et al. · 2020
“We discuss the roles of schema, identity, and context in knowledge graphs.”

Abstract · reviewed v6

Design question
Give nodes stable identity, explicit types, provenance and context; distinguish an imported fact from a proposed interpretation or missing input.
What to test
Can a reviewer determine each node’s meaning, source, scope and relationship type without guessing?
Limits
The survey explains methods and quality concerns. It does not claim that a larger graph or a hundred nodes automatically improves generation.

Read the reviewed paper ↗arXiv:2003.02320 · 2021-09-11

07From Local to Global: A Graph RAG Approach to Query-Focused SummarizationDarren Edge et al. · 2024
“derive an entity knowledge graph from the source documents”

Abstract · reviewed v2

Design question
Separate source-derived entities/relationships from summaries; use communities for corpus-wide questions and direct evidence for narrow factual checks.
What to test
When does a graph-supported global answer retain more relevant perspectives than flat retrieval?
Limits
The evaluation focused on sensemaking questions over two roughly million-token corpora; the paper calls for broader domain testing and fabrication-rate comparison. A dependency picture is not a GraphRAG implementation.

Read the reviewed paper ↗arXiv:2404.16130 · 2025-02-19

08HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language ModelsBernal Jiménez Gutiérrez et al. · 2024
“HippoRAG synergistically orchestrates LLMs, knowledge graphs, and the Personalized PageRank algorithm”

Abstract · reviewed v3

Design question
A typed graph can support multi-hop retrieval; benchmark a graph traversal against simple lexical/vector retrieval on actual cross-document questions.
What to test
Can a two-hop source relationship recover a missing but necessary piece of evidence?
Limits
Reported cost and performance results belong to that retrieval system and its multi-hop QA benchmarks, not to any graph visualization or source-count target.

Read the reviewed paper ↗arXiv:2405.14831 · 2025-01-14

09LightRAG: Simple and Fast Retrieval-Augmented GenerationZirui Guo et al. · 2024
“incorporates graph structures into text indexing and retrieval processes.”

Abstract · reviewed v3

Design question
Keep both specific entity evidence and higher-level thematic relationships, with incremental update rules and source provenance.
What to test
Do local and thematic retrieval routes cover complementary evidence without increasing unsupported claims?
Limits
The studied indexing and retrieval pipeline is more than a graph-shaped UI. Its benchmark claims cannot be transferred to an untested implementation.

Read the reviewed paper ↗arXiv:2410.05779 · 2025-04-28

Support agent workflows

10MemGPT: Towards LLMs as Operating SystemsCharles Packer et al. · 2023
“a system that intelligently manages different memory tiers”

Abstract · reviewed v2

Design question
Keep persistent knowledge and current working context distinct; expose bounded read operations and require an explicit task for each AI input.
What to test
Does the workflow retrieve the right prior evidence without carrying obsolete instructions into the present task?
Limits
A memory hierarchy is not the same as expert-domain quality. Its document/chat evaluations do not establish policy accuracy or Tacitus workflow gains.

Read the reviewed paper ↗arXiv:2310.08560 · 2024-02-12

11A-MEM: Agentic Memory for LLM AgentsWujiang Xu et al. · 2025
“a comprehensive note containing multiple structured attributes, including contextual descriptions, keywords, and tags.”

Abstract · reviewed v11

Design question
Represent reusable notes with explicit attributes and meaningful links; preserve revision history when a later observation changes a contextual interpretation.
What to test
Can a new evidence item revise a conclusion without erasing the source and rationale behind the previous version?
Limits
The reported experiments concern long-term conversational tasks. The limitations note that organization quality depends on underlying LLM capabilities. Generated links require validation before becoming shared policy knowledge.

Read the reviewed paper ↗arXiv:2502.12110 · 2025-10-08

12Generative Agents: Interactive Simulacra of Human BehaviorJoon Sung Park et al. · 2023
“store a complete record of the agent's experiences using natural language”

Abstract · reviewed v2

Design question
Separate observations, derived reflections and planned actions. Use simulated persona boards to find possible weaknesses, then validate with actual users and domain reviewers.
What to test
Do independent persona reviews identify different failure modes, and can those issues be verified against the actual object?
Limits
Believable simulated behavior is not evidence of an actual founder, investor or policy expert endorsing a product. The study does not establish that an agent board replaces human validation.

Read the reviewed paper ↗arXiv:2304.03442 · 2023-08-06

13Self-RAG: Learning to Retrieve, Generate, and Critique through Self-ReflectionAkari Asai et al. · 2023
“adaptively retrieves passages on-demand”

Abstract · reviewed v1

Design question
Use a need-to-retrieve gate and an answer-review stage instead of always adding more context; retain reviewable evidence checks.
What to test
Can a workflow identify when its selected evidence is insufficient and abstain or retrieve again?
Limits
The paper trains special reflection-token behavior. A hand-written checklist or an extra critic prompt is not its trained method and does not inherit the benchmark results.

Read the reviewed paper ↗arXiv:2310.11511 · 2023-10-17

Evaluate the result

14FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text GenerationSewon Min et al. · 2023
“breaks a generation into a series of atomic facts”

Abstract · reviewed v2

Design question
Evaluate a briefing claim by claim against original evidence; separately score supported facts, unsupported inference and useful abstention.
What to test
What proportion of factual claims is supported, and did the answer omit a decisive exception?
Limits
The main empirical domain is biographies. Policy analysis requires additional rubrics for scope, contested interpretations, operational usefulness and omitted perspectives. Automated judges also need calibration.

Read the reviewed paper ↗arXiv:2305.14251 · 2023-10-11

15Ragas: Automated Evaluation of Retrieval Augmented GenerationShahul Es et al. · 2023
“the ability of the retrieval system to identify relevant and focused context passages”

Abstract · reviewed v2

Design question
Score retrieval relevance, use of retrieved evidence and the resulting answer separately; do not equate a well-formed packet with a good briefing.
What to test
Did failure arise in selection, interpretation, unsupported generation or missing source coverage?
Limits
Reference-free automated metrics are screening tools, not a substitute for domain review or proof of factual completeness. Judge models can share the generator’s errors.

Read the reviewed paper ↗arXiv:2309.15217 · 2025-04-28

16ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation SystemsJon Saad-Falcon et al. · 2023
“context relevance, answer faithfulness, and answer relevance”

Abstract · reviewed v2

Design question
Use separate dimensions and human-calibrated judging; report uncertainty and disagreements rather than an unsupported single quality score.
What to test
Do automated assessments agree with blind human review on the actual policy domain?
Limits
ARES trains lightweight judges and uses human annotations with prediction-powered inference. A synthetic panel without calibration is not an implementation of its statistical guarantees.

Read the reviewed paper ↗arXiv:2311.09476 · 2024-03-31

What was reviewed?

Official arXiv abstract/metadata for every paper; selected HTML methods/limitations for GraphRAG, LLMLingua-2 and A-MEM. No claim of exhaustive full-paper review or literature coverage.

Years identify the first arXiv submission; publication and revision years may differ. Each citation links to the version checked. The register was researched on 2026-11-09.

Simulated founder, investor, engineering and practitioner reviews can uncover problems. They are not interviews, customer validation or independent expert approval.

An actual model test

Useful tests include failures.

We recorded 48 responses from two model families across six developer-visible tasks. Structured context fixed some narrow errors and introduced others. Longer instructions did not consistently improve the answers and used more input tokens. Quality and efficiency gains are not established.

The failures informed typed checks for source values, dates, reporting stages and reviewer roles. A field check does not certify the prose; domain review remains necessary.

Original test report ↗ · Raw responses ↗ · Debugging iteration ↗

Our evaluation standard

Useful work, supported
by the right evidence.

Compare task-only, retrieval and selected-context inputs on the same task and source pool. Check supported claims, citations, dates, exceptions, disagreement and unanswered questions. Record the model, source versions, tokens, cost and timing.

Report failed checks and reviewer disagreements. Assembly tests establish that context is bounded and traceable; model and domain evaluations answer different questions.

Inspect a working tool ↗

Source inspection