Skip to content
Henceforth Docs

From metadata to database

Henceforth metadata is the source of truth. The database is a compiled projection. You author JSON5 files that describe your data model, and the runtime compiles those files into PostgreSQL tables, views, indexes, foreign keys, and triggers. This page explains the authoring loop and what happens at each step.

The authoring loop

Section titled “The authoring loop”

Working with Henceforth metadata follows a three-step cycle:

  1. Edit – add, change, or remove JSON5 metadata files
  2. Plan – preview the resulting database changes
  3. Apply – execute the changes in PostgreSQL

Step 1: edit metadata

Section titled “Step 1: edit metadata”

Metadata lives in a configured directory (e.g., dev/metadata/). To make a change, you edit the JSON5 files directly:

  • Add a new field file to add a column
  • Change required: true on a scalar field to make it mandatory
  • Add a new object file to create a new table
  • Add a lookup field file to create a relationship between objects
  • Add a rollup field file to create a trigger-maintained aggregate
  • Add an object with extends to introduce inheritance

For file naming conventions, see Metadata file layout.

Step 2: plan

Section titled “Step 2: plan”

POST /v1/plan loads the metadata, compiles the desired schema, compares it to the live database, and returns a dry-run result. No DDL is executed.

The response tells you what would change. This was captured from a branch that adds a new Opportunity object with a name field and an indexed stage field, then updated by hand for the disposition field each change now carries – not a fresh live capture:

{
  "status": "planned",
  "source": { "kind": "branch", "branch": "add-opportunity", "commit": "1967ddf894c84e3f717dcccf1b9c51d716957fb0" },
  "options": { "destructive": false, "vault": true },
  "changes": [
    {
      "action": "createTable",
      "table": "rtdata.opportunity",
      "columns": [
        { "name": "id", "type": "UUID", "primaryKey": true },
        { "name": "stage", "type": "TEXT" },
        { "name": "name", "type": "TEXT", "notNull": true }
      ],
      "risk": "safe",
      "sql": [
        "CREATE TABLE IF NOT EXISTS \"rtdata\".\"opportunity\" (\n  \"id\" UUID PRIMARY KEY REFERENCES \"hf\".\"object\"(\"id\") ON DELETE CASCADE,\n  \"stage\" TEXT,\n  \"name\" TEXT NOT NULL\n)",
        "GRANT SELECT, INSERT, UPDATE, DELETE ON \"rtdata\".\"opportunity\" TO \"system:data_user\""
      ],
      "disposition": "execute"
    },
    {
      "action": "createIndex",
      "table": "rtdata.opportunity",
      "index": "idx_rtdata__opportunity__stage",
      "target": "column:stage",
      "risk": "blockingOnLargeTable",
      "sql": [
        "CREATE INDEX IF NOT EXISTS \"idx_rtdata__opportunity__stage\" ON \"rtdata\".\"opportunity\" (\"stage\")"
      ],
      "disposition": "execute"
    },
    {
      "action": "enableRls",
      "table": "rtdata.opportunity",
      "risk": "safe",
      "sql": [
        "ALTER TABLE \"rtdata\".\"opportunity\" ENABLE ROW LEVEL SECURITY",
        "ALTER TABLE \"rtdata\".\"opportunity\" FORCE ROW LEVEL SECURITY"
      ],
      "disposition": "execute"
    },
    {
      "action": "createPolicy",
      "table": "rtdata.opportunity",
      "policy": "owner_select",
      "command": "SELECT",
      "risk": "safe",
      "sql": [
        "DROP POLICY IF EXISTS \"owner_select\" ON \"rtdata\".\"opportunity\"",
        "CREATE POLICY \"owner_select\" ON \"rtdata\".\"opportunity\" FOR SELECT USING (EXISTS (SELECT 1 FROM \"hf\".\"object\" o WHERE o.id = \"opportunity\".id AND (pg_has_role(current_user, o.owner_id, 'MEMBER') OR pg_has_role(current_user, 'system:admin', 'MEMBER'))))"
      ],
      "disposition": "execute"
    }
  ],
  "summary": {
    "tablesCreated": 1,
    "tablesRenamed": 0,
    "tablesDropped": 0,
    "columnsAdded": 0,
    "columnsChanged": 0,
    "columnsDropped": 0,
    "columnsTypeChanged": 0,
    "constraintsChanged": 0,
    "indexesCreated": 1,
    "indexesReplaced": 0,
    "unchanged": 69
  },
  "diagnostics": []
}

Each change carries its DDL statements in sql, in the order they execute, a risk level – safe, requiresBackfill, blockingOnLargeTable, or destructive – which is the highest risk among that change’s own statements, and a disposition – execute, skip, or refuse – computed server-side from the same predicate the apply gate itself uses. source names the tree (and, for a cart branch, the commit) the plan was computed against; status is "planned" – nothing has run yet.

A new object’s columns are counted under tablesCreated, not columnsAdded – columnsAdded counts columns added to a table that already existed. This response is trimmed for length: the real one also carries an insert_access, owner_update, and owner_delete policy change, each shaped like owner_select above, for a table’s four base RLS policies.

Views are rebuilt on every plan and apply (they are derived, stateless artifacts, never diffed against their live definition) and are listed under no change: a converged plan whose only “change” would be re-running CREATE OR REPLACE VIEW stays empty.

By default, an orphan drop (a column, junction table, or module schema no longer in metadata) still appears in the plan – plan never hides a change – but carries "disposition": "skip" and is reported as a diagnostic warning; it does not run. Passing {"destructive": true} runs it. An apply may skip a step only when skipping leaves the database consistent with metadata. That holds for an orphan module schema, which nothing in metadata refers to. A skipped orphan column or junction table can still leave a NOT NULL column or foreign key behind; that gap is tracked in issue #1091.

An unsafe USING type narrowing – for example an integer field going from big to regular – cannot be skipped. Skipping its ALTER would leave the column wider than the metadata that describes it, and every read of the object would then fail. So it carries "disposition": "refuse" under destructive: false: apply refuses the whole run, naming the field and the narrowing, runs nothing, and does not advance metadata. Passing {"destructive": true} runs it, after vaulting the column’s current values (unless "vault": false). A type change the server cannot classify is refused the same way, but under its own error code, APPLY_UNRECOGNISED_TYPE_DRIFT, and its message asks you to review the types it shows and then report it (issue #1094; the error codes page documents the destructive: true escape hatch and its risk): it is almost always a gap in the server’s type comparison, not something you changed.

The managed foreign-key column drop a Lookup-to-Junction relationship-strategy migration itself decided is safe once its data has moved is exempt from the destructive gate and always "disposition": "execute", even under destructive: false – never "skip". Any other change rated destructive risk that apply cannot skip, such as a generated column’s storage mode changing (ReplaceColumn, a drop-and-re-add), carries "disposition": "refuse" instead under destructive: false: apply refuses the whole run over it rather than leaving just that one change in place. disposition is computed server-side from the same predicate the apply gate itself uses – read it directly rather than deriving “would this be refused?” from risk or any other field.

Planning is the safe way to understand the effect of a metadata edit before committing to it. Review the changes and diagnostics before proceeding.

Step 3: apply

Section titled “Step 3: apply”

POST /v1/apply performs the same load, compile, and diff as /v1/plan, then executes the generated DDL inside a database transaction. If any statement fails, the entire transaction rolls back.

The response has the same shape as /v1/plan, with "status": "applied". There is no "unchanged" status: an apply with nothing left to run still returns "applied", with an empty changes array (or with only "disposition": "skip" changes still on the array, orphan drops a previous apply already reported and this one still leaves in place).

Apply accepts an optional JSON body with options:

  • destructive (default false) – allow destructive operations such as dropping orphaned columns or narrowing column types. Without it, orphan drops are skipped and an unsafe narrowing refuses the whole run
  • vault (default true) – before destructive operations, preserve the affected column data in operations.vault so it can be restored later

Apply is idempotent: it computes a checksum of the desired schema and skips execution when the database already matches. Each successful apply is recorded in operations.migrations for audit and history.

What gets created in PostgreSQL

Section titled “What gets created in PostgreSQL”

The compiler produces these artifact types:

Artifact When it is created
Table One per object. Holds user-defined field columns. Primary key references hf_object (or the parent table for inherited objects).
View One per object. Joins hf_object with the object table(s) for a unified read surface. For inherited objects, the view joins the full table chain.
Column One per scalar field (stored). One per same-row formula field (virtual generated). One per lookup field (UUID + FK). One per rollup field (trigger-maintained stored column).
View expression One per cross-object formula field (correlated subquery in the view).
Index On indexed scalar fields, on lookup FK columns, and expression indexes on indexed formulas.
Foreign key On every lookup column: referencing the target table (single-target), or hf_object (multi-target). On inherited object tables: id referencing the parent table.
Trigger + function On immutable lookup fields (mutable: false), guarding against FK changes. On rollup fields, maintaining aggregate values on insert/update/delete of child records.

How views work

Section titled “How views work”

Each object gets a view that joins hf_object with the object table:

CREATE OR REPLACE VIEW "hf_core__account__view" AS
  SELECT o."id", o."object_type", o."created_at", o."updated_at",
         t."name", t."industry", t."active", t."contact_count"
  FROM "hf_object" o
  JOIN "hf_core__account" t ON t."id" = o."id";

For inherited objects, the view joins the full table chain from hf_object through each ancestor to the leaf:

CREATE OR REPLACE VIEW "hf_core__individual__view" AS
  SELECT o."id", o."object_type", o."created_at", o."updated_at",
         t0."email",
         t1."first_name", t1."last_name"
  FROM "hf_object" o
  JOIN "hf_core__party" t0 ON t0."id" = o."id"
  JOIN "hf_core__individual" t1 ON t1."id" = o."id";

Cross-object formula fields appear as computed expressions in the view’s SELECT clause (correlated subqueries), not as columns on the object table.

The data API reads from these views, which means every query returns both system fields and user-defined fields (including inherited fields and cross-object formulas) in a single result set. Writes go to the object table(s) and hf_object separately.

Diagnostics

Section titled “Diagnostics”

The compiler may emit diagnostics alongside the schema diff. These appear in the diagnostics array of the /v1/plan and /v1/apply responses, even when there are no changes to apply.

Diagnostics surface modeling feedback – for example, a warning that required: true with onParentDelete: "setNull" was auto-corrected to restrict, or an informational note that a new subtype widens existing lookups targeting its base type. For the full diagnostic catalog, see Diagnostics.

What’s next

Section titled “What’s next”

With metadata applied and the database schema in place, the data API serves CRUD endpoints against the compiled schema – see Create records through the API and the other data API How-to recipes.

For the internal mechanics of the loader, compiler, and diff engine, see Metadata and Apply Pipeline.

Reference

Section titled “Reference”
  • Metadata file layout – directory structure, filename conventions, and parsing rules
  • Diagnostics – severity levels, diagnostic catalog, and how diagnostics surface in /v1/plan and /v1/apply responses
  • HTTP API reference – the exact field set on ApplyResult, PlannedChange, and every Change variant shown above