ServiceNow Mirror Planner — Ingestion & Range Sharding

This guide covers operational use of the Cortex ServiceNow table mirror planner with the high-throughput ingestion pipeline (Phase 4a), optional sys_id range sharding across multiple Table API nodes (Phase 4b), and the runtime schema drift (Phase 5) and relational integrity (Phase 6) enhancements that run during and after mirror jobs.

For first-time setup (connections, table selection, archive database), start with Getting your ServiceNow data.

---

Overview

High-throughput ingestion

Mirror jobs copy data through a producer–consumer pipeline:

• The producer fetches keyset pages from the ServiceNow Table API.
• The consumer batches rows and upserts them into the archive database (DELETE + multi-row INSERT in a single transaction, with row-by-row fallback on failure).

Multiple tables can copy in parallel, bounded by MaxConcurrentMirrorTables and a job-scoped API throttle (GlobalApiConcurrency).

Range sharding

Very large tables can bottleneck on a single keyset cursor and one API endpoint. When sharding is enabled, Cortex splits the 128-bit sys_id hex space into uniform partitions using BigInteger math, then runs independent keyset pipelines per partition.

Each shard:

• Owns a non-overlapping sys_id range (for example, quartering the space when ShardCount = 4).
• Can be pinned to a different Table API base URL when you configure extra instance nodes on the mirror job.
• Persists its own cursor in ServiceNowMirrorJobTableShard so interrupted jobs resume shard-by-shard.

Sharding activates only when both EnableSysIdRangeSharding is true and the table’s expected row count meets ShardRowCountThreshold.

Runtime schema drift

While rows are copying, Cortex can detect new fields (for example custom u_* columns) that were not in the initial sys_dictionary snapshot:

Channel	When it runs	What it does
Proactive (dictionary poll)	Producer, every N keyset pages	Re-reads sys_dictionary and merges new column specs into the live registry
Reactive (JSON key scan)	Consumer, before each batch insert	Compares incoming API row keys against known fields; triggers online ALTER TABLE for unmapped columns

Schema mutations run autocommit under a per-physical-table lock so sharded pipelines do not corrupt each other. This feature is off by default (EnableRuntimeSchemaEvolution: false).

Relational integrity

After data lands, Cortex strengthens the physical mirror for archive navigation:

Stage	When	What
Per-table indexing (PhaseIndexing)	After row copy for each table	Creates idempotent non-clustered indexes on scalar reference columns (IX_sn_r_*)
Job-end reconciliation	After all tables finish	Metadata rescan, catch-up indexes, soft orphan validation, M2M junction views

Orphan validation never fails the job—it writes warnings to the sync log. Junction views (vw_sn_m2m_{parent}_{field}) accelerate many-to-many navigation in the archive viewer.

---

Prerequisites

• Cortex admin access and a ServiceNow API data source with admin Table API access.
• An empty ServiceNow archive database (MariaDB, PostgreSQL, or SQL Server).
• For multi-node sharding: one or more extra instance base URLs on the mirror job (same credentials as the primary node). Each URL becomes an API client in the job’s pool; shards round-robin across clients up to MaxConcurrentShardsPerTable.

Open the planner at Utilities → ServiceNow table mirror planner (/utilities/servicenow-table-planner) and monitor jobs at Utilities → ServiceNow mirror jobs (/utilities/servicenow-mirror-jobs).

---

Configuration

Sharding and ingestion settings live under ServiceNowMirror:Ingestion in appsettings.json (or environment variables). All keys have defaults in code—the JSON block is optional and documents tunables for operators.

"ServiceNowMirror": {
  "Ingestion": {
    "EnableSysIdRangeSharding": false,
    "ShardCount": 4,
    "ShardRowCountThreshold": 250000,
    "MaxConcurrentShardsPerTable": 4
  }
}

Environment variable override example:

ServiceNowMirror__Ingestion__EnableSysIdRangeSharding=true
ServiceNowMirror__Ingestion__ShardCount=4

Sharding keys

Key	Default	Operational meaning
EnableSysIdRangeSharding	false	Master switch. When true, eligible tables split into parallel sys_id range pipelines.
ShardCount	4	Number of hex partitions (1–32). Example: 4 quarters the 128-bit space at 4000…, 8000…, C000… boundaries.
ShardRowCountThreshold	250000	Minimum expected rows before sharding is considered. Smaller tables use a single pipeline even when sharding is enabled.
MaxConcurrentShardsPerTable	4	Max shard pipelines running at once per table (1–16). Set ≤ number of API nodes you configured on the job.

Related ingestion keys (not sharding-specific)

Key	Default	Role
MaxConcurrentMirrorTables	4	Parallel table copies per job
GlobalApiConcurrency	8	Max concurrent Table API calls per job
KeysetPageSize	500	Rows per keyset page (application tables)
PrefetchPageDepth	2	Producer prefetch depth
TargetBatchSize	50	Rows per consumer batch insert
ProgressCheckIntervalPages	50	Keyset pages between the mirror-row growth checks that stop a non-converging copy. 0 disables the watch
MaxProgressChecksWithoutGrowth	5	Consecutive growth checks without a new distinct row before the copy is stopped
SysMetadataKeysetPageSize	1000	Rows per page for the sys_metadata copy (1–10000)
AttachmentOperationTimeoutSeconds	600	Seconds allowed for one attachment's download, metadata read and inserts (30–7200). A timed-out attachment is recorded as a row-level failure and the phase continues
AttachmentPhaseTimeoutMinutes	360	Minutes allowed for the whole attachment phase (5–1440) before it stops and the job finalizes with warnings

Schema drift & relational integrity (optional)

Section	Key	Default	Role
ServiceNowMirror:SchemaDrift	EnableRuntimeSchemaEvolution	false	Turn on proactive + reactive drift handling
	DictionaryPollIntervalPages	50	Pages between dictionary polls
ServiceNowMirror:RelationalIntegrity	EnablePostCopyReferenceIndexes	true	Per-table + job-end reference indexes
	EnableJobEndReconciliation	true	Job-end metadata rescan and catch-up
	EnableOrphanValidation	true	Soft FK orphan audit (warnings only)
	EnableJunctionViews	true	Create vw_sn_m2m_* views for M2M fields

---

How it works

Planner workflow (with sharding)

1. Connection — Primary instance URL + credentials (or saved API data source).
2. Select tables — Catalog shows storage strategy (TPH, TPC, TPP, Standalone) and physical storage table name.
3. Analyze & export — Dependency expansion (metadata, references, inheritance, attachments), plus a row-size preflight that flags tables too wide for the archive engine (see Row-size preflight for wide tables) and an estimated mirror duration (see Estimated mirror duration).
4. Archive database — Target ServiceNow archive connection.
5. Extra instance URLs (recommended for sharding) — Add additional Table API base URLs for multi-node instances. The job builds one API client per URL; shards distribute across them.
6. Start mirror job — Background worker runs copy, indexing, and job-end reconciliation.

Row-size preflight for wide tables

A table-per-hierarchy (TPH) family such as task stores hundreds of subclasses on one physical table, and the mirror aggregates every subclass column onto it. That can push a single table past the archive engine's row-size limits, where CREATE TABLE is rejected and no table is created at all — which is why a wide TPH parent, and the child tables stored on it, can be missing from Browse entirely.

Cortex checks for this in two places.

Run analysis (planner). After dependency expansion, Cortex estimates each physical mirror table's aggregated column count and row width using the same type and size mapping the mirror itself uses, and writes the findings to the analysis log. If any table looks too wide for the archive engine's limits, a Row-size preflight alert names up to the first 20 at-risk tables. If ServiceNow metadata could not be read completely (a paging safety cap or an X-Total-Count mismatch), the alert says so, because the estimate may understate the risk.

The limits used are those of the archive database you selected on the Archive database step: MariaDB/MySQL tables are measured against the archive's own InnoDB page size, SQL Server and PostgreSQL against their own calibrated limits. A MariaDB/MySQL page size can only be read once an archive is selected and reachable; until then Cortex assumes the 16 KB default and says so in the analysis log, so an early flag is not mistaken for a real measurement. Because the findings describe one engine, changing the archive selection clears a completed analysis (and cancels one still running) - run Analysis again for findings that match the new archive.

Table creation (mirror job). Before a table's DDL, Cortex logs a row-size report for TPH parents and any table with 50 or more columns — aggregated column count, declared bytes, fixed-length bytes, off-pageable columns, the in-page limit, and the widest columns — and appends it to the job's sync log under the schema phase. If CREATE TABLE is still rejected with a row-size error, Cortex re-fits that table's aggregated columns in aggressive off-page mode, promoting widenable columns to off-page text types until the conservative in-page estimate fits, and retries the create once. Columns widened or dropped by that retry are recorded against the table, so it reports as Complete with warnings rather than failing.

Escalation only runs for a table that actually fails, so healthy wide tables are not altered. See ServiceNow TPH Mirrors for how aggregated parent columns and their limits work.

Shard lifecycle

Table copy starts
  → Expected rows ≥ ShardRowCountThreshold and sharding enabled?
       No  → Single keyset pipeline (standard copy)
       Yes → EnsureShardState: N shard rows in job DB
            → Up to MaxConcurrentShardsPerTable parallel pipelines
            → Each shard: sys_id range filter + dedicated API client index
            → Cursor saved per shard on each page
  → All shards complete → table proceeds to PhaseIndexing

Shard phases in the database: Pending → Copying → Complete (or Failed). Resume an interrupted job from the mirror jobs dashboard—shards with saved cursors continue where they left off.

How sys_metadata is copied

sys_metadata is the one table family the mirror does not copy with a keyset cursor. Some instances hold legitimate records whose sys_id is a semantic word rather than a 32-character GUID (ServiceNow's own sys_security_operation rows carry values such as write and delete). Because a hex GUID sorts below a word like write, a comparison of sys_id > <last value> excludes every row after that point: the copy stops partway through the table, and the table is finalized as Complete with warnings rather than copied in full.

Cortex therefore copies sys_metadata with absolute offset paging, ordered by sys_created_on, sys_id. sys_created_on is immutable, so a record update cannot move a row across a saved offset and make the next page skip it.

• The saved cursor is offset:<n>, and resume continues from the saved offset.
• A cursor from an older keyset-based run cannot be honoured, so the copy restarts at offset 0. That is a safe gap-fill rather than a duplicate: the mirror upserts by sys_id (DELETE + INSERT).
• Pages are SysMetadataKeysetPageSize rows (default 1000), and offset pages get the same transient retry/backoff and row-failure recording as the keyset path.
• Offset pages ask ServiceNow to omit its pagination header (sysparm_suppress_pagination_header). sys_metadata is wide enough that the header's URLs would exceed the request length limit and ServiceNow would reject the page with Pagination not supported.

Every other table keeps the normal keyset paging, and every whole-table copy (offset-paged or not) is still watched by the growth check described under A table that stops making distinct-row progress.

Schema drift channels

Proactive (dictionary poll)

• Runs on the producer every DictionaryPollIntervalPages keyset pages.
• Fetches updated sys_dictionary rows for the logical table.
• Merges new ColumnSpec entries into the shared ServiceNowMirrorRuntimeColumnRegistry (shared across shards on the same physical table).

Reactive (JSON key scan)

• Runs on the consumer immediately before each batch insert.
• Compares each row’s API keys to KnownLogicalElements (ignores _display_value and _link suffix keys).
• For new keys: infers SQL type (dictionary row first, then heuristics on batch values), then runs ALTER TABLE under the table schema coordinator lock.
• Updates the registry so subsequent pages include the new column in sysparm_fields and insert mapping.

Enable both channels with EnableRuntimeSchemaEvolution: true. Leave off until you have validated drift behavior in a non-production mirror.

Indexing and job-end reconciliation

Per table — PhaseIndexing

After row copy and row-failure retry, before count verification:

1. Table phase transitions to Indexing (UI label: Creating reference indexes).
2. ServiceNowMirrorReferenceIndexEvolver scans scalar reference fields whose target table is in the mirror schedule.
3. Creates idempotent indexes IX_sn_r_{table}_{column}_{seq} (truncated to 63 characters for PostgreSQL).
4. Phase advances to Verifying.

Job-end — relational integrity coordinator

When all scheduled tables have finished (and the job is not re-queued for resume):

1. Full metadata rescan (sys_dictionary, sys_m2m).
2. Catch-up reference indexes for any columns added during Phase 5 drift.
3. Soft orphan validation — anti-join counts for broken reference links (TRIM(fk) <> '' skips unassigned fields). Warnings only; job still completes.
4. Junction views — CREATE VIEW vw_sn_m2m_{parent}_{field} projecting parent_sys_id, child_sys_id, junction_sys_id from M2M link tables. Archive viewer M2M relationships prefer these views over heuristic junction-table queries.

Record-format check during Run analysis

Estimating table widths is not the only way a mirror can fail after it has started. Run analysis also samples the data itself, so a value the archive engine cannot store is reported while you are still configuring the job rather than hours into a copy.

After the row-size preflight, Cortex fetches one record from each selected table and checks every non-empty field whose mirror column is a typed column - a number, decimal, boolean or date/time - against the column the mirror would declare for it on the selected archive engine. The check uses the same value parser the mirror uses when it inserts rows, so "passes the check" and "the INSERT succeeds" mean the same thing. Text and reference columns are not format-checked, because there is nothing to parse.

A field whose sampled value cannot be read as its column type is reported as a finding:

u_custom_amount: value "1,250.00 USD" is not a valid decimal(18,2) — the mirror INSERT would fail for records with this value.

• The findings are written to the analysis log with a FORMAT: prefix, and the first 20 are listed in a Data-format preflight alert (the log holds the rest).
• Inherited fields are checked as well: a record taken from a table-per-hierarchy child is validated against the superclass field definitions merged in first, so a value in a field the subclass inherits is not missed.
• The check is bounded, and says so rather than implying full coverage. It samples at most 500 tables, warning when a run selects more. Where an instance answers a read with only sys_id, Cortex re-requests the declared fields - and because the Table API caps field names per request, a table with more than 250 declared fields is checked for the first 249 of them, again with a warning. Empty tables are skipped, and a table that cannot be sampled is reported as a warning naming it.
• It never blocks the analysis: if the check fails, a warning is logged and the rest of the analysis result stands. If the archive selection changes while it is running, its findings are discarded rather than shown against the new archive.

The check reads records from ServiceNow and writes nothing; it does not create tables or touch the archive.

---

Monitoring & telemetry

Mirror jobs dashboard

Utilities → ServiceNow mirror jobs shows macro job status and per-table phase. Watch for these table phase transitions during a healthy large-table run:

Phase	UI label	Meaning
Copying	Copying rows	Keyset / shard pipelines active
Indexing	Creating reference indexes	Post-copy reference index DDL
Verifying	Verifying	Row count and failure checks
Complete	Complete	Table finished successfully
CompleteWithWarnings	Complete with warnings	Table finished with non-fatal gaps recorded in its Last error column — counted as finished, not failed

For sharded tables, inspect shard rows in the job database (ServiceNowMirrorJobTableShard) if a table stalls in Copying—one failed shard can block completion.

ServiceNow database views (v_*) that cannot be mirrored

ServiceNow exposes derived database views under the v_ prefix — for example the Percona / MySQL monitoring views v_iostats, v_mysql_status and v_mysql_variables, or Performance Analytics views such as v_pa_par_combined_dashboard. Many of these views are not queryable through the Table API and always answer with HTTP 500.

A 500 from a v_* view is deterministic, not a transient blip, so Cortex does not retry or auto-resume such a table. The table is completed as Complete with warnings, and the reason is recorded in its Last error column:

ServiceNow view 'v_iostats' returned HTTP 500 from the Table API; derived views cannot be mirrored
through the Table API. Schema may exist but no rows were copied. Skipped so the mirror job can finish.

The view's schema may exist in the mirror database but no rows were copied, and the job moves on instead of re-queuing near completion. Ordinary (non-view) tables keep their normal transient-error retries when the Table API returns a 500.

A table that stops making distinct-row progress

A heavily churned ServiceNow table can defeat a keyset copy. Rows can be fetched and re-inserted again and again while the mirror's count of distinct rows stays flat, and because the cursor still advances the ordinary "cursor did not advance" stall check never fires. Before this check existed, a sys_metadata copy in that state kept inserting until it reached the 50,000-page safety cap.

Cortex now watches a whole-table copy for this. Every ProgressCheckIntervalPages keyset pages it re-counts the mirror's rows, and after MaxProgressChecksWithoutGrowth consecutive checks with no growth it stops the copy instead of spinning:

• sys_metadata is finished as Complete with warnings. That table family is legitimately re-copied whenever records change, so a flat row count does not mean the copy is broken; the reason is written to the table's Last error column.
• Any other table is marked Failed, with a Last error naming the table:

Keyset paging for <table> stopped: the cursor advanced but the mirror stopped gaining distinct rows. Partial
data may be in the mirror. The table was failed so the mirror job can advance; failed tables are not retried
automatically. Resuming or re-running the mirror re-copies this table from the start (the copy is idempotent
by sys_id).

Partial data may therefore be present in that table, and it is not retried automatically - resume the job or re-run it (Restart from beginning) to copy the table again. The copy is idempotent by sys_id, so re-copying does not create duplicate rows.

The check guards whole-table copies. A sharded table is not watched this way, because its shards share one physical table and a sibling shard's inserts would look like this shard's progress; the sys_metadata resume pass applies the same growth check itself. If a sharded table stalls in Copying, inspect its shard rows in the job database (ServiceNowMirrorJobTableShard) - one failed shard can block completion.

Bounding the attachment phase

The binary-attachment phase runs after the application tables finish copying. It is bounded in two places, so a hung download or a stalled database write cannot leave the job stuck at Running with every table already finished.

• Per attachment - AttachmentOperationTimeoutSeconds (default 600, range 30–7200) covers one attachment's file download, its metadata read and the sys_attachment / sys_attachment_doc inserts. An attachment that exceeds it is recorded on the Row-Level Failures tab with the reason Attachment operation timed out after <seconds> second(s)., and the phase continues with the remaining attachments - one pathological file does not hold up the rest.
• Whole phase - AttachmentPhaseTimeoutMinutes (default 360, range 5–1440). If the phase is still running after that, Cortex stops it and finalizes the job with warnings instead of leaving it Running.

Both limits are linked to the job's own cancellation token, so cancelling a job still cancels the phase immediately and is not mistaken for a timeout. The defaults are safety nets rather than expected steady-state values - raise them under ServiceNowMirror:Ingestion for very large attachment volumes. Timed-out attachments are listed with the other row-level failures and can be retried once the job has stopped, as described below.

Retrying failed rows

Row-level insert failures are listed on the Row-Level Failures tab of the mirror jobs page, with the actions below.

Action	What it does
Retry failed rows	Re-fetches every listed sys_id from ServiceNow and retries the insert, without re-running the whole job.
Retry selected	Retries only the rows you select in the grid.
Export failure logs	Downloads a CSV of the listed row-level failures.

Retries are only accepted once the job is no longer Pending or Running. While a job is running, the retry buttons are not offered and the warning banner tells you that Retry failed rows becomes available once the job stops or completes — so let the job pause, fail, or finish before retrying.

Job-level errors while a job is running

The job-level error banner is only shown for a job that is not running. A running job's stored error is a worker-pass snapshot that can lag the live per-table phases and contradict the progress summary shown above it, so it is held back until the job stops.

Per-table errors and the row-level failure list are still shown while a job runs — use the Last error column on the Table Sync Progress tab and the Row-Level Failures tab to see what is actually failing mid-run.

Per-table failure reasons survive the job's automatic resume: a table that already ended as Failed or Complete with warnings keeps its Last error text when the job requeues, so the reason it stopped is still readable on the next pass. Only a table that is actually re-processed — through Retry failed rows or Restart from beginning — starts again with a clean error state.

Sync log panel

Each job stores the last 100 structured log lines (SyncLogJson). Log entries include a phase tag:

Phase tag	When it appears
metadata	Catalog / sys_dictionary / sys_db_object loading
schema	DDL, topology, per-table index creation messages
datacopy	Page progress during row copy
relationalintegrity	Job-end reconciliation, orphan warnings, junction views

Messages to expect at job end (phase tag relationalintegrity):

• Job-end relational integrity reconciliation starting…
• Catch-up reference indexes ensured for N physical table(s).
• Relational integrity: {table}.{column} → {target}: N orphan reference linkage(s). Sample source sys_id(s): … (warnings only)
• Orphan validation complete: X field(s) scanned, Y with orphan linkages, Z total orphan reference(s).
• Junction navigation views ensured for N M2M link field(s).
• Job-end relational integrity reconciliation complete.

Orphan messages indicate data quality gaps in the mirror (missing target rows), not a failed job. Investigate source ACL filtering, partial table selection, or timing if counts are high.

Estimated mirror duration

A ServiceNow instance's size and speed vary enormously between customers, so Cortex does not estimate a mirror run from a fixed assumption. It measures throughput from a real sample and extrapolates from the selected tables' known row counts, in two stages.

At analysis time (rough). After the row-size preflight, Run analysis picks a representative application table and reads a bounded sample of it read-only from ServiceNow, then shows an Estimated mirror duration result panel in the Analyze step:

• a low-to-high duration band (for example between about 12 minutes and about 1 hour) rather than a single number, because the sample is small;
• a basis line saying what the estimate was measured from;
• how many rows across how many of the selected tables are known, and the sample's throughput in rows per second;
• any caveats, such as a wide band because the sample was small, or tables whose row count ServiceNow could not report.

The same line is written to the analysis Activity log. The estimate is skipped (with a log line) when there is no suitable table to sample, and the band widens when a sample read only the record sys_id because the table's full field set could not be discovered — that sample can be optimistic for wide tables. Sampling is a read-only source read: it writes nothing to the mirror database and nothing to ServiceNow.

At job start (real). When the mirror job begins, Cortex copies one representative application table end-to-end before the parallel pass, measures the real throughput, and stores the estimate on the job. That table is copied single-threaded and slightly ahead of the rest of the run; it is a table the job needs anyway, and the parallel pass then skips it. The estimate is skipped when the only candidate tables are large enough that a serial copy would delay the job.

On the mirror jobs dashboard. A job that has an estimate shows:

• Estimated total runtime — the same low-to-high band, with the sampled table and its measured rows per second;
• a progress bar of rows copied against the estimate's known total (for example 120,000 of about 450,000 rows copied), which is useful in the early part of a run before enough live throughput data exists to state a live ETA;
• Estimated completion (UTC) as soon as the job can compute a live ETA from its own progress, alongside any older estimate.

The duration estimate is best-effort and never fails or delays a mirror job beyond the serial copy described above: if measuring or extrapolating fails, it is logged and the job runs on without an estimate.

Server debug logs

When log level is Debug, Table API calls include ServiceNowApiCallContext tags:

(phase: datacopy, worker: 2, shard: 1)

Use these to correlate slow pages with a specific shard and API node. See Logging for enabling file or console debug output.

---

Operational checklist

Enabling sharding for a large mirror

• Set EnableSysIdRangeSharding to true (config or env var).
• Confirm ShardCount and MaxConcurrentShardsPerTable match your API node count.
• Add extra instance base URLs on the mirror job in the planner (one per node).
• Set ShardRowCountThreshold above your smallest table that should not shard.
• Monitor mirror jobs dashboard and sync log during first production run.
• Resume rather than restart if the job interrupts—shard cursors are persisted.

Enabling schema drift (optional)

• Set ServiceNowMirror:SchemaDrift:EnableRuntimeSchemaEvolution to true.
• Start with a test mirror; confirm new u_* fields appear without job failure.
• Watch for DDL timeout messages; increase SchemaMutationTimeoutSeconds if needed.

After job completion

• Confirm tables reached Complete or Complete with warnings.
• Review sync log for relationalintegrity orphan warnings.
• Spot-check archive viewer M2M relationships on tables with sys_m2m metadata.
• Register or refresh the ServiceNow archive data source in Cortex pointing at the mirror database.

---

Troubleshooting

Symptom	Likely cause	Action
Large table slow; single API node saturated	Sharding off or no extra URLs	Enable sharding; add extra instance URLs; raise MaxConcurrentShardsPerTable
Table stuck in Copying	One shard failed or stalled	Check shard LastError in job DB; resume job; review API rate limits
Job seems stuck near completion, or keeps re-queuing	A v_* database view returns HTTP 500 from the Table API	Those views are skipped automatically as Complete with warnings with no rows copied — check the per-table Last error column, and investigate only the tables that are still interrupted
Sharding not activating	Row count below threshold	Lower ShardRowCountThreshold or accept single-pipeline copy
New columns missing in mirror	Schema drift disabled	Enable EnableRuntimeSchemaEvolution
Job completes but orphan warnings	Expected with partial mirrors	Expand mirrored reference targets or accept soft warnings
M2M relationships slow in archive	Junction views missing	Confirm EnableJunctionViews and mirrored sys_m2m / junction tables
Retry failed rows button not offered	Job is still Pending or Running	Wait for the job to stop or complete; per-table errors and row-level failures stay listed while it runs
Job stays Running after every table is finished	A hung attachment download or a stalled database write in the attachment phase	The attachment phase is bounded: a timed-out attachment becomes a row-level failure and the phase continues, and a phase that exceeds AttachmentPhaseTimeoutMinutes stops and finalizes the job with warnings. Raise AttachmentOperationTimeoutSeconds / AttachmentPhaseTimeoutMinutes under ServiceNowMirror:Ingestion for very large attachment volumes
A TPH parent table is missing from Browse (and the child tables stored on it)	The aggregated parent table was rejected by the archive engine's row-size limit, so no table was created	Re-run the mirror job — Cortex re-fits the column set and retries the create once. Check the table's Last error column for the columns that were widened or dropped
Row-size preflight alert before a mirror is started	Selected tables look too wide for the selected archive engine's row-size limits (InnoDB row-size limits on MariaDB/MySQL, the engine's own calibrated limits on SQL Server and PostgreSQL — too many columns and/or columns too wide)	Review the listed tables in the analysis log; when a table does fail to create the mirror re-fits it and retries once, recording the affected columns

For connection, catalog, and first-time planner issues, see Getting your ServiceNow data.

---

See also

• Getting your ServiceNow data — Planner workflow, recommended tables, attachments
• Logging — Debug log levels and file output
• Browse — Using the archive after the mirror completes
• ServiceNow TPH Mirrors — how the mirror aggregates TPH subclass columns onto parent tables

---

New Features Added In V3.2.0

Automatic Retry of Failed Records

When a mirror job encounters errors during record retrieval (e.g., transient API failures, network timeouts), it now automatically retries the failed records rather than marking the entire batch as failed. The per-table status shows:

• Retrieve failures — records that failed on initial retrieval and were retried
• Each failure records a FailureKind (Insert or Retrieve) and the cursor position for resumable retries

Reliable Rebuilds

Mirror jobs now include enhanced completion checks:

• Retry at the record level — individual record failures don't abort the table
• Resume from cursor — sharded tables resume from the last successful LastSysIdCursor
• Admin retry UI — per-record detail dialog with Inspect and Retry buttons
• Failure reporting — clearer per-table failure counts and types in the mirror jobs dashboard

Adding Recommended DB Indexes

After restoring a ServiceNow export (rather than using the mirror planner), and building a Cortex data source connection to it, you can use Cortex to rebuild the recommended B-tree indexes from Data Sources > Your Service Now Data Source > Database tab > Add Recommended Db Indexes.

The tool resolves each recommended logical table to its physical mirror table (including *_archive and *_bak variants) and skips tables that are not present in the mirror.