Data Quality Assessment

Data Quality Assessment (DQA) checks are implemented at two levels to ensure data completeness, consistency, and accuracy across all MW-PrEP Visit Log records.

Two-layer architecture: Client-side validation catches errors at the point of collection — before the image is submitted. Server-side validation catches errors that are only visible at the population level, or that were skipped by the field worker on the device.

Part 1 — Client-Side Data Validation

When a field worker photographs a form page, ScanForm immediately runs these validation checks against the OCR results on the device. If any check fails, the app instantly prompts the worker to correct the paper form and retake the photo — before the data is submitted. There is no human verifier involved at this stage; validation is fully automated and real-time. 📸

Important constraints of client-side validation:

Validates OCR-extracted field values directly on the device — no network connection required.
Field workers can skip warnings and proceed with submission; client-side checks are advisory, not blocking.
Does not have access to the full dataset — cross-record and longitudinal checks must be done server-side.
Administrative fields (book number, page number, discard flags, photo taken) are not validated client-side.

Because field workers can bypass checks, server-side validation is always required as a second layer.

Row Eligibility

A row is considered active (i.e. validation rules apply) when:

At least one of the following key fields has ≥ 1 filled box: visit_date, prep_reg_id (≥ 3 filled boxes), sex_disagg, age, visit_outcome (≥ 2 filled boxes) — AND the discard_row bubble for that row is not marked.

This logic prevents spurious validation alerts on intentionally blank rows. The visit log contains 15 client rows per page spread; on any given day, many rows may be empty. Without this eligibility gate, every unfilled row would trigger a cascade of false “required field missing” alerts.

In addition, a page-level check validates that the Year field is filled if any row on the page is active. The year field is shared across all rows on the page and must always be present.

Active Checks

Check Types Explained

Check Type	Description
Client-Side Validation Check Types
enough_filled	At least one box or bubble in the field group is filled — prevents blank mandatory fields. Used for digit-box fields (IDs, dates, counts) and single-bubble triggers.
exactly_one	Exactly one bubble is filled — enforces single-select fields. Implemented as a pair of checks: enough_answers (≥ 1 selected) AND not_too_many_answers (≤ 1 selected).
not_too_many	No more than one bubble may be filled — caps a field at one selection. Used alone when the field is optional but cannot be multi-selected.

Fields Not Validated Client-Side

The following fields exist on the form but have no client-side validation rule. Errors in these fields are caught by server-side DQA instead.

Part 2 — Server-Side Data Validation

Server-side DQA checks run automatically each time the dbt pipeline executes against the full submitted dataset. 🤖 Unlike client-side validation, these checks can compare across records and across forms — catching errors that are only visible at the population level.

Capabilities beyond client-side:

Cross-references longitudinal records (e.g. detecting the same PrEP ID assigned to two different clients)
Validates values against external registers (PrEP register, HTS initial register)
Enforces clinical plausibility rules that require full cohort context
Applies to all submitted records, including those where client-side warnings were dismissed
Separates checks into raw-data-based (run before type casting) and clean-data-based (run after cleaning and cross-form joins)

Record Drop Logic

Warning

Records are excluded from the clean dataset — not deleted. Excluded records remain visible in the DQA dashboard and can be corrected and re-entered.

A record is dropped from the clean layer when:

It has one or more checks with severity Error.
It is structurally unverified, discarded, a duplicate, or missing (these are excluded at the raw model stage before checks run).
The discard_row bubble is marked — such rows are excluded at raw model ingestion.
Records where the page 2 scan is missing are flagged via the page_2_entry_url linkage check.

Note

Pipeline separation: Checks are split into two streams — Visit Card and Visit Log — which follow the same logic but are applied to their respective register types. The register_type column distinguishes between the two in the combined output.

Error Classification

Severity	Meaning	Action
Severity Levels
Error	The record violates a hard rule — a value is missing, unparseable, clinically implausible, or contradicts another linked record. Records with ≥ 1 Error are excluded from the clean dataset.	Exclude from analysis · Flag for correction and re-entry in DQA dashboard.
Warning	The record is unusual or internally inconsistent but may still be valid. The record is retained in the clean dataset but flagged for human review.	Keep in analysis · Flag for review · No automatic exclusion.

Full Check Catalogue

Summary by Category

Category	Errors	Warnings	Total
Server-Side Checks by Category
Missing Mandatory Value	5	0	5
Invalid Number	5	4	9
Invalid Date	7	0	7
Invalid Code	0	1	1
Multiple / Missing Answer	10	8	18
ID Transcription Check	4	0	4
Cross-Record Consistency	8	2	10
Clinical Logic	4	6	10
Total	43	21	64

Check Tag Reference

Each server-side check is tagged in the pipeline to enable filtering in the DQA dashboard. Tags allow data managers to view all checks of a given type across fields without scanning the full check catalogue.

Generated automatically from MW-PrEP Visit Log v3.2 source files. Last updated: 2026-06-09.