Summary: The Critical Difference Between NODUP and NODUPKEY

NODUPKEY evaluates uniqueness only across the variables listed in the BY statement, keeping the very first record encountered and purging all subsequent observations into the void. In contrast, NODUP (or NODUPRECS) compares every single variable across the entire observation in the Program Data Vector (PDV), only dropping records that are 100% identical duplicates. In longitudinal clinical trials and financial ledgers, using NODUPKEY blindly destroys valid follow-up visits without throwing a single warning in the SAS log.

PROC SORT NODUPKEY vs. NODUP Architectural Comparison

Figure 1: The 3-Row Pipeline Test — Comparing NODUPKEY Key Truncation vs. Audited NODUP Deduplication with DUPOUT Archive.

The 3-Row Technical Screen #

If you want to know whether a SAS programmer truly understands the mechanics of the Program Data Vector (PDV) or simply memorized syntax from a cheat sheet, present them with this 3-row dataset:

Row 1: Patient 101 | Visit 1 (Baseline) | ALT 24 U/L (Normal)
Row 2: Patient 101 | Visit 2 (Week 4) | ALT 58 U/L (Acute Liver Signal)
Row 3: Patient 101 | Visit 2 (Week 4) | ALT 58 U/L (EDC Double-Entry Glitch)

Now ask one question: "What happens if I execute PROC SORT DATA=labs NODUPKEY; BY PatientID;?"

A junior programmer smiles: "It removes the duplicate Row 3."

A senior statistical programmer cringes: "It keeps Row 1, and permanently deletes both Row 2 and Row 3, obliterating the patient's acute safety signal from the study."

That is the gap. NODUPKEY is not an intelligent deduplicator. It is an operational woodchipper.

How the Program Data Vector Evaluates Uniqueness #

To understand why this happens, you must look at how PROC SORT interfaces with the underlying record buffer:

  • NODUPKEY (Key-Level Comparison): SAS checks only the columns specified in your BY statement. As it streams through sorted observations, it compares the current record's BY-values against the previous record's BY-values. If the keys match, SAS drops the incoming row immediately. It never inspects visit numbers, lab parameters, timestamps, or adverse event codes.
  • NODUP / NODUPLICATES (Full-Row Comparison): SAS compares every single variable in the dataset. An observation is purged if and only if all character and numeric values are exact carbon copies of the immediately preceding record.

The Deceptive Blue Note in the SAS Log #

The primary reason NODUPKEY causes catastrophic errors in production is that SAS treats key truncation as an entirely routine operation. The SAS log produces no red ERROR and no green WARNING. Instead, it displays a polite blue note:

NOTE: 14000 observations with duplicate KEY values were deleted.

To an untrained programmer, seeing a blue NOTE feels like success: "Great, I cleaned up 14,000 duplicate rows!" But in a longitudinal dataset where each patient has 10 visits over 48 weeks, that single command just discarded 90% of the clinical follow-up data.

The Adjacent-Row Trap of NODUP #

While NODUPKEY is overly aggressive, NODUP has a subtle failure mode of its own: it only checks consecutive rows.

If you run:

PROC SORT DATA=patient_labs NODUP;
    BY USUBJID;
RUN;

SAS sorts the data by USUBJID, but the order of the other variables within that subject is non-deterministic. If two identical lab records are separated by an observation from a different visit, NODUP will fail to detect them because they are not adjacent in the sort buffer!

The Ironclad Rule: Whenever you use NODUP to eliminate true duplicate records, you must sort by all variables:

PROC SORT DATA=patient_labs NODUP DUPOUT=work.discarded_records;
    BY _ALL_;
RUN;

The Enterprise Production Standard #

In regulated GxP environments, clinical trials, and financial audits (21 CFR Part 11), you must be able to prove exactly what data was removed and why. High-performing teams enforce two mandatory standards:

1. Always Use DUPOUT=

Never run NODUPKEY or NODUP without the DUPOUT=dataset option. DUPOUT= routes every single discarded observation into a dedicated audit table. If an auditor asks why records were removed, you have complete traceability.

2. Explicit Intent via FIRST. and LAST.

If your business requirement is genuinely to keep one observation per subject (for example, the most recent blood pressure reading), never rely on NODUPKEY, which arbitrarily keeps whatever record was encountered first. Sort explicitly and use explicit DATA step BY-group logic:

/* Sort by Subject and Visit Date in chronological order */
PROC SORT DATA=work.vitals;
    BY USUBJID VisitDate;
RUN;

/* Explicitly keep only the latest visit per subject */
DATA work.latest_vitals;
    SET work.vitals;
    BY USUBJID VisitDate;
    IF LAST.USUBJID; /* Deterministic and Auditable */
RUN;
Authoritative Knowledge Hub Guide

Deep Dive: PROC SORT, NODUPKEY & FIRST./LAST. BY-Groups

Explore the official encyclopedic guide on data deduplication, BY-group processing flags, and Program Data Vector mechanics in our Base & Advanced SAS Knowledge Hub.