PROC SORT: NODUP vs. NODUPKEY & FIRST./LAST. BY-Groups
Eliminate duplicate records, order observations, and utilize FIRST. and LAST. automatic variables.
📌 Direct Answer / Executive Definition
PROC SORT arranges dataset observations in ascending or descending order by specified BY variables. The NODUPKEY option removes records with duplicate values in the BY variables. The NODUP option removes records only if the entire observation across all variables is an exact duplicate.
Why Deduplication & BY-Group Processing Matter for Data Integrity
Deduplication is a primary step in data pipelines. Using NODUP when NODUPKEY is required can allow duplicate patient visits or transactions into analysis datasets.
Standard Syntax, Derivation & Framework Pattern
/* Sorting and Deduplicating */
PROC SORT DATA=sashelp.class OUT=work.sorted_class NODUPKEY;
BY Age;
RUN;
/* BY-Group Processing with FIRST. and LAST. */
DATA work.age_summary;
SET work.sorted_class;
BY Age;
IF FIRST.Age THEN Group_Count = 0;
Group_Count + 1;
/* Output only the last observation per age group */
IF LAST.Age THEN OUTPUT;
RUN;Core Rules & Certification Takeaways
- NODUPKEY checks uniqueness across only the variables in the BY statement.
- NODUP (NODUPLICATES) compares every single column in the entire row.
- When a BY statement is used in a DATA step, SAS generates temporary boolean flags: FIRST.variable and LAST.variable (1 or 0).
- FIRST. and LAST. variables are not written to output datasets unless copied to user variables.
🔗 Related Architectural Concepts & Next Steps
Deepen your mastery with connected topics across our curriculum and knowledge hubs.
Frequently Asked Questions (FAQ)
What is the difference between NODUP and NODUPKEY in PROC SORT?
NODUPKEY checks for duplicates based only on the BY-variables and keeps the first record. NODUP checks all variables across the entire row and only drops identical rows.
What are FIRST. and LAST. variables in SAS?
They are temporary automatic flags created during BY-group processing in the DATA step that evaluate to 1 on the first or last observation of each BY-group.
Want complete video breakdowns & exercises?
Explore full step-by-step masterclass training in The Simplest Guide to SAS Programming.