Bio Clinical Biostatistics Cdisc Data

Version Compatibility

Reference examples tested with: pyreadstat 1.2+, pandas 2.1+, numpy 1.26+. CDISC standards referenced: SDTM 2.0 / SDTMIG 3.4 (SDTM 3.0 / SDTMIG 4.0 in public review through April 2026); ADaMIG v1.3 (2021); OCCDS v1.1 (Nov 2021); BDS-for-TTE v1.0; Define-XML 2.1 (FDA-recommended for studies starting on/after March 15, 2023); Dataset-JSON v1.1 (Dec 2024; FDA Federal Register notice April 2025); Pinnacle 21 Community 4.0+; CORE (CDISC Open Rules Engine, 2021). Define-XML 2.1 FDA support began March 15, 2021 and is required for studies starting on/after March 15, 2023.

Before using code patterns, verify installed versions match. If versions differ:

• Python: pip show <package> then help(module.function) to check signatures
• R packages cited (essential for ADaM derivation): admiral (Roche/openpharma), metacore, metatools, xportr

If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying.

CDISC SDTM and ADaM Data Handling

"Load clinical trial data" -> Parse CDISC SDTM domain files; build or consume ADaM analysis-ready datasets; preserve subject-level and event-level structure; respect traceability and validation expectations for regulatory submission.

• Python: pyreadstat.read_xport(), pd.read_sas(), pd.merge()
• R: haven::read_xpt(), admiral for ADaM derivation, Pinnacle21 or CORE for validation

Aggregation Strategy Taxonomy -- Choose the Right Question

These are NOT interchangeable. A drug might not change the proportion with AEs (binary: no effect) but increase events per patient (count: harmful). The choice must be pre-specified in the SAP based on the scientific question, not analytic convenience.

Decision Tree by Scenario

SDTM vs ADaM -- The Regulatory Layer Cake

The ADaM Fundamental Principles (the "ROT" document):

1. Analysis-ready (one procedure call -> the analysis result)
2. Traceability (every value links back to SDTM via metadata)
3. Clear/unambiguous communication via Define-XML
4. Naming conventions (PARAM, PARAMCD, AVAL, AVALC, BASE, CHG, PCHG, ABLFL, ANL01FL, ...)
5. Structural rules (ADSL one row per subject; BDS one row per subject/parameter/timepoint/analysis flag)

Postdoc reading: the ADaM IG v1.3 PDF (cdisc.org), ADaM ROT, FDA Study Data Technical Conformance Guide (current 2024 version), Pinnacle 21 validation rule catalog, PHUSE Connect 2023-2025 conference proceedings.

SDTM Domain Overview

USUBJID = STUDYID-SITEID-SUBJID is the universal merge key. Subject-level domains (DM) have one row per USUBJID; event-level domains have multiple.

ARM vs ACTARM: ARM is planned treatment from randomisation; ACTARM is actual treatment received. In crossover designs, ARM differs from ACTARM by definition; in parallel-arm trials, they diverge when subjects are randomised to one arm but receive another (per-protocol violations). Primary analyses use ARM (ITT); safety uses ACTARM.

RFSTDTC vs RFXSTDTC: RFSTDTC is "first study activity date" (typically screening start); RFXSTDTC is "first treatment date." For treatment-emergent adverse event (TEAE) calculations, ALWAYS use RFXSTDTC (per ICH E2A) — RFSTDTC includes screening AEs which are not treatment-emergent.

Reading .xpt Files

import pyreadstat
import pandas as pd

# pyreadstat (recommended -- handles SAS metadata)
dm, meta = pyreadstat.read_xport('dm.xpt')
# meta.column_names, meta.column_labels, meta.variable_value_labels

# pandas built-in (SAS XPORT v5)
dm = pd.read_sas('dm.xpt', format='xport', encoding='utf-8')

# CSV fallback (common in academic datasets)
dm = pd.read_csv('DM.csv')

When pyreadstat is available, the metadata object provides column labels, value labels, and format information lost with other readers. Critical for analysis-dataset derivation: the metadata carries the controlled-terminology codelist, essential for handling values like AESEV ('MILD'/'MODERATE'/'SEVERE') with semantic ordering.

SAS XPT v5 vs Dataset-JSON -- The 2025-2026 Transition

SAS XPT v5 is the current FDA submission format but dates to 1995, with constraints:

• 8-character variable names (so LBSTRESN is a max-length name)
• 200-character text values
• No UTF-8 (ASCII only) -> problematic for multilingual trials
• Single dataset per file

Dataset-JSON v1.1 (CDISC, December 2024; FDA Federal Register notice April 2025) is the modern replacement. PHUSE-CDISC-FDA pilot has demonstrated drop-in feasibility. FDA adoption timeline pending as of mid-2026; EMA and PMDA exploring in parallel.

Pragmatic position: for the next ~2 years, SAS XPT v5 will remain the de facto submission format; sponsors should architect for Dataset-JSON migration but maintain XPT compliance.

Joining Domains -- The Right Way

import pandas as pd

dm = pd.read_csv('DM.csv')
ae = pd.read_csv('AE.csv')

# WRONG: merging event-level directly onto subject-level inflates rows
# RIGHT: aggregate first, then merge
any_serious = ae.groupby('USUBJID')['AESER'].apply(lambda x: (x == 'Y').any()).reset_index()
any_serious.columns = ['USUBJID', 'HAD_SERIOUS_AE']

analysis = dm.merge(any_serious, on='USUBJID', how='left')
analysis['HAD_SERIOUS_AE'] = analysis['HAD_SERIOUS_AE'].fillna(False)

Always use how='left' when merging onto DM to preserve all randomised subjects, even those with no events. Fill missing event indicators with 0 or False.

Aggregation strategy must follow the scientific question

These are NOT interchangeable. A drug might not change the proportion with AEs (binary: no effect) but increase events per patient (count: harmful). The choice must follow the SAP, not analytic convenience.

# Count events per subject
ae_counts = ae.groupby('USUBJID').size().reset_index(name='AE_COUNT')

# Maximum severity per subject (map to numeric first -- string max is unreliable)
severity_map = {'MILD': 1, 'MODERATE': 2, 'SEVERE': 3}
ae['AESEV_NUM'] = ae['AESEV'].map(severity_map)
max_severity = ae.groupby('USUBJID')['AESEV_NUM'].max().reset_index()

# Specific event: COVID-19 adverse event
covid_ae = ae[ae['AEDECOD'] == 'COVID-19']
covid_ae['AESEV_NUM'] = covid_ae['AESEV'].map(severity_map)
had_covid = covid_ae.groupby('USUBJID')['AESEV_NUM'].max().reset_index()
had_covid.columns = ['USUBJID', 'COVID_SEVERITY']

analysis = dm.merge(had_covid, on='USUBJID', how='left')
analysis['HAD_COVID'] = analysis['COVID_SEVERITY'].notna().astype(int)

ADaM Architecture -- The Postdoc Deep Dive

ADSL (Subject-Level) -- The Spine

Exactly one row per subject. Every other ADaM dataset must merge to ADSL on USUBJID. Standard variables:

• USUBJID -- universal subject ID
• TRT01A / TRT01P / ACTARMCD / ARMCD -- planned and actual treatment, period 1
• TRTSDT / TRTEDT -- treatment start/end dates (derived from EX, not SDTM)
• AGE, SEX, RACE, ETHNIC -- demographics from DM
• RANDDT -- randomisation date
• DCSREAS / DCSREASP / DCSREASCD -- discontinuation reason (coded + verbatim)
• Population flags: ITTFL, FASFL, SAFFL, PPROTFL, EFFFL (Y/N flags for analysis populations)
• Stratification factors: STRATA1, STRATA2 (from randomisation)
• Baseline covariates that will be used as model covariates downstream

BDS (Basic Data Structure) -- Long Format Analysis Data

One row per subject per parameter per analysis timepoint per analysis flag. Used for ADVS, ADLB, ADEFF, ADQS, ADTTE.

Required variables:

• USUBJID, STUDYID -- merge keys
• PARAM, PARAMCD, PARAMN -- parameter name, code, number
• AVISIT, AVISITN -- analysis visit name, number
• ADT, ADY -- analysis date, analysis day (relative to TRTSDT)
• AVAL, AVALC -- analysis value (numeric, character)
• BASE -- baseline value (replicated per subject/parameter)
• CHG, PCHG -- change from baseline, percent change
• ABLFL -- 'Y' for the baseline record
• ANL01FL, ANL02FL -- analysis flags for primary/secondary analyses
• DTYPE -- derivation type ('LOCF', 'WOCF', 'AVERAGE', 'BOCF', or null for original)
• BASETYPE -- when multiple baselines per subject/parameter (crossover)
• EPOCH -- study period (SCREENING, TREATMENT, FOLLOW-UP)

# Example: derive ADLB BDS structure from LB SDTM
import pandas as pd

lb = pd.read_csv('LB.csv')
adsl = pd.read_csv('ADSL.csv')

# Filter to active tests
adlb = lb[lb['LBTESTCD'].isin(['ALT', 'AST', 'CREAT', 'HGB'])].copy()
adlb['AVAL'] = adlb['LBSTRESN']
adlb['PARAM'] = adlb['LBTEST']
adlb['PARAMCD'] = adlb['LBTESTCD']

# Merge subject-level treatment from ADSL
adlb = adlb.merge(adsl[['USUBJID', 'TRT01A', 'TRTSDT']], on='USUBJID')

# Compute analysis day
adlb['ADT'] = pd.to_datetime(adlb['LBDTC'], errors='coerce')
adlb['TRTSDT'] = pd.to_datetime(adlb['TRTSDT'], errors='coerce')
adlb['ADY'] = (adlb['ADT'] - adlb['TRTSDT']).dt.days + 1  # Day 1 = first dose

# Set ABLFL from LBBLFL
adlb['ABLFL'] = (adlb['LBBLFL'] == 'Y').map({True: 'Y', False: None})

# Derive BASE per subject/parameter
baselines = adlb[adlb['ABLFL'] == 'Y'][['USUBJID', 'PARAMCD', 'AVAL']]
baselines.columns = ['USUBJID', 'PARAMCD', 'BASE']
adlb = adlb.merge(baselines, on=['USUBJID', 'PARAMCD'], how='left')

# Compute CHG and PCHG
adlb['CHG'] = adlb['AVAL'] - adlb['BASE']
adlb['PCHG'] = 100 * adlb['CHG'] / adlb['BASE']

OCCDS (Occurrence Data Structure) -- One Row per Event

OCCDS v1.1 (Nov 2021) handles AE, CM, MH. Variables: AEDECOD, AEBODSYS, AESEV, AESER, ASTDT (analysis start date), AENDT, TRTEMFL.

OCCDS v1.1 added TRTEM01FL through TRTEM##FL for multi-period treatment-emergent flags — essential for crossover and multi-phase studies where a single TRTEMFL is ambiguous.

ADTTE (Time-to-Event) -- The CNSR Convention Trap

The ADaM BDS for TTE v1.0 uses BDS structure with extra variables for survival analysis:

• STARTDT -- time origin (typically TRTSDT for OS; RANDDT for PFS; response date for DOR)
• ADT -- analysis date (event date if event, censoring date if censored)
• AVAL = ADT - STARTDT (+1 if "first day = day 1" convention)
• AVALU = 'DAYS' (or 'MONTHS' for some endpoints)
• CNSR -- censoring indicator. CONVENTION: CNSR = 0 for events; positive integers for censoring, integer encodes censoring reason
• EVNTDESC -- text description ('Death due to disease', 'Last alive contact')
• CNSDTDSC, SRCDOM, SRCVAR, SRCSEQ -- traceability back to SDTM source

The CNSR convention is OPPOSITE to most statistical packages, which use 1 = event. R survival::Surv(time, event) expects event=1; SAS PROC LIFETEST takes CENSORED= statement that's opposite to CNSR convention. This is a perpetual bug source. When passing ADTTE to analysis:

# Convert CDISC ADTTE CNSR to R/Python statistical convention
adtte['event'] = (adtte['CNSR'] == 0).astype(int)  # 1 = event for survival packages

Define-XML 2.1

Every ADaM dataset requires variable-level metadata in Define-XML 2.1 (FDA-required for studies starting on/after March 15, 2023; support began March 15, 2021). Fields per variable:

• Origin -- CRF, derived, predecessor SDTM variable
• Derivation rule -- free text or controlled algorithm
• Codelist -- linked controlled terminology
• Length, datatype, label

The FDA reviewer's Analysis Data Reviewer's Guide (ADRG) is now expected in every NDA/BLA — walks reviewer through how each analysis dataset was built.

Two-level traceability expectation: SDTM raw -> ADaM analysis-ready, with no orphan derivations. FDA reviewers explicitly trace AE counts in CSR table -> ADAE rows -> AE SDTM rows. Any break is a flag.

Treatment-Emergent AE -- The Convention Variation

ICH E2A (1995) defines an AE generically. TEAE is sponsor-defined:

TRTEMFL = 'Y' if AE.ASTDT >= TRTSDT AND AE.ASTDT <= TRTEDT + X days

X = post-treatment follow-up window. Common values:

• Small molecules: 28 or 30 days
• Biologics with extended half-life: longer (e.g., 60-90 days for mAbs)
• Cell/gene therapy: indefinite (lifelong monitoring expected)

Sponsor variation:

• Day-of-first-dose AE included as TEAE (FDA preference) vs excluded (some EMA reviewers)
• Partial-date imputation: impute day 15 if only month/year known, vs censor as missing
• Worsening of pre-existing AE: flagged via SEV change vs requires new PT (preferred term)

MedDRA SOC/PT hierarchy: AEs coded to MedDRA Preferred Terms (PT), grouped by System Organ Class (SOC). Clinically related PTs (e.g., 'Diarrhea' / 'Frequent bowel movements' / 'Loose stools') often combined via Standardized MedDRA Queries (SMQs) or sponsor-defined groupings. ADAE typically carries both AEDECOD (PT) and SMQ/group flags.