Education Cloud Course Catalog Migrate

Migrating Course Catalog to Education Cloud

Import course catalog data from multiple sources (CSV, PDF, website) and bulk-create Learning Course records with structure detection and data quality validation.

Scope

• In scope: Multi-source ingestion (CSV, PDF, web), Learning + LearningCourse bulk creation via sObject REST API (Composite batches up to 200), linked course detection (lecture+lab via IsLinkedOnly), credit hour parsing (Duration/DurationUnit), course type mapping, data quality validation, conflict resolution, gap identification/remediation
• Out of scope: Creating individual Learning Course records manually (delegate to metadata generation skills), creating Learning Programs (Learning.Type = 'LearningProgram' + separate child object), creating External Learning (Learning.Type = 'ExternalLearning'), course sections/offerings (CourseOffering object), enrollment/registration workflows (no dedicated skill yet — ask the user), SIS sync (separate integration workflow)

---

Required Inputs

Gather or infer before proceeding:

• Target org: Active Salesforce org with Education Cloud enabled
• Primary data source: CSV file, PDF document, or website URL with course catalog data
• Business Account: the Academic Operations Business Account to link courses to (e.g., College of Engineering)
• Academic term (optional): context only, not stored on Learning Course

Additional sources (gathered during workflow):

• Supplementary data sources for missing fields (descriptions, types, mappings)
• Lecture-lab mapping files (if multi-component courses detected)
• Conflict resolution decisions (if existing courses found)

Defaults unless specified:

• Course status: Active (IsActive = true on Learning)
• Batch size: 25 records per Composite API call as an agent (API supports 200; tool-call payload limits are the binding constraint — see Step 7)
• Data quality validation: Enabled
• Learning.Type: 'LearningCourse' (required for course catalog entries)

---

Execution Transport

Resolve the API version first — do not hardcode. GET /services/data/ (no version segment) returns an endpoint map already resolved to the org's max version, not a {version, label, url} list. Pipe it into scripts/resolve_api_version.py to get {"version": "69.0"}; use as vXX.X everywhere below, including the transport probe.

Then probe transport health: GET /services/data/vXX.X/limits via headless-360 dispatch. 2xx → route all calls through it. On absent/4xx/5xx → probe other transports (other Salesforce MCP, custom MCP), use first healthy, announce it. None healthy → stop, ask user to connect one; never fabricate. (SOQL /query is routable on headless-360 — confirmed by live test; /tooling/query and ?fields= are unconfirmed, use describe + direct record reads instead.)

Never fall back to sf CLI or shell out for org credentials — a slow/oversized batch is a batch-size problem (shrink it, step 7), not a transport failure, and CLI shell-out trips an unbypassable approval gate on some surfaces. A version too new for the org returns 404 NOT_FOUND on an otherwise-healthy transport — re-read GET /services/data/, drop to the org's max version, retry, don't switch transport. See references/gotchas.md for both.

---

Workflow

All steps are sequential. Do not skip or reorder. If blocked, stop and ask for missing context.

Phase 1 — Ingest Primary Source

1. Identify data source type — determine if primary source is CSV, PDF, or website URL. Ask user to provide the source if not already specified.

2. Parse primary source — extract course data:

   • For CSV: Use Read tool to load file, parse columns
   • For PDF: Use Read tool (supports PDF extraction), parse text. Over 20 pages, read in sequential page-range chunks and merge parsed rows before validating completeness — see references/large-catalog-handling.md §1
   • For website: Use WebFetch tool to retrieve HTML, parse course listings

   Look for: course codes (identity — split into SubjectAbbreviation + CourseNumber, see step 8/9), course names (Name), credit hours (Duration + DurationUnit), descriptions (Description), course types (Lecture/Lab/Exam/Drill/Recitation), prerequisites, corequisites, linked course indicators (IsLinkedOnly), delivery modality (if the same course code repeats with different Duration/DurationUnit — see references/large-catalog-handling.md §2).

3. Validate data completeness — pipe the parsed records into the completeness helper:

   python3 scripts/validate_completeness.py --required "CourseCode,Name,Duration" --recommended "Description,CourseType" <<'JSON'
   [ { "CourseCode": "...", "Name": "...", ... }, ... ]
   JSON

   The report gives present/missing counts per field. Optional fields (prerequisites, corequisites, linked courses) are not checked — their absence never blocks progress.

   If any required field shows missing > 0, report error and stop. If recommended fields are missing, note the gaps and continue to Phase 2.

Phase 2 — Identify Gaps and Request Additional Sources

4. Report data quality — print a summary table showing which fields are present and which are missing. Example:

   Data Quality Report:
   • Course codes: 120/120 OK
   • Course names: 120/120 OK
   • Credit hours: 120/120 OK
   • Descriptions: 115/120 (5 missing)
   • Course types: 118/120 (2 missing)

5. Request supplementary sources — if key fields are missing (descriptions, types), ask user: "Do you have additional sources — a website, department document, or mapping file?"

   • If yes: receive additional source, parse it, and merge data with primary source
   • If no: offer to create courses with placeholder values or stop and wait for complete data

   Repeat this step until all gaps are filled or user confirms to proceed with incomplete data.

   If subject-to-Business-Account mapping is ambiguous, resolve every ambiguous case in one pass (source doc + live public catalog + course level/number patterns) and present them together, not one round at a time (see references/gotchas.md).

6. Detect multi-component courses — pipe the course codes into the lecture/lab pairing helper:

   python3 scripts/detect_linked_courses.py --suffixes "L,LAB" <<'JSON'
   [ "BIO-201", "BIO-201L", ... ]
   JSON

   Each {"lecture", "lab"} pair returned is a candidate multi-component course.

   • If any pairs are found, pause and explain: "I found X multi-component (lecture + lab) courses. Structure them as (1) separate standalone courses, or (2) linked courses (IsLinkedOnly flag, students must take both)?"

   If user selects "linked courses", request a mapping file or confirm the naming pattern is reliable.

   Detect modality duplicates — group parsed rows by course code. Same code, different Duration/DurationUnit/Description = a modality variant (e.g. Online vs. On-Campus), not a duplicate — never silently dedupe/merge. Resolve the representation once per run, not per-course. See references/large-catalog-handling.md §2.

Phase 3 — Batch Create Learning and LearningCourse Records

7. Prepare batch creation plan — pipe the parsed course records into the batching helper. The Composite API accepts up to 200 records/call, but an agent's tool-call payload has a smaller practical ceiling (~56KB/150 records). Use --batch-size 25 when executing as an agent (raise toward 200 only if the calling environment confirms it can handle larger payloads):

   python3 scripts/batch_courses.py --batch-size 25 <<'JSON'
   [ { "Name": "...", ... }, ... ]
   JSON

   Each element of the returned array is one Composite API call's worth of records. Print: "Starting course catalog import. I'll use Composite API to create records in batches. Total: X courses."

   If a batch still won't inline through a healthy dispatch transport, re-run with a smaller --batch-size (e.g. 10) instead of switching transport. Track dispatch vs. fallback counts and report both in the Phase 5 summary.

   Over 200 courses (8+ batches): disclose total batch count upfront, print the Batch N complete line after every batch (not just at the end), and — only if the surface exposes a parallel-task/subagent tool — follow the fan-out rules in references/large-catalog-handling.md §3-4. Never assume that capability exists; default to sequential batches.

8. Check for existing courses — before creating Batch 1, scope the check to this batch's own course codes, never a global fetch (an unrelated catalog entry already in the org is not a conflict).

   Split this batch's course codes first (scripts/split_course_code.py) — key the check on those identity fields, never CipCode (many orgs leave it null — see references/gotchas.md).

   Headless-360 note: SOQL /query is routable via the headless-360 dispatch tool (confirmed by live test) — query LearningCourse, matching each split {SubjectAbbreviation, CourseNumber} pair exactly (SOQL has no tuple IN, so OR the pairs): SELECT Id, LearningId, Name, SubjectAbbreviation, CourseNumber FROM LearningCourse WHERE (SubjectAbbreviation = 'BIO' AND CourseNumber = '201') OR .... If Ids are unknown and a SOQL lookup isn't available, ask the user to supply them (or a prior import manifest). Never do this via Bash/CLI — see references/gotchas.md.

   If any rows come back (i.e. a SubjectAbbreviation+CourseNumber pair in this batch already exists in the org), pause and explain conflict:

   • Show existing course details vs. new import data side-by-side
   • Check if existing courses have dependencies (course offerings, enrollments)
   • Offer options: (1) Update existing courses, (2) Skip existing courses, (3) Cancel and review manually

   Wait for user decision before proceeding.

9. Create Batch 1 — two-step process for each course:

   Field placement (verified via describe): CipCode, Duration, DurationUnit, ProviderId are createable on Learning ONLY — ProviderId on the child fails with INVALID_FIELD_FOR_INSERT_UPDATE. LearningCourse takes Name, LearningId, CourseType, IsLinkedOnly, SubjectAbbreviation, CourseNumber, Description, CourseLevelDescription, FieldOfStudy — it inherits the account link via LearningId. Confirm with describe before building the body.

   CipCode is opt-in. Federal CIP taxonomy field, not the course's identity — leave unset unless the source supplies a real CIP code and the user confirms.

   Step 1: Create Learning (parent) records via Composite API:

   POST /services/data/vXX.X/composite/sobjects
   {
     "allOrNone": false,
     "records": [
       {
         "attributes": {"type": "Learning", "referenceId": "learning1"},
         "Name": "<course-name>",
         "Type": "LearningCourse",
         "Description": "<description>",
         "ProviderId": "<account-id>",
         "Duration": <hours>,
         "DurationUnit": "Credit Hours",
         "IsActive": true
       },
       ...
     ]
   }

   Capture each returned Id (order matches the records array) to use as LearningId in Step 2.

   Step 2: Create LearningCourse (child) records via Composite API:

   POST /services/data/vXX.X/composite/sobjects
   {
     "allOrNone": false,
     "records": [
       {
         "attributes": {"type": "LearningCourse"},
         "Name": "<course-record-name>",
         "LearningId": "<learning-id-from-step-1>",
         "SubjectAbbreviation": "<subject-prefix>",
         "CourseNumber": "<course-number>",
         "CourseType": "<parsed-course-type>",
         "IsLinkedOnly": false
       },
       ...
     ]
   }

   Set CourseType from the parsed/mapped course type for that record (see the Gotchas table for value mapping) — do not hard-code it. A lab record must get CourseType: "Lab", not "Lecture".

   CourseType picklist values (verified): Lecture, Lab, Exam, Drill, Recitation.

   restrictedPicklist: true means only today's values are valid to submit, not that the set is closed. If a source value (e.g. DurationUnit = "Program Points") isn't in describe's list, don't silently drop/remap it — offer adding the picklist value via metadata alongside remapping or a placeholder (see references/gotchas.md).

   Print progress: Batch 1 complete — X Learning + X LearningCourse records created.

10. Handle multi-component courses — if user selected "linked courses" in step 6:

    • Create lecture courses with CourseType: "Lecture", IsLinkedOnly = false
    • Create lab courses with CourseType: "Lab", IsLinkedOnly = true
    • Use the detect_linked_courses.py pairs (or a user-supplied mapping file) to determine which labs are linked-only
    • Print: X linked lab courses created (IsLinkedOnly = true)

11. Repeat for remaining batches — continue steps 8-10 for Batch 2, Batch 3, etc. until all courses created.