Arize Evaluator
github/awesome-copilot
Handles LLM-as-judge evaluation workflows on Arize including creating/updating evaluators, running evaluations on spans or experiments, managing tasks, trigger-run operations, column mapping, and…
A skill your agent uses when reasoning about how a SOSP submission is evaluated — double-blind HotCRP reviewing, staged review rounds, PC-chair-audited conflicts, the pre-PC-meeting author response…
MITAuto-check passed
Install Sosp Review Process
$ npx skills add brycewang-stanford/Awesome-Journal-Skills --skill sosp-review-process -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install brycewang-stanford/Awesome-Journal-Skills sosp-review-process --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/brycewang-stanford/Awesome-Journal-Skills.git skills-src && mkdir -p .claude/skills && cp -r skills-src/SOSP-Skills/skills/sosp-review-process .claude/skills/sosp-review-process && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "sosp-review-process" agent skill from https://github.com/brycewang-stanford/Awesome-Journal-Skills/tree/main/SOSP-Skills/skills/sosp-review-process into .claude/skills/sosp-review-process/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "sosp-review-process", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/brycewang-stanford/Awesome-Journal-Skills/tree/main/SOSP-Skills/skills/sosp-review-processType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add brycewang-stanford/Awesome-Journal-Skills --skill sosp-review-process -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install brycewang-stanford/Awesome-Journal-Skills sosp-review-process --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/brycewang-stanford/Awesome-Journal-Skills.git skills-src && mkdir -p .agents/skills && cp -r skills-src/SOSP-Skills/skills/sosp-review-process .agents/skills/sosp-review-process && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "sosp-review-process" agent skill from https://github.com/brycewang-stanford/Awesome-Journal-Skills/tree/main/SOSP-Skills/skills/sosp-review-process into .agents/skills/sosp-review-process/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "sosp-review-process", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add brycewang-stanford/Awesome-Journal-Skills --skill sosp-review-process -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install brycewang-stanford/Awesome-Journal-Skills sosp-review-process --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/brycewang-stanford/Awesome-Journal-Skills.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/SOSP-Skills/skills/sosp-review-process .cursor/skills/sosp-review-process && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "sosp-review-process" agent skill from https://github.com/brycewang-stanford/Awesome-Journal-Skills/tree/main/SOSP-Skills/skills/sosp-review-process into .cursor/skills/sosp-review-process/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "sosp-review-process", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/brycewang-stanford/Awesome-Journal-Skills.git --path SOSP-Skills/skills/sosp-review-process--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add brycewang-stanford/Awesome-Journal-Skills --skill sosp-review-process -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install brycewang-stanford/Awesome-Journal-Skills sosp-review-process --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/brycewang-stanford/Awesome-Journal-Skills.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/SOSP-Skills/skills/sosp-review-process .gemini/skills/sosp-review-process && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "sosp-review-process" agent skill from https://github.com/brycewang-stanford/Awesome-Journal-Skills/tree/main/SOSP-Skills/skills/sosp-review-process into .gemini/skills/sosp-review-process/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "sosp-review-process", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install brycewang-stanford/Awesome-Journal-Skills sosp-review-processInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add brycewang-stanford/Awesome-Journal-Skills --skill sosp-review-process -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/brycewang-stanford/Awesome-Journal-Skills.git skills-src && mkdir -p .github/skills && cp -r skills-src/SOSP-Skills/skills/sosp-review-process .github/skills/sosp-review-process && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "sosp-review-process" agent skill from https://github.com/brycewang-stanford/Awesome-Journal-Skills/tree/main/SOSP-Skills/skills/sosp-review-process into .github/skills/sosp-review-process/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "sosp-review-process", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add brycewang-stanford/Awesome-Journal-Skills --skill sosp-review-process -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install brycewang-stanford/Awesome-Journal-Skills sosp-review-process --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/brycewang-stanford/Awesome-Journal-Skills.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/SOSP-Skills/skills/sosp-review-process .opencode/skills/sosp-review-process && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "sosp-review-process" agent skill from https://github.com/brycewang-stanford/Awesome-Journal-Skills/tree/main/SOSP-Skills/skills/sosp-review-process into .opencode/skills/sosp-review-process/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "sosp-review-process", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
Facts
- Skill name
sosp-review-process- GitHub stars
- 1.2k
- Token cost
- ~1.3k tokens
- SKILL.md length
- 561 words
- Files
- 1
- Skills in repo
- 2,387
- Repo updated
- First seen
- Licence
- MIT
At a glance
A skill your agent uses when reasoning about how a SOSP submission is evaluated — double-blind HotCRP reviewing, staged review rounds, PC-chair-audited conflicts, the pre-PC-meeting author response…
- Works in 3 steps: A rejected paper with one strong… → Reviews written for the meeting… → Because SOSP and its sibling OSDI both…
- Reasoning about how a SOSP submission is evaluated — double-blind HotCRP reviewing
- SKILL.md covers The pipeline, Reading SOSP reviews, Confidentiality and integrity… and Calibrating expectations, plus 1 more section
- Instructions only: no scripts, shell commands, URLs or credentials in SKILL.md
What it does
Sosp Review Process is an agent skill from brycewang-stanford/Awesome-Journal-Skills. Use when reasoning about how a SOSP submission is evaluated — double-blind HotCRP reviewing, staged review rounds, PC-chair-audited conflicts, the pre-PC-meeting author response, the single-track PC meeting where every decision is argued live, notification, and shepherded acceptance outcomes.
Its SKILL.md is about 1.3k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.
The repository describes itself as: Journal-specific Claude Code/Codex skill packs covering mainstream journals — AER, QJE, Nature, Cell, 管理世界, 经济研究 & 200+ more — your fast track to getting published. ｜ 覆盖主流期刊的… The licence is MIT.
When your agent uses it
- Reasoning about how a SOSP submission is evaluated — double-blind HotCRP reviewing
- Staged review rounds
- PC-chair-audited conflicts
- The pre-PC-meeting author response
Example prompts
- “/sosp-review-process”
Workflow steps
3 steps, taken from the first numbered list in SKILL.md.
- A rejected paper with one strong champion review is often a revise-and-retarget
- Reviews written for the meeting compress; a terse negative review may encode a
- Because SOSP and its sibling OSDI both now run annually, the cost of a rejection
What it can do on your machine
Read from SKILL.md and the folder at commit 932eb23. It shows what the files ask for, not the result of running them.
Tool permissions
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Runs code
No scripts in the folder and no shell commands in SKILL.md.
From the folder's file list and the shell code blocks in SKILL.md.
Network
No URLs in SKILL.md.
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Context cost
Sosp Review Process loads about 1.3k tokens when it runs. Until then it costs about 78 tokens; SKILL.md has 561 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
Safety
Auto-check passed
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
SKILL.md
The full file from brycewang-stanford/Awesome-Journal-Skills at commit 932eb23, republished under its MIT licence (© brycewang-stanford). 561 words, ~1,259 tokens.
Download SKILL.mdSave it as
.claude/skills/sosp-review-process/SKILL.md (or your agent's skills folder).- name
- sosp-review-process
- description
- Use when reasoning about how a SOSP submission is evaluated — double-blind HotCRP reviewing, staged review rounds, PC-chair-audited conflicts, the pre-PC-meeting author response, the single-track PC meeting where every decision is argued live, notification, and shepherded acceptance outcomes.
SOSP Review Process
Use this to model what happens to a SOSP paper between the April upload and the July
decision, and to time author-side actions to the phases that actually move outcomes.
Anchors below are from the 2026-cycle CFP and the sosp26.hotcrp.com schedule
(checked 2026-07-08): submission April 2026, author response ahead of the PC meeting,
notification July 3, 2026.
The pipeline
| Phase | Who acts | What authors can do |
|---|---|---|
| Conflict screening (from abstract registration) | PC chairs audit declared conflicts, add or remove entries | Declare defensibly; chairs treat strategic conflicts as misconduct-adjacent |
| Early reviewing | A subset of the PC reads every paper | Nothing — the paper speaks alone |
| Later-round reviewing | Papers still alive draw additional, often more senior, reviews | Nothing; round structure and counts vary by year (待核实 per cycle) |
| Author response | Authors correct factual errors, answer posed questions | The one sanctioned intervention (see sosp-author-response) |
| PC meeting | Full committee discusses candidates live, single track | Nothing — but the response was written for this room |
| Decision + shepherd assignment | Chairs notify July 3; accepted papers get a shepherding PC member | Camera-ready planning begins (see sosp-camera-ready) |
Two structural facts do most of the explanatory work. First, reviewing is staged:
a paper that reads as unsound in the first pass never reaches the reviewers most
likely to appreciate a subtle contribution, so the opening pages must survive a
skeptical fast read (see sosp-writing-style). Second, decisions are made in a
meeting, not by score averaging: one convinced advocate who can defend the paper
against live objections beats four mildly positive scores.
Reading SOSP reviews
Systems reviews concentrate their weight in predictable places:
- Soundness attacks ("the evaluation doesn't isolate the claimed mechanism", "this workload can't show the tail behavior claimed") are the score-drivers. Address these first in any response.
- Positioning complaints ("how is this different from X at OSDI two years ago")
signal a related-work gap the meeting will replay (see
sosp-related-work). - Taste judgments ("incremental", "not SOSP-worthy") rarely change through argument; they change when an advocate reframes the paper's principle at the meeting — which your response can quietly arm with one crisp sentence.
Post-review triage sheet (fill before drafting anything)
For each review R1..Rn:
overall lean: champion / positive / neutral / negative / kill
score-driving issue: <one sentence>
type: factual error | missed content | posed question | judgment
response-eligible?: yes (types 1-3) / no (type 4)
Meeting risk: <the objection an advocate could not answer alone>Show full SKILL.md (217 more words)Show less
Confidentiality and integrity edges
- Reviews and PC discussion are confidential; reviewers are anonymous forever, and guessing at reviewer identity in the response reads badly (reviews are anonymous in both directions).
- Double-blind holds until notification. Do not advertise the submission where PC members will see it; the CFP's anonymization duty does not lapse after upload.
- HotCRP is the channel of record. Deadline-adjacent questions go to the PC chairs through it, not to PC members you happen to know — that is a conflict violation in a culture that audits conflicts.
Calibrating expectations
SOSP is a low-acceptance venue where most submissions are competent; rejections
routinely carry reviews that would be accepts elsewhere. Per-year submission counts
and acceptance rates are published with each program (待核实 for 2026 until the
program appears; cross-check dl.acm.org/conference/sosp and dblp for history).
Practical consequences of the meeting-centric model:
- A rejected paper with one strong champion review is often a revise-and-retarget candidate for the next systems deadline, not a redesign candidate.
- Reviews written for the meeting compress; a terse negative review may encode a long live discussion. Mine every sentence — each one survived an argument.
- Because SOSP and its sibling OSDI both now run annually, the cost of a rejection
is measured in months, not years (see
sosp-workflowfor the retargeting map).
Output format
[Phase] where the paper sits in the pipeline now
[Review map] per-review lean + score-driving issue + response eligibility
[Meeting risk] the one live objection to defuse
[Action] respond / wait / prepare retarget
[Integrity check] anonymity holds? all contact via HotCRP?© brycewang-stanford, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Files
Just SKILL.md in SOSP-Skills/skills/sosp-review-process of brycewang-stanford/Awesome-Journal-Skills.
Open the folder on GitHubat commit 932eb23
Compare with similar skills
Sosp Review Process next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Sosp Review Process this skillbrycewang-stanford/Awesome-Journal-Skills | 1.2k | — | ~1.3k | Automated safety check: Pass | MIT | |
| Arize Evaluatorgithub/awesome-copilot | 40k | 1 repos | ~8.1k | Automated safety check: Notes | MIT | |
| LLM Evaluationdavila7/claude-code-templates | 33k | 12 repos | ~3.5k | Automated safety check: Pass | MIT | |
| Agent Evaluationsickn33/agentic-awesome-skills | 47k | 1 repos | ~2k | Automated safety check: Pass | MIT | |
| Deepseek Reasonruvnet/ruflo | 74k | — | ~626 | Automated safety check: Notes | MIT | |
| EvaluatorsArize-ai/phoenix | 12k | — | ~1.7k | Automated safety check: Pass | Custom licence |
Similar skills
- Official
LLM Evaluation
davila7/claude-code-templates
Master comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing.
Agent Evaluation
sickn33/agentic-awesome-skills
Evaluate agent behavior with versioned cases and explicit verifiers.
Deepseek Reason
ruvnet/ruflo
Reasoning-mode completion against DeepSeek's deepseek-reasoner model (R1) via /v1/chat/completions.
Evaluators
Arize-ai/phoenix
Author or refine a Phoenix evaluator — code or LLM-as-a-judge — that scores a run's output.
Agent Evaluation Reporting
sickn33/agentic-awesome-skills
A skill your agent uses when summarizing agent evaluations where autonomous, assisted, failed, timed-out, or invalid outcomes must remain distinct and comparable.
More from brycewang-stanford/Awesome-Journal-Skills
All 2,387 skills in this repoAaag Data Analysis
brycewang-stanford/Awesome-Journal-Skills
A skill your agent uses when running and reporting the analysis for an Annals of the American Association of Geographers manuscript — spatial statistics and modeling, remote-sensing accuracy, or…
Aaag Literature Positioning
brycewang-stanford/Awesome-Journal-Skills
A skill your agent uses when positioning an Annals of the American Association of Geographers manuscript in the literature — engaging geographic scholarship across the relevant area and the…
Aaag Rebuttal
brycewang-stanford/Awesome-Journal-Skills
A skill your agent uses when responding to an Annals of the American Association of Geographers decision letter (major/minor revision) — building a point-by-point response to the subject editor and…
Aaag Research Design
brycewang-stanford/Awesome-Journal-Skills
A skill your agent uses when defending the research design of an Annals of the American Association of Geographers manuscript — spatial/quantitative analysis and GIScience, remote-sensing and…
Aaag Review Process
brycewang-stanford/Awesome-Journal-Skills
A skill your agent uses when you need to understand how the Annals of the American Association of Geographers evaluates a manuscript — double-anonymous review routed through a subject editor by…
Aaag Submission
brycewang-stanford/Awesome-Journal-Skills
A skill your agent uses when running the final pre-submission preflight for the Annals of the American Association of Geographers via ScholarOne Manuscripts — area/article-type selection…
Questions about Sosp Review Process
What does Sosp Review Process do?
A skill your agent uses when reasoning about how a SOSP submission is evaluated — double-blind HotCRP reviewing, staged review rounds, PC-chair-audited conflicts, the pre-PC-meeting author response…. Sosp Review Process is an agent skill from brycewang-stanford/Awesome-Journal-Skills. Use when reasoning about how a SOSP submission is evaluated — double-blind HotCRP reviewing, staged review rounds, PC-chair-audited conflicts, the pre-PC-meeting author response, the single-track PC meeting where every decision is argued live, notification, and shepherded acceptance outcomes.
When should I use Sosp Review Process?
Sosp Review Process fits situations like: reasoning about how a SOSP submission is evaluated — double-blind HotCRP reviewing; staged review rounds; PC-chair-audited conflicts; the pre-PC-meeting author response.
How do I install Sosp Review Process in Claude Code?
Run `npx skills add brycewang-stanford/Awesome-Journal-Skills --skill sosp-review-process -a claude-code`. Or copy the skill folder (SOSP-Skills/skills/sosp-review-process in brycewang-stanford/Awesome-Journal-Skills) into .claude/skills/sosp-review-process in your project. Claude Code loads it when a task matches its description.
How do I install Sosp Review Process in Codex?
Run `npx skills add brycewang-stanford/Awesome-Journal-Skills --skill sosp-review-process -a codex`. Or copy the skill folder (SOSP-Skills/skills/sosp-review-process in brycewang-stanford/Awesome-Journal-Skills) into .agents/skills/sosp-review-process in your project. Codex loads it when a task matches its description.
Can I use Sosp Review Process in Cursor, Gemini CLI or GitHub Copilot?
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add brycewang-stanford/Awesome-Journal-Skills --skill sosp-review-process -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/sosp-review-process, .gemini/skills/sosp-review-process, .github/skills/sosp-review-process and .opencode/skills/sosp-review-process in your project.
What does Sosp Review Process need to run?
SKILL.md names no scripts, command-line tools or credentials: Sosp Review Process is instructions for the agent only.
Does Sosp Review Process access the network?
SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.
Is Sosp Review Process safe to install?
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.
What licence does Sosp Review Process use?
Sosp Review Process is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.
How many tokens does Sosp Review Process use?
About 1.3k tokens (SKILL.md is roughly 5k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
What are the alternatives to Sosp Review Process?
Skills that share tags, products or a category with Sosp Review Process: Arize Evaluator (github/awesome-copilot, 40k stars), LLM Evaluation (davila7/claude-code-templates, 33k stars), Agent Evaluation (sickn33/agentic-awesome-skills, 47k stars) and Deepseek Reason (ruvnet/ruflo, 74k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
Who maintains Sosp Review Process?
brycewang-stanford (a GitHub user) maintains it in brycewang-stanford/Awesome-Journal-Skills, which has 1,231 GitHub stars. The repository holds 2,387 skills in this directory. The repository was last updated on September 27, 2026.
Source: brycewang-stanford/Awesome-Journal-Skills on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.