Claude Code, Skill
Skill Eval Harness
Skill Eval Harness is a Python CLI that measures the causal lift of an Agent Skill: it runs the same case, model, and repetition with and without the skill, validates that exact experimental identity, then reports what changed, what passed, and whether the eval leaked its own
Compatibility: Not stated.
Grade C
One Medium finding. Read the findings before installing.
Not a guarantee. A hygiene check of the source at the commit shown, not a security audit.
Rescanned at commit 0cc7392473c3, unchanged since the check below.
SECRETS_PATH_CREDENTIAL
Why this grade
- SECRETS_PATH_CREDENTIAL Medium
Mentions a credential store path (~/.ssh, id_rsa, .npmrc, .aws/credentials, credentials.json, .netrc).
run_trigger_matrix.py, line 147
CLAUDE_PORTABLE_AUTH_FILES = (".credentials.json",)
Any High, or two or more Medium findings, give D. One Medium gives C. Low findings or a security Note give B. Each code names a finding behind the letter.
Grades are a hygiene check of the source at the commit shown, on the date checked, not a security audit or a guarantee. The free directory grades are automated. Before release, a PlugVet engineer reviews every paid report and any grade we hold back, including all F grades, against the source code and the evidence we quote.
What was checked
Plain notes from the hygiene checks.
Permissions
Concern
can delete files.
PERMISSION_DELETE_FILES
run_trigger_matrix.py, line 590
shutil.rmtree(self._config_dir(workspace), ignore_errors=True)
run_trigger_matrix.py, line 729
shutil.rmtree(self._codex_home(workspace), ignore_errors=True)
run_trigger_matrix.py, line 831
shutil.rmtree(self._pi_home(workspace), ignore_errors=True)
Show more
examples/adewale-workspace/run_pi_smoke.py, line 118
shutil.rmtree(dest)
examples/adewale-workspace/run_pi_smoke.py, line 129
shutil.rmtree(d)
examples/adewale-workspace/run_pi_smoke.py, line 191
shutil.rmtree(staging, ignore_errors=True)
examples/adewale-workspace/run_pi_smoke.py, line 271
proc = subprocess.run(
scripts/check_consumer_manifests.py, line 42
proc = subprocess.run(
scripts/check_installed_wheel.py, line 77
proc = subprocess.run(argv, cwd=cwd, env=env, capture_output=True, text=True, check=False)
scripts/check_test_collection_parity.py, line 47
proc = subprocess.run(
scripts/check_test_collection_parity.py, line 65
proc = subprocess.run(
Install and update
Pass
No install-time script found, and dependencies are pinned or absent.
Network
Pass
No network call found.
Secrets
Concern
reads a credential or environment file.
SECRETS_PATH_CREDENTIAL SECRETS_PATH_DOTENV
run_trigger_matrix.py, line 147
CLAUDE_PORTABLE_AUTH_FILES = (".credentials.json",)
run_trigger_matrix.py, line 151
".vibe-home/.env",
run_trigger_matrix.py, line 445
codex_source / "auth.json", codex_source / "config.toml", vibe_source / ".env",
Show more
LESSONS_LEARNED.md, line 545
`.env` files, `.agents/skills`, sessions, extensions, additional directories, user policy, and
LESSONS_LEARNED.md, line 601
parsed `--skip-trust`; pre-setting trust therefore enabled an ancestor `.gemini/.env` even though
LESSONS_LEARNED.md, line 602
the isolated settings ignored ordinary `.env` files.
README.md, line 526
- **Mistral Vibe**: use `run-agent --agent vibe`, `judge --judge-backend vibe`, and `skill-trigger-matrix --agent vibe`. The harness isolates `VIBE_HOME`, passes `--model` as `VIBE
docs/agent-backend-interface-spec.md, line 148
- Use a fresh `GEMINI_CLI_HOME` outside the model workdir. A valid configured `security.auth.selectedType` wins; environment-selector precedence applies only when settings do not c
Agent instructions
Pass
No instruction to ignore earlier instructions or to hide a step.
Obfuscation
Note
a file was over the read cap and was not fully read.
OBFUSCATION_FILE_OVER_READ_CAP
skill_benchmark.py, line 1
#!/usr/bin/env python3
Licence
Pass
A licence file is present (LICENSE).
Maintainer activity
Pass
The latest commit is within the last 365 days.
Note
Hygiene check of commit 0cc7392473c3. can delete files. reads a credential or environment file. a file was over the read cap and was not fully read.
Date and source
Date checked:
github.com
Commit graded: 0cc7392473c3d8e2e7de7412239ba39229101604
This link goes to the original host. PlugVet does not host or check what you download from there, and installing it is your own decision and risk.