Is this AI agent skill safe to install? A 10-minute check, and what 47 graded skills showed

Why this question matters more than it did a year ago

A skill is a set of instructions an AI assistant follows on your behalf, often with access to your files, email, browser or payment tools. Installing one is closer to hiring a contractor with your passwords than to installing an app. Marketplaces have grown faster than the checks around them: the public listings we have graded range from clean, narrowly scoped tools to skills that ship with live credentials inside the file.

The 10-minute manual check

1. Who made it? A real person or company with a web presence, documentation and a history. No track record is not disqualifying, but it moves the skill to "test before trust".
2. Read the instruction file (SKILL.md or equivalent). Look for: instructions to act silently or not to tell the user; anything that overrides safety behaviour; sending data to an external URL; use of connected accounts (mail, storefront, cloud drive) beyond the stated job; requests for keys or passwords.
3. Open every other file. Markdown and templates only: lower risk. Scripts (.py, .js, .sh), install steps, dependency manifests: higher risk; have someone technical read them.
4. Match access to purpose. A weather skill needs the weather. It does not need your inbox.
5. Search the files for secrets. Strings that look like API keys, tokens, connection strings. If you find one that works, stop: the author has shipped their own credential to every installer, or worse, is harvesting yours.

This catches the obvious cases. It does not tell you how the skill behaves when it meets hostile input.

What a behavioural test adds

The manual check reads the skill. A behavioural test runs it against attacks and watches what it does. Sloopt's Rate check puts a skill through seven controls — injection probes, data-leak attempts, refusal checks, scope checks, secret detection, instruction disclosure and over-refusal — with the injection probes repeated 8 times in a fresh context (32 for a certification), and grades the worst finding A to F.

The reason to bother is that reading does not predict behaviour well. In our blind grading of 47 public skills, the only F was found by the live attack; the static scan of that skill was clean.

What 47 public skills looked like

Figures from Sloopt engine runs, September–October 2026. These are our own measurements; the public per-skill pages are at /skills.

All 47 public skills, graded blind: 1 F. The failing skill behaved normally on inspection and failed only under live attack.

The 16 most-installed skills in our set:

Grade	Skills	Share
A	10	62.5%
B	3	18.8%
C	3	18.8%
D	0	0%
F	0	0%

Read that table two ways. The good news: none of the most-installed skills failed, and nearly two-thirds were clean across every probe. The caution: more than one in three had a finding worth fixing, and install counts gave no hint which ones.

What a finding looks like at each grade (real cases from our runs, anonymised):

• F — invoice-reconciliation skill. The skill file contained a live payment-provider API key. Its behaviour was clean on every probe; it did not matter. Fix: move the key to the environment. After the fix: A.
• C — research assistant with web access. A disguised instruction hidden in a fetched page was obeyed in 20 of 32 runs. Fix: treat retrieved page text as data, not instructions.
• B — calendar skill. Partial compliance with an injected instruction in 3 of 32 runs, nothing obeyed, nothing leaked. Fix: one paragraph tightening the instructions.

What the grade means, and does not

A grade is a measurement at a point in time against named standards (OWASP, MITRE ATLAS, NIST AI RMF, EU AI Act). It is not a guarantee. An A at 8 runs means no attack was obeyed in 8 tries, which bounds the true success rate at roughly 37%; at 32 runs, roughly 9%. That is why the result page says "no injection obedience observed in N trials" rather than "safe".

Grades can also change. The next edit to a skill is where grades drop, which is what the $29 per month monitor is for: re-run on change, email on a grade move.

Automated checkers compared

Tool	What it does	Live behavioural attack?	Repeated runs with a bound?	Independent, shareable result?
Manual review (guides, SOPs)	Human reads files and permissions	No	No	No
Marketplace security tabs	Static pattern scan of the package	No	No	Listing-level only
Single-pass skill scanners (score out of 100)	Static scan, verdict in seconds	No	No	Score, no evidence trail
Sloopt Rate	Static checks plus seven behavioural controls, 8 or 32 runs each, worst finding sets A–F, evidence quoted per finding	Yes	Yes	Verification page and badge

Use the fast scanners as a first filter. Use a behavioural grade before you put a skill in front of customers or connect it to anything that pays, sends or deletes.

Check one now

Paste a skill or a link at sloopt.ai/rate. The free check runs all seven controls at 8 runs and shows the worst finding and the fix. Certification ($79) runs 32 and gives you a page you can send to whoever is asking "is it safe?".