{ "title": "Gate demo: the same gates, on planted defects and on clean code", "subtitle": "The paid skills do not just advise. They run a security scan and a correctness gate that block a run from being reported done. Here is each gate driven against a fixture with known defects and against the same code with the defects fixed.", "method": "For each gate, a fixture is copied into an isolated, git-initialised temporary directory and the real shipped CLI is driven start -> gate. The security scan and the file discovery only ever see the fixture. Nothing is mocked: these are the exact binaries that ship in the skill zips.", "reproduce": "node scripts/gate-demo/run-gates.js", "honesty": "Both controls are run on purpose. A gate that flagged everything would also flag the clean fixture, so the clean pass is as much a part of the proof as the defective catch. The runner grades itself against an answer key; if any expected outcome is missing the verdict below turns false.", "boundary": "A third fixture (blindspot) plants two real defects that are deliberately out of each gate's scope: a path traversal (not one of harden's pattern rules) and a boot-clean logic bug (verify checks syntax and crash-on-boot, not whether the math is right). The gates pass both, on purpose. This shows what each gate catches AND what it does not, so \"catches what it claims\" cannot be misread as \"catches everything\".", "classification": "Every planted security defect is tagged with its public CWE id (see the cwe field on each entry under fixtures.defective.planted and fixtures.blindspot.planted_missed). CWE is the vendor-neutral Common Weakness Enumeration catalogue at cwe.mitre.org. The point is to settle the \"are these strawman defects the vendor invented to be easy?\" question without asking anyone to trust us: the defects map to canonical weakness classes (CWE-798 hardcoded credentials, CWE-89 SQL injection, CWE-78 OS command injection, CWE-79 XSS, CWE-328 weak hash, CWE-338 weak PRNG), and each label is verifiable by reading the few-line fixture file against the public CWE definition. The blindspot path traversal is CWE-22, a famous class harden deliberately does not cover. The three correctness defects (syntax error, crash on boot, discount logic bug) carry no CWE because they are not security weaknesses.", "text_normalisation": "Two cosmetic substitutions are applied to the captured output so the saved file is stable and on-style: the random temp path becomes and the run id becomes , and em/en-dashes in prose fields (narrate, instructions, smoke note) become commas. Every structural field (state, finding text, file, line, tag, exit code, syntax flag, smoke result) is the verbatim CLI output.", "environment": { "node": "v24.14.0", "platform": "win32 x64" }, "fixtures": { "defective": { "description": "Eight small files with eleven planted defects: nine security defects that exercise all eight of harden's pattern rules (the two compound categories, XSS and weak hashing, each exercise both of their rules), plus a syntax error and a crash on boot.", "planted": [ { "file": "config.js", "line": 4, "defect": "Hardcoded API key", "cwe": "CWE-798", "caught_by": "harden:hardcoded-secret" }, { "file": "config.js", "line": 5, "defect": "Hardcoded password", "cwe": "CWE-798", "caught_by": "harden:hardcoded-secret" }, { "file": "db.js", "line": 7, "defect": "SQL built from user input by string concatenation", "cwe": "CWE-89", "caught_by": "harden:sql-concat" }, { "file": "exec.js", "line": 7, "defect": "Shell command built from user input", "cwe": "CWE-78", "caught_by": "harden:command-injection" }, { "file": "render.js", "line": 6, "defect": "Untrusted value assigned to innerHTML", "cwe": "CWE-79", "caught_by": "harden:xss-innerhtml" }, { "file": "render.js", "line": 10, "defect": "Untrusted value passed to React dangerouslySetInnerHTML", "cwe": "CWE-79", "caught_by": "harden:xss-react" }, { "file": "crypto.js", "line": 7, "defect": "MD5 used to hash a password", "cwe": "CWE-328", "caught_by": "harden:weak-crypto-md5" }, { "file": "crypto.js", "line": 11, "defect": "SHA1 used to sign a payload", "cwe": "CWE-328", "caught_by": "harden:weak-crypto-sha1" }, { "file": "crypto.js", "line": 15, "defect": "Math.random used to mint a session token", "cwe": "CWE-338", "caught_by": "harden:weak-rng-secret" }, { "file": "total.js", "line": 5, "defect": "Syntax error (malformed function signature)", "caught_by": "verify:syntax" }, { "file": "server.js", "line": 8, "defect": "Throws on startup (crash on boot)", "caught_by": "verify:smoke" } ] }, "clean": { "description": "The same eight files with every defect fixed: env-var config, parameterised SQL, execFile with an argument array, textContent and escaping React markup, SHA-256 and HMAC with a CSPRNG, a valid function, and a server that boots and stays up." }, "blindspot": { "description": "A third fixture of real defects that sit outside each gate's scope on purpose: a path traversal (harden has no pattern rule for it) and a boot-clean logic bug (verify checks syntax and crash-on-boot, not whether the result is correct). The gates are supposed to pass these, and they do. This bounds the claim.", "planted_missed": [ { "file": "lookup.js", "line": 12, "defect": "Path traversal: a user-controlled filename is joined to a base dir and read with no containment check", "cwe": "CWE-22", "missed_by": "harden", "why": "path traversal (CWE-22) is a real, canonical weakness, and it is deliberately NOT one of harden's eight pattern rules, so the static scan stays clean. The miss is the point: the scan is scoped, not a guarantee of no vulnerabilities." }, { "file": "pricing.js", "line": 8, "defect": "Logic bug: applyDiscount returns the discount amount, not the discounted price", "missed_by": "verify", "why": "verify checks syntax and crash-on-boot; the file parses and the server boots, so a wrong-but-valid computation passes. This is a correctness bug, not a security weakness, so it carries no CWE." } ] } }, "free_tier": { "skill": "godmode-lite", "skill_version": "2.3.1", "ships_commands": [ "start", "discover", "check", "test", "polish", "end", "status" ], "note": "The free tier ships a lighter protocol (see ships_commands). It has no harden (security scan) and no smoke-boot verify, so it runs neither gate shown here. Each gate's free_tier_probe records the free binary answering \"Unknown command\" when asked. The free tier does have check, test and polish, so the delta is specifically these two paid-only gates, not \"the free tier does nothing\"." }, "gates": [ { "skill": "godmode", "skill_version": "2.3.2", "gate": "harden", "label": "Security scan", "checks": "Six defect classes by static scan: hardcoded secrets, SQL built by string concatenation, command injection, innerHTML / dangerouslySetInnerHTML XSS, MD5 and SHA1 weak hashing, and Math.random used for a secret or token. The defective fixture exercises all eight underlying pattern rules (both XSS variants and both weak-hash algorithms included).", "free_tier_has_this": false, "free_tier_probe": { "command": "harden", "available": false, "response": "Unknown command: harden" }, "pass_state": "hardened_clean", "fail_state": "hardened_dirty", "defective": { "state": "hardened_dirty", "narrate": "Found 9 candidate findings across 8 files. Triage with judgment, heuristic flags, not all true positives.", "summary": { "scanned_files": 8, "total_findings": 9, "by_pattern": { "hardcoded-secret": 2, "weak-crypto-md5": 1, "weak-crypto-sha1": 1, "weak-rng-secret": 1, "sql-concat": 1, "command-injection": 1, "xss-innerhtml": 1, "xss-react": 1 }, "findings": [ { "file": "config.js", "line": 4, "tag": "hardcoded-secret", "text": "const apiKey = \"EXAMPLE_FAKE_api_key_1234567890abcdef\";" }, { "file": "config.js", "line": 5, "tag": "hardcoded-secret", "text": "const dbPassword = \"EXAMPLE_FAKE_password_not_a_real_value\";" }, { "file": "crypto.js", "line": 7, "tag": "weak-crypto-md5", "text": "return crypto.createHash(\"MD5\").update(pw).digest('hex');" }, { "file": "crypto.js", "line": 11, "tag": "weak-crypto-sha1", "text": "return crypto.createHash(\"SHA1\").update(payload).digest('hex');" }, { "file": "crypto.js", "line": 15, "tag": "weak-rng-secret", "text": "const r = Math.random().toString(36).slice(2);" }, { "file": "db.js", "line": 7, "tag": "sql-concat", "text": "const query = \"SELECT * FROM users WHERE id = \" + req.params.id;" }, { "file": "exec.js", "line": 7, "tag": "command-injection", "text": "exec(\"rm -rf \" + userPath, (err) => {" }, { "file": "render.js", "line": 6, "tag": "xss-innerhtml", "text": "element.innerHTML = userContent;" }, { "file": "render.js", "line": 10, "tag": "xss-react", "text": "return { dangerouslySetInnerHTML: { __html: userContent } };" } ] }, "envelope": { "state": "hardened_dirty", "run_id": "", "narrate": "Found 9 candidate findings across 8 files. Triage with judgment, heuristic flags, not all true positives.", "next": "alternatives", "instructions": "Layer 4 hardening complete. Triage data.findings with judgment, these are heuristic flags, not all true positives.\n\nSeverity ladder for each finding:\n1. Confirm the pattern is real (read the file, not just the matched line).\n2. If real and exploitable: fix immediately.\n3. If real but not exploitable in context: leave a brief justification comment.\n4. If a false positive: ignore.\n\nAfter triage, run `node bin/godmode alternatives` to begin Layer 5.", "data": { "scanned_files": 8, "total_findings": 9, "by_pattern": { "hardcoded-secret": 2, "weak-crypto-md5": 1, "weak-crypto-sha1": 1, "weak-rng-secret": 1, "sql-concat": 1, "command-injection": 1, "xss-innerhtml": 1, "xss-react": 1 }, "findings": [ { "file": "config.js", "line": 4, "tag": "hardcoded-secret", "text": "const apiKey = \"EXAMPLE_FAKE_api_key_1234567890abcdef\";" }, { "file": "config.js", "line": 5, "tag": "hardcoded-secret", "text": "const dbPassword = \"EXAMPLE_FAKE_password_not_a_real_value\";" }, { "file": "crypto.js", "line": 7, "tag": "weak-crypto-md5", "text": "return crypto.createHash(\"MD5\").update(pw).digest('hex');" }, { "file": "crypto.js", "line": 11, "tag": "weak-crypto-sha1", "text": "return crypto.createHash(\"SHA1\").update(payload).digest('hex');" }, { "file": "crypto.js", "line": 15, "tag": "weak-rng-secret", "text": "const r = Math.random().toString(36).slice(2);" }, { "file": "db.js", "line": 7, "tag": "sql-concat", "text": "const query = \"SELECT * FROM users WHERE id = \" + req.params.id;" }, { "file": "exec.js", "line": 7, "tag": "command-injection", "text": "exec(\"rm -rf \" + userPath, (err) => {" }, { "file": "render.js", "line": 6, "tag": "xss-innerhtml", "text": "element.innerHTML = userContent;" }, { "file": "render.js", "line": 10, "tag": "xss-react", "text": "return { dangerouslySetInnerHTML: { __html: userContent } };" } ] } } }, "clean": { "state": "hardened_clean", "narrate": "Layer 4 complete. No security/quality red flags across 8 files.", "summary": { "scanned_files": 8, "total_findings": 0, "by_pattern": {}, "findings": [] }, "envelope": { "state": "hardened_clean", "run_id": "", "narrate": "Layer 4 complete. No security/quality red flags across 8 files.", "next": "alternatives", "instructions": "Layer 4 hardening complete. Triage data.findings with judgment, these are heuristic flags, not all true positives.\n\nSeverity ladder for each finding:\n1. Confirm the pattern is real (read the file, not just the matched line).\n2. If real and exploitable: fix immediately.\n3. If real but not exploitable in context: leave a brief justification comment.\n4. If a false positive: ignore.\n\nAfter triage, run `node bin/godmode alternatives` to begin Layer 5.", "data": { "scanned_files": 8, "total_findings": 0, "by_pattern": {}, "findings": [] } } }, "blindspot": { "state": "hardened_clean", "narrate": "Layer 4 complete. No security/quality red flags across 4 files.", "summary": { "scanned_files": 4, "total_findings": 0, "by_pattern": {}, "findings": [] }, "envelope": { "state": "hardened_clean", "run_id": "", "narrate": "Layer 4 complete. No security/quality red flags across 4 files.", "next": "alternatives", "instructions": "Layer 4 hardening complete. Triage data.findings with judgment, these are heuristic flags, not all true positives.\n\nSeverity ladder for each finding:\n1. Confirm the pattern is real (read the file, not just the matched line).\n2. If real and exploitable: fix immediately.\n3. If real but not exploitable in context: leave a brief justification comment.\n4. If a false positive: ignore.\n\nAfter triage, run `node bin/godmode alternatives` to begin Layer 5.", "data": { "scanned_files": 4, "total_findings": 0, "by_pattern": {}, "findings": [] } } } }, { "skill": "godmode-plus", "skill_version": "2.7.1", "gate": "verify", "label": "Correctness gate", "checks": "Every changed source file is parsed with the language's own checker (node --check, py_compile, tsc --noEmit, ruby -c, gofmt). If a start script exists the app is booted for up to ten seconds to catch a crash-on-boot. A run cannot be reported done while either fails.", "free_tier_has_this": false, "free_tier_probe": { "command": "verify", "available": false, "response": "Unknown command: verify" }, "pass_state": "verified_pass", "fail_state": "verified_fail", "defective": { "state": "verified_fail", "narrate": "Verify FAILED. Syntax failures: total.js; smoke check failed. Loop back to Phase 2.", "summary": { "file_source": "recent", "files_checked": 7, "syntax_failures": [ "total.js" ], "syntax": [ { "file": "config.js", "tool": "node --check", "ok": true }, { "file": "crypto.js", "tool": "node --check", "ok": true }, { "file": "db.js", "tool": "node --check", "ok": true }, { "file": "exec.js", "tool": "node --check", "ok": true }, { "file": "render.js", "tool": "node --check", "ok": true }, { "file": "server.js", "tool": "node --check", "ok": true }, { "file": "total.js", "tool": "node --check", "ok": false } ], "smoke": { "applicable": true, "ok": false, "timed_out": false, "note": "Process exited with code 1 inside 10s, likely crashed." }, "changed_files": [ "config.js", "crypto.js", "db.js", "exec.js", "render.js", "server.js", "total.js" ] }, "envelope": { "state": "verified_fail", "run_id": "", "narrate": "Verify FAILED. Syntax failures: total.js; smoke check failed. Loop back to Phase 2.", "next": "check (fix the failures, then re-verify)", "instructions": "Verify failed. Read data.syntax for files that did not parse and data.smoke for app boot failures.\nLoop back to Phase 2 for the broken pieces only, do not re-run the full protocol.\nAfter fixing, run `node bin/godmode-plus check`, `test`, then `verify` again.", "data": { "file_source": "recent", "syntax": [ { "file": "config.js", "ext": ".js", "tool": "node --check", "ok": true, "exit_code": 0, "stderr": "" }, { "file": "crypto.js", "ext": ".js", "tool": "node --check", "ok": true, "exit_code": 0, "stderr": "" }, { "file": "db.js", "ext": ".js", "tool": "node --check", "ok": true, "exit_code": 0, "stderr": "" }, { "file": "exec.js", "ext": ".js", "tool": "node --check", "ok": true, "exit_code": 0, "stderr": "" }, { "file": "render.js", "ext": ".js", "tool": "node --check", "ok": true, "exit_code": 0, "stderr": "" }, { "file": "server.js", "ext": ".js", "tool": "node --check", "ok": true, "exit_code": 0, "stderr": "" }, { "file": "total.js", "ext": ".js", "tool": "node --check", "ok": false, "exit_code": 1, "stderr": "\\total.js:5\r\nfunction computeTotal(items {\r\n ^\r\n\r\nSyntaxError: Unexpected token '{'\r\n at wrapSafe (node:internal/modules/cjs/loader:1743:18)\r\n at checkSyntax (node:internal/main/check_syntax:76:3)\r\n\r\nNode.js v24.14.0\r\n" } ], "smoke": { "applicable": true, "script": "node server.js", "ok": false, "timed_out": false, "exit_code": 1, "stderr": "ppData\\Local\\Temp\\godmode-gate-demo-\\server.js:8\r\n throw new Error('Fatal: missingRequiredSetting is not configured, cannot boot');\r\n ^\r\n\r\nError: Fatal: missingRequiredSetting is not configured, cannot boot\r\n at Object. (\\server.js:8:9)\r\n at Module._compile (node:internal/modules/cjs/loader:1812:14)\r\n at Object..js (node:internal/modules/cjs/loader:1943:10)\r\n at Module.load (node:internal/modules/cjs/loader:1533:32)\r\n at Module._load (node:internal/modules/cjs/loader:1335:12)\r\n at wrapModuleLoad (node:internal/modules/cjs/loader:255:19)\r\n at Module.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:154:5)\r\n at node:internal/main/run_main_module:33:47\r\n\r\nNode.js v24.14.0\r\n", "note": "Process exited with code 1 inside 10s, likely crashed." }, "entry_points": [], "changed_files": [ "config.js", "crypto.js", "db.js", "exec.js", "render.js", "server.js", "total.js" ] } } }, "clean": { "state": "verified_pass", "narrate": "Phase 6 verified: 7 files syntax-checked (recent), smoke ok, 0 entry points listed.", "summary": { "file_source": "recent", "files_checked": 7, "syntax_failures": [], "syntax": [ { "file": "config.js", "tool": "node --check", "ok": true }, { "file": "crypto.js", "tool": "node --check", "ok": true }, { "file": "db.js", "tool": "node --check", "ok": true }, { "file": "exec.js", "tool": "node --check", "ok": true }, { "file": "render.js", "tool": "node --check", "ok": true }, { "file": "server.js", "tool": "node --check", "ok": true }, { "file": "total.js", "tool": "node --check", "ok": true } ], "smoke": { "applicable": true, "ok": true, "timed_out": true, "note": "Process still running after 10s (tree-killed), treated as ok (no immediate crash)." }, "changed_files": [ "config.js", "crypto.js", "db.js", "exec.js", "render.js", "server.js", "total.js" ] }, "envelope": { "state": "verified_pass", "run_id": "", "narrate": "Phase 6 verified: 7 files syntax-checked (recent), smoke ok, 0 entry points listed.", "next": "polish", "instructions": "Phase 7 of 7: Polish-Report.\n\nFinal pass. The application is verified working. Now leave it cleaner than you found it.\n\n1. Run `node bin/godmode-plus polish` to run the formatter / linter and write the run report.\n2. Manually trace the user flow listed in data.entry_points, confirm each new entry point does what the task asked.\n3. List any adjacent issues you noticed but did not fix.\n4. Then `node bin/godmode-plus end` for the final summary.", "data": { "file_source": "recent", "syntax": [ { "file": "config.js", "ext": ".js", "tool": "node --check", "ok": true, "exit_code": 0, "stderr": "" }, { "file": "crypto.js", "ext": ".js", "tool": "node --check", "ok": true, "exit_code": 0, "stderr": "" }, { "file": "db.js", "ext": ".js", "tool": "node --check", "ok": true, "exit_code": 0, "stderr": "" }, { "file": "exec.js", "ext": ".js", "tool": "node --check", "ok": true, "exit_code": 0, "stderr": "" }, { "file": "render.js", "ext": ".js", "tool": "node --check", "ok": true, "exit_code": 0, "stderr": "" }, { "file": "server.js", "ext": ".js", "tool": "node --check", "ok": true, "exit_code": 0, "stderr": "" }, { "file": "total.js", "ext": ".js", "tool": "node --check", "ok": true, "exit_code": 0, "stderr": "" } ], "smoke": { "applicable": true, "script": "node server.js", "ok": true, "timed_out": true, "exit_code": null, "stderr": "", "note": "Process still running after 10s (tree-killed), treated as ok (no immediate crash)." }, "entry_points": [], "changed_files": [ "config.js", "crypto.js", "db.js", "exec.js", "render.js", "server.js", "total.js" ] } } }, "blindspot": { "state": "verified_pass", "narrate": "Phase 6 verified: 3 files syntax-checked (recent), smoke ok, 0 entry points listed.", "summary": { "file_source": "recent", "files_checked": 3, "syntax_failures": [], "syntax": [ { "file": "lookup.js", "tool": "node --check", "ok": true }, { "file": "pricing.js", "tool": "node --check", "ok": true }, { "file": "server.js", "tool": "node --check", "ok": true } ], "smoke": { "applicable": true, "ok": true, "timed_out": true, "note": "Process still running after 10s (tree-killed), treated as ok (no immediate crash)." }, "changed_files": [ "lookup.js", "pricing.js", "server.js" ] }, "envelope": { "state": "verified_pass", "run_id": "", "narrate": "Phase 6 verified: 3 files syntax-checked (recent), smoke ok, 0 entry points listed.", "next": "polish", "instructions": "Phase 7 of 7: Polish-Report.\n\nFinal pass. The application is verified working. Now leave it cleaner than you found it.\n\n1. Run `node bin/godmode-plus polish` to run the formatter / linter and write the run report.\n2. Manually trace the user flow listed in data.entry_points, confirm each new entry point does what the task asked.\n3. List any adjacent issues you noticed but did not fix.\n4. Then `node bin/godmode-plus end` for the final summary.", "data": { "file_source": "recent", "syntax": [ { "file": "lookup.js", "ext": ".js", "tool": "node --check", "ok": true, "exit_code": 0, "stderr": "" }, { "file": "pricing.js", "ext": ".js", "tool": "node --check", "ok": true, "exit_code": 0, "stderr": "" }, { "file": "server.js", "ext": ".js", "tool": "node --check", "ok": true, "exit_code": 0, "stderr": "" } ], "smoke": { "applicable": true, "script": "node server.js", "ok": true, "timed_out": true, "exit_code": null, "stderr": "", "note": "Process still running after 10s (tree-killed), treated as ok (no immediate crash)." }, "entry_points": [], "changed_files": [ "lookup.js", "pricing.js", "server.js" ] } } } } ], "verdict": { "all_pass": true, "summary": { "total": 19, "passed": 19 }, "checks": [ { "kind": "reject", "check": "godmode harden: defective fixture is rejected", "expected": "hardened_dirty", "actual": "hardened_dirty", "pass": true }, { "kind": "pass", "check": "godmode harden: clean fixture passes", "expected": "hardened_clean", "actual": "hardened_clean", "pass": true }, { "kind": "reject", "check": "godmode-plus verify: defective fixture is rejected", "expected": "verified_fail", "actual": "verified_fail", "pass": true }, { "kind": "pass", "check": "godmode-plus verify: clean fixture passes", "expected": "verified_pass", "actual": "verified_pass", "pass": true }, { "kind": "catch", "check": "harden catches config.js:4 (hardcoded-secret)", "expected": "found at file:line", "actual": "found at file:line", "pass": true }, { "kind": "catch", "check": "harden catches config.js:5 (hardcoded-secret)", "expected": "found at file:line", "actual": "found at file:line", "pass": true }, { "kind": "catch", "check": "harden catches db.js:7 (sql-concat)", "expected": "found at file:line", "actual": "found at file:line", "pass": true }, { "kind": "catch", "check": "harden catches exec.js:7 (command-injection)", "expected": "found at file:line", "actual": "found at file:line", "pass": true }, { "kind": "catch", "check": "harden catches render.js:6 (xss-innerhtml)", "expected": "found at file:line", "actual": "found at file:line", "pass": true }, { "kind": "catch", "check": "harden catches render.js:10 (xss-react)", "expected": "found at file:line", "actual": "found at file:line", "pass": true }, { "kind": "catch", "check": "harden catches crypto.js:7 (weak-crypto-md5)", "expected": "found at file:line", "actual": "found at file:line", "pass": true }, { "kind": "catch", "check": "harden catches crypto.js:11 (weak-crypto-sha1)", "expected": "found at file:line", "actual": "found at file:line", "pass": true }, { "kind": "catch", "check": "harden catches crypto.js:15 (weak-rng-secret)", "expected": "found at file:line", "actual": "found at file:line", "pass": true }, { "kind": "catch", "check": "verify catches total.js (syntax)", "expected": "syntax failure", "actual": "syntax failure", "pass": true }, { "kind": "catch", "check": "verify catches server.js (crash on boot)", "expected": "smoke failure", "actual": "smoke failure", "pass": true }, { "kind": "boundary", "check": "boundary: godmode harden stays silent on the out-of-scope blindspot defect (scoped, not catch-all)", "expected": "hardened_clean", "actual": "hardened_clean", "pass": true }, { "kind": "boundary", "check": "boundary: godmode-plus verify stays silent on the out-of-scope blindspot defect (scoped, not catch-all)", "expected": "verified_pass", "actual": "verified_pass", "pass": true }, { "kind": "tier", "check": "free tier (godmode-lite) does not ship harden", "expected": "unavailable", "actual": "unavailable", "pass": true }, { "kind": "tier", "check": "free tier (godmode-lite) does not ship verify", "expected": "unavailable", "actual": "unavailable", "pass": true } ] } }