Skill Evaluation Engine

Skill Evaluation Engine

The Skill Evaluation Engine (skill-eval.js) is an intelligent hook designed to analyze every prompt and automatically suggest relevant context-specific "Skills" to Claude. By matching user intent, file paths, and keywords against a predefined ruleset, the engine ensures that Claude always has the right domain knowledge active without requiring manual activation.

Core Functionality

The engine processes the user's prompt and current working context through several evaluation layers:

• Keyword & Pattern Matching: Detects specific terms or regex patterns that indicate a need for specialized knowledge (e.g., "GraphQL", "test", "schema").
• Path-Based Activation: Automatically triggers skills when specific file extensions (e.g., .tsx, .gql) or directory structures are mentioned or targeted for modification.
• Intent Detection: Uses regex patterns to identify the type of work being performed, such as "refactoring," "documentation," or "debugging."
• Scoring System: Each match contributes to a confidence score. Only skills that meet the required threshold are recommended to the agent.

Configuration (skill-rules.json)

The behavior of the engine is governed by a JSON configuration file. This allows you to map your repository's unique directory structure and naming conventions to specific skills.

Skill Rule Schema

Each skill entry in the ruleset supports the following properties:

| Property | Type | Description | | :--- | :--- | :--- | | triggers.keywords | string[] | Plain text strings that trigger the skill. | | triggers.keywordPatterns | string[] | Regex patterns for more complex keyword matching. | | triggers.intentPatterns | string[] | Regex patterns focused on the user's goal. | | excludePatterns | string[] | Patterns that, if matched, prevent the skill from activating (useful for avoiding false positives). | | priority | number | A weighting factor (default: 5) to resolve conflicts or prioritize certain skills. |

Usage Example

Below is an example of how to configure a skill for a React UI library:

{
  "skills": {
    "core-components": {
      "triggers": {
        "keywords": ["button", "modal", "design system", "theme"],
        "keywordPatterns": ["UI\\s?Components"],
        "intentPatterns": ["create a new component", "update the styles"]
      },
      "directoryMappings": {
        "src/components/core": "core-components"
      },
      "priority": 8
    },
    "testing-patterns": {
      "triggers": {
        "keywords": ["jest", "test", "spec", "coverage"],
        "keywordPatterns": ["\\.test\\.(ts|tsx)$"]
      },
      "excludePatterns": ["manual test plan"],
      "priority": 10
    }
  },
  "scoring": {
    "keyword": 10,
    "keywordPattern": 15,
    "directoryMatch": 25
  }
}

Path & Directory Mapping

The engine includes robust path detection logic:

1. Explicit File Detection: It parses the prompt for filenames with known extensions.
2. Glob Pattern Matching: Supports standard glob patterns to associate skills with specific file types (e.g., **/*.graphql triggers the graphql-schema skill).
3. Directory Mapping: If a user mentions a path or is editing a file within a mapped directory (like src/services/api), the engine automatically associates the relevant skill.

Integration as a Hook

The engine is typically executed as a pre-prompt or on-render hook within the Claude Code environment. When triggered, it outputs a structured recommendation block that provides Claude with the context it needs:

[SKILL EVALUATION]
Based on your request, I've identified relevant domain knowledge:
- Skill: "testing-patterns" (Reason: matched pattern /\.test\.(ts|tsx)$/)
- Skill: "core-components" (Reason: matched directory "src/components/core")

Internal Logic & Scoring

While the engine is configurable, it follows a standard internal workflow:

1. Extraction: Pulls potential file paths and normalizes the prompt text.
2. Matching: Iterates through the rules defined in skill-rules.json.
3. Calculation: Sums scores based on the scoring configuration (e.g., a directory match might be weighted more heavily than a single keyword).
4. Thresholding: Filters out matches that do not meet the minimum confidence level required for a recommendation.