AI Agents

Hybrid Activity Classification: Keyword Rules First, LLM Fallback

By Omer DemirApr 2, 2026⏱ 5 min read👁 — reads⧉ — copies

AI disclosure: This article was moderated by AI.

Not every classification needs a language model. In the field service assistant, most service activities are unambiguous. “Replaced the roller guides” is a parts replacement. Sending every clear case to an LLM would add cost and latency, so the activity classifier has two tiers: deterministic keyword rules first, then an LLM when the rules are not confident.

The two tiers

1. Keyword rules run first. They are fast, free, and deterministic. For the large fraction of activities that match a clear rule, classification ends there.
2. The LLM fallback handles the ambiguous remainder. When the keyword pass cannot decide, the classifier asks Azure OpenAI to make the judgment.

Both tiers return the same shape: category, subcategory, confidence, and reasoning. Downstream code does not need to know which tier produced the answer.

The design works like a funnel. Obvious cases leave early, reserving model judgment for the uncertain remainder.

The taxonomy

The rules operate against a fixed taxonomy modeled as enums. It has three top-level categories, each with its own subcategories, and the types prevent a typo from reaching the index:

class ActivityCategory(str, Enum):
    PLANNED_MAINTENANCE = "planned_maintenance"
    CALLOUT = "callout"
    REPAIR = "repair"

class ActivitySubcategory(str, Enum):
    PREVENTIVE = "preventive"; SAFETY_INSPECTION = "safety_inspection"       # planned
    EMERGENCY = "emergency"; URGENT = "urgent"; BREAKDOWN = "breakdown"        # callout
    CORRECTIVE = "corrective"; COMPONENT_REPLACEMENT = "component_replacement" # repair
    WARRANTY = "warranty"; MODIFICATION = "modification"                      # ... and more

class ClassificationMethod(str, Enum):
    KEYWORD_RULE = "keyword_rule"
    LLM = "llm"
    MANUAL_REVIEW = "manual_review"

The Pydantic result model records both the label and how the classifier reached it. The method field supports later audits of each tier:

class ActivityClassification(BaseModel):
    category: ActivityCategory
    subcategory: ActivitySubcategory
    confidence: float
    method: ClassificationMethod
    reasoning: str | None = None
    cited_clause: str | None = None

The keyword tier, concretely

The first tier is an ordered list of regex rules. Each rule includes the category, subcategory, and confidence assigned to a match. Order matters because the first matching pattern wins.

# (pattern, category, subcategory, confidence)
KEYWORD_RULES = [
    (r"routine maintenance",              PLANNED_MAINTENANCE, PREVENTIVE,            1.0),
    (r"scheduled (visit|maintenance|service)", PLANNED_MAINTENANCE, PREVENTIVE,       1.0),
    (r"safety (check|inspection|test)",   PLANNED_MAINTENANCE, SAFETY_INSPECTION,     1.0),
    (r"emergency call.?out",              CALLOUT,             EMERGENCY,             1.0),
    (r"(elevator|lift|unit) stuck",       CALLOUT,             BREAKDOWN,             1.0),
    (r"fault .+ found",                   CALLOUT,             URGENT,                0.9),
    (r"replaced .+ component",            REPAIR,              COMPONENT_REPLACEMENT, 1.0),
    (r"warranty (repair|replacement)",    REPAIR,              WARRANTY,              1.0),
]

The classify() method walks the rules and returns the first match. Unambiguous phrasing receives 1.0, while a hedged pattern such as fault .+ found receives 0.9. No match returns 0.5 with method=MANUAL_REVIEW, which signals an escalation:

def classify(self, description: str) -> ActivityClassification:
    description_lower = description.lower()
    for pattern, category, subcategory, confidence in KEYWORD_RULES:
        if re.search(pattern, description_lower):
            return ActivityClassification(
                category=category, subcategory=subcategory, confidence=confidence,
                method=ClassificationMethod.KEYWORD_RULE,
                reasoning=f"Matched keyword pattern: '{pattern}'",
            )
    return ActivityClassification(
        category=ActivityCategory.PLANNED_MAINTENANCE,
        subcategory=ActivitySubcategory.PREVENTIVE,
        confidence=0.5,
        method=ClassificationMethod.MANUAL_REVIEW,
        reasoning="No keyword match — requires manual review or LLM classification",
    )

The confidence threshold

A single setting controls whether the assistant trusts a classification or escalates it:

features:
  classification:
    confidence_threshold: ${CLASSIFICATION_THRESHOLD:0.80}
    keyword_rules_enabled: true

With the default 0.80, the two confidence tiers produce clear outcomes:

The classifier automatically accepts everything at or above 0.80. A 0.5 no-match goes to the LLM tier or a human. Because the threshold is configurable, each deployment can tune precision against automation without changing code. Raise it where mistakes are costly and lower it where throughput matters.

Why not just use the LLM for everything?

Using the LLM for every classification has three compounding costs:

• Keyword rules handle the bulk of traffic for free, so the LLM bill scales with the ambiguous fraction rather than total volume.
• A rule match returns immediately. Only difficult cases incur the model round-trip latency.
• Rules are reproducible and straightforward to test. Using them first reduces the surface where non-determinism, and therefore evaluation effort, matters.

The LLM handles cases where rules fall short, including novel phrasing, mixed activities, and the long tail of inputs that no rule covers.

The same pattern applies to parts, where the decision affects billing rather than routing. A PartIdentification includes a billing field with contract_covered, billable, warranty_claim, or included_in_visit, plus warranty_eligible. Misclassifying a part changes a financial decision, not only a label:

class PartIdentification(BaseModel):
    description: str
    catalog_match: str | None = None
    category: PartCategory              # oem_part | aftermarket | consumable | specialty
    confidence: float
    method: ClassificationMethod
    billing: str                        # contract_covered | billable | warranty_claim | ...
    warranty_eligible: bool = False

Reasoning is part of the contract

Every classification includes its reasoning alongside the label. This serves two purposes:

• During a debrief, the assistant can explain why it categorized an activity, which builds technician trust.
• During evaluation, the reasoning is a debugging artifact. When a classification is wrong, it shows which tier produced the result and why.

Takeaways

• Run the cheap, deterministic rules first and reserve the more expensive LLM for genuine ambiguity.
• Put the confidence threshold in configuration so each deployment can balance precision and automation.
• Return reasoning alongside the label to support trust and debugging.

This pattern, a fast deterministic path with a model fallback, applies beyond activity classification. When most inputs are easy and a few are difficult, a two-tier design retains most of the LLM’s value at a fraction of its cost.

#classification#llm#azure-openai#cost-optimization#multi-agent