AI Agents

Hybrid Activity Classification: Keyword Rules First, LLM Fallback

By Omer DemirApr 2, 2026⏱ 5 min read👁 — reads⧉ — copies

Not every classification needs a language model. In the field service assistant, most service activities are unambiguous — “replaced the roller guides” is a parts replacement, full stop. Sending every one of those to an LLM would be slow and expensive. So the activity classifier is two-tiered: deterministic keyword rules first, LLM only when the rules aren’t confident.

The two tiers

1. Keyword rules run first. They’re fast, free, and deterministic. For the large fraction of activities that match a clear rule, that’s the whole classification.
2. LLM fallback handles the ambiguous remainder. When the keyword pass can’t decide, the classifier escalates to Azure OpenAI for a judgment call.

Either way, the output is the same shape — category, subcategory, confidence, and reasoning — so downstream code never has to know which tier answered.

I think of the design as a funnel: let the obvious cases leave early, then spend model judgment on the uncertain remainder.

The taxonomy

Before any rules, there’s a fixed taxonomy modeled as enums. Three top-level categories, each with its own subcategories, all typed so a typo can’t leak into the index:

class ActivityCategory(str, Enum):
    PLANNED_MAINTENANCE = "planned_maintenance"
    CALLOUT = "callout"
    REPAIR = "repair"

class ActivitySubcategory(str, Enum):
    PREVENTIVE = "preventive"; SAFETY_INSPECTION = "safety_inspection"       # planned
    EMERGENCY = "emergency"; URGENT = "urgent"; BREAKDOWN = "breakdown"        # callout
    CORRECTIVE = "corrective"; COMPONENT_REPLACEMENT = "component_replacement" # repair
    WARRANTY = "warranty"; MODIFICATION = "modification"                      # ... and more

class ClassificationMethod(str, Enum):
    KEYWORD_RULE = "keyword_rule"
    LLM = "llm"
    MANUAL_REVIEW = "manual_review"

The result is a Pydantic model that carries not just the label but how it was reached — the method field is what lets you audit tier-by-tier later:

class ActivityClassification(BaseModel):
    category: ActivityCategory
    subcategory: ActivitySubcategory
    confidence: float
    method: ClassificationMethod
    reasoning: str | None = None
    cited_clause: str | None = None

The keyword tier, concretely

The first tier is an ordered list of regex rules, each carrying the category, subcategory, and a confidence it assigns on match. Order matters: the first pattern to fire wins.

# (pattern, category, subcategory, confidence)
KEYWORD_RULES = [
    (r"routine maintenance",              PLANNED_MAINTENANCE, PREVENTIVE,            1.0),
    (r"scheduled (visit|maintenance|service)", PLANNED_MAINTENANCE, PREVENTIVE,       1.0),
    (r"safety (check|inspection|test)",   PLANNED_MAINTENANCE, SAFETY_INSPECTION,     1.0),
    (r"emergency call.?out",              CALLOUT,             EMERGENCY,             1.0),
    (r"(elevator|lift|unit) stuck",       CALLOUT,             BREAKDOWN,             1.0),
    (r"fault .+ found",                   CALLOUT,             URGENT,                0.9),
    (r"replaced .+ component",            REPAIR,              COMPONENT_REPLACEMENT, 1.0),
    (r"warranty (repair|replacement)",    REPAIR,              WARRANTY,              1.0),
]

The classify() method walks the rules and returns the first match. Note the two confidence tiers: unambiguous phrasings get 1.0, a hedged pattern like fault .+ found gets 0.9, and a no-match returns 0.5 with method=MANUAL_REVIEW — the signal to escalate:

def classify(self, description: str) -> ActivityClassification:
    description_lower = description.lower()
    for pattern, category, subcategory, confidence in KEYWORD_RULES:
        if re.search(pattern, description_lower):
            return ActivityClassification(
                category=category, subcategory=subcategory, confidence=confidence,
                method=ClassificationMethod.KEYWORD_RULE,
                reasoning=f"Matched keyword pattern: '{pattern}'",
            )
    return ActivityClassification(
        category=ActivityCategory.PLANNED_MAINTENANCE,
        subcategory=ActivitySubcategory.PREVENTIVE,
        confidence=0.5,
        method=ClassificationMethod.MANUAL_REVIEW,
        reasoning="No keyword match — requires manual review or LLM classification",
    )

The confidence threshold

A single knob governs when the assistant trusts a classification versus escalating:

features:
  classification:
    confidence_threshold: ${CLASSIFICATION_THRESHOLD:0.80}
    keyword_rules_enabled: true

With the default 0.80, the two confidence tiers fall out cleanly:

Everything at or above 0.80 is accepted automatically; the 0.5 no-match falls through to the LLM tier or a human. Exposing the threshold as configuration means you can tune precision vs. automation per deployment without touching code — tighten it where mistakes are costly, loosen it where throughput matters.

Why not just use the LLM for everything?

Three reasons, and they compound:

• Cost. Keyword rules handle the bulk of traffic for free. The LLM bill scales with the ambiguous fraction, not total volume.
• Latency. A rule match returns instantly; only the hard cases pay the model round-trip.
• Determinism. Rules are reproducible and trivially testable. Keeping them as the first tier shrinks the surface where non-determinism — and therefore evaluation effort — actually matters.

The LLM earns its keep exactly where rules fall short: novel phrasing, mixed activities, and the long tail of things nobody wrote a rule for.

The same pattern extends to parts, where the stakes are billing rather than routing. A PartIdentification carries a billing field — contract_covered, billable, warranty_claim, or included_in_visit — plus warranty_eligible, because misclassifying a part is a money question, not just a label:

class PartIdentification(BaseModel):
    description: str
    catalog_match: str | None = None
    category: PartCategory              # oem_part | aftermarket | consumable | specialty
    confidence: float
    method: ClassificationMethod
    billing: str                        # contract_covered | billable | warranty_claim | ...
    warranty_eligible: bool = False

Reasoning is part of the contract

Every classification carries its reasoning, not just a label. That does double duty:

• During a debrief, the assistant can explain why it categorized an activity, which builds technician trust.
• During evaluation, the reasoning is a debugging artifact — when a classification is wrong, you can see whether the rule tier or the model tier produced it and why.

Takeaways

• Cheap and deterministic first, expensive and smart second. Let keyword rules absorb the easy cases and reserve the LLM for genuine ambiguity.
• Put the confidence threshold in config so precision vs. automation is a deployment decision.
• Return reasoning alongside the label for trust and debuggability.

This pattern — a fast deterministic path with a model fallback — generalizes well beyond activity classification. Anywhere most inputs are easy and a few are hard, a two-tier design buys you most of the LLM’s value at a fraction of its cost.

#classification#llm#azure-openai#cost-optimization#multi-agent