Small-business outsourcing research
How often do support macros cite a policy version that is no longer current?
A protocol for auditing saved replies against the current policy register, with a test for stale wording, missing versions, and untested macros.

Research question and scope
This protocol asks how often a saved reply cites a policy version that the business has retired. The unit is one saved reply in the support tool, identified by its name and the queue where it is available. The study measures whether the reply names a policy, whether that policy is current, and whether the reply was tested after the last policy change. It does not evaluate whether the reply is well written or whether customers like it.
Set the scope to one small business, one support tool, and one policy register. Record the sources that define the current version and the people who may update a saved reply. If the policy register changes during the window, keep the earlier definition for the full sample or analyze the change as a separate period. A new policy page can look like a clean register when it is only a new document.
Population and unit of analysis
Define the population as every saved reply available to the support queue during the window, including replies that are rarely used and replies that are disabled but still stored. The unit is the saved reply, not the policy, because one policy can appear in several replies. A reply that is copied into another tool remains one unit with a note about the copy. A reply that is split into two shorter replies creates two units after the split date.
Record the queue, the product area, and whether the reply is customer-facing or internal. These strata matter because an internal note and a customer message carry different risks. Include replies created outside the normal review process, because the base rate is part of the finding. Test the unit definition on two ordinary replies before collection begins.
A reply can quote a policy without naming its version, which makes staleness harder to detect. Record the exact wording that carries the rule and the policy it appears to reference. A reply that cannot be matched to any policy is still a unit, but it should be labeled unmatched. Do not drop unmatched replies, because they are often the oldest.
Definitions that make the measure reproducible
Define a policy-citing reply as one whose customer-facing text states a rule, a limit, a deadline, or an eligibility condition. Define the current version as the version marked effective in the policy register on the sample date. Define a stale reply as one that states a rule from a retired version or contradicts the current version. Define an untested reply as one with no test record after the most recent policy change.
Do not count a generic greeting or a shipping address as a policy-citing reply. Do not count an internal comment about a policy as customer-facing text. A reply that is disabled but still available to restore should be included in the population. Each definition should include a positive example and a boundary example so a second reviewer can apply it consistently.
Evidence standard and data dictionary
For each saved reply, capture the name, the queue, the text, and the date it was last edited. Add the policy it cites, the version named or implied, and the current version on the sample date. Record whether a test message exists and the date of the last test. Keep a copy of the reply text with the audit record so a later change does not erase the finding.
Also capture who owns the reply and whether the owner is still with the business. Record any policy page that is missing from the register. If the reply uses a placeholder that is filled at send time, record how the placeholder changes the meaning. A second reviewer should be able to compare the reply with the policy without opening a dozen tools.
If the support tool keeps version history, record the version that was live during the sample. A reply that was corrected after the audit should still be counted as stale for the sample date. Keep the correction date separately. The evidence should describe the state at the time of the audit, not the state after the cleanup.
Sampling and collection
Draw the sample from the support tool's reply list rather than from the policies the team remembers. Record the export date, the filters, the row count, and the selection rule. Include every queue so the result is not dominated by the busiest team. If the list is ordered by name, a fixed interval spreads the sample across the alphabet and the product areas.
Collect the reply text and the policy register as they existed on the sample date. If a policy was updated during the collection, record the date and analyze the replies against the version that was current when they were last edited. Do not infer a test from a reply that was used successfully. Note the collection date separately from the last-edit date.
Record the queue where each reply is available rather than only the folder where it is stored. A reply can be visible in several queues and disabled in one. Use the support tool's own permission view for this field. If the view is unreliable, note the limitation and test one reply by opening it under a support account.
Analysis and reporting
Report the number of replies, the share that cite a policy, the share that are stale, and the share with no test record. Show the counts by queue, product area, and owner status. Present the age of the last edit beside the staleness rate. Use the same denominator for every rate.
Inspect the stale replies and describe the pattern without naming the employee. A cluster in one queue may point to a policy change that was never communicated, while a cluster among replies with no owner may point to turnover. Compare the result with the date of the last policy change. Report disabled replies separately because they are easy to forget.
A useful additional view is the share of stale replies that are still being sent to customers. That view separates a housekeeping problem from a live customer risk. Show it beside the staleness rate and describe the difference. The owner can then decide whether to pause the replies or update them first.
Report the result by queue and by the date of the last policy change so a reader can see whether staleness follows a specific event. Add the number of replies that were disabled but still stored, because those are the easiest to miss. State the sample date at the top of the table. A reader should be able to tell whether the finding is current.
Limitations and sources of error
The record cannot show whether a customer received the wrong answer. It can show only what the saved reply said and whether a test was recorded. A stale reply may never be used, and a current reply may still be applied to the wrong case. The protocol also cannot capture a reply that an assistant types from memory instead of the tool.
The policy register may itself be incomplete, which makes a reply look stale when the register is behind. Record the register date and the person who confirmed it. If the same person owns the policy and the reply, note the independence limitation and analyze that subgroup separately. Do not treat a missing version number as proof of an old rule without reading the text.
Review cadence and next action
Give the result to the owner and the support lead and ask them to choose one observable change. The change might be a version field on every policy-citing reply, a quarterly test, or a named owner for each queue. Record the owner, the effective date, and the measure that will show whether the change worked. Avoid changing the reply template and the policy register at the same time.
Repeat the protocol with the same definitions after the change and compare the same fields. If staleness fell but the customer complaint rate did not, the replies may not be the main source of wrong answers. If complaints fell but stale replies remained, the team may be avoiding the tool. Close the review when the owner accepts the finding and the next sample is scheduled.
Sources
- U.S. Small Business Administration, manage your business, U.S. Small Business Administration; checked 2026-11-07
- NIST Cybersecurity Framework 2.0, National Institute of Standards and Technology; checked 2026-11-07
- Federal Trade Commission, business guidance, Federal Trade Commission; checked 2026-11-07
- U.S. Government Accountability Office, internal control standards, U.S. Government Accountability Office; checked 2026-11-07
Frequently asked questions
Does a favorable sample prove the routine caused the result?
No. The design is observational and supports a bounded operating decision, not a universal causal claim.
What should an owner review first?
Start with high-consequence defects, missing denominators, exceptions, and cases that required the owner to reconstruct the evidence.
Related research
How long do supplier bank-detail changes wait for independent verification?
A protocol for measuring the delay between a supplier bank-detail change and the independent check that confirms it.
What share of damaged-order claims include a usable photo and order reference?
A protocol for measuring claim evidence quality, with clear tests for a usable photo, a linked order, and a recorded delivery condition.
What share of new assistant assignments have a named trainer and a first-week check?
A protocol for measuring onboarding coverage, with tests for a named trainer, a first-week review, and a record the next supervisor can use.