Agent tool calls are customer signals
When an agent uses a SaaS product, it does not experience the product through the same interface as a person. It may use an MCP server, a CLI, an API, or a tool exposed by the browser.
That interface is still part of the customer experience. It is simply one layer further away from the customer.
A tool call can succeed while the customer's task still fails. The request was valid, the server returned 200, and nothing useful happened. This is where ordinary observability starts to feel incomplete.
The interface is the product surface
I am not talking about building the agent itself. I mean the contract a SaaS platform gives an agent: available tools, schemas, permissions, errors, and results.
I like interface contracts because they make a system easier to reason about. They also create a useful place to measure it. If an agent repeatedly calls the wrong tool, retries the same action, or falls back to a human workflow, the interface is telling us something about the customer's experience.WebMCP is one version of this idea Chrome describes WebMCP as a proposed way for websites to expose structured tools, schemas, and page state to agents. It replaces some fragile clicking and guessing with an explicit contract. Read Chrome's WebMCP overview
The same idea applies to an MCP server or CLI. The transport changes; the customer question does not: did the product help complete the job?
A successful call is not success
The obvious metrics are availability, latency, error rate, invocation count, and cost. I would want all of them. They tell me whether the interface works as software.
They do not tell me whether it works for the customer.Why I reference AWS here AWS makes this distinction directly: infrastructure metrics show that an agent is running, while task completion and user satisfaction help show whether it is working. Read the AWS guidance
For customer success, I would care more about a small set of outcomes:
| Signal | What I want to learn |
|---|---|
| Task completion | Did the customer's intended job finish? |
| Repeated attempts | Is the agent compensating for a confusing or unreliable tool? |
| Escalation | Did the workflow need a person to recover it? |
| Customer effort | How much work remained after the agent stopped? |
| Confirmed outcome | Did the customer consider the problem solved? |
The denominator matters. Ten thousand successful calls can describe ten thousand useful actions, or one broken workflow retrying all afternoon.
Rasa makes a similar point in support automation: deflection only means that a human handoff did not happen; it does not prove the customer's issue was resolved.Why I reference Rasa here The useful lesson is not specific to chatbots. An operational event and a customer outcome are different measurements, even when they happen in the same workflow. Read Rasa's metric guide
The useful part is the join
A tool trace knows that an operation failed. Customer success knows that an account is struggling. Those facts become useful together only when they share enough context.
At minimum, I would carry the account, user, workflow, tool, and intended task through the trace. Then I could ask questions that an infrastructure dashboard cannot answer:
Which accounts repeatedly fail the same agent-driven workflow?
Which tools lead to retries or human escalation?
Did a release improve task completion or merely reduce errors?
Is agent adoption creating value, or just more activity?
OpenTelemetry now has semantic conventions for MCP-related telemetry. That does not create the customer-level join automatically, but it gives the technical events a common vocabulary.Why I reference OpenTelemetry here It shows that MCP activity can be traced consistently. Account and workflow context are still product decisions that need to be designed into the interface. Read the MCP conventions
The flow I have in mind is small:
No enormous “AI analytics platform” is required to test the idea. One important workflow and a few honest events would be enough to learn whether the signal is useful.
What I do not know yet
I have not operated this model across enough customers to claim that it predicts churn or drives expansion. This is a design hypothesis, not a case study.
It comes from the same way I tend to learn other systems: build the smallest real version, observe where the abstraction leaks, and let the failures show what the diagram left out.
The hypothesis is simple: when customers use a SaaS product through agents, the interfaces those agents use become customer touchpoints. Their telemetry should therefore sit beside product usage, tickets, and customer feedback—not in an isolated agent dashboard.
That is the part I want to test.