You have a testing discipline that has held for twenty-five years, and it is a good one: validate what you configured, trust what the vendor shipped. It survived the move to Fiori without modification. On a conversational surface it stops being enough, and not because the rule is wrong. The rule depends on being able to point at what you built.
The rule, and why it worked
I have never seen anyone validate a stock requirements list by hand.
Not once. No project team I have worked with has counted the receipts and issues on an MD04 screen against the underlying tables to confirm that SAP got the arithmetic right. Nobody I know of sat down and verified, condition by condition, that every standard pricing condition on a sales order calculated correctly.
That was not laziness. That was the correct allocation of finite test effort. The mapping was deterministic, the vendor warranted it, and the failure mode would have been catastrophic and obvious rather than subtle and silent. So you tested your pricing routines, your custom fields, your user exits, your output determination - the things you built - and you trusted the rest.
Fiori inherited the rule without anyone needing to restate it. New tile, same logic: validate the custom fields, trust the delivered app. The underlying service was the same service, so the data was the same data.
The rule works because the boundary is visible. You can point at the custom field. You can point at the Z-program. The line between what is yours and what is the vendor’s is drawn in the repository, and it has been drawn there since the beginning.
What you built when you turned the skills on
Now activate a few dozen conversational skills.
Every one of them is SAP-delivered. By the old rule, the whole thing is trusted, and a test plan written by anyone experienced would say so - scope it, confirm a handful of representative responses, move on.
Except you did build something. You chose that set, skill by skill. Nobody at SAP decided which skills your business would run, in what combination, for which roles. That selection is a customer artifact with no vendor warranty attached to it, and the behavior a user experiences is determined as much by that selection as by any individual skill.
So is the vocabulary you taught the assistant. Every mapping from one of your words to one of SAP’s is yours, and nobody warrants that either.
None of it appears in a transport. It has no object type. There is nothing to point at in a scoping meeting, which is exactly why it falls off the test plan - not by decision, but by default.
The boundary did not move. It stopped being visible.
The five things between the question and the answer
Once the line is drawn correctly, what to test becomes obvious, and none of it is field validation.
Routing. Does the question reach the intended skill? With one skill this is trivial. With several dozen, more than one can plausibly match a given phrasing, and the selection is made across the set.
Extraction. Did it pull the right parameters - this order, this plant, this date range - out of how the user actually talks rather than how a test script writes?
Fidelity. Suppose the right skill ran, with the right parameters, and the service returned the right data. Does the answer represent that data faithfully, with the unit, the date and the context that give the number its meaning? A tile never restated the number in a sentence.
Completeness. The service returned fourteen fields and the answer mentioned three. Which three, and was the omission material to the decision the user was about to make?
Scope. Did it answer the question asked, or an adjacent question that sounded the same? This is where the ambiguity from two issues back lands - the assistant returning open order value to someone who meant unshipped units, or measuring past due against the confirmed date when the reviewer meant the date the customer asked for. Last issue’s gap lands here too: answering who are our top customers from the sold-to, for a sales lead who meant the accounts behind the distributor.
AI evaluation tooling has names for these: tool selection, tool input accuracy, tool output utilization, response completeness, intent resolution. What is new is that they now belong in an SAP test plan.
None of that is a question about the service. The service is the same service, and you should trust it exactly as far as you trusted the Fiori tile in front of it. What you are testing is whether the path from question to answer holds.
Negative tests, which you have never written
There is a category here with no precedent in any SAP test plan you have ever run.
SAP describes how selection works. Joule matches a request to the best-fitting skill, using the skill’s description, the user’s role, the conversation so far and the business context. The test trace in Joule Studio opens by showing which skill or agent was selected. Practitioners building custom skills advise testing in both directions: queries that should trigger the skill, and queries that should not.
That makes the test object selection.
You have never written a Fiori test script with a section headed queries that should not open this tile, because the user opening the tile was the user’s decision and the system had no opinion about it. Now the system has an opinion, and the opinion is testable, which means it is also wrong sometimes in a way that belongs to you.
That trace is a test tool. Find out what record you get in production, when a ticket arrives and somebody has to work out what the assistant actually did.
The part that is not a gate
The consequence changes the shape of the work rather than just its content.
If selection means the best fit among the skills available, then adding to the set can change behavior for what was already there. Activating one more skill is not an isolated change - it introduces a new candidate into every selection decision the system makes.
Nothing in your estate has ever worked that way. Activating a Fiori tile has never altered what an existing tile does. Adding a transaction has never changed what another transaction returns. Every surface you have ever added has been additive and inert with respect to the others.
That is worth establishing in your own tenant rather than taking on faith, and it is a half-day experiment: baseline a set of utterances, activate one more skill, run them again, and see whether anything moved. If it did, your acceptance testing is not a gate you pass once before go-live. It is a regression corpus, owned by someone, rerun every time the skill set changes.
And the set keeps growing. Every release delivers new skills, and like a support package, nothing runs until you activate it. Each activation is a change to the set.
Run this before your next activation. Write down twenty real questions your users would actually ask — their phrasing, not yours. For each, record which skill should answer, which parameters should be extracted, and what the answer must contain to be useful. Run them as one test user with a fixed role, starting a fresh conversation for each, because role and conversation history both feed selection. Then activate one more skill and run them again. If any answer changed, you have found the thing that has no precedent in twenty-five years of SAP testing, and you now know what your test plan is missing.
The rule still holds. Test what you built, trust what the vendor shipped.
What is new is that you built something you cannot point at - and testing it produces an artifact your estate has never needed. Not a test plan. A set of real utterances that goes stale the moment somebody activates a skill nobody thought was related.
There is no precedent for that one. You build it from nothing.