A Practical Acceptance Test for Maximo Assistant Answers

A fluent answer is not the same as an accepted maintenance decision. Build a repeatable test that checks scope, source records, permissions, freshness, and human review.

Share
A Practical Acceptance Test for Maximo Assistant Answers

A Practical Acceptance Test for Maximo Assistant Answers

A maintenance planner asks an AI assistant for overdue preventive work at a particular site. The answer arrives in plain language and sounds confident. Before anyone moves the work into tomorrow's schedule, a more important question remains: does the answer represent the right records, the right site, the right time, and the permissions of the person asking?

That is the acceptance test I want teams to run before treating an assistant as part of an operational workflow. The goal is not to demand identical wording every time. It is to establish whether an answer is fit for a defined decision, whether the user can inspect the underlying evidence, and whether missing or uncertain information is handled safely. A useful test also establishes what the assistant must not do, including crossing a permission boundary or quietly turning a recommendation into an authorized maintenance action.

IBM describes Maximo Assistant as configurable by an administrator. Depending on the deployment and enabled tools, it can retrieve records, generate condition insights, search IBM Documentation, or perform other supported analysis. IBM also states that the assistant cannot automate tasks or create or edit system data unless an administrator has configured a custom tool for that purpose. These are product boundaries to check in the deployed version, not assumptions that every environment has the same capabilities. See Using Maximo Assistant and Maximo Assistant architecture.

The approach below is a proposed test method, not an IBM certification procedure. Use a nonproduction environment with approved test data and users. Define pass criteria with the maintenance owner and security team, then record actual behavior rather than scoring the quality of the prose alone.

Define the decision and its evidence before the prompt

Begin with a narrow and carefully defined operational decision. A planner may need to identify overdue PM work orders for a particular site, while a reliability engineer may want to investigate the condition of one asset. These are different tasks. One is a record retrieval question with a date and status boundary; the other may involve an insight generated from several kinds of evidence. Do not combine them into a broad request and then guess which part of the answer failed.

Write a test card for the first question. It should name the intended user role, site, relevant record type, date interpretation, status exclusions, expected result set, and the authority responsible for acting on the result. Specify the business meaning of overdue before writing the prompt. Does it refer to a target date, a scheduled date, or a locally defined rule? Which statuses are excluded? The correct definition is the one approved for the workflow, not whichever interpretation makes a demonstration look best.

Build a small reference set of fictional records with known expected outcomes. Include a record that meets every condition, one at a different site, one excluded by status, and one with a missing date. Include an asset or work order that the test user is not authorized to see. The reference set should be maintained independently of the generated answer. An authorized reviewer can use an approved application view or report to establish the expected records, subject to the same access scope being tested.

Now write the prompt in complete, specific language. IBM recommends being specific and suggests narrowing large requests, because an administrator can set a limit on the number of records the assistant analyzes at one time. A sample test prompt might ask for overdue work orders at a named test site using the organization's approved definition. Record the exact prompt, test account, time, configured environment, and expected set. A successful answer must not silently broaden the site or imply that a partial list is complete.

Avoid confusing a useful explanation with a verified fact. A response can describe why a work order might be late without the record containing that reason. In the test, require a clear distinction between retrieved information, an inference, and a recommendation. If the assistant cannot identify the supporting record or the user cannot inspect it through an authorized route, the reviewer should not treat the explanation as established maintenance history.

Map the deployed architecture and permission boundary

Before testing content, establish which Assistant configuration is actually in use. IBM distinguishes agentic and non-agentic modes. Its architecture documentation describes an agentic path involving the ALM agent, AI Service, tools on the Maximo MCP server, and Maximo Manage APIs. The non-agentic path uses a data retrieval tool without the ALM agent. These descriptions are useful orientation, but the deployed version and enabled configuration determine what the team must test.

Create a short inventory with the environment, application version, enabled AI configurations, searchable object structures, test users, and change owner. Include whether condition insights and documentation search are enabled, if applicable. Do not record passwords, tokens, or other secrets in the test pack. When a test fails, this inventory helps distinguish a content issue from an unavailable tool or a missing permission. It also prevents teams from treating a feature described in documentation as proof that it is enabled locally.

IBM states that Assistant operates within Maximo security, permissions, and approval rules, and that users can only receive information they are authorized to access. Test that claim in the actual configuration with at least two controlled roles. Give the first role an approved view of the test site and give the second a deliberately narrower scope. Ask the same question from each role. Compare the answers with each role's authorized reference view, not with an unrestricted administrator's results.

A second test should request a record that the restricted role cannot access. The acceptance criterion is that protected content is not exposed. Agree with the security team on how much feedback is appropriate when data is unavailable, and do not require a particular refusal sentence unless the implementation explicitly promises it. Capture enough evidence for review without copying restricted data into a broadly accessible test report.

The configuration also matters for read and write boundaries. IBM's user documentation says the assistant cannot create or edit data or automate tasks unless a custom tool has been configured for that purpose. If no such tool is approved, include a request to change a test record and verify that no change occurs. If an organization has deliberately enabled a custom tool, define and test its authorization, approval, audit, and rollback behavior separately. A recommendation in chat is never, by itself, evidence that a work order was approved.

Finally, repeat permission tests after changes to roles, object structures, AI configurations, or custom tools. Do not assume that a successful administrator demonstration proves the experience or access boundary for a planner, contractor, or supervisor. Security is part of answer quality because an otherwise correct answer is still unacceptable if it discloses information to the wrong person.

Check completeness, freshness, and the meaning of a result

A test needs more than a plausible answer. Compare the returned records with the expected reference set and classify the differences. A missing eligible record is a completeness problem; an included record outside the defined scope is a precision problem; a record the user cannot access is a security problem. Keep those categories separate because each requires a different investigation and may have a different release consequence.

Test a case with no matching records. The desired outcome is a response consistent with the authorized reference view, not a made-up example. Then test a case where the underlying record is incomplete. A missing description or date should not be silently transformed into a confident explanation. Ask reviewers to mark statements that are directly supported, statements that are inferred, and statements that cannot be checked from the available evidence. Capture the exact wording where it matters, but score the underlying claim rather than style alone.

Freshness requires special attention for condition insights. IBM says that asset insights are not automatically updated: they update when a user asks the assistant to generate insights for a specific asset or refreshes from the Insights card. If that capability is enabled, include a test that changes approved test data, checks the existing insight, requests a refresh through the supported path, and then records the observed result. Do not assume a newly opened conversation refreshes an insight. The test should show the timestamp or other available evidence that lets the reviewer understand what was evaluated.

Document the difference between a record and an interpretation of the record. A work order may say a pump was inspected; that alone does not prove a fault was repaired. An insight may summarize patterns and suggest further investigation; it does not replace an engineering assessment or an approved work plan. For each consequential statement, the reviewer should be able to ask which authorized records support it and whether those records are current enough for the decision.

Check record limits with a dataset deliberately larger than the configured limit, using fictional records. IBM notes that administrators can set the number of records analyzed at one time and recommends breaking broad requests into smaller questions. Agree on a practical acceptance rule for partial results. For example, a planner may need a clearly bounded site and date range, plus a separate authoritative report for the complete population. Do not present a subset as an exhaustive list unless the team has verified completeness.

Finally, test changes in wording. Ask equivalent questions that retain the same site, record type, time boundary, and business definition. IBM notes that similar requests can be interpreted differently at different times. The aim is not to demand identical sentences. The aim is to find prompts that produce materially different or unsupported decisions and either improve the testable configuration or teach users a more precise question.

Run a controlled review and define release gates

A useful acceptance session brings together the product owner, a representative end user, a Maximo administrator, and a security reviewer. Give each a defined task. The end user judges whether the answer supports the workflow. The administrator identifies configuration and record behavior. The security reviewer checks access. The product owner decides whether remaining limitations are acceptable for the intended use. No one should be asked to infer another person's approval.

Run the baseline questions first, then the negative tests. Include an authorized matching record, an authorized nonmatching record, an inaccessible record, an incomplete record, and a question outside the agreed scope. Where condition insights are enabled, add a stale-versus-refreshed scenario. Where a custom write tool exists, add an independently approved write test in nonproduction. Record the actual prompt and output, the reference evidence, the reviewer, and the outcome. Use a redacted report if answers could contain sensitive information.

Define pass and stop conditions before the demonstration. An unauthorized disclosure or an unapproved state change should stop release and trigger the organization's security process. A missing record or unsupported assertion may also block the specific workflow until investigated. A harmless variation in phrasing should not. The severity depends on the decision the answer supports, so establish the decision boundary instead of applying a single generic accuracy score.

If a test fails, preserve the evidence and identify the layer to investigate. Check the source record, the role's normal access, the Assistant configuration, the exact prompt, enabled tools, and any applicable integration or service health. Do not change several settings at once and then declare victory because a later answer looks better. Change one controlled element where practical, rerun the failed case and its related negative cases, and record the observed result.

A release decision should include limitations as well as passes. The team might approve Assistant as a navigation aid while requiring the planner to confirm the authoritative work order list before scheduling. It might restrict the initial use to a small site or a limited record type. It might decide that a particular insight requires an engineer's review before any maintenance recommendation is acted on. These are operational decisions, not product defaults.

After release, maintain a small regression pack. Repeat it when a model configuration, tool, security group, object structure, source process, or product version changes. Review real user reports through the organization's approved support process, removing sensitive content from shared test artifacts. The goal is not to freeze an assistant's wording. It is to keep the business decision, evidence, and authorization boundary stable as the surrounding system evolves.

Practical Implications

For the maintenance owner, select one decision that can be checked against a known set of records. Write down the meaning of terms such as overdue, the acceptable freshness of evidence, and the action a person may take after reviewing an answer. Assign a reviewer for exceptions.

For the Maximo administrator, document the deployed mode, enabled tools, searchable data, and record limits. Compare them with the relevant IBM documentation for the installed version. Keep a repeatable nonproduction test set and retest when configuration changes.

For the security team, use controlled roles with intentionally different access. Confirm both authorized retrieval and non-disclosure, and treat any custom write tool as a separate change requiring explicit approval and testing. Do not use a privileged demonstration account as a proxy for ordinary users.

For AI and integration teams, keep the source record, generated statement, and reviewer decision distinct. Investigate missing or unsupported claims at the correct layer. If a test cannot establish that the answer is complete, label it as a starting point rather than a complete operational report.

For all participants, capture observations without exporting secrets or unnecessary personal data. A concise test record with the prompt, expected scope, observed behavior, and decision is more useful than a polished transcript that cannot be reproduced.

Bottom Line

Maximo Assistant can make enterprise records easier to explore, but a convincing answer does not automatically become an authorized maintenance decision. The test is whether the right user receives appropriately scoped, current, supportable information for a specific task, with a clear route back to the records and people responsible for action.

Start with a small question and a known reference set. Test the normal case, empty case, missing data, permission boundary, and freshness where insights are enabled. Separate an answer that retrieves records from a custom tool that changes them. Record the actual behavior of the deployed version rather than assuming that every documented feature is active.

Keep the results with the configuration and the business definition used for the test. If a later change affects the source data or the user's access, rerun the relevant cases instead of relying on an old sign-off. When the assistant cannot establish an answer, the safe result may be to refer the user to an authorized report or a human reviewer. That is not a failed maintenance process; it is a clear boundary around what the technology has actually demonstrated. Use the evidence from each review to improve the question, the source records, or the workflow, and avoid turning a fluent response into proof that those improvements are unnecessary.

A good acceptance process makes limitations visible before they become operational surprises. It lets planners use the assistant as a helpful starting point while retaining the checks that keep maintenance decisions accountable.