A Manage Rollback Needs an Operator Compatibility Test
Restoring a Manage database and requesting an older application version are not enough to prove recovery. IBM documents an Entity Manager failure in which a newer operator, an older version lock and missing resource permissions prevent the application from starting.
A Manage Rollback Needs an Operator Compatibility Test
A rollback rehearsal should answer a business question: can the maintenance organization resume using the intended earlier application state? Changing a version property is only one part of that answer. A restored database is another. The operator that reconciles the application, the resources it expects to watch and the permissions available to its service accounts also belong in the recovery boundary. Otherwise, a team can successfully restore data and still have no usable application.
IBM's documentation on restoring applications to earlier patch releases makes the first boundary explicit: an application restore does not automatically restore a database that was updated during the upgrade. IBM's separate troubleshooting note on Maximo Manage 9.1 adds a more specific failure mode. An older, version-locked Manage application can coexist with a newer Manage Operator until the Entity Manager fails during startup because it lacks permissions for a resource the newer code expects. These are distinct documents answering different questions, and a recovery plan needs both.
This article examines that documented failure, not the general proposition that Manage rollback is broken. It also does not turn a troubleshooting note into a universal downgrade procedure. IBM describes a particular mixed-version configuration and gives particular remediation choices. Your supported release combination, database procedure and operational restrictions still need to be checked against your environment.
The practical improvement is to make operator compatibility a rehearsal acceptance criterion rather than an assumption hidden behind a backup checklist. Record what must be restored, what remains newer, how the controllers behave and what evidence demonstrates business usability. The following test design is an engineering recommendation derived from the cited documentation, not a claim that it has been executed against your cluster.
1. Define the recovery target as a configuration, not a number
IBM's restoration procedure describes changing the version property in an application's custom resource and restoring its database. Its example distinguishes the requested application version from the controller version. In that example, changing the application request to the earlier patch changes the reconciled application version while the controller remains at the later version. The page also identifies status.versions.supported as the list of versions supported by the controller manager.
That distinction should change the first page of a recovery runbook. Do not write only a target such as an earlier Manage patch. Record the intended application request, the controller or operator that will perform reconciliation and the database backup selected for that application state. Then identify the release-specific documentation that supports the combination. A version displayed in one place is not a complete description of the system you intend to recover.
The supported-version list is useful evidence, but it should not replace a working rehearsal. IBM's troubleshooting note describes an initialization problem in a mixed-version configuration, showing why a platform team should examine actual behavior as well as a declared version. Treat the list as an input to compatibility review, then capture the application's observed reconciliation result and the dependent controller behavior. This is a proposed acceptance practice, not a new IBM support guarantee.
Build the baseline before the forward update. Preserve a restricted snapshot of the relevant application custom resource, operator identity, subscription approval policy and database backup reference. Include customizations and external dependencies in your local recovery inventory where they affect service. Avoid placing credentials, connection strings or unrestricted cluster exports in a broadly shared change ticket. The goal is reproducibility without converting a recovery package into a source of security exposure.
The baseline should distinguish objects you expect to restore from objects you expect to retain. For example, a runbook might request an older application while leaving an operator newer, as IBM's example illustrates. That is an explicit compatibility decision, not an incidental detail. Assign someone to verify the intended relationship and document the basis for accepting it. If the relationship changes during the update, the old rehearsal result should not silently stand in for the new configuration.
Finally, define what recovery success means to the maintenance service owner. Application reconciliation, authentication, a representative record read and a controlled business transaction are different observations. Write their expected outcomes before beginning the exercise. This prevents a technically interesting version change from becoming the sole evidence behind a much larger claim that the service has been restored.
2. Diagnose the documented Entity Manager failure precisely
IBM's Manage 9.1 troubleshooting note concerns entitymgr-primary-entity and entitymgr-ws pods that repeatedly enter CrashLoopBackOff. The note says their containers terminate with exit code 1, while expected Ansible progress does not appear and the container logs do not expose a clear error. It associates the condition with an older Manage version lock and a Manage Operator at 9.1.16 or later.
The documented cause is more specific than a generic bad deployment. The Entity Manager attempts to watch healthapps.apps.mas.ibm.com during startup, but its service account lacks the required role-based access control permissions. The Kubernetes API returns 403 Forbidden, initialization fails and the container restarts. IBM states that the required healthapps RBAC is not available from the v9-260430-amd64 catalog configuration. These identifiers matter because they distinguish the note's failure mechanism from other causes of a restarting pod.
IBM also describes a possible error on the ManageApp custom resource involving the undefined variable cert_issuer_internal_renewbefore. Treat that message as another clue to collect, not a sufficient diagnosis on its own. A runbook that reacts to the variable name by changing unrelated certificate settings could miss the operator and permission mismatch described in the note. Preserve the message together with the observed versions and authorization evidence.
The note's diagnostic path starts by comparing the Manage application version lock with the operator version. It then checks the container's termination details, examines healthapps permissions for the relevant service accounts and looks for the failed API access in Kubernetes audit logs. That sequence is useful because each observation tests a different part of the proposed explanation. The application request establishes the older lock; the operator establishes the newer controller; the denial identifies the startup dependency that is not satisfied.
Do not infer that every exit code 1 has this cause. A container exit code describes termination, not a unique product defect. Likewise, an empty or unhelpful container log is not proof that Kubernetes has no useful evidence. The API audit trail can answer an authorization question that the application's own startup logging does not reveal. Use the documented chain of evidence before selecting the corresponding remediation branch.
In a rehearsal record, retain a compact diagnosis rather than an uncontrolled log dump: the affected pod identity, relevant versions, termination observation, missing permission result and matching API denial. Keep full audit material in an appropriately restricted location. This makes the conclusion reviewable by another operator without exposing unrelated workload activity or presenting a single screenshot as a complete explanation.
3. Keep database recovery and controller reconciliation separate
The restoration documentation contains a warning that deserves its own acceptance gate: if the upgrade changed the database, restoring the earlier application does not automatically restore that database. IBM says the pre-upgrade backup must be applied to revert the database. The application request and the database state therefore need separate evidence. Neither a successful backup job nor an edited custom resource proves the other operation has occurred.
For a rehearsal, define a database recovery record owned by the database team. It should identify the approved backup, the intended recovery point, the restoration outcome and the restrictions on application access during recovery. The exact database commands and sequencing must come from the supported procedure for your database and deployment. This article deliberately does not supply a generic database rollback command or claim that one sequence fits every Manage installation.
Define a second record for the application platform team. It should capture the desired version, the reconciled version, operator identity and dependent startup outcome. IBM's example explains why a controller can remain newer than the restored application. Recording these fields separately prevents the team's notes from collapsing several different states into a single assertion that the old version is back. It also gives support engineers a clearer starting point if the restore fails.
The handoff between these records is where the change coordinator earns their keep. Agree who may request application reconciliation, when database restoration is considered complete and who can reopen integrations or user access. Those are recommended operating controls, not additional product features. Without them, different teams can each finish a valid local task while the combined system remains unsuitable for business use.
Rehearsals should include a deliberate checkpoint before access is reopened. Compare the observed state with the recovery target, not merely with the forward-update checklist. If the database is restored but the Entity Manager cannot initialize, the exercise has found a platform recovery problem, not achieved recovery with a cosmetic warning. Conversely, running application pods should not excuse an unverified database recovery point. Both observations belong in the same acceptance decision without being mistaken for each other.
Also decide what happens to transactions arriving from outside Manage during the exercise. Your integration owners should explain how they will hold, account for and later reconcile their traffic under the organization's existing controls. This is a design requirement proposed here, not a claim about automatic behavior of any particular queue. The recovery target must include business consistency, or an apparently successful application restore may leave the operational teams to discover missing or repeated work later.
4. Choose between upgrade and temporary operator restoration consciously
IBM gives two resolution options for the documented Entity Manager condition. Its recommended option is to remove the ManageApp version lock and allow Manage to upgrade, aligning it with the newer operator. Its alternative is a temporary workaround: uninstall and reinstall the Manage Operator at the required older version without removing dependent resources. IBM says to select manual subscription approval when installing that older operator so it does not automatically upgrade again.
These are not equivalent recovery outcomes. Removing the older lock may restore service on the newer application path, but it does not demonstrate that the organization can run the intended earlier application state. That distinction matters when the rollback was requested because of an application problem introduced by the update. The change authority should know whether the team is accepting a forward recovery or completing the original backward recovery objective.
The temporary operator option carries a different ownership requirement. Someone must determine the required older operator, preserve the dependent resources and manage the approval setting that keeps the configuration from moving again. IBM labels the option temporary and prefers upgrading when possible. Do not translate that into an indefinite operational policy of freezing components without review. Record the reason for the exception, the intended exit condition and the person responsible for revisiting it.
This is also why the article does not offer a paste-and-run uninstall sequence. Choosing the wrong operator resources or removing dependencies could change the scope of the incident. Use the exact IBM note, your deployment's approved procedure and support guidance where required. The value of a rehearsal is that it exposes these decisions before production is unavailable, not that it produces a shorter collection of commands with fewer safeguards.
Avoid a third, undocumented shortcut: granting broad administrative privileges simply because the symptom involves authorization. IBM's stated resolution options address the mixed-version configuration. A blanket privilege increase could conceal the immediate denial without establishing that the remaining combination behaves correctly. If a proposed permission change is necessary in your environment, review it through security and support channels rather than treating a successful restart as evidence that the change is suitable.
Write a decision card for the rehearsal before running it. State the original recovery objective, the evidence that triggers each branch, the authority that approves a changed objective and the checks required afterward. Include the subscription approval policy in that card. IBM explicitly notes automatic operator approval as a way the newer operator can arrive while the application stays locked. A temporary fix that allows the same uncontrolled transition has not removed the operating condition that made the mismatch possible.
5. Make the rehearsal prove useful service, not just running pods
A strong rehearsal produces a chain of evidence from configuration to business use. Start with the application request and database recovery reference. Add the observed controller and reconciliation state. Then record whether the Entity Manager initializes without the documented authorization failure. Only after those platform observations should the exercise move into business verification. This proposed sequence makes failures easier to locate and keeps the acceptance decision tied to the intended recovery target.
Select business checks with the maintenance organization rather than inventing them during the outage. A planner might need to retrieve a known work order, a supervisor might need to inspect its history and an authorized tester might need to perform a reversible change on a designated test record. These are suggested test roles, not descriptions of a completed deployment. Choose records and actions that represent your own service without creating uncontrolled maintenance obligations.
Keep technical identity separate from tester identity. A privileged administrator's successful session does not answer whether ordinary users retain the access they need. Similarly, an integration service account should be tested through its approved interface and permissions, not through an administrator substituting a manual operation. The point is not to maximize the number of green checks. It is to verify the pathways that the business will rely on when normal operations resume.
Define negative evidence as well. If a known service account receives the healthapps denial described by IBM, that is a failed platform gate even if another pod appears healthy. If a database recovery reference cannot be matched to the approved target, the data gate remains open. If a representative user cannot complete the agreed task, the business gate has not passed. Giving these conditions names before the rehearsal prevents an ambiguous result from being softened into an optimistic summary.
Require an independent review of the evidence before calling the exercise successful. A second operator should be able to follow the configuration record, understand any remediation branch and reproduce the conclusion from the retained observations. The business owner should confirm that the tested tasks are meaningful. This review does not need to become an elaborate ceremony, but it should be more than the original executor asserting that everything looked fine.
Finally, retire stale rehearsal claims when relevant components change. An operator update, a different application lock or a changed database recovery method can invalidate assumptions that the earlier exercise depended on. Define which changes require a new test under your own change-management policy. The useful artifact is a living recovery capability with a clear configuration boundary, not a slide stating that rollback was tested sometime before the current operator arrived.
Practical Implications
For platform owners, the immediate action is to add an operator compatibility line to the next patch rehearsal. Capture the application lock, actual operator version, supported-version evidence and subscription approval policy before updating. After restoration, capture the same observations again. Do not infer compatibility from the application version alone, and do not describe a forward upgrade as a completed rollback when the target changed.
For database and application teams, create separate acceptance entries with a named handoff. IBM's restoration warning makes the distinction unavoidable: the earlier application request does not restore the earlier database. Agree the recovery point and access restrictions using your existing supported database procedure. Then verify the combined service before reopening ordinary traffic.
For incident responders, bookmark the specific Entity Manager note and record its applicability conditions in the runbook. A restarting pod, an older version lock, a newer operator and a matching healthapps authorization denial form a more defensible diagnosis than any one symptom alone. Preserve restricted evidence, and choose the remediation branch deliberately rather than applying an unrelated certificate or privilege change.
For change authorities, ask the rehearsal team one final question: did the exercise demonstrate the original recovery objective, or did it find a different path to service? Either outcome can inform a decision, but they should not share the same label. That small reporting discipline prevents a platform workaround from becoming an unsupported promise to the maintenance organization.
Bottom Line
IBM documents both an application restoration mechanism and a specific mixed-version Entity Manager failure. Read together, they show why a Manage recovery target must include the database, the application request and the operator that makes the request real. An older version string is not a complete recovery state, and a database restore cannot compensate for a controller that fails during initialization.
The engineering response is not to declare every rollback unsafe. It is to make the relevant compatibility assumptions visible and test them. Establish the target configuration, collect the documented diagnostic evidence, separate backward recovery from forward remediation and verify useful business service before closing the rehearsal.
Your resulting runbook should be clear about what was proven, which configuration was tested and what changes would require another exercise. That is a more valuable deliverable than a successful edit to a custom resource. It gives the maintenance service owner a defensible basis for trusting recovery when a future patch creates a reason to use it.