What a second round of production testing found, and what changed in the apps because of it

Automated tests check that the code does what it says. Some faults only show when somebody uses the software the way a business does, on the real thing, such as a till telling a cashier they are short by money they never had. So on 11 October two testers went through the live apps: one through the shop’s own apps as staff and customers, one through the consoles we and our customers administer.

The rules were strict. Nothing was deployed or restarted for the test, and nothing was written outside the apps’ own screens. The testers never typed a password or created an account: the owner signed in and pressed every final button. Every fault found in the first round of testing was checked again, to see whether anything fixed then had come back. Nothing had.

What it found

The two blockers were serious. Staff of a shop whose people had only workspace accounts could not sign in through Rutba’s front door at all. And a cashier’s register left out every cash refund, so each till close showed a shortage that never happened. Among the major faults:

• A cash-on-delivery order showed as paid before any cash had come in, and the refund form offered to refund money that was never received.
• A shopper’s checkout note (“please ring the bell”) was stored but never reached the staff.
• Dispatch sent its “shipped” email twice and dropped the tracking code.
• A till’s branch and desk could not be changed once chosen, and returns used the browser’s own pop-up, which a kiosk browser blocks.
• Menus offered pages the person’s role could not open, and dashboards that failed to load showed a cheerful “nothing overdue” instead.
• Some error messages showed staff the system’s internal wording rather than what had gone wrong.

What changed

• Signing in. Everybody who works in a workspace now signs in at Rutba’s sign-in, with their workspace password if that is the account they have. One door went live with the release.
• The till. A return belongs to the cashier who processed it, so their own register shows the refund. The till asks and tells on the page, and its branch and desk can be changed from the ribbon.
• Orders. A cash-on-delivery order reads as unpaid until its cash is in, and a refund is held to the money actually received. The checkout note appears on the order page, in the list, at the rider’s door and on the consignment. Dispatch mails once, keeps its tracking code, and mails again on delivery and on a return.
• Honest screens. Menus show only what the acting role can open and say which role would show the rest. Twenty home dashboards say a read could not load instead of drawing an all-clear, and an app tells somebody with no role in it exactly that.
• Plain words. One formatter turns every failed call into what it means and what to do, in place of the machinery’s message.

The consoles had their own list, mostly smaller. Suspending a person now asks first and needs a reason, which is kept with the audit record. A failed read says which read failed and why, with no form left under it to fill in. And email notices from the Relay, which had been failing quietly, now send.

And the tests that run every time

Alongside the people, the automated suite was rebuilt to walk every business on a shared service in turn, checking that each one sees only its own data. Its full run on 9 October passed 9,846 checks with none failing and 822 skipped, and the separation between businesses held everywhere it was probed. The management release gate passed 172 checks, with none failing and 17 skipped, before the release went out.