A bug that stops is asking you for help. The difficult ones are not those; they are the numbers that come out wearing a correct face. We found one this month. What stung most is that we were not the ones who found it.
KAZENA Order is a register. It rings up a sale, prints a receipt, and at the end of the day produces the day's summary. The summary lists the breakdown by payment method and the day's total. The total agreed every day. Nobody was looking at the part that did not.
A message came in from a store that has been testing with us: the cash at closing does not agree. The screen said 18,650 pesos in cash, and there were 16,250 pesos in the drawer. The gap was 2,400 pesos.
At first I assumed a keying error. Those do happen. The next day it came up short in almost the same shape, and when I laid out 3 days side by side, the missing amount was exactly the day's e-wallet total, every time.
A payment method is resolved by its code. There is a table that turns a code into a name and a category, and the software says that when the lookup finds nothing, treat it as cash. I am the one who wrote that line. I put it there when the table was still empty, because I did not want the screen to fall over.
5 weeks ago we changed payment method codes from 2 characters to 3. The table was migrated. The documents were migrated. What was not migrated was the list of old codes the store's terminal had loaded when it started up. That terminal had been running since before the change, and had not been shut down since.
So the terminal was still sending 2-character codes. Ask a 3-character table with 2 characters and nothing comes back. When nothing comes back, it becomes cash.
It never once reproduced at the office. Here we relaunch the app every time we try something, so the list is always fresh. We were hunting in a place that starts and stops for something that only appears in a place that keeps running.
Over 5 weeks, 412 of 8,600 sales, 63,000 pesos, had been counted as cash. Not one peso had gone missing. It was simply in the wrong box. At this store almost all payment is cash and e-wallets are only about 5% of the total. That is why it stayed under the surface for 5 weeks, and also why it was allowed to.
The worst part was that a reconciliation existed. There is a step that matches the sum of the breakdown against the day's total, and it passed every day. Since anything unresolved was being poured into cash, the sums agree no matter what happens. What that reconciliation was checking was itself.
A bug that stops gets fixed the same day. A number that is quietly wrong stays trusted for the whole time it takes to find it. Those 412 sales flowed into the books, trusted, for 5 weeks.
We changed 2 things. Treating a failed lookup as cash is gone. A code we cannot resolve now stays unknown, on the document and in the day's summary alike. Unknown lines on a screen look bad, and things that look bad get corrected by people. The other change was the reconciliation: it now matches the amount the register accumulated as it went against the amount the summary query added up. Comparing 2 numbers that came out of the same table tells you nothing.
I estimated half a day and it took 5. 4 of those were putting the 412 past sales back where they belonged. In the end the person who found this was the one counting the cash in the drawer. That number is made outside the software, which is exactly why it was able to disagree with it. At the office now, when we add a reconciliation, the first question is whether the 2 numbers are made in separate places.