From the black box to the boardroom
October 23, 2026
Why AI transparency is a business imperative
By Tina Shah Paikeday and Frank Scarpelli
“Did AI decide I wasn’t good enough before a human ever looked at my application?”
A job seeker asked this question recently, and it captures one of the most important challenges facing organizations deploying artificial intelligence today.
The honest answer is: it depends.
It depends on how the organization is using AI. It depends on how much authority the system has been given. And, critically, it depends on whether the people relying on its outputs can explain the decisions it is making on their behalf.
That question is particularly urgent in hiring, where AI can influence decisions with meaningful consequences for people’s careers. But it is not unique to talent. The same challenge is emerging wherever organizations use AI to automate workflows, inform decisions, or reshape how work gets done.
As adoption accelerates, simply disclosing that AI is being used is no longer enough. Organizations need to be able to understand how it is being used, explain why it produces a particular outcome, and evaluate whether it is actually creating the value they expect.
In other words, AI transparency is becoming more than a compliance requirement. It is becoming a prerequisite for trust, and for measurable business value.
Transparency is also a business question
The same principle becomes important when AI moves beyond talent decisions and into the operating model of a business.
Across conversations with private equity operators, portfolio executives, and leaders implementing AI inside their organizations, one concern appears repeatedly: trust.
Not simply trust that the technology works.
Trust that the decision it produces is one a human can understand, stand behind, and, when necessary, defend.
That is where transparency begins to connect directly to business value.
Organizations often approach AI by asking, “What can AI do for us?”
A useful starting point is: Where is the friction, and what problem are we actually trying to solve?
In practice, many apparent AI opportunities are not fundamentally AI problems. They are integration gaps, data problems, or broken processes.
Across conversations with operators, roughly 80% of what organizations initially put on an “AI wish list” can turn out to be process or integration issues instead.
That distinction matters because AI cannot compensate for a process that has never been clearly designed or data that has never been properly organized.
A useful test is straightforward: if you fixed the underlying process without introducing AI, how much of the problem would disappear?
Whatever remains is a much clearer AI opportunity.
What transparency looks like in practice
Consider an order-entry workflow implemented at a private equity-backed portfolio company.
The organization had a very ordinary problem: paper order tickets were being manually entered into its systems. The process was slow and prone to error. It was also difficult to audit. If something went wrong, the organization had to reconstruct what had happened after the fact.
AI was introduced to read the order tickets using optical character recognition.
But the important design choice was not simply that AI could extract the information.
The system attached a confidence score to each field it read.
When confidence was high, the information could continue through the workflow. When confidence was low, the case was routed to a human for review before proceeding.
That confidence score became the explainability layer.
Instead of saying, “the AI decided,” the organization could see where the AI was confident, where it was uncertain, and where human judgment needed to intervene.
The result was an approximately 50% reduction in order-entry staffing. Not because humans had simply been replaced by a black box, but because human attention could be directed to the cases where it was actually needed.
The same underlying pattern can apply across very different use cases: the system shows its work, and the human remains involved at the moments that matter.
That is transparency operationalized.
If you cannot explain the decision, can you really measure the return?
The business case for explainability extends further.
Organizations are under growing pressure to demonstrate returns from their AI investments. But measuring those returns becomes difficult when leaders cannot clearly see how AI changed a decision or workflow in the first place.
A lower visible cost does not necessarily mean value was created.
Work may simply have moved from one team to another. A faster process may have accelerated a step nobody was waiting on. A successful pilot may deteriorate once exceptions begin accumulating.
Leaders therefore need a way to evaluate AI that goes beyond adoption rates or headline cost savings.
One practical governance scorecard asks four questions:
1. Business impact: Did a metric the organization already tracks actually move?
The strongest evidence is not a new metric invented after an AI deployment. It is improvement in something the business already cared about: revenue, cost, cycle time, or another established operating measure.
2. Decision speed: Are decisions happening faster at the points where speed actually matters?
AI may create value before that value appears directly in a financial statement. Reduced cycle time at critical approval or decision points can serve as a useful leading indicator.
3. Burden: Did the work actually disappear, or was it simply redistributed?
This is easy to overlook. An initiative can appear successful because one team’s workload fell while additional review, correction, or exception handling quietly moved elsewhere.
4. Culture: Can the people actually using the technology explain how it makes their work better?
If frontline employees cannot answer that question, an organization may have a mandate rather than genuine adoption.
These questions help shift AI governance from an abstract principle to an operating discipline.
They also make transparency central to ROI. If leaders cannot trace what changed, where human effort moved, and how decisions are being made differently, it becomes much harder to determine whether AI has created sustainable value.
Governance starts before the technology
There is another implication here that deserves attention.
AI can sometimes act as a microscope, exposing problems that were already present.
In talent, for example, implementing AI may force an organization to examine whether it is evaluating candidates against meaningful competencies or simply reproducing patterns based on who currently occupies a role.
In operations, an AI initiative may reveal that teams do not share a single source of truth, that integrations are weak, or that an existing workflow was never clearly designed.
The technology did not necessarily create those problems. It made them more visible.
That is why AI should not be treated as a silver bullet.
Organizations need sound processes, usable data, clear accountability, and people who understand how the technology fits into their work. Without those foundations, AI risks automating ambiguity rather than resolving it.
Change management is part of that equation as well.
Employees are already asking what AI means for their roles. If they cannot understand how a system works or see how it materially improves their work, resistance should not be surprising.
Transparency, therefore, is not only about explaining AI outputs. It is also about giving the people affected by AI enough visibility to participate intelligently in its adoption.
Transparency is not the same as explainability
The term “black box” is often used to describe AI systems, but it is worth being precise about what that means.
A black box is not simply a complicated system. It is a system in which the reasoning behind an output cannot be adequately understood or explained by the people responsible for acting on it.
Consider a recruiter reviewing candidates.
Humans are not inherently free from bias. We make assumptions based on our experiences and the limited information available to us. But when asked why we reached a decision, we can usually articulate some form of reasoning, even when that reasoning turns out to be flawed.
An opaque AI system may not provide that opportunity at all.
A recruiter may receive a candidate score, ranking, or recommendation without knowing which signals drove it, how those signals were weighted, or why one candidate was elevated over another.
That distinction points to two related but different governance requirements.
Transparency tells us what happened: which system was used, who interacted with it, and where AI played a role in the process.
Explainability helps us understand why: which factors contributed to an output and how the system arrived at its recommendation.
The National Institute of Standards and Technology’s AI Risk Management Framework similarly distinguishes characteristics such as accountability and transparency from explainability and interpretability. In practice, organizations need both if they want people to responsibly govern AI-assisted decisions.
Imagine, for example, an AI-generated signal assessing a candidate’s career trajectory. Accountability and transparency could provide a record of who viewed that signal and when.
Explainability would go further, showing the underlying factors contributing to it, such as role progression, tenure, skill growth, or changes in scope.
The goal is not simply to produce a score. It is to make the reasoning behind that score visible enough for a human to evaluate it.
Responsible AI should help people make better-informed decisions, not quietly substitute an opaque system for human judgment.
When process transparency is not enough
A useful example comes from one of the most widely cited cases of AI adoption in recruiting.
Unilever has been recognized for introducing game-based assessments and video interviews into its hiring process, generating significant efficiencies while retaining humans in final hiring decisions. The process reportedly saved more than 50,000 hours and approximately £1 million in recruiting costs.
On its face, the model contains several elements associated with responsible adoption: the use of AI is visible, humans remain involved, and the process produces measurable efficiency.
But more recent research illustrates why those protections alone do not resolve the explainability problem.
A 2026 Stanford-led study, Algorithmic Monocultures in Hiring, examined more than four million applications across 156 employers using the same hiring technology vendor. Looking at outcomes position by position, researchers found adverse impact against Black applicants in approximately 10.6% of positions.
The methodological issue is especially instructive.
According to the study, the vendor had evaluated bias by aggregating applicants and jobs rather than examining disparities at the position level. The researchers also described a system designed to identify candidates resembling existing employees rather than one validated against actual job performance.
The lesson is narrower—and more useful—than simply concluding that AI produces bias.
An organization can be transparent about the fact that it uses AI while still lacking sufficient visibility into what the AI is optimizing for and why it produces particular outcomes.
That is the difference between process transparency and decision explainability.
Knowing that an algorithm participated in a decision is not the same as knowing why it made the recommendation it did.
From the black box to the boardroom
The next phase of AI adoption will require organizations to move beyond the question of whether they are using AI.
The more consequential questions are becoming: Where is AI making or influencing decisions? Can we explain those decisions? Can humans intervene when necessary? And can we demonstrate that the technology is producing durable value?
Those questions connect responsible AI governance directly to business leadership.
A board does not need every director to become an AI engineer. But governing AI effectively requires enough understanding to ask meaningful questions about how a system fits into the broader business, what risks it introduces, how its decisions can be challenged, and what evidence demonstrates that it is working.
That may ultimately be the more important differentiator.
As AI deployment becomes increasingly commonplace, simply having the technology will say less about an organization than how deliberately it has integrated it.
The organizations that build trust will be the ones that can open the black box—not only to explain what their AI is doing, but to determine whether it has earned the authority and investment they have given it.
Because transparency is not the end of AI governance.
It is what makes meaningful governance possible.
