AI isn't the problem, but poor design can be

Originally published in the Human in the Loop newsletter on Substack.

‍

Artificial intelligence is rapidly becoming part of how organizations identify, assess, and select talent. As adoption accelerates, so too does the conversation around regulation, compliance, and bias.

‍

While these conversations are necessary, they often begin in the wrong place.

‍

Before we can talk about compliance, we first need to understand design. The fairness of an AI system should not be determined after existing processes have been deployed through audits or governance frameworks. It should be determined by intentional system design to build in de-biasing principles from the ground up.

‍

That requires us to step back and ask a more fundamental question: where does bias actually come from?

Human bias in hiring came before AI

At the center of my research on hiring decisions is Daniel Kahneman's dual-process theory of cognition.

‍

System 1 is fast, automatic, intuitive, and pattern-based. System 2 is slower, deliberate, and criteria-based.

‍

Unfortunately, most hiring systems have historically optimized for System 1. Search platforms, applicant tracking systems, and professional networks make it easier to identify candidates who resemble previous successful hires, but they do little to challenge the underlying cognitive patterns that shape those decisions.

‍

Representativeness, availability, and anchoring bias continue to influence who gets noticed, who advances, and ultimately who gets hired.

‍

AI does not eliminate these biases automatically. In fact, research by Suresh and Guttag shows that bias can enter AI systems at multiple points: through historical training data, representation in that data, feature construction, or the way a model is ultimately deployed. Each source requires a different intervention.

‍

Deployment bias deserves particular attention. Even a well-designed model can produce poor outcomes when used for purposes it was never intended to support.

‍

Governance, therefore, is not simply about evaluating outputs. It is about ensuring that AI is being used for the purpose it was designed to serve.

The architecture matters more than the label

When vendors say they use AI, they have communicated very little.

‍

The more meaningful question is: what kind of AI?

‍

General-purpose large language models are designed to infer patterns from broad training data. In hiring, that means they encode historical hiring biases by default. Recent research by Anzenberg and colleagues evaluating leading LLMs found demographic disparities affecting racial, ethnic, and intersectional groups.

‍

Domain-specific AI is fundamentally different.

‍

When trained on structured, job-specific attributes and designed with fairness constraints, these models evaluate candidates against competencies rather than credential proxies such as employer prestige, job titles, or educational pedigree. The same research found that domain-specific models improved hiring quality while maintaining equity across race, gender, and intersectional groups.

‍

This challenges one of the most persistent assumptions in talent acquisition: that organizations must choose between quality, diversity, and speed.

‍

When I tested domain-specific AI against humans last year in Findem's Innovation Accelerator, we found that responsibly designed, competency-anchored AI not only improves all three criteria simultaneously but beats humans on bias too.

‍

The architecture of the system, therefore, not simply the presence of AI, determines whether these outcomes are possible.

Human judgment still matters

None of this suggests that AI replaces human judgment. Rather, it changes where human judgment creates the greatest value.

‍

Humans continue to provide context, intuition, and discernment the dimensions of hiring that remain difficult to automate while AI supports sourcing and initial assessment.

‍

The recruiter's role therefore shifts upstream from expert intermediary to system architect: instead of manually evaluating every candidate, recruiters can focus on designing systems that enable more consistent and equitable decisions at scale.

Why passing an audit is not the finish line

Good design is necessary, but it is not sufficient. Organizations must also understand what meaningful auditing looks like.

‍

A recent Stanford Human-Centered AI study by Bommassani and colleagues analyzing 4.2 million job applications across 156 employers demonstrated that aggregate audits can conceal position-level disparities. While vendor-level analyses appeared satisfactory, position-level evaluation revealed adverse impact that would otherwise have remained hidden.

‍

The implication is straightforward.

‍

Organizations should ask not only whether an AI system has been audited, but how it has been audited. Position-level analysis can reveal risks that portfolio-level summaries may obscure.

‍

The study also raises a broader governance concern: algorithmic monocultures. As more organizations rely on the same AI providers, weaknesses in a single model have the potential to propagate across an entire sector rather than affecting only one employer.

A framework for responsible AI

As AI becomes increasingly embedded in employment decisions, responsible governance should begin with four practices.

‍

First, name the bias. Organizations cannot interrupt cognitive bias until they recognize where it exists.

‍

Second, design against it. Fairness should emerge from the architecture of the system through structured criteria and attribute-based evaluation, not from initiatives layered onto an inherently biased process.

‍

Third, audit the design. Look beyond aggregate reports and understand how systems perform at the position level.

‍

Finally, watch for algorithmic monocultures. Vendor selection is no longer simply a technology decision. It is a governance decision that requires understanding how models are trained, evaluated, and deployed.

‍

The future of employment fairness will not be determined by whether your organization adopts AI. It will be determined by the questions you ask about how that AI is designed, deployed, and governed.

‍

As you evaluate your own hiring processes, consider the upstream choices you make long before a hiring decision is ever made.