In this episode I’m joined by Robert Newry, Founder & CEO of the assessment company Arctic Shores and long time champion of doing assessment right!

Robert and I (and my AI co-host Mayda Tokens!) dig into one of the most urgent problems in hiring right now: the complete breakdown of traditional hiring signals.

We ponder the question- “How do we find the truth in an age where AI has flooded the top of the funnel, made credentials and resumes unreliable, and put enormous pressure on organizations to find new ways to identify talent?”

And we come up with some pretty good answers!

1. The Top of the Funnel Is in Chaos

The numbers are staggering. Accenture’s global resourcing lead told Robert they’re on pace for 12 million applications this year for roughly 100,000 hires — up from 4 million just three years ago. Same size team. Two and a half times the volume. The culprit isn’t a surge in qualified candidates; it’s AI-powered application tools that let candidates apply to jobs while they sleep. The moral contract between candidates and employers has been broken: candidates assume companies are using AI to screen, so they’re using AI to apply.

“It’s chaos out there. Candidates are using AI to fight AI — and we’re in a no-win scenario.”

2. Traditional Assessment Is Increasingly Gameable

Arctic Shores’ research from 18 months ago showed what most people didn’t want to admit: AI can ace virtually any traditional assessment format — personality tests, cognitive reasoning, multiple choice — with ease. And it’s not just about having a second screen open. Candidates can now point a phone at their screen, have the AI read the item, and get the answer instantly. Proctoring doesn’t solve this. The old protection mechanisms are obsolete.

3. The Answer Is Better Signal, Not More AI

The solution isn’t to ban AI from the process — it’s to design assessments that AI can’t easily game because they’re rooted in authentic behavior. Robert’s framework: if AI is being used to evaluate signals, those signals have to be grounded in high-fidelity behavioral data — not scraped from job descriptions, not inferred from keyword matching, not built on garbage in.

Job descriptions themselves are often the first failure point, and no amount of downstream AI sophistication fixes a weak foundation.

4. Stop Counting Leaves — Look at the Roots

Robert’s tree analogy is one of the sharpest frameworks in this episode. For decades, hiring has been obsessed with leaves — the skills on a resume, the credentials on a LinkedIn profile. But with the average shelf life of a skill now estimated at two and a half years, leaves are increasingly unreliable.

What matters is the root system: the durable human capabilities that allow someone to grow new skills, adapt to changing roles, and thrive in uncertainty.

5. Skills-Based Hiring Needs a Clearer Definition of “Skill”

Both Robert and I agree: the skills-based hiring movement is directionally right, but conceptually messy. Calling “innovation” or “persistence” a skill conflates what can be learned with what is innate. Durable traits — personality, cognitive style, learning orientation — don’t expire the way technical skills do. Measurement strategy has to account for these differences, or skills-based hiring just becomes the next echo chamber.

Final Takeaway

The hiring signal crisis is real — and it’s accelerating. AI has made it trivially easy to fake credentials, game traditional assessments, and flood the funnel with noise.

The organizations that receive the best signal won’t be the ones that deploy the most AI. They’ll be the ones that invest in the right signal: behavior-based, validated, and rooted in the durable human traits that no machine can fake.

*Claude.ai assisted with the creation of these show notes

“By the time you dot the final I’s and cross the final T’s, the assessment is already out of date.”

— Taylor Sullivan

Episode Overview

In this episode I’m joined by rising I/O rockstar Taylor Sullivan, IO psychologist and the architect of Workera’s assessment strategy. With Taylor’s guidance Workera, a verified skills intelligence platform, is doing something most of the industry is still afraid to do: going all in on using AI to build, deliver, and validate AI-based assessments.

Taylor and I (and my AI co-host Mayda Tokens) dig into how this actually works, why it’s scientifically defensible, and why the industry needs to stop waiting and start moving.

Topics Discussed & Key Insights

1. Traditional Assessment Development Is Already Broken

By the time a traditional assessment clears all the I-dotting and T-crossing, it’s often already out of date. AI changes that — enabling dynamic content generation, richer construct understanding, and real-time iteration that keeps pace with how work actually evolves.

2. Codifying Measurement Science Into a Multi-Agent System

Workera didn’t just bolt AI onto existing processes. They embedded IO psychology’s core principles — evidence-centered design, validity frameworks, job analysis — directly into a multi-agent authoring system. Experts define the standards. Agents execute to those standards. The science drives the machine, not the other way around.

Here’s a brief sketch of how it works in practice

* Define the purpose — Tell the agent what you’re measuring and why. This grounds everything that follows.

* Extract the construct — The agent probes the skill space using critical incident techniques, identifying what great performance actually looks like.

* Design the assessment — The agent selects question formats (multiple choice, drag and drop, voice interaction, sequencing) based on what will best elicit evidence of the skill.

* Automated quality review — Before anything goes live, the system checks for bias, language issues, and content alignment to the original skill definition.

* Monitor and improve — Once deployed, the agent tracks response patterns, flags problems, and learns from score appeals adjudicated by humans.

The skill domain is flexible — it works for cheeseburgers or cybersecurity. The methodology behind it is the same either way.

3. The “Harness” — Why This Is Safe

The key to responsible agentic AI isn’t less autonomy — it’s a well-designed harness (the constrained ecosystem where the agents do their thing). Human experts define what good looks like, set quality thresholds, and build in escalation points. The agents work within those constraints and loop back when they hit uncertainty. As Taylor puts it: “It’s not running completely autonomously unchecked.”

4. This Is About Development, Not Just Hiring

Workera’s primary focus is post-hire — workforce development, upskilling, and learning. Once an assessment identifies verified gaps in a person’s skills, the platform connects those gaps directly to personalized learning plans, curating from an organization’s existing content library. Two people can get the same score on an assessment and walk away with completely different development paths based on their specific pattern of strengths and gaps.

5. Verified Skills Intelligence — What It Actually Means

In a world where AI can write a perfect resume and LinkedIn profile for anyone, credentials are noise. Verified skills intelligence cuts through that — using assessment to generate actual evidence of what someone can do, fit for the stakes of the decision being made.

Final Takeaway

The tools to move beyond multiple choice, beyond static assessments, and beyond slow validation cycles exist today. The bottleneck isn’t technology — it’s the will to trust well-designed systems. When the science is built into the machine from the start, speed and rigor aren’t in conflict. They’re the same thing.

Quote:

“If you know what you’re doing, AI makes you faster. If you don’t, it just makes you wrong faster.”

–Louis Hickman

In this episode I’m joined by esteemed Psych Tech @ Work, Alumnus and AI research machine, Louis Hickman. Our incredible conversation taps into Louis’ myriad research studies to unpack AI’s direct impact on work, domain expertise, and talent assessment.

And of course, this episode also marks the return of the now new and improved AI podcast co-host Mayda Tokens (2.0).

Besides telling dumb jokes- Mayda’s job is to remind us that AI isn’t just a tool — it’s becoming an active participant in how we think, question, and explore ideas.

In the course of our conversation Mayda and I coax some PROFOUND take aways from our friend Louis as he shares the practical outcomes of his research:

1. AI is not removing the need for expertise — it’s making it more visible.

Scaling intelligence is easy.Scaling judgment is not.

The organizations that succeed won’t be the ones that adopt AI the fastest.

They’ll be the ones that:

* Understand what they’re measuring

* Use AI to enhance — not replace — that understanding’

* Maintain control over how decisions are made

2. AI allows us to scale both good science and bad measurement

Louis pushes back on the idea that recent advances represent a fundamental shift in how we measure people. Instead, what we’re seeing is:

* Better models

* Faster processing

* More scalable systems

But none of that replaces the need for valid, reliable, and job-relevant measurement.

3. AI doesn’t level the playing field — it often rewards those who already understand the game.

One of the most interesting ideas in this episode is how AI interacts with individual differences in expertise.

At a high level:

* For simple tasks, AI helps novices perform closer to experts

* For complex tasks, AI actually widens the gap- allowing experts to perform better

Why?

Because experts know how to ask better questions, recognize when AI is wrong, and refine its outputs—while novices often lack the ability to judge quality, diagnose errors, or course-correct when things go off track.

4. Replicability in LLMs Is Possible — if you know how to set it up right

A major “wow” moment in Louis’ research:

By running the model locally on the same class of hardware, fixing the model and prompt, and turning off sampling/randomness in the settings, you can make the system produce the same output for the same input every time.

5. AI should be used to scale decisions, but those decisions still need to be grounded in clearly defined constructs

At this point, AI adoption isn’t optional—it’s expected. Organizations are being pushed to move faster and scale, while vendors are rapidly building and deploying solutions, often without deep validation.

The resolution isn’t to slow down adoption—it’s to ensure we add and maintaining rigor.

6. AI makes it easy to scale assessment, but if the underlying design is weak, we’re just scaling bad measurement faster.

The resolution is to ensure what gets scaled is built on clear constructs, strong design, and validated measurement, so speed amplifies quality—not noise.

7. Working with AI is no longer just about what you can do—it’s about how effectively you can partner to make what you do better!

The tension is clear: AI can accelerate work, but over-reliance without critical evaluation leads to lower quality, missed errors, and reduced trust.

This shows up in real ways—unchecked outputs, declining attention to detail, and growing skepticism in collaborative work.

The resolution is that AI doesn’t replace accountability—users still need to apply judgment, review outputs, and take ownership of the final result.

Tune in to get the full story on these profound revelations and hear Mayda’s stand up comedy routine.