Episodes

  • Calling All TA Leaders: Success Requires Embedded Science

    Calling All TA Leaders: Success Requires Embedded Science
    Oct 15 2026

    “We need to have rigor adjusted for reality.” — Allie Wehling

    Episode Overview

    I am lucky to be joined by my friend Allie Wehling. I consider Allie a superstar. She has achieved what I call the gold standard for internal I/Os.

    Allie is a rare example of having an I/O helping guide the business of hiring. At Cisco, she owns both the IO content and the end-to-end hiring process. She sees the whole system and how every piece affects the others.

    Most hiring functions treat science as a bolt-on: a test added to a process nobody designed with measurement in mind. That approach has value, but it leaves results to chance. Consistent, defensible hiring requires a holistic approach, where science, process, and business needs are designed together.

    Join us as we dig into what an embedded science lens brings to hiring, and why TA can’t afford to go without it.

    Topics Discussed & Key Insights

    1. TA Leaders: Embed Science, Don’t Bolt It On

    Assessment usually gets added to hiring without anyone thinking about the whole process — piecemeal jamming assessments into legacy tools and processes. Allie’s role is built differently. Her teams cover both what gets measured and how it’s implemented, from requisition through pre-boarding. Having one leader across both is rare, and it gets results.

    “We have the WHAT, the content, the assessments from our COE, and then the HOW. How are we actually implementing it?”

    2. Science Sees the Whole Funnel, Not Just the Test

    The assessment is one input. Every handoff, from assessment to interview to decisioning, introduces the potential for bias AND the potential for ROI. The trick is pulling both levers in the right direction. And we can’t forget that candidate experience is critical. When top candidates drop out because of slow scheduling or untrained interviewers, the process fails, not the test.

    “because you can have the most beautifully designed and validated assessment and still walk away with a bad hire because the assessment is just one input to the much longer process.”

    3. Science Has to Be in Lockstep With the Business

    What’s scientifically best and what’s operationally viable are different questions, and both have to be solved. Process and technology have to be designed together. Allie’s team works “two in a box” with recruiting tech, and Cisco has hired an AI orchestration lead so the tools feel like one system.

    “they’re two different optimization problems. And if we pretend like they’re one, our systems are going to die quiet deaths.”

    4. Science Prepares the Org. for Transformation

    Allie listens for the problems leaders haven’t voiced yet. When executives kept asking how Cisco assesses for AI, her team built a library of work samples to measure it. Bespoke assessments built from validated items give hiring managers a hand in the content. An internal AI tool sets cut scores against predicted pass rates instead of gut thresholds.

    “I try to listen to what problem are they not even voicing yet.”

    5. Science Keeps You Defensible

    Job analysis documents job relatedness. Vendor claims get tested, not trusted. If a vendor can’t explain what its score predicts, or who’s on its science team, it’s not the right partner.

    “They’re predicting something, but is it the thing that we want to predict?”

    Final Takeaway

    Speed, volume, and AI are making hiring harder to get right. Science is what turns a pile of tools into a system that works effectively, consistently and holds up to scrutiny. It’s not a test you add. It’s a lens you build in.



    This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com
    Show More Show Less
    45 mins
  • You're Not Losing Your Job to AI. You're Handing It Over.

    You're Not Losing Your Job to AI. You're Handing It Over.
    Sep 24 2026
    “If you automate chaos you get chaos at scale.”— Tomas Chamorro-PremuzicEpisode OverviewIn this episode I’m joined for a third time by Tomas Chamorro-Premuzic — organizational psychologist, author, and now the first Chief Science Officer at Russell Reynolds Associates, one of the world’s top executive leadership advisory firms.We set out to role play this conversation as if it were happening in 2027. We didn’t stick to it. We kept sliding between the future, the present, and whatever is right around the corner — which says more about this moment than the role play ever could.We’ll come back in a year to see how we did. Underneath the predictions is one argument: machines can only take what has already been standardized.Topics Discussed & Key Insights1. The Future Is Already Here. It’s Just Not Evenly Distributed.Tomas recently returned from Shenzhen — stores staffed by humanoid robots, selling humanoid robots, with shoppers carrying them out in boxes. No human employees. His point, borrowing William Gibson: “the future is already here, just not evenly distributed.” What looks weird and niche today becomes mainstream fast. That’s the honest baseline for any one-year prediction: 95% of work will look the same next year. The question is which 5% won’t.2. You Can’t Automate What Hasn’t Been StandardizedThe core flip of the episode. Automation doesn’t take jobs — it takes standardized work. As Tomas puts it: “you cannot automate what hasn’t been standardized already.” Which raises his uncomfortable question: are knowledge workers standardizing themselves? Leaning on AI for everything produces what he calls “work slop and cognitive surrender” — the intellectual version of elevator music. Designed to blend in. Designed to be ignored. The more predictable your output, the easier you are to model, and the easier you are to replace.3. Hiring for Potential, Not Track RecordHis first concrete prediction: organizations will hire leaders less for what they have done and more for what they could do. AI enriches the signals needed to model curiosity, humility, coachability, and EQ — the things good assessment science has been chasing for decades. Technical skills, including the AI literacy everyone is worried about, “will be far less important than we think.” Human and humane skills will matter more. Track record measures what’s already been commoditized. Potential is what’s left.4. Autonomous Hiring Splits the MarketI ran a live query on my TA Tech Navigator during the episode. TA Tech Navigator is my online market research portal that contains 540 profiles of candidate evaluation and assessment tech. It has several frameworks that classify AI use and Science and can run trends reports using the aggregate data. The trend it flagged: fully autonomous hiring stacks are moving from fringe to viable. There are now vendors where no human touches the process until the final decision. Employers are splitting into two camps — buying efficiency or buying defensibility. Is the tool validated? Is there something here you can trust? That gap will widen, with regulation as the wildcard. The Navigator’s one-year read: faster, cheaper, more automated — but not fairer, more valid, or more defensible unless somebody demands it.5. The Zombie ProblemsTomas closes with a warning. Obsessing about the future is often an excuse for not dealing with present problems. Two have been with us for decades, and AI doesn’t touch either one: measuring leadership performance objectively, so tools predict actual performance instead of a popularity contest — and our stubborn preference for intuition over validated assessment. The same world that worships data, science, and validity is the world in which the MBTI is the number one personality assessment. New technology doesn’t kill these problems. Only deciding to solve them does.Final TakeawayMachines don’t take work. They take standardized work. The defensible position — for a knowledge worker, a leader, or a hiring process — is to stay hard to predict: keep thinking, keep surprising, and measure the things that actually differentiate people. That’s been the job all along. AI just raised the price of skipping it. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com
    Show More Show Less
    47 mins
  • AI Hiring Law: What Changed While You Weren't Looking?

    AI Hiring Law: What Changed While You Weren't Looking?
    Aug 11 2026
    “The way the law is working — the Workday case is exhibit A here — is that vendors are going to be on the hook if their systems are producing biased outcomes. Period.”— Marc WeinsteinEpisode OverviewIn this episode I’m joined by Marc Weinstein, an attorney with 27+ years of practice focused on organizations that use tests to make high-stakes decisions about people — licensure, certification, admissions, and employment selection. His work centers on the legal rights of people impacted by AI systems and the organizations that use them.The legal ground under AI hiring is moving fast. In the past few months a state AI law got sued into a rewrite, a court handed down a ruling that shields bias audit data from view, and a new lawsuit opened a line of attack nobody was watching. If you buy, sell, or use AI hiring tools, this episode helps you stay on top of the shifting sands.Topics Discussed & Key Insights1. First, how to roll: governance before headlines.Marc’s advice starts above the regulations. Your own governance is the key to successful risk mitigation. Get this part right and the rest gets much easier.* Before you track any law, ask why you want to use AI in the first place. If you have good answers, then ask how* Set principles for your organization — not for the whole world, for your organization. Let them drive your use cases, your vendor selection, and your evaluation criteria* Operate by them for real. A policy that lives on paper is worth nothing* Do this and you’re already in compliance with most of the laws that could touch you. Everything below is what’s shifting underneath that foundation2. Vendor liability is shifting fast — get ready now.For a long time, vendors selling hiring tools got a pass. How the customer used the tool was the customer’s problem. That arrangement is breaking down.* In Mobley v. Workday, the court is letting claims proceed on the theory that the platform acts as an agent of its employer customers — which puts the same federal anti-discrimination laws that apply to employers on the platform* Plaintiffs go where the deep pockets are. Mega vendors like Workday and 8fold are the deep pockets* Employers are still independently on the hook. Vendor liability adds to yours, it doesn’t replace it* Newer legislation points the same direction, and California’s new regulations make bias testing itself evidence in court — more on that below* Since we recorded: the court refused to dismiss most of the remaining claims against Workday. The case keeps moving, and the core theory keeps surviving3. Bias audit data can be withheld — and that says a lot.A discovery ruling in the Workday case protected the company’s bias testing data from the plaintiffs. Worth sitting with what that means.* The court held the data was privileged because lawyers curated it and the testing was done to provide legal advice — not for business use* Publicly advertising that you conduct bias testing did not waive the privilege* So the people alleging harm from the system may never see the data that would show whether the system harmed them* The bigger question: if the most important evidence about how these tools perform can be shielded, what does accountability actually look like? Proof may have to come from other directions4. There are new ways to hold these tools accountable.The Eightfold case shows creative lawyering finding routes into AI hiring that didn’t exist a year ago.* The claim is built on the Fair Credit Reporting Act and privacy law — the argument is that AI-generated candidate scores are consumer reports, compiled without the required disclosures and consent* These theories don’t require proving discrimination as a legal element, which makes them easier cases to bring* The target hasn’t changed. It’s still about how these tools treat people. There are just more ways to get there now5. Colorado is the cautionary tale: get aggressive, get crushed.Colorado passed the most rigorous state AI law in the country — mandatory risk management, impact assessments, a real duty of care. Then it got taken apart.* xAI sued to block the law. The DOJ intervened on xAI’s side — the first time the federal government moved to invalidate a state AI law* The legislature rewrote the law under pressure. The replacement is a notice-and-transparency framework, delayed to January 2027* The lesson for everyone else: states that regulate AI aggressively are going to get sued by the federal government and the private sector. Plan around that reality6. Meanwhile, California quietly became the strictest.While everyone watched Colorado, California put real requirements on the books — and nobody has sued to stop them.* Under the FEHA regulations, conducting a documented bias test supports an employer’s defense in discrimination litigation. Not conducting one supports the plaintiff. Records must be kept four years* The privacy agency’s automated decision-making rules ...
    Show More Show Less
    54 mins
  • You Bought the AI. So Why Aren't You Using It?

    You Bought the AI. So Why Aren't You Using It?
    Jun 29 2026
    “You buy the hiring platform, and then you don’t have these AI workflows enabled... here you are three, four years later. You’re still not able to do that.”— Nicole MundyEpisode OverviewIn this episode I’m joined by Nicole Mundy, Senior Research Analyst at Talentech Labs — a research and advisory firm that helps its enterprise clients evaluate and procure hiring technology systems.Nicole brings a valuable perspective to the table because she sees the dynamic between vendor and buyer up close and personal. When combined with my experience in this same realm from the science side- our discussion shines light on the reality of what is happening in AI tech adoption for TA.Topics Discussed & Key Insights1. Companies are buying AI hiring tools at scale — and then leaving them switched off.Among the world’s top enterprise organizations, Nicole estimates a surprisingly small amount are actually using AI to automatically assess active applicants.Why?Approvals never come - Companies buy the platform intending to enable the AI, “once the right approvals are in place- but years often pass without any change.Pilots underperform - Big companies test these tools and often conclude they can’t really use them, or they just didn’t work.Lack of solid ROI evidence - Despite the best intentions- most companies do not do the follow up work needed to demonstrate the impact of these tools on the bottom line. Legal ambiguity freezes decisions - With regulations constantly in flux, risk management often takes priority over business needs.2. Validation is misunderstood and absent.What vendors without I/O science guidance call validation isn’t what legal compliance actually requires.* Vendors are quick to speak about the validity of their solution and talk endlessly about validating their AI models — running statistical checks that the model predicts consistently and de-biasing its outputs across groups. This is purely empirical work.* But that’s IT-style validation. It confirms the system runs as built; it says nothing about whether the tool is fair or job-related* Validation for legal compliance, and sound science, demands a blend of rational and empirical work to document the job-relatedness of any tool used to make employment decisions3. “Skills” are everywhere, and nowhere.Skills-based hiring is the headline everyone wants. The problem is what counts as a skill.* Most platforms apply the “skill” label with no objective framework to define it. A skill ends up being little more than a tag like “Excel” or “communication”* The definitions behind these labels are usually poorly organized and loosely constructed. * There is no connection between the skills a platform claims to measure and any outcome on the job. Without that link, there is nothing for a buyer to trust or defend.* The companies doing skills-based hiring well are not buying one vendor and flipping a switch. They run multi-year programs: define the skills objectively, inventory what the organization has against what it needs, curate tools carefully on the front end, and collect assessment data at multiple points.4. Cheating is a zero-sum game, so let’s change the rulesAI-assisted candidate fraud, from AI completing assessments to coaching candidates through interviews, is driving enterprises toward more dynamic evaluation that is harder to game. But chasing detection is largely a losing battle.* Trying to catch and block AI use is whack-a-mole. It’s a zero-sum game, and it’s frustrating, because the tools keep getting better and the detection never really gets ahead* There’s a more useful way to think about it. People are going to use AI on the job, so why not let them use it in the application process in a controlled way? The question stops being “did they use AI” and becomes “how well do they use it”* Most large organizations are still just trying to get visibility on how much cheating is happening and what it looks like, which says how early everyone is on this.* Across the approaches Nicole sees, one thing tends to hold up whether a process is locked down or fully AI-assisted: competency-based follow-up questions that make candidates explain their own reasoning in their own words.Final TakeawayEnterprise isn’t slow on AI in hiring because it doesn’t understand the technology. It’s slow because the tools are bought on vendor claims that were never reviewed against real science, and the danger only becomes clear once the tool is in play. The companies getting it right aren’t chasing tools. They’re building programs, science first. Held to that bar, many of the tools on the market today wouldn’t survive the review, and the ones that would wouldn’t be sitting switched off. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com
    Show More Show Less
    46 mins
  • “The answer isn’t more AI — it’s better signal.”

    “The answer isn’t more AI — it’s better signal.”
    Jun 4 2026

    In this episode I’m joined by Robert Newry, Founder & CEO of the assessment company Arctic Shores and long time champion of doing assessment right!

    Robert and I (and my AI co-host Mayda Tokens!) dig into one of the most urgent problems in hiring right now: the complete breakdown of traditional hiring signals.

    We ponder the question- “How do we find the truth in an age where AI has flooded the top of the funnel, made credentials and resumes unreliable, and put enormous pressure on organizations to find new ways to identify talent?”

    And we come up with some pretty good answers!

    1. The Top of the Funnel Is in Chaos

    The numbers are staggering. Accenture’s global resourcing lead told Robert they’re on pace for 12 million applications this year for roughly 100,000 hires — up from 4 million just three years ago. Same size team. Two and a half times the volume. The culprit isn’t a surge in qualified candidates; it’s AI-powered application tools that let candidates apply to jobs while they sleep. The moral contract between candidates and employers has been broken: candidates assume companies are using AI to screen, so they’re using AI to apply.

    “It’s chaos out there. Candidates are using AI to fight AI — and we’re in a no-win scenario.”

    2. Traditional Assessment Is Increasingly Gameable

    Arctic Shores’ research from 18 months ago showed what most people didn’t want to admit: AI can ace virtually any traditional assessment format — personality tests, cognitive reasoning, multiple choice — with ease. And it’s not just about having a second screen open. Candidates can now point a phone at their screen, have the AI read the item, and get the answer instantly. Proctoring doesn’t solve this. The old protection mechanisms are obsolete.

    3. The Answer Is Better Signal, Not More AI

    The solution isn’t to ban AI from the process — it’s to design assessments that AI can’t easily game because they’re rooted in authentic behavior. Robert’s framework: if AI is being used to evaluate signals, those signals have to be grounded in high-fidelity behavioral data — not scraped from job descriptions, not inferred from keyword matching, not built on garbage in.

    Job descriptions themselves are often the first failure point, and no amount of downstream AI sophistication fixes a weak foundation.

    4. Stop Counting Leaves — Look at the Roots

    Robert’s tree analogy is one of the sharpest frameworks in this episode. For decades, hiring has been obsessed with leaves — the skills on a resume, the credentials on a LinkedIn profile. But with the average shelf life of a skill now estimated at two and a half years, leaves are increasingly unreliable.

    What matters is the root system: the durable human capabilities that allow someone to grow new skills, adapt to changing roles, and thrive in uncertainty.

    5. Skills-Based Hiring Needs a Clearer Definition of “Skill”

    Both Robert and I agree: the skills-based hiring movement is directionally right, but conceptually messy. Calling “innovation” or “persistence” a skill conflates what can be learned with what is innate. Durable traits — personality, cognitive style, learning orientation — don’t expire the way technical skills do. Measurement strategy has to account for these differences, or skills-based hiring just becomes the next echo chamber.

    Final Takeaway

    The hiring signal crisis is real — and it’s accelerating. AI has made it trivially easy to fake credentials, game traditional assessments, and flood the funnel with noise.

    The organizations that receive the best signal won’t be the ones that deploy the most AI. They’ll be the ones that invest in the right signal: behavior-based, validated, and rooted in the durable human traits that no machine can fake.

    *Claude.ai assisted with the creation of these show notes



    This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com
    Show More Show Less
    54 mins
  • Going All In: Using AI to Build Better Assessments

    Going All In: Using AI to Build Better Assessments
    Apr 27 2026

    “By the time you dot the final I’s and cross the final T’s, the assessment is already out of date.”

    — Taylor Sullivan

    Episode Overview

    In this episode I’m joined by rising I/O rockstar Taylor Sullivan, IO psychologist and the architect of Workera’s assessment strategy. With Taylor’s guidance Workera, a verified skills intelligence platform, is doing something most of the industry is still afraid to do: going all in on using AI to build, deliver, and validate AI-based assessments.

    Taylor and I (and my AI co-host Mayda Tokens) dig into how this actually works, why it’s scientifically defensible, and why the industry needs to stop waiting and start moving.

    Topics Discussed & Key Insights

    1. Traditional Assessment Development Is Already Broken

    By the time a traditional assessment clears all the I-dotting and T-crossing, it’s often already out of date. AI changes that — enabling dynamic content generation, richer construct understanding, and real-time iteration that keeps pace with how work actually evolves.

    2. Codifying Measurement Science Into a Multi-Agent System

    Workera didn’t just bolt AI onto existing processes. They embedded IO psychology’s core principles — evidence-centered design, validity frameworks, job analysis — directly into a multi-agent authoring system. Experts define the standards. Agents execute to those standards. The science drives the machine, not the other way around.

    Here’s a brief sketch of how it works in practice

    * Define the purpose — Tell the agent what you’re measuring and why. This grounds everything that follows.

    * Extract the construct — The agent probes the skill space using critical incident techniques, identifying what great performance actually looks like.

    * Design the assessment — The agent selects question formats (multiple choice, drag and drop, voice interaction, sequencing) based on what will best elicit evidence of the skill.

    * Automated quality review — Before anything goes live, the system checks for bias, language issues, and content alignment to the original skill definition.

    * Monitor and improve — Once deployed, the agent tracks response patterns, flags problems, and learns from score appeals adjudicated by humans.

    The skill domain is flexible — it works for cheeseburgers or cybersecurity. The methodology behind it is the same either way.

    3. The “Harness” — Why This Is Safe

    The key to responsible agentic AI isn’t less autonomy — it’s a well-designed harness (the constrained ecosystem where the agents do their thing). Human experts define what good looks like, set quality thresholds, and build in escalation points. The agents work within those constraints and loop back when they hit uncertainty. As Taylor puts it: “It’s not running completely autonomously unchecked.”

    4. This Is About Development, Not Just Hiring

    Workera’s primary focus is post-hire — workforce development, upskilling, and learning. Once an assessment identifies verified gaps in a person’s skills, the platform connects those gaps directly to personalized learning plans, curating from an organization’s existing content library. Two people can get the same score on an assessment and walk away with completely different development paths based on their specific pattern of strengths and gaps.

    5. Verified Skills Intelligence — What It Actually Means

    In a world where AI can write a perfect resume and LinkedIn profile for anyone, credentials are noise. Verified skills intelligence cuts through that — using assessment to generate actual evidence of what someone can do, fit for the stakes of the decision being made.

    Final Takeaway

    The tools to move beyond multiple choice, beyond static assessments, and beyond slow validation cycles exist today. The bottleneck isn’t technology — it’s the will to trust well-designed systems. When the science is built into the machine from the start, speed and rigor aren’t in conflict. They’re the same thing.



    This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com
    Show More Show Less
    48 mins
  • Quality Research Shows the Real Impact of AI @ Work

    Quality Research Shows the Real Impact of AI @ Work
    Mar 30 2026

    Quote:

    “If you know what you’re doing, AI makes you faster. If you don’t, it just makes you wrong faster.”

    –Louis Hickman

    In this episode I’m joined by esteemed Psych Tech @ Work, Alumnus and AI research machine, Louis Hickman. Our incredible conversation taps into Louis’ myriad research studies to unpack AI’s direct impact on work, domain expertise, and talent assessment.

    And of course, this episode also marks the return of the now new and improved AI podcast co-host Mayda Tokens (2.0).

    Besides telling dumb jokes- Mayda’s job is to remind us that AI isn’t just a tool — it’s becoming an active participant in how we think, question, and explore ideas.

    In the course of our conversation Mayda and I coax some PROFOUND take aways from our friend Louis as he shares the practical outcomes of his research:

    1. AI is not removing the need for expertise — it’s making it more visible.

    Scaling intelligence is easy.Scaling judgment is not.

    The organizations that succeed won’t be the ones that adopt AI the fastest.

    They’ll be the ones that:

    * Understand what they’re measuring

    * Use AI to enhance — not replace — that understanding’

    * Maintain control over how decisions are made

    2. AI allows us to scale both good science and bad measurement

    Louis pushes back on the idea that recent advances represent a fundamental shift in how we measure people. Instead, what we’re seeing is:

    * Better models

    * Faster processing

    * More scalable systems

    But none of that replaces the need for valid, reliable, and job-relevant measurement.

    3. AI doesn’t level the playing field — it often rewards those who already understand the game.

    One of the most interesting ideas in this episode is how AI interacts with individual differences in expertise.

    At a high level:

    * For simple tasks, AI helps novices perform closer to experts

    * For complex tasks, AI actually widens the gap- allowing experts to perform better

    Why?

    Because experts know how to ask better questions, recognize when AI is wrong, and refine its outputs—while novices often lack the ability to judge quality, diagnose errors, or course-correct when things go off track.

    4. Replicability in LLMs Is Possible — if you know how to set it up right

    A major “wow” moment in Louis’ research:

    By running the model locally on the same class of hardware, fixing the model and prompt, and turning off sampling/randomness in the settings, you can make the system produce the same output for the same input every time.

    5. AI should be used to scale decisions, but those decisions still need to be grounded in clearly defined constructs

    At this point, AI adoption isn’t optional—it’s expected. Organizations are being pushed to move faster and scale, while vendors are rapidly building and deploying solutions, often without deep validation.

    The resolution isn’t to slow down adoption—it’s to ensure we add and maintaining rigor.

    6. AI makes it easy to scale assessment, but if the underlying design is weak, we’re just scaling bad measurement faster.

    The resolution is to ensure what gets scaled is built on clear constructs, strong design, and validated measurement, so speed amplifies quality—not noise.

    7. Working with AI is no longer just about what you can do—it’s about how effectively you can partner to make what you do better!

    The tension is clear: AI can accelerate work, but over-reliance without critical evaluation leads to lower quality, missed errors, and reduced trust.

    This shows up in real ways—unchecked outputs, declining attention to detail, and growing skepticism in collaborative work.

    The resolution is that AI doesn’t replace accountability—users still need to apply judgment, review outputs, and take ownership of the final result.

    Tune in to get the full story on these profound revelations and hear Mayda’s stand up comedy routine.



    This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com
    Show More Show Less
    58 mins
  • The Truth About AI Based Talent Assessment

    The Truth About AI Based Talent Assessment
    Feb 23 2026
    “The rules haven’t changed. The technology has — but the rules haven’t.” — Nathan MondragonEpisode OverviewIn this episode, I’m joined by my old friend (and now co-worker!) Nathan Mondragon, an IO psychologist and long-time leader in creating the future at the intersection of assessment science, hiring technology, and applied AI.Nathan and I have lived through multiple waves of “this will change everything” technology — from early online testing to video interviewing, machine learning, and now generative AI. And the beat goes on!Nathan and I have recently joined forces at ProboTalent where we are creating defensible AI based assessment tools.We talk about where AI has genuinely moved the field forward, where it hasn’t, and why so many of the debates we’re having today are versions of conversations we’ve been having for decades. Along the way, we unpack Nathan’s paradigm busting work at HireVue’, and why the fundamentals of good measurement haven’t changed — even as the tools have.Topics Discussed & Key Insights1. The Rules of Good Assessment Haven’t Changed — We Just Keep Forgetting ThemNathan makes a point that anchors the entire episode: while technology has advanced dramatically, the core rules of good assessment — validity, relevance, interpretability, and fairness — are exactly the same.AI doesn’t get a pass on methodology. If anything, it raises the bar for rigor, because mistakes scale faster.2. Early Hiring Tech Was Built to Solve Operational Problems, Not Measurement ProblemsWe talk about the early days of online hiring and assessment, where the primary goal was digitization, not insight. Systems were designed to move paper processes online, not to improve how well we understand people.That legacy still shapes today’s platforms — and explains why so many tools feel efficient but shallow.3. HireVue Was a Real Paradigm Shift — and It Required Scientific CourageNathan reflects on the early days of HireVue and why it was genuinely revolutionary at the time. The breakthrough wasn’t just video — it was the larger shift toward digitizing and scaling structured assessment experiences in a way the field hadn’t seen before.What made this moment interesting from an IO psychology standpoint is that it required a different mindset as a scientist: being willing to engage with a new modality, even when the measurement implications weren’t fully understood yet. Innovation in assessment has always involved tension — between rigor and experimentation, between what’s proven and what’s possible.Nathan shares what it was like to help lead through that transition, and why thoughtful scientists have to be able to sit with uncertainty long enough to shape new approaches responsibly, rather than rejecting them outright.4. AI Didn’t Create Bad Measurement — It Made It Easier to ScaleA recurring theme: AI doesn’t magically improve weak constructs. If you feed it noisy proxies, you just get faster, more confident noise.We discuss why generative AI and machine learning don’t eliminate the need for careful construct definition — and why “it correlates” is not the same thing as “it measures something useful.”5. Interactivity Matters More Than ModalityOne of the most important takeaways: the future of assessment isn’t about whether something is text, video, or simulation-based — it’s about how interactive and information-rich the experience is.Nathan explains why dynamic interaction reveals far more about decision-making, reasoning, and capability than static prompts ever will.6. Native AI vs. Embedded AI Is a False DebateWe unpack the difference between “AI-native” products and traditional tools with AI layered on top — and why this distinction often misses the point.What matters isn’t where AI lives in the stack, but whether it’s being used to improve interpretation, not just automate scoring or classification.7. Skills and Knowledge Are Still Hard to Measure — and AI Has to Be Used CarefullyWe close by confronting a reality the market often underestimates: skills and knowledge testing have always been difficult to do well, and scaling them without losing rigor is even harder.We connect this directly to the work we’re doing at Probo Talent, where the focus is on a more responsible alternative: using AI to scale the parts of assessment that have historically been hardest to scale, while staying within safe, established modalities and an explainable, scientifically grounded wrapper. The goal is not novelty for its own sake, but a practical example of how AI can be used carefully to solve long-standing problems in skills-based hiring without sacrificing defensibility or trustFinal TakeawayAI changes how we can build hiring and assessment systems — but it doesn’t change what makes them good.If we ignore decades of psychological science in favor of speed, novelty, or convenience, AI will simply help us make the same ...
    Show More Show Less
    49 mins