How To Make The Most Of The Kirkpatrick Model To Measure Impact With Jordan Rawlinson

It'd be good to start off with how you got into L&D.  I believe you started in the higher education sector, and then you seem to have transitioned into learning design.

When I was in college, I started getting interested in working in higher education because I was involved in a lot of student leadership organisations, orientation programmes, and training initiatives focused on helping first-year college students transition into university life. I thought that was an interesting way to be impactful in someone's journey because students leave college if they don't have a successful transition, and that can affect the course of their lives in significant ways. 

Through those experiences, I was exposed to training and development, leadership development, team dynamics, and many of the foundational concepts that would later become part of my career in learning and development.

My first job after college was as a leadership development consultant with a higher education nonprofit. I travelled to different universities delivering leadership development programmes for students. In that role, I realised the work-life balance wasn't going to be ideal because you're operating on a college student schedule, which means evenings, weekends, and events outside a traditional nine-to-five environment. 

As I spoke with mentors and people in my network, they began recommending learning and development because the work is very similar, except you're working with adult learners in corporate or business settings rather than college students.

That led me to pivot my job search, and I eventually accepted a role here in Jacksonville, Florida, with a staffing and recruiting company that didn't have any formal training programme. They were frustrated because new hires were learning from their desk partners,  picking up incorrect habits and inconsistent ways of using systems. As you can imagine, that created a lot of bad data and operational challenges. 

I built the training programme from scratch, and that experience eventually led to another similar role with a different staffing company. From there, I became connected with Kirkpatrick Partners, where I am today. I tend to say that I fell into learning and development because I think that's the experience many of us have. But it's been a great way to combine my strengths with my interests, and it's been a really rewarding career path.

That’s a broad range of experience. Out of those situations, why do you think it can be so hard for learning to translate into behaviour change?  

There are certainly a lot of factors. If I think about the main ones I've seen, expectations going into training are one of the biggest barriers to transfer. If someone enters training knowing that they're expected to apply what they've learned on the job, that their manager will ask them about it, or that the training is genuinely important to their success, then motivation increases. 

People pay closer attention, ask better questions, and spend more time thinking about how to apply what they're learning in their own work environment. On the other hand, if someone feels they're only attending because they need to tick a box, or they don't understand why they're there, that naturally leads to less engagement and less behaviour change.

I also think having the resources to be successful after training is incredibly important. Someone may have learned everything they need to know, but if they don't have time to apply it, don't have access to the technology they need, don't know where to ask questions, or don't have resources they can reference afterwards, then applying the learning becomes difficult. There needs to be reinforcement, support, and access to resources, alongside the motivation to use what they've learned.

Those are probably the two biggest factors. If I think about barriers before, during, and after training, engagement and relevance during training obviously matter. If learners aren't engaged or can't see the relevance, transfer will suffer. But from my experience, the biggest issues usually happen before training begins and after it ends. Those are the stages that have the greatest impact on transfer.

Applying that thinking to what you do at Kirkpatrick Partners, what does impact look like to you, and how are you facilitating all of these resources that need to create that impact?

The primary focus of our certification programmes is evaluation. The Kirkpatrick Model is the industry standard for evaluating training effectiveness, and people come to us wanting to understand how to evaluate the impact of their own programmes. In many ways, we're training the trainers.

One advantage we have is that motivation is already fairly high. Most participants understand why they're there and have some familiarity with evaluation, so we don't struggle as much with that initial motivation barrier. Instead, our focus is on reinforcement, encouragement, and follow-up after the programme.

We use technology to help automate that process. One tool we use is Knowly. It's a great way to scale behavioural check-ins and automate follow-up conversations. It allows us to ask learners how they're progressing, what barriers they're encountering, and remind them of the goals they set for themselves. It keeps everything top of mind and helps direct people toward solutions when challenges arise.

We also offer ongoing coaching calls where participants can speak directly with facilitators like me. Rather than trying to solve problems on their own, they can join these sessions and receive personalised guidance. We run these several times each month.

We've also developed an AI chatbot that serves as an in-the-moment coach. If someone has a question between coaching sessions, they can use the AI tool to receive guidance and support. We also try to keep managers involved through Knowly so that there's an ongoing communication campaign supporting learners after training.

Those are some of the primary ways we try to create impact and provide transfer support after our certification programmes.

That’s good to hear. Feedback mechanisms are really important for gathering the data needed to make that impact. 

It occurs to me that one-to-one interviews and focus groups can be great opportunities to gather that feedback. From your perspective, are there any best practices for using those mechanisms to gather the right feedback, whether that's asking certain questions or making people feel comfortable enough to share what they need to share?

It's interesting because the coaching sessions I mentioned aren't technically called interviews or focus groups, but we're usually asking the same kinds of questions. What I find fascinating is that framing makes a huge difference.

If I asked someone, "Would you like to participate in an interview so we can hear how training impacted you?" many people would probably hesitate or say they don't have time. It's similar to how people react to surveys. But if I frame it as a coaching call where we can discuss challenges they're facing and help them work through them, suddenly there's a clear benefit for them. There's a "what's in it for me" element.

Because of that, I think the branding and framing of these conversations is important. You don't necessarily have to call them interviews or focus groups. The questions remain largely the same: How is it going? Are you using what you've learned? What challenges are you facing? Those are still interview-style questions.

Beyond that, I think the most important thing is focusing on behaviours. Whether you're using surveys, interviews, or focus groups, questions centred on the behaviours learners are expected to apply will provide the most valuable information. Rather than spending most of your time asking about favourite activities or what parts of training people enjoyed, it's more useful to focus on what they're actually doing differently as a result of the learning.

For example, if we've taught people how to evaluate training programmes, build surveys, or conduct interviews and focus groups, we can ask specifically about those behaviours. If someone tells us they haven't conducted any Level 2 evaluations, then we can ask why. Maybe they didn't receive enough practice during training. Maybe they encountered barriers in their workplace.

By focusing on behaviour first, we can then explore either the barriers preventing application or, if they are applying the learning successfully, the results they're seeing because of it. The conversation becomes much more meaningful when it's built around real-world application rather than simply reviewing the learning experience itself.

It also occurs to me that a big element of that is the concept of psychological safety. 

How important is it to create psychological safety, and how would you do that from your role in making sure behaviour change can happen?

I think a lot depends on the environment where people are expected to apply what they've learned. Within the learning experience itself, facilitators and trainers can create psychological safety by setting expectations from the beginning. You can make it clear that there are no stupid questions, that participation is encouraged, and that learning will be active and collaborative. Simply setting those expectations often helps people feel more comfortable engaging in the experience.

When it comes to application, however, psychological safety can sometimes be outside our control. During training, we can ask learners what barriers they anticipate facing when they try to apply what they've learned. They may tell us that their manager won't be receptive, that their team won't support the change, or that there are organisational obstacles.

In those situations, we can try to help them think through workarounds or strategies for overcoming those barriers. If we have the ability to influence the environment directly for example, by working with managers or acting as an internal consultant, then those conversations become valuable. If people don't feel safe experimenting with new behaviours because they fear criticism or resistance, that will naturally become a barrier to transfer.

In an ideal world, we'd address all of those barriers before they become problems, but reality isn't always that simple. 

I’m also fascinated by the balance between qualitative data and quantitative data for gathering the stories that ultimately demonstrate impact. The stories often lean toward the emotional and human side of things, although it's always context-specific. 

How can people strike the right balance between making sure both sides are represented when they're trying to demonstrate impact?

I completely agree that both are important. A lot of the Level 1 and Level 2 evaluation data within the Kirkpatrick Model tends to be more structured and straightforward. Questions about the training experience itself, whether learners found the content relevant, whether they felt more confident afterwards, or whether they believed they gained the intended knowledge can often be measured using scales and multiple-choice questions. 

Those types of questions naturally generate quantitative data and can be useful for identifying patterns across large groups of learners.

Where stories become especially valuable is when you're trying to understand the reasons behind those numbers. Qualitative data helps uncover the "why." It helps explain barriers to transfer, unexpected outcomes, and the lived experiences behind behaviour change. When you're dealing with behavioural application, there are often too many variables to capture through predefined survey options alone. You can't realistically anticipate every barrier someone might encounter when applying learning in the workplace, which is why qualitative questions become so important.

A lot of this comes down to thoughtful survey design, interview design, and the conversations you're having with learners after training. What's interesting is that some of the most valuable information I've gathered throughout my career hasn't always come through formal evaluation tools.

Sometimes it's happened in completely informal settings. It might be while grabbing coffee with someone and casually asking how training went, or speaking with a manager whose team recently completed a programme and asking whether they've noticed any changes in behaviour. Those conversations still provide valuable data, even though they're informal and qualitative in nature.

I think the goal should be to intentionally design opportunities for both quantitative and qualitative feedback from the beginning. At the same time, you should remain open to valuable insights emerging in unexpected ways because some of the best information doesn't always arrive through formal channels.

Another part of your roles in the past has involved LMS administration. 

What are your thoughts on how people can choose an LMS that works for them, and when should an LMS even enter the conversation for L&D and training?

I have to admit that learning management systems haven’t been my favourite part of my career, as I prefer the relationship-building aspects more, like spending my time directly with leaders and learners. That said, I do think LMS platforms can be extremely useful when it comes to scalability and automation. If an organisation relies heavily on e-learning, an LMS can help automate many of the administrative aspects of training, including data collection, learner tracking, and basic evaluation. It can also provide useful engagement data, such as how much time learners spend in particular courses and how actively they're interacting with content.

When it comes to selecting a platform, I think one of the most important questions is whether the system has been designed with behaviour change in mind. Many LMS platforms stop at learning. After all, they're learning management systems. Their primary focus is usually content delivery rather than application.

Because of that, I would encourage organisations to ask deeper questions during the selection process. What kinds of post-training automations are available? How does the platform support reinforcement after the learning event? How does it connect learning objectives to workplace behaviours? How does it support follow-up, accountability, and ongoing application?

This becomes particularly important when organisations purchase pre-packaged learning content. Often, the learning objectives are clearly defined, but the expected behaviours are much less clear. Even when the content itself is strong, there may be very little built into the system to reinforce application or measure whether the learning is actually being used.

Those are the kinds of questions I would prioritise because, ultimately, learning alone isn't enough. People can learn all they want, but if that learning doesn't lead to behaviour change, then it's unlikely to generate the results the organisation is hoping to achieve.

On the technology side, a common theme I hear when I ask people about AI is that it can help with productivity, administration, and efficiency, but that the human element can never truly be replaced. 

What are your thoughts on AI and L&D, whether that's positive, negative, or somewhere in between?

I think regardless of anyone's personal opinion, AI is here to stay. At this point, it's less about deciding whether we like it and more about deciding how we're going to work alongside it.

My perspective is that AI can help us become more human because it can remove many of the technical and administrative tasks that consume our time. By handling those responsibilities, AI gives us more space to focus on the things it can't do as effectively. That includes building relationships with stakeholders, asking thoughtful follow-up questions, recognising cultural dynamics within organisations, and reading situations with nuance and context.

AI is obviously valuable for productivity and efficiency, but I also think it can become a direct tool for supporting behaviour change. In our own certification programmes, we've explored building AI tools that learners can use after training. Instead of limiting AI to the design process, we can train tools using our own coaching philosophies and approaches. We can essentially teach the system how we would respond if a learner asked a particular question.

The result is a resource that can provide guidance even when a facilitator isn't immediately available. In a sense, it allows you to be in multiple places at once because the tool has been trained using your expertise and coaching style.

Even with all of that, I don't believe AI will fully replace human professionals. Business has always been built on relationships, and I don't see that changing. The best salespeople are usually the best listeners. They're the people who build trust and understand others. In many ways, everything we do involves persuasion. We're constantly trying to gain buy-in, communicate ideas, influence decisions, and build alignment with leadership.

AI can help create a presentation deck or draft a proposal, but it's the emotional connection that persuades people. That's still a human capability, and I think it will remain that way for a long time.

Another concept I find interesting is that in the post-pandemic world, e-learning has become significantly more common.

In some cases, it may even have created a form of learning fatigue, where people are constantly chasing certifications, courses, and new learning opportunities. It can become overwhelming. 

What are your thoughts on recognising learning fatigue and finding the right learning path for a particular individual and situation?

It still comes back to behaviour change and results. If I'm learning simply for the sake of learning, or because I want another certification on my résumé, that doesn't feel particularly purposeful. Instead, I think it's more valuable to start by identifying the result you're trying to achieve or the problem you're trying to solve.

For example, if my goal is to generate more revenue for my company, that's the result I'm aiming for. From there, I can work backwards and ask what behaviours would contribute to that outcome. Maybe I determine that improving certain sales behaviours would help. Once I've identified the behaviours that need to change, I can then ask what knowledge or skills I need to acquire in order to perform those behaviours effectively.

That's where the learning path becomes much clearer. I think working backwards from desired results is often more effective than simply asking what you're interested in learning. It's wonderful to be a lifelong learner, but time and resources are limited. At some point, we have to prioritise learning that we're going to apply.

When learning is connected to a meaningful outcome and specific behavioural goals, it becomes much easier to avoid learning fatigue because you're no longer trying to learn everything. You're learning with purpose.

How do you continue to evolve in your role? Are there any tools, methods, or approaches that you particularly enjoy learning through?

My biggest source of learning continues to be other people. When I watch someone do something particularly creative or effective, I learn from that. I learn from other trainers, other learning professionals, and other people working in this space. 

Mentors have also played a major role throughout my career. They provide guidance, perspective, and support that goes far beyond technical knowledge. They help you navigate challenges, think differently about opportunities, and continue growing professionally.

Something else I find valuable is the feedback that comes from people who genuinely understand your work. When someone sees you operating at your best and recognises your strengths, that's validating. It's the kind of insight you don't always gain through self-reflection or technology.