Can an Algorithm Replace a Recruiter?

We at SAP, recent winners of ERE awards for branding and technology, thrive on pushing the envelope, disrupting existing norms and perceived recruitment wisdoms. One of the hot topics in the recruitment industry today is whether an algorithm can replace a recruiter.

In seeking to answer this, we challenged the very traditional university recruiting model.

As far as the question we posed in the headline: WOW. That’s a controversial question!

Perhaps it’s a sensitive one for recruiters to read.

It’s an age-old question that has never truly been answered. Until today …

In a high-tech world, can a computer replace a recruiter? Or more precisely, can an algorithm replace a recruiter?

One of us — Matt — is going to talk about this more in San Diego this month on his panel, but let us say for now that the answer is yes. And we proved it in the field of university recruitment.

Cultural Fit Assessment_ScreenshotUniversity recruiting is an area that seriously needs disruption. In some ways it can be viewed as archaic, some may say elitist. Why elitist? Companies follow the same tired techniques of going to only a handful of the top universities. You know the ones: MIT, USC, Harvard, Oxford, Cambridge, etc. Companies build relations with the professors to curry favor and be able to attract the “best” graduates from that university, and if targeted, from a specific course. They sponsor the course or invest time in curriculum development to forge even closer relations with the university and be in a better position to attract the “best” graduates.

Does that feel right in 2015? Is that an effective strategy in an interconnected, high-tech society? Do the best students always go to the best universities? What if a student cannot afford to study at a top university? What if they have to stay in their local area to stay with family if they are old/ill? What if a student simply chooses a different university? Are companies best served by effectively excluding students from other universities? That surely can’t be the best strategy in 2015.

We quickly came to the conclusion that the best graduates for our Sales Academy may not study at the top universities. A great salesperson has many different skills and characteristics. Sales requires mental toughness. A great salesperson possesses drive and revels in the thrill of the chase of the sale. They are addicted to achieving results against the odds and they don’t get flustered when presented with an obstacle (a client saying no for the first or even tenth time). The best salespeople have a Zen-like ability to focus on the task at hand, accept rejection, and bounce right back. They close deals because they challenge decision-makers but build great trusting relationships. Those are skills that don’t just come from the best universities. Many are inherent, natural character traits. Why limit the talent pool?

Hence we set ourselves the goal at SAP to move away from the old “elitist” strategy and achieve the democratization of university recruiting. Let’s throw the SAP doors open to all the students of the world. A truly meritocratic selection process, no matter what university they attend. If they are a great salesperson and we see that potential, #BOOM, let’s hire them!

Not all innovatory ideas gain initial applause or recognition. Imagine the confused and scared reactions: “Oh my, you are opening the doors to thousands of applications, and we don’t have the recruiters to assess them all. How do we pre-screen them? A chorus of sceptics sang.”

This was very true. Of course, recruiters want to revert to traditional sifting techniques. They would select by: looking at the university the student is attending; exam results; degree grade projections; and maybe differentiating between the “usual crowd” if the student has work experience, be it an internship or gap year experience. But that would be reverting to type and employing the subjective stereotyping of the system we wanted to replace.

What if we generate applications through social media and digital marketing campaigns and then drive those thousands of interested applicants’ online response through an online assessment tool? That would remove the pre-screening of CVs/resumes by recruiters. What if the online assessments could provide the candidates with the ultimate experience: real-time candidate feedback. Getting a pass/fail in real time. Now that would be truly special.

Situational Judgement Assessment_ScreenshotSo that was our goal — create a unique process that would actually mean something to candidates. We partnered with an assessment specialist to create an online assessment tool that was split into two valuations. The first, a 10-minute SAP cultural test that was pass/fail. Next, if a graduate passed the cultural assessment, a 20-minute situational judgment assessment. Again, this second test was pass/fail. Pass and the recruiter calls and arranges a bootcamp, a day-long assessment. Fail, you get that feedback in real-time and can move on. No waiting for weeks of never hearing back from a recruiter wondering if your application would be successful. No resume rotting in an online database, gathering dust as each day passes with no feedback, all the while the employment brand deteriorates in the eye of that applicant. Sometimes “no” is the best answer to get. It is closure.

Online assessments can be tricky. We wanted it to act as the cultural guardian for SAP as well as test the candidates’ reactions in different situations. How did we create an online assessment that worked for us?  Hours of focus groups with business leaders, graduates, recruiters, HR. We needed to define WGLL (“What Great Looks Like” at SAP). Those key components were defined as:

Enterprising: Thrives with change, high energy, challenging the business

Team Orientation: Shares and cares for others

Impact:Leaves their mark. Entertains, engages, and influences

Tough-mindedness: Resilient. Able to work under pressure.

Curiosity: Insatiable curiosity for their area and clients. Thinks different.

Gathering info: The depth and breadth of inquisitiveness that someone will show when they are trying to find information and how they then keep and share this info.

Influencing Others: Creating win/win relationships within broad networks. Understanding the needs of key people and presenting mutually beneficial solutions.

Article Continues Below

Creating Ideas & Processing Information: The level to which someone will evaluate different ideas, selecting an effective solution. Forming ideas as a result of seeing patterns from an assortment of information.

Building Confidence: The ability to be decisive and build morale in others.

So we launched an ambitious online digital marketing campaign. We partnered with LinkedIn to use the many thousands of graduates in their network. We drove awareness via Facebook, Twitter, various targeted job boards, and university mailing lists. For us, targeted digital marketing was key. All these applicants were driven to our assessment tool, a tool that was designed to wow candidates, provide a good experience, and most importantly tell them, there and then, if they were successful in their application. Successful candidates went to a bootcamp at one of SAP’s key locations across the world to face a day of fun assessments and insights into our company.

The results:

In assessing the key achievements, several measurements were employed:

Number of website visitors: 1.2 million visitors to the site globally

Number of people ‘applying’ and starting assessment: 50,000+

Number of hires: 500 globally

Number of complaints received: Zero complaints about candidate experience

Projected cost saving: Projected cost saving of £250,000 in year one (less recruiter screening time)

Number of dropouts in application: Huge improvement from 93 percent to 25 percent

Applicant questionnaires revealed:

  • 75 percent of candidates said that the online tools boosted their motivation to progress to application
  • 88 percent said online tools were more engaging than other graduate applications
  • 100 percent said that the bootcamp increased their motivation to work at SAP, were happy they took part, and thought the experience provided a great insight into working at SAP


This new direction in graduate recruitment represented a strategic change for SAP. It was innovative. Some have told us that it is potentially industry redefining. You can be the judge of that.

What did we achieve?

  • Real-time candidate feedback: Immediate pass/fail result
  • A great candidate experience: You know where you stand
  • “Democratization of recruitment”: Candidates from hundreds of universities were targeted via social media and LinkedIn and could apply
  • “Meritocracy of application”: Selected on fit and ability, not the university
  • Defining SAP’s culture fit and what makes a great salesperson
  • Reduction of recruiter admin and time

So can an algorithm replace a recruiter? Yes. We have shown that. Technology blessed us with a high potential graduate pipeline that has demonstrated a quality-of-hire prowess through an ability to hit quota, faster than previous graduates, and at higher levels.

Matthew Jeffery, pictured at center, cited as one of the world's leading recruitment strategists and leaders, is VP, head of global sourcing and employment branding for SAP. Previously, he was head of EMEA talent acquisition and global employment brand for software giant Autodesk. Previous to Autodesk, he was the global director of recruitment brand for Electronic Arts.

Andrea Woolley is a marketing director at SAP, the market leader in enterprise application software. She has been part of the sourcing & employment brand team for the last six years, focused on large-scale global projects to increase SAP’s employment brand global reach and impact the quality of talent hired at SAP.


167 Comments on “Can an Algorithm Replace a Recruiter?

  1. Great experiment. Impressive results. Wrong conclusion. Saying recruiters can be replaced by algorithms is akin to saying that doctors can be replaced by X-ray machines. I understand the author wants to turn heads with this headline, but since you’re coming from SAP, please exercise a little restrain before burying recruiters with your technology! 😉

    1. Hi Max. Thanks for the compliment on the results. Also thank you for taking time out to read and also comment. Appreciated. The key aspect to consider here was the sheer number of applications we opened ourselves up to. It was huge. But that was what was needed to move to a new strategy and embrace applications from Universities right across the world and not focus on the select few. We did not have the internal resources, (i.e. numbers of recruiters), to be able to filter/pre select from those huge number of applicants. Plus……it would have been a huge challenge to even try as we would have been faced with having to try to filter based on the traditional screening techniques i.e. select based on University, Course, grades and work experience. We needed a new solution. Especially for the recruiters as they would not have appreciated the extra works on top of their busy day jobs.

      So in this case the simplest and most effective method was to embrace an online assessment. This was a huge help. And it was the most effective route.

      So in this article the takeaway is that in these cases technology plays a huge part and opens up the possibilities to what we can seek to achieve. It helps recruiters. It means that we can embrace bigger candidate pools for selection without adding an extra x number of recruiters. But guess what, when we get past the assessments it is the great skills of the recruiter that then come into play. Building relationships with candidates, providing a great experience, taking them through the boot camp and then selecting the best.

      In fact it takes the best of both worlds. Greater candidate pool selection and utilizing all the fantastic recruiters we have. This model works well with University Recruiting. But as you know it is not a model that lends itself to traditional professional recruiting where the need is to attract passive candidates and that requires the skills of a recruiter to direct source/headhunt.

      Hope this reply helps put into more context Max. Hope to see you at the ERE conference.

      1. Thanks Matthew… I think it is brilliant that SAP showed some initiative in this domain, and I strongly agree with both your premise (that the best sales people dont necessarily come from the best schools) and your solution (go broad, and let behavioral assessments narrow the selection). Full disclaimer: I used to work for SAP until 7 months ago, and I’ve now started a company in recruitment automation which tackles the same sort of issues… So you can see why this article got my attention!

        1. Thanks again Max. Congrats on the new company. Wishing you well. Entrepreneurs like you make the world go round. I see you have sent a LinkedIn request. Lets definitely keep in touch. Great you used to work for us. I hope you have fond memories.

          We will see what the judgement of the industry is but trying something different away from the traditional recruitment models is exciting and the Sales Academy is attracting sales people from Universities that would never of been on the radar and seeing them achieve & smash targets is a great win. Also embracing Social Media & channeling the response through assessments actually does work.

          Foremost to our mind was candidate experience and how people would feel if they failed. No is the worst word in the world. But people prefer that than waiting for a response. We spent some time reviewing questionnaire results of those that had failed and the positive response was great. candidates genuinely liked getting an answer and moving on.

          Hope you get to ERE Conference at the end of the month. It would be great to share a glass of vino and discuss the state of the recruitment world. Best wishes.

    1. Thanks Jim. Appreciate you commenting and your support. Looking forward to discussing more, answering questions & hopefully sharing a glass of wine with you at the reception. Best wishes.

  2. If the vision is to replace the recruiter who sits behind the desk pumping names and resumes in and out of an ATS or sitting on a college campus handing out memorabilia and answering questions as students pass by, then yes I am in agreement that your approach is better.

    Let’s not confuse recruiters with a glorified data entry clerk who periodically visits a location to get the brand name seen and heard.

    If the vision is to replace a recruiter that makes cold calls and can close the deal with a great prospect, then the answer is an emphatic no.

    1. Hi Gareth. Thanks for your comments. The goal isn’t to replace the recruiter. But to make life easier. To enhance the candidates they present to hiring managers. To lessen their workload AND raise quality. As per the article, we applied this thinking to University Recruiting.

      We also wanted to take the side of the graduates out there. What is the traditional University Recruiting strategy? Because companies have limited resources, (time, budget, recruiters), they select a handful of Universities to build relationships with. But what if the best sales graduate does not attend them? How do they get noticed? Is that fair to someone who has the skills?

      How are University students generally selected for interview? University attended. Course studied. Previous grades. Maybe work experience. That then reverts to the tried and tested preselection of picking from the ‘best universities’. That does not open up the field to other candidates who could be better suited.

      It must be frustrating to be a graduate out there applying for jobs, be highly talented, be driven and yet because they selected to go to a University outside the catchment area of a particular business that they don’t get a break? That’s surely our job to help them?

      We wanted to open up the candidate pool to Social Media, Digital marketing, Graduate Campaign marketing but knew that would open up to thousands of applications. Hiring 20/30/40 plus recruiters to screen would not make sense as you would revert to the screening above. That is not fair to the candidates out there.

      So in this instance, creating online assessments & using algorithms benefited EVERYONE. Recruiters because they did not get the extra burden of work & the subjective filtering of candidates. The candidate population as anyone, from anywhere, with the right skills can get through and hired. And SAP. Because we are benefiting from hiring great sales graduates who also fit our culture.

      When the recruiters come through the assessments who then takes them through the process? Who builds and develop the relationships? Yes the recruiters.

      Now will this work in professional hiring? You know better than me Gareth that recruiters are essential to source and reach out to passive candidates. its an art. Vital. Hopefully Gareth that helps gives more context and you would follow the same thought processes in University Recruiting. So this does not remove the need for recruiters it makes the process for hiring graduates fairer for all. Love your continued views if you are happy to share 🙂

      1. Matthew, it will work for campus recruiting or other high volume focused global recruiting initiatives. I applaud the initiative.

        The big take away from this whole experience is that “you and your team” closed a huge gap with a global solution. The algorithm is a significant piece in the puzzle. It is not the end all and be all of recruitment.

        In no way do I think your article is claiming that AI is the alpha and omega of recruitment.

        However, I do want to add that AI cannot and will not fix the recruitment process. Recruiters get into the trap of waiting for the market to deliver solutions to fix their underlying problems, not realizing that recruiting at its core is a very simple process at best. The market is delivering some very useful tools to make our lives easier but it is also convoluting the recruitment space with unnecessary levels of complexity and waste.

        How much money has been wasted already on new HR tech solutions in the name of catching up with the crowd?

        This all aside, You have demonstrated that recruiters can fix their own problems and that is what I hope people get from reading about this initiative.

        1. Gareth. Thanks again for commenting. We are on the same wavelength. You raise very deep and persuasive points. Many companies waste money chasing the holy grail and trust in tech. I am sure many millionaires have been made on the dreams of tech. Again it comes down to the recruitment leaders being savvy with their recruiters and selecting the right product and creating a bespoke solution that works for their companies.

          You are right that this works for graduate hiring. That industry needs disrupting. It is hardly meritocratic for companies to focus on hiring from the select few. But the model is broken when it comes to professional hiring….in relation to the passive pool.

          Fascinating discussions. Love when our industry debates and challenges itself. Hope we get to meet Gareth. You know your stuff and sound a great person to brainstorm with.

  3. A recruiter does more than screen. If you want to say that an algorithm can screen better than a recruiter, that’s fine. However, to say that an algorithm can replace a recruiter is just downright insulting to recruiters everywhere.

    1. Hi Ken. First off thanks for taking time out to comment. Very much appreciated. I totally agree. There is no insult to recruiters here. We faced a challenge. How do you move away from traditional University Recruiting models and embrace candidates from any University and from any country. We don’t have the resources, volume of recruiters, time to be present at all Universities. In the past we had to pre select the top ones. But does the best Sales graduate go to those select Universities? We thought not. They could come from anywhere…why exclude them? Hence we wanted to open it up.

      By embracing Social media we knew we would get thousands of applications. We would need to add additional recruiters to screen them. That was not feasible. Plus how would those recruiters screen/filter? They would, naturally, revert to selecting based on the University, course taken, exam grades and maybe work experience. That was not then benefiting candidates from new Universities.

      Hence in this case, we needed to look at something new. The online assessment. Which works perfectly in this instance. Our recruiters were happy as it did not add to their workloads. They are given shortlists to move to bootcamp. We did not have to hire lots more recruiters to get through pre screening. No insult to recruiters there.

      Can this work for professional hiring? As you know most of the BEST candidates are passive. That requires the skills of the recruiter to source and attract. That takes time effort…and that art of picking up the phone. That’s an art. Hopefully this clarifies more.

  4. The process is both intriguing and thought-provoking. I will be interested to see how this maps to mid and long term retention of these selected individuals, and what that will mean in terms of bottom-line revenue for SAP.

    1. Hi John. Yes. this is key. We can see that we are hiring great sales people. We have a clearer view that they are more suited to our culture, (thanks to cultural assessment both online and at the bootcamp assessments), and we have a clearer idea that they will react in business situations like an SAP salesperson would. hence in theory if they are a good fit, their retention should be stronger.

      We monitor their success by periodical review of their performance against quota. How many achieve <50, 50, 75, 100, 110%+ of their sales quotas at 6/12/18 months. That is real quality of hire data. And gives us a great benchmark to review hiring & historical performance.

      BUT. Here is the challenge as you state John. SAP have to ensure these salespeople feel that we are a Great Place to Work, that they are well remunerated, they get great training & development, have great career prospects, have work/life balance, enjoy the culture…etc etc etc all the things that it is up to a company to do to retain staff. If we don't, then they will leave and that then undermines the investments made….it is a clear goal for the company to keep them as we know the consequences if we don't. HR, the business invest great time and energy into ensuring we can keep them.

  5. I agree that university recruiting is in need of disruption but I assure you that there is a real change happening. It may not be all that visible to those outside of our niche but we’re seeing it every day.

    More and more of the decision makers at senior levels of talent acquisition grew up in the age of technology and so aren’t married to the idea that recruiting recent graduates and students means doing so through on-campus recruitment. That is part of the strategy but not the entire strategy. In fact, they correctly view on-campus recruitment as a tactic and not as a strategy. It is just another source as is a niche job board, an event at a hotel, a banner ad campaign on Facebook, whatever.

    Some of our clients have abandoned on-campus interviewing all together and that’s freaking out a number of our friends in career services. Far more of our clients have scaled back on the number of schools they visit so they can make those visits more impactful. They’re measuring results and understanding that a school at which they hire 10 people at a total cost of $50,000 generates a lower return on investment than a school at which they hire five people at a total cost of $10,000. As recently as a few years ago, few organizations tracked cost-of-hire to the school level. All travel and other such costs were dumped into a catch-all.

    1. Hi Steve. Thanks for taking time out of your busy day to read and comment. Appreciated.

      I think it also is FAIR. Say I am a great Graduate who has sales potential. I work hard. I get great results. Have a part time job. Constantly gaining experience. BUT….and its a big BUT…. I did not go to one of the, lets pick a number, 10/15/20 Universities that a Company selects for preferential treatment, does that make me any less valid? How do I get seen? I apply through a company careers site. But what if that recruiter selects me out because they are focused on the same Universities for selection. I may be the best hire for that company and they will never know. Hence, the model needs to be disrupted as you say.

      I would also look at the education system. I will refer to the UK here. A quick Google Search will tell you that many more thousands of students are passing exams. Many more are getting top grades. More and more students are going through University. And yes. More of them are getting 1st Class and 2.1 degrees. The degrees of differentiation are blurring. How do you select? back to the traditional methods.

      Technology here is a friend. An enabler. If we hire 50 more recruiters, no offense to them, how would they select from the thousands of graduate applications we benefitted from via Social Media, Digital marketing. There is little to differentiate. Plus, you can’t have everyone attend a boot camp. pre selection has to take place at the application stage.

      Technology here is a win for the candidate, (real time feedback, we select from any university, anywhere in the world and based purely on merit). The recruiter wins, (they have less screening to do and present better quality candidates). And the business…(quality of hire).

      Steven. The stats you present above tell a story. WOW. Thank you for sharing. Appreciated. I hope you are at ERE Conference at the end of this month.

      1. I can see that we are totally on the same page. I absolutely agree that far too many employers place far too much emphasis on the school a candidate attended, especially for positions like sales where soft skills matter so much.

        I feel that recruiters can add a lot more value for positions which require years of experience because those candidates often need to be sourced or at least convinced to engage with the potential employer.

        Yes, I will be in San Diego in 1.5 weeks. Hopefully we’ll bump into each and be able to catch-up. If you’ll be at The Candidate Experience party then I’ll be happy to buy you a free glass of wine. 🙂

        1. Steven. Totally agree.

          Those passive candidates, happy in their jobs, won’t be on job boards, they won’t be on recruitment agency books, they won’t respond to all the digital & social marketing we throw at them…why….they are happy in their job, passive. Thats when the recruiter earns their money. The art of convincing someone to join a new company.

          I had this discussion with a Senior Director the other day. Imagine a great Sales person. We want to hire them. Guess what. They have established pipelines of sales for months to come. They have built a network that delivers. They have repeat business in pipeline. Their company will be paying them well. They will lock them up with stock. How easy is it to get them to come to a new company and start again? That takes more than a LinkedIn Inmail or an e-mail. That requires the art of the recruiter to work out their psychology..what are their frustrations? Maybe it is they feel bored, maybe not enough responsibility, maybe not enough training & growth, maybe no promotion prospects & hit the glass ceiling, maybe they hate their manager…etc etc etc That is an art to convince them to start again….not easy and an algorithm won’t help here……

          Look forward to meeting you in San Diego

  6. As someone with a background in I/O psychology, HR technology (and specifically, recruiting tech), and years as a recruiter, the short answer is “yes,” an algorithm can replace a recruiter. Will it happen anytime soon? Again, the short answer is “nope.”

    In order to make this happen, an organization needs to develop a robust competency model for their jobs (or job) based on sound psychometric methodology (one that holds up in court) which then aids one in developing the resulting selection/assessment tool. Finally, any tool developed to leverage these selections should also have the communication workflow to enable the soft touch today’s applicant pools demand.

    Most companies are loathe to do these things due to the expense and expertise required to both implement AND maintain these tools. (I say “maintain” because jobs and competency models need to be revisited periodically.) As a result, it’s easier to just keep on keeping on with current recruiting models. Sure, there are vendors inching ever closer to enabling this (Workday, Oracle, SAP, etc.) but in the end, the final step always sits with HR (and Legal) to gain a level of comfort (and expertise) to make sure these tools work and are legally appropriate in their own organizations. Companies like this are few and far between.

    So relax, recruiters; your jobs are safe. For now.

    1. “As a result, it’s easier to just keep on keeping on with current recruiting models.”

      Bingo. It’s especially easier to stick with current models when the costs of their inefficiencies and errors aren’t captured as direct bottom line impacts. As long as companies don’t know how much their current recruitment models cost them in terms of losses, the relative economy of choosing other models won’t really be able to be assessed.

      1. I think it’s more a function of a skills gap that exists in HR. I can’t tell you how many senior HR leaders I’ve talked to whose eyes glaze over when you talk about “competency modeling” or “psychometric validation.” Many of them just don’t get it and there are just so many I/O psychologists to go around.

          1. Considering how many organizations I’ve seen who can’t even settle on a proper number of job codes (let alone the CONTENT of those job codes), none of this should be a surprise to anyone who’s spent any time in an HRIS group.

      2. Love your thinking Medieval Recruiter.

        Cost is a huge driver. Whilst it may not be captured directly, it can lead to interesting discussions.

        With our 50,000 + applications what is the cost associated with the recruiters screening time?

        We had a go at calculating and came up with a savings for us of 250,000 GBP, (rolling in some other considerations).

        There are other costs:

        The opportunity cost. Ie we are seeing more of the graduates hit sales quota than previous hires. Thats a measurable impact on the companies bottom line as more sales are made than previous hiring intakes.

        Cultural mis-match. Those graduates that come in and don’t adjust to the culture or don’t like it. They leave taking their months of training with them plus any contacts developed. Thats attrition costs! Then we also have to re hire and bring someone else up to speed. That is again significant cost.

        There is the brand cost. Those graduates who apply and may not have been selected in the past because of traditional screening techniques. They will not be happy bunnies with the company for not coming back to them.

        Hence, cost is a fascinating discussion. Even if cost is hidden…its still a cost.

        What is the cost of not hiring THE best available to a company….hmmmm Maybe an article for you to write Medieval 🙂

    2. Thanks Ben for reading and taking time to comment. Appreciated. Yup. In this day and age getting legal buy in, HR Confidence, IT data privacy compliance takes many an hour. As Andrea will attest. Thats not the fun part of launching a tool and takes considerable effort. But is worth it in the end.

      Ben fascinated in your background in I/O Psychology, HR Technology, (and recruiting tech). Will you be at ERE Conference? Love to catch up and have a chat. Maybe a glass of red wine or pint of bitter…whatever your drink of choice.

      1. Sadly I won’t be at ERE this year. Thanks for the offer though and please, feel free to LinkIn to me and we can talk shop any time! (And if you’re ever in Atlanta, I could always go for a bourbon.)

    3. Ben,
      Sadly you are wrong. An algorithm MAY be able to find some online presence of a person. But it cannot nor ever will be able to call and speak to a person in a caring, respectful voice. Coax more details out of them. Hear when they are fudging an answer. Push when they won’t directly answer a question.

      1. Totally agree Scott….but it has a place in University recruiting where many cv;s/resumes are similar 🙂

      2. Scott, I respectfully disagree with your statement. Research has consistently shown that interviews have the least reliability and predictive validity of all the selection measures, though structuring them in terms of competencies improves predictive validity. In fact, I posit that a computer, if given enough data (and in the future, scraping work and personal profiles, online postings, etc.,) could do a much better job of avoiding the biases inherent in a person doing selection. We’re all, after all, simply human.

        I’m not necessarily saying that it’s better per se though. As I mentioned before, applicants today want, no, DEMAND a higher touch. What an algorithm can do though is focus a recruiter into giving that white glove treatment to those an organization actually wants rather than those who don’t have a chance at being employed there. An algorithm frees the recruiter up to do this (as the original post states.) As a prior VP of TA once said to me, “Why are we spending all this time on rejections when we should be spending our time on the actual people we’re interviewing and hiring?”

  7. “We quickly came to the conclusion that the best graduates for our Sales Academy may not study at the top universities.”

    Congrats, you found out in 2015 what everyone with a shred of common sense and experience has known for decades.

    Anyway, the ‘yes’ answer needs qualification. If people are intent on adhering to old standards and practices – CV review, interviews, background/credit checks, reference checks, etc – then no, an algorithm can’t replace a recruiter because the process itself is flawed. You’ll just get the same chancy results quicker, and you’ll have to rely on active candidates. However, if you abandon the traditional process, figure out what’s actually needed for success in that particular job, and test for that, then yes, an algorithm can most certainly replace a recruiter, at least in the majority if not all of the assessment process. The results will be faster and more consistent. But, you will still need to fill the funnel sometimes. Having a rock solid assessment process doesn’t mean you’ll have people to feed into it.

    The problem you will run into, though, is people will not want to abandon those old practices and techniques. Nor will companies readily accept that their over inflated ‘job descriptions’ do not match a realistic assessment of what it takes to do X, Y, or Z at their particular company. That would mean admitting error, and that doing X, Y, or Z at their company is no more special or difficult doing it anywhere else.

    I’ve long been a proponent of the idea that the hiring process is broken in the extreme, and I applaud you guys for taking steps to build a significant portion of what a hiring process should be at this time, in this day and age. However, I think you have not thought of, or have vastly underestimated, how much the recruiting and hiring industry is maintained and driven by charlatans and hucksters who have made massive amounts of money by specifically getting people to ignore evidence and keep the hiring process broken. Nor will this overcome the problem of branding and the brand gap. No matter how well you assess people, if your company has a horrendous reputation and treats people badly and pays badly, you will not be able to fill that funnel. And, as is their tendency, company principles will blame the process and not themselves.

    1. Hello Medieval Recruiter. Thanks for your comments. Hoping we get to meet at ERE, you sound a good person to discuss recruitment with. We really appreciate you taking time out to read the article.

      I think you are right many people have always known that the best graduates can come from ANY university. So why are most companies still focusing on a select few Universities? How do those graduates not at the selected Universities break through the recruiter filters and get noticed? If you are a large company that gets many thousands of applications per year from students how do you select who is right? Most likely a company will select based on University attended, Course studied, previous exam grades attained. Then, given a recruiter has limited hours per day, they naturally, revert to selecting from those handful of Universities. You know that happens throughout the world. Hence, many have known it but few have sought to change it.

      Medieval Recruiter, remember that we are referring here to University hiring. Not professional hiring, where passive candidates are key to our success. Take the best sales person, (experienced hire). They are happy in their job. They are making great money. They have developed pipelines for months of revenue. They have built sustainable networks. Their company has locked them in with stock. Hence they are not on job boards. They are not with recruitment agencies and however much we pump into digital marketing, they are not going to respond. This is where the art of the recruiter remains and algorithms won’t work. That phone needs to be picked up and that person convinced. Not art!!

      Medieval Recruiter. We share the same view. There are people out there who like to see the broken models remain. They make money from it. This article will not be popular with some as it gives an alternative. Like you, we like to challenge the status quo. We want to share ideas. Others can take the idea and run with it, adapt it but at least we are open with what we are doing.

      Given this model above benefits the candidate, (real time feedback, any candidate can now win through on their merit). It benefits the recruiters, (better pre screened high quality graduates, less filtering work, candidates get instant feedback). And the company, (better quality of hire). i would be surprised if others are not doing the same, (and keeping quiet), or they will experiment.

      I hope we meet Medieval Recruiter. You sound like a great person to share a glass of wine and put the world to rights.

      1. Whiskey or rum when I’m out, wine when I’m home. Living in NY and having to spend close to 20 bucks for a drink sometimes, I just go for the diesel fuel.

        I think your approach is MORE than applicable to regular hiring as it is to university recruiting. The real benefit of it is removing bias from a process that is WAY too subjective already. However, the costs associated with such assessments were noted in another comment, and that is a sticking point, but I feel one that can be overcome.

        In the end I think it’s processes like yours which will get recruiters out of the business of trying to convince hiring manager to see candidates and make their total business to convince candidates to look at certain companies. Right now it’s an eve split, and a hiring manger asks for candidates that fit specification X, you find them, and then spend the next few weeks to months convincing them to at least speak to them. This creates a nice internal stress for them, because their manager will then be able to come to them and say, “You said you needed someone, you have people, they have passed our assessments, why have you not interviewed a single one?”

        1. ooohhh Medieval Recruiter now you are talking. Whiskey or rum. Why not mix them up and have both together. A great cocktail. You NY folk can drink it neat yeh? 😉 lol

          Agree…as always. Subjectivity is dangerous. When opinions rule. Not objectivity. I always laugh when a hiring manager says, they must be good, ‘they went to the same university as me’!!! lol. the subjectiveness being that EVERYONE that went to their University must be awesome…..that always makes me giggle. I went to a good University, (tope 3 uk), and I can say that not everyone who graduated there was great. They would agree….lol

          Medieval Recruiter you touch on another interesting area. The job spec. We all know it should be a great marketing doc, sells the company, sells the role, paints the path to a better future but often is a throwaway scrap that is tossed out on the internet to tick a box, that bears no relation to the role, does not describe the company or the potential advancement and has the greatest Employment Brand negative impact. Another area that needs disruption.

          Oho thats why I like Stacy Zapar as she tried that with Zappos and tried to remove them completely…better that that rubbish? Where do you stand on that medieval?

          Loving your thoughts…do you have a blog to follow….?

          1. I do, but I haven’t written on it in some time. I prefer snarking from comment sections, but it’s linked in my Disqus profile.

            I liked the Zappos move, but I think it would be very limited in its applicability to other companies. Having a known and favorable brand goes a long way toward removing recruiting difficulties. But a truly solid process would scale from Zappos all the way down to Joe’s Bait Shop, and still work. And most companies aren’t too far removed from the bait shop, the mega employers like Google employ a proportionately small part of the labor pool. The vast majority of employers are largely unknown, and largely behind the curve in terms of benefits, pay, and pretty much everything else.

            Don’t combine whiskey and rum, it’s horrible. Try this: Gin and Cuantro, splash of 7 Up or Sprite, and a lime wedge. It’s called an orange peel, and it’s very tasty.

          2. Great comments Medieval.

            Nothing wrong with a bit of snarking 😉

            Orange Peel….sounds cool. I will ask for that at ERE San Diego…

  8. As an Industrial-Organizational Psychology graduate, I’m so impressed by SAP’s adoption of research-based, data-driven best practices in hiring and recruitment. Bravo!

    I’d love to learn your insights on how to overcome pushback from both job candidates and employers in adapting to these more scientific methods vs. the traditional recruitment practice.

    1. Hi Ji-A Min — our internal stakeholders (Recruiting, HRBPs, our Talent Acquisition leadership team, etc) were extremely supportive of taking this more scientific approach. Assessments are a hot topic at SAP and they are either being used or being strongly considered across all levels of hiring — graduate through executive hiring. So we launched our program really at just the right time internally.
      We did survey candidates who passed the assessments (who may or may not have been hired), and the response was overwhelming positive. The feedback was that because the assessments were not overly time consuming, and because the feedback was real-time, they found them valuable and not necessarily a “roadblock.”

        1. Ji-A Min do you know of other companies doing this? We could not see. Are you helping companies with this? It is obviously an area that appeals to you and your skill sets.

          1. Ji-A Min. Thanks for sharing. We look forward to learning more from others and helping to disrupt an area in recruitment that needs reform. 🙂 Great you are leading the charge as well.

    2. Hi Ji-A Min. Thank you for your comments and your support. I am happy to give some more color. Love that you are qualified in Industrial-Organisational Psychology. A valuable skill & qualification. As you know ‘Big Data’ is all the buzz in recruitment. Data informs decisions….or lets rephrase that….help us reach better more qualified decisions. As humans we can still interpret data incorrectly but it helps us raise the bar and gives more information to decide.

      Being honest we had some concerns at the start. Would candidates get angry if they sit an online assessment and get a ‘no’. We ran focus groups and students told us that they get more frustrated when they apply for a job at a company and they never hear back and left wondering ‘what is happening with my application’ or ‘do they want me’ or ‘did they receive my application’. We also heard frustration at the ‘bounce back emails’ they receive after application i.e. the ‘Thank you for your application. If your skills & experience are of relevant for the role that you applied for a recruiter will be in touch’. Both were not liked by the focus groups. Hence real time candidate feedback was asked for. A no may be horrible…but it is an answer and it is immediate and people respect feedback.

      We needed to confirm this focus group test information for real so we polled candidates who did not pass the assessments. They were, maybe surprisingly, happy to get instant feedback and to reach closure and move on. That was great to hear for us and clarified what we learnt from the focus groups. Phew you could say.

      Ji-A it has been interesting to read some of the comments here about the article scaring Recruiters that computers will replace them. Technology in this instance is an enabler. It helps the recruiters do a better job from selecting from thousands of candidates with very similar cv’s/resumes. Recruiters actually were happy at SAP that this technology saved them time, helped give candidates feedback, helped them present better, high quality candidates to the hiring managers for the assessment boot camps.

      I hope this helps….let me know if you would like additional feedback. Happy to answer any questions.

      1. Thanks, Matthew – very helpful! I’ve noticed that external recruiters tend to be the ones who are the most resistant, internal recruiters usually welcome assessments for the time savings with open arms.

        Getting people to understand that relying on their gut instinct and first impressions when hiring is erroneous is also a challenge.

        1. Ji-A Min. You get it. Wish the market was more like you 🙂 Why is technology feared by some. It enables better performance and makes life easier.

          Is the answer to stick to the old past methods, precluding candidates because they choose a certain University.

          And when it comes to selecting Grads I don’t blame recruiters in how they have filtered cv’s / resumes. How else can they produce candidates for interview other than look at University attended, grades, courses etc. Not much else to go by. And with more students getting top grades and getting degrees. the differentiation point is really impossible. Technology enables fairness…for the candidate, the selecting recruiter and of course the company benefits from higher quality candidates win/win/win. Whats to fear 🙂

          Fun discussion.

          1. Thanks, I’m always happy to meet an ally 🙂

            Yes, it doesn’t matter that these methods are objectively more fair, meritocratic, and accurate if we can’t figure out how to persuasively message it to get greater buy-in. Your article is definitely adding to the discourse!

          2. That means a lot Ji-A Min…. Both Andrea and I appreciate that. Hope we get to meet someday. let’s keep our communication open and connect on LinkedIn 🙂

          3. Thanks Ji-A Min. Accepted the invite. lets stay connected. Thanks for the article great read. Andrea, please take a look. Loving the title:
            In Hiring, Algorithms Beat Instinct.

  9. Hi Andrea, Matt,

    This has always been an interesting topic to think about. We started it with my last organization. I’m curious what types of test/retest and blind comparisons you used. Mainly, will the applicant take the assessment and get the same score every time? And second, did you do any blind comparisons where you hired candidates that ‘passed’ and candidates that ‘failed’ the assessment to see if those that passed actually perform better on the job. My biggest concern with assessments is halo-effect. Is the success of a candidate that ‘passed’ really just a result of 1) everyone assuming they are great because they ‘passed’ and 2) the organization wanting the technology to work because of the investment that they are less critical of new hires once they start.


    1. Hi Logan — we absolutely did extensive testing, and many pass/fail calibrations, with our assessment partner to ensure that the algorithm behind the assessment was accurate. We tested more than triple the actual number of questions that we asked (both questions specially designed for SAP, and questions from a validated bank of content from our assessment partner), and made changes to the algorithm based on those responses. Test subjects were salespeople at SAP at all levels (entry level through executive).
      We considered allowing candidates who failed the assessment to continue in the process but ultimately decided, based on the sheer volume of candidates, that only those who passed both assessments would be invited to actually apply.

      1. But what do you mean by calibrate? You are ultimately trying to assign objective measures to subjective material. For instance…”insatiable curiosity”…how is an algorithm determining who passes/fails on this measure? Who gets to decide the universally (and legally) acceptable definition of and measurement for this quality. The algorithm isn’t measuring universally standard measurements like distance, weight, height, etc. So who gets to decide who has the SAP Culturally ‘passable’ level of insatiable curiosity? I hate to assume, but I imagine you just took a look at some of your top performers and reviewed their scores? And how are you guaranteeing that how a candidate answers the question once, is how they will answer weeks or months later.

        My biggest concern is that these assessments are really just using pseudoscience to tell someone they aren’t a culture fit. And then going to hiring managers and saying, these are the best candidates because it’s science.

        1. Certainly “cultural fit” has an element of subjectivity to it. Every company has a distinct culture, and companies have a responsibility to both current and future employees to really try to identify that “secret sauce” that makes a truly successful employee.
          That was the project we embarked on at SAP — let’s look at the thousands of successful salespeople we have at SAP and start drilling down into what makes them successful, be it personality traits, courses they studied at university, the industry they came from, etc. We also investigated why these successful employees love working for SAP — what did they stay year after year. These types of considerations influenced the questions we chose to ask in the assessments, and helped us to calibrate which answers were the ones we were looking for.

  10. No one ,and I mean no one that can gather that many comments and following on an ERE discussion as Mr MJ in such as short period of time Great to see and bringing ERE and TA discussions to life, – my goodness we have waited long enough and many of us have tried to get him out of ‘writing retirement’ Upwards and onwards and only another 375 or so comments to beat earlier made MJ ERE discussion record 🙂

    1. LOL. Thanks Jacob. We have two more articles in the pipeline. As you know, it is important to write from a position of credibility. No point talking the future if you can’t deliver in the present. You know the challenges that being innovatory in a large company are… SAP are starting to demonstrate that more and more…hence the pen will flow more freely again…..and Jacob love that you always have the passion to comment, debate, share, support, disagree….passion that many in the industry could benefit from sharing 😉 We only make a difference if we want to make a difference 🙂

      1. Making a difference is to me akin to being alive, and as we as people from birth through youth and adulthood evolve so should and does the world around us. To participate, to contribute, to discuss and to share for the benefit of the wider good is all that matters, for ourselves and for our children and those that will take over one day. It is a responsibility we often forget and think we can and should leave to others, never I say and it is the responsibility of everybody, just like taking part in deciding who to vote for in an election and for representing you.

        1. Yes. Totally agree Jacob. People always need to shape the industry they work in. Join in the discussion.

          What i loathe are those in the ‘Commentariat’ who seek to criticize any new idea. I remember when Stacy Zapar was gaining PR for removing job descriptions with Zappos. Oh some of the negative comments there. WOW. I was shocked. I mean come on. She was experimenting. Trying something new. We should applaud her as an innovator. See if it works. See the challenges. She was trailblazing. But oh no. Some of that ‘Commentariat’ felt the need to flame her.

          Very sad.

          And I look at that ‘Commentariat’. You can Google them. They live by gaining a presence on Social media, criticizing others, gaining followers….but oh…guess what….they rarely offer their own ideas, their own solutions….because they know…they will be flamed by others in the ‘Commentariat’. It is easier to sit on the sidelines and commentate and criticize than it is to write new ideas and innovate…………

          LOL. Soapbox rant over.

          For those interested I recently spoke about this with Rayanne Thorn on her excellent blogtalk radio:

          1. Plus jacob. I look at Recruiter Leaders in the Industry. How many share ideas? How many write articles? How many speak at Conferences? You can judge that 😉

      2. Writing from a perspective of knowledge of having something valid and relevant to say and from a perspective of credibility is why I believe people do listen when you talk, when you write a piece and when you report on other matters than going to a Prince concert (he-he) 🙂 If anyone are to pay attention, to learn and be inspired it h a s to come from rock solid foundation, otherwise it is like hot air, it rises and disappears!

        1. Thanks Jacob. That means a lot. In this industry you never expect universal appeal. I just love people to debate. Challenge. I have had little to say in recent times but feeling the need to share, debate and speak again.

          I won’t stop enjoying my Prince concerts. lol I have not been to one for a while. I feel the need to go to the US when he announces his US tour soon. BUT….what I like about Prince, he innovates, not afraid to experiment, changes his set list each gig, has mastered his craft. In his field he challenges exactly that music industry norm….especially breaking away from the confines of his restrictive Warner Bros music contract. Imagine being a music artist writing music you never own the rights to. The record company does. You may hate him, his music but he is a pioneer. And a great musician I learn from 😉 But that is a separate subject 😉

  11. What SAP and team behind done is actually pretty impressive and having been involved with SAP I can say that nothing is e v e r done with half measures and is as solid as it can ever be. Just look at the numbers that are behind this, look at the ratio, the conversion, the feed-back and the focus this is being given, really really impressive stuff. What has to be taken into consideration here is that SAP as a company not really that well known outside the inner IT circles, and even within these the company does not have an awareness amongst graduates like Microsoft, Apple Google or even Oracle. The structures of the company are complex, they cater for the B2B market and and not in everyday view why trust me to get this kind of response from graduates is seriously something. As for not going for the elitist top 10% of academic achievements when choosing sales ability raw talent this ought to be plainly clear. An analysis of top IT sales people those that exceed quotas with in excess of 130% nearly all come from backgrounds that are n o t top 10, or 20 or 30% top university leagues, which simply goes to show that it may be perceived that best results come from one a set of criteria, where in fact it is often the direct opposite.

    1. Jacob. Thanks for joining in and taking time to comment. You have a passion and a history in this area and so your comment carries significant weight. Appreciate your views.

      And you are right. We do have challenges with our brand reach that we will discuss in a forthcoming article on ERE. When you go on campus, we are competing with the likes of an Apple, Google, Microsoft, which have very attractive brands that candidates are familiar with, especially being very consumer focused. Being a B-2-B presents challenges with getting instant recognition amongst Grads but when we start to detail the vision of the company, the move from enterprise software to the cloud, the dynamism of our CEO Bill McDermott, the many pockets of start ups in SAP, (start ups backed by the financial security of a big company), we are finding more buy into the vision. The Cloud is so liberating & exciting. But it is not easy….and Jacob as you know reality has to demonstrate the marketing…if not then that is a spiraling issue for Employment Brand. SAP is so diverse. The culture and office ‘feel’ is different from Germany to say India, Palo Alto (USA), India, China, Singapore, ireland, Africa. Coupled with the differing companies under our umbrella, Concur, Successfactors, Ariba, then there is a different feel for all.

      I do wonder why Jacob more has not been made of Graduates being missed because they don’t go to a companies target University? The best hire is surely the best hire, wherever they study or come from.

      Most companies state millennials are the future of their company but do those companies get the ‘best’ millennial for them or just the ones from the Universities they decide to target…..not meritocratic at all.

      Its a fascinating area. Love discussing it. Its an area that needs disrupting……

  12. The industry is changing and disruptive technologies in HR are coming out daily. It is inevitable that technology will continue to provide solutions that are more advanced than the human touch. I see algorithms play a role in our day to day business providing companies a more effective way to begin engagement with search firms. The foundation of the algorithms are critical – they must grounded in real data and performance. However, Talent Acquisition professionals have a strong gut built on their personal experience. This is a soft skill that no technology would ever be able to replace, nor should it. No more than the ability to match a resume to a job description.

    Technology will continue to enhance the industry and our lives, but nothing alone will be the silver bullet.

    1. Sean. Thanks for reading. And taking time to comment. Appreciated. You are right. Technology is not to be feared. It is an enabler. There to help us be better at what we do. Help the candidates gain a better experience. Help recruiters present the best candidates.

      One of the things we are using at the moment, (and this is not a product pitch but just something I love using), is RMK, by Successfactors. It allows for us to develop a Talent Community. Candidates fill in a 30 sec business card, (register against job alerts), and we get them into the SAP Talent Community. (Candidates get relevant jobs emailed to them so keeps them up to speed on opportunities). BUT What I love is using RMK to help us review source of hire and channel effectiveness. Our goal is to channel all campaigns and sources into RMK, (which also leads into full job apply where relevant). So we have a COMPLETE source of hire overview.

      BUT the bit I love we get to see, when creating a campaign code, what the best source of hire channels are. Eg a job board. How many candidates apply? How many are interviewed? How many are hired? That makes investments so much easier and moving us away from the ….’Oh that job board feels right as it produces lots of response’ conversations we used to have. We can now see that ‘lots of response’ and it may be rubbish and we don’t hire. Maybe a job board/social media channel has less quantity of response BUT we interview and hire more.

      Technology again helps here. We can target money, time, resources to the best channels.

      The wins…we target more effectively. We don’t generate response that we don’t use i.e. candidates sitting on a database getting frustrated they applied but they were not right for the role & hence damaging the employment brand. This is ‘Big Data’ at its best.

      Disruptive technology that delivers ROI 😉

      Sean will you be at ERE Conference in San Diego…be great to say hi and share a drink at the reception?

  13. Posting an article like this in a recruiting forum is akin to posting in a CPA forum 15 years ago that a computer can calculate your taxes online. Not everyone is going to agree with the approach. Now, approximately 30 million people use TurboTax or a similar product. Nice work SAP team, especially how you were able to break through some conventional thinking and keeping the candidate experience in the forefront of your objectives.

    1. Thanks Matt. Appreciate the comments. We did not expect people to stand up and applaud us. Thats not the point! To many this it is controversial. But if we don’t share, challenge the status quo, then the industry is all the poorer. I love people who challenge existing ideas, Chris Hoyt, Stacy Zapar, Kat Drum, Glenn Cathey, are people I sit up and listen to what they are saying. People I admire very much 🙂

      Thanks for the supportive comment Sir. Hope you will be at ERE San Diego…it promises to be a good Conference…and a lot warmer for me than the UK!!!! 😉

      1. Your approach takes the BS out of recruiting, and it’s a profession founded on BS. Therefore, you will get applause from some, while others will be very put off by the fact that you’re trampling on their living.

        1. Thanks Medieval Recruiter. I learn from you. You like to shake it up. We are not here to be popular…. 😉 You like to challenge and disrupt….I like that. Debate is good 🙂

          1. Who is this masked person……? Why do they hide? It’s like Batman. He lives & circulates with us but we will never know his true identity…… lol

    2. Following what generally going on in corporate recruitment it takes 30 seconds to conclude that of all evolution and invention 80% come from vendors and only 20% from within companies themselves. As and when therefore someone from a large corporate do speak up, provide hugely impressive stats and show what can and has been done, it may generally be seen as small steps, however in the wider context of TA evolution it is actually a pretty big deal. Show me how many like sized companies like SAP that stand up and show how things can be done and with what results, and you will see that we are talking less than 10%

      1. Jacob. Surely the future of University Recruiting lies online. Embrace Social & Digital marketing, take the candidate pipeline through online assessments, then take the most relevant to interview/bootcamp.

        Isn’t that fairer to students….we focus on merit not on what University they study at.

        We free up recruiter time from trying to filter from graduate cv’s that look the same/little to differentiate from.

        And the company hires the best people?

        Surely that is the future? I know you think so……but others?

  14. This is an inspiring article and a great use of technology to prevent bias and put everyone on a level playing field.
    There are many facets to the role of recruiter and I cannot see that role ever totally being replaced by a computer – the human touch points are too many for that to be able to happen. And you know this too since you didn’t rely 100% on tech but held Assessment Days.
    So it would be interesting to know what surprises came out of this program – eg what %age of hires came from universities you previously weren’t interested in, what %age of hires actually don’t have a degree at all or even (and this depends on how you define your graduate requirements) the balance between those with 0-1 yr post-uni experience / 1-2 yr post-uni experience etc etc.
    Next on my wishlist is for a program to be developed that can go through the tons of showreels my team has to view each day 😉

    1. Thanks Lucy. Great to hear from you. Been too long. Thanks for reading & sharing your views. You are one of the best recruiters I have worked with, exceptionally talented, & I will always work with you again in the future in a heartbeat 🙂

      yeh. totally agree with what you say. The recruiter is key for professional hiring. Graduate hiring is where algorithm’s come into their own. I remember when we screened grads at EA. Thousands of cv’s. Only way to select was based on University, course, exam grades and whether they had a little work experience. A nightmare. Hence, maybe this is the future of Graduate hiring BUT professional hiring, that depends on the skill of people like you Lucy. the art of picking up a phone and convincing a great candidate to leave their current company and join a new one. Thats an art. You are awesome at it. And managing people to do it.

      As to the stats, will come back tomorrow when back in the office. Great questions.

      As to showreels….wowsers. Good luck. That is time consuming beyond belief….fun…but time consuming. Again an art trying to decipher if the candidate did the work, was part of a team, what work they did…lol….thats tough recruiting.

  15. WOW! What a comment stream, well done MJ and Andrea! More importantly the comment stream is well considered and has some fabulous insights. As the team who worked with SAP on this project, we have to take our hats off to SAP and the TA team, they were super brave and allowed us to really “stretch our legs” on the execution of the psychology combined with technology. What is fleetingly mentioned in this article is the co-creation of a radically different Assessment Centre (we actually banned that word!), it was called the SAP Bootcamp Challenge, where the candidates collaborated not competed to get the roles. So online tools aren’t THE answer if you want to improve Quality of Hire, they are part of the answer. It is the incremental gains achieved through an online to congruent offline process where the Quality of Hire magic happens. The technology is the enabler, it allows the recruiter to not be replaced but positioned further back in the process, removed from the administrator role, into a role where their skills can add the most value. It also means those recruiters are now not screening thousands of unqualified applicants but fewer pre-qualified candidates. The beauty of what SAP achieved was this end-end brand engaging and highly accurate experience, for both SAP and the candidates. Great company, groundbreaking implementation.

    As an aside we are applying a similar methodology in predicting compassion in applicants who are looking to join the NHS and predicting which Academy Footballers will make it to the first team (for a Championship side) and helping ex-servicemen and women, paralympians/anyone who the traditional CV is blocking from being able to articulate what they have the potential to do, not what they’ve done.

    The possibilities in predicting, using data, human potential and therefore future performance are incredibly exciting.

    Well done MJ, Andrea and team. Great work.

    1. Love that you bring more attention to the bootcamps Roger! That is really where the “high touch” recruiter came into play, but was only possible because the assessments whittled down tens of thousands of applicants, so that we could get 100 to attend the bootcamp and really give them the attention they deserve.
      Thanks for your support as always 🙂

      1. We could have gone more into the bootcamps Andrea in this article. But I suppose we can’t cover everything 😉 lol

    2. Back in 2007/2008 SAP actually had a grad sales academy program where I was Nordic region execution responsible. The assessment day/boot camp was where we really saw what these young people on paper and through the very extensive pre assessment/boot camp questionnaires (back then only semi automated) really made of. It was all about collaboration about seeing how they worked together, how they interacted and what role they played with those they in reality competing against for a place on the team, the other assessment centre/bootcamp candidates. If talking sales and believing in that ‘people buy from people’ then seeing someone in action (irrespective of whatever activity they were involved with) was where it became clear who stood out and who going on to what would then become a hugely intensive and demanding (weekly assignments, monthly topic specific assessment ‘exams’ and much much else. All a giant success until that fatal time that started the Global Financial Crisis and a complete global SAP shut down of absolutely everything recruitment. I would like to think that the foundation for what current team come up with laid back then, could appear that either 2007/08 structures dusted off and re-used albeit with an updated slant or that someone re-invented and come up with same solutions. What matters though is a solid solution that works and that very much appear to be the case.

      1. Jacob, unfortunately as is the way of the world, we started from scratch, just serendipitous (we could call it “great minds think alike” :)) that it sounds like your methodologies were akin to the ones we developed with the current SAP team. To quote Mark Twain,
        “There is no such thing as a new idea. It is impossible. We simply take a lot of old ideas and put them into a sort of mental kaleidoscope. We give them a turn and they make new and curious combinations. We keep on turning and making new combinations indefinitely; but they are the same old pieces of colored glass that have been in use through all the ages.”

        1. Just for the record. Ideas and structures were set in Walldorf, I was merely the local executing force. If an idea is right, if a thorough planning and thought process gone into it, then very often the solution and outcome will be the same, so perhaps 30-40% of using ideas/structures and solutions from back in 2007/08 but in latest version obviously with technology playing a much bigger role + the numbers of hires around 60% of that in this campaign. In was an awesome program back then (only snag was the brand piece of which making grads aware of something by the name of SAP, 90% responding that they did not have a clue what where and how this was) and it appear by all accounts that a success this time around as well.

          1. Jacob. Never say ‘merely the local executing force’ 🙂 Never merely. I am sure you delivered it with style and panache. It is just a shame that the team now did not get all that thinking from the past. As above, new leaders, new times, makes past information more challenging plus I suppose starting from scratch and building it from the ground up made sure we were adapting to the technology of the day.

            Jacob you would have loved the mechanics of it. Defining the culture of SAP. Defining successful sales competencies. What Great Looks Like or WGLL as it is affectionately known. Focus groups, questionnaires with the business, sales people, graduates, high school students, HR, legal, TA….all involved So much wealth of thought to really define ‘SAP’. I am sure Roger & his team developed a few extra grey hairs from the experience.

            It will be interesting to see what other companies do. Companies all want the best graduates. Many right now are pouring their resources into specific target Universities. But are they getting the best graduates? And yes….it is measurable…from a sales perspective. We review how many hit <50, 50, 75, 100, 110% of their quota in 6/12/18 months, (after then they are into the business and it is tougher to claim quality of hire). But you can compare results of graduates from the past to graduates of today. Review attrition rates. This is pure quality of hire. Plus it is being able to show in monetary terms the difference in quality.

            This whole process provides the candidates with what they want. A meritocratic process. To be judged for their skills not where they studied. Real time feedback…a yes or no on the spot…not a holding email or waiting in vain for a reply some 3/4 weeks later. The recruiter gets what they want. They get pre qualified candidates that they can present into the bootcamp and don't have to spend hours sifting through cv's/resumes trying to differentiate from people with familiar similar skills. Plus the recruiter has the data to show the hiring manager this is a cool hire. And of course the business benefits. Higher quality, culture fit, candidates. Win/Win/Win. It will be interesting to see who else develops this. I am sure Roger will gets lots of interest.

        2. Roger….quoting Mark Twain…very impressive 🙂 Deep and true. I think I can quote from Prince, Margaret Thatcher, Machiavelli & Ian Botham….lol

          1. Oh and Roger. I have fond memories of that night when we first set this snowball rolling. That Indian restaurant in London. Many the best ideas are formed over spicy curry & copious red wine lol

      2. Thanks Jacob. Great discussion and always love your passion. The shame is that we started from scratch again as Roger said. But it was also right to do so as so much has changed in this digital world. Many hours brainstorming went into this. I remember fondly sitting round a table with Roger, Gareth, Jess & Amy and Andrea, scratching our heads and brainstorming, flip charting, scribbling flow charts on whiteboards….I know Gareth loves that sort of stuff. As you know Jacob, Technology brings with it so many wins but also so many challenges. Especially globally. For example, Modern data privacy laws are very stringent about what could & could not be collected. That took some time to ensure compliance. I learnt a lot from this exercise about data privacy……probably more than I ever need to know. lol Then there is the whole candidate experience piece, the ‘apply’, the ‘look & feel’ and the defining the ‘rejection’. No is tough and we had to get that right. On top of that, setting the pass benchmark. We wanted to keep it tight and ensure a defined number ‘passed’ and hence control quality into a bootcamp. Automating all this was fun. But seeing that 50,000 + candidates were going to apply ensured we had to map it all out.

        It sounds like Jacob you guys has a great process set out when you were at SAP. I wish I had been privy to that thinking. I guess that leaders come and go, data & reports get filed, and hence knowledge gets harder to find from the past. That’s common to modern business. I am sure that much of what I did at EA and the reports I wrote are well hidden away and the current leadership team will never even see that thinking and ideas I presented. Much would still be relevant today as we mapped out some pretty radical ideas. BUT I suppose thats the way of the world.

    3. Thanks Roger. It was a pleasure working with you, Jess, Amy & Gareth on Phase 1. You guys are a delight to work with and really switched on cookies. I am sure as people start to get more into the ‘guts’ of this thinking that it will become the norm for the industry. The sheer volume of candidates from Social, giving depth & breadth to pipeline, the real time feedback, the pre qualified ‘passed’ candidates for the recruiter to move into the process, the move to opening up University Recruiting to any student, any where in the world, and selecting them on their merit. All could become the norm. We will see. I am sure that you will be VERY busy with enquiries and sharing what could be the disruptor in University Recruiting. Looking forward to seeing what the future holds 🙂

  16. How well would this approach work in markets where there aren’t tens of thousands of potential candidates?

    1. Hi Mitch. Great question. Hope you are well Sir. We did face that. There are certain locations that have less pipelines but we were able to use a combination of LinkedIn job ads, LinkedIn targeted campaigns, Facebook & other Social campaigns, database marketing, digital marketing, local graduate job board campaigns, (targeted), University mail list targeting, referral campaigns to bolster what pipelines we had.

      BUT…..If University pipeline is that bad in a certain country to be that small it probably makes you question why that office is expanding in a talent pool that lacks graduate candidates.

      Hope you will be at ERE San Diego Mitch. It would be great to see you.

  17. @matthew_jeffery:disqus @andrea Woolley Nothing like a bit of controversy and a great story to start a debate! The specifics of the project aside, the impact and implications are significant. One of the problems is that when we make progress – new technologies etc – we often judge them in the context of today, not tomorrow. For example, when the internet first hit mainstream in the late nineties, there was a chorus of “Print advertising will be dead in 3 years.” Publishers rallied and said “nonsense”. And of course, it wasn’t dead in 3 years. It was 10! Then it fell off a cliff. But publishing didn’t die, neither did job advertising. It just changed shape.

    Roger sums it up nicely, the role of the recruiter will change. But the impact on external recruiters, or more importantly the sourcing process, will be huge. And, as an aside, the impact on the assessment industry – currently very tired and lacking in innovation – will also be big.

    Technology is advancing rapidly, and the value derived from our increasing level of connectedness is a key driver. We can pull together and analyse data on a scale that was just not possible before, bringing in new data sets and correlating them with others that, in isolation, were meaningless but together are very insightful. If we can achieve the kind of results we did with the team at SAP today, just think what we will be able to do once the technology matures in say 5 or 10 years time.

    Matt & Andrea it was great working with you and a defining moment for us too. Dream team 😉

    1. Gareth. Wowsers. Thinking of what will be possible in 5 to 10 years is both scary and exciting. I am sure you have been mapping it already in Chemistry Towers 😉 This project is well over a year old BUT we needed history to judge it based on results and its been great to enter for awards to benchmark what peers think. Thankfully ERE gave us an award for it 🙂 I know you did a happy dance to that.

      It was a pleasure to spend time with you, Roger, Amy, Jess and the team devising this. Many thanks. I really can’t wait to see what you guys come up with next. Keep on disrupting Gareth…it suits you Sir 😉

    2. You know Mr. Jeffery doesn’t shy away from controversy! 🙂
      Our recruiters were crucial in this process, we all know that. But partnering with you on assessments was the key to make the work they did more valuable and efficient. I don’t know many recruiters who want to go through 2,000 candidates.

      1. Creating discussion is great for the industry. Its how we all progress. This process helped recruiters. It helped them filter candidates based on their competencies/skills. BUT after the assessments, it was the recruiters who took the ‘passed’ candidates, screened, built the relationships and then took them through boot camps. Kudos to the recruiters.

  18. Great to see SAP transforming traditional hiring strategies and leveraging the power of social media to promote their graduate campaign. These types of results can only be achieved when you have innovative people driving change within an organisation. Well done and thanks for sharing your story.

    1. Thanks Carly to you and all at LinkedIn. You guys played a major part in this success. We love working with you and the LinkedIn crew.

      Whats fascinating is before we started we were aware that LinkedIn was making big strides in adding graduates to your network but we were not aware of how BIG those steps were. You guys have developed an enviable graduate network.

      By utilizing LinkedIn job slots, LinkedIn campaigns, LinkedIn discussion groups, our LinkedIn company careers page, (along with other digital marketing activities), we developed a sizable pipeline across the globe. Without LinkedIn we would have struggled. So if anyone doubts LinkedIn’s graduate database you can certainly point to our success partnering with you and accessing your graduate network.

      I think that this model can apply to many companies. It is a new form of University Recruiting. Utilize social media, anchored on LinkedIn to drive graduate traffic to an online assessment tool that then produces a shortlist for interview/bootcamp. Thats fairer to the Graduates, (selection on merit not based on the University they attend), helps Recruiters, (they don’t have to have the impossible task of filtering lots of applications that have little to differentiate between each other). Better for companies as higher quality of hire kicks in.

      Thanks again Carly & LinkedIn. You guys rock!

  19. I totally agree accurate measurement of SOH is critical. I have talked with many leaders in TA where this is still a black hole. There are alot of great products/tools that can help companies do this better. There still is a challenge with data overload and while the intent to measure is there, actually pulling that back into buying decisions sometimes falls through. Our specific area of business involves third party search firms (we connect employers with specialty recruiters) and especially in this sliver of SOH tracking is almost non-existent and it creates a cycle of inefficiency within TA. That is where we are focused – driving innovation in that area.

    And yes we will be at ERE and I’ll take you up on the drink!

    1. Hey Sean. First up, looking forward to a drink. Will be great to discuss recruiting and put the world to rights.

      Key to data is how it is used and critically how it is interpreted. Look at politics. One set of economic statistics will be met with 2 different interpretations One very positive. One really negative. Hence how data is presented is key. It can tell the story that you want it to tell.

      BUT the data I love is factual data. We have a tool called RMK. It houses our talent community. But it is a tool that allows us to track the success of various sourcing channels. By creating a campaign code and driving response to RMK, (e.g. a job board), we can see how many candidates applied, how many were interviewed, how many were hired. This data helps us inform decisions on which job boards to use. That data I love.

      Sean it sounds like you are really innovating. Thanks for your support and positive support. We value it.

        1. I want to add my 2 cents. The key to data is that data are used to answer questions. Most of the time in HR, a senior leader will ask for data and then hunt for questions. I often push back by saying, “What are the questions you’re trying to answer and then let’s go find the relevant information?” That’s a much better use of data than just sheer data mining.

  20. Congratulations on the campaign. I’m absolutely in favour of democratising recruitment. The approach you have adopted looks similar to one I did at my previous organisation. Notwithstanding I’m in favour of extending approach to grads to one of a more holistic “Early Careers” proposition and certainly moving beyond an outdated trad university approach. You know how to get a headline Matthew though. I don’t think algorithms replace recruiters. They compliment them. They haven’t in your model. The activities, or skill sets of recruiters may have changed, but you still need people to define your content and presumably to assess at your “boot camp” (sexed up assessment centre?). It’s just that the role / activity/ purpose of recruiters has evolved due to the plethora of channels, social and otherwise, and yes insight provided through algorithms. The Times They continue to change. Without change we die. As Darwin said, he who adapts to change thrives. So algorithms compliment as opposed to replace.

    1. Hi Paul. Thanks for taking time to read and comment. ALWAYS value your opinion as you know.

      Democracy feels an apt term at the moment given the UK election, (and Hilary announcing her candidacy in the USA).

      Love to hear more of your approach at the previous organization. Guessing Unilever. Love your passion to disrupt the traditional model of going to a handful of Universities to hire talent.

      As to the headline. Yes. How we recruit Sales Graduates felt a little less conversation stimulating.

      BUT……its not as headline grabbing as you think.

      We seriously wanted to encourage many more people to apply. To embrace LinkedIn Graduate job ads, utilize LinkedIn campaigns, embrace candidates from Social, Facebook, Twitter, drive Digital Marketing Campaigns. And of course use University marketing to drive a healthy candidate pipeline.

      By driving thousands more candidates towards SAP we did not have the resources to sift/filter. We would have had to add an extra 20/30/40 recruiters. but how would they have sifted? They would have looked, naturally, at the University, course, exam grades and back to the traditional forms of focusing on candidates from the usual Universities. That is no criticism of recruiters but what else could they do? I did the same when I was a recruiter many years ago 😉 Hence the need to drive traffic into the online assessment tool.

      So rather than adding recruiters we replied on assessments & algorithms. Hence the headline.

      Our recruiters wee very grateful. They did not have to sift and of course they had better qualified candidates to then take forward and work their magic on.

      You are right the recruiters were at the assessments centre but so were business leaders,many business leaders. Complimenting each other. A great mix.

      Great discussion. Hope the headline makes more sense now 🙂

      1. Think I posted this twice. Sorry if it’s a repeat. I get the headline Matthew. Always did. Just wanted to provoke a response, rather as you like to stimulate debate. Great on senior leaders. They’re recruiters too btw 🙂

        1. Hope you get writing Paul. Always love reading what you are doing & saying. You are the benchmark 🙂

  21. The headline makes sense. I understand it. I just wanted to provoke a response, rather like you wishing to stimulate debate. The business leaders are also recruiters btw 🙂

    1. That’s true….everyone is a recruiter.

      You know how to provoke response. That’s why I have always looked up to you 🙂

      Love to hear more about the assessments & tools you used. Maybe over a nice drink in this gorgeous sunshine.

    1. Would love that. Just us. A blank canvas and a bottle of red. 🙂 (Plus I am waiting on the mighty Bobness to come to the UK. Still on my bucket list). I had to mention Bob as you did not quote him…quite shocked. lol

  22. Can an Algorithm Replace a Recruiter? No, but several thousand may be able to, maybe in the next decade or two.

    Or maybe not. The real-estate industry has not moved away from human agents; technology has improved the auspices of the meeting, but for high-stakes decisions, especially moving from one tribe to another, we may be hard-wired to seek a human guide from one world to the next

    It’s not surprising at all that the early progress is being made with a top tier employer of choice (SAP) in a strongly asymmetric market niche (i.e. recent grads are virtually ALL active candidates) in a relatively homogenous skillset role (sales) where the costs of negative selection are immaterial (SAP can afford to ignore any number of candidates). Certainly the methods here may not be so effective in a tight passive market for mid-career niche specialties for an unknown employer. IBM’s Watson and similar artificial intelligence technology may find doctoring and lawyering an easier nut to crack than recruiting, but crack it once, and it will be cracked but good.

    Nice comment stream- this is a subject that any thinking person in the business cares about.

    1. Hi Martin. Thanks for reading and even more thanks for taking time out to comment. Really appreciated. Its great that people are engaging and discussing. Thats why I love this industry. So many opinions and ideas.

    1. LOL. Yes…re-reading you did quote…lol. I am just mightily relieved that you did not say this article was ‘Blowin’ in the wind’!!! lol See you soon.

  23. Hi MJ, great to see you asking thought provoking questions for us to think about, debate and collaborate.

    In my opinion, algorithims and technology when used strategically, creatively, and responsibly empowers us to make a greater impact. It is something we should embrace and get excited about. The role of a recruiter is evolving. The opportunity to use data, allows us to be more strategic, and act as true consultants. And the use of tools like on-line assessments increases our efficiency, allows us to provide a more meaningful candidate experience, and may help us to make a far greater impact. I.e. reducing unconscious biases, resulting in a more diverse slate, and diverse workforce. It would be interesting to analyze, whether or not we are producing a more diverse slate after using on-line assessments.

    Re:realtime feedback. It would be great to take the idea of providing real-time feedback a step further by making it fun through gamification-reaching levels. I’d be interested to see what types of individual feedback are your candidates receiving? Did they learn and grow from the experience? Do they feel like SAP cares? Did they gain knowledge of what they need to do to build the skills they need for our 21st century jobs? Do they feel they know how to get those skills and are they encouraged to come back?

    In your article, you ask “why limit the talent pool?” I challenge us to all think about how we can expand it. Together, we need to address the shortage of tech talent, and provide our students the feedback, information, and resources to gain the skills they need. Great university recruiters are passionate about growing and nurturing talent, and if we build an experience where candidates:
    1) feel like they grew from their experience, regardless of outcome
    2) feel encouraged to develop x, y, and z skills
    3) came closer to finding their dream job
    4) would refer a friend
    5) increased their network
    Then, that is when we know we have created a great candidate experience.

    My advice embrace technology, integrate it in all that we do, so we can make a greater impact.

    All my best, @KMillerMorris

    1. Hey Karen. Really appreciate you taking time to comment especially as you are a thought leader in the University Recruiting space, with vast experience, (people should take a look at your LinkedIn profile). I have always looked up to your achievements and rich ideas. I know you love pushing the status quo. Without any doubt in my book you have had one of the biggest impacts in the field of University Recruiting in the recruitment industry. Kudos to you.

      We are totally at one on your views on data. Removing unconscious bias, widening, diversifying the talent pool all critical.

      Loving your ideas on real time feedback and gasification. Love to hear more of your thinking here 🙂 Yes. Key for us was ensuring that candidates got that real time feedback. Having the chance to understand why they did not pass the assessment. Hence when you dedicate time to this, relationships are built and one of the reasons that when we surveyed candidates, (who did not pass), that feedback was still positive. A no is not always a negative…..

      Given your passion on widening the talent pool in tech, I can’t wait to continue to follow what you do and your next wins.

  24. Really innovative idea and certainly with merit Matt.
    I can see it having advantages e.g. in reaching out to underrepresented and diverse groups, removing matters such as unconscious bias and providing a great candidate experience. I wonder however, how well it could be adopted by smaller as well as large firms and how it will remove the human experience of the recruiter replete with their unique understanding of managing the client and candidate expectation. Indeed, was Linkedin and other such media not predicted to remove the the role of the recruiter? Or has it made the role of the recruiter ever more important in a world where there is data overload and a need for recruitment to become a little less scientific and a bit more strategic and personable ? A case in hand may be the requirement to attract and retain autistic / disabled technology candidates for SAP (to achieve their prescribed target in this particular diversity strand ). Is the algorithm intelligent enough to appreciate the specific nuances of this type of placement?
    Or do we accept that one method will never entirely replace the other but collectively will provide the customer with the best result.

    1. Hi there Naheed. Thanks for taking time out to read the article and we are even more grateful you commented. Thank you.

      Thanks for the compliments to SAP. Appreciated.

      You pose some great questions. I think all attract methods pose challenges. Of course the ‘traditional’ model, of selecting graduates because of what University they studied at challenges the ethics of ‘meritocracy’. The best students could be studying anywhere. Hence opening up the talent pool wider, deeper, broader, certainly democratizes recruitment.

      But you are right to point to the role of the recruiter. They are critical. That personal experience & relationship they provide is key to making the online assessments work. Once the broad pipeline is filtered down, the recruiter works their magic and then assesses candidates at ‘hangout’ and also the bootcamp.

      This could work well for smaller companies, especially as they may not have the resources to be on campus / a quantity of campuses. Utilizing Social Media & an online assessment can again benefit those smaller companies.

      The intelligence of the algorithm? Its something that is always being reviewed, enhanced. But remember the system it replaces where those brilliant grads may not get into a company because of the University they choose to study at.

      Fascinating discussion.

      1. Hi Matthew, I’m all for democratisation in the recruitment process and love how SAP & IT are, not only enabling organsations, but are being agents of change. I’m looking forward to watching this space!

        1. Thanks Naheed. We appreciate your kind words and support.

          Will you be at ERE San Diego? Be great to say hi?

  25. Hi Keith. Thanks for joining the debate. You are never too late. Always love a debate with you Comrade 😉

    Article headline was in reference to the large number of applications we were opening ourselves up to by driving large Social Media, Digital Marketing campaigns. As you can see over 50,000 graduates started the assessments. Imagine that had been cv’s? We would not have had the recruiters to filter plus how would they filter? No doubt on the University attended, hence going back to the traditional model. Hence in this case, rather than add 20/30/40 recruiters to filter we utilized assessments & algorithms, hence the headline. Of course, headlines are written to attract attention. How we hire Sales Graduates was probably not going to catch the eye as much 🙂

    Agree on your other comments. This is a great method for University Recruiting but professional hiring, which for SAP is about the passive candidate, its back to basics and pick up the phone. Those basics again Keith that I know you love discussing. 🙂

    Look forward to seeing you soon Keith and having a pie, chips, gravy and mushy peas 🙂

  26. Hi Andrea & Matthew. Fascinating article. This is really very innovatory. Do you know of other companies doing this? I am interested to learn more. Thanks

    1. Hi TA Disruptor! Of course we cannot be aware of what everyone is doing in the market. When we set out on this path with our phase 1 assessment partner Chemistry, we understood this was very unique in the way we were tackling this. I hope other companies are doing something similar. We heard from Paul Maxin that he used to do something like this. I think key is that companies need to share more so others can learn. That is what we are trying to do. The industry benefits when we all share.

  27. Matt – Thanks for sharing this post. I applaud any effort that challenges traditional “norms” in recruiting, and brings the data to back it up.

    As you mentioned in the comments, I think this approach can be effective in University Recruiting where most background/CV’s are similar, and new grads are hungry to secure jobs so are willing to go through whatever steps/assessments/etc you put in front of them.

    I would challenge the headline as a statement of fact for all of recruiting, as you’re applying this to a small and specific subset of overall population. As the global economy continues to improve, I wonder how this approach would scale to senior level or “in demand” hires. Would that passive talent be willing to take time to go through several rounds of assessments before speaking with a recruiter? Particularly if they’re assessing fit themselves on an exploratory basis? I like the real time feedback from a candidate experience standpoint, but I don’t know if a Scala Developer is going to bother with that.

    I’d love to see how this scales to experienced hiring if you take this there. Keep pushing the envelope my friend!

    1. Hi ThisisLars — your comment on whether passive talent would be willing to take an assessment is one we are debating right now at SAP. We all know how valuable passives are, and we all want to make sure we find and then nurture that relationship with them. Stay tuned, we are thinking of doing a pilot with a small sub-set of candidates on just this topic shortly!

    2. Hey Lars. Adding to what Andrea says, I find this whole area fascinating. I am going to go off on a tangent 😉

      I still think that Sourcing ‘passives’ does not get the credit it deserves. It is an art. Many companies undervalue the concept and actually see it as a low cost activity. That is sacrilege. I really stand back in awe when I listen to the sourcing team in action.

      Look at sales. We, like any company, want to hire the best sales people. They win us contracts, repeat contracts, new market share. Are the best sales people on a job board or with a recruitment agency? Doubtful. They are passive.

      Now consider their situation. The are great sales people. They have locked in great clients, revenue pipelines, (often for months / years to come). Their company recognizes their value and so they will be well rewarded and no doubt locked in with stock. Why all away from this? There has to be a compelling proposition. These sales people are busy, on the road, being approached daily by competitors. A simple LinkedIn Inmail will not get these ‘talents’ to jump ship. Hence the sourcer has to press a number of psychological buttons to see what their ‘pressure points are’. Is it they need a new challenge; they reached the glass ceiling and won’t be promoted; they are frustrated with their manager; they don’t like latest product developments they have to sell; they don’t get recognized enough for what they do, (sales people often have big egos). Hence the Sourcer has to really work to get the hire. That is an art. It is not easy. I get frustrated when I hear leaders talk of sourcing as ‘simple activity’, ‘the role is a step to being a recruiter’, ‘a low cost location activity’. Sourcing is critical and an art!

      Now back to this subject. Driving passives into an assessment tool, in the early stages is a no/no. We want them. We are seeking to seduce them. No way can we drive into an assessment tool. BUT as Andrea says. We are looking at what role assessments can play in the later parts of a process. But positioning it that it helps the candidate if they are a fit for SAP etc. Fascinating area. I love discussing.

      Lars sounds like a late night discussion on an Ocean Liner with a bottle of wine lol Up for it? 😉

  28. I am not one who normally chimes in on comments sections (actually this is my first time ever), but I felt the need to here in light of some of the commentary i have seen in this thread. New ideas, new ways of thinking are always a good thing in my mind – whether they succeed or fail. There are people who try and there are people who sit back and yell “heresy”. Matthew is trying something new here, and at least for SAP, it seems to be working. I applaud this and the way he and the team at SAP are not sitting back and doing things the way they have always been done. As for those who think the sky is falling any time some new idea is discussed, I implore you to be more open minded.

    1. Thanks Ed. Appreciate you chiming in. Always love what you have to say as an Industry innovator and leader.

      I remember when Stacy Zapar launched the Zappos initiative to remove cv’s/resumes. She was flamed for daring try something new. Seems to be part of the territory for some.

      For SAP this has worked well for use both Phase one and the later phases. And it benefits the candidates. Real time feedback and we are selecting from a wider talent pool.

      Keep up the legendary work 🙂 Web Steam master 🙂

  29. Hi Kathy. Just Googling Roundpegg, it sounds very cool. Sounds like your company is on the cutting edge as well. Hoping to see you speak about it at ERE or LinkedIn conference 🙂

  30. Google Dave. Great to hear from you. I always look up to you and admire your passion my friend. You are on the cutting edge of marketing & candidate experience.

    I am so with you on the ‘Red bricks’. There seems this need to focus on a handful of ‘target’ universities, with companies pouring their investments & sponsorships in these Universities BUT when you put candidates through assessments, we are seeing many candidates from Universities that we have sat up and said ‘wow, wheres that’, ‘we would never have thought of going there’. Obviously naming those ‘hidden gem’ universities in this article is not something I will do as people will start to go to them 😉 But great graduates can truly come from anywhere.

    Lets hook up Dave. Love to discuss the CX journey with you more. Your research sounds priceless and fascinating.

  31. I find it pretty impressive that 0 complaints about candidate experience. I’m curious if the new millennial workforce will become more and more appreciative of assessments like this.
    Truly letting the equal opportunity play out not based on old bias.

    1. Hi Aaron. To be very honest we did expect complaints. It was a concern. When an assessment returns a ‘no’, (the hardest word in the dictionary), the expectation was that we would get some people who may wish to debate that. Obviously the assessment site was written in a very empathetic way & ‘no’ was delivered in as best a way as possible BUT human nature, being human nature, you would expect someone/people to challenge and complain. They did not. Now, we can’t say the system was perfect and I am sure people will complain but its good to get a sense that people like real time feedback, positive & negative. Maybe when a lot of companies do this then more will challenge. Maybe actually getting feedback, unusual in this industry, is such a novelty that it played in our favor?

    2. A lack of BS and a clear, timely, Yes or No answer, why would anyone complain about that? My guess is if complaints did surface, they were about the process being ‘impersonal’ and being evaluated by a computer rather than in person. But then that would just fall into the noise of anyone who fails to get a job, and wants to claim they’ve be improperly evaluated, rather than admit they potentially aren’t the right person for the job. Generally speaking though, you’ll find many job seeker just happy as hell to have a no BS process to engage in. My fear would be what happens when these people leave SAP, and eventually apply elsewhere and see what a cluster fark most hiring processes are.

      In the end, now that I think of it, it might help SAP retain people once they see that.

      1. Loving your comments and insights Medieval Recruiter 🙂 You are one switched on recruiter 🙂

  32. Hi Matthew,

    Thanks for inviting me to comment on your fine piece- I haven’t gone through all the commentary yet, but kudos to you and Andrea for writing a stimulating article.“Can an Algorithm Replace a Recruiter?”

    I’ve long maintained that in a world filled with logical, rational decision-makers, most of what many of us typically do as recruiters, which are low touch, and/or low value-add activities, should be no-sourced (eliminated), through-sourced (automated or algorithmicized[!?]), or out-sourced, leaving recruiters to perform high-touch and/or high value-add activities (like advising, mentoring, streamlining/improving recruiting processes, closing). However, I have also come to the conclusion (based on research in cognitive science and behavioral
    economics) that we are not living in a world of “rational actors,” but are
    actually dominated in our waking lives by unconscious and semiconscious cognitive biases which greatly limit are our ability to be cool and calm rational decision-makers. This results in organizations (which pride themselves in being profit-maximizing rationalists) actually being run by the GAFIS Principles: the Greed, Arrogance, Fear, Ignorance/Incompetence, and Stupidity of the Founders, CXOs, and other organizational power-brokers. What does this mean to us? It means you’ll typically get recruiting, employment, and most
    others policies based on the whims of the people in charge, rather than on any practical, fact-and logic-based reasons. ….

    How does this relate to the article? In many cases, hiring managers will fight tooth and nail NOT to have selections made algorithmically if it means a loss in perceived control/power, and it’s likely recruiting managers/directors/etc. will also fight against it if it results in a decrease in their budget or staff- it’s kind of hard to swagger about saying “my algorithm’s bigger/better” (unless you’re a hard-core techie).

    Furthermore, unless the people who made/oversaw the previous recruiting policies are either gone or too powerful to suffer consequences, anything that says “this is a substantially better way of doing things” makes those who did the earlier, less-effective way look wrong or at least ineffective, if only by comparison. For these reasons, it’s usually best to do little beyond tinkering a little at the edges, because too many powerful people can look bad or get angry if you challenge the basic assumptions of the way they do things…Net result: over time, you’ll probably see fewer college recruiters, but I don’t think much will substantially change in the long run, at least for the vast majority of employers in the world…

    But enough about you- what about ME? I didn’t fall off the edge of the world: for the past several months I’ve been working hard actually RECRUITING (and being on the boards of two non-profits which work to help employ people on the Autism Spectrum), and it’s taken up
    much of my time and energy. I’ve also had time to think about ERE. I realized that while I LOVE the attention, trying to prove (and often failing miserably) how witty and clever I am to you isn’t really a productive use of my time. I am also coming to realize there really isn’t much of a market for telling people what I believe they should know, as opposed to what they want to hear. I’ll be blunt (“How refreshing, Keith!”): There’s not much money in me ranting and raving likely a self-proclaimed 21st Century Jeremiah., so unless someone has something which isn’t more than about a single stage away from getting paid to do it (like speaking, writing, advising, or consulting), I’ll tend to keep a pretty low profile for now. (The exception is if someone with good resources/connections wants me to work with them to develop a fact- and logic-based Generally Accepted Recruiting Practices [GARPS], which I’d LOVE to do for free (if necessary), and I’m happy to point people in the direction of recruiting jobs, resources, etc.)

    So Gentle Readers, for now I shall leave the digital pages of the Recruitisphere to the *many corporate suck-ups and hucksters,and few genuine, intelligent writers like Matt and Andrea (sorry for leaving the other good ones out here).


    Keith Halperin SPHR Emeritus, 415.672.7326

    *Unless things have changed
    radically in the last few months.

    1. Hi Keith. First of all great to hear from you. I value your opinions very highly. I remember with fondness our discussions over Recruitment 3.0/4.0/5.0 back in 2011/12. How time flies.

      First of all I like your reference to the GAFIS principles. I will certainly use that in future presentations. Great insights. And I agree that hiring decisions in the industry are often made on whims & subjective decisions. Getting more data helps steer the decision. There will always be subjectively involved and hiring managers have to take risks but data informs and helps make better decisions.

      Keith you will be a HUGE loss to ERE. I have always valued reading your views and you challenging. There are many people who ‘watch’ ERE i.e. read the articles and all the comments but they don’t comment themselves. I would advise you not to lose your positive challenging spirit and not to retire away. The industry needs people who challenge and don’t accept the status quo. How does the industry change it it accepts the ‘norm’.

      Keith you are a great man. Any chance of you going to ERE San Diego. Would be awesome to see you.

    2. Nice to see you posting again, however briefly. You are correct, the world is ruled by people who think they are acting rationally, but at most they are acting in a goal oriented manager – soft rationalism – and are usually massively disconnected from reality when it comes to judging whether or not the actions they take will actually deliver the results they supposedly want.

      It’s a popular idea in economics that people have this homunculus living in them that makes ‘rational’ decisions. They don’t, they’re just people. Once you realize those in decision making positions are often no smarter than you, can often be much dumber and disconnected from normal channels of information the rest of us have easy access to, that they are just as tribal as any other human being, and just as dominated by biases and fear, you start to be able to make sense of their decisions. They make ‘sense’ of a sort, however they are also completely counterproductive toward good hiring and often even just good business practices.

      And when you couple that with the recruiting profession being dominates by sales! types, you get a recipe for hell. Sales people are notoriously resilient to negative feedback which is great in their profession, but once they start managing that same resilience leads to a near psychotic indifference to opportunity costs, which is damaging and often devastating to companies. It means those ‘clients’ for whom candidates never seem to work out or be a ‘fit’ for get more attention, not less. It means no matter what they do, since they’re paying the bills, it’s ‘okay,’ despite the questionable morality, ethicality, and legality of those actions. That, in my opinion, is a large part of what perpetuates the ridiculously arcane and poor practices of recruiting ever onward, because the people who should be questioning it most are under the control of used car salesmen who jump on every job from any client, no matter how questionable the opportunity or the client’s behavior in the past, like it’s manna from heaven.

      There’s a ‘manager’ at my current company that has received several requests recently from clients, they were desperate! They needed people! They didn’t know how many, they didn’t know what they were willing to pay them, they didn’t know when they were going to be able to start, but they knew they needed them! And being a good sales! guy he had his recruiters, including me, jump right on it. Each position turned into a cluster $%& as you can imagine, with numerous back fills already, and the initial rate were finding people at, well they cut that in half and demanded people a couple bucks above minimum wage. There were so many red flags there that anyone really should have seen that coming, but not a sales! guy. Nope, it’s an opportunity! to them, not an invitation to waste a ton of time for what amounted to barely any billing.

      I’ve seen this kind of crap repeat itself ad nauseam in the recruiting industry, at agencies I’ve worked at, at agencies colleagues have worked at, in several corporate settings, etc. It never ends, and of course every ‘prominent’ recruiter always says their agency is different, but it never seems to pan out in reality when you read reviews of their companies or talk to current or former employees. I haven’t lost your taste for ranting yet, Keith, but I do agree that this industry is likely unsalvageable. That’s the case because too many people make too much money perpetuating bad practices as a way to satisfy the neediness of those under the thrall of GAFIS principles. And there are too many sales! people in charge and managing when they should be relegated to actual sales, to which their resilience is very useful and well suited, rather than managing and operation roles where their resilience means casting every sub par employer as some misunderstood fantasy world where, if they could just hire ‘the right kinds of people,’ then they would turn into paradises where the fact that they barely pay anyone, offer next to no off-time, and have benefits from the Marquis De Sade Insurance Group, somehow won’t matter to people, because they’ll be so overwhelmed with ‘engagement’ and ‘job satisfaction’ and ‘INSERT HR/RECRUITING BUZZWORD HERE’ that it won’t matter that they’re treated like hell and can’t afford rent or food.

      1. Powerful arguments medieval. I hope that lots of people get to read your logic & passion for making a difference. And you are right. Many people benefit from making money from a system which is broken in many ways. But they will protect the status quo as that is where their money is made. Why would they seek to fix or embrace change. They would lose money 🙂

  33. Hi Tom. Sorry for the delay in replying. I have been traveling. First off thanks for taking time to read and also just to register we are even more appreciative you took time to comment.

    Thanks for the compliments on the idea & execution for our graduate sales roles. Appreciated.

    I will try and answer each question.

    1) We are looking at a number of different roles/disciplines to test further adoption. For professional hiring, for developers, we have introduced assessment tests. Again, when you hire developers it makes sense. Of course we wanted to see the execution in Sales first but we will keep you updated on other areas.
    2) Yes. We closely monitored the level of drop outs. It was very low because of the shortness of the assessments. The culture assessment is something that took 10 mins. Hence people completed it with few drop outs. I think culture assessments are also intriguing for candidates to complete and see if they are a fit. With a ‘pass’ awarded, psychologically people are excited and hence went through the situational judgement.
    3) Previously we were very traditional. We focused on select target universities. Like many in the market. But this allowed us to go wider, deeper, broader & more diverse.
    4) I will go back and check the numbers for the numbers of different universities covered by this scheme. Key was that it has substantially grown BUT also the quality has been superb. I remember we looked on the map for some countries and were Googling Universities to get an idea. But the best candidates for sales graduates can come from anywhere. And millennials are happy to move for their art and career.
    5) Yes. We are busy collecting and analysis data. Things like the number of graduates coming in and making <50% of quota target, then those making 50-74%, 75%-99%, 100% 110%. This is reviewed over 6/12/18 months. Attrition has been substantially lower as the culture fit assessment & extensive boot camp assessments help define 'fit'.

    Tom, Thanks for taking time to read, comment. Appreciated. If you are ERE San Diego I would love to catch up 🙂

    1. Ji-A Min — very thought provoking, and I couldn’t agree more that engagement is always key and will (hopefully!) always involve a knowledgeable recruiter.

  34. MJ – if this was truly a “Democratization of recruitment” and “Meritocracy of application” and “If they are a great salesperson and we see that potential, #BOOM, let’s hire them!” – why did you restrict it only to graduates?

      1. Thanks MJ – I’ve seen the PR release before. I think it is a great programme but wanted to get past the spin – I am interested to understand why you chose to restrict the sales programme to graduates? Additionally, where was an algorithm actually involved in this programme, as that is not jumping out for me? It seems to me that you put graduates through ability and some form of psych testing at the front end, having gathered core behaviours from your hiring managers previously. I was always taught that making selection decisions on the basis of psych tests was against best practice and that the results of these should at best be utilised to inform further questioning. Admittedly this may have been SHL trying to sell me more tests after the training!

        1. ha Ken. Not a PR piece that was CBS news reporting on Autism.

          Lets get past the spin as we always do 😉 Our Sales program is not restricted to graduates. The Sales Academy is just one particular initiative to attract talent.

          I think Ken on the testing we have to look at the big picture. We opened up the programme to a wider catchment pool. Not just the traditional universities. Which I know you applaud. We had over 50,000 start the assessment process. If we did not put in place an assessment, what would we have done? What if that was 50,000+ resumes/cv’s? That would have been a nightmare and we would have hired extra recruiters to filter. BUT The recruiters would have struggled to filter the graduates, given most grads look similar on paper. They would have naturally, (as I would have done), selected candidates for interview based on their University and the usual institutions would have been selected. That excludes. You know many companies are doing that. (They boldly state their preferred Universities on their web sites).

          SAP is not an easy culture. (As I have told you). lol. Hence the need to ensure that the first part emphasized that. Then the situational judgement questions, (20 mins), helped get into the skill sets.

          Now methods can always be improved, enhanced. We can debate algorithms. But I think that it is a step away from those traditional ‘best universities’ selection methods.

          Is any system perfect? No. But filtering according to University, course attended, is properly the least meritocratic way.

          Ken, we are long overdue a catch up. I can show you the mechanics behind the scenes. Always love debates with you 🙂

          1. I refer to this press release, Hopefully my request to understand does not make me one of the commentariat that you reference elsewhere, but I really do want to understand, how in an democratic and meritocratic recruitment programme you only selected graduates, as this seems to be contradictory. My other question, and perhaps the elephant in the room as I don’t think this is one of mechanics, is where exactly you have utilised an algorithm in this programme, as the title of the article still reads as “Can an algorithm replace a recruiter” – however great the programme undoubtedly is, I’m not sure whether the subsequent article follows up on this. Cheers, Ken

          2. Hi Ken. Ah ok. Never thought of you as one of the ‘commentariat’. Just someone who is inquisitive. Which is cool. What you refer to is one of a number of programs. There are a number of multiple access points into SAP. Both graduate and also non graduate.

            Even within a graduate program. Opening up to grads from all Universities is a positive move. Maybe you disagree. Maybe you prefer the traditional model of working with a select few ‘target’ universities? I don’t know. But widening it out feels a positive move and more meritocratic.

            You can debate whether there should be any focus on graduates at all. That becomes a different viewpoint. I think you find most companies have graduate programs.

            If your question is can people get into SAP without a degree then yes. But we also have to be targeted in our ‘attract’ hence we have University programs. Like most companies.

            In terms of the algorithm question. In this case, when opening it up, with the presumption being that if those 50,000 + people starting the assessments were to have applied, (a safe presumption), then we would have had to hire recruiters. 20/30. To filter. And then filter how? Based on University attended. So adding more recruiters here did not make sense and algorithms replaced the need.

            Cheers Matthew 🙂

          3. Thanks for taking the time to reply Matthew, even if my inquisitiveness is turning into an inquisition! I can accept your position as “more meritocratic” if not “meritocratic” in isolation – this presupposes that the skills of an effective sales person are only contained within a graduate – I’m not sure this is true from my own, admittedly, anecdotal experience, . I was trying to understand whether opening up to a wider pool of universities (who presumably would have a lower academic requirement for entry purely on the principle of supply and demand if nothing else), then why not open it up, through social media as you have already established this as an effective channel, to non graduates? I’m not arguing the case for a focus on a more select group of universities at all.

            With regards to the algorithm clarification, referencing again the title of your article, my point is that I cannot see where you have used an algorithm at all. You have used a selection of tests for culture and aptitude, but no actual algorithm or have I missed something?

    1. Gotta love a sun headline! I think part of the challenge here is the naming. This was the front end of the new SAP graduate hiring program, not a sales hiring program specifically. The sales academy was actually the graduate sales academy. It happened to be the first function to go live. similarly to other organisations that have graduate schemes, SAP was no different. they just chose to stream it more visibly and do sales first. So the focus on graduates is understandable given it was a graduate program.

      What i can say was that the applicant screening tools, hangout and bootcamp stages were all designed to look at the core motivations, values and behaviours required to be a successful hire in this role and were not selecting on university or grade.

      In terms of the algorithm, every online assessment has an algorithm sitting behind it, of varying degree of complexity. In this case, the algorithm resided within our technology at the applicant screening stage. There is no issue with using assessments for screening as they are accurate. The issue you mention re the use of assessments is valid for certain assessments in particular, its not a blanket guide.

      To address the sun headline – i think its like @roger says, it doesn’t replace the recruiter, it just frees them from work they shouldn’t be doing – applicant screening, especially at high volumes. The technology out there is more than adequate to do this job, freeing the recruiter to focus on sourcing, candidate selection etc.

  35. Kudos on the initiative guys, especially on coming up with ways to challenge the existing hiring biases. I found the article thorough and can see how a system like that can bring about the benefits you guys claimed in the article.

    I’m curious to hear your guys’ thoughts on a similar model for hiring more experienced candidates? Have you guys experimented with that, too? If so, what were the results like? Did the quality of the hires change in any way?

    Being a co-founder of a talent management startup, I believe there are certain functions of recruiting that algorithms can augment. For example, inbound sourcing of potential candidates is one area where algorithms are coming in to picture already.

    1. Hi there Shikhar. First off thank you for taking time to read the article and an even bigger thank you for commenting. We really appreciate it.

      Lastly thank you for the compliments. Appreciated. All too many people seek to criticize and knock down new ideas & initiatives and debate pedantics. So thank you.

      Obviously this method lends itself perfectly to Graduate Recruiting where there is little to differentiate between candidates on cv’s / resumes. It moves on from the traditional selection techniques used. i.e. which university do they go to etc

      You raise a great question. How can it be applied to more experienced candidates. Thus far we have not applied it to more experienced candidates apart from running a number of pilots across the world for development hiring, (hence technical assessments help differentiate here).

      I suppose Shikhar when you look at experienced hires you need to differentiate between the ‘active’ pool and the ‘passive’ pool. Asking a ‘passive’ candidate to do an assessment early on does not feel right. We have sourced/headhunted them. Hence ask them too early to do an assessment feels ‘cheeky’.

      However, there may be a case for asking ‘active’ candidates in certain roles to complete them. To help us screen candidates. I again look at the thousands of cv’s/resumes that are sent in through the careers site. Could they be prime for assessment? The candidate is asking us to consider them.

      Shikhar I am interested in your views here. What do you think? When can assessments be used for experienced hires?


  36. Totally awesome reading through this debate. Not understanding why some don’t understand what an algorithm is here. Checking the definition it is: ‘In mathematics and computer science, an algorithm (i/?æl??r?ð?m/ al-g?-ri-dh?m) is a self-contained step-by-step set of operations to be performed. Algorithms exist that perform calculation, data processing, and automated reasoning’. That sounds what is being done here. Or am I missing something? However, whilst I love this approach, key is that it maintains the human element. Recruitment is all about people. Relationships. Get that balance right and its a winning program.
    Fast forward 20 years and I think computers will do much of what recruiters do today. Artificial intelligence is fast gaining on human intelligence. In some cases of recruiters I have met probably exceeds it now!

    1. TA Disrupter – you’ve hit the nail on the head!

      Did you see John Sullivan’s article, posted here today? Recruiters can’t avoid technology (can anyone?), but that human element will always be important. For SAP, in this graduate hiring project, that was in the several post online assessment phases – the “hang out” phone conversation that recruiters had with all candidates who passed, and the in-person boot camps.

  37. Thanks for the follow-up Matthew.

    I agree that presenting assessments to passive candidates will meet with friction. In fact, I’ve not been a believer of assessment in the past, largely due to the ones that I had come across. But after the advantages you guys mentioned, I understand their value a lot more, especially if they are designed right (which you guys seemed to be doing).

    Coming back to the topic of experienced candidates. I feel we might be able to get to a place in foreseeable future where algorithms will be able to do the leg work of pre-evaluating potential candidates based on various criteria.

    Ideally, and I’m making a huge leap of imagination here, hiring for business should be more like hiring in sports: based on concrete data. But a big hindrance to that in the present setup is that our employment data is not public information. There are may reasons why that is so. Some reasons are due to proprietary and private rights and some purely cultural. My guess is cultural reasons might continue to evolve as people are increasingly sharing about their work in public domain places, forums, networks, conferences, etc. and technologies will then able to serve-up a unified view of this information.

  38. It’s interesting to ask the question if an algorithm will replace a recruiter. I don’t think it could and the reason why is because humans have abilities that we have never been able to recreate on the computer. An algorithm can help automate mundane tasks, but the real critical thinking comes from the human brain.

  39. Love the article and moving towards using great “clean” datasets like validated assessments for this work. I would take issue with one of the steps that – to me – sets up a completely biased experiment – and something HR and others need to move away from…

    “Hours of focus groups with business leaders, graduates, recruiters, HR. We needed to define WGLL (“What Great Looks Like” at SAP). Those key
    components were defined as …”

    To do a real data science experiment and truly get rid of bias – we need to let the data define what the culture is vs. having well meaning people define this. In the experiment defined in this article I can predict that the “culture” is going to reflect the preferences of the people who defined it.

    What would have been more rigorous would have been to use the assessment to sample current employees and let the data tell you what they culture is.

    Otherwise – love the article – the comments and the move towards using a more rigorous approach. Excellent.

Leave a Comment

Your email address will not be published. Required fields are marked *