Why You Should Care About Big Data

I am not sure about you, but I am reading more and more about the power of “big data.” Forrester, McKinsey, and IBM have all issued white papers or reports in the last month or two discussing the impact that the analysis of big data will have on business.

Big data refers to the totality of information available. This includes data in emails, instant messages, in video, and in audio files — all data that might help create a more complete understanding about an issue or person or provide an answer to some question. All the spreadsheets and databases we are currently using are made up of structured data, data that can be organized into columns or rows and then added or otherwise analyzed.

And, while this type of data is incredibly useful, access to unstructured data would add dimensions and depths that only the CIA can currently realize.

Historically, the volume and unstructured nature of so-called “big data” prevented much in way of analysis. An individual had to listen to the audio, watch the videos, read all the material, and integrate and analyze to form a conclusion. This is obviously very time-consuming, and requires training and the ability to assimilate many kinds of media. But we now have computers that are close to being able to look at large amounts of this kind of data and draw inferences, make suggestions, and provide summaries. The CIA and other government agencies undoubtedly already are using these tools to analyze email, voice mail, and phone calls in search of terrorists.

But these capabilities are about to be available to everyone. In the past few months Oracle announced it had acquired Endeca, a company that does dataanalysis and is building a Big Data Appliance — a computer specially designed to handle the volume of information found in unstructured data. IBM developed Watson, the computer that played against humans and won at Jeopardy, as a big data analysis machine.

HP announced a few days ago that it is integrating Autonomy, which it purchased earlier this year, into a new hardware platform for data analysis, SAS has developed a number of big-data applications, and EMC recently acquired Greenplum, another data analysis firm. Each of these firms is looking to mine the potential of the massive amounts of data that exist and that are being created.

Imagine the power these tools will potentially give to marketing and advertising folks. They may be able to specifically target individuals with messages that, based on the analysis of what they are writing or talking about, will entice them to buy a product or choose a suppler. On the more positive side, this level of understanding will make it possible for computers to take over call centers, much of customer support, and other jobs where knowing a lot about the caller as well as the products will be most useful.

What This Means for Recruiters

For recruiters, this may change everything about what we do and how we do it.

The capability this analytic software has is scary and threatening. To thrive in the coming world of big-data analysis, we will all have to learn to adapt and to develop very different skills from the ones we now have.

Here is just a cursory sampling of what may be in store.

Job Descriptions

By analyzing what a department produces, what data is gathered and used, who people talk to and interact with in meetings or in social networks, these programs will be able to identify key characteristics of successful people and from that develop a list of competencies, skills, and attitudes that are most likely to be successful. They can match this against current requirements and suggest changes or skills that might improve or complement whatever exists. But a job description or analysis will be much more complete and accurate than they are today.

This capability will be here in a year or two.


By tapping into a larger data-set than we can access or analyze today, we can find more people and learn more about them than ever. We can perhaps get referrals from whoever a person calls, what they talk about, and who they refer to in the conversations.

These tools will also completely eliminate the need for Boolean search or experts in using the various forms of search that are popular today. These will all be automated to a great extent. Imagine a computer akin to the Hal 9000 in 2001:Space Odyssey that can understand human languages (similar to Siri on the Apple iPhone 4S) and conduct a search independently of a recruiter. They will be able to dig much deeper and make inferences based on data that would be impossible for a human.

Article Continues Below

I look for some of this capability within two to three years at the most.


We have an increasing ability to learn more and more about people by gleaning bits of information about someone from scraping or extracting data from websites/public information/social networks and from information about the products or services someone buys or uses, and from their interests extracted from comments, Tweets, locations, and so forth.

This, combined with better analysis of the job as described above, will let us choose people with a higher probability of success than we can do today with all of our tools.

This is just around the corner and could probably be put into practice at some level today if the machines were available for commercial use (some are) and the costs were reasonable.

Metrics and Performance Analysis

With the power that these tools are already capable of, everything we do will be tracked and can be correlated to our performance.

We will be able to measure and track which calls resulted in the most candidates, what methods yielded the greatest returns, and how well our candidates performed once hired. Some of this capability is available today with tools incorporated into HRIS systems like SAP and Oracle.

Legal and Privacy Concerns are huge and will require another article to discuss.

What are your thoughts about all of this? Is this all just a pipe-dream or do you agree that it will happen?

Kevin Wheeler is a globally known speaker, author, futurist, and consultant in talent management, human capital acquisition and learning & development. He has founded a number of organizations including the Future of Talent Institute, Global Learning Resources, Inc. and the Australasian Talent Conference, Ltd. He hosts Future of Talent Retreats in the U.S., Europe, and Australia. He writes frequently on LinkedIn, is a columnist for ERE.net, keynotes, and speaks at conferences and events globally, and advises firms on talent strategy. He has authored two books and hundreds of articles and white papers. He has a new book on recruiting that will be out in late summer of 2016. Prior to his current work, he had a 20+year corporate career in several San Francisco area tech and financial service firms. He has also been on the faculty of San Francisco State University and the University of San Francisco. He can be reached at kwheeler@futureoftalent.org.


11 Comments on “Why You Should Care About Big Data

  1. Thank you, Kevin. This topic interests me quite a bit. We’re very close to being able to create (possible now, but complicated) what I call a “digital dossier” based entirely on publicly accessible and data-minable information. (It will be interesting to see who owns someone’s biometric data.) I think it will also radically transform sourcing- if you know where someone is and what they’re doing real-time, you don’t need to source them. (How often do you need to “source” your car keys?) It’ll be interesting when companies start going after (in a very low-key way) promising middle- and high-school students.

    Folks, what do you think?


    Keith “To Know Everything About Everyone” Halperin

  2. Big data will have some impact on HR, but my hunch is that it will be mostly hidden from users; embedded in tools like LinkedIn (it already is).

    For most internal applications data can’t really be described as big. A system like LinkedIn or facebook or even a retailer such as Amazon does have a big data issue – lots of events by a large number of users every day. HR typically doesn’t have this issue. HR datasets are pretty small but the key issue faced is probably one of sparseness. Go into any level of granularity and you quickly are forced to play with very small numbers of observations.

    HR data could become bigger if IT system log data was incorporated, which it can be at the individual-use level. You quickly have an issue about representativeness of the sample given much information won’t be captured by systems. Analysis of twitter data doesn’t tell you what the population thinks, it tells you what twitter users are interested in tweeting.

    Combining structured and unstructured HR data from multiple sources for reporting is something we’re working on at the moment. I think there is huge promise and the prospects we’re showing are excited how it helps answer the ‘why’ part of an issue to a far greater extent than previously available. But though we’re working with data sets many times larger than can be probably handled in typical tools like Excel I don’t think we need to be rushing to adopting something like Hadoop quite yet.

  3. @ Andrew: Without giving away the store: what are you and your colleagues trying to do? (Please use laypersons’ lingo.)



  4. Thanks Keith.

    What we’re doing is applying a marketing analytics approach to employees. We bring together data from a variety of sources, be that corporate HR systems, non-HR corporate systems and combine it with data which we capture, both structured and non-structured and data that is publicly available. All of this is linked together into a big multi-dimensional dataset (lots of variables) mostly at the employee level.

    After making this data available we then apply some models to score and rank employees based on the data combinations we see. For example we might apply a segmentation model based on employees’ needs. We might perform some sort of risk prediction.

    Finally we provide custom interactive dashboards designed to answer common questions in ways that general HR folk can understand, delivered through a web-browser. So a viewer route might be to look at turnover data, be guided to see where there are issues / trends and then be shown, let’s say for an ‘at risk’ group 18months into the firm, what this group were saying differently in their first month than folk who didn’t leave. This bit is based on work on data driven user interactions & perceptions. The statistics is very much ‘behind the scenes’.

    We don’t think that it’s fundamentally a technology problem so the positioning is a mixture of expertise and platform with strong domain expertise. We haven’t developed any technology, rather apply best-in-breed solutions few of which have been traditionally used in HR. Whilst I noted above we are not using technology like Hadoop (massive datastore used by most ‘big data’ environments) all of the tools we use are found in these environments, all could be connected to Hadoop and a ‘big data store’ if we chose to do so.

    I’ve recently been working on a project outside HR; working with a big global marketing function to define & implement what is called ‘closed loop marketing’ (see wikipedia). I guess you could call our approach as ‘closed loop HR’.

  5. Thanks, Andrew. If I understand you, you take employee data and see who is likely to screw up or burn out?


  6. Kevin

    Yes, there are HR opportunities for Big Data. Just how big is the opportunity? I think the answer is: It depends.

    The premise of all these points of view is that data can hold a key to insight. The opportunity is to either find a way to mine existing data, or devise a data collection method that can be mined. That is exactly what a pre-employment assessment validation analysis accomplishes.

    Your four call outs: Job Descriptions, Sourcing, Assessment and Metrics and Performance can be excellent sources of data. When properly gathered and combined for HR analytics, it can become Big Data, and provide big insights, insights with the ability to deliver high ROI.

    The Big Data you reference in IBM, McKinsey et al, are massive data sets, the scope of which most HR or recruiting functions may not have, or possess the skills and computers to analyze.

    Validation analysis, which examines the relationship among response patterns to a simulation for pre-employment candidate evaluation and metrics of on-the-job performance can easily reach a million data points with a modest sample size. Conclusions from this level of Big Data provide tremendous insight regarding the job-fit of candidates for one position.

    Companies that engage in local validation of assessment reap the benefit of big data on a micro scale.

    One of the downfalls or Achilles heels of Big Data, is that one begins by searching for order in chaos. Companies that invest in developing job-specific assessment content and then conduct local validation realize that beginning the data collection with the RIGHT data makes all the difference. This approach does not begin with chaos. The quality of the conclusions and the reliability of the algorithm at identifying and reporting on candidate-job fit reduces waste and low-end performance variation in the business process called staffing.

  7. This is a very interesting topic to me. Thank you Kevin.

    From a Recruiting perspective, I do agree that new technologies will offer us new ways to find better employees through the seas of applicants and candidates. But, my gut reaction is that there may be many challenges with things like EEO, PCI and other factors such as the privacy agreements of a company and it’s customers for example. The legal aspect is the area that I think will slow progress with high power analysis of Big Data.

    Still, I’m very excited to see this technology emerge to help us humans work through the massive amounts of data to help our clients (or client groups) hire better talent that add additional value to the jobs they do. I believe this kind of innovation is what makes America the greatest place on the earth to do business.

    Thanks again Kevin and to Keith, Andrew and Joseph for your thoughts !



Leave a Comment

Your email address will not be published. Required fields are marked *