January 2011 How Does Social Learning Measure Up - T+D - ASTD

E-dition	Email	PDF	Podcast	Print

T+D JANUARY 11 // FEATURE //

How Does Social Learning Measure Up? Social doesn’t have to mean unmeasurable. Seven methods can help learning leaders put hard numbers to soft investments. By Karie Willyerd and Gene A. Pease + From ASTD Press: The New Social Learning // + From Infoline: ASTD’s Best of Social Learning: A Digital Presentation // + From T+D: Learning Gets Social //

Measurement is convenient for the learning function when visible metrics such as revenue or time-to-quota are clear markers of the success of a learning initiative. Unfortunately, not every program or initiative deployed has such easily aligned metrics, and there can be skepticism from internal clients for claiming business results with so many variables affecting success. A learning function may be asked to implement many soft investments, such as leadership development, communication skills, or, more recently, social learning. For these investments, companies must go beyond surveys, smile sheets, and dashboards, and embrace a measurement toolbox that keeps pace with new learning approaches. In the last 40 years, the science of measurement has made exceptional strides—from advances in statistical modeling to computing power that simplifies the most complex mathematical calculations. Yet the learning industry has failed to take full advantage of this progress, and existing measurement models can fall short of delivering truly actionable information. So, what are the alternatives? Plan a measurement approach that can look beyond the business impact of past programs to apply statistical certainty to the success of future programs. We propose seven steps for organizations to use when both past and future investment is critical to business success: Assess the value of a business impact study. Align support and stakeholders. Identify strategic goals. Determine metrics. Collect the data. Calculate the business impact. Improve the impact for future investments. Although these steps can be used with any human capital investment, we will discuss each of these seven steps in the context of a social learning initiative (SLI) at a major technology firm in Silicon Valley. The company saw the potential for social learning to encourage collaboration, produce ideas, and share user-generated content on demand; however, the question remained: Was it a good business investment? 1. Assess the value of a business impact study. The first question that should be asked is whether the benefit of understanding past investment is worth the time and resources required to assess business impact. Particularly in today’s economic environment, every resource is precious, and there is no room for vanity projects. Business impact studies are ideal for initiatives that have high organizational visibility, high expected operational or strategic influence, larger-than-average cost, or greater-than-average risk. Long-term or ongoing investments in which intermediate findings could be applied for a better final return are also good candidates. In the case of SLI, the company had implemented a social learning strategy internally and was considering packaging the solution to sell externally. Knowing which corporate functions most benefitted from the use of social learning would help the company to strategically target and sell to future customers. Other initiatives well suited to an impact study include employee development programs such as peer-to-peer mentoring, leadership training, or diversity initiatives. Does the program have a positive effect on employee retention or advancement? Do particular positions or demographic types benefit more than others? The answers to these questions provide tangible bottom-line advantages for an organization. Proving that a past decision was good is not nearly as valuable as helping to predict where to make future investment trade-offs. 2. Align support and stakeholders. When undertaking a new approach such as social learning, getting buy-in from a variety of stakeholders is essential. Consider what the social learning experience will be within the organization. Is there an existing culture of innovation and technology, or is there a need to build acceptance of a nontraditional learning approach? Which departments will gain the most by participating in or supporting the effort? In the case of SLI, the application was developed first as a support tool for the sales and services team. This group was primed for ongoing training and was motivated to increase individual sales, making it the perfect pilot for the new learning mode. Building alliances with other groups has additional benefits. It raises the learning department to a strategic position within the company, underscoring the connection between investments in people and broader business goals. Additionally, it breaks down barriers, allowing access to other departmental data that will be critical to the success of the impact study. 3. Identify strategic goals. In planning an impact study, the best place to start is at the end—that is, what will success look like? What strategic goals does your initiative hope to meet? Companies eager to jump on the bandwagon of trends, such as social learning, often miss this critical principle: First identify the desired business outcome, then find the right tools to make it happen. The SLI system was tied both to the corporate vision and to specific business objectives. The vision of the company was to have collaborative network participation through shared innovation, community development, and open source leadership. The creation of SLI, a learning 2.0 model that embraces learning as collaborative and user-generated, was a real-world application of this concept. On a practical level, the development of a social learning model had the potential to offer bottom-line savings as an alternative to more cost-intensive instructor-led training and web-based training. Prior to the launch of SLI, the learning function recorded more than 6 million hours of formal training within an 11-month period at significant expense to the company. The portal allowed a greater diversity of content, reaching more unique users at the same or lower cost than before, and making more efficient use of subject-matter experts and content developers. 4. Determine metrics. At the outset of any human capital investment, learning leaders should ask, “How will I connect my investment to the company’s strategic goals?” Working backwards from the desired outcome can help identify the metrics that will help track the effectiveness of the initiative. This is an opportunity to break down the initiative into observable components, identifying both key performance indicators (KPIs) and business impact metrics. To assess the business impact of SLI, study directors identified several essential metrics: usage (unique titles developed, hours of content viewed, and overall usage time), investment (time and cost for development and deployment), and content quality (user feedback and content ratings). To accurately evaluate the newly introduced social learning system as compared with instructor-led and web-based training, metrics were collected for all three modes. Once metrics are determined, study managers must ensure that the desired data exist. Generally, data are housed in four areas: operational (including sales numbers, customer retention numbers, customer service handling times, units of production, and safety scores); HR (including hire date, current position, employment history, ethnicity, and gender); finance; and, often, within third-party entities who are contracted to provide a service such as payroll. In the case of SLI, study managers identified operational and financial data sources for development and deployment information and human resource systems for user demographics. 5. Collect the data. Now that the framework of the study is complete and relevant data identified, the data must be gathered to calculate business impact. Today’s companies are awash in data, but very often, the information is difficult to obtain, kept within departmental silos. This is the point at which alliances with stakeholders from other areas can allow access to rich, existing data sets. To assess the value of SLI, the company needed a clear view of the entire learning investments over a two-year period. While user data showed whether the social learning system was being accessed and how often, key information housed within operations enabled a comparative analysis of development expenses, delivery costs, and resource requirements for each of the learning modalities. Likewise, mining demographic information from HR began to create a profile of which employees benefitted most from the social learning system. 6. Calculate the business impact. Now we can begin to explore important findings about business impact: Who is using the social learning system and how frequently? What are the costs to develop and deploy content? How do these compare to other modalities? What is the quality of the content developed? From this information, a learning leader can gather some interesting descriptive demographics for further study. However, concluding at this stage that a certain indicator has an impact on another indicator is a risky assumption. Factors studied are often correlated, or related, but determining true causation requires more advanced statistical methodology such as general linear modeling. The good news is that the methodology to determine true causation does exist. The results can arm a learning leader with irrefutable proof for which aspects of an initiative work, where these aspects have the most impact, and whether particular groups experience more benefit. The company found that the use of SLI was widespread. From launch through the first 15 months of existence, nearly 11,000 active employees accessed SLI content at least once, representing nearly 40 percent of the company’s workforce. Sales and services led usage (with 76 percent of all content views), while software engineers led content creation (with 45 percent of all content hours created). In addition, the content was found to be of high quality, as measured by user feedback, along with a comparison of test scores for those trained traditionally and those who chose to access content on SLI for test preparation in lieu of formal training. In comparison to instructor-led and web-based training, SLI had several cost advantages. Although ILT requires less investment for development, the delivery costs for a live instructor are high. In contrast, WBT allows for low delivery costs and quick deployment, while demanding a large upfront investment in development (an average of 200 hours of development for every hour of final content). SLI shares the low delivery costs of web-based training but with greatly reduced development and labor costs: The average development time was two hours for every hour of content. Additionally, traditional instructional design and content development expertise is not necessary in SLI’s peer-to-peer learning environment. The company determined that this represented a 28 percent reduction in the development rate used in cost calculations. In all, SLI was shown to have a return-on-investment of 75:1 over web-based training, with a net gain of $29.5 million. Within any organization, this would be valuable insight. We know that the investment was worthwhile: employees are adopting the new learning mode, and the system provides significant cost savings while maintaining a high level of quality. But important questions, which make the difference between simply proving the value of a past initiative and actually improving future investments, remain unanswered. 7. Improve the impact for future investments. It is at this point that predictive analytics takes center stage. Through statistical modeling, companies can answer forward-looking questions that inform not merely whether an initiative had value, but precisely how and why. Let’s look at two specific questions. First, how do you know that the success was a result of a particular initiative and not another factor? A human capital initiative does not take place in a vacuum. A variety of factors exist that can complicate the results. For example, were investment shifts responsible for changes in hours delivered, or were there marketing differences? For many soft investments, there are likely to be a number of variables, such as executive sponsorship or companion organizational initiatives that can affect business results. Second, does the initiative have the same effect on all types of employees? We may be able to tell which users logged the most hours in a social learning system. But which users gained the most value? Did the learning environment contribute to employee retention? Did it lead to promotions into higher pay-scale positions? Predictive analytics allow companies to isolate the effects of individual investments by treating a corporate initiative as an observational study. We are able to move beyond relational correlations to true causation: Specific effects of a human capital program are reported when there is a 95 percent certainty of their impact. The resulting analysis opens up exciting possibilities for companies, turning data into truly actionable intelligence. Once a learning leader knows what aspects of a program have the greatest impact, future initiatives can be developed with known success. In the case of SLI, the learning function discovered that the greatest need area for users was in software, which changed more rapidly than the hardware the company sold. The search data and viewed content also informed the learning function where to focus its limited resources for developing formal content. An increased focus was placed on recruiting experts in software to contribute content and implementing other initiatives to reward usage of SLI to help reduce service calls and response time. Fortunately the tools required to conduct these types of analyses are now widely available and relatively inexpensive, and firms with expertise in these newer types of measurement are able to mentor a learning function early in their journey. Karie Willyerd is co-founder and CEO of Jambok, a social learning platform, and the co-author of The 2020 Workplace: How Innovative Companies Attract, Develop, and Keep Tomorrow’s Employees Today; karie@jambok.com. Gene A. Pease is co-founder and CEO of Capital Analytics Inc., a leading consultancy that enables organizations to measure and optimize the impact of their human capital investments; gpease@capanalytics.com.

 

Request more information or report issues with this page.

To add pages to your ASTD Favorites you must be logged in.

via astd.org