Question

A series of pipelines managed by Pacific Gas & Electric (PG&E) move natural gas across the state of California, providing a source of energy to over 4 million customers. Valve operators, equipment that automatically monitors and adjusts the pipeline pressure or gas flow, help ensure the safe flow of gas through the pipelines. Technicians need to perform scheduled maintenance on these valves to ensure they avoid over-pressuring or under-pressurizing the pipelines, which could cause them to rupture. PG&E redesigned its current training program that focused on these valves. It did so because the training was instructor-led in the classroom using props, as opposed to having technicians working with actual valves. PG&E also decided that the training did not allow the technicians enough time to practice. Finally, it was interested in reducing the number of times technicians had to redo maintenance tasks that were performed incorrectly and the time it took to complete maintenance tasks. PG&E’s new training program uses a 3-D simulation along with instructor-led training. The simulation shows how the valve actually operates and can simulate the tasks that technicians perform such as assembling, disassembling, and cleaning the valve. Technicians can see what happens inside the valve when they make adjustments. The simulation includes a tutorial that guides the technician through the task and a test that can be used to assess whether the technician can correctly perform the task. Technicians are also able to access the simulation on their iPad when they are working on a pipeline.

1) What outcomes should PG&E collect to determine the effectiveness of the new training program?

2) What evaluation design should it use?

3) Explain your choice of outcomes and design.

Answer 1

1. To determine the effectiveness of the new training program, the company must collect the following outcomes:

• The immediate reaction of the technicians to the training program
• The feedback of the supervisors regarding the change of behaviour of technicians after the training
• The average rate of performing maintenance tasks by the technicians, before and after the training
• The number of times technicians had to redo maintenance, before and after the training
• The overall performance of the technicians, before and after the training

2. Kirkpatrick’s Model of Learning Evaluation can be used to evaluate the overall effectiveness of the training program.

3. The Kirkpatrick’s model will evaluate the training at 4 levels:

• Reaction: The immediate reaction of trainees to the training program
• Learning: This level will evaluate the learning, knowledge and skills of the trainees, after attending the training program
• Behavior: The overall enhancement or improvement in the behaviour of the trainee after the training program
• Results: The business impact of the training i.e. the profitability or cost effectiveness brought by the training program. The above mentioned outcomes will depict the cost effectiveness as well as business results.

This will give a comprehensive evaluation of the simulation training and depict its effectiveness for the business model of the company.