CMU Statistician Bridges Prediction and Inference in Machine Learning
Carnegie Mellon University's Dr. Ryan Tibshirani, a statistician, has developed a novel method aiming to combine the strengths of modern machine learning and traditional statistics. This innovation seeks to preserve the adaptability and predictive power of current ML techniques while offering valuable statistical insights.
The new method, developed by Tibshirani and his colleagues, targets the gap between prediction and inference in statistics. It opens avenues to add inferential capabilities to other predictive modeling techniques, potentially providing significant value in complex environments with 'big data'.
Machine learning methods, while excelling at prediction, often lack the ability to teach lessons or provide generalizable insights. This is particularly evident in the lasso method, a widely used automated predictive modeling technique. The lasso, despite its effectiveness, lacks standard significance tests, hindering its inferential contributions. Tibshirani's team has addressed this by developing a special significance test for the lasso, enabling it to produce inferential contributions.
Dr. Tibshirani's innovative method bridges the divide between prediction and inference in statistics. By enhancing the lasso method's inferential capabilities, it paves the way for other predictive modeling techniques to provide generalizable insights. This development may help alleviate concerns about the 'end of theory' in statistics, as data volumes grow and automated prediction methods advance.
