Skip to content

Explore Courses | Elder Research | Contact | LMS Login

Statistics.com
  • Curriculum
    • Curriculum
    • About Us
    • Testimonials
    • Management Team
    • Faculty Search
    • Teach With Us
    • Credit & Credentialing
  • Courses
    • Explore Courses
    • Course Calendar
    • About Our Courses
    • Course Tour
    • Test Yourself!
  • Mastery Series
    • Mastery Series Program
    • Bayesian Statistics
    • Business Analytics
    • Healthcare Analytics
    • Marketing Analytics
    • Operations Research
    • Predictive Analytics
    • Python for Analytics
    • R Programming
    • Rasch & IRT
    • Spatial Statistics
    • Statistical Modeling
    • Survey Statistics
    • Text Mining and Analytics
  • Certificates
    • Certificate Program
    • Analytics for Data Science
    • Biostatistics
    • Programming for Data Science – R (Novice)
    • Programming for Data Science – R (Experienced)
    • Programming for Data Science – Python (Novice)
    • Programming for Data Science – Python (Experienced)
    • Social Science
  • Degrees
    • Degree Programs
    • Computational Data Analytics Certificate of Graduate Study from Rowan University
    • Health Data Management Certificate of Graduate Study from Rowan University
    • Data Science Analytics Master’s Degree from Thomas Edison State University (TESU)
    • Data Science Analytics Bachelor’s Degree – TESU
    • Mathematics with Predictive Modeling Emphasis BS from Bellevue University
  • Enterprise
    • Organizations
    • Higher Education
  • Resources
    • Blog
    • FAQs & Knowledge Base
    • Glossary
    • Site Map
    • Statistical Symbols
    • Weekly Brief Newsletter Signup
    • Word of the Week
Menu Close
  • Curriculum
    • Curriculum
    • About Us
    • Testimonials
    • Management Team
    • Faculty Search
    • Teach With Us
    • Credit & Credentialing
  • Courses
    • Explore Courses
    • Course Calendar
    • About Our Courses
    • Course Tour
    • Test Yourself!
  • Mastery Series
    • Mastery Series Program
    • Bayesian Statistics
    • Business Analytics
    • Healthcare Analytics
    • Marketing Analytics
    • Operations Research
    • Predictive Analytics
    • Python for Analytics
    • R Programming
    • Rasch & IRT
    • Spatial Statistics
    • Statistical Modeling
    • Survey Statistics
    • Text Mining and Analytics
  • Certificates
    • Certificate Program
    • Analytics for Data Science
    • Biostatistics
    • Programming for Data Science – R (Novice)
    • Programming for Data Science – R (Experienced)
    • Programming for Data Science – Python (Novice)
    • Programming for Data Science – Python (Experienced)
    • Social Science
  • Degrees
    • Degree Programs
    • Computational Data Analytics Certificate of Graduate Study from Rowan University
    • Health Data Management Certificate of Graduate Study from Rowan University
    • Data Science Analytics Master’s Degree from Thomas Edison State University (TESU)
    • Data Science Analytics Bachelor’s Degree – TESU
    • Mathematics with Predictive Modeling Emphasis BS from Bellevue University
  • Enterprise
    • Organizations
    • Higher Education
  • Resources
    • Blog
    • FAQs & Knowledge Base
    • Glossary
    • Site Map
    • Statistical Symbols
    • Weekly Brief Newsletter Signup
    • Word of the Week

Blog

Home » Blog » Blog Type » General Post » Handling the Noise – Boost It or Ignore It?

Handling the Noise – Boost It or Ignore It?

  • March 6, 2019
  • , 5:45 pm

In most statistical modeling or machine learning prediction tasks, there will be cases that can be easily predicted based on their predictor values (signal), as well as cases where predictions are unclear (noise). Two statistical learning methods, boosting and ProfWeight, use those difficult cases in exactly opposite ways – boosting up-weights them, and ProfWeight down-weights them.

Boosting

In boosting, a model is repeatedly fit to data, and successive iterations in the process up-weight the cases that are hard to predict. Specifically, the algorithm is as follows:

  1. Fit a model to the data

  2. Draw a new (often via cross-validation or the bootstrap) sample from the data so that erroneously predicted records have higher probability of selection.

  3. Fit a model to the new sample

  4. Repeat steps 2-3 multiple times

  5. Final predictions are weighted averages of all the models, giving higher weights to the more recent ones.

Boosting is most often employed with decision trees, where it produced significant gains in predictive performance. Why does it work? Boosting is a form of ensemble learning, in which predictions from multiple models are averaged, leveraging the “wisdom of the crowd,†reducing variance, and yielding better predictions than most of the individual models. By forcing the model to focus on the harder-to-predict cases and learn from them, boosting avoids the easy path of gaining a high accuracy score by relying mainly on the easy-to-predict cases.

ProfWeight

ProfWeight is a technology from IBM that does the reverse of boosting: it down-weights the hard-to-predict cases.

Here is a link – https://researcher.watson.ibm.com/researcher/files/us-adhuran/improving-simple-models-final.pdf

The scenario for using ProfWeight is very specific: You have a black-box neural net model that works well, and you need to find a simpler model, either for interpretability or to minimize computation.

Deep learning neural nets are gaining rapidly in popularity as superior performers in a wide variety of situations. However, they do not help you understand or explain the relationships between predictors and outcomes. This is often necessary to communicate the role of statistical models to others in an organization, to understand how predictions might be improved with other data, and to improve processes that give rise to the data. Deep learning is also computationally expensive, and impossible to implement in some sensors, and other memory-constrained environments.

In situations where the deep-learning model is very accurate in generating predictions, ProfWeight uses the information from the deep learning model to guide in the creation of a simpler model that would otherwise not do as well. Just as with boosting, ProfWeight uses the hard-to-predict cases, but it down-weights them rather than upweighting them. Specifically, reaching back into the intermediate layers of the neural net where prediction error is more prevalent, ProfWeight identifies the records that have higher prediction error and then tells the simpler network not to focus on them. The simpler network is thus trained on the easier-to-predict cases, i.e. the less noisy ones. The simpler model is improved by reducing the extent to which it is fit to noise, as opposed to signal.

How can both up-weighting and down-weighting work? The answer is that they are used in different situations:

Boosting is used when you have a weak model, and an ensemble approach can improve its predictive power. Up-weighting the difficult cases is used as part of the learning process, and focuses the ensemble effort where it is most needed.

ProfWeight is used when you already have a strong but complex black-box model, and you need to replace it with a simpler model. Down-weighting is used to remove the noisy cases before the learning process of the simpler model.

Subscribe to the Blog

You have Successfully Subscribed!

Categories

Recent Posts

  • Dec 14: Statistics in Practice December 11, 2020
  • PUZZLE OF THE WEEK – School in the Pandemic December 11, 2020
  • From Kaggle to Cancel: The Culture of AI December 11, 2020

About Statistics.com

Statistics.com offers academic and professional education in statistics, analytics, and data science at beginner, intermediate, and advanced levels of instruction. Statistics.com is a part of Elder Research, a data science consultancy with 25 years of experience in data analytics.

Latest Blogs

  • Dec 14: Statistics in Practice
    December 11, 2020/
    0 Comments
  • PUZZLE OF THE WEEK – School in the Pandemic
    December 11, 2020/
    0 Comments
  • From Kaggle to Cancel: The Culture of AI
    December 11, 2020/
    0 Comments

Social Networks

Linkedin
Twitter
Facebook
Youtube

Contact

The Institute for Statistics Education
4075 Wilson Blvd, 8th Floor
Arlington, VA 22203
(571) 281-8817

ourcourses@statistics.com

© Copyright 2021 - Statistics.com, LLC | All Rights Reserved | Privacy Policy | Terms of Use

By continuing to use this website, you consent to the use of cookies in accordance with our Cookie Policy.

Accept