Locally Robust Models for Optimization under Tail-based Data Imbalance
Event details
| Date | 23.06.2023 |
| Hour | 14:00 › 15:00 |
| Speaker | Professor Karthyek Murthy |
| Location | |
| Category | Conferences - Seminars |
| Event Language | English |
Abstract: Several problems in data-driven decision-making and risk management suffer from data imbalance, a term referring to settings where a small fraction of data has an outsized impact on estimating one or more decision-making criteria. Due to the paucity of relevant samples, such problems are usually approached with the “estimate, then optimize” workflow involving a model estimation from data in the first step before plugging in the trained model to solve various downstream optimization tasks. As biases due to model selection, misspecification, and overfitting to in-sample data are difficult to avoid in the first-step estimated model, particularly more so in settings affected by data imbalance, we construct novel locally robust optimization formulations in which the first-step estimation has no effect, locally, on the optimal solutions obtained. We show that this local insensitivity translates to improved out-of-sample performance freed from the first-order impact of model errors introduced in the first-step estimation.
A key ingredient in achieving this local robustness is a novel debiasing procedure that adds a non-parametric bias correction term to the objective. The debiased formulation retains convexity, and the imputation of the correction term relies only on a non-restrictive large deviations behavior conducive for transferring knowledge from representative data-rich regions to the data-scarce tail regions suffering from imbalance. The bias correction gets determined by the extent of model error in the estimation step and the specifics of the stochastic program in the optimization step, thereby serving as a scalable “smart-correction" step bridging the disparate goals in estimation and optimization.
Bio sketch: Karthyek Murthy serves as an Assistant Professor in Singapore University of Technology & Design. His research interests lie in data-driven Operations Research. Prior to joining SUTD, he was a postdoctoral researcher in Columbia University IEOR department and he received his PhD in Tata Institute of Fundamental Research, Mumbai. His research has been recognised with 2021 INFORMS Junior Faculty Forum (JFIG) Paper competition (Third place) and 2019 WSC Best Paper Award. Karthyek serves as an Associate Editor for the INFORMS journal Stochastic Systems and as a judge for the INFORMS Nicholson student paper competition.
A key ingredient in achieving this local robustness is a novel debiasing procedure that adds a non-parametric bias correction term to the objective. The debiased formulation retains convexity, and the imputation of the correction term relies only on a non-restrictive large deviations behavior conducive for transferring knowledge from representative data-rich regions to the data-scarce tail regions suffering from imbalance. The bias correction gets determined by the extent of model error in the estimation step and the specifics of the stochastic program in the optimization step, thereby serving as a scalable “smart-correction" step bridging the disparate goals in estimation and optimization.
Bio sketch: Karthyek Murthy serves as an Assistant Professor in Singapore University of Technology & Design. His research interests lie in data-driven Operations Research. Prior to joining SUTD, he was a postdoctoral researcher in Columbia University IEOR department and he received his PhD in Tata Institute of Fundamental Research, Mumbai. His research has been recognised with 2021 INFORMS Junior Faculty Forum (JFIG) Paper competition (Third place) and 2019 WSC Best Paper Award. Karthyek serves as an Associate Editor for the INFORMS journal Stochastic Systems and as a judge for the INFORMS Nicholson student paper competition.
Practical information
- Informed public
- Registration required
Organizer
- Prof. Daniel Kuhn