In this study, we explore the impact of noise handling strategies on optimization performance in the context of a real-world production planning problem. Uncertainties intrinsic to the production system are captured using a discrete event simulation (DES) model, and the production plan is optimized using an evolutionary algorithm. The stochastic nature of the fitness values (as returned by the DES simulation) may impact onoptimization performance, and we explore explicit and implicit averaging strategies to address this issue. Specifically, we evaluate the effectiveness of different strategies, when a limited budget of evaluations is available. Our results indicate a general advantage of implicit averaging in this setting, and a good degree of robustness with regard to population size. On the other hand, explicit averaging is found to be non-competitive, due to the cost of repeat-evaluations of the same solution. Finally, we explore a hybrid approach that uses explicit averaging to refine fitness estimates during final solution selection. Under increasing levels of fitness variability, this hybrid strategy starts to outperform pure implicit and explicit averaging strategies.