From https://arxiv.org/pdf/2310.08215

• ML often “does not work”
  • It does generalize, but not in the way that one would expect. They tend to generalize well, given sufficient amount of data, appropriate inductive biases, and if we stay in the same distribution as the training set (in-distribution (ID) generalization).
  • “Does not work”: Our models, however, need to cope with new situations in practice. Whenever there are changes in the deployment conditions, our model will usually work much worse
• ML has high operating costs
  • So, we usually need to constantly adjust our model to the new settings. This requires an ML engineer (O(100k USD/year)), collecting fresh data (on dedicated pipelines) or buying specialized proprietary data, and computing resources or credits for an ML cloud to adjust the models on the new data. From a business perspective, these points boil down to a money issue. ML has high operating costs if our model constantly has to be adapted to new scenarios. If we had a model that generalized well, we would have less or even none of these costs.
• ML is currently not trustworthy.
  • Even if we address the previous concerns, broad use of ML is not just a matter of whether our model works well or not – it is difficult to trust ML models. Extreme cases are when our life, health, or money is at stake.
    • Example 1: Ten AI doctors say we have stomach cancer and recommend chemo- and radiotherapy. Could we trust this diagnosis and start these treatments? The majority of people would want to have the cancer pointed out in the MRT images. This is an example of explanability.
    • Example 2: We are in a self-driving car driving through a curvy road along a cliff. Should we lift our hands off the wheel? Probably not. We likely could not even do that because these cars would insist on human intervention (e.g., by giving warning signs). The automatic detection of an unusual environment is an example of uncertainty quantification.
    • Example 3: It is also hard to trust images generated by DALL-E to be sensible: We often see absurd artifacts in otherwise great ML-generated art. This is a problem of OOD generalization, as our model only gives high-quality images for a restricted set of prompts.