Choose an AI task you can measure

Define the decision before the model

A prediction is useful only in relation to a task. The scikit learn evaluation guide distinguishes prediction from decision making and explains that metrics should follow the intended goal. A classification label, a probability, and a numerical forecast need different evaluation choices. A single accuracy figure is therefore an incomplete starting point for a project. [1]

Build an evaluation brief

Write one sentence describing what a user will do with the result. Then identify the reference data, the unit being predicted, and the cost of different mistakes. For an image classifier, a mislabeled image and a duplicated image are different problems. For a forecast, an error on an ordinary day and an error during a peak may have different practical consequences.

Our suggested brief includes a baseline, a fixed evaluation dataset, a chosen metric, and a review of representative errors. Keep the baseline simple enough to inspect. Reserve the evaluation data before experimenting with the model. Record how labels were obtained and which records are uncertain. This makes it easier to identify whether a later improvement comes from the model, the preprocessing, or the dataset itself.

Inspect errors, then decide what to build

Group mistakes by conditions that matter to the workflow. Review whether the model performs differently on new sources, rare classes, or difficult measurements. Decide how uncertain predictions reach a person for review. A project may need better data collection before it needs a larger network. The result of the first iteration should be an inspectable decision about the next step, rather than an unsupported promise that AI will automate everything.