A successful demonstration answer does not mean that the large model application covers all businesses. Assessment samples need to include wording changes, insufficient information, and confusing objects, and the judgment criteria need to be clear before the test to avoid showing only smooth questions and answers.

Sample selection

Organize the test set from real work problems to cover common questions, insufficient information and situations that are easy to confuse. The sample should have source basis and expected processing method, and should not just pick the easiest questions to answer.

Evaluation dimensions

Check whether the answers are accurate, whether the references are relevant, whether the permissions are valid, and whether the business actions are correct. Different scenarios require different standards, and it is not appropriate to summarize all performances into an unexplained score.

Version control

After the model, prompts and documents are changed, use the same set of samples to compare and record the differences. Evaluation helps determine where to improve and does not mean that the model will achieve the same performance on all new problems.

Scenario Check

A set of samples can contain questions that are easy to answer, have similar wording, and have insufficient information, and record the results separately. In this way, we can see where the problems lie, instead of forming an incomplete judgment of the effect by only showing successful cases.

Specific verification methods

Synonymous questions are checked to see if the same applicable basis is found; questions with similar terms are checked to see if the objects are confused; conclusions will not be made when there is insufficient data; tool samples verify authority and results, and each error record indicates the input, source and specific deviation, rather than just counting a passing percentage.

After the samples and the basis for evaluation are clear, the results can be used to determine where to improve. Data, retrieval and process issues are located separately, and subsequent adjustments are checked with the same set of questions to facilitate observation of actual changes.