Home / Blog / Data Science Digital Book / Model Evaluation Techniques

Model Evaluation Techniques

• July 15, 2023
• 2624
• 54

Meet the Author : Mr. Bharani Kumar

Bharani Kumar Depuru is a well known IT personality from Hyderabad. He is the Founder and Director of Innodatatics Pvt Ltd and 360DigiTMG. Bharani Kumar is an IIT and ISB alumni with more than 18+ years of experience, he held prominent positions in the IT elites like HSBC, ITC Infotech, Infosys, and Deloitte. He is a prevalent IT consultant specializing in Industrial Revolution 4.0 implementation, Data Analytics practice setup, Artificial Intelligence, Big Data Analytics, Industrial IoT, Business Intelligence and Business Management. Bharani Kumar is also the chief trainer at 360DigiTMG with more than Ten years of experience and has been making the IT transition journey easy for his students. 360DigiTMG is at the forefront of delivering quality education, thereby bridging the gap between academia and industry.

The set of error functions below can be used to assess the model if the output variable 'Y' is continuous.

Error = Predicted Value - Actual Value (Actual Value is also called as Ground Truth Value)

• Mean Error (ME)
• Mean Absolute Error (MAE) or Mean Absolute Deviation (MAD)
• Mean Squared Error (MSE)
• Root Mean Squared Error (RMSE)
• Mean Percentage Error (MPE)
• Mean Absolute Percentage Error (MAPE)
• Mean Absolute Scaled Error (MASE)
• Correlation Coefficient

Learn the core concepts of Data Science Course video on YouTube:

MAEin-sample naive is the mean absolute error produced by a naive forecast

If the ‘Y’ is Discrete variable (Classification Models) then we can use the following list:

Confusion Matrix:

can be used for multi-class classification models as well as binary classifications.

To compare anticipated values with actual values, confusion matrix is employed.

Binary Classification Confusion Matrix:

• True Positive (TP)

Patient with disease is told that he/she has disease
• True Negative (TN)

Patient with no disease is told that he/she has no disease
• False Positive (FP)

Patient with no disease is told that he/she has disease
• False Negative (FN)

Patient with disease is told that he/she has no disease

• Precision = Prob. of correctly identifying a random patient with disease as having disease
Precision is also called as Positive Predictive Value (PPV)
• Sensitivity (Recall or Hit rate or True Positive Rate) = Proportion of people with disease who are correctly identified as having disease
• Specificity (True negative rate) = TN/(TN+FP) = Proportion of people with no disease being characterized as not having disease
• FP rate (Alpha or type I error) = 1 - Specificity
• FN rate (Beta or type II error) = 1 - Sensitivity
• F1 = 2 * ((Precision * Recall) / (Precision + Recall)); F1: 1 to 0 & defines a measure that balances precision & recall

F1 score is the harmonic mean of precision and recall. Closer the ‘F1’ value to 1, better the accuracy.

Confusion Matrix

A cross table or contingency table are other names for a confusion matrix. Here is a multi-class categorization issue example.

Values off the diagonal are incorrectly predicted, whereas values along the diagonal are correctly predicted.

ROC Curve

Since World War II, the Receiver Operating Characteristic Curve has been employed to discern between genuine signals and false alarms.

The 'True Positive Rate (TPR)' and 'False Positive Rate (FPR)' are plotted on the Y-axis and X-axis, respectively, of the ROC curve.

Accuracy is represented graphically using the ROC curve.

The cut-off value is also determined using the ROC curve.

Examples include: Risk Neutral: The probability should be > 0.5 as the cut-off value to classify a customer under the "will default" category; Risk Taking: The probability should be > 0.8 as the cut-off value to classify a customer under the "will default" category; or Risk Averse: The probability should be > 0.3 as the cut-off value to classify a customer under the "will default" category.

AUC (Area Under the Curve) may be determined numerically if one needs to assess the accuracy.

0.9 - 1.0 = A (outstanding)

0.8 - 0.9 = B (excellent/good)

0.7 - 0.8 = C (acceptable/fair)

0.6 - 0.7 = D (poor)

0.5 - 0.6 = F (no discrimination)

Click here to learn Data Science Course, Data Science Course in Hyderabad, Data Science Course in Bangalore