CRISP - DM Data Collection

CRISP - DM Business Understanding

Articulate the business problem by understanding the client/customer requirements

A few examples on Business Objective and Business Constraints

• Business Problem : Significant proportion of customers who take loan are unable to repay
• Business Objective : Minimize Loan Defaulters
• Business Constraint : Maximize Profits
• Business Problem : Significant proportion of customers are complaining that they did not do the credit card transaction
• Business Objective : Minimize Fraud
• Business Constraint : Maximize Convenience
• Keys points to remember : Ensure that objectives and constraints are SMART
• Key Deliverable: Project Charter

CRISP - DM Data Collection

Understanding various data types is pivotal to proceed further with data collection.

Data Types

Structured vs Unstructured

Balanced vs Imbalanced Data

• Whether a person claims an insurance or not,
• Will a person 'pay on time, 'with a delay' or 'will default', etc,

• Balanced data is that data where the classes of output variables are more or less in equal proportion
• E.g. 47% of people have defaulted and 53% of data is not defaulted in the loan default variable
• When we have balanced data then we can simply apply random sampling techniques

• Imbalanced data is that data where the classes of output variables are in unequal proportion
• E.g. 23% of data is defaulted and 77% of data is not defaulted in the loan default variable
•  

Rule of thumb: If the percentage of the minority output class is less than 30%, the data are unbalanced.

Refer to page 2 for information on sampling for unbalanced data.

When we have imbalanced data then we apply different sampling techniques such as:

• Random Resampling - Undersampling and Oversampling
• Bootstrap Resampling
• K-Fold Cross Validation
• Repeated K-Fold Cross Validation
• Stratified K-Fold Cross-Validation
• Leave-One-Out (N-Fold Cross-Validation) LOOCV
• SMOTE (Synthetic Minority Oversampling Technique)
• MSMOTE (Modified SMOTE)
• Cluster-Based Sampling
• Ensemble Techniques

Click here to learn Artificial Intelligence in Bangalore

Click here to learn Data Analytics in Bangalore

Data Collection Sources

Primary Data: Data Collected at the Source

Secondary Data: Data Collected Before Hand

Primary Data

Examples of Primary Data

Surveys, Design of Experiments, IoT Sensors Data, Interviews, Focus Groups, etc.

Survey steps:

1. Understand the business reality and objective behind conducting the survey. E.g. Sales are low for a training company

2. Perform root cause analysis - Fishbone Analysis, 5-Why Analysis, etc. E.g. Product Pricing is uncompetitive

3. Formulate Decision Problem. E.g. Should product prices be changed

4. Formulate Research Objective. E.g. Determine the price elasticity of demand and the impact on sales and profits of various levels of price changes

5. List out Constructs. E.g. Training Enrolment

6. Deduce Aspects based on construct. E.g. Time aspect, Strength aspect, Constraint aspect

7. Devise Survey Questionnaire based on the Aspects. E.g. I am most likely to enroll for the training program in: In the next one week, In the next one month, In the next one quarter, etc.

Design of Experiments examples:

• Coupon marketing with a 10% discount vs 20% discount, to which of these customers are responding well
• Coupon targeting customers within 10 km radius versus 20 km radius
• Combinations of discount & distance to experiment

Secondary Data

• Oracle DB
• Microsoft DB
• MySQL
• NoSQL - MongoDB
• Big Data, etc.
• Industry reports
• Government reports
• Quasi-government reports, etc.

Click here to learn Data Analytics in Hyderabad

Meta Data Description: Data about Data

• Obtaining meta data description is mandatory before we proceed further in the project
• Understand the data volume details such as size, number of records, total databases, tables, etc.
• Understand the data attributes/variables - description and values which these variables take

Click here to learn Machine Learning in Hyderabad

Base Equation

Output and input variables are two major categories for random variables.

Mathematically the relation between these is expressed using base equation:

Random Variable

Upper case is always used to indicate random variables.

Lower case letters are used to indicate the values that a random variable can take.

Example: One dice roll X equals 1, 2, 3, 4, 5, and 6.

Probability Distribution

Probability Distribution is the process of tabulating or displaying the probability of all potential outcomes of an event.

The underlying probability distribution is known as a probability distribution if a random variable is continuous.

The underlying probability distribution is known as Discrete Probability Distribution if a random variable is discrete.

Inferential Statistics

Inferential statistical is a process of analysing the sample data and deriving statements / properties of a population.

Sampling Techniques

The default strategy for inferential statistics is known as probability sampling, often known as unbiased sampling. Every data point that has to be collected will have an equal chance of being chosen.

A few examples of Non-Probability Sampling:

Sampling Techniques

Unbiased Sampling also known as Probability sampling is the default approach for inferential statistics. Each data point to be collected will have equal opportunity to get selected.

A few examples of Probability Sampling:

Sampling Funnel