DS 4100 Day 15
DS 4100 Data Collection, Integration, and Analysis
Today we’re continuing our lesson on statistics.
Statistical Inference
Statistical inference combines the methods of descriptive statistics with the theory of probability to infer characteristics of a large population from small sample. The sample must be selected randomly and must be “large enough” to be statistically significant.
Statistical Significance
Statistical significance means that the characteristics of the sample are likely not due to chance or random error. Significance is expressed by a probability. In most sciences, a confidence level of 95% is generally accepted as statistically significant, i.e., we accept a 5% probability of being wrong about our conclusion.
Population vs. Sample
A sample is a small set of randomly selected representatives from a larger population. When it is impossible (or very difficult) to measure a population, then select a sample.
Standard Error
The standard error of a sample is a measure of the sampling error and can be used to estimate the standard deviation of a population:
SE = deviation / sqrt(n1)
where n is the sample size, i.e., the number of representative elements.
Random Sample
Samples must be drawn randomly which means that each element of the population has the same probability of being included in the sample. Random samples are often drawn through random events, such as assigning numeric identifiers to each element and selecting a set of random numbers through a computer or a random number table.
Concept of Correlation
A correlation is a relationship between two variables in which one variable changes in a quantifiable way with another.
For example, there are correlations between:
 Weight and cholesterol level
 Task completion time and task complexity
 Salary and years of education
Consider the following:
In the late 1940s, before there was a polio vaccine, public health experts in America noted that polio cases increased in step with the consumption of ice cream and soft drinks. Eliminating such treats was even recommended as part of an antipolio diet. It turned out that polio outbreaks were most common in the hot months of summer, when people naturally ate more ice cream, showing only an association or correlation.
So, does eating ice cream or hot weather cause polio?
Detecting and Measuring Correlation
Correlation is most easily detected through exploratory visualization with a scatter plot.
The strength or degree of correlation is measured by the coefficient of correlation, R.
The value of R ranges from 1 to +1.

Positive: as one variable increases, the other increases as well

Negative: as one variable increases, the other decreases

Once a correlation has been established, the relationship can be mathematically quantified in a formula through regression analysis:

Linear regression: line

Nonlinear regression: curve

Pearson Moment Coefficient R
Recall that the value of R is between 1 and +1.
An absolute value close to 1 is a strong correlation, whereas a value close to 0 indicates little to no correlation.
Values above 0.6 are generally considered to be useful, while values above 0.9 are indications of strong correlations.
Assessing Correlation
1) Identify the pairs of values to be tested for correlation 2) Create a scatter plot of the data 3) Visually inspect the clustering and trending of the data to determine if there is potential for correlation 4) Calculate the Pearson Moment coefficient of correlation, R 5) Evaluate R
Assumption of Normality
The Pearson Moment coefficient of correlation (R) assumes that the data is normally distributed.
When data is not normally distributed or when the presence of outliers gives a distorted picture of the association between two random variables, the Spearman’s rank correlation is a nonparametric test that can be used instead of the Pearson’s correlation coefficient.
The Spearman’s rank correlation (also called Spearman’s rho ρ) is the Pearson’s correlation coefficient on the ranks of the data.
Determining Normality
Data that is normally distributed would exhibit a “bell curve” distribution with most values around the mean and fewer values on each tail.
Normality can be determined in several ways:
* perform the ShapiroWilk test
* build a QQ plot
* visually inspect histogram
Pearson’s and Spearman’s Correlations
To calculate Spearman’s rho, we need to determine the rank for each of the IQ scores and each of the Rock scores, e.g., the rank of the first IQ score (cell A4) is =RANK.AVG(A4,A$4:A$13,1).
Next calculate both correlation coefficients as follows: Pearson’s =CORREL(A4:A13,B4:B13) = 0.036 Spearman’s =CORREL(C4:C13,D4:D13) = 0.115
Result: there is no significant correlation between IQ and listening to rock music based on the sample.
Correlation in the Presence of Outliers
 Outliers can affect the calculation of correlation, so plot the data in a scatter plot or histogram to determine if there are outliers.
 Calculate the Pearson’s correlation coefficient for the sample with and without the outliers. If there isn’t much difference, then the outliers are not influencing the results.
 Calculate the Spearman’s rank coefficient. If it is close to the Pearson’s correlation coefficient, it is also a good indicator that the outliers are not substantially influencing the results.
 If there are clear differences then you will need to determine how you treat the outliers, i.e., whether to remove them.
Summary
 Correlation is an important data analysis tool
 The strength of a correlation is expressed with the coefficient of correlation, R
 Correlations can be positive or negative
 Correlation does not equate to causation
 Use the Pearson Moment correlation if the data does not have outliers and is reasonably normally distributed, otherwise use the Spearman Rank correlation
Predictive Analytics
Predictive analytics is an area of data mining that deals with extracting information from data and using it to predict trends and behavior patterns. Often the unknown event of interest is in the future, but predictive analytics can be applied to any type of unknown whether it be in the past, present or future.
Another definition:
Predictive analytics is the practice of extracting information from existing data sets in order to determine patterns and predict future outcomes and trends. Predictive analytics does not tell you what will happen in the future. Predictive analytics is largely focused on forecasting under risk where different outcome scenarios occur at differing probabilities.
Forecasting
Forecasting of demand, storage growth, resources, network traffic, orders, and so forth is a key responsibility of an information scientist.
 There are four general approaches:
 TimeSeries Models
 Causal Models
 Qualitative Models
 MonteCarlo Simulation
 Qualitative Models
 Single and WideBand Delphi
 Expert Judgment
 BottomUp Composite
 Stakeholder Survey
 TimeSeries Models
 Moving Average
 Exponential Smoothing
 Trend Projection
 Causal Models
 Linear Regression
 Multiple Regression
 NonLinear Regression
Qualitative Models
Qualitative models incorporate expert opinions and subjective factors. Useful when subjective factors are thought to be important or when accurate quantitative data is difficult to obtain. Common qualitative techniques are
 Delphi (single and wideband)
 Expert Judgment
 BottomUp Composite
 Stakeholder Survey
TimeSeries Models
Timeseries models attempt to predict growth based on historical data. Common timeseries models are
 Moving average
 Exponential smoothing
 Trend projections
Causal Models
Causal models take other factors into account and are often more accurate. The objective is to build a model with the best statistical relationship between the variable being forecast and the independent variables. Regression analysis is the most common technique used in causal modeling.
Moving Average
Moving averages work best when the data does not contain any trend or cyclic patterns. The forecast is the average of the most recent n data values, Yt.
This method tends to smooth out shortterm irregularities in the data series.
Weighted Moving Average
Weighted moving averages use weights to put more emphasis on recent periods Often used when there is slow growth.
Selecting Weights
Start with equal weights (standard moving average) and then put progressively more weight on the most recent data if there’s growth. Exponential smoothing is best if there’s more emphasis on recent data. Regression trend models are better when there are growth or cycles.
Exponential Smoothing
The smoothing constant (α) is the percentage by which the forecast takes into account the most recent data point.
F_t+1 = F_t + α(Y_tF_t)
Start with a smoothing constant α = 0.3 and then adjust the model. Bootstrap the model by using the first data point as the forecast for the first time period.
Holt’s Method
Holt’s Method adds a trend component to exponential smoothing to produce a secondorder model.
FwT_t+1 = F_t+T_t FwT_t+1 = F_t+(1β)T_t1+β(F_tF_t1)
High values for β makes the model more responsive to recent changes in trend.
Comments