دانلود کتابچه راهنمای تجزیه و تحلیل آماری: کاربردهای هوش مصنوعی و ML

This book will help the novice user become familiar with data mining. Basically, data mining is performing data analysis (or statistics) on data sets (often large) that have been obtained from potentially many sources. As such, the miner may not have control of the input data but must rely on sources that have gathered the data. As such, there are problems that every data miner must be aware of as they begin (or complete) a mining operation. I strongly resonated with the material on “The Top 10 Data Mining Mistakes,” which gives a worthwhile checklist: • Ensure you have a response variable and predictor variables—and that they are correctly measured. • Beware of overfitting. With scads of variables, it is easy with most statistical programs to fit incredibly complex models, but they cannot be reproduced. It is good to save part of the sample to use to test the model. Various methods are offered in this book. • Don’t use only one method. Using only linear regression can be a problem. Try dichotomizing the response or categorizing it to remove nonlinearities in the response variable. Often, there are clusters of values at zero, which messes up any normality assumption. This, of course, loses information, so you may want to categorize a continuous response variable and use an alternative to regression. Similarly, predictor variables may need to be treated as factors rather than linear predictors. A classic example is using marital status or race as a linear predictor when there is no order. • Asking the wrong question—when looking for a rare phenomenon, it may be helpful to identify the most common pattern. These may lead to complex analyses, as in item 3, but they may also be conceptually simple. Again, you may need to take care that you don’t overfit the data. • Don’t become enamored with the data. There may be a substantial history from earlier data or from domain experts that can help with the modeling. • Be wary of using an outcome variable (or one highly correlated with the outcome variable) and becoming excited about the result. The predictors should be “proper” predictors in the sense that they (a) are measured prior to the outcome and (b) are not a function of the outcome. • Do not discard outliers without solid justification. Just because an observation is out of line with others, it shouldn’t be ignored. You must check the circumstances that led to the value. In any event, it is useful to conduct the analysis with the observation(s) included and excluded to determine the sensitivity of the results to the outlier. • Extrapolating is a fine way to go broke; the best example is the stock market. Stick within your data and if you must go outside, put plenty of caveats. Better still, restrain the impulse to extrapolate. Beware that pictures are often far too simple and we can be misled. Political campaigns oversimplify complex problems (“my opponent wants to raise taxes”; “my opponent will take us to war”) when the realities may imply we have some infrastructure needs that can be handled only with new funding or we have been attacked by some bad guys. Be wary of your data sources. If you are combining several sets of data, they need to meet a few standards: • The definitions of variables that are being merged should be identical. Often, they are close but not exact (especially in metaanalysis, where clinical studies may have somewhat different definitions due to different medical institutions or laboratories). • Be careful about missing values. Often, when multiple data sets are merged, missing values can be induced: one variable isn’t present in another data set; what you thought was a unique variable name was slightly different in the two sets, so you end up with two variables that both have a lot of missing values. • How you handle missing values can be crucial. In one example, I used complete cases and lost half of my sample; all variables had at least 85% completeness, but when put together, the sample lost half of the data. The residual sum of squares from a stepwise regression was about 8. When I included more variables using mean replacement, almost the same set of predictor variables surfaced, but the residual sum of squares was 20. I then used multiple imputations and found approximately the same set of predictors but had a residual sum of squares (median of 20 imputations) of 25. I find that mean replacement is rather optimistic but surely better than relying on only complete cases. Using stepwise regression, I find it useful to replicate it with a bootstrap or with multiple imputations. However, with large data sets, this approach may be expensive computationally. To conclude, there is a wealth of material in this handbook that will repay study.