AI Algorithm and Machine Learning vs. Traditional Statistics and Data Science – Part 2

In the first installment of this series, we discussed the distinct variations between AI, traditional statistics, and data science. We learned how AI's capacity to incorporate extensive amounts of data, along with sophisticated techniques like game theory, decision trees, random forests, and dynamic programming, can significantly enhance the usefulness of traditional methods. Lastly, we talked about how individuals, particularly recent graduates, who possess expertise in AI will hold a substantial advantage in the employment market. In this part, we delve into how AI can reshape the universe of parametric predictive modeling.

Parametric vs. Non-parametric Predictive Modeling

Sid: When creating a predictive model that relies on several independent variables, it's important to avoid multi-collinearity amongst them. To check for this issue, the modeler should analyze a correlation matrix and then move on to the regression analysis. Thus, the process for developing a regression model is often a manual, two-step procedure in most statistical software packages. Is it possible to teach machine-learning algorithms to combine these analyses without relying on a specific procedure?

ChatGPT: Yes, it is possible to teach machine learning algorithms to combine these analyses. Many machine learning algorithms make use of feature selection algorithms, which can automatically select the most relevant variables for model building, and can recognize and ignore highly correlated variables. Other algorithms, such as support vector machines and random forest, can also be used for model building, as they can determine the most important variable interactions without relying on a pre-defined procedure.

Bard: Yes, it is possible to teach machine learning algorithms to combine correlation matrices and regression analyses without relying on a specific procedure. One way to do this is to use a technique called regularization. Regularization is a way of preventing machine learning algorithms from overfitting the data. Overfitting occurs when an algorithm learns the training data too well, and as a result, it is not able to generalize to new data.

One type of regularization is called Lasso. Lasso stands for Least Absolute Shrinkage and Selection Operator. Lasso works by shrinking the coefficients of the independent variables towards zero. This helps to prevent multicollinearity, as it reduces the correlation between the independent variables.

Another type of regularization is called Ridge. Ridge stands for Tikhonov Regularization. Ridge works by shrinking the coefficients of the independent variables towards zero, but it does not shrink them as much as Lasso. This helps to prevent multicollinearity, but it also helps to improve the performance of the model on the training data.

Machine learning algorithms can be trained to use regularization techniques automatically. This means that the modeler does not need to manually check for multicollinearity or perform a two-step procedure. The algorithm will automatically learn to avoid multicollinearity and improve the performance of the model on the training data.

Here are some of the benefits of using machine learning algorithms to combine correlation matrices and regression analyses:

·         It is a more automated process, which can save time and effort.

·         It can help to prevent multicollinearity, which can improve the performance of the model.

·         It can improve the performance of the model on the training data.

Here are some of the challenges of using machine learning algorithms to combine correlation matrices and regression analyses:

·         It can be more computationally expensive than manual methods.

·         It can be more difficult to interpret the results of the analysis.

·         It can be more difficult to troubleshoot problems with the model.

Overall, using machine learning algorithms to combine correlation matrices and regression analyses can be a more efficient and effective way to develop predictive models. However, it is important to be aware of the challenges involved in this approach.

Sid: What modeling technique isn't affected by multi-collinearity?

ChatGPT: The modeling technique that is not affected by multi-collinearity is a decision tree. Decision trees are independent of multi-collinearity because they build decision rules to make predictions on the data without relying on linear relationships between variables.

Sid: In that case, will a decision tree be better than regression to develop a pricing model for a high-rise condo building with views on one side but not on the other?

ChatGPT: Yes, a decision tree will likely be better than a regression model to develop a pricing model for a high-rise condo building with views on one side but not on the other. This is because decision trees are not affected by multi-collinearity, and the decision trees can identify complex relationships between the independent variables. Additionally, decision trees are relatively easy to interpret, which is helpful when explaining complex pricing models.

Bard: There are a few modeling techniques that are not affected by multicollinearity, including:

·         Decision trees: Decision trees are a type of non-parametric model that does not make any assumptions about the underlying distribution of the data. This makes them less susceptible to multicollinearity than parametric models, such as linear regression.

·         Random forests: Random forests are an ensemble learning method that combines multiple decision trees. This helps to reduce the variance of the model and make it more robust to multicollinearity.

·         Support vector machines: Support vector machines are a type of machine learning algorithm that can be used for both classification and regression tasks. They are not affected by multicollinearity because they do not make any assumptions about the underlying distribution of the data.

In the case of a high-rise condo building with views on one side but not on the other, a decision tree or random forest model would be a good choice for developing a pricing model. These models can take into account the complex factors that affect condo prices, such as the size and layout of the unit, the floor plan, the amenities, and the view. They can also handle the presence of multicollinearity in the data.

Regression analysis is a parametric model that makes assumptions about the underlying distribution of the data. This makes it more susceptible to multicollinearity. In addition, regression analysis is not as good at handling complex factors, such as the view in a condo building.

Here are some additional details about each of the modeling techniques mentioned above:

·         Decision trees: Decision trees are a type of supervised learning algorithm that can be used for both classification and regression tasks. They work by splitting the data into smaller and smaller groups until each group is homogeneous. The decision tree is then used to predict the value of the target variable for new data points.

·         Random forests: Random forests are an ensemble learning method that combines multiple decision trees. This helps to reduce the variance of the model and make it more robust to noise and overfitting. Random forests are often used for classification and regression tasks, but they can also be used for other tasks, such as feature selection and dimensionality reduction.

·         Support vector machines: Support vector machines (SVMs) are a type of machine learning algorithm that can be used for both classification and regression tasks. They work by finding the hyperplane that best separates the data into two classes. SVMs are often used for high-dimensional data because they can handle nonlinear relationships.

Sid: When working with datasets, there are various types of variables including continuous, categorical, descriptive, and binary. To create a predictive model using multiple regression analysis, non-quantitative variables must first be transformed into quantitative variables. This allows them to be used as independent variables in the equation. One of the common methods for this transformation is linearization, where numerical values are assigned to categories or descriptions. However, this process can be subjective and dependent on the modeler's preferences. For instance, a modeler might assign 100 to "average," 75 to "below average," 125 to "above average," 150 to "good," and 175 to "excellent." But this approach may not always reflect real-world market behavior. How can AI overcome this challenge and handle a diverse set of variables in a model without relying on subjective judgments?

ChatGPT: AI can help to overcome this challenge by performing feature engineering. Feature engineering involves collecting or creating new features that can better explain the data if used as independent variables instead of non-quantitative variables. For example, the non-quantitative variables of the example could be converted to useable features such as the number of words in the description, the length of the description, or the position of the words in the sentence. AI algorithms such as natural language processing and other text analytics algorithms can analyze these new features in order to create more accurate predictive models without relying on subjective judgement. By doing so, AI can create a better understanding of how variables influence the outcome and use this to create a more accurate predictive model.

Bard: There are a number of ways that AI can overcome the challenge of handling a diverse set of variables in a model without relying on subjective judgments. One way is to use machine learning algorithms that can learn the relationships between variables without being explicitly programmed with them. This can be done by using algorithms such as decision trees, random forests, and neural networks. These algorithms can learn the relationships between variables by analyzing large datasets of data.

Another way that AI can overcome this challenge is to use natural language processing (NLP) techniques to extract meaning from text data. This can be done by using algorithms such as sentiment analysis, topic modeling, and named entity recognition. These algorithms can be used to extract information from text data that would be difficult or impossible to extract using traditional statistical methods.

By using these techniques, AI can create models that are more accurate and reliable than models that are created using traditional statistical methods. This is because AI models are not limited to the relationships that are explicitly programmed into them. Instead, they can learn the relationships between variables from the data itself. This allows AI models to be more flexible and adaptable to changes in the data.

Here are some specific examples of how AI can be used to handle a diverse set of variables in a model:

·         Decision trees can be used to create models that can handle both categorical and continuous variables. Decision trees work by splitting the data into smaller and smaller groups until each group contains only one type of variable. This allows the model to learn the relationships between variables without relying on subjective judgments.

·         Random forests are a type of ensemble learning algorithm that can be used to create models that are more accurate than decision trees. Random forests work by creating multiple decision trees and then averaging their predictions. This helps to reduce the variance in the model and makes it more accurate.

·         Neural networks are a type of machine learning algorithm that can be used to create models that can handle complex relationships between variables. Neural networks work by learning the weights of connections between neurons in a network. This allows the model to learn the relationships between variables without being explicitly programmed with them.

These are just a few examples of how AI can be used to handle a diverse set of variables in a model. As AI technology continues to develop, we can expect to see even more powerful and sophisticated techniques for handling complex data sets.

Sid: Do you think AI-powered systems can use a combination of decision trees, random forests, and neural networks, replacing old-fashioned regression-based predictive modeling altogether?

Bard: It is possible that AI-powered systems could use a combination of decision trees, random forests, and neural networks to replace old-fashioned regression-based predictive modeling altogether. However, it is important to note that regression-based predictive modeling is still a powerful tool that can be used in many cases.

Here are some of the advantages of using AI-powered systems to create predictive models:

·         Accuracy: AI-powered systems can often create models that are more accurate than traditional regression-based models. This is because AI models can learn the relationships between variables from the data itself, without relying on subjective judgments.

·         Flexibility: AI models are more flexible than traditional regression-based models. This is because AI models can handle complex relationships between variables, and they can be adapted to changes in the data.

·         Scalability: AI models can be scaled to handle large datasets. This is important for many applications, such as fraud detection and risk assessment.

However, there are also some disadvantages to using AI-powered systems to create predictive models:

·         Complexity: AI models can be more complex than traditional regression-based models. This can make them more difficult to understand and interpret.

·         Data requirements: AI models require large datasets to train. This can be a challenge for some applications, such as fraud detection.

·         Cost: AI models can be more expensive to develop than traditional regression-based models.

Overall, it is likely that AI-powered systems will continue to play an important role in predictive modeling. However, it is also likely that traditional regression-based models will continue to be used in many cases. The choice of which type of model to use will depend on the specific application and the available data.

Note: All conversations here are unedited. E&OE.

Note: ChatGPT is a registered trademark of OpenAI, and Bard is a registered trademark of Google.