Bridging the Gap: Using Forward Sales Samples to Improve CAMA Models

Many large municipalities face dire financial challenges resulting from shrinking commercial tax base due to rising office vacancies and falling property prices (e.g., some properties have recently sold for land values, etc.) from the continued work-from-home trends, so they must not only avoid wasteful expenditures but also explore new and innovative methods to enhance accuracy and reliability of Computer-Assisted Mass Appraisal (CAMA) models to implement stable assessment rolls, vastly reducing the incidence of appeals, refunds, and risks of expensive ratio trials.

In the world of CAMA, it is commonly understood that the quality of CAMA models should be assessed by performing a series of error tests on the sales samples used for modeling. However, these samples are often not subject to thorough testing to ensure they represent the entire population. As a result, the values obtained from these samples may need to accurately reflect the modeling statistics' overall quality. Therefore, evaluating a set of sales ratios from forward sales samples is essential to ensure the quality and effectiveness of the population values.

Forward samples derived from more recent sales and adjusted to the valuation date can significantly benefit CAMA modeling. If CAMA models based on Multiple Regression Analysis (MRA) are built efficiently and adhere to proper econometric requirements, the results from forward samples could closely resemble those from the initial modeling samples, thus improving model accuracy and reliability.

Unfortunately, many jurisdictions in the US and Canada still rely on CAMA packages with outdated, non-econometric methodologies. These packages often lack the necessary tests to ensure that modeling sales samples are representative, leading to a one-size-fits-all approach to modeling. This underscores the urgent need for improved methodologies and highlights the potential benefits of forward sample testing.

The need for forward sample testing arises because many CAMA models are often developed separately from their populations. In other words, modelers using outdated platforms may begin the modeling process when sales samples are created, without confirming whether those samples genuinely represent the unsold populations to which the models will eventually be applied. Even where the prior sample tests are meaningfully performed, tracking the forward sales samples and ratios would be a sound statistical practice to identify the geographic areas or the stretches on the value curve (e.g., the longer end of the value curve, etc.) where models tend to fail or return less than adequate results.

Potential Benefits

Forward sales samples can be beneficial for assessing the effectiveness of CAMA models, particularly in the context of the changing real estate landscape and the challenges municipalities face. Here are some key benefits of using forward sales samples in CAM:

Reflecting Current Market Conditions: Forward sales samples derived from more recent transactions and adjusted to the valuation date can help capture current market conditions more accurately. This can lead to more reliable and up-to-date review and validation for CAMA models.

Improving Model Accuracy: By incorporating forward sales samples into the testing and validation process of CAMA models, assessment departments can enhance the accuracy and reliability of their valuation models. This could potentially reduce the incidence of appeals and refunds, ultimately leading to more stable assessment rolls.

Identifying Model Weaknesses: Tracking forward sales samples and ratios can help identify areas where CAMA models may fail or produce less accurate results. This can enable assessment departments to make targeted improvements to their modeling methodologies and better address any shortcomings in the valuation process.

Reduced Appeals and Refunds: A more accurate model can lead to fewer property tax disputes, saving large tax jurisdictions significant time and money.

Risk Mitigation: By identifying potential issues early, assessment departments can proactively address them, reducing the risk of expensive ratio trials.

Evolving Methodologies: Given the rapid changes in the real estate market, it is crucial for assessment departments to continuously evolve their methodologies and embrace more modern and econometric-based approaches to CAMA modeling. Incorporating forward sample testing can be a step in the right direction toward modernizing valuation practices.

In summary, forward sales samples can potentially enhance the effectiveness of CAMA models. Assessment departments should seriously consider incorporating this method into their valuation processes. By adopting more dynamic and data-driven approaches to assessment, municipalities can better navigate the financial challenges they face and ensure the accuracy and reliability of their property valuations.

Linear CAMA Models

Since most CAMA models are based on linear multiple regression analysis (MRA), they tend to underperform at both ends of the value curve, i.e., below the 25th and above the 75th percentile of the curve, generally producing less efficient values and resulting in mass appeals.  

They consistently overpredict at the shorter end of the curve while grossly underpredicting at the longer end, causing severe regressive situations that warrant "cause for concern."

However, these models generally perform reasonably well in the middle of the curve, i.e., between the 25th and 75th percentile. Therefore, in addition to the model-wise sales ratio stats (COD, PRD, etc.), those who practice linear modeling should also examine those stats separately—below the 25th percentile, 25th to 75th percentile, and above the 75th percentile—to stay alert and cautious about the potential failures at either end of the curve.

Analysis

Many assessment departments face the challenge of MRA and CAMA models underperforming at the ends of the value curve due to the inherent limitations of linear models, which assume a linear relationship between the independent variables (e.g., property characteristics) and the dependent variable (property value).

Nonlinear Relationships: The relationship between property value and characteristics can often be nonlinear. For example, the value of a property might increase exponentially or logarithmically with the size of the lot rather than linearly.

Prevalence of Outliers: Outliers (extreme values) can disproportionately influence the model's coefficients, leading to inaccurate predictions at the extremes.

Data Sparsity: There may be fewer data points at the extremes of the value curve (significantly below the 10th and above the 90th), making it more difficult for the model to accurately capture the relationship between variables in those regions.

Model Performance: Linear MRA models often struggle to accurately predict property values at the extreme ends of the value curve. Below the 25th percentile and above the 75th percentile, these models tend to exhibit significant over-prediction and under-prediction, respectively. This can lead to mass appeals and regressive situations, causing concerns about the fairness and accuracy of property assessments.

Value Segments: Given the varying performance of CAMA models across different segments of the value curve, assessment departments must analyze model-wise sales ratio statistics separately for each segment. By examining performance metrics such as Coefficient of Dispersion (COD) and Price Related Differential (PRD) below the 25th percentile, between the 25th and 75th percentile, and above the 75th percentile, analysts can gain insights into where the models may be failing and adjust their methodologies accordingly.

Forward Sales Samples: Forward sales samples can be invaluable in monitoring these value segments and evaluating the effectiveness of CAMA models across the entire curve. By incorporating more recent sales data and adjusting it to the valuation date, analysts can track how well the models perform in real-time and identify any discrepancies or inaccuracies in their predictions.

Identifying Failures: By using forward sales samples to analyze value segments, assessment departments can proactively identify areas where the models may be underperforming. This early detection allows analysts to make targeted adjustments to their modeling techniques, address issues at the ends of the curve, and improve the overall accuracy of property valuations.

In summary, monitoring value segments and utilizing forward sales samples can help assessment departments address the challenges associated with linear MRA models. This approach ensures more accurate and reliable property assessments and allows significant improvement. By focusing on the performance of CAMA models at different points along the value curve and leveraging real-time data from forward samples, analysts can enhance the quality of their valuation processes and minimize the risks of appeals and potential regressive situations.

Hybrid CAMA Models

The forward sample test is paramount, in which final values are derived from a hybrid process—top-down MRA-based values and bottom-up comparable sales analyses. Many jurisdictions worldwide are still half-sold on the mass appraisal concept and modeling. Because they lack confidence in the MRA process, they try to supplement it by averaging (watering down) the statistically significant MRA values with highly subjective comparables (generally three to five comps).

Suppose the MRA returns a value of $500K when determining the value of parcel X, while the comps produce $400K. As a result, the final roll value for that parcel would be $450K. While jurisdictions aim for transparency, explicability, decomposability, and limited experimentation, they often unknowingly introduce significant subjectivity (and perhaps bias) into the valuation process, resulting in unsmoothed, jagged values throughout the roll.

In hybrid environments, where the comps have a constrained bottom-up contribution to the final values, the overall modeling COD would be significantly lower than the COD from the MRA model alone. For example, if the overall COD is 9, the Comps-only COD would be closer to 6, and the MRA-only COD would be around 12. Although these COD statistics seem favorable initially, the CODs from the forward samples are likely to be much higher, around 14, as the roll would already be published by then, and the original comps would be embedded.

It is crucial to avoid future confusion by only publishing the COD from the MRA component, as the CODs are only intended for mass appraisal models. This practice ensures transparency and clarity in the valuation process, as the CODs from efficient MRA models are comparable across all forward samples.

Continuously tracking results from the forward sales samples is critical, as it ensures the reliability and accuracy of the models and enables the identification and investigation of potential issues.

Analysis

The issue of a hybrid approach in mass appraisal models, combining top-down MRA-based values with bottom-up comparable sales analyses, is a critical consideration for assessment departments. Here's how forward sales samples can help monitor value consistency in hybrid CAMA modeling environments:

Hybrid Approach and Subjectivity: In hybrid modeling environments where MRA values are averaged with comparable sales analyses, there can be a risk of introducing subjectivity and bias into the valuation process. This hybrid approach may lead to unsmoothed and jagged values throughout the assessment roll, potentially impacting the accuracy and reliability of property valuations.

Impact on Coefficient of Dispersion (COD): Combining MRA values with comparable sales analyses can result in an overall COD lower than that obtained from the MRA model alone. While lower COD statistics may initially seem favorable, including highly subjective comps can lead to higher COD values when considering forward samples due to the presence of embedded original comps and potential inconsistencies in valuation.

Transparency and Clarity: To maintain transparency and clarity in the valuation process, it is crucial to avoid confusion by only publishing the COD derived from the MRA component. This practice ensures that the COD values reported are based on the statistical efficiency of the MRA models and can be compared consistently across all forward samples, providing a transparent and standardized measure of valuation accuracy.

Monitoring Value Consistency: Continuously tracking results from forward sales samples is essential for monitoring the consistency of values derived from CAMA models comprising the assessment roll. By analyzing how well the models perform over time and across different market conditions, assessors can identify any discrepancies or trends that may indicate the need to adjust the valuation methodology.

Role of Forward Sales Samples: Forward sales samples play a crucial role in helping assessors monitor the reliability and accuracy of CAMA models. By evaluating the models' performance against more recent sales data, assessors can ensure that the values produced reflect current market conditions and maintain consistency in property assessments.

In summary, forward sales samples can help monitor value consistency in CAMA models incorporating a hybrid valuation approach. By tracking the results from forward samples and focusing on the statistical efficiency of the MRA component, assessment departments can maintain transparency, accuracy, and reliability in their property valuations, ultimately improving the effectiveness of the assessment process.

Sales Chasing

Sales chasing is prohibited in mass appraisal modeling. If models are built chasing sales, they could be easily identified. Efficient MRA models generate market-significant values; therefore, forward samples would demonstrate similar stats to their modeled counterparts. Alternatively, those models must be investigated if the forward sales samples show much worse stats (much higher COD, irrational PRD, etc.).

State equalization boards must prioritize using forward sales samples to establish fair and equitable equalization rates. This approach, free from sales manipulation, will give all stakeholders a sense of security and confidence.

Local Value Adjustment Boards and Review Commissions can also use forward samples to identify inaccuracies in CAMA models, enabling them to manage resources and pinpoint higher-risk areas efficiently.

Local mass appeals filers can also benefit from forward sales samples. These samples can help them quickly identify model failures and optimize their marketing strategies.

Given the dynamic nature of local housing markets, forward samples are crucial in defining divergence points. This information is essential for all stakeholders, ensuring they are well-informed and prepared for market changes.

Although some independent consultants are proposing challenger models to reduce sales manipulation, this could be costly for financially struggling municipalities. Instead, promoting forward sales samples can level the playing field and save costs. IAAO and State Equalization boards should seriously consider and promote this option to avoid expensive ratio trials, providing reassurance and confidence to all stakeholders.

Forward Sales Samples to Mitigate Potential Sales Chasing

Sales chasing is prohibited in mass appraisal modeling. If models are built chasing sales, they could be easily identified. Efficient MRA models generate market-significant values; therefore, forward samples would demonstrate similar stats to their modeled counterparts. Alternatively, those models must be investigated if the forward sales samples show much worse stats (much higher COD, irrational PRD, etc.).

State equalization boards must prioritize the use of forward sales samples to establish fair and equitable equalization rates. Free from sales manipulation, this approach will give all stakeholders security and confidence.

Local Value Adjustment Boards and Review Commissions can also use forward samples to identify inaccuracies in CAMA models, enabling them to manage resources and pinpoint higher-risk areas efficiently.

Local mass appeal filers can also benefit from forward sales samples. These samples can help them quickly identify model failures and optimize their marketing strategies.

Given the dynamic nature of local housing markets, forward samples are crucial in defining divergence points. This information is essential for all stakeholders, ensuring they are well-informed and prepared for market changes.

Although some independent consultants are proposing challenger models to reduce sales manipulation, this could be costly for financially struggling municipalities. Instead, promoting forward sales samples can level the playing field and save costs. IAAO and State Equalization boards should seriously consider and promote this option to avoid expensive ratio trials, providing reassurance and confidence to all stakeholders.

Conclusion

This blog post highlights the importance of forward sales samples as a risk management tool, especially given many taxing jurisdictions' challenging financial circumstances. Here is a summary of the key points discussed:

1. Financial Challenges: Many municipalities face dire financial circumstances due to shrinking commercial tax bases, falling property values, and rising office vacancies resulting from continued work-from-home trends.

2. Risk Management with Forward Sales Samples: Forward sales samples are crucial for assessing the effectiveness of CAMA models, particularly in monitoring the accuracy and reliability of valuations across different segments of the value curve.

3. Challenges of Linear MRA Models: Linear multiple regression analysis (MRA) models often underperform at the ends of the value curve, leading to potential mass appeals and regressive situations. Forward sales samples can help identify these issues and improve model accuracy.

4. Hybrid Modeling Approaches: A hybrid approach combining MRA values with comparable sales analyses can introduce subjectivity and bias into the valuation process. Monitoring forward sales samples is essential to ensure consistency and reliability in the valuation models.

5. Avoiding Sales Chasing: Sales chasing, where models are built using recent sales data, can lead to inaccuracies in valuations. Forward sales samples can help minimize sales chasing and enhance model efficiency, stability, and customer confidence.

6. Promoting Cost-Effective Solutions: Instead of costly challenger models, promoting forward sales samples can level the playing field, save costs, and provide reassurance and confidence to stakeholders.

In conclusion, leveraging forward sales samples as a risk management tool is essential for taxing jurisdictions to navigate financial challenges, improve model accuracy, and ensure fair and transparent property valuations. By monitoring forward samples and proactively addressing issues identified, assessment departments can enhance the reliability of their valuation models and build trust among stakeholders.

Sid's Bookshelf: Elevate Your Personal and Business Potential

Representative Sales Sampling: The Not-So-Secret Ingredient of AVM Success

Target Audience: New Graduates/Analysts

When constructing an AVM, it's of utmost importance that the sales sample used in modeling accurately mirrors the population from which it's derived. This accuracy is a key factor in the AVM model's reliability. To achieve this, analysts can employ several methods:

1. Random Sampling: Analysts can use random sampling techniques, ensuring that each property in the population has an equal chance of being included in the sample. This reduces bias and ensures a more representative sample, giving them confidence in the data they are working with.

2. Stratified Sampling: If the population can be divided into different strata (such as different property types or locations), stratified sampling can be used to ensure that each stratum is represented proportionally in the sample.

3. Cluster Sampling: In cases where the population is divided into clusters (such as neighborhoods), they can use cluster sampling to select clusters randomly and include all properties within those clusters in the sample.

4. Weighting: They can apply weighting techniques to adjust the influence of different properties in the sample to better reflect the overall population, which is particularly useful when specific population segments are underrepresented in the sample.

5. Validation and Calibration: They can validate the AVM by comparing the model's estimates to actual sales prices for a separate set of properties. The model can then be calibrated to improve its accuracy and ensure validity when significant discrepancies arise.

Employing these techniques and ensuring that the sales sample represents the population can help analysts create a more robust and reliable model.

Data Selection and Stratification

Geographic location: The sample should reflect the population's geographic distribution, which is especially important when property values vary significantly across areas.

Property characteristics: Analysts should choose a sample that includes properties with characteristics similar to those of the population of interest, including factors such as land and building sizes, age, property type, and amenities, which can be achieved by categorizing the data based on these characteristics and ensuring that each category is proportionally represented in the final sample.

Time: Analysts must ensure that the data used to create the sales sample reflects current market conditions, choose a specific timeframe that represents a stable market, or consider making time-series adjustments. 

Sample Size: The sample size should be large enough to provide reliable estimates of property values, which depend on the model's complexity and data variability. 

By following these steps, analysts can increase the likelihood that the sales sample used in an AVM accurately represents the population from which it is derived, resulting in more accurate property valuations.

Example

Splitting a Sales Sample between Modeling (80%) and Holdout (20%) Samples

Dataset

Since an AVM will be developed using a sample of 11,767 home sales from a specific county, it is statistically necessary to divide the sample into modeling and holdout sets, with an 80%/20 % split. The table above displays the median sale prices and other key independent variables for both samples and the original (pre-split) sample.

Analysis

Analysts need to consider whether there are discernible patterns or biases in the data distribution between the modeling and holdout samples to assess the randomness of the split. In this case, the split has been implemented to maintain the overall characteristics of the entire dataset.

1. Count: The count of observations in the modeling and holdout samples reflects the intended 80-20 split, which suggests that the split was executed as planned without any noticeable bias.

2. Sale Price: The median sale prices in the modeling and holdout samples, $395,000 and $394,500, respectively, are remarkably close to each other and to the pre-split dataset's median. This negligible difference, likely due to random chance, indicates that the split has not significantly altered the sale price distribution between the two samples.

3. Land SF, Bldg Age, Living SF, Other SF, and Bath Count: The median values of independent variables such as Land SF, Bldg Age, Living SF, Other SF, and Bath Count in the modeling and holdout samples closely mirror those of the pre-split dataset. This similarity, particularly the identical square footage and building age, underscores the split's success in maintaining similar distributions of these variables in both samples.

As for the effectiveness of the split, an 80/20 split is a commonly used ratio in the AVM world. In this case, it provides a good balance between having enough data for developing the model (modeling sample) and having enough data to evaluate the model's performance (holdout sample).

In conclusion, the split is random and effective. It maintains the dataset's overall distribution and characteristics in both the modeling and holdout samples, enabling the development of a robust, accurate AVM model that can be validated on the holdout sample.

Sales Sampling in Mass Appraisal

Analysts should carefully consider the implications of using stratified or cluster sampling when developing an AVM to generate a countywide assessment roll to ensure fairness and equity. While these sampling techniques can help ensure a more representative sample and improve the model's accuracy, they can also introduce breaks in values along the stratification or cluster lines, such as towns or neighborhoods within the county.

When creating a countywide assessment roll, breaks in values along stratification or cluster lines may not be desirable, potentially leading to inconsistencies or disparities in property assessments within the same jurisdiction. This, in turn, can potentially result in unfair treatment or unequal tax burdens for property owners in different areas, a situation we all strive to avoid.

Therefore, analysts may need to balance the benefits of using stratified or cluster sampling to achieve a more representative sample with the need to maintain fairness and equity in the assessment roll. They may need to carefully evaluate the potential impact of these sampling techniques on the overall assessment results and consider alternative sampling approaches or weighting methods to address breaks in values along stratification or cluster lines, while still ensuring accuracy and fairness in the assessment roll.

Ultimately, the goal should be to develop an AVM that provides accurate property values while maintaining fairness and equity in the assessment process, taking into account the unique characteristics and challenges of the local jurisdiction.

Sales Sampling in the Private Sector

In the private sector, particularly for AVM vendors or companies focused on developing automated valuation models for commercial purposes, the primary goal is often to produce the most accurate property values possible. In this context, the vendor may prioritize accuracy and modeling performance over concerns about value breaks along a county's stratification or cluster lines.

Therefore, private-sector AVM vendors may use stratified or cluster sampling techniques to develop their models. These methods often lead to more precise and reliable valuations by ensuring a more representative population sample. By using these sampling techniques, the model can capture nuances and variations in property values across market segments, leading to more accurate assessments overall.

While the focus in the private sector may lean toward accuracy and model performance, AVM vendors need to be transparent about their sampling methods and the potential implications of breaks in values across stratification or cluster boundaries. Additionally, vendors should strive to continuously validate and refine their models to ensure they are providing reliable and fair property valuations that meet the needs of their clients and stakeholders.

Ultimately, the decision to use stratified or cluster sampling for AVM development in the private sector may depend on the vendor's specific objectives and the desired balance among accuracy, fairness, and market representation in the valuation process.

Crossover – When a Private Vendor Sells to the Public Sector

Suppose a private AVM vendor contracts to sell values to a County Assessor to assist in producing an assessment roll. In that case, it may be beneficial for the vendor to recalibrate its model using a top-line representative random sample. Recalibrating the model ensures that the AVM's estimates align more closely with the specific market conditions and property characteristics in the county for which the assessment roll is being generated.

Using a top-line, representative random sample, the AVM vendor can play a crucial role in addressing potential issues related to bias or inaccuracies in the original model that may have arisen from different data sources or methodologies. With this recalibration using a representative sample specific to the county where the assessment roll will be used, the AVM vendor can significantly enhance the accuracy and reliability of the property valuations provided to the County Assessor, making the County Assessor feel valued for their contribution.

Additionally, recalibrating the model allows the vendor to tailor the AVM's estimates to reflect the local real estate market's unique characteristics and trends, improving the relevance and usefulness of the valuation results for assessment purposes.

Collaboration is Key

The AVM vendor and County Assessor, both key players in this process, can collaborate to ensure the recalibration process is effective. This would benefit not only the AVM model but also the accuracy of property assessments.

Sharing Data: The County Assessor may have access to additional data on property characteristics and sales that could improve the randomness and representativeness of the sample.

Transparency: The vendor should be transparent about their sampling methods and the potential for bias in their original model, allowing the assessment staff to assess the need for recalibration and its effectiveness.

Overall, recalibrating the AVM model using a top-line representative random sample can help ensure that the property values generated by the AVM align with the County Assessor's specific needs and requirements and contribute to producing a fair and accurate assessment roll.

Using Artificial Intelligence (AI) in Sales Sampling

Artificial Intelligence (AI) can play a crucial role in ensuring that the modeling sales sample used in an AVM represents the population it derives from in both the private and public sectors. Here are some ways in which AI can help achieve a more representative sales sample:

1. Advanced Sampling Techniques: AI algorithms can optimize sampling techniques, such as random sampling, stratified sampling, or cluster sampling, to ensure that the sales sample is representative of the population. AI can help identify patterns in the data and select sample points that reflect the diversity of the overall population.

2. Feature Selection: AI can assist in identifying the most relevant features or variables to include in the sales sample, ensuring that the selected properties capture the key characteristics of the population. Feature selection algorithms can help prioritize variables that have the most significant impact on property values.

3. Data Quality Assessment: AI can analyze the quality of the data used in the sales sample, identifying and addressing any biases, errors, or missing values that could impact the sample's representativeness. AI algorithms can flag data inconsistencies and suggest corrective actions to improve data quality.

4. Model Validation and Calibration: AI can be used to validate AVM models against actual sales data and adjust model parameters through calibration to improve accuracy and ensure that the model reflects the population's actual characteristics. This iterative process helps refine the AVM and enhance its representativeness.

5. Dynamic Sampling Strategies: AI can enable dynamic sampling strategies that adapt to changes in the real estate market or population characteristics over time. By continuously monitoring data trends and adjusting the sampling approach accordingly, AI can help maintain the relevance and representativeness of the sales sample.

6. Fairness and Bias Mitigation: AI can be leveraged to detect and mitigate biases in the modeling sales sample, ensuring that the sample is fair and equitable. Fairness-aware machine learning algorithms can help address bias issues and promote inclusivity in the sample selection.

By incorporating AI-driven approaches into sampling, data processing, model optimization, and validation, AVMs in the private and public sectors can enhance the representativeness of the sales sample used in modeling, leading to more accurate and reliable property valuations that better reflect the target population's characteristics.

Sid's AI-Assisted Bookshelf: Elevate Your Personal and Business Potential