Bridging the Gap: Using Forward Sales Samples to Improve CAMA Models

Many large municipalities face dire financial challenges resulting from shrinking commercial tax base due to rising office vacancies and falling property prices (e.g., some properties have recently sold for land values, etc.) from the continued work-from-home trends, so they must not only avoid wasteful expenditures but also explore new and innovative methods to enhance accuracy and reliability of Computer-Assisted Mass Appraisal (CAMA) models to implement stable assessment rolls, vastly reducing the incidence of appeals, refunds, and risks of expensive ratio trials.

In the world of CAMA, it is commonly understood that the quality of CAMA models should be assessed by performing a series of error tests on the sales samples used for modeling. However, these samples are often not subject to thorough testing to ensure they represent the entire population. As a result, the values obtained from these samples may need to accurately reflect the modeling statistics' overall quality. Therefore, evaluating a set of sales ratios from forward sales samples is essential to ensure the quality and effectiveness of the population values.

Forward samples derived from more recent sales and adjusted to the valuation date can significantly benefit CAMA modeling. If CAMA models based on Multiple Regression Analysis (MRA) are built efficiently and adhere to proper econometric requirements, the results from forward samples could closely resemble those from the initial modeling samples, thus improving model accuracy and reliability.

Unfortunately, many jurisdictions in the US and Canada still rely on CAMA packages with outdated, non-econometric methodologies. These packages often lack the necessary tests to ensure that modeling sales samples are representative, leading to a one-size-fits-all approach to modeling. This underscores the urgent need for improved methodologies and highlights the potential benefits of forward sample testing.

The need for forward sample testing arises because many CAMA models are often developed separately from their populations. In other words, modelers using outdated platforms may start the modeling process when the sales samples are created without confirming whether those samples genuinely represent the unsold populations to which the models will eventually be applied. Even where the prior sample tests are meaningfully performed, tracking the forward sales samples and ratios would be a sound statistical practice to identify the geographic areas or the stretches on the value curve (e.g., the longer end of the value curve, etc.) where models tend to fail or return less than adequate results.

Potential Benefits

Forward sales samples can be beneficial for assessing the effectiveness of CAMA models, particularly in the context of the changing real estate landscape and the challenges municipalities face. Here are some key benefits of using forward sales samples in CAM:

Reflecting Current Market Conditions: Forward sales samples derived from more recent transactions and adjusted to the valuation date can help capture current market conditions more accurately. This can lead to more reliable and up-to-date review and validation for CAMA models.

Improving Model Accuracy: By incorporating forward sales samples into the testing and validation process of CAMA models, assessment departments can enhance the accuracy and reliability of their valuation models. This could potentially reduce the incidence of appeals and refunds, ultimately leading to more stable assessment rolls.

Identifying Model Weaknesses: Tracking forward sales samples and ratios can help identify areas where CAMA models may fail or produce less accurate results. This can enable assessment departments to make targeted improvements to their modeling methodologies and better address any shortcomings in the valuation process.

Reduced Appeals and Refunds: A more accurate model can lead to fewer property tax disputes, saving large tax jurisdictions significant time and money.

Risk Mitigation: By identifying potential issues early, assessment departments can proactively address them, reducing the risk of expensive ratio trials.

Evolving Methodologies: Given the rapid changes in the real estate market, it is crucial for assessment departments to continuously evolve their methodologies and embrace more modern and econometric-based approaches to CAMA modeling. Incorporating forward sample testing can be a step in the right direction toward modernizing valuation practices.

In summary, forward sales samples can potentially enhance the effectiveness of CAMA models. Assessment departments should seriously consider incorporating this method into their valuation processes. By adopting more dynamic and data-driven approaches to assessment, municipalities can better navigate the financial challenges they face and ensure the accuracy and reliability of their property valuations.

Linear CAMA Models

Since most CAMA models are based on linear multiple regression analysis (MRA), they tend to underperform at both ends of the value curve, i.e., below the 25th and above the 75th percentile of the curve, generally producing less efficient values and resulting in mass appeals.  

They consistently overpredict at the shorter end of the curve while grossly under-predicting at the longer end, causing severe regressive situations that lead to "cause for concern."

However, these models generally perform reasonably well in the middle of the curve, i.e., between the 25th and 75th percentile. Therefore, in addition to the model-wise sales ratio stats (COD, PRD, etc.), those who practice linear modeling should also examine those stats separately—below the 25th percentile, 25th to 75th percentile, and above the 75th percentile—to stay alert and cautious about the potential failures at either end of the curve.

Analysis

Many assessment departments face the challenge of MRA and CAMA models underperforming at the ends of the value curve due to the inherent limitations of linear models, which assume a linear relationship between the independent variables (e.g., property characteristics) and the dependent variable (property value).

Nonlinear Relationships: The relationship between property value and characteristics can often be nonlinear. For example, the value of a property might increase exponentially or logarithmically with the size of the lot rather than linearly.

Prevalence of Outliers: Outliers (extreme values) can disproportionately influence the model's coefficients, leading to inaccurate predictions at the extremes.

Data Sparsity: There may be fewer data points at the extremes of the value curve (significantly below the 10th and above the 90th), making it more difficult for the model to accurately capture the relationship between variables in those regions.

Model Performance: Linear MRA models often struggle to accurately predict property values at the extreme ends of the value curve. Below the 25th percentile and above the 75th percentile, these models tend to exhibit significant over-prediction and under-prediction, respectively. This can lead to mass appeals and regressive situations, causing concerns about the fairness and accuracy of property assessments.

Value Segments: Given the varying performance of CAMA models across different segments of the value curve, assessment departments must analyze model-wise sales ratio statistics separately for each segment. By examining performance metrics such as Coefficient of Dispersion (COD) and Price Related Differential (PRD) below the 25th percentile, between the 25th and 75th percentile, and above the 75th percentile, analysts can gain insights into where the models may be failing and adjust their methodologies accordingly.

Forward Sales Samples: Forward sales samples can be invaluable in monitoring these value segments and evaluating the effectiveness of CAMA models across the entire curve. By incorporating more recent sales data and adjusting it to the valuation date, analysts can track how well the models perform in real-time and identify any discrepancies or inaccuracies in their predictions.

Identifying Failures: By using forward sales samples to analyze value segments, assessment departments can proactively identify areas where the models may be underperforming. This early detection allows analysts to make targeted adjustments to their modeling techniques, address issues at the ends of the curve, and improve the overall accuracy of property valuations.

In summary, monitoring value segments and utilizing forward sales samples can help assessment departments address the challenges associated with linear MRA models. This approach ensures more accurate and reliable property assessments and allows significant improvement. By focusing on the performance of CAMA models at different points along the value curve and leveraging real-time data from forward samples, analysts can enhance the quality of their valuation processes and minimize the risks of appeals and potential regressive situations.

Hybrid CAMA Models

The forward sample test is paramount, where final values are created by a hybrid process—top-down MRA-based values and bottom-up comparable sales analyses. Many jurisdictions worldwide are still half-sold on the mass appraisal concept and modeling. Because they lack confidence in the MRA process, they try to supplement it by averaging (watering down) the statistically significant MRA values with highly subjective comparables (generally three to five comps).

Suppose the MRA returns a value of $500K when determining the value of parcel X, while the comps produce $400K. As a result, the final roll value for that parcel would be $450K. While jurisdictions aim for transparency, explicability, decomposability, and limited experimentation, they often unknowingly introduce high subjectivity (and perhaps bias) into the valuation process, resulting in unsmoothed and jagged values throughout the roll.

In hybrid environments, where the comps have a constrained bottom-up contribution to the final values, the overall modeling COD would be significantly lower than the COD from the MRA model alone. For example, if the overall COD is 9, the Comps-only COD would be closer to 6, and the MRA-only COD would be around 12. Although these COD statistics seem favorable initially, the CODs from the forward samples are likely to be much higher, around 14, as the roll would already be published by then, and the original comps would be embedded.

It is crucial to avoid future confusion by only publishing the COD from the MRA component, as the CODs are only intended for mass appraisal models. This practice ensures transparency and clarity in the valuation process, as the CODs from efficient MRA models are comparable across all forward samples.

Continuously tracking results from the forward sales samples is critical, as it ensures the reliability and accuracy of the models and allows for the identification and investigation of any potential issues.

Analysis

The issue of a hybrid approach in mass appraisal models, combining top-down MRA-based values with bottom-up comparable sales analyses, is a critical consideration for assessment departments. Here's how forward sales samples can help monitor value consistency in hybrid CAMA modeling environments:

Hybrid Approach and Subjectivity: In hybrid modeling environments where MRA values are averaged with comparable sales analyses, there can be a risk of introducing subjectivity and bias into the valuation process. This hybrid approach may lead to unsmoothed and jagged values throughout the assessment roll, potentially impacting the accuracy and reliability of property valuations.

Impact on Coefficient of Dispersion (COD): Combining MRA values and comparable sales analyses can result in an overall COD lower than what would be obtained from the MRA model alone. While lower COD statistics may initially seem favorable, including highly subjective comps can lead to higher COD values when considering forward samples due to the presence of embedded original comps and potential inconsistencies in valuation.

Transparency and Clarity: To maintain transparency and clarity in the valuation process, it is crucial to avoid confusion by only publishing the COD derived from the MRA component. This practice ensures that the COD values reported are based on the statistical efficiency of the MRA models and can be compared consistently across all forward samples, providing a transparent and standardized measure of valuation accuracy.

Monitoring Value Consistency: Continuously tracking results from forward sales samples is essential for monitoring the consistency of values derived from CAMA models comprising the assessment roll. By analyzing how well the models perform over time and across different market conditions, assessors can identify any discrepancies or trends that may indicate the need to adjust the valuation methodology.

Role of Forward Sales Samples: Forward sales samples play a crucial role in helping assessors monitor the reliability and accuracy of CAMA models. By evaluating the models' performance against more recent sales data, assessors can ensure that the values produced reflect current market conditions and maintain consistency in property assessments.

In summary, forward sales samples can help monitor value consistency in CAMA models incorporating a hybrid valuation approach. By tracking the results from forward samples and focusing on the statistical efficiency of the MRA component, assessment departments can maintain transparency, accuracy, and reliability in their property valuations, ultimately improving the effectiveness of the assessment process.

Sales Chasing

Sales chasing is prohibited in mass appraisal modeling. If models are built chasing sales, they could be easily identified. Efficient MRA models generate market-significant values; therefore, forward samples would demonstrate similar stats to their modeled counterparts. Alternatively, those models must be investigated if the forward sales samples show much worse stats (much higher COD, irrational PRD, etc.).

State equalization boards must prioritize using forward sales samples to establish fair and equitable equalization rates. This approach, free from sales manipulation, will give all stakeholders security and confidence.

Local Value Adjustment Boards and Review Commissions can also use forward samples to identify inaccuracies in CAMA models, enabling them to manage resources and pinpoint higher-risk areas efficiently.

Local mass appeals filers can also benefit from forward sales samples. These samples can help them quickly identify model failures and optimize their marketing strategies.

Given the dynamic nature of local housing markets, forward samples are crucial in defining divergence points. This information is essential for all stakeholders, ensuring they are well-informed and prepared for market changes.

Although some independent consultants are proposing challenger models to reduce sales manipulation, this could be costly for financially struggling municipalities. Instead, promoting forward sales samples can level the playing field and save costs. IAAO and State Equalization boards should seriously consider and promote this option to avoid expensive ratio trials, providing reassurance and confidence to all stakeholders.

Forward Sales Samples to Mitigate Potential Sales Chasing

Sales chasing is prohibited in mass appraisal modeling. If models are built chasing sales, they could be easily identified. Efficient MRA models generate market-significant values; therefore, forward samples would demonstrate similar stats to their modeled counterparts. Alternatively, those models must be investigated if the forward sales samples show much worse stats (much higher COD, irrational PRD, etc.).

State equalization boards must prioritize using forward sales samples to establish fair and equitable equalization rates. Free from sales manipulation, this approach will give all stakeholders security and confidence.

Local Value Adjustment Boards and Review Commissions can also use forward samples to identify inaccuracies in CAMA models, enabling them to manage resources and pinpoint higher-risk areas efficiently.

Local mass appeals filers can also benefit from forward sales samples. These samples can help them quickly identify model failures and optimize their marketing strategies.

Given the dynamic nature of local housing markets, forward samples are crucial in defining divergence points. This information is essential for all stakeholders, ensuring they are well-informed and prepared for market changes.

Although some independent consultants are proposing challenger models to reduce sales manipulation, this could be costly for financially struggling municipalities. Instead, promoting forward sales samples can level the playing field and save costs. IAAO and State Equalization boards should seriously consider and promote this option to avoid expensive ratio trials, providing reassurance and confidence to all stakeholders.

Conclusion

This blog post highlights the importance of forward sales samples as a risk management tool, especially given many taxing jurisdictions' challenging financial circumstances. Here is a summary of the key points discussed:

1. Financial Challenges: Many municipalities face dire financial circumstances due to shrinking commercial tax bases, falling property values, and rising office vacancies resulting from continued work-from-home trends.

2. Risk Management with Forward Sales Samples: Forward sales samples are crucial for assessing the effectiveness of CAMA models, particularly in monitoring the accuracy and reliability of valuations across different segments of the value curve.

3. Challenges of Linear MRA Models: Linear multiple regression analysis (MRA) models often underperform at the ends of the value curve, leading to potential mass appeals and regressive situations. Forward sales samples can help identify these issues and improve model accuracy.

4. Hybrid Modeling Approaches: A hybrid approach combining MRA values with comparable sales analyses can introduce subjectivity and bias into the valuation process. Monitoring forward sales samples is essential to ensure consistency and reliability in the valuation models.

5. Avoiding Sales Chasing: Sales chasing, where models are built using recent sales data, can lead to inaccuracies in valuations. Forward sales samples can help minimize sales chasing and enhance model efficiency, stability, and customer confidence.

6. Promoting Cost-Effective Solutions: Instead of costly challenger models, promoting forward sales samples can level the playing field, save costs, and provide reassurance and confidence to stakeholders.

In conclusion, leveraging forward sales samples as a risk management tool is essential for taxing jurisdictions to navigate financial challenges, improve model accuracy, and ensure fair and transparent property valuations. By monitoring forward samples and proactively addressing issues identified, assessment departments can enhance the reliability of their valuation models and build trust among stakeholders.

Sid's Bookshelf: Elevate Your Personal and Business Potential