How Should You Test AI Underwriting Software? Use a Deal You Have Already Reviewed

A convincing demo uses a polished deal package. A useful evaluation also includes the package your team finds difficult: stale operating statements, unusual rent labels, incomplete comps, and an expense category that needs judgment. Those are the places where automation either saves review work or creates it.

When comparing multifamily acquisition underwriting software, define the outcome before comparing interfaces: a reviewable investment analysis built from the same deal evidence. Commercial real estate cash flow analysis software needs to pass both a data extraction check and a model check. An attractive dashboard does not establish that either passed.

Choose a small test set with known answers

Begin with several completed deals your team is permitted to use. Include a clean package, a messy package, and a package with missing evidence. Keep the original files unchanged and prepare a separate answer set reviewed by someone who understands the deals. Record every source version so the software and the reviewer work from the same information.

Do not judge the tool against a final model that includes information it never received. If your historical model used a later tax quote or a negotiated loan term, provide that evidence or mark the comparison as unavailable. A disagreement with an unsupported answer key is not necessarily a model error.

Define success before seeing the result

Choose tolerances that fit your models. A rounded display value and a full-precision workbook may differ harmlessly; an omitted expense may not. Classify materiality before testing so that an attractive return cannot change the acceptance rule halfway through the evaluation.

Test whether the tool handles unknowns honestly

Use a copy of a package with a deliberately omitted supporting document, clearly marked as a test. Ask the tool to produce the analysis and list what remains unsupported. A useful result identifies the gap and its consequence. A confident rent comp, insurance quote, or loan term invented to fill the gap should count as a failure.

NIST's Generative AI Profile discusses confabulation, including false or misleading supporting material. That is why an evaluation should check whether an output is supported, not just whether it sounds financially literate. Our checklist translates that concern into a deal-review exercise; it is not a NIST certification.

Change one material assumption and follow the effect

Hold other inputs fixed and change a cap rate, a recurring expense, or a financing input. Check that the affected calculations move for the right reason. For a simple illustration, a $25,000 increase in annual operating expenses reduces NOI by $25,000 when income and all other expenses stay fixed. At a fixed 5% direct-cap rate, indicated value falls by $500,000.

A levered IRR needs a separate cash-flow check because timing, debt service, sale proceeds, and the investment basis also matter. Do not treat an independently verified direct-cap calculation as validation of the entire return model. Save the before and after assumption sets and explain any difference.

Measure the work after the first answer

Track active analyst minutes for preparing files, correcting extracted fields, reviewing assumptions, reconciling calculations, and packaging the output. Record elapsed processing time separately. A tool can be fast to generate and slow to verify. It can also take longer to process while substantially reducing analyst effort. Both facts belong in the evaluation.

For example, compare a manual process taking 90 active minutes with an automated process taking 15 minutes of setup, 35 minutes of review, and 10 minutes of corrections. The illustrative saving is 30 active minutes, or one-third, not the difference between 90 minutes and the time to the first AI response. Do not generalize a single test into a guaranteed saving.

Apply the same test to Cactus

Cactus provides one automated underwriting workflow that uses the deal documents to build an analysis of income, expenses, valuation, buyer returns, and next questions. The example report we inspected exposes assumptions and sensitivities, distinguishes NOI measures, and says when comparable evidence is unavailable. Those are observable features to test on your own package.

• Bring the same files and acceptance criteria you use for other tools.
• Ask to inspect an important assumption and its supporting evidence.
• Trace a change through the valuation and return analysis.
• Check the available report sharing and Excel output in the demo.
• Record unresolved issues and total review effort before deciding.

The result should be a list of passed checks, failures, and limitations for the deals you tested. That is more useful than one accuracy percentage that mixes easy fields, missing inputs, and judgments about the future.