Are you collecting apples or oranges?
What do fruits have to do with a sector-wide data project? In 2010, when we started working on HomeKeeper and the HomeKeeper National Data Hub, we envisioned creating a shared data hub of aggregated program data to help programs measure social impact and benchmark their success to their peers. Creating a national data hub already seemed like a daunting task, but it was complicated by the fact that comparing one program’s outcomes to another was a lot like comparing apples to oranges, as the saying goes.
The shared road to shared data
One our first tasks in building a shared measurement system was to develop a common set of inputs and standard data collection practices with our earliest pilot members. This would allow us to create common social impact metrics for the sector. Fortunately, three organizations in the Northwest, OPAL Community Land Trust, Homestead Community Land Trust and Kulshan Community Land Trust, had already begun aligning the way they tracked their program intake forms. But when we started looking at the existing data tracking spreadsheets from all the organizations in our pilot program, no two programs had the same list.
Some groups were tracking age, others were tracking date of birth. Some never asked if their applicants were first-time homebuyers and most programs couldn’t tell whether or not homeowners were moving on to market rate homeownership when they sold their deed-restricted home. Tracking income and area median income (AMI) was even more tricky since some programs were only tracking the eligibility income, while others relied only on gross household income. While every program agreed that tracking the property address and number of bedrooms was important, some programs had never tracked any appraised market values, the subsidies attached to those homes, or even the year the home was built.
In the beginning it was clear that transitioning to a common standard was going to mean change and compromise for everyone. It was a thoughtful process, with a lot of discussion and input over a series of webinar conference calls. Eventually the discussion yielded some consensus. For example, we decided we weren’t going to track congressional district, because it’s not found on an appraisal and we can probably map that later, we figured. And we decided to let programs define “first-time homebuyer” themselves because there is more than one official definition if you ask around. We agreed that everyone should start tracking date of birth of all household members, so we could count children and seniors housed dynamically. And that everyone should have an exit survey for outgoing homeowners in place. Some very good metrics were proposed, but did not make it on the list, like the energy ratings of homes and commute times before and after purchase. If programs wanted to track more information, they certainly had the flexibility to do so, but our task at hand was to identify what the group thought every program should always track at a very minimum. The data had to be reasonably accessible and of interest to the majority of the programs to make the cut.
Setting a new standard
Eventually, everyone was convinced it was well worth the effort. This was despite the fact that it meant asking more applicant details on the application and incorporating additional processes, like capturing appraisal information and reviewing HUD-1 settlement statements after close. By the end of our first year, we not only had buy-in on a common list of fields and practices, but the process had inspired programs to suggest new metrics that everyone wanted to add, like employment and educational status of individual household members. People were literally asking us to require them to input more data!
As a result of this year-long process working with program staff and stakeholders from across the country, along with researchers from the Urban Institute, we ultimately assembled a standardized list of data fields and data management practices for measuring program performance and impacts. These standards represents what the Cornerstone Partnership and the HomeKeeper community believe homeownership programs need to tell a more complete story about the impact of their programs and our sector as a whole. As we discover and learn how to improve the ways we measure program outcomes, these standards will no doubt be refined and expanded. But for now, they represent our best attempt at identifying and defining the data every program should track.
If you manage a homeownership program that preserves affordability over the long term, we invite you to align your program documents and procedures with this data collection checklist developed for you by your peers. This is intended to be used by all homeownership programs regardless of whether or not you are a HomeKeeper user. By going through the process of reviewing and updating your forms and procedures, your program will reflect many program design best practices. With a community of practitioners collecting the same data in the same way, we can more accurately measure the impact of the sector at national level and you can compare your program metrics to your peers knowing that you’re making a true apples to apples comparison.