Book review: Understanding spatial media

Book review: Understanding spatial media

by | Jul 5, 2017 | Data Sharing, Determinants of Homelessness, Poverty, Poverty Reduction, Research and Data, Technology

Book review: Understanding spatial media

Rob Kitchin, Tracey Lauriault and Matthew Wilson recently co-edited a book titled Understanding Spatial Media. Published by SAGE, this book is about technology, power, people, democracy and geography.

Here are 10 things to know about the book.

1. The book can help us make better decisions pertaining to the acquisition and use of technology and data. Technological advancements happen quickly; this book encourages the reader to think carefully about these changes. It encourages us to think sociologically and technologically at the same time. That’s because technology isn’t neutral; there are politics and values embedded within it. We shape technology, which in turn shapes us.

2. This book matters because homelessness is inherently geographical. As an example, many households currently experiencing homelessness in Calgary used to live in First Nations communities. Understanding what’s driving their ‘flow in’ to Calgary’s Homeless-Serving System of Care is an important part of understanding homelessness in Calgary. Across Canada, rates of child poverty are much higher ‘on reserve’ than ‘off reserve.’ The visual below, taken from this report, makes that point quite clear.

Child Poverty rates graph

3. The book is relevant to Point-in-Time (PiT) Counts of homelessness. Chapter 21 is about spatial profiling; it looks at the action of classifying people and then acting according to that classification. As noted by Calgary Homeless Foundation’s CEO, Diana Krecsy: “The language we use in PiT Counts gets translated into policy discussions.” For example, if communities choose to start counting veterans experiencing homelessness (as has happened in many Canadian communities over the past several years) it can translate into more attention being focused on the housing needs of veterans (which I would argue has also happened).[1] Indeed, categories and definitions used in PiT Counts are not value neutral.

4. Chapter 4, “Digitally Augmented Geographies,” highlights important differences in Internet accessibility globally. Authored by Mark Graham, I found this chapter very readable. Points raised in the chapter include the fact that approximately one-quarter of the world’s population “has still never used the internet” (p. 46). The chapter further notes that, of the approximately 3 billion people in the world who do use the Internet, “many are forced to use it in very restricted ways because of economic necessity (e.g. metered plans, bandwidth caps), technical limitations (e.g. slow connectivity speeds) and government restrictions (e.g. censorship or surveillance)” (p. 46). This is relevant for persons experiencing homelessness, and it’s especially relevant to northern regions of Canada; it’s also an important advocacy area for the Federation of Canadian Municipalities.[2]

5. Chapter 5, “Locative and Sousveillant Media,” discusses the use of smartphones to capture things on video…including things that weren’t intended to be captured on video. Readers learn that approximately two-thirds of the world’s population keeps a smartphone “within arms reach [sic] at all times, reaching for it 150 times per day…” (p. 57). The same chapter includes a discussion of people whipping out their phones and catching ‘incidents of interest’ on video—for me, this brought to mind a May 2017 incident in which a Toronto security guard was caught on camera while throwing objects at a man experiencing homelessness.

6. Chapter 7, titled “Urban Dashboards,” discusses a topic that is very relevant to my work at the Calgary Homeless Foundation (CHF). It notes that, since roughly 1990, the term “dashboard” has denoted “a screen giving a graphical summary of various types of information, typically used to give an overview of (part of) a business organization” (p. 75). It further notes: “Business and urban data displays often mimic the dashboard instrumentation of cards or aeroplanes. Where in a car you would find indicators for speed, oil and fuel levels, here you will find widgets representing an organization’s ‘key performance indicators’: cash flow, stocks, inventory and so forth” (p. 77). Below, readers can see an excerpt from a Calgary community dashboard.

ExternalDashboard_2017_07_05_CROP

 

This image is an excerpt from a Calgary community dashboard measuring collective impact on specific goals pertaining to homelessness to reach by the end of 2018.

7. Several of the book’s chapters are relevant to research currently being done at CHF. CHF is currently doing some ‘GIS mapping’ research. For clients receiving CHF-funded housing, we’re looking at the postal codes where they came from (before entering Calgary’s Homeless-Serving System of Care) as well as postal codes of the housing units where they end up (once we help house them). Similar research is discussed at various junctures of the book. The book’s co-editor, Dr. Tracey Lauriault, recently explained: “One of aims of the book in general is to give you the ‘critical thinking chops’ needed to do GIS mapping research.”

8. The book touches on many of the same issues discussed at the Canadian Homelessness Data Sharing Initiative. This annual event, co-hosted by CHF and the University of Calgary’s School of Public Policy, discusses infrastructure (i.e. where to store data); modeling and schemas (i.e. rules related to a classification system); analytics (i.e. the use of techniques, statistics and software), access to data, sensitivity (i.e. respecting people); linked data; and transparency. All of these topics are discussed in the book. (I’ve previously blogged about the inaugural data-sharing event here, and I’ve recently blogged about the second annual event here).

9. The authors who contributed to this book represent a diverse collection of scholars.Scholars from 18 universities, spanning six countries, participated in the effort. This is indicative of some impressive professional networking on the part of the book’s three editors—they opened a tent and let a lot of thinkers come inside it. This serves readers well.

10. This book is geared mostly to other researchers in geography and media studies. Most of the book’s intended readers are university professors and upper-level university students. I personally had great difficulty understanding several of the chapters. I even found Chapter 1 to be very difficult (it uses the words ontogenetic, neogeography, multivocal, geovisual and synoptic). I would therefore not recommend this book to people who lack prior knowledge in the realm of spatial media.

I wish to thank the following individuals for invaluable assistance with this blog post:   Vicki Ballance, Rachel Campbell, Chantal Hansen, Patrick Hunter, Diana Krecsy, Tracey Lauriault, Kara Layher, Michael Lenczner, Lindsay Lenny and  Joel Sinclair.  Any errors are mine.

 

[1] For examples of recent attention to veterans’ homelessness in Canada, see this this Globe and Mail article and this PowerPoint presentation. (I suspect that a variety of factors, not just PIT Counts, have likely led to increased policy attention on veterans’ homelessness in Canada over the past several years.)

[2] You can download a “pre-publication version” of this chapter here, free of charge. You can check out Professor Graham’s web site here, and you can see his contributions to The Guardian here.

 

You can view a PDF version of this blog post here: Book review- Understanding spatial media 

Second Annual Canadian Homelessness Data Sharing Initiative

Second Annual Canadian Homelessness Data Sharing Initiative

by | Jul 4, 2017 | Data Sharing, Ending Homelessness, Government, Housing, Pit Count, Poverty Reduction, Research and Data, Ron Kneebone

Second Annual Canadian Homelessness Data Sharing Initiative

On May 18, 2017, the Second Annual Canadian Homelessness Data Sharing Initiative took place in Calgary (all slide presentations, as well as photos from the event, are available here). The event was organized by the Calgary Homeless Foundation and the University of Calgary’s School of Public Policy, and the participants included:  people who build datasets (about persons experiencing homelessness); researchers who use that data; persons with lived experience; and public servants.

Here are 10 things to know about this year’s event:

1. For the second year in a row, there was strong representation from Canada’s federal government. Five officials from Employment and Social Development Canada (ESDC) participated, three of whom had formal speaking roles at the event. Aaron Segaert (from ESDC) presented data from more than 200 homeless shelters between 2005 and 2014, showing that:

  • the occupancy rate (i.e. % of beds filled each night) across these shelters rose from 83% in 2010 to 92% in 2014;
  • the average ‘duration of stay’ by households using these shelters is increasing, especially for families and seniors;
  • the number of seniors using homeless shelters annually nearly doubled across Canada between 2005 and 2014.

2. This year’s event had strong Quebec representation. Research presented by Annie Duchesne, for example, finds that certain subgroups of persons in Montreal’s largest homeless shelter are more likely to experience chronic homelessness (i.e. long-term homelessness) than others—those subgroups include persons over the age of 50, persons with mental health problems and persons with disabilities.

3. Indigenous perspectives were presented. Bonnie Healy’s presentation focused on the work of the Alberta First Nations Information Governance Centre. Topics raised in her presentation included First Nations OCAP principles, a publication titled First Nations – Health Trends Alberta, the First Nations health status report for the Alberta region, the work of the Alberta First Nations Governance Centre, and Indigenous logic models.

4. Several data-sharing advocates actively participated in this year’s event. Michael Lenczner, a data-sharing champion in Canada’s social sector, attended and spoke at this year’s event. He stated that, in terms of data sharing, he’s not aware of any other subsector of Canada’s non-profit sector that has an annual forum to discuss the importance of data. He also cited Alberta as a leader in data sharing, making reference mostly to PolicyWise, who’ve worked with government to link client administrative data from multiple ministries. They’re the leaders of this kind of data-linking in Canada, and possibly the world.

5. Difficulties with researchers accessing federal homelessness data were raised. Tracey Lauriault is a Carleton University professor who described her past difficulties in trying to access HIFIS data for research. When she did, she was told that her data requests must be sent to community coordinators; yet, federal officials were never able to provide her with a list of community coordinators.

6. One of the event highlights was a panel discussion on moving towards increased national integration of Homelessness Management Information Systems (HMIS). As I’ve written before, there are multiple software systems across Canada that keep data on persons experiencing homelessness; many people would like to see increased integration of these systems (possibly into one very large system, or at least the sharing of data among these systems so that researchers can have larger samples for their work). Henry Dagher (ESDC) discussed the evolution of the Homeless Individuals and Families Information System (HIFIS) software system (which is one type of HMIS system). This federally-administered HIFIS system is now operating in more than 100 communities. HIFIS 4 is now web-based and gaining strong momentum; BC Housing is now implementing HIFIS province-wide (spanning approximately 200 service providers). Several panel members suggest that community members need to gain more control of HIFIS (via a stakeholder advisory body with some clout that includes persons with lived experience). As Michael Lenczner puts it: “The tail shouldn’t wag the dog.” Jenn Legate (Calgary Homeless Foundation) raised several operational concerns that need to be kept in mind as we move forward on increased national integration of HIMIS systems—namely, the ongoing costs a server, the cost of migrating data from an old database system to a new system, challenges pertaining to customer service provided by software vendors, and legal barriers to data sharing.

7. Important findings were presented from Canada’s recent nationally-coordinated Point in Time Count of homeless persons. Patrick Hunter’s presentation noted that more than 25% of homeless persons enumerated during the 2016 count did not use an emergency shelter during the previous year—I think this speaks in part to conditions in emergency shelters, about which there’s virtually no research.[1] Hunter also reported that Indigenous peoples are nine times more likely to experience homelessness than the rest of Canada’s population; what’s more, more likely to experience longer homeless spells than non-Indigenous people.

8. One of the event highlights was a “review of the day” by Stephen Metraux. Metraux, the Director of the Health Policy Program at the University of the Sciences in Philadelphia, gave a ‘wrap up’ presentation that included a short slide presentation. He subsequently wrote a blog post in which he reflected on his experience at the event.

9. Several suggestions were made about a ‘way forward.’ Topics that need to be tackled in the future include:

10. This will continue to be an annual event that we expect to be held each year in (or near) Calgary. It may also evolve into a two-day format, with one day focusing on the operational aspects associated with building, maintaining and improving HMIS systems. All of these operational matters are the focus of this biannual event in the United States; yet, no Canadian equivalent currently exists.

In Sum. We hope this annual event will help communities across Canada get closer to ending homelessness. The event web page—with slide presentations and minutes from the event—can be found here.

A blog post written about the First Annual Canadian Homelessness Data Sharing Initiative can be found here.

The author wishes to thank the following individuals for invaluable assistance with this blog post: Vicki Ballance, Ron Kneebone, Eric Latimer, Tracey Lauriault, Kara Layher, Michael Lenczner, Lindsay Lenny, Stephen Metraux and one anonymous source. Any errors lie with the author.

 

[1] This 2016 report on conditions inside Out of the Cold facilities and Warming Centres is worth reading. However, it should be noted that the facilities that are the focus of this report are not “homeless shelters” as defined by City of Toronto officials; rather, they operate separately from the formal shelter system.

 

You can view a PDF version of this blog post here: Second Annual Canadian Homelessness Data Sharing Initiative

 

Monitoring Program Performance in Calgary’s Homeless-Serving System of Care

Monitoring Program Performance in Calgary’s Homeless-Serving System of Care

by | Jun 12, 2017 | Uncategorized

Monitoring Program Performance in Calgary’s Homeless-Serving System of Care

Calgary Homeless Foundation (CHF) recently unveiled new key performance indicators (KPIs)for programs they fund.[1] These new indicators were developed after nine months of community consultation and have been piloted over the course of the past year. A May 2017 slide presentation on the development of some of these KPIs can be found here, while a seven-page guide for staff in the sector who do data entry can be found here.

Here are 10 things to know.

  1. Once a client is referred to a housing program, staff who work for that program are expected to locate the client and move them into housing ASAP. Calgary’s Homeless-Serving System of Care uses a triage system called Coordinated Access and Assessment (CAA). On a weekly basis, staff from the sector meet to discuss persons who are experiencing homelessness and who need to be housed, available housing units and ‘who fits where.’ Program referrals are based on conversations that take place at that table. Once a referral is made, CHF makes sure program entry occurs in the shortest possible time. CHF’s KPIs also monitor the percentage of referrals that never materialize (possible reasons for a referral not materializing include: the referral was declined by the funded program or by the client, or the referral didn’t happen because the client couldn’t be located).[2]
  2. Funded programs are encouraged to provide staff only as required (i.e. to not ‘over support’). CHF takes the position that while some tenants will always need case management support (i.e., ongoing professional staff support) many tenants won’t require such support (and some never require it). CHF therefore encourages funded programs to remove unnecessary professional support and promote independence. In that way, CHF creates a smoother ‘flow’ through the system.
  3. Programs funded by CHF are encouraged to persevere in keeping challenging tenants housed. Programs in the single adults and families sector are encouraged to keep people housed for at least nine months (i.e., three consecutive quarters); whereas those in the youth sector are expected to keep a person housed for at least six months (i.e., two consecutive quarters). In all sectors, maintaining housing can include moving people to a new unit in cases where a specific tenancy hasn’t worked well. In effect, programs are encouraged to take on challenging clients and to not give up on them.
  4. Once a person is successfully housed, CHF encourages program staff to be mindful of ‘missing tenants.’ This incentivizes funded programs to track down clients who are missing and to go back to the ‘CAA table’ to offer the vacated spaces to a future tenant within a short time frame.
  5. CHF-funded programs are encouraged (and generally want) to beat the average score for their cohort. On most of the above measures, a cohort average figure (based on the previous year) is calculated for all programs in a specific category (for more on the different categories of programs funded by CHF, see this recent blog post on CHF’s System Planning Framework). CHF sets a benchmark score 10% above the average. Funded programs in that category are then encouraged to score at or better than that benchmark (i.e. at or better than 10% above the cohort average of the previous year).
  6. All of this data is tracked through Calgary’s HMIS system. Each quarter, CHF staff use HMIS data to calculate KPI results for all CHF-funded programs. KPI results are then emailed to funded programs each quarter. No follow-up is required for programs that perform well in their KPIs. However, if CHF staff finds a program’s KPIs to be problematic, they may contact the program for clarification. CHF recognizes that a problem may be simply technical in nature (e.g., a new staff person isn’t entering data into HMIS properly). Other times, it may be a performance issue.
  7. An accreditation process helps ensure accurate data entry. Programs funded by CHF go through an accreditation process with an external accrediting body called the Canadian Accreditation Council (CAC). This service is paid for by CHF. CHF has its own standardsthat it has developed with help from CAC, and CHF-funded programs are expected to meet these standards. This process involves client file review, staff interviews and client interviews. Detailed reports are then provided to both CHF and the funded program. Among other things, this process helps ensure accurate data entry. Client case notes and files are reviewed during each accreditation by the CAC team.
  8. HMIS training also helps ensure accurate data entry. This training, provided free of charge, is provided to programs by CHF staff on a regular basis. Follow-up HMIS technical support is offered throughout the year.
  9. The Government of Alberta (GoA) is a strong supporter of the CHF’s KPIs. The GoA continues to show very strong interest in CHF’s KPIs. GoA plans to report on similar KPIs for the 2018/19 Ministry of Community and Social Services Business Plan which will include results from across the homeless serving system. According to one GoA source: “We have drawn heavily on CHF work to inform this thinking.”
  10. CHF will soon unveil new KPIs for programs that specifically serve Indigenous peoples. These KPIs have been developed in collaboration with key members of Calgary’s Indigenous community. This will be the subject of a future blog post.

The author wishes to thank Brian Bechtel, Jennifer Eyford, Geoff Gillard, Chantal Hansen, Sarah Knopp, Friney Labranche, Kara Layher, Sara Mikhail, Angela Pye, Jaime Rogers, Ken Swift, Alina Turner and two anonymous reviewers for invaluable assistance with this blog post. Any errors lie with the author.

 

[1] For a general overview of the programs funded by CHF, see this previous blog post.

 

[2] It has recently come to light that, in Toronto, officials have trouble locating clients after they’re referred to subsidized housing (and, meanwhile, the units sit vacant).

 

 

You can view a PDF version of the present blog post here: Monitoring Program Performance in Calgary’s Homeless-Serving System of Care

 

Ten Things to Know About Social Assistance in Canada

Ten Things to Know About Social Assistance in Canada

by | May 8, 2017 | Alberta, Determinants of Homelessness, Ending Homelessness, Pit Count, Point in time Count, Poverty Reduction, Research and Data, Social Assistance

Ten Things to Know About Social Assistance in Canada

The present blog post is the first in a two-part series on social assistance. (Part two, which looks at social assistance in Alberta, can be viewed here). The series is inspired by recent data captured in Alberta’s 2016 Point-in-Time Count of Homelessness suggesting that just a small percentage of persons experiencing homelessness in Calgary receive social assistance (see point #7 of this previous post).

Ron Kneebone (Professor of Economics at the University of Calgary) and Katherine White (Yukon’s Deputy Finance Minister) have referred to social assistance as “the final layer of the public social safety net — designed to catch those people in need of support but unable to find it from family, friends or non-government agencies…”

(I’d argue that, in larger urban centres, social assistance is in fact the second-last layer before the homeless-serving sector…)

Here are 10 things to know:

  1. Every Canadian province and territory has its own social assistance system—that is, its own legislation, its own regulations and its own policies.First Nations with self-government agreements have their own “income assistance” programs. And for First Nations without self-government agreements, income assistance is funded by Indigenous and Northern Affairs Canada (but “aligned with the rates and eligibility criteria for off-reserve residents of the reference province or territory”).[1] In the words of Martin Papillon (Associate Professor of Political Science at the University of Ottawa): “First Nations administer income assistance on behalf of federal authorities, yet they follow rules and objectives established by provinces.”[2]
  2. There aren’t enough jobs to go around,[3] and it’s well-known that Employment Insurance benefits provide only temporary coverage (and only cover a small percentage of jobless persons).[4] Without social assistance, many people without jobs would be destitute. This places elected officials and public servants in a conundrum—while wanting to provide some basic income assistance for those without work, they don’t want to ‘make life so comfortable’ for those persons so as to discourage them from actively looking for work. They also don’t want workers to quit their jobs in the belief that social assistance provides a ‘good living.’ In other words, by design, social assistance has two contradictory objectives: 1) to give people enough money to live on; and 2) to not give people enough money to live on.
  3. In Canada, social assistance coverage expanded in the post-World War II era; it then contracted in the 1980s and 1990s. In the years following World War II, Canada experienced low unemployment, high levels of tax revenue and a strong feeling of collective solidarity. During this time, senior orders of government designed and funded a social assistance system with benefit levels and rules that were generous relative to today.[5] From the mid-1960s until the mid-1970s, this expansion was especially fast.[6] (For more on the political and economic factors that led to the post-1970s contraction, see this 2014 article by Jim Stanford.)
  4. Most people agree that social assistance benefit levels are insufficient to live on. Across Canada, 70% of households on social assistance are “food insecure.” In fact, it’s rare to see an elected official or senior public servant even attempt to make a case that social assistance benefit levels are sufficient. In 1995, an Ontario provincial cabinet minister attempted to do this; he was roundly ridiculed. In Alberta, a “single employable adult” on social assistance receives approximately $8,000 annually to live on. (To see social assistance benefit levels for yourself, check out the most recent Welfare in Canada)
  5. Very few immigrants (relative to Canada’s general population) receive social assistance.That’s a finding of research done by Tracy Smith-Carrier and Jennifer Mitchell (and that research is presented in Chapter 17 of this 2015 book on social assistance in Canada). However, a very large percentage of members of First Nations receive “income assistance” (this issue is discussed in detail by Martin Papillon in Chapter 18 of the aforementioned book).
  6. In recent years, there’s been a substantial increase in persons with disabilities receiving social assistance. At a national level, John Stapleton and Anne Tweddle have written about this here. They find this increase to be especially apparent in Ontario, Manitoba, Alberta and British Columbia (and they find it to be most pronounced in Alberta). For a recent review of what this trend looks like in Alberta, see this recent report.
  7. The inadequacy of social assistance puts a strain on other parts of Canada’s social welfare system. Three specific points are worth making here. First, if social assistance benefit levels were higher, there would be less demand for emergency shelter beds (that’s one of the findings of this recent report). Second, most of the government funding required for social housing in Canada is for the “rent supplement” component of the assistance (i.e. financial assistance to cover the gap between what it costs the operator to pay for the housing, on the one hand, and what a low-income household can afford, on the other). There’d be less need for social housing funding if social assistance benefit levels were higher. Third, low income is associated with poor health outcomes,[7] which in turn lead to higher health care costs. It’s therefore likely higher social assistance benefit levels would reduce health care costs in Canada.
  8. Many landlords discriminate against tenants who report social assistance as a source of income. This is commonly known by both social assistance recipients and their advocates. And in 2008, this theory was put to the test in a study where ‘mock phone calls’ were made to Toronto landlords; during the study, researchers found solid empirical support for the claim that landlords do indeed discriminate against social assistance recipients.
  9. Social assistance administrators do not track what happens to people who are denied coverage. In other words, when a person’s application for social assistance is rejected, there’s no systematic effort made to track what happens to them. However, researchers do sometimes look at what happens after people stop receiving social assistance; one such Canadian study is available here.
  10. A modest increase in social assistance benefit levels would likely reduce homelessness. A recent report estimates that modest increases in social assistance benefit levels would likely result in less need for emergency shelter beds for homeless persons. Specifically, the report suggests that a 15-20% increase in benefit levels for ‘single employables’ would likely result in a 15-20% decrease in demand for shelter beds.

In Sum. Across Canada, social assistance plays an important, but insufficient, role in poverty alleviation. Higher social assistance benefit levels would likely result in tangible outcomes, including less food insecurity, improved health outcomes and less homelessness. Part 2 of the present blog series will focus on the Alberta context.

The author wishes to thank Daniel Béland, Gerry Boychuk, Pierre-Marc Daigneault, Louise Gallagher, Seth Klein, Jennefer Laidley, Kara Layher, Lindsay Lenny, Michael Mendelson, Dionne Miazdyck-Shield, Munir Sheikh, Anne Tweddle and Donna Wood for invaluable assistance with this blog post. Any errors lie with the author.

 

Download a PDF of this blog post here: Ten Things to Know About Social Assistance in Canada

 

[1] An important exception is Ontario, where the provincial government is responsible for on-reserve income assistance. Martin Papillon briefly discusses this in Chapter 18 of this book.

[2] I’ve taken this quote from p. 334 of this book.

[3] For more on the relationship between the labour market and social assistance receipt, see Gerard Boychuk’s chapter in this 2015 book. Figure 2.2 in the chapter consists of a line graph suggestive of a strong correlation (R2 = – 0.88) between the percentage of Canada’s adult population receiving social assistance, and the employment rate, over time.

[4] For more on the inadequacy of Employment Insurance benefits, see the Employment Insurance chapter in the 2017 Alternative Federal Budget.

[5] This happened as part of an expansion of Canada’s entire social welfare system. For more on this, see this book by Dennis Guest.

[6] To learn more about this history, check out my PhD thesis, which can be downloaded here.

 

[7] This 2009 report, focusing on the Ontario context, looks specifically at health outcomes of social assistance recipients.

 

Ten things to know about program evaluation and the Calgary Homeless Foundation

Ten things to know about program evaluation and the Calgary Homeless Foundation

by | May 2, 2017 | Ending Homelessness, Research and Data

Ten things to know about program evaluation and the Calgary Homeless Foundation

The Canadian Evaluation Society (CES) recently invited me to speak on a panel discussion. I was asked to speak to how my organization, the Calgary Homeless Foundation (CHF), views program evaluation.

 

The CES defines evaluation as: “the systematic assessment of the design, implementation or results of an initiative for the purposes of learning or decision-making.”

With this in mind, here are 10 things to know about program evaluation in general and specifically its role at CHF:

  1. Formal program evaluation typically has a logic model.[1] In order to build the logic model, evaluators interview staff with direct knowledge of the program being evaluated. They also do a lot of readings—those readings include funding agreements and any reports that led to the genesis of the program being evaluated. The process of producing the logic model should be iterative, with multiple drafts and a feedback process involving program staff. Eventually, both the evaluators and program staff should agree on the logic model’s content.
  2. Some program evaluators learn on the job, while others have formal training. There are at least three ways to obtain program evaluation credentials in Canada: 1) there are several diploma and certificate programs offered by members of the Consortium of Universities for Evaluation Education; 2) the CES offers non-credit courses through its Essential Skills Series; and 3) evaluators can receive credentials via the CES Credentialed Evaluator program.
  3. How ‘arm’s length’ program evaluators should be is often the subject of debate. In Canada’s federal government, most evaluation is done internally. Having said that, when the federal government does evaluation internally, it often outsources many of the specific tasks (e.g., survey administration, the facilitation of focus groups, etc.).
  4. There are advantages to having program evaluation done externally. Those advantages include: 1) capacity (perhaps no in-house staff have comparable levels of expertise, including expertise pertaining to data analysis); 2) perspective (that external person might be able to guide the conversation in a way that insiders, who might have designed the program, might not think about it); and 3) independence (i.e. the avoidance of bias).
  5. There are advantages to having program evaluation done internally. An external evaluator won’t understand the program when they begin the evaluation and will take time to learn; by contrast, when a program evaluation is done internally, all basic information about the program under evaluation is known by evaluators up front. Put differently, when you pay an external person to come in and evaluate your program, your staff have to explain the program’s basics to the external person—and while this is happening, you’re paying both your staff and the external person! Another advantage of doing program evaluation in-house is that it builds capacity—i.e. staff get to know their work better.
  6. Like many non-profit organizations, the Calgary Homeless Foundation typically (CHF) chooses not to use external program evaluators. One major reason for this is cost.
  7. CHF uses a logic model for the housing programs that it funds. This logic model was built ‘in house’ via a process led by a CHF staff person (namely, my colleague Janice Chan). Our logic model begins by outlining the major contributing factors to homelessness. It then lists the major inputs required to successfully house persons who are experiencing homelessness. It then discusses outputs (e.g., number of clients housed within a given time frame), outcomes (i.e., housing stability) and impact (namely, more client independence). This logic model is presented below.

 

LogicModelFlowchart

 

8. CHF’s performance indicators are linked to the CHF’s logic model. CHF’s performance indicators[2] and quarterly monitoring of performance by programs it funds are linked to the logic model presented above (put differently, there’s ongoing monitoring against our logic model). That said, our performance indicators are only one part of our overall performance measurement. With our overall evaluative process, we also ask the following questions about programs that we fund: Has the program in question been providing us with good financial reporting on time? How collaborative have program staff been?
9. CHF contracts (for the programs we fund) contain explicit information about our performance indicators. Indeed, contracts spell out expected performance by that funded program on those indicators. Funded programs are expected to achieve outcomes that are 10% greater than their cohort’s average the previous year. (Click on this link to see the template version of the program outline section of one of our typical contracts; see pp. 10-11 of the template for information on performance indicators.)
10. Some programs funded by CHF have their own logic models. One reason for this is that CHF isn’t always that program’s sole funder. Likewise, CHF funds some non-housing programs (e.g., outreach, prevention) for which this logic model isn’t a neat fit.[3] In such cases, we develop arrangements (and funding contracts) that seek to reconcile this tension.

In sum. Program evaluation is an important tool to demonstrate performance and efficiency. Whether an internal or external approach is adopted, the outcome can provide important information to guide funding and programming decisions.

 

A PDF version of the present blog post is available to download here: Ten things to know about program evaluation and the Calgary Homeless Foundation

 

The author wishes to thank the following individuals for invaluable assistance with this blog post:  Carla Babiuk, John Burrett, Janice Chan, John Ecker, Louise Gallagher, Penny Hawkins, Kara Layher, Lindsay Lenny, Kevin McNichol, Natalie Noble, Rob Shepherd, Tim Veitch and Jeannette Waegemakers Schiff.  Any errors lie with the author.

 

 

 

[1] Behind the logic model is a theory of change.

 

[2] CHF’s performance indicators (for programs it funds) will be the subject of a future blog post.

 

[3] See this previous blog post for an overview of the various program types funded by CHF.