Most organizations don’t know what they don’t know, but let’s not confuse that with toxicity. The litmus test is whether the organization understands how important it is to learn. A toxic organization is more likely to cling to old, familiar behaviors and to actively resist efforts to turn foundational ignorance around. A learning organization–on the other hand–understands the value of being wrong and celebrates the journey necessary to getting better.  Toxic organizations already believe they are better and consequently are more likely to engage in consultant abuse.

Several weeks ago, I fired a toxic organization after a steadily devolving relationship spanning five years. This organization (whose name, for no other reason than professional courtesy, won’t be divulged here) had been given a sweetheart role far beyond its own experience to link the expertise of consultants to a range of education providers.  All was reasonable in the first years of this work as the organization and its first wave of consultants coalesced around the mantra of “building a bicycle while riding it.”  Sadly, though, as the work became more complex and trust between the organization and its partner organizations deteriorated, the organization began sliding down the slippery slope of trying to control its own consultants.

Most organizational consultants naively believe in the existence of surefire cures for bad organizations. I’m no exception. As the story goes, bring them to understand their own shortcomings and possibilities to do better–painful as that process may be–and they’ll get better by leaps and bounds, right?  Well, no. Organizations don’t get better unless they sincerely want to.  In that light, there are abundant clues that should guide a consultant’s decision about if and where to spend professional energy.   Here’s Rick Voorhees’ Checklist for Spotting Toxic Organizations:

1.  Foremost, does the organization understand the expertise that consultants bring and that–because the organization lacks the expertise that it is hiring–the consultants it hires are a critical internal customer?  Are consultants seen as partners or as units of expenditure to be controlled?

2.  What is the organization’s track record in dealing with consultants and other partners? Have other consultants told the organization to take a hike?  Why?  Finding out why others won’t work with the organization can save valuable time.

3. What is the trust quotient within the organization? How fearing of its own role is it?  The above organization became increasingly bureaucratic and controlling, requiring consultants to sign agreements that would have limited their consulting with other partners and to seek approval for all travel two months ahead of time. Organizations that truly understand that a networked world depends on expertise also understand that they can’t regulate that expertise

4.  Does the organization treat its consultants like de facto employees?  There’s a distinct role for consultants (and the expertise they bring) versus what the organization can require of its own employees. I was told by a junior staff member that the organization could require any behavior from its consultants since they were, in fact, employees. This has obvious and dangerous legal implications including workmen’s compensation, health insurance, Of course, this very junior person now denies ever saying this.

5.  Does the organization have have a systematic way of evaluating its own performance?  It seems disingenuous that a national organization the promotes data-driven decision making would avoid having concrete ways of gathering data about its own performance in the field. Another sign of dry rot.

6.  How does the organization communicate internally? Externally? This organization attempted to double their dues for constituent institutions one short week after a national meeting in which the leadership of these institutions were present.  Not a good communication strategy.

7. Does the organization hire its own staff and family members as consultants?  In other words, does the organization promote nepotism?

8. Are ego needs out of balance? Does the management’s ego needs outweigh the importance of attending to the work?  In the present example, the organization felt very threatened when given opposite opinions to what their controlling nature told them should be happening.

9. Finally, does the organization actually consult its own consultants?  Or, are they ignored except when very convenient.  Consultants are a critical internal resource and organizations who are too hidebound to learn from them probably ought not to have them.

None of these flaws alone is fatal if the organization can identify them and turn them around. Collectively, however, they are a shipwreck.  Toxic organizations are usually unable to identify any flaws without expert intervention.  And, when #9 is in play, the chances of a toxic organization getting better are zilch. Better to walk away than to be like Sisyphus, constantly pushing a ball up a hill. Sometimes, it’s just the wrong hill.

Others, including Michael Quinn Patton, have been seeking to widen educators’ knowledge about the possible in proving things work.  Disputing this false “Gold Standard,” Patton offers some common sense advice well worth considering.  Considering that standards for research are different from evaluation, Patton includes utility, feasibility, propriety, and accuracy as key goals.  How often to higher education researchers fold these touchstones into their work?  Not often in my opinion, especially when recommending RCT’s.  Remember, the goal of research in the area of student outcomes is to determine what types of students change in what way under what circumstances.

Patton indicates that there are times when RCT’s are appropriate:  drug studies, fertilizer and crop yield studies, and single health practices.  The fertilizer reference, of course, brings me a broad chuckle.  In my keynote to the Association for Institutional Research in 2003, I pointed out that treatment and control group methodologies were a call to “explain a multivariate world with a two variable model.”  I still think I’m right and Patton agrees.  Times when RCT’s are not appropriate include situations that are complex, multi-dimensional and highly context-specific.  Patton uses a community health interventions as an example; more broadly I use any intervention that seeks to change complex human behavior such as learning and skill interventions.  My monograph late in the 1990’s sought to explain, for example, how complex and interrelated factors come together to predict student learning and cognitive development.  Readers wanting a quick overview of how complex events might tie together can find a visual here.

So where does this leave all those well-intended souls wanting to prove that their programs work?  Having to learn much more than the common mantra, I suspect.  Patton talks about both the possible and appropriate, a good place to visit.   Multiple sources of data about each case, triangulation of sources, modus operandi analysis, and epidemiological field are his keywords.  To me, this all sounds a lot like context and generating meaning from each program’s reality rather than hammering on a RCT.  Patton goes further, though, and offers that RCT’s aren’t needed when face validity is high, the observed changes are dramatic, and the link between treatment and outcome is direct.  I think he’s arguing that educators and others frequently use RCT’s as a nail when the only tool they have is a hammer.

This is to say that there are times when RCT’s are appropriate.  Most often, however, the assumptions they carry limit what can be learned about the intervention under scrutiny.  In higher education, they also assume that small, mirco-level programs can either assign students randomly to control and treatment groups or have the sophistication to match treatment and control groups.  The former is virtually impossible from a moral as well as logistical perspective while the latter does violence to the complexity of an intervention.  My experience working widely with programs throughout the United States has taught me that matching subjects on gender, age, and race/ethnicity tells you only whether gender, age, and race/ethnicity have a bearing on an intervention.  Does anyone besides me think that student outcomes depend more on other key factors such as the structure of the intervention, student motivation, and the quality of teaching?  Does lack of a RCT mean that any other data gathered about that program is meaningless?  I hope not.

I’ve come lately to adopt the term developmental evaluation in my consulting practice to distinguish our approach from summative and formative evaluation.  Most funders are interested in summative and formative evaluation while some are moving toward developmental approaches.  The difference?  Summative evaluations are for making final judgments and formative evaluations are directed at improving programs.  Developmental evaluation, on the other hand, talks about ongoing development and knowledge building.  The programs I work with aspire to be innovative and cutting-edge; they don’t have a long history from which to draw.  They’re not ready for rigid summative judgments although they’re receptive to formative help.  They are a work-in-progress.

Innovative programs understand they have much to prove.  Especially in a world that usually believes that a RCT is the sine qua non in research and evaluation.  These programs have much to prove and much to lose.  Using other techniques to prove their worth is attractive including focus groups; comparing outcome data to other, similar programs; journaling and analyzing the insights of faculty about what works; and benchmarking a program’s progress against it’s own performance.  Each has pitfalls, certainly, but if RCT’s are sold as the only answer for judging a programs worth, and they take all the time and resources available for evaluation in a small-scale program, we may never get to the bottom of what really works in a complex world.

At Stellenbosch, two very bright South African higher education pacesetters, Lynda Murray and Pieter Vermeulen, joined with me to evaluate the university’s institutional research and planning function.  These evaluations are always much more than simply institutional research, however.  It’s hard to keep an evaluation of an institution’s use of information inside a tight box, without commenting on the bigger world inside the university and outside.  Stellenbosch was no different.

To say that South Africa’s higher education (tertiary) sector has undergone a major transformation since Reconciliation in 1994 could qualify as an all-time understatement.  The post-apartheid era has caused institutions to rethink not just diversity but their approaches to the World in the 21st century.  Earlier this year the education ministry was split into two parts with higher education joining training to cover private and public institutions.  The scope of this re-constituted ministry function includes universities, colleges, and the skills development sectors, which include the Sector Education and Training Authorities (SETA’s) and the National Skills Authority and the National Skills Fund.   Much more united and certainly ambitious than the American arrangement for higher education where fragmented policy is a fact of life.

Why talk about national changes?  Stellenbosch is one of the oldest universities on the continent and has been a traditional leader in South African higher education.  What Stellenbosch tries, many will emulate.  To ask an external panel to review its use of data and information is tribute to evolving leadership and a willingness to ask hard questions.  I’m not going to give you the details of what we found since that’s up to the University and its able institutional research and planning leader, Dr. Jan Botha, to distribute those details.  I can say, though, that Stellenbosch has committed to furthering its leadership journey in South African higher education by using its data strategically and to rethinking its approach to strategic planning.

Certain truths fall out of any planning situation. I’ve been fortunate to work with some very bright minds in this business, especially Byron and Kay McClenney at the University of Texas, who are constantly pushing institutions to use their own data.  With their help and the scar tissue that any good consultant accumulates, I’ve steadily been adding to my list of critical elements to gauge institutional planning that I sometimes title–somewhat humorously–“Rick’s Rubrics.”  I’ll share these here.  If you’re curious about how they played out at Stellenbosch or in South Africa, drop Jan an email.

• If you’re not planning, you’re planning to fail.  Many institutions have glossy strategic plans but fail to operationalize them by explaining exactly who is doing what, how large its commitment (staff and dollar resources), and how it will know whether it all works.
• Planning, unfortunately, oftentimes becomes a defensive activity.  Many institutions proliferate unit planning to keep things “about the way they’ve always been.”
• Perfect data don’t exist.  Most institutions won’t cross this threshold.  Fear of failure punctuates this stance as does some general ignorance about what data is on hand and what can be created.
• Thin to Win.  Who wants to read a long plan.  Thick plans are a fodder for doorstops, usually.  On the other hand, plans without a clear announcement to specific activities to bring about goals aren’t worth cost of paper.  There’s a balance here and precision wins the day over the ponderous.
• It’s not enough for planning to be participatory; it also had to be decisive.  Committees don’t carry out plans, but the wisdom of those who will carry out the plan is fundamental.  Another balancing act, but let’s error on the side of making decisions not keeping all parties feeling good.
• Select 3 (maybe 4) “main things” that make a real difference.  This is Byron’s critical lesson for me and others.  Not atypically, I evaluated one institution with 39 priorities.  I asked how in the wide world, they could handle 39 priorities the response was that “we meet and talk about them.”  I rest my case!
• Don’t expect a home run every time.  Definitely an “Americanism” and forgive me the sports analogy, but I’ve also seen institutions grow quite tired of planning simply because the results aren’t visible, say, in six months or even a year.  Planning is a journey, not an episode.
• Be flexible and ready to adjust strategies and goals.  Most institutions develop a strategic plan and never adjust it to fit emerging realities and new intelligence.  A periodic review once a year is advisable, prior to setting new action priorities for the next year is advisable.
• Show results widely (even if ugly).  Dirty news seldom survives at most institutions, unfortunately.  A courageous institution uses ugly data to calibrate changes needed to address new priorities.  Audiences sometimes hear from me that dirty data does not make you a bad person!  I hope you see both the humor and the imperative.
• Link clearly to resources.  A plan without a clear tie to human and dollar resources is not a plan, it’s a public relations piece.  Over last decade I’ve seen accrediting agencies awake to institutions with gloss without substance.  A good thing.
• Most critically:  separate the operational from the strategic.  Most institutional managers get hung up here by thinking that their day-to-day activities are strategic when, in fact, they are usually operational (but quite excellent, as I’ve found).  I always get back to doing three (or no more than four) things very well.  Most institutions would do well to define what they mean by “operational excellence” before they pursue strategic goals that are tipping points for the future.