The third hut: central vs. distributed IT
Why can't we all get along?
NB: This post is very specific to the organization of IT at universities. While I hope it is of some interest to others, it’s quite higher education focused.
In 2003, when I was new to a campus IT organization and new to cybersecurity - and a newly minted CISO - I was invited to a meeting of the “IT Alliance”. This was a meeting ostensibly to enhance the partnership between the IT staff working in each of the academic and administrative units, and those working in the campus central IT shop. Being relatively new to campus organization, I was just learning that almost every academic unit had its own small- to medium-sized IT staff (as few as 1 and as many as 50 people). There’s an old Jewish joke about a religious man who is the sole survivor of a shipwreck on a remote island. After a few years, a boat finds him, and the crew is amazed to see that he has built three huts. “What are they all for?” they ask. He smiles and replies, “This is my house. This is the synagogue I go to. And that? That's the synagogue I wouldn't be caught dead in.” On many campuses, that third hut is central IT.
This is not to say that third hut is particularly welcoming to the distributed unit-based IT staff. While many readers will push back saying they work well with their central IT organization, tales of central arrogance and poor communications abound, I surely saw this quite often myself. What took place at that initial IT Alliance meeting wasn’t a collaboration. It wasn’t even communication. It was thrown together at the last minute, and had central staff giving lazy presentations on already planned activities to a moribund and sleepy group.
I won’t bore you with the details of how it happened; being new I was too foolish to keep my head down, and when asked to take over the group I did so and (I think) completely transformed it. No doubt others who lived through this have their own recollection of how I did and how it evolved. The object lesson for me wasn’t anything specific I did, but the awareness of two things: universities run on relationships, not power structures; and worldviews are shaped by those human dimensions of control, ego, and status, not common purpose. As I’ve written about elsewhere, where worldviews collide, conflict arises.
This post will be concerned with the central question of “how should central IT and distributed IT be organized”, particularly in light of the crushing budget shortfalls wrought by the collapse of American scientific leadership. Whether there is a cost savings to be had by centralizing the distributed IT function is one that resurfaces regularly. My perspective on this has also changed over the years, as I have confronted the two conflicting pressures as CISO - how can I increase my span of control to enhance cybersecurity vs. the reality that campuses are huge and I need those distributed staff as partners, not obedient drones.
It’s worth trying to characterize the worldviews of central IT and distributed IT. Of course, regardless of any empathy I bring to this, except for my first 7 years in IT I’ve worked for the central shop. So I’ll just acknowledge my blinders and do the best I can1.
A common characteristic of the distributed IT workforce is that not only do they often more directly support faculty and faculty research projects, but it’s quite common that they themselves are graduates of the departments they work in. Graduate students in particular, if they have any technical bent, are drawn into more technical roles in research labs, and upon graduating students are often hired by the department as IT staff. This differs significantly from the management structure of most central IT staff - in university central units, research support is typically balkanized to some specialized sub unit, composed of research facilitators or HPC support. Thus while the typical employee in a central shop has a singular manager, in an academic unit the IT staff serve some arbitrarily large number of largely autonomous faculty. Each of whom may run their own lab entirely shaped in make up and architecture by the needs of their specific research program, and the expertise of graduate students working in the lab. IT is highly idiosyncratic in many research environments.
The assumption giving rise to discussions of centralization is so broadly believed, it’s rarely challenged. That is, “clearly centralizing will allow us to reduce redundancy, more efficiently manage staff, and thus save big bucks.” Like most assumptions, this is simultaneously true and false, depending on how you choose to look at it. I recall discovering one small unit on campus whose sole IT employee had deployed an Active Directory (AD) and an entire Exchange infrastructure, simply to provide email accounts to 12 staff members. Clearly an opportunity for consolidation by central IT whose AD had a quarter of a million accounts. From the unit’s perspective, (a unit which didn’t maintain enough control over their IT spend to understand what this was costing), they got two things from this, both vanity email addresses and direct support immediately upon request. No need to talk to someone who didn’t know who there were, by calling the help desk. I suppose I should point out the obvious: the central service received considerable attention from the cybersecurity team, thus offered far more robust security measures.
Central IT shops view these and similar scenarios almost in monochrome. We provide service X to vast numbers of people, with reasonable reliability, and most of our services are free2. Centrally provided services have a strong homogenizing effect on how customers are viewed. No one’s a researcher, or teaching faculty, or even a staff member, rather they’re merely users with accounts. A vast horizon of users at such a distance that individuals melt away.
For the distributed IT staff, the situation is far more colorful. They operate at one-degree-of-separation from the departmental users, even within larger academic units. Rarely does anyone call a help desk, instead they call Bob or Alice and expect an office visit. Many of the services they offer that are “redundant” to central services (file storage being the paradigmatic example) were created in direct response to faculty needs or demands. Typically with far greater constraints on cost than any specific performance requirement. Once a service infrastructure is established, word of mouth results in quick growth within the unit.
Thus we return to the clash of worldviews. One group views a service as responding directly to faculty needs, provided at a lower cost, and with responsive service. Everything you need to earn a gold star. The other views it as redundant, idiosyncratic in technical design and product selection, and undermining the equivalent offering provided centrally. As I said earlier, control, ego, and status.
Confounding the situation is often a distinct lack of respect between the two groups. Central IT staff tend to see the overly tailored solutions adopted by distributed IT as tinkering and hobbyist in nature. Shortcuts that diminish enterprise rigor. The distributed IT staff see the central services as unresponsive to actual user needs, slow to change, developed without user input, and typically “enterprise rigor” translates to costly and professionally arrogant. At best the central shop is viewed as necessary and the domain of commodity IT, while distributed IT can most charitably called “supplemental IT”.
Nor do the distributed IT staff necessarily understand or appreciate the management and governance overhead burden of central services. In my own case, the governance of campus cybersecurity included the CFO, audit, compliance, academic affairs, the faculty senate, and of course the internal IT management, all at a senior level before I could even consider departmental demands. One tries to maintain radical transparency, but with so many political dimensions at play, that can be quite difficult.
The wise CIO when being encouraged to absorb academic IT will be cautious if not outright resistant. Eliminating these redundant academic services is wrought with challenges. Research programs will have infrastructure that needs rejiggering to work with central services, if that’s even possible. The staff that have built these services will be demoralized to see their heroics treated as naïve mistakes. Enterprise solutions, which are typically more costly to run, will need to be expanded and any cost savings in the unit are probably not capturable. Units that have underinvested in IT staff represent a kind of deferred maintenance cost the central shop will have to absorb by hiring enough supplemental staff. Of course, the organizational damage to users is significant as those one-degree-of-separation relationships are broken.
Perhaps part of the answer to the central vs. distributed debate is to ask “what best serves the mission of the institution” rather than “how to cut costs”. Even if the need to cut costs is inevitable, without asking the former, we lack the ability to perform the latter with the least damage. In an era where so many of the services provided centrally are commodity cloud services, we may be better served by asking “how can we refactor central IT to minimize its scope and footprint, and expand the local support within academic units?”. I say that knowing heads will explode, but surely it’s worth exploring. I used to argue that there was a continuum between buttoned down enterprise rigor at one end of the curve, and agile creativity and responsiveness at the other. The goal should really be for central shops to move themselves further to the right, and the academic shops more to the left. Fundamentally I don’t believe they’re incompatible (mostly - there is a kind of overhead associated with true enterprise rigor that’s unavoidable.)
It may well be that the reason the center vs. distributed debate won’t die is that we’ve set up a competition based on a false premise. It’s less a question of what sort of management structure is best, or at least cheapest, than it is that these each represent different solutions to different problems. In one we ask “if we need to minimize risk, maximize robustness, predictability, and stability (i.e., slow change) how would we organize ourselves?” For the other we ask “if we can accept a lot of risk, and want to minimize costs locally, yet be flexible to meet a rather extraordinary range of demands, how would we then organize ourselves?” On one hand we’re saying what kind of BBQ should I cook for the meat eaters, while the other is what kind of vegetable should I serve the vegetarians. Choosing either for both makes no sense.
The optimistic reader is probably waiting for a section of recommendations. Is efficiency gained by consolidating distributed IT within central IT? Can costs be lowered by doing so? How should a CIO respond to the pressure to address the expansive distributed IT community and bring to it the kind of organizational governance and controls central IT lives with?
As I have argued in any number of previous posts, organizations are best understood as collections of people and the impulses that drive them. While it is tiresome to acknowledge, the common belief that “we’re special and different” that every university brings to every situation is more true than not. However that specialness reflects the unique personalities that in toto make up the organizational culture. Strictly speaking, as structures on an orgchart, informed by mission, universities are shockingly more similar than they want to admit.
As such, all I can recommend is a program that increases the permeability of both central and distributed IT. I’ve seen too many attempts, typically half-hearted, where an “IT Planning Council” is created, managed centrally (with a distributed co-chair, of course) but with no actual influence over central projects, nor distributed projects. Alternatively, creating independent governing committees for distributed IT run smack into the reality that their unit heads and unit customers drive activity and control budgets. In every case I’ve seen, both end up as coffee klatches for cynical complaining over slack during ineffective meetings.
What I mean by “increases the permeability” is that both groups need to directly address the challenges of control, ego, and status by simply stepping back from the collision of these two world views. Permeability works not just between central and distributed units, but also between the various distributed units. It is far more likely that locally developed services in one academic unit can be shared with others, than simply replaced with one by the centralized unit. Control and ego is challenging to overcome in these situations; consolidation will be viewed as putting at risk the opportunity for leadership. Unit-based IT staff gain authority by intermediating between faculty and services, a shared service runs the risk of removing that.
Permeability lies in creating greater mutual understanding, shared governance with genuine authority3, and systems that allow for the free exchange of talent and ideas. Permeability is achieved by openly discussing the pressures that shape and influence decisions, rather than the decision themselves. Transparency, honesty, and willingness to put mission ahead of control, ego, and status are the levers available to us. How this is achieved can require a plurality of tactics. From job shadowing and professional development, reading groups and collaborative budget development, nothing should be off the table. The truth is, cost savings can only emerge as a byproduct of a healthy culture, built on a foundation of trust and strong relationships.
Essentially I’m arguing that universities need to develop a healthier ecosystem of IT staffing, which is the precursor and enabler of optimizing for cost. This requires,
recognizing that central and distributed IT are intrinsically different and not merely variations of the same solution.
the unique dimensions of service provided by distributed IT (particularly support for research) need to be recognized, and if possible emulated where appropriate centrally. Often this requires stretching past the typically Microsoft monoculture central IT embraces, in that the open and highly adaptable world of Linux forms the foundation of research infrastructure.
administrative heads in academic units should work to mature their local IT from IT hobbyists and individual contributors, to shops - smaller IT organizations with similar expectations and oversight the central IT organization lives with4. This should include surfacing IT plans and budgets to senior campus leadership as part of the annual budgeting exercise. Campus leadership will expect to see details on sunk costs, equipment replacement, staffing analyses, and cybersecurity activities.
central units need to develop deployment and change management plans in sincere partnership with distributed IT, since they will be the last mile of support for central services. Further, without their advocacy even the finest service will be seen as disappointing. I’ve seen too many distributed IT staff quietly whisper in the ears of local faculty and staff, how poorly central IT is managing a service. This insistence on ‘us’ vs. ‘them’ as a critical token of identity may be the single most corrosive activity damaging the central vs. distributed relationship.
distributed IT leaders also need to recognize that being the “Director of IT” for a division, department, or college does not make you a peer of the institutional CIO. These are not the same job simply with differences in scope. CIOs operate in an entirely different governance and accountability structure, which has more in common with your Dean than you. Be their partner, speak their language, leverage that partnership to create opportunities for institutional service.
as I hinted at earlier, often what is tempting to view as a redundant service is in fact a supplemental one. Shadow IT does occasionally arise from ego and control, but often it addresses unmet needs or shortcomings in a central service. These need to be reviewed openly, and not merely as something to be swatted away.
As pressures on institutional budgets increase, it is inevitable that the question of what to do about distributed IT will pop to the surface like a buoy. Ultimately it’s a messy question. This seems as appropriate time as any to quote Mencken, “For every complex problem there is an answer that is clear, simple, and wrong.”
Naturally, since I’m discussing a community of hundreds or thousands of distributed IT staff, individuals and backgrounds are enormously varied. Be that as it may, it’s worth recognizing that I’ll be painting with a broad brush throughout this post.
How universities fund IT is itself shocking. Often academic units pay some sort of per head tax annually which contributes to the central IT shop’s budget. Perhaps this has been normalized to the point that no one’s surprised but me, but for activities as necessary to the institution as physical buildings and classrooms, IT funding is still handled in the most MacGyvering fashion. The complexity of what CFOs deal with has me lighting a candle for them in sorrow for their suffering.
The notion of “governance” just keeps flowing off my keyboard, but I’m really not thrilled with the idea. This is not out of a disbelief in the principle, but rather the differential of status at the institution between those who truly govern (executives and faculty) vs. those that comprise distributed IT who are generally line staff with a managerial title.
At one institution I saw a major unit consolidate a number of departmental IT shops into one larger team (~60 people) serving all of the individual units. This included naming a CIO for the college. This person became the single most effective partner for central IT and resulted in the college having an outsized influence on campus IT. I believe this is becoming more common, though at many places control, ego, and status remain hurdles to overcome.