OPINION
Broadcasting Board of Governors Information War: Long Over and Lost
The Blame Game (Power Failure Edition)
By The Federalist
As reported on BBG Watch, agency official Andre Mendes (Chief Information Officer, Chief Technology Officer) produced a memo detailing a December 9, 2015 power outage which severely crippled agency operations on that day as the agency worked for hours to restore power.
You can read the memorandum on BBG Watch.
We read through the Mendes memo and come away with some observations:
We begin at the end of the memo in which Mr. Mendes states,
“While completely outside of our control, this outage did nonetheless took take place (sic) on my watch, so please accept my personal apologies. As most of you know, our reliability record has come a long way in the last few years and an occurrence like this is not reflective of the way we operate.”
To the contrary, when you read the memo, you see that there were several points before the fact of both incidents where, in our view, catastrophic failures could have been prevented with proper oversight and other precautions: procedures that, in our view, were wholly within the agency’s control.
And as you read the memo in its entirety, you come away with the sense that Mr. Mendes tries to ascribe blame upon others in what appears to be an attempt to take the focus off him:
“A GSA contractor in the Switzer building drilling a hole between two floors with a pneumatic hammer struck the main power conduits that provide power to both the Switzer and Cohen building.”
That is certainly the fact of the matter. But the much larger issue is this:
These were main power conduits to two Federal structures. How is it that no one knew they were there before the work began? These conduits are critical infrastructure. In most normal circumstances, there are blueprints and/or schematics that should be on file not only with the General Services Administration (GSA) but also with Mr. Mendes and his office. Did these blueprints somehow go AWOL? Did anyone at BBG/IBB know that construction work was going to be done in that area and bothered to check whether it might lead to an accidental severing of an electrical power line?
Another thing that strikes us (pardon the pun)–we are reminded of a commercial heard periodically on Washington, DC radio station WTOP for the “Ms. Utility” service advising commercial contractors and homeowners to call a 1-800 number before digging.
In the context of these incidents, apparently no one thought to consult whatever documentation exists – as surely there must – regarding main power conduits in both the Cohen and Switzer buildings. Nor did the agency or GSA avail itself of available technology to determine what was between the floors before the contractor went to work.
Mr. Mendes states that GSA will do no further drilling without proper precautions. Are we then led to believe that proper precautions were not in place beforehand as part of a standard operating procedure? And where was the oversight by Mr. Mendes’ office?
The next part of this fiasco involves the agency’s back-up power system or “UPS:” (uninterruptible power supply, not the shipping company). According to the Mendes report, “a BBG contractor…failed to set a switch deep in the system’s operating menu to the ON position. Although the main screen showed the generator ready and able to transfer load, this setting prevented the load from actually being transferred from UPS#2 to the generator.”
Thus, we are to believe that there was no one in the agency’s Engineering or IT staff assigned to routinely double check something “deep in the system’s operating menu” to make certain that there were no disengaged switches? At the very least, systems checks should have been performed after any work on critical controls and any construction work that might affect power supply.
Who knows – maybe the Chinese got in there and reset the switch to OFF after the contractor finished work.
Never underestimate the Chinese, Mr. Mendes.
Since Mr. Mendes’ memo is loaded with apologies, we apologize for a bit of sarcasm. However, we might not be far off the mark in reality. We already know the Russians have the capability of doing this kind of thing. And frankly, so does the US Government. Cyber warfare: so many dimensions.
“Several mission critical rooms lost power after UPS#2 was completely drained resulting in the interruption of broadcasts for VOA, MBN, RFA and OCB.”
Impressive. On paper, that represents about 80 percent of the BBG broadcast entities.
Mr. Mendes goes on to state,
“More than 20+ years ago the decision was made to hook up the TV Origination and Satellite Uplink rooms only to UPS#1 due to insufficient budget to expand UPS#2 capacity to accommodate the extra load. Personnel responsible for that implementation retired many years ago and, because this setup never failed before, no one was aware of this construction idiosyncrasy.”
Here’s a big-time gambit in the blame game: lay the fiasco on people long gone from the agency.
But in reality, that begs the issue.
That does not take into consideration what the agency has done to load up power requirements for the enormous equipment array the agency has dropped into the Cohen Building during the intervening 20+ years, to use Mr. Mendes’ time frame.
On its face, what this says is that as more equipment was deployed inside the Cohen Building, somehow there was no periodic review of how power was being supplied to this equipment or routed to and from the universal power supplies.
That’s hard – very hard – to believe.
Mr. Mendes dives into the deep end of the pool called “ASSUME.” As he says himself, “this setup never failed before.” What that means to the outside observer is that no one was paying attention to a critical piece of operational infrastructure or anticipating that the demands of the agency’s power requirements might cause a catastrophic failure.
Once again, there are supposed to be schematics and blueprints of what is being fed into and out of different systems and components. Where are they? In the absence of the human component, this kind of documentation is supposed to represent an important part of the agency’s institutional/legacy technical memory. “Supposed to.” But perhaps in the agency’s growing reputation of being (yes, here it comes)
- Dysfunctional
- Defunct
- Broken
- Rudderless
Certain rational practices seem to have been lost along the way.
“Past results are no guarantee of future performance.” That’s a common disclaimer for buying and selling stocks on the stock market. It is just as relevant when running a large, power-dependent agency such as the BBG which requires a high level of system reliability. And also, it especially applies when placing increasing demands on that system. Nothing lasts forever. Mr. Mendes seems to have lost sight of that as well.
One refrain that comes up periodically about the agency’s technical infrastructure on Mr. Mendes’ tour is a seemingly dismissive attitude in the mode of “worry about it later.”
Well, “later” appears to be now.
We also note that Mr. Mendes throws out a favorite agency explanation for just about everything it doesn’t do right:
“Insufficient budget.”
The agency’s budget has not been flat. It has been growing. Nor has it been hammered by draconian cuts. But what has happened is a monumental increase in the bureaucracy: the proverbial tail wagging the dog. That increase costs money and with this agency, that usually means at the expense of something else: in this case, perhaps that of the agency’s power infrastructure. After the agency receives its congressional authorization, how that money is actually dispersed becomes something of a shell game.
As we pointed out in the beginning of this narrative, Mr. Mendes states,
“While completely outside of our control…”
The ultimate truth of the matter is that just about everything in Mr. Mendes’ narrative was well within the agency’s control. However, it is also readily apparent that steps to preserve that control were off the agency’s radar, particularly where it counts the most: with senior agency officials.
The Federalist
April 2016