BBG Watch
When something goes wrong at the Broadcasting Board of Governors (BBG), which happens often, it’s always somebody else’s fault. It’s never the fault of current BBG executives in BBG’s International Broadcasting Bureau (IBB).
Broadcasting Board of Governors (BBG) employees, including Voice of America (VOA) broadcasters, will no doubt give their opinion in the next Federal Employee Viewpoint Survey (FEVS) conducted anonymously each year by the Office of Personnel Management (OPM). Unfortunately, hundreds of BBG’s and VOA’s exploited contract employees are not included in these OPM surveys, but regular government employees have been putting the BBG at the bottom of all federal agencies in terms of their assessment of the management’s leadership skills and employee job satisfaction.
Some of BBG contract employees have expressed their views by joining the $400 million anti-discrimination class action lawsuit filed recently against the BBG in a federal court.
###
From: IBB Notices Admin
Sent: Wednesday, April 20, 2016 7:41:06 PM
To: IBB Notices Administration
Subject: December Power Outage – From BBG CIO/CTO André Mendes
Dear colleagues:
Yesterday, I had the opportunity to share with the Governors the information contained herein, including the letters from GSA and one of our contractors. So, I am now at liberty to share it with you all as promised after the December 9th, 2015 power outage. Below you will find a report on what caused it, and the steps that we have taken to ensure that it is not repeated.
There are three main reasons for the power outage and its effect on transmissions leaving the Cohen Building :
1. Power to the Cohen Building was cut off at approximately 8:32 AM
· Reason – A GSA contractor in the Switzer building drilling a hole between two floors with a pneumatic hammer struck the main power conduits that provide power to both the Switzer and Cohen buildings.
· Resolution – GSA has assured us that no further drilling in either building will be done without proper precautions (X-rays or Field Radar to discover hidden cables etc. before drilling)
· Outcome – – GSA has issued an apology letter to John Lansing and the Board of Governors (enclosed)
2. Generator #2 failed to transfer the load from UPS#2 (Back-up Batteries) and, after 44 minutes and when completely drained, UPS#2 shut down completely into alarm status.
· Reason – During routine maintenance on December 1st, 2015, a BBG contractor (Alban Cat Power Systems) failed to set a switch deep in the system’s operating menu to the ON position. Although the main screen showed the generator ready and able to transfer load, this setting prevented the load from actually being transferred from UPS#2 to the generator.
· Resolution – The BBG has installed a manual bypass that will enable load transfer even if that switch is wrongly set during maintenance again.
· Outcome – Alban Cat Power systems has issued an apology to the BBG and has been served with a cure notice due to this incident and two other instances of poor performance. See the communication from Alban below:
“All service reports from this FY have been sent in to upper management for review. It is obvious we need to make changes to bring more consistency to you, which will be addressed immediately. I understand your position and that has been communicated clearly within our group. Please accept our deepest apologies for the inconveniences your organization has experienced. We look forward to rebuilding our relationship with BBG, and learning from these events, in order to provide more reliable service to your critical mission. Thank you for your candor and patience during these times.
V/R,
Joel Canzanella
Product Support Representative
Alban CAT Power Systems”
3. Several mission critical rooms lost power after UPS#2 was completely drained resulting in the interruption of broadcasts for VOA, MBN, RFA and OCB
· Reason – More than 20+ years ago the decision was made to hook up the TV Origination and Satellite Uplink rooms only to UPS#1 due to insufficient budget to expand UPS#2 capacity to accommodate the extra load. Personnel responsible for that implementation retired many years ago and, because this setup never failed before, no one was aware of this construction idiosyncrasy.
· Resolution – Over the last 5 years, the BBG has disposed of over 100 servers as it moved its computing infrastructure to the Cloud. As a result, we now have enough capacity in UPS#2 to accommodate the extra load associated with these rooms and have commissioned new electrical panels to be deployed in those rooms. New panels are far cheaper than the replacement and expansion of UPS#2 capacity.
· Outcome – On or about April 30th, 2016, the BBG will bring online these new panels effectively connecting these two rooms and the equipment therein to both UPS’s and therefore both generators.
Summarizing:
An unfortunate decision by a GSA contractor on 12/9/15, combined with a setup error by a BBG contractor on 12/1/15 and some budget driven construction decisions made over two decades ago, came together in a unique set of circumstances to create a lengthy power cut at the Cohen Building and resulted in several hours of outage for most of the BBG broadcasts.
It took a couple of hours after power was stabilized to bring UPS#2 up to capacity and then to restart all of the systems connected just to that bank of batteries and the associated diesel generator.
This type of outage is something that has not taken place in decades. While we do not expect any such occurrence to take place going forward, we are working on solutions that would mitigate impact if such a problem occurred again.
While completely outside of our control, this outage did nonetheless took take place on my watch, so please accept my personal apologies. As most of you know, our reliability record has come a long way in the last few years and an occurrence like this is not reflective of the way we operate.
Please do not hesitate to reach out to me, should you have any further questions on this unfortunate episode.
With best regards,
André V. Mendes
CIO/CTO
BBG
