BBG Watch Commentary
BBG
Broadcasting Board of Governors (BBG) Chief Information Officer and former Interim CEO and Director André V. Mendes may have experienced a redux of his earlier troubles at PBS when the BBG-run Voice of America (VOA) saw a series of crippling electrical and digital breakdowns during the last few days. The network control center, which uplinks all the programming, lost all power on December 7 and was not able to distribute programs for many hours. The digital storage system used by VOA broadcasters malfunctioned several times during the last few days, preventing VOA broadcasters from producing some of their video reports for TV broadcasts and distribution overseas.
DCRTV.com Washington, DC / Baltimore area Radio/TV/Media website had a news item, “VOA Experiences ‘Meltdown’,” which referred to BBG Watch reporting on the power and equipment failures at the Broadcasting Board of Governors and Voice of America headquarters near the Capitol.
In an email sent out to staff at 12:49 PM, December 31, Mr. Mendes reported that two High Powered Content Directors (HPCD) “have arrived and are being installed and configured as you read this.” Their installation is supposed to bring the digital storage system back to functionality. Many employees wondered, however, why this upgrade was not done before the latest and many previous breakdowns of the system. Today’s email did not say when and how the emergency power supply issue will be solved.
In an email sent out at 4:08 PM, December 30, Mr. Mendes apologized to Voice of America broadcasters for “the problems caused by the periodic unavailability of the MediaGrid storage system.” Mr. Mendes wrote “please accept my personal apologies for this serious event. It is very distressing and equally humbling.”
Such technical and human relations controversies are apparently not new in Mr. Mendes’ IT career.
In October 2006, Broadcasting & Cable reported that Mr. Mendes left his PBS position as Chief Technology Integration Officer when PBS stations largely rejected the new automated system, called ACE, which they said was too expensive and didn’t live up to its billing. Mr. Mendes championed the ACE system, according to the Broadcasting & Cable report.
PBS later admitted that early installations experienced significant technical problems. Some station engineers said ACE wasn’t practical and duplicated functionality provided by existing servers. Broadcasting & Cable article also reported that according to sources, “the aversion to ACE had much to do with Mendes, who rubbed some veteran engineers the wrong way with his strong pitch for IT-based workflows.” “’Engineers don’t like having the boat rocked’, says one source close to the ACE project,” the article states, “’and I think Andre pissed people off’” “For his part, Mendes says he got along well with station engineers but imagines that there are ‘a wide range of opinions’ about him through the PBS community,” Broadcasting & Cable reported in 2006 “’Inherently’, he [Mr. Mendes] notes, ‘change is controversial’” the article quoted Mr. Mendes.
A few years after voluntarily leaving PBS, Mr. Mendes was hired by BBG.
The latest problems at VOA, however, are not related to any new digital changes or improvements initiated by Mr. Mendes. The digital and electrical systems that failed have been in place for several years and have experienced repeated failures under his watch to the great frustration of VOA broadcasters.
However, when BBG Chairman Jeff Shell confirmed in January 2014 that Mr. Mendes will serve as Director of Global Operations, he said that Mr. Mendes and two other members of the interim management team “leverage the strengths of three exceptional leaders and set the stage for important agency reforms.” The so-called troika was in charge until John F. Lansing, former successful cable and entertainment TV executive and president of Scripps Networks, but without any substantive experience in government operations and international information outreach, was appointed permanent BBG CEO and Director last August and assumed his duties last September.
It was interesting to see that in a recent radio interview, Mr. Lansing used the “change is controversial” theme in trying to explain low employee morale ranking for the agency in the Office of Personnel Management (OPM) 2015 Federal Employee Viewpoint Survey (FEVS) and in earlier such government-wide surveys. Mr. Lansing was not in charge of the agency when the latest 2015 survey was conducted. We were wondering whether he might have been given the “change is controversial” excuse narrative by Mr. Mendes or perhaps some of the other BBG executives who are viewed by employees as being primarily responsible for the dismal morale and blamed for lacking leadership and innovation skills. To the astonishment and frustration of many rank-and-file Voice of America broadcasters, Mr. Lansing charged two longtime IBB executives, Jeffrey N. Trimble and Marie Skiba Lennon, with leading administrative reform at the BBG.
The vast majority of VOA broadcasters welcomes technological change and is frustrated by the inability of their leaders to deliver it and make it work, sources told BBG Watch. Mr. Mendes made an all-out effort to get more employees to participate in the 2015 FEVS survey. But while the participation rate was much higher, probably due to his promise of time-off for employees who filled out the FEVS questionaries, there were no significant gains in employee morale. BBG retained its next to the last rating in the medium-size Federal agencies category in 2015.
It is not known whether BBG executive staff did due diligence research and informed Shell about Mr. Mendes’ tenure at PBS before he was named by him to lead the interim management team in 2014. Previously Mr. Mendes had directed the BBG International Broadcasting Bureau (IBB) Office of Technology, Services and Innovation (TSI). In announcing Mr. Lansing’s appointment as permanent CEO and Director in August 2015, Chairman Shell Chairman Shell “thanked André Mendes for his service as BBG Interim CEO,” a BBG press release said. “On behalf of the entire Board, I would like to express our deep appreciation for the tireless work of André and his dedicated team. During this transition period, André and his colleagues have worked diligently to produce outstanding results and to spur on forward momentum at the BBG,” Jeff Shell stated.
In 2011, the VOA website was successfully hacked for several hours by the Iranian Cyber Army which managed to replace VOA content with anti-American images and slogans, some of them directed at the then Secretary of State Hillary Clinton. She later said that the BBG is “practically defunct.” She was not referring specifically to the Iranian hacking incident but rather commenting on BBG’s overall failure to succeed in the “information war” against Russia and violent extremists — a broader accusation which would have applied more to former BBG members, of which she was one, and to other BBG executives more senior than Mendes, some of whom still hold key positions at the agency.
A BBG January 2014 press release said that Mr. Mendes “provides executive leadership in the planning, development, and operation of all agency engineering and technical systems, including a world-wide satellite and transmitting station network, as well as planning for the use of new technological improvements and efficiencies.”
“Before joining the agency, Mendes served as senior vice president, strategic planning and global CIO for Special Olympics International. Previously, he served as Chief Technology Integration Officer for the Public Broadcasting Service [PBS], where he was responsible for both Information Technology and Broadcast Engineering during a $1.8 billion transition to digital television,” the January 22, 2014 BBG press release said.
We still think Mr. Mendes should be applauded for apologizing for the latest digital meltdown at VOA while other BBG executives who are much more to blame for the agency being “practically defunct” (Hillary Clinton’s words) as usual ran for cover and remained silent.
While the apology does not get Mr. Mendes off the hook in the eyes of many employees, one high-level BBG manager commented to BBG Watch on the condition of anonymity, that Mr. Mendes, a relative newcomer to the federal agency in charge of U.S. international media outreach, behaved like “a grown up in a sea of brats; the others were silent.” A former VOA manager said that as someone who grew up in Portugal under a dictatorship Mr. Mendes may have a better appreciation for the mission of VOA’s foreign language broadcasters, but he was not in a position to judge whether this alone made Mr. Mendes a better manager.
Speaking on the condition of anonymity, a former BBG official said that while Mr. Mendes is prone to hyperbole, he did help to solve or ameliorate some, but not all, of the bureaucratic problems created earlier by longtime BBG executives who as usual are now silent and never do anything or say anything when serious crises develop. One of them seems to have been linked one way or another to nearly every serious crisis the agency has seen in recent years, the former official added, and has never been held accountable. It is not clear, however, whether Mr. Mendes initiated some of the positive changes himself while he was interim CEO, or whether he was carrying out orders from Chairman Shell and the BBG board. During Mr. Mendes’ tenure as interim CEO, several Radio and TV Marti broadcasters who had been illegally dismissed by BBG several years ago, were rehired. Some praised Mr. Mendes for his role in this process.
Many other current and former BBG and VOA employees, however, were much less generous in their comments about his overall performance.
Current VOA journalists and broadcasters have been complaining about the digital systems and digital outreach for years and have been utterly frustrated by the latest meltdown, yet even in his apology, Mr. Mendes said “…we will restore the MediaGrid to the stellar availability that has been its hallmark over the last 5 years.”
This reference to “stellar availability” is likely to produce howls of laughter and derision from Voice of America broadcasters who have struggled with the system for years, one source said.
“OMG!!! Down again!!! This is amateur hour. We need to send out a red alert and reach out to real broadcast professionals. The in house staff are clearly incapable of resolving this issue. The Board should step in and bring in their experts. We need help and we need it now!!!,” a VOA broadcaster wrote on December 30.
“This isn’t hyperbole: this is the complete and total failure of the agency. The fact that it is, apparently, self-inflicted makes it all the worse. We are doing what Putin would love to be able to do: the Voice of America is silenced,” a VOA journalist said on the same day.
“It looks like the fixes aren’t working. This is a full-fledged meltdown of essential systems. Is it a bug, an attack, poor system maintenance? Does the Board know what is going on? Congress? The White House?,” was another comment from a VOA broadcaster.
“Can this really be happening? Are we actually failing at the most basic of all broadcast tasks: staying on the air???,” another VOA broadcaster said.
“It would be nice if we would get notified when the grid goes down, not only when its back up for 15 min,” was another comment from a VOA employee.
“Take any private/commercial network — MSNBC, CNN, FOX, you name it — and they would be off the air, and people fired, if they allowed these kinds of continuous failures. But — it’s good enough for government work!,” a former senior broadcaster said.
An employee writing under the pseudonym “Dalet Victim” [Dalet is the name of one of digital systems at VOA] commented: “What’s interesting about all of this is that it comes at the end of 2015, going into 2016. As anyone who has had to work under the Dalet dictatorship at VOA will tell you, this system has been an absolute nightmare. By the way — it was jettisoned by NPR [may have meant PBS] quite a few years ago. Someone should conduct an investigation into the amount of money and time that has been wasted, including lost programming and shredded employee hours, because of Dalet problems. Yeah right — that will happen…”
VOA and other BBG employees, as well as outside observers, realize that Mr. Mendes is just one of many BBG top executives responsible for the agency’s meltdown and not necessarily the chief culprit by any means.
“Anyone want to bet? A year from today, we will have the same people milling around, acting important and doing little but praise themselves. Of course with a frequent jaunt about the globe to visit friends and relatives … er, I mean to attend Official Functions,” one observer commented.
A Voice of America employee wrote: “But senior VOA leaders don’t care about this [equipment failures]….know why? Because none of them use Dalet. None of them even have Dalet loaded onto their desktops. They are unfamiliar with the most basic procedures for television, i.e. finding and exporting news video. (They think it comes from the magic TV fairies.) That is why everything is a mess, forcing everyone to work off servers. That is why we spend thousands on personal hard drives for archiving. Our leadership can’t solve our problems because they don’t even understand the language we use to describe them. They think there is no difference between managing a toaster factory and managing VOA — it’s just about bossing people around!”
In responding to Mr. Mendes’ latest URGENT UPDATE – MediaGrid Remediation, one former agency employee observed, “Here’s the question — why is this being accomplished only now????? I’m sure Mr. Mendes has an answer to that.”
Many BBG journalists, former BBG journalists and executives, former BBG members and other experts have told BBG Watch that Mr. Shell and Mr. Lansing are wrong in claiming that problems are being solved at the agency and in opposing the bipartisan H.R. 2323 bill, the United States International Communications Reform Act, to restructure and reform the Broadcasting Board of Governors. In May 2015, Mr. Shell took issue with the Office of Inspector General’s “Ineffective Leadership” assessment of the Broadcasting Board of Governors.
The latest technical and management failures, including the $400 million class action discrimination suit filed by contract employees and the management crisis at Radio Free Europe / Radio Liberty (RFE/RL), prove beyond any doubt that the BBG lingers in a crisis mode. Its failed executives, not counting Mr. Lansing, cannot save the agency and should never be given more control over BBG’s individual entities. Mr. Lansing alone cannot save the agency. If Mr. Shell and Mr. Lansing know what is good for U.S. international media and for the country, and their own excellent private sector reputation, they should drop their opposition to H.R. 2323 and embrace the reform bill, BBG Watch was told by many distinguished Americans.
From: IBB Notices Admin
Sent: Thursday, December 31, 2015 12:49 PM
To: IBB Notices Administration
Subject: URGENT UPDATE – MediaGrid Remediation – Please read carefully
Colleagues:
The Content Directors have arrived and are being installed and configured as you read this.
We expect the upgrade process to begin at 1:30 and at this point we do not know exact duration but expect it to be substantially shorter than the firmware upgrade we undertook overnight last Monday.
During the process, the Dalet+ application will stay up but with limited functionality since the MediaGrid will have to be unavailable to end users.
During the maintenance window, the following services will be unavailable in both the full client and Webspace:· No new incoming video from any source (including watch folders)
· No editing or creation of video in DaletPlus
· No export of video from DaletPlus (Export To Portable Media)
· No fulfillment of Master Control placeholders
· No upload of video to Spectrum for control room playout
· No XTend Import/Export for FCP and Premiere
All other services such as newswriting functionality and news wires will remain available. Rundowns and teleprompter will also be available, although no new video can be uploaded to the Spectrum once the maintenance begins.
We will continue to keep you updated as the upgrade progresses.
Sincerely,
André V. Mendes
CIO/CTO
BBG
###
From: IBB Notices Admin
Sent: Wednesday, December 30, 2015 4:08 PM
To: IBB Notices Administration
Subject: MediaGrid Storage
Dear colleagues:
The last few days have been extremely challenging. I know that you are very frustrated by the problems caused by the periodic unavailability of the MediaGrid storage system.
I want you to know that the TSI team and our vendor are working around the clock to resolve the problem and hope that offers you some solace.
At this point, we know what the final resolution is and it involves the installation of two High Powered Content Directors (HPCD) that will be able to keep up with the fast rising amount of video being produced in this building.
Currently, the MediaGrid storage unit is host to over 2 million files and that numbers grows every day.
It is my expectation that we will receive one of the HPCDs via overnight shipping tomorrow morning and that in a few hours we can install it, configure it and proceed to index the content from our system. IF everything goes well, by early afternoon, we will restore the MediaGrid to the stellar availability that has been its hallmark over the last 5 years.
Should that not occur, we are working on a Plan B to substantially cut down on the content the system is hosted so that we can begin operating more reliably until the units arrive and are deployed.
It would not be productive, at this time, to go over the reasons why we find ourselves in this situation. I personally promise you a full report on the series of circumstances that led to this problem so that you fully understand the situation.
Meanwhile, please accept my personal apologies for this serious event. It is very distressing and equally humbling.
Sincerely,
André V. Mendes
CIO/CTO
BBG
Comments are closed.