BBG Watch Commentary
The latest digital equipment failure at the Voice of America (VOA) — one of many technical failures at VOA during the last few hours and days — appears to be over (for now), according to an email forwarded to BBG Watch by Broadcasting Board of Governors (BBG) employees. BBG is responsible for management and technical support for VOA through its International Broadcasting Bureau (IBB) and its Technology, Services, and Innovation office. IBB, TSI and other program and technical support, run by the same BBG/IBB top executives, have been getting about 34% of the BBG budget in recent years without producing any programs. In FY2016 budget request, IBB’s budget is: $61 million; TSI’s budget is: $178.6 million.
The bureaucratic bloat as the agency meltdown continues is beyond incredible. One of the emails forwarded to us was addressed to:
“Technology Support Services Directorate”; “TSS Managers”; “Information Technology and CIO Managers”
Another internal email was addressed to: “VOA Broadcast Operations Managers; Digital Support Team SysMonitors; VOA TV Studio Directors; Terry Balazs; Andre Mendes”
There is no shortage of directors and managers at the Broadcasting Board of Governors and the International Broadcasting Bureau. They have been incapable of solving any of these problems for years.
This is probably a shock to new BBG CEO John Lansing, a successful former private sector cable and entertainment TV executive but without any previous government service experience. Problems are getting worse.
As reported in our previous post, “VOA broadcasters and other BBG rank-and-file employees who contacted us reacted with disbelief when new BBG CEO John Lansing appointed two longtime executives with BBG’s International Broadcasting Bureau (IBB) to lead the administrative reform effort.”
Did he have any other choice?
It is not easy, but not impossible, to fire government-employed SES (Senior Executive Service) managers. IBB executives, however, found ways to eliminate programs and dismiss government employees by replacing them with poorly-paid employee contractors who also lack other employment benefits and rights, while the IBB bureaucracy kept expanding its ranks. The Office of Inspector General found these practices to be in violation of government rules and regulations, but no heads have rolled. BBG/IBB executives have been praised and promoted. Meanwhile, a $400 million class action discrimination suit has been filed against the BBG by contract employees.
According to a Concerned BBG Employee, “The IBB Office of Engineering and Office of Computing Systems, now jokingly called Technology, Services, and Innovation, has been systematically drained of top level talent as the current management practices cronyism at its best.”
The latest comment from Concerned BBG Employee:
“One noticeable thing lacking in the emails referenced in the article is the fact that no level of management, be it BBG, IBB, TSI, or VOA has said a word. Do they not care or this considered an acceptable level of performance within the agency?”
–A Concerned BBG Employee, Submitted on 2015/12/30 at 2:19 pm
UPDATE!: One of BBG’s top executives, André V. Mendes, has issued an apology at 4:08 PM. BBG Watch will have a report. Other top BBG and IBB executives have not been heard from. They have never been publicly held accountable or have taken publicly any responsibility for numerous earlier crises at BBG, VOA or Radio Free Europe / Radio Liberty.
From: Ralph Cassano
Sent: Wednesday, December 30, 2015 2:15 PM
To: VOA MEDIA TEAMS; VOA Newsroom
Cc: Technology Support Services Directorate; TSS Managers; Information Technology and CIO Managers; Terry Balazs; Andre Mendes
Subject: UPDATE : DaletPlus MediaGrid Available
At this time, the Digital Management Division has confirmed that the DaletPlus system has returned to full functionality. Our system administrators have restored the Omneon MediaGrid and all systems appear normal. Please resume regular operations.
We STRONGLY suggest that you utilize the BLACKMAGIC/emergency control room playout workflow throughout the rest of the day as a precaution.
Ralph Cassano
IT Specialist
Digital Media Support Division (T/SD)
Technology Services Support Directorate
Voice of America Digital Failure, saga continues, frustrations mount
December 30, 2015, 1:35 PM
BBG Watch Commentary
There is no end in sight to equipment and other failures at the Voice of America (VOA) as employees’ frustrations with the Broadcasting Board of Governors (BBG) agency management mount over these and other issues. VOA broadcasters and other BBG rank-and-file employees who contacted us reacted with disbelief when new BBG CEO John Lansing appointed two longtime executives with BBG’s International Broadcasting Bureau (IBB) to lead the administrative reform effort.
More frustration was expressed in comments and emails, copies of which were forwarded to BBG Watch by third-parties. These emails were not sent to BBG Watch by their authors.
A VOA journalist wrote:
“This isn’t hyperbole: this is the complete and total failure of the agency. The fact that it is, apparently, self-inflicted makes it all the worse. We are doing what Putin would love to be able to do: the Voice of America is silenced.
— A Voice of America Journalist, December 30, 2015
“It looks like the fixes aren’t working.
This is a full-fledged meltdown of essential systems.
Is it a bug, an attack, poor system maintenance?
Does the Board know what is going on? Congress? The White House?”
— A Voice of America Broadcaster, December 30, 2015
“Can this really be happening? Are we actually failing at the most basic of all broadcast tasks: staying on the air???”
— A Voice of America Broadcaster, December 30, 2015
“It would be nice if we would get notified when the grid goes down, not only when its back up for 15 min.”
— A Voice of America Broadcaster, Wednesday, December 30, 2015 12:07 PM
“Yes, the grid went down at 11:43 a.m. so, yes, the video is going to be red because it did not have a chance to copy to the Media Grid.”
— Broadcasting Board of Governors / International Broadcasting Bureau Consultant, Wednesday, December 30, 2015 12:05 PM
“What’s interesting about all of this is that it comes at the end of
2015, going into 2016. As anyone who has had to work under the Dalet
dictatorship at VOA will tell you, this system has been an absolute
nightmare. By the way — it was jettisoned by NPR quite a few years
ago. Someone should conduct an investigation into the amount of money
and time that has been wasted, including lost programming and shredded
employee hours, because of Dalet problems. Yeah right — that will happen…”
— Dalet Victim, Submitted on 2015/12/29 at 6:09 pm
“The IBB Office of Engineering and Office of Computing Systems, now jokingly called Technology, Services, and Innovation, has been systematically drained of top level talent as the current management practices cronyism at its best.”
— Concerned BBG Employee, Submitted on 2015/12/29 at 3:41 pm
“But senior VOA leaders don’t care about this….know why? Because none of them use Dalet. None of them even have Dalet loaded onto their desktops. They are unfamiliar with the most basic procedures for television, i.e. finding and exporting news video. (They think it comes from the magic TV fairies.) That is why everything is a mess, forcing everyone to work off servers. That is why we spend thousands on personal hard drives for archiving. Our leadership can’t solve our problems because they don’t even understand the language we use to describe them. They think there is no difference between managing a toaster factory and managing VOA — it’s just about bossing people around!”
— Anonymous Broadcaster, Submitted on 2015/12/29 at 7:43 am
“I’m a contractor at VOA. How do I join this class action law suit?”
— A., Submitted on 2015/12/28 at 6:57 pm
“Take any private/commercial network — MSNBC, CNN, FOX, you name it —
and they would be off the air, and people fired, if they allowed these
kinds of continuous failures. But — it’s good enough for government
work!”
— Former Senior Broadcaster, Submitted on 2015/12/28 at 2:40 pm
“Anyone want to bet? A year from today, we will have the same people milling around, acting important and doing little but praise themselves. Of course with a frequent jaunt about the globe to visit friends and relatives … er, I mean to attend Official Functions.
The U.S. Congress could restore U.S. International Broadcasting to its former glory, but that isn’t likely. The American voters just aren’t that informed or interested — and if it’s not likely to gather votes, politicians don’t care a lot.
Also, any real remedial effort would involve severe and sweeping changes, perhaps the removal of all employees and officers at or about the equivalent of GS-14. It’s like removing the mold from the cheddar in your fridge; you can’t just scrape the top and think that you’ve solved the problem.
Dear Hon. Congress people: Please give the WORLD a Christmas gift by actually fixing BBG and its spawn. (Until that happens, Dear Incumbents: Bah humbug. Begone!)”
— C.A., Submitted on 2015/12/22 at 11:28 pm
###
From: Renee Turner
Sent: Wednesday, December 30, 2015 12:02 PM
To: VOA MEDIA TEAMS; Digital Support Team
Subject: RE: TV PRODUCERS–MUST READ!!!
If the embedded video did not fully render on the Media Grid before the Media Grid went down the video will not play in the control room. That’s why you are seeing red on your rundown.
If the video was embedded earlier this morning when the media grid was operational, and the video fully rendered, then the video will playback in the control room. This is why we are asking all services to get the video you need for your show into your rundowns as soon as possible.
Right now, the media grid is down again. So, if you trying to put video into your rundown now for and upcoming show, most likely it will not play because it cannot transfer to the playback server (media grid).
The Media Grid is unstable. This is why you must also load any video you want to play during your show onto the Black Magic backup drive.
Also, when playing out from Black Magic, you should transfer the vide to the computer and not play the video from the Black Magic Drive.
Renee Turner
TV Consultant/Trainer
Dalet Launch Team/ VOA TV
###
From: Ralph Cassano
Sent: Wednesday, December 30, 2015 12:22 PM
To: VOA MEDIA TEAMS; VOA Newsroom
Cc: Technology Support Services Directorate; TSS Managers; Information Technology and CIO Managers; Terry Balazs; Andre Mendes
Subject: DaletPlus MediaGrid Unavailable
Starting at about 11:45am this morning, we again started experiencing a global issue with the Omneon Media Grid. This problem is similar to the one we have encountered in the past week. All language services must be prepared with a backup plan for video playback. The system administrators are currently looking into the issue to see what the cause may be and how to quickly resolve the issue.
The Dalet application remains available but no new DaletPlus media will be available in the control rooms until the MediaGrid is back online. Please use the emergency pre-production workflow to prepare for broadcast production.
We will keep you informed of any changes or updates.
Ralph Cassano
IT Specialist
Digital Media Support Division (T/SD)
Technology Services Support Directorate
###
From: Ross McCallum
Sent: Wednesday, December 30, 2015 11:16 AM
To: VOA MEDIA TEAMS
Subject: TV PRODUCERS–MUST READ!!!
Reminder to all Dalet TV program producers: Continue to embed video files in Dalet rundowns for transfer to the playout server even when the Media Grid is unstable/not fully operating…
Video that loads to the playout server WILL PLAY ON-AIR even if Media Grid is down….
So, it is best to quickly review newly edited video for quality upon arrival in your Dalet Production folder, and then embed it in Dalet immediately to increase chances that video will load for playout…..
Ross MacCallum
Consultant/TV Production & Training
Dalet Team
VOA-TV
###
From: Renee Turner
Sent: Wednesday, December 30, 2015 12:02 PM
To: Mehrnoush Karimian; Ross McCallum; VOA MEDIA TEAMS; Renee Stevens; Digital Support Team
Subject: RE: TV PRODUCERS–MUST READ!!!
If the embedded video did not fully render on the Media Grid before the Media Grid went down the video will not play in the control room. That’s why you are seeing red on your rundown.
If the video was embedded earlier this morning when the media grid was operational, and the video fully rendered, then the video will playback in the control room. This is why we are asking all services to get the video you need for your show into your rundowns as soon as possible.
Right now, the media grid is down again. So, if you trying to put video into your rundown now for and upcoming show, most likely it will not play because it cannot transfer to the playback server (media grid).
The Media Grid is unstable. This is why you must also load any video you want to play during your show onto the Black Magic backup drive.
Also, when playing out from Black Magic, you should transfer the vide to the computer and not play the video from the Black Magic Drive.
Renee Turner
TV Consultant/Trainer
Dalet Launch Team/ VOA TV
###
From: Mehrnoush Karimian
Sent: Wednesday, December 30, 2015 11:55 AM
To: Ross McCallum; VOA MEDIA TEAMS; Renee Stevens; Digital Support Team
Subject: RE: TV PRODUCERS–MUST READ!!!
Just as a heads up to everyone: we’ve been having some issues with videos that are showing as embedded in the rundown. Despite the fact that they show up as fully embedded in to the rundown, we are unable to play them in the control room. The video shows up red on the playback system of the director and technical director and does not play at all.
If you could please advise on this Ross, I would really appreciate it.
Thank you!
Best wishes,
Mehrnoush Karimian
###
From: Ross McCallum
Sent: Wednesday, December 30, 2015 11:16 AM
To: VOA MEDIA TEAMS
Subject: TV PRODUCERS–MUST READ!!!
Reminder to all Dalet TV program producers: Continue to embed video files in Dalet rundowns for transfer to the playout server even when the Media Grid is unstable/not fully operating…
Video that loads to the playout server WILL PLAY ON-AIR even if Media Grid is down….
So, it is best to quickly review newly edited video for quality upon arrival in your Dalet Production folder, and then embed it in Dalet immediately to increase chances that video will load for playout…..
Ross MacCallum
Consultant/TV Production & Training
Dalet Team
VOA-TV
###
From: Steve McClain
Sent: Wednesday, December 30, 2015 10:30 AM
To: VOA MEDIA TEAMS; VOA Newsroom; IT Digital Management Division
Cc: Technology Support Services Directorate; TSS Managers; Information Technology and CIO Managers; VOA Broadcast Operations Managers; Digital Support Team SysMonitors; VOA TV Studio Directors; Terry Balazs; Andre Mendes
Subject: UPDATE DaletPlus Omneon Media Grid Is Available
ALL:
At this time, Digital Management Division has confirmed that the DaletPlus System has returned to full functionality. Our System Administrators have restored the Omneon Media Grid and all systems appear normal. Please resume regular operations.
We STRONGLY suggest that you utilize the BLACK MAGIC backup plan throughout the rest of the day as a precaution.
We apologize to our Broadcast Community for this inconvenience and appreciate your support in this matter.
*NOTE*
The DaletPlus Trainers have put together some tools to assist the Broadcast Community in the event of any TV production emergency when the actual DaletPlus application is not operational. Please find those tools in the following location:
…
If you have any questions on these procedures please call the DP Trainers or email them at …
Steve M. McClain
IT Specialist
Digital Media Support Division
###
From: Steve McClain
Sent: Wednesday, December 30, 2015 07:55 AM
To: VOA MEDIA TEAMS; VOA Newsroom; IT Digital Management Division
Cc: Technology Support Services Directorate; TSS Managers; Information Technology and CIO Managers; VOA Broadcast Operations Managers; Digital Support Team SysMonitors; VOA TV Studio Directors; Terry Balazs; Andre Mendes
Subject: DaletPlus Omneon Media Grid Is Unavailable
Starting at about 7:45am this morning, we started experiencing a house-wide issue again with the Omneon Media Grid. This problem is similar to the one we have encountered earlier in the week. All language services must be prepared with a backup plan for video playback. The system administrators are currently looking into the issue to see what the cause may be and how to quickly resolve the issue.
The Dalet application remains available but no media operations in the Control Rooms will be possible until the Media Grid is back online. Please use the emergency pre-production workflow to prepare for broadcast production.
We will keep you informed of any changes or updates. We apologize to our Broadcast Community for this inconvenience and appreciate your support in this matter.
*NOTE*
The DaletPlus Trainers have put together some tools to assist the Broadcast Community in the event of any TV production emergency when the actual DaletPlus application is not operational. Please find those tools in the following location:
…
If you have any questions on these procedures please call the DP Trainers or email them at …
Steve M. McClain
IT Specialist
Digital Media Support Division