BBG Watch Commentary
Voice of America (VOA) broadcasters are completely fed up with repeated technical mishaps at their taxpayer-funded Broadcasting Board of Governors (BBG) federal agency headquarters, which house radio and TV broadcasting studios and digital media equipment. They keep sending us announcements they receive about new problems. They also point out that these mishaps are nothing new. Power and equipment outages have been happening regularly over many years without effective plans, precautions or actions being implemented by senior management of the BBG’s bloated bureaucracy and technical support division, the International Broadcasting Bureau (IBB).
Below is the latest email forwarded to us. Below you will also find copies other email sent recently and over the years. Nothing much has changed, while IBB staff had grown by 37% in a recent seven year period.
Voice of America broadcasters hope that new BBG CEO John Lansing will see these emails if they are posted by BBG Watch.
“Those are just a few of them [e-mails]. As you recall, the problems with Dalet go back to at least 2010, actually to 2009 and before,” a Voice of America reporter wrote.
From: IBB Notices Admin
Sent: Wednesday, December 23, 2015 6:16 PM
To: IBB Notices Administration
Subject: MediaGrid Maintenance
All:
Over the last week, we have experienced two incidents with our Video Storage Appliance (Media Grid) that caused its unavailability. Working with the vendor we have identified the root cause of the problem and the steps necessary to prevent a re-occurrence.
Due to availability of specific vendor personnel during this holiday week, we will be performing the maintenance this coming Monday the 28th. Maintenance is expected to begin around 6 PM and last up to 6 hours.
Until the maintenance is completed, there is a possibility that we might experience another incident, so it is essential that all language services continue to be prepared to utilize our control room video play-out backup system (BlackMagic). Attached is the Black Magic Video Playout Instructions.
We apologize to our Broadcast Community for this inconvenience and appreciate your support in this matter as we address this maintenance issue.
As usual, this scheduled downtime may be delayed or aborted at the last minute in the event of major breaking news. If you have any questions regarding this notice, please contact the Technical Support Services Directorate’s Computer Support Center … .
Larry G. Rubendall
Deputy for Information Technology
Larry G. Rubendall
Deputy for Information Technology
U.S. International Broadcasting Bureau
330 Independence Avenue, SW
Workstation 4061A
Washington, DC 20237
ALSO SEE: Power outage disrupts Voice of America broadcasts, BBG Watch, December 7, 2015
ALSO SEE: Voice of America power and broadcasting outage UPDATE, BBG Watch, December 8, 2015
ALSO SEE: FedSmith, FEDweek list Broadcasting Board of Governors as one of the worst federal agencies for employees, BBG Watch, December 10, 2015
ALSO SEE: BBG employees ‘crippled’ by technical breakdowns ‘lose confidence’ in leaders, BBG Watch, December 22, 2015
2015
###
From: IBB Notices Admin
Sent: Monday, December 7, 2015 1:28 PM
To: IBB Notices Administration
Subject: Power Outage Update
Colleagues:
Earlier this morning a GSA contractor working on a construction project in the Switzer building inadvertently hit a power line.
Unfortunately, this incident brought down power in both the Switzer and Cohen buildings completely disrupting almost all infrastructure systems.
At this point, we are working diligently with all of our vendors to ensure that we can restore full operations as soon as possible.
Portions of the building are up to full operations but many of our broadcast transmission systems are still being hampered by this incident.
We will notify you immediately once we have restored full functionality.
We deeply regret the inconvenience to all of our colleagues and programs.
Sincerely,
André V. Mendes
###
From: IBB Notices Admin
Sent: Monday, December 7, 2015 6:11 PM
To: IBB Notices Administration
Subject: Power Outage Update
All,
The power issue has been resolved and we are working on bringing all systems back online. In the meantime, please note the following:
• Radio: All radio systems are up and all radio shows should continue as normally scheduled.
• Telephones: All phones are back up and running
• Internet/Dalet: All work stations should have access to the network and Dalet. If you have any difficulty accessing the internet, Dalet, the network drives, etc., please contact the helpdesk at …
We currently only have one area still experiencing difficulties:
• TV: We are still working to bring the TV systems back on line. Most major systems are already operational and we anticipate that all morning shows should proceed as planned, but please make sure that you back up your files for playout for the morning shows.
Thank you for your patience and creativity today as we dealt with these challenges.
Kelu Chao
###
2013
Response To Problem Saving Work In DaletPlus
Ted R. Tate
Sent: Thursday, March 28, 2013 8:23 PM
To: ALLTV OPERATIONS
Cc: VOA Newsroom; VOA Newsroom Central Desk; Information Technology Managers; Andre Mendes; John Johnson; IT Digital Management Division; Help Desk Email; Juli M. Fletcher
This afternoon some users of DaletPlus trying to save news stories were encountering a “Save as” message and an intermittent loss of the ability to save within DaletPlus. At this time the Digital Management Division believes the problem has been isolated and are testing the fix.
To be sure you have the best opportunity for normal operations within DaletPlus, if you were logged onto DaletPlus before 6:30PM, you should log out and then log back in.
We will plan to update you later as we continue to monitor the situation. If you encounter any further problems, please contact the Help Desk. Thank you for your help as we resolve this issue.
Ted R. Tate
Lead IT Specialist
Digital Management Division
###
UPD 1 on Widespread Connectivity Issues
Raymond Perez
Sent: Saturday, November 16, 2013 1:21 PM
To: ALLTV OPERATIONS; VOA TV Teams
Cc: IT Digital Management Division; Information Technology Managers; Help Desk Email; Andre Mendes; John Johnson; Juli M. Fletcher; VOA Newsroom; VOA Radio Users
Importance: High
ALL:
We are beginning to see vast improvements throughout the Agency when connecting to various applications. I will outline below what is currently up and running at this time.
DaletPlus TV – though connectivity seems to be fine when logging in to the application, there may still be some issues with Media Migration or transferring video into the Rundown to Omneon. We would continue to advise all
Users to prepare their emergency play-out measures in case it is needed. Transfers to Corktree appear to be working and FIFO’s appear to be recording correctly again.
Radio Suite HD – appears to be up and functional at this time.
Scheduall – appears to be up and functional at this time.
FootPrints – appears to be up and functional at this time.
MAC – transfers using Xtend appear to be functional at this time
SharePoint – was reported down but appears to be functional at this time.
We still strongly urge everyone to be prepared with any and all emergency plans for their shows until we can be completely sure that every system is fully operational. Thank you for your cooperation and we sincerely apologize for any inconvenience this may have caused.
Ray
Raymond Perez
Lead IT Specialist
Digital Management Division
###
From: Brian Payton
To: ALLTV OPERATIONS
To: VOA Teams
Cc: IT Digital Management Division
Cc: Information Technology Managers
Cc: Help Desk Email
Cc: Andre Mendes
Cc: John Johnson
Cc: Juli M. Fletcher
Cc: VOA Newsroom
Subject: Dalet Plus System Issues
Sent: Sep 7, 2013 6:34 AM
ALL:
At this time all users are unable to log into daletplus. Our System
Administrators are currently looking into the issue to see what the cause may
be. As a work around for our production groups, please use Black Magic (via
Final Cut) to import clips for EMERGENCY PLAYOUT. Additionally, please consult
your individual EMERGENCY SOP’s for broadcast as well.
We will keep you informed of any updates while we resolve this issue. Thank You
for you patience and we apologize for this inconvenience
Brian F. Payton
IT Specialist
Digital Management Division
###
From: Christopher Novack
To: ALLTV OPERATIONS
To: VOA Teams
Cc: IT Digital Management Division
Cc: Information Technology Managers
Cc: Help Desk Email
Cc: Andre Mendes
Cc: John Johnson
Cc: Juli M. Fletcher
Cc: VOA Newsroom
Subject: DaletPlus UPDATE: DaletPlus search issue
Sent: Aug 28, 2013 9:39 AM
ALL
While emergency maintenance was performed on the system overnight, at this time we are still seeing the corruption issue within the DaletPlus full client search index. An urgent trouble ticket has been entered with Dalet support for this issue.
However, the Webspace search index is 100% complete and may be used as an alternative.
Searching in the DaletPlus full client will continue to return results with varying degrees of accuracy. Results will tend to be more accurate with newer content, such as just-arrived news wires, but there will be gaps.
We thank you for your understanding in this matter.
Chris Novack
Operations Manager
Digital Management Division
###
From: IBB Notices Admin
Sent: Wednesday, August 14, 2013 12:56 PM
To: IBB Notices Administration
Subject: DaletPlus WebSpace upgrade tonight and possible impact on DaletPlus users
During the upgrade of the DaletPlus WebSpace system this morning, a defect was discovered that requires the installation of a small patch to the DaletPlus servers. The original version of WebSpace is unaffected and remains available at this time, and will continue to be available until we have determined that the defect is completely resolved.
This work will begin at approximately 7pm this evening. While this fix is not related to the DaletPlus full client system, the installation of the fix will require that a number of core servers be reset. These resets will be performed individually to minimize the impact to the rest of the system, however it is possible that DaletPlus full client users will briefly see a “Reconnecting…” message while work is taking place.
A notice will be sent prior to commencement of the upgrade. At that time it is strongly recommended that users save their work.
Thank you.
Chris Novack
Operations Manager, Digital Management Division
###
Sent: Sunday, July 21, 2013 5:34 PM
To:
ALLTV OPERATIONS; VOA TV Teams
Cc: IT Digital Management Division; Information Technology Managers; Help Desk Email; Andre Mendes; John Johnson; Juli M. Fletcher; VOA Newsroom
Importance: High
ALL:
At this time we have found what we believe may be the cause of the SEARCH to automatically re-index. An error message from a Wire Filter agent indicated that the agent appears to have been misconfigured and set to run continuously. This agent has now been configured correctly and to run at appropriately intervals. It is currently processing about 180,000 assets per hour and at this rate should be completed by 11 PM this evening.
Though we cannot positively rule out any data corruption from the failed Hard Disk Drive that was previously mentioned, we do know this misconfiguration would have continued to cause very serious problems. We will continue to monitor all aspects of the re-indexing and test again to ensure it is working correctly.
Again, we apologize for any inconvenience this has caused and we thank you for your patience in this matter.
Ray
###
From: Kathleen Dawson
To: VOA Newsroom
Subject: Dalet Problems…
Sent: Jun 5, 2013 9:35 AM
If your Dalet Plus is opening strangely and won’t give you access to files, the Dalet Plus techs say log off Dalet and restart it again. You may have to do it a few times.
The problem apparently isn’t with individual computers or accounts – it has to do with servers.
They’re trying to find a way to prevent the problem.
Kate
###
From: SEANMARTIN P. HANSBERRY
To: ALLTV OPERATIONS
To: VOA TV Teams
To: DP Trainers
To: IT Digital Management Division
Cc: Andre Mendes
Cc: John Johnson
Cc: Juli M. Fletcher
Cc: Michael Hunt
Cc: Christopher Novack
Subject: ES_Proxies Are Down
Sent: May 15, 2013 2:15 PM
All:
We are currently experiencing an issue with one of the file storage systems used
by DaletPlus. This is currently preventing access to Webspace proxy videos and
some full-client proxy videos.
We have temporarily redirected new Webspace proxy videos to a different storage
system, however attempting to view existing material will result in an error
message that there is no media available.
For full client users, most current material remains available. Material
ingested overnight last night may present an error message.
We are working with the Networks and Storage team to bring this storage system
back online as quickly as possible.
Sean Hansberry
SeanMartin P. Hansberry
Lead IT Specialist – DST
Digital Management Division
###
IMPORTANT – DaletPlus Restart at 100 AM
Ted R. Tate
Sent: Tuesday, April 30, 2013 10:52 PM
To:
ALLTV OPERATIONS; VOA TV Teams
Cc: VOA Newsroom; Help Desk Email; Andre Mendes; John Johnson; Juli M. Fletcher; IT Digital Management Division; Information Technology Managers
After further troubleshooting it has been determined that system administrators must perform a full reset of the DaletPlus system in order to correct stability issues resulting from an unscheduled Windows security patch deployment that occurred this past weekend. This reset will occur at 1am.
The DaletPlus system is expected to be unavailable for up to 2 hours. This includes the desktop client as well as Webspace. A house notice will be sent prior to work beginning and again when complete.
It is strongly recommended that after work is complete, you restart your computer before using the DaletPlus client.
###
From: Brian Payton
To: ALLTV OPERATIONS
Cc: Andre Mendes
Cc: John Johnson
Cc: Information Technology Managers
Cc: IT Digital Management Division
Cc: Juli M. Fletcher
Cc: VOA Newsroom
Subject: DaletPlus Wires Search issue
Sent: Dec 29, 2012 10:16 AM
ALL:
There have been reports that wire searching is not polling updated results. We have investigated these claims and although the re-indexing was completed this morning, we have confirmed that issues with the wires search(and up to date results) are not polling.
Thank you for your patience and understanding as we attempt to resolve this issue.
Brian F. Payton
IT Specialist
Digital Management Division
###
2012
From: Brian Payton
Sent: Thursday, October 11, 2012 10:21:08 AM
To: ALLTV OPERATIONS
Cc: IT Digital Management Division; Information Technology Managers; Help Desk Email; John Johnson; Andre Mendes
Subject: **RESOLVED** – Dalet Plus Files, Placeholders and transfering issues
ALL:
At this time, our System Administrators have resolved the various issues related to file transfers and we have investigated the status of the DaletPlus jobs and are happy to report they are processing properly. The root issue of this disruption of service seems to stem from a large amount of files being transferred during a single point of time this morning which cause the dalet system to act erratically. This trickled down to the boarder user community with the following reported symptoms:
· Users who save to placeholders will notice that after a change in status(MCA in particular), high-res video is not displayed
· Language services staff who are in preparing for upcoming shows will only see WMV Proxy video (Red “X”) as “ONLINE”
We will continue to monitor these support systems for any peculiarities and let our Broadcast Community know if any other issues arise. As always, we thank you for your patience in this matter and apologize for any inconvenience.
Brian F. Payton
IT Specialist
Digital Management Division
###
Sent: Thursday, September 27, 2012 1:40 PM
To:
Carol Pearson; Julia V. Taboh; Amy Katz; Carolyn Presutti; Brian Allen; Greg Flakus; Susan Koster; David Jones; NEB Correspondents
Cc: Michael Burke; Mike Kornely; Adam Greenbaum
And when this ENG material is ingested by DALET Plus, it often takes an hour just to get the video into the system. In some cases it takes two or three hours. It takes still longer for the ingest group to log and tag the material.
There has to be a faster way to at least get video into the system so it is physically possible to share good stuff.
###
From: IBB Notices Administration [notices@bbg.gov]
Sent: Sunday, February 26, 2012 8:52 AM
To: IBB Staff
Subject: DaletPlus system update – 8:30AM Sunday
The DaletPlus high-resolution video storage is partially fixed and we
will return it to service to enable broadcast this morning. The system
will be running on only one of three components called Content
Directors, so we are lacking redundancy and further maintenance will be
required to fully restore the system. We will work with the vendor to
keep the system running and fully restore it as soon as possible.
Thank you for your continued patience and understanding.
Regards,
James W. Russell
T/IN, Enterprise Networks & Storage Division
U.S. International Broadcasting Bureau
###
2011
From: Brian F Payton
Date: Thu, Dec 22, 2011 at 10:52 AM
Subject: Dalet Plus TV Operation or Rundown Problem
To: VOA TV Teams, IT Digital Management Division, IT Help Desk, IT Computer Systems Support
Cc: Christopher Novack, Andre Mendes, Michael Hunt
Dalet Plus TV Operation or Rundown Problem
Users are having TV playout issues using Dalet Plus and Clip tool. Both are encountering problems entering files into Dalet Plus rundowns. We are working to quickly resolve this problem. Users in control rooms are seeing the following errors:
**Illegal media data error in X and Y channel headings
*Unable to log into clip tool and\or the application is asking for license information*
We ask for your patience and will contact you when the problem is resolved or when more information is available.
—
Brian F. Payton
IT Specialist
Digital Management Division
###
From: Michelle Armes
To: VOA TV Teams
Date: Thu, 19 May 2011 10:58:26 -0400
Subject: Dalet Plus Udate
To all,
We are experiencing problems with the digital newsroom system. Users
will experience issues uploading spectrum for playout. We ask for your
patience and will advise you when the problem is resolved or more
information is available.
Thank you for your understanding in this matter.
###
From: Raymond Perez
To: VOA TV Teams, All TV Operations,
Cc: Information Technology Managers, IT Customer Support Help Desk, Andre Mendes, John Johnson
Date: Fri, 02 Sep 2011 11:04:29 -0400
Subject: DaletPlus Import/Export Problem with FCP
ALL:
Users are reporting problems using the Export to Dalet Plus function (XTend) in the transfer of files between Final Cut Pro and the digital newsroom system. At this time it appears this issue is isolated to the Language Services on the third floor exclusively. We and CSS are looking at the current problem and are working toward a quick resolution. We ask for your patience and will contact you when the problem is resolved or more information is available.
Ray
—
Raymond Perez
Lead IT Specialist
Digital Management Division