Former Analyst Challenges USAGM Audience Measurement Methods, Claims of Sharp Increases

BBG – USAGM Watch Guest Commentary

Former Analyst Challenges USAGM Audience Measurement Methods, Claims of Sharp Increases

By Dan Robinson

 

A former analyst for the U.S. Agency for Global Media, which oversees taxpayer-funded broadcast and online media directed at overseas audiences, has publicly challenged the methods used by the agency in making audience size claims.

An audience research analyst for the U.S. Agency for Global Media (formerly the Broadcasting Board of Governors) for many years, Kim Elliott, Ph.D., is the first former official to raise questions about USAGM figures.

His views, published online by the University of Southern California Center for Public Diplomacy blog, also appeared first in a small circulation subscription journal published by NASWA (North American Shortwave Association). This article is based on both pieces.

 

THE USAGM AUDIENCE INCREASE: LESS STARTLING THAN MEETS THE EYE. By Kim Andrew Elliott, CPD Blog, March 27, 2019

 

Retired from USAGM since 2017, Elliott’s articles about U.S. international broadcasting have appeared in newspapers and foreign affairs journals, and are highlighted on his Twitter feed.

He refers to the claim by USAGM CEO John Lansing, that agency programming “was consumed by 345 million adults weekly worldwide — an unprecedented year-on-year increase of 67 million from 2017.”

“There really is no “year-on-year” increase,” says Elliott, “because that would require surveys every year in every country reached by the USAGM entities. There certainly is not enough budget for such an undertaking.”

Elliott says figures are based on a “change of methodology” in which the agency uses survey questions focusing with even greater specificity on “all the programs and blog names” associated with key brand broadcasters, such as Voice of America.

China plays a major role in USAGM global estimate claims.  Elliott asserts that the 6.2 percent increase claimed by the agency in China “beggars belief and indeed is not to be believed.” 

This description does not appear in Elliott’s USC article, but Elliott still states that the new methodology “would explain an apparent increase of audience in China from 0.3% in 2014 to 6.2% in 2018 — an increase that strains credibility.”

CRITICS QUESTION SERIES OF AUDIENCE CLAIMS

USAGM’s assertion of a 67 million increase is the latest in a series of claims in recent years. In 2016, it claimed a 50 million audience size increase in one year.  The latest USAGM press release claims “a record weekly audience of 345 million across radio, television and the internet – a 24 percent increase from 278 million in 2017.

Critics within and outside the agency have raised questions, alleging that USAGM has been cooking the numbers, which also help agency appeals to Congress to sustain and increase its budget. 

USAGM claims expansion and “dramatic growth” in use of agency programs by people in China, on digital and social media platforms that are “censored on mainstream platforms.” 

However, “as the skeptical research analyst” Elliott writes that he saw “increases or decreases from one survey to the next that were attributable to something in the methodology, such as the wording of the questionnaire, the conduct of the interviewers, or the selection of the sample.”

Agency officials did not respond immediately to a request for comment on Elliott’s article.

USAGM is facing a proposed $180 million cut in its budget as part of the spending proposals for FY 2020 the Trump administration has sent to Congress.  In FY 2019, Congress ended up restoring some funding for the agency, boosting its actual budget to the $808 million level.

FALSE POSITIVES

Elliott says audience claims may also be based on what he calls the “false-positive trap”, and include the assertion that USAGM has a “market-leading audience share within Iran.”

In recent months, USAGM launched a 24/7 Persian language television operation. But Elliott notes that based on “anecdotal discussion and reaction from the Tehran regime, BBC is mentioned more often than U.S. international broadcasting outlets.”

Before he retired in 2017, Elliott said he made known his concerns about methodologies used to buttress agency claims about its reach in Iran, “but the audience research results for Iran were not changed.”

“Apparently [USAGM] management sees false positives from program names that resemble program names from domestic broadcasting as a feature rather than a bug,” Elliott writes.

“This “feature” brings higher audience numbers, which are touted by management.”

SKEPTICISM ABOUT AGENCY RUSSIA, IRAN CLAIMS

Elliott does not rule out that USAGM has “exploited” false positives for other audience increase claims, for example its claim of increases in the agency’s audience in Russia.

USAGM launched what it calls a 24 hour Russian language television “network” called “Current Time”, and maintains that the joint operation by VOA and Radio Free Europe/Radio Liberty has attracted wide viewership in Russia.

The agency counts online “views” and what are called “impressions” in its calculations of audience size. USAGM has not released detailed explanations of how it arrives at final figures that form the basis of press releases and reports to Congress.

BBG/USAGM Watch, the independent watchdog website monitoring USAGM operations, says actual outcomes for “Current Time” have been dismal.

In 2018, BBG/USAGM Watch noted that in one 30 day period, “Current Time” lagged far behind Russian TV operations, and accused USAGM of “[trying] to fool a lot of people. . . with selective and misleading use of meaningless statistics.”

EMAIL WARNED AGENCY BROADCAST ELEMENTS ABOUT METRICS TACTICS

In 2017, Elliott suggested that agency officials may have been using research tools to deliberately target certain foreign language broadcast services for downgrading or elimination by inaccurately portraying them as having failed to reach certain performance review targets.

In an email, apparently not approved in advance by his bosses, Elliot warned Voice of America service chiefs to be wary of the process being used “against your services or yourselves” and pointed to what he called “volatile” Internet (“digital”) metrics.

A copy of the email was provided to this reporter by a source at USAGM.

The agency has faced pressure to save money by eliminating duplicative language services. Individual broadcast services have launched lobbying efforts, internally and on Capitol Hill, to defend against partial or complete shutdown.

Despite calls over the years by the Government Accountability Office (GAO) to trim operations, VOA and Radio Free Asia, for example, still operate competing broadcast services for countries such as Burma, North Korea, and Cambodia. 

Some members of Congress also continue to advocate to begin new language broadcasts to areas of the world that are of specific interest to them or to their local constituencies.

“FLAWED” REVIEW PROCESS

Elliott’s 2017 email referred to a 2016 strategy review survey by management involving [performance] targets “based on the results of survey-based audience research or Internet metrics.”

Lower performance targets he had proposed were “increased up [the management] chain”, said Elliott, “producing a high failure rate.”

“One might conclude from this high failure rate that VOA language services are performing very poorly,” Elliott wrote at the time. “I think the real reason is that the review process is flawed.” 

FIRST KNOWN PUBLIC CRITICISM

Elliott’s articles mark the first public criticism to emerge from a former official of audience measurement methods employed by USAGM, which re-branded itself in 2018.

The agency has faced sharp criticism, including by this reporter, for numerous performance and efficiency issues, technological breakdowns, and journalistic lapses, as well as advocacy journalism, including a string of scandals at VOA under Obama-era appointees Lansing, and VOA director Amanda Bennett.

Through presidential and congressional actions, the agency in general has been moved closer to the U.S. national security and foreign policymaking structure, a process that began under the Obama administration which felt not enough was being done to help combat Russian disinformation and online activities of ISIS and al-Qaeda. 

Though annoyed by numerous scandals and low morale plaguing the agency, members of Congress remain generally supportive, but are insistent that it be thoroughly immersed in whole-of-government CVE (Countering Violent Extremism) efforts. 

CRITICIZES “CHERRY-PICKING RESULTS”, URGES CHANGE IN APPROACH

“The best solution would be for international media audience research to be conducted by a third party, or by a consortium of the players”, Elliott writes in his article in the USC public diplomacy site.

“For the sake of transparency, instead of cherry-picking results for press releases, the datasets and questionnaires should be made available to scholars and researchers. This could be done in a full-day deep dive involving researchers inside the international media business as well as outside observers, with lots of questions and answers.”

AGENCY FACES FOIA REQUESTS

In 2018, I submitted a Freedom of Information Act Request (FOIA) to USAGM asking for documentation relating to its claim two years earlier of a 50 million audience increase, and methodology used to support that claim.

As of this writing, the outcome of that request, to obtain information that would more completely inform taxpayers about how USAGM comes up with audience size and reach figures that seem impressive on their face, is still awaited.

The Trump administration was late in moving to make leadership changes at the agency. The nomination of Michael Pack as new CEO to replace Lansing is still pending on Capitol Hill.

USAGM and VOA officials and some of government journalists are thought to have encouraged a series of articles in major U.S. media since Donald Trump was elected, warning that any ideological “takeover” of the agency by the new administration would be a disaster. 

However, every U.S. president has used his authority to make changes in the agency. It used to be the U.S. Information Agency (USIA) and was later abolished by the U.S. Congress and President Bill Clinton who created the former Broadcasting Board of Governors. 

Legislation signed by President Barack Obama before he left office formally abolished the BBG Board, which was supposed to be transitioned to an advisory board for the CEO. 

Under the 2017 legislation, yet another board called the International Broadcasting Advisory Board consisting of five members, including Secretary of State, serving three three-year terms was supposed to be formed.  What progress has been made, if any, toward creating that board, is unknown. 

USAGM held an event in September of 2018 touting its re-branding from the former Broadcasting Board of Governors, but limited media attendance.  The agency at first declined to allow this reporter, representing BBG – USAGM Watch, to attend. 

Officials took no questions about agency audience claims or any other issue, at the after-hours event which was held in the First Amendment Lounge of the National Press Club.

###

Dan Robinson retired in 2014 after 34 years with the Voice of America. In addition to his assignment as senior White House correspondent from 2010 to 2014, he served as bureau chief in Nairobi, Kenya and Bangkok, Thailand. He was also the chief of the VOA Burmese Service and the Capitol Hill correspondent. Views expressed here are his own.