Methodology

Methodology of the Platform for Combating Media Disinformation – Raskrinkavanje.ba

Raskrinkavanje aims to combat media disinformation, recognised in the growing trend of disseminating inaccurate information represented as facts, placement of false and unverified news, and other forms of violations of good practices of journalism – which should ensure independent, objective, and truthful informing of the public. In the methodology of our work we will present the basic principles of how the platform operates, including: the principles of selection of the content and media outlets which are monitored and rated on the platform; rating of specific items of media content and media themselves; use of sources for research, structure of fact-checking analyses, and methodology for creating lists of media with questionable credibility.

Choice of media to be monitored and sources

a) Choice of media

Media that will be monitored for the purposes of this platform are selected based on two criteria:

1. that the media has audience/readership in BiH,
2. that it publishes and promotes articles and topics which indicate questionable professional ethics and standards.

According to the first criterion, we will monitor all media which have a significant readership in BiH based on available data (www.alexa.com and similar); specifically: that they are among the 100 most visited web domains in BiH; or that BiH ranks high among the countries where the visitors of that websites come from. According to the second criterion, we will monitor all media found to have some of the following characteristics: they do not have clearly indicated impressum and editorial staff; they publish content without providing the author’s name or a clear indication that the content is redistributed; the claims in the articles are not verified or supported by sources; they have already received some negative ratings on the raskrinkavanje.ba website. In the course of researching particular topics, the editorial staff of the Raskrinkavanje will, according to the aforementioned criteria, narrow down or expand the list of media it monitors.

In addition to media sources, social media profiles can also be monitored if they become, or tend to become, a source of fake news or some form of disinformation that may spread in the public sphere.

b) Sources for analyses

Ratings on the Raskrinkavanje platform are based on verification of claims published in the media and on establishing of relevant facts. Fact-checking research is based on the use of credible sources and methods. Every fact-checking analysis will have clearly stated sources and links to them. Some of the sources to be used in the research are the following:

• Official data and statistics
• Responses and data from official institutions
• Relevant media sources
• Official statements by relevant persons or institutions
• Research made by relevant institutions or individuals
• Expert opinions
• Legal acts and official documents of different kinds
• Own research
• Other verifiable and relevant sources

Structure of fact-checking analyses

Fact-checking analyses represent independent verification of various media content. Each such analysis contains: clearly marked claims from media reports that are checked and rated; a clear explanation of the ratings given on the basis of established facts and rating methodology; the rating; link to the original article or articles that are the source of these claims.

One fact-checking analysis can cover multiple original articles if they deal with the same topic and redistribute claims from one another.

Each analysed article contains the following elements on the “Raskrinkavanje” website, which ensure that the rated content is saved as it was found, should it be removed or modified by the authors after verification:

• The original headline, lead and text of the article, copied in the form of a post on raskrinkavanje.ba, as well as the URL of the original article;
• Original article as it was published by the media outlet, saved in PDF format, with the original URL of the article in the title of the document;
• Link to a copy of the article saved at archive.org.

A detailed overview of the elements of fact-checking analysis

A claim which is rated, published in an article of a monitored media outlet. One or more claims may be taken from one article. Each claim is verified and rated and the process include the following steps: selection of the claim to be rated – fact checking – presentation of established facts – rating.

In the event that the rated claims are redistributed by other media, each analysis, in addition to the original article, covers all the articles where one or more rated claims appear. Where multiple media have redistributed the same claim without citing the original source, the source is identified based on the timing of the publication of content. The oldest publication is treated as the original article which has subsequently been fully or partially redistributed by other media, if this can be determined using all available known sources. The extent and frequency of the redistribution of the claim is monitored from the time the original content was created to the time when the analysis was published, which is clearly visible (date and time wjen the article was published). The analysis can be supplemented with new data if such need arises (resurfacing of outdated news, continuation of intense news coverage after the published analysis, etc.), with a clear indication of when, for what reason and in what way the original content was expanded.

The claims to be rated are chosen primarily because they are presented as facts, and their verification aims to determine the extent to which they are factual and how the information was obtained in the context of professional journalism standards that should ensure accurate informing. As the platform deals with the media scene, the claims that are rated come primarily from the journalists, i.e. the persons who produce the media content.

Exceptions can be made for claims which are quotes or paraphrased statements of third parties, which may also be rated in the following cases:

• The statements quoted in the article are rated when they are the source of the disputed claim and have not been approached critically by the journalist. If the disputed statement is published in an article with a critical approach respecting the standards of the profession, it will not be rated.
• Under uncritical treatment, we mean that 1) the statement is disputable with respect to being grounded in facts, but is treated as a fact, especially in cases where the author further elaborates the presented argument without verification and questioning its truthfulness or offering evidence for the stated matter; 2) in cases where the statement is potentially harmful to certain persons/groups, the statement is presented as an established fact without contacting the other party; 3) in cases of a topic subject to public debate, the statement is presented as an established fact without giving other relevant information, especially if there are already known facts that do not support or dispute the point made.
• It is understood that when rating quoted statements, statements that seek to make factual claims are taken into account, rather than those that represent personal opinions and interpretations of occurrences, i.e., statements that are clearly formulated as subjective.
• The statement can also be taken into account when dealing with a person whose credibility has already been called into question by previous checking or facts known. The credibility of the source is considered to the extent relevant for the analysis of the claim made. These are, in particular, cases where the primary source are persons who have been found to deceive the public by making claims about the subject under consideration that are proven to be incorrect; to misrepresent themselves as experts in a particular topic or area where they lack expertise; to falsely represent their position or status in relation to a particular topic or area (cases where persons present themselves as independent experts, even though they have a clear and demonstrable personal interest associated with one of the opposing parties to the story); and when such statements are uncritically redistributed without verification.

Verification of the claim, which involves applying methodological criteria for the ratings used on the platform, and determining whether (and what) ratings relate to the extracted claim. Once the potential applicability of the predefined criteria has been determined, each individual claim shall be verified in accordance with the methodology. Examples of verification for some of the ratings include:

• If it is fake news, the research determines whether the information is presented in a separate claim of fact or whether it is an incorrect statement. All facts determined in the research, as well as their relation to the claim made, are stated in the explanation of the rating.
• In the case of biased reporting, the way the contextualised claim is made while neglecting or distorting other relevant information, or the way a particular narrative is favoured over another that opposes it (failing to contact the other party; omitting known facts that do not support the claim made) is identified and stated. Also, if the article uses anonymous sources while: it does not offer any evidence for their claims, it is not explicitly stated that the other party was contacted for verification, or that the information was verified from any other source, such behaviours are treated as a form of biased reporting.
• When it comes to manipulation of facts, verification determines what is wrong with the way the information is presented or interpreted, which may not be incorrect itself.
• In the case of a spin, the verification establishes a link between the claim being published and some other event or news, which may include a review of other reports by the same media on the topic, occurrence, or stakeholders involved. An example is cases where there is a current topic or report unfavourable to certain stakeholders, which is completely ignored in a given media, or their purpose is knowingly diminished; while at the same time reversed content and reports (favourable to given stakeholders, unfavourable to opposite parties) are being published.

All of the findings of the verification, relevant to the rating of the verified claim are clearly, thoroughly and accurately presented in the analysis of the content processed. All subsequent information (finding out about new facts, publication of corrections by the analysed media, detected changes in the original content after the publication of the analysis, new information about the dissemination of the original claim, etc.) will be clearly indicated as subsequently added content in the text of the analysis.

Rating given based on the above verification

Based on the above verification, each extracted claim is classified into one of the categories defined by the methodology, i.e., it is given an appropriate rating.

Each individual rating is automatically assigned to the processed article, as well as the media that published it. One article may have multiple ratings, depending on 1) how many rated claims it has 2) how many disputable elements have been identified.

Rating methodology

Ratings that can be obtained by a single claim or article from a media are as follows:

Satire

“Satire” rating is given to media content that uses fictitious news with the aim of satirical representation of the reality. This is another example of a media report that is not problematic if the media clearly indicates that it is satirical content. The purpose of incorporating this rating into the methodology is to point out this type of content in situations where it has the potential of being mistaken for a true media report and become potentially harmful in case of further redistribution without verification and a clear indication that it is satire. In our methodology, only media content whose author clearly emphasises that it is satire receives this rating. The media that publish this type of content cannot be found on one of the negative lists simply because of the publication of such content.

Clickbait

“Clickbait” rating is given to media reports whose title does not have a foothold in the text that follows. Such texts and articles aim to capture the attention of a consumer with a sensational title, promising content that does not actually exist and are mostly created out of financial interest, i.e., to increase readership.

Hidden advertisement

“Hidden advertisement” rating is given to a paid ad that is presented as editorial content. Reports like this look like journalistic/authorial texts, even though they are sponsored content. This rating is given in cases where such content is not clearly marked as advertisement or sponsored articles, as required by the rules of the profession.

Error

“Error” rating is given to a media report in which the media editorial board makes a mistake by posting inaccurate information or redistributing unverified news from another source without the obvious intention of disinforming the public. This kind of report can be very harmful in some cases, and after redistributing news from unreliable sources several times, we can also talk about the intention of some media to influence the public through redistribution of inaccurate content.

Biased reporting

“Biased reporting” rating is given to a media report that can be clearly identified as favouring facts, views and conclusions that fit a particular narrative, often disregarding the other party’s rules of contact when it comes to claims that are harmful to one’s reputation or portray certain stakeholders with a negative connotation. One of the forms of biased reporting is selective presentation of facts, where facts that support a particular thesis are emphasised, while facts that do not confirm it tend to be omitted.

Such media reports are mostly accompanied by a very emotional and passionate way of writing. They may or may not be inaccurate, but as a rule they do not show the whole picture and all sides of the story, but present only the facts that fit the preferred narrative.

Conspiracy theory

“Conspiracy Theory” rating is given to a media report that portrays a false or unverifiable description of an occurrence, event, or person, presenting it as part or result of a hidden plan (“conspiracy”). Typical of such content is that they make a series of claims, presented as facts, between which cause and effect relationships are established, without offering any credible evidence.

Pseudoscience

“Pseudoscience” rating is given to a media report that attempts to present certain opinions, views, values, or findings obtained by a non-scientific method as scientific findings or facts. Such reports often misinterpret or manipulate existing scientific research, or invoke research that is inconsistent with scientific verification and fact-finding.

Manipulation of facts

“Manipulation of facts” rating is given to a media report that uses known and accurate facts, but interprets them in a misleading manner. These reports generally use accurate information to make inaccurate conclusions or claims, thereby directing the conclusions of media content consumers in the wrong direction compared to the true meaning of the facts presented.

Disinformation

“Disinformation” rating is given to a media report that contains a “mix” of facts and false or semi-true content. In such cases, the media may not necessarily be aware of the inaccurate information that is published along with the accurate one. Also, this rating will be given to reports that have false attribution or titles that do not reflect the text in terms of accuracy of information.

Spin

“Spin” rating is given to a media report by which the author tries to divert the public’s attention from other content or facts that are reported in the media at that moment. It is characteristic of such media reports that they occur mainly in reaction to some other media report or current event that needs to be diverted from, which is unfavourable to some, mainly political, stakeholder. This rating is therefore specific in that it may depend on external factors, such as cases where a particular media ignores current news unfavourable to one party and at the same time publishes content that is unfavourable to the other party. In addition, spin can be recognised by the fact that the current news is casually mentioned in a media report, but the focus then shifts to 1) positive reporting on the stakeholder who is “affected” by the unfavourable news, or 2) news unfavourable to the other party.

Fake news

“Fake news” rating is given to an original media report (entirely produced by the media that published it) that contains factually incorrect claims or information. Content rated as fake news can be reliably identified as having been created and disseminated with the intent of disinforming the public, that is, to present a completely fake claim as a fact.

Redistributing fake news

This rating is given to media reports that redistribute the original content of other media that contains fake news. The rating is only given for redistributing content that is categorised as fake news. The redistribution of content rated as negative is treated as the creation of such content, since it does not entail the classification of the media as one that publishes fake news (see “Lists of Media Sources of Questionable Credibility”).

Corrected

“Corrected” rating is an additional rating given when the media determines that the content it publishes falls within one of the categories of negative ratings listed above, and corrects it

1. in accordance with the facts
2. in a clear and visible manner, in accordance with the principles of publishing the correction.

If media that have published original content subject to verification have visible contact information on their websites, they will always be contacted for comment on claims that prove disputable, giving each of them the opportunity to correct any errors in this way. In such cases, the analysis published on the Raskrinkavanje platform will further indicate that the rated content has subsequently been corrected.

Censorship

“Censorship” label marks media content that turns out to be partially or fully censored. This is usually content that was removed shortly after it was published, without a clear explanation by the editorial staff. In this sense, a distinction will be made between style and other types of updates, since censorship is primarily about content that deals with politically sensitive and topics of public interest that have “disappeared” for no apparent reason.

Unverifiable

“Unverifiable” rating is given to a media report that did not provide enough information to support the claims made in the report, nor can they be verified due to the lack or unavailability of reliable sources.

Lists of media sources of questionable credibility

In addition to rating individual content, Raskrinkavanje platform will automatically create lists of media and media sources of questionable credibility based on all the ratings recorded.

The purpose of making such lists is to establish a reliable, independent and methodologically based source that enables consumers of media content to check the credibility of the media they follow. In this way, media consumers can, by monitoring media lists, differentiate between professionally and quality reporting media and those found to be unreliable or inaccurate, or the ones whose credibility is compromised in any other way, in accordance with the criteria set by Raskrinkavanje.

Therefore, through the introduction of new content, Raskrinkavanje will continuously populate and update two media lists:

1. “Red flag” list, or List of media that publish fake news
2. “High-risk” list or List of media where there is reasonable doubt that they may publish content of questionable veracity

Red-flag list or List of media that publish fake news is a list of media who published and are proven to be the author of fake news, as fake news is defined by the methodology. Every media whose one media report is rated “Fake news” is automatically placed on this list. Every media whose 3 media reports receive a “Redistributing fake news” rating is also on the list.

Once a media is on this list, it will be removed from the list if within 3 months of the last such rating does not receive a new “Fake news” or “Redistributing fake news” report for any of its media reports, i.e. if it does not publish any fake news within 3 months. In the event that the media publishes a correction for fake news according to the standards prescribed in the methodology, after such correction it is removed from the list.

High-risk list or List of media that may publish reports of questionable veracity is a list containing media that, for a maximum period of three months, published at least three articles that had some of the elements of disinformation as defined by the methodology. Therefore, any media whose at least 3 media reports receive a rating other than “Fake news”, “Error”, or “Satire” ratings within 3 months, becomes a part of this list.

Once a media is on this list, it will be removed from the list if within 3 months after the last rating it does not receive any new rating from the methodology (except the rating “Fake news”, “Error” or “Satire”) for some of their media reports.

All media whose content has been rated on Raskrinkavanje, permanently maintain their “profiles” on the platform, regardless of whether they are on one of the two lists at a given time. The media profile, in addition to reviewing its ratings assigned and articles rated, contains all relevant information available, such as: portal address (URL), information about other related platforms (magazines, radio and television channels, etc.) information about ownership and registration, impressum, time since active etc.