United Kingdom General electionby 15 Aug 2029in 999 days Majority for any party 34% Labour largest party 93% Calendar Track record Research
Research · United Kingdom · Data and engineering

The UK data pipeline: collecting, reading and checking British pollsters' tables

Abstract. Every poll figure the site uses is read by a pollster-specific reader from the pollster's own spreadsheet or PDF, checked arithmetically, matched to its poll and stored with the file's SHA-256, so any number can be traced back to the file it came from. A daily check collects new files politely and stores them automatically only when every check passes and nothing already stored would be replaced.

Polling Forecast · updated 9 November 2026 · 24 min read · data pipeline, pollster tables, PDF extraction, provenance, SHA-256, automated checks, robots.txt

Why the pipeline matters to the method

The forecasting method on this site depends on numbers that most poll aggregators never collect. Our projection starts from each poll's raw answers, weighted but before the pollster's own turnout and don't-know steps (Correcting the raw answers: the method behind our UK projection, Why we start from British pollsters' raw answers rather than the published headline), and polling error is measured in England (Measuring British polling error in England, 2015–2024). Neither the raw stage nor an England figure appears in press reports. Both have to be read out of the pollsters' own data tables, published as spreadsheets and PDFs in formats that differ between pollsters and drift between releases.

This technical paper describes how files are found, read, checked and stored so that every number can be traced to its source. The rules given are the rules in the code; where the decisions log differs, the code has been followed and the difference noted.

Five principles govern the design.

  1. Everything traces to a file. Every stored table carries the name and SHA-256 hash of the file it came from, and the file itself is archived in the database under that hash.
  2. Nothing is overwritten. A corrected table, a revised method description or a changed headline is added as a new row, and the old row is marked as superseded.
  3. Readers find tables by their labels, never by fixed positions. A pollster that adds a column or moves a table should not silently shift the numbers.
  4. A failed check stops storage. A file whose arithmetic does not add up is kept for a person to look at; it is never stored as it is.
  5. Blocks are never worked around. If a site's robots.txt disallows a file, or the site answers with a refusal or a challenge page, the refusal is logged and the file is left for a person to save by hand in a browser.

The stages at a glance

A pollster's file passes through the same sequence whether it arrives through the daily automatic check or is uploaded by hand on the admin page. The only differences are the first two steps, which the automatic check adds, and the decision to store, which a person makes on the admin page.

Stage What happens Code Written to
Discover List the pollster's newest files from its own site (WordPress media library or RSS feed) PollSources, PoliteWeb nothing yet
Skip seen Ignore any URL already logged, or a file of the same name already in the pollster's folder PollFetcher poll.FetchedFile
Download Fetch to a temporary file, respecting robots.txt, spacing and refusals PoliteWeb temporary folder
Read The first reader that recognises the file extracts its tables, banner and method IngestService, Readers/* in memory
Check Arithmetic checks on every table, then the rebuilt England headline against the stored poll TableTools, IngestService in memory
Match Find the poll the file belongs to, by pollster, fieldwork dates and sample size SourceTableWriter.FindPoll in memory
Store Add the poll if new; write tables, columns, cells, checks and method PollCreator, SourceTableWriter poll.Poll, poll.PollResult, poll.SourceTable*, poll.PollMethod, poll.ExtractionCheck
Archive Keep the original file, once per hash IngestService.ArchiveFile poll.SourceFile
Refresh Clear the site's caches so the forecast is rebuilt with the new poll SiteRefresher none

Table: The pipeline, in order. The first three stages belong to the automatic check; the rest are shared with the admin page and the pipeline's command-line tools.

The readers and loading logic are one library (Elections.Core/Ingest) shared by the admin page, the daily check and the command-line pipeline, so a file produces the same result by any route.

The database layout

Polls are stored in SQL Server. The tables fall into four groups.

Polls and their headlines. poll.Poll holds one row per poll (pollster, fieldwork dates, sample size, source grade, polling basis, how England was obtained) and poll.PollResult the GB and England headline share for each party. A result is never edited; a correction is a new row that points to the one it supersedes. A poll first found by the automatic check is created from its own headline table with a code of the form NX-<pollster>-<yyyymmdd>, source grade A (the pollster's own table) and basis "HeadlineEquivalent".

The pollster's tables as printed (script 08). poll.SourceTable has one row per table extracted from a file. It records the poll, the table's code, its stage and base scope, whether totals are GB or UK, the question and base text, whether values are proportions, percentages or counts, the file name and SHA-256, the sheet or page, the reader's name and version, and the row that superseded it, if any. poll.SourceTableColumn holds the banner: every column as printed (Total, age groups, regions, 2024 vote and so on) with its group label, a normalised code, whether a region column lies inside England, and its weighted and unweighted bases. poll.SourceTableCell holds one value per row and column, exactly as printed, with the printed weighted count where the pollster gives one. poll.PollMethod records how the pollster turns answers into a headline (mode, weighting targets, turnout method, don't-know method, question wording, and whether the headline is model output). poll.ExtractionCheck records every check run on every table, with the expected and actual values and the tolerance.

The stage codes describe where a table sits on the path from raw answers to headline.

Stage Meaning
S0 Unweighted answers, including don't know, would not vote and refused. Only Ipsos prints this.
S1 Weighted, everyone asked (or the widest base printed), don't knows included, before any turnout step
S2 After the pollster's turnout filter or weight, still including don't knows
S3 The published headline
SQUEEZE The follow-up question put to don't knows
LTV The likelihood-to-vote question

Table: Stage codes in poll.SourceTable.

Base scope records who was asked: all adults, everyone except those certain not to vote, likely voters, don't knows only, the headline base, or "Other" for a comparison table that is kept but not used as the main raw stage. Where a file prints two versions of the raw question (YouGov prints both its constituency question and the standard question), the raw stage used by the method is the one whose scope is not "Other".

The files themselves (script 13). poll.SourceFile holds the original bytes of every file from which tables have been stored, keyed by SHA-256, with its name, source URL, size and the time it was saved. Because poll.SourceTable records the same hash, any stored table can be re-read from the archived file without the working folder of downloads, and a file can be shown to be the one that was read by recomputing its hash.

The fetch log (script 13). poll.FetchRun has one row per automatic check and poll.FetchedFile one row per file found, with its URL, where it was listed, its status and a message.

A separate set of tables (script 09) holds the raw and headline England figures for the 19 historical final polls from 2015 to 2024 that the correction is estimated from. Those were loaded once from recovered tables rather than through the readers described here.

Tracing a number to its source

The claim that every number can be traced to a file and a cell has a precise meaning. Any value used by the method is a row in poll.SourceTableCell. That row points to its column (with the column's position in the sheet or the joined PDF banner, and its label as printed), carries its row label as printed and its position within the table, and belongs to a poll.SourceTable row that names the file, its hash, the sheet or pages, the table title and the reader version. The file is in poll.SourceFile under the same hash. Given a number on the site, one can therefore open the exact file it came from, go to the named sheet or page and table, and find the printed row and column.1

Reading spreadsheets

Four pollsters publish their tables as Excel workbooks: Find Out Now, Survation, More in Common and Opinium. Each workbook is loaded into a plain grid of cell values per sheet, so that .xls and .xlsx files behave the same and blanks, dashes and #DIV/0! are all treated as empty. Each pollster has its own reader class, and the first reader whose quick test recognises the file (by its sheet names and sometimes its file name or cover-page text) reads it. A file no reader recognises is reported as "not a known pollster format" and nothing is stored.

Reader Version Where the stages are
Find Out Now 1.1 Sheets Voting (S1), Squeeze, Turnout (LTV) and Headline VI (S3); values are proportions down to a "Filtered N" row of weighted bases; commissioned files' "(alt)" sheets are ignored
Survation 1.1 One Tables sheet with stacked tables: Table_V1 (LTV), Table_V2_1 (S1, excluding those certain not to vote), Table_V2_2 (S2, weighted by likelihood to vote), Table_V2 (S3); totals are UK-wide
More in Common 1.1 votingintention (raw) (S1), votingintention (headline) (S3), likelihoodtovote (LTV), and a squeeze table in some files; two layouts, the later one interleaving a proportion column and an unweighted-N column for every banner cell
Opinium 1.0 V003 (S1) and HeadlineVI (S3), each answer printed as a proportion with its weighted count on the row below

Table: Spreadsheet readers and what each takes from its pollster's workbook.

Every reader works the same way within a sheet. It finds the header row and base rows by their text ("All", "Filtered N", "Base: … (Weighted)"), reads the banner from the group and label rows, and reads answer rows until a terminating label. A banner group is carried rightwards until the next group label, so "Labour" under "GE 2024" is a past-vote column. The cover sheet supplies fieldwork dates, sample size and method text, each found by its label.

Dates are parsed from the forms pollsters actually print, including "30 – 5 November 2025", where the start day falls in the previous month. If the cover date cannot be read, Find Out Now's and Opinium's readers fall back to the date in the file name and say so.

Survation deserves one note. Some of its releases print proportions to only two decimal places. When every proportion in a table is a whole percentage, the reader stores the printed weighted counts instead, which are more precise, and records that it has done so.

Reading PDFs

Ipsos and YouGov publish their tables only as PDFs. A PDF has no cells: it is a set of words placed at coordinates on a page. The PDF readers use PdfPig, a .NET library that returns each word with its bounding box. The problem is then to rebuild rows and columns from positions without relying on fixed coordinates, which change between releases.

Lines. Words are sorted top to bottom and left to right, and a word joins the current line if its top lies within 2.5 points of the line's. The label of a line is the run of non-numeric words before its first number.

Columns. Every banner table in these files has a line of unweighted sample sizes ("Unweighted Total 727 194 162 …" in Ipsos, "Unweighted Sample …" in YouGov). The key idea is that each number on that line marks the centre of a column. The readers find that line by its text, and the horizontal midpoints of its numbers become the column centres \(c_1 < c_2 < \dots < c_K\). Heading words from the lines above are stacked into the column whose centre is nearest, and hyphenated breaks are joined ("Eng-" and "land" become "England").

Values. A number at horizontal midpoint \(x\) on any later line is assigned to column

\[ k^{*}(x) = \arg\min_{k} \left| x - c_k \right|, \quad \text{accepted only if} \quad \left| x - c_{k^{*}} \right| \le \max\!\left( \tfrac{1}{2} \min(c_{k^{*}} - c_{k^{*}-1},\ c_{k^{*}+1} - c_{k^{*}}),\ 4 \right), \tag{1} \]

that is, it must lie within half the gap to the nearer neighbouring column (and never less than 4 points), or it is discarded. A number that falls between columns is therefore dropped rather than assigned to the wrong one.

This rule has a useful side effect. YouGov prints, to the left of its Total column, the previous poll's figures for comparison. That column has no sample size on the unweighted line, so no column centre is created for it, and its numbers are ignored without any special code.

Pages. A banner too wide for one page is continued on the next page under the same title. The readers read each page, then join the pages' columns on their normalised codes, skipping codes already read (Total is repeated on every page). Ipsos pages whose row labels differ from the first page of the same title are skipped with a message rather than joined out of line.

The Ipsos reader stores the weighted counts rather than the printed percentages. Ipsos prints each answer as a count line followed by a whole-number percentage line; the counts are exact to the nearest respondent, while the percentages are rounded to a whole point. Every printed percentage is then checked against the counts. The headline (S3) is the same likely-voter table with the parties divided by the printed "Total voting" sub-base, which reproduces Ipsos's published headline. Ipsos's "UnW Tot." column becomes S0, the only unweighted stage any pollster prints.

The YouGov reader handles the quirks found across 90 files: two runs of party columns on some pages (current vote, then 2024 vote), a "Weighetd Sample" misspelling, and "Skipped" and "Reufsed" rows. Because YouGov's headline has been MRP output since 2025, the reader flags it as model output and takes as S1 the constituency-prompted question the model is built from, keeping the standard question as a comparison table.

The pipeline command pdf-dump <file> [page] [--columns] prints what a reader sees: the lines, and the columns found from the base line. It is the tool for diagnosing a reader failure or writing a new reader.

Label normalisation

Pollsters print the same thing in many ways: "Conservative", "Conservatives", "Con", "Cons"; "Liberal Democrats (Lib Dem)"; "Reform UK"; "Some other party"; "I would not vote"; "Don't know"; "Prefer not to say". Before any arithmetic, each row label is turned into a stable code by Labels.Answer, after lower-casing, collapsing spaces and replacing curly apostrophes and non-breaking spaces. The codes are CON, LAB, LD, REF, GRN, SNP, PC, RB (Restore Britain), YP (Your Party), OTH, DK, WNV and REFUSED. UKIP and the Brexit Party map to REF, all Green parties to GRN, and Northern Ireland parties and the Alba Party to OTH. Likelihood-to-vote labels map to LTV0 to LTV10 or to verbal codes ("definitely", "very likely" and so on).

Banner columns are coded in the same way by Labels.Column, using the group label to decide what the column means: TOTAL; NATION:ENG, NATION:SCT, NATION:WLS, NATION:NI; the nine ONS regions (RGN:NE to RGN:SW) or the coarse four some pollsters use (RGN:NORTH, RGN:MIDS, RGN:LON, RGN:SOUTH), each flagged as inside or outside England; PV24:CON, PV24:DNV and so on for 2024 vote; and AGE, SEX, EDU and EU16 groups. Two columns with the same code get a numeric suffix so they remain distinct.

The important rule is what happens to a label that is not recognised. It is not guessed and it is not lumped into Others. The row is stored with the code UNKNOWN and the reader reports "unrecognised row label" as an issue, which appears on the admin page and in the fetch log. Because UNKNOWN is not a party code, such a row is left out of party shares rather than silently inflating one of them.

From a printed banner to GB and England

The stage tables are GB or UK tables with regional columns. The GB and England figures the method uses are derived from them by fixed rules (StageMath).

GB is the Total column when the table covers Great Britain. When the totals are UK-wide (Survation), Northern Ireland is removed using counts:

\[ g_r = 100 \, \frac{N^{\text{tot}}_{r} - N^{\text{NI}}_{r}}{B^{\text{tot}} - B^{\text{NI}}}, \tag{2} \]

where \(N_r\) is the weighted count for answer \(r\) in the column (or the percentage times the base where no count is printed) and \(B\) is the column's weighted base.

England is taken by the first of three methods that the table allows. First, the pollster's own England column, where printed ("Column"). Second, the English regions added up by their printed weighted bases ("RegionSum"), using the nine ONS regions if all are present and otherwise the coarse four:

\[ E_r = \frac{\sum_{g \in G} p_{g,r} \, B_g}{\sum_{g \in G} B_g}, \tag{3} \]

where \(p_{g,r}\) is the printed share for answer \(r\) in region \(g\) and \(B_g\) is that region's weighted base. Third, where no weighted bases are printed (More in Common's recent files), GB minus Scotland and Wales at their shares of GB adults ("PopulationShares"):

\[ E_r = \frac{G_r - 0.086 \, S_r - 0.049 \, W_r}{1 - 0.086 - 0.049}, \tag{4} \]

where 0.086 and 0.049 are Scotland's and Wales's shares of GB adults (ONS mid-2023). Each derived figure records its method, and the poll pages say which was used. Party shares are then taken over parties only, with don't knows, would-not-vote and refused removed and the remainder rescaled to 100. The table below shows, for the polls in today's reading, which method supplied each England figure.

PollsterFieldwork endEngland fromRawPublishedOurs
Find Out Now19 Nov 2026RegionSum27.7 / 20.329.7 / 20.926.7 / 22.2
YouGov16 Nov 2026Column28.0 / 24.027.0 / 24.027.0 / 26.0
More in Common15 Nov 2026PopulationShares30.2 / 23.730.2 / 26.329.2 / 25.7
Survation12 Nov 2026RegionSum31.5 / 21.631.1 / 22.530.5 / 23.5
Opinium6 Nov 2026RegionSum28.5 / 20.828.5 / 20.827.5 / 22.8
Live The polls behind our projection: the latest from each pollster with full tables in the last 28 days, England, Labour / Conservative. Computed from the database on 19 November 2026.

The automatic checks

Every table is checked as it is read, and every result is stored in poll.ExtractionCheck. A file is stored only if all checks on all its tables pass and at least one voting-intention table was found. The checks and their tolerances are as follows.

Check Rule Tolerance
TOTAL_SUMS_100 Answers in the Total column add up to 100% ±1.5 points by default; ±1.0 for Ipsos's parties-only headline; ±3.0 for YouGov's whole-number tables
COUNTS_MATCH_SHARES Each printed count divided by its column base matches the printed percentage Per cell, \((50 + 0.5\,p)/B + 0.15\) points; Survation's two-decimal tables allow 0.5 more; Ipsos allows 0.5 for whole-number percentages plus \((50 + 0.5\,p)/B + 0.1\)
ENGLAND_MATCHES_REGIONS Where both an England column and English regions with weighted bases are printed, England equals the regions added up ±1.0 point; ±2.0 for YouGov
S1_BASE_IS_SAMPLE The raw table's weighted base equals the sample size stated on the cover or footer ±1 respondent
HEADLINE_MATCHES_DATABASE The England headline rebuilt from the tables matches the England figures already stored for the poll ±0.25 points; ±0.6 when England comes from population shares; at least ±0.75 for whole-number files

Table: Checks run on every file. \(p\) is the printed percentage and \(B\) the column's weighted base.

The allowance in COUNTS_MATCH_SHARES comes from rounding. A weighted count printed as a whole number may be out by half a respondent, which is \(50/B\) percentage points; the base itself is rounded, contributing about \(0.5\,p/B\); and the printed percentage has its own precision, covered by the constant. Ipsos's percentages are whole numbers, so they may be out by a further half point.

The last check is the most important, because it ties the extracted tables to a headline entered and checked independently. Its tolerance is 0.25 points where England is measured directly or from weighted regions, 0.6 where it is estimated from population shares, and at least 0.75 for YouGov. YouGov prints whole-number percentages, so the stored England figures are whole numbers as printed, while the figures rebuilt from the tables are rescaled to 100 over the parties; the two can legitimately differ by more than half a point on one party, and the worst gap found across 90 files was 0.47. The check is skipped for a poll the database does not yet hold, since there is nothing to compare against; such a poll is created from its own headline table only after every other check has passed.

A file is matched to its poll by pollster and fieldwork end date, within two days either side. When two polls from the same pollster end on the same day, the closer start date and then the closer sample size decide. This rule was added after YouGov's three-week combined samples, which end on the same day as a weekly poll, were at first matched to the weekly poll.

The daily automatic check

The website runs a check for new polls once a day. After 07:00 local time it looks, every 15 minutes, for a scheduled run that has already happened today, and runs if there is none; so if the site starts later in the day, the check runs a couple of minutes after start-up. Only one run can happen at a time. The admin page has a "Check now" button and the command-line pipeline has fetch-polls. The hour, whether the check runs at all, and whether it may store automatically are settings.

Where it looks

Each pollster has one source, chosen by testing its site in September 2026. Four are WordPress sites whose media library can be listed through its public JSON interface, newest first, 20 files at a time. Two publish an RSS feed of articles, and the check opens recent matching articles (published in the last 21 days, at most 8, or 12 for Ipsos) and takes the links to data tables from each.

Pollster Source Files taken
Find Out Now WordPress media library (cms.findoutnow.co.uk) names ending -VI-Find-Out-Now.xlsx
More in Common WordPress media library names containing "voting intention", excluding hypothetical, MRP and tactical releases
Survation WordPress media library Surv…_VI_yyyy-mm-dd_Tables
Opinium WordPress media library Observer voting-intention tables; Politico, Spectator, by-election and leadership releases excluded
Ipsos en-uk RSS feed, then the article's PDF links politmkp or "Political Monitor … Tables", excluding charts and slides
YouGov en-gb RSS feed, weekly "voting-intention" articles, then the PDF link the linked "VotingIntention" PDF

Table: Sources for the automatic check. Commissioned and one-off polls are left out by file name; they can still be uploaded by hand.

How it behaves on other people's sites

All web access goes through a single class, PoliteWeb, which enforces the rules for every request.

  • robots.txt first. Before any address on a host is requested, that host's robots.txt is fetched and parsed. The rules for our named agent apply if there is a group for it, otherwise the rules for all agents; the longest matching rule wins, with * wildcards and $ endings supported. A disallowed address is not requested. A missing robots.txt, or one that cannot be read, is treated as setting no rules, which is the usual convention, and the fact is logged.
  • Spacing. Requests are at least 1.5 seconds apart.
  • A named user agent. Every request identifies itself as PollingForecast/1.0 with a link to the site (www.pollingforecast.com).
  • No evasion. Requests are plain GETs, without cookies or logins. A 401, 403, 429 or 503 response is treated as a refusal. A page containing a known bot-challenge marker is treated as a refusal. A download whose first bytes are an HTML page rather than a spreadsheet or PDF is refused as "sent a web page instead of the file". A refusal is logged with the status "Blocked" and is never retried in a different way.

A URL that has been logged is not downloaded again, and a file whose name matches one already in the pollster's working folder (for example, one saved by hand) is recorded as "Skipped" without being downloaded.2

What it stores, and what it leaves for a person

Each new file is downloaded to a temporary location and read with the same readers and checks as the admin page.

Status When What happens to the file
AddedPoll Every check passed and no poll matched, so a new poll was created and its tables stored Stored; original archived in poll.SourceFile
Stored Every check passed, the poll existed and had no tables from any other file Stored; original archived
AlreadyStored Every table from this exact file is already in the database Discarded
Waiting The poll already has tables from a different file, or automatic storing is switched off Kept in PollFiles/<Pollster>/ for the admin page
CheckFailed At least one check failed, or no voting-intention table was found Kept for the admin page, with the failed checks in the log
NotRead No reader recognises the file, or a reader failed Unrecognised files are discarded; files a reader failed on are kept
Skipped A file of the same name was already in the working folder Not downloaded
Blocked / DownloadFailed Refused by robots.txt or the site, or the download failed Nothing downloaded

Table: Outcomes of the automatic check, as recorded in poll.FetchedFile.

Storing is automatic only in the first two cases, which together require that every check passes and the poll has no stored tables from a different file. The second condition matters: a second file for a poll is usually a re-upload or a correction, and storing it would supersede the tables held, which is a judgement for a person. Such files wait on the admin page with their status, their England Labour–Conservative lead at each stage and every check. If anything was stored, the site's caches are cleared so the forecast is rebuilt.

The first real run illustrates the rules. It found a new Find Out Now poll (23 September 2026) and created it; it stored tables for three More in Common polls and one Survation poll that were already recorded in the database; and it held back two More in Common files, re-uploads of polls that already had tables, for a person to decide.

Versioning of readers and of models

Two kinds of version are recorded, for different reasons.

Readers. Every reader declares a name and a version, and every stored table records both. The rule is to bump the version whenever the reader's logic changes, so that a table can always be linked to the logic that produced it. Tables are not silently re-extracted when a reader changes; tables read by an earlier version keep that version's label until the file is read again. When a file is re-read and stored, the writer compares table by table: if the current table for that poll and table code already came from a file with the same hash, it is skipped; otherwise the new table is inserted and the old one is marked as superseded. The method description in poll.PollMethod is superseded in the same way, but only if it has actually changed.

Models. The statistical models are versioned separately in model.PollModelVersion, each with a status (Draft, Experimental, Live or Retired) and its configuration recorded as JSON. The first versions, PollModel 1.0 and 1.0-exp, were retired unchanged when new final polls were recovered, with their values frozen, and replaced by 1.1 and 1.1-exp. The principle is the same as for data: a change of method is a new version, and the old one is kept, so that any published figure can be reproduced from the version that made it.

A case study: the Opinium reader

The Opinium reader shows what reading real files involves. Opinium's fortnightly Observer workbooks have a front page, V003 (the full question with don't know and would-not-vote rows) and HeadlineVI. England is the four English regions (North, Midlands, London, South) added up by their weighted bases; the Northern Ireland column is empty because the sample is GB.

Reading all 54 files from October 2024 to September 2026 turned up five kinds of oddity, each handled by an explicit rule rather than a manual fix.

  1. Missing raw table. Two October 2024 files have no V003 sheet. Only the headline is stored, and the reader reports why.
  2. A short headline. Two February 2026 files print a HeadlineVI without the SNP and Plaid Cymru rows, so it adds up to about 97%. If the headline lacks those rows while the full question has them, and the headline fails the sum-to-100 check, the headline table is not stored and the full question is. This is safe because Opinium's headline is the full question with don't knows removed.
  3. Changing bases. The base of V003 is described as "who might vote" in most files, "likely to vote" in some 2025 files, and "all respondents" in others. The reader records the scope accordingly.
  4. Wrong or missing years. A front page reading "November 2023" in a 2024 file, or field dates with no year, would put the poll in the wrong place. The reader compares the front-page date with the date in the file name, and if they are more than 40 days apart but agree once the year is taken from the file name, it uses the file name's year and says so.
  5. A new party label. "Restore" is read as Restore Britain.

With these rules all 54 files read, every check passes, and the England headline rebuilt from the tables matches the stored figures to within 0.001 points. Reading Opinium also taught something about its method: there is no turnout weight between V003 and the headline, only the base filter and the removal of don't knows, so its raw stage and its headline agree on the parties.

What can go wrong, and how it is caught

Failure How it is caught
A pollster changes its layout, moving or adding columns Readers find tables and columns by label; a missing sheet, header or base row is reported; a mis-read table fails the sum, count or England checks
A number is placed in the wrong PDF column The nearest-column rule discards numbers between columns; misplacement usually breaks COUNTS_MATCH_SHARES or ENGLAND_MATCHES_REGIONS
A new party or answer label appears Stored as UNKNOWN and reported, never folded into Others
The file belongs to a different poll than assumed Matching uses dates within two days, then start date and sample size; HEADLINE_MATCHES_DATABASE fails if the tables do not reproduce that poll's headline
A table is incomplete (the Opinium short headline) TOTAL_SUMS_100 fails; the reader-specific rule keeps the full table instead
A working-draft file is published (Ipsos, March 2026, with significance letters inside the cells) The Ipsos reader recognises the layout and refuses the file with a message; its figures must be entered by hand
The site serves an error or challenge page in place of the file PoliteWeb refuses HTML where a file was expected and recognises challenge pages
A corrected version replaces a file already stored The automatic check will not store it; a person decides on the admin page, and the old tables are superseded, not deleted
The same file arrives twice Recognised by URL, by name in the working folder, or by hash and table code in poll.SourceTable

Table: Failure modes and the mechanism that catches each.

Two gaps should be stated. First, issues and checks are different things. A failed check blocks storage. An issue, such as an unrecognised row label or a missing optional sheet, is reported on the admin page and in the fetch log but does not by itself stop a file whose checks all pass from being stored, including by the automatic check. The arithmetic checks make it very likely that a structural problem affecting the voting-intention numbers will also fail a check, but not certain. Second, the checks confirm that a table is internally consistent and reproduces the poll's headline. They cannot detect an error the pollster made consistently in its own file, and they do not test whether the pollster's method is sound; that is the subject of Rating the British pollsters: accuracy, the Labour–Conservative lead and openness.

Coverage is the other limitation. The automatic check covers six pollsters. Others either publish no tables, publish them in formats with no reader yet, or sit behind robots rules or challenge pages. Their polls are recorded from their headline figures and enter the published average where they qualify, but their raw stages cannot be read, so they do not enter our projection. Today that projection uses the raw tables of 5 pollsters (Find Out Now, YouGov, More in Common, Survation and Opinium). Tables that cannot be reached automatically are listed for saving by hand in a browser; no pollster is contacted and no block is bypassed.

Summary

Each file is read by a reader written for its pollster's layout, each table must pass checks with tolerances derived from rounding, and each poll's rebuilt England headline must match a figure recorded independently. Everything stored carries its file's hash, the file itself is kept, and nothing is overwritten. The daily check does the routine work but stops whenever a file fails a check or would replace something already stored. A forecast whose claim is that the published headlines are wrong in predictable ways has to be able to show, number by number, what the pollsters actually printed.

What has changed since this paper was written

Updated 9 November 2026. The rest of the paper is as first published, except where a sentence below says it was corrected.

  • More frequent checks (13 October). After the morning run, the readers check for new polls every three hours until 11pm.
  • Polls without tables (26 October). The Wikipedia check now stores any GB poll we do not hold, graded C, marked as a Wikipedia transcription and counted in the GB average only. When the pollster's own tables arrive, the same poll is upgraded in place with its England figures and grade A; no duplicate is created. YouGov and Ipsos are still read by hand.
  • Pollster grades rescaled (13 October) (Rating the British pollsters: accuracy, the Labour–Conservative lead and openness).

  1. For most readers the recorded row position is the answer's position within the table, and the stored first row of the table is 1, so the cell is located by the table title, printed row label and column rather than by a spreadsheet row number. Survation's reader records the sheet row where each stacked table starts.↩

  2. The "already seen" test excludes files logged as Blocked or DownloadFailed, so those URLs are tried again at the next daily check. Every other logged URL is never requested again.↩

How to cite

Polling Forecast (2026). The UK data pipeline: collecting, reading and checking British pollsters' tables. Retrieved 20 November 2026 from https://www.pollingforecast.com/research/uk-data-pipeline. Live figures change as new polls arrive; the date above is when the text was last revised.

You may also like