A city-wide effort to digitise historical and planning records has stalled at the deduplication stage, forcing staff and contractors to work through a backlog that has grown larger than initially expected.
Townsville City Council's multi-year digitisation program ran into a significant technical hurdle this week after staff identified a large cache of duplicate images embedded across the council's planning, heritage, and flood-recovery document archives. The problem, which surfaced during a routine audit of the digital records system on Monday, has temporarily slowed public access to several online planning portals used by residents and businesses across the city.
The timing matters. Council has been racing to complete the digitisation work ahead of a Queensland State Archives compliance deadline later this year, and the duplicate image problem threatens to push that schedule. The program is also tied directly to Townsville's ongoing flood-resilience effort — accurate, searchable records of pre- and post-2019 flood infrastructure are central to how the city plans and funds future mitigation works, including upgrades along Rowes Bay and along the Ross River corridor.
The issue is not simple file duplication. Council's IT and records management teams found that scanned images — many of them engineering drawings, zoning maps, and post-flood damage assessments — had been ingested into the system multiple times through separate batch uploads conducted between 2021 and 2024. Each duplicate image occupies storage space and, more critically, appears as a separate record in the public-facing search tool hosted on the council's website. Staff searching for a single heritage property on Flinders Street or a drainage easement in Mundingburra can return dozens of near-identical results, making it difficult to confirm which version is the authoritative record.
The council's Records and Information Services team, based at the Civic Theatre administration precinct on Boundary Street, is working with a contracted software vendor to run automated deduplication across the affected directories. The process involves hash-matching scanned files and flagging records where metadata conflicts — a time-consuming step when the files in question include large-format engineering PDFs and multi-page TIFF images. Council has not publicly confirmed how many files are involved, but the scope of the audit covers records generated across at least four separate council departments since the post-2019 flood recovery effort began in earnest.
Two programs with direct community impact are caught in the slowdown. The Townsville Local Disaster Management Group relies on the council's document system to access infrastructure condition reports when activating flood or cyclone response. A spokesperson for the group confirmed this week that staff have been asked to use legacy hard-copy backups while the deduplication work is underway — a workaround, but an inconvenient one given the volume of updates generated since the 2019 event.
The second affected program is the Townsville Heritage Register project, a joint initiative between Council and the Townsville Local History Collection at the Thuringowa Central branch of Townsville City Libraries. Volunteers cataloguing pre-war buildings across North Ward and Belgian Gardens had been using the digital archive to cross-reference council permit records with photographic evidence. That access is currently restricted to staff-only terminals while the cleanup proceeds.
The deduplication work has also drawn attention to a broader question about procurement. The software platform initially selected for the digitisation project was adopted in 2021 under a Queensland Government state-purchasing contract, which was intended to reduce per-council licensing costs. Whether the duplicate ingestion problem stems from a platform configuration issue or from the way contractors uploaded batches of files is, according to council officers, still being determined.
For residents and businesses waiting on planning documents or heritage assessments, council's customer service centre on Walker Street is the current point of contact. Staff there can arrange manual retrieval of specific records while the online system is partially restricted. Council has indicated it expects the automated deduplication pass to be completed within the next two to three weeks, after which a manual review of flagged records will follow before full public access is restored. Anyone with an urgent planning matter pending is advised to contact the Development Services counter directly rather than rely on the online portal until further notice.
About this article
Published by The Daily Townsville
Spread the word
Newsletter