How Townsville's Records Ended Up Full of Doubles: The Story Behind the Duplicate Image ProblemUpdated

Townsville City Council's digital asset library contains tens of thousands of photographs accumulated over roughly two decades of scanning, photographing and importing. A significant portion of those files — internal estimates have put the figure at more than 15 percent of the total catalogue — are duplicates: identical or near-identical images stored under different file names, across different folders, sometimes on entirely separate servers. The rationalisation program to fix that problem formally began in the first quarter of 2026, but the conditions that created it stretch back much further.

The timing matters because the council is mid-way through a broader digital infrastructure upgrade ahead of the Townsville Hydrogen Hub's anticipated operational milestones and the ongoing reconstruction work tied to the 2019 flood recovery. Both projects generate enormous volumes of photographic documentation — site inspections, progress records, community consultation imagery — and feeding that material into an already cluttered system risks compounding an existing mess rather than creating a clean record.

How the Duplicates Accumulated

Three distinct events drove the problem. The first was the 2019 Ross River Dam flood emergency, when the priority was capturing damage evidence quickly. Staff across multiple departments — engineering, community services, disaster recovery — were photographing the same sites at Rosslea, Idalia and along the Haughton River corridor simultaneously, often uploading to whichever shared drive was accessible rather than the designated repository. No deduplication protocol was in place at the time.

The second factor was a 2021 migration away from the council's legacy content management system to a newer platform. File transfers of that scale routinely produce duplicate entries when folders are mapped incorrectly or when staff manually re-upload files they cannot locate in the new environment. The Townsville City Libraries system went through an analogous problem during its own catalogue digitisation work centred on the Aitkenvale branch around the same period, with scanned historical photographs from the North Queensland collection appearing in multiple catalogue records.

Third, and perhaps most structural, was the growth of smartphone-based documentation across council field teams. Officers attached to the Riverway precinct maintenance crews, the Castle Hill lookout infrastructure team and environmental monitoring operations along the Belgian Gardens foreshore all began submitting images via personal devices from around 2020 onward. Without a centralised intake point, duplicates entered the system with every batch upload.

What Deduplication Actually Involves

Replacing or removing a duplicate image sounds straightforward. In practice, each file may have been referenced by multiple records — a development application, a media release, an internal report — meaning deletion without checking those links breaks content across the organisation's published and internal materials. The council's ICT team, working alongside the records unit based at Thuringowa Drive, has had to build a reference-mapping process before any file can be safely retired.

Software tools that use perceptual hashing — algorithms that compare images by visual content rather than file name or metadata — have been central to identifying near-matches, particularly where the same photograph was saved in both JPEG and PNG formats, or at different resolutions for different publishing contexts. The James Cook University library system, which ran a comparable audit of its digital collections in 2023, found that perceptual hashing identified roughly 30 percent more duplicates than filename matching alone, according to publicly available conference proceedings from the Australian Library and Information Association.

For community organisations managing their own archives — the Townsville Pacific Festival committee, for example, or the First Nations community groups documenting treaty process consultations on Palm Island — the same logic applies at a smaller scale. Duplicate images waste storage, create confusion about which version is the authoritative record, and can result in outdated photographs being used in publications after a preferred image has been identified.

The council's ICT team has indicated the first full deduplication pass across the primary asset library is scheduled for completion by September 2026. Organisations running their own image collections — whether heritage groups along Flinders Street or sporting clubs at Dairy Farmers Stadium — would be well placed to audit their own holdings before the wet season documentation season begins in earnest later this year. Free tools including digiKam and Google's open-source duplicate-finder libraries are available for non-enterprise users, and the Townsville City Libraries digital literacy program runs sessions at the Central Library on Denham Street that cover basic file management.