Migration log · August 2026

Finally migrating failheap‑challenge

Discourse ships an official vBulletin import script and there is a well-known community guide for it. Both were the starting point. Neither survived contact with a real forum of this age — the script does not run at all on current Discourse without patching, and several things it does import, it imports wrongly.

I tried this on my own first and got nowhere. What finally moved it was working through the whole thing with Claude: four weeks of planning, test runs and false starts, then a four-day push to get it live. What follows is what actually broke and how it was fixed.

01

The encoding problem

The source database declared every text column latin1_swedish_ci. The bytes inside those columns were UTF‑8.

Nobody chose this. latin1_swedish_ci was MySQL's compiled-in default from 4.1 (2004) until 8.0 (2018), and this server runs 5.0.95 — so every CREATE TABLE that did not name a charset silently got latin1. Neither phpBB nor vBulletin ever named one:

tables at latin1_swedish_ci        204
tables at utf8_general_ci            1   # tsx_server, a 2014 add-on
columns with explicit non-latin1     4   # all four in that same add-on table

The post table makes it plain — pagetext carries no CHARACTER SET clause of its own, and the table ends with a bare DEFAULT CHARSET=latin1 it inherited from the server. In 204 tables the forum software specified a charset exactly zero times; the only table that did came from a third party, three years after the migration.

Both applications got away with it because each stored UTF‑8 bytes and told MySQL the connection was latin1. MySQL passed the bytes through untouched and handed back exactly what it was given. Byte in, byte out — the declared charset never mattered, because nothing in the chain ever tried to interpret it. That held for fifteen years.

The UTF‑8 bytes themselves arrived with the phpBB import, and the database can date that precisely. vBulletin's import*id columns sit empty on a forum that was never imported into; here they are populated on 133,388 posts and 1,986 users, and the seam is sharp enough to date to the hour:

last post carrying an import id   2011-07-01 21:20:56
first post with none              2011-07-01 22:34:06   # 73 minutes later

imported content spans            2011-04-09 → 2011-07-01   # ~12 weeks
native vBulletin content spans    2011-07-01 → 2026-06-24   # everything since

So the forum ran roughly three months on phpBB and moved to vBulletin on 1 July 2011, carrying its history across. The oldest posts in the archive — a thread where people are racing to reply "second" and arguing about "'11s" — are the phpBB forum's opening day, preserved through the move with their original timestamps.

Left alone this is harmless — vBulletin read back the same bytes it wrote. It becomes destructive the moment you run mysqldump normally, because MySQL then "helpfully" re-encodes on the way out.

Naive dump

C3 83 C2 A9
Renders as é — four bytes where two belong. The dump declares itself utf8, so nothing errors. The corruption is silent and permanent.

Byte passthrough

C3 A9
Renders as é. The original bytes reach the destination untouched and are simply relabelled as what they always were.

Confirming it was safe to fix

A uniform mislabelling is recoverable with one rule. A mixed encoding — some rows genuinely latin1, others UTF‑8 — is not, and always loses something. So the first job was proving which case this was.

-- how many posts contain any high byte?
SELECT COUNT(*) FROM post
WHERE CONVERT(pagetext USING BINARY)
      RLIKE CONCAT('[', UNHEX('80'), '-', UNHEX('FF'), ']');
-- 43,125

-- of those, how many are valid UTF-8 when reinterpreted?
SELECT COUNT(*) FROM post
WHERE CONVERT(pagetext USING BINARY)
      RLIKE CONCAT('[', UNHEX('80'), '-', UNHEX('FF'), ']')
  AND CONVERT(CONVERT(pagetext USING BINARY) USING utf8) IS NOT NULL;
-- 43,125  ← every single one. Zero exceptions.

Both numbers matching is the green light: the whole archive is UTF‑8 wearing the wrong label, and one uniform rule fixes it.

The fix. Dump with byte passthrough so MySQL never converts anything, then rewrite the charset declarations while loading into a modern database:

# on the source
mysqldump --default-character-set=latin1 --skip-set-charset ...

# while loading
sed 's/DEFAULT CHARSET=latin1/DEFAULT CHARSET=utf8mb4/g'

The dumps that were already there

Three earlier dumps sat in /root on the old server. All three carry SET NAMES utf8, meaning they were taken without the passthrough flags. Verified byte-for-byte against the new dump: they are double-encoded throughout. Importing from any of them would have baked é into 43,125 posts permanently.

Verification, not assumption. After loading, a search across 1.22 million imported posts for the classic mojibake signatures returned one hit — and that post is corrupted in the source vBulletin too, from someone pasting broken text years ago. Nothing was damaged in transit.

What that looks like in practice

A French post from 2012, as it survives in the correct dump, and as the same sentence appears in the three dumps that were already sitting on the server:

Correct dump

bonjour,mon copain écoute son ipod tous les soirs
Dumped with byte passthrough. Two bytes, C3 A9, correctly labelled as UTF‑8 on load.

The dumps already on the server

bonjour,mon copain écoute son ipod
Same post, three separate backups from 2024 and 2025. Four bytes, C3 83 C2 A9. Every accented character in the archive, wrong.

Those three dumps were the obvious thing to import from — they were already made, already on disk, and nothing about them looks broken until you go looking for a specific character. Using any of them would have baked that second column into 43,125 posts permanently.

02

What was actually in there

ContentIn sourceImportableNote
Posts1,892,3221,873,461Difference is deleted content
Threads20,79120,70883 soft-deleted, 49 redirect stubs
Users4,8584,84117 skipped — duplicate/invalid emails
Forums → categories6565All imported
Private messages35,38335,383 
Custom avatars1,2111,21130 MB of blobs, all intact
Profile backgrounds510510Needed a second dump pass
Attachments00Never enabled — not data loss

Roughly half the database was junk that should never be migrated: 382 MB of spam-filter logs, 977 MB of search index that Discourse rebuilds itself, plus edit history and BBCode caches. Dumping only the 36 content tables cut 1.4 GB of dead weight.

Deleted content is why progress bars stop short of 100%. 83 threads were soft-deleted in vBulletin — flagged as removed but left in the table. The importer correctly skips them, so counting against the raw table total makes a finished import look stuck at 99.9%. The real target is 20,708 threads and 1,873,461 posts.

03

The stack

Discourse's importer is not a standalone converter — it is a Ruby script that runs inside a working Discourse instance, reading from a live MySQL copy of the old forum. So both halves have to exist at once.

Source serverCentOS 5.10 · MySQL 5.0.95 Reached over SSH with legacy SHA‑1 algorithms re-enabled — OpenSSH 9 refuses them by default. The box runs OpenSSH 4.3 from 2006.
vbmysql containerMySQL 8.0.46 · utf8mb4 Holds the corrected copy of the vBulletin database. MyISAM tables converted to InnoDB on load.
discourse containerRuby 3.4 · PG 18 · Redis Runs the importer. Postgres data and node_modules live on Docker volumes, not Windows bind mounts.
Targetbare metal · 4 cores · 8 GB · SSD Cloudflare → router:443 → HAProxy → Discourse. Mail via Mailgun.

An OpenSSH from 2006 sounds worse than it was. Re-enabling SHA‑1 key exchange to reach a nineteen-year-old daemon is the kind of thing that deserves a second look, and it got one. That server accepted keys only — no password authentication to brute‑force — and its SSH port was reachable only from whitelisted addresses at the firewall, so the internet could not talk to it at all. The version was old; the exposure was not. Most of the vulnerabilities that make people flinch at OpenSSH 4.3 need either a password prompt or the ability to reach the port.

The forum's actual problem was never SSH. It was a public web application on an operating system that stopped receiving security updates in 2017, being scanned continuously by bots that got past Cloudflare.

Things that had to be fixed to get this far

ProblemCauseFix
SSH refused to connectOpenSSH 9 dropped SHA‑1 KEX; server offers nothing elseRe-enable diffie-hellman-group14-sha1, ssh-rsa
Transfer silently truncatedscp reported success after sending 209 MB of 375 MBSplit into chunks, verify each by MD5, retry
mysql2 gem wouldn't buildNo MySQL client headers in the imageapt-get install default-libmysqlclient-dev
Bundler refused to resolveDiscourse's own Gemfile declares sqlite3 twice with conflicting pinsRemove the duplicate
Importer crashed on first userTZInfo 2.x rejects integer timestamps; the script still passes themConvert to Time first
Crashed again on user 2Some accounts have a NULL last-visit dateMake the timestamp parser nil-safe
pnpm install failedWindows bind mounts can't do atomic renamesMove node_modules to a Docker volume
Postgres wouldn't startWindows mounts can't express Unix ownershipDocker volume for the data directory
Missing table mid-importcustomprofilepic wasn't in the original dumpDump and load it separately (510 images)

Four of those are upstream bugs in Discourse's vBulletin importer — it has not been updated for current TZInfo and Gemfile state. The patched copies are kept alongside .bak originals so the same fixes can be reapplied on the production build instead of rediscovered.

04

Why moving a forum is painful

The work took about four weeks. Three of those were planning and test runs; the final push ran from a Saturday to the following Tuesday. Almost none of that was the import itself.

A full move takes roughly three days, and it has to be done more than once. Each pass turns up damage the previous one hid, so the sequence is: run it, find what broke, fix it, run it again. The import is the cheap part. Finding out what it quietly got wrong is the expensive part.

PhaseRecordsWall clock
Users, avatars, profile images4,841~12 minutes
Topics and their first posts20,708~25 minutes
Posts1,892,172~26 hours
Private messages35,383~75 minutes
Closing topics20,708seconds, one bulk UPDATE
Post-processing1,927,108~15 hours
Permalinks, bans20,700minutes

Two full passes over 1.9 million posts, back to back, single-threaded. The importer is CPU-bound in Ruby at about twenty posts a second, on one core, with the database idle and fifteen others doing nothing. There is no parallelism switch. That is simply how long it takes.

The repairs cost more than the import. Every pass ended with hundreds of thousands of posts needing their text rewritten and then re-rendered: 355,515 posts for the nested quotes alone, 55,245 for BBCode, 401,719 rebakes. Each of those is another multi-hour single-threaded walk over the same data. The import finishes and the real work starts.

It also has to survive being interrupted, and it does: every record is stamped with its original vBulletin id as it lands, so a restart reloads that index and skips what already exists. Killing it loses nothing. That property is what made repeated passes affordable, and it is worth checking for before starting anything this long.

The other reason for the repeat runs is that damage does not announce itself. A row count that drifts by forty. A progress bar attributed to the wrong phase. Avatars present in the database and invisible on the page. None of it raises an error; all of it is found by looking at the result rather than the exit code.

05

Two ways it got stuck

Getting the data out

Before any of it could be imported it had to be copied off a 2006-era server, and scp lied about doing so. It reported success, exited zero, and delivered 209 MB of a 375 MB file. The truncation was only visible because the resulting dump would not import.

The fix was to stop trusting the transfer: split the file into chunks, checksum each one on both ends, and retry any that disagreed. Every large transfer in this project has been checksum-verified since, and that habit caught nothing else — which is the point. You do not find out you needed it until you did not do it.

The hang at 91%

Sixteen hours into a run, at 1,704,559 posts, the import stopped dead. It did not crash and did not error: the process stayed alive pinned at 100% CPU, holding an open Postgres transaction with 181 locks and no query issued for fifteen minutes. The log stopped growing at the same instant.

The open transaction named the topic, and walking the source table to that offset found the post responsible:

post 1776968, by "Sp4m"
length      99,545 characters
[IMG] tags  1,326   ← the same image URL, over and over

A joke post: one image repeated thirteen hundred times. The BBCode converter parses every tag and resolves it against the surrounding text, and on that input it went quadratic and never came back. Pinned CPU, no disk activity, no memory growth, no progress — catastrophic regex backtracking, which looks exactly like healthy work from the outside.

Collapsing the repeats in the staging database took the post from 99,545 characters to 188, keeping the first image and a note about what was removed. The post keeps its meaning and its place in the thread.

The post had already been deleted once. A moderator soft-deleted it in 2018 with the reason “forum breaking post”. It broke vBulletin too, seven years earlier, and it was still sitting in the database waiting to break the next thing that read it.

06

The guest posts

Deleting an account in vBulletin asks what to do with its posts, and the answer here was usually to keep them. Removing fifteen years of somebody's replies would gut the threads they appear in, leaving conversations that answer nobody. So the account went and the posts stayed: post.userid set to zero, the display name left sitting in post.username, and the thread still reading exactly as it always had.

That happened for two quite different reasons. Some people asked for their account to be removed, in which case it would be renamed first and then deleted, so the posts survived under a name that was no longer theirs. Others lost the account as a punishment. Either way the intent was the same: the person leaves, the conversation stays intact.

Discourse's importer reads only post.userid. When that lookup fails it falls through to the system account and the username in the very same database row is discarded.

# script/import_scripts/vbulletin.rb, line 428
user_id: user_id_from_imported_user_id(post["userid"]) || Discourse::SYSTEM_USER_ID
#                                                          ^ post["username"] is never read
 Count
Posts by deleted accounts71,761
… of which still carry the original name71,728
Distinct authors affected365
smuggo40,475
XenosisReaper24,247
Venec4,983

Two people account for 65,000 of them. On vBulletin those posts showed as written by smuggo and XenosisReaper; after import they show as written by the system account, with the names gone from threads that are still actively read.

In the vBulletin database

userid 0
username smuggo
Post 94: "Cyclone/Thrasher hulls. Yeah, fit arty and stuff. And MWDs…" — the account is gone, the name is not.

After import

author system
Same post, same text, no author. Repeated 71,728 times across the archive.

The deleted accounts are pre-existing. The lost names are not. Those accounts were removed years before this migration, deliberately, and nothing should bring them back. But keeping the posts readable was the whole point of how they were removed — the name was retained on purpose — and the import is what throws that away. Calling this purely a source-data problem was wrong.

The fix

The stored ids let the posts be joined back to post.username afterwards, with no re-import. For each of the 365 names an account is created and the posts are moved onto it. Threads then read the way they did on vBulletin.

The question is what kind of account. Discourse offers a staged user — an account that cannot be logged into but that anyone able to receive mail at its address can claim. That is the usual choice for imported guest content, and it is the wrong one here.

Every one of these accounts was deleted on purpose. Some at the owner's request, some as a punishment — but none of them by accident, and the database does not record which was which. Recreating those names as claimable accounts would quietly reverse every one of those decisions at once, and hand the names to whoever asks first. So all 365 are created suspended instead: the name stays attached to the posts, the account is visibly closed, and nobody can register into it. Each carries a note explaining that it was recreated during the migration, so any individual one can be reopened from the admin UI for somebody who asks.

./fix-orphan-authors.sh check   # read-only, shows what would change
./fix-orphan-authors.sh apply   # create the accounts and reassign

These 365 are not the same as the forum's ban list, and the two are easy to confuse. vBulletin's userban table holds exactly 15 rows — accounts that still exist and were banned. The importer's suspend_users phase applies those automatically, which is why the ban list needs no manual step.

The 365 are a different population entirely: accounts that were deleted, so no user row survives for a ban to point at. They exist now only as a name stamped on their old posts. They cannot appear in userban, and the importer cannot suspend them, because as far as it is concerned they were never users. They are created and suspended together by fix-orphan-authors.sh after the import finishes — 380 closed accounts in total, arrived at by two completely separate routes.

One name needed special handling. FraXy is the only one of the 365 who also still has a live account — seven posts under the surviving account, sixteen more orphaned under the same name. Creating a suspended duplicate would have split one person across two accounts, so the script detects that case and returns the orphaned posts to the existing account instead.

The script refuses to run while the importer is active, and re-running it is harmless. A handful of posts — 33 of the 71,761 — have no username recorded at all and stay on the system account, because there is nothing to restore.

07

The BBCode that never converted

This is the largest defect in the migration, and it went unnoticed for most of it because the import reports nothing wrong. Posts arrive, counts match, text is intact — and nearly half of them render their markup as visible text.

Quotes: 905,000 posts, and a lesson about reading the whole program

Discourse converts BBCode during import with ruby-bbcode-to-md. Its quote pattern matches lowercase [quote]. vBulletin's Reply‑With‑Quote button writes uppercase, which is how almost every quote on a fifteen-year-old forum was made. So mid-import the database looks like this:

[QUOTE=spasm;148785]The Indians would probably respond with pig-bombs.[/QUOTE]
← what a reader sees, verbatim, instead of a quote block

uppercase, untranslated .... 828,413 posts
lowercase, translated fine ..  87,282 posts

Nearly half the forum rendering as raw markup, with no error and every count reconciling. The obvious conclusion was that this is the migration's biggest defect and needs a repair script. A script was written, tested, and very nearly run.

It was not a defect. It was a phase that had not run yet. The importer has five stages after the one that creates posts, and one of them — post_process_posts — exists precisely to do this. Its regex is case-insensitive, and it does considerably more than case-folding:

[QUOTE=spasm;148785]…[/QUOTE]
        ↓
[quote="spasm,post:12,topic:4567"]…[/quote]

That is Discourse's native quote format: attributed, and clickable, jumping the reader to the post being quoted. To build it, the importer resolves the old vBulletin post id through the same import-id mapping used everywhere else. The same phase rewrites [THREAD], [THREAD=id], [POST] and [POST=id] into working internal URLs, and strips dead [attach] tags.

The repair script produced plain markdown blockquotes. Correct-looking, and strictly worse — no link, no jump target. Worse still, it would have consumed the [QUOTE=user;postid] tags that post_process_posts needs as input. Running it first would have permanently downgraded 905,000 quotes from linked to inert, and the designed phase would then have found nothing to do.

This is the trap worth knowing about. The importer does have real defects, and they present exactly like this one — same symptom, same scale, no error message. The difference was only visible by reading the part of the program that had not run yet. An unfinished process is not a broken one, and a repair applied to a half-finished pipeline can destroy the input the remaining half depends on.

Video: 53,662 posts

vBulletin 4's own video tag — [video=youtube;CUUiHQRu6Xo]…[/video] — is not in the converter at all. 48,897 YouTube, 4,165 of a second YouTube variant, 555 Vimeo, 45 Dailymotion. Discourse embeds a bare video URL sitting on its own line, so the repair is to reduce each tag to the URL it wrapped and let oneboxing do the rest.

Seven custom tags

No generic converter could have handled these: they were defined on this forum, and they live in a bbcode table that is not part of the content dump at all. Finding them meant going back to the source server and reading it directly.

These are the counts measured in Discourse after the import, restricted to tags with a matching closer — prose mentions of a tag and markdown links that merely look like one are excluded, which is why some numbers are lower than a naive text search suggests.

TagPostsTagsBecomes
[spoiler]39,56050,497Discourse [details] block
[video=…]5,1415,698Bare URL, oneboxed
[h5loop]5,1206,832Bare .webm URL
[h5video]3,4614,048Bare URL
[soundcloud]347412Bare URL, oneboxed
[strike]296394Markdown ~~strikethrough~~
[y2kpp]22Dropped — always empty

[y2kpp] is the smallest entry and the easiest decision: it was a one-shot tag built for a single event, an incursion into the forum's EVE Online community, and it has no use beyond it. Both surviving instances are [y2kpp][/y2kpp] with nothing inside. Dropped without ceremony.

[strike] carries a fingerprint of the older migration. Some instances read [strike:2z0scad4] — that suffix is a phpBB uid, attached to every tag phpBB parsed so it could tell real markup from text a user had typed. It survived the 2011 move into vBulletin and sat in the database untouched for fifteen years.

The spoiler tag is the one that matters. 40,265 posts used it, and its vBulletin definition is a block of inline JavaScript with a show/hide button — markup that means nothing to Discourse. It maps cleanly onto a native collapsible block.

What it took

55,245 posts were rewritten. Every media tag turned out to wrap a plain URL, so reducing each one to the URL on its own line lets Discourse's own link handling produce a real embed — no bespoke rendering needed:

[h5VIDEO]http://i.imgur.com/aTeNcGA.webm[/h5VIDEO]        →   http://i.imgur.com/aTeNcGA.webm
[SOUNDCLOUD]http://soundcloud.com/matas/hobnotropic[/…]  →   http://soundcloud.com/matas/hobnotropic
[video=youtube_share;CUUiHQRu6Xo]http://youtu.be/…[/video] →   http://youtu.be/CUUiHQRu6Xo
[strike:2z0scad4]Navy Raven (Jita - 560~)[/strike:…]   →   ~~Navy Raven (Jita - 560~)~~

The pattern has to refuse to match across a broken tag. Some posts contain [strike with no closing bracket — an author's typo, preserved for a decade. A plain non-greedy match runs straight past it to the next item's closing tag and deletes everything in between. One sale thread lost 806 characters that way before the guard was added. The same applies to spoilers, which nest several deep, so the conversion runs repeatedly and unwraps one layer at a time: 39,560 posts, then 1,190, then 297, then 79, down to a handful.

The quotes themselves were handled by the importer, not by this script — and handled better than a hand-written fixer could. This is a real post from the forum, four people quoting each other in a chain:

Before

[QUOTE=Nimbly Bimbly;635596][QUOTE=Eard;635546]
[QUOTE=Rep;635488][QUOTE=Hels;635478]Plan to
pick it up again after the standalone
releases.[/QUOTE][/QUOTE][/QUOTE][/QUOTE]

After

[quote="Nimbly Bimbly,post:412,topic:8891"]
[quote="Eard,post:408,topic:8891"]
[quote="Rep,post:405,topic:8891"]
[quote="Hels,post:403,topic:8891"]
Plan to pick it up again after the
standalone releases.
[/quote][/quote][/quote][/quote]

Each name resolves to the post it quotes, so the rendered quote is clickable and jumps to the original — the post ids come from the same import-id mapping used everywhere else. That is the part a hand-rolled converter could not have reproduced, and the reason for not pre-empting the importer with one.

The spoiler tag, including the uppercase variant the converter also missed:

Before

test
[spoiler]icle[/spoiler]

[SPOILER]testicle[/SPOILER]

After

test

[details="Spoiler"]
icle
[/details]

[details="Spoiler"]
testicle
[/details]

And video, where the tag is reduced to the URL it wrapped so that Discourse embeds it:

Before

PVP Video:

[video=youtube_share;CUUiHQRu6Xo]
http://youtu.be/CUUiHQRu6Xo[/video]

After

PVP Video:

http://youtu.be/CUUiHQRu6Xo

← renders as an embedded player

Not every media tag can become a bare URL. [soundcloud] wraps real SoundCloud track links, which Discourse embeds on sight. But [h5video] and [h5loop] wrap direct .webm file links — 6,275 of them on imgur, the rest scattered across gfycat, webmup and hosts that no longer exist. A bare .webm URL is not oneboxable, so imgur links are rewritten to the .gifv page form that Discourse does embed, and everything else becomes a labelled link rather than a dead-looking URL.

The script clears each post's baked_version, so Discourse re-renders them in the background afterwards.

08

What the crash left behind

The host had crashed mid-import days earlier. The import resumed and finished, but the database kept a souvenir. The first query run against the finished forum failed like this:

ERROR:  missing chunk number 0 for toast value 567848 in pg_toast_26316
CONTEXT:  parallel worker

Postgres stores oversized column values — here, post bodies — outside the main table in a side table called TOAST. This says a row points at content that is no longer there. It is the error people find when they search for whether their database is corrupt.

What made it confusing is that the same query worked when restricted to a range of post ids, and failed only when run against the whole table. That difference is the diagnosis: a narrow query uses an index and reads only live rows, while an unrestricted one falls back to a sequential scan and walks every physical row on disk, including dead ones no query can return.

The damage was smaller than the error suggests. A row-by-row walk of all 1,927,108 posts — each read individually, failures caught and counted — found zero unreadable live posts. One genuinely broken post surfaced first and was restored from the source database. Everything else was dead tuples: rows already superseded, waiting to be reclaimed, which only a full scan ever touches.

scanned 1927108 rows, 0 bad

The practical effect is narrow. Discourse itself never trips over it, because Discourse queries by id and stays on the index. It only appears when something sweeps the whole table — which is exactly what measuring the leftover BBCode was about to do.

09

Counting the quotes correctly

With the import finished, the obvious question is how many quotes post_process_posts actually fixed. The obvious query gives a frightening answer:

posts matching [quote=   928,069   # unchanged from before the phase ran

Which would mean fifteen hours of post-processing had achieved nothing. It had not. The pattern matches both formats — the vBulletin tag it was looking for and the Discourse tag it had been rewritten into. The converted form puts a double quote straight after quote=; the vBulletin form never does, and that one character separates them.

So the phase worked, and a second query said 13,574 posts still held raw tags — 98.5% converted, a rounding error's worth of work left. That number was wrong by a factor of twenty-five, and the way it was wrong is worth more than the number.

A query that could not see the problem

The count asked for posts holding a vBulletin tag and no converted one:

raw ILIKE '%[quote=%'  AND  raw NOT LIKE '%[quote="%'

That second condition sounds like a sensible way to exclude finished work. It is in fact a way to exclude the evidence. The importer converts only the outermost quote in a post and stops, so a quote-of-a-quote ends up holding both forms at once:

[quote="Frug,post:151,topic:18768"]              converted
[QUOTE=erichkknaar;1695312]                      left as literal text
[QUOTE=Approaching Walrus;1695153]have some pure ideology
[/QUOTE][/QUOTE]

Every post like that contains [quote=", so the query threw it away. What it measured was the small population of posts where the importer had converted nothing at all — and reported that as the whole problem. Dropping the second condition changed the answer from 13,574 to 350,254.

This is the second time in this project a count was wrong in the same direction. Earlier a progress counter was attributed to the phase that had announced itself rather than the phase producing it. Both errors share a shape: a measurement built on an assumption about the thing being measured, which then cannot contradict that assumption. The check that catches it is not a better query — it is looking at a handful of actual posts.

What was actually left

Resolving them works the same way the importer does: look the vBulletin post id up in the import-id map and build a linked quote, falling back to a plain attributed one where the target was never imported.

posts changed          355,515
tags → linked quote     627,137
tags → plain quote        22,885
stray [/QUOTE] closers   1,046,087

1,871 tags carry no post id at all — [quote=Wired News], [quote=the article] — because they quote a source rather than a member. There is nothing to link to, and they become plain attributed blockquotes, which is what they always were.

179 posts still match the pattern and none of them should change: their [quote= sits inside a code block, where it is content rather than markup.

The videos were links, not videos

Converting the video tag to the URL it wrapped was correct, but vBulletin had stored those URLs without a scheme — //youtu.be/WSKi8HfcxEk. Discourse only embeds a URL it can resolve, so 51,326 posts rendered a bare blue link where a player belonged. Restoring https: turned every one of them back into a video.

10

The setting that eats accounts

vBulletin had 4,858 users. Discourse ended up with 4,840. The eighteen missing accounts left no error, no warning and no log line — the importer simply did not create them.

Twelve were lost to a single setting. Discourse's normalize_emails defaults to true, which strips Gmail dots and +tags before comparing addresses. Two accounts that vBulletin considered entirely separate look identical to it:

58644  jsmith      [email protected]   ✓ imported
58645  jsmith2     [email protected]   ✗ silently skipped
58647  jsmith_b    [email protected]   ✗ silently skipped
58696  smithj      [email protected]   ✗ silently skipped

all four normalize to  [email protected]

The other six collided on username instead — different people who happened to share a name with someone already imported. Between them the eighteen account for 1,279 posts, which landed either on the system account or on the same-named account that won the race.

All eighteen were recreated afterwards, keeping their real vBulletin address and join date, and their posts moved back onto them. It is a small job — two of the eighteen carried almost all those posts and had already been recovered by hand, leaving sixteen accounts and 107 posts — but the alternative was leaving sixteen people's authorship wrong to avoid touching a database that had already been reviewed. Every one of the 4,858 vBulletin accounts now exists.

The fix is to turn the setting off for the import and back on afterwards. normalize_emails is doing something useful in normal operation — it stops one person farming unlimited accounts from a single mailbox. It is only wrong during a migration, where those aliases are not abuse but fifteen years of accumulated history, already accepted by the software people were actually using.

A ban evasion the merge would have erased

One of the twelve is worth telling in full, because it is the clearest illustration of what gets lost. Two accounts, same person, and both banned:

vB idUsernamePostsBannedReason given
1496Rakshasa The Cat12,4602017‑10‑02“3 in a row laters, feel free to start again”
80180Honk Honkler172019‑02‑16“Hello Rakshasha”

Banned in 2017 with an explicit invitation to come back. Returned sixteen months later under [email protected] — a plus-alias of the original address — and was recognised after seventeen posts. The second ban reason is two words long and makes the whole story legible.

Because the alias normalized to the original, the second account was never created and its seventeen posts fell to the system account. Both bans collapse into one, and the return, the alias and the recognition all disappear — leaving a single 2017 ban and no evidence anything else ever happened.

Splitting them back apart is straightforward once the cause is known, because nothing had to be moved off the surviving account: the posts were never merged onto it in the first place. Recreate the second account from its vBulletin id, move its seventeen posts across, and apply its own ban with its own date and reason. The only obstacle is that normalize_emails blocks the fix for the same reason it caused the problem — so it is switched off for the duration of the operation and restored immediately afterwards.

11

Ten videos, recovered sideways

A couple of hundred posts held a video tag that was simply broken:

[video=youtu /video]        [video=  video]        [video=y  eo]

Checking the source database showed the damage was not caused by the migration — vBulletin had been displaying these as literal text for years. The provider name and video id were gone. The obvious conclusion is that nothing can be done, and the obvious conclusion is wrong.

Look at where the broken tags sit:

[QUOTE=Jack bubu;164505][video=youtu /video]

:O[/QUOTE]

Every one is inside a quote. These were not posts that contained a video — they were posts quoting a post that contained a video, and vBulletin's Reply‑With‑Quote mangled the tag while copying it. The original is untouched, and the quote records exactly which post it came from.

post 164505 (the original, intact):
  [video=youtube;BpA6TC0T_Lw]http://www.youtube.com/watch?v=BpA6TC0T_Lw[/video]

So the id is recoverable: read the quoted post id out of the broken post, look up that post in the source database, and pull the video id from its intact tag. Ten videos came back that way, each restored to the quote that had lost it. Five more could not be — the post they quoted had no video tag of its own, having been mangled further up a longer chain.

This was nearly written off. The first pass concluded the ids were unrecoverable and stripped the broken markup, which was defensible and also wrong — the check that mattered, whether the id survived somewhere else, had not been run. Damaged data is worth one more look before it is discarded, particularly on a forum where quoting is how every conversation works: anything worth quoting exists in more than one place.

12

Two things that were there all along

A fingerprint from 2011

4,370 posts carried tags that looked like this:

[quote:2u7yqhbz]...[/quote:2u7yqhbz]        [spoiler:3p26ccso]...[/spoiler:3p26ccso]

That suffix is a phpBB uid. phpBB stamped every tag it parsed with a short random string, one per post, so its renderer could tell markup it had produced from text a user had merely typed. The suffixes came across in the 2011 migration, meant nothing to vBulletin, and sat in the database being displayed as literal text for fifteen years. Removing the suffix leaves an ordinary tag, which the conversions used everywhere else then handle.

These are the oldest posts on the forum, and their markup is the last trace of the software it ran on before.

The avatars that were imported and invisible

The import brought 1,207 avatars across, and the files sat correctly on disk. The forum showed letter circles for everyone.

Discourse does not serve an avatar from the uploaded file. It serves a resized copy from an optimized_images table, generated by a background job the first time one is needed. That job runs under Sidekiq, and nothing had started Sidekiq on a machine being used for repair work. The table was empty, so every avatar request fell through to the fallback.

“1,207 avatars imported” was true and useless. The data was present, correct, and completely invisible. It is the same trap as the quote count in section 9 from the opposite direction: there, a query said the work was done when it was not; here, the database said the data was there when nobody could see it. Neither is caught by querying harder. Both are caught by looking at the page.

Generating the 10,827 thumbnails directly took a few minutes. On the production machine none of this applies — Sidekiq runs, and the job does it unprompted.

13

The videos that would not play

Every YouTube link on the forum rendered as a plain blue link. The URL was right, the markup Discourse produced was right, and no video ever appeared.

The obvious conclusion is that something in the imported posts is malformed, and that conclusion survived three rounds of increasingly precise edits to the stored HTML — each one comparing the imported markup against what Discourse generates itself, finding a small difference, correcting it, and changing nothing on screen.

The test that ended it took thirty seconds. Post a new YouTube URL through Discourse, as a user, in a fresh topic. It rendered as a bare link too. Nothing imported was involved, so nothing imported was at fault — this instance could not onebox anything. Three rounds of work had been spent correcting data against a broken baseline.

Two environmental causes, neither in the data:

Sidekiq was not running. Resolving a onebox — fetching a video's title and thumbnail — is a background job. Without the worker, Discourse marks a link for oneboxing and never finishes the job, leaving exactly the bare link that was showing.

Plugin JavaScript had never been built. The frontend was built earlier with pnpm build, which builds Discourse core and deliberately refuses to import plugin modules — plugins compile through a separate task:

rake assets:precompile:build_plugins        # 44 plugins, 64 seconds

Without it discourse-lazy-videos never loads, so nothing turns a video container into a player. The container was in the page the whole time, inert.

Both are artefacts of running repair work on a development image. The production container compiles plugin assets during bootstrap and supervises Sidekiq, so neither failure can occur there — which is the argument for deploying onto the official image rather than promoting the machine the repairs were done on.

A million jobs that were not a problem

Starting Sidekiq immediately produced an admin warning: 1,386,127 queued jobs, almost all of them PullHotlinkedImages — Discourse offering to download and re-host every externally linked image on the forum.

That reads like days of crawling and a serious decision about link rot. It is neither. The queue holds one job per post per rebake, and the posts had been rebaked several times; nearly every job opens a post, finds nothing external, and exits. Counting the actual images gives a very different picture:

external image URLs in cooked posts   47,783
  — of which img.youtube.com          46,997   # thumbnails generated locally, regenerable
  — genuinely external                   786   # across 426 posts

786 images, in 426 posts — imgur albums, CCP dev blogs, personal sites. Those are the ones that vanish when a host lapses, and they are worth keeping. Running them directly took minutes rather than days, and the rest of the queue was discarded: clearing it removes no data, since the URLs stay in the posts and the images keep loading from wherever they live. The job only ever made local copies.

14

The passwords nobody knows

vBulletin stored passwords as md5(md5(password) + salt). Discourse uses PBKDF2 and cannot verify a vBulletin hash, so no old password can survive a migration. What is easy to miss is what the importer does instead.

vBulletin row      md5hash, salt
importer builds    "md5hash:salt"     # joined, and passed as the PLAINTEXT password
Discourse stores   pbkdf2(that string, 600,000 iterations)

The hash and salt are concatenated and handed to Discourse as though a user had typed them. Discourse dutifully hashes the result. Every one of the 4,841 accounts ends up with a valid, working credential — one derived from the old database and known to nobody, not even its owner.

The practical effect is not a security hole, but it is not nothing either. No old password opens an account, because nobody can type a 32‑character MD5 digest they have never seen. But the credential rows are live rather than absent, and they are a deterministic function of a database that has been dumped, copied and archived several times over. Anyone holding an old dump holds the input.

So the credentials were deleted outright rather than left in place. All 4,842 rows are gone, along with every session token. Every account now has no password at all, and the only way into one is a reset link sent to its registered address — which is the intended route for a forum that has been offline for a year regardless.

=== deleting all user_passwords rows ===
DELETE 4842
  credentials remaining: 0
  expiring all active sessions

This ran last, after an administrator account existed — wiping every password on a forum with no admin is a memorable way to lock yourself out of your own site.

15

Where things stand

The import is finished and the repairs are done. What follows is what came out of it, and what is now running in production.

 CountNote
Posts1,927,109Every post from every live thread
Topics45,11520,708 threads + 24,346 private conversations
Categories65One per vBulletin forum
Users5,172All 4,858 vBulletin accounts, plus 314 recreated from guest posts
Suspended32914 from the ban list, the rest deleted accounts
Subscriptions292,900217,074 tracking, 75,826 watching
Polls300Results appended to their opening posts
Stored passwords0Deliberately — see section 11

What the repairs actually moved

None of this needed a re-import — the stored ids made every one of these a matter of joining back to the old database and correcting what was found.

RepairScopeOutcome
Quote tags left unconverted355,515 posts627,137 tags became linked quotes
Videos stored without a scheme51,326 postsBare links became embedded players
phpBB uid suffixes4,370 posts[quote:2u7yqhbz] and friends, from 2011
Avatar thumbnails10,827Generated for 1,207 users
Hotlinked images archived656Across 426 posts, against future link rot
BBCode with no converter55,245 postsspoiler, video, h5loop, h5video, soundcloud, strike
Guest-post authorship~71,000 posts365 names restored as suspended accounts
Thread subscriptions234,258Of 234,428; the rest reference deleted users
Poll results300Of 303; 3 threads were not imported
Bans14Applied by hand after the phase crashed
Collided accounts4Two name collisions and one alias, separated
Truncated video tags10URLs recovered from the posts they quoted
Accounts lost to email normalizing16Recreated with 93 of their posts
Old-URL redirects20,700showthread.php links resolve again

Every one of the 4,858 vBulletin accounts now exists in Discourse. 10,076 posts remain on the system account — the ones vBulletin recorded no name for at all, anonymous in the source data and equally anonymous now.

The BBCode needed several passes, not one. Spoiler tags nest, and the pattern that converts them is deliberately forbidden from matching across another spoiler tag — without that guard, a malformed tag lets the match run on to the next closer and swallow everything between. So each pass unwraps one layer: 39,560 posts, then 1,190, then 297, then 79, down to a handful. Converging slowly is the correct behaviour here; matching greedily would have been fast and would have eaten text.

Old links still work

The importer writes a mapping of every old thread id to its new topic, but with URLs (thread/3) that nothing on the internet actually links to. This forum used vBulletin's SEO-friendly form, so the shapes that need to resolve are:

/showthread.php?3332-GANK-NIGHT-GN-31-Crowes-and-Bummers-II
/showthread.php?t=3332
/showthread.php?9095-R.I.P.-SHC&p=206497&viewfull=1

Discourse rewrites incoming URLs through a permalink_normalizations setting before looking them up, so one rule per shape is enough — with one catch worth recording, because it is not documented anywhere obvious. The setting is split on | and the boundary between pattern and replacement is found by counting unescaped /. A rule can therefore contain neither an alternation nor a slash in its replacement, which makes thread/ impossible to express. Renaming the stored permalinks to thread-3332 sidesteps it entirely.

All three shapes now return a 301 to the right topic, so fifteen years of links from elsewhere survive the move.

16

Going live

The forum has been live at failheap-challenge.com since 25 August 2026, on a four-core machine behind HAProxy behind Cloudflare. The deployment itself was the last day of a four-week job, and three things went wrong during it that are worth writing down, because each of them is invisible until it bites.

The bootstrap failed on a plugin that no longer needs installing

The container build aborted with git clone exiting 128:

fatal: destination path 'discourse-solved' already exists and is not an empty directory

That plugin now ships with Discourse core. Cloning it into a directory that already exists kills the whole bootstrap, forty minutes in. The fix is a one-word guard — test -d <dir> || git clone … — which also means the build cannot break the same way when upstream absorbs the next plugin.

The database would not dump

Then pg_dump refused to finish, on two different tables:

ERROR:  missing chunk number 0 for toast value 567834 in pg_toast_28637

This is the same host crash from section 8, and it was worse than it had looked. A row-by-row walk of all 1.93 million posts read cleanly — every live row was fine. But REINDEX gave the real answer:

ERROR:  could not create unique index "posts_pkey"
DETAIL:  Key (id)=(157942) is duplicated.

The table held 42 duplicated ids — 1,927,151 physical rows where the primary key allowed 1,927,109 — and the index itself was corrupt, which is why index scans could return the same row twice and why every attempt to copy the table failed. Each duplicate had one readable copy and one whose content was unreachable. Keeping the readable copy, identified by physical row address, and rebuilding the index brought the count back to exactly what the primary key expected.

This had been quietly distorting things for hours. Row counts drifted by small amounts across the session and were written off as rounding between different queries. They were not: the table genuinely contained more rows than it should. A count that will not settle is worth chasing, not explaining away.

Everything a database dump does not contain

The restore brought 1.93 million posts across intact, and the forum still looked wrong, three times over:

That last one is the general lesson. Restoring a database overwrites every site setting. The development database had email switched off and registration open; both came across and silently undid deliberate production decisions. It happened twice in one evening before the pattern was obvious.

997 emails

Everyone's password had been deliberately deleted, so the only way back in was a reset link. That made the announcement email load-bearing rather than decorative.

It went to 997 people in one batch: everyone who had logged in within five years, plus everyone who joined in the forum's first five years and wrote more than a hundred posts. One message failed on a network timeout and was retried.

 Rate 
Delivered92.56%~930 people reached
Failed6.21%~62 addresses dead after fifteen years
Spam complaints0%the number that protects the domain

One line in the draft claimed “your account is still there, under the same name”. Checking it before sending showed that 1,104 of 4,858 accounts — 22.7% — had been renamed, almost all spaces becoming underscores. More than one recipient in five would have read that sentence and found it false. It was replaced with an explanation of what changed and why their posts are still attached either way.

Bounces now feed back automatically: a Mailgun webhook tells Discourse when an address fails, and two hard bounces disable it. Those 62 dead addresses will retire themselves rather than being mailed forever.