Migration log · August 2026
Discourse ships an official vBulletin import script and there is a well-known community guide for it. Both were the starting point. Neither survived contact with a real forum of this age — the script does not run at all on current Discourse without patching, and several things it does import, it imports wrongly.
I tried this on my own first and got nowhere. What finally moved it was working through the whole thing with Claude: four weeks of planning, test runs and false starts, then a four-day push to get it live. What follows is what actually broke and how it was fixed.
The source database declared every text column latin1_swedish_ci. The bytes inside
those columns were UTF‑8.
Nobody chose this. latin1_swedish_ci was MySQL's compiled-in default from 4.1
(2004) until 8.0 (2018), and this server runs 5.0.95 — so every
CREATE TABLE that did not name a charset silently got latin1. Neither phpBB nor
vBulletin ever named one:
tables at latin1_swedish_ci 204 tables at utf8_general_ci 1 # tsx_server, a 2014 add-on columns with explicit non-latin1 4 # all four in that same add-on table
The post table makes it plain — pagetext carries no
CHARACTER SET clause of its own, and the table ends with a bare
DEFAULT CHARSET=latin1 it inherited from the server. In 204 tables the forum software
specified a charset exactly zero times; the only table that did came from a third party, three
years after the migration.
Both applications got away with it because each stored UTF‑8 bytes and told MySQL the connection was latin1. MySQL passed the bytes through untouched and handed back exactly what it was given. Byte in, byte out — the declared charset never mattered, because nothing in the chain ever tried to interpret it. That held for fifteen years.
The UTF‑8 bytes themselves arrived with the phpBB import, and the database can date that
precisely. vBulletin's import*id columns sit empty on a forum that was never imported
into; here they are populated on 133,388 posts and 1,986 users,
and the seam is sharp enough to date to the hour:
last post carrying an import id 2011-07-01 21:20:56 first post with none 2011-07-01 22:34:06 # 73 minutes later imported content spans 2011-04-09 → 2011-07-01 # ~12 weeks native vBulletin content spans 2011-07-01 → 2026-06-24 # everything since
So the forum ran roughly three months on phpBB and moved to vBulletin on 1 July 2011, carrying its history across. The oldest posts in the archive — a thread where people are racing to reply "second" and arguing about "'11s" — are the phpBB forum's opening day, preserved through the move with their original timestamps.
Left alone this is harmless — vBulletin read back the same bytes it wrote. It becomes
destructive the moment you run mysqldump normally, because MySQL then "helpfully"
re-encodes on the way out.
utf8, so nothing errors. The corruption is silent and permanent.A uniform mislabelling is recoverable with one rule. A mixed encoding — some rows genuinely latin1, others UTF‑8 — is not, and always loses something. So the first job was proving which case this was.
-- how many posts contain any high byte? SELECT COUNT(*) FROM post WHERE CONVERT(pagetext USING BINARY) RLIKE CONCAT('[', UNHEX('80'), '-', UNHEX('FF'), ']'); -- 43,125 -- of those, how many are valid UTF-8 when reinterpreted? SELECT COUNT(*) FROM post WHERE CONVERT(pagetext USING BINARY) RLIKE CONCAT('[', UNHEX('80'), '-', UNHEX('FF'), ']') AND CONVERT(CONVERT(pagetext USING BINARY) USING utf8) IS NOT NULL; -- 43,125 ← every single one. Zero exceptions.
Both numbers matching is the green light: the whole archive is UTF‑8 wearing the wrong label, and one uniform rule fixes it.
The fix. Dump with byte passthrough so MySQL never converts anything, then rewrite the charset declarations while loading into a modern database:
# on the source mysqldump --default-character-set=latin1 --skip-set-charset ... # while loading sed 's/DEFAULT CHARSET=latin1/DEFAULT CHARSET=utf8mb4/g'
Three earlier dumps sat in /root on the old server. All three carry
SET NAMES utf8, meaning they were taken without the passthrough flags. Verified
byte-for-byte against the new dump: they are double-encoded throughout. Importing from any of them
would have baked é into 43,125 posts permanently.
Verification, not assumption. After loading, a search across 1.22 million imported posts for the classic mojibake signatures returned one hit — and that post is corrupted in the source vBulletin too, from someone pasting broken text years ago. Nothing was damaged in transit.
A French post from 2012, as it survives in the correct dump, and as the same sentence appears in the three dumps that were already sitting on the server:
C3 A9, correctly
labelled as UTF‑8 on load.C3 83 C2 A9. Every accented character in the archive, wrong.Those three dumps were the obvious thing to import from — they were already made, already on disk, and nothing about them looks broken until you go looking for a specific character. Using any of them would have baked that second column into 43,125 posts permanently.
| Content | In source | Importable | Note |
|---|---|---|---|
| Posts | 1,892,322 | 1,873,461 | Difference is deleted content |
| Threads | 20,791 | 20,708 | 83 soft-deleted, 49 redirect stubs |
| Users | 4,858 | 4,841 | 17 skipped — duplicate/invalid emails |
| Forums → categories | 65 | 65 | All imported |
| Private messages | 35,383 | 35,383 | |
| Custom avatars | 1,211 | 1,211 | 30 MB of blobs, all intact |
| Profile backgrounds | 510 | 510 | Needed a second dump pass |
| Attachments | 0 | 0 | Never enabled — not data loss |
Roughly half the database was junk that should never be migrated: 382 MB of spam-filter logs, 977 MB of search index that Discourse rebuilds itself, plus edit history and BBCode caches. Dumping only the 36 content tables cut 1.4 GB of dead weight.
Deleted content is why progress bars stop short of 100%. 83 threads were soft-deleted in vBulletin — flagged as removed but left in the table. The importer correctly skips them, so counting against the raw table total makes a finished import look stuck at 99.9%. The real target is 20,708 threads and 1,873,461 posts.
Discourse's importer is not a standalone converter — it is a Ruby script that runs inside a working Discourse instance, reading from a live MySQL copy of the old forum. So both halves have to exist at once.
vbmysql containerMySQL 8.0.46 · utf8mb4
Holds the corrected copy of the vBulletin database. MyISAM tables converted to InnoDB on load.
discourse containerRuby 3.4 · PG 18 · Redis
Runs the importer. Postgres data and node_modules live on Docker volumes, not Windows bind mounts.
An OpenSSH from 2006 sounds worse than it was. Re-enabling SHA‑1 key exchange to reach a nineteen-year-old daemon is the kind of thing that deserves a second look, and it got one. That server accepted keys only — no password authentication to brute‑force — and its SSH port was reachable only from whitelisted addresses at the firewall, so the internet could not talk to it at all. The version was old; the exposure was not. Most of the vulnerabilities that make people flinch at OpenSSH 4.3 need either a password prompt or the ability to reach the port.
The forum's actual problem was never SSH. It was a public web application on an operating system that stopped receiving security updates in 2017, being scanned continuously by bots that got past Cloudflare.
| Problem | Cause | Fix |
|---|---|---|
| SSH refused to connect | OpenSSH 9 dropped SHA‑1 KEX; server offers nothing else | Re-enable diffie-hellman-group14-sha1, ssh-rsa |
| Transfer silently truncated | scp reported success after sending 209 MB of 375 MB | Split into chunks, verify each by MD5, retry |
mysql2 gem wouldn't build | No MySQL client headers in the image | apt-get install default-libmysqlclient-dev |
| Bundler refused to resolve | Discourse's own Gemfile declares sqlite3 twice with conflicting pins | Remove the duplicate |
| Importer crashed on first user | TZInfo 2.x rejects integer timestamps; the script still passes them | Convert to Time first |
| Crashed again on user 2 | Some accounts have a NULL last-visit date | Make the timestamp parser nil-safe |
pnpm install failed | Windows bind mounts can't do atomic renames | Move node_modules to a Docker volume |
| Postgres wouldn't start | Windows mounts can't express Unix ownership | Docker volume for the data directory |
| Missing table mid-import | customprofilepic wasn't in the original dump | Dump and load it separately (510 images) |
Four of those are upstream bugs in Discourse's vBulletin importer — it
has not been updated for current TZInfo and Gemfile state. The patched copies are kept alongside
.bak originals so the same fixes can be reapplied on the production build instead of
rediscovered.
The work took about four weeks. Three of those were planning and test runs; the final push ran from a Saturday to the following Tuesday. Almost none of that was the import itself.
A full move takes roughly three days, and it has to be done more than once. Each pass turns up damage the previous one hid, so the sequence is: run it, find what broke, fix it, run it again. The import is the cheap part. Finding out what it quietly got wrong is the expensive part.
| Phase | Records | Wall clock |
|---|---|---|
| Users, avatars, profile images | 4,841 | ~12 minutes |
| Topics and their first posts | 20,708 | ~25 minutes |
| Posts | 1,892,172 | ~26 hours |
| Private messages | 35,383 | ~75 minutes |
| Closing topics | 20,708 | seconds, one bulk UPDATE |
| Post-processing | 1,927,108 | ~15 hours |
| Permalinks, bans | 20,700 | minutes |
Two full passes over 1.9 million posts, back to back, single-threaded. The importer is CPU-bound in Ruby at about twenty posts a second, on one core, with the database idle and fifteen others doing nothing. There is no parallelism switch. That is simply how long it takes.
The repairs cost more than the import. Every pass ended with hundreds of thousands of posts needing their text rewritten and then re-rendered: 355,515 posts for the nested quotes alone, 55,245 for BBCode, 401,719 rebakes. Each of those is another multi-hour single-threaded walk over the same data. The import finishes and the real work starts.
It also has to survive being interrupted, and it does: every record is stamped with its original vBulletin id as it lands, so a restart reloads that index and skips what already exists. Killing it loses nothing. That property is what made repeated passes affordable, and it is worth checking for before starting anything this long.
The other reason for the repeat runs is that damage does not announce itself. A row count that drifts by forty. A progress bar attributed to the wrong phase. Avatars present in the database and invisible on the page. None of it raises an error; all of it is found by looking at the result rather than the exit code.
Before any of it could be imported it had to be copied off a 2006-era server, and
scp lied about doing so. It reported success, exited zero, and delivered
209 MB of a 375 MB file. The truncation was only visible because the
resulting dump would not import.
The fix was to stop trusting the transfer: split the file into chunks, checksum each one on both ends, and retry any that disagreed. Every large transfer in this project has been checksum-verified since, and that habit caught nothing else — which is the point. You do not find out you needed it until you did not do it.
Sixteen hours into a run, at 1,704,559 posts, the import stopped dead. It did not crash and did not error: the process stayed alive pinned at 100% CPU, holding an open Postgres transaction with 181 locks and no query issued for fifteen minutes. The log stopped growing at the same instant.
The open transaction named the topic, and walking the source table to that offset found the post responsible:
post 1776968, by "Sp4m"
length 99,545 characters
[IMG] tags 1,326 ← the same image URL, over and over
A joke post: one image repeated thirteen hundred times. The BBCode converter parses every tag and resolves it against the surrounding text, and on that input it went quadratic and never came back. Pinned CPU, no disk activity, no memory growth, no progress — catastrophic regex backtracking, which looks exactly like healthy work from the outside.
Collapsing the repeats in the staging database took the post from 99,545 characters to 188, keeping the first image and a note about what was removed. The post keeps its meaning and its place in the thread.
The post had already been deleted once. A moderator soft-deleted it in 2018 with the reason “forum breaking post”. It broke vBulletin too, seven years earlier, and it was still sitting in the database waiting to break the next thing that read it.
Deleting an account in vBulletin asks what to do with its posts, and the answer here was
usually to keep them. Removing fifteen years of somebody's replies would gut the threads they
appear in, leaving conversations that answer nobody. So the account went and the posts stayed:
post.userid set to zero, the display name left sitting in
post.username, and the thread still reading exactly as it always had.
That happened for two quite different reasons. Some people asked for their account to be removed, in which case it would be renamed first and then deleted, so the posts survived under a name that was no longer theirs. Others lost the account as a punishment. Either way the intent was the same: the person leaves, the conversation stays intact.
Discourse's importer reads only post.userid. When that lookup fails it falls
through to the system account and the username in the very same database row is discarded.
# script/import_scripts/vbulletin.rb, line 428 user_id: user_id_from_imported_user_id(post["userid"]) || Discourse::SYSTEM_USER_ID # ^ post["username"] is never read
| Count | |
|---|---|
| Posts by deleted accounts | 71,761 |
| … of which still carry the original name | 71,728 |
| Distinct authors affected | 365 |
smuggo | 40,475 |
XenosisReaper | 24,247 |
Venec | 4,983 |
Two people account for 65,000 of them. On vBulletin those posts showed as written by
smuggo and XenosisReaper; after import they show as written by the
system account, with the names gone from threads that are still actively read.
The deleted accounts are pre-existing. The lost names are not. Those accounts were removed years before this migration, deliberately, and nothing should bring them back. But keeping the posts readable was the whole point of how they were removed — the name was retained on purpose — and the import is what throws that away. Calling this purely a source-data problem was wrong.
The stored ids let the posts be joined back to post.username afterwards, with no
re-import. For each of the 365 names an account is created and the posts are moved onto it. Threads then read the way
they did on vBulletin.
The question is what kind of account. Discourse offers a staged user — an account that cannot be logged into but that anyone able to receive mail at its address can claim. That is the usual choice for imported guest content, and it is the wrong one here.
Every one of these accounts was deleted on purpose. Some at the owner's request, some as a punishment — but none of them by accident, and the database does not record which was which. Recreating those names as claimable accounts would quietly reverse every one of those decisions at once, and hand the names to whoever asks first. So all 365 are created suspended instead: the name stays attached to the posts, the account is visibly closed, and nobody can register into it. Each carries a note explaining that it was recreated during the migration, so any individual one can be reopened from the admin UI for somebody who asks.
./fix-orphan-authors.sh check # read-only, shows what would change ./fix-orphan-authors.sh apply # create the accounts and reassign
These 365 are not the same as the forum's ban list, and the two are easy to
confuse. vBulletin's userban table holds exactly 15 rows
— accounts that still exist and were banned. The importer's suspend_users
phase applies those automatically, which is why the ban list needs no manual step.
The 365 are a different population entirely: accounts that were deleted, so no user
row survives for a ban to point at. They exist now only as a name stamped on their old posts.
They cannot appear in userban, and the importer cannot suspend them, because as far
as it is concerned they were never users. They are created and suspended together by
fix-orphan-authors.sh after the import finishes — 380 closed accounts in
total, arrived at by two completely separate routes.
One name needed special handling. FraXy is the only one of the 365 who
also still has a live account — seven posts under the surviving account, sixteen
more orphaned under the same name. Creating a suspended duplicate would have split one person
across two accounts, so the script detects that case and returns the orphaned posts to the
existing account instead.
The script refuses to run while the importer is active, and re-running it is harmless. A handful of posts — 33 of the 71,761 — have no username recorded at all and stay on the system account, because there is nothing to restore.
This is the largest defect in the migration, and it went unnoticed for most of it because the import reports nothing wrong. Posts arrive, counts match, text is intact — and nearly half of them render their markup as visible text.
Discourse converts BBCode during import with ruby-bbcode-to-md. Its quote pattern
matches lowercase [quote]. vBulletin's Reply‑With‑Quote button writes
uppercase, which is how almost every quote on a fifteen-year-old forum was made.
So mid-import the database looks like this:
[QUOTE=spasm;148785]The Indians would probably respond with pig-bombs.[/QUOTE] ← what a reader sees, verbatim, instead of a quote block uppercase, untranslated .... 828,413 posts lowercase, translated fine .. 87,282 posts
Nearly half the forum rendering as raw markup, with no error and every count reconciling. The obvious conclusion was that this is the migration's biggest defect and needs a repair script. A script was written, tested, and very nearly run.
It was not a defect. It was a phase that had not run yet. The importer has
five stages after the one that creates posts, and one of them —
post_process_posts — exists precisely to do this. Its regex is
case-insensitive, and it does considerably more than case-folding:
[QUOTE=spasm;148785]…[/QUOTE]
↓
[quote="spasm,post:12,topic:4567"]…[/quote]
That is Discourse's native quote format: attributed, and
clickable, jumping the reader to the post being quoted. To build it, the importer resolves
the old vBulletin post id through the same import-id mapping used everywhere else. The same
phase rewrites [THREAD], [THREAD=id], [POST] and
[POST=id] into working internal URLs, and strips dead [attach] tags.
The repair script produced plain markdown blockquotes. Correct-looking, and strictly worse
— no link, no jump target. Worse still, it would have consumed the
[QUOTE=user;postid] tags that post_process_posts needs as input.
Running it first would have permanently downgraded 905,000 quotes from linked to inert, and the
designed phase would then have found nothing to do.
This is the trap worth knowing about. The importer does have real defects, and they present exactly like this one — same symptom, same scale, no error message. The difference was only visible by reading the part of the program that had not run yet. An unfinished process is not a broken one, and a repair applied to a half-finished pipeline can destroy the input the remaining half depends on.
vBulletin 4's own video tag —
[video=youtube;CUUiHQRu6Xo]…[/video] — is not in the converter at all.
48,897 YouTube, 4,165 of a second YouTube variant, 555 Vimeo, 45 Dailymotion. Discourse embeds a
bare video URL sitting on its own line, so the repair is to reduce each tag to the URL it wrapped
and let oneboxing do the rest.
No generic converter could have handled these: they were defined on this forum, and they live
in a bbcode table that is not part of the content dump at all. Finding them meant
going back to the source server and reading it directly.
These are the counts measured in Discourse after the import, restricted to tags with a matching closer — prose mentions of a tag and markdown links that merely look like one are excluded, which is why some numbers are lower than a naive text search suggests.
| Tag | Posts | Tags | Becomes |
|---|---|---|---|
[spoiler] | 39,560 | 50,497 | Discourse [details] block |
[video=…] | 5,141 | 5,698 | Bare URL, oneboxed |
[h5loop] | 5,120 | 6,832 | Bare .webm URL |
[h5video] | 3,461 | 4,048 | Bare URL |
[soundcloud] | 347 | 412 | Bare URL, oneboxed |
[strike] | 296 | 394 | Markdown ~~strikethrough~~ |
[y2kpp] | 2 | 2 | Dropped — always empty |
[y2kpp] is the smallest entry and the easiest decision: it was a one-shot tag
built for a single event, an incursion into the forum's EVE Online community, and it has no
use beyond it. Both surviving instances are [y2kpp][/y2kpp] with nothing inside.
Dropped without ceremony.
[strike] carries a fingerprint of the older migration. Some instances read
[strike:2z0scad4] — that suffix is a phpBB uid, attached to every tag phpBB
parsed so it could tell real markup from text a user had typed. It survived the 2011 move into
vBulletin and sat in the database untouched for fifteen years.
The spoiler tag is the one that matters. 40,265 posts used it, and its vBulletin definition is a block of inline JavaScript with a show/hide button — markup that means nothing to Discourse. It maps cleanly onto a native collapsible block.
55,245 posts were rewritten. Every media tag turned out to wrap a plain URL, so reducing each one to the URL on its own line lets Discourse's own link handling produce a real embed — no bespoke rendering needed:
[h5VIDEO]http://i.imgur.com/aTeNcGA.webm[/h5VIDEO] → http://i.imgur.com/aTeNcGA.webm [SOUNDCLOUD]http://soundcloud.com/matas/hobnotropic[/…] → http://soundcloud.com/matas/hobnotropic [video=youtube_share;CUUiHQRu6Xo]http://youtu.be/…[/video] → http://youtu.be/CUUiHQRu6Xo [strike:2z0scad4]Navy Raven (Jita - 560~)[/strike:…] → ~~Navy Raven (Jita - 560~)~~
The pattern has to refuse to match across a broken tag. Some posts contain
[strike with no closing bracket — an author's typo, preserved for a decade. A
plain non-greedy match runs straight past it to the next item's closing tag and deletes
everything in between. One sale thread lost 806 characters that way before the guard was added.
The same applies to spoilers, which nest several deep, so the conversion runs repeatedly and
unwraps one layer at a time: 39,560 posts, then 1,190, then 297, then 79, down to a handful.
The quotes themselves were handled by the importer, not by this script — and handled better than a hand-written fixer could. This is a real post from the forum, four people quoting each other in a chain:
[QUOTE=Nimbly Bimbly;635596][QUOTE=Eard;635546] [QUOTE=Rep;635488][QUOTE=Hels;635478]Plan to pick it up again after the standalone releases.[/QUOTE][/QUOTE][/QUOTE][/QUOTE]
[quote="Nimbly Bimbly,post:412,topic:8891"] [quote="Eard,post:408,topic:8891"] [quote="Rep,post:405,topic:8891"] [quote="Hels,post:403,topic:8891"] Plan to pick it up again after the standalone releases. [/quote][/quote][/quote][/quote]
Each name resolves to the post it quotes, so the rendered quote is clickable and jumps to the original — the post ids come from the same import-id mapping used everywhere else. That is the part a hand-rolled converter could not have reproduced, and the reason for not pre-empting the importer with one.
The spoiler tag, including the uppercase variant the converter also missed:
test [spoiler]icle[/spoiler] [SPOILER]testicle[/SPOILER]
test [details="Spoiler"] icle [/details] [details="Spoiler"] testicle [/details]
And video, where the tag is reduced to the URL it wrapped so that Discourse embeds it:
PVP Video: [video=youtube_share;CUUiHQRu6Xo] http://youtu.be/CUUiHQRu6Xo[/video]
PVP Video:
http://youtu.be/CUUiHQRu6Xo
← renders as an embedded player
Not every media tag can become a bare URL. [soundcloud] wraps
real SoundCloud track links, which Discourse embeds on sight. But [h5video] and
[h5loop] wrap direct .webm file links — 6,275 of them on imgur,
the rest scattered across gfycat, webmup and hosts that no longer exist. A bare
.webm URL is not oneboxable, so imgur links are rewritten to the
.gifv page form that Discourse does embed, and everything else becomes a
labelled link rather than a dead-looking URL.
The script clears each post's baked_version, so Discourse re-renders them in the
background afterwards.
The host had crashed mid-import days earlier. The import resumed and finished, but the database kept a souvenir. The first query run against the finished forum failed like this:
ERROR: missing chunk number 0 for toast value 567848 in pg_toast_26316 CONTEXT: parallel worker
Postgres stores oversized column values — here, post bodies — outside the main table in a side table called TOAST. This says a row points at content that is no longer there. It is the error people find when they search for whether their database is corrupt.
What made it confusing is that the same query worked when restricted to a range of post ids, and failed only when run against the whole table. That difference is the diagnosis: a narrow query uses an index and reads only live rows, while an unrestricted one falls back to a sequential scan and walks every physical row on disk, including dead ones no query can return.
The damage was smaller than the error suggests. A row-by-row walk of all 1,927,108 posts — each read individually, failures caught and counted — found zero unreadable live posts. One genuinely broken post surfaced first and was restored from the source database. Everything else was dead tuples: rows already superseded, waiting to be reclaimed, which only a full scan ever touches.
scanned 1927108 rows, 0 bad
The practical effect is narrow. Discourse itself never trips over it, because Discourse queries by id and stays on the index. It only appears when something sweeps the whole table — which is exactly what measuring the leftover BBCode was about to do.
With the import finished, the obvious question is how many quotes
post_process_posts actually fixed. The obvious query gives a frightening answer:
posts matching [quote= 928,069 # unchanged from before the phase ran
Which would mean fifteen hours of post-processing had achieved nothing. It had not. The pattern
matches both formats — the vBulletin tag it was looking for and the Discourse tag it
had been rewritten into. The converted form puts a double quote straight after
quote=; the vBulletin form never does, and that one character separates them.
So the phase worked, and a second query said 13,574 posts still held raw tags — 98.5% converted, a rounding error's worth of work left. That number was wrong by a factor of twenty-five, and the way it was wrong is worth more than the number.
The count asked for posts holding a vBulletin tag and no converted one:
raw ILIKE '%[quote=%' AND raw NOT LIKE '%[quote="%'
That second condition sounds like a sensible way to exclude finished work. It is in fact a way to exclude the evidence. The importer converts only the outermost quote in a post and stops, so a quote-of-a-quote ends up holding both forms at once:
[quote="Frug,post:151,topic:18768"] converted [QUOTE=erichkknaar;1695312] left as literal text [QUOTE=Approaching Walrus;1695153]have some pure ideology [/QUOTE][/QUOTE]
Every post like that contains [quote=", so the query threw it away. What it
measured was the small population of posts where the importer had converted nothing at all
— and reported that as the whole problem. Dropping the second condition changed the answer
from 13,574 to 350,254.
This is the second time in this project a count was wrong in the same direction. Earlier a progress counter was attributed to the phase that had announced itself rather than the phase producing it. Both errors share a shape: a measurement built on an assumption about the thing being measured, which then cannot contradict that assumption. The check that catches it is not a better query — it is looking at a handful of actual posts.
Resolving them works the same way the importer does: look the vBulletin post id up in the import-id map and build a linked quote, falling back to a plain attributed one where the target was never imported.
posts changed 355,515 tags → linked quote 627,137 tags → plain quote 22,885 stray [/QUOTE] closers 1,046,087
1,871 tags carry no post id at all — [quote=Wired News],
[quote=the article] — because they quote a source rather than a member. There is
nothing to link to, and they become plain attributed blockquotes, which is what they always were.
179 posts still match the pattern and none of them should change: their [quote=
sits inside a code block, where it is content rather than markup.
Converting the video tag to the URL it wrapped was correct, but vBulletin had stored those URLs
without a scheme — //youtu.be/WSKi8HfcxEk. Discourse only embeds a URL it can
resolve, so 51,326 posts rendered a bare blue link where a player belonged. Restoring
https: turned every one of them back into a video.
vBulletin had 4,858 users. Discourse ended up with 4,840. The eighteen missing accounts left no error, no warning and no log line — the importer simply did not create them.
Twelve were lost to a single setting. Discourse's normalize_emails defaults to
true, which strips Gmail dots and +tags before comparing addresses.
Two accounts that vBulletin considered entirely separate look identical to it:
58644 jsmith [email protected] ✓ imported 58645 jsmith2 [email protected] ✗ silently skipped 58647 jsmith_b [email protected] ✗ silently skipped 58696 smithj [email protected] ✗ silently skipped all four normalize to [email protected]
The other six collided on username instead — different people who happened to share a name with someone already imported. Between them the eighteen account for 1,279 posts, which landed either on the system account or on the same-named account that won the race.
All eighteen were recreated afterwards, keeping their real vBulletin address and join date, and their posts moved back onto them. It is a small job — two of the eighteen carried almost all those posts and had already been recovered by hand, leaving sixteen accounts and 107 posts — but the alternative was leaving sixteen people's authorship wrong to avoid touching a database that had already been reviewed. Every one of the 4,858 vBulletin accounts now exists.
The fix is to turn the setting off for the import and back on afterwards.
normalize_emails is doing something useful in normal operation — it stops one
person farming unlimited accounts from a single mailbox. It is only wrong during a migration,
where those aliases are not abuse but fifteen years of accumulated history, already accepted by
the software people were actually using.
One of the twelve is worth telling in full, because it is the clearest illustration of what gets lost. Two accounts, same person, and both banned:
| vB id | Username | Posts | Banned | Reason given |
|---|---|---|---|---|
| 1496 | Rakshasa The Cat | 12,460 | 2017‑10‑02 | “3 in a row laters, feel free to start again” |
| 80180 | Honk Honkler | 17 | 2019‑02‑16 | “Hello Rakshasha” |
Banned in 2017 with an explicit invitation to come back. Returned sixteen months later under
[email protected] — a plus-alias of the original address
— and was recognised after seventeen posts. The second ban reason is two words long and
makes the whole story legible.
Because the alias normalized to the original, the second account was never created and its seventeen posts fell to the system account. Both bans collapse into one, and the return, the alias and the recognition all disappear — leaving a single 2017 ban and no evidence anything else ever happened.
Splitting them back apart is straightforward once the cause is known, because nothing had to be
moved off the surviving account: the posts were never merged onto it in the first place. Recreate
the second account from its vBulletin id, move its seventeen posts across, and apply its own ban
with its own date and reason. The only obstacle is that normalize_emails blocks the
fix for the same reason it caused the problem — so it is switched off for the duration of
the operation and restored immediately afterwards.
A couple of hundred posts held a video tag that was simply broken:
[video=youtu /video] [video= video] [video=y eo]
Checking the source database showed the damage was not caused by the migration — vBulletin had been displaying these as literal text for years. The provider name and video id were gone. The obvious conclusion is that nothing can be done, and the obvious conclusion is wrong.
Look at where the broken tags sit:
[QUOTE=Jack bubu;164505][video=youtu /video] :O[/QUOTE]
Every one is inside a quote. These were not posts that contained a video — they were posts quoting a post that contained a video, and vBulletin's Reply‑With‑Quote mangled the tag while copying it. The original is untouched, and the quote records exactly which post it came from.
post 164505 (the original, intact): [video=youtube;BpA6TC0T_Lw]http://www.youtube.com/watch?v=BpA6TC0T_Lw[/video]
So the id is recoverable: read the quoted post id out of the broken post, look up that post in the source database, and pull the video id from its intact tag. Ten videos came back that way, each restored to the quote that had lost it. Five more could not be — the post they quoted had no video tag of its own, having been mangled further up a longer chain.
This was nearly written off. The first pass concluded the ids were unrecoverable and stripped the broken markup, which was defensible and also wrong — the check that mattered, whether the id survived somewhere else, had not been run. Damaged data is worth one more look before it is discarded, particularly on a forum where quoting is how every conversation works: anything worth quoting exists in more than one place.
4,370 posts carried tags that looked like this:
[quote:2u7yqhbz]...[/quote:2u7yqhbz] [spoiler:3p26ccso]...[/spoiler:3p26ccso]
That suffix is a phpBB uid. phpBB stamped every tag it parsed with a short random string, one per post, so its renderer could tell markup it had produced from text a user had merely typed. The suffixes came across in the 2011 migration, meant nothing to vBulletin, and sat in the database being displayed as literal text for fifteen years. Removing the suffix leaves an ordinary tag, which the conversions used everywhere else then handle.
These are the oldest posts on the forum, and their markup is the last trace of the software it ran on before.
The import brought 1,207 avatars across, and the files sat correctly on disk. The forum showed letter circles for everyone.
Discourse does not serve an avatar from the uploaded file. It serves a resized copy from an
optimized_images table, generated by a background job the first time one is needed.
That job runs under Sidekiq, and nothing had started Sidekiq on a machine being used for repair
work. The table was empty, so every avatar request fell through to the fallback.
“1,207 avatars imported” was true and useless. The data was present, correct, and completely invisible. It is the same trap as the quote count in section 9 from the opposite direction: there, a query said the work was done when it was not; here, the database said the data was there when nobody could see it. Neither is caught by querying harder. Both are caught by looking at the page.
Generating the 10,827 thumbnails directly took a few minutes. On the production machine none of this applies — Sidekiq runs, and the job does it unprompted.
Every YouTube link on the forum rendered as a plain blue link. The URL was right, the markup Discourse produced was right, and no video ever appeared.
The obvious conclusion is that something in the imported posts is malformed, and that conclusion survived three rounds of increasingly precise edits to the stored HTML — each one comparing the imported markup against what Discourse generates itself, finding a small difference, correcting it, and changing nothing on screen.
The test that ended it took thirty seconds. Post a new YouTube URL through Discourse, as a user, in a fresh topic. It rendered as a bare link too. Nothing imported was involved, so nothing imported was at fault — this instance could not onebox anything. Three rounds of work had been spent correcting data against a broken baseline.
Two environmental causes, neither in the data:
Sidekiq was not running. Resolving a onebox — fetching a video's title and thumbnail — is a background job. Without the worker, Discourse marks a link for oneboxing and never finishes the job, leaving exactly the bare link that was showing.
Plugin JavaScript had never been built. The frontend was built earlier with
pnpm build, which builds Discourse core and deliberately refuses to import plugin
modules — plugins compile through a separate task:
rake assets:precompile:build_plugins # 44 plugins, 64 seconds
Without it discourse-lazy-videos never loads, so nothing turns a video container
into a player. The container was in the page the whole time, inert.
Both are artefacts of running repair work on a development image. The production container compiles plugin assets during bootstrap and supervises Sidekiq, so neither failure can occur there — which is the argument for deploying onto the official image rather than promoting the machine the repairs were done on.
Starting Sidekiq immediately produced an admin warning: 1,386,127 queued jobs,
almost all of them PullHotlinkedImages — Discourse offering to download and
re-host every externally linked image on the forum.
That reads like days of crawling and a serious decision about link rot. It is neither. The queue holds one job per post per rebake, and the posts had been rebaked several times; nearly every job opens a post, finds nothing external, and exits. Counting the actual images gives a very different picture:
external image URLs in cooked posts 47,783 — of which img.youtube.com 46,997 # thumbnails generated locally, regenerable — genuinely external 786 # across 426 posts
786 images, in 426 posts — imgur albums, CCP dev blogs, personal sites. Those are the ones that vanish when a host lapses, and they are worth keeping. Running them directly took minutes rather than days, and the rest of the queue was discarded: clearing it removes no data, since the URLs stay in the posts and the images keep loading from wherever they live. The job only ever made local copies.
vBulletin stored passwords as md5(md5(password) + salt). Discourse uses PBKDF2 and
cannot verify a vBulletin hash, so no old password can survive a migration. What is easy to miss
is what the importer does instead.
vBulletin row md5hash, salt
importer builds "md5hash:salt" # joined, and passed as the PLAINTEXT password
Discourse stores pbkdf2(that string, 600,000 iterations)
The hash and salt are concatenated and handed to Discourse as though a user had typed them. Discourse dutifully hashes the result. Every one of the 4,841 accounts ends up with a valid, working credential — one derived from the old database and known to nobody, not even its owner.
The practical effect is not a security hole, but it is not nothing either. No old password opens an account, because nobody can type a 32‑character MD5 digest they have never seen. But the credential rows are live rather than absent, and they are a deterministic function of a database that has been dumped, copied and archived several times over. Anyone holding an old dump holds the input.
So the credentials were deleted outright rather than left in place. All 4,842 rows are gone, along with every session token. Every account now has no password at all, and the only way into one is a reset link sent to its registered address — which is the intended route for a forum that has been offline for a year regardless.
=== deleting all user_passwords rows === DELETE 4842 credentials remaining: 0 expiring all active sessions
This ran last, after an administrator account existed — wiping every password on a forum with no admin is a memorable way to lock yourself out of your own site.
The import is finished and the repairs are done. What follows is what came out of it, and what is now running in production.
| Count | Note | |
|---|---|---|
| Posts | 1,927,109 | Every post from every live thread |
| Topics | 45,115 | 20,708 threads + 24,346 private conversations |
| Categories | 65 | One per vBulletin forum |
| Users | 5,172 | All 4,858 vBulletin accounts, plus 314 recreated from guest posts |
| Suspended | 329 | 14 from the ban list, the rest deleted accounts |
| Subscriptions | 292,900 | 217,074 tracking, 75,826 watching |
| Polls | 300 | Results appended to their opening posts |
| Stored passwords | 0 | Deliberately — see section 11 |
None of this needed a re-import — the stored ids made every one of these a matter of joining back to the old database and correcting what was found.
| Repair | Scope | Outcome |
|---|---|---|
| Quote tags left unconverted | 355,515 posts | 627,137 tags became linked quotes |
| Videos stored without a scheme | 51,326 posts | Bare links became embedded players |
| phpBB uid suffixes | 4,370 posts | [quote:2u7yqhbz] and friends, from 2011 |
| Avatar thumbnails | 10,827 | Generated for 1,207 users |
| Hotlinked images archived | 656 | Across 426 posts, against future link rot |
| BBCode with no converter | 55,245 posts | spoiler, video, h5loop, h5video, soundcloud, strike |
| Guest-post authorship | ~71,000 posts | 365 names restored as suspended accounts |
| Thread subscriptions | 234,258 | Of 234,428; the rest reference deleted users |
| Poll results | 300 | Of 303; 3 threads were not imported |
| Bans | 14 | Applied by hand after the phase crashed |
| Collided accounts | 4 | Two name collisions and one alias, separated |
| Truncated video tags | 10 | URLs recovered from the posts they quoted |
| Accounts lost to email normalizing | 16 | Recreated with 93 of their posts |
| Old-URL redirects | 20,700 | showthread.php links resolve again |
Every one of the 4,858 vBulletin accounts now exists in Discourse. 10,076 posts remain on the system account — the ones vBulletin recorded no name for at all, anonymous in the source data and equally anonymous now.
The BBCode needed several passes, not one. Spoiler tags nest, and the pattern that converts them is deliberately forbidden from matching across another spoiler tag — without that guard, a malformed tag lets the match run on to the next closer and swallow everything between. So each pass unwraps one layer: 39,560 posts, then 1,190, then 297, then 79, down to a handful. Converging slowly is the correct behaviour here; matching greedily would have been fast and would have eaten text.
The importer writes a mapping of every old thread id to its new topic, but with URLs
(thread/3) that nothing on the internet actually links to. This forum used
vBulletin's SEO-friendly form, so the shapes that need to resolve are:
/showthread.php?3332-GANK-NIGHT-GN-31-Crowes-and-Bummers-II /showthread.php?t=3332 /showthread.php?9095-R.I.P.-SHC&p=206497&viewfull=1
Discourse rewrites incoming URLs through a permalink_normalizations setting before
looking them up, so one rule per shape is enough — with one catch worth recording, because
it is not documented anywhere obvious. The setting is split on | and the boundary
between pattern and replacement is found by counting unescaped /. A rule can
therefore contain neither an alternation nor a slash in its replacement, which makes
thread/ impossible to express. Renaming the stored permalinks to
thread-3332 sidesteps it entirely.
All three shapes now return a 301 to the right topic, so fifteen years of links from elsewhere survive the move.
The forum has been live at failheap-challenge.com since 25 August 2026, on a four-core machine behind HAProxy behind Cloudflare. The deployment itself was the last day of a four-week job, and three things went wrong during it that are worth writing down, because each of them is invisible until it bites.
The container build aborted with git clone exiting 128:
fatal: destination path 'discourse-solved' already exists and is not an empty directory
That plugin now ships with Discourse core. Cloning it into a directory that already exists
kills the whole bootstrap, forty minutes in. The fix is a one-word guard —
test -d <dir> || git clone … — which also means the build cannot
break the same way when upstream absorbs the next plugin.
Then pg_dump refused to finish, on two different tables:
ERROR: missing chunk number 0 for toast value 567834 in pg_toast_28637
This is the same host crash from section 8, and it was worse than it had looked. A
row-by-row walk of all 1.93 million posts read cleanly — every live row was fine.
But REINDEX gave the real answer:
ERROR: could not create unique index "posts_pkey" DETAIL: Key (id)=(157942) is duplicated.
The table held 42 duplicated ids — 1,927,151 physical rows where the primary key allowed 1,927,109 — and the index itself was corrupt, which is why index scans could return the same row twice and why every attempt to copy the table failed. Each duplicate had one readable copy and one whose content was unreachable. Keeping the readable copy, identified by physical row address, and rebuilding the index brought the count back to exactly what the primary key expected.
This had been quietly distorting things for hours. Row counts drifted by small amounts across the session and were written off as rounding between different queries. They were not: the table genuinely contained more rows than it should. A count that will not settle is worth chasing, not explaining away.
The restore brought 1.93 million posts across intact, and the forum still looked wrong, three times over:
avatar_sizes list — 24, 48, 72, 96, 144, 288 — and silently falls
back to a letter for anything else. Six more had an extension of unknown,
inherited from vBulletin storing avatars as nameless database blobs.That last one is the general lesson. Restoring a database overwrites every site setting. The development database had email switched off and registration open; both came across and silently undid deliberate production decisions. It happened twice in one evening before the pattern was obvious.
Everyone's password had been deliberately deleted, so the only way back in was a reset link. That made the announcement email load-bearing rather than decorative.
It went to 997 people in one batch: everyone who had logged in within five years, plus everyone who joined in the forum's first five years and wrote more than a hundred posts. One message failed on a network timeout and was retried.
| Rate | ||
|---|---|---|
| Delivered | 92.56% | ~930 people reached |
| Failed | 6.21% | ~62 addresses dead after fifteen years |
| Spam complaints | 0% | the number that protects the domain |
One line in the draft claimed “your account is still there, under the same name”. Checking it before sending showed that 1,104 of 4,858 accounts — 22.7% — had been renamed, almost all spaces becoming underscores. More than one recipient in five would have read that sentence and found it false. It was replaced with an explanation of what changed and why their posts are still attached either way.
Bounces now feed back automatically: a Mailgun webhook tells Discourse when an address fails, and two hard bounces disable it. Those 62 dead addresses will retire themselves rather than being mailed forever.