WiredTiger File Forensics Part 2: wt dump

This article contains one normal section – how to print information directly from the raw WiredTiger files using wt dump – followed by really important, otherwise undocumented information about how to get to the MongoDB binary data inside WT tables.

See “WiredTiger File Forensics (Part 1: Building “wt”)” for:

• How to build the “wt” utility command
• A short description of its commandline syntax
• How to set the wiredtiger_open() configuration string it needs

The examples in this page assume the following:

• WiredTiger.config or WIREDTIGER_CONFIG or the -C option is being used to give “wt” the right config string. Having said that – an empty “wt” config might work OK for you, at least until wt printlog.
• xxd, bsondump, jq are installed. (See “Shell script formatting tricks” section for why.)

wt dump

The wt dump command can be used to show the full content of a WiredTiger table file. It will output the table object’s header then every key and value.

wt dump has one compulsory argument: uri. A uri is usually “table:” + WT ident string, or “file:” + (relative) file path. Eg. “table:collection-4–9876544321″ or “file:collection-4–9876544321.wt“.

“table:” is automatically prepended to the uri by wt dump if you don’t specify any type yourself.

Documentation at source.wiredtiger.com

The format from wt dump can be:

• “print”: Keys and values will be strings of printable ascii chars and \-escaped hex values of non-printable bytes. (A.k.a. pretty-print.)
  Eg. \82\00\00\00\03_id\00\0a\00\00\00\ff_id\00\00\03max\00…..
• “hex”: Keys and values will be show as continuous [0-9af] hexadecimal characters. No escaping or quotes.
  Eg. 82000000035f6964000a000000ff5f696400…
• “json” Keys will be shown as natural type (only a plain number as of MongoDB 4.4 / WiredTiger 10.0.0, I think). Values will be shown as UTF-16, which doesn’t make sense – it is inserting null bytes every second byte. (I suspect this is a current-version bug.)
  Eg. “\u0082\u0000\u0000\u0000\u0003\u005f\u0069\u0064\u0000\u000a\u0000\u0000\u0000\u00ff\u005f\u0069\u0064\u0000….”

For “print” or “hex” the output has a fixed line format:

The -j JSON output mode of wt dump is much easier to read by eye, but values are currently encoded in UTF-16 strings which is practically useless. I can’t ingest that using my preferred JSON tools at least. Eg. what is the binary value 82 00 00 00 03 5f 69 64 (eight bytes) will be output as “\u0082\u0000\u0000\u0000\u0003\u005f\u0069\u0064” (sixteen bytes). The binary length has been doubled by null bytes between every intended value byte.

Shell script formatting tricks used in this article

• xxd -r -p converts plain, [0-9a-fA-Z]* hex strings to raw bytes. (Note “xxd -rp” fails, and fails silently. Keep the -r and -p flags separated in the shell command.)
• bsondump is used to convert BSON from binary to JSON. Use –quiet.
• tail -n +7 to skip the header output of wt dump.
• awk is used to filter out just the keys (line number NR%2 == 1) or values (line number NR%2 == 0) in the pretty-print or hex mode of wt dump.
• jq is used to filter and/or reformat bsondump and wt printlog output.
  (Ideally jq would be used to ingest the output of wt dump -j directly, but the falsely null-padded, bytes-as-UTF-16 string encoding it currently outputs defeats that.)

Record output is not influenced by the WT binary format

The output of wt dump is unaffected by the WiredTiger binary format used. It will come out the same regardless of table format type, and modifiers such as max page size, key prefixing, etc.

This means wt dump won’t teach you about the ‘physical’ format of the data. The only thing you can see related to this is the config string that was used in WT_SESSION::create when a table was first made. It is shown in the header section every time wt dump is run.

$ wt dump file:collection-0--4131298130356306083.wt 
WiredTiger Dump (WiredTiger Version 10.0.0)
Format=print
Header
file:collection-0--4131298130356306083.wt
access_pattern_hint=none,allocation_size=4KB,app_metadata=(formatVersion=1),assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none),block_allocation=best,block_compressor=snappy,cache_resident=false,checksum=on,colgroups=,collator=,columns=,dictionary=0,encryption=(keyid=,name=),exclusive=false,extractor=,format=btree,huffman_key=,huffman_value=,ignore_in_memory_cache_size=false,immutable=false,import=(enabled=false,file_metadata=,repair=false),internal_item_max=0,internal_key_max=0,internal_key_truncate=true,internal_page_max=4KB,key_format=q,key_gap=10,leaf_item_max=0,leaf_key_max=0,leaf_page_max=32KB,leaf_value_max=64MB,log=(enabled=false),lsm=(auto_throttle=true,bloom=true,bloom_bit_count=16,bloom_config=,bloom_hash_count=8,bloom_oldest=false,chunk_count_limit=0,chunk_max=5GB,chunk_size=10MB,merge_custom=(prefix=,start_generation=0,suffix=),merge_max=15,merge_min=0),memory_page_image_max=0,memory_page_max=10m,os_cache_dirty_max=0,os_cache_max=0,prefix_compression=false,prefix_compression_min=4,readonly=false,source=,split_deepen_min_child=0,split_deepen_per_child=0,split_pct=90,tiered=(chunk_size=1GB,tiers=),tiered_storage=(auth_token=,bucket=,local_retention=300,name=,object_target_size=10M),type=file,value_format=u,verbose=[],write_timestamp_usage=none

FYI there’s a lot of empty and/or unused options in the config strings, so don’t take the presence of an option to imply it is being used. Eg. above we see lsm options, even though all the tables mongodb makes are btrees (which you can confirm by the “format=btree” option).

The wt verify subcommand with a -d option does show something about the physical layout. Arguably wt stat does too. But these will no be covered here.

Metadata tables to explore in MongoDB data directory

WiredTiger

Yawn. Just a version string. View it as plain text.

WiredTiger.turtle

Fairly boring. Version information, a URI pointing to WiredTiger.wt, a config string to open it. View it as plain text.

The turtle file will be recreated auto-magically if it is absent.

WiredTiger.config

This is not a WiredTiger data file. It is kind of like a resource file for the “wt” utility command. It won’t be created by any program. It will only be made by you, or some nice colleague, to save you from specifying the -C option each time you run the “wt” command. See the previous blog article to learn what to put in it.

WiredTiger.lock

Just a classic lock file. Contains nothing except the plain text “WiredTiger lock file”. Created and overwritten automatically.

WiredTiger.wt

This is the main metadata table of the WiredTiger storage engine. It is also, itself, a WiredTiger table, one the WiredTiger.turtle file points to.

• The keys in this table are URI strings of:
  • table: Wiredtiger tables
  • file: Files
  • system: A couple of special system values, and 
  • colgroup: Colgroups.
    (Colgroups are 1-to-1 to tables the way MongoDB uses WiredTiger; I tend to forget about their presence as a result.)
• The values are the table/file/colgroup WiredTiger-flavoured JSON config strings that WiredTiger uses when it opens those objects in other API functions.

The entire content of this file is ascii-printable, keys and values alike. WT identifiers and config values are all print-safe c strings. You won’t have it as easy with the other table files though! They all contain binary numeric type and BSON keys and/or values.

WiredTiger.wt is not the MongoDB collection and index etc. catalog. That is in _mdb_catalog.wt (see later).

There is a special command that is hard-coded to dump only this collection – wt list. By default, it filters the output to just the keys, and furthermore, it excludes the “system:” ones plus the table/file listing for itself. Even if you use the “-v” option it will still exclude itself.

Bonus exercise: Dump the content of the WiredTiger.wt metadata file like it was just any other wt table

Amongst other records, the WiredTiger.wt file includes “table:xxxx” keys that point to every table’s URI. Every table except itself, that is.

So you can run commands such as wt dump collection-0–4131298130356306083, or wt dump index-8-5834121039240263510 or wt dump _mdb_catalog or wt dump sizeStorer, but you can’t run wt dump WiredTiger.

The trick is using the URI with “file:” at the front. You also need the full filename with the “.wt” suffix on the end, because it is a file, not a table URI. (wt dump assumes “table:” as the type prefix if not supplied.)

$ wt dump file:WiredTiger.wt

WiredTiger Dump (WiredTiger Version 10.0.0)
Format=print
Header
file:WiredTiger.wt
access_pattern_hint=none,allocation_size=4KB,app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none),block_allocation=best,block_compressor=,cache_resident=false,checksum=uncompressed,colgroups=,collator=,columns=,dictionary=0,encryption=(keyid=,name=),exclusive=false,extractor=,format=btree,huffman_key=,.....
Data
colgroup:_mdb_catalog\00
app_metadata=(formatVersion=1),collator=,columns=,source="file:_mdb_catalog.wt",type=file\00
colgroup:collection-0--4131298130356306083\00
app_metadata=(formatVersion=1),collator=,columns=,source="file:collection-0--4131298130356306083.wt",type=file\00
colgroup:collection-0--6422702119521843596\00
app_metadata=(formatVersion=1),collator=,columns=,source="file:collection-0--6422702119521843596.wt",type=file\00
colgroup:collection-0-5834121039240263510\00
app_metadata=(formatVersion=1),collator=,columns=,source="file:collection-0-5834121039240263510.wt",type=file\00
colgroup:collection-0-5841128870485063129\00
app_metadata=(formatVersion=1),collator=,columns=,source="file:collection-0-5841128870485063129.wt",type=file\00
colgroup:collection-10--4131298130356306083\00
app_metadata=(formatVersion=1),collator=,columns=,source="file:collection-10--4131298130356306083.wt",type=file\00
colgroup:collection-10-5841128870485063129\00
app_metadata=(formatVersion=1,oplogKeyExtractionVersion=1),collator=,columns=,source="file:collection-10-5841128870485063129.wt",type=file\00
colgroup:collection-11-5841128870485063129\0
...
colgroup:sizeStorer\00
app_metadata=,collator=,columns=,source="file:sizeStorer.wt",type=file\00
file:WiredTigerHS.wt\00
access_pattern_hint=none,allocation_size=4KB,app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none),block_allocation=best,block_compressor=snappy,cache_resident=false,checksum=uncompressed,...
file:_mdb_catalog.wt\00
access_pattern_hint=none,allocation_size=4KB,app_metadata=(formatVersion=1),assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none),block_allocation=best,....
file:collection-0--4131298130356306083.wt\00
access_pattern_hint=none,allocation_size=4KB,app_metadata=(formatVersion=1),assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none),block_allocation=best,
...
file:sizeStorer.wt\00
access_pattern_hint=none,allocation_size=4KB,app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none),block_allocation=best,block_compressor=,...
system:checkpoint\00
checkpoint_timestamp="6088966b00000001"\00
system:checkpoint_base_write_gen\00
base_write_gen=104333\00
system:checkpoint_snapshot\00
snapshot_min=1,snapshot_max=1,snapshot_count=0\00
system:oldest\00
oldest_timestamp="6088966600000001"\00
table:_mdb_catalog\00
app_metadata=(formatVersion=1),colgroups=,collator=,columns=,key_format=q,value_format=u\00
table:collection-0--4131298130356306083\00
app_metadata=(formatVersion=1),colgroups=,collator=,columns=,key_format=q,value_format=u\00
table:collection-0--6422702119521843596\00
app_metadata=(formatVersion=1),colgroups=,collator=,columns=,key_format=q,value_format=u\00
table:collection-0-5834121039240263510\00
app_metadata=(formatVersion=1),colgroups=,collator=,columns=,key_format=q,value_format=u\00
table:collection-0-5841128870485063129\00
app_metadata=(formatVersion=1),colgroups=,collator=,columns=,key_format=q,value_format=u\00
table:collection-10--4131298130356306083\00
app_metadata=(formatVersion=1),colgroups=,collator=,columns=,key_format=q,value_format=u\00
table:collection-10-5841128870485063129\00
app_metadata=(formatVersion=1,oplogKeyExtractionVersion=1),colgroups=,collator=,columns=,key_format=q,value_format=u\00
table:collection-11-5841128870485063129\00
app_metadata=(formatVersion=1),colgroups=,collator=,columns=,key_format=q,value_format=u\00
...
table:index-9-5841128870485063129\00
app_metadata=(formatVersion=8),colgroups=,collator=,columns=,key_format=u,value_format=u\00
table:sizeStorer\00
app_metadata=,colgroups=,collator=,columns=,key_format=u,value_format=u\00

_mdb_catalog.wt

This is the catalog/schema that the MongoDB user sees.

$ #N.b. URI can be "file:_mdb_catalog.wt", "table:_mdb_catalog", or just "_mdb_catalog"
$ wt dump -x table:_mdb_catalog | tail -n +7 | awk 'NR%2 == 0 { print }' | xxd -r -p | bsondump --quiet

{"md":{"ns":"local.startup_log","options":{"uuid":{"$binary":{"base64":"9TLH+oBmQXW7eou0SZFXPQ==","subType":"04"}},"capped":true,"size":{"$numberInt":"10485760"}},"indexes":[{"spec":{"v":{"$numberInt":"2"},"key":{"_id":{"$numberInt":"1"}},"name":"_id_","ns":"local.startup_log"},"ready":true,"multikey":false,"multikeyPaths":{"_id":{"$binary":{"base64":"AA==","subType":"00"}}},"head":{"$numberLong":"0"},"prefix":{"$numberLong":"-1"},"backgroundSecondary":false,"runTwoPhaseBuild":false,"versionOfBuild":{"$numberLong":"1"}}],"prefix":{"$numberLong":"-1"}},"idxIdent":{"_id_":"index-1-5841128870485063129"},"ns":"local.startup_log","ident":"collection-0-5841128870485063129"}
{"isFeatureDoc":true,"ns":null,"nonRepairable":{"$numberLong":"0"},"repairable":{"$numberLong":"1"}}
{"md":{"ns":"local.replset.oplogTruncateAfterPoint","options":{"uuid":{"$binary":{"base64":"8kuKi7n6TliHYwjFxEEKWw==","subType":"04"}}},"indexes":[{"spec":{"v":{"$numberInt":"2"},"key":{"_id":{"$numberInt":"1"}},"name":"_id_","ns":"local.replset.oplogTruncateAfterPoint"},"ready":true,"multikey":false,"multikeyPaths":{"_id":{"$binary":{"base64":"AA==","subType":"00"}}},"head":{"$numberLong":"0"},"prefix":{"$numberLong":"-1"},"backgroundSecondary":false,"runTwoPhaseBuild":false,"versionOfBuild":{"$numberLong":"1"}}],"prefix":{"$numberLong":"-1"}},"idxIdent":{"_id_":"index-3-5841128870485063129"},"ns":"local.replset.oplogTruncateAfterPoint","ident":"collection-2-5841128870485063129"}
...
...

The content of each value is BSON, as you can in the deserialized output above. There is just one document that isn’t for a collection, the “isFeatureDoc” (see above). Apart from that single, trivial exception, the fields of each object are:

• “ns” Eg. “test.foo”, “config.chunks”, “local.oplog.rs”
• “ident” The unique identifier string for the WT table of this mongodb collection. Eg. “collection-7-5834121039240263510”
• “idxIdent”: object listing the ident(ifier)s of each index on the collection (Annoyingly they are separated from the index definitions inside the “md.indexes” object; see below.)
• “md” Metadata of the collection. It has these fields:
  • “ns” (again)
  • “options” Options you would pass to collection create command. Most collections will only have the (automatically-assigned) collection UUID value in this sub-object, but if a collection has a non-default option such as capped: true it is defined here.
  • “indexes” Array of index specs, the ones you’d see if you were running db.collection.getIndexes().

Confirming things match the MongoDB picture

If you want to prove to yourself the information in _mdb_catalog WiredTiger table matches what the result of listDatabases (‘show dbs’), listCollections (‘show collections’), and listIndexes (db.collection.getIndexes()) in MongoDB, you can search the file with these shell commands:

List of collections and their index names, vs. the WT ident(ifier) strings

$ wt dump -x table:_mdb_catalog | tail -n +7 | awk 'NR%2 == 0 { print }' | xxd -r -p | bsondump --quiet | jq 'select(. | has("md")) | {ns: .ns, ident: .ident, idxIdent: .idxIdent}'
{
  "ns": "local.startup_log",
  "ident": "collection-0-5841128870485063129",
  "idxIdent": {
    "_id_": "index-1-5841128870485063129"
  }
}
{
  "ns": "local.replset.oplogTruncateAfterPoint",
  "ident": "collection-2-5841128870485063129",
  "idxIdent": {
    "_id_": "index-3-5841128870485063129"
  }
}
...
...

List of collections and the options strings they were created with:

$ wt dump -x table:_mdb_catalog | tail -n +7 | awk 'NR%2 == 0 { print }' | xxd -r -p | bsondump --quiet | jq 'select(. | has("md")) | .md | {ns: .ns, options: .options}'
{
  "ns": "local.startup_log",
  "options": {
    "uuid": { "$binary": {  "base64": "9TLH+oBmQXW7eou0SZFXPQ==", "subType": "04" } },
    "capped": true,
    "size": { "$numberInt": "10485760" }
  }
}
....

Index definitions (a.k.a. index specs):

$ wt dump -x table:_mdb_catalog | tail -n +7 | awk 'NR%2 == 0 { print }' | xxd -r -p | bsondump --quiet | jq 'select(. | has("md")) | .md | {ns: .ns, indexes: .indexes}'
{
  "ns": "local.startup_log",
  "indexes": [
    {
      "spec": { "v": { "$numberInt": "2" },
        "key": {
          "_id": { "$numberInt": "1" }
        },
        "name": "_id_",
        "ns": "local.startup_log"
      },
      "ready": true,
      "multikey": false,
      "multikeyPaths": {
        "_id": { "$binary": { "base64": "AA==", "subType": "00" }
        }
      },
      "head": { "$numberLong": "0" },
....

Output a tab-delimited table of WT ident(ifier) values vs. collection namespace

$ wt dump -x table:_mdb_catalog | tail -n +7 | awk 'NR%2 == 0 { print }' | xxd -r -p | bsondump --quiet | jq -r 'select(. | has("md")) | [.ident, .ns] | @tsv' | sort
collection-0--4131298130356306083	config.cache.chunks.test.bar
collection-0-5834121039240263510	local.replset.initialSyncId
collection-0-5841128870485063129	local.startup_log
collection-0--6422702119521843596	config.system.sessions
collection-10--4131298130356306083	config.cache.chunks.test.foo
...
...

Output a tab-delimited table of WT ident(ifier) values vs. indexes

The following outputs three columns: WT ident value, collection namespace, and index name.

$ wt dump -x table:_mdb_catalog | tail -n +7 | awk 'NR%2 == 0 { print }' | xxd -r -p | bsondump --quiet | jq -r 'select(. | has("idxIdent")) | .ns as $nsT | .idxIdent | to_entries[] | [.value, $nsT, .key] | @tsv' | sort
index-11--4131298130356306083	config.cache.chunks.test.foo	_id_
index-12--4131298130356306083	config.cache.chunks.test.foo	lastmod_1
index-12-5841128870485063129	local.system.replset	_id_
index-1--4131298130356306083	config.cache.chunks.test.bar	_id_
index-14-5841128870485063129	admin.system.version	_id_
...
...

sizeStorer.wt

Just a table of BSON values. MongoDB stores the sum BSON document data size and record count for each mongodb collection here. The keys are not the mongodb collection ns value though – they are the WT URI/ident values. You’ll have to map to the output of wt dump _mdb_catalog to work out which counts and sizes are for which collections.

$ #print the keys
$ wt dump table:sizeStorer | tail -n +7 | awk 'NR%2 == 1 { print }' | head 
table:_mdb_catalog
table:collection-0--2823106041799591351
table:collection-0--4131298130356306083
table:collection-0--6422702119521843596
...
...

$ #print the bson data values
$ wt dump -x table:sizeStorer | tail -n +7 | awk 'NR%2 == 0 { print }' | xxd -r -p | bsondump --quiet | jq -c .
{"numRecords":{"$numberLong":"37"},"dataSize":{"$numberLong":"16930"}}
{"numRecords":{"$numberLong":"1"},"dataSize":{"$numberLong":"22"}}
{"numRecords":{"$numberLong":"1"},"dataSize":{"$numberLong":"130"}}
{"numRecords":{"$numberLong":"0"},"dataSize":{"$numberLong":"0"}}
...

storage.bson

Not a part of WiredTiger. A few storage-engine-related options are persisted into a file in the dbPath data directory so MongoDB layer above can restart knowing if the directoryPerDB etc. options were used last time the data was used.

View it using bsondump [–quiet] storage.bson .

Application (= MongoDB) tables

The collection-*.wt and index-*.wt files contain the data of the MongoDB collections and indexes that you observe as a client connected to a MongoDB instance.

These are not WiredTiger metadata so we’ll cover them in the next blog post (WiredTiger File Forensics Part 3: Viewing all the MongoDB data). For starters though you can use the wt dump -x collection-*** | tail -n +7 | awk ‘{NR%2 = 0}’ | bsondump –quiet method shown for _mdb_catalog earlier to see the bsondump of collection-*.wt.

FYI these files do not contain any of their own MongoDB catalog information. You can see the data in them, but you can’t see what indexes are for which collections, or even what a collection’s mongodb ns (namespace) is. You will always need to dump the BSON data from the _mdb_catalog wiredtiger table information to get that.

WiredTiger’s transactional log files

The journal/WiredTigerLog.NNNNNNNN files are WiredTiger’s transactional log files, its write-ahead log. This is the only place on disk that preserves writes that haven’t yet been flushed to collection-*.wt and index-*.wt files during a checkpoint.

You usually don’t need to pay attention to the exact log file numbers as you work with them – they’re treated as a sequence and are automatically iterated from the start.

You view them with wt printlog instead of wt dump. The output is always JSON, but additional hex versions of the keys and values in ops can be included as well by using the -x option.

$ wt printlog | head -n 20
[
  { "lsn" : [72,128],
    "hdr_flags" : "",
    "rec_len" : 128,
    "mem_len" : 128,
    "type" : "system",
    "ops": [
      { "optype": "prev_lsn",
        "prev_lsn": [1, 0]
      }
    ]
  },
  { "lsn" : [72,256],
    "hdr_flags" : "",
    "rec_len" : 128,
    "mem_len" : 128,
    "type" : "system",
    "ops": [
      { "optype": "checkpoint_start",
...

This is just a starter for using looking at the wiredtiger transaction log files (a.k.a. mongodb journal files). The meaning of lsn numbers; how to identify collections from ops[].fileid; how to find oplog writes; why there are no writes for many collections, is material for another day.

You can ignore the WiredTigerPreplog.* files, they are empty. They’re pre-allocated in the filesystem to reduce latency that can occur doing the open() syscall; they are simply renamed as the next WiredTigerLog.* file when it is needed.

Summary

wt dump is the main command you will use to look at WiredTiger tables. wt list is a shortcut for dumping a subset of records from the top-most of those tables (WiredTiger.wt).

wt printlog is the command needed to view the (completely different) transaction log files. These are the write-ahead log of WiredTiger, known as journal in MongoDB.

Knowing how to follow the hierarchy of WiredTiger metadata, followed by the MongoDB catalog, is crucial to begin introspection of a MongoDB data directory.

Another crucial understanding is that WiredTiger stores keys and values as raw binary values with any type indicator, so you have to know ahead of time what those are, and/or be a skilled binary detective. The long and the short of it though is: WiredTiger’s metadata is simple, ascii-printable strings; all other tables are usually BSON or plain integer types. (Ok, one extra: A special ‘dehydrated keystring’ packing format, defined by the MongoDB layer, is in the index-*.wt files.)

Wait for the next blog post (WiredTiger File Forensics Part 3: Viewing all the MongoDB data) to see how to put it all together and extract any information, as you think of it as a MongoDB user.