HashiDays One conference. Three cities. Find a city near you
Vault
Monitor & troubleshoot

Inspect data in BoltDB

  • 12min
  • |
  • Vault

Vault clusters configured to use Integrated Storage (Raft), the cluster persists all encrypted data to a bbolt key/value store.

You can inspect data in clusters which use Integrated Storage by operating Vault in recovery mode and using the sys/raw API as described in Inspect Data in Integrated Storage. That process is useful for troubleshooting issues in operable Vault clusters and provides the greatest detail possible for issue resolution and troubleshooting.

You can also use information in this tutorial to access low level details from the database by directly examining the database files.

This is a tool of last resort, but it's helpful when you suspect database corruption, when a Vault cluster is inoperable, or for querying the free list.

Warning

DO NOT inspect data in-place on a running production Vault cluster with the techniques and tools described in this tutorial. Use a copy of the database files as required instead.

Prerequisites

To follow the examples in this tutorial, you need the following.

Learn monitoring repository

You can get some example Vault database files to use when following along in this tutorial by cloning the learn-vault-monitoring repository or downloading a zip file from GitHub.

  1. Clone the repository:

    $ git clone https://github.com/hashicorp-education/learn-vault-monitoring

    Or, download the repository contents:

    Download

    Tip

    If you downloaded the zip archive, be sure to unzip the file before proceeding.

    This repository has supporting content for Vault learn tutorials. The content specific to this tutorial is in a subdirectory.

  2. Change the working directory to learn-vault-monitoring/example-raft-database.

    $ cd learn-vault-monitoring/example-raft-database

  3. Examine the directory structure of a data directory from a Vault server configured to use Integrated Storage.

    $ tree
.
├── raft
│   ├── raft.db
│   └── snapshots
└── vault.db

    This is the most minimal example, with a raft directory containing the cluster coordinating data in raft.db, and a vault.db that holds the Vault operational data and secrets.

    Were this from a busier cluster, the snapshots directory would also contain some snapshot data.

With your example Vault data in place, you are now ready to inspect the files with the bbolt CLI and via a web browser with boltd.

Notes about storage architecture and terms

bbolt uses a database file represented by pages. The first 2 pages store important database metadata, while the third page holds the "free list". The free list keeps track of pages marked as deleted, which are free for writing to again.

All remaining pages contain the buckets. All keys and values get stored in buckets, represented by B-tree data structures.

Using the bbolt CLI

The bbolt CLI tool is part of the bbolt distribution.

  1. Install bbolt into your $GOBIN path where GOBIN is the directory where go install and go get will place binaries after building main packages. 

    $ go install go.etcd.io/bbolt/cmd/bbolt@latest

  2. Check the bbolt installation to make sure it's on your $PATH.

    $ bbolt help
Bbolt is a tool for inspecting bbolt databases.

Usage:

  bbolt command [arguments]

The commands are:

    bench       run synthetic benchmark against bbolt
    buckets     print a list of buckets
    check       verifies integrity of bbolt database
    compact     copies a bbolt database, compacting it in the process
    dump        print a hexadecimal dump of a single page
    get         print the value of a key in a bucket
    info        print basic info
    keys        print a list of keys in a bucket
    help        print this screen
    page        print one or more pages in human readable format
    pages       print list of pages with their types
    page-item   print the key and value of a page item.
    stats       iterate over all pages and generate usage stats

Use "bbolt [command] -h" for more information about a command.

    Tip

    If you meet with an error instead, ensure that the go install succeeded and that bbolt is in your $PATH with which bbolt.

Check database file integrity

One useful feature of bbolt is the check command. It verifies that all pages are accessible or marked as freed, and also checks that no pages are double referenced.

If you suspect issues with the database file, the first thing you can do is check its integrity.

$ bbolt check vault.db
OK

The expected successful result is OK.

If there are issues with the database, the tool will emit errors.

Statistics

You can also gather essential statistics about the database.

$ bbolt stats vault.db
Aggregate statistics for 2 buckets

Page count statistics
  Number of logical branch pages: 11
  Number of physical branch overflow pages: 0
  Number of logical leaf pages: 297
  Number of physical leaf overflow pages: 135
Tree statistics
  Number of keys/value pairs: 1314
  Number of levels in B+tree: 3
Page size utilization
  Bytes allocated for physical branch pages: 45056
  Bytes actually used for branch data: 22775 (50%)
  Bytes allocated for physical leaf pages: 1769472
  Bytes actually used for leaf data: 1238220 (69%)
Bucket statistics
  Total number of buckets: 2
  Total number on inlined buckets: 1 (50%)
  Bytes used for inlined buckets: 317 (0%)

The stats command performs an extensive search of the database to track every page reference. It starts at the current meta page and recursively iterates through every accessible bucket. The command will emit errors if the database is corrupt.

From the example output, you learn that this database consists of 1314 key/value pairs and that there have been 1238220 bytes actually used for the leaf data.

More information is available from the help output with bbolt stats -h.

List buckets

The Vault database starts with 2 buckets: config and data.

You can list them like this:

$ bbolt buckets vault.db
config
data

List and get keys

The goal of this section is for you to learn how to list keys in more detail.

  1. List all keys present in the config bucket.

    $ bbolt keys vault.db config
latest_config
latest_indexes
local_node_config

    The keys in the config bucket contain information about the latest cluster configuration, including a list of cluster members, latest index state, and whether the local node is a voter.

  2. Get count of the keys in the data bucket.

    $ bbolt keys vault.db data | wc -l
    1311

    The data bucket holds the majority of all data in Vault, and should also represent the majority of the keys found in the database.

    Since there are several keys present in this example, you can explore a smaller subset of the keys contained in the data bucket.

  3. List just the keys which make up the Vault core information.

    $ bbolt keys vault.db data | grep core
core/audit
core/auth
core/autoloaded-license
core/cluster/feature-flags
core/cluster/local/info
core/hsm/barrier-unseal-keys
core/index-header-hmac-key
core/keyring
core/leader/dceb963e-72a2-f1d4-ea8f-c903dbb57dd1
core/local-audit
core/local-auth
core/local-mounts
core/lock
core/master
core/mounts
core/raft/tls
core/seal-config
core/shamir-kek
core/wrapping/jwtkey

    These keys represent the Vault core, and Vault uses them internally to contain critical configuration information.

  4. A more practical example than demonstrating exploration is to find the count of valid tokens in the current Vault data. You can do so by counting the entries under the path sys/token/id/.

    $ bbolt keys vault.db data | grep 'sys/token/id/' | less | wc -l
    228

  5. Another practical example is confirming the existence of a particular policy by name. If for some reason you need to verify that the cluster has a particular named policy, you can list all names.

    $ bbolt keys vault.db data | grep 'sys/policy/' | cut -d '/' -f3
admins
control-group
default
lab-user
ldap-dev
ldap-user
prometheus
response-wrapping
vaultron-example-namespace-ns
vaultron-example-root-ns
vaultron-example-token-admin
vaultron-example-token-identity
vaultron-sudo

    Note

    Along with listing keys, you can get key values for a limited subset of keys which allow unauthenticated access. Vault encrypts all user supplied inputs at rest, and you will not be able to decrypt them using the techniques and tools in this tutorial.

    One such exception is the list of cluster members found in the latest_config key in the config bucket.

  6. Get the latest_config key values.

    $ bbolt get vault.db config latest_config
}%vaultron-vault-010.10.42.200:8201%vaultron-vault-110.10.42.201:8201%vaultron-vault-210.10.42.202:8201%vaultron-vault-310.10.42.203:8201%vaultron-vault-410.10.42.204:8201

    These are the 5 servers which made up the Vault cluster these data are from.

Warning

Attempting to get key values results in binary data, which when directed to the standard output can result in terminal session issues. You might instead consider redirecting the output to a file and examining the file in an editor that can handle binary files.

If you have concerns about your terminal session, you can also try testing the output with file. Using the earlier examples from listing keys making up the Vault core, you can check if it might be safe to view the server's seal configuration in core/seal-config.

$ bbolt get vault.db data core/seal-config | file -
/dev/stdin: JSON data

This is JSON data that provides some helpful information about the server's seal configuration.

$ bbolt get vault.db data core/seal-config
{"type":"shamir","secret_shares":1,"secret_threshold":1,"pgp_keys":null,"nonce":"","backup":false,"stored_shares":1}

While generally not as useful as the vault.db, you can also inspect data in the raft.db. For example, to find the cluster member which was the last candidate voted a leader, you can check the LastVoteTerm key in the conf bucket.

$ bbolt get raft/raft.db conf LastVoteCand
10.10.42.201:8201

Pages and the free list

This section is most useful for users of Vault 1.8.0 and higher who wish to learn more about the free list status.

  1. To begin, list the first 2 pages.

    $ bbolt pages vault.db | head -n4
ID       TYPE       ITEMS  OVRFLW
======== ========== ====== ======
0        meta       0
1        meta       0

    The list has 2 meta pages. You can use the most current one with the highest transaction ID value to find the freelist location.

  2. Check the transaction ID for page 0.

    $ bbolt page vault.db 0 | grep -B2 'Txn ID'
Freelist:   <pgid=306>
HWM:        <pgid=477>
Txn ID:     1712

  3. Check the transaction ID for page 1.

    $ bbolt page vault.db 1 | grep -B2 'Txn ID'
Freelist:   <pgid=174>
HWM:        <pgid=477>
Txn ID:     1711

    Page 0 has the highest Txn ID value, and is the meta page you should examine to find the root of the tree. Note that its Freelist value points to the correct freelist page ID.

  4. List page 0 to find the root page ID.

    $ bbolt page vault.db 0 | grep Root
Root:       <pgid=235>

  5. The root page ID is 235, and from here you can walk the entire tree of pages. Begin by listing page 235.

    $  bbolt page vault.db 235
Page ID:    235
Page Type:  leaf
Total Size: 4096 bytes
Item Count: 2

"config": <pgid=0,seq=0>
"data": <pgid=212,seq=0>

    The output shows the pages representing the config and data buckets.

  6. You can retrieve the data page at 212.

    $ bbolt page vault.db 212
Page ID:    212
Page Type:  branch
Total Size: 4096 bytes
Item Count: 10

"audit/4487ee65-4449-6a32-d0af-518c380a0ede/salt": <pgid=139>
"index-dr/pages/99": <pgid=58>
"sys/expire/id/auth/userpass/login/lab-user-1/   h04a2ea39ac9f9b5fca932aab0f7ed25140db7e06e620926ffb43530a6911b61b": <pgid=162>
"sys/expire/id/auth/userpass/login/lab-user-3/   h999915ecf7b1956d2287c913979ebe3ae6e2d35cca151f7da5e840ecf56b212d": <pgid=302>
"sys/expire/id/auth/userpass/login/lab-user-5/   h74f9c9511f932e06e8f3e574947dd0e8f7279f2cc50cc3d3957320d539e2cfe7": <pgid=221>
"sys/expire/id/auth/userpass/login/lab-user-7/   h44ba7953b65aceec11658b9371546743162fa86f999d7c86ecb3ce39abfa42e7": <pgid=217>
"sys/expire/id/auth/userpass/login/lab-user-7/   hc6386db098e528405526f7b288703038a0358c4e90372e8f5b77e6f62e8b4222": <pgid=301>
"sys/token/accessor/d1db3bdfd1cae11be408a41157996f32646f62af": <pgid=281>
"sys/token/id/h60a215b77e0bc911cae8e988de36a8c1df736c57cf0e9023c0bb2430ea8538f3": <pgid=196>
"sys/token/id/he32264f2dc56be09df07c4ed7b2b45987b440e459c9c622183527ddbf4ca41dc": <pgid=194>

  7. Page 162 represents a username and password auth method lease id value. You can list this page ID to view its contents.

    $ bbolt page vault.db 162
Page ID:    162
Page Type:  branch
Total Size: 4096 bytes
Item Count: 17

"sys/expire/id/auth/userpass/login/lab-user-1/h04a2ea39ac9f9b5fca932aab0f7ed25140db7e06e620926ffb43530a6911b61b": <pgid=64>
"sys/expire/id/auth/userpass/login/lab-user-1/h4a13e2e920de1a69720984c2c643b172d7a451bcbf4d1bd90b1d5ba3bc56ed2d": <pgid=98>
"sys/expire/id/auth/userpass/login/lab-user-1/h9f7b901d7bddc6407d82ef442c79bb6407804c5717c999ea74cc81a434f4efde": <pgid=41>
"sys/expire/id/auth/userpass/login/lab-user-1/he93aa7f0493831dbdb5fada98f790f6b66eb1cbde985dbc49962629b7cd95fe0": <pgid=125>
"sys/expire/id/auth/userpass/login/lab-user-3/h021e7aadf3008a2bb47ea917a6de8402439bd4637b78b11d56b372f5f1872627": <pgid=169>
"sys/expire/id/auth/userpass/login/lab-user-3/h0cf7e2818d1f43250793cc3adf37d2cd65ea6421fe2ca1411b8ed59885f23b8b": <pgid=53>
"sys/expire/id/auth/userpass/login/lab-user-3/h16a29ff005344742a5ca44a27d04bcc26467012229b9aba5977266e4a7e5888c": <pgid=118>
"sys/expire/id/auth/userpass/login/lab-user-3/h20fef3012a68b1f3d9a0e3f42d08d5f22dfc9d5584582777538e8be1aeda9d78": <pgid=198>
"sys/expire/id/auth/userpass/login/lab-user-3/h32be213efb25de70f7401a6955a3b716f0e4dcddc654629edfacaa5754742e41": <pgid=134>
"sys/expire/id/auth/userpass/login/lab-user-3/h4109fc3b1d299624d81eabc9d7e5be4e0bf03d7bfe28d4b4a1de439b99f0f03e": <pgid=155>
"sys/expire/id/auth/userpass/login/lab-user-3/h44e2ad5e382b80ceb2ab45d10710d9feb1468d0baf853c7f94df57e2b0db7aa8": <pgid=4>
"sys/expire/id/auth/userpass/login/lab-user-3/h494323e1dcad8ae427e163d0fffcc6bbf317c8a6d9c3489d98d3cf22fe98cfdb": <pgid=181>
"sys/expire/id/auth/userpass/login/lab-user-3/h5d7b4c21ae89c98952a45bb6bbae4698f2a1c0d7715022e2f9456092ab423b7f": <pgid=105>
"sys/expire/id/auth/userpass/login/lab-user-3/h69290ffe0e01a35568c381ac6b4a6cea66c2947d30c4266d8410a45f65794216": <pgid=66>
"sys/expire/id/auth/userpass/login/lab-user-3/h7396c835ae280c4115c01835e5f83b072eafef5480e758fe254bc7135fd610a2": <pgid=138>
"sys/expire/id/auth/userpass/login/lab-user-3/h78b7a2a0688a3f6afe17e6d34cb9b96cbf926fc26a891300c220390fbc5adba8": <pgid=122>
"sys/expire/id/auth/userpass/login/lab-user-3/h885ab1f14922f3310924bac61019c8a894dc3d55bbb1fa98f1cc71ddb1e2d318": <pgid=123>

  8. Now you can view the leaf page for an individual lease id, for example the one at page ID 64.

    $ bbolt page vault.db 64
Page ID:    64
Page Type:  leaf
Total Size: 4096 bytes
Item Count: 3

"sys/expire/id/auth/userpass/login/lab-user-1/h04a2ea39ac9f9b5fca932aab0f7ed25140db7e06e620926ffb43530a6911b61b": 000000010232b3d254a9581ad479364b7ba235aa11a91bc225adc8c447e0ec267ba2c489492ff4d5df7891aa3fc91de61cd30bab462a91674f13117f613e9f6ae7bb1e903eaaed485db5bde4e98debd601a0f5303fcc5f0244df3d4795f1b5a5db7fb8ff9101e8dabd43b092902043af6dc59755acafe0730850410d37c8717f65cd09e015c15ddc665515b37ca8be8f51d67abc7ab093183a8d11a4c2edf56bce6a53691e0b066c7b2c37ff3d7062169850f79fa8d89602ad4b6bb5fd3c7d741939fa8bc3dd84373a1e86f79ebabcc1ff91c38728d57806483d2d4e03d6aef414ba0c8826ddc9b4d953220c7dda59cde87b71a4db778e365dd41f27e6d0f7089074aa82eb0ced12d5cf3cd9c241094e0ba034fdaab13f264adf4a73341a3944c08b16ba7581cc5c546cfea21909adde5792530aec8c45d6b3a491541f48ad194164a8ffd773908247c7f649f9d07e962a2addabc2742aaea6864417ae8e3e86a914652f2f72cc1b8abab2be7348208c46ca485887a7bb12579ea0e6e13f6d61ada5f339f1a5f2fe8999a46efb05ce9582d942094e2055828642c69e67648de867d9e1e14bdfc5f526e10c569de74e695b1c58e649ce50d02eab8268456010a3126fc079012af2dfede424913fcd756ea1f7498a7eb584f03bfc59b408a3205bbed61c931131cd2ae67a2dd985439a2924efb73f9b49ba7acdeea23e67226fbc4225f158e2975b7cd363661f7efb1ee3ae2ac7ff0930e2af8fe59478920f3b86fd840dd1280c901d2ad1ea174c850b893612d58fcb5d45a32e5e08a1167925a2a506d57aef0c0e0cf5d11205d346737904dfa3f5c89c624728241d7b98176a26dc8deb5e6e7e9ddbe6df7847922d7976920ad699f79a94c2465627f8b41b1680170ad2b5c07d872739b73e78f3b3b44a8b9a00438316c611d66d99ec6db0bc07522324d3b1c9be79f76cfd179e6b2e65a3afc771c3e9a84dac58471fc1e54390b00a3c4f0166e67fce6ff121527abe45b363e63404c6f0290687bdc7fa1ff29ae61f35f4bad69ccadc71da8e2af4632ac565e931cfa0ad4919fa802a2c1b5e557796b9e8ca4b93b25b2c991a87079c8d18fc03debb416d8be6ae3f4d10b49f65fef46415428c14111d4f20a767b1218da34f7f3343bde1268461e547fd74d8841cd0fc22acd1116dda4e4be9d8bbd100b125318e22ec04568ba2603c9aebc249b9fdc7837557d87bf77ff8a496c159a25bd531c3f5fcb5a11267f52c1fdfd51bc330fb3ed84903d7755850be54fa7cdf9726a9da77e330f1dd934d84c3e6b4afe13d4ba574fe95695ece561c8e9b936b56ff905fc045b07c9bc2494bf48ce60588bef4ec5b213384ad97af5e9a82dc3ffdd8721931a30aa707262d8ef796f300eed4eb3f89d2ae7b0586a4d4a889ccb55d29ce348a517fe40622b1addb8e2f24d92b0271ea37bd7674de741749af6ded672abcbc5d238aad4f39c8496f996db894cb9029ab8fbbe4b7c29924b67606daaef0f2f686cc5362544306f1fba53d380f1ed5777e8f38d47e9fbf011f45
...snip...

    The result is a key sys/expire/id/auth/userpass/login/lab-user-1/h04a2ea39ac9f9b5fca932aab0f7ed25140db7e06e620926ffb43530a6911b61b with a long hexadecimal string representing the encrypted value.

    You can explore the entire tree this way.

  9. With the freelist itself, recall that meta page 0 was the page ID containing the most recent Txn ID. Check the Freelist value for meta page 0.

    $ bbolt page vault.db 1 | grep -B2 'Txn ID'
Freelist:   <pgid=174>

  10. The freelist page ID is 174. Go ahead and list it next.

    $ bbolt page vault.db 174 | head -n 15
Page ID:    174
Page Type:  freelist
Total Size: 4096 bytes
Item Count: 31
Overflow: 0

57
58
151
189
194
212
235
306
313

    Note the presence of 31 pages in the freelist.

    The other way to verify a page is free, is to check for its page ID in the pages output, and ensure that free appears next to it.

  11. Confirm that page 313 is free.

    $ bbolt pages vault.db | grep -w 313
313      free

Using the boltd web UI

Note

You will just read from the database using this boltd workflow, and the tool doesn't have any potentially destructive functionality.

Another tool from the original BoltDB project, boltd exists to explore the database visually.

You can use boltd by attaching it to a Vault database and connecting a web browser to the web UI it exposes.

  1. Install boltd.

    $ go get github.com/boltdb/boltd/...

  2. Start boltd and attach it to vault.db.

    $ boltd vault.db
Listening on http://localhost:9000

  3. Access http://localhost:9000 in a browser.

    boltd web user interface

    The user interface is minimal, and defaults to displaying a view of pages.

    You can use it to browse facets of your Vault data like the bbolt CLI tool, by navigating links or jumping directly to page numbers in the Go to page input.

    You can also explore a page usage histogram by selecting Show Page Usage.

    page usage histogram

Summary

You have learned about how to inspect data in the bbolt database file for a Vault server using the Integrated Storage backend. You learned about 2 tools which you can use for this purpose along with caveats around their usage.

You can learn more about inspecting Vault data in Inspect Data in Integrated Storage and Inspecting Data in Consul Storage.

Clean up

When you finish the hands on lab, you can clean up by changing to the parent directory and removing the learn-vault-monitoring directory.

$ cd ../..

rm -rf learn-vault-monitoring

Help and reference

Previous

Monitor telemetry & audit devices

 Next

Inspect data in Integrated Storage