Hyperledger Fabric CouchDB is killing my cloud storage bills

Hyperledger Fabric is a nice DLT platform and offers great customization options. One of which is the ability to choose different databases to store blockchain data. The recommended and best supported option is to use a CouchDB. It offers the ability to use complex queries on the blockchain data via Hyperledger Fabric Chaincode via different programming languages like go, JavaScript or Java. To find out more about Hyperledger with CouchDB take a look at the offical documentation .

In the picture below you can see the CouchDB overview of an example Hyperledger Fabric node with one channel “compactionchannel” and the chaincode “cdbc”. This is just an example to demonstrate the CouchDB structure and doesn’t contain much data.

Blockchains like Hyperledger store data indefinitely and so the overall amount only increases. To enable the mentioned complex queries the current “world state” is represented in CouchDB. When data changes the corresponding “world state” also changes – CouchDB only appends the new value to its internal representation. Normally the automatic compaction would remove unused data continuously and keep the size of the database small. 

In some circumstances the automatic compaction will not start. We encountered this situation at one of our customers. This lead to continuous growth of the database and tons of unnecessary GBs of storage. In this case you have two options, reconfigure the automation trigger or start the compaction manually.

Status of your Hyperledger CouchDB compaction

You can query your compaction configuration with the following command:

$ curl -H "Content-Type: application/json" -H "Authorization: Basic <Basic-Auth>" http://<CouchDB>:5984/_node/nonode@nohost/_config

{
  ...
  "compactions": {
    "_default": "[{db_fragmentation, \"30%\"}, {view_fragmentation, \"30%\"}]"
  },
  "compaction_daemon": {
    "check_interval": "300",
    "min_file_size": "256000"
  }
}

Manual compaction

You can start the compaction manually via the following REST command.

$ curl -H "Content-Type: application/json" -H "Authorization: Basic <Basic-Auth>" -X POST http://<CouchDB>:5984/<channel-name>_<chaincode>/_compact

{"ok":true}

To check the process of compaction:

$ curl -H "Content-Type: application/json" -H "Authorization: Basic <Basic_Auth>" http://<CouchDB>:5984/<channel-name>_<chaincode>

{
  ...
  "compaction_running": true,
  ...
}

Furthermore not only the database but also views can be compacted. To do so another REST call can be used.

curl -H "Content-Type: application/json" -H "Authorization: Basic <Basic-Auth>" -X POST http://<CouchDB>:5984/<channel-name>_<chaincode>/_compact/<view-name>

{"ok":true}

Depending on the size of the Hyperledger Fabric database or view the compaction can take quite some time.

Detailed information about CouchDB compaction can be found in the compaction page of the official documentation. The German version of this post can be found here .