Off-Chain Meta Data For Blockchain Solutions - What Choices Exist?

Off-Chain Data:   Typical Blockchain  network  is  meant  to  host  business  transactions  that  are of relevance  to  all  consortium members.  However  there  are  lot  of  information which will  be useful  to  the successful  execution  of  Blockchain  transactions  yet  not  part  of the core  business to business  transaction. This  set  of  data  typically  a  meta-data  about  the  blockchain  is  known as Off-chain  data. 

The  following  are  typical  use  of  Off-chain  in a Blockchain   scenario.

• Enables  storage  of  meta-data  to  help the successful functioning of Eco System, which includes authentication, authorization, role based security, fine grained access, External data needed for successful  execution of  smart contracts.
• Helps  with Scalability  of  Blockchain  solutions  so that  storage  intensive  data can be stored  outside of blockchain network
• Need  not  be  constrained  by  Data Format, Syntax  and Semantics  of  blockchain platforms but rather  can  store  any  type  of  data that  may be useful to the  over all business use case.

One  of  the biggest  challenges  with  Off-chain  data  is  that,  still  off-chain  needs  to be available to  all the  blockchain  users   i.e  to all consortium members  when it comes to   private permissioned blockchain. So  the  off-chain  data  itself  needs  to  follow  the  tenants  of  blockchain network  such that the  data is quickly  replicated to all  participants.

So  what  are  our  choices to effectively  implement  Off-Chain  metadata  as  part  of  our  Blockchain solutions.

Microsoft Coco Framework :     Microsoft  has  recently  released  the initial  technical  details  about  Coco  Framework,  which  provides a  architecture  to  segregate  the  core  of  the  blockchain network  from  the  metadata  about the eco system. Coco  framework  implements  a  key-value pair persistent  layer  which  itself  is  distributed  on top of the   Blockchain core. Coco framework also talks  about  two  types  of  transactions,  one is the  application  transaction which are meant for the underlying  blockchain  and  the  second one is  the  administrative  transactions which support the  functioning  of  the  blockchain network.  At  this  time  further  details  about  the  Coco  framework are  yet to be available,  however  this  provides  a  promising  option for  off-chain metadata for blockchain. However  at  this  time  it  supports  key,value  pair  persistent  store,  so  there needs to be some work for  storing  semantically  rich  metadata  like  images, blobs, multi media content  etc...

Bigchain DB :  Bigchain DB is the  first  attempt  to  integrate  big data like scalability into the blockchain databases  and  to  make  both one on the  same.  This approach  practically  relieves the need  for  off-chain  rather  the  Blockchain (which itself on  a  Bigdata platform)  stores  all the data both on-chain and off-chain. Bigchain DB  has  two layers  of  database, the lower level is handled by Mongo DB  federation and the higher level is handled by the  Bigchain Core DB. As  Bigchain DB is implemented  on  a rich  document  centric  database,  storing  metadata would be  much easier with a support  of  larger  set  of  data types  and  richer data models. However BigchainDB  in itself a blockchain  database also,  which means  its  Smart Contract framework  has to be robust  enough like  Ethereum, HyperLedger blockchains.  At  this  time  it is not  sure  whether  Bigchain database  can be integrated  with  Ethereum or Hyperledger to make it a effective off-chain solution. But considering the  Scale ,  Bigchain DB  will be a  good choice.

Bigchain DB  has lot of potential  to replace  traditional  transactional databases  especially when there is a need  for  peer to peer replication  as well  as  IoT  scenarios.

Azure Cosmos DB :    Azure Cosmos DB  is a globally distributed multi modal  database,  which means that  it  provides  global  distribution easily with   a  simple  configuration  option,  also it supports  multiple data formats like  Document DB, Mongo DB, Graph DB, Table DB. This means any kind of  metadata semantics  can be represented  using  Cosmos DB and yet  it will be geographically scalable to be available  to all  consortium members. However  in a classic way it is disconnected  from the  Blockchain  database.  Also  Multi Master  distribution  is  difficult  to implement though it is possible,  where  as  the  Read  Only  Snapshots  are  much  easier to implement.

The  following  diagram  courtesy  of  Microsoft  Azure,  talks  about  Multi Master Replication is Cosmos DB.

However  it  requires  that  lot of  manual  setup  behind  the scenes  especially  the need for application to  UNION all the data  from  multiple regions for a  global  read perspective.

AZURE SQL With Geo Replication and Chaining :    Azure SQL  which is the  Relational database as a service  on Azure,  has a  Geo  Replication feature which facilitates multiple  Read Only Secondary copies  of a  primary  database thus enabling  distribution of meta data across  consortium members. We  can  create  Secondary  on Secondary  (which is known as Chaining)  there by achieving  unlimited  scale. Considering  that  most  of the off-chain metadata of a consortium blockchain may  be initiated  from  a single member  who would  typically  be the  leader of the blockchain consortium, this  approach of Azure SQL  with Geo Replication can still be a  effective off-chain database solution.

Multi Master Scenarios  can  be  implemented  programmatically using the  Distributed Transaction Approach.  This  facilitates that  multiple members  can  write  to their own node  and yet keep it sync across  the members. However this approach  may  have to worry a  lot about  integrity  of  data across  nodes.

The following  code  piece  Courtesy of Microsoft Azure  indicates  a  distributed  transaction across databases.

using (var scope = new TransactionScope())
{
    using (var conn1 = new SqlConnection(connStrDb1))
    {
        conn1.Open();
        SqlCommand cmd1 = conn1.CreateCommand();
        cmd1.CommandText = string.Format("insert into T1 values(1)");
        cmd1.ExecuteNonQuery();
    }

    using (var conn2 = new SqlConnection(connStrDb2))
    {
        conn2.Open();
        var cmd2 = conn2.CreateCommand();
        cmd2.CommandText = string.Format("insert into T2 values(2)");
        cmd2.ExecuteNonQuery();
    }

    scope.Complete();
}

There  could  be more approach  to  help enterprises  solve  the  Blockchain Scale  and  Metadata needs  with  a  distributed  off-chain store.  Let  me know of any other options that  come to your mind as  this is more  of Azure  and Microsoft  centric  approaches.