Kwil DB: Decentralised Open Source SQL
Kwil describes itself as the “first free to use fully decentralised web3 communication platform”. In a tweet on Thursday Brennan Lamey, the founder of Kwil, announced that they were opening up their custom decentralised SQL database (Kwil DB) for use by anyone essentially open-sourcing it. Kwil DB is used to power the backend of Kwil Social.
We stripped out the backend of @kwil_app, so that now anyone can utilize a custom decentralized SQL database on top of @ArweaveTeam .
Kwil DB allows for responsive, scalable, community owned decentralized SQL databases. Curious how it works, and how you can use it today?
— Brennan Lamey (@brennan_lamey) February 16, 2022
What is Kwil DB?
Kwil DB is an SQL database built on Arweave which has its own file system, auto-bundling functionality, and a data verification method called “Proof of Secret”, as well as taking pools to ensure that the nodes store the correct data, as if they don’t they get slashed, essentially losing part of their stake.
Proof of secret is a method that essentially allows each database query to be only as big as it needs to be without excessive data sizes and transactions.
In order to start using it you first need to create a data moat, which is essentially a history of all the previous states that data mode has found itself in – just like a history of all transactions on a specific Blockchain that is specific to that data.
When someone creates a data moat they are provided with an RSA key pair – a public-key cryptosystem that is widely used for secure data transmission – and three secrets – random 32 bytes strings. The three secrets provided are as follows:
- A past secret
- A current secret
- A future secret
Incoming data writes contain contain the query, a timestamp, a hash of the query and timestamp, a unique query ID and an RSA signature. All these are used and hashed in certain combinations that allow the nodes to simply check to see if their data matches the data of another node, and if not take action by voting to slash the malicious node’s stake and resubmit the query.
If a consensus cannot be met the current secret is revealed to help validate and reconstruct all affected queries, while the future secret becomes the current secret and a new future secret is generated. If you want to get into the real technical side of how proof of secret works then check out Brennan’s extensive thread on the topic here.
The benefit of using Proof Of Secret is, as Brennan himself puts it:
What is the benefit of using this “proof of secret” model?
If Kwil DB were to store RSA signatures for every query, then there would be an additional 1024 bytes of data for each data write. When SQL queries can be as small as <100 bytes, this is pretty inefficient.
— Brennan Lamey (@brennan_lamey) February 16, 2022
Proof of secret and data moats allow you to have multiple branches of data. Thus if there is an issue with one data moat, new queries can keep on coming in whilst the validity of previous ones is being checked. Again, in the words of Brennan:
Secondly, the ability for secrets to evolve means that, even if the secret gets leaked, communities can take action to correct the situation without disrupting the flow of incoming data, and simultaneously allow new nodes to sync old data without checking the hash of each query.
— Brennan Lamey (@brennan_lamey) February 16, 2022
Kwil DB is currently operating a test validator which supports all the SQL queries and file system writes which you can utilise for free via their NodeJS API.
By combining the ease-of-use of a SQL database with the capabilities of the Permaweb, Kwil DB enables a new form of open data composability previously impossible in Web3.
We're excited to see the amazing things the community will be able to build with a Permaweb SQL database!
— Brennan Lamey (@brennan_lamey) February 16, 2022
The problem with most databases is the same problem that many web2 applications and storage solutions face which is that of centralisation. Having a decentralised database and methods to prove the validity of the information stored within might be something highly crucial to the development of Web3 centralised apps (dApps).
Join our
