Have you heard of BitDB? (If you haven't, check out this article to learn more)
It's a universal bitcoin database that lets you easily build serverless decentralized applications. People are building all kinds of cool applications using bitdb. (Now you can even serve a full website from the blockchain using BitDB! : Launching a Website on the Bitcoin Cash Network Is Now a Reality)

Previously on BitDB

If you've tried BitDB, you've been probably using the built-in JSON based query language.
While useful, this language lacked some critical features and was far from satisfactory in terms of its expressive power and extensibility.
I've been internally experimenting and iterating on different approaches, and have finally reached a point where I can officially announce it (as well as give it a name).

Introducing Bitquery

Today I am releasing Bitquery, a Turing Complete Bitcoin Query Language for Building Immutable Unstoppable APIs.
Also, I'm open sourcing Bitqueryd, the query engine that lets you interact with BitDB using bitquery language.
The main innovation is that it's a programmable query language. It has the expressive power to not only "query" but also "process". Through its programmability you can now easily build application-specific immutable API endpoints on bitcoin.
Note that while Bitquery is the default query language of BitDB, it's completely decoupled from the core bitd module to maximize extensibility of each module going forward.
  • Bitd: An autonomous engine that writes to BitDB
  • Bitqueryd: A query engine that reads from BitDB through the bitquery query language.

Quick Intro to BitDB

Before we discuss Bitquery, let's quickly go through what BitDB does:
  1. It crawls Bitcoin
  2. Parses raw transaction into a structured JSON document
  3. Stores it into MongoDB
Here's what a raw transaction looks like:
BitDB transforms it into a structured object and stores it into database, like this:
You can learn more about what each attribute means here
As you can see, we now have a structure that can be easily queried against.
So now that we've written data into bitdb, how do we read from bitdb?
One way would be to leave it up to app developers to write their own custom code to handle querying, encoding, and processing the DB response on the client side.
But this will mean every application writes custom client side code for interacting with BitDB. And this can hurt interoperability, portability, transparency, and decentralization.
A much better solution is to use a uniform high level query language that includes all of this logic in a single request, which would ensure application portability--if one bitdb node goes down, an application can easily migrate to another node without interruption because they all speak the same language to talk to bitdb.
And THIS is Bitquery.

Bitquery: Build Immutable APIs on Bitcoin

Bitquery lets you build immutable, unstoppable APIs on Bitcoin.

1. Before

Let's take a look at an example raw BitDB object. The structure and the attribute names are not exactly "human readable". You can't really call this an "API". It's more like a raw database dump:

2. After

On the other hand, here's what a Bitquery response looks like:
As you can see, the query result is human readable, and INDISTINGUISHABLE from any mainstream API providers like Twitter, Facebook, etc...
So we get the best of both worlds--the usability of mainstream APIs, and the security of Bitcoin. This is possible because Bitquery:
  1. not only lets you Query Bitcoin through BitDB (which is deterministically derived from Bitcoin)
  2. but also lets you Process the BitDB response with a Turing complete language
  3. all within a single query language, letting us create "API endpoints" that are containerized within a single portable JSON object.
So how does this Bitquery actually work?

How Bitquery Works

Bitquery is a declarative query language that has the expressive power to contain all of the following within a single JSON object:
  1. Querying: Query BitDB using the native JSON based query language for MongoDB.
  2. Encoding: A built-in encoding scheme for dealing with bitcoin script push data.
  3. Processing: Process the BitDB raw response and turn it into whichever format you desire through a built-in programming language.

Here's an example:
Let's go through each highlighted part:
  1. Protocol Version: The query language is 100% self-contained (doesn't require a 3rd party schema to interpret its semantics) and will evolve over time, so it's important to declare the version.
  2. Query: Bitquery implements a 1:1 mapping from JSON query object to native MongoDB query API. Basically it's a pure MongoDB query.
  3. Process Function: This part is powered by jq, a Turing complete, stack based functional programming language.

1. Querying

The query part is simple, it's just a mongodb query. Once you understand how each bitcoin transaction is indexed in BitDB (as explained above), this part is as simple as making a regular mongodb query request.
So if you're looking for a transaction output that contains OP_RETURN as the first push data (index 0), you can write:
The opcode for OP_RETURN is 106
And to search for output scripts that are OP_RETURN and and contain the text "hello" in any of the push data, you can write:
Again, it's just a regular MongoDB query. You can learn more here: https://docs.bitdb.network/docs/query_v3#b-query

2. Encoding

Another important part about Bitquery is its encoding handler.
Internally BitDB stores each script push data under "b0", "b1", "b2", ... as base64 encoded string (or as an object if it's an opcode)
But each application protocol powered by bitcoin script can have their own custom encoding scheme per script push data. One app might be using hex encoding for a certain push data, while another might be using UTF8 encoding for some of its push data.
Therefore we need to specify the encoding of the attributes we're querying for. Bitquery makes this easy through the built-in attribute name convention:
  • b0, b1, b2, b3, b4, ... : use these to query the push data in base64 encoding (or an opcode)
  • h0, h1, h2, h3, h4, ... : use these to query the push data in hex encoding
  • s0, s1, s2, s3, s4, ... : use these to query the push data in UTF8 encoding (The s-attributes are also indexed for full-text search)

In the earlier versions the query language included an extra object to describe these encoding schemes, but this was too verbose, not flexible enough, and not very intuitive.

For example, if you want to fetch Memo.cash posts whose output script looks like this:
You can write a query that looks like this:
Quick explanation:
  1. OP_RETURN: It's index 0 push data, so the query looks for out.b0 of {"op": 106}
  2. 0x6d02: It's index 1, and is hex format, so the query looks for out.h1 of "6d02"
  3. Hello: It's index 2, and is UTF8, so the query looks for out.s2 of "Hello"
You can learn more about the encoding scheme here

3. Processing

Finally, here's the last piece of the puzzle that's been missing from the previous versions of bitdb's query language--programmability.
Without programmability, bitdb simply returns a full transaction object, which as we saw above, is too raw and not immediately usable unless developers write additional code to process them.
The ideal approach would be if we could somehow introduce programmability into the query language itself so everyone who wants to make use of an API endpoint can use exactly the same standard query, without having to write any extra processing code.
This point becomes obvious when we compare with mainstream API providers such as Twitter. Twitter has clearly defined, purpose-driven API endpoints that return objects with attributes that are immediately usable and human readable, instead of returning a full database dump.
To achieve this, Bitquery adopts jq, a popular Turing complete functional programming language that fits into a single line string.
jq was designed to be used in a unix command line setting, and this "command line" constraint gave rise to a programming language that is Turing complete yet can be packed into a single line. And this quality is perfect for something like Bitquery.
For example, here's a bitquery that makes a query to BitDB (q.find), and then passes it to the response processing function (r.f) to extract out the .blk.i, .blk.t, .out[1].s2 attributes and assign them to the human-readable attributes block, timestamp, and content, respectively:
This is just one example but remember, jq is Turing complete, which means you can process the query response to transform it into ANY format you desire.
You can learn more about jq syntax here: https://stedolan.github.io/jq/manual/#Basicfilters
And THIS is how you can create an immutable and unstoppable API on Bitcoin.


Enough with the theory. Let's actually try something tangible.
Go ahead and copy and paste the code in a text editor, save it as an HTML file, and open it in a browser:
<html> <script> // Bitquery for filtering memo.cash posts (6d02) // and extract out only the messages var query = { "v": 3, "q": { "find": { "out.b0": { "op": 106 }, "out.h1": "6d02" }, "project": { "out.$": 1 } }, "r": { "f": "[ .[] | {msg: .out[0].s2} ]" } }; // Make an HTTP request to bitdb.network // public endpoint with an API key fetch( "https://bitdb.network/q/" + btoa(JSON.stringify(query)), { "headers": { key: "qqjtyk2qmphm4d2ddqfeg855we2u4ly7m59a0tuenn" } } ).then(function(res) { return res.json() }).then(function(res) { // Render the response! res.c.forEach(function(item) { document.write(item.msg + "<br>") }) }) </script> </html>
It just works!
The best part about bitdb is, you can get started immediately without setting up some complicated "developer environment".
This simplicity is possible because Bitdb doesn't invent some new convoluted technology for decentralization. Instead it depends on Bitcoin which is super simple and minimal, and Bitdb can afford to provide ideal developer and user experience without sacrificing security. Other "blockchain" projects have given up this simplicity to build their own uber-complex solutions by adding new unproven concepts and data structures, and that's one of the main reasons why it's so difficult to work with those technologies and even the very developers who build applications on top of those platforms mostly have no idea what's going on underneath.
With BitDB, it's all just Bitcoin. All you have to remember is that BitDB turns Bitcoin transactions into a structured format and stores it in a MongoDB instance. And you can add additional levels of security simply by cross-validating with multiple Bitcoin nodes.
Of course, the HTTP API is not the only way to access bitdb, there are two ways: 1. Direct: Directly connect to the MongoDB instance through bitqueryd. You can easily do this if your own bitdb node. 2. HTTP API: Connect to a public BitDB node via HTTP API. Most developers shouldn't have to worry about running a server. You can simply connect to a public bitdb node via HTTP with no hassle.

Conclusion: What Does This All Mean?

1. Deterministic like a chain of math functions

Here's a high level view: We can think of the whole thing as a series of mathematical functions.
  1. Index: BitDB crawls, parses, and indexes Bitcoin's raw transaction in a structured manner. The whole process is deterministic, therefore we can be sure that querying BitDB is equivalent to querying Bitcoin. We can think of this as a "function" that takes the entire Bitcoin universe as input and returns the derived BitDB as output.
  2. Query: The first step of Bitquery is the actual query request into the MongoDB instance. This could be seen as a "function" as well. It takes the query object as "input", and returns the response as "output".
  3. Process: The second step of Bitquery is the processing part. This is where you can manipulate the DB response into any format you want using the transparent programming language (jq). This is also a "function" that takes the BitDB query response as input and returns Application API as output.

2. Build Immutable Unstoppable APIs

Because the entire derivation process (Bitcoin to BitDB to Bitquery to Application APIs) is completely open source, transparent, and has zero side-effect, we can conclude that the resulting application api transitively benefits from the immutability and determinism of Bitcoin, secured by Proof of Work.
At the same time, we end up with an API that can be so user-friendly and developer-friendly that it's indistinguishable from any existing mainstream cloud API providers.

3. API that can seamlessly talk to one another

Because we can trust that each API created this way is immutably derived from bitcoin, we can trust the resulting immutability and build one API on top of another. Multiple parties can build and publish their own APIs, and these APIs can communicate with one another.
Also, now that we have human-readable APIs, application developers can easily come up with an additional layer of open standards for such inter-application communications.
There are many more benefits and much more potential I haven't covered here, but I will leave it up to the reader's imagination.


11 of 11 reviewers say it's worth paying for

0 of 11 reviewers say it's not worth paying for
  earned 0.0¢
This is astonishing! Good job!
   1yr ago
  earned 0.0¢
Amazing work as always. Does the SLP tokengraph use bitquery? thx
   1yr ago
  earned 10.0¢
@Jonald_Fyookball thanks!
Yes, Bitquery was extracted out from Tokengraph so it's been in "production" for a while now (especially the "processing" part), it's just that it took some time for me to actually formulate this into an open-source extensible query language.
I'm trying to break down all the moving pieces into interoperable modules, so at the end of the day something like Tokengraph will be just a matter of composing multiple minimal modules together.
   1yr ago
  earned 0.0¢
Great work
   1yr ago
  earned 0.0¢
I am working on integrating this into my site's database right now.
Will I be able to have my MySQL database update automatically from bitDB using this new tool?
   1yr ago