Apache Calcite Architecture and Streaming SQL

What is Apache Calcite?

Apache Calcite is an open source dynamic data management framework which is licensed by Apache software foundation and is written in Java programming language. It consists of many things that comprises a general database management system, but does not have the key features of it like storing the data and processing of that data, which is done by some specialized engines. There are several Apache projects which uses it and those are -

Apache Hive
Apache Flink
Apache Drill
Apache Storm
Apache Samza

It includes many components like Sql parser, Query optimizer, Sql generator, Sql validator and an API for building expressions in relational algebra.

A community-driven open source framework for shared Big Data Analytics. Click to explore about, Apache Flink Architecture

What are the benefits of Apache Calcite?

Though Apache Calcite Architecture does not support storing and the processing of data, but it have sevaral features which maks it beneficial to use, and that are as follows -

Open Source Friendliness - It is an open source framework maintained by Apache Software Foundation. As it is written in Java which makes it easier to operate with many data processing engines that are also written in java or runs on JVM based environment, especially when talking about the Hadoop ecosystem.
Multiple Data Models - As Calcite supports query optimization and query execution but does not supports storing and processing of data, which makes it good for mediating between the applications having more than one data storage locations and also using multiple data processing engines.
Cross System Support - Here, cross system support means that Apache Calcite framework is able to run and optimize queries across several query processing systems or engines and database backends like Apache Spark, Hive, Flink, Drill and many others.
Support for SQL and its Extensions - As many systems does not provide their own query language and prefers the existing one like SQL. So, for those systems, Calcite provides supports for SQL dialects and extensions.
Reliability - Calcite is much reliable as it contains an extensive test suite which validates all the components of the system and also includes the query optimizer rules and the integration with the backend data sources.

How it works?

In this before passing any querry, we have to tell the query planner that what schema and tables are currently available. So , the first step is to parse the model and should generate a schema object, and for this purpose a Calcite’s ModelHandler is used and for writing a model handler we have to write a dummy implementation of Calcite connection interface and use it to pass state to ModelHandler. After invoking the ModelHandler, from that dummy connection object we will get the schemaobject and use it with the query planner.

An open-source, distributed processing engine and framework of stateful computations written in JAVA and Scala. Click to explore about, Data Processing with Apache Flink

What is the architecture of Apache Calcite?

There are various components which comprises in Apache Calcite Architecture - SQL query Parser and Validator - It translates the sql query to a tree of relational operators. As, it does not supports storing of data, so it provides a method to define a table schema and view external storage engines with the help of adapters.

Simple Planning Queries
Optimized Query

Although Calcite provides optimized SQL support to the systems which needs it, it also gives optimization support to the systems that already have their own language parsing and interpretation.

What are the challenges of Query Optimizer?

Some of the challenges facing in query optimization and making an efficient query are join optimization and the table size when using tables. As a join combines rows from two tables with the help of the values of the fields which are common in both. So, the order which is used to join the tables will impact on the data set size and will impact on the performance. So, one of the main advantage of query optimizer is the ability to find the best join order. The join order is not defined in the execution plan of the query. The query optimizer considers two different type of trees and those are -

Left Deep Nodes

All the internal nodes of the left deep tree have atleast one leaf node as a child node.Althogh it is simple to implement but this technique does not gives an efficient results.In this technique we firstly joins the two tables and after that the resultant table is joined with the third table, and the resultant of that table gets joined with theanother one and so on until the join operation gets performed.

Bushy Trees Nodes

In this a join operates on the result of two join. These trees also contains subtrees and their joins with other subtrees are more efficient than that of left deep. In this technique, if there exists four tables to be join, then we first apply join operation on the two tables, after that join is performed on another two tables are join and after getting the resultant table of both the join operations, the final join operation is performed in order to perform join non all the tables.

Our solutions cater to diverse industries with a focus on serving ever-changing marketing needs. Click here to talk to our Technology Specialists

Conclusion

As Apache Calcite Architecture does not support storing and processing of the data, so this feature of it helps in mediating between the applications which are having more than one data storage locations and also using multiple data processing engines. Some of the processing engines which adopts Calcite are Apache Hive, Drill, Storm and Flink. If you are looking for more information regarding databases, we recommend you to give a read to