MatrixOne-v1.0.0-RC1

We are thrilled to announce the release of MatrixOne 1.0.0-RC1! This milestone marks we have basically achieved the initial design goal of the MatrixOne open-source project. Cheers!

MatrixOne 1.0.0-RC1 Review

MatrixOne is designed to provide a unified and scalable database management solution for transactional, analytical and streaming workloads and powers a wide range of applications. In general, MatrixOne 1.0.0-RC1 has delivered a cloud-native architecture with separated storage and computing, presenting a fully-functional database with high performance queries and elastic scaling with familiar relational SQL. The OLTP and OLAP performance of MatrixOne has also reached the industry's average level, meanwhile MatrixOne provides an experimental function of Stream table and Kafka connector for streaming processing.

SQL

• Syntax: MatrixOne SQL syntax is highly compatible with MySQL 8.0.
• DDL: Supports common DDL statements such as CREATE, ALTER, and DELETE databases, tables, views, and partitioned tables.
• DML: Supports common DML statements including INSERT, UPDATE, DELETE, and data import/export statements.
• Basic SQL query: Supports basic query capabilities such as grouping, deduplication, filtering, sorting, limiting, regular expressions, etc.
• Advanced SQL query: Supports advanced query capabilities including views, subqueries, joins, unions, Common Table Expressions (CTE), window functions, prepared statements, etc.
• Data Types: Supports data types including integer, floating-point, string, date/time, boolean, enum, binary, and JSON types.
• Aggregate Functions: Supports common aggregate functions such as AVG, COUNT, MIN, MAX, SUM, etc.
• System Functions and Operators: Supports common string, date/time, mathematical functions, and operators.
• Indexes and Constraints: Supports primary keys, unique constraints, non-null constraints, foreign keys, auto-increment constraints, and secondary indexes.
• Streaming: Supports the creation of streaming tables and integration with Kafka data sources.
• Multi-tenancy: Supports the creation and management of internal tenants within the database.

Data Import and Export

• Supports batch insert data using INSERT, LOAD DATA, SOURCE, etc.
• Supports data export using SELECT INTO and modump methods.
• Supports direct import from local files and object storage.

Transactions

• Supports ACID (Atomicity, Consistency, Isolation, Durability) transaction properties.
• Supports transaction initiation, commit, and rollback operations.
• Supports both pessimistic and optimistic transactions, allowing users to switch between them.
• Provides configuration options for isolation levels, including Snapshot Isolation and Read Committed.

Deployment

• Supports both standalone and distributed deployment architectures.
• Supports deployment through source code, binary packages, Docker, and Kubernetes.
• Supports deployment on public cloud, private cloud, and physical machines.

Application Development

• Supports native connectors for popular development languages such as Java, Python, and Golang, including JDBC, pymysql, go-sql-driver.
• Supports common ORM frameworks for various languages such as MyBatis, Spring JPA, SQLAlchemy, GORM.
• Supports popular MySQL management tools like Navicat, MySQL Workbench, DBeaver, HeidiSQL.
• Provides a standalone and distributed command-line management tool named mo_ctl.
• Includes comprehensive database logging and query recording capabilities, with the ability to integrate with visualization components like Grafana for real-time cluster monitoring.

Security and Privileges

• Supports Transport Layer Security (TLS) encrypted connections.
• Supports Role-Based Access Control (RBAC).

Backup and Restore

• Supports logical backups using the modump tool.
• Supports physical backups using the mobr tool.

MatrixOne Use Cases

Operational Analytics

As the data volume of common IT systems such as OA, ERP, CRM, etc., increases with business growth, traditional single-node databases may not meet their performance requirements. Many companies set up a separate analytical database system to meet the needs of important reporting requirements at month-end or quarter-end, or use sharding techniques to reduce query loads. An operational analytics system is one that allows you to make quick decisions based on the current operational system. MatrixOne can fulfill the requirements of both the business system and the analytical system with a single database, while providing powerful scalability that can seamlessly expand as the business grows.

Real-time Analytics

For typical OLAP (Online Analytical Processing) applications in business, such as dashboards and BI reports, massive data analysis is often required. When the data volume becomes significant, performance bottlenecks may arise, resulting in poor timeliness. MatrixOne offers fast data ingestion, powerful analytical performance and scalability, enabling accelerated processing of complex and large-scale SQL queries, achieving sub-second response times, and improving the agility of enterprise decision-making and analysis.

IoT Monitoring

With the widespread application of sensors and network technologies, various IoT devices generate massive amounts of data, such as manufacturing plant production lines, renewable energy vehicles, city surveillance cameras, and more. The scale of data can easily reach hundreds of terabytes or even petabytes. There is an increasing demand for enterprises to store and utilize this data. Traditional database solutions cannot handle the real-time data ingestion and processing required in such massive and large-scale scenarios. MatrixOne provides powerful capabilities for streaming data ingestion and processing, along with robust scalability to handle any workload and data volume, fully meeting the requirements of this scenario.

Fluctuating Web Applications

For internet-based web applications such as gaming, e-commerce, entertainment, social media, news, etc., where user numbers are large and business fluctuations are frequent and significant, substantial computing resources are often required to support business demands during peak events. MatrixOne, with its fully cloud-native architecture, offers exceptional scalability, automatically scaling up or down in response to changes in the business, thereby greatly reducing the operational complexity for users.

SaaS Applications

SaaS applications have experienced explosive growth in recent years. In SaaS application development, a multi-tenant model needs to be considered. Traditional solutions often involve either shared database instances for multiple tenants or dedicated database instances for each tenant, facing a trade-off between management costs and isolation. MatrixOne comes with built-in multi-tenancy capabilities, providing natural workload isolation between tenants and independent scalability. It also offers centralized management capabilities, effectively addressing cost, ease of management, and isolation requirements, making it the ideal choice for SaaS applications.

Quick Start

Docker

docker pull matrixorigin/matrixone:1.0.0-rc1

To get started with MatrixOne 1.0.0-rc1, please visit our official website at https://docs.matrixorigin.cn/en. You can find detailed documentation, installation instructions, and tutorials to help you explore the features and capabilities of MatrixOne. Additionally, our community forums are available for any questions, discussions, or feedback you may have.

What's changed from 0.8 to 1.0

From 0.8 to 1.0, we mainly developed some new functions and improve usability for production-level deployment.

Key New Features

Physical Backup(Enterprise Edition)

In this version, we have introduced the physical backup feature, allowing users to easily backup and restore databases. Now, you can effortlessly create snapshots of your database and restore to previous states when needed, ensuring data integrity and reliability.

Stream Computing

As the final piece of the HSTAP architecture puzzle, we have completed the framework design for stream computing. In this iteration, we have added the ability to create streaming tables and implemented a Kafka connector to meet the streaming data ingestion needs of various time-series scenarios.

Recursive CTE

Recursive CTE (Common Table Expression) is a feature that allows for repeatedly executing an initial CTE to return subsets of data until the complete result set is obtained. The implementation in this iteration enables users to easily handle hierarchical data and build more complex and flexible queries using recursive queries.

Key Improvements

MySQL DDL Compatibility

We have further improved MatrixOne's compatibility with MySQL, enabling better support for MySQL table creation statements. Now, users can seamlessly migrate existing MySQL applications to MatrixOne without modifying existing table creation statements.

Session Migration during Scaling

We have added the capability to migrate sessions seamlessly during distributed instance scaling operations. Users can easily adjust the capacity of MatrixOne without impacting existing sessions and business operations.

Simplified System Configuration

We have significantly simplified the startup configuration options for both single-node and distributed versions, allowing users to quickly and easily start the database.

mo_ctl Tool

We have optimized the functionality of the mo_ctl maintenance tool for both single-node and distributed setups, providing more powerful and user-friendly maintenance capabilities. Now, you can effortlessly deploy, upgrade, scale, and perform o...

MatrixOne-v0.8.0

We are excited to announce MatrixOne 0.8.0 release. After four months' development, MatrixOne completed its architecture design with Proxy module, which ensures workload and tenant isolation. MatrixOne has also made much improvement with its OLTP and OLAP performance, along with its scalability, stability and user experience. This is also a Beta release, open for developers for testing and feedback. Cheers!

Docker

docker pull matrixorigin/matrixone:0.8.0

Key Points

Performance

-OLTP performance: 3-7x improvement, reaching MySQL performance in standalone version.
-OLAP performance: SSB and TPCH performance align with Snowflake and Clickhouse.
-Scalability: archiving near-linear performance growth with CN horizontal scaling.

Improved Usability

-Backward compatibility of data storage format.
-Deployment and administration tool for both standalone and distributed version.
-Improved MySQL compatibility in case sensitivity, information_schema and DDL statements.
-Read committed with pessimistic locking implemented in transaction mode. (Experimental)

Other Features

-Support window function RANK(), ROW_NUMBER() and DENSE_RANK()
-Support the BINARY type and related functions.
-Support data sharing between tenants and PUBLISH/SUBSCRIBE functions.
-Support INSERT...ON DUPLICATE KEY UPDATE statement.
-Add Sequence and related statements.
-Support ADD [COLUMN] | DROP [COLUMN] in the ALTER TABLE statement.
-Support multi-layer foreign key.
-Support RAND() built-in function.
-Support setting global variables in configuration files.
-Secure initial MatrixOne account by password replacement.
-Several types of partitioning are supported. (Experimental)

Known Issues:

-0.8.0 data format is not compatible with the previous versions.
-Secondary Key doesn't improve any performance.
-Memory leak occasionally happens and may lead to an OOM error.
-Workload isolation is only supported by JDBC.
-DN is a single point of failure for distributed version.
-Occasional system hung under high concurrency workload.
-Pessimistic transaction has a few fatal bugs remaining.

New Contributors

• @zengyan1 made their first contribution at #8244
• @forsaken628 made their first contribution at #8728
• @gavinyue made their first contribution at #8825
• @gouhongshen made their first contribution at #9775
• @WitcherTheWhite made their first contribution at #9941

Full Changelog: v0.7.0...v0.8.0

matrixone-v0.7.0

We are excited to announce MatrixOne 0.7.0 release. MatrixOne has made much progress with its stability and performance in this release, along with an increase in MySQL compatibility and many new features. Cheers!

Docker

docker pull matrixorigin/matrixone:0.7.0

Features

-Support foreign key
-Support adding "cluster by" in DDL statements
-Add a MYSQL_COMPATBILITY_MODE to manage MySQL-compatible behaviors
-Improve Unique Index behavior
-Support to load data from the client machine
-Support Alter View
-Add query_result() function to get saved query results
-Support account suspending
-Support system table sharing to accounts
-Add aggregation function: group_concat
-Add built-in functions: format, replace, curdate, field, substring_index

Known Issues:

-0.7.0 data format is not compatible with the previous versions.
-Transactional performance doesn't compete with mature DBMS products.
-Secondary Key doesn't improve any performance.
-Only support up to 100GB benchmark testing.
-Out of memory(OOM) error or context deadline will occur after 5 hours of frequent testing.
-Query as count(*) performance is unsatisfactory.
-Loading several GB tables in distributed cluster will result in OOM error.

Contributors

This release includes 656 commits by 43 authors.

New Contributors

• @sourcelliu made their first contribution in #6747
• @iceTTTT made their first contribution in #7201
• @chrisxu333 made their first contribution in #6968
• @songjiayang made their first contribution in #7450
• @dr-lab made their first contribution in #7647
• @arjunsk made their first contribution in #7849
• @Morranto made their first contribution in #7471

Full Changelog

v0.6.0...v0.7.0

matrixone-v0.6.0

We are excited to announce MatrixOne 0.6.0 release. In this release, MatrixOne has upgraded to a disaggregated compute and storage architecture, with a cloud native HTAP engine and many new features. Cheers!

Docker

docker pull matrixorigin/matrixone:0.6.0

Features

• Support distributed ACID transaction with a Snapshot Isolation level.
• Support TEXT, BLOB, TIME, JSON data type.
• Support multi-tenancy with account, user and role.
• Support VIEW.
• Support Java, Python, Golang connector and Mybatis, Spring JPA, SQLAlchemy ORM connection.
• Support import CSV and JSON data from local file system and cloud S3 storage.
• Implemented a MatrixOne dedicated backup tool modump.
• Support composite primary key, unique key and auto-increment constraint.
• Add system_metrics database in the catalog to monitor instance status.
• Add system database in the catalog to record user statements and system logs.
• Support timezone.
• Support TLS encrypted connection.
• Support pre-compilation statements PREPARE, EXECUTE, DEALLOCATE.
• Support EXPLAIN ANALYZE.
• Support UNION, UNION ALL, INTERSECT, MINUS operators.
• Support TEMPORARY TABLE.
• Support EXTERNAL TABLE.
• Support PARTITION BY.
• Add many system variables and tables to maintain MySQL compatibility.
• Add many new JSON, Datetime functions.

Known Issues:

• 0.6.0 data format is not compatible with the previous versions.
• Transactional performance doesn't compete with mature DBMS products.
• Composite Primary Key, Unique Key and Secondary Key doesn't improve any performance #6028 .
• Insert/update into select having more than 100MB data will fail in a distributed setting #6780.
• Execute load data in a loop may cause out of memory #6793.
• Data race happens in a distributed setting #6855, #6926.
• Background jobs have a low probability causing user transaction being aborted due to r-w conflict #6049.
• Insert file into BLOB type might fail #6302.

Contributors

This release includes 1520 commits by 97 authors.

New Contributors

• @lokax made their first contribution in #4186
• @triump2020 made their first contribution in #4414
• @Abirdcfly made their first contribution in #4450
• @yjw1268 made their first contribution in #4472
• @Juneezee made their first contribution in #4609
• @ZoranPandovski made their first contribution in #4767
• @Toms1999 made their first contribution in #4778
• @xy2398437254 made their first contribution in #4927
• @goodMan-code made their first contribution in #5033
• @DanielZhangQD made their first contribution in #5140
• @taofengliu made their first contribution in #5349
• @TszKitLo40 made their first contribution in #5333
• @TheR1sing3un made their first contribution in #5352
• @qqIsAProgrammer made their first contribution in #5382

Full Changelog

v0.5.1...v0.6.0

matrixone-v0.5.1

This release resolved some log replaying and storage GC issues.

Docker

docker pull matrixorigin/matrixone:0.5.1

Bugs

• #4213
• #3961

These bugs are fixed by PRs below.

• #4214
• #4052
• #4051

These bugs lead to some stability issues of a MatrixOne instance.
An upgrade from 0.5.0 to 0.5.1 only requires to replace the mo-server binary file.

matrixone-v0.5.0

We are excited to announce MatrixOne 0.5.0 release. In this release, MatrixOne has a stand-alone columnar storage engine which can support HTAP workloads. Cheers!

Docker

docker pull matrixorigin/matrixone:0.5.0

Features

• Support ACID transaction with a Snapshot Isolation level.
• Support UPDATE, DELETE and INSERT INTO ... SELECT ... statement.
• Support BOOL and Timestamp data type.
• Support LEFT / RIGHT / OUTER / NATURAL JOIN.
• Support Having expression.
• Support subquery.
• Support Common Table Expression.
• Support CASE ... WHEN expression.
• Support Interval expression.
• Support Explain plan tree.
• Support new aggregate function: any_value
• Lots of new functions are supported.

Known issues

• Hotfix on 0.5.0 may cause data format incompatable.
• 'Too many open files' error might be issued when user insert/update/delete data or create/drop table frequently. User need to increase the 'max open files' to resolve the issue.
• Running 1GB TPCH benchmark test with memory less than 64GB may result in an out of memory error.
• Loading large size of CSV file than 100GB might lead to system hanging. #3858
• Mixed TP and AP work loads running for a long time, might cause system panic. #3947 #3961

Contributors

This release includes 811 commits by 73 authors.

New Contributors

• @lawrshen
• @lyfer233
• @wuliuqii
• @ericsyh
• @dongdongyang33
• @aylei
• @richelleguice
• @aressu1985
• @mklzl

We appreciate your contribution! Welcome to MatrixOne community.

Full Changelog

v0.4.0...v0.5.0

matrixone-v0.4.0

We are excited to announce MatrixOne 0.4.0 release. Cheers!

Docker

docker pull matrixorigin/matrixone:0.4.0

Features

• Support inner join with equality operator.
• Support subquery within from clause.
• Support decimal data type.
• Support following builtin functions:
  • Mathematical: Abs, Log, Ln, Ceil, Exp, Power, Pi, Sin, Sinh, Cos, ACos, Tan, ATan, Cot
  • Datetime: Month, Weekday, Date, DayOfYear
  • String: Space, Reverse, Substring, Ltrim, Rtrim, StartsWith, EndsWith, Lpad, Rpad, Empty, LengthUTF8
• Support following aggregate functions:
  • Bit_and, Bit_or, Bit_xor, Stddev_pop, Var

Known issues

• For continuous table creation actions in a script, the parameter of pre-allocated-group-num should be set to a bigger value such in cluster scenarios. Otherwise, "no available raft group" may be notified during tables creation sequence.

Contributors

This release includes 259 commits by 50 authors.

New Contributors

• @BePPPower
• @JackTan25
• @Charlie17Li
• @domingozhang
• @Fungx
• @JasonPeng1310
• @jiajunhuang
• @NTH19
• @noneback
• @RinChanNOWWW
• @chaixuqing
• @Y7n05h
• @yuxubinchen
• @adlternative
• @ajian2002
• @bxiiiiii
• @coderzc
• @e11jah
• @fengttt
• @florashi181
• @hiyoyolumi
• @jinfuchiang
• @ouyuanning
• @qingxinhome
• @supermario1990
• @whileskies
• @xiw5
• @yclchuxue
• @ZtXavier

We appreciate your contribution! Welcome to MatrixOne community.

Full Changelog

v0.3.0...v0.4.0

matrixone-v0.3.0

We are excited to announce MatrixOne 0.3.0 release. Cheers!

Docker

docker pull matrixorigin/matrixone:0.3.0

Features

• Exports data into CSV file.
• Introduces parallel execution and improve performance for projection query.
• Introduces IN operator and allows you to specify multiple values in a WHERE clause.
• Support NULLABLE column in group by expression.
• Introduces new functions: round and floor.
• Introduces a Chaos Testing framework for distributed system.

Known issues

• For continuous table creation actions in a script, the parameter of pre-allocated-group-num should be set to a bigger value such in cluster scenarios. Otherwise, "no available raft group" may be notified during tables creation sequence.

Contributors

This release includes 157 commits by 21 authors.

• broccoliSpicy(@broccoliSpicy)
• Chen Mingsong(@m-schen)
• hanfang (@aptend)
• O2 (@ikenchina)
• Jin Hai (@JinHai-CN)
• Jiang xinmeng (@jiangxinmeng1)
• Lin Junhong (@iamlinjunhong)
• Long Ran (@aunjgr)
• Nan Deng (@dengn)
• Otter(@WenhaoKong2001)
• Peng Zhen (@daviszhen)
• Qin Shuqi (@sukki37)
• Sundy Li(@sundy-li)
• Shen JiangWei(@LeftHandCold)
• Jian Wang(@jianwan0214)
• Wan Hanbo(@wanhanbo )
• Xu Peng (@XuPeng-SH)
• Yan Wenze(@nnsgmsone)
• Yuesheng Li(@reusee)
• Zilong Zhou (@zzl200012)
• Zhang Yingfeng (@yingfeng)

New Contributors

• @wanhanbo made their first contribution in #1600
• @ikenchina made their first contribution in #1685
• @sundy-li made their first contribution in #1704
• @WenhaoKong2001 made their first contribution in #1838

We appreciate your contribution! Welcome to MatrixOne community.

Full Changelog

v0.2.0...v0.3.0

matrixone-v0.2.0

We are excited to announce MatrixOne 0.2.0 release. Cheers!

Docker

docker pull matrixorigin/matrixone:0.2.0

Features

• Support automatic rebalancing for the built-in AOE columnar storage engine.
• Introduce new SQL parser.
• Introduce factorized SQL execution engine which is the fastest MPP in Golang.
• Support CREATE / DROP INDEX.
• Support PRIMARY KEY in CREATE TABLE.
• Support more data types for SQL binary operators.
• Support equi join with group by or aggregation statements.
• Support new types of DATE and DATETIME.

Known issues

• For continuous table creation actions in a script, the parameter of pre-allocated-group-num should be set to a bigger value such in cluster scenarios. Otherwise, "no available raft group" may be notified during tables creation sequence.

Contributors

This release includes 243 commits by 21 authors.

• BingLin Chang (@decster)
• Chen Mingsong（@m-schen）
• Nan Deng (@dengn)
• Jin Hai (@JinHai-CN)
• Jiang xinmeng (@jiangxinmeng1)
• Li Yang (@lignay)
• Lin Junhong (@iamlinjunhong)
• lni (@lni)
• Long Ran (@aunjgr)
• Peng Zhen (@daviszhen)
• Qin Shuqi (@sukki37)
• Shen JiangWei(@LeftHandCold)
• Jian Wang(@jianwan0214)
• broccoliSpicy(@broccoliSpicy)
• Ryan Wang(@wanglei4687)
• Xiong Jingjuan (@anitajjx)
• Xu Peng (@XuPeng-SH)
• Yan Wenze(@nnsgmsone)
• Yuesheng Li(@reusee)
• Zilong Zhou (@zzl200012)
• Zhang Yingfeng (@yingfeng)

matrixone-v0.1.0

We are excited to announce the first release of MatrixOne 0.1.0 on 2021/10/24. Cheers!

Docker

docker pull matrixorigin/matrixone:0.1.0

Features

SQL

Following SQL statements are supported in this release.

DDL

• CREATE / DROP DATABASE
• CREATE / DROP TABLE

DML

• INSERT
• LOAD DATA
• SELECT
  • WHERE
  • GROUP BY
  • ORDER BY
  • LIMIT, OFFSET
• SHOW
  • DATABASES
  • TABLES
• USE

Data Types

• TINYINT / SMALLINT / INT / BIGINT, SIGNED / UNSIGNED
• FLOAT / DOUBLE
• CHAR / VARCHAR

Operators

• >, >=, <, <=, <>, !=, =
• BETWEEN … AND …, NOT BETWEEN … AND …
• AND, &&, OR, ||
• +, -, *, /, %, MOD, DIV, NEG
• CAST

Aggregation Functions

• COUNT
• SUM
• AVG
• MAX
• MIN

Database Engine

• MySQL dialect is supported.
• Build-in Append only columnar engine(AOE) has been supported as the first fusion database engine.
• Real-time analytical queries are supported.
• Vectorized query execution engine, based on MPP solution, is introduced.
• Part of the expression execution has been vectorized through SIMD instructions.
• Distributed AOE with strong consistency guaranteed by RAFT has been supported.
• The performance of replicated state machine has been greatly improved through the unique design of reusing RAFT log as well as Write-Ahead log without any redundancy.

Contributors

This release includes 453 commits by 16 authors. Special thanks to the first author: Yan Wenze (@nnsgmsone)

• Chen Mingsong（@m-schen）
• Jin Hai (@JinHai-CN)
• Jiang xinmeng (@jiangxinmeng1)
• Li Yang (@lignay)
• Lin Junhong (@iamlinjunhong)
• lni (@lni)
• Long Ran (@aunjgr)
• Peng Zhen (@daviszhen)
• Qin Shuqi (@sukki37)
• Shen JiangWei(@LeftHandCold)
• Wei Ziran (@w-zr)
• Xiong Jingjuan (@anitajjx)
• Xu Peng (@XuPeng-SH)
• Yan Wenze(@nnsgmsone)
• Zilong Zhou (@zzl200012)
• Zhang Yingfeng (@yingfeng)