v2.0.1-hotfix-20241211

Release date: December 12, 2024
MatrixOne version: v2.0.1-hotfix-20241211
MOI PROD Impact Scope: All CN Affected

This hotfix resolves compatibility issues in v2.0.1 to ensure improved stability and consistent functionality.

What's Changed

• Add debug log for 2.0.1 by @XuPeng-SH in #20729
• fix functionid by @daviszhen in #20743

Full Changelog: v2.0.1...v2.0.1-hotfix-20241211

MatrixOne-v2.0.1

Release date: December 10, 2024
MatrixOne version: v2.0.1

MatrixOne 2.0.1 introduces a series of improvements and bug fixes that enhance system performance, stability, and usability. Below are the major updates and new features.

Key Improvements

• Index and Query Optimization
• Memory and Resource Optimization
• Password Management
• Backup and Restore Enhancements
• Experimental support for full text search

Bugfix

This release addresses several critical bugs that improve the overall stability and reliability of the system. Some of the key bug fixes include:

• Transaction Management:

  • Fixed deadlocks and transaction state inconsistencies in distributed environments.
  • Resolved race conditions in transaction handling and session management.
  • Improved error handling for lock conflicts and concurrent delete operations.

• SQL Execution:

  • Fixed issues with recursive CTEs, view creation, and ON DUPLICATE KEY UPDATE statements.
  • Resolved problems with incorrect row counts and handling of compressed files during LOAD DATA.

• Logtail and Cache:

  • Addressed data race conditions and performance issues in logtail processing.
  • Fixed issues with cache usage and hit status monitoring.

• Full-Text Index:

  • Fixed panics related to full-text index creation and runtime filters.
  • Improved delete operations for tables with full-text indexes.

• Backup and Restore:

  • Fixed missing workspace data during the restore process.
  • Resolved errors when restoring tables or databases with special keywords or case-sensitive names.

• Performance and Stability:

  • Addressed memory allocation issues in batch processing and deduplication operations.
  • Resolved panics and race conditions in various modules, including query execution and system diagnostics.

Additionally, many other minor bug fixes were implemented to further enhance system reliability. For the full list of changes, refer to the What's Changed.

What's Changed

• Fix missing workspace data in restore process by @aptend in #19736
• uses CI release/2.0-dev for 2.0-dev by @Rosyrain in #19692
• refactor: improve checkTableDefChange by @huby2358 in #19681
• sync to 2.0: reduce the comparison during the reader primary key iterator scan by @gouhongshen in #19677
• optimize folding in EvalExpression(2.0-dev) by @zengyan1 in #19686
• [enhancement] proxy: reduce the interval time of refresh cn nodes by @volgariver6 in #19704
• Operator analyze counter set reuse2.0-dev by @qingxinhome in #19708
• abort all active txn on invalid service and make new txn can work to 2.0 by @zhangxu19830126 in #19749
• Fix test global checkpoint5 2.0 by @LeftHandCold in #19742
• fix lockservice dead lock with txn by @zhangxu19830126 in #19773
• fix dedup by @jiangxinmeng1 in #19775
• [bug] return retry error if snapshot ts is smaller than ts of catalog by @volgariver6 in #19780
• malloc: add IgnoreMunmapError by @reusee in #19782
• Password management policy to 2.0-dev by @YANGGMM in #19785
• fix group concat without data by @iamlinjunhong in #19778
• set sink mysql timeout to 10min by @ck89119 in #19791
• Fix TestAppendAndGC for 2.0 by @LeftHandCold in #19795
• handle index table missing table_kind identifier by @qingxinhome in #19796
• Backup adapts to 2.0 by @LeftHandCold in #19768
• Improve cdc send sql performance by @ck89119 in #19797
• fix: fetch result from correct resultset 2.0-dev by @xzxiong in #19809
• fix txn status invalid to 2.0 by @zhangxu19830126 in #19817
• fix ut by @ck89119 in #19836
• check user after open check switch by @YANGGMM in #19830
• fix transfer deletes by @jiangxinmeng1 in #19831
• [bug] logtail client: change the channel to buffered one by @volgariver6 in #19839
• Fix TestAppendAndGC2 by @LeftHandCold in #19824
• refactor: reduce memory alloc in makeBatchRows by @huby2358 in #19793
• refactor DML plan (cp #18406) by @aunjgr in #19848
• objectio: make metadata cache capacity dynamic adjustable by @reusee in #19829
• use customized driver by @ck89119 in #19849
• allow now function pushdown to block filters by @badboynt1 in #19855
• fix a bug cause remote run cannot stop immediately. by @m-schen in #19841
• fix Incorrect time unit for password lock check(#2.0-dev) by @YANGGMM in #19853
• fifocache: add context params to methods by @reusee in #19805
• [fix] : Instead of sending the value scan's batch all at once, split it up to reuse memory. by @jensenojs in #19845
• make ut for multi_update faster (#19861) by @ouyuanning in #19874
• [bug] proxy: fix UT by @volgariver6 in #19880
• metric: redefine duration buckets by @reusee in #19878
• to 2.0: fix the batchApplyTombstone got unsorted rowOffsets by @gouhongshen in #19870
• fix nested recursive cte by @iamlinjunhong in #19899
• [Bug]: use load_file() function to load stage data, mo report 'ERROR 20101 (HY000): internal error: Invalid alloc size 7775731712 by @cpegeric in #19905
• malloc: do not unmap to avoid VMA fragmentation by @reusee in #19886
• bug fix fulltext phrase search by @cpegeric in #19902
• add debug tool for remote run (#19833) by @ouyuanning in #19892
• feat: keep S3IOInput as 6-prec float value. by @xzxiong in #19865
• refactor: remove openSysTable 2.0 by @aptend in #19823
• Add metric for GC by @LeftHandCold in #19887
• cherry-pick optimize performance for filling hashkeys on string vectors (#19881) by @badboynt1 in #19918
• fix bug: sysbench delete pk will flush s3 by @ouyuanning in #19927
• support ip whitelist feat (#2.0-dev) by @YANGGMM in #19919
• cherry pick 'support pushdown cast expression to block filters for date time type' by @badboynt1 in #19938
• ignore bvt case 2.0-dev by @w-zr in #19935
• update log by @jiangxinmeng1 in #19932
• do some lock if lock no rows after lock_op by @ouyuanning in #19943
• [bug] clusterservice: ignore tn store which is expired by @volgariver6 in #19954
• support password_history and password_reuse_interval take effect separately(#2.0-dev) by @YANGGMM in #19950
• enable transfer in metric and cu table on 2.0-dev by @w-zr in #19916
• optimize workspace size by @Wenbin1002 in #19929
• cherry-pick: support observe cache usage and hit status (#19914) by @xzxiong in #19926
• add system variable profiling_history_size(#2.0-dev) by @YANGGMM in #19946
• cherry-pick: couple of updates by @daviszhen in #19968
• update logtail entry type by @jiangxinmeng1 in #19901
• [bug] fix ut, which raises "address already in use" error by @volgariver6 in #19957
• fix panic in fillStrhashmap by @badboynt1 in #19971
• fix create view with hint(#2.0-dev) by @YANGGMM in #19978
• fix bug: panic on create fulltext idx by @ouyuanning in #19975
• Fix TestBackupData2 for 2.0 by @LeftHandCold in #19986
• fix show index panic(#2.0-d...

v1.2.3-hotfix-20241101

What's Changed

• fix vector free panic(1.2.3-tmp-shulian) by @zengyan1 in #19680
• Fix alter table column null property-shulian by @qingxinhome in #19688
• fix decimal output by @daviszhen in #19643
• cherry-pick: fix select version by @daviszhen in #19695
• fix decimal panic(1.2.3-shulian) by @zengyan1 in #19716

Full Changelog: v1.2.3-hotfix-20240916...v1.2.3-hotfix-20241101

MatrixOne-v2.0.0

We are excited to announce the release of MatrixOne v24.2.0.0!

MatrixOne is a hyper-converged, cloud-native database built for modern data demands. Designed to deliver high performance, scalability, and MySQL compatibility, MatrixOne provides a seamless HTAP (Hybrid Transactional/Analytical Processing) experience, allowing users to efficiently handle transactions, analytics, time-series data, and streaming processing in one unified platform.

Highlights of this Release

MatrixOne v24.2.0.0 introduces significant enhancements for enterprise-level high availability, disaster recovery, and expanded AIGC support.
This version features robust improvements to support Generative AI applications, including unstructured data processing, full-text search, optimized vector search, and enhanced MySQL compatibility. Key features include external data access, snapshot backups, point-in-time recovery (PITR), CDC, and disaster recovery through log-based replication for primary-standby clusters. MatrixOne continues to evolve as a leading platform for intelligent, AI-driven data management, providing enterprises with the ideal solution for their data infrastructure needs.

Use Cases

MatrixOne is well-suited for the following application scenarios. We welcome users facing similar challenges to reach out and explore a trial deployment with us.

Generative AI Applications

MatrixOne’s hyper-converged architecture is ideal for Generative AI, offering comprehensive support for multimodal data, real-time data retrieval, and intelligent data processing. In text and image generation scenarios, MatrixOne facilitates rapid response times and high-quality outputs through efficient data management, vector and hybrid search capabilities, data preprocessing with Python UDFs, and GPU-accelerated real-time inference. MatrixOne’s low-latency architecture supports Generative AI workloads like large-scale data storage, online inference, and adaptive feedback, empowering enterprises to drive innovation with speed and efficiency.

Time-Series Data Applications

Modern IoT ecosystems generate massive volumes of real-time data from diverse sources like industrial production lines, smart grids, and autonomous systems. MatrixOne is built to handle such demands with millisecond-level, high-concurrency writes and rapid, scalable data retrieval. MatrixOne’s real-time analytics seamlessly integrates with machine learning models, making it an ideal solution for predictive maintenance, energy optimization, and intelligent monitoring.

Mixed Workload Scenarios

Traditional single-node databases often struggle with the simultaneous processing demands of transactional and analytical workloads in enterprise applications like ERP, CRM, and OA systems. MatrixOne’s support for mixed workloads within a single database enables real-time analytics, continuous reporting, and efficient data-driven decision-making without the need for additional analytical databases or sharding. With MatrixOne’s scalability and high concurrency support, enterprises can confidently meet growing data demands while keeping performance at peak levels.

Enterprise SaaS

With the rise of enterprise SaaS applications, supporting multi-tenancy while ensuring cost-efficiency and data isolation is essential. MatrixOne’s native multi-tenant architecture provides load isolation, independent scaling, and unified management for each tenant. This architecture reduces management overhead, ensures data separation, and improves operational efficiency, making MatrixOne an optimal database choice for SaaS platforms.

Key New Features

Multi-mode Data Management

MatrixOne now supports direct access to external object storage, remote file systems, and local storage through Stage objects, as well as datalink access to files in storage systems. This capability significantly simplifies data pipeline construction for Generative AI applications, reducing development overhead and maintenance costs.

Full-text Indexing for Text and JSON Data

Full-text indexing on JSON and TEXT columns greatly enhances performance in AIoT applications, especially when combined with MatrixOne’s JSON data type, which minimizes data redundancy and boosts efficiency.

Vector Search

Enhanced vector search capabilities now provide rapid, large-scale vector retrieval, a critical feature for Generative AI applications involving large language models (LLMs) and retrieval-augmented generation (RAG).

Snapshot-based Backup and Recovery

Cluster and tenant-level data snapshots capture the database state at specific points in time, ensuring rapid recovery while minimally impacting performance. Snapshots support cross-tenant restoration, bolstering MatrixOne’s disaster recovery.

Primary-standby Log Replication for High Availability

Log replication enables transaction log synchronization between primary and standby databases, supporting high availability and disaster recovery. Standby databases can take over in case of primary database failure, ensuring uninterrupted operations.

Point-in-time Recovery (PITR)

PITR captures all data changes post-snapshot, allowing precise restoration to a historical moment in case of accidental operations or data loss. This approach reduces storage costs, enhances recovery efficiency, and provides flexibility for critical business continuity and compliance.

MatrixOne to MySQL CDC

Change Data Capture (CDC) from MatrixOne to MySQL supports real-time disaster recovery for users transitioning from MySQL, maintaining data continuity.

Table-level Publish-Subscribe

Building on previous database-level publish/subscribe, table-level publish-subscribe in this release enables more granular control over data change synchronization, providing enhanced flexibility for data management.

Additional Updates

SQL Enhancements

• Added support for rename table, create pitr, drop pitr, alter pitr, restore pitr, show pitrs.
• Optimized show publications and show subscriptions.
• Enhanced load data infile command to support user-defined column order.

Data Types

• Support datalink data type

Indexes and Constraints

• Support Full-text Index

Functions and Operators

• Support JSON functions: json_row, jq, try_jq, json_extract_string, json_extract_float64 functions.
• Enhanced date manipulation for now() function.

Tools

• mo-backup: Supports PiTR management.
• mo_cdc: supports CDC task management.

MySQL Compatibility

• Support Encode()/Decode() function

Quick Start

Community users and enterprise developers can try MatrixOne with the following command:

docker pull matrixorigin/matrixone:2.0.0

For more details, including architectural insights, installation guides, and tutorials, visit our documentation site. Join our discussions or share feedback on GitHub or in our community WeChat group.

Known Issues

• Standby clusters currently do not support synchronization of data in external tables or stages.
• Standby clusters support only cold backups and cannot be opened in read-only mode.
• CDC supports only table-level data synchronization.
• Snapshot backups support cluster and tenant levels, with restoration possible at the cluster, tenant, database, or table level.
• Snapshots and PiTR backups cannot recover deleted tenant data.

New Contributors

• @zuyu made their first contribution in #16402
• @Aoang made their first contribution in #14897
• @Cyberleu made their first contribution in #19014

Full Changelog: v1.2.4...v2.0.0

v1.2.3-hotfix-20241016

What's Changed

• fix Intersection2Vector panic by @gouhongshen in #19383

Full Changelog: v1.2.3-hotfix-20241010...v1.2.3-hotfix-20241016

v1.2.3-hotfix-20241010

Full Changelog: v1.2.3-hotfix-20240916...v1.2.3-hotfix-20241010

v1.2.4

Release Date: September 23, 2024
MatrixOne Version: v1.2.4

This hotfix addresses several critical issues and includes minor improvements aimed at enhancing stability and performance.

What's Changed

• make query with limit run in single parallel by @badboynt1 in #18803
• cp to 1.2-dev 'fix ndv calculation to make it more accurate' by @badboynt1 in #18848
• fix json extract function(#1.2 hotfix) by @YANGGMM in #18849
• cp to 1.2-dev 'fix the condition for runtime filters (#18496)' by @badboynt1 in #18860

Full Changelog: v1.2.3-hotfix-20240916...v1.2.4

v1.2.3-hotfix-20240916

Release Date: September 16, 2024
MatrixOne Version: v1.2.3-hotfix-20240916

This hotfix addresses several critical issues and includes minor improvements aimed at enhancing stability and performance.

Bug Fixes

• Resolved a transaction retrieval issue in GetOrCreateTxnWithMeta.
• Fixed incorrect value retrieval for unique and secondary keys.
• Corrected a panic issue in KillRoutineConnections.
• Fixed a proxy panic when killing a connection.
• Resolved a bug where sessions could not be transferred after using LOAD DATA LOCAL INFILE.
• Fixed range checkpoint inconsistencies.

What's Changed

• add summary for txnBlock(1.2-dev only) by @aptend in #18691
• fix GetOrCreateTxnWithMeta not get txn error by @w-zr in #18705
• bug fix: unique key/secondary key get incorrect value (#18711) by @ouyuanning in #18717
• set srv.pu before creating hakeeper client by @w-zr in #18672
• Method for reading query results during upgrade modification by @qingxinhome in #18731
• fix KillRoutineConnections panic by @YANGGMM in #18726
• [bug] proxy: fix panic when kill connection by @volgariver6 in #18756
• support order by limit optimization for non-sorted columns by @badboynt1 in #18760
• cp to 1.2-dev 'fix a bug that cause hashmap data wrong' by @badboynt1 in #18779
• [tech] pprof: dump malloc profile by @volgariver6 in #18768
• remove the 1024bytes limit for group_concat result. by @m-schen in #18786
• add support rollback running txn to 1.2 by @zhangxu19830126 in #18775
• [bug] proxy: session cannot be transferred after load data local infile by @volgariver6 in #18787
• fix range checkpoint by @jiangxinmeng1 in #18799

Full Changelog: v1.2.3...v1.2.3-hotfix-20240916

MatrixOne-v1.2.3

Release Date: September 11, 2024
MatrixOne Version: v1.2.3

Compared to the previous version, v1.2.3 focuses on bug fixes, performance optimizations, and minor feature enhancements, without introducing major new features.

Improvements

• Memory Optimization: Enhancements in memory usage, particularly for duplicate checks, SEMI Join, and TableScan operations.
• Enhanced Logging: Added logs for account restrictions, account suspensions, and additional operations, improving debugging and monitoring capabilities.
• Performance Enhancements: Optimized handling of LIMIT 0 queries, improved dynamic cache management, and refined query performance for JSON type ordering.

Bug Fixes

This release addresses multiple issues related to memory usage, transaction handling, proxy connection stability, and more. These fixes improve the overall stability and reliability of MatrixOne.

What’s Changed

• [cherry-pick] : cherry-pick 17161 17020 16522 to save memory useage for dup check and clean up code by @jensenojs in #17211
• add log for account restricted by @YANGGMM in #17327
• fileservice: fix TestFuzzingDiskS3 by @reusee in #17470
• [CherryPick]: Reduce mp.Grow2 in SEMI Join and TableScan Operator by @arjunsk in #17482
• fix restore slow （#1.2-dev） by @YANGGMM in #17288
• Cp 1.2.2 hotfix by @sukki37 in #17554
• fileservice: fix missing free in tests by @reusee in #17544

For a full list of changes and improvements, refer to the complete changelog.

This release is part of our ongoing effort to ensure MatrixOne’s stability, performance, and usability, offering refinements that contribute to a smoother and more efficient database experience.

MatrixOne-v1.2.2

Release date: July 12, 2024
MatrixOne version: v1.2.2

Compared with the previous v1.2.1, v1.2.2 doesn't introduce new features but only focuses on bug fixes and minor optimizations.

Improvements

• Support GROUP BY 1, MAKEDATE
• Add runtime metrics
• Support reload auto increment cache
• Allow loading file info asynchronously for the disk cache
• Optimize flush and merge

Bugfix

Fixed bugs related to partition state, proxy, MySQL compatibility, lockservice, stats, and merge memory control. Check the What's Changed section for details.

What's Changed

• [fix] remove unnecessary values from the second parameter of prefix_in (cp #17111 to 1.2) by @aunjgr in #17186
• Add CoreDump when CachedData is nil by @LeftHandCold in #17219
• memorycache: do not allow nil Data by @reusee in #17222
• [bug] proxy: fix the connection id when send request to cn server. by @volgariver6 in #17228
• fix unlock fatal when tn restart by @iamlinjunhong in #17237
• fix always return illegal state error on special connection to 1.2 by @zhangxu19830126 in #17251
• cherry-pick stats related fix into 1.2-dev by @badboynt1 in #17260
• cp 17265 into 1.2-dev by @badboynt1 in #17268
• support last_day and makedate (#1.2-dev) by @YANGGMM in #17277
• add debug info for remap error by @daviszhen in #17256
• cherry-pick: remove compilectx by @daviszhen in #17257
• fix show create External table-1.2-dev by @qingxinhome in #17172
• Fix show columns compatibility issues1.2 dev by @qingxinhome in #17199
• fix bvt test(#1.2-dev) by @YANGGMM in #17284
• cherry-pick to 1.2-dev 'fix a bug that cause wrong parallelism for tpch' by @badboynt1 in #17297
• cherry-pick runtime metrics changes by @reusee in #17272
• Fix set global var do not work cross cns by @ck89119 in #17273
• Cherry-pick-fix-bugs[Dup/WW/Data-corruption] by @triump2020 in #17312
• When DDL statement has been executed within a transaction, force the use of a single CN by @ouyuanning in #17282
• fix dual by @daviszhen in #17279
• cherry-pick to 1.2-dev 'fix a bug that cause tpch performance regression ' by @badboynt1 in #17315
• retry lock when cn down by @iamlinjunhong in #17320
• Cherry pick- fix-snapshot-read-bug[txn is stale: timestamp is too small] (#17271) by @triump2020 in #17306
• fix data race when get deltaloc on 1.2-dev by @w-zr in #17319
• Merge delta loc 1.2 dev by @w-zr in #17328
• fix enum type op in one transaction (#1.2-dev) by @YANGGMM in #17285
• [pick to 1.2] fix a bug that process's cancel not match to its context by @m-schen in #17349
• Fix Location's Extend is 0 when call UpdateTableStats for 1.2 by @LeftHandCold in #17346
• fix makedate and lastday func(#1.2-dev) by @YANGGMM in #17343
• fileservice: use wrapped standard retryer in AWS sdk (#17250) by @reusee in #17267
• update delete by deltaloc by @jiangxinmeng1 in #17246
• [enhancement] stats: reduce the parallel goroutines to update stats by @volgariver6 in #17342
• [bug] stats: fix hang issue by @volgariver6 in #17364
• fix cannot restart lockservice by @iamlinjunhong in #17371
• support reload auto increment cache to 1.2 by @zhangxu19830126 in #17369
• only create buffer when err is nil by @w-zr in #17373
• fileservice: allow loading file infos asynchronously in disk cache by @reusee in #17360
• Expansion 1.2.2 Upgrade Package by @qingxinhome in #17383
• cp to 1.2-dev "release unused memory in shuffleRange" by @badboynt1 in #17386
• add clean commit state log by @iamlinjunhong in #17390
• cp to 1.2-dev "don't calculate stats for limit 0 query" by @badboynt1 in #17400
• [enhancement] proxy: remove error logs caused by hearteabt of LB. by @volgariver6 in #17401
• [enhancement] stats: add factor to update worker number. by @volgariver6 in #17410
• [enhancement] stats: do not update the same key at the same time. by @volgariver6 in #17422
• fileservice: set finalizer on read closer to avoid potential leaks by @reusee in #17398
• fix change morpc backend state to 1.2 by @zhangxu19830126 in #17432
• metric: fix typo and register fsWriteCounter by @reusee in #17434
• cherry-pick some fileservice changes from main branch by @reusee in #17429
• using process memory usage for background task memory throttle 1.2 by @aptend in #17436
• [Cherry-pick] Alter table add column with lower_case_table_names = 0 by @ck89119 in #17407
• [plan] support "GROUP BY 1" (cp to 1.2) by @aunjgr in #17424
• 1.2 optimize flush and merge prints by @XuPeng-SH in #17437
• [fix] don't push LIMIT clause to index table if main table has filters by @aunjgr in #17446
• fix ut Test_saveQueryResultMeta panic by @CJKkkk-315 in #17450
• Fix GC MaxConsumed For Backup by @LeftHandCold in #17464
• remove an info-level log. by @m-schen in #17461
• The global checkpoint only collects the node of the last deltaLoc by @LeftHandCold in #17440
• Add bvt by @ck89119 in #17457
• Fix TestSnapshotMeta data race&failed for 1.2 by @LeftHandCold in #17460
• sort objects in ascending order by @w-zr in #17454
• Fix merge mem contro 1.2 by @aptend in #17479
• [enhancement] add concurrency when fetch objects of table by @volgariver6 in #17449
• fix makedate and last_day func (#1.2-dev) by @YANGGMM in #17473

Full Changelog: v1.2.1...v1.2.2