Clickhouse mode of study view #11224

alisman · 2024-11-22T20:24:16Z

Fix # (see https://help.github.com/en/articles/closing-issues-using-keywords)

Describe changes proposed in this pull request:

a
b

Checks

The commit log is comprehensible. It follows 7 rules of great commit messages. We can fix this during merge by using a squash+merge if necessary
Has tests or has a separate issue that describes the types of test that should be created. If no test is included it should explicitly be mentioned in the PR why there is no test.
Is this PR adding logic based on one or more clinical attributes? If yes, please make sure validation for this attribute is also present in the data validation / data loading layers (in backend repo) and documented in File-Formats Clinical data section!
Make sure your PR has one of the labels defined in https://github.com/cBioPortal/cbioportal/blob/master/.github/release-drafter.yml

Any screenshots or GIFs?

If this is a new visual feature please add a before/after screenshot or gif
here with e.g. Giphy CAPTURE or Peek

Notify reviewers

Read our Pull request merging
policy. It can help to figure out who worked on the
file before you. Please use git blame <filename> to determine that
and notify them either through slack or by assigning them as a reviewer on the PR

alisman · 2024-11-22T20:25:36Z

src/main/resources/org/cbioportal/persistence/mybatis/ClinicalEventMapper.xml

@@ -181,7 +181,7 @@
                </if>
            </if>
        </where>
-        Group by clinical_event.EVENT_TYPE, patient.STABLE_ID
+        Group by clinical_event.EVENT_TYPE, patient.INTERNAL_ID


@haynescd this is one fix i did. i think we can keep it. the stable id is not unique across study

alisman · 2024-11-22T21:10:11Z

src/main/java/org/cbioportal/service/impl/AlterationCountServiceImpl.java

@@ -223,6 +263,197 @@ public Pair<List<CopyNumberCountByGene>, Long> getPatientCnaGeneCounts(List<Mole
        );
    }

+    @Override


@haynescd what's all this stuff?

This is the stuff for the new clickhouse implementation

alisman · 2024-11-22T21:16:33Z

src/main/resources/db-scripts/clickhouse/materialized_views.sql

@@ -0,0 +1,32 @@
+DROP TABLE IF EXISTS sample_list_columnstore;


@haynescd we can kill this file, right?

alisman · 2024-11-22T21:18:22Z

src/main/resources/org/cbioportal/persistence/mybatis/GenePanelMapper.xml

@@ -121,6 +121,7 @@
        <include refid="selectGenePanelData"/>
        <include refid="fromGenePanelData"/>
        WHERE
+        SAMPLE_ID IS NOT NULL AND 


@haynescd this is one we might want to undo because it changes profile counts. this is what allows the system to recover from incomplete sample_profile table issue.

alisman · 2024-11-22T21:19:11Z

src/main/resources/templates/index.html

@@ -32,7 +32,7 @@
        window.netlify = localStorage.netlify;

        if (window.localdev || window.localdist) {
-            window.frontendConfig.frontendUrl = "//localhost:3000/"
+            window.frontendConfig.frontendUrl = "https://localhost:3000/"


might possible break localdb tests

Yea, probably

* Add Columnar SQL file to init Clickhouse DB * Refactored Mapper xml to extract StudyViewFilterMapper

* ✅ Add Unit test for StudyViewMapper Clickhouse * ✅ Update db props to include mysql and clickhouse datasources to fix tests * Address comments * Rename package to clickhouse * Update to static final * Use bean name instead of qualifier

* Create new wide table sql file and rename package * Remove genomic_event view * Add AlterationFilter to mutated_genes endpoint * Add AlterationFilter to mutated-genes endpoint * Fix unit test * Fix sonar issues * Add test for mutation types and status * remove unused imports

* add missing poc clinical data binning function

* Add sample_mv materialized view and use it in mappers

* Add Support for TotalProfiledCase Counts for Mutated-genes endpoint. * Create sql files to create new tables * Add unit test for totalProfiledCount * Add matching gene panel ids * Add TotalProfiledCountsWithoutPanelData * Add profileCount for genes without gene panel data * Add Comments for SQL * Update matching Gene Panel Ids * Clean up code * Fix test * Add query to get correct Gene Panels * Fix unit test * Add comments

* working poc * refactor logic into service, so clean * refactor for parameters builder, simplify min max logic, streamline service call * remove unused services and imports * remove more unused imports

* Implement molecular profile count endpoint using Clickhouse * Cleanup

* ✨ Add CNA Gene Endpoint * 🐛 Fix StudyViewFilterMapper.xml to allow ability to filter on gene and alteration * Fix merge conflict * Address comments * Fix unit tests * Fix sonar issues

* ✨ Add StructuralVariant-genes endpoint * Fix sonar issues * Update MatchingGenePanel request to return list * Create and use sample_derive * Update where sample_derived is stored to fix unit test

Co-authored-by: Bryan Lai <[email protected]>

* use clinical_data_derived instead of sample_clinical_attribute_numeric_mv and patient_clinical_attribute_numeric_mv * use clinical_attribute_meta instead of sample_clinical_attribute_numeric_mv and patient_clinical_attribute_numeric_mv * remove unused clinical data count methods and SQL * fix numericalClinicalDataCountFilter * Move CategoricalClinicalAttributeFilter to repository * remove unused columns * Add override to methods --------- Co-authored-by: haynescd <[email protected]>

…0857) * Add patient_id column to genomic_event_derived * Update sql to convert list of patients to list of samples

* refactor to use clickhouse * filter out empty attr values * edit comment * fix sonarcloud issues * use parallel stream, shaves off 5s * use newer mapping annotation

Co-authored-by: Bryan Lai <[email protected]>

haynescd · 2024-11-25T02:26:12Z

src/main/java/org/cbioportal/persistence/util/RedisCacheUtils.java

@@ -14,7 +14,9 @@
 import java.util.stream.Collectors;

 @Component
-@ConditionalOnProperty(name = "persistence.cache_type", havingValue = "redis")
+@ConditionalOnExpression(
+    "#{environment['persistence.cache_type'] == 'redis' or environment['persistence.cache_type_clickhouse'] == 'redis'}"


I think we should remove this and be enable caching or disable caching for the whole system... going forward

agreed. this is so that we can assess performance without cheating with cache. we need caching on for legacy because otherwise, the product is totally unusable!

and at the same time, when we deploy for demonstration purposes, we want initial load of studyview to compete with legacy, i.e. cache ON.

haynescd · 2024-11-25T02:33:06Z

src/main/java/org/cbioportal/service/impl/RedisCacheStatisticsServiceImpl.java

@@ -19,7 +20,9 @@
 import java.util.stream.Collectors;

 @Service
-@ConditionalOnProperty(name = "persistence.cache_type", havingValue = {"redis"})
+@ConditionalOnExpression(
+    "#{environment['persistence.cache_type'] == 'redis' or environment['persistence.cache_type_clickhouse'] == 'redis'}"


Same as above... Should we still separate these?

* Hide select properties of ClinicalDataFilter from frontend * Update swagger decorators on clickhouse controller

… obj)

…#11155) * Add patient level filtering for aggregation * Patient level filtering works for non-NA * Categorical patient level filtering & clean up * Use new generic assay table schema

…y clickhouse_enabled is set (#11256) * Update cBioPortal to dynamically load ch Components only when property clickhouse_enabled is set * Update env var for circleCi

* Merge genomic data bins working * Workaround for clickhouse bug in numerical data parsing --------- Co-authored-by: alisman <[email protected]>

* fix CNA query for genomic data filter * rename one of the cna_query statements to cna_count_query to avoid table name clash

sonarcloud · 2024-12-11T15:03:49Z

Quality Gate failed

Failed conditions
4 New Bugs (required ≤ 0)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

alisman commented Nov 22, 2024

View reviewed changes

haynescd and others added 25 commits November 23, 2024 23:28

➕ Add infra to support multi db and Columnar DataSource Properties

bd2918f

Add Columnar support for Mybatis

86bd2cc

Add Columnar Study View Controller /api/column-store

9db2295

Feature/ch/update poc base branch (#10776)

b4a8a38

* Add Columnar SQL file to init Clickhouse DB * Refactored Mapper xml to extract StudyViewFilterMapper

use regular views instead of materialized views (#10793)

12b7b07

Add missing POC clinical data binning function (#10778)

63858cf

* add missing poc clinical data binning function

Create sample materialized view and use it (#10822)

b5fb29e

* Add sample_mv materialized view and use it in mappers

RFC80:density plot endpoint(#10837)

a03ec54

* working poc * refactor logic into service, so clean * refactor for parameters builder, simplify min max logic, streamline service call * remove unused services and imports * remove more unused imports

Fix clinical data counts query/filter using new derived table (#10833)

1051dea

Molecular-profile-counts service implementation (#10840)

ed26ca7

* Implement molecular profile count endpoint using Clickhouse * Cleanup

Fix clinical data scatterplot (point at new derived cd table) (#10846)

2015f69

Rfc80/add cna genes endpoint (#10845)

6e8e6c7

* ✨ Add CNA Gene Endpoint * 🐛 Fix StudyViewFilterMapper.xml to allow ability to filter on gene and alteration * Fix merge conflict * Address comments * Fix unit tests * Fix sonar issues

✨ Add StructuralVariant-genes endpoint (#10854)

5b0f2bd

* ✨ Add StructuralVariant-genes endpoint * Fix sonar issues * Update MatchingGenePanel request to return list * Create and use sample_derive * Update where sample_derived is stored to fix unit test

Implement study view sample-lists-counts endpoint in Clickhouse (#10849)

3cac3a2

Co-authored-by: Bryan Lai <[email protected]>

Fix class import (#10856)

3dd1e2a

Rfc80/fix patient level filtering for alteration counts endpoints (#1…

ab3000a

…0857) * Add patient_id column to genomic_event_derived * Update sql to convert list of patients to list of samples

🐛 Fix totalCount calculation when WES not available (#10858)

fc03406

Consolidate SQL files used to create CH Tables (#10867)

604568f

Demo rfc80 poc violin plot (#10863)

a569ec6

* refactor to use clickhouse * filter out empty attr values * edit comment * fix sonarcloud issues * use parallel stream, shaves off 5s * use newer mapping annotation

replace sample_mv references with sample_derived (#10880)

783b608

🔧 Update Sonar Cloud Token (#10888)

fc2f99d

zainasir and others added 4 commits November 23, 2024 23:41

require approval for restricted tests

a6ace50

rename approval job for circleci tests

ef25b13

add integration tests for cna-genes and sv-genes (#11221)

53c7126

Co-authored-by: Bryan Lai <[email protected]>

Fix Rebase issues with master

2b45019

haynescd force-pushed the demo-rfc80-poc branch from 4d4b302 to 2b45019 Compare November 25, 2024 02:21

Merge branch 'master' into demo-rfc80-poc

64fe5d0

haynescd reviewed Nov 25, 2024

View reviewed changes

zainasir and others added 2 commits November 25, 2024 10:34

Use upstream repo for api-tests

b44eb6e

Fix swagger decorators for rfc80 clickhouse controller (#11230)

b780409

* Hide select properties of ClinicalDataFilter from frontend * Update swagger decorators on clickhouse controller

alisman changed the title ~~Clickhouse api~~ Clickhouse mode of study view Nov 25, 2024

onursumer and others added 17 commits November 25, 2024 18:03

check for non-numerical values in numerical clinical data count filter

2aac75e

unify numerical data filters in one sql

9075f37

do not normalize attribute value if not NA

17a5dad

add unit tests

c97fc06

Add TransactionManager to handle lazy loading of data from db (cursor…

97c5c5e

… obj)

add unit tests for logic in service and repository methods

219c111

fix error with case list counts method

cc16f2c

Add categorical patient level filtering for generic-assay-data-counts (…

b72ddaa

…#11155) * Add patient level filtering for aggregation * Patient level filtering works for non-NA * Categorical patient level filtering & clean up * Use new generic assay table schema

Fix unfiltered method call in cacheable condition (#11250)

04c9b42

skip web-and-data image build for api tests (#11255)

68dfdaf

run cbioportal instance in clickhouse mode for apitests

955f85e

pass env var to circleci environment

8828759

Merge pull request #11257 from cBioPortal/fix/circleci-clickhouse

a2cc213

Update cBioPortal to dynamically load ch Components only when propert…

a8d1e95

…y clickhouse_enabled is set (#11256) * Update cBioPortal to dynamically load ch Components only when property clickhouse_enabled is set * Update env var for circleCi

Optimize bins filtering by merging bins (#11248)

b31fe9e

* Merge genomic data bins working * Workaround for clickhouse bug in numerical data parsing --------- Co-authored-by: alisman <[email protected]>

Fix CNA query (#11265)

94e6381

* fix CNA query for genomic data filter * rename one of the cna_query statements to cna_count_query to avoid table name clash

make CaseListDataCount serializable (#11267)

9c109f6

Fix CH primary keying of genetic-alteration-derived (#11268)

1107607

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clickhouse mode of study view #11224

Clickhouse mode of study view #11224

alisman commented Nov 22, 2024

alisman Nov 22, 2024

haynescd Nov 25, 2024

alisman Nov 22, 2024

haynescd Nov 25, 2024

alisman Nov 22, 2024

haynescd Nov 25, 2024

alisman Nov 22, 2024

alisman Nov 22, 2024

haynescd Nov 25, 2024

haynescd Nov 25, 2024

alisman Nov 25, 2024

alisman Nov 25, 2024

haynescd Nov 25, 2024

sonarcloud bot commented Dec 11, 2024

		@@ -0,0 +1,32 @@
		DROP TABLE IF EXISTS sample_list_columnstore;

Clickhouse mode of study view #11224

Are you sure you want to change the base?

Clickhouse mode of study view #11224

Conversation

alisman commented Nov 22, 2024

Checks

Any screenshots or GIFs?

Notify reviewers

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarcloud bot commented Dec 11, 2024

Quality Gate failed