From daa6ba7b444662ceaebda1840cb01c7b71857f8e Mon Sep 17 00:00:00 2001 From: Bobby Ullman Date: Tue, 23 Aug 2016 09:34:49 -0700 Subject: [PATCH] merge develop into perf cli branch (#820) * Nit: use single-element ImmutableSet for clarity * SweepTaskRunnerImpl.getTimestampsToSweep: minor refactor * 0.11.0 release notes edits * Add ETE test for Atlas configured with no leader * Remove wrong config value from ETE tests * Refactor out on-demand loading and add tests for it. Clean-up files, convert/refactor tests to assertJ * Switch error to warn since it isn't actually an error * Rename SweepStrategySweeper to StrategySweeper * Get rid of NOTHING shortcut (essentially the same atm) * Make timestamp test work with all KVSes * Add release notes * Make test more consistent for DB KVS * StrategySweeper -> Sweeper, new test to maker sure we don't call the kvs when there aren't any rows to sweep * Get rid of Sweep at the beginning of two classes * Log messages should use AtlasDB * move to next release * Fix weird compile error * Remove bad optional.absent check * Update circle.yml to include container-logs in artifacts * Add split out logs for ETE tests * 0.11.1 release note * release notes for 0.11.2 * Add CassandraServerVersion with tests * 0.11.3 release notes * Update release-notes.rst * Test for cassandra duplicate table creation bug * Change the copyright header to use the regular one * Passwords should not be output in generated toString() methods Prevents an issue seen where a stack trace helpfully tries to print the config but also prints the password to the log files. Updated copyright of all modified classes to 2016. * Bump version * dont log loading a tracing prefs that isn't enabled at some point we started shipping an example prefs file that always gets loaded, so you get this 'error' message at startup by default now. * cleanup temp tables Necessary for a corresponding internal change. The general gist is open atlas never wanted this application-specific garbage anyways, so I've mostly removed it here and made it so someone can just implement an interface. Real reason for all of this is that we're moving to lazily initialized temp tables because postgres' rubbish temp table creation speeds screw with Hikari pool resizing, among other things. * Switch to DockerComposeRule instead of DockerComposition * Switch to DockerComposeRule instead of DockerComposition for TimelockServer * Switch to DockerComposeRule instead of DockerComposition for EteSetup * Switch to DockerComposeRule instead of DockerComposition for DbkvsTestSuite * Adding diagnostics, as per Oracle Support * Rename composeRule -> docker, per naming convention in DCR example code * Add entry in ValueTypes table for UUID A UUID is just two longs (a 16-byte value). We encode them as big-endian, so range scans are supported. * Clean up explanation for VAR_LONG and VAR_SIGNED_LONG * Reword introductory paragraph * Update release notes for PR #755. * Add advice for contributing code * Add explanation about FIXED_LONG_LITTLE_ENDIAN * Use footnotes because they're prettier * Fix typos in contribution guidelines * don't always log sql errors especially ones that are expected errors / caller logic is set up to deal with the failure. Fixes #709 * Add comment explaining the version * Clean up tests * Switch to shared static logger * Adjust table widths so the footnotes don't spill over * Add additional logging * Unused imports * ssl: false should override the presence of an SslConfiguration We want to add a default SslConfiguration to our internal application. However, users may have explicitly disabled Ssl for communication with their Cassandra, which will be broken by the presence of the new block overriding the `ssl()` param.. Despite being deprecated, the `ssl()` param should override the presence of an ssl config, meaning that we can apply this without being a breaking change. * License * Fix copy-paste error * Upgrade checkstyle to 6.18 * Add .baseline files * Remove a few of the checkstyle/findbugs. Change import order to match existing order. Change copyright notice to BSD-3. Auto-add copyright notice in IntelliJ * Add baseline to the gradle files * Enable checkstyle and findbugs report extraction in CircleCI * CassandraTestSuite: Add a Classrule to ensure Cassandra images are created * Switch from whitelisted projects to blacklisted projects * Remove unused variable * Clean up docker-compose * CassandraSchemaLockTest: Automated test for detecting multiple table creation error * Cleanup * Add matchers to get better error messages * Improve error messages * Move the cassandra multinode test to a separate directory * Revert copy-paste error * Fix typo: SIGNED_VAR_LONG -> VAR_SIGNED_LONG * Move checkstyle version into versions file, remove META-INF exclude, and add comment about why we have the findbugs excludes * Ensure C* image is created before docker-compose * Remove stray empty lines * Prep and enable baseline for atlasdb-dropwizard-bundle * Enable baseline for atlasdb-hikari * Prep atlasdb-api for baseline * Prep atlasdb-api tests for baseline * Fix-up findbugs errors for atlasdb-api * Enable baseline for atlasdb-api * Run cassandra-multinode-tests in a separate container * Create fromBoolean method in LockLeader enum * Docs: cassandra_KVS_config should have more information on ssl and leader block. * Improve the wording in docs * Add release notes * Upgrade to gradle 2.14.1 and jacoco to 0.7.7.201606060606. Fixes #767 * Revert "Test for cassandra duplicate table creation bug" * Revert "Revert "Test for cassandra duplicate table creation bug"" * Add container logs directory * Add .idea to .gitignore * Move to package, blacklist * Increase timeout * Implement the timestamp dropwizard command * Convert timestamp CLI dropwizard class into a generic one for all CLIs * Fix checkstyle and similar * Add temporary workaround for airlift/airline#51 * Convert leader blocks into TS/LOCK blocks * Refactor configuration conversion and argument extraction out * Add offline flag to CLIs * Add offline support to console and propagate it through for CLIs * Refector AtlasDbConfigs as part of #goldentime to play around with tell rather than ask code structure * Update nullables and unused import * Add missing copyright * Revert "Update nullables and unused import" This reverts commit 8ee3b4d8c5a67dcaf343ef195f750baca6e64aeb. * Revert "Refector AtlasDbConfigs as part of #goldentime to play around with tell rather than ask code structure" This reverts commit 5248e3f9a28111728b09c297fca73d74b43fc395. * Parameterise TestTimestampCommand * Further refactoring of timestamp command tests * Implement some ETE dropwizard tests * Refactor runCommand to take in a String * Remove random generated files * Add tests for AtlasDbCommandUtils * Ignore dropwizard ETE tests until docker-compose-rule#95 is merged and a new version is tagged * Added dropwizard bundle to release notes * Clean-up some code, fix an EMPTY_NAMESPACE bug when serialising/deserialising * Switch to docker run and re-enable tests * Add @Nullable to AtlasDbConfigs * Set of minor modifications * Set of minor modifications * Annotate why we JsonIgnore onAcquireConnectionVisitor * Add tests for AtlasDbCliCommand#getOptionTypesForCommandMetadata * Refactor out addOptionToParser and add tests * Refactor out serialisation command * Fix-up checkstyle * Move timestamp file clean up into method * Add a new Leader configuration page * Make timeout 2 minutes * Fix checkstyle * Use correct jacoco version * Use assserThat from unit * Refactor CassandraSchemaLockTest and cassandra yml * Cleanup container logs * Cleanup build.gradle * Reduce timeout, extract CONTAINER_LOGS_DIRECTORY * minor doc nits * Prep atlasdb-dbkvs for baseline * Prepare atlasdb-dbkvs for findbugs * Enable baseline on atlasdb-dbkvs * Remove @After method * Increase timeout * Remove dependencies on ete1 as this is causing hanging builds. Fixed in #793 * Newline for Ben * Select the first leader according to alphabetical order to make sure lockCreator is deterministic. * Add tests * Fix CassandraConnectionTest by setting a valid quorumSize * Add docs change * Fix directory structure in atlasdb-impl-shared/src/test/java * Add profiling KVS wrapper to default kvs startup, use same order as internal product * Add necessary documentation for profiling kvs' * Docs nit: add comma before "which" * Add check to make sure it's possible to reach quorum * Wait for DB start * Prep atlasdb-impl-shared for baseline. Disable abbreviation checking in files with KV in their name. * Prep atlasdb-impl-shared tests for baseline * Get atlasdb-impl-shared and tests working with findbugs. Exclude DynamicPartitionMapImpl from synchronisation checks until later * Enable baseline for atlasdb-impl-shared * Code review fixes * Rename atlasdb-server -> atlasdb-service, atlasdb-server-dropwizard -> atlasdb-service-server * Add release notes * Prep atlasdb-cassandra for baseline * Fix-up some FindBugs for atlasdb-cassandra * Enable baseline on atlasdb-cassandra * Fix-up code review comments * ignore eclipse generated stuff * Reduce number of threads and timeout * Expand title line and fix link to file * Cleanup docs * Docs changes * Docs: Add link from Logging page to Enabling C* Tracing page. * Docs: minor fixes, mostly capitalisation * release notes edits for 0.12.0 * Rename link * Check for unique key constraints when inserting tables into metadata * Fix-up ETE dependencies so that we wait for the DB box to be created * another atlas -> atlasdb * Refactor DbkvsTestSuite so it runs more consistently locally * Remove test that checks duplicate table names throws. Cassandra doesn't throw, so we shouldn't really either * fix index.rst pages * edit breaking change note * final 0.12.0 release notes nits * Convert findbugs annotation version to a constant * Prep atlasdb-config for baseline * Enable baseline for atlasdb-config * Move release notes to right version * fix compile errors * publish test artifacts * add licenses * fix checkstyle errors * fixed errors from bad merge * fix checkstyle rule failures caused by merge * fix findbugs errors --- atlasdb-api/build.gradle | 3 + .../palantir/atlasdb/config/LeaderConfig.java | 3 + .../api/AtlasDbConstraintCheckingMode.java | 11 + ...alueServiceTransactionIntegrationTest.java | 10 + .../CassandraClientPoolIntegrationTest.java | 20 ++ .../cassandra/CassandraTestSuite.java | 11 + .../cassandra/CassandraApiVersionTest.java | 65 ++++++ .../CassandraKeyValueServiceConfigTest.java | 64 ++++++ ...andraKeyValueServiceTableCreationTest.java | 94 ++++++++ .../CassandraKeyValueServiceTest.java | 99 +++++++++ .../keyvalue/cassandra/HiddenTablesTest.java | 58 +++++ .../SchemaMutationLockTablesTest.java | 117 ++++++++++ .../cassandra/SchemaMutationLockTest.java | 133 +++++++++++ .../UniqueSchemaMutationLockTableTest.java | 152 +++++++++++++ atlasdb-cassandra/build.gradle | 7 + .../cassandra/CQLKeyValueService.java | 89 ++++++++ .../cassandra/CQLKeyValueServices.java | 9 + .../cassandra/CassandraApiVersion.java | 3 + .../cassandra/CassandraClientPool.java | 26 +++ .../CassandraClientPoolingContainer.java | 3 + .../cassandra/CassandraKeyValueService.java | 207 ++++++++++++++++++ .../cassandra/CassandraKeyValueServices.java | 37 ++++ .../keyvalue/cassandra/CassandraVerifier.java | 8 + .../cassandra/SchemaMutationLock.java | 20 ++ .../jmx/CassandraJmxCompactionClient.java | 2 +- atlasdb-cli/build.gradle | 11 + .../cli/command/SingleBackendCommand.java | 7 + .../timestamp/AbstractTimestampCommand.java | 4 + .../timestamp/CleanTransactionRange.java | 7 + .../timestamp/FastForwardTimestamp.java | 4 + .../cli/command/timestamp/FetchTimestamp.java | 4 + .../cli/command/TestTimestampCommand.java | 7 + .../runner/SingleBackendCliTestRunner.java | 5 + .../atlasdb/config/AtlasDbConfigs.java | 14 ++ .../atlasdb/config/ServerListConfig.java | 5 + .../atlasdb/factory/TransactionManagers.java | 78 +++++++ .../atlasdb/http/FailoverFeignTarget.java | 9 + .../console/module/AtlasCoreModule.groovy | 4 + .../dbkvs/impl/AbstractDbReadTable.java | 6 + .../atlasdb/keyvalue/dbkvs/impl/DbKvs.java | 12 + .../dbkvs/impl/ranges/DbKvsGetRanges.java | 157 +++++++------ .../commands/AtlasDbCommandUtils.java | 28 +++ .../commands/AtlasDbCommandUtilsTest.java | 13 ++ atlasdb-ete-tests/build.gradle | 6 + .../palantir/atlasdb/AtlasDbEteServer.java | 14 ++ .../CassandraDropwizardEteTest.java | 4 + .../CassandraNoLeaderDropwizardEteTest.java | 4 + .../dropwizard/DbKvsDropwizardEteTest.java | 4 + .../transaction/impl/SnapshotTransaction.java | 4 + atlasdb-perf/build.gradle | 3 +- .../atlasdb/performance/BenchmarkParam.java | 17 ++ .../performance/PerformanceResults.java | 58 +++-- .../backend/CassandraPhysicalStore.java | 31 +-- .../backend/KeyValueServiceConnector.java | 19 +- .../performance/backend/PhysicalStore.java | 4 +- .../backend/PostgresPhysicalStore.java | 25 ++- .../performance/benchmarks/KvsBenchmarks.java | 53 +++-- .../benchmarks/KvsGetDynamicBenchmarks.java | 52 +++-- .../benchmarks/KvsGetRangeBenchmarks.java | 33 ++- .../benchmarks/KvsPutBenchmarks.java | 9 +- .../performance/cli/AtlasDbPerfCli.java | 15 +- .../atlasdb/server/AtlasDbServiceServer.java | 8 + .../atlasdb/sweep/AbstractSweeperTest.java | 5 + atlasdb-timelock-server/build.gradle | 4 + build.gradle | 6 + circle.yml | 4 + .../com/palantir/nexus/db/sql/SQLString.java | 5 + docs/source/cluster_management/clis.rst | 3 + docs/source/cluster_management/index.rst | 4 + .../cassandra_KVS_configuration.rst | 9 + docs/source/configuration/index.rst | 5 + docs/source/configuration/leader_config.rst | 45 +++- docs/source/release_notes/release-notes.rst | 177 +++++++++++++++ docs/source/schemas/index.rst | 40 ++++ gradle/baseline.gradle | 4 + gradle/versions.gradle | 7 + settings.gradle | 11 + 77 files changed, 2148 insertions(+), 170 deletions(-) create mode 100644 atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraApiVersionTest.java create mode 100644 atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraKeyValueServiceConfigTest.java create mode 100644 atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraKeyValueServiceTableCreationTest.java create mode 100644 atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraKeyValueServiceTest.java create mode 100644 atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/HiddenTablesTest.java create mode 100644 atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/SchemaMutationLockTablesTest.java create mode 100644 atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/SchemaMutationLockTest.java create mode 100644 atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/UniqueSchemaMutationLockTableTest.java diff --git a/atlasdb-api/build.gradle b/atlasdb-api/build.gradle index 4f7806aec7e..79cbf471b67 100644 --- a/atlasdb-api/build.gradle +++ b/atlasdb-api/build.gradle @@ -13,9 +13,12 @@ dependencies { compile 'com.fasterxml.jackson.core:jackson-databind:' + libVersions.jackson compile 'javax.validation:validation-api:1.1.0.Final' +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 compile ('com.palantir.remoting:ssl-config:' + libVersions.http_remoting) { exclude group: 'com.fasterxml.jackson.core', module: 'jackson-databind' } +======= +>>>>>>> merge develop into perf cli branch (#820) testCompile(group: 'junit', name: 'junit', version: libVersions.junit) { exclude group: 'org.hamcrest' diff --git a/atlasdb-api/src/main/java/com/palantir/atlasdb/config/LeaderConfig.java b/atlasdb-api/src/main/java/com/palantir/atlasdb/config/LeaderConfig.java index 1c34f9aa649..f77f067fe21 100644 --- a/atlasdb-api/src/main/java/com/palantir/atlasdb/config/LeaderConfig.java +++ b/atlasdb-api/src/main/java/com/palantir/atlasdb/config/LeaderConfig.java @@ -51,8 +51,11 @@ public File acceptorLogDir() { @Size(min = 1) public abstract Set leaders(); +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56:atlasdb-api/src/main/java/com/palantir/atlasdb/config/LeaderConfig.java public abstract Optional sslConfiguration(); +======= +>>>>>>> merge develop into perf cli branch (#820):atlasdb-api/src/main/java/com/palantir/atlasdb/config/LeaderConfig.java @JsonProperty("lockCreator") @Value.Default public String lockCreator() { diff --git a/atlasdb-api/src/main/java/com/palantir/atlasdb/transaction/api/AtlasDbConstraintCheckingMode.java b/atlasdb-api/src/main/java/com/palantir/atlasdb/transaction/api/AtlasDbConstraintCheckingMode.java index f63c8c2aadd..0eb9bd1eb9f 100644 --- a/atlasdb-api/src/main/java/com/palantir/atlasdb/transaction/api/AtlasDbConstraintCheckingMode.java +++ b/atlasdb-api/src/main/java/com/palantir/atlasdb/transaction/api/AtlasDbConstraintCheckingMode.java @@ -27,6 +27,7 @@ private enum ShouldThrow { NO } +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 private final ShouldThrow shouldThrow; AtlasDbConstraintCheckingMode(ShouldThrow shouldThrow) { @@ -35,6 +36,16 @@ private enum ShouldThrow { public boolean shouldThrowException() { return shouldThrow == ShouldThrow.YES; +======= + private final ShouldThrow shouldThrowException; + + AtlasDbConstraintCheckingMode(ShouldThrow shouldThrowException) { + this.shouldThrowException = shouldThrowException; + } + + public boolean shouldThrowException() { + return shouldThrowException == ShouldThrow.YES; +>>>>>>> merge develop into perf cli branch (#820) } } diff --git a/atlasdb-cassandra-integration-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CQLKeyValueServiceTransactionIntegrationTest.java b/atlasdb-cassandra-integration-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CQLKeyValueServiceTransactionIntegrationTest.java index 30d4a09599d..b8d0078cc8a 100644 --- a/atlasdb-cassandra-integration-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CQLKeyValueServiceTransactionIntegrationTest.java +++ b/atlasdb-cassandra-integration-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CQLKeyValueServiceTransactionIntegrationTest.java @@ -17,12 +17,17 @@ import com.palantir.atlasdb.cassandra.CassandraKeyValueServiceConfigManager; import com.palantir.atlasdb.keyvalue.api.KeyValueService; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56:atlasdb-cassandra-integration-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CQLKeyValueServiceTransactionIntegrationTest.java import com.palantir.atlasdb.transaction.impl.AbstractTransactionTest; public class CQLKeyValueServiceTransactionIntegrationTest extends AbstractTransactionTest { +======= +import com.palantir.atlasdb.sweep.AbstractSweeperTest; +>>>>>>> merge develop into perf cli branch (#820):atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraKeyValueServiceSweeperTest.java @Override protected KeyValueService getKeyValueService() { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56:atlasdb-cassandra-integration-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CQLKeyValueServiceTransactionIntegrationTest.java return CQLKeyValueService.create( CassandraKeyValueServiceConfigManager.createSimpleManager(CQLTestSuite.CQLKVS_CONFIG)); } @@ -32,4 +37,9 @@ protected boolean supportsReverse() { return false; } +======= + return CassandraKeyValueService.create( + CassandraKeyValueServiceConfigManager.createSimpleManager(CassandraTestSuite.CASSANDRA_KVS_CONFIG), CassandraTestSuite.LEADER_CONFIG); + } +>>>>>>> merge develop into perf cli branch (#820):atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraKeyValueServiceSweeperTest.java } diff --git a/atlasdb-cassandra-integration-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraClientPoolIntegrationTest.java b/atlasdb-cassandra-integration-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraClientPoolIntegrationTest.java index 0aef72f1050..4b7c139d347 100644 --- a/atlasdb-cassandra-integration-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraClientPoolIntegrationTest.java +++ b/atlasdb-cassandra-integration-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraClientPoolIntegrationTest.java @@ -42,6 +42,10 @@ import com.google.common.collect.Range; import com.palantir.atlasdb.AtlasDbConstants; import com.palantir.atlasdb.cassandra.CassandraKeyValueServiceConfigManager; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56:atlasdb-cassandra-integration-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraClientPoolIntegrationTest.java +======= +import com.palantir.atlasdb.cassandra.ImmutableCassandraKeyValueServiceConfig; +>>>>>>> merge develop into perf cli branch (#820):atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraClientPoolTest.java import com.palantir.atlasdb.keyvalue.cassandra.CassandraClientPool.LightweightOppToken; import com.palantir.atlasdb.keyvalue.cassandra.CassandraClientPool.WeightedHosts; import com.palantir.common.base.FunctionCheckedException; @@ -52,7 +56,23 @@ public class CassandraClientPoolIntegrationTest { @Before public void setUp() { kv = CassandraKeyValueService.create( +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56:atlasdb-cassandra-integration-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraClientPoolIntegrationTest.java CassandraKeyValueServiceConfigManager.createSimpleManager(CassandraTestSuite.CASSANDRA_KVS_CONFIG), +======= + CassandraKeyValueServiceConfigManager.createSimpleManager( + ImmutableCassandraKeyValueServiceConfig.builder() + .addServers(new InetSocketAddress("localhost", 9160)) + .poolSize(20) + .keyspace("atlasdb") + .ssl(false) + .replicationFactor(1) + .mutationBatchCount(10000) + .mutationBatchSizeBytes(10000000) + .fetchBatchCount(1000) + .safetyDisabled(true) + .autoRefreshNodes(true) + .build()), +>>>>>>> merge develop into perf cli branch (#820):atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraClientPoolTest.java CassandraTestSuite.LEADER_CONFIG); kv.initializeFromFreshInstance(); kv.dropTable(AtlasDbConstants.TIMESTAMP_TABLE); diff --git a/atlasdb-cassandra-integration-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraTestSuite.java b/atlasdb-cassandra-integration-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraTestSuite.java index 025769c9ec3..6126ccf4ba1 100644 --- a/atlasdb-cassandra-integration-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraTestSuite.java +++ b/atlasdb-cassandra-integration-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraTestSuite.java @@ -43,6 +43,7 @@ @RunWith(Suite.class) @SuiteClasses({ +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56:atlasdb-cassandra-integration-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraTestSuite.java CassandraKeyValueServiceTransactionIntegrationTest.class, CassandraClientPoolIntegrationTest.class, CassandraConnectionIntegrationTest.class, @@ -53,6 +54,16 @@ CassandraKeyValueServiceIntegrationTest.class, SchemaMutationLockIntegrationTest.class, SchemaMutationLockTablesIntegrationTest.class, +======= + CassandraConnectionTest.class, + CassandraKeyValueServiceTableCreationTest.class, + CassandraKeyValueServiceSerializableTransactionTest.class, + CassandraKeyValueServiceSweeperTest.class, + CassandraTimestampTest.class, + CassandraKeyValueServiceTest.class, + SchemaMutationLockTest.class, + SchemaMutationLockTablesTest.class, +>>>>>>> merge develop into perf cli branch (#820):atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraTestSuite.java }) public class CassandraTestSuite { diff --git a/atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraApiVersionTest.java b/atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraApiVersionTest.java new file mode 100644 index 00000000000..4e30bff6db6 --- /dev/null +++ b/atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraApiVersionTest.java @@ -0,0 +1,65 @@ +/** + * Copyright 2016 Palantir Technologies + * + * Licensed under the BSD-3 License (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://opensource.org/licenses/BSD-3-Clause + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.palantir.atlasdb.keyvalue.cassandra; + +import static org.hamcrest.MatcherAssert.assertThat; +import static org.hamcrest.core.Is.is; + +import org.junit.Test; + +public class CassandraApiVersionTest { + @Test public void + version_19_36_0_does_not_support_cas() { + CassandraApiVersion version = new CassandraApiVersion("19.36.0"); + assertThat(version.supportsCheckAndSet(), is(false)); + } + + @Test public void + version_19_37_0_supports_cas() { + CassandraApiVersion version = new CassandraApiVersion("19.37.0"); + assertThat(version.supportsCheckAndSet(), is(true)); + } + + @Test public void + version_19_38_0_supports_cas() { + CassandraApiVersion version = new CassandraApiVersion("19.38.0"); + assertThat(version.supportsCheckAndSet(), is(true)); + } + + @Test public void + version_20_1_0_supports_cas() { + CassandraApiVersion version = new CassandraApiVersion("20.1.0"); + assertThat(version.supportsCheckAndSet(), is(true)); + } + + @Test public void + version_18_40_0_does_not_support_cas() { + CassandraApiVersion version = new CassandraApiVersion("18.40.0"); + assertThat(version.supportsCheckAndSet(), is(false)); + } + + @Test public void + version_20_40_1_supports_cas() { + CassandraApiVersion version = new CassandraApiVersion("20.40.1"); + assertThat(version.supportsCheckAndSet(), is(true)); + } + + @Test(expected=UnsupportedOperationException.class) public void + invalid_version_strings_throw_an_error() { + new CassandraApiVersion("20_4.1"); + } + +} diff --git a/atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraKeyValueServiceConfigTest.java b/atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraKeyValueServiceConfigTest.java new file mode 100644 index 00000000000..86722019d94 --- /dev/null +++ b/atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraKeyValueServiceConfigTest.java @@ -0,0 +1,64 @@ +/** + * Copyright 2015 Palantir Technologies + * + * Licensed under the BSD-3 License (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://opensource.org/licenses/BSD-3-Clause + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.palantir.atlasdb.keyvalue.cassandra; + +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import java.net.InetSocketAddress; +import java.nio.file.Paths; + +import org.junit.Test; + +import com.palantir.atlasdb.cassandra.ImmutableCassandraKeyValueServiceConfig; +import com.palantir.remoting.ssl.SslConfiguration; + +public class CassandraKeyValueServiceConfigTest { + private static final InetSocketAddress SERVER_ADDRESS = InetSocketAddress.createUnresolved("localhost", 9160); + private static final SslConfiguration SSL_CONFIGURATION = SslConfiguration.of(Paths.get("./trustStore.jks")); + + private static final ImmutableCassandraKeyValueServiceConfig CASSANDRA_CONFIG = ImmutableCassandraKeyValueServiceConfig.builder() + .addServers(SERVER_ADDRESS) + .replicationFactor(1) + .keyspace("atlasdb") + .build(); + + + @Test + public void usingSslIfSslParamPresentAndTrue() { + assertTrue(CASSANDRA_CONFIG.withSsl(true).usingSsl()); + } + + @Test + public void notUsingSslIfSslParamPresentAndFalse() { + assertFalse(CASSANDRA_CONFIG.withSsl(false).usingSsl()); + } + + @Test + public void notUsingSslIfSslParamFalseAndSslConfigurationPresent() { + assertFalse(CASSANDRA_CONFIG.withSsl(false).withSslConfiguration(SSL_CONFIGURATION).usingSsl()); + } + + @Test + public void usingSslIfSslParamNotPresentAndSslConfigurationPresent() { + assertTrue(CASSANDRA_CONFIG.withSslConfiguration(SSL_CONFIGURATION).usingSsl()); + } + + @Test + public void notUsingSslIfSslParamNotPresentAndSslConfigurationNotPresent() { + assertFalse(CASSANDRA_CONFIG.usingSsl()); + } +} diff --git a/atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraKeyValueServiceTableCreationTest.java b/atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraKeyValueServiceTableCreationTest.java new file mode 100644 index 00000000000..49ea066e474 --- /dev/null +++ b/atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraKeyValueServiceTableCreationTest.java @@ -0,0 +1,94 @@ +/** + * Copyright 2015 Palantir Technologies + * + * Licensed under the BSD-3 License (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://opensource.org/licenses/BSD-3-Clause + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.palantir.atlasdb.keyvalue.cassandra; + +import java.util.concurrent.BrokenBarrierException; +import java.util.concurrent.CyclicBarrier; +import java.util.concurrent.ForkJoinPool; +import java.util.stream.IntStream; + +import org.junit.After; +import org.junit.Before; +import org.junit.Test; + +import com.palantir.atlasdb.AtlasDbConstants; +import com.palantir.atlasdb.cassandra.CassandraKeyValueServiceConfigManager; +import com.palantir.atlasdb.cassandra.ImmutableCassandraKeyValueServiceConfig; +import com.palantir.atlasdb.keyvalue.api.TableReference; + +public class CassandraKeyValueServiceTableCreationTest { + public static final TableReference GOOD_TABLE = TableReference.createFromFullyQualifiedName("foo.bar"); + public static final TableReference BAD_TABLE = TableReference.createFromFullyQualifiedName("foo.b@r"); + + protected CassandraKeyValueService kvs; + protected CassandraKeyValueService slowTimeoutKvs; + + @Before + public void setUp() { + ImmutableCassandraKeyValueServiceConfig quickTimeoutConfig = CassandraTestSuite.CASSANDRA_KVS_CONFIG + .withSchemaMutationTimeoutMillis(500); + kvs = CassandraKeyValueService.create( + CassandraKeyValueServiceConfigManager.createSimpleManager(quickTimeoutConfig), CassandraTestSuite.LEADER_CONFIG); + + ImmutableCassandraKeyValueServiceConfig slowTimeoutConfig = CassandraTestSuite.CASSANDRA_KVS_CONFIG + .withSchemaMutationTimeoutMillis(60 * 1000); + slowTimeoutKvs = CassandraKeyValueService.create( + CassandraKeyValueServiceConfigManager.createSimpleManager(slowTimeoutConfig), CassandraTestSuite.LEADER_CONFIG); + + kvs.dropTable(AtlasDbConstants.TIMESTAMP_TABLE); + } + + @After + public void tearDown() { + kvs.teardown(); + } + + @Test (timeout = 10 * 1000) + public void testTableCreationCanOccurAfterError() { + try { + kvs.createTable(BAD_TABLE, AtlasDbConstants.GENERIC_TABLE_METADATA); + } catch (Exception e) { + // failure expected + } + kvs.createTable(GOOD_TABLE, AtlasDbConstants.GENERIC_TABLE_METADATA); + kvs.dropTable(GOOD_TABLE); + } + + @Test + public void testCreatingMultipleTablesAtOnce() { + int threadCount = 16; + CyclicBarrier barrier = new CyclicBarrier(threadCount); + ForkJoinPool threadPool = new ForkJoinPool(threadCount); + + threadPool.submit(() -> { + IntStream.range(0, threadCount).parallel().forEach(i -> { + try { + barrier.await(); + slowTimeoutKvs.createTable(GOOD_TABLE, AtlasDbConstants.GENERIC_TABLE_METADATA); + } catch (BrokenBarrierException | InterruptedException e) { + // Do nothing + } + }); + }); + + slowTimeoutKvs.dropTable(GOOD_TABLE); + } + + @Test + public void describeVersionBehavesCorrectly() throws Exception { + kvs.clientPool.runWithRetry(CassandraVerifier.underlyingCassandraClusterSupportsCASOperations); + } +} diff --git a/atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraKeyValueServiceTest.java b/atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraKeyValueServiceTest.java new file mode 100644 index 00000000000..6f12543c497 --- /dev/null +++ b/atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraKeyValueServiceTest.java @@ -0,0 +1,99 @@ +/** + * Copyright 2016 Palantir Technologies + * + * Licensed under the BSD-3 License (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://opensource.org/licenses/BSD-3-Clause + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.palantir.atlasdb.keyvalue.cassandra; + +import static org.mockito.Matchers.startsWith; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; + +import java.util.Set; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; + +import org.apache.thrift.TException; +import org.junit.After; +import org.junit.Before; +import org.junit.Ignore; +import org.junit.Test; +import org.slf4j.Logger; + +import com.google.common.base.Preconditions; +import com.palantir.atlasdb.AtlasDbConstants; +import com.palantir.atlasdb.cassandra.CassandraKeyValueServiceConfigManager; +import com.palantir.atlasdb.keyvalue.api.KeyValueService; +import com.palantir.atlasdb.keyvalue.api.TableReference; +import com.palantir.atlasdb.keyvalue.impl.AbstractAtlasDbKeyValueServiceTest; + +public class CassandraKeyValueServiceTest extends AbstractAtlasDbKeyValueServiceTest { + private KeyValueService keyValueService; + private ExecutorService executorService; + private Logger logger = mock(Logger.class); + + @Before + public void setupKVS() { + keyValueService = getKeyValueService(); + executorService = Executors.newFixedThreadPool(4); + } + + @After + public void cleanUp() { + executorService.shutdown(); + } + + @Override + protected KeyValueService getKeyValueService() { + return CassandraKeyValueService.create( + CassandraKeyValueServiceConfigManager.createSimpleManager(CassandraTestSuite.CASSANDRA_KVS_CONFIG), CassandraTestSuite.LEADER_CONFIG, logger); + } + + @Override + protected boolean reverseRangesSupported() { + return false; + } + + @Override + @Ignore + public void testGetRangeWithHistory() { + // + } + + @Override + @Ignore + public void testGetAllTableNames() { + // + } + + @Test + public void testCreateTableCaseInsensitive() throws TException { + TableReference table1 = TableReference.createFromFullyQualifiedName("ns.tAbLe"); + TableReference table2 = TableReference.createFromFullyQualifiedName("ns.table"); + TableReference table3 = TableReference.createFromFullyQualifiedName("ns.TABle"); + keyValueService.createTable(table1, AtlasDbConstants.GENERIC_TABLE_METADATA); + keyValueService.createTable(table2, AtlasDbConstants.GENERIC_TABLE_METADATA); + keyValueService.createTable(table3, AtlasDbConstants.GENERIC_TABLE_METADATA); + Set allTables = keyValueService.getAllTableNames(); + Preconditions.checkArgument(allTables.contains(table1)); + Preconditions.checkArgument(!allTables.contains(table2)); + Preconditions.checkArgument(!allTables.contains(table3)); + } + + @Test + public void shouldNotErrorForTimestampTableWhenCreatingCassandraKVS() throws Exception { + verify(logger, never()).error(startsWith("Found a table " + AtlasDbConstants.TIMESTAMP_TABLE)); + } +} \ No newline at end of file diff --git a/atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/HiddenTablesTest.java b/atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/HiddenTablesTest.java new file mode 100644 index 00000000000..5cf7a029597 --- /dev/null +++ b/atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/HiddenTablesTest.java @@ -0,0 +1,58 @@ +/** + * Copyright 2016 Palantir Technologies + * + * Licensed under the BSD-3 License (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://opensource.org/licenses/BSD-3-Clause + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.palantir.atlasdb.keyvalue.cassandra; + +import static org.hamcrest.MatcherAssert.assertThat; +import static org.hamcrest.core.Is.is; + +import org.junit.Test; + +import com.palantir.atlasdb.AtlasDbConstants; +import com.palantir.atlasdb.keyvalue.api.TableReference; + +public class HiddenTablesTest { + private final HiddenTables hiddenTables = new HiddenTables(); + + @Test public void + shouldSayTimestampIsHidden() { + assertThat(hiddenTables.isHidden(AtlasDbConstants.TIMESTAMP_TABLE), is(true)); + } + + @Test public void + shouldSayMetadataIsHidden() { + assertThat(hiddenTables.isHidden(AtlasDbConstants.METADATA_TABLE), is(true)); + } + + @Test public void + shouldSayAnOldStyleLocksTableIsHidden() { + assertThat(hiddenTables.isHidden(TableReference.createUnsafe("_locks")), is(true)); + } + + @Test public void + shouldSayANewStyleLocksTableIsHidden() { + assertThat(hiddenTables.isHidden(TableReference.createUnsafe("_locks_aaaa_123")), is(true)); + } + + @Test public void + shouldSayANamespacedTableIsNotHidden() { + assertThat(hiddenTables.isHidden(TableReference.createFromFullyQualifiedName("namespace.table")), is(false)); + } + + @Test public void + shouldSayANonNamespacedVisibleTableIsNotHidden() { + assertThat(hiddenTables.isHidden(TableReference.createWithEmptyNamespace("table")), is(false)); + } +} diff --git a/atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/SchemaMutationLockTablesTest.java b/atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/SchemaMutationLockTablesTest.java new file mode 100644 index 00000000000..c6e8a31baee --- /dev/null +++ b/atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/SchemaMutationLockTablesTest.java @@ -0,0 +1,117 @@ +/** + * Copyright 2016 Palantir Technologies + * + * Licensed under the BSD-3 License (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://opensource.org/licenses/BSD-3-Clause + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.palantir.atlasdb.keyvalue.cassandra; + +import static java.util.Collections.synchronizedList; +import static java.util.stream.IntStream.range; + +import static org.hamcrest.MatcherAssert.assertThat; +import static org.hamcrest.Matchers.empty; +import static org.hamcrest.Matchers.hasItem; +import static org.hamcrest.Matchers.hasSize; +import static org.hamcrest.Matchers.is; +import static org.hamcrest.Matchers.startsWith; + +import java.util.ArrayList; +import java.util.List; +import java.util.Set; +import java.util.UUID; +import java.util.concurrent.Callable; +import java.util.concurrent.CyclicBarrier; +import java.util.function.IntConsumer; + +import org.apache.thrift.TException; +import org.junit.Before; +import org.junit.Test; + +import com.google.common.collect.Iterables; +import com.palantir.atlasdb.cassandra.CassandraKeyValueServiceConfig; +import com.palantir.atlasdb.keyvalue.api.TableReference; + +public class SchemaMutationLockTablesTest { + private SchemaMutationLockTables lockTables; + private CassandraKeyValueServiceConfig config; + private CassandraClientPool clientPool; + + @Before + public void setupKVS() throws TException, InterruptedException { + config = CassandraTestSuite.CASSANDRA_KVS_CONFIG + .withKeyspace(UUID.randomUUID().toString().replace('-', '_')); // Hyphens not allowed in C* schema + clientPool = new CassandraClientPool(config); + clientPool.runOneTimeStartupChecks(); + lockTables = new SchemaMutationLockTables(clientPool, config); + } + + @Test + public void startsWithNoTables() throws TException { + assertThat(lockTables.getAllLockTables(), is(empty())); + } + + @Test + public void tableShouldExistAfterCreation() throws Exception { + lockTables.createLockTable(); + assertThat(lockTables.getAllLockTables(), hasSize(1)); + } + + @Test + public void multipleLockTablesExistAfterCreation() throws Exception { + lockTables.createLockTable(); + lockTables.createLockTable(); + assertThat(lockTables.getAllLockTables(), hasSize(2)); + } + + @Test + public void multipleSchemaMutationLockTablesObjectsShouldReturnSameLockTables() throws Exception { + SchemaMutationLockTables lockTables2 = new SchemaMutationLockTables(clientPool, config); + lockTables.createLockTable(); + assertThat(lockTables.getAllLockTables(), is(lockTables2.getAllLockTables())); + } + + @Test + public void shouldCreateLockTablesStartingWithCorrectPrefix() throws TException { + lockTables.createLockTable(); + + assertThat(Iterables.getOnlyElement(lockTables.getAllLockTables()).getTablename(), startsWith("_locks_")); + } + + @Test + public void whenTablesAreCreatedConcurrentlyAtLeastOneThreadShouldSeeBothTables() { + CyclicBarrier barrier = new CyclicBarrier(2); + + List> lockTablesSeen = synchronizedList(new ArrayList<>()); + + range(0,2).parallel() + .forEach(ignoringExceptions( () -> { + barrier.await(); + lockTables.createLockTable(); + lockTablesSeen.add(lockTables.getAllLockTables()); + return null; + })); + + assertThat("Only one table was seen by both creation threads", lockTablesSeen, hasItem(hasSize(2))); + } + + private IntConsumer ignoringExceptions(Callable function) { + return (i) -> { + try { + function.call(); + } catch (Exception e) { + throw new RuntimeException(e); + } + }; + } +} diff --git a/atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/SchemaMutationLockTest.java b/atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/SchemaMutationLockTest.java new file mode 100644 index 00000000000..55f41c4b577 --- /dev/null +++ b/atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/SchemaMutationLockTest.java @@ -0,0 +1,133 @@ +/** + * Copyright 2016 Palantir Technologies + * + * Licensed under the BSD-3 License (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://opensource.org/licenses/BSD-3-Clause + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.palantir.atlasdb.keyvalue.cassandra; + +import static org.hamcrest.CoreMatchers.is; +import static org.hamcrest.MatcherAssert.assertThat; + +import java.util.Arrays; +import java.util.Collection; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import java.util.concurrent.atomic.AtomicInteger; + +import org.apache.cassandra.thrift.ConsistencyLevel; +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.ExpectedException; +import org.junit.runner.RunWith; +import org.junit.runners.Parameterized; + +import com.palantir.atlasdb.cassandra.CassandraKeyValueServiceConfigManager; +import com.palantir.atlasdb.cassandra.ImmutableCassandraKeyValueServiceConfig; +import com.palantir.atlasdb.config.LockLeader; +import com.palantir.common.exception.PalantirRuntimeException; + +@RunWith(Parameterized.class) +public class SchemaMutationLockTest { + public static final SchemaMutationLock.Action DO_NOTHING = () -> {}; + protected SchemaMutationLock schemaMutationLock; + private final ExecutorService executorService = Executors.newFixedThreadPool(4); + + @Parameterized.Parameter(value = 0) + public boolean casEnabled; + + @Parameterized.Parameter(value = 1) + public String expectedTimeoutErrorMessage; + + @Parameterized.Parameters + public static Collection parameters() { + return Arrays.asList(new Object[][] { + { true, "We have timed out waiting on the current schema mutation lock holder." }, + { false, "unable to get a lock on Cassandra system schema mutations" }}); + } + + @Rule + public ExpectedException expectedException = ExpectedException.none(); + + @Before + public void setUp() throws Exception { + setUpWithCasSupportSetTo(casEnabled); + } + + protected void setUpWithCasSupportSetTo(boolean supportsCas) throws Exception { + ImmutableCassandraKeyValueServiceConfig quickTimeoutConfig = CassandraTestSuite.CASSANDRA_KVS_CONFIG + .withSchemaMutationTimeoutMillis(500); + CassandraKeyValueServiceConfigManager simpleManager = CassandraKeyValueServiceConfigManager.createSimpleManager(quickTimeoutConfig); + ConsistencyLevel writeConsistency = ConsistencyLevel.EACH_QUORUM; + CassandraClientPool clientPool = new CassandraClientPool(simpleManager.getConfig()); + + UniqueSchemaMutationLockTable lockTable = new UniqueSchemaMutationLockTable(new SchemaMutationLockTables(clientPool, quickTimeoutConfig), LockLeader.I_AM_THE_LOCK_LEADER); + schemaMutationLock = new SchemaMutationLock(supportsCas, simpleManager, clientPool, writeConsistency, lockTable); + } + + @Test + public void testLockAndUnlockWithoutContention() { + schemaMutationLock.runWithLock(() -> {}); + } + + @Test + public void doesNotPerformAnActionIfTheLockIsAlreadyHeld() { + schemaMutationLock.runWithLock(() -> { + Future getLockAgain = CassandraTestTools.async( + executorService, + () -> schemaMutationLock.runWithLock(DO_NOTHING)); + + Thread.sleep(3*1000); + + CassandraTestTools.assertThatFutureDidNotSucceedYet(getLockAgain); + }); + } + + @Test(timeout = 10 * 1000) + public void canRunAnotherActionOnceTheFirstHasBeenCompleted() { + AtomicInteger counter = new AtomicInteger(); + SchemaMutationLock.Action increment = () -> counter.incrementAndGet(); + + schemaMutationLock.runWithLock(increment); + schemaMutationLock.runWithLock(increment); + + assertThat(counter.get(), is(2)); + } + + @Test + public void shouldWrapCheckedExceptionsInARuntimeException() { + Exception error = new Exception(); + + expectedException.expect(PalantirRuntimeException.class); + expectedException.expectCause(is(error)); + + schemaMutationLock.runWithLock(() -> { throw error; }); + } + + @Test + public void testLocksTimeout() throws InterruptedException, ExecutionException, TimeoutException { + schemaMutationLock.runWithLock(() -> { + expectedException.expect(PalantirRuntimeException.class); + expectedException.expectMessage(expectedTimeoutErrorMessage); + + Future async = CassandraTestTools.async( + executorService, + () -> schemaMutationLock.runWithLock(DO_NOTHING)); + async.get(10, TimeUnit.SECONDS); + }); + } +} diff --git a/atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/UniqueSchemaMutationLockTableTest.java b/atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/UniqueSchemaMutationLockTableTest.java new file mode 100644 index 00000000000..98c1f30724b --- /dev/null +++ b/atlasdb-cassandra-tests/src/test/java/com/palantir/atlasdb/keyvalue/cassandra/UniqueSchemaMutationLockTableTest.java @@ -0,0 +1,152 @@ +/** + * Copyright 2016 Palantir Technologies + * + * Licensed under the BSD-3 License (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://opensource.org/licenses/BSD-3-Clause + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.palantir.atlasdb.keyvalue.cassandra; + +import static org.hamcrest.MatcherAssert.assertThat; +import static org.hamcrest.Matchers.is; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import java.util.Collections; + +import org.apache.thrift.TException; +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.ExpectedException; + +import com.google.common.collect.ImmutableSet; +import com.palantir.atlasdb.config.LockLeader; +import com.palantir.atlasdb.keyvalue.api.TableReference; + +public class UniqueSchemaMutationLockTableTest { + + private UniqueSchemaMutationLockTable uniqueLockTable; + private SchemaMutationLockTables lockTables; + private TableReference lockTable1 = TableReference.createWithEmptyNamespace(SchemaMutationLockTables.LOCK_TABLE_PREFIX + "_1"); + private TableReference lockTable2 = TableReference.createWithEmptyNamespace(SchemaMutationLockTables.LOCK_TABLE_PREFIX + "_2"); + + @Rule + public ExpectedException exception = ExpectedException.none(); + + @Before + public void setupKVS() throws TException, InterruptedException { + lockTables = mock(SchemaMutationLockTables.class); + uniqueLockTable = new UniqueSchemaMutationLockTable(lockTables, LockLeader.I_AM_THE_LOCK_LEADER); + } + + @Test + public void shouldReturnALockTableIfNoneExist() throws TException { + when(lockTables.getAllLockTables()).thenReturn(Collections.EMPTY_SET, ImmutableSet.of(lockTable1)); + when(lockTables.createLockTable()).thenReturn(lockTable1); + + assertThat(uniqueLockTable.getOnlyTable(), is(lockTable1)); + } + + @Test + public void shouldReturnTheSameLockTableOnMultipleCalls() throws TException { + when(lockTables.getAllLockTables()).thenReturn(ImmutableSet.of(lockTable1)); + + assertThat(uniqueLockTable.getOnlyTable(), is(uniqueLockTable.getOnlyTable())); + } + + @Test + public void shouldNotCreateALockTableIfOneAlreadyExists() throws Exception { + when(lockTables.getAllLockTables()).thenReturn(ImmutableSet.of(lockTable1)); + + uniqueLockTable.getOnlyTable(); + + verify(lockTables, never()).createLockTable(); + } + + @Test + public void shouldReturnTheLockTableIfItIsTheOnlyOne() throws TException { + when(lockTables.getAllLockTables()).thenReturn(ImmutableSet.of(lockTable1)); + + assertThat(uniqueLockTable.getOnlyTable(), is(lockTable1)); + } + + @Test + public void shouldThrowExceptionIfMultipleTablesExist() throws Exception { + when(lockTables.getAllLockTables()).thenReturn(ImmutableSet.of(lockTable1, lockTable2)); + + exception.expect(IllegalStateException.class); + exception.expectMessage("Multiple schema mutation lock tables have been created.\n"); + + try { + uniqueLockTable.getOnlyTable(); + } finally { + verify(lockTables, never()).createLockTable(); + } + } + + @Test + public void shouldThrowExceptionIfMultipleTablesExistSuddenlyDueToConcurrency() throws Exception { + when(lockTables.getAllLockTables()).thenReturn(ImmutableSet.of(), ImmutableSet.of(lockTable1, lockTable2)); + + exception.expect(IllegalStateException.class); + exception.expectMessage("Multiple schema mutation lock tables have been created.\n"); + + try { + uniqueLockTable.getOnlyTable(); + } finally { + verify(lockTables).createLockTable(); + } + } + + @Test + public void shouldNotCreateLockTableIfNotLockLeader() throws Exception { + when(lockTables.getAllLockTables()).thenReturn(ImmutableSet.of(), ImmutableSet.of(lockTable1)); + + uniqueLockTable = new UniqueSchemaMutationLockTable(lockTables, LockLeader.SOMEONE_ELSE_IS_THE_LOCK_LEADER); + + uniqueLockTable.getOnlyTable(); + + verify(lockTables, never()).createLockTable(); + } + + @Test + public void shouldThrowAnExceptionIfMultipleLockTablesAreCreatedWhenWeAreNotTheLockTable() throws TException { + when(lockTables.getAllLockTables()).thenReturn(ImmutableSet.of(lockTable1, lockTable2)); + + uniqueLockTable = new UniqueSchemaMutationLockTable(lockTables, LockLeader.SOMEONE_ELSE_IS_THE_LOCK_LEADER); + + exception.expect(IllegalStateException.class); + exception.expectMessage("Multiple schema mutation lock tables have been created.\n"); + + uniqueLockTable.getOnlyTable(); + } + + @Test(expected = RuntimeException.class) + public void shouldWrapThriftExceptions() throws TException { + when(lockTables.createLockTable()).thenThrow(TException.class); + + uniqueLockTable.getOnlyTable(); + } + + @Test + public void shouldWaitWhenNotTheLockLeader() throws TException { + when(lockTables.getAllLockTables()).thenReturn(ImmutableSet.of(), ImmutableSet.of(lockTable1)); + + uniqueLockTable = new UniqueSchemaMutationLockTable(lockTables, LockLeader.SOMEONE_ELSE_IS_THE_LOCK_LEADER); + + assertThat(uniqueLockTable.getOnlyTable(), is(lockTable1)); + } +} + diff --git a/atlasdb-cassandra/build.gradle b/atlasdb-cassandra/build.gradle index db3a8d7ae4f..ac25e2db258 100644 --- a/atlasdb-cassandra/build.gradle +++ b/atlasdb-cassandra/build.gradle @@ -20,14 +20,21 @@ dependencies { compile 'org.apache.commons:commons-pool2:2.4.2' +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 compile ('com.palantir.remoting:ssl-config:' + libVersions.http_remoting) { +======= + compile ('com.palantir.remoting:ssl-config:0.13.0') { +>>>>>>> merge develop into perf cli branch (#820) exclude group: 'com.fasterxml.jackson.core', module: 'jackson-databind' } compile group: 'com.google.code.findbugs', name: 'annotations', version: libVersions.findbugsAnnotations +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 testCompile 'org.mockito:mockito-core:' + libVersions.mockito +======= +>>>>>>> merge develop into perf cli branch (#820) processor 'org.immutables:value:' + libVersions.immutables processor 'com.google.auto.service:auto-service:1.0-rc2' } diff --git a/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CQLKeyValueService.java b/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CQLKeyValueService.java index 2af02b749eb..5b69e650fb4 100644 --- a/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CQLKeyValueService.java +++ b/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CQLKeyValueService.java @@ -126,6 +126,15 @@ public class CQLKeyValueService extends AbstractKeyValueService { private Cluster cluster; private Cluster longRunningQueryCluster; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 +======= + Session session; + Session longRunningQuerySession; + + CQLStatementCache cqlStatementCache; + protected CQLKeyValueServices cqlKeyValueServices; + +>>>>>>> merge develop into perf cli branch (#820) private final CassandraKeyValueServiceConfigManager configManager; private final Optional compactionManager; private ConsistencyLevel readConsistency = ConsistencyLevel.LOCAL_QUORUM; @@ -381,9 +390,16 @@ private Map getRowsAllColsInternal(final TableReference tableRef, throw Throwables.throwUncheckedException(t); } for (Row row : resultSet.all()) { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 Cell cell = Cell.create(getRowName(row), getColName(row)); if ((getTs(row) < startTs) && (!result.containsKey(cell) || (result.get(cell).getTimestamp() < getTs(row)))) { +======= + Cell cell = Cell.create(CQLKeyValueServices.getRowName(row), CQLKeyValueServices.getColName(row)); + if ((CQLKeyValueServices.getTs(row) < startTs) + && (!result.containsKey(cell) + || (result.get(cell).getTimestamp() < CQLKeyValueServices.getTs(row)))) { +>>>>>>> merge develop into perf cli branch (#820) result.put( Cell.create(getRowName(row), getColName(row)), Value.create(getValue(row), getTs(row))); @@ -435,7 +451,25 @@ private void loadWithTs(final TableReference tableRef, boolean loadAllTs, final Visitor> visitor, final ConsistencyLevel consistency) throws Exception { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 final CassandraKeyValueServiceConfig config = configManager.getConfig(); +======= + String loadWithTsQuery = "SELECT * FROM " + getFullTableName(tableRef) + + " WHERE " + CassandraConstants.ROW_NAME + " = ?" + + " AND " + CassandraConstants.COL_NAME_COL + " = ?" + + " AND " + CassandraConstants.TS_COL + " > ?" + + (!loadAllTs ? " LIMIT 1" : ""); + CassandraKeyValueServiceConfig config = configManager.getConfig(); + if (cells.size() > config.fetchBatchCount()) { + log.warn("A call to " + tableRef + + " is performing a multiget " + cells.size() + + " cells; this may indicate overly-large batching on a higher level.\n" + + CassandraKeyValueServices.getFilteredStackTrace("com.palantir")); + } + PreparedStatement preparedStatement = getPreparedStatement(tableRef, loadWithTsQuery, session) + .setConsistencyLevel(consistency); + List resultSetFutures = Lists.newArrayListWithCapacity(cells.size()); +>>>>>>> merge develop into perf cli branch (#820) List resultSetFutures = Lists.newArrayListWithCapacity(cells.size()); SortedSetMultimap cellsByCol = @@ -479,7 +513,11 @@ private void loadWithTs(final TableReference tableRef, + " = ? AND " + fieldNameProvider.timestamp() + " > ?" + (!loadAllTs ? " LIMIT 1" : ""); for (ResultSetFuture rsf : resultSetFutures) { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 visitResults(rsf.getUninterruptibly(), visitor, loggedLoadWithTsQuery, loadAllTs); +======= + visitResults(rsf.getUninterruptibly(), visitor, loadWithTsQuery, loadAllTs); +>>>>>>> merge develop into perf cli branch (#820) } } @@ -529,12 +567,21 @@ private Map getLatestTimestampsInternal(final TableReference tableRe + (timestampByCell.size() % fetchBatchCount > 0 ? 1 : 0); List>> futures = Lists.newArrayListWithCapacity(numPartitions); String loadOnlyTsQuery = "SELECT " +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 + fieldNameProvider.row() + ", " + fieldNameProvider.column() + ", " + fieldNameProvider.timestamp() + " FROM " + getFullTableName(tableRef) + " WHERE " + fieldNameProvider.row() + " = ?" + " AND " + fieldNameProvider.column() + " = ?" +======= + + CassandraConstants.ROW_NAME + ", " + + CassandraConstants.COL_NAME_COL + ", " + + CassandraConstants.TS_COL + + " FROM " + getFullTableName(tableRef) + + " WHERE " + CassandraConstants.ROW_NAME + " = ?" + + " AND " + CassandraConstants.COL_NAME_COL + " = ?" +>>>>>>> merge develop into perf cli branch (#820) + " LIMIT 1"; if (timestampByCell.size() > fetchBatchCount) { log.warn("Re-batching in getLatestTimestamps a call to " + tableRef @@ -561,7 +608,14 @@ public Map call() throws Exception { for (ResultSetFuture resultSetFuture : resultSetFutures) { ResultSet resultSet = resultSetFuture.getUninterruptibly(); for (Row row : resultSet.all()) { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 res.put(Cell.create(getRowName(row), getColName(row)), getTs(row)); +======= + res.put(Cell.create( + CQLKeyValueServices.getRowName(row), + CQLKeyValueServices.getColName(row)), + CQLKeyValueServices.getTs(row)); +>>>>>>> merge develop into perf cli branch (#820) } cqlKeyValueServices.logTracedQuery( loadOnlyTsQuery, @@ -734,10 +788,17 @@ private String getPutUnlessExistsQuery(TableReference tableRef, int ttl) { protected String getPutQuery(TableReference tableName, int ttl) { String putQuery = "INSERT INTO " + getFullTableName(tableName) +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 + " (" + fieldNameProvider.row() + ", " + fieldNameProvider.column() + ", " + fieldNameProvider.timestamp() + ", " + fieldNameProvider.value() + ")" +======= + + " (" + CassandraConstants.ROW_NAME + ", " + + CassandraConstants.COL_NAME_COL + ", " + + CassandraConstants.TS_COL + ", " + + CassandraConstants.VALUE_COL + ")" +>>>>>>> merge develop into perf cli branch (#820) + " VALUES (?, ?, ?, ?)"; if (ttl >= 0) { putQuery += " USING TTL " + ttl; @@ -819,9 +880,15 @@ public void truncateTables(final Set tablesToTruncate) { public void delete(final TableReference tableRef, final Multimap keys) { int cellCount = 0; String deleteQuery = "DELETE FROM " + getFullTableName(tableRef) +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 + " WHERE " + fieldNameProvider.row() + " = ?" + " AND " + fieldNameProvider.column() + " = ?" + " AND " + fieldNameProvider.timestamp() + " = ?"; +======= + + " WHERE " + CassandraConstants.ROW_NAME + " = ?" + + " AND " + CassandraConstants.COL_NAME_COL + " = ?" + + " AND " + CassandraConstants.TS_COL + " = ?"; +>>>>>>> merge develop into perf cli branch (#820) CassandraKeyValueServiceConfig config = configManager.getConfig(); int fetchBatchCount = config.fetchBatchCount(); for (final List batch : Iterables.partition(keys.keySet(), fetchBatchCount)) { @@ -941,7 +1008,11 @@ public ClosableIterator> getRangeWithPageCreator( } bindQuery.append("LIMIT " + batchHint); final String getLastRowQuery = "SELECT * FROM " + getFullTableName(tableRef) + " WHERE " +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 + fieldNameProvider.row() + " = ?"; +======= + + CassandraConstants.ROW_NAME + " = ?"; +>>>>>>> merge develop into perf cli branch (#820) return ClosableIterators.wrap( new AbstractPagingIterable, TokenBackedBasicResultsPage, byte[]>>() { @Override @@ -974,7 +1045,11 @@ TokenBackedBasicResultsPage, byte[]> getPage(final byte[] startKey) byte[] maxRow = null; ResultsExtractor extractor = resultsExtractor.get(); for (Row row : rows) { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 byte[] rowName = getRowName(row); +======= + byte[] rowName = CQLKeyValueServices.getRowName(row); +>>>>>>> merge develop into perf cli branch (#820) if (maxRow == null) { maxRow = rowName; } else { @@ -990,7 +1065,11 @@ TokenBackedBasicResultsPage, byte[]> getPage(final byte[] startKey) // get the rest of the last row BoundStatement boundLastRow = getPreparedStatement(tableRef, getLastRowQuery, session).bind(); +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 boundLastRow.setBytes(fieldNameProvider.row(), ByteBuffer.wrap(maxRow)); +======= + boundLastRow.setBytes(CassandraConstants.ROW_NAME, ByteBuffer.wrap(maxRow)); +>>>>>>> merge develop into perf cli branch (#820) try { resultSet = session.execute(boundLastRow); } catch (com.datastax.driver.core.exceptions.UnavailableException e) { @@ -1004,10 +1083,17 @@ TokenBackedBasicResultsPage, byte[]> getPage(final byte[] startKey) extractor.internalExtractResult( timestamp, selection, +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 getRowName(row), getColName(row), getValue(row), getTs(row)); +======= + CQLKeyValueServices.getRowName(row), + CQLKeyValueServices.getColName(row), + CQLKeyValueServices.getValue(row), + CQLKeyValueServices.getTs(row)); +>>>>>>> merge develop into perf cli branch (#820) } SortedMap> resultsByRow = Cells.breakCellsUpByRow(extractor.asMap()); @@ -1293,6 +1379,7 @@ PreparedStatement getPreparedStatement(TableReference tableRef, String query, Se throw Throwables.throwUncheckedException(e.getCause()); } } +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 private boolean shouldTraceQuery(TableReference tableRef) { return tracingPrefs.shouldTraceQuery(tableRef.getQualifiedName()); @@ -1313,4 +1400,6 @@ long getTs(Row row) { byte[] getValue(Row row) { return CassandraKeyValueServices.getBytesFromByteBuffer(row.getBytes(fieldNameProvider.value())); } +======= +>>>>>>> merge develop into perf cli branch (#820) } diff --git a/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CQLKeyValueServices.java b/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CQLKeyValueServices.java index 797004149db..50e0a1aef5d 100644 --- a/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CQLKeyValueServices.java +++ b/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CQLKeyValueServices.java @@ -277,11 +277,20 @@ void createTableWithSettings(TableReference tableRef, byte[] rawMetadata, CQLKey chunkLength = explicitCompressionBlockSizeKB; } +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 queryBuilder.append("CREATE TABLE IF NOT EXISTS " + kvs.getFullTableName(tableRef) + " ( " + kvs.fieldNameProvider.row() + " blob, " + kvs.fieldNameProvider.column() + " blob, " + kvs.fieldNameProvider.timestamp() + " bigint, " + kvs.fieldNameProvider.value() + " blob, " +======= + queryBuilder.append("CREATE TABLE IF NOT EXISTS " + + kvs.getFullTableName(tableRef) + " ( " // full table name (ks.cf) + + CassandraConstants.ROW_NAME + " blob, " + + CassandraConstants.COL_NAME_COL + " blob, " + + CassandraConstants.TS_COL + " bigint, " + + CassandraConstants.VALUE_COL + " blob, " +>>>>>>> merge develop into perf cli branch (#820) + "PRIMARY KEY (" + kvs.fieldNameProvider.row() + ", " + kvs.fieldNameProvider.column() + ", " diff --git a/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraApiVersion.java b/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraApiVersion.java index ad0e7222585..3abc9579a00 100644 --- a/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraApiVersion.java +++ b/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraApiVersion.java @@ -50,9 +50,12 @@ public boolean supportsCheckAndSet() { return supportsCheckAndSet; } +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 @Override public String toString() { return versionString; } +======= +>>>>>>> merge develop into perf cli branch (#820) } diff --git a/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraClientPool.java b/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraClientPool.java index fdb04605785..389b13c7c52 100644 --- a/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraClientPool.java +++ b/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraClientPool.java @@ -19,7 +19,10 @@ import java.net.InetSocketAddress; import java.net.SocketException; import java.net.SocketTimeoutException; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 import java.net.UnknownHostException; +======= +>>>>>>> merge develop into perf cli branch (#820) import java.util.Arrays; import java.util.HashMap; import java.util.List; @@ -278,6 +281,10 @@ private CassandraClientPoolingContainer getRandomGoodHost() { public InetSocketAddress getRandomHostForKey(byte[] key) { List hostsForKey = tokenMap.get(new LightweightOppToken(key)); +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 +======= + SetView liveOwnerHosts; +>>>>>>> merge develop into perf cli branch (#820) if (hostsForKey == null) { log.debug("We attempted to route your query to a cassandra host that already contains the relevant data." @@ -372,6 +379,7 @@ protected static String internalTableName(TableReference tableRef) { return tableName.replaceFirst("\\.", "__"); } +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 private InetSocketAddress getAddressForHostThrowUnchecked(String host) { try { return getAddressForHost(host); @@ -402,6 +410,8 @@ InetSocketAddress getAddressForHost(String host) throws UnknownHostException { } } +======= +>>>>>>> merge develop into perf cli branch (#820) private void refreshTokenRanges() { try { ImmutableRangeMap.Builder> newTokenRing = @@ -413,12 +423,23 @@ private void refreshTokenRanges() { // RangeMap needs a little help with weird 1-node, 1-vnode, this-entire-feature-is-useless case if (tokenRanges.size() == 1) { String onlyEndpoint = Iterables.getOnlyElement(Iterables.getOnlyElement(tokenRanges).getEndpoints()); +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 InetSocketAddress onlyHost = getAddressForHost(onlyEndpoint); newTokenRing.put(Range.all(), ImmutableList.of(onlyHost)); } else { // normal case, large cluster with many vnodes for (TokenRange tokenRange : tokenRanges) { List hosts = tokenRange.getEndpoints().stream() .map(host -> getAddressForHostThrowUnchecked(host)).collect(Collectors.toList()); +======= + InetSocketAddress onlyHost = new InetSocketAddress( + onlyEndpoint, + CassandraConstants.DEFAULT_THRIFT_PORT); + newTokenRing.put(Range.all(), ImmutableList.of(onlyHost)); + } else { // normal case, large cluster with many vnodes + for (TokenRange tokenRange : tokenRanges) { + List hosts = Lists.transform(tokenRange.getEndpoints(), + endpoint -> new InetSocketAddress(endpoint, CassandraConstants.DEFAULT_THRIFT_PORT)); +>>>>>>> merge develop into perf cli branch (#820) LightweightOppToken startToken = new LightweightOppToken( BaseEncoding.base16().decode(tokenRange.getStart_token().toUpperCase())); LightweightOppToken endToken = new LightweightOppToken( @@ -655,8 +676,13 @@ private static NavigableMap buildHostsWeightedByActi int lowerBoundInclusive = 0; NavigableMap weightedHosts = new TreeMap<>(); +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 for (Entry entry : openRequestsByHost.entrySet()) { // We want the weight to be inversely proportional to the number of open requests so that we pick +======= + for (Entry entry : activeConnectionsByHost.entrySet()) { + // We want the weight to be inversely proportional to the number of active connections so that we pick +>>>>>>> merge develop into perf cli branch (#820) // less-active hosts. We add 1 to make sure that all ranges are non-empty int weight = totalOpenRequests - entry.getValue() + 1; weightedHosts.put(lowerBoundInclusive + weight, entry.getKey()); diff --git a/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraClientPoolingContainer.java b/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraClientPoolingContainer.java index 997c810afd4..7360680f5de 100644 --- a/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraClientPoolingContainer.java +++ b/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraClientPoolingContainer.java @@ -90,7 +90,10 @@ public V runWithPooledResource(FunctionCheckedException + " started at " + DateTimeFormatter.ISO_INSTANT.format(Instant.now()) + " - " + count.getAndIncrement()); try { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 openRequests.getAndIncrement(); +======= +>>>>>>> merge develop into perf cli branch (#820) return runWithGoodResource(fn); } catch (Throwable t) { log.warn("Error occurred talking to host '{}': {}", host, t.toString()); diff --git a/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraKeyValueService.java b/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraKeyValueService.java index be83a013b82..b8fa7f0bfc1 100644 --- a/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraKeyValueService.java +++ b/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraKeyValueService.java @@ -33,6 +33,10 @@ import java.util.concurrent.Future; import java.util.concurrent.TimeUnit; import java.util.concurrent.TimeoutException; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 +======= +import java.util.stream.Collectors; +>>>>>>> merge develop into perf cli branch (#820) import org.apache.cassandra.thrift.CASResult; import org.apache.cassandra.thrift.Cassandra.Client; @@ -42,6 +46,11 @@ import org.apache.cassandra.thrift.ColumnParent; import org.apache.cassandra.thrift.ConsistencyLevel; import org.apache.cassandra.thrift.Deletion; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 +======= +import org.apache.cassandra.thrift.KeyRange; +import org.apache.cassandra.thrift.KeySlice; +>>>>>>> merge develop into perf cli branch (#820) import org.apache.cassandra.thrift.KsDef; import org.apache.cassandra.thrift.Mutation; import org.apache.cassandra.thrift.SlicePredicate; @@ -154,8 +163,11 @@ public class CassandraKeyValueService extends AbstractKeyValueService { private final ConsistencyLevel writeConsistency = ConsistencyLevel.EACH_QUORUM; private final ConsistencyLevel deleteConsistency = ConsistencyLevel.ALL; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 private final TracingQueryRunner queryRunner; +======= +>>>>>>> merge develop into perf cli branch (#820) public static CassandraKeyValueService create( CassandraKeyValueServiceConfigManager configManager, Optional leaderConfig) { @@ -192,8 +204,11 @@ protected CassandraKeyValueService(Logger log, SchemaMutationLockTables lockTables = new SchemaMutationLockTables(clientPool, configManager.getConfig()); this.schemaMutationLockTable = new UniqueSchemaMutationLockTable(lockTables, whoIsTheLockCreator()); +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 this.queryRunner = new TracingQueryRunner(log, tracingPrefs); +======= +>>>>>>> merge develop into perf cli branch (#820) } private LockLeader whoIsTheLockCreator() { @@ -211,8 +226,13 @@ protected void init() { clientPool.runOneTimeStartupChecks(); +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 boolean supportsCas = !configManager.getConfig().scyllaDb() && clientPool.runWithRetry(CassandraVerifier.underlyingCassandraClusterSupportsCASOperations); +======= + boolean supportsCas = clientPool.runWithRetry( + CassandraVerifier.underlyingCassandraClusterSupportsCASOperations); +>>>>>>> merge develop into perf cli branch (#820) schemaMutationLock = new SchemaMutationLock( supportsCas, @@ -1018,16 +1038,51 @@ private void batchMutateInternal(Client client, batchMutateInternal(client, ImmutableSet.of(tableRef), map, consistency); } +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 private void batchMutateInternal(Client client, Set tableRefs, Map>> map, ConsistencyLevel consistency) throws TException { queryRunner.run(client, tableRefs, () -> { +======= + private void batchMutateInternal(final Client client, + final Set tableRefs, + final Map>> map, + final ConsistencyLevel consistency) throws TException { + run(client, tableRefs, () -> { +>>>>>>> merge develop into perf cli branch (#820) client.batch_mutate(map, consistency); return true; }); } +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 +======= + private boolean shouldTraceQuery(Set tableRefs) { + for (TableReference tableRef : tableRefs) { + if (shouldTraceQuery(tableRef)) { + return true; + } + } + return false; + } + + private void logFailedCall(Set tableRefs) { + log.error("A call to table(s) {} failed with an exception.", + tableRefs.stream().map(TableReference::getQualifiedName).collect(Collectors.joining(", "))); + } + + private void logTraceResults(long duration, Set tableRefs, ByteBuffer recvTrace, boolean failed) { + if (failed || duration > getMinimumDurationToTraceMillis()) { + log.error("Traced a call to {} that {}took {} ms. It will appear in system_traces with UUID={}", + tableRefs.stream().map(TableReference::getQualifiedName).collect(Collectors.joining(", ")), + failed ? "failed and " : "", + duration, + CassandraKeyValueServices.convertCassandraByteBufferUuidToString(recvTrace)); + } + } + +>>>>>>> merge develop into perf cli branch (#820) private Map> multigetInternal( Client client, TableReference tableRef, @@ -1035,7 +1090,11 @@ private Map> multigetInternal( ColumnParent colFam, SlicePredicate pred, ConsistencyLevel consistency) throws TException { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 return queryRunner.run(client, tableRef, () -> client.multiget_slice(rowNames, colFam, pred, consistency)); +======= + return run(client, tableRef, () -> client.multiget_slice(rowNames, colFam, pred, consistency)); +>>>>>>> merge develop into perf cli branch (#820) } @Override @@ -1074,7 +1133,11 @@ private void truncateInternal(Client client, TableReference tableRef) throws TEx for (int tries = 1; tries <= CassandraConstants.MAX_TRUNCATION_ATTEMPTS; tries++) { boolean successful = true; try { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 queryRunner.run(client, tableRef, () -> { +======= + run(client, tableRef, () -> { +>>>>>>> merge develop into perf cli branch (#820) client.truncate(internalTableName(tableRef)); return true; }); @@ -1273,6 +1336,7 @@ public ClosableIterator>> getRangeOfTimestamps( TableReference tableRef, RangeRequest rangeRequest, long timestamp) { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 Optional timestampsGetterBatchSize = configManager.getConfig().timestampsGetterBatchSize(); if (timestampsGetterBatchSize.isPresent()) { return getTimestampsInBatchesWithPageCreator( @@ -1289,6 +1353,14 @@ public ClosableIterator>> getRangeOfTimestamps( deleteConsistency, TimestampExtractor.SUPPLIER); } +======= + return getRangeWithPageCreator( + tableRef, + rangeRequest, + timestamp, + deleteConsistency, + TimestampExtractor.SUPPLIER); +>>>>>>> merge develop into perf cli branch (#820) } @Override @@ -1305,6 +1377,7 @@ public ClosableIterator>> getRangeWithHistory( HistoryExtractor.SUPPLIER); } +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 private ClosableIterator>> getTimestampsInBatchesWithPageCreator( TableReference tableRef, RangeRequest rangeRequest, @@ -1320,11 +1393,15 @@ private ClosableIterator>> getTimestampsInBatchesWithPageCre } private ClosableIterator> getRangeWithPageCreator( +======= + public ClosableIterator> getRangeWithPageCreator( +>>>>>>> merge develop into perf cli branch (#820) TableReference tableRef, RangeRequest rangeRequest, long timestamp, ConsistencyLevel consistency, Supplier> resultsExtractor) { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 RowGetter rowGetter = new RowGetter(clientPool, queryRunner, consistency, tableRef, ColumnFetchMode.FETCH_ALL); ColumnGetter columnGetter = new ThriftColumnGetter(); @@ -1337,12 +1414,15 @@ private ClosableIterator> getRangeWithPageCreator( RangeRequest rangeRequest, Supplier> resultsExtractor, long timestamp) { +======= +>>>>>>> merge develop into perf cli branch (#820) if (rangeRequest.isReverse()) { throw new UnsupportedOperationException(); } if (rangeRequest.isEmptyRange()) { return ClosableIterators.wrap(ImmutableList.>of().iterator()); } +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 CassandraRangePagingIterable rowResults = new CassandraRangePagingIterable<>( rowGetter, @@ -1351,6 +1431,85 @@ private ClosableIterator> getRangeWithPageCreator( resultsExtractor, timestamp ); +======= + final int batchHint = rangeRequest.getBatchHint() == null ? 100 : rangeRequest.getBatchHint(); + SliceRange slice = new SliceRange( + ByteBuffer.wrap(PtBytes.EMPTY_BYTE_ARRAY), + ByteBuffer.wrap(PtBytes.EMPTY_BYTE_ARRAY), + false, + Integer.MAX_VALUE); + final SlicePredicate pred = new SlicePredicate(); + pred.setSlice_range(slice); + + final ColumnParent colFam = new ColumnParent(internalTableName(tableRef)); + final ColumnSelection selection = rangeRequest.getColumnNames().isEmpty() ? ColumnSelection.all() + : ColumnSelection.create(rangeRequest.getColumnNames()); + return ClosableIterators.wrap( + new AbstractPagingIterable, TokenBackedBasicResultsPage, byte[]>>() { + @Override + protected TokenBackedBasicResultsPage, byte[]> getFirstPage() throws Exception { + return page(rangeRequest.getStartInclusive()); + } + + @Override + protected TokenBackedBasicResultsPage, byte[]> getNextPage( + TokenBackedBasicResultsPage, byte[]> previous) throws Exception { + return page(previous.getTokenForNextPage()); + } + + TokenBackedBasicResultsPage, byte[]> page(final byte[] startKey) throws Exception { + InetSocketAddress host = clientPool.getRandomHostForKey(startKey); + return clientPool.runWithRetryOnHost(host, new FunctionCheckedException< + Client, + TokenBackedBasicResultsPage, byte[]>, + Exception>() { + @Override + public TokenBackedBasicResultsPage, byte[]> apply(Client client) + throws Exception { + final byte[] endExclusive = rangeRequest.getEndExclusive(); + + KeyRange keyRange = new KeyRange(batchHint); + keyRange.setStart_key(startKey); + if (endExclusive.length == 0) { + keyRange.setEnd_key(endExclusive); + } else { + // We need the previous name because this is inclusive, not exclusive + keyRange.setEnd_key(RangeRequests.previousLexicographicName(endExclusive)); + } + + List firstPage; + + try { + firstPage = run(client, tableRef, + () -> client.get_range_slices(colFam, pred, keyRange, consistency)); + } catch (UnavailableException e) { + if (consistency.equals(ConsistencyLevel.ALL)) { + throw new InsufficientConsistencyException("This operation requires all" + + " Cassandra nodes to be up and available.", e); + } else { + throw e; + } + } + + Map> colsByKey = + CassandraKeyValueServices.getColsByKey(firstPage); + TokenBackedBasicResultsPage, byte[]> page = resultsExtractor.get() + .getPageFromRangeResults(colsByKey, timestamp, selection, endExclusive); + if (page.moreResultsAvailable() && firstPage.size() < batchHint) { + // If get_range_slices didn't return the full number of results, there's no + // point to trying to get another page + page = SimpleTokenBackedResultsPage.create(endExclusive, page.getResults(), false); + } + return page; + } + + @Override + public String toString() { + return "get_range_slices(" + colFam + ")"; + } + }); + } +>>>>>>> merge develop into perf cli branch (#820) return ClosableIterators.wrap(rowResults.iterator()); } @@ -1447,7 +1606,11 @@ private void createTablesWithLock(final Map tableNamesTo if (!existingTablesLowerCased.contains(tableRefLowerCased)) { client.system_add_column_family(getCfForTable(table, metadata)); } else { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 log.debug("Ignored call to create table ({}) that already existed (case insensitive).", table); +======= + log.warn("Ignored call to create a table ({}) that already existed (case insensitive).", table); +>>>>>>> merge develop into perf cli branch (#820) } } if (!tablesToCreate.isEmpty()) { @@ -1693,7 +1856,11 @@ public Void apply(Client client) throws Exception { col.setName(colName); col.setValue(contents); col.setTimestamp(timestamp); +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 CASResult casResult = queryRunner.run(client, tableRef, () -> client.cas( +======= + CASResult casResult = run(client, tableRef, () -> client.cas( +>>>>>>> merge develop into perf cli branch (#820) rowName, tableName, ImmutableList.of(), @@ -1710,6 +1877,46 @@ public Void apply(Client client) throws Exception { }); } catch (Exception e) { throw Throwables.throwUncheckedException(e); +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 +======= + } + } + + private interface Action { + V run() throws TException; + } + + private V run(Client client, Set tableRefs, Action action) throws TException { + if (shouldTraceQuery(tableRefs)) { + return trace(action, client, tableRefs); + } else { + try { + return action.run(); + } catch (TException e) { + logFailedCall(tableRefs); + throw e; + } + } + } + + private V run(Client client, TableReference tableRef, Action action) throws TException { + return run(client, ImmutableSet.of(tableRef), action); + } + + private V trace(Action action, Client client, Set tableRefs) throws TException { + ByteBuffer traceId = client.trace_next_query(); + Stopwatch stopwatch = Stopwatch.createStarted(); + boolean failed = false; + try { + return action.run(); + } catch (TException e) { + failed = true; + logFailedCall(tableRefs); + throw e; + } finally { + long duration = stopwatch.elapsed(TimeUnit.MILLISECONDS); + logTraceResults(duration, tableRefs, traceId, failed); +>>>>>>> merge develop into perf cli branch (#820) } } diff --git a/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraKeyValueServices.java b/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraKeyValueServices.java index 87cf16f87f0..b813df39c48 100644 --- a/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraKeyValueServices.java +++ b/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraKeyValueServices.java @@ -28,6 +28,10 @@ import org.apache.cassandra.thrift.CfDef; import org.apache.cassandra.thrift.Column; import org.apache.cassandra.thrift.ColumnOrSuperColumn; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 +======= +import org.apache.cassandra.thrift.KeySlice; +>>>>>>> merge develop into perf cli branch (#820) import org.apache.commons.lang3.Validate; import org.apache.thrift.TException; import org.slf4j.Logger; @@ -283,21 +287,36 @@ protected static int convertTtl(final long durationMillis, TimeUnit sourceTimeUn public static boolean isMatchingCf(CfDef clientSide, CfDef clusterSide) { String tableName = clientSide.name; if (!Objects.equal(clientSide.compaction_strategy_options, clusterSide.compaction_strategy_options)) { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 logMismatch("compaction strategy", +======= + log.debug("Found client/server disagreement on compaction strategy options for {}." + + " (client = ({}), server = ({}))", +>>>>>>> merge develop into perf cli branch (#820) tableName, clientSide.compaction_strategy_options, clusterSide.compaction_strategy_options); return false; } if (clientSide.gc_grace_seconds != clusterSide.gc_grace_seconds) { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 logMismatch("gc_grace_seconds period", +======= + log.debug("Found client/server disagreement on gc_grace_seconds for {}." + + " (client = ({}), server = ({}))", +>>>>>>> merge develop into perf cli branch (#820) tableName, clientSide.gc_grace_seconds, clusterSide.gc_grace_seconds); return false; } if (clientSide.bloom_filter_fp_chance != clusterSide.bloom_filter_fp_chance) { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 logMismatch("bloom filter false positive chance", +======= + log.debug("Found client/server disagreement on bloom filter false positive chance for {}." + + " (client = ({}), server = ({}))", +>>>>>>> merge develop into perf cli branch (#820) tableName, clientSide.bloom_filter_fp_chance, clusterSide.bloom_filter_fp_chance); @@ -305,13 +324,19 @@ public static boolean isMatchingCf(CfDef clientSide, CfDef clusterSide) { } if (!(clientSide.compression_options.get(CassandraConstants.CFDEF_COMPRESSION_CHUNK_LENGTH_KEY).equals( clusterSide.compression_options.get(CassandraConstants.CFDEF_COMPRESSION_CHUNK_LENGTH_KEY)))) { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 logMismatch("compression chunk length", +======= + log.debug("Found client/server disagreement on compression chunk length for {}." + + " (client = ({}), server = ({}))", +>>>>>>> merge develop into perf cli branch (#820) tableName, clientSide.compression_options.get(CassandraConstants.CFDEF_COMPRESSION_CHUNK_LENGTH_KEY), clusterSide.compression_options.get(CassandraConstants.CFDEF_COMPRESSION_CHUNK_LENGTH_KEY)); return false; } if (!Objects.equal(clientSide.compaction_strategy, clusterSide.compaction_strategy)) { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 // consider equal "com.whatever.LevelledCompactionStrategy" and "LevelledCompactionStrategy" if (clientSide.compaction_strategy != null && clusterSide.compaction_strategy != null @@ -326,6 +351,18 @@ public static boolean isMatchingCf(CfDef clientSide, CfDef clusterSide) { } if (clientSide.isSetPopulate_io_cache_on_flush() != clusterSide.isSetPopulate_io_cache_on_flush()) { logMismatch("populate_io_cache_on_flush", +======= + log.debug("Found client/server disagreement on compaction_strategy for {}." + + " (client = ({}), server = ({}))", + tableName, + clientSide.compaction_strategy, + clusterSide.compaction_strategy); + return false; + } + if (clientSide.isSetPopulate_io_cache_on_flush() != clusterSide.isSetPopulate_io_cache_on_flush()) { + log.debug("Found client/server disagreement on populate_io_cache_on_flush for {}." + + " (client = ({}), server = ({}))", +>>>>>>> merge develop into perf cli branch (#820) tableName, clientSide.isSetPopulate_io_cache_on_flush(), clusterSide.isSetPopulate_io_cache_on_flush()); diff --git a/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraVerifier.java b/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraVerifier.java index 999927b565f..925825543da 100644 --- a/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraVerifier.java +++ b/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/CassandraVerifier.java @@ -195,11 +195,16 @@ public Void apply(Cassandra.Client client) throws TException { throw ire; } } catch (Exception f) { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 log.warn("Couldn't use host {} to create keyspace." + " It returned exception \"{}\" during the attempt." + " We will retry on other nodes, so this shouldn't be a problem unless all nodes failed." + " See the debug-level log for the stack trace.", host, f.toString(), f); log.debug("Specifically, creating the keyspace failed with the following stack trace", f); +======= + log.error("Couldn't use host {} to create keyspace, it returned exception \"{}\" during" + + " the attempt.", host, f.toString(), f); +>>>>>>> merge develop into perf cli branch (#820) } } if (!someHostWasAbleToCreateTheKeyspace) { @@ -283,7 +288,10 @@ static void checkAndSetReplicationFactor( underlyingCassandraClusterSupportsCASOperations = client -> { try { CassandraApiVersion serverVersion = new CassandraApiVersion(client.describe_version()); +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 log.debug("Connected cassandra thrift version is: " + serverVersion); +======= +>>>>>>> merge develop into perf cli branch (#820) return serverVersion.supportsCheckAndSet(); } catch (TException ex) { throw new UnsupportedOperationException("Couldn't determine underlying cassandra version;" diff --git a/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/SchemaMutationLock.java b/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/SchemaMutationLock.java index 7d9075fab63..74689da8c34 100644 --- a/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/SchemaMutationLock.java +++ b/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/SchemaMutationLock.java @@ -154,9 +154,20 @@ private long waitForSchemaMutationLock() { int mutationTimeoutMillis = configManager.getConfig().schemaMutationTimeoutMillis(); // possibly dead remote locker if (stopwatch.elapsed(TimeUnit.MILLISECONDS) > mutationTimeoutMillis * 4) { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 TimeoutException schemaLockTimeoutError = generateSchemaLockTimeoutException(stopwatch); LOGGER.error(schemaLockTimeoutError.getMessage(), schemaLockTimeoutError); throw Throwables.rewrapAndThrowUncheckedException(schemaLockTimeoutError); +======= + throw new TimeoutException(String.format("We have timed out waiting on the current" + + " schema mutation lock holder. We have tried to grab the lock for %d milliseconds" + + " unsuccessfully. Please try restarting the AtlasDB client. If this occurs" + + " repeatedly it may indicate that the current lock holder has died without" + + " releasing the lock and will require manual intervention. This will require" + + " restarting all atlasDB clients and then using cqlsh to truncate the _locks table." + + " Please contact support for help with this in important situations.", + stopwatch.elapsed(TimeUnit.MILLISECONDS))); +>>>>>>> merge develop into perf cli branch (#820) } long timeToSleep = CassandraConstants.TIME_BETWEEN_LOCK_ATTEMPT_ROUNDS_MILLIS @@ -168,7 +179,10 @@ private long waitForSchemaMutationLock() { } // we won the lock! +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 LOGGER.info("Successfully acquired schema mutation lock."); +======= +>>>>>>> merge develop into perf cli branch (#820) return null; }); } catch (Exception e) { @@ -178,6 +192,7 @@ private long waitForSchemaMutationLock() { return perOperationNodeIdentifier; } +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 private TimeoutException generateSchemaLockTimeoutException(Stopwatch stopwatch) { return new TimeoutException(String.format("We have timed out waiting on the current" + " schema mutation lock holder. We have tried to grab the lock for %d milliseconds" @@ -188,6 +203,8 @@ private TimeoutException generateSchemaLockTimeoutException(Stopwatch stopwatch) stopwatch.elapsed(TimeUnit.MILLISECONDS))); } +======= +>>>>>>> merge develop into perf cli branch (#820) private void schemaMutationUnlock(long perOperationNodeIdentifier) { if (!supportsCas) { schemaMutationLockForEarlierVersionsOfCassandra.unlock(); @@ -222,8 +239,11 @@ private void schemaMutationUnlock(long perOperationNodeIdentifier) { + " from underneath us. Our ID, which we expected, was %s, the value we saw in the" + " database was instead %s.", Long.toString(perOperationNodeIdentifier), remoteLock)); } +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 LOGGER.info("Successfully released schema mutation lock."); +======= +>>>>>>> merge develop into perf cli branch (#820) return null; }); } catch (Exception e) { diff --git a/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/jmx/CassandraJmxCompactionClient.java b/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/jmx/CassandraJmxCompactionClient.java index c51ea442423..0105cd75dc3 100644 --- a/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/jmx/CassandraJmxCompactionClient.java +++ b/atlasdb-cassandra/src/main/java/com/palantir/atlasdb/keyvalue/cassandra/jmx/CassandraJmxCompactionClient.java @@ -38,7 +38,7 @@ /** * Maintains a JMX client for each node in C* cluster. */ -public class CassandraJmxCompactionClient { +public final class CassandraJmxCompactionClient { private static final Logger log = LoggerFactory.getLogger(CassandraJmxCompactionClient.class); private final String host; private final int port; diff --git a/atlasdb-cli/build.gradle b/atlasdb-cli/build.gradle index 92bb36c5404..a55b238f79b 100644 --- a/atlasdb-cli/build.gradle +++ b/atlasdb-cli/build.gradle @@ -7,13 +7,24 @@ mainClassName = 'com.palantir.atlasdb.cli.AtlasCli' applicationName = 'atlasdb' dependencies { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 compile project(':atlasdb-dagger') +======= + compile project(':atlasdb-service') + compile project(':atlasdb-impl-shared') +>>>>>>> merge develop into perf cli branch (#820) compile 'io.airlift:airline:0.7' runtime project(':atlasdb-cassandra') runtime project(':atlasdb-rocksdb') testCompile project(':atlasdb-rocksdb') +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 +======= + + processor 'com.google.dagger:dagger-compiler:2.0.2' + processor 'org.immutables:value:' + libVersions.immutables +>>>>>>> merge develop into perf cli branch (#820) } configurations.matching({ it.name in ['compile', 'runtime'] }).all { diff --git a/atlasdb-cli/src/main/java/com/palantir/atlasdb/cli/command/SingleBackendCommand.java b/atlasdb-cli/src/main/java/com/palantir/atlasdb/cli/command/SingleBackendCommand.java index aff7853f501..faf77e171e4 100644 --- a/atlasdb-cli/src/main/java/com/palantir/atlasdb/cli/command/SingleBackendCommand.java +++ b/atlasdb-cli/src/main/java/com/palantir/atlasdb/cli/command/SingleBackendCommand.java @@ -19,10 +19,17 @@ import com.google.common.annotations.VisibleForTesting; import com.google.common.base.Preconditions; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 import com.palantir.atlasdb.services.AtlasDbServices; import com.palantir.atlasdb.services.AtlasDbServicesFactory; import com.palantir.atlasdb.services.DaggerAtlasDbServices; import com.palantir.atlasdb.services.ServicesConfigModule; +======= +import com.palantir.atlasdb.cli.services.AtlasDbServices; +import com.palantir.atlasdb.cli.services.AtlasDbServicesFactory; +import com.palantir.atlasdb.cli.services.DaggerAtlasDbServices; +import com.palantir.atlasdb.cli.services.ServicesConfigModule; +>>>>>>> merge develop into perf cli branch (#820) import com.palantir.common.base.Throwables; public abstract class SingleBackendCommand extends AbstractCommand { diff --git a/atlasdb-cli/src/main/java/com/palantir/atlasdb/cli/command/timestamp/AbstractTimestampCommand.java b/atlasdb-cli/src/main/java/com/palantir/atlasdb/cli/command/timestamp/AbstractTimestampCommand.java index c03060d4e9c..a094579259f 100644 --- a/atlasdb-cli/src/main/java/com/palantir/atlasdb/cli/command/timestamp/AbstractTimestampCommand.java +++ b/atlasdb-cli/src/main/java/com/palantir/atlasdb/cli/command/timestamp/AbstractTimestampCommand.java @@ -29,7 +29,11 @@ import com.google.common.collect.Iterables; import com.google.common.collect.Sets; import com.palantir.atlasdb.cli.command.SingleBackendCommand; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 import com.palantir.atlasdb.services.AtlasDbServices; +======= +import com.palantir.atlasdb.cli.services.AtlasDbServices; +>>>>>>> merge develop into perf cli branch (#820) import io.airlift.airline.Option; import io.airlift.airline.OptionType; diff --git a/atlasdb-cli/src/main/java/com/palantir/atlasdb/cli/command/timestamp/CleanTransactionRange.java b/atlasdb-cli/src/main/java/com/palantir/atlasdb/cli/command/timestamp/CleanTransactionRange.java index a53d2a3381b..1ef53a5d2ed 100644 --- a/atlasdb-cli/src/main/java/com/palantir/atlasdb/cli/command/timestamp/CleanTransactionRange.java +++ b/atlasdb-cli/src/main/java/com/palantir/atlasdb/cli/command/timestamp/CleanTransactionRange.java @@ -20,12 +20,19 @@ import com.google.common.collect.HashMultimap; import com.google.common.collect.Multimap; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 +======= +import com.palantir.atlasdb.cli.services.AtlasDbServices; +>>>>>>> merge develop into perf cli branch (#820) import com.palantir.atlasdb.keyvalue.api.Cell; import com.palantir.atlasdb.keyvalue.api.KeyValueService; import com.palantir.atlasdb.keyvalue.api.RangeRequest; import com.palantir.atlasdb.keyvalue.api.RowResult; import com.palantir.atlasdb.keyvalue.api.Value; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 import com.palantir.atlasdb.services.AtlasDbServices; +======= +>>>>>>> merge develop into perf cli branch (#820) import com.palantir.atlasdb.transaction.impl.TransactionConstants; import com.palantir.common.base.ClosableIterator; diff --git a/atlasdb-cli/src/main/java/com/palantir/atlasdb/cli/command/timestamp/FastForwardTimestamp.java b/atlasdb-cli/src/main/java/com/palantir/atlasdb/cli/command/timestamp/FastForwardTimestamp.java index ff901d26306..eed992de26f 100644 --- a/atlasdb-cli/src/main/java/com/palantir/atlasdb/cli/command/timestamp/FastForwardTimestamp.java +++ b/atlasdb-cli/src/main/java/com/palantir/atlasdb/cli/command/timestamp/FastForwardTimestamp.java @@ -18,7 +18,11 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 import com.palantir.atlasdb.services.AtlasDbServices; +======= +import com.palantir.atlasdb.cli.services.AtlasDbServices; +>>>>>>> merge develop into perf cli branch (#820) import com.palantir.timestamp.PersistentTimestampService; import com.palantir.timestamp.TimestampService; diff --git a/atlasdb-cli/src/main/java/com/palantir/atlasdb/cli/command/timestamp/FetchTimestamp.java b/atlasdb-cli/src/main/java/com/palantir/atlasdb/cli/command/timestamp/FetchTimestamp.java index c4e8cb51b71..3eaa960138d 100644 --- a/atlasdb-cli/src/main/java/com/palantir/atlasdb/cli/command/timestamp/FetchTimestamp.java +++ b/atlasdb-cli/src/main/java/com/palantir/atlasdb/cli/command/timestamp/FetchTimestamp.java @@ -21,7 +21,11 @@ import org.slf4j.LoggerFactory; import com.palantir.atlasdb.cleaner.KeyValueServicePuncherStore; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 import com.palantir.atlasdb.services.AtlasDbServices; +======= +import com.palantir.atlasdb.cli.services.AtlasDbServices; +>>>>>>> merge develop into perf cli branch (#820) import io.airlift.airline.Command; import io.airlift.airline.Option; diff --git a/atlasdb-cli/src/test/java/com/palantir/atlasdb/cli/command/TestTimestampCommand.java b/atlasdb-cli/src/test/java/com/palantir/atlasdb/cli/command/TestTimestampCommand.java index 859878f50bf..618cd6713b4 100644 --- a/atlasdb-cli/src/test/java/com/palantir/atlasdb/cli/command/TestTimestampCommand.java +++ b/atlasdb-cli/src/test/java/com/palantir/atlasdb/cli/command/TestTimestampCommand.java @@ -47,10 +47,17 @@ import com.palantir.atlasdb.cli.command.timestamp.FetchTimestamp; import com.palantir.atlasdb.cli.runner.InMemoryTestRunner; import com.palantir.atlasdb.cli.runner.SingleBackendCliTestRunner; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 import com.palantir.atlasdb.services.AtlasDbServicesFactory; import com.palantir.atlasdb.services.ServicesConfigModule; import com.palantir.atlasdb.services.test.DaggerTestAtlasDbServices; import com.palantir.atlasdb.services.test.TestAtlasDbServices; +======= +import com.palantir.atlasdb.cli.services.AtlasDbServicesFactory; +import com.palantir.atlasdb.cli.services.DaggerTestAtlasDbServices; +import com.palantir.atlasdb.cli.services.ServicesConfigModule; +import com.palantir.atlasdb.cli.services.TestAtlasDbServices; +>>>>>>> merge develop into perf cli branch (#820) import com.palantir.common.time.Clock; import com.palantir.lock.LockClient; import com.palantir.lock.LockDescriptor; diff --git a/atlasdb-cli/src/test/java/com/palantir/atlasdb/cli/runner/SingleBackendCliTestRunner.java b/atlasdb-cli/src/test/java/com/palantir/atlasdb/cli/runner/SingleBackendCliTestRunner.java index 2426c157782..df78816ca92 100644 --- a/atlasdb-cli/src/test/java/com/palantir/atlasdb/cli/runner/SingleBackendCliTestRunner.java +++ b/atlasdb-cli/src/test/java/com/palantir/atlasdb/cli/runner/SingleBackendCliTestRunner.java @@ -17,8 +17,13 @@ import java.net.URISyntaxException; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 import com.palantir.atlasdb.services.AtlasDbServices; import com.palantir.atlasdb.services.AtlasDbServicesFactory; +======= +import com.palantir.atlasdb.cli.services.AtlasDbServices; +import com.palantir.atlasdb.cli.services.AtlasDbServicesFactory; +>>>>>>> merge develop into perf cli branch (#820) public interface SingleBackendCliTestRunner extends AutoCloseable { diff --git a/atlasdb-config/src/main/java/com/palantir/atlasdb/config/AtlasDbConfigs.java b/atlasdb-config/src/main/java/com/palantir/atlasdb/config/AtlasDbConfigs.java index a54c3518ff9..90641ca0f42 100644 --- a/atlasdb-config/src/main/java/com/palantir/atlasdb/config/AtlasDbConfigs.java +++ b/atlasdb-config/src/main/java/com/palantir/atlasdb/config/AtlasDbConfigs.java @@ -24,10 +24,14 @@ import com.fasterxml.jackson.databind.JsonNode; import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.dataformat.yaml.YAMLFactory; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 import com.fasterxml.jackson.dataformat.yaml.YAMLGenerator; import com.fasterxml.jackson.datatype.guava.GuavaModule; import com.fasterxml.jackson.datatype.jdk7.Jdk7Module; import com.google.common.base.Optional; +======= +import com.fasterxml.jackson.datatype.guava.GuavaModule; +>>>>>>> merge develop into perf cli branch (#820) import com.google.common.base.Strings; import com.palantir.remoting.ssl.SslConfiguration; @@ -37,14 +41,21 @@ public final class AtlasDbConfigs { public static final String ATLASDB_CONFIG_ROOT = "/atlasdb"; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 public static final ObjectMapper OBJECT_MAPPER = new ObjectMapper(new YAMLFactory() .disable(YAMLGenerator.Feature.USE_NATIVE_TYPE_ID) .disable(YAMLGenerator.Feature.WRITE_DOC_START_MARKER)); +======= + private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper(new YAMLFactory()); +>>>>>>> merge develop into perf cli branch (#820) static { OBJECT_MAPPER.setSubtypeResolver(new DiscoverableSubtypeResolver()); OBJECT_MAPPER.registerModule(new GuavaModule()); +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 OBJECT_MAPPER.registerModule(new Jdk7Module()); +======= +>>>>>>> merge develop into perf cli branch (#820) } private AtlasDbConfigs() { @@ -97,6 +108,7 @@ private static JsonNode findRoot(JsonNode node, @Nullable String configRoot) { return null; } return root; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 } public static AtlasDbConfig addFallbackSslConfigurationToAtlasDbConfig( @@ -126,5 +138,7 @@ private static Optional addFallbackSslConfigurationToServerLis .from(serverList) .sslConfiguration(serverList.sslConfiguration().or(sslConfiguration)) .build()); +======= +>>>>>>> merge develop into perf cli branch (#820) } } diff --git a/atlasdb-config/src/main/java/com/palantir/atlasdb/config/ServerListConfig.java b/atlasdb-config/src/main/java/com/palantir/atlasdb/config/ServerListConfig.java index fa85d43785a..8f4e8423dac 100644 --- a/atlasdb-config/src/main/java/com/palantir/atlasdb/config/ServerListConfig.java +++ b/atlasdb-config/src/main/java/com/palantir/atlasdb/config/ServerListConfig.java @@ -30,10 +30,15 @@ @JsonSerialize(as = ImmutableServerListConfig.class) @Value.Immutable public interface ServerListConfig { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 @Size(min = 1) Set servers(); Optional sslConfiguration(); +======= + @Size(min = 1) + Set servers(); +>>>>>>> merge develop into perf cli branch (#820) } diff --git a/atlasdb-config/src/main/java/com/palantir/atlasdb/factory/TransactionManagers.java b/atlasdb-config/src/main/java/com/palantir/atlasdb/factory/TransactionManagers.java index a30c76d5944..2b23b7072ff 100644 --- a/atlasdb-config/src/main/java/com/palantir/atlasdb/factory/TransactionManagers.java +++ b/atlasdb-config/src/main/java/com/palantir/atlasdb/factory/TransactionManagers.java @@ -83,6 +83,7 @@ private TransactionManagers() { } /** +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 * Create a {@link SerializableTransactionManager} with provided configuration, {@link Schema}, * and an environment in which to register HTTP server endpoints. */ @@ -92,6 +93,19 @@ public static SerializableTransactionManager create( Environment env, boolean allowHiddenTableAccess) { return create(config, ImmutableSet.of(schema), env, allowHiddenTableAccess); +======= + * Create a {@link SerializableTransactionManager} with provided configuration, + * {@link SSLSocketFactory}, {@link Schema}, and an environment in which to + * register HTTP server endpoints. + */ + public static SerializableTransactionManager create( + AtlasDbConfig config, + Optional sslSocketFactory, + Schema schema, + Environment env, + boolean allowHiddenTableAccess) { + return create(config, sslSocketFactory, ImmutableSet.of(schema), env, allowHiddenTableAccess); +>>>>>>> merge develop into perf cli branch (#820) } /** @@ -100,6 +114,10 @@ public static SerializableTransactionManager create( */ public static SerializableTransactionManager create( AtlasDbConfig config, +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 +======= + Optional sslSocketFactory, +>>>>>>> merge develop into perf cli branch (#820) Set schemas, Environment env, boolean allowHiddenTableAccess) { @@ -109,14 +127,23 @@ public static SerializableTransactionManager create( LockAndTimestampServices lts = createLockAndTimestampServices( config, +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 +======= + sslSocketFactory, +>>>>>>> merge develop into perf cli branch (#820) env, LockServiceImpl::create, atlasFactory::getTimestampService); KeyValueService kvs = NamespacedKeyValueServices.wrapWithStaticNamespaceMappingKvs(rawKvs); +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 kvs = ValidatingQueryRewritingKeyValueService.create(kvs); kvs = ProfilingKeyValueService.create(kvs); kvs = SweepStatsKeyValueService.create(kvs, lts.time()); +======= + kvs = ProfilingKeyValueService.create(kvs); + kvs = new SweepStatsKeyValueService(kvs, lts.time()); +>>>>>>> merge develop into perf cli branch (#820) TransactionTables.createTables(kvs); @@ -195,7 +222,11 @@ public static LockAndTimestampServices createLockAndTimestampServices( Supplier lock, Supplier time) { LockAndTimestampServices lockAndTimestampServices = +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 createRawServices(config, env, lock, time); +======= + createRawServices(config, sslSocketFactory, env, lock, time); +>>>>>>> merge develop into perf cli branch (#820) return withRefreshingLockService(lockAndTimestampServices); } @@ -209,6 +240,10 @@ private static LockAndTimestampServices withRefreshingLockService( private static LockAndTimestampServices createRawServices( AtlasDbConfig config, +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 +======= + Optional sslSocketFactory, +>>>>>>> merge develop into perf cli branch (#820) Environment env, Supplier lock, Supplier time) { @@ -221,6 +256,7 @@ private static LockAndTimestampServices createRawServices( } } +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 private static LockAndTimestampServices createRawLeaderServices( LeaderConfig leaderConfig, Environment env, @@ -237,6 +273,40 @@ private static LockAndTimestampServices createRawLeaderServices( .lock(createService(sslSocketFactory, leaderConfig.leaders(), RemoteLockService.class)) .time(createService(sslSocketFactory, leaderConfig.leaders(), TimestampService.class)) .build(); +======= + warnIf(config.lock().isPresent(), + "Ignoring lock server configuration because leadership election is enabled"); + warnIf(config.timestamp().isPresent(), + "Ignoring timestamp server configuration because leadership election is enabled"); + + return ImmutableLockAndTimestampServices.builder() + .lock(createService(sslSocketFactory, config.leader().get().leaders(), RemoteLockService.class)) + .time(createService(sslSocketFactory, config.leader().get().leaders(), TimestampService.class)) + .build(); + } else { + warnIf(config.lock().isPresent() != config.timestamp().isPresent(), + "Using embedded instances for one (but not both) of lock and timestamp services"); + + RemoteLockService lockService = config.lock() + .transform(new ServiceCreator<>(sslSocketFactory, RemoteLockService.class)) + .or(lock); + TimestampService timeService = config.timestamp() + .transform(new ServiceCreator<>(sslSocketFactory, TimestampService.class)) + .or(time); + + if (!config.lock().isPresent()) { + env.register(lockService); + } + if (!config.timestamp().isPresent()) { + env.register(timeService); + } + + return ImmutableLockAndTimestampServices.builder() + .lock(lockService) + .time(timeService) + .build(); + } +>>>>>>> merge develop into perf cli branch (#820) } private static LockAndTimestampServices createRawRemoteServices(AtlasDbConfig config) { @@ -249,6 +319,7 @@ private static LockAndTimestampServices createRawRemoteServices(AtlasDbConfig co .build(); } +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 private static LockAndTimestampServices createRawEmbeddedServices( Environment env, Supplier lock, @@ -272,6 +343,8 @@ public static Optional createSslSocketFactory(Optional SslSocketFactories.createSslSocketFactory(config)); } +======= +>>>>>>> merge develop into perf cli branch (#820) private static T createService( Optional sslSocketFactory, Set uris, @@ -282,7 +355,12 @@ private static T createService( private static class ServiceCreator implements Function { private Class serviceClass; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 ServiceCreator(Class serviceClass) { +======= + ServiceCreator(Optional sslSocketFactory, Class serviceClass) { + this.sslSocketFactory = sslSocketFactory; +>>>>>>> merge develop into perf cli branch (#820) this.serviceClass = serviceClass; } diff --git a/atlasdb-config/src/main/java/com/palantir/atlasdb/http/FailoverFeignTarget.java b/atlasdb-config/src/main/java/com/palantir/atlasdb/http/FailoverFeignTarget.java index 6848868e983..d281d76e6b4 100644 --- a/atlasdb-config/src/main/java/com/palantir/atlasdb/http/FailoverFeignTarget.java +++ b/atlasdb-config/src/main/java/com/palantir/atlasdb/http/FailoverFeignTarget.java @@ -112,14 +112,23 @@ private void checkAndHandleFailure(RetryableException ex) { boolean failedDueToNumSwitches = numSwitches.get() >= numServersToTryBeforeFailing; if (failedDueToFastFailover) { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 log.error("This connection has been instructed to fast failover for " +======= + log.warn("This connection has been instructed to fast failover for " +>>>>>>> merge develop into perf cli branch (#820) + TimeUnit.MILLISECONDS.toSeconds(fastFailoverTimeoutMillis) + " seconds without establishing a successful connection." + " The remote hosts have been in a fast failover state for too long."); } else if (failedDueToNumSwitches) { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 log.error("This connection has tried " + numServersToTryBeforeFailing + " hosts rolling across " + servers.size() + " servers, each " + failuresBeforeSwitching + " times and has failed out.", ex); +======= + log.warn("This connection has tried " + numServersToTryBeforeFailing + + " hosts each " + failuresBeforeSwitching + " times and has failed out.", ex); +>>>>>>> merge develop into perf cli branch (#820) } if (failedDueToFastFailover || failedDueToNumSwitches) { diff --git a/atlasdb-console/src/main/groovy/com/palantir/atlasdb/console/module/AtlasCoreModule.groovy b/atlasdb-console/src/main/groovy/com/palantir/atlasdb/console/module/AtlasCoreModule.groovy index 58b603b3b5d..cda39f5a7c1 100644 --- a/atlasdb-console/src/main/groovy/com/palantir/atlasdb/console/module/AtlasCoreModule.groovy +++ b/atlasdb-console/src/main/groovy/com/palantir/atlasdb/console/module/AtlasCoreModule.groovy @@ -141,7 +141,11 @@ class AtlasCoreModule implements AtlasConsoleModule { } private setupConnection(AtlasDbConfig config) { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 SerializableTransactionManager tm = TransactionManagers.create(config, ImmutableSet.of(), +======= + SerializableTransactionManager tm = TransactionManagers.create(config, Optional.absent(), ImmutableSet.of(), +>>>>>>> merge develop into perf cli branch (#820) new com.palantir.atlasdb.factory.TransactionManagers.Environment() { @Override public void register(Object resource) { diff --git a/atlasdb-dbkvs/src/main/java/com/palantir/atlasdb/keyvalue/dbkvs/impl/AbstractDbReadTable.java b/atlasdb-dbkvs/src/main/java/com/palantir/atlasdb/keyvalue/dbkvs/impl/AbstractDbReadTable.java index df52f51032c..216ebd11e97 100644 --- a/atlasdb-dbkvs/src/main/java/com/palantir/atlasdb/keyvalue/dbkvs/impl/AbstractDbReadTable.java +++ b/atlasdb-dbkvs/src/main/java/com/palantir/atlasdb/keyvalue/dbkvs/impl/AbstractDbReadTable.java @@ -250,6 +250,7 @@ public ClosableIterator getOverflow(Collection>> futures = Queues.newArrayDeque(); for (FullQuery query : queries) { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 futures.add(getSupplierFuture(() -> run(query))); } return new LazyClosableIterator<>(futures); @@ -278,6 +279,11 @@ public T get(long timeout, TimeUnit unit) { return get(); } }; +======= + futures.add(submit(MoreExecutors.directExecutor(), query)); + } + return new LazyClosableIterator<>(futures); +>>>>>>> merge develop into perf cli branch (#820) } private boolean isSingleton(Iterable iterable) { diff --git a/atlasdb-dbkvs/src/main/java/com/palantir/atlasdb/keyvalue/dbkvs/impl/DbKvs.java b/atlasdb-dbkvs/src/main/java/com/palantir/atlasdb/keyvalue/dbkvs/impl/DbKvs.java index 2f7a9109a14..248bbe5a251 100644 --- a/atlasdb-dbkvs/src/main/java/com/palantir/atlasdb/keyvalue/dbkvs/impl/DbKvs.java +++ b/atlasdb-dbkvs/src/main/java/com/palantir/atlasdb/keyvalue/dbkvs/impl/DbKvs.java @@ -527,7 +527,11 @@ private TokenBackedBasicResultsPage>, byte[]> getTimestampsP try { return runRead(tableRef, table -> getTimestampsPageInternal(table, range, timestamp)); } finally { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 log.debug("Call to KVS.getTimestampsPage on table {} took {} ms.", +======= + log.info("Call to KVS.getTimestampsPage on table {} took {} ms.", +>>>>>>> merge develop into perf cli branch (#820) tableRef, watch.elapsed(TimeUnit.MILLISECONDS)); } } @@ -674,7 +678,11 @@ private Map>> getFirstRowsColumnRangePage( try { return runRead(tableRef, table -> extractRowColumnRangePage(table, columnRangeSelection, ts, rows)); } finally { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 log.debug("Call to KVS.getFirstRowColumnRangePage on table {} took {} ms.", +======= + log.info("Call to KVS.getFirstRowColumnRangePage on table {} took {} ms.", +>>>>>>> merge develop into perf cli branch (#820) tableRef, watch.elapsed(TimeUnit.MILLISECONDS)); } } @@ -1003,7 +1011,11 @@ private T runWriteForceAutocommit(TableReference tableRef, Function T runWriteFreshConnection( ConnectionSupplier conns, TableReference tableRef, Function runner) { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 log.debug("Running in a new thread to turn autocommit on for write"); +======= + log.info("Running in a new thread to turn autocommit on for write"); +>>>>>>> merge develop into perf cli branch (#820) AtomicReference result = Atomics.newReference(); Thread writeThread = new Thread(() -> { SqlConnection freshConn = conns.getFresh(); diff --git a/atlasdb-dbkvs/src/main/java/com/palantir/atlasdb/keyvalue/dbkvs/impl/ranges/DbKvsGetRanges.java b/atlasdb-dbkvs/src/main/java/com/palantir/atlasdb/keyvalue/dbkvs/impl/ranges/DbKvsGetRanges.java index 426987fcf08..9b5c7d4a47c 100644 --- a/atlasdb-dbkvs/src/main/java/com/palantir/atlasdb/keyvalue/dbkvs/impl/ranges/DbKvsGetRanges.java +++ b/atlasdb-dbkvs/src/main/java/com/palantir/atlasdb/keyvalue/dbkvs/impl/ranges/DbKvsGetRanges.java @@ -33,6 +33,7 @@ import com.google.common.collect.Lists; import com.google.common.collect.Maps; import com.google.common.collect.Ordering; +import com.google.common.collect.SortedSetMultimap; import com.google.common.collect.TreeMultimap; import com.google.common.primitives.UnsignedBytes; import com.palantir.atlasdb.keyvalue.api.Cell; @@ -65,24 +66,27 @@ public class DbKvsGetRanges { private static final OperationTimer logTimer = LoggingOperationTimer.create(log); private static final byte[] SMALLEST_NAME = Cells.createSmallestCellForRow(new byte[] {0}).getColumnName(); private static final byte[] LARGEST_NAME = Cells.createLargestCellForRow(new byte[] {0}).getColumnName(); + private final DbKvs kvs; private final DdlConfig config; private final DBType dbType; private final Supplier connectionSupplier; - public DbKvsGetRanges(DbKvs kvs, - DdlConfig config, - DBType dbType, - Supplier connectionSupplier) { + public DbKvsGetRanges( + DbKvs kvs, + DdlConfig config, + DBType dbType, + Supplier connectionSupplier) { this.kvs = kvs; this.config = config; this.dbType = dbType; this.connectionSupplier = connectionSupplier; } - public Map, byte[]>> getFirstBatchForRanges(final TableReference tableRef, - Iterable rangeRequests, - final long timestamp) { + public Map, byte[]>> getFirstBatchForRanges( + TableReference tableRef, + Iterable rangeRequests, + long timestamp) { Map, byte[]>> results = Maps.newHashMap(); for (List batch : Iterables.partition(rangeRequests, 500)) { results.putAll(getFirstPages(tableRef, batch, timestamp)); @@ -90,9 +94,10 @@ public Map, byte[]>> return results; } - private Map, byte[]>> getFirstPages(TableReference tableRef, - List requests, - long timestamp) { + private Map, byte[]>> getFirstPages( + TableReference tableRef, + List requests, + long timestamp) { List subQueries = Lists.newArrayList(); List argsList = Lists.newArrayList(); for (int i = 0; i < requests.size(); i++) { @@ -120,13 +125,14 @@ private Map, byte[]>> } } - private Map, byte[]>> getFirstPagesFromDb(TableReference tableRef, - List requests, - long timestamp, - final SqlConnection conn, - String query, - Object[] args) { - TreeMultimap rowsForBatches = getRowsForBatches(conn, query, args); + private Map, byte[]>> getFirstPagesFromDb( + TableReference tableRef, + List requests, + long timestamp, + final SqlConnection conn, + String query, + Object[] args) { + SortedSetMultimap rowsForBatches = getRowsForBatches(conn, query, args); Map cells = kvs.getRows(tableRef, rowsForBatches.values(), ColumnSelection.all(), timestamp); NavigableMap> cellsByRow = Cells.breakCellsUpByRow(cells); @@ -134,9 +140,12 @@ private Map, byte[]>> return breakUpByBatch(requests, rowsForBatches, cellsByRow); } - private static TreeMultimap getRowsForBatches(SqlConnection c, String query, Object[] args) { - AgnosticResultSet results = c.selectResultSetUnregisteredQuery(query, args); - TreeMultimap ret = TreeMultimap.create( + private static SortedSetMultimap getRowsForBatches( + SqlConnection connection, + String query, + Object[] args) { + AgnosticResultSet results = connection.selectResultSetUnregisteredQuery(query, args); + SortedSetMultimap ret = TreeMultimap.create( Ordering.natural(), UnsignedBytes.lexicographicalComparator()); for (AgnosticResultRow row : results.rows()) { @@ -150,20 +159,13 @@ private static TreeMultimap getRowsForBatches(SqlConnection c, return ret; } - private Pair> getRangeQueryAndArgs(String tableName, - byte[] startRow, - byte[] endRow, - boolean reverse, - int numRowsToGet, - int queryNum) { - if (startRow.length == 0) { - if (reverse) { - startRow = LARGEST_NAME; - } else { - startRow = SMALLEST_NAME; - } - } - + private Pair> getRangeQueryAndArgs( + String tableName, + byte[] startRow, + byte[] endRow, + boolean reverse, + int numRowsToGet, + int queryNum) { String extraWhere; List args = Lists.newArrayList(); args.add(queryNum); @@ -172,7 +174,11 @@ private Pair> getRangeQueryAndArgs(String tableName, } else { extraWhere = " t.row_name >= ? "; } - args.add(startRow); + if (startRow.length > 0) { + args.add(startRow); + } else { + args.add(reverse ? LARGEST_NAME : SMALLEST_NAME); + } if (endRow.length > 0) { if (reverse) { @@ -207,9 +213,10 @@ private Pair> getRangeQueryAndArgs(String tableName, * This tablehod expects the input to be sorted by rowname ASC for both rowsForBatches and * cellsByRow. */ - private Map, byte[]>> breakUpByBatch(List requests, - TreeMultimap rowsForBatches, - NavigableMap> cellsByRow) { + private Map, byte[]>> breakUpByBatch( + List requests, + SortedSetMultimap rowsForBatches, + NavigableMap> cellsByRow) { Map, byte[]>> ret = Maps.newHashMap(); for (int i = 0; i < requests.size(); i++) { RangeRequest request = requests.get(i); @@ -220,7 +227,13 @@ private Map, byte[]>> SortedMap> cellsForBatch = Maps.filterKeys( request.isReverse() ? cellsByRow.descendingMap() : cellsByRow, Predicates.in(rowNames)); - validateRowNames(cellsForBatch.keySet(), request.getStartInclusive(), request.getEndExclusive(), request.isReverse()); + + validateRowNames( + cellsForBatch.keySet(), + request.getStartInclusive(), + request.getEndExclusive(), + request.isReverse()); + IterableView> rows = RowResults.viewOfMap(cellsForBatch); if (!request.getColumnNames().isEmpty()) { rows = filterColumnSelection(rows, request); @@ -250,20 +263,25 @@ private void validateRowNames(Iterable rows, byte[] startInclusive, byte for (byte[] row : rows) { if (reverse) { AssertUtils.assertAndLog(startInclusive.length == 0 - || UnsignedBytes.lexicographicalComparator().compare(startInclusive, row) >= 0, "row was out of range"); + || UnsignedBytes.lexicographicalComparator().compare(startInclusive, row) >= 0, + "row was out of range"); AssertUtils.assertAndLog(endExclusive.length == 0 - || UnsignedBytes.lexicographicalComparator().compare(row, endExclusive) > 0, "row was out of range"); + || UnsignedBytes.lexicographicalComparator().compare(row, endExclusive) > 0, + "row was out of range"); } else { AssertUtils.assertAndLog(startInclusive.length == 0 - || UnsignedBytes.lexicographicalComparator().compare(startInclusive, row) <= 0, "row was out of range"); + || UnsignedBytes.lexicographicalComparator().compare(startInclusive, row) <= 0, + "row was out of range"); AssertUtils.assertAndLog(endExclusive.length == 0 - || UnsignedBytes.lexicographicalComparator().compare(row, endExclusive) < 0, "row was out of range"); + || UnsignedBytes.lexicographicalComparator().compare(row, endExclusive) < 0, + "row was out of range"); } } } - private IterableView> filterColumnSelection(IterableView> rows, - final RangeRequest request) { + private IterableView> filterColumnSelection( + IterableView> rows, + RangeRequest request) { return rows.transform(RowResults.createFilterColumns(new Predicate() { @Override public boolean apply(byte[] col) { @@ -283,7 +301,11 @@ private static void closeSql(SqlConnection conn) { } } - private String getSimpleRowSelectOneQueryPostgres(String tableName, String minMax, String extraWhere, String order) { + private String getSimpleRowSelectOneQueryPostgres( + String tableName, + String minMax, + String extraWhere, + String order) { return String.format( SIMPLE_ROW_SELECT_ONE_POSTGRES_TEMPLATE, tableName, @@ -293,10 +315,11 @@ private String getSimpleRowSelectOneQueryPostgres(String tableName, String minMa order); } - private String getSimpleRowSelectOneQueryOracle(String tableName, - String minMax, - String extraWhere, - String order) { + private String getSimpleRowSelectOneQueryOracle( + String tableName, + String minMax, + String extraWhere, + String order) { return String.format( SIMPLE_ROW_SELECT_ONE_ORACLE_TEMPLATE, tableName, @@ -311,25 +334,25 @@ private String prefixTableName(String tableName) { } private static final String SIMPLE_ROW_SELECT_TEMPLATE = - " /* SIMPLE_ROW_SELECT_TEMPLATE (%s) */ " + - " SELECT /*+ INDEX(t pk_%s) */ " + - " DISTINCT row_name, ? as batch_num " + - " FROM %s t " + - " WHERE %s " + - " ORDER BY row_name %s "; + " /* SIMPLE_ROW_SELECT_TEMPLATE (%s) */ " + + " SELECT /*+ INDEX(t pk_%s) */ " + + " DISTINCT row_name, ? as batch_num " + + " FROM %s t " + + " WHERE %s " + + " ORDER BY row_name %s "; private static final String SIMPLE_ROW_SELECT_ONE_POSTGRES_TEMPLATE = - " /* SIMPLE_ROW_SELECT_ONE_TEMPLATE_PSQL (%s) */ " + - " SELECT /*+ INDEX(t pk_%s) */ " + - " DISTINCT row_name, ? as batch_num " + - " FROM %s t " + - " WHERE %s " + - " ORDER BY row_name %s LIMIT 1"; + " /* SIMPLE_ROW_SELECT_ONE_TEMPLATE_PSQL (%s) */ " + + " SELECT /*+ INDEX(t pk_%s) */ " + + " DISTINCT row_name, ? as batch_num " + + " FROM %s t " + + " WHERE %s " + + " ORDER BY row_name %s LIMIT 1"; private static final String SIMPLE_ROW_SELECT_ONE_ORACLE_TEMPLATE = - " /* SIMPLE_ROW_SELECT_ONE_TEMPLATE_ORA (%s) */ " + - " SELECT /*+ INDEX(t pk_%s) */ " + - " %s(row_name) as row_name, ? as batch_num " + - " FROM %s t " + - " WHERE %s"; + " /* SIMPLE_ROW_SELECT_ONE_TEMPLATE_ORA (%s) */ " + + " SELECT /*+ INDEX(t pk_%s) */ " + + " %s(row_name) as row_name, ? as batch_num " + + " FROM %s t " + + " WHERE %s"; } diff --git a/atlasdb-dropwizard-bundle/src/main/java/com/palantir/atlasdb/dropwizard/commands/AtlasDbCommandUtils.java b/atlasdb-dropwizard-bundle/src/main/java/com/palantir/atlasdb/dropwizard/commands/AtlasDbCommandUtils.java index 65ac5ca2cd2..ea5042844a9 100644 --- a/atlasdb-dropwizard-bundle/src/main/java/com/palantir/atlasdb/dropwizard/commands/AtlasDbCommandUtils.java +++ b/atlasdb-dropwizard-bundle/src/main/java/com/palantir/atlasdb/dropwizard/commands/AtlasDbCommandUtils.java @@ -22,16 +22,37 @@ import java.util.stream.Stream; import com.fasterxml.jackson.core.JsonProcessingException; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 import com.google.common.base.Optional; import com.google.common.base.Preconditions; import com.palantir.atlasdb.config.AtlasDbConfig; import com.palantir.atlasdb.config.AtlasDbConfigs; +======= +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.dataformat.yaml.YAMLFactory; +import com.fasterxml.jackson.dataformat.yaml.YAMLGenerator; +import com.fasterxml.jackson.datatype.guava.GuavaModule; +import com.google.common.base.Optional; +import com.google.common.base.Preconditions; +import com.palantir.atlasdb.config.AtlasDbConfig; +>>>>>>> merge develop into perf cli branch (#820) import com.palantir.atlasdb.config.ImmutableAtlasDbConfig; import com.palantir.atlasdb.config.ImmutableServerListConfig; import com.palantir.atlasdb.config.ServerListConfig; public final class AtlasDbCommandUtils { public static final Object ZERO_ARITY_ARG_CONSTANT = ""; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 +======= + private static final ObjectMapper OBJECT_MAPPER; + + static { + YAMLFactory yamlFactory = new YAMLFactory(); + yamlFactory.configure(YAMLGenerator.Feature.USE_NATIVE_TYPE_ID, false); + OBJECT_MAPPER = new ObjectMapper(yamlFactory); + OBJECT_MAPPER.registerModule(new GuavaModule()); + } +>>>>>>> merge develop into perf cli branch (#820) private AtlasDbCommandUtils() { // Static utility class @@ -43,7 +64,10 @@ public static AtlasDbConfig convertServerConfigToClientConfig(AtlasDbConfig serv ServerListConfig leaders = ImmutableServerListConfig.builder() .servers(serverConfig.leader().get().leaders()) +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 .sslConfiguration(serverConfig.leader().get().sslConfiguration()) +======= +>>>>>>> merge develop into perf cli branch (#820) .build(); return ImmutableAtlasDbConfig.builder() @@ -71,6 +95,10 @@ public static List gatherPassedInArguments(Map allArgs) } public static String serialiseConfiguration(AtlasDbConfig cliConfiguration) throws JsonProcessingException { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 return AtlasDbConfigs.OBJECT_MAPPER.writeValueAsString(cliConfiguration); +======= + return OBJECT_MAPPER.writeValueAsString(cliConfiguration); +>>>>>>> merge develop into perf cli branch (#820) } } diff --git a/atlasdb-dropwizard-bundle/src/test/java/com/palantir/atlasdb/dropwizard/commands/AtlasDbCommandUtilsTest.java b/atlasdb-dropwizard-bundle/src/test/java/com/palantir/atlasdb/dropwizard/commands/AtlasDbCommandUtilsTest.java index 980241d1963..2ac03a022f2 100644 --- a/atlasdb-dropwizard-bundle/src/test/java/com/palantir/atlasdb/dropwizard/commands/AtlasDbCommandUtilsTest.java +++ b/atlasdb-dropwizard-bundle/src/test/java/com/palantir/atlasdb/dropwizard/commands/AtlasDbCommandUtilsTest.java @@ -18,9 +18,12 @@ import static org.assertj.core.api.Assertions.assertThat; import static org.mockito.Mockito.mock; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 import java.io.File; import java.io.IOException; import java.net.InetSocketAddress; +======= +>>>>>>> merge develop into perf cli branch (#820) import java.util.HashMap; import java.util.List; import java.util.Map; @@ -29,6 +32,7 @@ import com.google.common.collect.ImmutableList; import com.google.common.collect.ImmutableMap; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 import com.google.common.collect.ImmutableSet; import com.palantir.atlasdb.cassandra.ImmutableCassandraKeyValueServiceConfig; import com.palantir.atlasdb.config.AtlasDbConfig; @@ -37,6 +41,12 @@ import com.palantir.atlasdb.config.ImmutableLeaderConfig; import com.palantir.atlasdb.spi.KeyValueServiceConfig; import com.palantir.remoting.ssl.SslConfiguration; +======= +import com.palantir.atlasdb.config.AtlasDbConfig; +import com.palantir.atlasdb.config.ImmutableAtlasDbConfig; +import com.palantir.atlasdb.config.ImmutableLeaderConfig; +import com.palantir.atlasdb.spi.KeyValueServiceConfig; +>>>>>>> merge develop into perf cli branch (#820) public class AtlasDbCommandUtilsTest { private static final String LOCAL_SERVER_NAME = "Local Server"; @@ -126,6 +136,7 @@ public void argumentsWithTheZeroArityStringHaveOnlyTheKeyKept() { assertThat(gatheredArgs).containsExactly("--zero-arity-arg"); } +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 @Test public void canSerializeAndDeserializeAtlasDbConfig() throws IOException { @@ -161,4 +172,6 @@ public void canSerializeAndDeserializeAtlasDbConfig() throws IOException { assertThat(bigConfig).isEqualTo(deserializedConfig); } +======= +>>>>>>> merge develop into perf cli branch (#820) } diff --git a/atlasdb-ete-tests/build.gradle b/atlasdb-ete-tests/build.gradle index da3bc7cc113..9e59a10db91 100644 --- a/atlasdb-ete-tests/build.gradle +++ b/atlasdb-ete-tests/build.gradle @@ -24,6 +24,8 @@ dependencies { runtime project(':atlasdb-dbkvs') + runtime project(':atlasdb-dbkvs') + runtime(project(':atlasdb-cassandra')) { exclude(module:'log4j-over-slf4j') exclude(module:'jcl-over-slf4j') @@ -42,7 +44,11 @@ dependencies { compile 'org.apache.thrift:libthrift:' + libVersions.libthrift processor 'org.immutables:value:' + libVersions.immutables +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 processor('com.google.dagger:dagger-compiler:' + libVersions.dagger) { +======= + processor('com.google.dagger:dagger-compiler:2.0.2') { +>>>>>>> merge develop into perf cli branch (#820) // We need to explicitly exclude these so that intellij does not label them as provided if(gradle.startParameter.taskNames.contains('idea')) { exclude group: 'com.google.guava' diff --git a/atlasdb-ete-tests/src/main/java/com/palantir/atlasdb/AtlasDbEteServer.java b/atlasdb-ete-tests/src/main/java/com/palantir/atlasdb/AtlasDbEteServer.java index 22467e19d06..689e5569a70 100644 --- a/atlasdb-ete-tests/src/main/java/com/palantir/atlasdb/AtlasDbEteServer.java +++ b/atlasdb-ete-tests/src/main/java/com/palantir/atlasdb/AtlasDbEteServer.java @@ -17,6 +17,12 @@ import java.util.Set; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 +======= +import javax.net.ssl.SSLSocketFactory; + +import com.google.common.base.Optional; +>>>>>>> merge develop into perf cli branch (#820) import com.google.common.collect.ImmutableSet; import com.palantir.atlasdb.cas.CheckAndSetClient; import com.palantir.atlasdb.cas.CheckAndSetSchema; @@ -37,6 +43,10 @@ public class AtlasDbEteServer extends Application { private static final boolean DONT_SHOW_HIDDEN_TABLES = false; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 +======= + private static final Optional NO_SSL = Optional.absent(); +>>>>>>> merge develop into perf cli branch (#820) private static final Set ETE_SCHEMAS = ImmutableSet.of( CheckAndSetSchema.getSchema(), TodoSchema.getSchema()); @@ -53,7 +63,11 @@ public void initialize(Bootstrap bootstrap) { @Override public void run(AtlasDbEteConfiguration config, final Environment environment) throws Exception { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 TransactionManager transactionManager = TransactionManagers.create(config.getAtlasDbConfig(), ETE_SCHEMAS, environment.jersey()::register, DONT_SHOW_HIDDEN_TABLES); +======= + TransactionManager transactionManager = TransactionManagers.create(config.getAtlasDbConfig(), NO_SSL, ETE_SCHEMAS, environment.jersey()::register, DONT_SHOW_HIDDEN_TABLES); +>>>>>>> merge develop into perf cli branch (#820) environment.jersey().register(new SimpleTodoResource(new TodoClient(transactionManager))); environment.jersey().register(new SimpleCheckAndSetResource(new CheckAndSetClient(transactionManager))); } diff --git a/atlasdb-ete-tests/src/test/java/com/palantir/atlasdb/ete/dropwizard/CassandraDropwizardEteTest.java b/atlasdb-ete-tests/src/test/java/com/palantir/atlasdb/ete/dropwizard/CassandraDropwizardEteTest.java index ae852a72f51..b0d55587c82 100644 --- a/atlasdb-ete-tests/src/test/java/com/palantir/atlasdb/ete/dropwizard/CassandraDropwizardEteTest.java +++ b/atlasdb-ete-tests/src/test/java/com/palantir/atlasdb/ete/dropwizard/CassandraDropwizardEteTest.java @@ -22,5 +22,9 @@ public class CassandraDropwizardEteTest extends DropwizardEteTest { @ClassRule +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 public static final RuleChain COMPOSITION_SETUP = EteSetup.setupComposition("cassandra-ha-dropwizard", "docker-compose.cassandra.yml"); +======= + public static final RuleChain COMPOSITION_SETUP = EteSetup.setupComposition("cassandra-ha", "docker-compose.cassandra.yml"); +>>>>>>> merge develop into perf cli branch (#820) } diff --git a/atlasdb-ete-tests/src/test/java/com/palantir/atlasdb/ete/dropwizard/CassandraNoLeaderDropwizardEteTest.java b/atlasdb-ete-tests/src/test/java/com/palantir/atlasdb/ete/dropwizard/CassandraNoLeaderDropwizardEteTest.java index 0d4afa4934a..8de31a7a14d 100644 --- a/atlasdb-ete-tests/src/test/java/com/palantir/atlasdb/ete/dropwizard/CassandraNoLeaderDropwizardEteTest.java +++ b/atlasdb-ete-tests/src/test/java/com/palantir/atlasdb/ete/dropwizard/CassandraNoLeaderDropwizardEteTest.java @@ -22,5 +22,9 @@ public class CassandraNoLeaderDropwizardEteTest extends DropwizardEteTest { @ClassRule +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 public static final RuleChain COMPOSITION_SETUP = EteSetup.setupComposition("cassandra-no-leader-dropwizard", "docker-compose.no-leader.cassandra.yml"); +======= + public static final RuleChain COMPOSITION_SETUP = EteSetup.setupComposition("cassandra-no-leader", "docker-compose.no-leader.cassandra.yml"); +>>>>>>> merge develop into perf cli branch (#820) } diff --git a/atlasdb-ete-tests/src/test/java/com/palantir/atlasdb/ete/dropwizard/DbKvsDropwizardEteTest.java b/atlasdb-ete-tests/src/test/java/com/palantir/atlasdb/ete/dropwizard/DbKvsDropwizardEteTest.java index 976b544cb48..ce252393d12 100644 --- a/atlasdb-ete-tests/src/test/java/com/palantir/atlasdb/ete/dropwizard/DbKvsDropwizardEteTest.java +++ b/atlasdb-ete-tests/src/test/java/com/palantir/atlasdb/ete/dropwizard/DbKvsDropwizardEteTest.java @@ -22,5 +22,9 @@ public class DbKvsDropwizardEteTest extends DropwizardEteTest { @ClassRule +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 public static final RuleChain COMPOSITION_SETUP = EteSetup.setupComposition("dbkvs-dropwizard", "docker-compose.dbkvs.yml"); +======= + public static final RuleChain COMPOSITION_SETUP = EteSetup.setupComposition("dbkvs", "docker-compose.dbkvs.yml"); +>>>>>>> merge develop into perf cli branch (#820) } diff --git a/atlasdb-impl-shared/src/main/java/com/palantir/atlasdb/transaction/impl/SnapshotTransaction.java b/atlasdb-impl-shared/src/main/java/com/palantir/atlasdb/transaction/impl/SnapshotTransaction.java index e4452b85b6e..ca08fd96c5a 100644 --- a/atlasdb-impl-shared/src/main/java/com/palantir/atlasdb/transaction/impl/SnapshotTransaction.java +++ b/atlasdb-impl-shared/src/main/java/com/palantir/atlasdb/transaction/impl/SnapshotTransaction.java @@ -1472,7 +1472,11 @@ private boolean rollbackFailedTransactions( } try { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 log.debug("For table: " + tableRef +======= + log.warn("For table: " + tableRef +>>>>>>> merge develop into perf cli branch (#820) + " we are deleting values of an uncommitted transaction: " + keysToDelete); keyValueService.delete(tableRef, Multimaps.forMap(keysToDelete)); } catch (RuntimeException e) { diff --git a/atlasdb-perf/build.gradle b/atlasdb-perf/build.gradle index 08144bb9ebf..cad89c86957 100644 --- a/atlasdb-perf/build.gradle +++ b/atlasdb-perf/build.gradle @@ -7,7 +7,7 @@ mainClassName = 'com.palantir.atlasdb.performance.cli.AtlasDbPerfCli' applicationName = 'atlasdb-perf' dependencies { - compile project(':atlasdb-spi') + compile project(':atlasdb-api') compile project(':atlasdb-dbkvs') compile project(':atlasdb-cassandra') @@ -28,6 +28,7 @@ dependencies { configurations.matching({ it.name in ['compile', 'runtime'] }).all { resolutionStrategy { + force 'javax.validation:validation-api:1.1.0.Final' force 'com.google.code.findbugs:annotations:2.0.3' force 'org.slf4j:slf4j-api:' + libVersions.slf4j force 'org.apache.commons:commons-lang3:' + libVersions.commons_lang3 diff --git a/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/BenchmarkParam.java b/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/BenchmarkParam.java index f65ef0c5e8a..35cdd07fd32 100644 --- a/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/BenchmarkParam.java +++ b/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/BenchmarkParam.java @@ -1,3 +1,20 @@ +/** + * Copyright 2016 Palantir Technologies + * + * Licensed under the BSD-3 License (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://opensource.org/licenses/BSD-3-Clause + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package com.palantir.atlasdb.performance; /** diff --git a/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/PerformanceResults.java b/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/PerformanceResults.java index a998b32c5b3..73059fdd79b 100644 --- a/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/PerformanceResults.java +++ b/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/PerformanceResults.java @@ -1,10 +1,30 @@ +/** + * Copyright 2016 Palantir Technologies + * + * Licensed under the BSD-3 License (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://opensource.org/licenses/BSD-3-Clause + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package com.palantir.atlasdb.performance; import java.io.BufferedWriter; import java.io.File; -import java.io.FileWriter; +import java.io.FileNotFoundException; +import java.io.FileOutputStream; import java.io.IOException; +import java.io.OutputStreamWriter; import java.lang.reflect.Field; +import java.nio.charset.StandardCharsets; import java.util.Collection; import java.util.List; import java.util.concurrent.TimeUnit; @@ -30,31 +50,35 @@ public PerformanceResults(Collection results) { } public void writeToFile(File file) throws IOException { - try (BufferedWriter fout = new BufferedWriter(new FileWriter(file))) { + try (BufferedWriter fout = openFileWriter(file)) { long date = System.currentTimeMillis(); - List newResults = results.stream().map(r -> { - String[] benchmarkParts = r.getParams().getBenchmark().split("\\."); + List newResults = results.stream().map(rs -> { + String[] benchmarkParts = rs.getParams().getBenchmark().split("\\."); String benchmarkSuite = benchmarkParts[benchmarkParts.length - 2]; String benchmarkName = benchmarkParts[benchmarkParts.length - 1]; return ImmutablePerformanceResult.builder() .date(date) .suite(benchmarkSuite) .benchmark(benchmarkName) - .backend(r.getParams().getParam((BenchmarkParam.BACKEND.getKey()))) - .samples(r.getPrimaryResult().getStatistics().getN()) - .std(r.getPrimaryResult().getStatistics().getStandardDeviation()) - .mean(r.getPrimaryResult().getStatistics().getMean()) - .data(getData(r)) - .units(r.getParams().getTimeUnit()) - .p50(r.getPrimaryResult().getStatistics().getPercentile(50.0)) - .p90(r.getPrimaryResult().getStatistics().getPercentile(90.0)) - .p99(r.getPrimaryResult().getStatistics().getPercentile(99.0)) + .backend(rs.getParams().getParam(BenchmarkParam.BACKEND.getKey())) + .samples(rs.getPrimaryResult().getStatistics().getN()) + .std(rs.getPrimaryResult().getStatistics().getStandardDeviation()) + .mean(rs.getPrimaryResult().getStatistics().getMean()) + .data(getData(rs)) + .units(rs.getParams().getTimeUnit()) + .p50(rs.getPrimaryResult().getStatistics().getPercentile(50.0)) + .p90(rs.getPrimaryResult().getStatistics().getPercentile(90.0)) + .p99(rs.getPrimaryResult().getStatistics().getPercentile(99.0)) .build(); }).collect(Collectors.toList()); new ObjectMapper().writeValue(fout, newResults); } } + private BufferedWriter openFileWriter(File file) throws FileNotFoundException { + return new BufferedWriter(new OutputStreamWriter(new FileOutputStream(file), StandardCharsets.UTF_8)); + } + private List getData(RunResult result) { return result.getBenchmarkResults().stream() .flatMap(b -> getRawResults(b.getPrimaryResult().getStatistics()).stream()) @@ -63,9 +87,9 @@ private List getData(RunResult result) { private List getRawResults(Statistics statistics) { try { - Field f = statistics.getClass().getDeclaredField("values"); - f.setAccessible(true); - Multiset rawResults = (Multiset) f.get(statistics); + Field field = statistics.getClass().getDeclaredField("values"); + field.setAccessible(true); + Multiset rawResults = (Multiset) field.get(statistics); return rawResults.entrySet().stream() .flatMap(e -> DoubleStream.iterate(e.getKey(), d -> d).limit(e.getValue()).boxed()) .collect(Collectors.toList()); @@ -77,7 +101,7 @@ private List getRawResults(Statistics statistics) { @JsonDeserialize(as = ImmutablePerformanceResult.class) @JsonSerialize(as = ImmutablePerformanceResult.class) @Value.Immutable - static abstract class PerformanceResult { + abstract static class PerformanceResult { public abstract long date(); public abstract String suite(); public abstract String benchmark(); diff --git a/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/backend/CassandraPhysicalStore.java b/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/backend/CassandraPhysicalStore.java index f96ed39d080..45d5bf8ba39 100644 --- a/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/backend/CassandraPhysicalStore.java +++ b/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/backend/CassandraPhysicalStore.java @@ -23,6 +23,7 @@ import org.joda.time.Duration; +import com.google.common.base.Optional; import com.jayway.awaitility.Awaitility; import com.palantir.atlasdb.cassandra.CassandraKeyValueServiceConfig; import com.palantir.atlasdb.cassandra.CassandraKeyValueServiceConfigManager; @@ -40,7 +41,7 @@ * * @author mwakerman */ -public class CassandraPhysicalStore extends PhysicalStore { +public final class CassandraPhysicalStore extends PhysicalStore { private static final String KEYSPACE = "atlasdb"; private static final int THRIFT_PORT_NUMBER = 9160; @@ -50,19 +51,21 @@ public class CassandraPhysicalStore extends PhysicalStore { private static final String DOCKER_LOGS_DIR = "container-logs"; public static CassandraPhysicalStore create() { - DockerComposeRule docker = DockerComposeRule.builder() - .file(getDockerComposeFileAbsolutePath()) - .waitingForHostNetworkedPort(THRIFT_PORT_NUMBER, toBeOpen(), Duration.standardMinutes(1)) - .saveLogsTo(DOCKER_LOGS_DIR) - .build(); + DockerComposeRule docker = DockerComposeRule.builder() + .file(getDockerComposeFileAbsolutePath()) + .waitingForHostNetworkedPort(THRIFT_PORT_NUMBER, toBeOpen(), Duration.standardMinutes(1)) + .saveLogsTo(DOCKER_LOGS_DIR) + .build(); - return new CassandraPhysicalStore(docker); + return new CassandraPhysicalStore(docker); } private static String getDockerComposeFileAbsolutePath() { try { - return PhysicalStores.writeResourceToTempFile(CassandraPhysicalStore.class, CASSANDRA_DOCKER_COMPOSE_PATH).getAbsolutePath(); + return PhysicalStores + .writeResourceToTempFile(CassandraPhysicalStore.class, CASSANDRA_DOCKER_COMPOSE_PATH) + .getAbsolutePath(); } catch (IOException e) { throw new RuntimeException("Unable to write docker compose file to a temporary file.", e); } @@ -87,9 +90,7 @@ public KeyValueService connect() { docker.before(); } } catch (IOException | InterruptedException | IllegalStateException e) { - System.err.println("Could not run docker compose rule for cassandra."); - e.printStackTrace(); - return null; + throw new RuntimeException("Could not run docker compose rule for cassandra.", e); } ImmutableCassandraKeyValueServiceConfig connectionConfig = ImmutableCassandraKeyValueServiceConfig.builder() @@ -114,13 +115,17 @@ public KeyValueService connect() { .pollInterval(com.jayway.awaitility.Duration.FIVE_SECONDS) .until(canCreateKeyValueService(connectionConfig)); - return CassandraKeyValueService.create(CassandraKeyValueServiceConfigManager.createSimpleManager(connectionConfig)); + return CassandraKeyValueService.create( + CassandraKeyValueServiceConfigManager.createSimpleManager(connectionConfig), + Optional.absent()); } private static Callable canCreateKeyValueService(CassandraKeyValueServiceConfig config) { return () -> { try { - CassandraKeyValueService.create(CassandraKeyValueServiceConfigManager.createSimpleManager(config)); + CassandraKeyValueService.create( + CassandraKeyValueServiceConfigManager.createSimpleManager(config), + Optional.absent()); return true; } catch (Exception e) { return false; diff --git a/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/backend/KeyValueServiceConnector.java b/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/backend/KeyValueServiceConnector.java index 3d950528396..c2ecc007579 100644 --- a/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/backend/KeyValueServiceConnector.java +++ b/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/backend/KeyValueServiceConnector.java @@ -1,3 +1,20 @@ +/** + * Copyright 2016 Palantir Technologies + * + * Licensed under the BSD-3 License (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://opensource.org/licenses/BSD-3-Clause + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package com.palantir.atlasdb.performance.backend; import org.openjdk.jmh.annotations.Param; @@ -10,7 +27,7 @@ public class KeyValueServiceConnector extends PhysicalStore { /** - * Edit this instance variable name ("backend") with care -- it must match {@code BenchmarkParam.BACKEND}.getKey() + * Edit this instance variable name ("backend") with care -- it must match {@code BenchmarkParam.BACKEND}.getKey(). */ @Param private KeyValueServiceType backend; diff --git a/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/backend/PhysicalStore.java b/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/backend/PhysicalStore.java index 40b798f48be..7d8b77a93ec 100644 --- a/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/backend/PhysicalStore.java +++ b/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/backend/PhysicalStore.java @@ -20,8 +20,8 @@ import com.palantir.atlasdb.keyvalue.api.KeyValueService; /** - * The physical store backing the AtlasDB key-value service. Any new physical store should extend this class (currently only Postgres is - * implemented). + * The physical store backing the AtlasDB key-value service. Any new physical store should extend this class + * (currently only Postgres is implemented). */ public abstract class PhysicalStore implements AutoCloseable { diff --git a/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/backend/PostgresPhysicalStore.java b/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/backend/PostgresPhysicalStore.java index 12afe6d3d27..fb50f507260 100644 --- a/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/backend/PostgresPhysicalStore.java +++ b/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/backend/PostgresPhysicalStore.java @@ -27,6 +27,7 @@ import com.palantir.docker.compose.connection.DockerPort; import com.palantir.docker.compose.connection.waiting.HealthCheck; import com.palantir.docker.compose.connection.waiting.SuccessOrFailure; +import com.palantir.nexus.db.pool.config.ImmutableMaskedValue; import com.palantir.nexus.db.pool.config.ImmutablePostgresConnectionConfig; /** @@ -34,7 +35,7 @@ * * @author mwakerman */ -public class PostgresPhysicalStore extends PhysicalStore { +public final class PostgresPhysicalStore extends PhysicalStore { private static final String POSTGRES_DB_NAME = "atlas"; private static final int POSTGRES_PORT_NUMBER = 5432; @@ -44,17 +45,19 @@ public class PostgresPhysicalStore extends PhysicalStore { private static final String POSTGRES_DOCKER_LOGS_DIR = "container-logs"; public static PostgresPhysicalStore create() { - DockerComposeRule docker = DockerComposeRule.builder() - .file(getDockerComposeFileAbsolutePath()) - .waitingForHostNetworkedPort(POSTGRES_PORT_NUMBER, toBeOpen()) - .saveLogsTo(POSTGRES_DOCKER_LOGS_DIR) - .build(); - return new PostgresPhysicalStore(docker); + DockerComposeRule docker = DockerComposeRule.builder() + .file(getDockerComposeFileAbsolutePath()) + .waitingForHostNetworkedPort(POSTGRES_PORT_NUMBER, toBeOpen()) + .saveLogsTo(POSTGRES_DOCKER_LOGS_DIR) + .build(); + return new PostgresPhysicalStore(docker); } private static String getDockerComposeFileAbsolutePath() { try { - return PhysicalStores.writeResourceToTempFile(PostgresPhysicalStore.class, POSTGRES_DOCKER_COMPOSE_PATH).getAbsolutePath(); + return PhysicalStores + .writeResourceToTempFile(PostgresPhysicalStore.class, POSTGRES_DOCKER_COMPOSE_PATH) + .getAbsolutePath(); } catch (IOException e) { throw new RuntimeException("Unable to write docker compose file to a temporary file.", e); } @@ -79,15 +82,13 @@ public KeyValueService connect() { docker.before(); } } catch (IOException | InterruptedException | IllegalStateException e) { - System.err.println("Could not run docker compose rule for postgres."); - e.printStackTrace(); - return null; + throw new RuntimeException("Could not run docker compose rule for postgres.", e); } ImmutablePostgresConnectionConfig connectionConfig = ImmutablePostgresConnectionConfig.builder() .dbName(POSTGRES_DB_NAME) .dbLogin(POSTGRES_USER_LOGIN) - .dbPassword(POSTGRES_USER_PASSWORD) + .dbPassword(ImmutableMaskedValue.of(POSTGRES_USER_PASSWORD)) .host(docker.containers().ip()) .port(POSTGRES_PORT_NUMBER) .build(); diff --git a/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/benchmarks/KvsBenchmarks.java b/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/benchmarks/KvsBenchmarks.java index 7ee9d916e01..baaf0b96687 100644 --- a/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/benchmarks/KvsBenchmarks.java +++ b/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/benchmarks/KvsBenchmarks.java @@ -17,7 +17,7 @@ package com.palantir.atlasdb.performance.benchmarks; -import org.apache.commons.lang.Validate; +import org.apache.commons.lang3.Validate; import com.palantir.atlasdb.keyvalue.api.KeyValueService; import com.palantir.atlasdb.keyvalue.api.TableReference; @@ -50,34 +50,45 @@ public static void validate(boolean assertion, String formatString, Object... ar * @param tableName the name of the table being created. * @param rowComponent the name of the row being created. * @param columnName the name of the column being created. - * @return a reference to the newly created table. + * @return a reference to the newly created table */ - public static TableReference createTable(KeyValueService kvs, String tableName, String rowComponent, String columnName) { + public static TableReference createTable(KeyValueService kvs, + String tableName, + String rowComponent, + String columnName) { TableReference tableRef = TableReference.createFromFullyQualifiedName(tableName); - TableDefinition tableDef = new TableDefinition() {{ - rowName(); - rowComponent(rowComponent, ValueType.STRING); - columns(); - column(columnName, columnName, ValueType.BLOB); - conflictHandler(ConflictHandler.IGNORE_ALL); - sweepStrategy(TableMetadataPersistence.SweepStrategy.NOTHING); - }}; + TableDefinition tableDef = new TableDefinition() { + { + rowName(); + rowComponent(rowComponent, ValueType.STRING); + columns(); + column(columnName, columnName, ValueType.BLOB); + conflictHandler(ConflictHandler.IGNORE_ALL); + sweepStrategy(TableMetadataPersistence.SweepStrategy.NOTHING); + } + }; kvs.createTable(tableRef, tableDef.toTableMetadata().persistToBytes()); return tableRef; } - public static TableReference createTableWithDynamicColumns(KeyValueService kvs, String tableName, String rowComponent, String columnComponent) { + public static TableReference createTableWithDynamicColumns(KeyValueService kvs, + String tableName, + String rowComponent, + String columnComponent) { TableReference tableRef = TableReference.createFromFullyQualifiedName(tableName); - TableDefinition tableDef = new TableDefinition() {{ - rowName(); - rowComponent(rowComponent, ValueType.STRING); - dynamicColumns(); - columnComponent(columnComponent, ValueType.FIXED_LONG); - value(ValueType.FIXED_LONG); - conflictHandler(ConflictHandler.IGNORE_ALL); - sweepStrategy(TableMetadataPersistence.SweepStrategy.NOTHING); - }}; + TableDefinition tableDef = new TableDefinition() { + { + rowName(); + rowComponent(rowComponent, ValueType.STRING); + dynamicColumns(); + columnComponent(columnComponent, ValueType.FIXED_LONG); + value(ValueType.FIXED_LONG); + conflictHandler(ConflictHandler.IGNORE_ALL); + sweepStrategy(TableMetadataPersistence.SweepStrategy.NOTHING); + } + }; kvs.createTable(tableRef, tableDef.toTableMetadata().persistToBytes()); return tableRef; } + } diff --git a/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/benchmarks/KvsGetDynamicBenchmarks.java b/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/benchmarks/KvsGetDynamicBenchmarks.java index b8993b945fe..ae8fe8da851 100644 --- a/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/benchmarks/KvsGetDynamicBenchmarks.java +++ b/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/benchmarks/KvsGetDynamicBenchmarks.java @@ -1,7 +1,24 @@ +/** + * Copyright 2016 Palantir Technologies + * + * Licensed under the BSD-3 License (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://opensource.org/licenses/BSD-3-Clause + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package com.palantir.atlasdb.performance.benchmarks; import java.io.UnsupportedEncodingException; -import java.util.Collection; +import java.nio.charset.StandardCharsets; import java.util.Collections; import java.util.Map; import java.util.concurrent.TimeUnit; @@ -49,7 +66,7 @@ public class KvsGetDynamicBenchmarks { private static final long DUMMY_TIMESTAMP = 1L; private static final long READ_TIMESTAMP = 2L; - private final int NUM_COLS = 50000; + private static final int NUM_COLS = 50000; private KeyValueServiceConnector connector; private KeyValueService kvs; @@ -60,18 +77,19 @@ public class KvsGetDynamicBenchmarks { private Map firstCell2ReadTimestamp; @Setup - public void setup(KeyValueServiceConnector connector) throws UnsupportedEncodingException { - this.connector = connector; - kvs = connector.connect(); + public void setup(KeyValueServiceConnector conn) throws UnsupportedEncodingException { + this.connector = conn; + kvs = conn.connect(); tableRef1 = KvsBenchmarks.createTableWithDynamicColumns(kvs, TABLE_NAME_1, ROW_COMPONENT, COLUMN_COMPONENT); - byte[] rowBytes = ROW_COMPONENT.getBytes("UTF-8"); - Map values = Maps.newHashMap(); + byte[] rowBytes = ROW_COMPONENT.getBytes(StandardCharsets.UTF_8); + Map values = Maps.newHashMap(); allCells2ReadTimestamp = Maps.newHashMap(); - firstCell2ReadTimestamp = ImmutableMap.of(Cell.create(rowBytes, ("col_0").getBytes("UTF-8")), READ_TIMESTAMP); + firstCell2ReadTimestamp = ImmutableMap.of( + Cell.create(rowBytes, "col_0".getBytes(StandardCharsets.UTF_8)), READ_TIMESTAMP); for (int i = 0; i < NUM_COLS; i++) { - Cell c = Cell.create(rowBytes, ("col_"+i).getBytes("UTF-8")); - values.put(c, Ints.toByteArray(i)); - allCells2ReadTimestamp.put(c, READ_TIMESTAMP); + Cell cell = Cell.create(rowBytes, ("col_" + i).getBytes(StandardCharsets.UTF_8)); + values.put(cell, Ints.toByteArray(i)); + allCells2ReadTimestamp.put(cell, READ_TIMESTAMP); } kvs.put(tableRef1, values, DUMMY_TIMESTAMP); } @@ -86,14 +104,20 @@ public void cleanup() throws Exception { @Benchmark public Map getAllColumnsExplicitly() { Map result = kvs.get(tableRef1, allCells2ReadTimestamp); - KvsBenchmarks.validate(result.size() == NUM_COLS, "Should be %s columns, but were: %s", NUM_COLS, result.size()); + KvsBenchmarks.validate(result.size() == NUM_COLS, + "Should be %s columns, but were: %s", NUM_COLS, result.size()); return result; } @Benchmark public Map getAllColumnsImplicitly() throws UnsupportedEncodingException { - Map result = kvs.getRows(tableRef1, Collections.singleton(ROW_COMPONENT.getBytes("UTF-8")), ColumnSelection.all(), READ_TIMESTAMP); - KvsBenchmarks.validate(result.size() == NUM_COLS, "Should be %s columns, but were: %s", NUM_COLS, result.size()); + Map result = kvs.getRows( + tableRef1, + Collections.singleton(ROW_COMPONENT.getBytes("UTF-8")), + ColumnSelection.all(), + READ_TIMESTAMP); + KvsBenchmarks.validate(result.size() == NUM_COLS, + "Should be %s columns, but were: %s", NUM_COLS, result.size()); return result; } diff --git a/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/benchmarks/KvsGetRangeBenchmarks.java b/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/benchmarks/KvsGetRangeBenchmarks.java index 652c830af00..7776c14663b 100644 --- a/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/benchmarks/KvsGetRangeBenchmarks.java +++ b/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/benchmarks/KvsGetRangeBenchmarks.java @@ -1,5 +1,23 @@ +/** + * Copyright 2016 Palantir Technologies + * + * Licensed under the BSD-3 License (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://opensource.org/licenses/BSD-3-Clause + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package com.palantir.atlasdb.performance.benchmarks; +import java.nio.charset.StandardCharsets; import java.util.ArrayList; import java.util.Arrays; import java.util.List; @@ -8,7 +26,7 @@ import java.util.Set; import java.util.concurrent.TimeUnit; -import org.apache.commons.lang.Validate; +import org.apache.commons.lang3.Validate; import org.openjdk.jmh.annotations.Benchmark; import org.openjdk.jmh.annotations.BenchmarkMode; import org.openjdk.jmh.annotations.Level; @@ -46,7 +64,7 @@ public class KvsGetRangeBenchmarks { private static final String TABLE_NAME_1 = "performance.table1"; private static final String ROW_COMPONENT = "key"; private static final String COLUMN_NAME = "value"; - private static final byte [] COLUMN_NAME_IN_BYTES = COLUMN_NAME.getBytes(); + private static final byte [] COLUMN_NAME_IN_BYTES = COLUMN_NAME.getBytes(StandardCharsets.UTF_8); private static final long DUMMY_TIMESTAMP = 1L; private static final long QUERY_TIMESTAMP = 2L; @@ -64,9 +82,9 @@ public class KvsGetRangeBenchmarks { private static final int NUM_REQUESTS = 1000; @Setup(Level.Trial) - public void setup(KeyValueServiceConnector connector) { - this.connector = connector; - this.kvs = connector.connect(); + public void setup(KeyValueServiceConnector conn) { + this.connector = conn; + this.kvs = conn.connect(); this.tableRef1 = KvsBenchmarks.createTable(kvs, TABLE_NAME_1, ROW_COMPONENT, COLUMN_NAME); storeData(); } @@ -74,8 +92,6 @@ public void setup(KeyValueServiceConnector connector) { private void storeData() { Validate.isTrue(NUM_ROWS % PUT_BATCH_SIZE == 0); - - for (int i = 0; i < NUM_ROWS; i += PUT_BATCH_SIZE) { Map> multiPutMap = Maps.newHashMap(); multiPutMap.put(tableRef1, generateBatch(i, PUT_BATCH_SIZE)); @@ -153,7 +169,8 @@ public void getMultiRange() { int numRequests = Iterables.size(requests); KvsBenchmarks.validate(numRequests == results.size(), - "Got %s requests and %s results, requests %s, results %s", numRequests, results.size(), requests, results); + "Got %s requests and %s results, requests %s, results %s", + numRequests, results.size(), requests, results); results.forEach((request, result) -> { KvsBenchmarks.validate(1 == result.getResults().size(), "Key %s, List size is %s", diff --git a/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/benchmarks/KvsPutBenchmarks.java b/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/benchmarks/KvsPutBenchmarks.java index e563ce7b216..1d9c4efa5b5 100644 --- a/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/benchmarks/KvsPutBenchmarks.java +++ b/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/benchmarks/KvsPutBenchmarks.java @@ -17,6 +17,7 @@ package com.palantir.atlasdb.performance.benchmarks; +import java.nio.charset.StandardCharsets; import java.util.Map; import java.util.Random; import java.util.concurrent.TimeUnit; @@ -56,7 +57,7 @@ public class KvsPutBenchmarks { private static final String TABLE_NAME_2 = "performance.table2"; private static final String ROW_COMPONENT = "key"; private static final String COLUMN_NAME = "value"; - private static final byte [] COLUMN_NAME_IN_BYTES = COLUMN_NAME.getBytes(); + private static final byte [] COLUMN_NAME_IN_BYTES = COLUMN_NAME.getBytes(StandardCharsets.UTF_8); private static final long DUMMY_TIMESTAMP = 1L; private static final int VALUE_BYTE_ARRAY_SIZE = 100; @@ -72,9 +73,9 @@ public class KvsPutBenchmarks { private TableReference tableRef2; @Setup - public void setup(KeyValueServiceConnector connector) { - this.connector = connector; - this.kvs = connector.connect(); + public void setup(KeyValueServiceConnector conn) { + this.connector = conn; + this.kvs = conn.connect(); this.tableRef1 = KvsBenchmarks.createTable(kvs, TABLE_NAME_1, ROW_COMPONENT, COLUMN_NAME); this.tableRef2 = KvsBenchmarks.createTable(kvs, TABLE_NAME_2, ROW_COMPONENT, COLUMN_NAME); } diff --git a/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/cli/AtlasDbPerfCli.java b/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/cli/AtlasDbPerfCli.java index eabbac579b6..c03d71ec852 100644 --- a/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/cli/AtlasDbPerfCli.java +++ b/atlasdb-perf/src/main/java/com/palantir/atlasdb/performance/cli/AtlasDbPerfCli.java @@ -45,11 +45,7 @@ /** * The Atlas Perf(ormance) CLI is a tool for making and running AtlasDB performance tests. * - * - * Note, you must have a docker machine running, and put the environment variables in your run configuration (if you are in an IDE). - * Something like this: - * DOCKER_TLS_VERIFY=1;DOCKER_HOST=tcp://192.168.99.100:2376;DOCKER_CERT_PATH=/Users/dcohen/.docker/machine/machines/atlas;DOCKER_MACHINE_NAME=atlas - * Check these against your actual env, e.g. "docker-machine env atlas" + * This requires you to have a docker-machine running and configured correctly. * * @author mwakerman, bullman */ @@ -68,8 +64,9 @@ public class AtlasDbPerfCli { @Option(name = {"-l", "--list-tests"}, description = "Lists all available benchmarks.") private boolean listTests; - @Option(name = {"-o", "--output"}, description = "The file in which to store the test results. Leave blank to only write results to " + - "the console.") + @Option(name = {"-o", "--output"}, + description = "The file in which to store the test results. " + + "Leave blank to only write results to the console.") private String outputFile; public static void main(String[] args) throws Exception { @@ -125,7 +122,9 @@ private static void listAllBenchmarks() { } private static Set getAllBenchmarks() { - Reflections reflections = new Reflections("com.palantir.atlasdb.performance.benchmarks", new MethodAnnotationsScanner()); + Reflections reflections = new Reflections( + "com.palantir.atlasdb.performance.benchmarks", + new MethodAnnotationsScanner()); return reflections.getMethodsAnnotatedWith(Benchmark.class).stream() .map(method -> method.getDeclaringClass().getSimpleName() + "." + method.getName()) .collect(Collectors.toSet()); diff --git a/atlasdb-service-server/src/main/java/com/palantir/atlasdb/server/AtlasDbServiceServer.java b/atlasdb-service-server/src/main/java/com/palantir/atlasdb/server/AtlasDbServiceServer.java index d2f68500fc0..05280210329 100644 --- a/atlasdb-service-server/src/main/java/com/palantir/atlasdb/server/AtlasDbServiceServer.java +++ b/atlasdb-service-server/src/main/java/com/palantir/atlasdb/server/AtlasDbServiceServer.java @@ -15,6 +15,10 @@ */ package com.palantir.atlasdb.server; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56:atlasdb-service-server/src/main/java/com/palantir/atlasdb/server/AtlasDbServiceServer.java +======= +import com.google.common.base.Optional; +>>>>>>> merge develop into perf cli branch (#820):atlasdb-service-server/src/main/java/com/palantir/atlasdb/server/AtlasDbServiceServer.java import com.google.common.collect.ImmutableSet; import com.palantir.atlasdb.factory.TransactionManagers; import com.palantir.atlasdb.impl.AtlasDbServiceImpl; @@ -35,6 +39,10 @@ public static void main(String[] args) throws Exception { public void run(AtlasDbServiceServerConfiguration config, final Environment environment) throws Exception { SerializableTransactionManager tm = TransactionManagers.create( config.getConfig(), +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56:atlasdb-service-server/src/main/java/com/palantir/atlasdb/server/AtlasDbServiceServer.java +======= + Optional.absent(), +>>>>>>> merge develop into perf cli branch (#820):atlasdb-service-server/src/main/java/com/palantir/atlasdb/server/AtlasDbServiceServer.java ImmutableSet.of(), environment.jersey()::register, false); diff --git a/atlasdb-tests-shared/src/main/java/com/palantir/atlasdb/sweep/AbstractSweeperTest.java b/atlasdb-tests-shared/src/main/java/com/palantir/atlasdb/sweep/AbstractSweeperTest.java index b64e9de8c54..e7f4e58493c 100644 --- a/atlasdb-tests-shared/src/main/java/com/palantir/atlasdb/sweep/AbstractSweeperTest.java +++ b/atlasdb-tests-shared/src/main/java/com/palantir/atlasdb/sweep/AbstractSweeperTest.java @@ -14,6 +14,11 @@ * limitations under the License. */ package com.palantir.atlasdb.sweep; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56:atlasdb-tests-shared/src/main/java/com/palantir/atlasdb/sweep/AbstractSweeperTest.java +======= + +import static com.palantir.atlasdb.schema.generated.SweepProgressTable.SweepProgressRowResult; +>>>>>>> merge develop into perf cli branch (#820):atlasdb-tests-shared/src/main/java/com/palantir/atlasdb/sweep/AbstractSweeperTest.java import java.util.List; import java.util.Set; diff --git a/atlasdb-timelock-server/build.gradle b/atlasdb-timelock-server/build.gradle index 8bbc7721652..6acea54019c 100644 --- a/atlasdb-timelock-server/build.gradle +++ b/atlasdb-timelock-server/build.gradle @@ -27,7 +27,11 @@ dependencies { compile 'org.apache.thrift:libthrift:' + libVersions.libthrift processor 'org.immutables:value:' + libVersions.immutables +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 processor('com.google.dagger:dagger-compiler:' + libVersions.dagger) { +======= + processor('com.google.dagger:dagger-compiler:2.0.2') { +>>>>>>> merge develop into perf cli branch (#820) // We need to explicitly exclude these so that intellij does not label them as provided if(gradle.startParameter.taskNames.contains('idea')) { exclude group: 'com.google.guava' diff --git a/build.gradle b/build.gradle index ba5888ba46f..67f9e790f41 100644 --- a/build.gradle +++ b/build.gradle @@ -25,7 +25,10 @@ plugins { } repositories { +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 mavenCentral() +======= +>>>>>>> merge develop into perf cli branch (#820) maven { url 'https://dl.bintray.com/palantir/releases/' } @@ -41,12 +44,15 @@ task clean(type: Delete) { delete buildDir } +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 apply plugin: 'com.palantir.jacoco-full-report' jacoco { toolVersion = libVersions.jacoco } +======= +>>>>>>> merge develop into perf cli branch (#820) subprojects { task allDeps(type: DependencyReportTask) {} } diff --git a/circle.yml b/circle.yml index a859a767140..50d47fc02b8 100644 --- a/circle.yml +++ b/circle.yml @@ -15,7 +15,11 @@ general: artifacts: - "build/reports/profile" - "atlasdb-ete-tests/container-logs" +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 - "atlasdb-cassandra-integration-tests/container-logs" +======= + - "atlasdb-cassandra-tests/container-logs" +>>>>>>> merge develop into perf cli branch (#820) - "atlasdb-cassandra-multinode-tests/container-logs" dependencies: diff --git a/commons-db/src/main/java/com/palantir/nexus/db/sql/SQLString.java b/commons-db/src/main/java/com/palantir/nexus/db/sql/SQLString.java index 61973321d34..de80c2a7a41 100644 --- a/commons-db/src/main/java/com/palantir/nexus/db/sql/SQLString.java +++ b/commons-db/src/main/java/com/palantir/nexus/db/sql/SQLString.java @@ -29,8 +29,13 @@ import javax.annotation.concurrent.GuardedBy; +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 import org.apache.commons.lang3.StringUtils; import org.apache.commons.lang3.Validate; +======= +import org.apache.commons.lang.StringUtils; +import org.apache.commons.lang.Validate; +>>>>>>> merge develop into perf cli branch (#820) import com.google.common.collect.ImmutableMap; import com.google.common.collect.Sets; diff --git a/docs/source/cluster_management/clis.rst b/docs/source/cluster_management/clis.rst index 9f35e5830e7..12439ab881e 100644 --- a/docs/source/cluster_management/clis.rst +++ b/docs/source/cluster_management/clis.rst @@ -44,6 +44,7 @@ timestamp --------- Read or recalculate the immutable timestamp. Run ``./bin/atlasdb help timestamp`` for more information. +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 Offline CLIs ============ @@ -59,6 +60,8 @@ For example, to run the ``fast-forward`` command with default settings, run ``./ The ``--offline`` flag will make the CLI ignore the leader, timestamp, and lock configuration blocks, and start an embedded timestamp and lock server. Once the CLI has completed, you can resume your AtlasDB clients. +======= +>>>>>>> merge develop into perf cli branch (#820) Running commands without any servers being up --------------------------------------------- diff --git a/docs/source/cluster_management/index.rst b/docs/source/cluster_management/index.rst index 61e2022a588..e58337d2c9d 100644 --- a/docs/source/cluster_management/index.rst +++ b/docs/source/cluster_management/index.rst @@ -9,4 +9,8 @@ Cluster Management backup-restore clis console +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 dropwizard-bundle +======= + +>>>>>>> merge develop into perf cli branch (#820) diff --git a/docs/source/configuration/cassandra_KVS_configuration.rst b/docs/source/configuration/cassandra_KVS_configuration.rst index 78b871c2ae8..65cf39d0546 100644 --- a/docs/source/configuration/cassandra_KVS_configuration.rst +++ b/docs/source/configuration/cassandra_KVS_configuration.rst @@ -1,5 +1,8 @@ +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 .. _cassandra-configuration: +======= +>>>>>>> merge develop into perf cli branch (#820) ========================================= Cassandra Key Value Service Configuration ========================================= @@ -19,8 +22,11 @@ e.g. runtime 'com.palantir.atlasdb:atlasdb-cassandra:0.7.0' +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 .. _cassandra-kvs-config: +======= +>>>>>>> merge develop into perf cli branch (#820) Configuring a Running Application to Use Cassandra ================================================== @@ -97,6 +103,7 @@ ssl This property is a boolean value saying whether or not to use ssl. When ``true``, it will use java system properties that are passed in as jvm arguments to determine how to set up the ssl connection. For example, you would use the jvm option ``-Djavax.net.ssl.trustStore=`` to tell atlas where to find the truststore to use. +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 .. _cassandra-sweep-config: Column Paging for Sweep (experimental) @@ -113,3 +120,5 @@ the Cassandra node to run out of memory. In such cases, limiting the value of ``timestampsGetterBatchSize`` (which is infinite by default) could result in greater reliability. On the other hand, more aggressive paging could lead to slower sweep performance. +======= +>>>>>>> merge develop into perf cli branch (#820) diff --git a/docs/source/configuration/index.rst b/docs/source/configuration/index.rst index c59358b0c47..e4828fcddfb 100644 --- a/docs/source/configuration/index.rst +++ b/docs/source/configuration/index.rst @@ -2,6 +2,8 @@ Configuration ============= +The AtlasDB configuration has two main blocks - keyValueService and leader. Please look at the keyValueService config for the KVS(Cassandra/Postgres) you are using and the Leader configuration page for configuring the leader block. + .. toctree:: :maxdepth: 1 :titlesonly: @@ -12,6 +14,9 @@ Configuration cassandra_KVS_configuration leader_config logging +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 The AtlasDB configuration has two main parts - keyValueService and leader. Please look at the keyValueService config for the KVS you are using (either :ref:`Cassandra ` or :ref:`Postgres `) and the :ref:`Leader Configuration ` page for configuring the leader block. +======= +>>>>>>> merge develop into perf cli branch (#820) diff --git a/docs/source/configuration/leader_config.rst b/docs/source/configuration/leader_config.rst index 1e86f6f9c6d..15df6c6b947 100644 --- a/docs/source/configuration/leader_config.rst +++ b/docs/source/configuration/leader_config.rst @@ -1,6 +1,7 @@ .. _leader-config: ==================== +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 Leader Configuration ==================== @@ -15,10 +16,34 @@ If no leader configuration is specified, then AtlasDB clients will create an emb If you wish to run more than one AtlasDB client with Cassandra KVS, then you **must** provide a leader block. Failure to do so can lead to data corruption. +======= +Leader configuration +==================== + +A minimal AtlasDB configuration for the leader block will look like : + +.. code-block:: yaml + + leader: + # This should be at least half the number of nodes in your cluster + quorumSize: 2 + learnerLogDir: var/data/paxosLogs + acceptorLogDir: var/data/paxosLogs + # This should be different for every node. If ssl is not enabled, then the host must be specified as http + localServer: https://:3828 + # This should be the same for every node. If ssl is not enabled, then the host must be specified as http + lockCreator: https://host1:3828 + # This should be the same for every node + leaders: + - https://host1:3828 # If ssl is not enabled, then the hosts must be specified as http + - https://host2:3828 + - https://host3:3828 +>>>>>>> merge develop into perf cli branch (#820) Leader ====== +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 Required parameters: .. list-table:: @@ -205,4 +230,22 @@ If you are running multiple AtlasDB clients, ensure your CLI is pointing at the - "http://host2:3828/api" - "http://host3:3828/api" sslConfiguration: - trustStorePath: var/security/truststore.jks \ No newline at end of file + trustStorePath: var/security/truststore.jks +======= +The leader block is where you specify the leaders (all hosts that participate in leader election) for paxos and a lockCreator for creating the schema mutation lock table. + +Required parameters: + +- ``quorumSize`` : Number of hosts that form a majority. This number must be greater than half of the total number of hosts. +- ``leaders`` : A list of all hosts. The protocol must be http/https depending on if ssl is configured. +- ``localhost`` : The ``protocol://hostname:port`` eg: ``https://myhost:3828`` of the host on which this config exists. + +Optional parameters: + +- ``learnerLogDir`` : Path to the paxos learner logs (defaults to var/data/paxos/learner) +- ``acceptorLogDir`` : Path to the paxos acceptor logs (defaults to var/data/paxos/acceptor) +- ``lockCreator`` : The host responsible for creation of the schema mutation lock table. If specified, this must be same across all hosts. (defaults to the first host in the sorted leaders list) +- ``pingRateMs`` : defaults to 5000 +- ``randomWaitBeforeProposingLeadershipMs`` : defaults to 1000 +- ``leaderPingResponseWaitMs`` : defaults to 5000 +>>>>>>> merge develop into perf cli branch (#820) diff --git a/docs/source/release_notes/release-notes.rst b/docs/source/release_notes/release-notes.rst index 6fa09b5e9ab..a81cbb67bb9 100644 --- a/docs/source/release_notes/release-notes.rst +++ b/docs/source/release_notes/release-notes.rst @@ -31,6 +31,7 @@ Changelog .. <<<<------------------------------------------------------------------------------------------------------------->>>> ======= +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 v0.14.0 ======= @@ -77,6 +78,8 @@ v0.14.0 .. <<<<------------------------------------------------------------------------------------------------------------->>>> ======= +======= +>>>>>>> merge develop into perf cli branch (#820) v0.13.0 ======= @@ -88,6 +91,7 @@ v0.13.0 - Change * - |breaking| +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 - ``AtlasDbServer`` has been renamed to ``AtlasDbServiceServer``. Any products that are using this should switch to using the standard AtlasDB java API instead. (`Pull Request `__) @@ -117,6 +121,11 @@ v0.13.0 (`Pull Request `__) +======= + - ``AtlasDbServer`` has been renamed to ``AtlasDbServiceServer``. Any products that are using this should switch to using the Java Feign client instead. + (`Pull Request `__) + +>>>>>>> merge develop into perf cli branch (#820) .. <<<<------------------------------------------------------------------------------------------------------------->>>> ======= @@ -192,6 +201,8 @@ v0.11.2 ======= v0.11.1 +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 +======= ======= .. list-table:: @@ -212,6 +223,172 @@ v0.11.1 .. <<<<------------------------------------------------------------------------------------------------------------->>>> +======= +v0.11.0 +>>>>>>> merge develop into perf cli branch (#820) +======= + +.. list-table:: + :widths: 5 40 + :header-rows: 1 + + * - Type + - Change + + * - |improved| + - Clarified the logging when multiple timestamp servers are running to state that CLIs could be causing the issue. + (`Pull Request `__) + + * - |changed| + - Updated cassandra client from 2.2.1 to 2.2.7 and cassandra docker testing version from 2.2.6 to 2.2.7. + (`Pull Request `__) + + * - |fixed| + - The leader config now contains a new ``lockCreator`` option, which specifies the single node that creates the locks table when starting your cluster for the very first time. + This configuration prevents an extremely unlikely race condition where multiple clients can create the locks table simultaneously. + Full details on the failure scenario can be found on `#444 `__. + + If left blank, ``lockCreator`` will default to the first host in the ``leaders`` list, but we recommend setting this explicitly to ensure that the lockCreater is the same value across all your clients for a specific service. + This configuration is only relevant for new clusters and does not affect existing AtlasDB clusters. + + Full details for configuring the leader block, see `cassandra configuration `__. + (`Pull Request `__) + + * - |fixed| + - A utility method was removed in the previous release, breaking an internal product that relied on it. + This method has now been added back. + (`Pull Request `__) + + * - |fixed| + - Removed unnecessary error message for missing _timestamp metadata table. + _timestamp is a hidden table, and it is expected that _timestamp metadata should not be retrievable from public API. + (`Pull Request `__) + + * - |improved| + - Trace logging is more informative and will log all failed calls. + To enable trace logging, see `Enabling Cassandra Tracing `__. + (`Pull Request `__) + + * - |deprecated| + - The Cassandra KVS now supports specifying SSL options via the new ``sslConfiguration`` block, which takes precedence over the now deprecated ``ssl`` property. + The ``ssl`` property will be removed in a future release, and consumers leveraging the Cassandra KVS are encouraged to use the ``sslConfiguration`` block instead. + See the `Cassandra SSL Configuration `__ documentation for more details. + (`Pull Request `__) + +.. <<<<------------------------------------------------------------------------------------------------------------->>>> + +======= +v0.10.0 +======= + +.. list-table:: + :widths: 5 40 + :header-rows: 1 + + * - Type + - Change + + * - |changed| + - Updated HikariCP dependency from 2.4.3 to 2.4.7 to comply with updates in internal products. + Details of the HikariCP changes can be found `here `__. + (`Pull Request `__) + + * - |new| + - AtlasDB currently allows you to create dynamic columns (wide rows), but you can only retrieve entire rows or specific columns. + Typically with dynamic columns, you do not know all the columns you have in advance, and this features allows you to page through dynamic columns per row, reducing pressure on the underlying KVS. + Products or clients (such as AtlasDB Sweep) making use of wide rows should consider using ``getRowsColumnRange`` instead of ``getRows`` in ``KeyValueService``. + (`Pull Request `__) + + Note: This is considered a beta feature and is not yet being used by AtlasDB Sweep. + + * - |fixed| + - We properly check that cells are not set to empty (zero-byte) or null. + (`Pull Request `__) + + * - |improved| + - Cassandra client connection pooling will now evict idle connections over a longer period of time and has improved logic + for deciding whether or not a node should be blacklisted. This should result in less connection churn + and therefore lower latency. (`Pull Request `__) + +.. <<<<------------------------------------------------------------------------------------------------------------->>>> + +====== +v0.9.0 +====== + +.. list-table:: + :widths: 5 40 + :header-rows: 1 + + * - Type + - Change + + * - |breaking| + - Inserting an empty (size = 0) value into a ``Cell`` will now throw an ``IllegalArgumentException``. (`#156 `__) Likely empty + values include empty strings and empty protobufs. + + AtlasDB cannot currently distinguish between empty and deleted cells. In previous versions of AtlasDB, inserting + an empty value into a ``Cell`` would delete that cell. Thus, in this snippet, + + .. code-block:: java + + Transaction.put(table, ImmutableMap.of(myCell, new byte[0])) + Transaction.get(table, ImmutableSet.of(myCell)).get(myCell) + + the second line will return ``null`` instead of a zero-length byte array. + + To minimize confusion, we explicitly disallow inserting an empty value into a cell by throwing an + ``IllegalArgumentException``. + + In particular, this change will break calls to ``Transaction.put(TableReference tableRef, Map values)``, + as well as generated code which uses this method, if any entry in ``values`` contains a zero-byte array. If your + product does not need to distinguish between empty and non-existent values, simply make sure all the ``values`` + entries have positive length. If the distinction is necessary, you will need to explicitly differentiate the + two cases (for example, by introducing a sentinel value for empty cells). + + If any code deletes cells by calling ``Transaction.put(...)`` with an empty array, use + ``Transaction.delete(...)`` instead. + + *Note*: Existing cells with empty values will be interpreted as deleted cells, and will not lead to Exceptions when read. + (`Pull Request `__) + + * - |improved| + - The warning emitted when an attempted leadership election fails is now more descriptive. + (`Pull Request `__) + + * - |fixed| + - Code generation for the ``hashCode`` of ``*IdxColumn`` classes now uses ``deepHashCode`` for its arrays such that it returns + consistent hash codes for use with hash-based collections (HashMap, HashSet, HashTable). + This issue will only appear if you are instantiating columns in multiple places and storing columns in hash collections. + + If you are using `Indices `__ we recommend you upgrade as a precaution and ensure you are not relying on logic related to the ``hashCode`` of auto-generated ``*IdxColumn`` classes. + You will need to regenerate your schema code in order to see this fix. + (`Pull Request `__) + +.. <<<<------------------------------------------------------------------------------------------------------------->>>> + +====== +v0.8.0 +====== + +.. list-table:: + :widths: 5 40 + :header-rows: 1 + + * - Type + - Change + + * - |fixed| + - Removed a check enforcing a leader block config when one was not required. + This prevents AtlasDB 0.11.0 clients from starting if a leader configuration is not specified (i.e. single node clusters). + (`Pull Request `__) + + * - |improved| + - Updated schema table generation to optimize reads with no ColumnSelection specified against tables with fixed columns. To benefit from this improvement you will need to re-generate your schemas. + (`Pull Request `__) + +.. <<<<------------------------------------------------------------------------------------------------------------->>>> + ======= v0.11.0 ======= diff --git a/docs/source/schemas/index.rst b/docs/source/schemas/index.rst index 24c7b3bec31..10ff30299fb 100644 --- a/docs/source/schemas/index.rst +++ b/docs/source/schemas/index.rst @@ -128,6 +128,7 @@ characteristics, among other things. The supported types are: | Name | Java Type | Format | Anywhere in | Range | | | | | table? | Scans? | +===============================+==============+===============+================+==========+ +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 | FIXED_LONG | long | byte[8] | YES | YES | +-------------------------------+--------------+---------------+----------------+----------+ | FIXED_LONG_LITTLE_ENDIAN | long | byte[8] | YES | NO [4]_ | @@ -162,12 +163,49 @@ characteristics, among other things. The supported types are: .. [2] A ``STRING`` or ``BLOB`` can only be a row or column component +======= +| FIXED\_LONG | long | byte[8] | YES | YES | ++-------------------------------+--------------+---------------+----------------+----------+ +| FIXED\_LONG\_LITTLE\_ENDIAN | long | byte[8] | YES | NO | +| [3]_ | | | | | ++-------------------------------+--------------+---------------+----------------+----------+ +| NULLABLE\_FIXED\_LONG | long | byte[9] | YES | YES | ++-------------------------------+--------------+---------------+----------------+----------+ +| VAR\_LONG | long | byte[len] [1]_| YES | YES | ++-------------------------------+--------------+---------------+----------------+----------+ +| VAR\_SIGNED\_LONG | long | byte[len] [1]_| YES | YES | ++-------------------------------+--------------+---------------+----------------+----------+ +| UUID | UUID | byte[16] | YES | YES | ++-------------------------------+--------------+---------------+----------------+----------+ +| SHA\_256\_HASH | Sha256Hash | byte[32] | YES | YES | ++-------------------------------+--------------+---------------+----------------+----------+ +| STRING | String | byte[] | NO [2]_ | YES | ++-------------------------------+--------------+---------------+----------------+----------+ +| VAR\_STRING | String | byte[len] | YES | NO | ++-------------------------------+--------------+---------------+----------------+----------+ +| BLOB | byte[] | byte[] | NO [2]_ | YES | ++-------------------------------+--------------+---------------+----------------+----------+ +| SIZED\_BLOB | byte[] | byte[len] | YES | NO | ++-------------------------------+--------------+---------------+----------------+----------+ + +.. [1] + All long data types are signed, but VAR\_SIGNED\_LONG is + encoded in a manner which stores negative numbers more efficiently than + VAR\_LONG. + In particular, 0 to 127 are 1 byte for VAR\_LONG and -64 to 63 are 1 byte for + VAR\_SIGNED\_LONG. VAR\_LONG will always use 10 bytes to encode negative + numbers. + +.. [2] + A STRING or BLOB can only be a row or column component +>>>>>>> merge develop into perf cli branch (#820) if it is the last component of the component list. .. [3] This type can be useful on some key value stores because keys next to each other won't be written next to each other. This can be good because it will spread out the load of writes to many different ranges. +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 .. [4] If a type does not support range-scanning, **range scans will still be available @@ -180,6 +218,8 @@ characteristics, among other things. The supported types are: - (“ab”, 10) will match - (“ab”, 20) will match - (“abc”, 30) will not match +======= +>>>>>>> merge develop into perf cli branch (#820) Protobufs and Persistables ~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/gradle/baseline.gradle b/gradle/baseline.gradle index 322cd80191a..6d04fca92f1 100644 --- a/gradle/baseline.gradle +++ b/gradle/baseline.gradle @@ -1,5 +1,9 @@ List blacklistedBaselineProjects = [ +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 'atlasdb-cassandra-integration-tests', +======= + 'atlasdb-cassandra-tests', +>>>>>>> merge develop into perf cli branch (#820) 'atlasdb-cli', 'atlasdb-client', 'atlasdb-client-protobufs', diff --git a/gradle/versions.gradle b/gradle/versions.gradle index a3828a0c55c..46207904f78 100644 --- a/gradle/versions.gradle +++ b/gradle/versions.gradle @@ -20,7 +20,10 @@ ext.libVersions = joda_time: '2.7', json_simple: '1.1.1', snappy: '1.1.1.7', +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 dagger: '2.4', +======= +>>>>>>> merge develop into perf cli branch (#820) immutables: '2.2.8', jackson: '2.5.1', jackson_annotation: '2.5.0', @@ -35,8 +38,12 @@ ext.libVersions = findbugs: '1.3.9', hikariCP: '2.4.7', checkstyle: '6.18', +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 findbugsAnnotations: '2.0.3', +======= +>>>>>>> merge develop into perf cli branch (#820) snakeyaml: '1.12', + findbugsAnnotations: '2.0.3', // Danger, Will Robinson! // diff --git a/settings.gradle b/settings.gradle index 012aa5f40d2..5978470795f 100644 --- a/settings.gradle +++ b/settings.gradle @@ -1,13 +1,20 @@ include ":atlasdb-api" include ":atlasdb-cassandra" +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 include ":atlasdb-cassandra-integration-tests" +======= +include ":atlasdb-cassandra-tests" +>>>>>>> merge develop into perf cli branch (#820) include ":atlasdb-cassandra-multinode-tests" include ":atlasdb-cli" include ":atlasdb-client" include ":atlasdb-client-protobufs" include ":atlasdb-config" include ":atlasdb-console" +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 include ":atlasdb-dagger" +======= +>>>>>>> merge develop into perf cli branch (#820) include ":atlasdb-dropwizard-bundle" include ":atlasdb-dropwizard-tests" include ":atlasdb-dbkvs" @@ -29,6 +36,10 @@ include ":atlasdb-timelock-ete" include ":atlasdb-timelock-server" include ":atlasdb-commons" include ":atlasdb-partition-manager" +<<<<<<< 7033b8fc57203bf309772ac48101c6126fb91d56 +======= +include ":atlasdb-perf" +>>>>>>> merge develop into perf cli branch (#820) include ":atlasdb-service" include ":atlasdb-service-server" include ":commons-annotations"