Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

replaced isAvailable call #331

Merged
merged 2 commits into from
Oct 4, 2024
Merged

replaced isAvailable call #331

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
971b749
replaced isAvailable call
patduin Oct 3, 2024
3cd069d
Update CHANGELOG.md
JayGreeeen Oct 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
## [3.13.4] - 2024-10-04
### Fix
- Removed call to primary and made it only when Kerberos (SASL) is enabled.
- Fixed Handler creation for SASL.

## [3.13.3] - 2024-10-02
### Added
- Metric for monitoring open transports. `<prefix>.open_transports_gauge`
- Metric for monitoring open transports. `<prefix>.com_hotels_bdp_waggledance_open_transports_gauge`
### Changed
- Removed RetryingHMSHandler. Retries are done in the client there should be no need to wrap everything in retry logic again.

Expand Down
5 changes: 0 additions & 5 deletions ...va/com/hotels/bdp/waggledance/mapping/service/impl/PrefixBasedDatabaseMappingService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,11 +97,6 @@ private void add(AbstractMetaStore metaStore) {

if (metaStore.getFederationType() == PRIMARY) {
primaryDatabaseMapping = databaseMapping;
if (!metaStoreMapping.isAvailable()) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was added as part of Kerberos support: https://github.com/ExpediaGroup/waggle-dance/pull/266/files#diff-8c2a82b373b4f06981a5086c6e6f2cbafea8480451366fcf1ba3c1490450388a

For non Kerberos this is not needed and unwanted as Healtcheck connections just opening the processor will cause this to call the primary metastore. In production environments this is slowing things down. I moved it to the SASL code path.

throw new WaggleDanceException(
String.format("Primary metastore is unavailable %s", metaStore.getRemoteMetaStoreUris())
);
}
}

mappingsByPrefix.put(metaStoreMapping.getDatabasePrefix(), databaseMapping);
Expand Down
10 changes: 9 additions & 1 deletion ...e-core/src/main/java/com/hotels/bdp/waggledance/server/TSetIpAddressProcessorFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
import org.apache.hadoop.hive.conf.HiveConf;
import org.apache.hadoop.hive.metastore.IHMSHandler;
import org.apache.hadoop.hive.metastore.TSetIpAddressProcessor;
import org.apache.thrift.TException;
import org.apache.thrift.TProcessor;
import org.apache.thrift.TProcessorFactory;
import org.apache.thrift.transport.TSocket;
Expand Down Expand Up @@ -59,7 +60,14 @@ public TProcessor getProcessor(TTransport transport) {

boolean useSASL = hiveConf.getBoolVar(HiveConf.ConfVars.METASTORE_USE_THRIFT_SASL);
if (useSASL) {
IHMSHandler handler = TokenWrappingHMSHandler.newProxyInstance(baseHandler, useSASL);
try {
baseHandler.getStatus();
} catch (TException e) {
throw new RuntimeException("Error creating TProcessor. Could not get status.", e);
}
IHMSHandler tokenHandler = TokenWrappingHMSHandler.newProxyInstance(baseHandler, useSASL);
IHMSHandler handler = ExceptionWrappingHMSHandler.newProxyInstance(tokenHandler);
transportMonitor.monitor(transport, baseHandler);
return new TSetIpAddressProcessor<>(handler);
} else {
IHMSHandler handler = ExceptionWrappingHMSHandler.newProxyInstance(baseHandler);
Expand Down
15 changes: 14 additions & 1 deletion ...re/src/test/java/com/hotels/bdp/waggledance/server/TSetIpAddressProcessorFactoryTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,12 +42,13 @@ public class TSetIpAddressProcessorFactoryTest {
private @Mock TTransportMonitor transportMonitor;
private @Mock TTransport transport;

private final HiveConf hiveConf = new HiveConf();
private HiveConf hiveConf;
private TSetIpAddressProcessorFactory factory;

@Before
public void init() {
when(federatedHMSHandlerFactory.create()).thenReturn(federatedHMSHandler);
hiveConf = new HiveConf();
factory = new TSetIpAddressProcessorFactory(hiveConf, federatedHMSHandlerFactory, transportMonitor);
}

Expand All @@ -67,4 +68,16 @@ public void connectionIsMonitored() throws Exception {
assertThat(transportCaptor.getValue(), is(transport));
assertThat(handlerCaptor.getValue(), is(instanceOf(FederatedHMSHandler.class)));
}

@Test
public void connectionIsMonitoredSasl() throws Exception {
hiveConf.setBoolVar(HiveConf.ConfVars.METASTORE_USE_THRIFT_SASL, Boolean.TRUE);
factory.getProcessor(transport);

ArgumentCaptor<TTransport> transportCaptor = ArgumentCaptor.forClass(TTransport.class);
ArgumentCaptor<Closeable> handlerCaptor = ArgumentCaptor.forClass(Closeable.class);
verify(transportMonitor).monitor(transportCaptor.capture(), handlerCaptor.capture());
assertThat(transportCaptor.getValue(), is(transport));
assertThat(handlerCaptor.getValue(), is(instanceOf(FederatedHMSHandler.class)));
}
}
Loading