Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add logic back to try to reconnect to the app websocket if socket is closed #279

Merged
merged 15 commits into from
Oct 28, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -179,8 +179,9 @@ You need `holochain` and `hc` on your path, best to get them from nix with `nix-

To perform the pre-requisite DNA compilation steps, and run the Nodejs test, run:
```bash
nix-shell
./run-test.sh
nix develop
./build-fixture.sh
npm run test
```

## Contribute
Expand Down
32 changes: 19 additions & 13 deletions src/api/app/websocket.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,12 @@
import Emittery, { UnsubscribeFunction } from "emittery";
import { omit } from "lodash-es";
import { AgentPubKey, CellId, InstalledAppId, RoleName } from "../../types.js";
import { AppInfo, CellType, MemproofMap } from "../admin/index.js";
import {
AppAuthenticationToken,
AppInfo,
CellType,
MemproofMap,
} from "../admin/index.js";
import {
catchError,
DEFAULT_TIMEOUT,
Expand Down Expand Up @@ -77,6 +82,7 @@ export class AppWebsocket implements AppClient {
CallZomeResponseGeneric<Uint8Array>,
CallZomeResponse
>;
private readonly appAuthenticationToken: AppAuthenticationToken;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't believe this is very high risk, but I don't feel comfortable not pointing this out. Storing this token may expose it to an injected script. If it is exfiltrated then it is re-usable elsewhere.

I much prefer the single-use tokens but I understand that this is being done for UX.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is, and it's similar to the zome call signing credentials. They're equally stored in JavaScript memory for convenience, which compromises security to a degree.


cachedAppInfo?: AppInfo | null;

Expand Down Expand Up @@ -110,6 +116,7 @@ export class AppWebsocket implements AppClient {
private constructor(
client: WsClient,
appInfo: AppInfo,
token: AppAuthenticationToken,
callZomeTransform?: CallZomeTransform,
defaultTimeout?: number
) {
Expand All @@ -118,6 +125,7 @@ export class AppWebsocket implements AppClient {
this.installedAppId = appInfo.installed_app_id;
this.defaultTimeout = defaultTimeout ?? DEFAULT_TIMEOUT;
this.callZomeTransform = callZomeTransform ?? defaultCallZomeTransform;
this.appAuthenticationToken = token;
this.emitter = new Emittery<AppEvents>();
this.cachedAppInfo = appInfo;

Expand Down Expand Up @@ -204,18 +212,15 @@ export class AppWebsocket implements AppClient {

const client = await WsClient.connect(options.url, options.wsClientOptions);

if (env?.APP_INTERFACE_TOKEN) {
// Note: This will only work for multiple connections if a single_use = false token is provided
await client.authenticate({ token: env.APP_INTERFACE_TOKEN });
} else {
if (!options.token) {
throw new HolochainError(
"AppAuthenticationTokenMissing",
`unable to connect to Conductor API - no app authentication token provided.`
);
}
await client.authenticate({ token: options.token });
}
const token = options.token ?? env?.APP_INTERFACE_TOKEN;

if (!token)
throw new HolochainError(
"AppAuthenticationTokenMissing",
`unable to connect to Conductor API - no app authentication token provided.`
);

await client.authenticate({ token });

const appInfo = await (
AppWebsocket.requester(client, "app_info", DEFAULT_TIMEOUT) as Requester<
Expand All @@ -233,6 +238,7 @@ export class AppWebsocket implements AppClient {
return new AppWebsocket(
client,
appInfo,
token,
options.callZomeTransform,
options.defaultTimeout
);
Expand Down
29 changes: 29 additions & 0 deletions src/api/client.ts
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ export class WsClient extends Emittery {
options: WsClientOptions;
private pendingRequests: Record<number, HolochainRequest>;
private index: number;
private authenticationToken: AppAuthenticationToken | undefined;

constructor(socket: IsoWebSocket, url?: URL, options?: WsClientOptions) {
super();
Expand Down Expand Up @@ -178,6 +179,7 @@ export class WsClient extends Emittery {
* @param request - The authentication request, containing an app authentication token.
*/
async authenticate(request: AppAuthenticationRequest): Promise<void> {
this.authenticationToken = request.token;
return this.exchange(request, (request, resolve) => {
const encodedMsg = encode({
type: "authenticate",
Expand Down Expand Up @@ -212,6 +214,33 @@ export class WsClient extends Emittery {
sendHandler(request, resolve, reject);
});
return promise as Promise<Response>;
} else if (this.url && this.authenticationToken) {
const response = new Promise<unknown>((resolve, reject) => {
// typescript forgets in this promise scope that `this.url` is not undefined
const socket = new IsoWebSocket(this.url as URL, this.options);
this.socket = socket;
socket.onerror = (errorEvent) => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this tell the difference between an invalid token and other kinds of error?

It seems like the user experience wouldn't be great if requests always result in a reconnect that fails when a temporary token was in use. If we can tell the token is invalid here, it would be good to clear it so that the client stops trying to reconnect

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With an invalid token what happens is that you get the "Client disconnected with pending requests" error and this onError callback here is never called in the first place.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't actually understand where the "client disconnected with pending requests" occurs...but if I log something in that onError callback it's not getting logged when it fails due to an invalid token.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting. Well Holochain won't send back anything to tell the client that the connection is going to be closed, it'll just close the connection from its end. Where is that error thrown, can/should it be caught?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I confirm that his happens. It comes in as a close event on the websocket with code 1005 which means "No status received". It's indistinguishable from another unexpected close event which usually is 1005 or 1006. For this reconnect we could register another close handler that unsets the token, but it's not 100 % certain that the close was due to an invalid token.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a specific InvalidTokenError now which only happens on failed automatic websocket reconnect. If the reconnect fails, the stored token is deleted. No risk of this happening in a loop, because the reconnects are only attempted as part of a request being made by the client consumer.

reject(
new HolochainError(
"ConnectionError",
`could not connect to Holochain Conductor API at ${this.url} - ${errorEvent.error}`
)
);
};
socket.onopen = () => {
// Send authentication token
const encodedMsg = encode({
type: "authenticate",
data: encode({
token: this.authenticationToken as AppAuthenticationToken,
}),
});
this.socket.send(encodedMsg);
sendHandler(request, resolve, reject);
};
this.setupSocket();
});
return response as Promise<Response>;
} else {
return Promise.reject(new Error("Socket is not open"));
}
Expand Down
5 changes: 4 additions & 1 deletion test/e2e/common.ts
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,8 @@ export const withConductor =
};

export const installAppAndDna = async (
adminPort: number
adminPort: number,
nonExpiringToken?: boolean
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to call this parameter singleUseToken to stay close to the conductor API naming?

Copy link
Contributor Author

@matthme matthme Oct 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And then have it be true by default? We can do that yes. It's single-use and expiring...I could also make two parameters for expiry and single use.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed it to take two separate parameters now to be aligned with the conductor API naming: d6a7db7

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That works yes, just expose the API with defaults. I was thinking that single use could have a timeout and when it's false then it could have no expiry. I wouldn't be opposed to the client exposing the API that way because those are the two expected configurations really

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Though it's probably useful to let people just have access to these

): Promise<{
installed_app_id: InstalledAppId;
cell_id: CellId;
Expand Down Expand Up @@ -127,6 +128,8 @@ export const installAppAndDna = async (
});
const issued = await admin.issueAppAuthenticationToken({
installed_app_id,
single_use: !nonExpiringToken,
expiry_seconds: nonExpiringToken ? 0 : 30,
});
const client = await AppWebsocket.connect({
url: new URL(`ws://localhost:${appPort}`),
Expand Down
46 changes: 46 additions & 0 deletions test/e2e/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1441,6 +1441,52 @@ test(
})
);

test(
"client reconnects websocket if closed before making a zome call",
withConductor(ADMIN_PORT, async (t) => {
const { cell_id, client, admin } = await installAppAndDna(ADMIN_PORT, true);
await admin.authorizeSigningCredentials(cell_id);
await client.client.close();
const call = client.callZome({
cell_id,
zome_name: TEST_ZOME_NAME,
fn_name: "bar",
provenance: cell_id[1],
payload: null,
});
try {
await call;
t.pass("websocket was reconnected successfully");
} catch (error) {
t.fail("websocket was not reconnected");
matthme marked this conversation as resolved.
Show resolved Hide resolved
}
})
);

test(
"client fails to reconnect to websocket if closed before making a zome call and does not end up in a loop",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't quite testing what I was concerned about with looping. It's repeatedly reconnecting with an invalid token that I was concerned about. If the caller isn't careful about which error is which then they might treat this error as retryable and keep trying to reconnect. I'd prefer if this test could make two zome calls and the second one got a different error message back asking the caller to provide new credentials

Copy link
Contributor Author

@matthme matthme Oct 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, configuring the message to ask for new credentials doesn't work I think due to the "client closed with pending requests" error that doesn't even let us define the error message here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I can look into handling this close event better in the first place.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But I'm not sure whether we should conflate this with this PR

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no way of modifying the error in a way to ask the user to provide a valid token. Any zome call with an invalid token will now lead to the following error:

ClientClosedWithPendingRequests: client closed with pending requests - close event code: 1005, request id: 1

And this seems to be a result of how the conductor handles things and cannot be affected at the js side as I understand it. This error however is distinguishable by a caller from a "normal" zome call error if required so I don't think there's more we can do here without touching the conductor behavior.

withConductor(ADMIN_PORT, async (t) => {
const { cell_id, client, admin } = await installAppAndDna(ADMIN_PORT);
await admin.authorizeSigningCredentials(cell_id);
await client.client.close();
const call = client.callZome({
cell_id,
zome_name: TEST_ZOME_NAME,
fn_name: "bar",
provenance: cell_id[1],
payload: null,
});
try {
await call;
t.fail(
"reconnecting to websocket should have failed due to an invalid token."
);
} catch (error) {
ThetaSinner marked this conversation as resolved.
Show resolved Hide resolved
t.pass("reconnecting to websocket failed");
}
})
);

test(
"Rust enums are serialized correctly",
withConductor(ADMIN_PORT, async (t) => {
Expand Down
Loading