Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature/local_cloud_toggle #296

Open
wants to merge 71 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
1d8d722
feature/local_cloud_toggle
BrianWhitneyAI Oct 23, 2024
f2a0559
feature/toggle-modal
BrianWhitneyAI Oct 25, 2024
f52d31c
add conditional show move menu item
BrianWhitneyAI Oct 28, 2024
77a52bd
modal list to table
BrianWhitneyAI Oct 28, 2024
1201a84
reorder imports
BrianWhitneyAI Oct 28, 2024
ae06c75
use existing filesize method
BrianWhitneyAI Oct 28, 2024
e159438
add placeholder action
BrianWhitneyAI Oct 28, 2024
8c810a6
add placeholder logic
BrianWhitneyAI Oct 28, 2024
1e3c38b
remove duplicate reducer
BrianWhitneyAI Oct 28, 2024
588447d
update styling for move modal
BrianWhitneyAI Oct 28, 2024
741d415
remove off nas
BrianWhitneyAI Oct 28, 2024
1bc859f
update move only structure
BrianWhitneyAI Oct 29, 2024
034549d
remove loc target
BrianWhitneyAI Oct 31, 2024
a450c41
add gradient
BrianWhitneyAI Oct 31, 2024
dbb368e
Merge pull request #311 from AllenInstitute/feature/toggle-modal
BrianWhitneyAI Nov 4, 2024
ed34524
feature/move-files-styling
BrianWhitneyAI Nov 8, 2024
52bd1a9
feature/copy-to-local-cache-request
BrianWhitneyAI Nov 11, 2024
3e9a0eb
add test
BrianWhitneyAI Nov 11, 2024
31d4abf
add LoadBalancerBaseURl gloabl variable
BrianWhitneyAI Nov 14, 2024
6abbb7c
update put request
BrianWhitneyAI Nov 18, 2024
46bae5d
add username to cache request
BrianWhitneyAI Nov 18, 2024
f61ded5
disable caching for web
BrianWhitneyAI Nov 18, 2024
4f4e570
feature/refactor-environment
BrianWhitneyAI Nov 20, 2024
cf02d0b
add back LazilyRenderedThumbnail test
BrianWhitneyAI Nov 20, 2024
4020326
Merge branch 'main' into feature/local_cloud_toggle
BrianWhitneyAI Nov 20, 2024
f697ebd
Merge branch 'feature/local_cloud_toggle' into feature/copy-to-local-…
BrianWhitneyAI Nov 20, 2024
8c60c6f
Merge branch 'feature/copy-to-local-cache-request' into feature/refac…
BrianWhitneyAI Nov 20, 2024
9d4668b
Merge branch 'feature/local_cloud_toggle' into feature/move-files-sty…
BrianWhitneyAI Nov 20, 2024
07a2715
refactor enum and persisted env
BrianWhitneyAI Nov 22, 2024
ccca39f
add back dlt comment
BrianWhitneyAI Nov 22, 2024
390a804
Merge pull request #339 from AllenInstitute/feature/refactor-environment
BrianWhitneyAI Nov 22, 2024
44b3f16
comment resolution
BrianWhitneyAI Nov 23, 2024
63796fd
Merge pull request #345 from AllenInstitute/feature/refactor-environment
BrianWhitneyAI Nov 23, 2024
86e32f3
Merge pull request #322 from AllenInstitute/feature/copy-to-local-cac…
BrianWhitneyAI Nov 23, 2024
6fc75d8
Merge branch 'main' into feature/local_cloud_toggle
BrianWhitneyAI Nov 25, 2024
f04b7c6
feature/toggle_status_report
BrianWhitneyAI Nov 25, 2024
3726ec5
comment res
BrianWhitneyAI Nov 26, 2024
60cf69a
adjust error case
BrianWhitneyAI Nov 26, 2024
422c8de
Update packages/core/state/interaction/logics.ts
BrianWhitneyAI Nov 26, 2024
e8809bc
Merge pull request #350 from AllenInstitute/feature/toggle_status_report
BrianWhitneyAI Nov 26, 2024
921421e
369 Fix broken FileAnnotationList test
pgarrison Dec 18, 2024
17e9980
Merge remote-tracking branch 'origin/feature/local_cloud_toggle' into…
BrianWhitneyAI Dec 18, 2024
656b8ea
368 Update arrange step of test to match cloud-first FMS (failing)
pgarrison Dec 18, 2024
a9a2940
368 Display Local File Path annotation as File Path (Local) (test pas…
pgarrison Dec 18, 2024
dfb7384
368 Test expects custom /allen mount point from OS
pgarrison Dec 18, 2024
7e94c3c
seperate table
BrianWhitneyAI Dec 18, 2024
4932dc8
368 Set custom /allen mount point for File Path (Local)
pgarrison Dec 18, 2024
a94c95e
update wording
BrianWhitneyAI Dec 18, 2024
1c54d47
368 Remove File Path (Local) top-level annotation to avoid duplication
pgarrison Dec 19, 2024
1126cd9
368 Rename annotation at the HTTPAnnotationService level
pgarrison Dec 19, 2024
022377f
368 Test expects specific annotation from LabKey (not top-level annot…
pgarrison Dec 19, 2024
1642a68
update table gradient and summeries
BrianWhitneyAI Dec 19, 2024
083456f
368 Test for new HTTPAnnotationService special case
pgarrison Dec 19, 2024
da16e23
File Path (Vast) instead of File Path (Local)
pgarrison Dec 20, 2024
6215d9e
367 FMS file_path is the cloud path (tests failing)
pgarrison Dec 18, 2024
160c0fe
367 Read cloud path from FileDetail object with http://... prefix
pgarrison Dec 20, 2024
66f572c
367 Hoist AICS_FMS_S3_URL_PREFIX to top-level constant
pgarrison Dec 20, 2024
8817332
367 More detailed file_path comment
pgarrison Dec 20, 2024
3cab1ac
368 import ordering
pgarrison Dec 20, 2024
efccc6c
remove NAS wording
BrianWhitneyAI Dec 20, 2024
29423b0
368 Clearer comment about loading localPath
pgarrison Dec 20, 2024
4e48e39
Merge pull request #320 from AllenInstitute/feature/move-files-styling
BrianWhitneyAI Dec 20, 2024
2736712
376 Downloads use labkey URL when file is available on Vast
pgarrison Dec 20, 2024
f706be0
376 Test for download path
pgarrison Dec 20, 2024
bbcd3e9
Merge branch 'main' into feature/local_cloud_toggle
BrianWhitneyAI Dec 20, 2024
71dd109
update local path label
BrianWhitneyAI Jan 2, 2025
ca2154b
Merge pull request #370 from AllenInstitute/feature/368-use-local-fil…
BrianWhitneyAI Jan 2, 2025
db08ad2
Merge branch 'feature/local_cloud_toggle' into feature/367-canonical-…
BrianWhitneyAI Jan 2, 2025
0425736
update annotation name
BrianWhitneyAI Jan 2, 2025
e76d13c
Merge pull request #375 from AllenInstitute/feature/367-canonical-pat…
BrianWhitneyAI Jan 2, 2025
de641da
Merge pull request #377 from AllenInstitute/feature/376-download-via-…
BrianWhitneyAI Jan 2, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 9 additions & 5 deletions packages/core/App.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import GlobalActionButtonRow from "./components/GlobalActionButtonRow";
import StatusMessage from "./components/StatusMessage";
import TutorialTooltip from "./components/TutorialTooltip";
import QuerySidebar from "./components/QuerySidebar";
import { FileExplorerServiceBaseUrl } from "./constants";
import { Environment } from "./constants";
import { interaction, selection } from "./state";
import useLayoutMeasurements from "./hooks/useLayoutMeasurements";

Expand All @@ -39,11 +39,11 @@ interface AppProps {
// Localhost: "https://localhost:9081"
// Stage: "http://stg-aics-api.corp.alleninstitute.org"
// From the web (behind load balancer): "/"
fileExplorerServiceBaseUrl?: string;
environment?: Environment;
}

export default function App(props: AppProps) {
const { fileExplorerServiceBaseUrl = FileExplorerServiceBaseUrl.PRODUCTION } = props;
const { environment = Environment.PRODUCTION } = props;

const dispatch = useDispatch();
const hasQuerySelected = useSelector(selection.selectors.hasQuerySelected);
Expand Down Expand Up @@ -79,8 +79,12 @@ export default function App(props: AppProps) {

// Set data source base urls
React.useEffect(() => {
dispatch(interaction.actions.initializeApp(fileExplorerServiceBaseUrl));
}, [dispatch, fileExplorerServiceBaseUrl]);
dispatch(
interaction.actions.initializeApp({
environment,
})
);
}, [dispatch, environment]);

// Respond to screen size changes
React.useEffect(() => {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import Annotation from "../../../entity/Annotation";
import FileFilter from "../../../entity/FileFilter";
import { initialState, reducer, reduxLogics, interaction, selection } from "../../../state";
import HttpAnnotationService from "../../../services/AnnotationService/HttpAnnotationService";
import { FESBaseUrl } from "../../../constants";

describe("<AnnotationFilterForm />", () => {
const LISTROW_TESTID_PREFIX = "default-button-";
Expand All @@ -31,14 +32,14 @@ describe("<AnnotationFilterForm />", () => {
it("shows all values as unchecked at first", async () => {
// arrange
const responseStub = {
when: `test/file-explorer-service/1.0/annotations/${fooAnnotation.name}/values`,
when: `${FESBaseUrl.TEST}/file-explorer-service/1.0/annotations/${fooAnnotation.name}/values`,
respondWith: {
data: { data: ["a", "b", "c", "d"] },
},
};
const mockHttpClient = createMockHttpClient(responseStub);
const annotationService = new HttpAnnotationService({
baseUrl: "test",
fileExplorerServiceBaseUrl: FESBaseUrl.TEST,
httpClient: mockHttpClient,
});
sandbox.stub(interaction.selectors, "getAnnotationService").returns(annotationService);
Expand Down Expand Up @@ -68,14 +69,14 @@ describe("<AnnotationFilterForm />", () => {
it("deselects and selects a value", async () => {
// arrange
const responseStub = {
when: `test/file-explorer-service/1.0/annotations/${fooAnnotation.name}/values`,
when: `${FESBaseUrl.TEST}/file-explorer-service/1.0/annotations/${fooAnnotation.name}/values`,
respondWith: {
data: { data: ["a", "b", "c", "d"] },
},
};
const mockHttpClient = createMockHttpClient(responseStub);
const annotationService = new HttpAnnotationService({
baseUrl: "test",
fileExplorerServiceBaseUrl: FESBaseUrl.TEST,
httpClient: mockHttpClient,
});
sandbox.stub(interaction.selectors, "getAnnotationService").returns(annotationService);
Expand Down Expand Up @@ -124,14 +125,14 @@ describe("<AnnotationFilterForm />", () => {
it("naturally sorts values", async () => {
// arrange
const responseStub = {
when: `test/file-explorer-service/1.0/annotations/${fooAnnotation.name}/values`,
when: `${FESBaseUrl.TEST}/file-explorer-service/1.0/annotations/${fooAnnotation.name}/values`,
respondWith: {
data: { data: ["AICS-24", "AICS-0", "aics-32", "aICs-2"] },
},
};
const mockHttpClient = createMockHttpClient(responseStub);
const annotationService = new HttpAnnotationService({
baseUrl: "test",
fileExplorerServiceBaseUrl: FESBaseUrl.TEST,
httpClient: mockHttpClient,
});
sandbox.stub(interaction.selectors, "getAnnotationService").returns(annotationService);
Expand Down Expand Up @@ -172,14 +173,14 @@ describe("<AnnotationFilterForm />", () => {
});

const responseStub = {
when: `test/file-explorer-service/1.0/annotations/${fooAnnotation.name}/values`,
when: `${FESBaseUrl.TEST}/file-explorer-service/1.0/annotations/${fooAnnotation.name}/values`,
respondWith: {
data: { data: [true, false] },
},
};
const mockHttpClient = createMockHttpClient(responseStub);
const annotationService = new HttpAnnotationService({
baseUrl: "test",
fileExplorerServiceBaseUrl: FESBaseUrl.TEST,
httpClient: mockHttpClient,
});

Expand Down Expand Up @@ -279,14 +280,14 @@ describe("<AnnotationFilterForm />", () => {
it("naturally sorts values", async () => {
// arrange
const responseStub = {
when: `test/file-explorer-service/1.0/annotations/${fooAnnotation.name}/values`,
when: `${FESBaseUrl.TEST}/file-explorer-service/1.0/annotations/${fooAnnotation.name}/values`,
respondWith: {
data: { data: [5, 8, 6.3, -12, 10000000000, 0] },
},
};
const mockHttpClient = createMockHttpClient(responseStub);
const annotationService = new HttpAnnotationService({
baseUrl: "test",
fileExplorerServiceBaseUrl: FESBaseUrl.TEST,
httpClient: mockHttpClient,
});
sandbox.stub(interaction.selectors, "getAnnotationService").returns(annotationService);
Expand Down Expand Up @@ -334,14 +335,14 @@ describe("<AnnotationFilterForm />", () => {
it("naturally sorts values", async () => {
// arrange
const responseStub = {
when: `test/file-explorer-service/1.0/annotations/${fooAnnotation.name}/values`,
when: `${FESBaseUrl.TEST}/file-explorer-service/1.0/annotations/${fooAnnotation.name}/values`,
respondWith: {
data: { data: [446582220, 125, 10845000, 86400000] },
},
};
const mockHttpClient = createMockHttpClient(responseStub);
const annotationService = new HttpAnnotationService({
baseUrl: "test",
fileExplorerServiceBaseUrl: FESBaseUrl.TEST,
httpClient: mockHttpClient,
});
sandbox.stub(interaction.selectors, "getAnnotationService").returns(annotationService);
Expand Down
17 changes: 7 additions & 10 deletions packages/core/components/DirectoryTree/test/DirectoryTree.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ import {
import { Provider } from "react-redux";
import { createSandbox } from "sinon";

import { TOP_LEVEL_FILE_ANNOTATIONS } from "../../../constants";
import { FESBaseUrl, TOP_LEVEL_FILE_ANNOTATIONS } from "../../../constants";
import Annotation from "../../../entity/Annotation";
import AnnotationName from "../../../entity/Annotation/AnnotationName";
import { FmsFileAnnotation } from "../../../services/FileService";
Expand Down Expand Up @@ -53,17 +53,13 @@ describe("<DirectoryTree />", () => {
type: "Text",
});

const baseUrl = "http://test-aics.corp.alleninstitute.org";
const baseDisplayAnnotations = TOP_LEVEL_FILE_ANNOTATIONS.filter(
(a) => a.name === AnnotationName.FILE_NAME
);
const state = mergeState(initialState, {
metadata: {
annotations: [...baseDisplayAnnotations, fooAnnotation, barAnnotation],
},
interaction: {
fileExplorerServiceBaseUrl: baseUrl,
},
selection: {
annotationHierarchy: [fooAnnotation.name, barAnnotation.name],
columns: [...baseDisplayAnnotations, fooAnnotation, barAnnotation].map((a) => ({
Expand Down Expand Up @@ -194,9 +190,13 @@ describe("<DirectoryTree />", () => {
},
];
const mockHttpClient = createMockHttpClient(responseStubs);
const annotationService = new HttpAnnotationService({ baseUrl, httpClient: mockHttpClient });
const fileExplorerServiceBaseUrl = FESBaseUrl.TEST;
const annotationService = new HttpAnnotationService({
fileExplorerServiceBaseUrl: fileExplorerServiceBaseUrl,
httpClient: mockHttpClient,
});
const fileService = new HttpFileService({
baseUrl,
fileExplorerServiceBaseUrl: fileExplorerServiceBaseUrl,
httpClient: mockHttpClient,
downloadService: new FileDownloadServiceNoop(),
});
Expand Down Expand Up @@ -362,9 +362,6 @@ describe("<DirectoryTree />", () => {
metadata: {
annotations: [...baseDisplayAnnotations, fooAnnotation, barAnnotation],
},
interaction: {
fileExplorerServiceBaseUrl: baseUrl,
},
selection: {
annotationHierarchy: [fooAnnotation.name],
columns: [...baseDisplayAnnotations, fooAnnotation, barAnnotation].map((a) => ({
Expand Down
61 changes: 26 additions & 35 deletions packages/core/components/FileDetails/FileAnnotationList.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,14 @@ export default function FileAnnotationList(props: FileAnnotationListProps) {
return;
}

const path = await executionEnvService.formatPathForHost(fileDetails.path);
let path;
if (fileDetails.localPath === null) {
// The Local File Path annotation is not defined because the file is not available
// on-premises
path = fileDetails.localPath;
} else {
path = await executionEnvService.formatPathForHost(fileDetails.localPath);
}
if (!active) {
return;
}
Expand Down Expand Up @@ -65,13 +72,29 @@ export default function FileAnnotationList(props: FileAnnotationListProps) {
return accum;
}

const annotationValue = annotation.extractFromFile(fileDetails);
let annotationValue = annotation.extractFromFile(fileDetails);
if (annotationValue === Annotation.MISSING_VALUE) {
// Nothing to show for this annotation -- skip
return accum;
}

const ret = [
if (annotation.name === AnnotationName.LOCAL_FILE_PATH) {
if (localPath === null) {
// localPath hasn't loaded yet, but it should eventually because there is an
// annotation named AnnotationName.LOCAL_FILE_PATH
return accum;
} else {
// Use the user's /allen mount point, if known
annotationValue = localPath;
}
}

if (annotation.name === AnnotationName.FILE_PATH) {
// Display the full http://... URL
annotationValue = fileDetails.cloudPath;
}

return [
...accum,
<FileAnnotationRow
key={annotation.displayName}
Expand All @@ -80,38 +103,6 @@ export default function FileAnnotationList(props: FileAnnotationListProps) {
value={annotationValue}
/>,
];

// Special case for file paths: we want to display both the "canonical" FMS path
// (i.e. POSIX path held in the database; what we have an annotation for)
// as well as the path at which the file is *actually* accessible on _this_ computer ("local" file path)
if (annotation.name === AnnotationName.FILE_PATH) {
if (fileDetails.path !== fileDetails.cloudPath) {
ret.push(
<FileAnnotationRow
key="file-path-cloud"
className={styles.row}
name="File Path (Cloud)"
value={fileDetails.cloudPath}
/>
);
}

// In certain circumstances (i.e., linux), the path at which a file is accessible is === the canonical path
if (localPath && localPath !== annotationValue && !localPath.startsWith("http")) {
ret.splice(
-1, // Insert before the "canonical" path so that it is the first path-like row to be seen
0, // ...don't delete the "canonical" path
<FileAnnotationRow
key="file-path-local"
className={styles.row}
name="File Path (Local)"
value={localPath}
/>
);
}
}

return ret;
}, [] as JSX.Element[]);
}, [annotations, fileDetails, isLoading, localPath]);

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,12 @@ import FileDetail from "../../../entity/FileDetail";
import ExecutionEnvServiceNoop from "../../../services/ExecutionEnvService/ExecutionEnvServiceNoop";
import { initialState } from "../../../state";
import { TOP_LEVEL_FILE_ANNOTATIONS } from "../../../constants";
import Annotation from "../../../entity/Annotation";
import { AnnotationType } from "../../../entity/AnnotationFormatter";

describe("<FileAnnotationList />", () => {
describe("file path representation", () => {
it("has both canonical file path and file path adjusted to OS & allen mount point", async () => {
it("has both cloud file path and local file path adjusted to OS & allen mount point", async () => {
// Arrange
const hostMountPoint = "/some/path";

Expand All @@ -24,6 +26,17 @@ describe("<FileAnnotationList />", () => {

const { store } = configureMockStore({
state: mergeState(initialState, {
metadata: {
annotations: [
...TOP_LEVEL_FILE_ANNOTATIONS,
new Annotation({
annotationName: "Local File Path",
annotationDisplayName: "File Path (Local VAST)",
description: "Path to file in on-premises storage.",
type: AnnotationType.STRING,
}),
],
},
interaction: {
platformDependentServices: {
executionEnvService: new FakeExecutionEnvService(),
Expand All @@ -33,19 +46,18 @@ describe("<FileAnnotationList />", () => {
});

const filePathInsideAllenDrive = "path/to/MyFile.txt";

const canonicalFilePath = `/allen/${filePathInsideAllenDrive}`;
const filePath = `production.files.allencell.org/${filePathInsideAllenDrive}`;
const fileDetails = new FileDetail({
file_path: canonicalFilePath,
file_path: filePath,
file_id: "abc123",
file_name: "MyFile.txt",
file_size: 7,
uploaded: "01/01/01",
annotations: [],
annotations: [
{ name: "Local File Path", values: [`/allen/${filePathInsideAllenDrive}`] },
],
});

const expectedLocalPath = `${hostMountPoint}/${filePathInsideAllenDrive}`;

// Act
const { findByText } = render(
<Provider store={store}>
Expand All @@ -54,17 +66,17 @@ describe("<FileAnnotationList />", () => {
);

// Assert
[
"File Path (Canonical)",
canonicalFilePath,
"File Path (Local)",
expectedLocalPath,
].forEach(async (cellText) => {
for (const cellText of [
"File Path (Cloud)",
`https://s3.us-west-2.amazonaws.com/${filePath}`,
"File Path (Local VAST)",
`${hostMountPoint}/${filePathInsideAllenDrive}`,
]) {
expect(await findByText(cellText)).to.not.be.undefined;
});
}
});

it("has only canonical file path when no allen mount point is found", () => {
it("has only cloud file path when no allen mount point is found", () => {
// Arrange
class FakeExecutionEnvService extends ExecutionEnvServiceNoop {
public formatPathForHost(posixPath: string): Promise<string> {
Expand All @@ -86,7 +98,7 @@ describe("<FileAnnotationList />", () => {
});

const filePathInsideAllenDrive = "path/to/MyFile.txt";
const filePath = `/allen/${filePathInsideAllenDrive}`;
const filePath = `production.files.allencell.org/${filePathInsideAllenDrive}`;
const fileDetails = new FileDetail({
file_path: filePath,
file_id: "abc123",
Expand All @@ -104,10 +116,12 @@ describe("<FileAnnotationList />", () => {
);

// Assert
expect(() => getByText("File Path (Local)")).to.throw();
["File Path (Canonical)", filePath].forEach((cellText) => {
expect(getByText(cellText)).to.not.be.undefined;
});
expect(() => getByText("File Path (Local VAST)")).to.throw();
["File Path (Cloud)", `https://s3.us-west-2.amazonaws.com/${filePath}`].forEach(
(cellText) => {
expect(getByText(cellText)).to.not.be.undefined;
}
);
});
});
});
Loading
Loading