-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix listing of files in AlluxioFileSystem #23815
Conversation
Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: tanyajun.
|
Could you add a test for this problem ? |
lib/trino-filesystem-alluxio/src/main/java/io/trino/filesystem/alluxio/AlluxioFileIterator.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@awkwardd Add the following extractSchema
method to AlluxioUtils
, and update the next()
method in AlluxioFileIterator
can pass the test successfully. Please help update the code in this PR.
public static Location extractSchema(Location location)
{
String path = location.toString();
if (path.startsWith("alluxio://")) {
int index = path.indexOf("/", "alluxio://".length());
if (index != -1) {
String schema = path.substring(0, index);
return Location.of(schema + "/");
}
else {
return location;
}
}
return location;
}
public FileEntry next()
throws IOException
{
if (!hasNext()) {
return null;
}
URIStatus fileStatus = files.next();
String path = fileStatus.getPath();
if (path.startsWith("/")) {
path = path.substring(1);
}
Location schema = extractSchema(dirLocation);
Location fileLocation = schema.appendSuffix(path);
return new FileEntry(
fileLocation,
fileStatus.getLength(),
Instant.ofEpochMilli(fileStatus.getLastModificationTimeMs()),
Optional.empty());
}
Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: tanyajun.
|
@awkwardd you may also need to sign the CLA https://trino.io/development/process#contribution-process |
I have signed the cla file and snet it last night. |
@cla-bot check |
Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: tanyajun.
|
The cla-bot has been summoned, and re-checked this pull request! |
@awkwardd Please take a look at #23815 (comment) |
e580e63
to
d9b84d2
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This change looks good to me. @raunaqmorarka Could you help merge this PR?
d9b84d2
to
370b1c0
Compare
Can we add a test here ? The existing tests didn't catch this right ? |
This pull request has gone a while without any activity. Tagging for triage help: @mosabua |
Co-authored by: JiamingMai <[email protected]>
370b1c0
to
26d7b0e
Compare
Description
When use alluxio as hive fs,cant get data from alluxio.
I finaly found that the AlluxioIterator.java didn't generate correct file location as alluxio://192...../dir/filename but alluxio:///dir/filename
Additional context and related issues
Fixes #23778
Release notes
( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text: