Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Switch to Mina SSHD for the implementation of the SSH crawler #1225

Merged
merged 4 commits into from
Sep 12, 2024
Merged

Switch to Mina SSHD for the implementation of the SSH crawler #1225

merged 4 commits into from
Sep 12, 2024

Conversation

dadoonet
Copy link
Owner

@dadoonet dadoonet commented Aug 4, 2021 •
edited
Loading

We switch to https://github.com/apache/mina-sshd for the implementation of the SSH crawler.
It also helps us to now run integration tests.

@dadoonet dadoonet added the test Related to tests only label Aug 4, 2021
@dadoonet dadoonet added this to the 2.7 milestone Aug 4, 2021
@dadoonet dadoonet self-assigned this Aug 4, 2021
@dadoonet
Copy link
Owner Author

dadoonet commented Aug 5, 2021

After digging into this one, I found that as soon you are adding this dependency:

<dependency>
    <groupId>org.apache.sshd</groupId>
    <artifactId>sshd-sftp</artifactId>
    <version>2.7.0</version>
</dependency>

It generates 2 threads although you are not calling any kind of code from this dependency.
The JVM seems to be loading a default class and run it.

 1> 11:15:07,166 ERROR [RandomizedRunner] 2 threads leaked from SUITE scope at fr.pilato.elasticsearch.crawler.fs.test.integration.elasticsearch.FsCrawlerTestSshIT: 
  1>    1) Thread[id=27, name=sshd-SshClient[7f58f5ef]-timer-thread-1, state=TIMED_WAITING, group=TGRP-FsCrawlerTestSshIT]
  1>         at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
  1>         at [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  1>         at [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1661)
  1>         at [email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  1>         at [email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  1>         at [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1056)
  1>         at [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1116)
  1>         at [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
  1>         at [email protected]/java.lang.Thread.run(Thread.java:832)
  1>    2) Thread[id=28, name=Thread-0, state=RUNNABLE, group=TGRP-FsCrawlerTestSshIT]
  1>         at [email protected]/sun.nio.ch.KQueue.poll(Native Method)
  1>         at [email protected]/sun.nio.ch.KQueuePort$EventHandlerTask.poll(KQueuePort.java:200)
  1>         at [email protected]/sun.nio.ch.KQueuePort$EventHandlerTask.run(KQueuePort.java:287)
  1>         at [email protected]/java.lang.Thread.run(Thread.java:832)

@dadoonet dadoonet modified the milestones: 2.7, 2.8 Aug 5, 2021
@dadoonet dadoonet modified the milestones: 2.8, 2.9 Dec 14, 2021
@dadoonet dadoonet modified the milestones: 2.9, 2.10 Jan 10, 2022
@dadoonet dadoonet added update When updating an existing feature dependencies Pull requests that update a dependency file labels Aug 30, 2024
@dadoonet dadoonet changed the title Add tests for SSH Service Switch to Mina SSHD for the implementation of the SSH crawler Aug 30, 2024
@dadoonet
Try to reproduce the Zombie Thread pb
f38d6e2
@dadoonet
Solve the Zombie threads issue
41eb711 
This was due to the fact we were using this line to decompress a zip file:

```java
FileSystems.newFileSystem(zipFile, zipProperties)
```

This apparently loads and starts some classes behind the scene, which we don't want.
github-advanced-security[bot]
github-advanced-security bot found potential problems Sep 10, 2024
View reviewed changes
framework/src/main/java/fr/pilato/elasticsearch/crawler/fs/framework/FsCrawlerUtil.java Outdated
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
Files.copy(file, destination.resolve(rootPath.relativize(file).toString()), StandardCopyOption.COPY_ATTRIBUTES, StandardCopyOption.REPLACE_EXISTING);
return FileVisitResult.CONTINUE;
try (InputStream is = jar.getInputStream(file); FileOutputStream fos = new FileOutputStream(f)) {

Check failure

Code scanning / SonarCloud

Extracting archives should not lead to zip slip vulnerabilities High

Change this code to not construct the path from file name entry of an archive. See more on SonarCloud
@sonarqubecloud SonarQubeCloud
Copy link

Quality Gate Failed Quality Gate failed

Failed conditions
E Security Rating on New Code (required ≥ A)

See analysis details on SonarCloud

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

@dadoonet dadoonet merged commit 779a38c into master Sep 12, 2024
13 of 14 checks passed
@dadoonet dadoonet deleted the test/ssh branch September 12, 2024 08:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@dadoonet dadoonet

Labels
dependencies Pull requests that update a dependency file test Related to tests only update When updating an existing feature
Projects
None yet
Milestone
2.10
Development

Successfully merging this pull request may close these issues.
1 participant
@dadoonet