-
Notifications
You must be signed in to change notification settings - Fork 357
Build with Pivotal HD 1.1
If you build your own applications with Maven targeting the Pivotal HD 1.1 distribution then you can use the Hadoop artifacts that are now available in the Spring IO Release Repository. The version to use for the artifacts is 2.0.5-alpha-gphd-2.1.0.0.
You need to add the following repository to your Maven pom:
<repository>
<id>spring-releases</id>
<name>Spring Release Repository</name>
<url>http://repo.spring.io/libs-release</url>
</repository>
We have recently started publishing "flavors" of the Spring for Apache Hadoop project built against different Hadoop distributions. This allows for better transitive dependency resolution.
The dependency to use for the Pivotal HD 1.1 "flavor" of spring-data-hadoop is:
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-hadoop</artifactId>
<version>2.0.4.RELEASE-phd1</version>
</dependency>
If you are submitting YARN jobs from your application, then you should also add the following dependency to your applications pom:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-jobclient</artifactId>
<version>2.0.5-alpha-gphd-2.1.0.0</version>
<scope>runtime</scope>
</dependency>
Note: The phd1 suffix for the spring-data-hadoop version will select the correct build that resolves the transitive dependencies for Pivotal HD 1.1.
If you prefer to use Spring for Apapche Hadoop version 1.x when building your own applications with Maven targeting Pivotal HD 1.1 then you can use the Spring for Apache Hadoop 1.1 release using a version of 1.1.0.RELEASE-phd1.
We have recently started publishing "flavors" of the Spring for Apache Hadoop project built against different Hadoop distributions. This allows for better transitive dependency resolution.
The dependency to use for the Pivotal HD 1.1 "flavor" of spring-data-hadoop is:
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-hadoop</artifactId>
<version>1.1.0.RELEASE-phd1</version>
</dependency>
If you are submitting YARN jobs from your application, then you should also add the following dependency to your applications pom:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-jobclient</artifactId>
<version>2.0.5-alpha-gphd-2.1.0.0</version>
<scope>runtime</scope>
</dependency>
Note: The phd1 suffix for the spring-data-hadoop version will select the correct build that resolves the transitive dependencies for Pivotal HD 1.1.
With the repository mentioned above added and by using 2.0.5-alpha-gphd-2.1.0.0 as the version for any Hadoop dependencies the correct jars should be resolved.