diff --git a/README.md b/README.md index 9f4f6fece..77679bffc 100644 --- a/README.md +++ b/README.md @@ -74,12 +74,11 @@ Fiware/Cygnus has four sub-modules [cygnus-common](cygnus-common#welcome-to-cygn Following are the links of install section of Fiware/Cygnus sub-modules: * cygnus-common: - [Install from sources](doc/cygnus-common/installation_and_administration_guide/install_from_sources.md) - - [Install with docker](doc/cygnus-common/installation_and_administration_guide/install_with_docker.md) - - [Install with rpm](doc/cygnus-common/installation_and_administration_guide/install_with_rpm.md) + - [Install with docker](doc/cygnus-common/installation_and_administration_guide/install_with_docker.md) * cygnus-ngsi: - [Install from sources](doc/cygnus-ngsi/installation_and_administration_guide/install_from_sources.md) - [Install with docker](doc/cygnus-ngsi/installation_and_administration_guide/install_with_docker.md) - - [Install with rpm](doc/cygnus-ngsi/installation_and_administration_guide/install_with_rpm.md) + * cygnus-twitter: - [Install from sources](doc/cygnus-twitter/installation_and_administration_guide/install_from_sources.md) - [Install with docker](doc/cygnus-twitter/installation_and_administration_guide/install_with_docker.md) diff --git a/cygnus-ngsi-ld/README.md b/cygnus-ngsi-ld/README.md index 47d7625ed..e670151f9 100755 --- a/cygnus-ngsi-ld/README.md +++ b/cygnus-ngsi-ld/README.md @@ -206,7 +206,7 @@ $ curl -X GET "http://localhost:8081/v1/stats" | python -m json.tool ## Advanced topics and further reading Detailed information regarding cygus-ngsi can be found in the [Installation and Administration Guide](../doc/cygnus-ngsi-ld/installation_and_administration_guide/introduction.md), the [User and Programmer Guide](../doc/cygnus-ngsi-ld/user_and_programmer_guide/introduction.md) and the [Flume extensions catalogue](../doc/cygnus-ngsi-ld/flume_extensions_catalogue/introduction.md). The following is just a list of shortcuts regarding the most popular topics: -* [Installation with docker](../doc/cygnus-ngsi-ld/installation_and_administration_guide/install_with_docker.md). An alternative to RPM installation, docker is one of the main options when installing FIWARE components. +* [Installation with docker](../doc/cygnus-ngsi-ld/installation_and_administration_guide/install_with_docker.md). The preffered installation way, docker is one of the main options when installing FIWARE components. * [Installation from sources](../doc/cygnus-ngsi-ld/installation_and_administration_guide/install_from_sources.md). Sometimes you will need to install from sources, particularly when some of the dependencies must be modified. [Top](#top) diff --git a/cygnus-ngsi/README.md b/cygnus-ngsi/README.md index 12f0475ff..1e87eaffb 100644 --- a/cygnus-ngsi/README.md +++ b/cygnus-ngsi/README.md @@ -226,7 +226,7 @@ Many other operations, like getting/setting the log level, can be found in Manag ## Advanced topics and further reading Detailed information regarding cygus-ngsi can be found in the [Installation and Administration Guide](../doc/cygnus-ngsi/installation_and_administration_guide/introduction.md), the [User and Programmer Guide](../doc/cygnus-ngsi/user_and_programmer_guide/introduction.md) and the [Flume extensions catalogue](../doc/cygnus-ngsi/flume_extensions_catalogue/introduction.md). The following is just a list of shortcuts regarding the most popular topics: -* [Installation with docker](../doc/cygnus-ngsi/installation_and_administration_guide/install_with_docker). An alternative to RPM installation, docker is one of the main options when installing FIWARE components. +* [Installation with docker](../doc/cygnus-ngsi/installation_and_administration_guide/install_with_docker). Thre preferred installation method, docker is one of the main options when installing FIWARE components. * [Installation from sources](../doc/cygnus-ngsi/installation_and_administration_guide/install_from_sources.md). Sometimes you will need to install from sources, particularly when some of the dependencies must be modified, e.g. the `hadoop-core` libraries. * [Running as a process](../doc/cygnus-ngsi/installation_and_administration_guide/running_as_process.md). Running cygus-ngsi as a process is very useful for testing and debugging purposes. * [Management Interface](../doc/cygnus-ngsi/installation_and_administration_guide/management_interface.md). REST-based management interface for administration purposes. diff --git a/doc/contributing/contributing_guidelines.md b/doc/contributing/contributing_guidelines.md index 4060d90ad..8de30339e 100644 --- a/doc/contributing/contributing_guidelines.md +++ b/doc/contributing/contributing_guidelines.md @@ -16,7 +16,6 @@ Content: * [Commits and squashing](#section3.8) * [Releasing](#section3.9) * [Deployers](#section4) - * [RPMs](#section4.1) * [Dockers](#section4.2) * [Documentation](#section5) * [Repository documentation](#section5.1) @@ -120,7 +119,6 @@ Each folder MUST have, at least, the following subdirectories and files: * `docker/` → everything about deploying Cygnus by means of Docker * `test/` → acceptance tests, e2e tests, performance tests, others * `conf/` → templates for configuration files required to run the agent -* `spec/` → spec file for generating a RPM * `pom.xml` → Maven’s Project Object Model A folder with common content named `cygnus-common` MUST exist. It will be a Maven project in charge of building a Cygnus common library that SHOULD be used by all the agents, enforcing the reusability of code. @@ -254,17 +252,6 @@ Releases MUST be published in the releases section of the main repository As a result of the release, `CHANGES_NEXT_RELEASE` file MUST be emptied in Github repo. ## Deployers -### RPMs -There MUST exist a `rpm/` folder at the root of the main repository. A packaging script MUST generate a RPM based on the spec file of each Cygnus agent, including `cygnus-common`. Such a spec file MUST live at the `spec` subfolder within the agent folder. - -Upon releasing, these RPMs MUST be created and uploaded to some repository in order they are available. As an example, `cygnus-ngsi` agent's RPM is uploaded to `http://repositories.testbed.fiware.org`. - -Agents' RPMs MUST depend on `cygnus-common` RPM, which MUST be in charge of installing not only the common classes to all the agents, but installing Apache Flume, default configuration templates and provisioning the Cygnus plugin. `cygnus-common` RPM is typically uploaded to `http://repositories.testbed.fiware.org` as well. - -All RPMs spec files (spec for `cygnus-common` and any other agent) MUST contain a copy of the content of `CHANGES_NEXT_RELEASE` file. - -[Top](#top) - ### Dockers There MUST exist a `docker/` folder at the root of the main repository. Every Cygnus agent MUST include a docker subfolder as per the following rules: @@ -391,6 +378,6 @@ In addition, a table MUST be included in charge of defining a set of alarm condi ## Configuration When adding a new agent to Cygnus, it MUST include an agent configuration template in [Flume format](https://flume.apache.org/FlumeUserGuide.html#setup). Other configuration files MAY be added as well. -The specific agent configuration template MUST replace the one handled by `cygnus-common` in the Flume deployment donde by `cygnus-common` RPM. +The specific agent configuration template MUST replace the one handled by `cygnus-common` in the Flume deployment done by `cygnus-common`. [Top](#top) diff --git a/doc/cygnus-common/installation_and_administration_guide/README.md b/doc/cygnus-common/installation_and_administration_guide/README.md index 5de3e9ab7..3c958ad9a 100644 --- a/doc/cygnus-common/installation_and_administration_guide/README.md +++ b/doc/cygnus-common/installation_and_administration_guide/README.md @@ -10,7 +10,6 @@ * [log4j configuration](./log4j_conf.md) * Running: * [Running as a process](./running_as_process.md) - * [Running as a service](./running_as_service.md) * [YAFS](./yafs.md) * [Testing](./testing.md) * [Logs and alarm](./logs_and_alarms.md) diff --git a/doc/cygnus-common/installation_and_administration_guide/install_from_sources.md b/doc/cygnus-common/installation_and_administration_guide/install_from_sources.md index b6989c3ee..9925f5cad 100644 --- a/doc/cygnus-common/installation_and_administration_guide/install_from_sources.md +++ b/doc/cygnus-common/installation_and_administration_guide/install_from_sources.md @@ -34,7 +34,7 @@ Maven is installed by downloading it from [maven.apache.org](http://maven.apache [Top](#top) ## `cygnus` user creation -It is highly recommended to create a `cygnus` Unix user, under which Cygnus will be installed and run. By the way, this is how the [RPM](./install_with_rpm.md) proceeds. +It is highly recommended to create a `cygnus` Unix user, under which Cygnus will be installed and run. Creating such a user is quite simple. As a sudoer user (root or any other allowed), type the following: diff --git a/doc/cygnus-common/installation_and_administration_guide/install_with_rpm.md b/doc/cygnus-common/installation_and_administration_guide/install_with_rpm.md deleted file mode 100644 index 35edc6d03..000000000 --- a/doc/cygnus-common/installation_and_administration_guide/install_with_rpm.md +++ /dev/null @@ -1,11 +0,0 @@ -# Installing cygnus-common with RPM (CentOS/RedHat) - -Simply configure the FIWARE release repository if not yet configured: -``` -sudo wget -P /etc/yum.repos.d/ https://nexus.lab.fiware.org/repository/raw/public/repositories/el/7/x86_64/fiware-release.repo -``` -And use your applications manager in order to install the latest version of cygnus-common: -``` -sudo yum install cygnus-common -``` -The above will install cygnus-common at `/usr/cygnus/`. diff --git a/doc/cygnus-common/installation_and_administration_guide/running_as_service.md b/doc/cygnus-common/installation_and_administration_guide/running_as_service.md deleted file mode 100644 index 061f866a5..000000000 --- a/doc/cygnus-common/installation_and_administration_guide/running_as_service.md +++ /dev/null @@ -1,24 +0,0 @@ -# Running cygnus-common as a service -**Note**: Cygnus can only be run as a service if you installed it through the RPM. - -Running cygnus-common is the same than running a plain Flume. Once the `cygnus_instance_.conf` and `agent_.conf` files are properly configured, just use the `service` command to start, restart, stop or get the status (as a sudoer): - - $ sudo service cygnus status - - $ sudo service cygnus start - - $ sudo service cygnus restart - - $ sudo service cygnus stop - -Previous commands affects to **all** of Cygnus instances configured. If only one instance is wanted to be managed by the service script then the instance identifier after the action must be specified: - - $ sudo service cygnus status - - $ sudo service cygnus start - - $ sudo service cygnus restart - - $ sudo service cygnus stop - -Where `` is the suffix at the end of the `cygnus_instace_.conf` or `agent_.conf` files you used to configure the instance. diff --git a/doc/cygnus-common/installation_and_administration_guide/yafs.md b/doc/cygnus-common/installation_and_administration_guide/yafs.md index 5c95a9f2a..c5bee8216 100644 --- a/doc/cygnus-common/installation_and_administration_guide/yafs.md +++ b/doc/cygnus-common/installation_and_administration_guide/yafs.md @@ -7,7 +7,7 @@ Nevertheless, using YAFS has side effects, specially when a thread dies as part Of special interest is the case involving a configuration change on the fly. Flume is designed for that purpose, nevertheless using YAFS produce very weird results in conjunction with on the fly configuration changes. This is because reloading a configuration in Flume implies to stop previous running components, i.e. thread stops that YAFS detects and result in a complete Cygnus stop. -In order to avoid scenarios as the ones described above, YAFS can be disabled when configuring the agent, by simply using the `--no-yafs` option. Please check the sections about running a Cygnus agent as a [process](./runninh_as_process.md) or as a [service](./running_as_service.md) for further details. +In order to avoid scenarios as the ones described above, YAFS can be disabled when configuring the agent, by simply using the `--no-yafs` option. Please check the sections about running a Cygnus agent as a [process](./runninh_as_process.md) for further details. **NOTE**: Please observe Flume 1.4.0 shows a problem when changing the configuration on the fly, as described in [FLUME-2310](https://issues.apache.org/jira/browse/FLUME-2310). Basically, new components cannot register for JMX monitoring because old ones are not properly deregistered. This problem is inherited by Cygnus since it extends Flume. In any case, it is a minor issue not affecting Flume nor Cygnus functionalityu (except for proper JMX monitoring, of course). You will see this error in Cygnus in traces like this one: diff --git a/doc/cygnus-ngsi-ld/installation_and_administration_guide/README.md b/doc/cygnus-ngsi-ld/installation_and_administration_guide/README.md index 78194f0a8..8d9682f43 100644 --- a/doc/cygnus-ngsi-ld/installation_and_administration_guide/README.md +++ b/doc/cygnus-ngsi-ld/installation_and_administration_guide/README.md @@ -10,7 +10,6 @@ * log4j configuration (FIXME: we don't have yet and specific configuration for log4j but it will be done in the future) * Running: * [Running as a process](./running_as_process.md) - * [Running as a service](./running_as_service.md) * Advanced topics: * [Testing](./testing.md) * [Sanity checks](./sanity_checks.md) diff --git a/doc/cygnus-ngsi-ld/installation_and_administration_guide/introduction.md b/doc/cygnus-ngsi-ld/installation_and_administration_guide/introduction.md index 5e36d072e..0b0dd7ac3 100644 --- a/doc/cygnus-ngsi-ld/installation_and_administration_guide/introduction.md +++ b/doc/cygnus-ngsi-ld/installation_and_administration_guide/introduction.md @@ -33,7 +33,7 @@ Apart from this introduction, this Installation and Administration Guide mainly * [Configuration examples](./configuration_examples.md) * Running: * [Running as a process](./running_as_process.md) - * [Running as a service](./running_as_service.md) + * Advanced topics: * [Testing](./testing.md) * [Sanity checks](./sanity_checks.md) diff --git a/doc/cygnus-ngsi-ld/installation_and_administration_guide/running_as_service.md b/doc/cygnus-ngsi-ld/installation_and_administration_guide/running_as_service.md deleted file mode 100644 index 5f7b9187f..000000000 --- a/doc/cygnus-ngsi-ld/installation_and_administration_guide/running_as_service.md +++ /dev/null @@ -1,24 +0,0 @@ -# Running cygnus-ngsi-ld as a service -**Note**: Cygnus can only be run as a service if you installed it through the RPM. - -Once the `cygnus_instance_.conf` and `agent_.conf` files are properly configured, just use the `service` command to start, restart, stop or get the status (as a sudoer): - - $ sudo service cygnus status - - $ sudo service cygnus start - - $ sudo service cygnus restart - - $ sudo service cygnus stop - -Previous commands affects to **all** of Cygnus instances configured. If only one instance is wanted to be managed by the service script then the instance identifier after the action must be specified: - - $ sudo service cygnus status - - $ sudo service cygnus start - - $ sudo service cygnus restart - - $ sudo service cygnus stop - -Where `` is the suffix at the end of the `cygnus_instace_.conf` or `agent_.conf` files you used to configure the instance. diff --git a/doc/cygnus-ngsi/installation_and_administration_guide/README.md b/doc/cygnus-ngsi/installation_and_administration_guide/README.md index e27bc767a..2ce622826 100644 --- a/doc/cygnus-ngsi/installation_and_administration_guide/README.md +++ b/doc/cygnus-ngsi/installation_and_administration_guide/README.md @@ -2,7 +2,6 @@ * [Introduction](./introduction.md) * Installation: - * [Installation via RPM](./install_with_rpm.md) * [Installation via docker](./install_with_docker.md) * [Installation from sources](./install_from_sources.md) * Configuration: @@ -11,7 +10,6 @@ * [Configuration examples](./configuration_examples.md) * Running: * [Running as a process](./running_as_process.md) - * [Running as a service](./running_as_service.md) * Advanced topics: * [Testing](./testing.md) * [Backends as short-term historics](./backends_as_sth.md) diff --git a/doc/cygnus-ngsi/installation_and_administration_guide/install_with_rpm.md b/doc/cygnus-ngsi/installation_and_administration_guide/install_with_rpm.md deleted file mode 100644 index 8b97b0f44..000000000 --- a/doc/cygnus-ngsi/installation_and_administration_guide/install_with_rpm.md +++ /dev/null @@ -1,11 +0,0 @@ -# Installing cygnus-ngsi with RPM (CentOS/RedHat) - -Simply configure the FIWARE release repository if not yet configured: -``` -sudo wget -P /etc/yum.repos.d/ https://nexus.lab.fiware.org/repository/raw/public/repositories/el/7/x86_64/fiware-release.repo -``` -And use your applications manager in order to install the latest version of cygnus-ngsi: -``` -sudo yum install cygnus-ngsi -``` -The above will install cygnus-ngsi at `/usr/cygnus/`, together with cygnus-common (it contains all the common dependencies to all Cygnus agents, NGSI agent included). diff --git a/doc/cygnus-ngsi/installation_and_administration_guide/running_as_service.md b/doc/cygnus-ngsi/installation_and_administration_guide/running_as_service.md deleted file mode 100644 index c75a2a6df..000000000 --- a/doc/cygnus-ngsi/installation_and_administration_guide/running_as_service.md +++ /dev/null @@ -1,24 +0,0 @@ -# Running cygnus-ngsi as a service -**Note**: Cygnus can only be run as a service if you installed it through the RPM. - -Once the `cygnus_instance_.conf` and `agent_.conf` files are properly configured, just use the `service` command to start, restart, stop or get the status (as a sudoer): - - $ sudo service cygnus status - - $ sudo service cygnus start - - $ sudo service cygnus restart - - $ sudo service cygnus stop - -Previous commands affects to **all** of Cygnus instances configured. If only one instance is wanted to be managed by the service script then the instance identifier after the action must be specified: - - $ sudo service cygnus status - - $ sudo service cygnus start - - $ sudo service cygnus restart - - $ sudo service cygnus stop - -Where `` is the suffix at the end of the `cygnus_instace_.conf` or `agent_.conf` files you used to configure the instance. diff --git a/doc/cygnus-ngsi/integration/orion_cygnus_spark.md b/doc/cygnus-ngsi/integration/orion_cygnus_spark.md index 9bf87668f..60485adbc 100644 --- a/doc/cygnus-ngsi/integration/orion_cygnus_spark.md +++ b/doc/cygnus-ngsi/integration/orion_cygnus_spark.md @@ -92,7 +92,7 @@ Please refere to Orion Context Broker [official documentation](http://fiware-ori [Top](#top) ## Setting up Cygnus -Regarding installation, do it from [FIWARE yum repository](https://github.com/telefonicaid/fiware-cygnus/blob/master/doc/cygnus-ngsi/installation_and_administration_guide/install_with_rpm.md). Once installed in its latest version, a Cygnus agent must be configured as follows: +Regarding installation, do it [using Dockerhub](https://github.com/telefonicaid/fiware-cygnus/blob/master/doc/cygnus-ngsi/installation_and_administration_guide/install_with_docker.md). Once installed in its latest version, a Cygnus agent must be configured as follows: ``` $ cat /usr/cygnus/conf/agent_spark.conf diff --git a/mkdocs.yml b/mkdocs.yml index d2c3ee905..0df1dc3c8 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -16,8 +16,7 @@ pages: - 'cygnus-common library': - 'Installation and Administration Guide': - 'Introduction': 'cygnus-common/installation_and_administration_guide/introduction.md' - - 'Hardware requirements': 'cygnus-common/installation_and_administration_guide/hw_requirements.md' - - 'Installation via RPM': 'cygnus-common/installation_and_administration_guide/install_with_rpm.md' + - 'Hardware requirements': 'cygnus-common/installation_and_administration_guide/hw_requirements.md' - 'Installation from sources': 'cygnus-common/installation_and_administration_guide/install_from_sources.md' - 'Installation via docker': 'cygnus-common/installation_and_administration_guide/install_with_docker.md' - 'Cygnus agent configuration': 'cygnus-common/installation_and_administration_guide/cygnus_agent_conf.md' @@ -50,12 +49,10 @@ pages: - 'Quick Start Guide': 'cygnus-ngsi/quick_start_guide.md' - 'Installation and Administration Guide': - 'Introduction': 'cygnus-ngsi/installation_and_administration_guide/introduction.md' - - 'Installation via RPM': 'cygnus-ngsi/installation_and_administration_guide/install_with_rpm.md' - 'Installation via docker': 'cygnus-ngsi/installation_and_administration_guide/install_with_docker.md' - 'Installation from sources': 'cygnus-ngsi/installation_and_administration_guide/install_from_sources.md' - 'NGSI agent configuration': 'cygnus-ngsi/installation_and_administration_guide/ngsi_agent_conf.md' - 'Configuration examples': 'cygnus-ngsi/installation_and_administration_guide/configuration_examples.md' - - 'Running as a service': 'cygnus-ngsi/installation_and_administration_guide/running_as_service.md' - 'Testing': 'cygnus-ngsi/installation_and_administration_guide/testing.md' - 'Backends as short-term historics': 'cygnus-ngsi/installation_and_administration_guide/backends_as_sth.md' - 'Name mappings': 'cygnus-ngsi/installation_and_administration_guide/name_mappings.md' diff --git a/rpm/README.md b/rpm/README.md deleted file mode 100644 index 1fc1e5c2c..000000000 --- a/rpm/README.md +++ /dev/null @@ -1,53 +0,0 @@ -In order to build the RPM packages, follow the following steps: - -* Build the Cygnus .jars both _common_ and _ngsi_ with `mvn clean compile exec:exec assembly:single` (see [Cygnus-common README](../cygnus-common/README.md) - and [Cygnus-NGSI README](../cygnus-ngsi/README.md) for additional detail). -* Run the `package.sh` script (inside the `rpm` directory) which, upon finalization, will generate the `.rpm`'s - files in the `cygnus-common/spec/RPMS/x86_64` and `cygnus-ngsi/spec/RPMS/x86_64` directories. You must specify the version number (matching with the one in the pom.xml - file) with the `-v` argument. Release for the rpm package must also specified whit `-r` option. - ``` - ./package.sh -v 0.13.0 -r 5.ge58dffa - ``` - You can see full options of script package.sh typing `./package.sh -h` - - The results packages contains: - - - `cygnus-common/spec/RPMS/x86_64`: has the Cygnus common features and the Apache Flume SW - - `cygnus-ngsi/spec/RPMS/x86_64`: has only the jar and config templates for NGSI connector. - -When a package is built it introduces three features in Cygnus SW: - -* Daily log rotation: use logrotate tool to perform this operation. View - [logrotate config file](../cygnus-common/spec/SOURCES/logrotate.d/logrotate-cygnus-daily) for more info -* Automatic deletion of files older than 30 days programmed in cron: - [cron file](../cygnus-common/spec/SOURCES/cron.d/cleanup_old_cygnus_logfiles) -* init.d service script: which is a common service script (named as `cygnus`) with the usual operations - `start`, `stop`, `restart` and `status`. It is invoked as other service script: `service cygnus start` - and this actilon starts **all** instances configured for Cygnus but service script has a special - feature: if a second parameter is provided (after the action) the script only acts over the - instance indicated. I.E. - `service cygnus start test1` In this case service script tries to start instance of cygnus called test1. If test1 - instance is not configured a special error will be shown. - `service cygnus stop test1` Only test1 will be stopped. And so for `restart` and `status` operations. - -**Note:** due to a bug in rpm generation ([issue #329](https://github.com/telefonicaid/fiware-cygnus/issues/329)) all packages -that were built before 2015/03/05 and installed should be erased manually and install a new one build after 2015/03/05. - -The date of the building of the installed package can be viewed with `rpm -qi cygnus` and look into **Build Date** field: - -```shell -$ rpm -qi cygnus -Name : cygnus Relocations: (not relocatable) -Version : 0.6.0 Vendor: Telefonica I+D -Release : 35.g26e07c4 Build Date: Mon 16 Feb 2015 03:56:58 PM CET -Install Date: Mon 16 Feb 2015 04:14:04 PM CET Build Host: ci-iot-deven-01 -Group : Applications/cygnus Source RPM: cygnus-0.6.0-35.g26e07c4.src.rpm -Size : 107280119 License: AGPLv3 -Signature : (none) -Summary : Package for cygnus component -Description : -This connector is a (conceptual) derivative work of ngsi2cosmos, and implements -a Flume-based connector for context data coming from Orion Context Broker. -``` - -All packages build before 2015/03/05 should be erased and generated again. diff --git a/rpm/colors_shell.sh b/rpm/colors_shell.sh deleted file mode 100755 index 3300474a1..000000000 --- a/rpm/colors_shell.sh +++ /dev/null @@ -1,82 +0,0 @@ -#!/bin/bash - -# Copyright 2014 Telefonica Investigación y Desarrollo, S.A.U -# -# This file is part of fiware-cygnus (FI-WARE project). -# -# fiware-cygnus is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General -# Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any -# later version. -# fiware-cygnus is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied -# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more -# details. -# -# You should have received a copy of the GNU Affero General Public License along with fiware-cygnus. If not, see -# http://www.gnu.org/licenses/. -# -# For those usages not covered by the GNU Affero General Public License please contact with iot_support at tid dot es - -########################## -### COLORS FOR LOGS ### -########################## - -## LOG FUNCTIONS -_message(){ - _error_color="\\033[37m\\033[1;31m" # BG light grey, FG red - _stage_color="\\033[37m\\033[1;33m" # FG yellow - _ok_color="\\033[37m\\033[1;32m" # FG light green - _user_color="\\033[37m\\033[1;30m" # FG Dark Grey - _default_log_color_pre="\\033[0m\\033[1;34m" - _default_log_color_suffix="\\033[0m" - isError=$(echo "$1"|grep -i "\\[ERROR\\]") - isStage=$(echo "$1"|grep -i "\\[INFO\\]") - isOk=$(echo "$1"|grep -i "\\[OK\\]") - isUser=$(echo "$1"|grep -i "\\[USER\\]") - isDash=`echo -en|grep "\-en"` - outLog="/dev/stdout" - if ! [ -z "$isError" ]; then - messageColor=${_error_color} && outLog="/dev/stderr" - else - if ! [ -z "$isStage" ]; then - messageColor=${_stage_color} - elif ! [ -z "$isOk" ]; then - messageColor=${_ok_color} - elif ! [ -z "$isUser" ]; then - messageColor=${_user_color} - else - messageColor=${_default_log_color_pre} - fi - fi - [ -z "$isDash" ] && echo -en "${messageColor}$1${_default_log_color_suffix}\n" >$outLog|| echo "${messageColor}$1${_default_log_color_suffix}\n" >$outLog -} - -## -# Logs with colors into Console and, if $2 has value, -# without them into log file -# Input: -# *$1: Message -# $2: Log file -## -_log(){ - _message "[`date '+%X'`] $1" - # - if [[ -f "$2" ]]; then - echo "[`date '+%X'`] $1" >> "./$2" - fi -} - -_logError(){ - _log "[ERROR] $1" $2 -} - -_logStage(){ - _log "[INFO] $1" $2 -} - -_logOk(){ - _log "[OK] $1" $2 -} - -_logUser(){ - _log "[USER] $1" $2 -} diff --git a/rpm/get_version_string.sh b/rpm/get_version_string.sh deleted file mode 100755 index fcac316b9..000000000 --- a/rpm/get_version_string.sh +++ /dev/null @@ -1,108 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2013 Telefonica Investigacion y Desarrollo, S.A.U -# -# This file is part of fiware-cygnus (FI-WARE project). -# -# fiware-cygnus is free software: you can redistribute it and/or -# modify it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# fiware-cygnus is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero -# General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with fiware-cygnus. If not, see http://www.gnu.org/licenses/. -# -# For those usages not covered by this license please contact with -# iot_support at tid dot es - -# Bash lib to know the RPM version and revision from a GitHub repository -# Call method get_rpm_version_string to obtain them for rpmbuild -# - -if [[ $(ps -hp $$ | grep bash) ]]; then - shopt -s extglob -elif [[ $(ps -hp $$ | grep zsh) ]]; then - setopt kshglob -fi - -get_branch() -{ - git rev-parse --abbrev-ref HEAD -} - -## Specific functions according the TID workflow -get_branch_type() -{ - local branch="$(get_branch)" - case $branch in - release/*) echo "release";; - develop) echo "develop";; - master) echo "stable";; - *) echo "other";; - esac -} - -get_version_string() -{ - local branch branch_name describe_tags version ancestor release - case $(get_branch_type) in - stable) - # If we are on stable branch get last tag as the version, but transform to x.x.x-x-SHA1 - describe_tags="$(git describe --tags --long --match "[[:digit:]]*.[[:digit:]]*.[[:digit:]]*" 2>/dev/null)" - version="${describe_tags%-*-*}" - echo "${version%.*}-${version#*.*.*.}-$(git log --pretty=format:'%h' -1)" - ;; - develop) - ## If we are in develop use the total count of commits of the repo - total_commit_number=$(git rev-list --all --count) - short_hash=$(git rev-parse --short HEAD) - version="$(git describe --tags --long --match "[[:digit:]]*.[[:digit:]]*.[[:digit:]]*" 2>/dev/null)" - version="${version%-*-*}" - version="${version%KO}" - echo "${version}-${total_commit_number}-${short_hash}" - ;; - release) - ## in release branches the version is a tag named - branch_name="$(get_branch)" - branch_name="${branch_name#*/}" - describe_tags="$(git describe --tags --long --match ${branch_name} 2>/dev/null)" - version="${describe_tags%-*-*}" - version="${version%KO}" - release=${describe_tags#*.*.*-} - echo "${version}-${release}" - ;; - other) - ## We are in detached mode, use the last x-y-z tag - version="$(git describe --tags --long --match "[[:digit:]]*.[[:digit:]]*.[[:digit:]]*" 2>/dev/null)" - version="${version%-*-*}" - version="${version%KO}" - echo "${version}" - ;; - *) - # RMs don't stablish any standard here, we use branch name as version - version=$(get_branch) - # Using always develop as parent branch does not describe correctly the number of revision - # for branches not starting there, but works as an incremental rev - ancestor="$(git merge-base $version develop)" - version=${version#*/} - local res="$(git log --oneline ${ancestor}.. --pretty='format:%h')" - ## wc alone does not get the last line when there's no new line - [[ -z $res ]] && rel=0 || rel=$(echo "$res" | wc -l | tr -d ' ') - echo "${version}-${rel}-g$(git log --pretty=format:'%h' -1)" - esac -} - -get_rpm_version_string() { - local version_string ver rel - version_string="$(get_version_string)" - ver="${version_string%-*-*}" - rel="${version_string:$((${#ver}+1))}" - echo "${ver//[[:space:]-\/#]}" "${rel//[-]/.}" -} - -#cd $1 -#get_version_string| cut -d "-" -f $2 diff --git a/rpm/package.sh b/rpm/package.sh deleted file mode 100755 index 7ed17a560..000000000 --- a/rpm/package.sh +++ /dev/null @@ -1,261 +0,0 @@ -#!/usr/bin/env bash - -# Copyright 2014 Telefonica Investigación y Desarrollo, S.A.U -# -# This file is part of fiware-cygnus (FI-WARE project). -# -# fiware-cygnus is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General -# Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any -# later version. -# fiware-cygnus is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied -# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more -# details. -# -# You should have received a copy of the GNU Affero General Public License along with fiware-cygnus. If not, see -# http://www.gnu.org/licenses/. -# -# For those usages not covered by the GNU Affero General Public License please contact with iot_support at tid dot es - -################################################# -### Main script for Cygnus RPM Stage ### -################################################# - -function download_flume(){ - # download artifact from external URL and unzip it into ${RPM_BASE_DIR} - _logStage "######## Preparing the apache-flume component... ########" - - local ARTIFACT_FLUME_URL=${1} - local FLUME_TAR=${2} - - local TMP_DIR="tmp_deleteme" - mkdir -p ${TMP_DIR} - pushd ${TMP_DIR} &> /dev/null - #remove .tar.gz so twice is executed - local FLUME_WO_TAR=${FLUME_TAR%.*} - FLUME_WO_TAR=${FLUME_WO_TAR%.*} - - - _log "#### The version of the component is (${FLUME_TAR}) ####" - _log "#### Downloading apache-flume: ${FLUME_TAR}... ####" - curl -s -o ${FLUME_TAR} ${ARTIFACT_FLUME_URL}/${FLUME_TAR} - if [[ $? -ne 0 ]]; then - _logError "cannot download apache-flume.tar.gz (${FLUME_TAR}) from ${ARTIFACT_FLUME_URL}" - return 1 - else - _logOk ".............. Done! .............." - fi - - _log "#### Uncompresing apache-flume: ${FLUME_TAR}... ####" - tar xvzf ${FLUME_TAR} &> /dev/null - if [[ $? -ne 0 ]]; then - _logError ".............. Cannot untar flume.tar.gz (${FLUME_TAR}) .............." - return 1 - else - _logOk ".............. Done! .............." - fi - - - _log "### Disable httpclient and httpcore libraries distributed within apache-flume bundle... ###" - mv ${FLUME_WO_TAR}/lib/httpclient-4.2.1.jar ${FLUME_WO_TAR}/lib/httpclient-4.2.1.jar.old - mv ${FLUME_WO_TAR}/lib/httpcore-4.2.1.jar ${FLUME_WO_TAR}/lib/httpcore-4.2.1.jar.old - - _log "### Disable the bundled version of libthrift within Apache Flume... ###" - mv ${FLUME_WO_TAR}/lib/libthrift-0.7.0.jar ${FLUME_WO_TAR}/lib/libthrift-0.7.0.old - - _log "#### Cleaning the temporal folders... ####" - rm -rf ${RPM_SOURCE_DIR}/${FLUME_WO_TAR} - rm -rf ${FLUME_WO_TAR}/docs # erase flume documentation - rm ${FLUME_WO_TAR}/conf/flume-conf.properties.template # we will add our own templates - rm ${FLUME_WO_TAR}/conf/log4j.properties # we will add our own templates - rm -rf ${RPM_PRODUCT_SOURCE_DIR} - mkdir -p ${RPM_PRODUCT_SOURCE_DIR} - cp -R ${FLUME_WO_TAR}/* ${RPM_PRODUCT_SOURCE_DIR}/ - popd &> /dev/null - rm -rf ${TMP_DIR} - return 0 - - _logStage "######## The apache-flume is ready for use! ... ########" -} - -function copy_cygnus_startup_script(){ - _logStage "######## Copying the cygnus startup script into the apache-flume... ########" - rm -rf ${RPM_PRODUCT_SOURCE_DIR}/bin/cygnus-flume-ng - cp ${BASE_DIR}/cygnus-common/src/main/resources/cygnus-flume-ng ${RPM_PRODUCT_SOURCE_DIR}/bin -} - -function copy_cygnus_jar_to_flume_directory(){ - local component_name=${1} - _logStage "######## Copying the ${component_name} jar into the apache-flume... ########" - mkdir -p ${RPM_PRODUCT_SOURCE_DIR}/plugins.d/cygnus - if [[ ${component_name} == "cygnus-common" ]]; then - mkdir ${RPM_PRODUCT_SOURCE_DIR}/plugins.d/cygnus/libext - cp $BASE_DIR/${component_name}/target/${component_name}-${PRODUCT_VERSION}-jar-with-dependencies.jar ${RPM_PRODUCT_SOURCE_DIR}/plugins.d/cygnus/libext - elif [[ ${component_name} == "cygnus-ngsi" ]]; then - mkdir ${RPM_PRODUCT_SOURCE_DIR}/plugins.d/cygnus/lib - cp $BASE_DIR/${component_name}/target/${component_name}-${PRODUCT_VERSION}-jar-with-all-dependencies.jar ${RPM_PRODUCT_SOURCE_DIR}/plugins.d/cygnus/lib - else - mkdir ${RPM_PRODUCT_SOURCE_DIR}/plugins.d/cygnus/lib - cp $BASE_DIR/${component_name}/target/${component_name}-${PRODUCT_VERSION}-jar-with-dependencies.jar ${RPM_PRODUCT_SOURCE_DIR}/plugins.d/cygnus/lib - fi -} - -function copy_cygnus_conf() { - local component_name=${1} - _logStage "######## Copying ${component_name} template config files to destination config directory... ########" - rm -rf ${RPM_SOURCE_DIR}/config - mkdir -p ${RPM_SOURCE_DIR}/config - cp ${BASE_DIR}/${component_name}/conf/* ${RPM_SOURCE_DIR}/config/ # templates are copied - if [[ -f ${RPM_SOURCE_DIR}/config/log4j.properties.template ]]; then - mv ${RPM_SOURCE_DIR}/config/log4j.properties.template ${RPM_SOURCE_DIR}/config/log4j.properties - fi - mv ${RPM_SOURCE_DIR}/config/README.md ${RPM_SOURCE_DIR}/config/README-${component_name}.md -} - -function clean_up_previous_builds() { - _logStage "######## Cleaning up previous builds of rpm... ########" - rm -rf ${RPM_BASE_DIR}/{RPMS,BUILDROOT,BUILD,SRPMS} - rm -rf ${RPM_SOURCE_DIR}/{config,usr} - return 0 -} - -function get_name_suffix() { - # At this moment Cygnus package name only has a Hadoop core version - HADOOP_VERSION=$(grep -A1 "hadoop-core" ${BASE_DIR}/cygnus-common/pom.xml | tail -1 | tr -d " ") - if [[ -z "${HADOOP_VERSION}" || ${HADOOP_VERSION} == "" ]]; then - return 1 - fi - NAME_SUFFIX="_hadoopcore_${HADOOP_VERSION}" -} - -function usage() { - SCRIPT=$(basename $0) - - printf "\n" >&2 - printf "usage: ${SCRIPT} [options] \n" >&2 - printf "\n" >&2 - printf "Options:\n" >&2 - printf "\n" >&2 - printf " -h show usage\n" >&2 - printf " -v VERSION Mandatory parameter. Version for rpm product preferably in format x.y.z \n" >&2 - printf " -r RELEASE Mandatory parameter. Release for product. I.E. 0.ge58dffa \n" >&2 - printf " -u ARTIFACT_URL Optional parameter. Url to server that contains flume package. Default value is http://archive.apache.org/dist/flume/1.4.0/ \n" >&2 - printf " -a ARTIFACT_NAME Optional parameter. Artifact name. Default value is apache-flume-1.4.0-bin.tar.gz. It is important that artifact be in .tar.gz format\n" >&2 - printf "\n" >&2 - exit 1 -} - -while getopts ":v:r:u:a:h" opt - -do - case $opt in - v) - VERSION_ARG=${OPTARG} - ;; - r) - RELEASE_ARG=${OPTARG} - ;; - u) - ARTIFACT_URL=${OPTARG} - ;; - a) - ARTIFACT_NAME=${OPTARG} - ;; - h) - usage - ;; - *) - echo "invalid argument: '${OPTARG}'" - exit 1 - ;; - esac -done - -# Setting folders this scrtipt is in neore/scripts BASE_DIR. Note: $0/.. is CWD -BASE_DIR=$(python -c 'import os,sys;print os.path.realpath(sys.argv[1])' $0/../..) - -# Import the colors for deployment script -source ${BASE_DIR}/rpm/colors_shell.sh - -# check user -if [[ $(id -u) == "0" ]]; then - _logError "${0}: shouldn't be executed as root" - exit 1 -fi - - -if [[ ! -z ${VERSION_ARG} ]]; then - PRODUCT_VERSION=${VERSION_ARG} -else - _logError "A product version must be specified with -v parameter." - usage - exit 2 -fi - -if [[ ! -z ${RELEASE_ARG} ]]; then - PRODUCT_RELEASE=${RELEASE_ARG} -else - _logError "A product release must be specified with -r parameter." - usage - exit 2 -fi - -if [[ -z ${ARTIFACT_URL} ]]; then - ARTIFACT_URL="http://archive.apache.org/dist/flume/1.4.0/" -fi - -if [[ -z ${ARTIFACT_NAME} ]]; then - ARTIFACT_NAME="apache-flume-1.4.0-bin.tar.gz" -fi - -_logStage "######## Setting the environment... ########" - - -_log "#### Iterate over every SPEC file ####" - -for SPEC_FILE in $(find "${BASE_DIR}" -type f -name *.spec) -do - - RPM_BASE_DIR="$(dirname ${SPEC_FILE})" - RPM_BASE_DIR=${RPM_BASE_DIR%*SPECS} # remove trailing SPECS - RPM_SOURCE_DIR="${RPM_BASE_DIR}/SOURCES" - RPM_PRODUCT_SOURCE_DIR="${RPM_SOURCE_DIR}/usr/cygnus/" - - CYGNUS_COMPONENT_NAME=${RPM_BASE_DIR%*/spec/} - CYGNUS_COMPONENT_NAME=${CYGNUS_COMPONENT_NAME##*/} - - - clean_up_previous_builds - - if [[ ${CYGNUS_COMPONENT_NAME} == "cygnus-common" ]]; then - - download_flume ${ARTIFACT_URL} ${ARTIFACT_NAME} - [[ $? -ne 0 ]] && exit 1 - - copy_cygnus_startup_script - [[ $? -ne 0 ]] && _logError "Cygnus startup script copy has failed. Did you run 'mvn clean compile exec:exec assembly:single'? Does the version in pom.xml file match $PRODUCT_VERSION?" && exit 1 - - get_name_suffix - [[ $? -ne 0 ]] && _logError "Can't get the name suffix" && exit 1 - - else - # dummy NAME_SUFFIX - NAME_SUFFIX="dummy_suffix" - fi - - copy_cygnus_jar_to_flume_directory ${CYGNUS_COMPONENT_NAME} - [[ $? -ne 0 ]] && _logError "Cygnus jar copy has failed. Did you run 'mvn clean compile exec:exec assembly:single'? Does the version in pom.xml file match $PRODUCT_VERSION?" && exit 1 - - copy_cygnus_conf ${CYGNUS_COMPONENT_NAME} - [[ $? -ne 0 ]] && exit 1 - - _logStage "######## Executing the rpmbuild ... ########" - - _log "#### Packaging using: ${SPEC_FILE}... ####" - # Execute command to create RPM - RPM_BUILD_COMMAND="rpmbuild -v -bb ${SPEC_FILE} --define '_topdir '${RPM_BASE_DIR} --define '_product_version '${PRODUCT_VERSION} --define '_product_release '${PRODUCT_RELEASE} --define '_name_suffix '${NAME_SUFFIX}" - _log "Rpm construction command: ${RPM_BUILD_COMMAND}" - rpmbuild -v -bb ${SPEC_FILE} --define '_topdir '${RPM_BASE_DIR} --define '_product_version '${PRODUCT_VERSION} --define '_product_release '${PRODUCT_RELEASE} --define '_name_suffix '${NAME_SUFFIX} - _logStage "######## rpmbuild finished! ... ########" -done - -_logStage "######## RPM Stage Finished! ... ########"