diff --git a/.github/CODE_OF_CONDUCT.md b/.github/CODE_OF_CONDUCT.md new file mode 100644 index 00000000..c768083b --- /dev/null +++ b/.github/CODE_OF_CONDUCT.md @@ -0,0 +1,76 @@ +# Contributor Covenant Code of Conduct + +## Our Pledge + +In the interest of fostering an open and welcoming environment, we as +contributors and maintainers pledge to making participation in our project and +our community a harassment-free experience for everyone, regardless of age, body +size, disability, ethnicity, sex characteristics, gender identity and expression, +level of experience, education, socio-economic status, nationality, personal +appearance, race, religion, or sexual identity and orientation. + +## Our Standards + +Examples of behavior that contributes to creating a positive environment +include: + +* Using welcoming and inclusive language +* Being respectful of differing viewpoints and experiences +* Gracefully accepting constructive criticism +* Focusing on what is best for the community +* Showing empathy towards other community members + +Examples of unacceptable behavior by participants include: + +* The use of sexualized language or imagery and unwelcome sexual attention or + advances +* Trolling, insulting/derogatory comments, and personal or political attacks +* Public or private harassment +* Publishing others' private information, such as a physical or electronic + address, without explicit permission +* Other conduct which could reasonably be considered inappropriate in a + professional setting + +## Our Responsibilities + +Project maintainers are responsible for clarifying the standards of acceptable +behavior and are expected to take appropriate and fair corrective action in +response to any instances of unacceptable behavior. + +Project maintainers have the right and responsibility to remove, edit, or +reject comments, commits, code, wiki edits, issues, and other contributions +that are not aligned to this Code of Conduct, or to ban temporarily or +permanently any contributor for other behaviors that they deem inappropriate, +threatening, offensive, or harmful. + +## Scope + +This Code of Conduct applies both within project spaces and in public spaces +when an individual is representing the project or its community. Examples of +representing a project or community include using an official project e-mail +address, posting via an official social media account, or acting as an appointed +representative at an online or offline event. Representation of a project may be +further defined and clarified by project maintainers. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported by contacting the project team at julian.pawlowski@gmail.com. All +complaints will be reviewed and investigated and will result in a response that +is deemed necessary and appropriate to the circumstances. The project team is +obligated to maintain confidentiality with regard to the reporter of an incident. +Further details of specific enforcement policies may be posted separately. + +Project maintainers who do not follow or enforce the Code of Conduct in good +faith may face temporary or permanent repercussions as determined by other +members of the project's leadership. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, +available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html + +[homepage]: https://www.contributor-covenant.org + +For answers to common questions about this code of conduct, see +https://www.contributor-covenant.org/faq diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md new file mode 100644 index 00000000..6611bca3 --- /dev/null +++ b/.github/CONTRIBUTING.md @@ -0,0 +1,10 @@ +# Contributing to the FHEM Docker image + +:+1::tada: First off, thanks for taking the time to contribute! :tada::+1: + +The following is a set of guidelines for contributing to the FHEM Docker image, which is hosted in the [FHEM Organization](https://github.com/fhem) on GitHub. These are mostly guidelines, not rules. Use your best judgment, and feel free to propose changes to this document in a pull request. + +#### Table Of Contents + +[Code of Conduct](CODE_OF_CONDUCT.md) +[How to create a pull request](https://help.github.com/articles/creating-a-pull-request-from-a-fork/) diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 00000000..dd84ea78 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,38 @@ +--- +name: Bug report +about: Create a report to help us improve +title: '' +labels: '' +assignees: '' + +--- + +**Describe the bug** +A clear and concise description of what the bug is. + +**To Reproduce** +Steps to reproduce the behavior: +1. Go to '...' +2. Click on '....' +3. Scroll down to '....' +4. See error + +**Expected behavior** +A clear and concise description of what you expected to happen. + +**Screenshots** +If applicable, add screenshots to help explain your problem. + +**Desktop (please complete the following information):** + - OS: [e.g. iOS] + - Browser [e.g. chrome, safari] + - Version [e.g. 22] + +**Smartphone (please complete the following information):** + - Device: [e.g. iPhone6] + - OS: [e.g. iOS8.1] + - Browser [e.g. stock browser, safari] + - Version [e.g. 22] + +**Additional context** +Add any other context about the problem here. diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md new file mode 100644 index 00000000..bbcbbe7d --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -0,0 +1,20 @@ +--- +name: Feature request +about: Suggest an idea for this project +title: '' +labels: '' +assignees: '' + +--- + +**Is your feature request related to a problem? Please describe.** +A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] + +**Describe the solution you'd like** +A clear and concise description of what you want to happen. + +**Describe alternatives you've considered** +A clear and concise description of any alternative solutions or features you've considered. + +**Additional context** +Add any other context or screenshots about the feature request here. diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 00000000..e69de29b diff --git a/.travis.yml b/.travis.yml index 535a682b..3873b845 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,6 +1,11 @@ language: bash services: docker +branches: + only: + - dev + - /v\d+.\d+.*/ + # amd64 toolchain (used as a base for non-sudo compilation, which is faster) include: &toolchain_amd64 sudo: false @@ -74,7 +79,7 @@ include: &toolchain_linux_arm32v7 - ARCH=arm32v7 - DPKG_ARCH=armhf - QEMU_ARCH=arm - - NPM_ARCH=arm + - NPM_ARCH=armhf - GNU_TRIPLET=arm-linux-gnueabihf - GNU_MULTILIB_TRIPLET=arm-linux-gnueabihf - GPP_COMPILER=arm-linux-gnueabihf-g++ @@ -89,7 +94,7 @@ include: &toolchain_linux_arm64v8 - ARCH=arm64v8 - DPKG_ARCH=arm64 - QEMU_ARCH=aarch64 - - NPM_ARCH=arm + - NPM_ARCH=arm64 - GNU_TRIPLET=aarch64-linux-gnu - GNU_MULTILIB_TRIPLET=arm-linux-gnueabihf - GPP_COMPILER=aarch64-linux-gnu-g++ @@ -112,31 +117,30 @@ matrix: notifications: email: false -before_install: - - export CXX="${GPP_COMPILER}" CC="${GCC_COMPILER}" DOCKER_CONFIG="./"; - - if [[ "${CROSS_TOOLCHAIN}" == "true" ]]; then - sudo rm -rf /etc/apt/sources.list.d/**; - sudo rm /etc/apt/sources.list; - echo "deb [arch=amd64,i386] http://archive.ubuntu.com/ubuntu xenial main universe multiverse restricted" | sudo tee /etc/apt/sources.list; - echo "deb [arch=${DPKG_ARCH}] http://ports.ubuntu.com/ubuntu-ports xenial main universe multiverse restricted" | sudo tee -a /etc/apt/sources.list; - echo "deb-src http://archive.ubuntu.com/ubuntu xenial main universe multiverse restricted" | sudo tee -a /etc/apt/sources.list; - sudo dpkg --add-architecture ${DPKG_ARCH}; - sudo apt-get update -yq; - curl -fsSL https://github.com/multiarch/qemu-user-static/releases/download/v3.0.0/x86_64_qemu-${QEMU_ARCH}-static.tar.gz | tar zx -C ./src/; - fi - # Update FHEM repository - - svn co https://svn.fhem.de/fhem/ ./src/fhem >/dev/null; +before_install: | + export CXX="${GPP_COMPILER}" CC="${GCC_COMPILER}" DOCKER_CONFIG="./"; + if [[ "${CROSS_TOOLCHAIN}" == "true" ]]; then + sudo rm -rf /etc/apt/sources.list.d/**; + sudo rm /etc/apt/sources.list; + echo "deb [arch=amd64,i386] http://archive.ubuntu.com/ubuntu xenial main universe multiverse restricted" | sudo tee /etc/apt/sources.list; + echo "deb [arch=${DPKG_ARCH}] http://ports.ubuntu.com/ubuntu-ports xenial main universe multiverse restricted" | sudo tee -a /etc/apt/sources.list; + echo "deb-src http://archive.ubuntu.com/ubuntu xenial main universe multiverse restricted" | sudo tee -a /etc/apt/sources.list; + sudo dpkg --add-architecture ${DPKG_ARCH}; + sudo apt-get update -yq; + curl -fsSL https://github.com/multiarch/qemu-user-static/releases/download/v3.1.0-2/x86_64_qemu-${QEMU_ARCH}-static.tar.gz | tar zx -C ./src/; + fi + svn co https://svn.fhem.de/fhem/trunk ./src/fhem/trunk >/dev/null; -install: - - travis_wait 120 sleep infinity & ./scripts/build.sh; +install: | + travis_wait 120 sleep infinity & ./scripts/build.sh; -script: - - ./scripts/test-integration.sh; +script: | + ./scripts/test-integration.sh; cache: directories: - $TRAVIS_BUILD_DIR/src/fhem/ - -after_success: - - ./scripts/push-docker.sh; - - ./scripts/push-docker-manifest.sh; + +after_success: | + ./scripts/push-docker.sh; + ./scripts/push-docker-manifest.sh; diff --git a/Dockerfile b/Dockerfile index 77483b3b..fc4d74e4 100644 --- a/Dockerfile +++ b/Dockerfile @@ -59,14 +59,25 @@ LABEL org.fhem.licenses=${L_LICENSES_FHEM} LABEL org.fhem.description=${L_DESCR_FHEM} ENV TERM xterm - -# Configure environment -COPY ./src/qemu-* /usr/bin/ - -RUN echo "org.opencontainers.image.created=${BUILD_DATE}\norg.opencontainers.image.authors=${L_AUTHORS}\norg.opencontainers.image.url=${L_URL}\norg.opencontainers.image.documentation=${L_USAGE}\norg.opencontainers.image.source=${L_VCS_URL}\norg.opencontainers.image.version=${IMAGE_VERSION}\norg.opencontainers.image.revision=${IMAGE_VCS_REF}\norg.opencontainers.image.vendor=${L_VENDOR}\norg.opencontainers.image.licenses=${L_LICENSES}\norg.opencontainers.image.title=${L_TITLE}\norg.opencontainers.image.description=${L_DESCR}\norg.fhem.authors=${L_AUTHORS_FHEM}\norg.fhem.url=${L_URL_FHEM}\norg.fhem.documentation=${L_USAGE_FHEM}\norg.fhem.source=${L_VCS_URL_FHEM}\norg.fhem.version=${FHEM_VERSION}\norg.fhem.revision=${VCS_REF}\norg.fhem.vendor=${L_VENDOR_FHEM}\norg.fhem.licenses=${L_LICENSES_FHEM}\norg.fhem.description=${L_DESCR_FHEM}" > /image_info +ENV LANG en_US.UTF-8 +ENV LANGUAGE en_US:en +ENV LC_ALL en_US.UTF-8 # Install base environment -RUN DEBIAN_FRONTEND=noninteractive apt-get update \ +COPY ./src/qemu-* /usr/bin/ +COPY src/entry.sh /entry.sh +COPY src/ssh_known_hosts.txt /ssh_known_hosts.txt +COPY src/health-check.sh /health-check.sh +COPY src/find-missing-deb-packages.sh /usr/local/bin/find-missing-deb-packages.sh +COPY src/find-missing-perl-modules.sh /usr/local/bin/find-missing-perl-modules.sh +COPY src/99_DockerImageInfo.pm /fhem/FHEM/ +ADD https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py /usr/local/bin/speedtest-cli +RUN chmod 755 /*.sh /usr/local/bin/speedtest-cli \ + && echo "org.opencontainers.image.created=${BUILD_DATE}\norg.opencontainers.image.authors=${L_AUTHORS}\norg.opencontainers.image.url=${L_URL}\norg.opencontainers.image.documentation=${L_USAGE}\norg.opencontainers.image.source=${L_VCS_URL}\norg.opencontainers.image.version=${IMAGE_VERSION}\norg.opencontainers.image.revision=${IMAGE_VCS_REF}\norg.opencontainers.image.vendor=${L_VENDOR}\norg.opencontainers.image.licenses=${L_LICENSES}\norg.opencontainers.image.title=${L_TITLE}\norg.opencontainers.image.description=${L_DESCR}\norg.fhem.authors=${L_AUTHORS_FHEM}\norg.fhem.url=${L_URL_FHEM}\norg.fhem.documentation=${L_USAGE_FHEM}\norg.fhem.source=${L_VCS_URL_FHEM}\norg.fhem.version=${FHEM_VERSION}\norg.fhem.revision=${VCS_REF}\norg.fhem.vendor=${L_VENDOR_FHEM}\norg.fhem.licenses=${L_LICENSES_FHEM}\norg.fhem.description=${L_DESCR_FHEM}" > /image_info \ + && sed -i "s/stretch main/stretch main contrib non-free/g" /etc/apt/sources.list \ + && sed -i "s/stretch-updates main/stretch-updates main contrib non-free/g" /etc/apt/sources.list \ + && sed -i "s/stretch\/updates main/stretch\/updates main contrib non-free/g" /etc/apt/sources.list \ + && DEBIAN_FRONTEND=noninteractive apt-get update \ && DEBIAN_FRONTEND=noninteractive apt-get install -qqy --no-install-recommends \ apt-transport-https \ apt-utils \ @@ -83,31 +94,45 @@ RUN DEBIAN_FRONTEND=noninteractive apt-get update \ \ && DEBIAN_FRONTEND=noninteractive apt-get install -qqy --no-install-recommends \ avahi-daemon \ + avrdude \ bluez \ - build-essential \ - cpanminus \ curl \ dfu-programmer \ dnsutils \ + espeak \ etherwake \ + git-core \ + i2c-tools \ inetutils-ping \ jq \ + lame \ + libav-tools \ + libttspico-utils \ + lsb-release \ + mariadb-client \ + mp3wrap \ + mplayer \ netcat \ + nmap \ openssh-client \ - perl \ - python \ sendemail \ snmp \ sox \ - subversion \ sqlite3 \ + subversion \ sudo \ telnet \ telnet-ssl \ unzip \ usbutils \ wget \ - \ + && apt-get autoremove -qqy && apt-get clean \ + && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* + +# Add Perl app layer for pre-compiled packages +RUN DEBIAN_FRONTEND=noninteractive apt-get update \ + && DEBIAN_FRONTEND=noninteractive apt-get install -qqy --no-install-recommends \ + perl \ libalgorithm-merge-perl \ libauthen-*-perl \ libavahi-compat-libdnssd-dev \ @@ -120,20 +145,27 @@ RUN DEBIAN_FRONTEND=noninteractive apt-get update \ libcrypt-*-perl \ libdata-dump-perl \ libdatetime-format-strptime-perl \ + libdbd-mysql \ + libdbd-pg-perl \ libdbd-sqlite3-perl \ libdbi-perl \ libdevel-size-perl \ libdevice-serialport-perl \ + libdevice-usb-perl \ libdigest-*-perl \ libdpkg-perl \ + libencode-perl \ liberror-perl \ libev-perl \ libfile-copy-recursive-perl \ libfile-fcntllock-perl \ + libfinance-quote-perl \ libgd-graph-perl \ libgd-text-perl \ + libgnupg-interface-perl \ libhtml-strip-perl \ libhtml-treebuilder-xpath-perl \ + libimage-imlib2-perl \ libimage-info-perl \ libimage-librsvg-perl \ libio-file-withpath-perl \ @@ -141,20 +173,26 @@ RUN DEBIAN_FRONTEND=noninteractive apt-get update \ libjson-perl \ libjson-xs-perl \ liblist-moreutils-perl \ + libmail-gnupg-perl \ libmail-imapclient-perl \ libmail-sendmail-perl \ libmime-base64-perl \ libmime-lite-perl \ libmodule-pluggable-perl \ libmojolicious-perl \ + libmoox-late-perl \ libnet-bonjour-perl \ libnet-jabber-perl \ + libnet-oauth-perl \ + libnet-oauth2-perl \ libnet-server-perl \ libnet-sip-perl \ libnet-snmp-perl \ libnet-ssleay-perl \ libnet-telnet-perl \ libnet-xmpp-perl \ + libnmap-parser-perl \ + librivescript-perl \ librpc-xml-perl \ libsnmp-perl \ libsnmp-session-perl \ @@ -181,41 +219,81 @@ RUN DEBIAN_FRONTEND=noninteractive apt-get update \ libxml-stream-perl \ libxml-treebuilder-perl \ libxml-xpath-perl \ - && cpanm \ - Net::MQTT::Constants \ - Net::MQTT::Simple \ - && if [ "${ARCH}" != "i386" ]; then \ - cpanm \ - Crypt::Random \ - ; fi \ - && if [ "${ARCH}" = "amd64" ] || [ "${ARCH}" = "i386" ]; then \ - cpanm \ - Crypt::Cipher::AES \ - ; fi \ - && rm -rf /root/.cpanm \ - && if [ -d ./src/fhem/ ]; then \ - svn up ./src/fhem >/dev/null \ - ; else \ - svn co https://svn.fhem.de/fhem/ ./src/fhem \ - ; fi \ - && apt-get purge -qqy \ - build-essential \ - cpanminus \ - unzip \ - subversion \ + libxml-xpathengine-perl \ + libyaml-libyaml-perl \ + libyaml-perl \ && apt-get autoremove -qqy && apt-get clean \ && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* -ENV LANG en_US.UTF-8 -ENV LANGUAGE en_US:en -ENV LC_ALL en_US.UTF-8 +# Add Perl app layer for self-compiled software +# * exclude any ARM platforms due too long build time +# * manually pre-compiled ARM packages may be applied here +RUN if [ "${ARCH}" = "amd64" ] || [ "${ARCH}" = "i386" ]; then \ + DEBIAN_FRONTEND=noninteractive apt-get update \ + && DEBIAN_FRONTEND=noninteractive apt-get install -qqy --no-install-recommends \ + build-essential \ + cpanminus \ + libssl-dev \ + && cpanm \ + Crypt::OpenSSL::AES \ + CryptX \ + Device::SMBus \ + Net::MQTT::Constants \ + Net::MQTT::Simple \ + && if [ "${ARCH}" = "amd64" ]; then \ + cpanm \ + Crypt::Random \ + Math::Pari \ + ; fi \ + && rm -rf /root/.cpanm \ + && apt-get purge -qqy \ + build-essential \ + cpanminus \ + libssl-dev \ + && apt-get autoremove -qqy && apt-get clean \ + && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* \ + ; fi -COPY src/entry.sh /entry.sh -COPY src/health-check.sh /health-check.sh +# Add nodejs app layer +RUN if [ "${ARCH}" != "arm32v5" ]; then \ + if [ "${ARCH}" = "i386" ]; then \ + curl -sL https://deb.nodesource.com/setup_8.x | bash - \ + ; else \ + curl -sL https://deb.nodesource.com/setup_10.x | bash - \ + ; fi \ + && DEBIAN_FRONTEND=noninteractive apt-get install -qqy --no-install-recommends \ + build-essential \ + libssl-dev \ + nodejs \ + && if [ "${ARCH}" = "arm32v7" ] || [ "${ARCH}" = "arm64v8" ]; then \ + NPM_CONFIG_UNSAFE_PERM=true npm install -g \ + alexa-fhem \ + ; else \ + npm install -g \ + alexa-fhem \ + ; fi \ + && rm -rf ~/.npm* \ + && apt-get purge -qqy \ + build-essential \ + libssl-dev \ + && apt-get autoremove -qqy && apt-get clean \ + && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* \ + ; fi + +# Add Python app layer +RUN DEBIAN_FRONTEND=noninteractive apt-get update \ + && DEBIAN_FRONTEND=noninteractive apt-get install -qqy --no-install-recommends \ + libinline-python-perl \ + python3 \ + python3-pychromecast \ + youtube-dl \ + && apt-get autoremove -qqy && apt-get clean \ + && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* + +# Add FHEM app layer +# Note: Manual checkout is required if build is not run by Travis: +# svn co https://svn.fhem.de/fhem/trunk ./src/fhem/trunk COPY src/fhem/trunk/fhem/ /fhem/ -COPY src/99_DockerImageInfo.pm /fhem/FHEM/ -ADD https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py /usr/local/bin/speedtest-cli -RUN chmod 755 /*.sh /usr/local/bin/speedtest-cli VOLUME [ "/opt/fhem" ] diff --git a/scripts/build.sh b/scripts/build.sh index 8553c4f6..7e19a307 100755 --- a/scripts/build.sh +++ b/scripts/build.sh @@ -1,5 +1,8 @@ #!/bin/bash set -e +echo "Building for platform: `uname -a`" +TRAVIS_BRANCH=${TRAVIS_BRANCH:-`git branch | sed -n -e 's/^\* \(.*\)/\1/p'`} +LABEL=${LABEL:-`uname -m`_linux} echo "TRAVIS_BRANCH = ${TRAVIS_BRANCH}" echo "TRAVIS_TAG = ${TRAVIS_TAG}" [[ -n "${TRAVIS_BRANCH}" && "${TRAVIS_BRANCH}" != "master" ]] && set -x @@ -10,7 +13,12 @@ BUILD_DATE=$( date --iso-8601=seconds --utc ) BASE="fhem/fhem-${LABEL}" BASE_IMAGE="debian" BASE_IMAGE_TAG="stretch" -FHEM_VERSION="$( cd ./src/fhem; svn log -v ./tags | grep "A /tags/FHEM_" | sort | tail -n 1 | cut -d / -f 3 | cut -d " " -f 1 |cut -d _ -f 2- | sed s/_/./g )" + +# Download dependencies if not existing +if [ ! -d ./src/fhem ]; then + svn co https://svn.fhem.de/fhem/trunk ./src/fhem/trunk; +fi +FHEM_VERSION="$( svn ls "^/tags" https://svn.fhem.de/fhem/ | grep "FHEM_" | sort | tail -n 1 | cut -d / -f 1 | cut -d " " -f 1 |cut -d _ -f 2- | sed s/_/./g )" FHEM_REVISION_LATEST="$( cd ./src/fhem; svn info -r HEAD | grep "Revision" | cut -d " " -f 2 )" if [[ -n "${ARCH}" && "${ARCH}" != "amd64" ]]; then @@ -22,7 +30,7 @@ if [[ -n "${ARCH}" && "${ARCH}" != "amd64" ]]; then fi IMAGE_VERSION=$(git describe --tags --dirty --match "v[0-9]*") -IMAGE_VERSION=${IMAGE_VERSION:1} +IMAGE_VERSION=${IMAGE_VERSION:-1} IMAGE_BRANCH=$( [[ -n "${TRAVIS_BRANCH}" && "${TRAVIS_BRANCH}" != "master" && "${TRAVIS_BRANCH}" != "${TRAVIS_TAG}" ]] && echo -n "${TRAVIS_BRANCH}" || echo -n "" ) VARIANT_FHEM="${FHEM_VERSION}-s${FHEM_REVISION_LATEST}" VARIANT_IMAGE="${IMAGE_VERSION}$( [ -n "${IMAGE_BRANCH}" ] && echo -n "-${IMAGE_BRANCH}" || echo -n "" )" @@ -32,6 +40,9 @@ echo -e "\n\nNow building variant ${VARIANT} ...\n\n" # Only run build if not existing on Docker hub yet function docker_tag_exists() { + if [[ "x${DOCKER_USER}" == "x" || "x${DOCKER_PASS}" == "x" ]]; then + return 1 + fi set +x TOKEN=$(curl -s -H "Content-Type: application/json" -X POST -d '{"username": "'${DOCKER_USER}'", "password": "'${DOCKER_PASS}'"}' https://hub.docker.com/v2/users/login/ | jq -r .token) EXISTS=$(curl -s -H "Authorization: JWT ${TOKEN}" https://hub.docker.com/v2/repositories/$1/tags/?page_size=10000 | jq -r "[.results | .[] | .name == \"$2\"] | any") @@ -59,11 +70,6 @@ else echo "No prior build found for ${BASE}:${TAG} on Docker Hub registry" fi -# Download dependencies if not existing -if [ ! -d ./src/fhem ]; then - svn co https://svn.fhem.de/fhem/ ./src/fhem; -fi - docker build \ $( [ -n "${CACHE_TAG}" ] && echo -n "--cache-from "${BASE}:${CACHE_TAG}"" ) \ --tag "${BASE}:${VARIANT}" \ diff --git a/scripts/compile-perl-arm.sh b/scripts/compile-perl-arm.sh new file mode 100755 index 00000000..51ade409 --- /dev/null +++ b/scripts/compile-perl-arm.sh @@ -0,0 +1,76 @@ +#!/bin/bash + +sudo apt update +sudo apt install -y build-essential devscripts quilt dh-autoreconf dh-systemd ubuntu-dev-tools sbuild debhelper moreutils +sudo adduser $USER sbuild + +echo "\$apt_allow_unauthenticated = 1; +\$environment_filter = [ + '^PATH$', + '^DEB(IAN|SIGN)?_[A-Z_]+$', + '^(C(PP|XX)?|LD|F)FLAGS(_APPEND)?$', + '^USER(NAME)?$', + '^LOGNAME$', + '^HOME$', + '^TERM$', + '^SHELL$', + '^no_proxy$', + '^http_proxy$', + '^https_proxy$', + '^ftp_proxy$', +]; + +# Directory for writing build logs to +\$log_dir=\$ENV{HOME}."/ubuntu/logs"; + +# don't remove this, Perl needs it: +1;" > ~/.sbuildrc +mkdir -p $HOME/ubuntu/{build,logs} + +echo "SCHROOT_CONF_SUFFIX=\"source-root-users=root,sbuild,admin +source-root-groups=root,sbuild,admin +preserve-environment=true\" +# you will want to undo the below for stable releases, read \`man mk-sbuild\` for details +# during the development cycle, these pockets are not used, but will contain important +# updates after each release of Ubuntu +# if you have e.g. apt-cacher-ng around +# DEBOOTSTRAP_PROXY=http://127.0.0.1:3142/" > ~/.mk-sbuild.rc + +sg sbuild +newgrp sbuild +sudo sbuild-update --keygen +sudo chown -R $USER:sbuild ~/.gnupg/ +sudo su -c "grep -q /etc/hosts /etc/schroot/sbuild/copyfiles || echo /etc/hosts >> /etc/schroot/sbuild/copyfiles" + +mk-sbuild --target=armel stretch +mk-sbuild --target=armhf stretch +mk-sbuild --target=arm64 stretch + +curl -fsSL https://github.com/multiarch/qemu-user-static/releases/download/v3.1.0-2/x86_64_qemu-arm-static.tar.gz | tar zx -C ~/ +curl -fsSL https://github.com/multiarch/qemu-user-static/releases/download/v3.1.0-2/x86_64_qemu-aarch64-static.tar.gz | tar zx -C ~/ +chmod a+x qemu-*-static +sudo cp -f qemu-arm-static /var/lib/schroot/chroots/stretch-amd64-armel/usr/bin +sudo cp -f qemu-arm-static /var/lib/schroot/chroots/stretch-amd64-armhf/usr/bin +sudo cp -f qemu-aarch64-static /var/lib/schroot/chroots/stretch-amd64-arm64/usr/bin + +sbuild --chroot stretch-amd64-armhf --arch armhf -j8 + +ARCHLIST="arm32v5 arm32v7 aarch64" +for ARCH in $ARCHLIST; do + + # platforms independent packages + dh-make-perl make --install --build --cpan Statistics::ChiSquare + dh-make-perl make --build --cpan Net::MQTT::Constants + dh-make-perl make --build --cpan Net::MQTT::Simple + + # platform packages + dh-make-perl make --build --cpan CryptX + dh-make-perl make --build --cpan Crypt::OpenSSL::AES + dh-make-perl make --build --cpan Device::SMBus + dh-make-perl make --install --build --cpan Math::Pari + dh-make-perl make --bdepends libmath-pari-perl --depends libmath-pari-perl --depends libclass-loader-perl --depends libstatistics-chisquare-perl --build --cpan Crypt::Random + + shopt -s extglob + rm -rf !(*.deb) + shopt -u extglob +done diff --git a/scripts/find-perl-modules.sh b/scripts/find-perl-modules.sh deleted file mode 100755 index 2d54c8ea..00000000 --- a/scripts/find-perl-modules.sh +++ /dev/null @@ -1,33 +0,0 @@ -#!/bin/bash - -# Find missing Perl modules that are in use by official modules - -# use -for module in `find $1./FHEM -type f -name "*.pm" -print0 | xargs -0 grep -oP "^use .*;" | sed 's|[;"'\'']||g' | sed 's/$attr{global}{modpath}//g' | sed 's/$main::attr{global}{modpath}//g' | sort -u -f -k2,2 | cut -d : -f 2- | cut -d " " -f 2 | grep -v -i -E "^(\$|\@|warnings|vars|feature|inline|strict|constant|(\d\.\d+)|POSIX|utf8)"`; do - NAME=`echo $module | cut -d " " -f 1 | cut -d ";" -f 1 | cut -d ":" -f 1` - if [[ -e "$1./$NAME" || -e "$1./FHEM/$NAME" || -e "$1./$NAME.pm" || -e "$1./FHEM/$NAME.pm" ]]; then - # This is a FHEM internal Perl module - continue - fi - CHK=`perl -e "use $module" 2>1 >/dev/null` - ret=$? - - if [ $ret != "0" ]; then - echo "Checking external module '$module' ... MISSING" - fi -done - -# require -for module in `find $1./FHEM -type f -name "*.pm" -print0 | xargs -0 grep -oP "^require .*;" | sed 's|[;"'\'']||g' | sed 's/$attr{global}{modpath}//g' | sed 's/$main::attr{global}{modpath}//g' | sort -u -f -k2,2 | cut -d : -f 2- | cut -d " " -f 2 | grep -v -i -E "^(\$|\@|warnings|vars|feature|inline|strict|constant|(\d\.\d+)|POSIX|utf8)"`; do - NAME=`echo $module | cut -d " " -f 1 | cut -d ";" -f 1 | cut -d ":" -f 1` - if [[ -e "$1./$NAME" || -e "$1./FHEM/$NAME" || -e "$1./$NAME.pm" || -e "$1./FHEM/$NAME.pm" ]]; then - # This is a FHEM internal Perl module - continue - fi - CHK=`perl -e "use $module" 2>1 >/dev/null` - ret=$? - - if [ $ret != "0" ]; then - echo "Checking external module '$module' ... MISSING" - fi -done diff --git a/src/entry.sh b/src/entry.sh index fc9a4d7e..da957989 100755 --- a/src/entry.sh +++ b/src/entry.sh @@ -90,20 +90,36 @@ if [ ! -s ${FHEM_DIR}/.ssh/id_ed25519 ]; then ssh-keygen -t ed25519 -f ${FHEM_DIR}/.ssh/id_ed25519 -q -N "" -o -a 100 sed -i "s/root@.*/fhem@fhem-docker/" ${FHEM_DIR}/.ssh/id_ed25519.pub fi -chmod -v 600 ${FHEM_DIR}/.ssh/id_ed25519 -chmod -v 644 ${FHEM_DIR}/.ssh/id_ed25519.pub -chown fhem.root ${FHEM_DIR}/.ssh/id_ed25519* +chmod 600 ${FHEM_DIR}/.ssh/id_ed25519 +chmod 644 ${FHEM_DIR}/.ssh/id_ed25519.pub # SSH key: RSA if [ ! -s ${FHEM_DIR}/.ssh/id_rsa ]; then echo -e " - Generating SSH RSA client certificate for user 'fhem' ..." rm -f ${FHEM_DIR}/.ssh/id_rsa* - ssh-keygen -t rsa -f ${FHEM_DIR}/.ssh/id_rsa -q -N "" -o -a 100 + ssh-keygen -t rsa -b 4096 -f ${FHEM_DIR}/.ssh/id_rsa -q -N "" -o -a 100 sed -i "s/root@.*/fhem@fhem-docker/" ${FHEM_DIR}/.ssh/id_rsa.pub fi -chmod -v 600 ${FHEM_DIR}/.ssh/id_rsa -chmod -v 644 ${FHEM_DIR}/.ssh/id_rsa.pub -chown fhem.root ${FHEM_DIR}/.ssh/id_rsa* +chmod 600 ${FHEM_DIR}/.ssh/id_rsa +chmod 644 ${FHEM_DIR}/.ssh/id_rsa.pub + +# SSH client hardening +if [ ! -f ${FHEM_DIR}/.ssh/config ]; then +echo "IdentityFile ~/.ssh/id_ed25519 +IdentityFile ~/.ssh/id_rsa + +Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr +HostKeyAlgorithms ssh-ed25519,ssh-rsa +KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha256 +MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,umac-128-etm@openssh.com +" > ${FHEM_DIR}/.ssh/config +fi + +# SSH key pinning +touch ${FHEM_DIR}/.ssh/known_hosts +cat ${FHEM_DIR}/.ssh/known_hosts /ssh_known_hosts.txt | grep -v ^# | sort -u -k2,3 > ${FHEM_DIR}/.ssh/known_hosts.tmp +mv -f ${FHEM_DIR}/.ssh/known_hosts.tmp ${FHEM_DIR}/.ssh/known_hosts +chown -R fhem.fhem ${FHEM_DIR}/.ssh/ # Function to print FHEM log in incremental steps to the docker log. [ -s "$( date +"$LOGFILE" )" ] && OLDLINES=$( wc -l < "$( date +"$LOGFILE" )" ) || OLDLINES=0 diff --git a/src/find-missing-deb-packages.sh b/src/find-missing-deb-packages.sh new file mode 100755 index 00000000..8820cf13 --- /dev/null +++ b/src/find-missing-deb-packages.sh @@ -0,0 +1,20 @@ +#!/bin/bash + +# Find missing Debian packages that are in use by official modules + +FPATH="${1:-/opt/fhem}" +[ ! -d "$FPATH" ] && echo FPATH="." + +find $FPATH/FHEM -type f -name "*.pm" -print0 | xargs -0 grep -oP "(?:(?:apt(?:-get)? install )[^\\\\\n<()\"\'&;]+)" | cut -d : -f 2- | sed "s/\[//g" | sed "s/\]//g" | sed "s/apt install //g" | sed "s/apt-get install install //g" | sed "s/apt-get install //g" | sed "s/on Debian and derivatives//g" | sed "s/und noch die mp3 Unterst//g" | sed "s/ /\n/g" | grep -v '^$\|^\s*\#' | sort -u -f > /tmp/missing.apt + +for package in `cat /tmp/missing.apt`; do + + CHK=`dpkg -s $package 2>&1 >/dev/null` + ret=$? + + if [ $ret != "0" ]; then + echo $package + fi +done + +rm -f /tmp/missing.apt diff --git a/src/find-missing-perl-modules.sh b/src/find-missing-perl-modules.sh new file mode 100755 index 00000000..49b5689f --- /dev/null +++ b/src/find-missing-perl-modules.sh @@ -0,0 +1,37 @@ +#!/bin/bash + +# Find missing Perl modules that are in use by official modules + +FPATH="${1:-/opt/fhem}" +[ ! -d "$FPATH" ] && echo FPATH="." + +for module in `find $FPATH/FHEM -type f -name "*.pm" -print0 | xargs -0 grep -oP "[^\w#](?:(?:use|require) (?:[A-Z\'\"$][\w:./{}()$\->\"\']+))" | sed 's|[;"'\'']||g' | sed 's/FHEM\///g' | sed 's/.*\///g' | sed 's/$attr{global}{modpath}//g' | sed 's/$main::attr{global}{modpath}//g' | sort -u -f -k2,2 | cut -d : -f 2- | cut -d " " -f 2 | grep -v -i -E "^(use|require|warnings|vars|feature|inline|strict|constant|POSIX|utf8)" | grep -v "[(){}$]" | grep -v "[.:]$"`; do + NAME=`echo $module | cut -d " " -f 1 | cut -d ";" -f 1 | cut -d ":" -f 1` + if [[ -e "$FPATH/$NAME" || -e "$FPATH/FHEM/$NAME" || -f "$FPATH/FHEM/lib/$NAME" || -e "$FPATH/$NAME.pm" || -e "$FPATH/FHEM/$NAME.pm" || -f "$FPATH/FHEM/lib/$NAME.pm" ]]; then + continue + fi + if [[ "x`find $FPATH/FHEM -type f -name "*_$NAME.pm"`" != "x" ]]; then + continue + fi + if [[ "x`find $FPATH/FHEM -type f -name "*_$NAME"`" != "x" ]]; then + continue + fi + if [ -d "$FPATH/FHEM/lib/$NAME" ]; then + NAME2=`echo $module | cut -d " " -f 1 | cut -d ";" -f 1 | cut -d ":" -f 2` + if [[ -f "$FPATH/FHEM/lib/$NAME/$NAME2" || -f "$FPATH/FHEM/lib/$NAME/$NAME2.pm" ]]; then + continue + fi + NAME3=`echo $module | cut -d " " -f 1 | cut -d ";" -f 1 | cut -d ":" -f 3` + if [[ -f "$FPATH/FHEM/lib/$NAME/$NAME2/$NAME3" || -f "$FPATH/FHEM/lib/$NAME/$NAME2/$NAME3.pm" ]]; then + continue + fi + fi + + + CHK=`perl -e "use $module" 2>1 >/dev/null` + ret=$? + + if [ $ret != "0" ]; then + echo $module + fi +done diff --git a/src/ssh_known_hosts.txt b/src/ssh_known_hosts.txt new file mode 100644 index 00000000..4e6def16 --- /dev/null +++ b/src/ssh_known_hosts.txt @@ -0,0 +1,2 @@ +# ssh-keyscan -p 58824 fhem-va.fhem.de 2>/dev/null +[fhem-va.fhem.de]:58824,[88.99.31.202]:58824,[2a01:4f8:10a:806::f2]:58824 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCcTyjPHgeBFgxtOmq8aZWOXyJU1K57cGyllhV1YhIbzM8MAjdhLV54vTcDKyoDszjug24luyU+OnCHfgyeo9mFZdM93vJI0LrB8fQWSHXD2tjBhDwxGhxm0ksqlDFP3h3ZFP6HoXzrOP69ucqLSKv8/cfkvpp2kfxRMxHGjsfroNHOHmwUtBy80wh/XNcUikOBqQ7aZiCQWGkdJLHEFWgTT1VQ9P7ZkNe33bG+TICc4LF05DqDIZbD4zqhfKj/oNCgmm+vNQU2GQc/FZBjxO6+qtkcO7Ne0kksI3L2xcEyvkJOm1GUvwB0tDJQSiNfbc6lEzEZx6MOUL3SY00WtJHd