File ardana-spark-8.0+git.1523316674.611a69f.obscpio of Package ardana-spark

Overview Repositories Revisions Requests Users Attributes Meta

File ardana-spark-8.0+git.1523316674.611a69f.obscpio of Package ardana-spark (Revision ee9710ea21fda51a86150ba9fdbad6a9)

Currently displaying revision ee9710ea21fda51a86150ba9fdbad6a9 , Show latest

070701000DE640000081A40000000000000000000000015ACBF7C20000007C000000FD0000000200000000000000000000003300000000ardana-spark-8.0+git.1523316674.611a69f/.gitreview[gerrit]
host=gerrit.suse.provo.cloud
port=29418
project=ardana/spark-ansible.git
defaultremote=ardana
defaultbranch=master
070701000DE63C000081A40000000000000000000000015ACBF7C20000000C000000FD0000000200000000000000000000003600000000ardana-spark-8.0+git.1523316674.611a69f/.rsync-filter- ardana-ci
070701000DE5FF000081A40000000000000000000000015ACBF7C20000279F000000FD0000000200000000000000000000003000000000ardana-spark-8.0+git.1523316674.611a69f/LICENSE
                                 Apache License
                           Version 2.0, January 2004
                        http://www.apache.org/licenses/

TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

1. Definitions.

"License" shall mean the terms and conditions for use, reproduction,
      and distribution as defined by Sections 1 through 9 of this document.

"Licensor" shall mean the copyright owner or entity authorized by
      the copyright owner that is granting the License.

"Legal Entity" shall mean the union of the acting entity and all
      other entities that control, are controlled by, or are under common
      control with that entity. For the purposes of this definition,
      "control" means (i) the power, direct or indirect, to cause the
      direction or management of such entity, whether by contract or
      otherwise, or (ii) ownership of fifty percent (50%) or more of the
      outstanding shares, or (iii) beneficial ownership of such entity.

"You" (or "Your") shall mean an individual or Legal Entity
      exercising permissions granted by this License.

"Source" form shall mean the preferred form for making modifications,
      including but not limited to software source code, documentation
      source, and configuration files.

"Object" form shall mean any form resulting from mechanical
      transformation or translation of a Source form, including but
      not limited to compiled object code, generated documentation,
      and conversions to other media types.

"Work" shall mean the work of authorship, whether in Source or
      Object form, made available under the License, as indicated by a
      copyright notice that is included in or attached to the work
      (an example is provided in the Appendix below).

"Derivative Works" shall mean any work, whether in Source or Object
      form, that is based on (or derived from) the Work and for which the
      editorial revisions, annotations, elaborations, or other modifications
      represent, as a whole, an original work of authorship. For the purposes
      of this License, Derivative Works shall not include works that remain
      separable from, or merely link (or bind by name) to the interfaces of,
      the Work and Derivative Works thereof.

"Contribution" shall mean any work of authorship, including
      the original version of the Work and any modifications or additions
      to that Work or Derivative Works thereof, that is intentionally
      submitted to Licensor for inclusion in the Work by the copyright owner
      or by an individual or Legal Entity authorized to submit on behalf of
      the copyright owner. For the purposes of this definition, "submitted"
      means any form of electronic, verbal, or written communication sent
      to the Licensor or its representatives, including but not limited to
      communication on electronic mailing lists, source code control systems,
      and issue tracking systems that are managed by, or on behalf of, the
      Licensor for the purpose of discussing and improving the Work, but
      excluding communication that is conspicuously marked or otherwise
      designated in writing by the copyright owner as "Not a Contribution."

"Contributor" shall mean Licensor and any individual or Legal Entity
      on behalf of whom a Contribution has been received by Licensor and
      subsequently incorporated within the Work.

2. Grant of Copyright License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      copyright license to reproduce, prepare Derivative Works of,
      publicly display, publicly perform, sublicense, and distribute the
      Work and such Derivative Works in Source or Object form.

3. Grant of Patent License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      (except as stated in this section) patent license to make, have made,
      use, offer to sell, sell, import, and otherwise transfer the Work,
      where such license applies only to those patent claims licensable
      by such Contributor that are necessarily infringed by their
      Contribution(s) alone or by combination of their Contribution(s)
      with the Work to which such Contribution(s) was submitted. If You
      institute patent litigation against any entity (including a
      cross-claim or counterclaim in a lawsuit) alleging that the Work
      or a Contribution incorporated within the Work constitutes direct
      or contributory patent infringement, then any patent licenses
      granted to You under this License for that Work shall terminate
      as of the date such litigation is filed.

4. Redistribution. You may reproduce and distribute copies of the
      Work or Derivative Works thereof in any medium, with or without
      modifications, and in Source or Object form, provided that You
      meet the following conditions:

(a) You must give any other recipients of the Work or
          Derivative Works a copy of this License; and

(b) You must cause any modified files to carry prominent notices
          stating that You changed the files; and

(c) You must retain, in the Source form of any Derivative Works
          that You distribute, all copyright, patent, trademark, and
          attribution notices from the Source form of the Work,
          excluding those notices that do not pertain to any part of
          the Derivative Works; and

(d) If the Work includes a "NOTICE" text file as part of its
          distribution, then any Derivative Works that You distribute must
          include a readable copy of the attribution notices contained
          within such NOTICE file, excluding those notices that do not
          pertain to any part of the Derivative Works, in at least one
          of the following places: within a NOTICE text file distributed
          as part of the Derivative Works; within the Source form or
          documentation, if provided along with the Derivative Works; or,
          within a display generated by the Derivative Works, if and
          wherever such third-party notices normally appear. The contents
          of the NOTICE file are for informational purposes only and
          do not modify the License. You may add Your own attribution
          notices within Derivative Works that You distribute, alongside
          or as an addendum to the NOTICE text from the Work, provided
          that such additional attribution notices cannot be construed
          as modifying the License.

You may add Your own copyright statement to Your modifications and
      may provide additional or different license terms and conditions
      for use, reproduction, or distribution of Your modifications, or
      for any such Derivative Works as a whole, provided Your use,
      reproduction, and distribution of the Work otherwise complies with
      the conditions stated in this License.

5. Submission of Contributions. Unless You explicitly state otherwise,
      any Contribution intentionally submitted for inclusion in the Work
      by You to the Licensor shall be under the terms and conditions of
      this License, without any additional terms or conditions.
      Notwithstanding the above, nothing herein shall supersede or modify
      the terms of any separate license agreement you may have executed
      with Licensor regarding such Contributions.

6. Trademarks. This License does not grant permission to use the trade
      names, trademarks, service marks, or product names of the Licensor,
      except as required for reasonable and customary use in describing the
      origin of the Work and reproducing the content of the NOTICE file.

7. Disclaimer of Warranty. Unless required by applicable law or
      agreed to in writing, Licensor provides the Work (and each
      Contributor provides its Contributions) on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
      implied, including, without limitation, any warranties or conditions
      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
      PARTICULAR PURPOSE. You are solely responsible for determining the
      appropriateness of using or redistributing the Work and assume any
      risks associated with Your exercise of permissions under this License.

8. Limitation of Liability. In no event and under no legal theory,
      whether in tort (including negligence), contract, or otherwise,
      unless required by applicable law (such as deliberate and grossly
      negligent acts) or agreed to in writing, shall any Contributor be
      liable to You for damages, including any direct, indirect, special,
      incidental, or consequential damages of any character arising as a
      result of this License or out of the use or inability to use the
      Work (including but not limited to damages for loss of goodwill,
      work stoppage, computer failure or malfunction, or any and all
      other commercial damages or losses), even if such Contributor
      has been advised of the possibility of such damages.

9. Accepting Warranty or Additional Liability. While redistributing
      the Work or Derivative Works thereof, You may choose to offer,
      and charge a fee for, acceptance of support, warranty, indemnity,
      or other liability obligations and/or rights consistent with this
      License. However, in accepting such obligations, You may act only
      on Your own behalf and on Your sole responsibility, not on behalf
      of any other Contributor, and only if You agree to indemnify,
      defend, and hold each Contributor harmless for any liability
      incurred by, or claims asserted against, such Contributor by reason
      of your accepting any such warranty or additional liability.

070701000DE6A9000081A40000000000000000000000015ACBF7C2000003C4000000FD0000000200000000000000000000003200000000ardana-spark-8.0+git.1523316674.611a69f/README.md<!--
(c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP
(c) Copyright 2017 SUSE LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
-->

README
======

This repo contains the following roles

- FND-SPA:         Spark service

The verbs:

- install   - install the service/role
- configure - configure the service/role
- start     - start the service/role
- stop      - stop the service/role

The operations:

- deploy       - deploy the service (install, configure and start)
070701000DE6AA000081A40000000000000000000000015ACBF7C2000002DF000000FD0000000200000000000000000000004400000000ardana-spark-8.0+git.1523316674.611a69f/_spark-schedule-restart.yml#
# (c) Copyright 2018 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
# Schedule a restart of all spark services using ardana_notify_... variables
- hosts: all
  tasks:
    - include: roles/FND-SPA/tasks/_schedule_restart.yml070701000DE680000041ED0000000000000000000000045ACBF7C200000000000000FD0000000200000000000000000000003200000000ardana-spark-8.0+git.1523316674.611a69f/ardana-ci070701000DE681000041ED0000000000000000000000035ACBF7C200000000000000FD0000000200000000000000000000003A00000000ardana-spark-8.0+git.1523316674.611a69f/ardana-ci/project070701000DE682000041ED0000000000000000000000035ACBF7C200000000000000FD0000000200000000000000000000004600000000ardana-spark-8.0+git.1523316674.611a69f/ardana-ci/project/input-model070701000DE683000041ED0000000000000000000000025ACBF7C200000000000000FD0000000200000000000000000000004B00000000ardana-spark-8.0+git.1523316674.611a69f/ardana-ci/project/input-model/data070701000DE684000081A40000000000000000000000015ACBF7C2000009BE000000FD0000000200000000000000000000005D00000000ardana-spark-8.0+git.1523316674.611a69f/ardana-ci/project/input-model/data/control_plane.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
  product:
    version: 2

control-planes:
    - name: ccp
      control-plane-prefix: ccp
      region-name: region1
      failure-zones:
        - AZ1
        - AZ2
        - AZ3

common-service-components:
        - lifecycle-manager-target
        - monasca-agent

clusters:
        - name: cluster0
          cluster-prefix: c0
          server-role:
            - SERVER1-ROLE
          member-count: 1
          allocation-policy: strict
          service-components:
            - lifecycle-manager
            - tempest
            - ntp-client
            # Required for testing in (run-test.sh)
            - openstack-client

- name: cluster1
          cluster-prefix: c1
          server-role:
            - SERVER2-ROLE
          member-count: 1
          allocation-policy: strict
          service-components:
            - ntp-client
            # Monasca server
            - kafka
            - zookeeper
            - vertica
            - monasca-api
            - monasca-persister
            - spark

- name: cluster2
          cluster-prefix: c2
          server-role:
            - SERVER3-ROLE
          member-count: 1
          allocation-policy: strict
          service-components:
            - ntp-server
            - mysql
            - rabbitmq
            - keystone-api

- name: cluster3
          cluster-prefix: c3
          server-role:
            - SERVER4-ROLE
            - SERVER5-ROLE
            - SERVER6-ROLE
          member-count: 1
          allocation-policy: strict
          service-components:
            - ntp-client
            - ip-cluster

resources:
        - name: resource
          resource-prefix: res
          server-role:
            - SERVER7-ROLE
          allocation-policy: any
          service-components:
            - ntp-client
070701000DE685000041ED0000000000000000000000025ACBF7C200000000000000FD0000000200000000000000000000003800000000ardana-spark-8.0+git.1523316674.611a69f/ardana-ci/tests070701000DE6A8000081ED0000000000000000000000015ACBF7C20000034A000000FD0000000200000000000000000000004C00000000ardana-spark-8.0+git.1523316674.611a69f/ardana-ci/tests/exercise_spark.bash#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#!/usr/bin/env bash

ansible-playbook -i hosts/verb_hosts spark-submit.yml
SPARK_STATUS=$?
echo "Spark submit quick sum"
if [ $SPARK_STATUS -eq 0 ]
then
  echo "Ok"
else
  echo "Fail"
  exit 1
fi

exit 0070701000DE686000081A40000000000000000000000015ACBF7C20000049C000000FD0000000200000000000000000000004700000000ardana-spark-8.0+git.1523316674.611a69f/ardana-ci/tests/test-plan.yaml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
- name: Exercise spark
  logfile: exercise-spark.log
  exec:
    - exercise_spark.bash

- name: Validate spark-up
  logfile: testsuite-test_up.log
  exec:
    - validate-spark-up.bash

- name: Validate stop-start spark
  logfile: testsuite-test_stop-start.log
  exec:
    - validate-spark-stop-start.bash

- name: Validate spark-upgrade
  logfile: testsuite-test_upgrade.log
  exec:
    - validate-spark-upgrade.bash

- name: Validate spark-reconfigure
  logfile: testsuite-test_reconfigure.log
  exec:
    - validate-spark-reconfigure.bash
070701000DE69B000081ED0000000000000000000000015ACBF7C200000402000000FD0000000200000000000000000000004900000000ardana-spark-8.0+git.1523316674.611a69f/ardana-ci/tests/update-conf.bash#!/bin/bash
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
set -vxe
echo "Changing $1 in $2"
pushd ~/openstack/my_cloud/config
sed -e "$1" -i "$2" --follow-symlinks
git add -A
git commit --allow-empty -m "My Config Change"
pushd ~/openstack/ardana/ansible/
ansible-playbook -i hosts/localhost config-processor-run.yml -e encrypt="" \
                     -e rekey=""
ansible-playbook -i hosts/localhost ready-deployment.yml
popd
popd
070701000DE69F000081ED0000000000000000000000015ACBF7C2000004A8000000FD0000000200000000000000000000005800000000ardana-spark-8.0+git.1523316674.611a69f/ardana-ci/tests/validate-spark-reconfigure.bash#!/bin/bash
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
ansible-playbook -i hosts/verb_hosts spark-reconfigure.yml
SPARK_STATUS=$?
echo "Spark reconfigure"
if [ $SPARK_STATUS -eq 0 ]
then
  echo "Ok"
else
  echo "Fail"
  exit 1
fi

ansible-playbook -i hosts/verb_hosts spark-status.yml
SPARK_STATUS=$?
echo "Spark Service Status is $SPARK_STATUS"
if [ $SPARK_STATUS -eq 0 ]
then
  echo "Ok"
else
  echo "Fail"
  exit 1
fi

ansible-playbook -i hosts/verb_hosts spark-submit.yml
SPARK_STATUS=$?
echo "Spark submit quick sum"
if [ $SPARK_STATUS -eq 0 ]
then
  echo "Ok"
else
  echo "Fail"
  exit 1
fi070701000DE687000081ED0000000000000000000000015ACBF7C20000060D000000FD0000000200000000000000000000005700000000ardana-spark-8.0+git.1523316674.611a69f/ardana-ci/tests/validate-spark-stop-start.bash#!/bin/bash
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
ansible-playbook -i hosts/verb_hosts spark-stop.yml
SPARK_STATUS=$?
echo "Spark Service Stop"
if [ $SPARK_STATUS -eq 0 ]
then
  echo "Ok"
else
  echo "Fail"
  exit 1
fi

ansible-playbook -i hosts/verb_hosts spark-status.yml
SPARK_STATUS=$?
echo "Spark Service Status is $SPARK_STATUS"
if [ $SPARK_STATUS -eq 0 ]
then
  echo "Fail"
  exit 1
else
  echo "Ok"
fi

ansible-playbook -i hosts/verb_hosts spark-start.yml
SPARK_STATUS=$?
echo "Spark Service Start"
if [ $SPARK_STATUS -eq 0 ]
then
  echo "Ok"
else
  echo "Fail"
  exit 1
fi

ansible-playbook -i hosts/verb_hosts spark-status.yml
SPARK_STATUS=$?
echo "Spark Service Status is $SPARK_STATUS"
if [ $SPARK_STATUS -eq 0 ]
then
  echo "Ok"
else
  echo "Fail"
  exit 1
fi

ansible-playbook -i hosts/verb_hosts spark-submit.yml
SPARK_STATUS=$?
echo "Spark submit quick sum"
if [ $SPARK_STATUS -eq 0 ]
then
  echo "Ok"
else
  echo "Fail"
  exit 1
fi070701000DE6A0000081ED0000000000000000000000015ACBF7C200000405000000FD0000000200000000000000000000004F00000000ardana-spark-8.0+git.1523316674.611a69f/ardana-ci/tests/validate-spark-up.bash#!/bin/bash
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
ansible-playbook -i hosts/verb_hosts spark-status.yml
spark_test_status=$?
echo "Spark Test Status is $spark_test_status"
if [ $spark_test_status -eq 0 ]
then
  echo "Ok"
else
  echo "Fail"
  exit 1
fi

ansible-playbook -i hosts/verb_hosts spark-submit.yml
SPARK_STATUS=$?
echo "Spark submit quick sum"
if [ $SPARK_STATUS -eq 0 ]
then
  echo "Ok"
else
  echo "Fail"
  exit 1
fi
070701000DE69D000081ED0000000000000000000000015ACBF7C2000004A0000000FD0000000200000000000000000000005400000000ardana-spark-8.0+git.1523316674.611a69f/ardana-ci/tests/validate-spark-upgrade.bash#!/bin/bash
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
ansible-playbook -i hosts/verb_hosts spark-upgrade.yml
SPARK_STATUS=$?
echo "Spark upgrade"
if [ $SPARK_STATUS -eq 0 ]
then
  echo "Ok"
else
  echo "Fail"
  exit 1
fi

ansible-playbook -i hosts/verb_hosts spark-status.yml
SPARK_STATUS=$?
echo "Spark Service Status is $SPARK_STATUS"
if [ $SPARK_STATUS -eq 0 ]
then
  echo "Ok"
else
  echo "Fail"
  exit 1
fi

ansible-playbook -i hosts/verb_hosts spark-submit.yml
SPARK_STATUS=$?
echo "Spark submit quick sum"
if [ $SPARK_STATUS -eq 0 ]
then
  echo "Ok"
else
  echo "Fail"
  exit 1
fi070701000DE638000041ED0000000000000000000000025ACBF7C200000000000000FD0000000200000000000000000000002F00000000ardana-spark-8.0+git.1523316674.611a69f/config070701000DE639000081A40000000000000000000000015ACBF7C2000003CF000000FD0000000200000000000000000000004200000000ardana-spark-8.0+git.1523316674.611a69f/config/spark-symlinks.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#

# The following relative symlinks are created under the
# my_cloud/config directory.
---
symlinks:
  "spark/spark-defaults.conf.j2": "roles/FND-SPA/templates/spark-defaults.conf.j2"
  "spark/spark-env.sh.j2": "roles/FND-SPA/templates/spark-env.sh.j2"
  "spark/spark-worker-env.sh.j2": "roles/FND-SPA/templates/spark-worker-env.sh.j2"

070701000DE609000041ED0000000000000000000000035ACBF7C200000000000000FD0000000200000000000000000000002E00000000ardana-spark-8.0+git.1523316674.611a69f/roles070701000DE60A000041ED0000000000000000000000075ACBF7C200000000000000FD0000000200000000000000000000003600000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA070701000DE635000041ED0000000000000000000000025ACBF7C200000000000000FD0000000200000000000000000000003F00000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/defaults070701000DE636000081A40000000000000000000000015ACBF7C2000012A4000000FD0000000200000000000000000000004800000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/defaults/main.yml#
# (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017-2018 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
spark_component_name: spark
spark_hosts: "{{ FND_SPA.members.master }}"
spark_group_name: "{{ verb_hosts.FND_SPA }}"
spark_local_hostname: >
  {{ (spark_hosts |
      selectattr('ardana_ansible_host','equalto', inventory_hostname) |
      first).host }}
spark_ver: 1.6.3
spark_hadoop_ver: "{{ '2.6' if deployer_media_legacy_layout|bool else '2.4' }}"
spark_tarball_name: spark-{{ spark_ver }}-bin-hadoop{{ spark_hadoop_ver }}.tgz
spark_streaming_kafka_version_jar: spark-streaming-kafka_2.10-{{ spark_ver }}.jar
spark_streaming_kafka_jar: spark-streaming-kafka.jar
spark_root_dir: "{{ spark_component_name | bin_dir() }}/.."

spark_venv_lib_dir: "{%- if deployer_media_legacy_layout|bool -%}
                        {{ spark_component_name | jar_dir() }}
                     {%- else -%}
                        /usr/share/spark/lib/
                     {%- endif -%}"

spark_jdbc_jar: "{%- if deployer_media_legacy_layout|bool -%}
                    drizzle-jdbc-1.3.jar
                 {%- else -%}
                    mysql-connector-java.jar
                 {%- endif -%}"

spark_libs:
  - ["kafka_2.10-0.8.1.1.jar", "kafka_2.10-0.8.1.1/libs"]
  - ["scala-library-2.10.1.jar", "kafka_2.10-0.8.1.1/libs"]
  - ["metrics-core-2.2.0.jar", "kafka_2.10-0.8.1.1/libs"]
  - ["{{ spark_streaming_kafka_version_jar }}", ""]
  - ["{{ spark_jdbc_jar }}", ""]
spark_current_dir: "{{ spark_root_dir }}/current"
spark_versioned_dir: "{{ spark_root_dir }}/spark-{{ spark_ver }}-bin-hadoop{{ spark_hadoop_ver }}"
spark_lib_jars_location: "{{ spark_versioned_dir }}/lib"
spark_local_dir: /var/spark
spark_run_dir: /var/run/spark
spark_worker_directory: "{{ spark_run_dir }}/work"
spark_conf_dir: /etc/spark/conf
spark_init_dir: /etc/spark/init
spark_log_dir: /var/log/spark
spark_tools_dir: "{{ spark_versioned_dir }}/tools"
spark_master_log_file: "{{ spark_log_dir }}/{{ spark_component_name }}-master.log"
spark_worker_log_file: "{{ spark_log_dir }}/{{ spark_component_name }}-worker.log"
spark_event_log_directory: "{{ spark_log_dir }}/events"
spark_group: spark
spark_user: spark
# spark configuration properties
spark_executor_cores: 1
spark_executor_memory: 512m
spark_cores_max: 3
spark_driver_memory: 512m
spark_python_worker_memory: 16m
spark_worker_memory: 512m
spark_worker_cores: 1
spark_daemon_x_ms: 16m
spark_daemon_x_mx: 16m
spark_daemon_max_perm_size: 32m
spark_sql_shuffle_partitions: 32
spark_worker_cleanup_enabled: true
spark_cleaner_ttl: 900
spark_sql_ui_retainedExecutions: 10
spark_streaming_ui_retainedBatches: 10
spark_worker_ui_retainedExecutors: 10
spark_worker_ui_retainedDrivers: 10
spark_ui_retainedJobs: 10
spark_ui_retainedStages: 10
spark_executor_logs_rolling_maxRetainedFiles: 24
spark_executor_logs_rolling_strategy: time
spark_executor_logs_rolling_time_interval: hourly
spark_master_port: "{{ host.bind.FND_SPA.master.port }}"
spark_worker_port: "{{ host.bind.FND_SPA.worker.port }}"
spark_master_webui_port: "{{ host.bind.FND_SPA.master_web_ui.port }}"
spark_worker_webui_port: "{{ host.bind.FND_SPA.worker_web_ui.port }}"
spark_blockmanager_port: "{{ host.bind.FND_SPA.block_manager.port }}"
spark_broadcast_port: "{{ host.bind.FND_SPA.broadcast.port }}"
spark_driver_port: "{{ host.bind.FND_SPA.driver.port }}"
spark_executor_port: "{{ host.bind.FND_SPA.executor.port }}"
spark_fileserver_port: "{{ host.bind.FND_SPA.fileserver.port }}"
spark_masters: "{% for node in FND_SPA.members.master %}{{ node.host }}:{{ node.port }}{% if not loop.last %},{% endif %}{% endfor %}"

# zookeeper
spark_zookeeper_nodes: "{{ FND_SPA.consumes_FND_ZOO.members.private }}"
spark_zookeeper_hosts: "{% for node in spark_zookeeper_nodes %}{{ node.host }}:{{ node.port }}{% if not loop.last %},{% endif %}{% endfor %}"

spark_init: systemd
# upstart
spark_master_upstart_conf: /etc/init/spark-master.conf
spark_worker_upstart_conf: /etc/init/spark-worker.conf

# systemd
spark_master_systemd_service: /etc/systemd/system/spark-master.service
spark_worker_systemd_service: /etc/systemd/system/spark-worker.service

# monitoring
spark_agent_detection_plugin_dir: "{{ monasca_agent_detection_plugin_dir }}"

070701000DE60E000041ED0000000000000000000000025ACBF7C200000000000000FD0000000200000000000000000000003C00000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/files070701000DE60F000081A40000000000000000000000015ACBF7C200000547000000FD0000000200000000000000000000004B00000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/files/SparkDetect.py#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#

import logging

import monasca_setup.detection

log = logging.getLogger(__name__)

class SparkDetect(monasca_setup.detection.ServicePlugin):

"""Detect Spark daemons and setup configuration to monitor them."""

def __init__(self, template_dir, overwrite=True, args=None):
        log.info("\tWatching the spark processes.")
        service_params = {
            'args': {},
            'template_dir': template_dir,
            'overwrite': overwrite,
            'service_name': 'spark',
            'process_names': ['org.apache.spark.deploy.master.Master',
                              'org.apache.spark.deploy.worker.Worker']
        }

super(SparkDetect, self).__init__(service_params)

070701000DE610000081A40000000000000000000000015ACBF7C200000422000000FD0000000200000000000000000000004900000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/files/quick_sum.py#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#

from __future__ import print_function
from pyspark import SparkConf
from pyspark import SparkContext

if __name__ == "__main__":

my_spark_conf = SparkConf().setAppName("Quicksum")
    spark_context = SparkContext(conf=my_spark_conf)
    data = [1, 2, 3, 4, 5]
    distData = spark_context.parallelize(data)
    total = distData.reduce(lambda a, b: a + b)
    print("Total is %s" % total)
    spark_context.stop()
070701000DE612000041ED0000000000000000000000025ACBF7C200000000000000FD0000000200000000000000000000003C00000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/tasks070701000DE616000081A40000000000000000000000015ACBF7C20000030F000000FD0000000200000000000000000000005200000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/tasks/_schedule_restart.yml#
# (c) Copyright 2018 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
- name: FND-SPA | _schedule_restart | Schedule a restart for all services
  debug:
    msg: "Trigger a change notification in spark"
  changed_when: true
  register: ardana_notify_spark_restart_required
070701000DE620000081A40000000000000000000000015ACBF7C200000604000000FD0000000200000000000000000000004A00000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/tasks/configure.yml#
# (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017-2018 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
- name: FND-SPA | configure | Link current version
  become: yes
  file:
    path: "{{ spark_current_dir }}"
    state: link
    src: "{{ spark_versioned_dir }}"
  when: deployer_media_legacy_layout|bool

- name: FND-SPA | configure | Set up spark-env (rpm)
  become: yes
  template:
    src: spark-env.j2
    dest: /etc/spark/spark-env
    owner: "{{ spark_user }}"
    group: "{{ spark_group }}"
    mode: 0644
  when: deployer_media_legacy_layout|bool == False

- name: FND-SPA | configure | Set up spark-defaults (rpm)
  become: yes
  template:
    src: spark-defaults.conf.j2
    dest: /etc/spark/spark-defaults.conf
    owner: "{{ spark_user }}"
    group: "{{ spark_group }}"
    mode: 0644
  when: deployer_media_legacy_layout|bool == False

- name: FND-SPA | configure | include configure tasks (legacy)
  include: configure_legacy.yml
  when: deployer_media_legacy_layout|bool
070701000DE615000081A40000000000000000000000015ACBF7C200000E88000000FD0000000200000000000000000000005100000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/tasks/configure_legacy.yml#
# (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017-2018 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
- name: FND-SPA | configure_legacy | Link spark commands into /usr/bin
  become: yes
  file:
    path: /usr/bin/{{ item }}
    state: link
    src: "{{ spark_current_dir }}/bin/{{ item }}"
  with_items:
    - spark-submit
    - spark-class
    - spark-shell
    - spark-sql

- name: FND-SPA | configure_legacy | Link spark libs
  become: yes
  file:
    path: "{{ spark_lib_jars_location }}/{{ spark_streaming_kafka_jar }}"
    state: link
    src: "{{ spark_lib_jars_location }}/{{ spark_streaming_kafka_version_jar }}"

- name: FND-SPA | configure_legacy | Set up spark-env
  become: yes
  template:
    src: spark-env.sh.j2
    dest: /etc/spark/conf/spark-env.sh
    owner: "{{ spark_user }}"
    group: "{{ spark_group }}"
    mode: 0644

- name: FND-SPA | configure_legacy | Set up spark-worker-env
  become: yes
  template:
    src: spark-worker-env.sh.j2
    dest: /etc/spark/conf/spark-worker-env.sh
    owner: "{{ spark_user }}"
    group: "{{ spark_group }}"
    mode: 0644

- name: FND-SPA | configure_legacy | Set up spark-defaults (legacy)
  become: yes
  template:
    src: spark-defaults.conf.j2
    dest: /etc/spark/conf/spark-defaults.conf
    owner: "{{ spark_user }}"
    group: "{{ spark_group }}"
    mode: 0644

- name: FND-SPA | configure_legacy | Create spark-master start script
  become: yes
  template:
    dest: "{{ spark_init_dir }}/start-spark-master.sh"
    owner: root
    group: root
    mode: 0755
    src: start-spark-master.sh.j2

- name: FND-SPA | configure_legacy | Create spark-worker start script
  become: yes
  template:
    dest: "{{ spark_init_dir }}/start-spark-worker.sh"
    owner: root
    group: root
    mode: 0755
    src: start-spark-worker.sh.j2

- name: FND-SPA | configure_legacy | Create spark-master systemd config
  become: yes
  template:
    src: spark-master.service.j2
    dest: "{{ spark_master_systemd_service }}"
    owner: root
    group: root
    mode: 0644

- name: FND-SPA | configure_legacy | Create spark-worker systemd config
  become: yes
  template:
    src: spark-worker.service.j2
    dest: "{{ spark_worker_systemd_service }}"
    owner: root
    group: root
    mode: 0644

# See: http://www.freedesktop.org/software/systemd/man/tmpfiles.d.html
- name: FND-SPA | configure_legacy | Create spark-master.conf
  become: yes
  template:
    src: spark-master.conf.j2
    dest: "/usr/lib/tmpfiles.d/spark-master.conf"
    owner: root
    group: root
    mode: 0644

- name: FND-SPA | configure_legacy | Create spark-worker.conf
  become: yes
  template:
    src: spark-worker.conf.j2
    dest: "/usr/lib/tmpfiles.d/spark-worker.conf"
    owner: root
    group: root
    mode: 0644

- name: FND-SPA | configure_legacy | Reload systemctl
  become: yes
  ignore_errors: True
  command: /bin/systemctl daemon-reload

- name: FND-SPA | configure_legacy | Link config version
  become: yes
  file:
    path: "{{ spark_current_dir }}/conf/{{ item }}"
    state: link
    src: "{{ spark_conf_dir }}/{{ item }}"
  with_items:
    - "spark-env.sh"
    - "spark-worker-env.sh"
    - "spark-defaults.conf"
070701000DE61D000081A40000000000000000000000015ACBF7C20000085B000000FD0000000200000000000000000000004800000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/tasks/install.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017-2018 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
- name: FND-SPA | install | include OS specific variables
  include_vars: "{{ ansible_os_family | lower }}.yml"

- name: FND-SPA | install | Install the spark package from repo
  become: yes
  package:
    name: "{{ item }}"
    state: present
  when: deployer_media_legacy_layout|bool == False
  with_items:
    - "spark"
    - "kafka"
    - "mysql-connector-java"

- name: FND-SPA | install | create symlink to mysql connector jdbc jar
  become: yes
  file:
    src: "/usr/share/java/{{ spark_jdbc_jar }}"
    dest: "{{ spark_venv_lib_dir }}/{{ spark_jdbc_jar }}"
    state: link
  when: deployer_media_legacy_layout|bool == False

- name: FND-SPA | install | include legacy layout tasks
  include: install_legacy.yml
  when: deployer_media_legacy_layout|bool

- name: FND-SPA | install | Create Spark log and local directories
  become: yes
  file:
    path: "{{ item }}"
    state: directory
    owner: "{{ spark_user }}"
    group: "{{ spark_group }}"
    mode: 0775
  with_items:
    - "{{ spark_local_dir }}"
    - "{{ spark_log_dir }}"
    - "{{ spark_run_dir }}"
    - "{{ spark_worker_directory }}"
    - "{{ spark_conf_dir }}"
    - "{{ spark_init_dir }}"
    - "{{ spark_event_log_directory }}"
    - "{{ spark_tools_dir }}"

- name: FND-SPA | install | Touch logfiles
  become: true
  file:
    path: "{{ item }}"
    state: touch
    owner: "{{ spark_user }}"
    group: adm
    mode: 0640
  with_items:
    - "{{ spark_master_log_file }}"
    - "{{ spark_worker_log_file }}"

070701000DE61E000081A40000000000000000000000015ACBF7C200000907000000FD0000000200000000000000000000004F00000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/tasks/install_legacy.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017-2018 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
- name: FND-SPA | install_legacy | Setup group
  become_user: root
  become: True
  group:
    name: "{{ spark_group }}"
    system: yes

- name: FND-SPA | install_legacy | Setup user
  become: yes
  user:
    name: "{{ spark_user }}"
    system: yes
    group: "{{ spark_group }}"
    createhome: no
    home: /nonexistent
    shell: /usr/sbin/nologin
    comment: "Apache Spark system user"

- name: FND-SPA | install_legacy | Install OS specific required packages
  become: yes
  package:
    name: "{{ item }}"
    state: present
  with_items: fnd_spark_required_packages

- name: FND-SPA | install_legacy | Update installed packager cache
  become: yes
  install_package:
    cache: update

- name: FND-SPA | install_legacy | Install the spark venv
  become: yes
  install_package:
    name: spark
    service: spark
    state: present

- name: FND-SPA | install_legacy | Uncompress the spark tar
  become: yes
  unarchive:
    copy: no
    creates: >
      "{{ spark_root_dir }}/spark-{{ spark_ver }}-
      bin-hadoop{{ spark_hadoop_ver }}"
    dest: "{{ spark_root_dir }}"
    src: "{{ spark_venv_lib_dir }}/{{ spark_tarball_name }}"

- name: FND-SPA | install_legacy | Untar kafka tarball
  become: yes
  unarchive:
    copy: no
    creates: "{{ spark_venv_lib_dir }}/kafka_2.10-0.8.1.1"
    dest: "{{ spark_venv_lib_dir }}"
    src: "{{ spark_venv_lib_dir }}/kafka_2.10-0.8.1.1.tgz"

- name: FND-SPA | install_legacy | Link lib jars to spark lib directory
  become: yes
  file:
    state: link
    dest: "{{ spark_lib_jars_location }}/{{ item[0] }}"
    src: "{{ spark_venv_lib_dir }}/{{ item[1] }}/{{ item[0] }}"
  with_items:
    - "{{ spark_libs }}"
070701000DE61B000081A40000000000000000000000015ACBF7C200000417000000FD0000000200000000000000000000005200000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/tasks/monasca_configure.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
- name: FND-SPA | monasca_configure |
    Copy Spark Custom plugin for Ardana only
  become: yes
  copy:
    src: SparkDetect.py
    dest: "{{ spark_agent_detection_plugin_dir }}/SparkDetect.py"
    owner: root
    group: root
    mode: 0440

- name: FND-SPA | monasca_configure |
    Run Monasca agent Ardana specific Spark detection plugin
  become: yes
  monasca_agent_plugin:
    name: "SparkDetect"
070701000DE61F000081A40000000000000000000000015ACBF7C2000002A2000000FD0000000200000000000000000000005000000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/tasks/password-change.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---

# nothing to do I don't think
070701000DE613000081A40000000000000000000000015ACBF7C2000005F5000000FD0000000200000000000000000000004D00000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/tasks/spark_submit.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
- name: FND-SPA | spark_submit | Push quick_sum.py script for validation
  become: yes
  copy:
    src: quick_sum.py
    dest: "{{ spark_tools_dir }}/quick_sum.py"
    owner: "{{ spark_user }}"
    group: "{{ spark_group }}"
    mode: 0775

- name: FND-SPA | spark_submit | Submit quick_sum.py to spark
  become_user: "{{ spark_user }}"
  become: yes
  shell: >
    export SPARK_HOME={{ spark_current_dir }} &&
    spark-submit --supervise --master spark://{{ spark_masters }} quick_sum.py
  register:  spark_quick_sum_result
  args:
    chdir: "{{ spark_tools_dir }}"
  run_once: True

- name: FND-SPA | spark_submit | Announce quick sum result if failed
  fail:
    msg: |
      Quick sum failed with output -
      {{ spark_quick_sum_result.stdout }}
      {{ spark_quick_sum_result.stderr }}
  when: not ("{{ spark_quick_sum_result.stdout }}" == "Total is 15")
  run_once: true
070701000DE619000081A40000000000000000000000015ACBF7C200000592000000FD0000000200000000000000000000004600000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/tasks/start.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017-2018 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---

- name: FND-SPA | start | restart spark master
  # start spark master
  service:
    name: spark-master
    state: restarted
  become: yes
  when: ardana_notify_spark_restart_required is defined and
        ardana_notify_spark_restart_required.changed

- name: FND-SPA | start | start spark master
  # start spark master
  service:
    name: spark-master
    state: started
  become: yes

- name: FND-SPA | start | restart spark worker
  # start spark worker
  service:
    name: spark-worker
    state: restarted
  become: yes
  when: ardana_notify_spark_restart_required is defined and
        ardana_notify_spark_restart_required.changed

- name: FND-SPA | start | start spark worker
  # start spark worker
  service:
    name: spark-worker
    state: started
  become: yes
070701000DE617000081A40000000000000000000000015ACBF7C20000071B000000FD0000000200000000000000000000004700000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/tasks/status.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
- name: FND-SPA | status | Print service being checked
  debug:
    msg: "Running service check for spark"
  run_once: true

- name: FND-SPA | status | Check systemd service running spark-master
  command: systemctl status spark-master
  ignore_errors: yes
  changed_when: false
  register: spark_systemctl_status_master_result

- name: FND-SPA | status | Report status of spark-master
  fail:
    msg: |
      spark-master is not running.
      systemctl status spark-master output:
      {{ spark_systemctl_status_master_result.stdout }}
      {{ spark_systemctl_status_master_result.stderr }}
  when: spark_systemctl_status_master_result | failed

- name: FND-SPA | status | Check systemd service running spark-worker
  command: systemctl status spark-worker
  ignore_errors: yes
  changed_when: false
  register: spark_systemctl_status_worker_result

- name: FND-SPA | status | Report status of spark-worker
  fail:
    msg: |
      spark-worker is not running.
      systemctl status spark-worker output:
      {{ spark_systemctl_status_worker_result.stdout }}
      {{ spark_systemctl_status_worker_result.stderr }}
  when: spark_systemctl_status_worker_result | failed
070701000DE614000081A40000000000000000000000015ACBF7C200000360000000FD0000000200000000000000000000004500000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/tasks/stop.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
- name: FND-SPA | stop | stop spark worker
  become: yes
  service:
    name: spark-worker
    state: stopped

- name: FND-SPA | stop | stop spark master
  become: yes
  service:
    name: spark-master
    state: stopped
070701000DE618000081A40000000000000000000000015ACBF7C200000434000000FD0000000200000000000000000000004700000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/tasks/unlink.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017-2018 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---

- name: FND-SPA | unlink | Unlink current version
  become: yes
  file:
    path: "{{ spark_current_dir }}"
    state: absent
  when: deployer_media_legacy_layout|bool

- name: FND-SPA | unlink | Unlink spark commands in /usr/bin
  become: yes
  file:
    path: /usr/bin/{{ item }}
    state: absent
  with_items:
    - spark-submit
    - spark-class
    - spark-shell
    - spark-sql
  when: deployer_media_legacy_layout|bool
070701000DE61A000081A40000000000000000000000015ACBF7C20000035B000000FD0000000200000000000000000000004800000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/tasks/upgrade.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
## upgrade is not yet implemented as this is the initial release

- include: stop.yml

- include: unlink.yml

- include: install.yml

- include: configure.yml

- include: start.yml

- include: monasca_configure.yml070701000DE61C000081A40000000000000000000000015ACBF7C20000040B000000FD0000000200000000000000000000004D00000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/tasks/upstart_init.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
- name: create spark-master upstart config
  become: yes
  template:
    dest: "{{ spark_master_upstart_conf }}"
    owner: root
    group: root
    mode: 0644
    src: upstart/spark-master.conf.j2

- name: create spark-worker upstart config
  become: yes
  template:
    dest: "{{ spark_worker_upstart_conf }}"
    owner: root
    group: root
    mode: 0644
    src: upstart/spark-worker.conf.j2070701000DE621000041ED0000000000000000000000025ACBF7C200000000000000FD0000000200000000000000000000004000000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/templates070701000DE62D000081ED0000000000000000000000015ACBF7C20000033F000000FD0000000200000000000000000000005500000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/templates/exercise_spark.sh.j2#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#!/usr/bin/env bash

export SPARK_HOME=/opt/stack/service/spark/venv/bin/../current
CALC_VALUE=`bash $SPARK_HOME/bin/spark-submit quick_sum.py`
echo The calculated value is $CALC_VALUE

exit 0070701000DE62A000081A40000000000000000000000015ACBF7C2000008A9000000FD0000000200000000000000000000005700000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/templates/spark-defaults.conf.j2{#
#
# (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017-2018 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#}
spark.driver.extraClassPath {{ spark_venv_lib_dir }}/{{ spark_jdbc_jar }}
spark.executor.extraClassPath {{ spark_venv_lib_dir }}/{{ spark_jdbc_jar }}

spark.blockManager.port {{ spark_blockmanager_port }}
spark.broadcast.port {{ spark_broadcast_port }}
spark.cores.max {{ spark_cores_max }}
spark.driver.memory {{ spark_driver_memory }}
spark.driver.port {{ spark_driver_port }}
spark.eventLog.dir {{ spark_event_log_directory }}
spark.executor.cores {{ spark_executor_cores }}
spark.executor.memory {{ spark_executor_memory }}
spark.executor.port {{ spark_executor_port }}
spark.fileserver.port {{ spark_fileserver_port }}
spark.python.worker.memory {{ spark_python_worker_memory }}
spark.speculation true
spark.speculation.interval 200
spark.sql.shuffle.partitions {{ spark_sql_shuffle_partitions }}
spark.worker.cleanup.enabled {{ spark_worker_cleanup_enabled }}
spark.cleaner.ttl {{ spark_cleaner_ttl }}
spark.sql.ui.retainedExecutions {{ spark_sql_ui_retainedExecutions }}
spark.streaming.ui.retainedBatches {{ spark_streaming_ui_retainedBatches }}
spark.worker.ui.retainedExecutors {{ spark_worker_ui_retainedExecutors }}
spark.worker.ui.retainedDrivers {{ spark_worker_ui_retainedDrivers }}
spark.ui.retainedJobs {{ spark_ui_retainedJobs }}
spark.ui.retainedStages {{ spark_ui_retainedStages }}
spark.executor.logs.rolling.maxRetainedFiles {{spark_executor_logs_rolling_maxRetainedFiles}}
spark.executor.logs.rolling.strategy {{spark_executor_logs_rolling_strategy}}
spark.executor.logs.rolling.time.interval {{spark_executor_logs_rolling_time_interval}}
070701000DE62E000081A40000000000000000000000015ACBF7C200000575000000FD0000000200000000000000000000004D00000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/templates/spark-env.j2{#
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2018 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#}
SPARK_HOME=/usr/share/spark
SPARK_LOCAL_IP={{ spark_local_hostname }}
SPARK_MASTER_IP={{ spark_local_hostname }}
SPARK_MASTER_PORT={{ spark_master_port }}
SPARK_MASTER_WEBUI_PORT={{ spark_master_webui_port }}

SPARK_WORKER_PORT={{ spark_worker_port }}
SPARK_WORKER_WEBUI_PORT={{ spark_worker_webui_port }}
SPARK_WORKER_DIR={{ spark_worker_directory }}

SPARK_WORKER_MEMORY={{ spark_worker_memory }}
SPARK_WORKER_CORES={{ spark_worker_cores }}

SPARK_DAEMON_JAVA_OPTS="-Xms{{ spark_daemon_x_ms }} -Xmx{{ spark_daemon_x_mx }} -XX:MaxPermSize={{ spark_daemon_max_perm_size }} -Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url={{ spark_zookeeper_hosts }} -Dspark.deploy.zookeeper.dir={{ spark_run_dir }}"

SPARK_MASTERS={{ spark_masters }}070701000DE629000081A40000000000000000000000015ACBF7C200000667000000FD0000000200000000000000000000005000000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/templates/spark-env.sh.j2#!/usr/bin/env bash
{#
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#}
export SPARK_LOCAL_IP={{ spark_local_hostname }}
export SPARK_MASTER_IP={{ spark_local_hostname }}
export SPARK_MASTER_PORT={{ spark_master_port }}
export SPARK_MASTERS={{ spark_masters }}
export SPARK_MASTER_WEBUI_PORT={{ spark_master_webui_port }}

export SPARK_WORKER_PORT={{ spark_worker_port }}
export SPARK_WORKER_WEBUI_PORT={{ spark_worker_webui_port }}
export SPARK_WORKER_DIR={{ spark_worker_directory }}

export SPARK_WORKER_MEMORY={{ spark_worker_memory }}
export SPARK_WORKER_CORES={{ spark_worker_cores }}

export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.logDirectory=file://{{ spark_event_log_directory }} -Dspark.history.ui.port=18082"
export SPARK_DAEMON_JAVA_OPTS="$SPARK_DAEMON_JAVA_OPTS -Xms{{ spark_daemon_x_ms }} -Xmx{{ spark_daemon_x_mx }} -XX:MaxPermSize={{ spark_daemon_max_perm_size }} -Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url={{ spark_zookeeper_hosts }} -Dspark.deploy.zookeeper.dir={{ spark_run_dir }}"
070701000DE62C000081A40000000000000000000000015ACBF7C2000002CC000000FD0000000200000000000000000000005500000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/templates/spark-master.conf.j2{#
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#}
d {{ spark_worker_directory }} 0755 {{ spark_user }} {{ spark_group }}
070701000DE632000081A40000000000000000000000015ACBF7C200000338000000FD0000000200000000000000000000005800000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/templates/spark-master.service.j2{#
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#}
[Unit]
Description=Spark Master Daemon
After=zookeeper.service

[Service]
Type=simple
ExecStart={{ spark_init_dir }}/start-spark-master.sh &
[Install]
WantedBy=multi-user.target

070701000DE633000081A40000000000000000000000015ACBF7C2000006A8000000FD0000000200000000000000000000005700000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/templates/spark-worker-env.sh.j2#!/usr/bin/env bash
{#
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#}
export SPARK_LOCAL_IP={{ spark_local_hostname }}
export SPARK_MASTER_IP={{ spark_local_hostname }}
export SPARK_MASTER_PORT={{ spark_master_port }}
export SPARK_MASTERS={{ spark_masters }}
export SPARK_MASTER_WEBUI_PORT={{ spark_master_webui_port }}

export SPARK_WORKER_PORT={{ spark_worker_port }}
export SPARK_WORKER_WEBUI_PORT={{ spark_worker_webui_port }}
export SPARK_WORKER_DIR={{ spark_worker_directory }}

export SPARK_WORKER_MEMORY={{ spark_worker_memory }}
export SPARK_WORKER_CORES={{ spark_worker_cores }}

# history server
export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.logDirectory=file://{{ spark_event_log_directory }} -Dspark.history.ui.port=18082"

# set up for multiple masters using zookeeper
export SPARK_DAEMON_JAVA_OPTS="$SPARK_DAEMON_JAVA_OPTS  -Xms{{ spark_daemon_x_ms }} -Xmx{{ spark_daemon_x_mx }} -XX:MaxPermSize={{ spark_daemon_max_perm_size }} -Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url={{ spark_zookeeper_hosts }} -Dspark.deploy.zookeeper.dir={{ spark_run_dir }}"
070701000DE62B000081A40000000000000000000000015ACBF7C2000002CC000000FD0000000200000000000000000000005500000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/templates/spark-worker.conf.j2{#
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#}
d {{ spark_worker_directory }} 0755 {{ spark_user }} {{ spark_group }}
070701000DE628000081A40000000000000000000000015ACBF7C200000337000000FD0000000200000000000000000000005800000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/templates/spark-worker.service.j2{#
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#}
[Unit]
Description=Spark Worker Daemon
After=zookeeper.service

[Service]
Type=simple
ExecStart={{ spark_init_dir }}/start-spark-worker.sh &
[Install]
WantedBy=multi-user.target
070701000DE62F000081A40000000000000000000000015ACBF7C2000007E3000000FD0000000200000000000000000000005900000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/templates/start-spark-master.sh.j2#!/usr/bin/env bash
{#
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#}
. {{ spark_current_dir }}/conf/spark-env.sh
export EXEC_CLASS=org.apache.spark.deploy.master.Master
export INSTANCE_ID=1
export SPARK_CLASSPATH={{ spark_current_dir }}/conf/:{{ spark_current_dir }}/lib/spark-assembly-{{ spark_ver }}-hadoop{{ spark_hadoop_ver }}.0.jar:{{ spark_current_dir }}/lib/datanucleus-core-3.2.10.jar:{{ spark_current_dir }}/lib/datanucleus-rdbms-3.2.9.jar:{{ spark_current_dir }}/lib/datanucleus-api-jdo-3.2.6.jar
export SPARK_HOME={{ spark_current_dir }}

{% if ansible_os_family | lower == 'debian' %}
start-stop-daemon -c {{ spark_user }}:{{ spark_group }} --pidfile {{ spark_run_dir }}/{{ spark_user }}-{{ spark_group }}-"$EXEC_CLASS"-"$INSTANCE_ID".pid --name spark-master --start --exec  /usr/bin/java -- -cp $SPARK_CLASSPATH $SPARK_DAEMON_JAVA_OPTS "$EXEC_CLASS" --ip "$SPARK_MASTER_IP" --port "$SPARK_MASTER_PORT" --webui-port "$SPARK_MASTER_WEBUI_PORT" >> "{{ spark_master_log_file }}" 2>&1 < /dev/null
{% elif ansible_os_family | lower == 'suse' %}
start_daemon -u {{ spark_user }} -g {{ spark_group }} -p {{ spark_run_dir }}/{{ spark_user }}-{{ spark_group }}-"$EXEC_CLASS"-"$INSTANCE_ID".pid /usr/bin/java -cp $SPARK_CLASSPATH $SPARK_DAEMON_JAVA_OPTS "$EXEC_CLASS" --ip "$SPARK_MASTER_IP" --port "$SPARK_MASTER_PORT" --webui-port "$SPARK_MASTER_WEBUI_PORT" >> "{{ spark_master_log_file }}" 2>&1 < /dev/null
{% endif %}070701000DE634000081A40000000000000000000000015ACBF7C2000007CE000000FD0000000200000000000000000000005900000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/templates/start-spark-worker.sh.j2#!/usr/bin/env bash
{#
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#}
. {{ spark_current_dir }}/conf/spark-env.sh
export EXEC_CLASS=org.apache.spark.deploy.worker.Worker
export INSTANCE_ID=1
export SPARK_CLASSPATH={{ spark_current_dir }}/conf/:{{ spark_current_dir }}/lib/spark-assembly-{{ spark_ver }}-hadoop{{ spark_hadoop_ver }}.0.jar:{{ spark_current_dir }}/lib/datanucleus-core-3.2.10.jar:{{ spark_current_dir }}/lib/datanucleus-rdbms-3.2.9.jar:{{ spark_current_dir }}/lib/datanucleus-api-jdo-3.2.6.jar
export SPARK_HOME={{ spark_current_dir }}

{% if ansible_os_family | lower == 'debian' %}
start-stop-daemon -c {{ spark_user }}:{{ spark_group }} --pidfile {{ spark_run_dir }}/{{ spark_user }}-{{ spark_group }}-"$EXEC_CLASS"-"$INSTANCE_ID".pid --name spark-worker --start --exec  /usr/bin/java -- -cp $SPARK_CLASSPATH $SPARK_DAEMON_JAVA_OPTS "$EXEC_CLASS" --webui-port "$SPARK_WORKER_WEBUI_PORT" --port $SPARK_WORKER_PORT $SPARK_MASTERS >> "{{ spark_worker_log_file }}" 2>&1 < /dev/null
{% elif ansible_os_family | lower == 'suse' %}
start_daemon -u {{ spark_user }} -g {{ spark_group }} -p {{ spark_run_dir }}/{{ spark_user }}-{{ spark_group }}-"$EXEC_CLASS"-"$INSTANCE_ID".pid /usr/bin/java -cp $SPARK_CLASSPATH $SPARK_DAEMON_JAVA_OPTS "$EXEC_CLASS" --webui-port "$SPARK_WORKER_WEBUI_PORT" --port $SPARK_WORKER_PORT $SPARK_MASTERS >> "{{ spark_worker_log_file }}" 2>&1 < /dev/null
{% endif %}
070701000DE60B000041ED0000000000000000000000025ACBF7C200000000000000FD0000000200000000000000000000003B00000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/vars070701000DE60C000081A40000000000000000000000015ACBF7C2000002AF000000FD0000000200000000000000000000004600000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/vars/debian.yml#
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
fnd_spark_required_packages:
  - default-jre-headless

fnd_spark_restart_packages:
  - default-jre-headless070701000DE60D000081A40000000000000000000000015ACBF7C2000002BD000000FD0000000200000000000000000000004400000000ardana-spark-8.0+git.1523316674.611a69f/roles/FND-SPA/vars/suse.yml#
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
fnd_spark_required_packages:
  - java-1_8_0-openjdk-headless

fnd_spark_restart_packages:
  - java-1_8_0-openjdk-headless070701000DE5FD000081A40000000000000000000000015ACBF7C200000347000000FD0000000200000000000000000000004A00000000ardana-spark-8.0+git.1523316674.611a69f/spark-deploy-monitoring-agent.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
- name: deploy spark-monasca-agent-plugin
  hosts: FND-SPA
  roles:
  roles:
    - {role: monasca-agent, run_mode: Use}
    - FND-SPA
  tasks:
    - include: roles/FND-SPA/tasks/monasca_configure.yml
070701000DE608000081A40000000000000000000000015ACBF7C200000373000000FD0000000200000000000000000000004100000000ardana-spark-8.0+git.1523316674.611a69f/spark-deploy-service.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development Company LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
- name: deploy spark-master and spark-worker
  hosts: FND-SPA
  roles:
    - FND-SPA
  tasks:
    - include: roles/FND-SPA/tasks/install.yml
    - include: roles/FND-SPA/tasks/configure.yml
    - include: roles/FND-SPA/tasks/start.yml

070701000DE63B000081A40000000000000000000000015ACBF7C20000031D000000FD0000000200000000000000000000003900000000ardana-spark-8.0+git.1523316674.611a69f/spark-deploy.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#

# deploy the service itself
- include: spark-deploy-service.yml

# create monitoring checks for spark Services
- include: spark-deploy-monitoring-agent.yml

070701000DE63F000081A40000000000000000000000015ACBF7C200000392000000FD0000000200000000000000000000004200000000ardana-spark-8.0+git.1523316674.611a69f/spark-password-change.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
- name: reconfigure spark service
  hosts: FND-SPA
  roles:
    - FND-SPA
  tasks:
    - include: roles/FND-SPA/tasks/stop.yml
    - include: roles/FND-SPA/tasks/password-change.yml
    - include: roles/FND-SPA/tasks/configure.yml
    - include: roles/FND-SPA/tasks/start.yml070701000DE63E000081A40000000000000000000000015ACBF7C200000367000000FD0000000200000000000000000000003E00000000ardana-spark-8.0+git.1523316674.611a69f/spark-reconfigure.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
- name: reconfigure spark service
  hosts: FND-SPA
  roles:
    - FND-SPA
  serial: 1
  tasks:
    - include: roles/FND-SPA/tasks/stop.yml
    - include: roles/FND-SPA/tasks/configure.yml
    - include: roles/FND-SPA/tasks/start.yml070701000DE637000081A40000000000000000000000015ACBF7C200000287000000FD0000000200000000000000000000003A00000000ardana-spark-8.0+git.1523316674.611a69f/spark-restart.yml#
# (c) Copyright 2018 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
- include: _spark-schedule-restart.yml

- include: spark-start.yml
070701000DE63D000081A40000000000000000000000015ACBF7C200000308000000FD0000000200000000000000000000003800000000ardana-spark-8.0+git.1523316674.611a69f/spark-start.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
- name: start spark-master and spark-worker
  hosts: FND-SPA
  roles:
    - FND-SPA
  tasks:
    - include: roles/FND-SPA/tasks/start.yml070701000DE607000081A40000000000000000000000015ACBF7C20000030A000000FD0000000200000000000000000000003900000000ardana-spark-8.0+git.1523316674.611a69f/spark-status.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
- name: status spark-master and spark-worker
  hosts: FND-SPA
  roles:
    - FND-SPA
  tasks:
    - include: roles/FND-SPA/tasks/status.yml070701000DE67E000081A40000000000000000000000015ACBF7C200000306000000FD0000000200000000000000000000003700000000ardana-spark-8.0+git.1523316674.611a69f/spark-stop.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
- name: stop spark-master and spark-worker
  hosts: FND-SPA
  roles:
    - FND-SPA
  tasks:
    - include: roles/FND-SPA/tasks/stop.yml070701000DE606000081A40000000000000000000000015ACBF7C200000306000000FD0000000200000000000000000000003900000000ardana-spark-8.0+git.1523316674.611a69f/spark-submit.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
- name: submit sample job to spark
  hosts: FND-SPA
  roles:
    - FND-SPA
  tasks:
    - include: roles/FND-SPA/tasks/spark_submit.yml070701000DE67F000081A40000000000000000000000015ACBF7C20000031A000000FD0000000200000000000000000000003A00000000ardana-spark-8.0+git.1523316674.611a69f/spark-upgrade.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
- hosts: FND-SPA
  roles:
    - {role: monasca-agent, run_mode: Use}
    - FND-SPA
  serial: 1
  tasks:
    - include: roles/FND-SPA/tasks/upgrade.yml
07070100000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000B00000000TRAILER!!!175 blocks

Places

File ardana-spark-8.0+git.1523316674.611a69f.obscpio of Package ardana-spark (Revision ee9710ea21fda51a86150ba9fdbad6a9)

Places