File 0001-SCRD-2502-Check-for-and-install-required-r... of Package ardana-osconfig

Overview Repositories Revisions Requests Users Attributes Meta

File 0001-SCRD-2502-Check-for-and-install-required-repos.patch of Package ardana-osconfig (Revision 17c5b0621a0e87869e5737db462edbba)

Currently displaying revision 17c5b0621a0e87869e5737db462edbba , Show latest

From 04f2ede06cf2294253f34405b85dd6ced6e0aab5 Mon Sep 17 00:00:00 2001
From: Colleen Murphy <colleen.murphy@suse.de>
Date: Fri, 2 Feb 2018 16:01:24 +0100
Subject: [PATCH] SCRD-2502 Check for and install required repos

In order for Ardana-managed nodes to receive updates for SLES and from
the Cloud add-on extension, we must ensure that the required installer
media and subscription channels exist on the nodes. This patch does two
things:

- Adds an ansible module called zypper_check_repositories which examines
  all of the current repositories found in /etc/zypp/repos.d, checking
  the tags in their metadata to identify, without relying on repository
  names or aliases, what repositories are actually installed on the node
  and determine if any are missing.

- Runs the new module as a task in the setup-zypp.yml playbook. If the
  task fails, it expects to find those repositories being served from
  the deployer node and will attempt to install them from there. If that
  fails, it will abort the play on that node.

For this setup to be successful in a SUSEified model, this requires
changes in the deployer setup to allow the deployer to serve these
repositories[1]. For legacy mode, the deployer is already serving the
"OS" and "SDK" from ISO media, and cloud media repositories as "extras"
repositories, but these are unsigned.

This patch also removes the installation of python-xml from the
setup-zypp playbook since python-xml is pulled in by ardana-packager and
is not needed for anything specific to this playbook.

[1] https://github.com/SUSE-Cloud/automation/pull/2391

Change-Id: Ibf78a2674811fb729ce59dca2e524cf5f7bb8cd7
---
 library/zypper_check_repositories.py | 171 +++++++++++++++++++++++++++++++++++
 roles/osconfig/tasks/setup-zypp.yml  | 116 ++++++++++++++++++++----
 roles/osconfig/vars/repos-cloud.yml  |  37 ++++++++
 3 files changed, 308 insertions(+), 16 deletions(-)
 create mode 100644 library/zypper_check_repositories.py
 create mode 100644 roles/osconfig/vars/repos-cloud.yml

diff --git a/library/zypper_check_repositories.py b/library/zypper_check_repositories.py
new file mode 100644
index 0000000..5c61c6a
--- /dev/null
+++ b/library/zypper_check_repositories.py
@@ -0,0 +1,171 @@
+#!/usr/bin/python
+#
+# (c) Copyright 2018 SUSE LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+#
+
+DOCUMENTATION = '''
+---
+module: zypper_check_repositories
+short_description: Compare installed zypper repositories to required repos
+description: |
+  Check the zypper repositories installed on the system and compare them with
+  a dictionary that describes the metadata of required repositories. Fails if
+  all required repositories are not installed on the system.
+author: SUSE Linux GmbH
+options:
+  expected_repodata:
+    description: Dictionary describing the repomd of required repositories.
+    default: {}
+'''
+
+EXAMPLES = '''
+- set_fact:
+    expected_repodata:
+      cloud:
+        name: "Cloud"
+        repomd:
+          tags:
+            - "obsproduct://build.suse.de/SUSE:SLE-12-SP3:Update:Products:Cloud8/suse-openstack-cloud/8/cd/x86_64"
+            - "obsproduct://build.suse.de/Devel:Cloud:8:Staging/suse-openstack-cloud/8/POOL/x86_64"
+- name: osconfig | setup-zypp | Check that SLES and Cloud repos are installed
+  zypper_check_repositories:
+    expected_repodata: expected_repodata
+'''
+
+import glob
+import re
+import os
+import urllib2
+from xml.etree import ElementTree
+
+
+try:
+    import urlparse # python2
+except ModuleNotFoundError:
+    import urllib.parse as urlparse # python3
+
+
+from ansible.module_utils.urls import * # Provides the open_url method
+
+
+def _required_tags(expected_repodata):
+    tags = {repo['name']: repo['repomd']['tags']
+            for repo in expected_repodata.values()}
+    return tags
+
+
+def _repomd_sources():
+    repomd_sources = []
+    for repo in glob.glob('/etc/zypp/repos.d/*.repo'):
+        with open(repo) as f:
+            baseurl = [line.split('=')[1].strip() for line in f.readlines()
+                       if re.search(r'^baseurl=.*$', line)][0]
+            url_parts = urlparse.urlparse(baseurl)
+            if url_parts.scheme not in ('http', 'https', 'file', 'dir'):
+                # NOTE(colleen): There are many posible URI formats that zypper
+                # could potentially be using as a repository source. We don't
+                # support them all right now.
+                # See: https://en.opensuse.org/openSUSE:Libzypp_URIs
+                continue
+            repomd_source = url_parts._replace(
+                path=urlparse.urljoin(
+                    url_parts.path + '/', 'repodata/repomd.xml')).geturl()
+            repomd_sources.append(repomd_source)
+    return repomd_sources
+
+
+def _fetch_url(url):
+    resp = open_url(url)
+    if resp.code and resp.code != 200:
+        raise Exception("Could not reach remote repository: %s" % url)
+    return resp
+
+
+def _fetch_repomd(repo_uri):
+    url = re.sub(r'^dir:', 'file:', repo_uri)
+    try:
+        repomd = _fetch_url(url).read()
+    except urllib2.URLError:
+        # ISO media might be under suse/
+        url.replace('repodata', 'suse/repodata')
+        try:
+            repomd = _fetch_url(url).read()
+        except urllib2.URLError:
+            # This is not a repomd-format repo, move on
+           return
+    return repomd
+
+
+def _find_repotags():
+    found_repotags = {}
+    repomd_sources = _repomd_sources()
+    for repomd_source in repomd_sources:
+        repomd = _fetch_repomd(repomd_source)
+        if not repomd:
+            continue
+
+        xmlns = {'repo': 'http://linux.duke.edu/metadata/repo'}
+        try:
+            xmlroot = ElementTree.fromstring(repomd)
+            taglist = xmlroot.findall('repo:tags', xmlns)
+            for repotag in taglist:
+                repos = repotag.findall('repo:repo', xmlns)
+                for r in repos:
+                    found_repotags[r.text] = 1
+        except Exception as e:
+            raise Exception("Could not understand repo metadata located at "
+                            "%s: %s" % (repomd_source, e.message))
+    return found_repotags
+
+
+def run(expected_repodata):
+
+    found_repotags = _find_repotags()
+
+    unfound_repos = []
+    for name, tags in _required_tags(expected_repodata).items():
+        found_tag = False
+        for tag in tags:
+            if tag in found_repotags:
+                found_tag = True
+        if not found_tag:
+            unfound_repos.append(name)
+    if any(unfound_repos):
+        raise Exception("Missing required repositories: %s" % ', '.join(
+            unfound_repos))
+
+
+def main():
+
+    argument_spec = dict(
+        expected_repodata=dict(default=dict())
+    )
+    module = AnsibleModule(argument_spec=argument_spec,
+                           supports_check_mode=False)
+    params = module.params
+
+    try:
+        run(params['expected_repodata'])
+    except Exception as e:
+        module.fail_json(msg=e.message)
+    module.exit_json(rc=0, changed=False,
+                     msg="All required repositories are installed.")
+
+
+from ansible.module_utils.basic import *
+
+
+if __name__ == '__main__':
+    main()
diff --git a/roles/osconfig/tasks/setup-zypp.yml b/roles/osconfig/tasks/setup-zypp.yml
index 90b4f0f..3104e30 100644
--- a/roles/osconfig/tasks/setup-zypp.yml
+++ b/roles/osconfig/tasks/setup-zypp.yml
@@ -21,6 +21,10 @@
     zypp_server: "{{ hlinux_server }}"
   when: zypp_server is not defined and hlinux_server is defined
 
+##########################
+# Legacy media mode only #
+##########################
+
 - name: osconfig | setup-zypp | Check SLES OS zypper repo location
   delegate_to: localhost
   stat:
@@ -65,9 +69,86 @@
     disable_gpg_check: yes
   when: _sles_sdk_repo_stat_result.stat.exists
 
-- name: osconfig | setup-zypp | Set extras repository list
+##############################
+# Set up required SUSE Media #
+##############################
+
+- name: osconfig | setup-zypp | Add SLES and Cloud repos
   set_fact:
     _repositories:
+      - alias: SLES12-SP3-Pool
+        name: SLES12-SP3-Pool
+      - alias: SLES12-SP3-Updates
+        name: SLES12-SP3-Updates
+      - alias: SUSE-OpenStack-Cloud-8-Pool
+        name:  SUSE-OpenStack-Cloud-8-Pool
+      - alias: SUSE-OpenStack-Cloud-8-Updates
+        name:  SUSE-OpenStack-Cloud-8-Updates
+
+- name: osconfig | setup-zypp | Load expected repodata
+  include_vars: repos-cloud.yml
+
+- set_fact:
+    suse_release: suse-{{ansible_distribution_version}}
+
+- name: osconfig | setup-zypp | Check that SLES and Cloud repos are installed
+  zypper_check_repositories:
+      expected_repodata: "{{ expected_repodata[suse_release][ansible_architecture] }}"
+  register: maintenance_repos_installed
+  # If these aren't already here, that's okay, try to get them from the deployer node.
+  ignore_errors: true
+  when: deployer_media_legacy_layout|bool == false
+
+- name: osconfig | setup-zypp | Check if specified SLES and Cloud repos exist on deployer
+  delegate_to: localhost
+  stat:
+    path: "{{ PACKAGE_CONSTANTS.REPO_DIR }}/ardana/sles12/zypper/suse/{{ item.name }}"
+  register: _sles_repos_stat_result
+  with_items: _repositories
+  when: (maintenance_repos_installed is defined and maintenance_repos_installed.failed)
+
+- name: osconfig | setup-zypp | Generate SLES repos exists map
+  set_fact:
+    _repo_exists: >
+      {%- set _map_els = dict() -%}
+      {%- for r in _sles_repos_stat_result.results -%}
+      {%-   set _ = _map_els.update({r.item.name: r.stat.isdir is defined and r.stat.isdir}) -%}
+      {%- endfor -%}
+      {{ _map_els }}
+  when: (maintenance_repos_installed is defined and
+         maintenance_repos_installed.failed)
+
+- name: osconfig | setup-zypp | Configure SLES and Cloud repos on nodes
+  become: yes
+  zypper_repository:
+    name: "{{ item.alias }}"
+    description: "{{ item.name }}"
+    repo: "{{ zypp_server }}/ardana/sles12/zypper/suse/{{ item.name }}"
+  when: _repo_exists is defined and _repo_exists[item.name]
+  with_items: _repositories
+
+- name: osconfig | setup-zypp | Check that SLES and Cloud repos are installed
+  zypper_check_repositories:
+    expected_repodata: "{{ expected_repodata[suse_release][ansible_architecture] }}"
+  register: maintenance_repos_installed
+  ignore_errors: true
+  when: deployer_media_legacy_layout|bool == false
+
+# Finally, if the above didn't work, fail.
+- name: osconfig | setup-zypp | Fail if unable to configure SLES and Cloud repos
+  delegate_to: localhost
+  fail:
+    msg: "Installing the SUSE Linux Enterprise Server 12 SP3 updates repositories and SUSE OpenStack Cloud extension is required."
+  when: (maintenance_repos_installed is defined and
+         maintenance_repos_installed.failed)
+
+############################
+# Set up SLES extras repos #
+############################
+
+- name: osconfig | setup-zypp | Set extras repository list
+  set_fact:
+    _extras_repositories:
       - alias: openstack-newton
         name: cloud-openstack-newton
       - alias: openstack-pike
@@ -82,11 +163,11 @@
   stat:
     path: "{{ zypper_extras_dir }}/{{ item.name }}"
   register: _sles_extras_repos_stat_result
-  with_items: _repositories
+  with_items: _extras_repositories
 
-- name: osconfig | setup-zypp | Generate SLES extras repo exists map
+- name: osconfig | setup-zypp | Generate SLES repos exists map
   set_fact:
-    _repo_exists: >
+    _extras_repo_exists: >
       {%- set _map_els = dict() -%}
       {%- for r in _sles_extras_repos_stat_result.results -%}
       {%-   set _ = _map_els.update({r.item.name: r.stat.isdir is defined and r.stat.isdir}) -%}
@@ -98,8 +179,8 @@
   zypper_repository:
     name: "{{ item.alias }}"
     state: absent
-  with_items: _repositories
-  when: _repo_exists[item.name]
+  with_items: _extras_repositories
+  when: _extras_repo_exists[item.name]
 
 - name: osconfig | setup-zypp | Configure the extras zypp repo on SLES nodes
   become: yes
@@ -108,10 +189,14 @@
     description: "{{ item.name }}"
     repo: "{{ zypp_server }}/{{ zypper_extras_loc }}/{{ item.name }}"
     disable_gpg_check: yes
-  with_items: _repositories
-  when: _repo_exists[item.name]
+  with_items: _extras_repositories
+  when: _extras_repo_exists[item.name]
 
-- name: osconfig | setup-zypp | Get symlink source
+############################
+# Set up third party repos #
+############################
+
+- name: osconfig | setup-zypp | Get thirdparty symlink source
   delegate_to: localhost
   stat:
     path: "{{ zypp_thirdparty_dir }}/latest"
@@ -134,13 +219,9 @@
     disable_gpg_check: yes
   when: thirdparty_zypp_repo_present
 
-- name: osconfig | setup-zypp | Install required packages
-  become: yes
-  zypper:
-    name: "{{ item }}"
-    state: latest
-  with_items:
-    - python-xml
+#####################
+# Refresh all repos #
+#####################
 
 - name: osconfig | setup-zypp | Generate list of zypper repos to refresh
   set_fact:
@@ -152,6 +233,9 @@
       {%- if _sles_sdk_repo_stat_result.stat.exists -%}
       {%-   set _ = _rr_list.append(osconfig_zypper_sdk_repo_name) -%}
       {%- endif -%}
+      {%- for r in _extras_repositories -%}
+      {%-   set _ = _rr_list.append(r.name) -%}
+      {%- endfor -%}
       {%- for r in _repositories -%}
       {%-   set _ = _rr_list.append(r.name) -%}
       {%- endfor -%}
diff --git a/roles/osconfig/vars/repos-cloud.yml b/roles/osconfig/vars/repos-cloud.yml
new file mode 100644
index 0000000..738dbb8
--- /dev/null
+++ b/roles/osconfig/vars/repos-cloud.yml
@@ -0,0 +1,37 @@
+#
+# (c) Copyright 2018 SUSE LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+#
+---
+expected_repodata:
+  suse-12.3:
+    x86_64:
+      sles12-sp3-pool:
+        name: "SLES12-SP3-Pool"
+        repomd:
+          tags: ["obsproduct://build.suse.de/SUSE:SLE-12-SP3:GA/SLES/12.3/POOL/x86_64"]
+      sles12-sp3-updates:
+        name: "SLES12-SP3-Updates"
+        repomd:
+          tags: ["obsrepository://build.suse.de/SUSE:Updates:SLE-SERVER:12-SP3:x86_64/update"]
+      suse-openstack-cloud-8-pool:
+        name: "SUSE-OpenStack-Cloud-8-Pool"
+        repomd:
+          tags: ["obsproduct://build.suse.de/SUSE:SLE-12-SP3:Update:Products:Cloud8/suse-openstack-cloud/8/POOL/x86_64",
+                 "obsproduct://build.suse.de/SUSE:SLE-12-SP3:Update:Products:Cloud8/hpe-helion-openstack-cloud/8/POOL/x86_64"]
+      suse-openstack-cloud-8-updates:
+        name: "SUSE-OpenStack-Cloud-8-Updates"
+        repomd:
+          tags: ["obsrepository://build.suse.de/SUSE:Updates:OpenStack-Cloud:8:x86_64/update",
+                 "obsrepository://build.suse.de/SUSE:Updates:HPE-Helion-OpenStack-Cloud:8:x86_64/update"]
-- 
2.16.1

Places

File 0001-SCRD-2502-Check-for-and-install-required-repos.patch of Package ardana-osconfig (Revision 17c5b0621a0e87869e5737db462edbba)

Places