File _service:obs_scm:obs-scm-bridge-0.1~pre.1642586... of Package obs_scm_bridge

Overview Repositories Revisions Requests Users Attributes Meta

File _service:obs_scm:obs-scm-bridge-0.1~pre.1642586655.f1db846.obscpio of Package obs_scm_bridge

07070100000000000081A400000000000000000000000161E7E21F00000179000000000000000000000000000000000000003300000000obs-scm-bridge-0.1~pre.1642586655.f1db846/Makefileprefix = /usr
PYTHON ?= python

servicedir = ${prefix}/lib/obs/service

all:

install:
	install -d $(DESTDIR)$(servicedir)
	install -m 0755 obs_scm_bridge $(DESTDIR)$(servicedir)

test:
	flake8 set_version tests/
	${PYTHON} -m unittest discover tests/

clean:
	find -name "*.pyc" -exec rm {} \;
	find -name '*.pyo' -exec rm {} \;
	rm -rf set_versionc

.PHONY: all install test
07070100000001000081A400000000000000000000000161E7E21F00000820000000000000000000000000000000000000003400000000obs-scm-bridge-0.1~pre.1642586655.f1db846/README.md
Native OBS SCM bridge helper
============================

Native OBS scm support for the build recipies and additional files. This is bridging an external authorative
scm repository into OBS. Any source change or merge workflow must be provided via the scm repository 
hoster in this scenario.

Only git is supported atm, but this can be extended later to further systems.

It is not recommended to put large binary files into git directly as this won't scale. Use the
asset support instead, which is described in pbuild documentation:

http://opensuse.github.io/obs-build/pbuild.html#_remote_assets

These assets will be downloaded by osc and OBS. The verification via sha256 sum is optional.

HOWTO manage a single package
=============================

The current way to define a git repository for an OBS package is using the `scmsync`
element inside the package meta.

```
<scmsync>https://github.com/foo/bar</scmsync>
```

For doing a local checkout use the currently experimental osc from

https://download.opensuse.org/repositories/home:/adrianSuSE:/OBSGIT/

This version allows you to do

# osc co $project <$package>

which will create a git repository inside of the classic osc checkout.

The only further tested functionality is to do local builds atm.

HOWTO manage an entire project
==============================

A git repository can also get defined for entire project. This can be done
via the scmsync element in project meta.

Any top level subdirectory will be handled as package container.

It is recomended to use git submodules for each package if it is a larger
project. This allows partial cloning of the specific package.

TODO
====

* Monitoring changes in referenced repository. (can currently be workarounded
   via "osc service rr")

* osc upstream integration

* signature validation

* find a better way to store files in .osc and .assets of the checkout, as
   they do not belong to the git repository
    auto extending .gitignore? (esp. when downloading asset files?)

* make cpio generation bit identical (avoid mtime from clone)

07070100000002000081A400000000000000000000000161E7E21F00000506000000000000000000000000000000000000003E00000000obs-scm-bridge-0.1~pre.1642586655.f1db846/obs-scm-bridge.spec#
# spec file
#
# Copyright (c) 2021 SUSE LLC
#
# All modifications and additions to the file contributed by third parties
# remain the property of their copyright owners, unless otherwise agreed
# upon. The license for this file, and modifications and additions to the
# file, is the same license as for the pristine package itself (unless the
# license for the pristine package is not an Open Source License, in which
# case the license is the MIT License). An "Open Source License" is a
# license that conforms to the Open Source Definition (Version 1.9)
# published by the Open Source Initiative.

# Please submit bugfixes or comments via https://bugs.opensuse.org/
#

%if 0%{?fedora} || 0%{?rhel}
%define build_pkg_name obs-build
%else
%define build_pkg_name build
%endif

Name:           obs-scm-bridge
Version:        0.0.1
Release:        0
Summary:        A help service to work with git repositories in OBS
License:        GPL-2.0-or-later
URL:            https://github.com/openSUSE/obs-scm-bridge
Source0:        %{name}-%{version}.tar.xz
Requires:       %{build_pkg_name} >= 20211125
BuildArch:      noarch
Recommends:     python3-packaging

%description

%prep
%autosetup

%build

%install
make DESTDIR=%{buildroot} install

%files
%{_prefix}/lib/obs/service

%changelog
07070100000003000081ED00000000000000000000000161E7E21F000031F0000000000000000000000000000000000000003900000000obs-scm-bridge-0.1~pre.1642586655.f1db846/obs_scm_bridge#!/usr/bin/python3
# -*- coding: utf-8 -*-

# scm (only git atm) cloning and packaging for Open Build Service
# 
# (C) 2021 by Adrian Schröter <adrian@suse.de>
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
# See http://www.gnu.org/licenses/gpl-2.0.html for full license text.

import argparse
import os
import re
import shutil
import sys
import logging
import subprocess
import tempfile
from html import escape
import urllib.parse
import configparser

outdir = None
download_assets = '/usr/lib/build/download_assets'
export_debian_orig_from_git = '/usr/lib/build/export_debian_orig_from_git'
pack_directories = False
get_assets = False

if os.environ.get('DEBUG_SET_VERSION') == "1":
    logging.getLogger().setLevel(logging.DEBUG)
if os.environ.get('OBS_SERVICE_DAEMON'):
    pack_directories = True
    get_assets = True
if os.environ.get('OSC_VERSION'):
    get_assets = True
os.environ['LANG'] = "C"

class ObsGit(object):
    def __init__(self, outdir, url):
        self.outdir   = outdir
        self.revision = None
        self.subdir   = None
        self.url = list(urllib.parse.urlparse(url))
        query = urllib.parse.parse_qs(self.url[4]);
        if "subdir" in query:
            self.subdir = query['subdir'][0]
            del query['subdir']
            self.url[4] = urllib.parse.urlencode(query)
        if self.url[5]:
            self.revision = self.url[5]
            self.url[5] = ''

def run_cmd(self, cmd, cwd=None, stdout=subprocess.PIPE, fatal=None):
        logging.debug("COMMAND: %s" % cmd)
        stderr = subprocess.PIPE
        if stdout == subprocess.PIPE:
            stderr = subprocess.STDOUT
        proc = subprocess.Popen(cmd,
                                shell=False,
                                stdout=stdout,
                                stderr=stderr,
                                cwd=cwd)
        output = proc.communicate()[0]
        if isinstance(output, bytes):
            output = output.decode('UTF-8')
        logging.debug("RESULT(%d): %s", proc.returncode, repr(output))
        if fatal and proc.returncode != 0:
            print("ERROR: " + fatal + " failed: ", output)
            sys.exit(proc.returncode)
        return (proc.returncode, output)

def do_clone_commit(self, outdir):
        cmd = [ 'git', 'init', outdir ]
        self.run_cmd(cmd, fatal="git init")
        cmd = [ 'git', '-C', outdir, 'remote', 'add', 'origin', urllib.parse.urlunparse(self.url) ]
        self.run_cmd(cmd, fatal="git remote add origin")
        cmd = [ 'git', '-C', outdir, 'fetch', '--depth', '1', 'origin', self.revision ]
        self.run_cmd(cmd, fatal="git fetch")
        cmd = [ 'git', '-C', outdir, 'checkout', '-q', self.revision ]
        self.run_cmd(cmd, fatal="git checkout")

def do_clone(self, outdir):
        if self.revision and re.match(r"^[0-9a-fA-F]{40,}$", self.revision):
            return self.do_clone_commit(outdir)
        cmd = [ 'git', 'clone', '--depth', '1', urllib.parse.urlunparse(self.url), outdir ]
        if self.revision:
            cmd.insert(2, '-b')
            cmd.insert(3, self.revision)
        self.run_cmd(cmd, fatal="git clone")

def clone(self):
        if not self.subdir:
            self.do_clone(self.outdir)
            return
        clonedir = tempfile.mkdtemp(prefix="obs-scm-bridge")
        self.do_clone(clonedir)
        fromdir = os.path.join(clonedir, self.subdir)
        if not os.path.isdir(fromdir):
            print("ERROR: subdir " + self.subdir + " does not exist")
            sys.exit(1)
        if not os.path.isdir(self.outdir):
            os.makedirs(self.outdir)
        for name in os.listdir(fromdir):
            shutil.move(os.path.join(fromdir, name), self.outdir)
        shutil.rmtree(clonedir)

def fetch_tags(self):
        cmd = [ 'git', '-C', self.outdir, 'fetch', '--tags', 'origin', '+refs/heads/*:refs/remotes/origin/*' ]
        logging.info("fetching all tags")
        self.run_cmd(cmd, fatal="fetch --tags")

def cpio_directory(self, directory):
        logging.info("create archivefile for %s", directory)
        cmd = [ download_assets, '--create-cpio', '--', directory ]
        archivefile = open(directory + '.obscpio', 'w')
        self.run_cmd(cmd, stdout=archivefile, fatal="cpio creation")
        archivefile.close()

def cpio_specials(self, specials):
        if not specials:
            return
        logging.info("create archivefile for specials")
        cmd = [ download_assets, '--create-cpio', '--', '.' ] + specials
        archivefile = open('build.specials.obscpio', 'w')
        self.run_cmd(cmd, stdout=archivefile, fatal="cpio creation")
        archivefile.close()

def cpio_directories(self):
        logging.debug("walk via %s", self.outdir)
        os.chdir(self.outdir)
        listing = sorted(os.listdir("."))
        specials = []
        for name in listing:
            if name == '.git':
                # we do not store git meta data service side atm to avoid bloat storage
                # however, this will break some builds, so we will need an opt-out in future
                shutil.rmtree(name)
                continue
            if name[0:1] == '.':
                specials.append(name)
                continue
            if os.path.islink(name):
                specials.append(name)
                continue
            if os.path.isdir(name):
                logging.info("CPIO %s ", name)
                self.cpio_directory(name)
                shutil.rmtree(name)
        if specials:
            self.cpio_specials(specials)
            for name in specials:
                if os.path.isdir(name):
                    shutil.rmtree(name)
                else:
                    os.unlink(name)

def get_assets(self):
        logging.info("downloading assets")
        if pack_directories:
            cmd = [ download_assets, '--noassetdir', '--', self.outdir ]
        else:
            cmd = [ download_assets, '--unpack', '--noassetdir', '--', self.outdir ]
        self.run_cmd(cmd, fatal="asset download")

def copyfile(self, src, dst):
        cmd = [ 'cp', '-af', self.outdir + "/" + src, self.outdir + "/" + dst ]
        self.run_cmd(cmd, fatal="file copy")

def export_debian_files(self):
        if os.path.isfile(self.outdir + "/debian/control") and \
                not os.path.isfile(self.outdir + "/debian.control"):
            self.copyfile("debian/control", "debian.control")
        if os.path.isfile(self.outdir + "/debian/changelog") and \
                not os.path.isfile(self.outdir + "/debian.changelog"):
            self.copyfile("debian/changelog", "debian.changelog")

def get_debian_origtar(self):
        if os.path.isfile(self.outdir + "/debian/control"):
            # need up get all tags 
            if not self.subdir:
                self.fetch_tags()
            cmd = [ export_debian_orig_from_git, self.outdir ]
            logging.info("exporting debian origtar")
            self.run_cmd(cmd, fatal="debian origtar export")

def get_subdir_info(self, dir):
        cmd = [ download_assets, '--show-dir-srcmd5', '--', dir ]
        rcode, info = self.run_cmd(cmd, fatal="download_assets --show-dir-srcmd5")
        return info.strip()

def write_info_file(self, filename, info):
        infofile = open(filename, 'w')
        infofile.write(info + '\n')
        infofile.close()

def add_service_info(self):
        info = None
        if self.subdir:
            info = self.get_subdir_info(self.outdir)
        else:
            cmd = [ 'git', '-C', outdir, 'show', '-s', '--pretty=format:%H', 'HEAD' ]
            rcode, info = self.run_cmd(cmd, fatal="git show -s HEAD")
            info = info.strip()
        if info:
            self.write_info_file(os.path.join(self.outdir, "_service_info"), info)

def write_package_xml_file(self, name, url):
        xmlfile = open(name + '.xml', 'w')
        xmlfile.write('<package name="' + escape(name) + '">\n')
        xmlfile.write('  <scmsync>' + escape(url) + '</scmsync>\n')
        xmlfile.write('</package>\n')
        xmlfile.close()

def generate_package_xml_files(self):
        logging.debug("walk via %s", self.outdir)
        os.chdir(self.outdir)
        export_files = set("_config")

# find all top level git submodules
        gitsubmodules = set()
        if os.path.isfile('.gitmodules'):
            gsmconfig = configparser.ConfigParser()
            gsmconfig.read('.gitmodules')
            for section in gsmconfig.sections():
                if not 'path' in gsmconfig[section]:
                    logging.warn("path not defined for git submodule " + section)
                    continue
                if not 'url' in gsmconfig[section]:
                    logging.warn("url not defined for git submodule " + section)
                    continue
                path = gsmconfig[section]['path']
                url = gsmconfig[section]['url']

if '/' in path:
                    # we handle only top level submodules in project mode
                    continue

# find revision of submodule
                cmd = [ 'git', 'ls-tree', 'HEAD', path ]
                rcode, output = self.run_cmd(cmd, fatal="git ls-tree")
                lstree = output.split()
                revision = lstree[2]
                if lstree[1] != 'commit' or len(revision) < 40:
                    logging.error("Invalid output of ls-tree for git submodule " + section)
                    sys.exit(1)

# all good, write xml file and register the module
                gitsubmodules.add(path)
                self.write_package_xml_file(path, url + '#' + revision)
                self.write_info_file(path + ".info", revision)
                export_files.add(path + ".xml")
                export_files.add(path + ".info")
                shutil.rmtree(path)

# handle plain files and directories
        listing = sorted(os.listdir("."))
        regexp =  re.compile(r"^[a-zA-Z0-9\.\-\_\+]*$");
        for name in listing:
            if name == '.git':
                shutil.rmtree(name)
                continue
            if os.path.isdir(name):
                if name in gitsubmodules:
                    # already handled as git submodule
                    continue

info = self.get_subdir_info(name)
                shutil.rmtree(name)
                if not regexp.match(name):
                    logging.warn("directory name contains invalid char: " + name)
                    continue

# add subdir info file
                self.write_info_file(name + ".info", info)

# add subdir parameter to url
                url = self.url
                query = urllib.parse.parse_qs(url[4])
                query['subdir'] = name
                url[4] = urllib.parse.urlencode(query)

self.write_package_xml_file(name, urllib.parse.urlunparse(url))
            else:
                if not name in export_files:
                    os.unlink(name)

if __name__ == '__main__':

parser = argparse.ArgumentParser(
        description='Open Build Service source service for managing packaging files in git.'
        'This is a special service for OBS git integration.')
    parser.add_argument('--outdir', required=True,
                        help='output directory for modified sources')
    parser.add_argument('--url',
                        help='REQUIRED: url to git repository')
    parser.add_argument('--projectmode',
                        help='just return the package list based on the subdirectories')
    parser.add_argument('--debug',
                        help='verbose debug mode')
    args = vars(parser.parse_args())

url = args['url']
    outdir = args['outdir']
    project_mode = args['projectmode']

if not outdir:
        print("no outdir specified")
        sys.exit(-1)

if not url:
        print("no url specified")
        sys.exit(-1)

if args['debug']:
        logging.getLogger().setLevel(logging.DEBUG)
        logging.debug("Running in debug mode")

# workflow
    obsgit = ObsGit(outdir, url)
    obsgit.clone()
    if project_mode == 'true' or project_mode == '1':
        obsgit.generate_package_xml_files()
        sys.exit(0)

if pack_directories:
        obsgit.add_service_info()
    if get_assets:
        obsgit.get_assets()
        obsgit.get_debian_origtar()
    if pack_directories:
        obsgit.export_debian_files()
        obsgit.cpio_directories()

07070100000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000B00000000TRAILER!!!34 blocks

Places

File _service:obs_scm:obs-scm-bridge-0.1~pre.1642586655.f1db846.obscpio of Package obs_scm_bridge

Places