Sign Up
Log In
Log In
or
Sign Up
Places
All Projects
Status Monitor
Collapse sidebar
Please login to access the resource
openSUSE:Leap:15.1:ARM:Staging
skopeo
bsc1115165-0001-Introduce-the-sync-command.patch
Overview
Repositories
Revisions
Requests
Users
Attributes
Meta
File bsc1115165-0001-Introduce-the-sync-command.patch of Package skopeo
From d09e51f4183a02e32e238aa5e9d2168cde4c4445 Mon Sep 17 00:00:00 2001 From: Flavio Castelli <fcastelli@suse.com> Date: Mon, 9 Jul 2018 15:27:01 +0200 Subject: [PATCH 1/3] Introduce the sync command The skopeo sync command can sync images between a SOURCE and a destination. The purpose of this command is to assist with the mirroring of container images from different docker registries to a single docker registry. Right now the following transport matrix is implemented: * `docker://` -> `docker://` * `docker://` -> `dir:` * `dir:` -> `docker://` The `dir:` transport is supported to handle the use case of air-gapped environments. In this context users can perform an initial sync on a trusted machine connected to the internet; that would be a `docker://` -> `dir:` sync. The target directory can be copied to a removable drive that can then be plugged into a node of the air-gapped environment. From there a `dir:` -> `docker://` sync will import all the images into the registry serving the air-gapped environment. The image namespace is changed during the `docker://` to `docker://` or `dir:` copy. The FQDN of the registry hosting the image will be added as new root namespace of the image. For example, the image `registry.example.com/busybox:latest` will be copied to `registry.local.lan/registry.example.com/busybox:latest`. The image namespace is not changed when doing a `dir:` -> `docker://` sync operation. The alteration of the image namespace is used to nicely scope images coming from different registries (the Docker Hub, quay.io, gcr, other registries). That allows all of them to be hosted on the same registry without incurring in clashes and making their origin explicit. Signed-off-by: Flavio Castelli <fcastelli@suse.com> Co-authored-by: Marco Vedovati <mvedovati@suse.com> --- cmd/skopeo/main.go | 1 + cmd/skopeo/sync.go | 564 ++++++++++++++++++++++++++++++++++++++++++++ cmd/skopeo/utils.go | 14 ++ docs/skopeo.1.md | 60 ++++- 4 files changed, 638 insertions(+), 1 deletion(-) create mode 100644 cmd/skopeo/sync.go diff --git a/cmd/skopeo/main.go b/cmd/skopeo/main.go index 78fa2af..f3ee825 100644 --- a/cmd/skopeo/main.go +++ b/cmd/skopeo/main.go @@ -80,6 +80,7 @@ func createApp() *cli.App { layersCmd, deleteCmd, manifestDigestCmd, + syncCmd, standaloneSignCmd, standaloneVerifyCmd, untrustedSignatureDumpCmd, diff --git a/cmd/skopeo/sync.go b/cmd/skopeo/sync.go new file mode 100644 index 0000000..8093a18 --- /dev/null +++ b/cmd/skopeo/sync.go @@ -0,0 +1,564 @@ +package main + +import ( + "context" + "fmt" + "net/url" + "os" + "path" + "path/filepath" + "strings" + + "github.com/containers/image/copy" + "github.com/containers/image/directory" + "github.com/containers/image/docker" + "github.com/containers/image/docker/reference" + "github.com/containers/image/transports" + "github.com/containers/image/types" + "github.com/pkg/errors" + "github.com/sirupsen/logrus" + "github.com/urfave/cli" +) + +type repoDescriptor struct { + DirBasePath string // base path when source is 'dir' + TaggedImages []types.ImageReference + Context *types.SystemContext +} + +type registrySyncCfg struct { + Images map[string][]string + Credentials types.DockerAuthConfig + TLSVerify bool `yaml:"tls-verify"` + CertDir string `yaml:"cert-dir"` +} + +type sourceCfg map[string]registrySyncCfg + +// Generates a config structure from a YAML file. +func newSourceConfig(yamlFile string) (cfg sourceCfg, err error) { + err = yamlUnmarshal(yamlFile, &cfg) + return +} + +// Checks if a given transport is supported by the sync operation. +func validSyncTransport(transport types.ImageTransport) bool { + switch transport { + case docker.Transport: + return true + case directory.Transport: + return true + } + + return false +} + +// Return a URL object from an input string +func parseURL(urlString string) (*url.URL, error) { + var parsedURL *url.URL + + parsedURL, err := url.Parse(urlString) + if err != nil { + return parsedURL, errors.WithMessage(err, "Error while parsing source") + } + + valid := validSyncTransport(transports.Get(parsedURL.Scheme)) + if !valid { + return parsedURL, errors.New("Invalid transport") + } + + return parsedURL, nil +} + +// Given a tranport and an image name (without the transport), returns an +// ImageReference. +func getImageReference(transport types.ImageTransport, imgName string) (types.ImageReference, error) { + ref, err := transport.ParseReference(imgName) + if err != nil { + return nil, errors.WithMessage(err, fmt.Sprintf("Cannot obtain a valid image reference from '%s'", imgName)) + } + + return ref, nil +} + +// Given a directory as a URL object, returns its string representation suitable +// to be used as a filesystem path +func dirPathFromURL(dirURL *url.URL) (string, error) { + var dirPath string + if dirURL.Scheme != directory.Transport.Name() { + return "", fmt.Errorf("Not an dir URL: %v", dirURL) + } + + if dirURL.Opaque != "" { + // relative dir path, e.g. dir:localdir (without "/" or "//") + dirPath = dirURL.Opaque + } else { + dirPath = path.Join("/", dirURL.Host, dirURL.Path) + } + + return dirPath, nil +} + +// Builds a destination image reference from a source image reference and +// a destination URL. +// Eg: +// source reference: docker://registry.example.com/library/busybox:stable +// destination URL: docker://my-registry.local.lan +// will return +// docker://my-registry.local.lan/registry.example.com/library/busybox:stable +// +// Note: when the source is a local directory, trimDirPath is trimmed from the +// source directory path, so that the destination scope is limited to what's inside +// host. +// Eg: +// source reference: dir:/home/user/syncfolder/registry.example.com/library/busybox:stable +// destination URL: docker://my-registry.local.lan +// will return +// docker://my-registry.local.lan/registry.example.com/library/busybox:stable +func buildFinalDestination(srcRef types.ImageReference, destURL *url.URL, trimDirPath string) (types.ImageReference, error) { + var destPath string + var finalDest string + + switch srcRef.Transport() { + case docker.Transport: + // docker -> dir or docker -> docker + destPath = srcRef.DockerReference().String() + case directory.Transport: + // dir -> docker (we don't allow `dir` -> `dir` sync operations) + destPath = strings.TrimPrefix(srcRef.StringWithinTransport(), trimDirPath) + // if source is a full path to an image, have destPath scoped to repo:tag + if destPath == "" { + destPath = path.Base(trimDirPath) + } + } + + destTransport := transports.Get(destURL.Scheme) + switch destTransport { + case docker.Transport: + finalDest = fmt.Sprintf("//%s", path.Join(destURL.Host, destURL.Path, destPath)) + case directory.Transport: + basePath, err := dirPathFromURL(destURL) + if err != nil { + return nil, errors.WithMessage(err, "Error processing destination URL") + } + finalDest = path.Join(basePath, destPath) + + logrus.Debugf("Creating dir path: %s", finalDest) + // the final directory holding the image must exist otherwise + // the directory ImageReference instance won't be created + if _, err := os.Stat(finalDest); err != nil { + if os.IsNotExist(err) { + if err := os.MkdirAll(finalDest, 0755); err != nil { + return nil, errors.WithMessage(err, fmt.Sprintf("Error creating directory for image %s", + finalDest)) + } + } else { + return nil, errors.WithMessage(err, fmt.Sprintf("Error checking existence of directory %s", + finalDest)) + } + } + } + logrus.Debugf("Final destination: %s", finalDest) + + destRef, err := getImageReference(destTransport, finalDest) + if err != nil { + return nil, err + } + + return destRef, nil +} + +// Retrieves all the tags associated to an image stored on a container registry. +func getImageTags(ctx context.Context, sysCtx *types.SystemContext, imgRef types.ImageReference) ([]string, error) { + name := imgRef.DockerReference().Name() + logrus.WithFields(logrus.Fields{ + "image": name, + }).Info("Getting tags") + tags, err := docker.GetRepositoryTags(ctx, sysCtx, imgRef) + + if err != nil { + // Some registries may decide to block the "list all tags" endpoint. + // Gracefully allow the sync to continue in this case. + if !strings.Contains(err.Error(), "401") { + return tags, errors.WithMessage(err, fmt.Sprintf("Error determining repository tags for image %s", name)) + } + logrus.Warnf("Registry disallows tag list retrieval: %s", err) + } + + return tags, nil +} + +// Checks if an image name name includes a tag. +func isTagSpecified(imageName string) (bool, error) { + normNamed, err := reference.ParseNormalizedNamed(imageName) + if err != nil { + return false, err + } + + tagged := !reference.IsNameOnly(normNamed) + logrus.WithFields(logrus.Fields{ + "imagename": imageName, + "tagged": tagged, + }).Info("Tag presence check") + return tagged, nil +} + +// Given an image reference on a container registry, returns a list of image +// references, one for each of the tags available for the given input image. +func imagesToCopyFromRegistry(srcRef types.ImageReference, src string, sourceCtx *types.SystemContext) (sourceReferences []types.ImageReference, retErr error) { + tags, err := getImageTags(context.Background(), sourceCtx, srcRef) + if err != nil { + return []types.ImageReference{}, err + } + for _, tag := range tags { + imageAndTag := fmt.Sprintf("%s:%s", src, tag) + ref, err := getImageReference(docker.Transport, imageAndTag) + if err != nil { + return []types.ImageReference{}, + errors.WithMessage(err, fmt.Sprintf("Error while building reference of %s", imageAndTag)) + } + sourceReferences = append(sourceReferences, ref) + } + return sourceReferences, retErr +} + +// Given an image reference as a local directory, returns all the image +// references available at the given path. +func imagesToCopyFromDir(dirPath string) (sourceReferences []types.ImageReference, retErr error) { + + if _, err := os.Stat(dirPath); err != nil { + return []types.ImageReference{}, + errors.WithMessage(err, fmt.Sprintf("Error checking for images in source path %q", dirPath)) + } + + err := filepath.Walk(dirPath, func(path string, info os.FileInfo, err error) error { + if err != nil { + return err + } + if !info.IsDir() && info.Name() == "manifest.json" { + ref, err := getImageReference(directory.Transport, fmt.Sprintf("%s", filepath.Dir(path))) + if err != nil { + return errors.WithMessage(err, fmt.Sprintf("Error while creating image referenced for path %s", + filepath.Dir(path))) + } + sourceReferences = append(sourceReferences, ref) + return filepath.SkipDir + } + return nil + }) + + if err != nil { + return []types.ImageReference{}, + errors.WithMessage(err, fmt.Sprintf("Error walking the path %q", dirPath)) + } + + return +} + +// Given a source URL and context, returns a list of tagged image references to +// be used as sync source. +func syncFromURL(sourceURL *url.URL, sourceCtx *types.SystemContext) (repoDescriptor, error) { + repoDesc := repoDescriptor{ + Context: sourceCtx, + } + + switch transports.Get(sourceURL.Scheme) { + case docker.Transport: + srcRef, err := getImageReference(docker.Transport, fmt.Sprintf("//%s%s", sourceURL.Host, sourceURL.Path)) + if err != nil { + return repoDesc, errors.WithMessage(err, "Error while parsing destination") + } + + imageTagged, err := isTagSpecified(sourceURL.Host + sourceURL.Path) + if err != nil { + return repoDesc, err + } + if imageTagged { + repoDesc.TaggedImages = append(repoDesc.TaggedImages, srcRef) + break + } + + repoName := fmt.Sprintf("//%s", path.Join(sourceURL.Host, sourceURL.Path)) + repoDesc.TaggedImages, err = imagesToCopyFromRegistry(srcRef, repoName, sourceCtx) + if err != nil { + return repoDesc, err + } + case directory.Transport: + dirPath, err := dirPathFromURL(sourceURL) + if err != nil { + return repoDesc, errors.WithMessage(err, "Error processing source URL") + } + + repoDesc.DirBasePath = dirPath + repoDesc.TaggedImages, err = imagesToCopyFromDir(dirPath) + if err != nil { + return repoDesc, err + } + } + + if len(repoDesc.TaggedImages) == 0 { + return repoDesc, errors.New("No images to sync found in SOURCE") + } + + return repoDesc, nil +} + +// Given a yaml file and a source context, returns a list of repository descriptors, +// each containing a list of tagged image references, to be used as sync source. +func syncFromYaml(yamlFile string, sourceCtx *types.SystemContext) (repoDescList []repoDescriptor, err error) { + cfg, err := newSourceConfig(yamlFile) + if err != nil { + return + } + + for server, serverCfg := range cfg { + if len(serverCfg.Images) == 0 { + logrus.WithFields(logrus.Fields{ + "registry": server, + }).Warnf("No images specified for registry") + continue + } + + for imageName, tags := range serverCfg.Images { + repoName := fmt.Sprintf("//%s", path.Join(server, imageName)) + logrus.WithFields(logrus.Fields{ + "repo": imageName, + "registry": server, + }).Info("Processing repo") + + serverCtx := sourceCtx + // override ctx with per-server options + serverCtx.DockerCertPath = serverCfg.CertDir + serverCtx.DockerDaemonCertPath = serverCfg.CertDir + serverCtx.DockerDaemonInsecureSkipTLSVerify = !serverCfg.TLSVerify + serverCtx.DockerInsecureSkipTLSVerify = !serverCfg.TLSVerify + serverCtx.DockerAuthConfig = &serverCfg.Credentials + + var sourceReferences []types.ImageReference + for _, tag := range tags { + source := fmt.Sprintf("%s:%s", repoName, tag) + + imageRef, err := docker.ParseReference(source) + if err != nil { + logrus.WithFields(logrus.Fields{ + "tag": source, + }).Error("Error processing tag, skipping") + logrus.Errorf("Error getting image reference: %s", err) + continue + } + sourceReferences = append(sourceReferences, imageRef) + } + + if len(tags) == 0 { + logrus.WithFields(logrus.Fields{ + "repo": imageName, + "registry": server, + }).Info("Querying registry for image tags") + + imageRef, err := docker.ParseReference(repoName) + if err != nil { + logrus.WithFields(logrus.Fields{ + "repo": imageName, + "registry": server, + }).Error("Error processing repo, skipping") + logrus.Error(err) + continue + } + + sourceReferences, err = imagesToCopyFromRegistry(imageRef, repoName, serverCtx) + if err != nil { + logrus.WithFields(logrus.Fields{ + "repo": imageName, + "registry": server, + }).Error("Error processing repo, skipping") + logrus.Error(err) + continue + } + } + + if len(sourceReferences) == 0 { + logrus.WithFields(logrus.Fields{ + "repo": imageName, + "registry": server, + }).Warnf("No tags to sync found") + continue + } + repoDescList = append(repoDescList, repoDescriptor{ + TaggedImages: sourceReferences, + Context: serverCtx}) + } + } + + return +} + +func syncHandler(c *cli.Context) (retErr error) { + if len(c.Args()) != 2 { + cli.ShowCommandHelp(c, "sync") + return errors.New("Exactly one argument expected") + } + + policyContext, err := getPolicyContext(c) + if err != nil { + return errors.WithMessage(err, "Error loading trust policy") + } + defer policyContext.Destroy() + + destinationURL, err := parseURL(c.Args()[1]) + if err != nil { + return errors.WithMessage(err, "Error while parsing destination") + } + destinationCtx, err := contextFromGlobalOptions(c, "dest-") + if err != nil { + return err + } + + signBy := c.String("sign-by") + removeSignatures := c.Bool("remove-signatures") + + sourceCtx, err := contextFromGlobalOptions(c, "src-") + if err != nil { + return err + } + sourceArg := c.Args()[0] + + var srcRepoList []repoDescriptor + + if c.IsSet("source-yaml") { + srcRepoList, err = syncFromYaml(sourceArg, sourceCtx) + if err != nil { + return err + } + } else { + sourceURL, err := parseURL(sourceArg) + if err != nil { + return errors.WithMessage(err, "Error while parsing source") + } + + if transports.Get(sourceURL.Scheme) == directory.Transport && + sourceURL.Scheme == destinationURL.Scheme { + return errors.New("sync from 'dir:' to 'dir:' not implemented, use something like rsync instead") + } + + srcRepo, err := syncFromURL(sourceURL, sourceCtx) + if err != nil { + return err + } + srcRepoList = append(srcRepoList, srcRepo) + } + + ctx, cancel := commandTimeoutContextFromGlobalOptions(c) + defer cancel() + + var imgCounter int + for _, srcRepo := range srcRepoList { + options := copy.Options{ + RemoveSignatures: removeSignatures, + SignBy: signBy, + ReportWriter: os.Stdout, + DestinationCtx: destinationCtx, + SourceCtx: srcRepo.Context, + } + + for counter, ref := range srcRepo.TaggedImages { + destRef, err := buildFinalDestination(ref, destinationURL, srcRepo.DirBasePath) + if err != nil { + return err + } + + logrus.WithFields(logrus.Fields{ + "from": transports.ImageName(ref), + "to": transports.ImageName(destRef), + }).Infof("Copying image tag %d/%d", counter+1, len(srcRepo.TaggedImages)) + + _, err = copy.Image(ctx, policyContext, destRef, ref, &options) + if err != nil { + return errors.WithMessage(err, fmt.Sprintf("Error copying tag '%s'", transports.ImageName(ref))) + } + imgCounter++ + } + } + + logrus.Infof("Synced %d images from %d sources", imgCounter, len(srcRepoList)) + + return nil +} + +var syncCmd = cli.Command{ + Name: "sync", + Usage: "Sync one or more images from one location to another", + Description: fmt.Sprint(` + + Copy all the images from SOURCE to DESTINATION. + + Useful to keep in sync a local container registry mirror. Can be used + to populate also registries running inside of air-gapped environments. + + SOURCE can be either a repository hosted on a container registry + (eg: docker://registry.example.com/busybox) or a local directory + (eg: dir:/media/usb/). + + If --source-yaml is specified, then SOURCE points to a YAML file with + a list of source images from different container registries + (local directories are not supported). + + When the source location is a container registry and no tags are specified, + skopeo sync will copy all the tags associated to the source image. + + DESTINATION can be either a container registry + (eg: docker://my-registry.local.lan) or a local directory + (eg: dir:/media/usb). + + When DESTINATION is a local directory, one directory per 'image:tag' is going + to be created. + `), + ArgsUsage: "[--source-yaml] SOURCE DESTINATION", + Action: syncHandler, + // FIXME: Do we need to namespace the GPG aspect? + Flags: []cli.Flag{ + cli.StringFlag{ + Name: "authfile", + Usage: "path of the authentication file. Default is ${XDG_RUNTIME_DIR}/containers/auth.json", + }, + cli.BoolFlag{ + Name: "remove-signatures", + Usage: "Do not copy signatures from SOURCE images", + }, + cli.StringFlag{ + Name: "sign-by", + Usage: "Sign the image using a GPG key with the specified `FINGERPRINT`", + }, + cli.BoolFlag{ + Name: "source-yaml", + Usage: "Interpret SOURCE as a YAML file with a list of images from different container registries", + }, + cli.StringFlag{ + Name: "src-creds, screds", + Value: "", + Usage: "Use `USERNAME[:PASSWORD]` for accessing the source registry", + }, + cli.StringFlag{ + Name: "dest-creds, dcreds", + Value: "", + Usage: "Use `USERNAME[:PASSWORD]` for accessing the destination registry", + }, + cli.StringFlag{ + Name: "src-cert-dir", + Value: "", + Usage: "use certificates at `PATH` (*.crt, *.cert, *.key) to connect to the source registry or daemon", + }, + cli.BoolTFlag{ + Name: "src-tls-verify", + Usage: "require HTTPS and verify certificates when talking to the container source registry or daemon (defaults to true)", + }, + cli.StringFlag{ + Name: "dest-cert-dir", + Value: "", + Usage: "use certificates at `PATH` (*.crt, *.cert, *.key) to connect to the destination registry or daemon", + }, + cli.BoolTFlag{ + Name: "dest-tls-verify", + Usage: "require HTTPS and verify certificates when talking to the container destination registry or daemon (defaults to true)", + }, + }, +} diff --git a/cmd/skopeo/utils.go b/cmd/skopeo/utils.go index 4dd5308..4f9e798 100644 --- a/cmd/skopeo/utils.go +++ b/cmd/skopeo/utils.go @@ -3,6 +3,8 @@ package main import ( "context" "errors" + "gopkg.in/yaml.v2" + "io/ioutil" "strings" "github.com/containers/image/transports/alltransports" @@ -102,3 +104,15 @@ func parseImageSource(ctx context.Context, c *cli.Context, name string) (types.I } return ref.NewImageSource(ctx, sys) } + +func yamlUnmarshal(yamlFile string, cfg interface{}) error { + source, err := ioutil.ReadFile(yamlFile) + if err != nil { + return err + } + err = yaml.Unmarshal(source, cfg) + if err != nil { + return err + } + return nil +} diff --git a/docs/skopeo.1.md b/docs/skopeo.1.md index 91afa63..29394a3 100644 --- a/docs/skopeo.1.md +++ b/docs/skopeo.1.md @@ -194,6 +194,45 @@ Verify a signature using local files, digest will be printed on success. **Note:** If you do use this, make sure that the image can not be changed at the source location between the times of its verification and use. +## skopeo sync +**skopeo sync** [**--source-yaml**] _source_ _destination_ + +Copy all the images from _source_ to _destination_. + +Useful to keep in sync with a local container registry mirror. It can also be used to populate registries running inside of air-gapped environments. + +_source_ can be either a repository hosted on a container registry (eg: docker://registry.example.com/busybox) or a local directory (eg: dir:/media/usb/). + +If **--source-yaml** is specified, then _source_ points to a YAML file with a list of source images from different container registries (local directories are not supported). + +When the source location is a container registry and no tags are specified, **skopeo sync** will copy all the tags associated to the source image. + +_destination_ can be either a container registry (eg: docker://my-registry.local.lan) or a local directory (eg: dir:/media/usb). + +When _destination_ is a local directory, one directory per 'image:tag' will be created. + + **--authfile** _path_ + + Path of the authentication file. Default is ${XDG_RUNTIME_DIR}/containers/auth.json + + **--remove-signatures** do not copy signatures from _source_ images + + **--sign-by** _fingerprint_ Sign the image using a GPG key with the specified _fingerprint_ + + **--source-yaml** Interpret _source_ as a YAML file with a list of images from different container registries + + **--src-creds** _username[:password]_, --screds _username[:password]_ Use _username[:password]_ for accessing the source registry + + **--dest-creds** _username[:password]_, --dcreds _username[:password]_ Use _username[:password]_ for accessing the destination registry + + **--src-cert-dir** _path_ Use certificates at _path_ (*.crt, *.cert, *.key) to connect to the source registry or daemon + + **--src-tls-verify** Require HTTPS and verify certificates when talking to the container source registry or daemon (defaults to true) + + **--dest-cert-dir** _path_ Use certificates at _path_ (*.crt, *.cert, *.key) to connect to the destination registry or daemon + + **--dest-tls-verify** Require HTTPS and verify certificates when talking to the container destination registry or daemon (defaults to true) + ## skopeo help show help for `skopeo` @@ -284,9 +323,28 @@ See `skopeo copy` above for the preferred method of signing images. $ skopeo standalone-verify busybox-manifest.json registry.example.com/example/busybox 1D8230F6CDB6A06716E414C1DB72F2188BB46CC8 busybox.signature Signature verified, digest sha256:20bf21ed457b390829cdbeec8795a7bea1626991fda603e0d01b4e7f60427e55 ``` +## skopeo sync +Example of the YAML file content when using **--source-yaml**: +```yaml +docker.io: + images: + busybox: [] + redis: + - "1.0" + - "2.0" + credentials: + username: john + password: this is a secret + tls-verify: true + cert-dir: /home/john/certs +quay.io: + images: + coreos/etcd: + - latest +``` # SEE ALSO -kpod-login(1), docker-login(1) +podman-login(1), docker-login(1) # AUTHORS -- 2.19.1
Locations
Projects
Search
Status Monitor
Help
OpenBuildService.org
Documentation
API Documentation
Code of Conduct
Contact
Support
@OBShq
Terms
openSUSE Build Service is sponsored by
The Open Build Service is an
openSUSE project
.
Sign Up
Log In
Places
Places
All Projects
Status Monitor