working commit

This commit is contained in:
2026-02-21 13:16:56 +02:00
parent d650d58a6d
commit 7be3cf8de7
1136 changed files with 722443 additions and 0 deletions
+117
View File
@@ -0,0 +1,117 @@
# `remote`
[![GoDoc](https://godoc.org/github.com/google/go-containerregistry/pkg/v1/remote?status.svg)](https://godoc.org/github.com/google/go-containerregistry/pkg/v1/remote)
The `remote` package implements a client for accessing a registry,
per the [OCI distribution spec](https://github.com/opencontainers/distribution-spec/blob/master/spec.md).
It leans heavily on the lower level [`transport`](/pkg/v1/remote/transport) package, which handles the
authentication handshake and structured errors.
## Usage
```go
package main
import (
"github.com/google/go-containerregistry/pkg/authn"
"github.com/google/go-containerregistry/pkg/name"
"github.com/google/go-containerregistry/pkg/v1/remote"
)
func main() {
ref, err := name.ParseReference("gcr.io/google-containers/pause")
if err != nil {
panic(err)
}
img, err := remote.Image(ref, remote.WithAuthFromKeychain(authn.DefaultKeychain))
if err != nil {
panic(err)
}
// do stuff with img
}
```
## Structure
<p align="center">
<img src="/images/remote.dot.svg" />
</p>
## Background
There are a lot of confusingly similar terms that come up when talking about images in registries.
### Anatomy of an image
In general...
* A tag refers to an image manifest.
* An image manifest references a config file and an orderered list of _compressed_ layers by sha256 digest.
* A config file references an ordered list of _uncompressed_ layers by sha256 digest and contains runtime configuration.
* The sha256 digest of the config file is the [image id](https://github.com/opencontainers/image-spec/blob/master/config.md#imageid) for the image.
For example, an image with two layers would look something like this:
![image anatomy](/images/image-anatomy.dot.svg)
### Anatomy of an index
In the normal case, an [index](https://github.com/opencontainers/image-spec/blob/master/image-index.md) is used to represent a multi-platform image.
This was the original use case for a [manifest
list](https://docs.docker.com/registry/spec/manifest-v2-2/#manifest-list).
![image index anatomy](/images/index-anatomy.dot.svg)
It is possible for an index to reference another index, per the OCI
[image-spec](https://github.com/opencontainers/image-spec/blob/master/media-types.md#compatibility-matrix).
In theory, both an image and image index can reference arbitrary things via
[descriptors](https://github.com/opencontainers/image-spec/blob/master/descriptor.md),
e.g. see the [image layout
example](https://github.com/opencontainers/image-spec/blob/master/image-layout.md#index-example),
which references an application/xml file from an image index.
That could look something like this:
![strange image index anatomy](/images/index-anatomy-strange.dot.svg)
Using a recursive index like this might not be possible with all registries,
but this flexibility allows for some interesting applications, e.g. the
[OCI Artifacts](https://github.com/opencontainers/artifacts) effort.
### Anatomy of an image upload
The structure of an image requires a delicate ordering when uploading an image to a registry.
Below is a (slightly simplified) figure that describes how an image is prepared for upload
to a registry and how the data flows between various artifacts:
![upload](/images/upload.dot.svg)
Note that:
* A config file references the uncompressed layer contents by sha256.
* A manifest references the compressed layer contents by sha256 and the size of the layer.
* A manifest references the config file contents by sha256 and the size of the file.
It follows that during an upload, we need to upload layers before the config file,
and we need to upload the config file before the manifest.
Sometimes, we know all of this information ahead of time, (e.g. when copying from remote.Image),
so the ordering is less important.
In other cases, e.g. when using a [`stream.Layer`](https://godoc.org/github.com/google/go-containerregistry/pkg/v1/stream#Layer),
we can't compute anything until we have already uploaded the layer, so we need to be careful about ordering.
## Caveats
### schema 1
This package does not support schema 1 images, see [`#377`](https://github.com/google/go-containerregistry/issues/377),
however, it's possible to do _something_ useful with them via [`remote.Get`](https://godoc.org/github.com/google/go-containerregistry/pkg/v1/remote#Get),
which doesn't try to interpret what is returned by the registry.
[`crane.Copy`](https://godoc.org/github.com/google/go-containerregistry/pkg/crane#Copy) takes advantage of this to implement support for copying schema 1 images,
see [here](https://github.com/google/go-containerregistry/blob/main/pkg/internal/legacy/copy.go).
+159
View File
@@ -0,0 +1,159 @@
// Copyright 2019 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package remote
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/url"
"github.com/google/go-containerregistry/pkg/name"
"github.com/google/go-containerregistry/pkg/v1/remote/transport"
)
type Catalogs struct {
Repos []string `json:"repositories"`
Next string `json:"next,omitempty"`
}
// CatalogPage calls /_catalog, returning the list of repositories on the registry.
func CatalogPage(target name.Registry, last string, n int, options ...Option) ([]string, error) {
o, err := makeOptions(options...)
if err != nil {
return nil, err
}
f, err := newPuller(o).fetcher(o.context, target)
if err != nil {
return nil, err
}
uri := url.URL{
Scheme: target.Scheme(),
Host: target.RegistryStr(),
Path: "/v2/_catalog",
RawQuery: fmt.Sprintf("last=%s&n=%d", url.QueryEscape(last), n),
}
req, err := http.NewRequest(http.MethodGet, uri.String(), nil)
if err != nil {
return nil, err
}
resp, err := f.client.Do(req.WithContext(o.context))
if err != nil {
return nil, err
}
defer resp.Body.Close()
if err := transport.CheckError(resp, http.StatusOK); err != nil {
return nil, err
}
var parsed Catalogs
if err := json.NewDecoder(resp.Body).Decode(&parsed); err != nil {
return nil, err
}
return parsed.Repos, nil
}
// Catalog calls /_catalog, returning the list of repositories on the registry.
func Catalog(ctx context.Context, target name.Registry, options ...Option) ([]string, error) {
o, err := makeOptions(options...)
if err != nil {
return nil, err
}
// WithContext overrides the ctx passed directly.
if o.context != context.Background() {
ctx = o.context
}
return newPuller(o).catalog(ctx, target, o.pageSize)
}
func (f *fetcher) catalogPage(ctx context.Context, reg name.Registry, next string, pageSize int) (*Catalogs, error) {
if next == "" {
uri := &url.URL{
Scheme: reg.Scheme(),
Host: reg.RegistryStr(),
Path: "/v2/_catalog",
}
if pageSize > 0 {
uri.RawQuery = fmt.Sprintf("n=%d", pageSize)
}
next = uri.String()
}
req, err := http.NewRequestWithContext(ctx, "GET", next, nil)
if err != nil {
return nil, err
}
resp, err := f.client.Do(req)
if err != nil {
return nil, err
}
if err := transport.CheckError(resp, http.StatusOK); err != nil {
return nil, err
}
parsed := Catalogs{}
if err := json.NewDecoder(resp.Body).Decode(&parsed); err != nil {
return nil, err
}
if err := resp.Body.Close(); err != nil {
return nil, err
}
uri, err := getNextPageURL(resp)
if err != nil {
return nil, err
}
if uri != nil {
parsed.Next = uri.String()
}
return &parsed, nil
}
type Catalogger struct {
f *fetcher
reg name.Registry
pageSize int
page *Catalogs
err error
needMore bool
}
func (l *Catalogger) Next(ctx context.Context) (*Catalogs, error) {
if l.needMore {
l.page, l.err = l.f.catalogPage(ctx, l.reg, l.page.Next, l.pageSize)
} else {
l.needMore = true
}
return l.page, l.err
}
func (l *Catalogger) HasNext() bool {
return l.page != nil && (!l.needMore || l.page.Next != "")
}
+72
View File
@@ -0,0 +1,72 @@
// Copyright 2019 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package remote
import (
"context"
"fmt"
"net/http"
"github.com/google/go-containerregistry/pkg/authn"
"github.com/google/go-containerregistry/pkg/name"
"github.com/google/go-containerregistry/pkg/v1/remote/transport"
)
// CheckPushPermission returns an error if the given keychain cannot authorize
// a push operation to the given ref.
//
// This can be useful to check whether the caller has permission to push an
// image before doing work to construct the image.
//
// TODO(#412): Remove the need for this method.
func CheckPushPermission(ref name.Reference, kc authn.Keychain, t http.RoundTripper) error {
auth, err := kc.Resolve(ref.Context().Registry)
if err != nil {
return fmt.Errorf("resolving authorization for %v failed: %w", ref.Context().Registry, err)
}
scopes := []string{ref.Scope(transport.PushScope)}
tr, err := transport.NewWithContext(context.TODO(), ref.Context().Registry, auth, t, scopes)
if err != nil {
return fmt.Errorf("creating push check transport for %v failed: %w", ref.Context().Registry, err)
}
// TODO(jasonhall): Against GCR, just doing the token handshake is
// enough, but this doesn't extend to Dockerhub
// (https://github.com/docker/hub-feedback/issues/1771), so we actually
// need to initiate an upload to tell whether the credentials can
// authorize a push. Figure out how to return early here when we can,
// to avoid a roundtrip for spec-compliant registries.
w := writer{
repo: ref.Context(),
client: &http.Client{Transport: tr},
}
loc, _, err := w.initiateUpload(context.Background(), "", "", "")
if loc != "" {
// Since we're only initiating the upload to check whether we
// can, we should attempt to cancel it, in case initiating
// reserves some resources on the server. We shouldn't wait for
// cancelling to complete, and we don't care if it fails.
go w.cancelUpload(loc)
}
return err
}
func (w *writer) cancelUpload(loc string) {
req, err := http.NewRequest(http.MethodDelete, loc, nil)
if err != nil {
return
}
_, _ = w.client.Do(req)
}
+28
View File
@@ -0,0 +1,28 @@
// Copyright 2018 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package remote
import (
"github.com/google/go-containerregistry/pkg/name"
)
// Delete removes the specified image reference from the remote registry.
func Delete(ref name.Reference, options ...Option) error {
o, err := makeOptions(options...)
if err != nil {
return err
}
return newPusher(o).Delete(o.context, ref)
}
@@ -0,0 +1,198 @@
// Copyright 2018 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package remote
import (
"context"
"errors"
"fmt"
"github.com/google/go-containerregistry/pkg/logs"
"github.com/google/go-containerregistry/pkg/name"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/partial"
"github.com/google/go-containerregistry/pkg/v1/types"
)
var allManifestMediaTypes = append(append([]types.MediaType{
types.DockerManifestSchema1,
types.DockerManifestSchema1Signed,
}, acceptableImageMediaTypes...), acceptableIndexMediaTypes...)
// ErrSchema1 indicates that we received a schema1 manifest from the registry.
// This library doesn't have plans to support this legacy image format:
// https://github.com/google/go-containerregistry/issues/377
var ErrSchema1 = errors.New("see https://github.com/google/go-containerregistry/issues/377")
// newErrSchema1 returns an ErrSchema1 with the unexpected MediaType.
func newErrSchema1(schema types.MediaType) error {
return fmt.Errorf("unsupported MediaType: %q, %w", schema, ErrSchema1)
}
// Descriptor provides access to metadata about remote artifact and accessors
// for efficiently converting it into a v1.Image or v1.ImageIndex.
type Descriptor struct {
fetcher fetcher
v1.Descriptor
ref name.Reference
Manifest []byte
ctx context.Context
// So we can share this implementation with Image.
platform v1.Platform
}
func (d *Descriptor) toDesc() v1.Descriptor {
return d.Descriptor
}
// RawManifest exists to satisfy the Taggable interface.
func (d *Descriptor) RawManifest() ([]byte, error) {
return d.Manifest, nil
}
// Get returns a remote.Descriptor for the given reference. The response from
// the registry is left un-interpreted, for the most part. This is useful for
// querying what kind of artifact a reference represents.
//
// See Head if you don't need the response body.
func Get(ref name.Reference, options ...Option) (*Descriptor, error) {
return get(ref, allManifestMediaTypes, options...)
}
// Head returns a v1.Descriptor for the given reference by issuing a HEAD
// request.
//
// Note that the server response will not have a body, so any errors encountered
// should be retried with Get to get more details.
func Head(ref name.Reference, options ...Option) (*v1.Descriptor, error) {
o, err := makeOptions(options...)
if err != nil {
return nil, err
}
return newPuller(o).Head(o.context, ref)
}
// Handle options and fetch the manifest with the acceptable MediaTypes in the
// Accept header.
func get(ref name.Reference, acceptable []types.MediaType, options ...Option) (*Descriptor, error) {
o, err := makeOptions(options...)
if err != nil {
return nil, err
}
return newPuller(o).get(o.context, ref, acceptable, o.platform)
}
// Image converts the Descriptor into a v1.Image.
//
// If the fetched artifact is already an image, it will just return it.
//
// If the fetched artifact is an index, it will attempt to resolve the index to
// a child image with the appropriate platform.
//
// See WithPlatform to set the desired platform.
func (d *Descriptor) Image() (v1.Image, error) {
switch d.MediaType {
case types.DockerManifestSchema1, types.DockerManifestSchema1Signed:
// We don't care to support schema 1 images:
// https://github.com/google/go-containerregistry/issues/377
return nil, newErrSchema1(d.MediaType)
case types.OCIImageIndex, types.DockerManifestList:
// We want an image but the registry has an index, resolve it to an image.
return d.remoteIndex().imageByPlatform(d.platform)
case types.OCIManifestSchema1, types.DockerManifestSchema2:
// These are expected. Enumerated here to allow a default case.
default:
// We could just return an error here, but some registries (e.g. static
// registries) don't set the Content-Type headers correctly, so instead...
logs.Warn.Printf("Unexpected media type for Image(): %s", d.MediaType)
}
// Wrap the v1.Layers returned by this v1.Image in a hint for downstream
// remote.Write calls to facilitate cross-repo "mounting".
imgCore, err := partial.CompressedToImage(d.remoteImage())
if err != nil {
return nil, err
}
return &mountableImage{
Image: imgCore,
Reference: d.ref,
}, nil
}
// Schema1 converts the Descriptor into a v1.Image for v2 schema 1 media types.
//
// The v1.Image returned by this method does not implement the entire interface because it would be inefficient.
// This exists mostly to make it easier to copy schema 1 images around or look at their filesystems.
// This is separate from Image() to avoid a backward incompatible change for callers expecting ErrSchema1.
func (d *Descriptor) Schema1() (v1.Image, error) {
i := &schema1{
ref: d.ref,
fetcher: d.fetcher,
ctx: d.ctx,
manifest: d.Manifest,
mediaType: d.MediaType,
descriptor: &d.Descriptor,
}
return &mountableImage{
Image: i,
Reference: d.ref,
}, nil
}
// ImageIndex converts the Descriptor into a v1.ImageIndex.
func (d *Descriptor) ImageIndex() (v1.ImageIndex, error) {
switch d.MediaType {
case types.DockerManifestSchema1, types.DockerManifestSchema1Signed:
// We don't care to support schema 1 images:
// https://github.com/google/go-containerregistry/issues/377
return nil, newErrSchema1(d.MediaType)
case types.OCIManifestSchema1, types.DockerManifestSchema2:
// We want an index but the registry has an image, nothing we can do.
return nil, fmt.Errorf("unexpected media type for ImageIndex(): %s; call Image() instead", d.MediaType)
case types.OCIImageIndex, types.DockerManifestList:
// These are expected.
default:
// We could just return an error here, but some registries (e.g. static
// registries) don't set the Content-Type headers correctly, so instead...
logs.Warn.Printf("Unexpected media type for ImageIndex(): %s", d.MediaType)
}
return d.remoteIndex(), nil
}
func (d *Descriptor) remoteImage() *remoteImage {
return &remoteImage{
ref: d.ref,
ctx: d.ctx,
fetcher: d.fetcher,
manifest: d.Manifest,
mediaType: d.MediaType,
descriptor: &d.Descriptor,
}
}
func (d *Descriptor) remoteIndex() *remoteIndex {
return &remoteIndex{
ref: d.ref,
ctx: d.ctx,
fetcher: d.fetcher,
manifest: d.Manifest,
mediaType: d.MediaType,
descriptor: &d.Descriptor,
}
}
+17
View File
@@ -0,0 +1,17 @@
// Copyright 2018 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package remote provides facilities for reading/writing v1.Images from/to
// a remote image registry.
package remote
+317
View File
@@ -0,0 +1,317 @@
// Copyright 2023 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package remote
import (
"bytes"
"context"
"fmt"
"io"
"net/http"
"net/url"
"strings"
"github.com/google/go-containerregistry/internal/redact"
"github.com/google/go-containerregistry/internal/verify"
"github.com/google/go-containerregistry/pkg/authn"
"github.com/google/go-containerregistry/pkg/name"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/remote/transport"
"github.com/google/go-containerregistry/pkg/v1/types"
)
const (
kib = 1024
mib = 1024 * kib
manifestLimit = 100 * mib
)
// fetcher implements methods for reading from a registry.
type fetcher struct {
target resource
client *http.Client
}
func makeFetcher(ctx context.Context, target resource, o *options) (*fetcher, error) {
auth := o.auth
if o.keychain != nil {
kauth, err := authn.Resolve(ctx, o.keychain, target)
if err != nil {
return nil, err
}
auth = kauth
}
reg, ok := target.(name.Registry)
if !ok {
repo, ok := target.(name.Repository)
if !ok {
return nil, fmt.Errorf("unexpected resource: %T", target)
}
reg = repo.Registry
}
tr, err := transport.NewWithContext(ctx, reg, auth, o.transport, []string{target.Scope(transport.PullScope)})
if err != nil {
return nil, err
}
return &fetcher{
target: target,
client: &http.Client{Transport: tr},
}, nil
}
func (f *fetcher) Do(req *http.Request) (*http.Response, error) {
return f.client.Do(req)
}
type resource interface {
Scheme() string
RegistryStr() string
Scope(string) string
authn.Resource
}
// url returns a url.Url for the specified path in the context of this remote image reference.
func (f *fetcher) url(resource, identifier string) url.URL {
u := url.URL{
Scheme: f.target.Scheme(),
Host: f.target.RegistryStr(),
// Default path if this is not a repository.
Path: "/v2/_catalog",
}
if repo, ok := f.target.(name.Repository); ok {
u.Path = fmt.Sprintf("/v2/%s/%s/%s", repo.RepositoryStr(), resource, identifier)
}
return u
}
func (f *fetcher) get(ctx context.Context, ref name.Reference, acceptable []types.MediaType, platform v1.Platform) (*Descriptor, error) {
b, desc, err := f.fetchManifest(ctx, ref, acceptable)
if err != nil {
return nil, err
}
return &Descriptor{
ref: ref,
ctx: ctx,
fetcher: *f,
Manifest: b,
Descriptor: *desc,
platform: platform,
}, nil
}
func (f *fetcher) fetchManifest(ctx context.Context, ref name.Reference, acceptable []types.MediaType) ([]byte, *v1.Descriptor, error) {
u := f.url("manifests", ref.Identifier())
req, err := http.NewRequest(http.MethodGet, u.String(), nil)
if err != nil {
return nil, nil, err
}
accept := []string{}
for _, mt := range acceptable {
accept = append(accept, string(mt))
}
req.Header.Set("Accept", strings.Join(accept, ","))
resp, err := f.client.Do(req.WithContext(ctx))
if err != nil {
return nil, nil, err
}
defer resp.Body.Close()
if err := transport.CheckError(resp, http.StatusOK); err != nil {
return nil, nil, err
}
manifest, err := io.ReadAll(io.LimitReader(resp.Body, manifestLimit))
if err != nil {
return nil, nil, err
}
digest, size, err := v1.SHA256(bytes.NewReader(manifest))
if err != nil {
return nil, nil, err
}
mediaType := types.MediaType(resp.Header.Get("Content-Type"))
contentDigest, err := v1.NewHash(resp.Header.Get("Docker-Content-Digest"))
if err == nil && mediaType == types.DockerManifestSchema1Signed {
// If we can parse the digest from the header, and it's a signed schema 1
// manifest, let's use that for the digest to appease older registries.
digest = contentDigest
}
// Validate the digest matches what we asked for, if pulling by digest.
if dgst, ok := ref.(name.Digest); ok {
if digest.String() != dgst.DigestStr() {
return nil, nil, fmt.Errorf("manifest digest: %q does not match requested digest: %q for %q", digest, dgst.DigestStr(), ref)
}
}
var artifactType string
mf, _ := v1.ParseManifest(bytes.NewReader(manifest))
// Failing to parse as a manifest should just be ignored.
// The manifest might not be valid, and that's okay.
if mf != nil && !mf.Config.MediaType.IsConfig() {
artifactType = string(mf.Config.MediaType)
}
// Do nothing for tags; I give up.
//
// We'd like to validate that the "Docker-Content-Digest" header matches what is returned by the registry,
// but so many registries implement this incorrectly that it's not worth checking.
//
// For reference:
// https://github.com/GoogleContainerTools/kaniko/issues/298
// Return all this info since we have to calculate it anyway.
desc := v1.Descriptor{
Digest: digest,
Size: size,
MediaType: mediaType,
ArtifactType: artifactType,
}
return manifest, &desc, nil
}
func (f *fetcher) headManifest(ctx context.Context, ref name.Reference, acceptable []types.MediaType) (*v1.Descriptor, error) {
u := f.url("manifests", ref.Identifier())
req, err := http.NewRequest(http.MethodHead, u.String(), nil)
if err != nil {
return nil, err
}
accept := []string{}
for _, mt := range acceptable {
accept = append(accept, string(mt))
}
req.Header.Set("Accept", strings.Join(accept, ","))
resp, err := f.client.Do(req.WithContext(ctx))
if err != nil {
return nil, err
}
defer resp.Body.Close()
if err := transport.CheckError(resp, http.StatusOK); err != nil {
return nil, err
}
mth := resp.Header.Get("Content-Type")
if mth == "" {
return nil, fmt.Errorf("HEAD %s: response did not include Content-Type header", u.String())
}
mediaType := types.MediaType(mth)
size := resp.ContentLength
if size == -1 {
return nil, fmt.Errorf("GET %s: response did not include Content-Length header", u.String())
}
dh := resp.Header.Get("Docker-Content-Digest")
if dh == "" {
return nil, fmt.Errorf("HEAD %s: response did not include Docker-Content-Digest header", u.String())
}
digest, err := v1.NewHash(dh)
if err != nil {
return nil, err
}
// Validate the digest matches what we asked for, if pulling by digest.
if dgst, ok := ref.(name.Digest); ok {
if digest.String() != dgst.DigestStr() {
return nil, fmt.Errorf("manifest digest: %q does not match requested digest: %q for %q", digest, dgst.DigestStr(), ref)
}
}
// Return all this info since we have to calculate it anyway.
return &v1.Descriptor{
Digest: digest,
Size: size,
MediaType: mediaType,
}, nil
}
func (f *fetcher) fetchBlob(ctx context.Context, size int64, h v1.Hash) (io.ReadCloser, error) {
u := f.url("blobs", h.String())
req, err := http.NewRequest(http.MethodGet, u.String(), nil)
if err != nil {
return nil, err
}
resp, err := f.client.Do(req.WithContext(ctx))
if err != nil {
return nil, redact.Error(err)
}
if err := transport.CheckError(resp, http.StatusOK); err != nil {
resp.Body.Close()
return nil, err
}
// Do whatever we can.
// If we have an expected size and Content-Length doesn't match, return an error.
// If we don't have an expected size and we do have a Content-Length, use Content-Length.
if hsize := resp.ContentLength; hsize != -1 {
if size == verify.SizeUnknown {
size = hsize
} else if hsize != size {
return nil, fmt.Errorf("GET %s: Content-Length header %d does not match expected size %d", u.String(), hsize, size)
}
}
return verify.ReadCloser(resp.Body, size, h)
}
func (f *fetcher) headBlob(ctx context.Context, h v1.Hash) (*http.Response, error) {
u := f.url("blobs", h.String())
req, err := http.NewRequest(http.MethodHead, u.String(), nil)
if err != nil {
return nil, err
}
resp, err := f.client.Do(req.WithContext(ctx))
if err != nil {
return nil, redact.Error(err)
}
if err := transport.CheckError(resp, http.StatusOK); err != nil {
resp.Body.Close()
return nil, err
}
return resp, nil
}
func (f *fetcher) blobExists(ctx context.Context, h v1.Hash) (bool, error) {
u := f.url("blobs", h.String())
req, err := http.NewRequest(http.MethodHead, u.String(), nil)
if err != nil {
return false, err
}
resp, err := f.client.Do(req.WithContext(ctx))
if err != nil {
return false, redact.Error(err)
}
defer resp.Body.Close()
if err := transport.CheckError(resp, http.StatusOK, http.StatusNotFound); err != nil {
return false, err
}
return resp.StatusCode == http.StatusOK, nil
}
+277
View File
@@ -0,0 +1,277 @@
// Copyright 2018 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package remote
import (
"bytes"
"context"
"io"
"net/http"
"net/url"
"sync"
"github.com/google/go-containerregistry/internal/redact"
"github.com/google/go-containerregistry/internal/verify"
"github.com/google/go-containerregistry/pkg/name"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/partial"
"github.com/google/go-containerregistry/pkg/v1/remote/transport"
"github.com/google/go-containerregistry/pkg/v1/types"
)
var acceptableImageMediaTypes = []types.MediaType{
types.DockerManifestSchema2,
types.OCIManifestSchema1,
}
// remoteImage accesses an image from a remote registry
type remoteImage struct {
fetcher fetcher
ref name.Reference
ctx context.Context
manifestLock sync.Mutex // Protects manifest
manifest []byte
configLock sync.Mutex // Protects config
config []byte
mediaType types.MediaType
descriptor *v1.Descriptor
}
func (r *remoteImage) ArtifactType() (string, error) {
// kind of a hack, but RawManifest does appropriate locking/memoization
// and makes sure r.descriptor is populated.
if _, err := r.RawManifest(); err != nil {
return "", err
}
return r.descriptor.ArtifactType, nil
}
var _ partial.CompressedImageCore = (*remoteImage)(nil)
// Image provides access to a remote image reference.
func Image(ref name.Reference, options ...Option) (v1.Image, error) {
desc, err := Get(ref, options...)
if err != nil {
return nil, err
}
return desc.Image()
}
func (r *remoteImage) MediaType() (types.MediaType, error) {
if string(r.mediaType) != "" {
return r.mediaType, nil
}
return types.DockerManifestSchema2, nil
}
func (r *remoteImage) RawManifest() ([]byte, error) {
r.manifestLock.Lock()
defer r.manifestLock.Unlock()
if r.manifest != nil {
return r.manifest, nil
}
// NOTE(jonjohnsonjr): We should never get here because the public entrypoints
// do type-checking via remote.Descriptor. I've left this here for tests that
// directly instantiate a remoteImage.
manifest, desc, err := r.fetcher.fetchManifest(r.ctx, r.ref, acceptableImageMediaTypes)
if err != nil {
return nil, err
}
if r.descriptor == nil {
r.descriptor = desc
}
r.mediaType = desc.MediaType
r.manifest = manifest
return r.manifest, nil
}
func (r *remoteImage) RawConfigFile() ([]byte, error) {
r.configLock.Lock()
defer r.configLock.Unlock()
if r.config != nil {
return r.config, nil
}
m, err := partial.Manifest(r)
if err != nil {
return nil, err
}
if m.Config.Data != nil {
if err := verify.Descriptor(m.Config); err != nil {
return nil, err
}
r.config = m.Config.Data
return r.config, nil
}
body, err := r.fetcher.fetchBlob(r.ctx, m.Config.Size, m.Config.Digest)
if err != nil {
return nil, err
}
defer body.Close()
r.config, err = io.ReadAll(body)
if err != nil {
return nil, err
}
return r.config, nil
}
// Descriptor retains the original descriptor from an index manifest.
// See partial.Descriptor.
func (r *remoteImage) Descriptor() (*v1.Descriptor, error) {
// kind of a hack, but RawManifest does appropriate locking/memoization
// and makes sure r.descriptor is populated.
_, err := r.RawManifest()
return r.descriptor, err
}
func (r *remoteImage) ConfigLayer() (v1.Layer, error) {
if _, err := r.RawManifest(); err != nil {
return nil, err
}
m, err := partial.Manifest(r)
if err != nil {
return nil, err
}
return partial.CompressedToLayer(&remoteImageLayer{
ri: r,
ctx: r.ctx,
digest: m.Config.Digest,
})
}
// remoteImageLayer implements partial.CompressedLayer
type remoteImageLayer struct {
ri *remoteImage
ctx context.Context
digest v1.Hash
}
// Digest implements partial.CompressedLayer
func (rl *remoteImageLayer) Digest() (v1.Hash, error) {
return rl.digest, nil
}
// Compressed implements partial.CompressedLayer
func (rl *remoteImageLayer) Compressed() (io.ReadCloser, error) {
urls := []url.URL{rl.ri.fetcher.url("blobs", rl.digest.String())}
// Add alternative layer sources from URLs (usually none).
d, err := partial.BlobDescriptor(rl, rl.digest)
if err != nil {
return nil, err
}
if d.Data != nil {
return verify.ReadCloser(io.NopCloser(bytes.NewReader(d.Data)), d.Size, d.Digest)
}
// We don't want to log binary layers -- this can break terminals.
ctx := redact.NewContext(rl.ctx, "omitting binary blobs from logs")
for _, s := range d.URLs {
u, err := url.Parse(s)
if err != nil {
return nil, err
}
urls = append(urls, *u)
}
// The lastErr for most pulls will be the same (the first error), but for
// foreign layers we'll want to surface the last one, since we try to pull
// from the registry first, which would often fail.
// TODO: Maybe we don't want to try pulling from the registry first?
var lastErr error
for _, u := range urls {
req, err := http.NewRequest(http.MethodGet, u.String(), nil)
if err != nil {
return nil, err
}
resp, err := rl.ri.fetcher.Do(req.WithContext(ctx))
if err != nil {
lastErr = err
continue
}
if err := transport.CheckError(resp, http.StatusOK); err != nil {
resp.Body.Close()
lastErr = err
continue
}
return verify.ReadCloser(resp.Body, d.Size, rl.digest)
}
return nil, lastErr
}
// Manifest implements partial.WithManifest so that we can use partial.BlobSize below.
func (rl *remoteImageLayer) Manifest() (*v1.Manifest, error) {
return partial.Manifest(rl.ri)
}
// MediaType implements v1.Layer
func (rl *remoteImageLayer) MediaType() (types.MediaType, error) {
bd, err := partial.BlobDescriptor(rl, rl.digest)
if err != nil {
return "", err
}
return bd.MediaType, nil
}
// Size implements partial.CompressedLayer
func (rl *remoteImageLayer) Size() (int64, error) {
// Look up the size of this digest in the manifest to avoid a request.
return partial.BlobSize(rl, rl.digest)
}
// ConfigFile implements partial.WithManifestAndConfigFile so that we can use partial.BlobToDiffID below.
func (rl *remoteImageLayer) ConfigFile() (*v1.ConfigFile, error) {
return partial.ConfigFile(rl.ri)
}
// DiffID implements partial.WithDiffID so that we don't recompute a DiffID that we already have
// available in our ConfigFile.
func (rl *remoteImageLayer) DiffID() (v1.Hash, error) {
return partial.BlobToDiffID(rl, rl.digest)
}
// Descriptor retains the original descriptor from an image manifest.
// See partial.Descriptor.
func (rl *remoteImageLayer) Descriptor() (*v1.Descriptor, error) {
return partial.BlobDescriptor(rl, rl.digest)
}
// See partial.Exists.
func (rl *remoteImageLayer) Exists() (bool, error) {
return rl.ri.fetcher.blobExists(rl.ri.ctx, rl.digest)
}
// LayerByDigest implements partial.CompressedLayer
func (r *remoteImage) LayerByDigest(h v1.Hash) (partial.CompressedLayer, error) {
return &remoteImageLayer{
ri: r,
ctx: r.ctx,
digest: h,
}, nil
}
+287
View File
@@ -0,0 +1,287 @@
// Copyright 2018 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package remote
import (
"bytes"
"context"
"fmt"
"sync"
"github.com/google/go-containerregistry/internal/verify"
"github.com/google/go-containerregistry/pkg/name"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/partial"
"github.com/google/go-containerregistry/pkg/v1/types"
)
var acceptableIndexMediaTypes = []types.MediaType{
types.DockerManifestList,
types.OCIImageIndex,
}
// remoteIndex accesses an index from a remote registry
type remoteIndex struct {
fetcher fetcher
ref name.Reference
ctx context.Context
manifestLock sync.Mutex // Protects manifest
manifest []byte
mediaType types.MediaType
descriptor *v1.Descriptor
}
// Index provides access to a remote index reference.
func Index(ref name.Reference, options ...Option) (v1.ImageIndex, error) {
desc, err := get(ref, acceptableIndexMediaTypes, options...)
if err != nil {
return nil, err
}
return desc.ImageIndex()
}
func (r *remoteIndex) MediaType() (types.MediaType, error) {
if string(r.mediaType) != "" {
return r.mediaType, nil
}
return types.DockerManifestList, nil
}
func (r *remoteIndex) Digest() (v1.Hash, error) {
return partial.Digest(r)
}
func (r *remoteIndex) Size() (int64, error) {
return partial.Size(r)
}
func (r *remoteIndex) RawManifest() ([]byte, error) {
r.manifestLock.Lock()
defer r.manifestLock.Unlock()
if r.manifest != nil {
return r.manifest, nil
}
// NOTE(jonjohnsonjr): We should never get here because the public entrypoints
// do type-checking via remote.Descriptor. I've left this here for tests that
// directly instantiate a remoteIndex.
manifest, desc, err := r.fetcher.fetchManifest(r.ctx, r.ref, acceptableIndexMediaTypes)
if err != nil {
return nil, err
}
if r.descriptor == nil {
r.descriptor = desc
}
r.mediaType = desc.MediaType
r.manifest = manifest
return r.manifest, nil
}
func (r *remoteIndex) IndexManifest() (*v1.IndexManifest, error) {
b, err := r.RawManifest()
if err != nil {
return nil, err
}
return v1.ParseIndexManifest(bytes.NewReader(b))
}
func (r *remoteIndex) Image(h v1.Hash) (v1.Image, error) {
desc, err := r.childByHash(h)
if err != nil {
return nil, err
}
// Descriptor.Image will handle coercing nested indexes into an Image.
return desc.Image()
}
// Descriptor retains the original descriptor from an index manifest.
// See partial.Descriptor.
func (r *remoteIndex) Descriptor() (*v1.Descriptor, error) {
// kind of a hack, but RawManifest does appropriate locking/memoization
// and makes sure r.descriptor is populated.
_, err := r.RawManifest()
return r.descriptor, err
}
func (r *remoteIndex) ImageIndex(h v1.Hash) (v1.ImageIndex, error) {
desc, err := r.childByHash(h)
if err != nil {
return nil, err
}
return desc.ImageIndex()
}
// Workaround for #819.
func (r *remoteIndex) Layer(h v1.Hash) (v1.Layer, error) {
index, err := r.IndexManifest()
if err != nil {
return nil, err
}
for _, childDesc := range index.Manifests {
if h == childDesc.Digest {
l, err := partial.CompressedToLayer(&remoteLayer{
fetcher: r.fetcher,
ctx: r.ctx,
digest: h,
})
if err != nil {
return nil, err
}
return &MountableLayer{
Layer: l,
Reference: r.ref.Context().Digest(h.String()),
}, nil
}
}
return nil, fmt.Errorf("layer not found: %s", h)
}
func (r *remoteIndex) imageByPlatform(platform v1.Platform) (v1.Image, error) {
desc, err := r.childByPlatform(platform)
if err != nil {
return nil, err
}
// Descriptor.Image will handle coercing nested indexes into an Image.
return desc.Image()
}
// This naively matches the first manifest with matching platform attributes.
//
// We should probably use this instead:
//
// github.com/containerd/containerd/platforms
//
// But first we'd need to migrate to:
//
// github.com/opencontainers/image-spec/specs-go/v1
func (r *remoteIndex) childByPlatform(platform v1.Platform) (*Descriptor, error) {
index, err := r.IndexManifest()
if err != nil {
return nil, err
}
for _, childDesc := range index.Manifests {
// If platform is missing from child descriptor, assume it's amd64/linux.
p := defaultPlatform
if childDesc.Platform != nil {
p = *childDesc.Platform
}
if matchesPlatform(p, platform) {
return r.childDescriptor(childDesc, platform)
}
}
return nil, fmt.Errorf("no child with platform %+v in index %s", platform, r.ref)
}
func (r *remoteIndex) childByHash(h v1.Hash) (*Descriptor, error) {
index, err := r.IndexManifest()
if err != nil {
return nil, err
}
for _, childDesc := range index.Manifests {
if h == childDesc.Digest {
return r.childDescriptor(childDesc, defaultPlatform)
}
}
return nil, fmt.Errorf("no child with digest %s in index %s", h, r.ref)
}
// Convert one of this index's child's v1.Descriptor into a remote.Descriptor, with the given platform option.
func (r *remoteIndex) childDescriptor(child v1.Descriptor, platform v1.Platform) (*Descriptor, error) {
ref := r.ref.Context().Digest(child.Digest.String())
var (
manifest []byte
err error
)
if child.Data != nil {
if err := verify.Descriptor(child); err != nil {
return nil, err
}
manifest = child.Data
} else {
manifest, _, err = r.fetcher.fetchManifest(r.ctx, ref, []types.MediaType{child.MediaType})
if err != nil {
return nil, err
}
}
if child.MediaType.IsImage() {
mf, _ := v1.ParseManifest(bytes.NewReader(manifest))
// Failing to parse as a manifest should just be ignored.
// The manifest might not be valid, and that's okay.
if mf != nil && !mf.Config.MediaType.IsConfig() {
child.ArtifactType = string(mf.Config.MediaType)
}
}
return &Descriptor{
ref: ref,
ctx: r.ctx,
fetcher: r.fetcher,
Manifest: manifest,
Descriptor: child,
platform: platform,
}, nil
}
// matchesPlatform checks if the given platform matches the required platforms.
// The given platform matches the required platform if
// - architecture and OS are identical.
// - OS version and variant are identical if provided.
// - features and OS features of the required platform are subsets of those of the given platform.
func matchesPlatform(given, required v1.Platform) bool {
// Required fields that must be identical.
if given.Architecture != required.Architecture || given.OS != required.OS {
return false
}
// Optional fields that may be empty, but must be identical if provided.
if required.OSVersion != "" && given.OSVersion != required.OSVersion {
return false
}
if required.Variant != "" && given.Variant != required.Variant {
return false
}
// Verify required platform's features are a subset of given platform's features.
if !isSubset(given.OSFeatures, required.OSFeatures) {
return false
}
if !isSubset(given.Features, required.Features) {
return false
}
return true
}
// isSubset checks if the required array of strings is a subset of the given lst.
func isSubset(lst, required []string) bool {
set := make(map[string]bool)
for _, value := range lst {
set[value] = true
}
for _, value := range required {
if _, ok := set[value]; !ok {
return false
}
}
return true
}
+77
View File
@@ -0,0 +1,77 @@
// Copyright 2019 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package remote
import (
"context"
"io"
"github.com/google/go-containerregistry/internal/redact"
"github.com/google/go-containerregistry/internal/verify"
"github.com/google/go-containerregistry/pkg/name"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/types"
)
// remoteImagelayer implements partial.CompressedLayer
type remoteLayer struct {
ctx context.Context
fetcher fetcher
digest v1.Hash
}
// Compressed implements partial.CompressedLayer
func (rl *remoteLayer) Compressed() (io.ReadCloser, error) {
// We don't want to log binary layers -- this can break terminals.
ctx := redact.NewContext(rl.ctx, "omitting binary blobs from logs")
return rl.fetcher.fetchBlob(ctx, verify.SizeUnknown, rl.digest)
}
// Compressed implements partial.CompressedLayer
func (rl *remoteLayer) Size() (int64, error) {
resp, err := rl.fetcher.headBlob(rl.ctx, rl.digest)
if err != nil {
return -1, err
}
defer resp.Body.Close()
return resp.ContentLength, nil
}
// Digest implements partial.CompressedLayer
func (rl *remoteLayer) Digest() (v1.Hash, error) {
return rl.digest, nil
}
// MediaType implements v1.Layer
func (rl *remoteLayer) MediaType() (types.MediaType, error) {
return types.DockerLayer, nil
}
// See partial.Exists.
func (rl *remoteLayer) Exists() (bool, error) {
return rl.fetcher.blobExists(rl.ctx, rl.digest)
}
// Layer reads the given blob reference from a registry as a Layer. A blob
// reference here is just a punned name.Digest where the digest portion is the
// digest of the blob to be read and the repository portion is the repo where
// that blob lives.
func Layer(ref name.Digest, options ...Option) (v1.Layer, error) {
o, err := makeOptions(options...)
if err != nil {
return nil, err
}
return newPuller(o).Layer(o.context, ref)
}
+152
View File
@@ -0,0 +1,152 @@
// Copyright 2018 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package remote
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/url"
"strings"
"github.com/google/go-containerregistry/pkg/name"
"github.com/google/go-containerregistry/pkg/v1/remote/transport"
)
// ListWithContext calls List with the given context.
//
// Deprecated: Use List and WithContext. This will be removed in a future release.
func ListWithContext(ctx context.Context, repo name.Repository, options ...Option) ([]string, error) {
return List(repo, append(options, WithContext(ctx))...)
}
// List calls /tags/list for the given repository, returning the list of tags
// in the "tags" property.
func List(repo name.Repository, options ...Option) ([]string, error) {
o, err := makeOptions(options...)
if err != nil {
return nil, err
}
return newPuller(o).List(o.context, repo)
}
type Tags struct {
Name string `json:"name"`
Tags []string `json:"tags"`
Next string `json:"next,omitempty"`
}
func (f *fetcher) listPage(ctx context.Context, repo name.Repository, next string, pageSize int) (*Tags, error) {
if next == "" {
uri := &url.URL{
Scheme: repo.Scheme(),
Host: repo.RegistryStr(),
Path: fmt.Sprintf("/v2/%s/tags/list", repo.RepositoryStr()),
}
if pageSize > 0 {
uri.RawQuery = fmt.Sprintf("n=%d", pageSize)
}
next = uri.String()
}
req, err := http.NewRequestWithContext(ctx, "GET", next, nil)
if err != nil {
return nil, err
}
resp, err := f.client.Do(req)
if err != nil {
return nil, err
}
if err := transport.CheckError(resp, http.StatusOK); err != nil {
return nil, err
}
parsed := Tags{}
if err := json.NewDecoder(resp.Body).Decode(&parsed); err != nil {
return nil, err
}
if err := resp.Body.Close(); err != nil {
return nil, err
}
uri, err := getNextPageURL(resp)
if err != nil {
return nil, err
}
if uri != nil {
parsed.Next = uri.String()
}
return &parsed, nil
}
// getNextPageURL checks if there is a Link header in a http.Response which
// contains a link to the next page. If yes it returns the url.URL of the next
// page otherwise it returns nil.
func getNextPageURL(resp *http.Response) (*url.URL, error) {
link := resp.Header.Get("Link")
if link == "" {
return nil, nil
}
if link[0] != '<' {
return nil, fmt.Errorf("failed to parse link header: missing '<' in: %s", link)
}
end := strings.Index(link, ">")
if end == -1 {
return nil, fmt.Errorf("failed to parse link header: missing '>' in: %s", link)
}
link = link[1:end]
linkURL, err := url.Parse(link)
if err != nil {
return nil, err
}
if resp.Request == nil || resp.Request.URL == nil {
return nil, nil
}
linkURL = resp.Request.URL.ResolveReference(linkURL)
return linkURL, nil
}
type Lister struct {
f *fetcher
repo name.Repository
pageSize int
page *Tags
err error
needMore bool
}
func (l *Lister) Next(ctx context.Context) (*Tags, error) {
if l.needMore {
l.page, l.err = l.f.listPage(ctx, l.repo, l.page.Next, l.pageSize)
} else {
l.needMore = true
}
return l.page, l.err
}
func (l *Lister) HasNext() bool {
return l.page != nil && (!l.needMore || l.page.Next != "")
}
+108
View File
@@ -0,0 +1,108 @@
// Copyright 2018 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package remote
import (
"github.com/google/go-containerregistry/pkg/name"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/partial"
)
// MountableLayer wraps a v1.Layer in a shim that enables the layer to be
// "mounted" when published to another registry.
type MountableLayer struct {
v1.Layer
Reference name.Reference
}
// Descriptor retains the original descriptor from an image manifest.
// See partial.Descriptor.
func (ml *MountableLayer) Descriptor() (*v1.Descriptor, error) {
return partial.Descriptor(ml.Layer)
}
// Exists is a hack. See partial.Exists.
func (ml *MountableLayer) Exists() (bool, error) {
return partial.Exists(ml.Layer)
}
// mountableImage wraps the v1.Layer references returned by the embedded v1.Image
// in MountableLayer's so that remote.Write might attempt to mount them from their
// source repository.
type mountableImage struct {
v1.Image
Reference name.Reference
}
// Layers implements v1.Image
func (mi *mountableImage) Layers() ([]v1.Layer, error) {
ls, err := mi.Image.Layers()
if err != nil {
return nil, err
}
mls := make([]v1.Layer, 0, len(ls))
for _, l := range ls {
mls = append(mls, &MountableLayer{
Layer: l,
Reference: mi.Reference,
})
}
return mls, nil
}
// LayerByDigest implements v1.Image
func (mi *mountableImage) LayerByDigest(d v1.Hash) (v1.Layer, error) {
l, err := mi.Image.LayerByDigest(d)
if err != nil {
return nil, err
}
return &MountableLayer{
Layer: l,
Reference: mi.Reference,
}, nil
}
// LayerByDiffID implements v1.Image
func (mi *mountableImage) LayerByDiffID(d v1.Hash) (v1.Layer, error) {
l, err := mi.Image.LayerByDiffID(d)
if err != nil {
return nil, err
}
return &MountableLayer{
Layer: l,
Reference: mi.Reference,
}, nil
}
// Descriptor retains the original descriptor from an index manifest.
// See partial.Descriptor.
func (mi *mountableImage) Descriptor() (*v1.Descriptor, error) {
return partial.Descriptor(mi.Image)
}
// ConfigLayer retains the original reference so that it can be mounted.
// See partial.ConfigLayer.
func (mi *mountableImage) ConfigLayer() (v1.Layer, error) {
l, err := partial.ConfigLayer(mi.Image)
if err != nil {
return nil, err
}
return &MountableLayer{
Layer: l,
Reference: mi.Reference,
}, nil
}
@@ -0,0 +1,46 @@
// Copyright 2020 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package remote
import (
"github.com/google/go-containerregistry/pkg/name"
"golang.org/x/sync/errgroup"
)
// MultiWrite writes the given Images or ImageIndexes to the given refs, as
// efficiently as possible, by deduping shared layer blobs while uploading them
// in parallel.
func MultiWrite(todo map[name.Reference]Taggable, options ...Option) (rerr error) {
o, err := makeOptions(options...)
if err != nil {
return err
}
if o.progress != nil {
defer func() { o.progress.Close(rerr) }()
}
p := newPusher(o)
g, ctx := errgroup.WithContext(o.context)
g.SetLimit(o.jobs)
for ref, t := range todo {
ref, t := ref, t
g.Go(func() error {
return p.Push(ctx, ref, t)
})
}
return g.Wait()
}
+354
View File
@@ -0,0 +1,354 @@
// Copyright 2018 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package remote
import (
"context"
"errors"
"io"
"net"
"net/http"
"syscall"
"time"
"github.com/google/go-containerregistry/internal/retry"
"github.com/google/go-containerregistry/pkg/authn"
"github.com/google/go-containerregistry/pkg/logs"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/remote/transport"
)
// Option is a functional option for remote operations.
type Option func(*options) error
type options struct {
auth authn.Authenticator
keychain authn.Keychain
transport http.RoundTripper
context context.Context
jobs int
userAgent string
allowNondistributableArtifacts bool
progress *progress
retryBackoff Backoff
retryPredicate retry.Predicate
retryStatusCodes []int
// Only these options can overwrite Reuse()d options.
platform v1.Platform
pageSize int
filter map[string]string
// Set by Reuse, we currently store one or the other.
puller *Puller
pusher *Pusher
}
var defaultPlatform = v1.Platform{
Architecture: "amd64",
OS: "linux",
}
// Backoff is an alias of retry.Backoff to expose this configuration option to consumers of this lib
type Backoff = retry.Backoff
var defaultRetryPredicate retry.Predicate = func(err error) bool {
// Various failure modes here, as we're often reading from and writing to
// the network.
if retry.IsTemporary(err) || errors.Is(err, io.ErrUnexpectedEOF) || errors.Is(err, io.EOF) || errors.Is(err, syscall.EPIPE) || errors.Is(err, syscall.ECONNRESET) || errors.Is(err, net.ErrClosed) {
logs.Warn.Printf("retrying %v", err)
return true
}
return false
}
// Try this three times, waiting 1s after first failure, 3s after second.
var defaultRetryBackoff = Backoff{
Duration: 1.0 * time.Second,
Factor: 3.0,
Jitter: 0.1,
Steps: 3,
}
// Useful for tests
var fastBackoff = Backoff{
Duration: 1.0 * time.Millisecond,
Factor: 3.0,
Jitter: 0.1,
Steps: 3,
}
var defaultRetryStatusCodes = []int{
http.StatusRequestTimeout,
http.StatusInternalServerError,
http.StatusBadGateway,
http.StatusServiceUnavailable,
http.StatusGatewayTimeout,
499, // nginx-specific, client closed request
522, // Cloudflare-specific, connection timeout
}
const (
defaultJobs = 4
// ECR returns an error if n > 1000:
// https://github.com/google/go-containerregistry/issues/1091
defaultPageSize = 1000
)
// DefaultTransport is based on http.DefaultTransport with modifications
// documented inline below.
var DefaultTransport http.RoundTripper = &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: (&net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
}).DialContext,
ForceAttemptHTTP2: true,
MaxIdleConns: 100,
IdleConnTimeout: 90 * time.Second,
TLSHandshakeTimeout: 10 * time.Second,
ExpectContinueTimeout: 1 * time.Second,
// We usually are dealing with 2 hosts (at most), split MaxIdleConns between them.
MaxIdleConnsPerHost: 50,
}
func makeOptions(opts ...Option) (*options, error) {
o := &options{
transport: DefaultTransport,
platform: defaultPlatform,
context: context.Background(),
jobs: defaultJobs,
pageSize: defaultPageSize,
retryPredicate: defaultRetryPredicate,
retryBackoff: defaultRetryBackoff,
retryStatusCodes: defaultRetryStatusCodes,
}
for _, option := range opts {
if err := option(o); err != nil {
return nil, err
}
}
switch {
case o.auth != nil && o.keychain != nil:
// It is a better experience to explicitly tell a caller their auth is misconfigured
// than potentially fail silently when the correct auth is overridden by option misuse.
return nil, errors.New("provide an option for either authn.Authenticator or authn.Keychain, not both")
case o.auth == nil:
o.auth = authn.Anonymous
}
// transport.Wrapper is a signal that consumers are opt-ing into providing their own transport without any additional wrapping.
// This is to allow consumers full control over the transports logic, such as providing retry logic.
if _, ok := o.transport.(*transport.Wrapper); !ok {
// Wrap the transport in something that logs requests and responses.
// It's expensive to generate the dumps, so skip it if we're writing
// to nothing.
if logs.Enabled(logs.Debug) {
o.transport = transport.NewLogger(o.transport)
}
// Using customized retry predicate if provided, and fallback to default if not.
predicate := o.retryPredicate
if predicate == nil {
predicate = defaultRetryPredicate
}
// Wrap the transport in something that can retry network flakes.
o.transport = transport.NewRetry(o.transport, transport.WithRetryBackoff(o.retryBackoff), transport.WithRetryPredicate(predicate), transport.WithRetryStatusCodes(o.retryStatusCodes...))
// Wrap this last to prevent transport.New from double-wrapping.
if o.userAgent != "" {
o.transport = transport.NewUserAgent(o.transport, o.userAgent)
}
}
return o, nil
}
// WithTransport is a functional option for overriding the default transport
// for remote operations.
// If transport.Wrapper is provided, this signals that the consumer does *not* want any further wrapping to occur.
// i.e. logging, retry and useragent
//
// The default transport is DefaultTransport.
func WithTransport(t http.RoundTripper) Option {
return func(o *options) error {
o.transport = t
return nil
}
}
// WithAuth is a functional option for overriding the default authenticator
// for remote operations.
// It is an error to use both WithAuth and WithAuthFromKeychain in the same Option set.
//
// The default authenticator is authn.Anonymous.
func WithAuth(auth authn.Authenticator) Option {
return func(o *options) error {
o.auth = auth
return nil
}
}
// WithAuthFromKeychain is a functional option for overriding the default
// authenticator for remote operations, using an authn.Keychain to find
// credentials.
// It is an error to use both WithAuth and WithAuthFromKeychain in the same Option set.
//
// The default authenticator is authn.Anonymous.
func WithAuthFromKeychain(keys authn.Keychain) Option {
return func(o *options) error {
o.keychain = keys
return nil
}
}
// WithPlatform is a functional option for overriding the default platform
// that Image and Descriptor.Image use for resolving an index to an image.
//
// The default platform is amd64/linux.
func WithPlatform(p v1.Platform) Option {
return func(o *options) error {
o.platform = p
return nil
}
}
// WithContext is a functional option for setting the context in http requests
// performed by a given function. Note that this context is used for _all_
// http requests, not just the initial volley. E.g., for remote.Image, the
// context will be set on http requests generated by subsequent calls to
// RawConfigFile() and even methods on layers returned by Layers().
//
// The default context is context.Background().
func WithContext(ctx context.Context) Option {
return func(o *options) error {
o.context = ctx
return nil
}
}
// WithJobs is a functional option for setting the parallelism of remote
// operations performed by a given function. Note that not all remote
// operations support parallelism.
//
// The default value is 4.
func WithJobs(jobs int) Option {
return func(o *options) error {
if jobs <= 0 {
return errors.New("jobs must be greater than zero")
}
o.jobs = jobs
return nil
}
}
// WithUserAgent adds the given string to the User-Agent header for any HTTP
// requests. This header will also include "go-containerregistry/${version}".
//
// If you want to completely overwrite the User-Agent header, use WithTransport.
func WithUserAgent(ua string) Option {
return func(o *options) error {
o.userAgent = ua
return nil
}
}
// WithNondistributable includes non-distributable (foreign) layers
// when writing images, see:
// https://github.com/opencontainers/image-spec/blob/master/layer.md#non-distributable-layers
//
// The default behaviour is to skip these layers
func WithNondistributable(o *options) error {
o.allowNondistributableArtifacts = true
return nil
}
// WithProgress takes a channel that will receive progress updates as bytes are written.
//
// Sending updates to an unbuffered channel will block writes, so callers
// should provide a buffered channel to avoid potential deadlocks.
func WithProgress(updates chan<- v1.Update) Option {
return func(o *options) error {
o.progress = &progress{updates: updates}
o.progress.lastUpdate = &v1.Update{}
return nil
}
}
// WithPageSize sets the given size as the value of parameter 'n' in the request.
//
// To omit the `n` parameter entirely, use WithPageSize(0).
// The default value is 1000.
func WithPageSize(size int) Option {
return func(o *options) error {
o.pageSize = size
return nil
}
}
// WithRetryBackoff sets the httpBackoff for retry HTTP operations.
func WithRetryBackoff(backoff Backoff) Option {
return func(o *options) error {
o.retryBackoff = backoff
return nil
}
}
// WithRetryPredicate sets the predicate for retry HTTP operations.
func WithRetryPredicate(predicate retry.Predicate) Option {
return func(o *options) error {
o.retryPredicate = predicate
return nil
}
}
// WithRetryStatusCodes sets which http response codes will be retried.
func WithRetryStatusCodes(codes ...int) Option {
return func(o *options) error {
o.retryStatusCodes = codes
return nil
}
}
// WithFilter sets the filter querystring for HTTP operations.
func WithFilter(key string, value string) Option {
return func(o *options) error {
if o.filter == nil {
o.filter = map[string]string{}
}
o.filter[key] = value
return nil
}
}
// Reuse takes a Puller or Pusher and reuses it for remote interactions
// rather than starting from a clean slate. For example, it will reuse token exchanges
// when possible and avoid sending redundant HEAD requests.
//
// Reuse will take precedence over other options passed to most remote functions because
// most options deal with setting up auth and transports, which Reuse intetionally skips.
func Reuse[I *Puller | *Pusher](i I) Option {
return func(o *options) error {
if puller, ok := any(i).(*Puller); ok {
o.puller = puller
} else if pusher, ok := any(i).(*Pusher); ok {
o.pusher = pusher
}
return nil
}
}
@@ -0,0 +1,76 @@
// Copyright 2022 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package remote
import (
"io"
"sync"
"sync/atomic"
v1 "github.com/google/go-containerregistry/pkg/v1"
)
type progress struct {
sync.Mutex
updates chan<- v1.Update
lastUpdate *v1.Update
}
func (p *progress) total(delta int64) {
p.Lock()
defer p.Unlock()
atomic.AddInt64(&p.lastUpdate.Total, delta)
}
func (p *progress) complete(delta int64) {
p.Lock()
defer p.Unlock()
p.updates <- v1.Update{
Total: p.lastUpdate.Total,
Complete: atomic.AddInt64(&p.lastUpdate.Complete, delta),
}
}
func (p *progress) err(err error) error {
if err != nil && p.updates != nil {
p.updates <- v1.Update{Error: err}
}
return err
}
func (p *progress) Close(err error) {
_ = p.err(err)
close(p.updates)
}
type progressReader struct {
rc io.ReadCloser
count *int64 // number of bytes this reader has read, to support resetting on retry.
progress *progress
}
func (r *progressReader) Read(b []byte) (int, error) {
n, err := r.rc.Read(b)
if err != nil {
return n, err
}
atomic.AddInt64(r.count, int64(n))
// TODO: warn/debug log if sending takes too long, or if sending is blocked while context is canceled.
r.progress.complete(int64(n))
return n, nil
}
func (r *progressReader) Close() error { return r.rc.Close() }
+222
View File
@@ -0,0 +1,222 @@
// Copyright 2023 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package remote
import (
"context"
"sync"
"github.com/google/go-containerregistry/pkg/name"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/partial"
"github.com/google/go-containerregistry/pkg/v1/types"
)
type Puller struct {
o *options
// map[resource]*reader
readers sync.Map
}
func NewPuller(options ...Option) (*Puller, error) {
o, err := makeOptions(options...)
if err != nil {
return nil, err
}
return newPuller(o), nil
}
func newPuller(o *options) *Puller {
if o.puller != nil {
return o.puller
}
return &Puller{
o: o,
}
}
type reader struct {
// in
target resource
o *options
// f()
once sync.Once
// out
f *fetcher
err error
}
// this will run once per reader instance
func (r *reader) init(ctx context.Context) error {
r.once.Do(func() {
r.f, r.err = makeFetcher(ctx, r.target, r.o)
})
return r.err
}
func (p *Puller) fetcher(ctx context.Context, target resource) (*fetcher, error) {
v, _ := p.readers.LoadOrStore(target, &reader{
target: target,
o: p.o,
})
rr := v.(*reader)
return rr.f, rr.init(ctx)
}
// Head is like remote.Head, but avoids re-authenticating when possible.
func (p *Puller) Head(ctx context.Context, ref name.Reference) (*v1.Descriptor, error) {
f, err := p.fetcher(ctx, ref.Context())
if err != nil {
return nil, err
}
return f.headManifest(ctx, ref, allManifestMediaTypes)
}
// Get is like remote.Get, but avoids re-authenticating when possible.
func (p *Puller) Get(ctx context.Context, ref name.Reference) (*Descriptor, error) {
return p.get(ctx, ref, allManifestMediaTypes, p.o.platform)
}
func (p *Puller) get(ctx context.Context, ref name.Reference, acceptable []types.MediaType, platform v1.Platform) (*Descriptor, error) {
f, err := p.fetcher(ctx, ref.Context())
if err != nil {
return nil, err
}
return f.get(ctx, ref, acceptable, platform)
}
// Layer is like remote.Layer, but avoids re-authenticating when possible.
func (p *Puller) Layer(ctx context.Context, ref name.Digest) (v1.Layer, error) {
f, err := p.fetcher(ctx, ref.Context())
if err != nil {
return nil, err
}
h, err := v1.NewHash(ref.Identifier())
if err != nil {
return nil, err
}
l, err := partial.CompressedToLayer(&remoteLayer{
fetcher: *f,
ctx: ctx,
digest: h,
})
if err != nil {
return nil, err
}
return &MountableLayer{
Layer: l,
Reference: ref,
}, nil
}
// List lists tags in a repo and handles pagination, returning the full list of tags.
func (p *Puller) List(ctx context.Context, repo name.Repository) ([]string, error) {
lister, err := p.Lister(ctx, repo)
if err != nil {
return nil, err
}
tagList := []string{}
for lister.HasNext() {
tags, err := lister.Next(ctx)
if err != nil {
return nil, err
}
tagList = append(tagList, tags.Tags...)
}
return tagList, nil
}
// Lister lists tags in a repo and returns a Lister for paginating through the results.
func (p *Puller) Lister(ctx context.Context, repo name.Repository) (*Lister, error) {
return p.lister(ctx, repo, p.o.pageSize)
}
func (p *Puller) lister(ctx context.Context, repo name.Repository, pageSize int) (*Lister, error) {
f, err := p.fetcher(ctx, repo)
if err != nil {
return nil, err
}
page, err := f.listPage(ctx, repo, "", pageSize)
if err != nil {
return nil, err
}
return &Lister{
f: f,
repo: repo,
pageSize: pageSize,
page: page,
err: err,
}, nil
}
// Catalog lists repos in a registry and handles pagination, returning the full list of repos.
func (p *Puller) Catalog(ctx context.Context, reg name.Registry) ([]string, error) {
return p.catalog(ctx, reg, p.o.pageSize)
}
func (p *Puller) catalog(ctx context.Context, reg name.Registry, pageSize int) ([]string, error) {
catalogger, err := p.catalogger(ctx, reg, pageSize)
if err != nil {
return nil, err
}
repoList := []string{}
for catalogger.HasNext() {
repos, err := catalogger.Next(ctx)
if err != nil {
return nil, err
}
repoList = append(repoList, repos.Repos...)
}
return repoList, nil
}
// Catalogger lists repos in a registry and returns a Catalogger for paginating through the results.
func (p *Puller) Catalogger(ctx context.Context, reg name.Registry) (*Catalogger, error) {
return p.catalogger(ctx, reg, p.o.pageSize)
}
func (p *Puller) catalogger(ctx context.Context, reg name.Registry, pageSize int) (*Catalogger, error) {
f, err := p.fetcher(ctx, reg)
if err != nil {
return nil, err
}
page, err := f.catalogPage(ctx, reg, "", pageSize)
if err != nil {
return nil, err
}
return &Catalogger{
f: f,
reg: reg,
pageSize: pageSize,
page: page,
err: err,
}, nil
}
func (p *Puller) referrers(ctx context.Context, d name.Digest, filter map[string]string) (v1.ImageIndex, error) {
f, err := p.fetcher(ctx, d.Context())
if err != nil {
return nil, err
}
return f.fetchReferrers(ctx, filter, d)
}
+573
View File
@@ -0,0 +1,573 @@
// Copyright 2023 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package remote
import (
"bytes"
"context"
"errors"
"fmt"
"net/http"
"net/url"
"sync"
"github.com/google/go-containerregistry/pkg/logs"
"github.com/google/go-containerregistry/pkg/name"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/partial"
"github.com/google/go-containerregistry/pkg/v1/remote/transport"
"github.com/google/go-containerregistry/pkg/v1/stream"
"github.com/google/go-containerregistry/pkg/v1/types"
"golang.org/x/sync/errgroup"
)
type manifest interface {
Taggable
partial.Describable
}
// key is either v1.Hash or v1.Layer (for stream.Layer)
type workers struct {
// map[v1.Hash|v1.Layer]*sync.Once
onces sync.Map
// map[v1.Hash|v1.Layer]error
errors sync.Map
}
func nop() error {
return nil
}
func (w *workers) err(digest v1.Hash) error {
v, ok := w.errors.Load(digest)
if !ok || v == nil {
return nil
}
return v.(error)
}
func (w *workers) Do(digest v1.Hash, f func() error) error {
// We don't care if it was loaded or not because the sync.Once will do it for us.
once, _ := w.onces.LoadOrStore(digest, &sync.Once{})
once.(*sync.Once).Do(func() {
w.errors.Store(digest, f())
})
err := w.err(digest)
if err != nil {
// Allow this to be retried by another caller.
w.onces.Delete(digest)
}
return err
}
func (w *workers) Stream(layer v1.Layer, f func() error) error {
// We don't care if it was loaded or not because the sync.Once will do it for us.
once, _ := w.onces.LoadOrStore(layer, &sync.Once{})
once.(*sync.Once).Do(func() {
w.errors.Store(layer, f())
})
v, ok := w.errors.Load(layer)
if !ok || v == nil {
return nil
}
return v.(error)
}
type Pusher struct {
o *options
// map[name.Repository]*repoWriter
writers sync.Map
}
func NewPusher(options ...Option) (*Pusher, error) {
o, err := makeOptions(options...)
if err != nil {
return nil, err
}
return newPusher(o), nil
}
func newPusher(o *options) *Pusher {
if o.pusher != nil {
return o.pusher
}
return &Pusher{
o: o,
}
}
func (p *Pusher) writer(ctx context.Context, repo name.Repository, o *options) (*repoWriter, error) {
v, _ := p.writers.LoadOrStore(repo, &repoWriter{
repo: repo,
o: o,
})
rw := v.(*repoWriter)
return rw, rw.init(ctx)
}
func (p *Pusher) Put(ctx context.Context, ref name.Reference, t Taggable) error {
w, err := p.writer(ctx, ref.Context(), p.o)
if err != nil {
return err
}
m, err := taggableToManifest(t)
if err != nil {
return err
}
return w.commitManifest(ctx, ref, m)
}
func (p *Pusher) Push(ctx context.Context, ref name.Reference, t Taggable) error {
w, err := p.writer(ctx, ref.Context(), p.o)
if err != nil {
return err
}
return w.writeManifest(ctx, ref, t)
}
func (p *Pusher) Upload(ctx context.Context, repo name.Repository, l v1.Layer) error {
w, err := p.writer(ctx, repo, p.o)
if err != nil {
return err
}
return w.writeLayer(ctx, l)
}
func (p *Pusher) Delete(ctx context.Context, ref name.Reference) error {
w, err := p.writer(ctx, ref.Context(), p.o)
if err != nil {
return err
}
u := url.URL{
Scheme: ref.Context().Scheme(),
Host: ref.Context().RegistryStr(),
Path: fmt.Sprintf("/v2/%s/manifests/%s", ref.Context().RepositoryStr(), ref.Identifier()),
}
req, err := http.NewRequest(http.MethodDelete, u.String(), nil)
if err != nil {
return err
}
resp, err := w.w.client.Do(req.WithContext(ctx))
if err != nil {
return err
}
defer resp.Body.Close()
return transport.CheckError(resp, http.StatusOK, http.StatusAccepted)
// TODO(jason): If the manifest had a `subject`, and if the registry
// doesn't support Referrers, update the index pointed to by the
// subject's fallback tag to remove the descriptor for this manifest.
}
type repoWriter struct {
repo name.Repository
o *options
once sync.Once
w *writer
err error
work *workers
}
// this will run once per repoWriter instance
func (rw *repoWriter) init(ctx context.Context) error {
rw.once.Do(func() {
rw.work = &workers{}
rw.w, rw.err = makeWriter(ctx, rw.repo, nil, rw.o)
})
return rw.err
}
func (rw *repoWriter) writeDeps(ctx context.Context, m manifest) error {
if img, ok := m.(v1.Image); ok {
return rw.writeLayers(ctx, img)
}
if idx, ok := m.(v1.ImageIndex); ok {
return rw.writeChildren(ctx, idx)
}
// This has no deps, not an error (e.g. something you want to just PUT).
return nil
}
type describable struct {
desc v1.Descriptor
}
func (d describable) Digest() (v1.Hash, error) {
return d.desc.Digest, nil
}
func (d describable) Size() (int64, error) {
return d.desc.Size, nil
}
func (d describable) MediaType() (types.MediaType, error) {
return d.desc.MediaType, nil
}
type tagManifest struct {
Taggable
partial.Describable
}
func taggableToManifest(t Taggable) (manifest, error) {
if m, ok := t.(manifest); ok {
return m, nil
}
if d, ok := t.(*Descriptor); ok {
if d.MediaType.IsIndex() {
return d.ImageIndex()
}
if d.MediaType.IsImage() {
return d.Image()
}
if d.MediaType.IsSchema1() {
return d.Schema1()
}
return tagManifest{t, describable{d.toDesc()}}, nil
}
desc := v1.Descriptor{
// A reasonable default if Taggable doesn't implement MediaType.
MediaType: types.DockerManifestSchema2,
}
b, err := t.RawManifest()
if err != nil {
return nil, err
}
if wmt, ok := t.(withMediaType); ok {
desc.MediaType, err = wmt.MediaType()
if err != nil {
return nil, err
}
}
desc.Digest, desc.Size, err = v1.SHA256(bytes.NewReader(b))
if err != nil {
return nil, err
}
return tagManifest{t, describable{desc}}, nil
}
func (rw *repoWriter) writeManifest(ctx context.Context, ref name.Reference, t Taggable) error {
m, err := taggableToManifest(t)
if err != nil {
return err
}
needDeps := true
digest, err := m.Digest()
if errors.Is(err, stream.ErrNotComputed) {
if err := rw.writeDeps(ctx, m); err != nil {
return err
}
needDeps = false
digest, err = m.Digest()
if err != nil {
return err
}
} else if err != nil {
return err
}
// This may be a lazy child where we have no ref until digest is computed.
if ref == nil {
ref = rw.repo.Digest(digest.String())
}
// For tags, we want to do this check outside of our Work.Do closure because
// we don't want to dedupe based on the manifest digest.
_, byTag := ref.(name.Tag)
if byTag {
if exists, err := rw.manifestExists(ctx, ref, t); err != nil {
return err
} else if exists {
return nil
}
}
// The following work.Do will get deduped by digest, so it won't happen unless
// this tag happens to be the first commitManifest to run for that digest.
needPut := byTag
if err := rw.work.Do(digest, func() error {
if !byTag {
if exists, err := rw.manifestExists(ctx, ref, t); err != nil {
return err
} else if exists {
return nil
}
}
if needDeps {
if err := rw.writeDeps(ctx, m); err != nil {
return err
}
}
needPut = false
return rw.commitManifest(ctx, ref, m)
}); err != nil {
return err
}
if !needPut {
return nil
}
// Only runs for tags that got deduped by digest.
return rw.commitManifest(ctx, ref, m)
}
func (rw *repoWriter) writeChildren(ctx context.Context, idx v1.ImageIndex) error {
children, err := partial.Manifests(idx)
if err != nil {
return err
}
g, ctx := errgroup.WithContext(ctx)
g.SetLimit(rw.o.jobs)
for _, child := range children {
child := child
if err := rw.writeChild(ctx, child, g); err != nil {
return err
}
}
return g.Wait()
}
func (rw *repoWriter) writeChild(ctx context.Context, child partial.Describable, g *errgroup.Group) error {
switch child := child.(type) {
case v1.ImageIndex:
// For recursive index, we want to do a depth-first launching of goroutines
// to avoid deadlocking.
//
// Note that this is rare, so the impact of this should be really small.
return rw.writeManifest(ctx, nil, child)
case v1.Image:
g.Go(func() error {
return rw.writeManifest(ctx, nil, child)
})
case v1.Layer:
g.Go(func() error {
return rw.writeLayer(ctx, child)
})
default:
// This can't happen.
return fmt.Errorf("encountered unknown child: %T", child)
}
return nil
}
// TODO: Consider caching some representation of the tags/digests in the destination
// repository as a hint to avoid this optimistic check in cases where we will most
// likely have to do a PUT anyway, e.g. if we are overwriting a tag we just wrote.
func (rw *repoWriter) manifestExists(ctx context.Context, ref name.Reference, t Taggable) (bool, error) {
f := &fetcher{
target: ref.Context(),
client: rw.w.client,
}
m, err := taggableToManifest(t)
if err != nil {
return false, err
}
digest, err := m.Digest()
if err != nil {
// Possibly due to streaming layers.
return false, nil
}
got, err := f.headManifest(ctx, ref, allManifestMediaTypes)
if err != nil {
var terr *transport.Error
if errors.As(err, &terr) {
if terr.StatusCode == http.StatusNotFound {
return false, nil
}
// We treat a 403 here as non-fatal because this existence check is an optimization and
// some registries will return a 403 instead of a 404 in certain situations.
// E.g. https://jfrog.atlassian.net/browse/RTFACT-13797
if terr.StatusCode == http.StatusForbidden {
logs.Debug.Printf("manifestExists unexpected 403: %v", err)
return false, nil
}
}
return false, err
}
if digest != got.Digest {
// Mark that we saw this digest in the registry so we don't have to check it again.
rw.work.Do(got.Digest, nop)
return false, nil
}
if tag, ok := ref.(name.Tag); ok {
logs.Progress.Printf("existing manifest: %s@%s", tag.Identifier(), got.Digest)
} else {
logs.Progress.Print("existing manifest: ", got.Digest)
}
return true, nil
}
func (rw *repoWriter) commitManifest(ctx context.Context, ref name.Reference, m manifest) error {
if rw.o.progress != nil {
size, err := m.Size()
if err != nil {
return err
}
rw.o.progress.total(size)
}
return rw.w.commitManifest(ctx, m, ref)
}
func (rw *repoWriter) writeLayers(pctx context.Context, img v1.Image) error {
ls, err := img.Layers()
if err != nil {
return err
}
g, ctx := errgroup.WithContext(pctx)
g.SetLimit(rw.o.jobs)
for _, l := range ls {
l := l
g.Go(func() error {
return rw.writeLayer(ctx, l)
})
}
mt, err := img.MediaType()
if err != nil {
return err
}
if mt.IsSchema1() {
return g.Wait()
}
cl, err := partial.ConfigLayer(img)
if errors.Is(err, stream.ErrNotComputed) {
if err := g.Wait(); err != nil {
return err
}
cl, err := partial.ConfigLayer(img)
if err != nil {
return err
}
return rw.writeLayer(pctx, cl)
} else if err != nil {
return err
}
g.Go(func() error {
return rw.writeLayer(ctx, cl)
})
return g.Wait()
}
func (rw *repoWriter) writeLayer(ctx context.Context, l v1.Layer) error {
// Skip any non-distributable things.
mt, err := l.MediaType()
if err != nil {
return err
}
if !mt.IsDistributable() && !rw.o.allowNondistributableArtifacts {
return nil
}
digest, err := l.Digest()
if err != nil {
if errors.Is(err, stream.ErrNotComputed) {
return rw.lazyWriteLayer(ctx, l)
}
return err
}
return rw.work.Do(digest, func() error {
if rw.o.progress != nil {
size, err := l.Size()
if err != nil {
return err
}
rw.o.progress.total(size)
}
return rw.w.uploadOne(ctx, l)
})
}
func (rw *repoWriter) lazyWriteLayer(ctx context.Context, l v1.Layer) error {
return rw.work.Stream(l, func() error {
if err := rw.w.uploadOne(ctx, l); err != nil {
return err
}
// Mark this upload completed.
digest, err := l.Digest()
if err != nil {
return err
}
rw.work.Do(digest, nop)
if rw.o.progress != nil {
size, err := l.Size()
if err != nil {
return err
}
rw.o.progress.total(size)
}
return nil
})
}
@@ -0,0 +1,117 @@
// Copyright 2023 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package remote
import (
"bytes"
"context"
"errors"
"io"
"net/http"
"strings"
"github.com/google/go-containerregistry/pkg/name"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/empty"
"github.com/google/go-containerregistry/pkg/v1/mutate"
"github.com/google/go-containerregistry/pkg/v1/remote/transport"
"github.com/google/go-containerregistry/pkg/v1/types"
)
// Referrers returns a list of descriptors that refer to the given manifest digest.
//
// The subject manifest doesn't have to exist in the registry for there to be descriptors that refer to it.
func Referrers(d name.Digest, options ...Option) (v1.ImageIndex, error) {
o, err := makeOptions(options...)
if err != nil {
return nil, err
}
return newPuller(o).referrers(o.context, d, o.filter)
}
// https://github.com/opencontainers/distribution-spec/blob/main/spec.md#referrers-tag-schema
func fallbackTag(d name.Digest) name.Tag {
return d.Context().Tag(strings.Replace(d.DigestStr(), ":", "-", 1))
}
func (f *fetcher) fetchReferrers(ctx context.Context, filter map[string]string, d name.Digest) (v1.ImageIndex, error) {
// Check the Referrers API endpoint first.
u := f.url("referrers", d.DigestStr())
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u.String(), nil)
if err != nil {
return nil, err
}
req.Header.Set("Accept", string(types.OCIImageIndex))
resp, err := f.client.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if err := transport.CheckError(resp, http.StatusOK, http.StatusNotFound, http.StatusBadRequest, http.StatusNotAcceptable); err != nil {
return nil, err
}
var b []byte
if resp.StatusCode == http.StatusOK && resp.Header.Get("Content-Type") == string(types.OCIImageIndex) {
b, err = io.ReadAll(resp.Body)
if err != nil {
return nil, err
}
} else {
// The registry doesn't support the Referrers API endpoint, so we'll use the fallback tag scheme.
b, _, err = f.fetchManifest(ctx, fallbackTag(d), []types.MediaType{types.OCIImageIndex})
var terr *transport.Error
if errors.As(err, &terr) && terr.StatusCode == http.StatusNotFound {
// Not found just means there are no attachments yet. Start with an empty manifest.
return empty.Index, nil
} else if err != nil {
return nil, err
}
}
h, sz, err := v1.SHA256(bytes.NewReader(b))
if err != nil {
return nil, err
}
idx := &remoteIndex{
fetcher: *f,
ctx: ctx,
manifest: b,
mediaType: types.OCIImageIndex,
descriptor: &v1.Descriptor{
Digest: h,
MediaType: types.OCIImageIndex,
Size: sz,
},
}
return filterReferrersResponse(filter, idx), nil
}
// If filter applied, filter out by artifactType.
// See https://github.com/opencontainers/distribution-spec/blob/main/spec.md#listing-referrers
func filterReferrersResponse(filter map[string]string, in v1.ImageIndex) v1.ImageIndex {
if filter == nil {
return in
}
v, ok := filter["artifactType"]
if !ok {
return in
}
return mutate.RemoveManifests(in, func(desc v1.Descriptor) bool {
return desc.ArtifactType != v
})
}
+118
View File
@@ -0,0 +1,118 @@
// Copyright 2023 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package remote
import (
"bytes"
"context"
"encoding/json"
"github.com/google/go-containerregistry/pkg/name"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/partial"
"github.com/google/go-containerregistry/pkg/v1/types"
)
type schema1 struct {
ref name.Reference
ctx context.Context
fetcher fetcher
manifest []byte
mediaType types.MediaType
descriptor *v1.Descriptor
}
func (s *schema1) Layers() ([]v1.Layer, error) {
m := schema1Manifest{}
if err := json.NewDecoder(bytes.NewReader(s.manifest)).Decode(&m); err != nil {
return nil, err
}
layers := []v1.Layer{}
for i := len(m.FSLayers) - 1; i >= 0; i-- {
fsl := m.FSLayers[i]
h, err := v1.NewHash(fsl.BlobSum)
if err != nil {
return nil, err
}
l, err := s.LayerByDigest(h)
if err != nil {
return nil, err
}
layers = append(layers, l)
}
return layers, nil
}
func (s *schema1) MediaType() (types.MediaType, error) {
return s.mediaType, nil
}
func (s *schema1) Size() (int64, error) {
return s.descriptor.Size, nil
}
func (s *schema1) ConfigName() (v1.Hash, error) {
return partial.ConfigName(s)
}
func (s *schema1) ConfigFile() (*v1.ConfigFile, error) {
return nil, newErrSchema1(s.mediaType)
}
func (s *schema1) RawConfigFile() ([]byte, error) {
return []byte("{}"), nil
}
func (s *schema1) Digest() (v1.Hash, error) {
return s.descriptor.Digest, nil
}
func (s *schema1) Manifest() (*v1.Manifest, error) {
return nil, newErrSchema1(s.mediaType)
}
func (s *schema1) RawManifest() ([]byte, error) {
return s.manifest, nil
}
func (s *schema1) LayerByDigest(h v1.Hash) (v1.Layer, error) {
l, err := partial.CompressedToLayer(&remoteLayer{
fetcher: s.fetcher,
ctx: s.ctx,
digest: h,
})
if err != nil {
return nil, err
}
return &MountableLayer{
Layer: l,
Reference: s.ref.Context().Digest(h.String()),
}, nil
}
func (s *schema1) LayerByDiffID(v1.Hash) (v1.Layer, error) {
return nil, newErrSchema1(s.mediaType)
}
type fslayer struct {
BlobSum string `json:"blobSum"`
}
type schema1Manifest struct {
FSLayers []fslayer `json:"fsLayers"`
}
@@ -0,0 +1,129 @@
# `transport`
[![GoDoc](https://godoc.org/github.com/google/go-containerregistry/pkg/v1/transport?status.svg)](https://godoc.org/github.com/google/go-containerregistry/pkg/v1/transport)
The [distribution protocol](https://github.com/opencontainers/distribution-spec) is fairly simple, but correctly [implementing authentication](../../../authn/README.md) is **hard**.
This package [implements](https://godoc.org/github.com/google/go-containerregistry/pkg/v1/remote/transport#New) an [`http.RoundTripper`](https://godoc.org/net/http#RoundTripper)
that transparently performs:
* [Token
Authentication](https://docs.docker.com/registry/spec/auth/token/) and
* [OAuth2
Authentication](https://docs.docker.com/registry/spec/auth/oauth/)
for registry clients.
## Raison d'être
> Why not just use the [`docker/distribution`](https://godoc.org/github.com/docker/distribution/registry/client/auth) client?
Great question! Mostly, because I don't want to depend on [`prometheus/client_golang`](https://github.com/prometheus/client_golang).
As a performance optimization, that client uses [a cache](https://github.com/docker/distribution/blob/a8371794149d1d95f1e846744b05c87f2f825e5a/registry/client/repository.go#L173) to keep track of a mapping between blob digests and their [descriptors](https://github.com/docker/distribution/blob/a8371794149d1d95f1e846744b05c87f2f825e5a/blobs.go#L57-L86). Unfortunately, the cache [uses prometheus](https://github.com/docker/distribution/blob/a8371794149d1d95f1e846744b05c87f2f825e5a/registry/storage/cache/cachedblobdescriptorstore.go#L44) to track hits and misses, so if you want to use that client you have to pull in all of prometheus, which is pretty large.
![docker/distribution](../../../../images/docker.dot.svg)
> Why does it matter if you depend on prometheus? Who cares?
It's generally polite to your downstream to reduce the number of dependencies your package requires:
* Downloading your package is faster, which helps our Australian friends and people on airplanes.
* There is less code to compile, which speeds up builds and saves the planet from global warming.
* You reduce the likelihood of inflicting dependency hell upon your consumers.
* [Tim Hockin](https://twitter.com/thockin/status/958606077456654336) prefers it based on his experience working on Kubernetes, and he's a pretty smart guy.
> Okay, what about [`containerd/containerd`](https://godoc.org/github.com/containerd/containerd/remotes/docker)?
Similar reasons! That ends up pulling in grpc, protobuf, and logrus.
![containerd/containerd](../../../../images/containerd.dot.svg)
> Well... what about [`containers/image`](https://godoc.org/github.com/containers/image/docker)?
That just uses the the `docker/distribution` client... and more!
![containers/image](../../../../images/containers.dot.svg)
> Wow, what about this package?
Of course, this package isn't perfect either. `transport` depends on `authn`,
which in turn depends on docker's config file parsing and handling package,
which you don't strictly need but almost certainly want if you're going to be
interacting with a registry.
![google/go-containerregistry](../../../../images/ggcr.dot.svg)
*These graphs were generated by
[`kisielk/godepgraph`](https://github.com/kisielk/godepgraph).*
## Usage
This is heavily used by the
[`remote`](https://godoc.org/github.com/google/go-containerregistry/pkg/v1/remote)
package, which implements higher level image-centric functionality, but this
package is useful if you want to interact directly with the registry to do
something that `remote` doesn't support, e.g. [to handle with schema 1
images](https://github.com/google/go-containerregistry/pull/509).
This package also includes some [error
handling](https://github.com/opencontainers/distribution-spec/blob/60be706c34ee7805bdd1d3d11affec53b0dfb8fb/spec.md#errors)
facilities in the form of
[`CheckError`](https://godoc.org/github.com/google/go-containerregistry/pkg/v1/remote/transport#CheckError),
which will parse the response body into a structured error for unexpected http
status codes.
Here's a "simple" program that writes the result of
[listing tags](https://github.com/opencontainers/distribution-spec/blob/60be706c34ee7805bdd1d3d11affec53b0dfb8fb/spec.md#tags)
for [`gcr.io/google-containers/pause`](https://gcr.io/google-containers/pause)
to stdout.
```go
package main
import (
"io"
"net/http"
"os"
"github.com/google/go-containerregistry/pkg/authn"
"github.com/google/go-containerregistry/pkg/name"
"github.com/google/go-containerregistry/pkg/v1/remote/transport"
)
func main() {
repo, err := name.NewRepository("gcr.io/google-containers/pause")
if err != nil {
panic(err)
}
// Fetch credentials based on your docker config file, which is $HOME/.docker/config.json or $DOCKER_CONFIG.
auth, err := authn.DefaultKeychain.Resolve(repo.Registry)
if err != nil {
panic(err)
}
// Construct an http.Client that is authorized to pull from gcr.io/google-containers/pause.
scopes := []string{repo.Scope(transport.PullScope)}
t, err := transport.New(repo.Registry, auth, http.DefaultTransport, scopes)
if err != nil {
panic(err)
}
client := &http.Client{Transport: t}
// Make the actual request.
resp, err := client.Get("https://gcr.io/v2/google-containers/pause/tags/list")
if err != nil {
panic(err)
}
// Assert that we get a 200, otherwise attempt to parse body as a structured error.
if err := transport.CheckError(resp, http.StatusOK); err != nil {
panic(err)
}
// Write the response to stdout.
if _, err := io.Copy(os.Stdout, resp.Body); err != nil {
panic(err)
}
}
```
@@ -0,0 +1,62 @@
// Copyright 2018 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package transport
import (
"encoding/base64"
"fmt"
"net/http"
"github.com/google/go-containerregistry/pkg/authn"
)
type basicTransport struct {
inner http.RoundTripper
auth authn.Authenticator
target string
}
var _ http.RoundTripper = (*basicTransport)(nil)
// RoundTrip implements http.RoundTripper
func (bt *basicTransport) RoundTrip(in *http.Request) (*http.Response, error) {
if bt.auth != authn.Anonymous {
auth, err := authn.Authorization(in.Context(), bt.auth)
if err != nil {
return nil, err
}
// http.Client handles redirects at a layer above the http.RoundTripper
// abstraction, so to avoid forwarding Authorization headers to places
// we are redirected, only set it when the authorization header matches
// the host with which we are interacting.
// In case of redirect http.Client can use an empty Host, check URL too.
if in.Host == bt.target || in.URL.Host == bt.target {
if bearer := auth.RegistryToken; bearer != "" {
hdr := fmt.Sprintf("Bearer %s", bearer)
in.Header.Set("Authorization", hdr)
} else if user, pass := auth.Username, auth.Password; user != "" && pass != "" {
delimited := fmt.Sprintf("%s:%s", user, pass)
encoded := base64.StdEncoding.EncodeToString([]byte(delimited))
hdr := fmt.Sprintf("Basic %s", encoded)
in.Header.Set("Authorization", hdr)
} else if token := auth.Auth; token != "" {
hdr := fmt.Sprintf("Basic %s", token)
in.Header.Set("Authorization", hdr)
}
}
}
return bt.inner.RoundTrip(in)
}
@@ -0,0 +1,407 @@
// Copyright 2018 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package transport
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net"
"net/http"
"net/url"
"strings"
"sync"
authchallenge "github.com/docker/distribution/registry/client/auth/challenge"
"github.com/google/go-containerregistry/internal/redact"
"github.com/google/go-containerregistry/pkg/authn"
"github.com/google/go-containerregistry/pkg/logs"
"github.com/google/go-containerregistry/pkg/name"
)
type Token struct {
Token string `json:"token"`
AccessToken string `json:"access_token,omitempty"`
RefreshToken string `json:"refresh_token"`
ExpiresIn int `json:"expires_in"`
}
// Exchange requests a registry Token with the given scopes.
func Exchange(ctx context.Context, reg name.Registry, auth authn.Authenticator, t http.RoundTripper, scopes []string, pr *Challenge) (*Token, error) {
if strings.ToLower(pr.Scheme) != "bearer" {
// TODO: Pretend token for basic?
return nil, fmt.Errorf("challenge scheme %q is not bearer", pr.Scheme)
}
bt, err := fromChallenge(reg, auth, t, pr, scopes...)
if err != nil {
return nil, err
}
authcfg, err := authn.Authorization(ctx, auth)
if err != nil {
return nil, err
}
tok, err := bt.Refresh(ctx, authcfg)
if err != nil {
return nil, err
}
return tok, nil
}
// FromToken returns a transport given a Challenge + Token.
func FromToken(reg name.Registry, auth authn.Authenticator, t http.RoundTripper, pr *Challenge, tok *Token) (http.RoundTripper, error) {
if strings.ToLower(pr.Scheme) != "bearer" {
return &Wrapper{&basicTransport{inner: t, auth: auth, target: reg.RegistryStr()}}, nil
}
bt, err := fromChallenge(reg, auth, t, pr)
if err != nil {
return nil, err
}
if tok.Token != "" {
bt.bearer.RegistryToken = tok.Token
}
return &Wrapper{bt}, nil
}
func fromChallenge(reg name.Registry, auth authn.Authenticator, t http.RoundTripper, pr *Challenge, scopes ...string) (*bearerTransport, error) {
// We require the realm, which tells us where to send our Basic auth to turn it into Bearer auth.
realm, ok := pr.Parameters["realm"]
if !ok {
return nil, fmt.Errorf("malformed www-authenticate, missing realm: %v", pr.Parameters)
}
service := pr.Parameters["service"]
scheme := "https"
if pr.Insecure {
scheme = "http"
}
return &bearerTransport{
inner: t,
basic: auth,
realm: realm,
registry: reg,
service: service,
scopes: scopes,
scheme: scheme,
}, nil
}
type bearerTransport struct {
mx sync.RWMutex
// Wrapped by bearerTransport.
inner http.RoundTripper
// Basic credentials that we exchange for bearer tokens.
basic authn.Authenticator
// Holds the bearer response from the token service.
bearer authn.AuthConfig
// Registry to which we send bearer tokens.
registry name.Registry
// See https://tools.ietf.org/html/rfc6750#section-3
realm string
// See https://docs.docker.com/registry/spec/auth/token/
service string
scopes []string
// Scheme we should use, determined by ping response.
scheme string
}
var _ http.RoundTripper = (*bearerTransport)(nil)
var portMap = map[string]string{
"http": "80",
"https": "443",
}
func stringSet(ss []string) map[string]struct{} {
set := make(map[string]struct{})
for _, s := range ss {
set[s] = struct{}{}
}
return set
}
// RoundTrip implements http.RoundTripper
func (bt *bearerTransport) RoundTrip(in *http.Request) (*http.Response, error) {
sendRequest := func() (*http.Response, error) {
// http.Client handles redirects at a layer above the http.RoundTripper
// abstraction, so to avoid forwarding Authorization headers to places
// we are redirected, only set it when the authorization header matches
// the registry with which we are interacting.
// In case of redirect http.Client can use an empty Host, check URL too.
if matchesHost(bt.registry.RegistryStr(), in, bt.scheme) {
bt.mx.RLock()
localToken := bt.bearer.RegistryToken
bt.mx.RUnlock()
hdr := fmt.Sprintf("Bearer %s", localToken)
in.Header.Set("Authorization", hdr)
}
return bt.inner.RoundTrip(in)
}
res, err := sendRequest()
if err != nil {
return nil, err
}
// If we hit a WWW-Authenticate challenge, it might be due to expired tokens or insufficient scope.
if challenges := authchallenge.ResponseChallenges(res); len(challenges) != 0 {
// close out old response, since we will not return it.
res.Body.Close()
newScopes := []string{}
bt.mx.Lock()
got := stringSet(bt.scopes)
for _, wac := range challenges {
// TODO(jonjohnsonjr): Should we also update "realm" or "service"?
if want, ok := wac.Parameters["scope"]; ok {
// Add any scopes that we don't already request.
if _, ok := got[want]; !ok {
newScopes = append(newScopes, want)
}
}
}
// Some registries seem to only look at the first scope parameter during a token exchange.
// If a request fails because it's missing a scope, we should put those at the beginning,
// otherwise the registry might just ignore it :/
newScopes = append(newScopes, bt.scopes...)
bt.scopes = newScopes
bt.mx.Unlock()
// TODO(jonjohnsonjr): Teach transport.Error about "error" and "error_description" from challenge.
// Retry the request to attempt to get a valid token.
if err = bt.refresh(in.Context()); err != nil {
return nil, err
}
return sendRequest()
}
return res, err
}
// It's unclear which authentication flow to use based purely on the protocol,
// so we rely on heuristics and fallbacks to support as many registries as possible.
// The basic token exchange is attempted first, falling back to the oauth flow.
// If the IdentityToken is set, this indicates that we should start with the oauth flow.
func (bt *bearerTransport) refresh(ctx context.Context) error {
auth, err := authn.Authorization(ctx, bt.basic)
if err != nil {
return err
}
if auth.RegistryToken != "" {
bt.mx.Lock()
bt.bearer.RegistryToken = auth.RegistryToken
bt.mx.Unlock()
return nil
}
response, err := bt.Refresh(ctx, auth)
if err != nil {
return err
}
// Some registries set access_token instead of token. See #54.
if response.AccessToken != "" {
response.Token = response.AccessToken
}
// Find a token to turn into a Bearer authenticator
if response.Token != "" {
bt.mx.Lock()
bt.bearer.RegistryToken = response.Token
bt.mx.Unlock()
}
// If we obtained a refresh token from the oauth flow, use that for refresh() now.
if response.RefreshToken != "" {
bt.basic = authn.FromConfig(authn.AuthConfig{
IdentityToken: response.RefreshToken,
})
}
return nil
}
func (bt *bearerTransport) Refresh(ctx context.Context, auth *authn.AuthConfig) (*Token, error) {
var (
content []byte
err error
)
if auth.IdentityToken != "" {
// If the secret being stored is an identity token,
// the Username should be set to <token>, which indicates
// we are using an oauth flow.
content, err = bt.refreshOauth(ctx)
var terr *Error
if errors.As(err, &terr) && terr.StatusCode == http.StatusNotFound {
// Note: Not all token servers implement oauth2.
// If the request to the endpoint returns 404 using the HTTP POST method,
// refer to Token Documentation for using the HTTP GET method supported by all token servers.
content, err = bt.refreshBasic(ctx)
}
} else {
content, err = bt.refreshBasic(ctx)
}
if err != nil {
return nil, err
}
var response Token
if err := json.Unmarshal(content, &response); err != nil {
return nil, err
}
if response.Token == "" && response.AccessToken == "" {
return &response, fmt.Errorf("no token in bearer response:\n%s", content)
}
return &response, nil
}
func matchesHost(host string, in *http.Request, scheme string) bool {
canonicalHeaderHost := canonicalAddress(in.Host, scheme)
canonicalURLHost := canonicalAddress(in.URL.Host, scheme)
canonicalRegistryHost := canonicalAddress(host, scheme)
return canonicalHeaderHost == canonicalRegistryHost || canonicalURLHost == canonicalRegistryHost
}
func canonicalAddress(host, scheme string) (address string) {
// The host may be any one of:
// - hostname
// - hostname:port
// - ipv4
// - ipv4:port
// - ipv6
// - [ipv6]:port
// As net.SplitHostPort returns an error if the host does not contain a port, we should only attempt
// to call it when we know that the address contains a port
if strings.Count(host, ":") == 1 || (strings.Count(host, ":") >= 2 && strings.Contains(host, "]:")) {
hostname, port, err := net.SplitHostPort(host)
if err != nil {
return host
}
if port == "" {
port = portMap[scheme]
}
return net.JoinHostPort(hostname, port)
}
return net.JoinHostPort(host, portMap[scheme])
}
// https://docs.docker.com/registry/spec/auth/oauth/
func (bt *bearerTransport) refreshOauth(ctx context.Context) ([]byte, error) {
auth, err := authn.Authorization(ctx, bt.basic)
if err != nil {
return nil, err
}
u, err := url.Parse(bt.realm)
if err != nil {
return nil, err
}
v := url.Values{}
bt.mx.RLock()
v.Set("scope", strings.Join(bt.scopes, " "))
bt.mx.RUnlock()
if bt.service != "" {
v.Set("service", bt.service)
}
v.Set("client_id", defaultUserAgent)
if auth.IdentityToken != "" {
v.Set("grant_type", "refresh_token")
v.Set("refresh_token", auth.IdentityToken)
} else if auth.Username != "" && auth.Password != "" {
// TODO(#629): This is unreachable.
v.Set("grant_type", "password")
v.Set("username", auth.Username)
v.Set("password", auth.Password)
v.Set("access_type", "offline")
}
client := http.Client{Transport: bt.inner}
req, err := http.NewRequest(http.MethodPost, u.String(), strings.NewReader(v.Encode()))
if err != nil {
return nil, err
}
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
// We don't want to log credentials.
ctx = redact.NewContext(ctx, "oauth token response contains credentials")
resp, err := client.Do(req.WithContext(ctx))
if err != nil {
return nil, err
}
defer resp.Body.Close()
if err := CheckError(resp, http.StatusOK); err != nil {
if bt.basic == authn.Anonymous {
logs.Warn.Printf("No matching credentials were found for %q", bt.registry)
}
return nil, err
}
return io.ReadAll(resp.Body)
}
// https://docs.docker.com/registry/spec/auth/token/
func (bt *bearerTransport) refreshBasic(ctx context.Context) ([]byte, error) {
u, err := url.Parse(bt.realm)
if err != nil {
return nil, err
}
b := &basicTransport{
inner: bt.inner,
auth: bt.basic,
target: u.Host,
}
client := http.Client{Transport: b}
v := u.Query()
bt.mx.RLock()
v["scope"] = bt.scopes
bt.mx.RUnlock()
v.Set("service", bt.service)
u.RawQuery = v.Encode()
req, err := http.NewRequest(http.MethodGet, u.String(), nil)
if err != nil {
return nil, err
}
// We don't want to log credentials.
ctx = redact.NewContext(ctx, "basic token response contains credentials")
resp, err := client.Do(req.WithContext(ctx))
if err != nil {
return nil, err
}
defer resp.Body.Close()
if err := CheckError(resp, http.StatusOK); err != nil {
if bt.basic == authn.Anonymous {
logs.Warn.Printf("No matching credentials were found for %q", bt.registry)
}
return nil, err
}
return io.ReadAll(resp.Body)
}
@@ -0,0 +1,18 @@
// Copyright 2018 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package transport provides facilities for setting up an authenticated
// http.RoundTripper given an Authenticator and base RoundTripper. See
// transport.New for more information.
package transport
@@ -0,0 +1,196 @@
// Copyright 2018 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package transport
import (
"encoding/json"
"fmt"
"io"
"net/http"
"strings"
"github.com/google/go-containerregistry/internal/redact"
)
// Error implements error to support the following error specification:
// https://github.com/distribution/distribution/blob/aac2f6c8b7c5a6c60190848bab5cbeed2b5ba0a9/docs/spec/api.md#errors
type Error struct {
Errors []Diagnostic `json:"errors,omitempty"`
// The http status code returned.
StatusCode int
// The request that failed.
Request *http.Request
// The raw body if we couldn't understand it.
rawBody string
// Bit of a hack to make it easier to force a retry.
temporary bool
}
// Check that Error implements error
var _ error = (*Error)(nil)
// Error implements error
func (e *Error) Error() string {
prefix := ""
if e.Request != nil {
prefix = fmt.Sprintf("%s %s: ", e.Request.Method, redact.URL(e.Request.URL))
}
return prefix + e.responseErr()
}
func (e *Error) responseErr() string {
switch len(e.Errors) {
case 0:
if len(e.rawBody) == 0 {
if e.Request != nil && e.Request.Method == http.MethodHead {
return fmt.Sprintf("unexpected status code %d %s (HEAD responses have no body, use GET for details)", e.StatusCode, http.StatusText(e.StatusCode))
}
return fmt.Sprintf("unexpected status code %d %s", e.StatusCode, http.StatusText(e.StatusCode))
}
return fmt.Sprintf("unexpected status code %d %s: %s", e.StatusCode, http.StatusText(e.StatusCode), e.rawBody)
case 1:
return e.Errors[0].String()
default:
var errors []string
for _, d := range e.Errors {
errors = append(errors, d.String())
}
return fmt.Sprintf("multiple errors returned: %s",
strings.Join(errors, "; "))
}
}
// Temporary returns whether the request that preceded the error is temporary.
func (e *Error) Temporary() bool {
if e.temporary {
return true
}
if len(e.Errors) == 0 {
_, ok := temporaryStatusCodes[e.StatusCode]
return ok
}
for _, d := range e.Errors {
if _, ok := temporaryErrorCodes[d.Code]; !ok {
return false
}
}
return true
}
// Diagnostic represents a single error returned by a Docker registry interaction.
type Diagnostic struct {
Code ErrorCode `json:"code"`
Message string `json:"message,omitempty"`
Detail any `json:"detail,omitempty"`
}
// String stringifies the Diagnostic in the form: $Code: $Message[; $Detail]
func (d Diagnostic) String() string {
msg := fmt.Sprintf("%s: %s", d.Code, d.Message)
if d.Detail != nil {
msg = fmt.Sprintf("%s; %v", msg, d.Detail)
}
return msg
}
// ErrorCode is an enumeration of supported error codes.
type ErrorCode string
// The set of error conditions a registry may return:
// https://github.com/distribution/distribution/blob/aac2f6c8b7c5a6c60190848bab5cbeed2b5ba0a9/docs/spec/api.md#errors-2
const (
BlobUnknownErrorCode ErrorCode = "BLOB_UNKNOWN"
BlobUploadInvalidErrorCode ErrorCode = "BLOB_UPLOAD_INVALID"
BlobUploadUnknownErrorCode ErrorCode = "BLOB_UPLOAD_UNKNOWN"
DigestInvalidErrorCode ErrorCode = "DIGEST_INVALID"
ManifestBlobUnknownErrorCode ErrorCode = "MANIFEST_BLOB_UNKNOWN"
ManifestInvalidErrorCode ErrorCode = "MANIFEST_INVALID"
ManifestUnknownErrorCode ErrorCode = "MANIFEST_UNKNOWN"
ManifestUnverifiedErrorCode ErrorCode = "MANIFEST_UNVERIFIED"
NameInvalidErrorCode ErrorCode = "NAME_INVALID"
NameUnknownErrorCode ErrorCode = "NAME_UNKNOWN"
SizeInvalidErrorCode ErrorCode = "SIZE_INVALID"
TagInvalidErrorCode ErrorCode = "TAG_INVALID"
UnauthorizedErrorCode ErrorCode = "UNAUTHORIZED"
DeniedErrorCode ErrorCode = "DENIED"
UnsupportedErrorCode ErrorCode = "UNSUPPORTED"
TooManyRequestsErrorCode ErrorCode = "TOOMANYREQUESTS"
UnknownErrorCode ErrorCode = "UNKNOWN"
// This isn't defined by either docker or OCI spec, but is defined by docker/distribution:
// https://github.com/distribution/distribution/blob/6a977a5a754baa213041443f841705888107362a/registry/api/errcode/register.go#L60
UnavailableErrorCode ErrorCode = "UNAVAILABLE"
)
// TODO: Include other error types.
var temporaryErrorCodes = map[ErrorCode]struct{}{
BlobUploadInvalidErrorCode: {},
TooManyRequestsErrorCode: {},
UnknownErrorCode: {},
UnavailableErrorCode: {},
}
var temporaryStatusCodes = map[int]struct{}{
http.StatusRequestTimeout: {},
http.StatusInternalServerError: {},
http.StatusBadGateway: {},
http.StatusServiceUnavailable: {},
http.StatusGatewayTimeout: {},
}
// CheckError returns a structured error if the response status is not in codes.
func CheckError(resp *http.Response, codes ...int) error {
for _, code := range codes {
if resp.StatusCode == code {
// This is one of the supported status codes.
return nil
}
}
b, err := io.ReadAll(resp.Body)
if err != nil {
return err
}
return makeError(resp, b)
}
func makeError(resp *http.Response, body []byte) *Error {
// https://github.com/distribution/distribution/blob/aac2f6c8b7c5a6c60190848bab5cbeed2b5ba0a9/docs/spec/api.md#errors
structuredError := &Error{}
// This can fail if e.g. the response body is not valid JSON. That's fine,
// we'll construct an appropriate error string from the body and status code.
_ = json.Unmarshal(body, structuredError)
structuredError.rawBody = string(body)
structuredError.StatusCode = resp.StatusCode
structuredError.Request = resp.Request
return structuredError
}
func retryError(resp *http.Response) error {
b, err := io.ReadAll(resp.Body)
if err != nil {
return err
}
rerr := makeError(resp, b)
rerr.temporary = true
return rerr
}
@@ -0,0 +1,91 @@
// Copyright 2020 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package transport
import (
"fmt"
"net/http"
"net/http/httputil"
"time"
"github.com/google/go-containerregistry/internal/redact"
"github.com/google/go-containerregistry/pkg/logs"
)
type logTransport struct {
inner http.RoundTripper
}
// NewLogger returns a transport that logs requests and responses to
// github.com/google/go-containerregistry/pkg/logs.Debug.
func NewLogger(inner http.RoundTripper) http.RoundTripper {
return &logTransport{inner}
}
func (t *logTransport) RoundTrip(in *http.Request) (out *http.Response, err error) {
// Inspired by: github.com/motemen/go-loghttp
// We redact token responses and binary blobs in response/request.
omitBody, reason := redact.FromContext(in.Context())
if omitBody {
logs.Debug.Printf("--> %s %s [body redacted: %s]", in.Method, in.URL, reason)
} else {
logs.Debug.Printf("--> %s %s", in.Method, in.URL)
}
// Save these headers so we can redact Authorization.
savedHeaders := in.Header.Clone()
if in.Header != nil && in.Header.Get("authorization") != "" {
in.Header.Set("authorization", "<redacted>")
}
b, err := httputil.DumpRequestOut(in, !omitBody)
if err == nil {
logs.Debug.Println(string(b))
} else {
logs.Debug.Printf("Failed to dump request %s %s: %v", in.Method, in.URL, err)
}
// Restore the non-redacted headers.
in.Header = savedHeaders
start := time.Now()
out, err = t.inner.RoundTrip(in)
duration := time.Since(start)
if err != nil {
logs.Debug.Printf("<-- %v %s %s (%s)", err, in.Method, in.URL, duration)
}
if out != nil {
msg := fmt.Sprintf("<-- %d", out.StatusCode)
if out.Request != nil {
msg = fmt.Sprintf("%s %s", msg, out.Request.URL)
}
msg = fmt.Sprintf("%s (%s)", msg, duration)
if omitBody {
msg = fmt.Sprintf("%s [body redacted: %s]", msg, reason)
}
logs.Debug.Print(msg)
b, err := httputil.DumpResponse(out, !omitBody)
if err == nil {
logs.Debug.Println(string(b))
} else {
logs.Debug.Printf("Failed to dump response %s %s: %v", in.Method, in.URL, err)
}
}
return
}
@@ -0,0 +1,217 @@
// Copyright 2018 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package transport
import (
"context"
"errors"
"fmt"
"io"
"net/http"
"strings"
"time"
authchallenge "github.com/docker/distribution/registry/client/auth/challenge"
"github.com/google/go-containerregistry/pkg/logs"
"github.com/google/go-containerregistry/pkg/name"
)
// 300ms is the default fallback period for go's DNS dialer but we could make this configurable.
var fallbackDelay = 300 * time.Millisecond
type Challenge struct {
Scheme string
// Following the challenge there are often key/value pairs
// e.g. Bearer service="gcr.io",realm="https://auth.gcr.io/v36/tokenz"
Parameters map[string]string
// Whether we had to use http to complete the Ping.
Insecure bool
}
// Ping does a GET /v2/ against the registry and returns the response.
func Ping(ctx context.Context, reg name.Registry, t http.RoundTripper) (*Challenge, error) {
// This first attempts to use "https" for every request, falling back to http
// if the registry matches our localhost heuristic or if it is intentionally
// set to insecure via name.NewInsecureRegistry.
schemes := []string{"https"}
if reg.Scheme() == "http" {
schemes = append(schemes, "http")
}
if len(schemes) == 1 {
return pingSingle(ctx, reg, t, schemes[0])
}
return pingParallel(ctx, reg, t, schemes)
}
func pingSingle(ctx context.Context, reg name.Registry, t http.RoundTripper, scheme string) (*Challenge, error) {
client := http.Client{Transport: t}
url := fmt.Sprintf("%s://%s/v2/", scheme, reg.RegistryStr())
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
return nil, err
}
resp, err := client.Do(req.WithContext(ctx))
if err != nil {
return nil, err
}
defer func() {
// By draining the body, make sure to reuse the connection made by
// the ping for the following access to the registry
io.Copy(io.Discard, resp.Body)
resp.Body.Close()
}()
insecure := scheme == "http"
switch resp.StatusCode {
case http.StatusOK:
// If we get a 200, then no authentication is needed.
return &Challenge{
Insecure: insecure,
}, nil
case http.StatusUnauthorized:
if challenges := authchallenge.ResponseChallenges(resp); len(challenges) != 0 {
// If we hit more than one, let's try to find one that we know how to handle.
wac := pickFromMultipleChallenges(challenges)
return &Challenge{
Scheme: wac.Scheme,
Parameters: wac.Parameters,
Insecure: insecure,
}, nil
}
// Otherwise, just return the challenge without parameters.
return &Challenge{
Scheme: resp.Header.Get("WWW-Authenticate"),
Insecure: insecure,
}, nil
default:
return nil, CheckError(resp, http.StatusOK, http.StatusUnauthorized)
}
}
// Based on the golang happy eyeballs dialParallel impl in net/dial.go.
func pingParallel(ctx context.Context, reg name.Registry, t http.RoundTripper, schemes []string) (*Challenge, error) {
returned := make(chan struct{})
defer close(returned)
type pingResult struct {
*Challenge
error
primary bool
done bool
}
results := make(chan pingResult)
startRacer := func(ctx context.Context, scheme string) {
pr, err := pingSingle(ctx, reg, t, scheme)
select {
case results <- pingResult{Challenge: pr, error: err, primary: scheme == "https", done: true}:
case <-returned:
if pr != nil {
logs.Debug.Printf("%s lost race", scheme)
}
}
}
var primary, fallback pingResult
primaryCtx, primaryCancel := context.WithCancel(ctx)
defer primaryCancel()
go startRacer(primaryCtx, schemes[0])
fallbackTimer := time.NewTimer(fallbackDelay)
defer fallbackTimer.Stop()
for {
select {
case <-fallbackTimer.C:
fallbackCtx, fallbackCancel := context.WithCancel(ctx)
defer fallbackCancel()
go startRacer(fallbackCtx, schemes[1])
case res := <-results:
if res.error == nil {
return res.Challenge, nil
}
if res.primary {
primary = res
} else {
fallback = res
}
if primary.done && fallback.done {
return nil, multierrs{primary.error, fallback.error}
}
if res.primary && fallbackTimer.Stop() {
// Primary failed and we haven't started the fallback,
// reset time to start fallback immediately.
fallbackTimer.Reset(0)
}
}
}
}
func pickFromMultipleChallenges(challenges []authchallenge.Challenge) authchallenge.Challenge {
// It might happen there are multiple www-authenticate headers, e.g. `Negotiate` and `Basic`.
// Picking simply the first one could result eventually in `unrecognized challenge` error,
// that's why we're looping through the challenges in search for one that can be handled.
allowedSchemes := []string{"basic", "bearer"}
for _, wac := range challenges {
currentScheme := strings.ToLower(wac.Scheme)
for _, allowed := range allowedSchemes {
if allowed == currentScheme {
return wac
}
}
}
return challenges[0]
}
type multierrs []error
func (m multierrs) Error() string {
var b strings.Builder
hasWritten := false
for _, err := range m {
if hasWritten {
b.WriteString("; ")
}
hasWritten = true
b.WriteString(err.Error())
}
return b.String()
}
func (m multierrs) As(target any) bool {
for _, err := range m {
if errors.As(err, target) {
return true
}
}
return false
}
func (m multierrs) Is(target error) bool {
for _, err := range m {
if errors.Is(err, target) {
return true
}
}
return false
}
@@ -0,0 +1,111 @@
// Copyright 2018 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package transport
import (
"net/http"
"time"
"github.com/google/go-containerregistry/internal/retry"
)
// Sleep for 0.1 then 0.3 seconds. This should cover networking blips.
var defaultBackoff = retry.Backoff{
Duration: 100 * time.Millisecond,
Factor: 3.0,
Jitter: 0.1,
Steps: 3,
}
var _ http.RoundTripper = (*retryTransport)(nil)
// retryTransport wraps a RoundTripper and retries temporary network errors.
type retryTransport struct {
inner http.RoundTripper
backoff retry.Backoff
predicate retry.Predicate
codes []int
}
// Option is a functional option for retryTransport.
type Option func(*options)
type options struct {
backoff retry.Backoff
predicate retry.Predicate
codes []int
}
// Backoff is an alias of retry.Backoff to expose this configuration option to consumers of this lib
type Backoff = retry.Backoff
// WithRetryBackoff sets the backoff for retry operations.
func WithRetryBackoff(backoff Backoff) Option {
return func(o *options) {
o.backoff = backoff
}
}
// WithRetryPredicate sets the predicate for retry operations.
func WithRetryPredicate(predicate func(error) bool) Option {
return func(o *options) {
o.predicate = predicate
}
}
// WithRetryStatusCodes sets which http response codes will be retried.
func WithRetryStatusCodes(codes ...int) Option {
return func(o *options) {
o.codes = codes
}
}
// NewRetry returns a transport that retries errors.
func NewRetry(inner http.RoundTripper, opts ...Option) http.RoundTripper {
o := &options{
backoff: defaultBackoff,
predicate: retry.IsTemporary,
}
for _, opt := range opts {
opt(o)
}
return &retryTransport{
inner: inner,
backoff: o.backoff,
predicate: o.predicate,
codes: o.codes,
}
}
func (t *retryTransport) RoundTrip(in *http.Request) (out *http.Response, err error) {
roundtrip := func() error {
out, err = t.inner.RoundTrip(in)
if !retry.Ever(in.Context()) {
return nil
}
if out != nil {
for _, code := range t.codes {
if out.StatusCode == code {
return retryError(out)
}
}
}
return err
}
retry.Retry(roundtrip, t.predicate, t.backoff)
return
}
@@ -0,0 +1,44 @@
// Copyright 2019 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package transport
import (
"net/http"
"github.com/google/go-containerregistry/pkg/name"
)
type schemeTransport struct {
// Scheme we should use, determined by ping response.
scheme string
// Registry we're talking to.
registry name.Registry
// Wrapped by schemeTransport.
inner http.RoundTripper
}
// RoundTrip implements http.RoundTripper
func (st *schemeTransport) RoundTrip(in *http.Request) (*http.Response, error) {
// When we ping() the registry, we determine whether to use http or https
// based on which scheme was successful. That is only valid for the
// registry server and not e.g. a separate token server or blob storage,
// so we should only override the scheme if the host is the registry.
if matchesHost(st.registry.String(), in, st.scheme) {
in.URL.Scheme = st.scheme
}
return st.inner.RoundTrip(in)
}
@@ -0,0 +1,24 @@
// Copyright 2018 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package transport
// Scopes suitable to qualify each Repository
const (
PullScope string = "pull"
PushScope string = "push,pull"
// For now DELETE is PUSH, which is the read/write ACL.
DeleteScope string = PushScope
CatalogScope string = "catalog"
)
@@ -0,0 +1,109 @@
// Copyright 2018 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package transport
import (
"context"
"net/http"
"strings"
"github.com/google/go-containerregistry/pkg/authn"
"github.com/google/go-containerregistry/pkg/name"
)
// New returns a new RoundTripper based on the provided RoundTripper that has been
// setup to authenticate with the remote registry "reg", in the capacity
// laid out by the specified scopes.
//
// Deprecated: Use NewWithContext.
func New(reg name.Registry, auth authn.Authenticator, t http.RoundTripper, scopes []string) (http.RoundTripper, error) {
return NewWithContext(context.Background(), reg, auth, t, scopes)
}
// NewWithContext returns a new RoundTripper based on the provided RoundTripper that has been
// set up to authenticate with the remote registry "reg", in the capacity
// laid out by the specified scopes.
// In case the RoundTripper is already of the type Wrapper it assumes
// authentication was already done prior to this call, so it just returns
// the provided RoundTripper without further action
func NewWithContext(ctx context.Context, reg name.Registry, auth authn.Authenticator, t http.RoundTripper, scopes []string) (http.RoundTripper, error) {
// When the transport provided is of the type Wrapper this function assumes that the caller already
// executed the necessary login and check.
switch t.(type) {
case *Wrapper:
return t, nil
}
// The handshake:
// 1. Use "t" to ping() the registry for the authentication challenge.
//
// 2a. If we get back a 200, then simply use "t".
//
// 2b. If we get back a 401 with a Basic challenge, then use a transport
// that just attachs auth each roundtrip.
//
// 2c. If we get back a 401 with a Bearer challenge, then use a transport
// that attaches a bearer token to each request, and refreshes is on 401s.
// Perform an initial refresh to seed the bearer token.
// First we ping the registry to determine the parameters of the authentication handshake
// (if one is even necessary).
pr, err := Ping(ctx, reg, t)
if err != nil {
return nil, err
}
// Wrap t with a useragent transport unless we already have one.
if _, ok := t.(*userAgentTransport); !ok {
t = NewUserAgent(t, "")
}
scheme := "https"
if pr.Insecure {
scheme = "http"
}
// Wrap t in a transport that selects the appropriate scheme based on the ping response.
t = &schemeTransport{
scheme: scheme,
registry: reg,
inner: t,
}
if strings.ToLower(pr.Scheme) != "bearer" {
return &Wrapper{&basicTransport{inner: t, auth: auth, target: reg.RegistryStr()}}, nil
}
bt, err := fromChallenge(reg, auth, t, pr)
if err != nil {
return nil, err
}
bt.scopes = scopes
if err := bt.refresh(ctx); err != nil {
return nil, err
}
return &Wrapper{bt}, nil
}
// Wrapper results in *not* wrapping supplied transport with additional logic such as retries, useragent and debug logging
// Consumers are opt-ing into providing their own transport without any additional wrapping.
type Wrapper struct {
inner http.RoundTripper
}
// RoundTrip delegates to the inner RoundTripper
func (w *Wrapper) RoundTrip(in *http.Request) (*http.Response, error) {
return w.inner.RoundTrip(in)
}
@@ -0,0 +1,94 @@
// Copyright 2019 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package transport
import (
"fmt"
"net/http"
"runtime/debug"
)
var (
// Version can be set via:
// -ldflags="-X 'github.com/google/go-containerregistry/pkg/v1/remote/transport.Version=$TAG'"
Version string
ggcrVersion = defaultUserAgent
)
const (
defaultUserAgent = "go-containerregistry"
moduleName = "github.com/google/go-containerregistry"
)
type userAgentTransport struct {
inner http.RoundTripper
ua string
}
func init() {
if v := version(); v != "" {
ggcrVersion = fmt.Sprintf("%s/%s", defaultUserAgent, v)
}
}
func version() string {
if Version != "" {
// Version was set via ldflags, just return it.
return Version
}
info, ok := debug.ReadBuildInfo()
if !ok {
return ""
}
// Happens for crane and gcrane.
if info.Main.Path == moduleName {
return info.Main.Version
}
// Anything else.
for _, dep := range info.Deps {
if dep.Path == moduleName {
return dep.Version
}
}
return ""
}
// NewUserAgent returns an http.Roundtripper that sets the user agent to
// The provided string plus additional go-containerregistry information,
// e.g. if provided "crane/v0.1.4" and this modules was built at v0.1.4:
//
// User-Agent: crane/v0.1.4 go-containerregistry/v0.1.4
func NewUserAgent(inner http.RoundTripper, ua string) http.RoundTripper {
if ua == "" {
ua = ggcrVersion
} else {
ua = fmt.Sprintf("%s %s", ua, ggcrVersion)
}
return &userAgentTransport{
inner: inner,
ua: ua,
}
}
// RoundTrip implements http.RoundTripper
func (ut *userAgentTransport) RoundTrip(in *http.Request) (*http.Response, error) {
in.Header.Set("User-Agent", ut.ua)
return ut.inner.RoundTrip(in)
}
+711
View File
@@ -0,0 +1,711 @@
// Copyright 2018 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package remote
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"sort"
"strings"
"sync"
"github.com/google/go-containerregistry/internal/redact"
"github.com/google/go-containerregistry/internal/retry"
"github.com/google/go-containerregistry/pkg/authn"
"github.com/google/go-containerregistry/pkg/logs"
"github.com/google/go-containerregistry/pkg/name"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/remote/transport"
"github.com/google/go-containerregistry/pkg/v1/stream"
"github.com/google/go-containerregistry/pkg/v1/types"
)
// Taggable is an interface that enables a manifest PUT (e.g. for tagging).
type Taggable interface {
RawManifest() ([]byte, error)
}
// Write pushes the provided img to the specified image reference.
func Write(ref name.Reference, img v1.Image, options ...Option) (rerr error) {
return Push(ref, img, options...)
}
// writer writes the elements of an image to a remote image reference.
type writer struct {
repo name.Repository
auth authn.Authenticator
transport http.RoundTripper
client *http.Client
progress *progress
backoff Backoff
predicate retry.Predicate
scopeLock sync.Mutex
// Keep track of scopes that we have already requested.
scopeSet map[string]struct{}
scopes []string
}
func makeWriter(ctx context.Context, repo name.Repository, ls []v1.Layer, o *options) (*writer, error) {
auth := o.auth
if o.keychain != nil {
kauth, err := authn.Resolve(ctx, o.keychain, repo)
if err != nil {
return nil, err
}
auth = kauth
}
scopes := scopesForUploadingImage(repo, ls)
tr, err := transport.NewWithContext(ctx, repo.Registry, auth, o.transport, scopes)
if err != nil {
return nil, err
}
scopeSet := map[string]struct{}{}
for _, scope := range scopes {
scopeSet[scope] = struct{}{}
}
return &writer{
repo: repo,
client: &http.Client{Transport: tr},
auth: auth,
transport: o.transport,
progress: o.progress,
backoff: o.retryBackoff,
predicate: o.retryPredicate,
scopes: scopes,
scopeSet: scopeSet,
}, nil
}
// url returns a url.Url for the specified path in the context of this remote image reference.
func (w *writer) url(path string) url.URL {
return url.URL{
Scheme: w.repo.Scheme(),
Host: w.repo.RegistryStr(),
Path: path,
}
}
func (w *writer) maybeUpdateScopes(ctx context.Context, ml *MountableLayer) error {
if ml.Reference.Context().String() == w.repo.String() {
return nil
}
if ml.Reference.Context().Registry.String() != w.repo.Registry.String() {
return nil
}
scope := ml.Reference.Scope(transport.PullScope)
w.scopeLock.Lock()
defer w.scopeLock.Unlock()
if _, ok := w.scopeSet[scope]; !ok {
w.scopeSet[scope] = struct{}{}
w.scopes = append(w.scopes, scope)
logs.Debug.Printf("Refreshing token to add scope %q", scope)
wt, err := transport.NewWithContext(ctx, w.repo.Registry, w.auth, w.transport, w.scopes)
if err != nil {
return err
}
w.client = &http.Client{Transport: wt}
}
return nil
}
// nextLocation extracts the fully-qualified URL to which we should send the next request in an upload sequence.
func (w *writer) nextLocation(resp *http.Response) (string, error) {
loc := resp.Header.Get("Location")
if len(loc) == 0 {
return "", errors.New("missing Location header")
}
u, err := url.Parse(loc)
if err != nil {
return "", err
}
// If the location header returned is just a url path, then fully qualify it.
// We cannot simply call w.url, since there might be an embedded query string.
return resp.Request.URL.ResolveReference(u).String(), nil
}
// checkExistingBlob checks if a blob exists already in the repository by making a
// HEAD request to the blob store API. GCR performs an existence check on the
// initiation if "mount" is specified, even if no "from" sources are specified.
// However, this is not broadly applicable to all registries, e.g. ECR.
func (w *writer) checkExistingBlob(ctx context.Context, h v1.Hash) (bool, error) {
u := w.url(fmt.Sprintf("/v2/%s/blobs/%s", w.repo.RepositoryStr(), h.String()))
req, err := http.NewRequest(http.MethodHead, u.String(), nil)
if err != nil {
return false, err
}
resp, err := w.client.Do(req.WithContext(ctx))
if err != nil {
return false, err
}
defer resp.Body.Close()
if err := transport.CheckError(resp, http.StatusOK, http.StatusNotFound); err != nil {
return false, err
}
return resp.StatusCode == http.StatusOK, nil
}
// initiateUpload initiates the blob upload, which starts with a POST that can
// optionally include the hash of the layer and a list of repositories from
// which that layer might be read. On failure, an error is returned.
// On success, the layer was either mounted (nothing more to do) or a blob
// upload was initiated and the body of that blob should be sent to the returned
// location.
func (w *writer) initiateUpload(ctx context.Context, from, mount, origin string) (location string, mounted bool, err error) {
u := w.url(fmt.Sprintf("/v2/%s/blobs/uploads/", w.repo.RepositoryStr()))
uv := url.Values{}
if mount != "" && from != "" {
// Quay will fail if we specify a "mount" without a "from".
uv.Set("mount", mount)
uv.Set("from", from)
if origin != "" {
uv.Set("origin", origin)
}
}
u.RawQuery = uv.Encode()
// Make the request to initiate the blob upload.
req, err := http.NewRequest(http.MethodPost, u.String(), nil)
if err != nil {
return "", false, err
}
req.Header.Set("Content-Type", "application/json")
resp, err := w.client.Do(req.WithContext(ctx))
if err != nil {
if from != "" {
// https://github.com/google/go-containerregistry/issues/1679
logs.Warn.Printf("retrying without mount: %v", err)
return w.initiateUpload(ctx, "", "", "")
}
return "", false, err
}
defer resp.Body.Close()
if err := transport.CheckError(resp, http.StatusCreated, http.StatusAccepted); err != nil {
if from != "" {
// https://github.com/google/go-containerregistry/issues/1404
logs.Warn.Printf("retrying without mount: %v", err)
return w.initiateUpload(ctx, "", "", "")
}
return "", false, err
}
// Check the response code to determine the result.
switch resp.StatusCode {
case http.StatusCreated:
// We're done, we were able to fast-path.
return "", true, nil
case http.StatusAccepted:
// Proceed to PATCH, upload has begun.
loc, err := w.nextLocation(resp)
return loc, false, err
default:
panic("Unreachable: initiateUpload")
}
}
// streamBlob streams the contents of the blob to the specified location.
// On failure, this will return an error. On success, this will return the location
// header indicating how to commit the streamed blob.
func (w *writer) streamBlob(ctx context.Context, layer v1.Layer, streamLocation string) (commitLocation string, rerr error) {
reset := func() {}
defer func() {
if rerr != nil {
reset()
}
}()
blob, err := layer.Compressed()
if err != nil {
return "", err
}
getBody := layer.Compressed
if w.progress != nil {
var count int64
blob = &progressReader{rc: blob, progress: w.progress, count: &count}
getBody = func() (io.ReadCloser, error) {
blob, err := layer.Compressed()
if err != nil {
return nil, err
}
return &progressReader{rc: blob, progress: w.progress, count: &count}, nil
}
reset = func() {
w.progress.complete(-count)
}
}
req, err := http.NewRequest(http.MethodPatch, streamLocation, blob)
if err != nil {
return "", err
}
if _, ok := layer.(*stream.Layer); !ok {
// We can't retry streaming layers.
req.GetBody = getBody
// If we know the size, set it.
if size, err := layer.Size(); err == nil {
req.ContentLength = size
}
}
req.Header.Set("Content-Type", "application/octet-stream")
resp, err := w.client.Do(req.WithContext(ctx))
if err != nil {
return "", err
}
defer resp.Body.Close()
if err := transport.CheckError(resp, http.StatusNoContent, http.StatusAccepted, http.StatusCreated); err != nil {
return "", err
}
// The blob has been uploaded, return the location header indicating
// how to commit this layer.
return w.nextLocation(resp)
}
// commitBlob commits this blob by sending a PUT to the location returned from
// streaming the blob.
func (w *writer) commitBlob(ctx context.Context, location, digest string) error {
u, err := url.Parse(location)
if err != nil {
return err
}
v := u.Query()
v.Set("digest", digest)
u.RawQuery = v.Encode()
req, err := http.NewRequest(http.MethodPut, u.String(), nil)
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/octet-stream")
resp, err := w.client.Do(req.WithContext(ctx))
if err != nil {
return err
}
defer resp.Body.Close()
return transport.CheckError(resp, http.StatusCreated)
}
// incrProgress increments and sends a progress update, if WithProgress is used.
func (w *writer) incrProgress(written int64) {
if w.progress == nil {
return
}
w.progress.complete(written)
}
// uploadOne performs a complete upload of a single layer.
func (w *writer) uploadOne(ctx context.Context, l v1.Layer) error {
tryUpload := func() error {
ctx := retry.Never(ctx)
var from, mount, origin string
if h, err := l.Digest(); err == nil {
// If we know the digest, this isn't a streaming layer. Do an existence
// check so we can skip uploading the layer if possible.
existing, err := w.checkExistingBlob(ctx, h)
if err != nil {
return err
}
if existing {
size, err := l.Size()
if err != nil {
return err
}
w.incrProgress(size)
logs.Progress.Printf("existing blob: %v", h)
return nil
}
mount = h.String()
}
if ml, ok := l.(*MountableLayer); ok {
if err := w.maybeUpdateScopes(ctx, ml); err != nil {
return err
}
from = ml.Reference.Context().RepositoryStr()
origin = ml.Reference.Context().RegistryStr()
// This keeps breaking with DockerHub.
// https://github.com/google/go-containerregistry/issues/1741
if w.repo.RegistryStr() == name.DefaultRegistry && origin != w.repo.RegistryStr() {
from = ""
origin = ""
}
}
location, mounted, err := w.initiateUpload(ctx, from, mount, origin)
if err != nil {
return err
} else if mounted {
size, err := l.Size()
if err != nil {
return err
}
w.incrProgress(size)
h, err := l.Digest()
if err != nil {
return err
}
logs.Progress.Printf("mounted blob: %s", h.String())
return nil
}
// Only log layers with +json or +yaml. We can let through other stuff if it becomes popular.
// TODO(opencontainers/image-spec#791): Would be great to have an actual parser.
mt, err := l.MediaType()
if err != nil {
return err
}
smt := string(mt)
if !strings.HasSuffix(smt, "+json") && !strings.HasSuffix(smt, "+yaml") {
ctx = redact.NewContext(ctx, "omitting binary blobs from logs")
}
location, err = w.streamBlob(ctx, l, location)
if err != nil {
return err
}
h, err := l.Digest()
if err != nil {
return err
}
digest := h.String()
if err := w.commitBlob(ctx, location, digest); err != nil {
return err
}
logs.Progress.Printf("pushed blob: %s", digest)
return nil
}
return retry.Retry(tryUpload, w.predicate, w.backoff)
}
type withMediaType interface {
MediaType() (types.MediaType, error)
}
// This is really silly, but go interfaces don't let me satisfy remote.Taggable
// with remote.Descriptor because of name collisions between method names and
// struct fields.
//
// Use reflection to either pull the v1.Descriptor out of remote.Descriptor or
// create a descriptor based on the RawManifest and (optionally) MediaType.
func unpackTaggable(t Taggable) ([]byte, *v1.Descriptor, error) {
if d, ok := t.(*Descriptor); ok {
return d.Manifest, &d.Descriptor, nil
}
b, err := t.RawManifest()
if err != nil {
return nil, nil, err
}
// A reasonable default if Taggable doesn't implement MediaType.
mt := types.DockerManifestSchema2
if wmt, ok := t.(withMediaType); ok {
m, err := wmt.MediaType()
if err != nil {
return nil, nil, err
}
mt = m
}
h, sz, err := v1.SHA256(bytes.NewReader(b))
if err != nil {
return nil, nil, err
}
return b, &v1.Descriptor{
MediaType: mt,
Size: sz,
Digest: h,
}, nil
}
// commitSubjectReferrers is responsible for updating the fallback tag manifest to track descriptors referring to a subject for registries that don't yet support the Referrers API.
// TODO: use conditional requests to avoid race conditions
func (w *writer) commitSubjectReferrers(ctx context.Context, sub name.Digest, add v1.Descriptor) error {
// Check if the registry supports Referrers API.
// TODO: This should be done once per registry, not once per subject.
u := w.url(fmt.Sprintf("/v2/%s/referrers/%s", w.repo.RepositoryStr(), sub.DigestStr()))
req, err := http.NewRequest(http.MethodGet, u.String(), nil)
if err != nil {
return err
}
req.Header.Set("Accept", string(types.OCIImageIndex))
resp, err := w.client.Do(req.WithContext(ctx))
if err != nil {
return err
}
defer resp.Body.Close()
if err := transport.CheckError(resp, http.StatusOK, http.StatusNotFound, http.StatusBadRequest); err != nil {
return err
}
if resp.StatusCode == http.StatusOK {
// The registry supports Referrers API. The registry is responsible for updating the referrers list.
return nil
}
// The registry doesn't support Referrers API, we need to update the manifest tagged with the fallback tag.
// Make the request to GET the current manifest.
t := fallbackTag(sub)
u = w.url(fmt.Sprintf("/v2/%s/manifests/%s", w.repo.RepositoryStr(), t.Identifier()))
req, err = http.NewRequest(http.MethodGet, u.String(), nil)
if err != nil {
return err
}
req.Header.Set("Accept", string(types.OCIImageIndex))
resp, err = w.client.Do(req.WithContext(ctx))
if err != nil {
return err
}
defer resp.Body.Close()
var im v1.IndexManifest
if err := transport.CheckError(resp, http.StatusOK, http.StatusNotFound); err != nil {
return err
} else if resp.StatusCode == http.StatusNotFound {
// Not found just means there are no attachments. Start with an empty index.
im = v1.IndexManifest{
SchemaVersion: 2,
MediaType: types.OCIImageIndex,
Manifests: []v1.Descriptor{add},
}
} else {
if err := json.NewDecoder(resp.Body).Decode(&im); err != nil {
return err
}
if im.SchemaVersion != 2 {
return fmt.Errorf("fallback tag manifest is not a schema version 2: %d", im.SchemaVersion)
}
if im.MediaType != types.OCIImageIndex {
return fmt.Errorf("fallback tag manifest is not an OCI image index: %s", im.MediaType)
}
for _, desc := range im.Manifests {
if desc.Digest == add.Digest {
// The digest is already attached, nothing to do.
logs.Progress.Printf("fallback tag %s already had referrer", t.Identifier())
return nil
}
}
// Append the new descriptor to the index.
im.Manifests = append(im.Manifests, add)
}
// Sort the manifests for reproducibility.
sort.Slice(im.Manifests, func(i, j int) bool {
return im.Manifests[i].Digest.String() < im.Manifests[j].Digest.String()
})
logs.Progress.Printf("updating fallback tag %s with new referrer", t.Identifier())
return w.commitManifest(ctx, fallbackTaggable{im}, t)
}
type fallbackTaggable struct {
im v1.IndexManifest
}
func (f fallbackTaggable) RawManifest() ([]byte, error) { return json.Marshal(f.im) }
func (f fallbackTaggable) MediaType() (types.MediaType, error) { return types.OCIImageIndex, nil }
// commitManifest does a PUT of the image's manifest.
func (w *writer) commitManifest(ctx context.Context, t Taggable, ref name.Reference) error {
// If the manifest refers to a subject, we need to check whether we need to update the fallback tag manifest.
raw, err := t.RawManifest()
if err != nil {
return err
}
var mf struct {
MediaType types.MediaType `json:"mediaType"`
Subject *v1.Descriptor `json:"subject,omitempty"`
Config struct {
MediaType types.MediaType `json:"mediaType"`
} `json:"config"`
}
if err := json.Unmarshal(raw, &mf); err != nil {
return err
}
tryUpload := func() error {
ctx := retry.Never(ctx)
raw, desc, err := unpackTaggable(t)
if err != nil {
return err
}
u := w.url(fmt.Sprintf("/v2/%s/manifests/%s", w.repo.RepositoryStr(), ref.Identifier()))
// Make the request to PUT the serialized manifest
req, err := http.NewRequest(http.MethodPut, u.String(), bytes.NewBuffer(raw))
if err != nil {
return err
}
req.Header.Set("Content-Type", string(desc.MediaType))
resp, err := w.client.Do(req.WithContext(ctx))
if err != nil {
return err
}
defer resp.Body.Close()
if err := transport.CheckError(resp, http.StatusOK, http.StatusCreated, http.StatusAccepted); err != nil {
return err
}
// If the manifest referred to a subject, we may need to update the fallback tag manifest.
// TODO: If this fails, we'll retry the whole upload. We should retry just this part.
if mf.Subject != nil {
h, size, err := v1.SHA256(bytes.NewReader(raw))
if err != nil {
return err
}
desc := v1.Descriptor{
ArtifactType: string(mf.Config.MediaType),
MediaType: mf.MediaType,
Digest: h,
Size: size,
}
if err := w.commitSubjectReferrers(ctx,
ref.Context().Digest(mf.Subject.Digest.String()),
desc); err != nil {
return err
}
}
// The image was successfully pushed!
logs.Progress.Printf("%v: digest: %v size: %d", ref, desc.Digest, desc.Size)
w.incrProgress(int64(len(raw)))
return nil
}
return retry.Retry(tryUpload, w.predicate, w.backoff)
}
func scopesForUploadingImage(repo name.Repository, layers []v1.Layer) []string {
// use a map as set to remove duplicates scope strings
scopeSet := map[string]struct{}{}
for _, l := range layers {
if ml, ok := l.(*MountableLayer); ok {
// we will add push scope for ref.Context() after the loop.
// for now we ask pull scope for references of the same registry
if ml.Reference.Context().String() != repo.String() && ml.Reference.Context().Registry.String() == repo.Registry.String() {
scopeSet[ml.Reference.Scope(transport.PullScope)] = struct{}{}
}
}
}
scopes := make([]string, 0, len(scopeSet)+1)
// Push scope should be the first element because a few registries just look at the first scope to determine access.
scopes = append(scopes, repo.Scope(transport.PushScope))
for scope := range scopeSet {
scopes = append(scopes, scope)
}
return scopes
}
// WriteIndex pushes the provided ImageIndex to the specified image reference.
// WriteIndex will attempt to push all of the referenced manifests before
// attempting to push the ImageIndex, to retain referential integrity.
func WriteIndex(ref name.Reference, ii v1.ImageIndex, options ...Option) (rerr error) {
return Push(ref, ii, options...)
}
// WriteLayer uploads the provided Layer to the specified repo.
func WriteLayer(repo name.Repository, layer v1.Layer, options ...Option) (rerr error) {
o, err := makeOptions(options...)
if err != nil {
return err
}
if o.progress != nil {
defer func() { o.progress.Close(rerr) }()
}
return newPusher(o).Upload(o.context, repo, layer)
}
// Tag adds a tag to the given Taggable via PUT /v2/.../manifests/<tag>
//
// Notable implementations of Taggable are v1.Image, v1.ImageIndex, and
// remote.Descriptor.
//
// If t implements MediaType, we will use that for the Content-Type, otherwise
// we will default to types.DockerManifestSchema2.
//
// Tag does not attempt to write anything other than the manifest, so callers
// should ensure that all blobs or manifests that are referenced by t exist
// in the target registry.
func Tag(tag name.Tag, t Taggable, options ...Option) error {
return Put(tag, t, options...)
}
// Put adds a manifest from the given Taggable via PUT /v1/.../manifest/<ref>
//
// Notable implementations of Taggable are v1.Image, v1.ImageIndex, and
// remote.Descriptor.
//
// If t implements MediaType, we will use that for the Content-Type, otherwise
// we will default to types.DockerManifestSchema2.
//
// Put does not attempt to write anything other than the manifest, so callers
// should ensure that all blobs or manifests that are referenced by t exist
// in the target registry.
func Put(ref name.Reference, t Taggable, options ...Option) error {
o, err := makeOptions(options...)
if err != nil {
return err
}
return newPusher(o).Put(o.context, ref, t)
}
// Push uploads the given Taggable to the specified reference.
func Push(ref name.Reference, t Taggable, options ...Option) (rerr error) {
o, err := makeOptions(options...)
if err != nil {
return err
}
if o.progress != nil {
defer func() { o.progress.Close(rerr) }()
}
return newPusher(o).Push(o.context, ref, t)
}