working commit
This commit is contained in:
-202
@@ -1,202 +0,0 @@
|
||||
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
-756
@@ -1,756 +0,0 @@
|
||||
/*
|
||||
Copyright The containerd Authors.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
/*
|
||||
Copyright 2019 The Go Authors. All rights reserved.
|
||||
Use of this source code is governed by a BSD-style
|
||||
license that can be found in the LICENSE file.
|
||||
*/
|
||||
|
||||
package estargz
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path"
|
||||
"runtime"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
|
||||
"github.com/containerd/stargz-snapshotter/estargz/errorutil"
|
||||
"github.com/klauspost/compress/zstd"
|
||||
digest "github.com/opencontainers/go-digest"
|
||||
"golang.org/x/sync/errgroup"
|
||||
)
|
||||
|
||||
type GzipHelperFunc func(io.Reader) (io.ReadCloser, error)
|
||||
|
||||
type options struct {
|
||||
chunkSize int
|
||||
compressionLevel int
|
||||
prioritizedFiles []string
|
||||
missedPrioritizedFiles *[]string
|
||||
compression Compression
|
||||
ctx context.Context
|
||||
minChunkSize int
|
||||
gzipHelperFunc GzipHelperFunc
|
||||
}
|
||||
|
||||
type Option func(o *options) error
|
||||
|
||||
// WithChunkSize option specifies the chunk size of eStargz blob to build.
|
||||
func WithChunkSize(chunkSize int) Option {
|
||||
return func(o *options) error {
|
||||
o.chunkSize = chunkSize
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
// WithCompressionLevel option specifies the gzip compression level.
|
||||
// The default is gzip.BestCompression.
|
||||
// This option will be ignored if WithCompression option is used.
|
||||
// See also: https://godoc.org/compress/gzip#pkg-constants
|
||||
func WithCompressionLevel(level int) Option {
|
||||
return func(o *options) error {
|
||||
o.compressionLevel = level
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
// WithPrioritizedFiles option specifies the list of prioritized files.
|
||||
// These files must be complete paths that are absolute or relative to "/"
|
||||
// For example, all of "foo/bar", "/foo/bar", "./foo/bar" and "../foo/bar"
|
||||
// are treated as "/foo/bar".
|
||||
func WithPrioritizedFiles(files []string) Option {
|
||||
return func(o *options) error {
|
||||
o.prioritizedFiles = files
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
// WithAllowPrioritizeNotFound makes Build continue the execution even if some
|
||||
// of prioritized files specified by WithPrioritizedFiles option aren't found
|
||||
// in the input tar. Instead, this records all missed file names to the passed
|
||||
// slice.
|
||||
func WithAllowPrioritizeNotFound(missedFiles *[]string) Option {
|
||||
return func(o *options) error {
|
||||
if missedFiles == nil {
|
||||
return fmt.Errorf("WithAllowPrioritizeNotFound: slice must be passed")
|
||||
}
|
||||
o.missedPrioritizedFiles = missedFiles
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
// WithCompression specifies compression algorithm to be used.
|
||||
// Default is gzip.
|
||||
func WithCompression(compression Compression) Option {
|
||||
return func(o *options) error {
|
||||
o.compression = compression
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
// WithContext specifies a context that can be used for clean canceleration.
|
||||
func WithContext(ctx context.Context) Option {
|
||||
return func(o *options) error {
|
||||
o.ctx = ctx
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
// WithMinChunkSize option specifies the minimal number of bytes of data
|
||||
// must be written in one gzip stream.
|
||||
// By increasing this number, one gzip stream can contain multiple files
|
||||
// and it hopefully leads to smaller result blob.
|
||||
// NOTE: This adds a TOC property that old reader doesn't understand.
|
||||
func WithMinChunkSize(minChunkSize int) Option {
|
||||
return func(o *options) error {
|
||||
o.minChunkSize = minChunkSize
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
// WithGzipHelperFunc option specifies a custom function to decompress gzip-compressed layers.
|
||||
// When a gzip-compressed layer is detected, this function will be used instead of the
|
||||
// Go standard library gzip decompression for better performance.
|
||||
// The function should take an io.Reader as input and return an io.ReadCloser.
|
||||
// If nil, the Go standard library gzip.NewReader will be used.
|
||||
func WithGzipHelperFunc(gzipHelperFunc GzipHelperFunc) Option {
|
||||
return func(o *options) error {
|
||||
o.gzipHelperFunc = gzipHelperFunc
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
// Blob is an eStargz blob.
|
||||
type Blob struct {
|
||||
io.ReadCloser
|
||||
diffID digest.Digester
|
||||
tocDigest digest.Digest
|
||||
readCompleted *atomic.Bool
|
||||
uncompressedSize *atomic.Int64
|
||||
}
|
||||
|
||||
// DiffID returns the digest of uncompressed blob.
|
||||
// It is only valid to call DiffID after Close.
|
||||
func (b *Blob) DiffID() digest.Digest {
|
||||
return b.diffID.Digest()
|
||||
}
|
||||
|
||||
// TOCDigest returns the digest of uncompressed TOC JSON.
|
||||
func (b *Blob) TOCDigest() digest.Digest {
|
||||
return b.tocDigest
|
||||
}
|
||||
|
||||
// UncompressedSize returns the size of uncompressed blob.
|
||||
// UncompressedSize should only be called after the blob has been fully read.
|
||||
func (b *Blob) UncompressedSize() (int64, error) {
|
||||
switch {
|
||||
case b.uncompressedSize == nil || b.readCompleted == nil:
|
||||
return -1, fmt.Errorf("readCompleted or uncompressedSize is not initialized")
|
||||
case !b.readCompleted.Load():
|
||||
return -1, fmt.Errorf("called UncompressedSize before the blob has been fully read")
|
||||
default:
|
||||
return b.uncompressedSize.Load(), nil
|
||||
}
|
||||
}
|
||||
|
||||
// Build builds an eStargz blob which is an extended version of stargz, from a blob (gzip, zstd
|
||||
// or plain tar) passed through the argument. If there are some prioritized files are listed in
|
||||
// the option, these files are grouped as "prioritized" and can be used for runtime optimization
|
||||
// (e.g. prefetch). This function builds a blob in parallel, with dividing that blob into several
|
||||
// (at least the number of runtime.GOMAXPROCS(0)) sub-blobs.
|
||||
func Build(tarBlob *io.SectionReader, opt ...Option) (_ *Blob, rErr error) {
|
||||
var opts options
|
||||
opts.compressionLevel = gzip.BestCompression // BestCompression by default
|
||||
for _, o := range opt {
|
||||
if err := o(&opts); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
if opts.compression == nil {
|
||||
opts.compression = newGzipCompressionWithLevel(opts.compressionLevel)
|
||||
}
|
||||
layerFiles := newTempFiles()
|
||||
ctx := opts.ctx
|
||||
if ctx == nil {
|
||||
ctx = context.Background()
|
||||
}
|
||||
done := make(chan struct{})
|
||||
defer close(done)
|
||||
go func() {
|
||||
select {
|
||||
case <-done:
|
||||
// nop
|
||||
case <-ctx.Done():
|
||||
layerFiles.CleanupAll()
|
||||
}
|
||||
}()
|
||||
defer func() {
|
||||
if rErr != nil {
|
||||
if err := layerFiles.CleanupAll(); err != nil {
|
||||
rErr = fmt.Errorf("failed to cleanup tmp files: %v: %w", err, rErr)
|
||||
}
|
||||
}
|
||||
if cErr := ctx.Err(); cErr != nil {
|
||||
rErr = fmt.Errorf("error from context %q: %w", cErr, rErr)
|
||||
}
|
||||
}()
|
||||
tarBlob, err := decompressBlob(tarBlob, layerFiles, opts.gzipHelperFunc)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
entries, err := sortEntries(tarBlob, opts.prioritizedFiles, opts.missedPrioritizedFiles)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var tarParts [][]*entry
|
||||
if opts.minChunkSize > 0 {
|
||||
// Each entry needs to know the size of the current gzip stream so they
|
||||
// cannot be processed in parallel.
|
||||
tarParts = [][]*entry{entries}
|
||||
} else {
|
||||
tarParts = divideEntries(entries, runtime.GOMAXPROCS(0))
|
||||
}
|
||||
writers := make([]*Writer, len(tarParts))
|
||||
payloads := make([]*os.File, len(tarParts))
|
||||
var mu sync.Mutex
|
||||
var eg errgroup.Group
|
||||
for i, parts := range tarParts {
|
||||
i, parts := i, parts
|
||||
// builds verifiable stargz sub-blobs
|
||||
eg.Go(func() error {
|
||||
esgzFile, err := layerFiles.TempFile("", "esgzdata")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
sw := NewWriterWithCompressor(esgzFile, opts.compression)
|
||||
sw.ChunkSize = opts.chunkSize
|
||||
sw.MinChunkSize = opts.minChunkSize
|
||||
if sw.needsOpenGzEntries == nil {
|
||||
sw.needsOpenGzEntries = make(map[string]struct{})
|
||||
}
|
||||
for _, f := range []string{PrefetchLandmark, NoPrefetchLandmark} {
|
||||
sw.needsOpenGzEntries[f] = struct{}{}
|
||||
}
|
||||
if err := sw.AppendTar(readerFromEntries(parts...)); err != nil {
|
||||
return err
|
||||
}
|
||||
mu.Lock()
|
||||
writers[i] = sw
|
||||
payloads[i] = esgzFile
|
||||
mu.Unlock()
|
||||
return nil
|
||||
})
|
||||
}
|
||||
if err := eg.Wait(); err != nil {
|
||||
rErr = err
|
||||
return nil, err
|
||||
}
|
||||
tocAndFooter, tocDgst, err := closeWithCombine(writers...)
|
||||
if err != nil {
|
||||
rErr = err
|
||||
return nil, err
|
||||
}
|
||||
var rs []io.Reader
|
||||
for _, p := range payloads {
|
||||
fs, err := fileSectionReader(p)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
rs = append(rs, fs)
|
||||
}
|
||||
diffID := digest.Canonical.Digester()
|
||||
pr, pw := io.Pipe()
|
||||
readCompleted := new(atomic.Bool)
|
||||
uncompressedSize := new(atomic.Int64)
|
||||
go func() {
|
||||
var size int64
|
||||
var decompressFunc func(io.Reader) (io.ReadCloser, error)
|
||||
if _, ok := opts.compression.(*gzipCompression); ok && opts.gzipHelperFunc != nil {
|
||||
decompressFunc = opts.gzipHelperFunc
|
||||
} else {
|
||||
decompressFunc = opts.compression.Reader
|
||||
}
|
||||
decompressR, err := decompressFunc(io.TeeReader(io.MultiReader(append(rs, tocAndFooter)...), pw))
|
||||
if err != nil {
|
||||
pw.CloseWithError(err)
|
||||
return
|
||||
}
|
||||
defer decompressR.Close()
|
||||
if size, err = io.Copy(diffID.Hash(), decompressR); err != nil {
|
||||
pw.CloseWithError(err)
|
||||
return
|
||||
}
|
||||
uncompressedSize.Store(size)
|
||||
readCompleted.Store(true)
|
||||
pw.Close()
|
||||
}()
|
||||
return &Blob{
|
||||
ReadCloser: readCloser{
|
||||
Reader: pr,
|
||||
closeFunc: layerFiles.CleanupAll,
|
||||
},
|
||||
tocDigest: tocDgst,
|
||||
diffID: diffID,
|
||||
readCompleted: readCompleted,
|
||||
uncompressedSize: uncompressedSize,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// closeWithCombine takes unclosed Writers and close them. This also returns the
|
||||
// toc that combined all Writers into.
|
||||
// Writers doesn't write TOC and footer to the underlying writers so they can be
|
||||
// combined into a single eStargz and tocAndFooter returned by this function can
|
||||
// be appended at the tail of that combined blob.
|
||||
func closeWithCombine(ws ...*Writer) (tocAndFooterR io.Reader, tocDgst digest.Digest, err error) {
|
||||
if len(ws) == 0 {
|
||||
return nil, "", fmt.Errorf("at least one writer must be passed")
|
||||
}
|
||||
for _, w := range ws {
|
||||
if w.closed {
|
||||
return nil, "", fmt.Errorf("writer must be unclosed")
|
||||
}
|
||||
defer func(w *Writer) { w.closed = true }(w)
|
||||
if err := w.closeGz(); err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
if err := w.bw.Flush(); err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
}
|
||||
var (
|
||||
mtoc = new(JTOC)
|
||||
currentOffset int64
|
||||
)
|
||||
mtoc.Version = ws[0].toc.Version
|
||||
for _, w := range ws {
|
||||
for _, e := range w.toc.Entries {
|
||||
// Recalculate Offset of non-empty files/chunks
|
||||
if (e.Type == "reg" && e.Size > 0) || e.Type == "chunk" {
|
||||
e.Offset += currentOffset
|
||||
}
|
||||
mtoc.Entries = append(mtoc.Entries, e)
|
||||
}
|
||||
if w.toc.Version > mtoc.Version {
|
||||
mtoc.Version = w.toc.Version
|
||||
}
|
||||
currentOffset += w.cw.n
|
||||
}
|
||||
|
||||
return tocAndFooter(ws[0].compressor, mtoc, currentOffset)
|
||||
}
|
||||
|
||||
func tocAndFooter(compressor Compressor, toc *JTOC, offset int64) (io.Reader, digest.Digest, error) {
|
||||
buf := new(bytes.Buffer)
|
||||
tocDigest, err := compressor.WriteTOCAndFooter(buf, offset, toc, nil)
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
return buf, tocDigest, nil
|
||||
}
|
||||
|
||||
// divideEntries divides passed entries to the parts at least the number specified by the
|
||||
// argument.
|
||||
func divideEntries(entries []*entry, minPartsNum int) (set [][]*entry) {
|
||||
var estimatedSize int64
|
||||
for _, e := range entries {
|
||||
estimatedSize += e.header.Size
|
||||
}
|
||||
unitSize := estimatedSize / int64(minPartsNum)
|
||||
var (
|
||||
nextEnd = unitSize
|
||||
offset int64
|
||||
)
|
||||
set = append(set, []*entry{})
|
||||
for _, e := range entries {
|
||||
set[len(set)-1] = append(set[len(set)-1], e)
|
||||
offset += e.header.Size
|
||||
if offset > nextEnd {
|
||||
set = append(set, []*entry{})
|
||||
nextEnd += unitSize
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
var errNotFound = errors.New("not found")
|
||||
|
||||
// sortEntries reads the specified tar blob and returns a list of tar entries.
|
||||
// If some of prioritized files are specified, the list starts from these
|
||||
// files with keeping the order specified by the argument.
|
||||
func sortEntries(in io.ReaderAt, prioritized []string, missedPrioritized *[]string) ([]*entry, error) {
|
||||
|
||||
// Import tar file.
|
||||
intar, err := importTar(in)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to sort: %w", err)
|
||||
}
|
||||
|
||||
// Sort the tar file respecting to the prioritized files list.
|
||||
sorted := &tarFile{}
|
||||
picked := make(map[string]struct{})
|
||||
for _, l := range prioritized {
|
||||
if err := moveRec(l, intar, sorted, picked); err != nil {
|
||||
if errors.Is(err, errNotFound) && missedPrioritized != nil {
|
||||
*missedPrioritized = append(*missedPrioritized, l)
|
||||
continue // allow not found
|
||||
}
|
||||
return nil, fmt.Errorf("failed to sort tar entries: %w", err)
|
||||
}
|
||||
}
|
||||
if len(prioritized) == 0 {
|
||||
sorted.add(&entry{
|
||||
header: &tar.Header{
|
||||
Name: NoPrefetchLandmark,
|
||||
Typeflag: tar.TypeReg,
|
||||
Size: int64(len([]byte{landmarkContents})),
|
||||
},
|
||||
payload: bytes.NewReader([]byte{landmarkContents}),
|
||||
})
|
||||
} else {
|
||||
sorted.add(&entry{
|
||||
header: &tar.Header{
|
||||
Name: PrefetchLandmark,
|
||||
Typeflag: tar.TypeReg,
|
||||
Size: int64(len([]byte{landmarkContents})),
|
||||
},
|
||||
payload: bytes.NewReader([]byte{landmarkContents}),
|
||||
})
|
||||
}
|
||||
|
||||
// Dump prioritized entries followed by the rest entries while skipping picked ones.
|
||||
return append(sorted.dump(nil), intar.dump(picked)...), nil
|
||||
}
|
||||
|
||||
// readerFromEntries returns a reader of tar archive that contains entries passed
|
||||
// through the arguments.
|
||||
func readerFromEntries(entries ...*entry) io.Reader {
|
||||
pr, pw := io.Pipe()
|
||||
go func() {
|
||||
tw := tar.NewWriter(pw)
|
||||
defer tw.Close()
|
||||
for _, entry := range entries {
|
||||
if err := tw.WriteHeader(entry.header); err != nil {
|
||||
pw.CloseWithError(fmt.Errorf("failed to write tar header: %v", err))
|
||||
return
|
||||
}
|
||||
if _, err := io.Copy(tw, entry.payload); err != nil {
|
||||
pw.CloseWithError(fmt.Errorf("failed to write tar payload: %v", err))
|
||||
return
|
||||
}
|
||||
}
|
||||
pw.Close()
|
||||
}()
|
||||
return pr
|
||||
}
|
||||
|
||||
func importTar(in io.ReaderAt) (*tarFile, error) {
|
||||
tf := &tarFile{}
|
||||
pw, err := newCountReadSeeker(in)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to make position watcher: %w", err)
|
||||
}
|
||||
tr := tar.NewReader(pw)
|
||||
|
||||
// Walk through all nodes.
|
||||
for {
|
||||
// Fetch and parse next header.
|
||||
h, err := tr.Next()
|
||||
if err != nil {
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
return nil, fmt.Errorf("failed to parse tar file, %w", err)
|
||||
}
|
||||
switch cleanEntryName(h.Name) {
|
||||
case PrefetchLandmark, NoPrefetchLandmark:
|
||||
// Ignore existing landmark
|
||||
continue
|
||||
}
|
||||
|
||||
// Add entry. If it already exists, replace it.
|
||||
if _, ok := tf.get(h.Name); ok {
|
||||
tf.remove(h.Name)
|
||||
}
|
||||
tf.add(&entry{
|
||||
header: h,
|
||||
payload: io.NewSectionReader(in, pw.currentPos(), h.Size),
|
||||
})
|
||||
}
|
||||
|
||||
return tf, nil
|
||||
}
|
||||
|
||||
func moveRec(name string, in *tarFile, out *tarFile, picked map[string]struct{}) error {
|
||||
name = cleanEntryName(name)
|
||||
if name == "" { // root directory. stop recursion.
|
||||
if e, ok := in.get(name); ok {
|
||||
// entry of the root directory exists. we should move it as well.
|
||||
// this case will occur if tar entries are prefixed with "./", "/", etc.
|
||||
if _, done := picked[name]; !done {
|
||||
out.add(e)
|
||||
picked[name] = struct{}{}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
_, okIn := in.get(name)
|
||||
_, okOut := out.get(name)
|
||||
_, okPicked := picked[name]
|
||||
if !okIn && !okOut && !okPicked {
|
||||
return fmt.Errorf("file: %q: %w", name, errNotFound)
|
||||
}
|
||||
|
||||
parent, _ := path.Split(strings.TrimSuffix(name, "/"))
|
||||
if err := moveRec(parent, in, out, picked); err != nil {
|
||||
return err
|
||||
}
|
||||
if e, ok := in.get(name); ok && e.header.Typeflag == tar.TypeLink {
|
||||
if err := moveRec(e.header.Linkname, in, out, picked); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if _, done := picked[name]; done {
|
||||
return nil
|
||||
}
|
||||
if e, ok := in.get(name); ok {
|
||||
out.add(e)
|
||||
picked[name] = struct{}{}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type entry struct {
|
||||
header *tar.Header
|
||||
payload io.ReadSeeker
|
||||
}
|
||||
|
||||
type tarFile struct {
|
||||
index map[string]*entry
|
||||
stream []*entry
|
||||
}
|
||||
|
||||
func (f *tarFile) add(e *entry) {
|
||||
if f.index == nil {
|
||||
f.index = make(map[string]*entry)
|
||||
}
|
||||
f.index[cleanEntryName(e.header.Name)] = e
|
||||
f.stream = append(f.stream, e)
|
||||
}
|
||||
|
||||
func (f *tarFile) remove(name string) {
|
||||
name = cleanEntryName(name)
|
||||
if f.index != nil {
|
||||
delete(f.index, name)
|
||||
}
|
||||
var filtered []*entry
|
||||
for _, e := range f.stream {
|
||||
if cleanEntryName(e.header.Name) == name {
|
||||
continue
|
||||
}
|
||||
filtered = append(filtered, e)
|
||||
}
|
||||
f.stream = filtered
|
||||
}
|
||||
|
||||
func (f *tarFile) get(name string) (e *entry, ok bool) {
|
||||
if f.index == nil {
|
||||
return nil, false
|
||||
}
|
||||
e, ok = f.index[cleanEntryName(name)]
|
||||
return
|
||||
}
|
||||
|
||||
func (f *tarFile) dump(skip map[string]struct{}) []*entry {
|
||||
if len(skip) == 0 {
|
||||
return f.stream
|
||||
}
|
||||
var out []*entry
|
||||
for _, e := range f.stream {
|
||||
if _, ok := skip[cleanEntryName(e.header.Name)]; ok {
|
||||
continue
|
||||
}
|
||||
out = append(out, e)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
type readCloser struct {
|
||||
io.Reader
|
||||
closeFunc func() error
|
||||
}
|
||||
|
||||
func (rc readCloser) Close() error {
|
||||
return rc.closeFunc()
|
||||
}
|
||||
|
||||
func fileSectionReader(file *os.File) (*io.SectionReader, error) {
|
||||
info, err := file.Stat()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return io.NewSectionReader(file, 0, info.Size()), nil
|
||||
}
|
||||
|
||||
func newTempFiles() *tempFiles {
|
||||
return &tempFiles{}
|
||||
}
|
||||
|
||||
type tempFiles struct {
|
||||
files []*os.File
|
||||
filesMu sync.Mutex
|
||||
cleanupOnce sync.Once
|
||||
}
|
||||
|
||||
func (tf *tempFiles) TempFile(dir, pattern string) (*os.File, error) {
|
||||
f, err := os.CreateTemp(dir, pattern)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
tf.filesMu.Lock()
|
||||
tf.files = append(tf.files, f)
|
||||
tf.filesMu.Unlock()
|
||||
return f, nil
|
||||
}
|
||||
|
||||
func (tf *tempFiles) CleanupAll() (err error) {
|
||||
tf.cleanupOnce.Do(func() {
|
||||
err = tf.cleanupAll()
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
func (tf *tempFiles) cleanupAll() error {
|
||||
tf.filesMu.Lock()
|
||||
defer tf.filesMu.Unlock()
|
||||
var allErr []error
|
||||
for _, f := range tf.files {
|
||||
if err := f.Close(); err != nil {
|
||||
allErr = append(allErr, err)
|
||||
}
|
||||
if err := os.Remove(f.Name()); err != nil {
|
||||
allErr = append(allErr, err)
|
||||
}
|
||||
}
|
||||
tf.files = nil
|
||||
return errorutil.Aggregate(allErr)
|
||||
}
|
||||
|
||||
func newCountReadSeeker(r io.ReaderAt) (*countReadSeeker, error) {
|
||||
pos := int64(0)
|
||||
return &countReadSeeker{r: r, cPos: &pos}, nil
|
||||
}
|
||||
|
||||
type countReadSeeker struct {
|
||||
r io.ReaderAt
|
||||
cPos *int64
|
||||
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
func (cr *countReadSeeker) Read(p []byte) (int, error) {
|
||||
cr.mu.Lock()
|
||||
defer cr.mu.Unlock()
|
||||
|
||||
n, err := cr.r.ReadAt(p, *cr.cPos)
|
||||
if err == nil {
|
||||
*cr.cPos += int64(n)
|
||||
}
|
||||
return n, err
|
||||
}
|
||||
|
||||
func (cr *countReadSeeker) Seek(offset int64, whence int) (int64, error) {
|
||||
cr.mu.Lock()
|
||||
defer cr.mu.Unlock()
|
||||
|
||||
switch whence {
|
||||
default:
|
||||
return 0, fmt.Errorf("unknown whence: %v", whence)
|
||||
case io.SeekStart:
|
||||
case io.SeekCurrent:
|
||||
offset += *cr.cPos
|
||||
case io.SeekEnd:
|
||||
return 0, fmt.Errorf("unsupported whence: %v", whence)
|
||||
}
|
||||
|
||||
if offset < 0 {
|
||||
return 0, fmt.Errorf("invalid offset")
|
||||
}
|
||||
*cr.cPos = offset
|
||||
return offset, nil
|
||||
}
|
||||
|
||||
func (cr *countReadSeeker) currentPos() int64 {
|
||||
cr.mu.Lock()
|
||||
defer cr.mu.Unlock()
|
||||
|
||||
return *cr.cPos
|
||||
}
|
||||
|
||||
func decompressBlob(org *io.SectionReader, tmp *tempFiles, gzipHelperFunc GzipHelperFunc) (*io.SectionReader, error) {
|
||||
if org.Size() < 4 {
|
||||
return org, nil
|
||||
}
|
||||
src := make([]byte, 4)
|
||||
if _, err := org.Read(src); err != nil && err != io.EOF {
|
||||
return nil, err
|
||||
}
|
||||
var dR io.Reader
|
||||
if bytes.Equal([]byte{0x1F, 0x8B, 0x08}, src[:3]) {
|
||||
// gzip
|
||||
var dgR io.ReadCloser
|
||||
var err error
|
||||
if gzipHelperFunc != nil {
|
||||
dgR, err = gzipHelperFunc(io.NewSectionReader(org, 0, org.Size()))
|
||||
} else {
|
||||
dgR, err = gzip.NewReader(io.NewSectionReader(org, 0, org.Size()))
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer dgR.Close()
|
||||
dR = io.Reader(dgR)
|
||||
} else if bytes.Equal([]byte{0x28, 0xb5, 0x2f, 0xfd}, src[:4]) {
|
||||
// zstd
|
||||
dzR, err := zstd.NewReader(io.NewSectionReader(org, 0, org.Size()))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer dzR.Close()
|
||||
dR = io.Reader(dzR)
|
||||
} else {
|
||||
// uncompressed
|
||||
return io.NewSectionReader(org, 0, org.Size()), nil
|
||||
}
|
||||
b, err := tmp.TempFile("", "uncompresseddata")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if _, err := io.Copy(b, dR); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return fileSectionReader(b)
|
||||
}
|
||||
-40
@@ -1,40 +0,0 @@
|
||||
/*
|
||||
Copyright The containerd Authors.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package errorutil
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Aggregate combines a list of errors into a single new error.
|
||||
func Aggregate(errs []error) error {
|
||||
switch len(errs) {
|
||||
case 0:
|
||||
return nil
|
||||
case 1:
|
||||
return errs[0]
|
||||
default:
|
||||
points := make([]string, len(errs)+1)
|
||||
points[0] = fmt.Sprintf("%d error(s) occurred:", len(errs))
|
||||
for i, err := range errs {
|
||||
points[i+1] = fmt.Sprintf("* %s", err)
|
||||
}
|
||||
return errors.New(strings.Join(points, "\n\t"))
|
||||
}
|
||||
}
|
||||
-1232
File diff suppressed because it is too large
Load Diff
-237
@@ -1,237 +0,0 @@
|
||||
/*
|
||||
Copyright The containerd Authors.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
/*
|
||||
Copyright 2019 The Go Authors. All rights reserved.
|
||||
Use of this source code is governed by a BSD-style
|
||||
license that can be found in the LICENSE file.
|
||||
*/
|
||||
|
||||
package estargz
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"hash"
|
||||
"io"
|
||||
"strconv"
|
||||
|
||||
digest "github.com/opencontainers/go-digest"
|
||||
)
|
||||
|
||||
type gzipCompression struct {
|
||||
*GzipCompressor
|
||||
*GzipDecompressor
|
||||
}
|
||||
|
||||
func newGzipCompressionWithLevel(level int) Compression {
|
||||
return &gzipCompression{
|
||||
&GzipCompressor{level},
|
||||
&GzipDecompressor{},
|
||||
}
|
||||
}
|
||||
|
||||
func NewGzipCompressor() *GzipCompressor {
|
||||
return &GzipCompressor{gzip.BestCompression}
|
||||
}
|
||||
|
||||
func NewGzipCompressorWithLevel(level int) *GzipCompressor {
|
||||
return &GzipCompressor{level}
|
||||
}
|
||||
|
||||
type GzipCompressor struct {
|
||||
compressionLevel int
|
||||
}
|
||||
|
||||
func (gc *GzipCompressor) Writer(w io.Writer) (WriteFlushCloser, error) {
|
||||
return gzip.NewWriterLevel(w, gc.compressionLevel)
|
||||
}
|
||||
|
||||
func (gc *GzipCompressor) WriteTOCAndFooter(w io.Writer, off int64, toc *JTOC, diffHash hash.Hash) (digest.Digest, error) {
|
||||
tocJSON, err := json.MarshalIndent(toc, "", "\t")
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
gz, _ := gzip.NewWriterLevel(w, gc.compressionLevel)
|
||||
gw := io.Writer(gz)
|
||||
if diffHash != nil {
|
||||
gw = io.MultiWriter(gz, diffHash)
|
||||
}
|
||||
tw := tar.NewWriter(gw)
|
||||
if err := tw.WriteHeader(&tar.Header{
|
||||
Typeflag: tar.TypeReg,
|
||||
Name: TOCTarName,
|
||||
Size: int64(len(tocJSON)),
|
||||
}); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if _, err := tw.Write(tocJSON); err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
if err := tw.Close(); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := gz.Close(); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if _, err := w.Write(gzipFooterBytes(off)); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return digest.FromBytes(tocJSON), nil
|
||||
}
|
||||
|
||||
// gzipFooterBytes returns the 51 bytes footer.
|
||||
func gzipFooterBytes(tocOff int64) []byte {
|
||||
buf := bytes.NewBuffer(make([]byte, 0, FooterSize))
|
||||
gz, _ := gzip.NewWriterLevel(buf, gzip.NoCompression) // MUST be NoCompression to keep 51 bytes
|
||||
|
||||
// Extra header indicating the offset of TOCJSON
|
||||
// https://tools.ietf.org/html/rfc1952#section-2.3.1.1
|
||||
header := make([]byte, 4)
|
||||
header[0], header[1] = 'S', 'G'
|
||||
subfield := fmt.Sprintf("%016xSTARGZ", tocOff)
|
||||
binary.LittleEndian.PutUint16(header[2:4], uint16(len(subfield))) // little-endian per RFC1952
|
||||
gz.Extra = append(header, []byte(subfield)...)
|
||||
gz.Close()
|
||||
if buf.Len() != FooterSize {
|
||||
panic(fmt.Sprintf("footer buffer = %d, not %d", buf.Len(), FooterSize))
|
||||
}
|
||||
return buf.Bytes()
|
||||
}
|
||||
|
||||
type GzipDecompressor struct{}
|
||||
|
||||
func (gz *GzipDecompressor) Reader(r io.Reader) (io.ReadCloser, error) {
|
||||
return gzip.NewReader(r)
|
||||
}
|
||||
|
||||
func (gz *GzipDecompressor) ParseTOC(r io.Reader) (toc *JTOC, tocDgst digest.Digest, err error) {
|
||||
return parseTOCEStargz(r)
|
||||
}
|
||||
|
||||
func (gz *GzipDecompressor) ParseFooter(p []byte) (blobPayloadSize, tocOffset, tocSize int64, err error) {
|
||||
if len(p) != FooterSize {
|
||||
return 0, 0, 0, fmt.Errorf("invalid length %d cannot be parsed", len(p))
|
||||
}
|
||||
zr, err := gzip.NewReader(bytes.NewReader(p))
|
||||
if err != nil {
|
||||
return 0, 0, 0, err
|
||||
}
|
||||
defer zr.Close()
|
||||
extra := zr.Extra
|
||||
si1, si2, subfieldlen, subfield := extra[0], extra[1], extra[2:4], extra[4:]
|
||||
if si1 != 'S' || si2 != 'G' {
|
||||
return 0, 0, 0, fmt.Errorf("invalid subfield IDs: %q, %q; want E, S", si1, si2)
|
||||
}
|
||||
if slen := binary.LittleEndian.Uint16(subfieldlen); slen != uint16(16+len("STARGZ")) {
|
||||
return 0, 0, 0, fmt.Errorf("invalid length of subfield %d; want %d", slen, 16+len("STARGZ"))
|
||||
}
|
||||
if string(subfield[16:]) != "STARGZ" {
|
||||
return 0, 0, 0, fmt.Errorf("STARGZ magic string must be included in the footer subfield")
|
||||
}
|
||||
tocOffset, err = strconv.ParseInt(string(subfield[:16]), 16, 64)
|
||||
if err != nil {
|
||||
return 0, 0, 0, fmt.Errorf("legacy: failed to parse toc offset: %w", err)
|
||||
}
|
||||
return tocOffset, tocOffset, 0, nil
|
||||
}
|
||||
|
||||
func (gz *GzipDecompressor) FooterSize() int64 {
|
||||
return FooterSize
|
||||
}
|
||||
|
||||
func (gz *GzipDecompressor) DecompressTOC(r io.Reader) (tocJSON io.ReadCloser, err error) {
|
||||
return decompressTOCEStargz(r)
|
||||
}
|
||||
|
||||
type LegacyGzipDecompressor struct{}
|
||||
|
||||
func (gz *LegacyGzipDecompressor) Reader(r io.Reader) (io.ReadCloser, error) {
|
||||
return gzip.NewReader(r)
|
||||
}
|
||||
|
||||
func (gz *LegacyGzipDecompressor) ParseTOC(r io.Reader) (toc *JTOC, tocDgst digest.Digest, err error) {
|
||||
return parseTOCEStargz(r)
|
||||
}
|
||||
|
||||
func (gz *LegacyGzipDecompressor) ParseFooter(p []byte) (blobPayloadSize, tocOffset, tocSize int64, err error) {
|
||||
if len(p) != legacyFooterSize {
|
||||
return 0, 0, 0, fmt.Errorf("legacy: invalid length %d cannot be parsed", len(p))
|
||||
}
|
||||
zr, err := gzip.NewReader(bytes.NewReader(p))
|
||||
if err != nil {
|
||||
return 0, 0, 0, fmt.Errorf("legacy: failed to get footer gzip reader: %w", err)
|
||||
}
|
||||
defer zr.Close()
|
||||
extra := zr.Extra
|
||||
if len(extra) != 16+len("STARGZ") {
|
||||
return 0, 0, 0, fmt.Errorf("legacy: invalid stargz's extra field size")
|
||||
}
|
||||
if string(extra[16:]) != "STARGZ" {
|
||||
return 0, 0, 0, fmt.Errorf("legacy: magic string STARGZ not found")
|
||||
}
|
||||
tocOffset, err = strconv.ParseInt(string(extra[:16]), 16, 64)
|
||||
if err != nil {
|
||||
return 0, 0, 0, fmt.Errorf("legacy: failed to parse toc offset: %w", err)
|
||||
}
|
||||
return tocOffset, tocOffset, 0, nil
|
||||
}
|
||||
|
||||
func (gz *LegacyGzipDecompressor) FooterSize() int64 {
|
||||
return legacyFooterSize
|
||||
}
|
||||
|
||||
func (gz *LegacyGzipDecompressor) DecompressTOC(r io.Reader) (tocJSON io.ReadCloser, err error) {
|
||||
return decompressTOCEStargz(r)
|
||||
}
|
||||
|
||||
func parseTOCEStargz(r io.Reader) (toc *JTOC, tocDgst digest.Digest, err error) {
|
||||
tr, err := decompressTOCEStargz(r)
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
dgstr := digest.Canonical.Digester()
|
||||
toc = new(JTOC)
|
||||
if err := json.NewDecoder(io.TeeReader(tr, dgstr.Hash())).Decode(&toc); err != nil {
|
||||
return nil, "", fmt.Errorf("error decoding TOC JSON: %v", err)
|
||||
}
|
||||
if err := tr.Close(); err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
return toc, dgstr.Digest(), nil
|
||||
}
|
||||
|
||||
func decompressTOCEStargz(r io.Reader) (tocJSON io.ReadCloser, err error) {
|
||||
zr, err := gzip.NewReader(r)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("malformed TOC gzip header: %v", err)
|
||||
}
|
||||
zr.Multistream(false)
|
||||
tr := tar.NewReader(zr)
|
||||
h, err := tr.Next()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to find tar header in TOC gzip stream: %v", err)
|
||||
}
|
||||
if h.Name != TOCTarName {
|
||||
return nil, fmt.Errorf("TOC tar entry had name %q; expected %q", h.Name, TOCTarName)
|
||||
}
|
||||
return readCloser{tr, zr.Close}, nil
|
||||
}
|
||||
-2403
File diff suppressed because it is too large
Load Diff
-342
@@ -1,342 +0,0 @@
|
||||
/*
|
||||
Copyright The containerd Authors.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
/*
|
||||
Copyright 2019 The Go Authors. All rights reserved.
|
||||
Use of this source code is governed by a BSD-style
|
||||
license that can be found in the LICENSE file.
|
||||
*/
|
||||
|
||||
package estargz
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"hash"
|
||||
"io"
|
||||
"os"
|
||||
"path"
|
||||
"time"
|
||||
|
||||
digest "github.com/opencontainers/go-digest"
|
||||
)
|
||||
|
||||
const (
|
||||
// TOCTarName is the name of the JSON file in the tar archive in the
|
||||
// table of contents gzip stream.
|
||||
TOCTarName = "stargz.index.json"
|
||||
|
||||
// FooterSize is the number of bytes in the footer
|
||||
//
|
||||
// The footer is an empty gzip stream with no compression and an Extra
|
||||
// header of the form "%016xSTARGZ", where the 64 bit hex-encoded
|
||||
// number is the offset to the gzip stream of JSON TOC.
|
||||
//
|
||||
// 51 comes from:
|
||||
//
|
||||
// 10 bytes gzip header
|
||||
// 2 bytes XLEN (length of Extra field) = 26 (4 bytes header + 16 hex digits + len("STARGZ"))
|
||||
// 2 bytes Extra: SI1 = 'S', SI2 = 'G'
|
||||
// 2 bytes Extra: LEN = 22 (16 hex digits + len("STARGZ"))
|
||||
// 22 bytes Extra: subfield = fmt.Sprintf("%016xSTARGZ", offsetOfTOC)
|
||||
// 5 bytes flate header
|
||||
// 8 bytes gzip footer
|
||||
// (End of the eStargz blob)
|
||||
//
|
||||
// NOTE: For Extra fields, subfield IDs SI1='S' SI2='G' is used for eStargz.
|
||||
FooterSize = 51
|
||||
|
||||
// legacyFooterSize is the number of bytes in the legacy stargz footer.
|
||||
//
|
||||
// 47 comes from:
|
||||
//
|
||||
// 10 byte gzip header +
|
||||
// 2 byte (LE16) length of extra, encoding 22 (16 hex digits + len("STARGZ")) == "\x16\x00" +
|
||||
// 22 bytes of extra (fmt.Sprintf("%016xSTARGZ", tocGzipOffset))
|
||||
// 5 byte flate header
|
||||
// 8 byte gzip footer (two little endian uint32s: digest, size)
|
||||
legacyFooterSize = 47
|
||||
|
||||
// TOCJSONDigestAnnotation is an annotation for an image layer. This stores the
|
||||
// digest of the TOC JSON.
|
||||
// This annotation is valid only when it is specified in `.[]layers.annotations`
|
||||
// of an image manifest.
|
||||
TOCJSONDigestAnnotation = "containerd.io/snapshot/stargz/toc.digest"
|
||||
|
||||
// StoreUncompressedSizeAnnotation is an additional annotation key for eStargz to enable lazy
|
||||
// pulling on containers/storage. Stargz Store is required to expose the layer's uncompressed size
|
||||
// to the runtime but current OCI image doesn't ship this information by default. So we store this
|
||||
// to the special annotation.
|
||||
StoreUncompressedSizeAnnotation = "io.containers.estargz.uncompressed-size"
|
||||
|
||||
// PrefetchLandmark is a file entry which indicates the end position of
|
||||
// prefetch in the stargz file.
|
||||
PrefetchLandmark = ".prefetch.landmark"
|
||||
|
||||
// NoPrefetchLandmark is a file entry which indicates that no prefetch should
|
||||
// occur in the stargz file.
|
||||
NoPrefetchLandmark = ".no.prefetch.landmark"
|
||||
|
||||
landmarkContents = 0xf
|
||||
)
|
||||
|
||||
// JTOC is the JSON-serialized table of contents index of the files in the stargz file.
|
||||
type JTOC struct {
|
||||
Version int `json:"version"`
|
||||
Entries []*TOCEntry `json:"entries"`
|
||||
}
|
||||
|
||||
// TOCEntry is an entry in the stargz file's TOC (Table of Contents).
|
||||
type TOCEntry struct {
|
||||
// Name is the tar entry's name. It is the complete path
|
||||
// stored in the tar file, not just the base name.
|
||||
Name string `json:"name"`
|
||||
|
||||
// Type is one of "dir", "reg", "symlink", "hardlink", "char",
|
||||
// "block", "fifo", or "chunk".
|
||||
// The "chunk" type is used for regular file data chunks past the first
|
||||
// TOCEntry; the 2nd chunk and on have only Type ("chunk"), Offset,
|
||||
// ChunkOffset, and ChunkSize populated.
|
||||
Type string `json:"type"`
|
||||
|
||||
// Size, for regular files, is the logical size of the file.
|
||||
Size int64 `json:"size,omitempty"`
|
||||
|
||||
// ModTime3339 is the modification time of the tar entry. Empty
|
||||
// means zero or unknown. Otherwise it's in UTC RFC3339
|
||||
// format. Use the ModTime method to access the time.Time value.
|
||||
ModTime3339 string `json:"modtime,omitempty"`
|
||||
modTime time.Time
|
||||
|
||||
// LinkName, for symlinks and hardlinks, is the link target.
|
||||
LinkName string `json:"linkName,omitempty"`
|
||||
|
||||
// Mode is the permission and mode bits.
|
||||
Mode int64 `json:"mode,omitempty"`
|
||||
|
||||
// UID is the user ID of the owner.
|
||||
UID int `json:"uid,omitempty"`
|
||||
|
||||
// GID is the group ID of the owner.
|
||||
GID int `json:"gid,omitempty"`
|
||||
|
||||
// Uname is the username of the owner.
|
||||
//
|
||||
// In the serialized JSON, this field may only be present for
|
||||
// the first entry with the same UID.
|
||||
Uname string `json:"userName,omitempty"`
|
||||
|
||||
// Gname is the group name of the owner.
|
||||
//
|
||||
// In the serialized JSON, this field may only be present for
|
||||
// the first entry with the same GID.
|
||||
Gname string `json:"groupName,omitempty"`
|
||||
|
||||
// Offset, for regular files, provides the offset in the
|
||||
// stargz file to the file's data bytes. See ChunkOffset and
|
||||
// ChunkSize.
|
||||
Offset int64 `json:"offset,omitempty"`
|
||||
|
||||
// InnerOffset is an optional field indicates uncompressed offset
|
||||
// of this "reg" or "chunk" payload in a stream starts from Offset.
|
||||
// This field enables to put multiple "reg" or "chunk" payloads
|
||||
// in one chunk with having the same Offset but different InnerOffset.
|
||||
InnerOffset int64 `json:"innerOffset,omitempty"`
|
||||
|
||||
nextOffset int64 // the Offset of the next entry with a non-zero Offset
|
||||
|
||||
// DevMajor is the major device number for "char" and "block" types.
|
||||
DevMajor int `json:"devMajor,omitempty"`
|
||||
|
||||
// DevMinor is the major device number for "char" and "block" types.
|
||||
DevMinor int `json:"devMinor,omitempty"`
|
||||
|
||||
// NumLink is the number of entry names pointing to this entry.
|
||||
// Zero means one name references this entry.
|
||||
// This field is calculated during runtime and not recorded in TOC JSON.
|
||||
NumLink int `json:"-"`
|
||||
|
||||
// Xattrs are the extended attribute for the entry.
|
||||
Xattrs map[string][]byte `json:"xattrs,omitempty"`
|
||||
|
||||
// Digest stores the OCI checksum for regular files payload.
|
||||
// It has the form "sha256:abcdef01234....".
|
||||
Digest string `json:"digest,omitempty"`
|
||||
|
||||
// ChunkOffset is non-zero if this is a chunk of a large,
|
||||
// regular file. If so, the Offset is where the gzip header of
|
||||
// ChunkSize bytes at ChunkOffset in Name begin.
|
||||
//
|
||||
// In serialized form, a "chunkSize" JSON field of zero means
|
||||
// that the chunk goes to the end of the file. After reading
|
||||
// from the stargz TOC, though, the ChunkSize is initialized
|
||||
// to a non-zero file for when Type is either "reg" or
|
||||
// "chunk".
|
||||
ChunkOffset int64 `json:"chunkOffset,omitempty"`
|
||||
ChunkSize int64 `json:"chunkSize,omitempty"`
|
||||
|
||||
// ChunkDigest stores an OCI digest of the chunk. This must be formed
|
||||
// as "sha256:0123abcd...".
|
||||
ChunkDigest string `json:"chunkDigest,omitempty"`
|
||||
|
||||
children map[string]*TOCEntry
|
||||
|
||||
// chunkTopIndex is index of the entry where Offset starts in the blob.
|
||||
chunkTopIndex int
|
||||
}
|
||||
|
||||
// ModTime returns the entry's modification time.
|
||||
func (e *TOCEntry) ModTime() time.Time { return e.modTime }
|
||||
|
||||
// NextOffset returns the position (relative to the start of the
|
||||
// stargz file) of the next gzip boundary after e.Offset.
|
||||
func (e *TOCEntry) NextOffset() int64 { return e.nextOffset }
|
||||
|
||||
func (e *TOCEntry) addChild(baseName string, child *TOCEntry) {
|
||||
if e.children == nil {
|
||||
e.children = make(map[string]*TOCEntry)
|
||||
}
|
||||
if child.Type == "dir" {
|
||||
e.NumLink++ // Entry ".." in the subdirectory links to this directory
|
||||
}
|
||||
e.children[baseName] = child
|
||||
}
|
||||
|
||||
// isDataType reports whether TOCEntry is a regular file or chunk (something that
|
||||
// contains regular file data).
|
||||
func (e *TOCEntry) isDataType() bool { return e.Type == "reg" || e.Type == "chunk" }
|
||||
|
||||
// Stat returns a FileInfo value representing e.
|
||||
func (e *TOCEntry) Stat() os.FileInfo { return fileInfo{e} }
|
||||
|
||||
// ForeachChild calls f for each child item. If f returns false, iteration ends.
|
||||
// If e is not a directory, f is not called.
|
||||
func (e *TOCEntry) ForeachChild(f func(baseName string, ent *TOCEntry) bool) {
|
||||
for name, ent := range e.children {
|
||||
if !f(name, ent) {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// LookupChild returns the directory e's child by its base name.
|
||||
func (e *TOCEntry) LookupChild(baseName string) (child *TOCEntry, ok bool) {
|
||||
child, ok = e.children[baseName]
|
||||
return
|
||||
}
|
||||
|
||||
// fileInfo implements os.FileInfo using the wrapped *TOCEntry.
|
||||
type fileInfo struct{ e *TOCEntry }
|
||||
|
||||
var _ os.FileInfo = fileInfo{}
|
||||
|
||||
func (fi fileInfo) Name() string { return path.Base(fi.e.Name) }
|
||||
func (fi fileInfo) IsDir() bool { return fi.e.Type == "dir" }
|
||||
func (fi fileInfo) Size() int64 { return fi.e.Size }
|
||||
func (fi fileInfo) ModTime() time.Time { return fi.e.ModTime() }
|
||||
func (fi fileInfo) Sys() interface{} { return fi.e }
|
||||
func (fi fileInfo) Mode() (m os.FileMode) {
|
||||
// TOCEntry.Mode is tar.Header.Mode so we can understand the these bits using `tar` pkg.
|
||||
m = (&tar.Header{Mode: fi.e.Mode}).FileInfo().Mode() &
|
||||
(os.ModePerm | os.ModeSetuid | os.ModeSetgid | os.ModeSticky)
|
||||
switch fi.e.Type {
|
||||
case "dir":
|
||||
m |= os.ModeDir
|
||||
case "symlink":
|
||||
m |= os.ModeSymlink
|
||||
case "char":
|
||||
m |= os.ModeDevice | os.ModeCharDevice
|
||||
case "block":
|
||||
m |= os.ModeDevice
|
||||
case "fifo":
|
||||
m |= os.ModeNamedPipe
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// TOCEntryVerifier holds verifiers that are usable for verifying chunks contained
|
||||
// in a eStargz blob.
|
||||
type TOCEntryVerifier interface {
|
||||
|
||||
// Verifier provides a content verifier that can be used for verifying the
|
||||
// contents of the specified TOCEntry.
|
||||
Verifier(ce *TOCEntry) (digest.Verifier, error)
|
||||
}
|
||||
|
||||
// Compression provides the compression helper to be used creating and parsing eStargz.
|
||||
// This package provides gzip-based Compression by default, but any compression
|
||||
// algorithm (e.g. zstd) can be used as long as it implements Compression.
|
||||
type Compression interface {
|
||||
Compressor
|
||||
Decompressor
|
||||
}
|
||||
|
||||
// Compressor represents the helper mothods to be used for creating eStargz.
|
||||
type Compressor interface {
|
||||
// Writer returns WriteCloser to be used for writing a chunk to eStargz.
|
||||
// Everytime a chunk is written, the WriteCloser is closed and Writer is
|
||||
// called again for writing the next chunk.
|
||||
//
|
||||
// The returned writer should implement "Flush() error" function that flushes
|
||||
// any pending compressed data to the underlying writer.
|
||||
Writer(w io.Writer) (WriteFlushCloser, error)
|
||||
|
||||
// WriteTOCAndFooter is called to write JTOC to the passed Writer.
|
||||
// diffHash calculates the DiffID (uncompressed sha256 hash) of the blob
|
||||
// WriteTOCAndFooter can optionally write anything that affects DiffID calculation
|
||||
// (e.g. uncompressed TOC JSON).
|
||||
//
|
||||
// This function returns tocDgst that represents the digest of TOC that will be used
|
||||
// to verify this blob when it's parsed.
|
||||
WriteTOCAndFooter(w io.Writer, off int64, toc *JTOC, diffHash hash.Hash) (tocDgst digest.Digest, err error)
|
||||
}
|
||||
|
||||
// Decompressor represents the helper mothods to be used for parsing eStargz.
|
||||
type Decompressor interface {
|
||||
// Reader returns ReadCloser to be used for decompressing file payload.
|
||||
Reader(r io.Reader) (io.ReadCloser, error)
|
||||
|
||||
// FooterSize returns the size of the footer of this blob.
|
||||
FooterSize() int64
|
||||
|
||||
// ParseFooter parses the footer and returns the offset and (compressed) size of TOC.
|
||||
// payloadBlobSize is the (compressed) size of the blob payload (i.e. the size between
|
||||
// the top until the TOC JSON).
|
||||
//
|
||||
// If tocOffset < 0, we assume that TOC isn't contained in the blob and pass nil reader
|
||||
// to ParseTOC. We expect that ParseTOC acquire TOC from the external location and return it.
|
||||
//
|
||||
// tocSize is optional. If tocSize <= 0, it's by default the size of the range from tocOffset until the beginning of the
|
||||
// footer (blob size - tocOff - FooterSize).
|
||||
// If blobPayloadSize < 0, blobPayloadSize become the blob size.
|
||||
ParseFooter(p []byte) (blobPayloadSize, tocOffset, tocSize int64, err error)
|
||||
|
||||
// ParseTOC parses TOC from the passed reader. The reader provides the partial contents
|
||||
// of the underlying blob that has the range specified by ParseFooter method.
|
||||
//
|
||||
// This function returns tocDgst that represents the digest of TOC that will be used
|
||||
// to verify this blob. This must match to the value returned from
|
||||
// Compressor.WriteTOCAndFooter that is used when creating this blob.
|
||||
//
|
||||
// If tocOffset returned by ParseFooter is < 0, we assume that TOC isn't contained in the blob.
|
||||
// Pass nil reader to ParseTOC then we expect that ParseTOC acquire TOC from the external location
|
||||
// and return it.
|
||||
ParseTOC(r io.Reader) (toc *JTOC, tocDgst digest.Digest, err error)
|
||||
}
|
||||
|
||||
type WriteFlushCloser interface {
|
||||
io.WriteCloser
|
||||
Flush() error
|
||||
}
|
||||
Reference in New Issue
Block a user