docker-api-ng 0.3.0

data/docs/connecting.md

@@ -0,0 +1,111 @@
+# Connecting
+
+## Resolution order
+
+`Docker::API::Client.new` with no arguments resolves its daemon the way the
+`docker` CLI does:
+
+1. `DOCKER_HOST`, or `DOCKER_URL`
+2. the active Docker context — `DOCKER_CONTEXT`, then `currentContext` from
+   `~/.docker/config.json`
+3. `unix:///var/run/docker.sock` on Linux and macOS
+4. `npipe:////./pipe/docker_engine` on Windows
+
+`docker context use` exports nothing; it writes `currentContext` into
+`config.json`. Reading only `DOCKER_HOST` therefore disagrees with the CLI on
+the same machine, and falls back to `/var/run/docker.sock` — which Docker
+Desktop symlinks but Colima, Rancher Desktop, rootless Docker and Podman
+generally do not. Set `DOCKER_CONFIG` to point the lookup somewhere else.
+
+A context may carry its own TLS material, which is picked up with it. An
+unreadable store, a malformed `meta.json` or a context that no longer exists
+all mean "no context" rather than an error, because falling back to the
+platform default is what the CLI does.
+
+TLS material comes from `DOCKER_CERT_PATH`, which is expected to contain
+`ca.pem`, `cert.pem` and `key.pem`. `DOCKER_TLS_VERIFY` is a presence flag
+rather than a boolean: the CLI treats an exported-but-empty value as off, and
+so does this gem.
+
+Explicit arguments always win over the environment.
+
+## URL forms
+
+| Form | Transport |
+| --- | --- |
+| `unix:///var/run/docker.sock` | `Transport::Unix` |
+| `/var/run/docker.sock` | `Transport::Unix` — a bare path is a socket |
+| `tcp://host:2375` | `Transport::Tcp` |
+| `tcp://host:2376` with `tls:` | `Transport::Tls` |
+| `https://host:2376` | `Transport::Tls` |
+| `npipe:////./pipe/docker_engine` | `Transport::NamedPipe` |
+| `tcp://` | `Transport::Tcp` at localhost:2375 |
+
+## Version negotiation
+
+Docker versions its API by URL prefix: `/v1.55/containers/json`. On first use a
+client pings `/_ping`, reads the `Api-Version` header, and settles on
+`min(what this gem vendors, what the daemon speaks)`.
+
+```ruby
+client = Docker::API::Client.new                          # negotiate (default)
+client = Docker::API::Client.new(api_version: "1.44")     # pin; no ping is sent
+client = Docker::API::Client.new(api_version: :none)      # send unprefixed paths
+```
+
+Negotiation happens once per client, not once per request. A daemon older than
+`Docker::API::MIN_API_VERSION` raises `VersionUnsupported` naming both
+versions, at connection time rather than as a confusing 404 later.
+
+Pinning is worth doing in CI, where an unexpected daemon upgrade changing
+behaviour underneath a test suite is more disruptive than a version mismatch
+you can see in the diff.
+
+## Timeouts
+
+```ruby
+client = Docker::API::Client.new(read_timeout: 120, open_timeout: 5)
+```
+
+`read_timeout` is the wait for response data, not a deadline for the whole
+request. Long-running streams — `logs(follow: true)`, `system.events`, a slow
+build — are not subject to a total limit, only to a gap between chunks.
+
+## Logging
+
+```ruby
+require "logger"
+client = Docker::API::Client.new(logger: Logger.new($stdout))
+```
+
+One debug line per request, naming the operation, the verb and the resolved
+path. Request bodies are not logged: they routinely carry registry credentials
+and environment variables.
+
+## TLS
+
+```ruby
+client = Docker::API::Client.new(
+  url: "tcp://build.internal:2376",
+  tls: {
+    ca_file: "/certs/ca.pem",
+    cert_file: "/certs/cert.pem",
+    key_file: "/certs/key.pem",
+    verify: true,
+  }
+)
+```
+
+The handshake happens in the transport rather than in `Net::HTTP`, so the layer
+above receives an already-encrypted socket and there is one code path for every
+transport. TLS 1.2 is the minimum. `verify: false` exists for self-signed
+development daemons; it is not a default.
+
+## Windows named pipes
+
+```ruby
+client = Docker::API::Client.new(url: "npipe:////./pipe/docker_engine")
+```
+
+Both the forward-slash form that appears in `DOCKER_HOST` and the backslash
+form Windows itself uses are accepted.

data/docs/errors.md

@@ -0,0 +1,89 @@
+# Errors
+
+Everything this gem raises descends from `Docker::API::Error`. Nothing from
+beneath the abstraction escapes: no `Errno`, no `SocketError`, no `OpenSSL`
+exception and no `Net::` class reaches your rescue clause. The original is
+always retained as `#cause`, so nothing is lost.
+
+That is a deliberate difference from the `docker-api` gem, where
+`Excon::Error::Socket` reaches callers and couples them to an HTTP library they
+never chose.
+
+## The hierarchy
+
+```
+Docker::API::Error
+├── ConnectionError      the daemon could not be reached at all
+├── TimeoutError         it accepted the connection but did not answer in time
+├── ClientError          4xx
+│   ├── BadRequest       400
+│   ├── Unauthorized     401
+│   ├── Forbidden        403
+│   ├── NotFound         404
+│   ├── NotModified      304
+│   └── Conflict         409
+├── ServerError          5xx
+├── VersionUnsupported   the daemon speaks an API version this gem cannot use
+└── StreamError          a response stream was malformed or truncated
+```
+
+`NotModified` is grouped with the client errors rather than treated as a
+redirect because that is the meaning Docker gives it: starting an
+already-running container returns 304.
+
+## What an error carries
+
+```ruby
+begin
+  client.containers.get("missing")
+rescue Docker::API::Error => e
+  e.operation   #=> "container_inspect"
+  e.status      #=> 404
+  e.response    #=> #<Docker::API::Response status=404 bytes=42>
+  e.message     #=> "container_inspect failed (HTTP 404): No such container: missing"
+  e.cause       #=> the underlying exception, when there was one
+end
+```
+
+The message is composed from what was attempted, the status, and the daemon's
+own `message` field. When the body is not JSON the raw body is used instead,
+because an unparsed body still helps a human more than nothing does.
+
+## Rescuing well
+
+Rescue the narrowest thing that describes what you are handling:
+
+```ruby
+# Absence is a normal answer to a question.
+container = client.containers.find("web")   # nil rather than an exception
+
+# A name collision usually means somebody else won a race.
+begin
+  client.networks.create("shared")
+rescue Docker::API::Conflict
+  client.networks.get("shared")
+end
+
+# Connection and timeout failures are the ones worth retrying.
+begin
+  client.system.info
+rescue Docker::API::ConnectionError, Docker::API::TimeoutError => e
+  retry if (attempts += 1) < 3
+  raise
+end
+```
+
+Retrying a `ClientError` is almost always wrong: the request was malformed or
+the object was not there, and it will not be there in a second.
+
+## Build failures are not HTTP failures
+
+The daemon reports a failed build inside a `200` response, as an `error` key in
+the JSON-lines stream. Anything that only checks the status code concludes the
+build succeeded. `Images#build` reads the stream and raises, so a build failure
+is an exception like any other.
+
+## Streaming errors
+
+When a streaming request fails, the error response is buffered and raised
+rather than handed to your block. Your block only ever sees output.

data/docs/exec.md

@@ -0,0 +1,100 @@
+# Running commands in containers
+
+## exec
+
+```ruby
+result = container.exec(%w{chef-client -z})
+
+result.stdout     #=> "Starting Chef Infra Client...\n"
+result.stderr     #=> ""
+result.exit_code  #=> 0
+result.success?   #=> true
+result.output     #=> stdout and stderr together
+```
+
+`exec` creates an exec instance, starts it, reads the stream to completion, and
+then asks the daemon for the exit code. All four steps are one call.
+
+### Streaming as it runs
+
+```ruby
+container.exec(%w{make test}) do |stream, chunk|
+  logger << chunk if stream == :stdout
+end
+```
+
+The block receives chunks as they arrive, and the returned `ExecResult` still
+carries the complete output. Both are available; you do not have to choose.
+
+### Options
+
+```ruby
+container.exec(
+  %w{bundle exec rake},
+  env: { "RAILS_ENV" => "test" },
+  user: "app",
+  working_dir: "/srv/app",
+  tty: false,
+  privileged: false
+)
+```
+
+`tty: true` asks the daemon for a terminal, which means output arrives
+unframed. Everything is then reported as `:stdout`, because with a TTY there is
+genuinely only one stream.
+
+### Failure is returned, not raised
+
+A non-zero exit is a result, not an exception. Whether a failing command is an
+error depends entirely on why it was run — a test runner expects failures, a
+provisioning step does not — so the decision stays with the caller:
+
+```ruby
+result = container.exec(%w{rspec})
+warn "tests failed" unless result.success?
+
+container.exec(%w{apt-get update}).check!   # raises when this one fails
+```
+
+`check!` raises a `Docker::API::Error` whose message includes the exit code and
+whatever the command said.
+
+## Logs
+
+```ruby
+container.logs                                   # everything, as a string
+container.logs(tail: 100)
+container.logs(since: Time.now.to_i - 3600)
+container.logs(timestamps: true)
+
+container.logs(follow: true) do |stream, chunk|
+  $stdout << chunk
+end
+```
+
+Without a block the whole log is returned. With one, chunks are yielded as they
+arrive. `follow: true` does not end on its own — break out of the block.
+
+## Attach
+
+`exec` runs a new command. `attach` connects to the container's own process,
+and returns the socket so you can write to its stdin:
+
+```ruby
+io = container.attach(stdin: true, stdout: true, stderr: true)
+io.write("ls /\n")
+puts io.readpartial(4096)
+io.close
+```
+
+## Copying files
+
+```ruby
+archive = Docker::API::Tar.pack_dockerfile("unused", files: { "config.yml" => yaml })
+container.archive_in(archive.read, path: "/etc/app")
+
+tar_bytes = container.archive_out("/var/log/app.log")
+```
+
+Both take and produce tar archives, which is the only format the daemon's copy
+endpoints speak.

data/docs/extending.md

@@ -0,0 +1,109 @@
+# Extending and upgrading
+
+## How the generated layer is made
+
+`data/swagger/v1.55.yaml` is a vendored copy of Docker's own Swagger 2.0
+definition of the Engine API. Three emitters read it:
+
+| Emitter | Output |
+| --- | --- |
+| `tools/generator/operations_emitter.rb` | `lib/docker/api/operations.rb` |
+| `tools/generator/conformance_emitter.rb` | `spec/generated/operations_conformance_spec.rb` |
+| `tools/generator/rbs_emitter.rb` | `sig/docker/api/operations.rbs` |
+
+```console
+$ bundle exec rake api:generate
+```
+
+The output is plain, committed Ruby — no `define_method`, no runtime
+dispatch. Every method is spelled out so it can be read, grepped, documented by
+YARD and stepped through in a debugger. A metaprogrammed layer would be shorter
+to generate and considerably worse to live with.
+
+Because the output is committed, installing the gem needs no toolchain.
+
+## Upgrading to a new API version
+
+```console
+$ bundle exec rake api:sync[1.56]
+$ git diff --stat
+```
+
+`api:sync` fetches the specification, updates `MAX_API_VERSION`, regenerates
+everything, and prints the diff. Then read it:
+
+- A new endpoint appears as a new method.
+- A new parameter appears in a signature and in the RBS.
+- A removed parameter vanishes, and `steep check` names every caller.
+- A changed path or verb shows up as a failing conformance test.
+
+`rake api:verify` fails if the committed files do not match what the
+specification currently produces, so CI catches a hand-edit or a forgotten
+regeneration.
+
+## Adding an ergonomic wrapper
+
+The generated layer is complete. The ergonomic layer is sugar, and it grows
+when sugar is warranted — when there is normalisation to do, several calls to
+sequence, or a stream to decode.
+
+A wrapper is a thin call into `client.operations`:
+
+```ruby
+module Docker
+  module API
+    class Containers < Collection
+      def restart_all(filters: nil)
+        all(filters: filters).each(&:restart)
+      end
+    end
+  end
+end
+```
+
+Two rules keep the layers honest:
+
+1. **The ergonomic layer never talks to the connection directly.** It goes
+   through `operations`, so there is one place where requests are made.
+2. **It never hides capability.** If a wrapper cannot express something the
+   endpoint supports, that is a reason to widen the wrapper, not a reason for
+   callers to have no way to say it.
+
+## Adding a transport
+
+A transport makes a socket. That is the whole contract:
+
+```ruby
+class MyTransport < Docker::API::Transport::Base
+  def connect
+    dial("my://endpoint") { SomeSocket.new(...) }
+  end
+
+  def host_header
+    "localhost"
+  end
+end
+
+Docker::API::Client.new(transport: MyTransport.new)
+```
+
+`dial` converts every way the operating system reports failure into
+`Docker::API::ConnectionError`, which is what keeps `Errno` out of callers'
+rescue clauses.
+
+## Where the parameter names come from
+
+Method and parameter names are derived from the specification's `operationId`
+and parameter names:
+
+- `ContainerCreate` becomes `container_create`
+- `fromImage` becomes `from_image`
+- `one-shot` becomes `one_shot`
+- `X-Registry-Auth` becomes `x_registry_auth`
+- `until` becomes `until_`, because Ruby will accept `def f(until: nil)` and
+  then refuse to parse any reference to it
+- Every body parameter becomes `body`, whatever the specification calls it —
+  there are thirteen different names in v1.55
+
+Renamed parameters say so in their generated documentation, naming the spelling
+that goes on the wire.

data/docs/migrating-from-docker-api.md

@@ -0,0 +1,156 @@
+# Migrating from the docker-api gem
+
+This gem keeps no compatibility with
+[docker-api](https://github.com/upserve/docker-api). Everything lives under
+`Docker::API` and nothing is added to `::Docker`, so **both gems can be loaded
+into one process** while a migration is in progress. There is no flag day.
+
+## The shape of the change
+
+| docker-api | docker-api-ng |
+| --- | --- |
+| Class methods plus a connection argument | Everything hangs off a client |
+| `Docker.url = host` (process-global) | `Client.new(url: host)` (per client) |
+| `Docker.creds = ...` (process-global) | Resolved per call from `~/.docker/config.json` |
+| `Docker::Error::DockerError` | `Docker::API::Error` |
+| `Excon::Error::Socket` reaches you | `Docker::API::ConnectionError`, cause preserved |
+| `.info` — shape depends on the call | Normalised accessors, plus `#raw` |
+
+## Call by call
+
+### The daemon
+
+| docker-api | docker-api-ng |
+| --- | --- |
+| `Docker.info(conn)` | `client.system.info` |
+| `Docker.version(conn)` | `client.system.version` |
+| `Docker.ping(conn)` | `client.system.ping?` |
+| `Docker.authenticate!(creds)` | `client.system.authenticate(username:, password:)` |
+| `Docker.options` / `Docker.url=` | `Docker::API::Client.new(url:, tls:)` |
+| `Docker::Connection.new(url, opts)` | `Docker::API::Client.new(url:, tls:)` |
+
+### Containers
+
+| docker-api | docker-api-ng |
+| --- | --- |
+| `Docker::Container.get(name, {}, conn)` | `client.containers.get(name)` |
+| `Docker::Container.all({ all: true }, conn)` | `client.containers.all(all: true)` |
+| `Docker::Container.create(args, conn)` | `client.containers.create(**args)` |
+| `container.start` | `container.start` |
+| `container.stop` | `container.stop(timeout:)` |
+| `container.delete(force: true)` | `container.remove(force: true)` |
+| `container.json` | `container.raw`, or a named accessor |
+| `container.exec(cmd, wait:, "e" => env) { }` | `container.exec(cmd, env: env) { }` |
+| `container.info["Names"].include?("/x")` | `container.name == "x"` |
+
+`exec` returns a `Docker::API::ExecResult` with `#stdout`, `#stderr` and
+`#exit_code`, rather than a three-element array:
+
+```ruby
+# docker-api
+out = container.exec(cmd, wait: 60) { |_stream, chunk| logger << chunk }
+raise if out[2] != 0
+
+# docker-api-ng
+result = container.exec(cmd) { |_stream, chunk| logger << chunk }
+raise unless result.success?
+```
+
+### Images
+
+| docker-api | docker-api-ng |
+| --- | --- |
+| `Docker::Image.exist?(name, {}, conn)` | `client.images.exist?(name)` |
+| `Docker::Image.get(name, {}, conn)` | `client.images.get(name)` |
+| `Docker::Image.create({ "fromImage" => x }, creds, conn)` | `client.images.pull(x)` |
+| `Docker::Image.build(dockerfile, opts, conn)` | `client.images.build(dockerfile:, tag:)` |
+| `Docker::Image.build_from_dir(dir, opts)` | `client.images.build(context: dir, tag:)` |
+| `image.tag("repo" => r, "tag" => t)` | `image.tag("#{r}:#{t}")` |
+| `image.remove` | `image.remove` |
+
+### Networks and volumes
+
+| docker-api | docker-api-ng |
+| --- | --- |
+| `Docker::Network.get(name, {}, conn)` | `client.networks.get(name)` |
+| `Docker::Network.create(name, settings)` | `client.networks.create(name, **settings)` |
+| get-or-create with a rescue | `client.networks.ensure(name)` |
+| `network.info["EnableIPv6"]` | `network.ipv6?` |
+| `Docker::Volume.create(name)` | `client.volumes.create(name)` |
+
+### Errors
+
+| docker-api | docker-api-ng |
+| --- | --- |
+| `Docker::Error::DockerError` | `Docker::API::Error` |
+| `Docker::Error::NotFoundError` | `Docker::API::NotFound` |
+| `Docker::Error::ConflictError` | `Docker::API::Conflict` |
+| `Docker::Error::ServerError` | `Docker::API::ServerError` |
+| `Docker::Error::UnexpectedResponseError` | `Docker::API::BadRequest` |
+| `Docker::Error::TimeoutError` | `Docker::API::TimeoutError` |
+| `Docker::Error::IOError` | `Docker::API::ConnectionError` |
+| `Excon::Error::Socket` | `Docker::API::ConnectionError` |
+
+## Workarounds you can now delete
+
+### The dropped `platform` parameter
+
+`Docker::Container.create` forwards only `name` to the query string, so
+`platform` never reaches the daemon. The usual workaround is to create the
+container and re-fetch it:
+
+```ruby
+# docker-api
+Docker::Container.create(args, conn)
+Docker::Container.get(args["name"], {}, conn)
+
+# docker-api-ng
+client.containers.create(name: name, platform: "linux/arm64", **args)
+```
+
+### Listing everything to find one container
+
+`Docker::Container.get` is unreliable in docker-api 2.0.0, and the workaround
+is to list every container and match by name:
+
+```ruby
+# docker-api
+found = Docker::Container.all({ all: true }, conn)
+  .select { |c| c.info["Names"].include?("/#{name}") }
+raise Docker::Error::NotFoundError if found.empty?
+
+# docker-api-ng
+client.containers.get(name)
+```
+
+### Per-host caching around global state
+
+Because `Docker.url=` is process-global, code that talks to more than one
+daemon has to cache results per host to avoid handing the second caller the
+first daemon's answers. Two clients need no such care:
+
+```ruby
+local = Docker::API::Client.new
+build = Docker::API::Client.new(url: "tcp://build:2376")
+```
+
+### Rescuing the HTTP library
+
+```ruby
+# docker-api
+rescue Excon::Error::Socket => e
+
+# docker-api-ng
+rescue Docker::API::ConnectionError => e
+  e.cause   # the Errno, if you actually want it
+```
+
+## Anything not listed here
+
+The complete Engine API is available on `client.operations`, named after the
+specification's own operation ids:
+
+```ruby
+client.operations.container_prune(filters: { "until" => ["24h"] })
+client.operations.image_history(name: "alpine:3.20")
+```

data/docs/streaming.md

@@ -0,0 +1,91 @@
+# Streaming
+
+The Engine API streams in three different wire formats, and which one you get
+depends on the endpoint and on how the container was created.
+
+## The formats
+
+| Format | Where it appears | Decoder |
+| --- | --- | --- |
+| Multiplexed | `logs`, `exec`, `attach` on a container with no TTY | `Stream::Demultiplexer` |
+| Raw | the same endpoints when the container has a TTY | `Stream::Raw` |
+| JSON lines | `pull`, `push`, `build`, `/events` | `Stream::JSONLines` |
+
+The ergonomic layer picks the right decoder for you. `Container#logs` and
+`Container#exec` check whether the container has a TTY, because that is what
+decides between the first two.
+
+## Multiplexed framing
+
+Without a TTY, stdout and stderr share one connection and are separated by an
+eight-byte header on every frame: a stream id, three bytes of padding, then a
+big-endian payload length.
+
+```
++--------+--------+--------+--------+--------+--------+--------+--------+
+| stream |   0    |   0    |   0    |          payload length           |
++--------+--------+--------+--------+--------+--------+--------+--------+
+```
+
+```ruby
+container.exec(%w{make test}) do |stream, chunk|
+  case stream
+  when :stdout then $stdout << chunk
+  when :stderr then $stderr << chunk
+  end
+end
+```
+
+## Why the decoders buffer
+
+HTTP delivers chunks at byte boundaries that have nothing to do with Docker's
+framing. A frame header can be split across two chunks; a JSON object can be
+split mid-string. Every decoder here accumulates and emits only complete units.
+
+A decoder that assumes a chunk is a message passes its tests — where chunks
+happen to align — and interleaves garbage in production. It is worth knowing
+this is handled if you are ever tempted to read the raw stream yourself.
+
+## JSON-lines progress
+
+```ruby
+client.images.pull("alpine:3.20") do |event|
+  puts "#{event["status"]} #{event["progress"]}"
+end
+
+client.images.build(context: ".", tag: "app:dev") do |event|
+  print event["stream"] if event["stream"]
+end
+```
+
+Build streams carry the resulting image id in an `aux` object near the end, and
+report failures in an `error` key. `Images#build` reads both.
+
+## Endless streams
+
+`system.events` and `logs(follow: true)` do not end on their own. Break out of
+the block, or run them on a thread you can stop.
+
+```ruby
+client.system.events(filters: { "type" => ["container"] }) do |event|
+  break if event["Action"] == "die"
+end
+```
+
+`read_timeout` applies to the gap between chunks, not to the total duration, so
+a quiet stream will not be cut off mid-flow.
+
+## Taking the socket
+
+For genuinely interactive work — sending stdin — `Container#attach` returns the
+bidirectional socket itself:
+
+```ruby
+io = container.attach(stdin: true, stdout: true, stderr: true)
+io.write("echo hello\n")
+puts io.readpartial(4096)
+io.close
+```
+
+This path bypasses `Net::HTTP` deliberately, so the socket comes back with
+nothing consumed but the response head.