itsi-server 0.1.1 → 0.2.2

data/lib/itsi/server/config/middleware/endpoint/schemas.md

@@ -0,0 +1,122 @@
+---
+title: Schemas
+url: /middleware/endpoint/schemas
+next: error_response/
+---
+
+Endpoints also support basic schema enforcement for both request and and response bodies. Endpoint **Schemas** are simply plain-old Ruby hashes, which map named keys to allowed types.
+
+E.g.
+```ruby {filename=Itsi.rb}
+# This is a valid schema
+SimpleSchema = {
+  score: Integer
+}
+```
+
+A few special rules/conventions apply:
+
+* Regexp patterns are used in place of symbol keys to define dynamic parameters that must match a specific pattern (A key of `/.*/` matches any string).
+* A `:Boolean` symbol can be used instead of a class to require boolean parameters (as a proxy for the combination of TrueClass and FalseClass)
+* `Array` keys are used to define arrays of values.
+* Objects can be nested
+* A special `_required` key indicates which parameters in the current object are required.
+* Arrays must be homogeneous (No support for union types).
+* Schema validation *only* supports primitive Ruby types, Date, Time, Datetime, arrays and hashes (no unions or advanced coercion).
+
+
+## Supported types
+```ruby {filename=Itsi.rb}
+# Demo schema with all supported types included
+SupportedTypesSchema = {
+  _required: %i[name age active preferences created_at last_login],
+  name: String,               # A user's name
+  age: Integer,               # The user's age
+  active: :Boolean,           # Whether the user is active
+  preferences: {
+    _required: %i[theme notifications],
+    theme: Symbol,            # Preferred theme (e.g., :dark, :light)
+    notifications: :Boolean   # Whether notifications are enabled
+  },
+  scores: Array[Float],       # A list of user scores
+  metadata: {
+    _required: %i[signup_date last_purchase],
+    signup_date: Date,        # The date the user signed up
+    last_purchase: DateTime   # The timestamp of the last purchase
+  },
+  last_login: Time            # The last login time
+}
+```
+
+{{< callout >}}
+Itsi will check for the presence and structure of all:
+* [Schemas](/middleware/schemas)
+* [Controller methods](/middleware/controller)
+
+at **boot time** ensuring that you can't be caught out by naming mismatches at runtime, or during a hot-reload.
+If Itsi boots successfully, the referenced schemas objects can be found.
+{{</ callout >}}
+
+
+## Request Body Schemas
+
+```ruby {filename=Itsi.rb}
+AddressSchema = {
+  _required: %i[street city postcode],
+  street: String,
+  city: String,
+  postcode: String,
+  country: String # Optional
+}
+
+UserInputSchema = {
+  _required: %i[first_name last_name email address],
+  first_name: String,
+  last_name: String,
+  email: String,
+  age: Integer,
+  active: :Boolean,
+  roles: Array[String],
+  address: AddressSchema
+}
+
+UserResponseSchema = {
+  _required: %i[id email full_name],
+  id: Integer,
+  email: String,
+  full_name: String,
+  created_at: String,
+  address: AddressSchema
+}
+
+post "/users" do |request, params: UserInputSchema, response_format: UserResponseSchema|
+  user = User.create!(params)
+  request.created \
+    json: {
+      id: user.id,
+      email: user.email,
+      full_name: "#{user.first_name} #{user.last_name}",
+      created_at: user.created_at.iso8601,
+      address: {
+        street: user.address.street,
+        city: user.address.city,
+        postcode: user.address.postcode
+      }
+    },
+    as: response_format
+end
+```
+Endpoints with schema validation applied can count on requests *only* being invoked with correctly formed parameters (and descriptive errors being returned if schema enforcement fails.)
+
+You can *optionally* choose to apply a corresponding response schema validation to response objects.
+
+## Response Body Schemas
+```ruby {filename=Itsi.rb}
+PingSchema = {
+  status: String
+}
+
+get "/ping" do |req, response_format: PingSchema|
+  req.ok json: { status: "ok" }, as: response_format
+end
+```

data/lib/itsi/server/config/middleware/error_response.md

@@ -0,0 +1,61 @@
+---
+title: Error Response
+url: /middleware/error_response
+prev: schemas/
+---
+
+All of the below middlewares allow error responses to be customized (though will select a sane default if left unspecified).
+* [`allow_list`](/middleware/allow_list)
+* [`auth_api_key`](/middleware/auth_api_key)
+* [`auth_jwt`](/middleware/auth_jwt)
+* [`deny_list`](/middleware/deny_list)
+* [`intrusion_protection`](/middleware/intrusion_protection)
+* [`max_body`](/middleware/max_body)
+* [`proxy`](/middleware/proxy)
+* [`rate_limit`](/middleware/rate_limit)
+
+You can override the default error responses by providing a custom error response, either selecting a different built-in type,
+or providing a completely overridden options.
+The following built-in types are available:
+* `internal_server_error` (500)
+* `not_found` (404)
+* `unauthorized` (401)
+* `forbidden` (403)
+* `payload_too_large` (413)
+* `too_many_requests` (429)
+* `bad_gateway` (502)
+* `service_unavailable` (503)
+* `gateway_timeout` (504)
+
+## Reuse a built-in type
+To reuse a built-in type you can provide a string option for the `error_response` property.
+E.g.
+```ruby
+auth_api_key .. other options.., error_response: 'forbidden'
+```
+
+## Override the error response
+You may instead wish to completely override the error response. You can provide a status code, and a message in up to three
+formats: plain-text, JSON, or HTML (at least one must be provided). Itsi will serve the appropriate type based on the `Accept` header of the incoming request, or fall back to the default if the requested type is not available.
+Each of these three response formats can either be provided as an in-memory string, or a file path.
+
+E.g.
+
+```ruby
+auth_api_key .. other options.., error_response: {
+  status: 403,
+  plaintext: nil, # No plain-text response provided
+  json: { inline: {"message": "Forbidden"} }, # We provide the JSON response inline
+  html: { file: './forbidden.html'}, # We provide the HTML response as a file path
+  default: 'json' # When the Accept header doesn't match a supported response type, we'll default to JSON
+}
+
+# Or, e.g.
+auth_api_key .. other options.., error_response: {
+  status: 401,
+  plaintext: "Unauthorized",
+  json: { file: "unauthorized.json" },
+  html: { inline: "<h1>Unauthorized</h1>" },
+  default: 'html'
+}
+```

data/lib/itsi/server/config/middleware/error_response.rb

@@ -0,0 +1,36 @@
+module Itsi
+  class Server
+    module Config
+      InlineContentSource = TypedStruct.new do
+        {
+          inline: Type(String)
+        }
+      end
+
+      FileContentSource = TypedStruct.new do
+        {
+          file: Type(String)
+        }
+      end
+
+      ContentSource = TypedStruct.new do
+        Or(Type(InlineContentSource), Type(FileContentSource))
+      end
+
+      ErrorResponse = TypedStruct.new do
+        {
+          code: Type(Integer) & Required(),
+          plaintext: Type(ContentSource),
+          html: Type(ContentSource),
+          json: Type(ContentSource),
+          default: Enum(["plaintext", "html", "json"]) & Required()
+        }
+      end
+
+      ErrorResponseDef = TypedStruct.new do
+        Or(Enum(%w[internal_server_error not_found unauthorized forbidden payload_too_large
+        too_many_requests bad_gateway service_unavailable gateway_timeout]), Type(ErrorResponse)) & Required()
+      end
+    end
+  end
+end

data/lib/itsi/server/config/middleware/etag.md

@@ -0,0 +1,59 @@
+---
+title: ETag
+url: /middleware/etag
+---
+
+The ETag middleware enables HTTP entity tag generation for responses. It provides cache validation by automatically computing and attaching an ETag to outgoing responses, and optionally responding with `304 Not Modified` if the client includes a matching `If-None-Match` header.
+
+ETags are useful for optimizing client-side caching, conditional GETs, and reducing unnecessary data transfer.
+
+
+## ETag configuration
+```ruby {filename=Itsi.rb}
+etag \
+  type: "strong",
+  algorithm: "sha256",
+  min_body_size: 0,
+  handle_if_none_match: true
+```
+
+## ETag Applied to a sub-location
+```ruby {filename=Itsi.rb}
+location "/assets" do
+  etag \
+    type: "weak",
+    algorithm: "md5",
+    min_body_size: 1024,
+    handle_if_none_match: true
+end
+```
+
+## Configuration Options
+
+- **type**: Specifies whether the generated ETag is `"strong"` or `"weak"`.
+  - `strong`: ETag changes with any byte-level difference in the body.
+  - `weak`: Indicates a semantic equivalence rather than byte-level identity.
+
+- **algorithm**: Specifies the hash algorithm used to compute the ETag.
+  - `sha256` (default)
+  - `md5`
+
+- **min_body_size**: Minimum response body size (in bytes) required before an ETag is generated. Use this to skip ETags for small or trivial responses.
+
+- **handle_if_none_match**: When `true`, incoming requests with a matching `If-None-Match` header will receive a `304 Not Modified` response (instead of a full body), if the ETag matches the computed value.
+
+## How It Works
+
+### Before the Response
+If `handle_if_none_match` is enabled and the request includes an `If-None-Match` header, the value is stored in the request context for comparison later.
+
+### After the Response
+
+1. If the status code is cacheable (e.g., 200 OK, 201 Created), the middleware proceeds.
+2. If an ETag header is already present or if `Cache-Control: no-store` is set, it skips computation.
+3. If the response body is not streamable and meets the `min_body_size`, it is buffered.
+4. A hash (SHA-256 or MD5) is computed from the full body content.
+5. The ETag header is inserted using either strong (`"abc123"`) or weak (`W/"abc123"`) formatting.
+6. If the incoming request had a matching `If-None-Match`, the response is replaced with a 304 Not Modified.
+
+---

data/lib/itsi/server/config/middleware/etag.rb

@@ -0,0 +1,27 @@
+module Itsi
+  class Server
+    module Config
+      class Etag < Middleware
+
+        insert_text <<~SNIPPET
+        etag \\
+          type: ${1|"strong","weak"|},
+          algorithm: ${2|"sha256","md5"|},
+          min_body_size: ${3|0,1024|},
+          handle_if_none_match: ${4|true,false|}
+        SNIPPET
+
+        detail "Enables ETag generation for the server."
+
+        schema do
+          {
+            type: (Enum(["strong", "weak"]) & Required()).default("strong"),
+            algorithm: (Enum(["sha256", "md5"]) & Required()).default("sha256"),
+            min_body_size: Range(0...1024 ** 3).default(0),
+            handle_if_none_match: Bool().default(true)
+          }
+        end
+      end
+    end
+  end
+end

data/lib/itsi/server/config/middleware/grpc.md

@@ -0,0 +1,172 @@
+---
+title: gRPC
+url: /middleware/grpc
+---
+
+The **gRPC** middleware lets you expose one or more Ruby gRPC service handlers directly in Itsi.
+This allows you to use Itsi's efficient asynchronous server to serve requests and gain enhanced performance and asynchronous capabilities.
+
+Under the covers it:
+
+1. Serves a full HTTP/2 gRPC endpoint (with binary frames, trailers, `HTTP/2` compression, and reflection).
+2. Provides a **JSON‑over‑HTTP** gateway for each unary or streaming method—so you can POST JSON and receive JSON arrays without a gRPC client.
+3. Automatically enables `gRPC` reflection (so client like `evans`, `grpcurl` or `Postman` can discover your service endpoints without needing access to raw `.proto` files).
+4. Supports optional per‑call compression (`none`, `deflate`, `gzip`) and a non‑blocking execution mode.
+
+---
+
+## Usage
+
+```ruby
+# Define (or require) your service implementation:
+# Mount it in your Itsi.rb:
+grpc EchoServiceImpl.new,
+     nonblocking: false do         # prepend response with grpc‑encoding:gzip
+  # any additional middleware can nest here, e.g.:
+  response_headers additions: { "X-Service" => ["Echo"] }
+end
+```
+
+## Options
+
+| Option        | Type    | Default | Description                                                   |
+|---------------|---------|---------|---------------------------------------------------------------|
+| `*handlers`  | Object  | –       | One or more gRPC service implementations  |
+| `nonblocking`   | Boolean | false   | Run handler in fiber/thread‑pool (nonblocking mode). Only effective if using [hybrid mode](/options/scheduler_threads), otherwise the gRPC handler will adopt the server global concurrency mode (threads or fibers)         |
+| `reflection`   | Boolean | true   | Determines whether to serve reflection endpoints. (Useful for clients to auto-discover services)         |
+| `&block`        | Proc    | –       | **Optional** You can add additional middleware inside an optional block, to apply to all gRPC requests (e.g. rate limiters, deny lists, max body etc. etc.).                |
+
+Reflection is served only over HTTP/2; JSON‑over‑HTTP works with HTTP/1.1.
+
+
+
+## Walkthrough
+The below is a simple walkthrough of using Itsi and the [Ruby gRPC](https://grpc.io/docs/languages/ruby/basics/) library to expose a simple "Echo" service in Itsi,  complete with both native gRPC/HTTP2 and a JSON‑over‑HTTP gateway.
+
+{{% steps %}}
+
+### Step 1 — Define your gRPC contract
+
+Create a file `echo.proto` with:
+```proto
+syntax = "proto3";
+package echo;
+
+service EchoService {
+  rpc Echo(EchoRequest) returns (EchoResponse);
+}
+
+message EchoRequest {
+  string message = 1;
+}
+
+message EchoResponse {
+  string message = 1;
+}
+```
+
+### Step 2 — Generate Ruby stubs
+
+Install the Ruby gRPC tools and run:
+```bash
+gem install grpc-tools
+grpc_tools_ruby_protoc -I . --ruby_out=./ --grpc_out=./ echo.proto
+```
+This produces `echo_services_pb.rb` and supporting files.
+
+### Step 3 — Implement the service
+
+Create `echo_service_impl.rb`:
+```ruby {filename=echo_service_impl.rb}
+require_relative 'echo_services_pb'
+
+class EchoServiceImpl < Echo::EchoService::Service
+  # Unary RPC implementation
+  def echo(req, _unused_call)
+    Echo::EchoResponse.new(message: req.message)
+  end
+end
+```
+
+### Step 4 — Mount in Itsi
+
+In your `Itsi.rb` at the project root, add:
+
+```ruby {filename=Itsi.rb}
+require_relative 'echo_service_impl'
+
+bind "https://localhost:3000"
+grpc EchoServiceImpl.new,
+      nonblocking: false,
+      compression: 'gzip' do
+  # Nested middleware still works:
+  response_headers additions: { 'X-Service' => ['Echo'] }
+end
+```
+### Step 5 — Start the server
+```bash
+itsi serve
+```
+
+### Step 6 — Test with a gRPC client
+
+E.g. using [Evans](https://github.com/ktr0731/evans)
+
+```bash
+evans --host localhost --port 3000 repl
+
+  ______
+ |  ____|
+ | |__    __   __   __ _   _ __    ___
+ |  __|   \ \ / /  / _. | | '_ \  / __|
+ | |____   \ V /  | (_| | | | | | \__ \
+ |______|   \_/    \__,_| |_| |_| |___/
+
+ more expressive universal gRPC client
+
+
+echo.EchoService@127.0.0.1:3000> call Echo
+message (TYPE_STRING) => Hello
+{
+  "message": "Hello"
+}
+
+echo.EchoService@127.0.0.1:3000>
+```
+
+### Step 7 — Try the JSON gateway
+
+You can also POST plain JSON (works over HTTP/1.1 or HTTP/2):
+```bash
+curl \
+  -H 'Content-Type: application/json' \
+  -H "Content-Type: application/json" \
+  -X POST http://localhost:3000/echo.EchoService/Echo \
+  -d '{"message":"world"}'
+
+{"message":"world"}
+
+```
+
+{{% /steps %}}
+
+
+## JSON gateway
+Itsi exposes gRPC services via a secondary JSON gateways for use with simple HTTP clients.
+Once mounted, each service‐method is *also* reachable via HTTP/2 and via simple JSON POSTs:
+
+```bash
+# Unary RPC (Echo)
+curl -X POST http://0.0.0.0:3000/EchoService/Echo \
+     -H "Content-Type: application/json" \
+     -d '{"message":"hello"}'
+# → 200 OK, body: {"message":"hello"}
+
+# Server streaming RPC (e.g. "Numbers"): POST an array, receive JSON array:
+curl -X POST http://0.0.0.0:3000/NumberService/Stream \
+     -H "Content-Type: application/json" \
+     -d '[{"n":1},{"n":2},{"n":3}]'
+# → 200 OK, body: [{"n":1},{"n":2},{"n":3}]
+```
+
+Under the hood, the same framing machinery is used—you just get plain JSON arrays instead of gRPC frames.

data/lib/itsi/server/config/middleware/grpc.rb

@@ -0,0 +1,54 @@
+module Itsi
+  class Server
+    module Config
+      class Grpc < Middleware
+        insert_text [
+          <<~SNIPPET,
+          grpc ${1:MyServiceImpl.new}, nonblocking: ${2|false,true|} do
+            ${3:# nested middleware…}
+          end
+          SNIPPET
+
+          <<~SNIPPET,
+          grpc ${1:MyServiceImpl.new}, nonblocking: ${2|false,true|}
+          SNIPPET
+        ]
+
+        detail [
+          "gRPC service with middleware  (with HTTP/2, compression, reflection and JSON gateway)",
+          "gRPC service (with HTTP/2, compression, reflection and JSON gateway)"
+        ]
+
+        schema do
+          {
+            handlers: Array(Type(Object)) & Required(),
+            reflection: Bool().default(true),
+            nonblocking: Bool().default(false),
+            inner_block: Type(Proc)
+          }
+        end
+
+        def initialize(location, *handlers, reflection: true, nonblocking: false, &block)
+          super(location, {
+            handlers: handlers,
+            reflection: reflection,
+            nonblocking: nonblocking,
+            inner_block: block
+          })
+        end
+
+        def build!
+          location.grpc_reflection(@params[:handlers]) if @params[:reflection]
+          nonblocking = @params[:nonblocking]
+          blk = @params[:inner_block]
+          @params[:handlers].each do |handler|
+            location.location(Regexp.new("#{Regexp.escape(handler.class.service_name)}/(?:#{handler.class.rpc_descs.keys.map(&:to_s).join("|")})")) do
+              @middleware[:app] = { preloader: -> { Itsi::Server::GrpcInterface.for(handler) }, request_type: "grpc", nonblocking: nonblocking }
+              instance_exec(&blk) if blk
+            end
+          end
+        end
+      end
+    end
+  end
+end

data/lib/itsi/server/config/middleware/intrusion_protection.md

@@ -0,0 +1,124 @@
+---
+title: Intrusion Protection
+url: /middleware/intrusion_protection
+---
+
+The **Intrusion Protection** middleware detects and automatically bans clients that attempt to access suspicious URLs or send malicious header values. It combines pattern‑based detection (on request paths and header values) with a back‑end ban manager to temporarily block offending IPs.
+
+- **URL Patterns**: a list of regexes; any matching request path causes an immediate ban.
+- **Header Patterns**: per‑header regex lists; any matching header value causes a ban.
+- **Ban Duration**: how long (in seconds) to block the client IP.
+- **Store**: in‑memory or Redis‑backed (`store_config`) for both tracking and bans.
+- **Error Response**: customizable (default is `forbidden`).
+
+## Configuration
+
+```ruby {filename=Itsi.rb}
+intrusion_protection \
+  banned_url_patterns: [
+    "/admin/login",         # brute‑force login attempts
+    /\.php$/               # any PHP‑extension request
+  ],
+  banned_header_patterns: {
+    "User-Agent" => [
+      "sqlmap",             # SQL injection scanner
+      "curl"                # script‑based probing
+    ]
+  },
+  banned_time_seconds: 300, # ban for 5 minutes
+  store_config: "in_memory",# or { redis: { connection_url: "redis://…" } }
+  error_response: "forbidden"
+```
+
+### Using `KnownPaths` Helpers
+
+Itsi provides a `KnownPaths` module with many pre‑assembled lists of common attack targets taken from [FuzzDB](https://blog.mozilla.org/security/2013/08/16/introducing-fuzzdb/) (e.g. typical login or backup file locations). Each helper returns an `Array<String>` you can pass directly:
+
+```ruby {filename=Itsi.rb}
+# ban all common WordPress plugin endpoints
+intrusion_protection \
+  banned_url_patterns: Itsi::Server::KnownPaths.cms_wp_plugins,
+  banned_time_seconds: 600
+
+# ban both login files and directory‑brute paths
+intrusion_protection \
+  banned_url_patterns: (
+    Itsi::Server::KnownPaths.login_file_locations_logins +
+    Itsi::Server::KnownPaths.filename_dirname_bruteforce_common_web_extensions
+  ).uniq,
+  banned_time_seconds: 900
+```
+
+Available helper methods live under `Itsi::Server::KnownPaths`—for example:
+
+- `login_file_locations_logins`
+- `filename_dirname_bruteforce_test_demo`
+- `cms_wp_plugins`
+- `php_php_common_backdoors`
+- …and many more.
+To see all options, execute
+```ruby
+Itsi::Server::KnownPaths::ALL
+```
+in a REPL or see the raw input files [here](https://github.com/wouterken/itsi/tree/main/gems/server/lib/itsi/server/config/known_paths).
+
+### Options
+
+- **banned_url_patterns** (Array<String>)
+  Regexes applied to the full `path_and_query` of each request. A match → immediate ban+403.
+- **banned_header_patterns** (Hash<String,Array<String>>)
+  For each header name, a list of regexes tested against that header’s value. A match → ban+403.
+- **banned_time_seconds** (Integer)
+  Duration (in seconds) to keep the client IP banned.
+- **store_config** (`"in_memory"` or `{ redis: { connection_url: String } }`)
+  Backend for counters and ban state.
+- **error_response** (String or detailed ErrorResponse)
+  Response returned on detection or if IP is already banned (default: `forbidden`).
+- **trusted_proxies** (Hash<String,Hash>)
+  Map of trusted proxy IP addresses to their forwarded header configuration.
+
+## How It Works
+
+1. **Initialization**
+   - Compile `banned_url_patterns` into a `RegexSet`.
+   - Compile each set of header patterns into its own `RegexSet`.
+   - Instantiate a `RateLimiter` and `BanManager` (in‑memory or Redis).
+
+2. **Per‑Request**
+   - **Check ban status**: if the IP is already banned, return `error_response` immediately.
+   - **URL check**: if the request’s `path_and_query` matches any banned URL pattern, ban the IP for `banned_time_seconds` and return `error_response`.
+   - **Header check**: for each configured header, if its value matches any banned pattern, ban the IP and return `error_response`.
+   - Otherwise, allow the request to proceed.
+
+Banned IPs are automatically un‑banned after the specified TTL.
+
+## Trusted Proxies
+
+By default, an intrusion protection middleware uses the IP address from the underlying socket (remote_addr). However, if your server is behind a reverse proxy, all requests will appear to come from the proxy’s IP address. This can break IP-based rules or cause rate-limiting to group all users together.
+
+To address this, you can declare trusted proxies and instruct the server to extract the original client IP from forwarded headers only if the request came from one of these proxies.
+
+
+### Configuring trusted_proxies
+
+To trust one or more upstream proxies, provide a trusted_proxies map in the middleware configuration.
+E.g.
+```ruby {filename=Itsi.rb}
+intrusion_protection \
+  banned_url_patterns: [
+    "/admin/login",         # brute‑force login attempts
+    /\.php$/               # any PHP‑extension request
+  ],
+  banned_header_patterns: {
+    "User-Agent" => [
+      "sqlmap",             # SQL injection scanner
+      "curl"                # script‑based probing
+    ]
+  },
+  banned_time_seconds: 300, # ban for 5 minutes
+  store_config: "in_memory",# or { redis: { connection_url: "redis://…" } }
+  error_response: "forbidden",
+  trusted_proxies: {
+    "192.168.1.1" => { header: { name: "X-Forwarded-For" } }
+  }
+```