aws-cdk-neuronx-patterns 0.0.2 → 0.0.4

package/.jsii

@@ -7,6 +7,7 @@
     ]
   },
   "dependencies": {
+    "@aws-cdk/aws-sagemaker-alpha": "2.149.0-alpha.0",
     "aws-cdk-lib": "^2.149.0",
     "constructs": "^10.0.5"
   },
@@ -89,6 +90,37 @@
         }
       }
     },
+    "@aws-cdk/aws-sagemaker-alpha": {
+      "targets": {
+        "dotnet": {
+          "iconUrl": "https://raw.githubusercontent.com/aws/aws-cdk/main/logo/default-256-dark.png",
+          "namespace": "Amazon.CDK.AWS.Sagemaker.Alpha",
+          "packageId": "Amazon.CDK.AWS.Sagemaker.Alpha"
+        },
+        "go": {
+          "moduleName": "github.com/aws/aws-cdk-go",
+          "packageName": "awscdksagemakeralpha"
+        },
+        "java": {
+          "maven": {
+            "artifactId": "sagemaker-alpha",
+            "groupId": "software.amazon.awscdk"
+          },
+          "package": "software.amazon.awscdk.services.sagemaker.alpha"
+        },
+        "js": {
+          "npm": "@aws-cdk/aws-sagemaker-alpha"
+        },
+        "python": {
+          "classifiers": [
+            "Framework :: AWS CDK",
+            "Framework :: AWS CDK :: 2"
+          ],
+          "distName": "aws-cdk.aws-sagemaker-alpha",
+          "module": "aws_cdk.aws_sagemaker_alpha"
+        }
+      }
+    },
     "aws-cdk-lib": {
       "submodules": {
         "aws-cdk-lib.alexa_ask": {
@@ -3819,7 +3851,7 @@
   },
   "name": "aws-cdk-neuronx-patterns",
   "readme": {
-    "markdown": "# Neuronx patterns Construct Library\n\nThis library provides high-level architectural patterns using neuronx (e.g. Inferentia2 and Trainium1). It contains:\n\n- Neuronx Compile\n\n## Neuronx Compile\n\n:::note warn\nThis construct uses an Inferentia2 instance on EC2. You may need to increase your request limit for your AWS account.\n:::\n\nThis construct compiles models supported by Neuronx and uploads them to the specified S3 bucket.\n\nThis is NeuronxCompile architecture.\n![NeuronxCompile architecture](./docs/neuronx-compile-architecture.png)\n\nTo define\n\n```ts\nimport { Vpc } from \"aws-cdk-lib/aws-ec2\";\nimport { Bucket } from \"aws-cdk-lib/aws-s3\";\n\ndeclare const vpc: Vpc;\ndeclare const bucket: Bucket;\nconst compile = new NeuronxCompile(stack, \"NeuronxCompile\", {\n  vpc,\n  bucket,\n  model: Model.fromHuggingFace(\"example/example-7b-chat\", {\n    parameters: Parameters.billion(7),\n  }),\n});\n\n// Get the compiled artifacts from this S3 URL\nnew CfnOutput(stack, \"CompiledArtifact\", {\n  value: compile.compiledArtifactS3Url,\n});\n```\n\nThis construct assumes the required instance type depending on the number of model parameters.\n\nAfter compiled, you can see like the this file tree in the S3 bucket.\n\n```txt\n{compiledArtifactS3Url}/\n├── model\n│   ├── config.json\n│   ├── tokenizer_config.json\n│   ├── xxx.safetensors\n│   └── xxx.safetensors\n└── compiled\n    ├── xxx.neff\n    ├── xxx.neff\n    └── xxx.neff\n```\n\n### Spot Instance\n\n:::note warn\nIf you use Spot Instances, check if the request limit for Spot has been increased.\n:::\n\nYou can also use Spot Instances.\n\n```ts\nimport { Vpc } from \"aws-cdk-lib/aws-ec2\";\nimport { Bucket } from \"aws-cdk-lib/aws-s3\";\n\ndeclare const vpc: Vpc;\ndeclare const bucket: Bucket;\nnew NeuronxCompile(stack, \"NeuronxCompile\", {\n  vpc,\n  bucket,\n  model: Model.fromHuggingFace(\"example/example-7b-chat\", {\n    parameters: Parameters.billion(7),\n  }),\n  spot: true,\n});\n```\n\n### Compile Options\n\nIf you are familiar with Neuronx, you can also specify compilation options to better meet your requirements.\n\n```ts\nimport { Vpc } from \"aws-cdk-lib/aws-ec2\";\nimport { Bucket } from \"aws-cdk-lib/aws-s3\";\n\ndeclare const vpc: Vpc;\ndeclare const bucket: Bucket;\nnew NeuronxCompile(stack, \"NeuronxCompile\", {\n  vpc,\n  bucket,\n  model: Model.fromHuggingFace(\"example/example-22b-chat\", {\n    parameters: Parameters.billion(22),\n  }),\n  compileOptions: {\n    nPositions: 1024,\n    quantDtype: QuantDtype.S8,\n    optLevel: OptLevel.MODEL_EXECUTION_PERFORMANCE,\n  },\n});\n```\n"
+    "markdown": "# Neuronx patterns Construct Library\n\n> [!WARNING]\n> This library is experimental module.\n\nThis library provides high-level architectural patterns using neuronx (e.g. Inferentia2 and Trainium1). It contains:\n\n- Transformers Neuronx SageMaker Real-time Inference Endpoint\n- Neuronx Compile\n\n## Transformers Neuronx SageMaker Real-time Inference Endpoint\n\n> [!WARNING]\n> This construct uses an Inferentia2 instance on SageMaker. You may need to increase your request limit for your AWS account.\n\nBy using the `NeuronxCompile` construct included in this construct library, models published on HuggingFace can be easily deployed to SageMaker Real-time inference. To define using the `NeuronxCompile` construct:\n\n```ts\nimport * as ec2 from \"aws-cdk-lib/aws-ec2\";\nimport * as s3 from \"aws-cdk-lib/aws-s3\";\n\ndeclare const vpc: ec2.Vpc;\ndeclare const bucket: s3.Bucket;\nconst compile = new NeuronxCompile(this, \"NeuronxCompile\", {\n  vpc,\n  bucket,\n  model: Model.fromHuggingFace(\"example/example-7b-chat\", {\n    parameters: Parameters.billion(7),\n  }),\n});\nnew TransformersNeuronxSageMakerRealtimeInferenceEndpoint(\n  this,\n  \"RealtimeInference\",\n  {\n    modelData:\n      TransformersNeuronxSageMakerInferenceModelData.fromNeuronxCompile(\n        compile,\n      ),\n  },\n);\n```\n\n### Default inference code\n\nBy default, default inference code is deployed to implement the chat interface. The default inference code takes an object array like [transformers' conversations](https://huggingface.co/docs/transformers/main/en/conversations) and responds to the generated text. The following code is an example using the AWS SDK for JavaScript v3.\n\n```ts\nimport {\n  InvokeEndpointCommand,\n  SageMakerRuntimeClient,\n} from \"@aws-sdk/client-sagemaker-runtime\";\n\nconst client = new SageMakerRuntimeClient({\n  region: \"us-east-1\",\n});\nclient\n  .send(\n    new InvokeEndpointCommand({\n      EndpointName: \"my-endpoint-id\",\n      Body: JSON.stringify({\n        // Optional. You can change answer role.\n        role: \"ai\",\n        // Require. The messages like conversation.\n        messages: [\n          {\n            role: \"system\",\n            content: `You are helpfull assistant.`,\n          },\n          {\n            role: \"user\",\n            content:\n              \"please answer '1+1=?'. You must answer only answer numeric.\",\n          },\n        ],\n      }),\n      ContentType: \"application/json\",\n      Accept: \"application/json\",\n    }),\n  )\n  .then((res) => {\n    // { generated_text: \"2\" }\n    console.log(JSON.parse(res.Body.transformToString()));\n  });\n```\n\nTo change your own inference code, you can pass the code source.\n\n```ts\nimport * as s3Deplyment from \"aws-cdk-lib/aws-s3-deployment\";\n\ndeclare const compile: NeuronxCompile;\nnew TransformersNeuronxSageMakerRealtimeInferenceEndpoint(\n  this,\n  \"RealtimeInference\",\n  {\n    modelData:\n      TransformersNeuronxSageMakerInferenceModelData.fromNeuronxCompile(\n        compile,\n        s3Deplyment.Source.asset(\"path/to/my/code/directory\"),\n      ),\n  },\n);\n```\n\n## Neuronx Compile\n\n> [!WARNING]\n> This construct uses an Inferentia2 instance on EC2. You may need to increase your request limit for your AWS account.\n\nThis construct compiles models supported by Neuronx and uploads them to the specified S3 bucket. To define\n\n```ts\nimport * as ec2 from \"aws-cdk-lib/aws-ec2\";\nimport * as s3 from \"aws-cdk-lib/aws-s3\";\n\ndeclare const vpc: ec2.Vpc;\ndeclare const bucket: s3.Bucket;\nconst compile = new NeuronxCompile(this, \"NeuronxCompile\", {\n  vpc,\n  bucket,\n  model: Model.fromHuggingFace(\"example/example-7b-chat\", {\n    parameters: Parameters.billion(7),\n  }),\n});\n\n// Get the compiled artifacts from this S3 URL\nnew CfnOutput(this, \"CompiledArtifact\", {\n  value: compile.compiledArtifactS3Url,\n});\n```\n\nThis construct assumes the required instance type depending on the number of model parameters.\n\nAfter compiled, you can see like the this file tree in the S3 bucket.\n\n```txt\n{compiledArtifactS3Url}/\n├── model\n│   ├── config.json\n│   ├── tokenizer_config.json\n│   ├── xxx.safetensors\n│   └── xxx.safetensors\n└── compiled\n    ├── xxx.neff\n    ├── xxx.neff\n    └── xxx.neff\n```\n\nThis is NeuronxCompile architecture.\n![NeuronxCompile architecture](./docs/neuronx-compile-architecture.png)\n\n### Spot Instance\n\n> [!WARNING]\n> If you use Spot Instances, check if the request limit for Spot has been increased.\n\nYou can also use Spot Instances.\n\n```ts\nimport * as ec2 from \"aws-cdk-lib/aws-ec2\";\nimport * as s3 from \"aws-cdk-lib/aws-s3\";\n\ndeclare const vpc: ec2.Vpc;\ndeclare const bucket: s3.Bucket;\nnew NeuronxCompile(this, \"NeuronxCompile\", {\n  vpc,\n  bucket,\n  model: Model.fromHuggingFace(\"example/example-7b-chat\", {\n    parameters: Parameters.billion(7),\n  }),\n  spot: true,\n});\n```\n\n### Compile Options\n\nIf you are familiar with Neuronx, you can also specify compilation options to better meet your requirements.\n\n```ts\nimport * as ec2 from \"aws-cdk-lib/aws-ec2\";\nimport * as s3 from \"aws-cdk-lib/aws-s3\";\n\ndeclare const vpc: ec2.Vpc;\ndeclare const bucket: s3.Bucket;\nnew NeuronxCompile(this, \"NeuronxCompile\", {\n  vpc,\n  bucket,\n  model: Model.fromHuggingFace(\"example/example-22b-chat\", {\n    parameters: Parameters.billion(22),\n  }),\n  compileOptions: {\n    nPositions: 1024,\n    quantDtype: QuantDtype.S8,\n    optLevel: OptLevel.MODEL_EXECUTION_PERFORMANCE,\n  },\n});\n```\n"
   },
   "repository": {
     "type": "git",
@@ -3832,6 +3864,42 @@
     }
   },
   "types": {
+    "aws-cdk-neuronx-patterns.BucketCompiledModelOptions": {
+      "assembly": "aws-cdk-neuronx-patterns",
+      "datatype": true,
+      "docs": {
+        "stability": "stable"
+      },
+      "fqn": "aws-cdk-neuronx-patterns.BucketCompiledModelOptions",
+      "interfaces": [
+        "aws-cdk-neuronx-patterns.CompiledModelOptions"
+      ],
+      "kind": "interface",
+      "locationInModule": {
+        "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+        "line": 44
+      },
+      "name": "BucketCompiledModelOptions",
+      "properties": [
+        {
+          "abstract": true,
+          "docs": {
+            "stability": "stable",
+            "summary": "The number of parameters of model."
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 48
+          },
+          "name": "parameters",
+          "type": {
+            "fqn": "aws-cdk-neuronx-patterns.Parameters"
+          }
+        }
+      ],
+      "symbolId": "src/transformers-neuronx-sagemaker-realtime-inference:BucketCompiledModelOptions"
+    },
     "aws-cdk-neuronx-patterns.CompileOptions": {
       "assembly": "aws-cdk-neuronx-patterns",
       "datatype": true,
@@ -3842,21 +3910,21 @@
       "fqn": "aws-cdk-neuronx-patterns.CompileOptions",
       "kind": "interface",
       "locationInModule": {
-        "filename": "src/neuronx-compile.ts",
-        "line": 60
+        "filename": "src/model.ts",
+        "line": 34
       },
       "name": "CompileOptions",
       "properties": [
         {
           "abstract": true,
           "docs": {
-            "default": "4092",
+            "default": "4096",
             "stability": "stable"
           },
           "immutable": true,
           "locationInModule": {
-            "filename": "src/neuronx-compile.ts",
-            "line": 72
+            "filename": "src/model.ts",
+            "line": 46
           },
           "name": "nPositions",
           "optional": true,
@@ -3872,8 +3940,8 @@
           },
           "immutable": true,
           "locationInModule": {
-            "filename": "src/neuronx-compile.ts",
-            "line": 76
+            "filename": "src/model.ts",
+            "line": 50
           },
           "name": "optLevel",
           "optional": true,
@@ -3889,8 +3957,8 @@
           },
           "immutable": true,
           "locationInModule": {
-            "filename": "src/neuronx-compile.ts",
-            "line": 68
+            "filename": "src/model.ts",
+            "line": 42
           },
           "name": "quantDtype",
           "optional": true,
@@ -3906,8 +3974,8 @@
           },
           "immutable": true,
           "locationInModule": {
-            "filename": "src/neuronx-compile.ts",
-            "line": 64
+            "filename": "src/model.ts",
+            "line": 38
           },
           "name": "tpDegree",
           "optional": true,
@@ -3916,7 +3984,7 @@
           }
         }
       ],
-      "symbolId": "src/neuronx-compile:CompileOptions"
+      "symbolId": "src/model:CompileOptions"
     },
     "aws-cdk-neuronx-patterns.CompileRuntime": {
       "assembly": "aws-cdk-neuronx-patterns",
@@ -3929,7 +3997,7 @@
       "kind": "interface",
       "locationInModule": {
         "filename": "src/neuronx-compile.ts",
-        "line": 18
+        "line": 25
       },
       "name": "CompileRuntime",
       "properties": [
@@ -3942,7 +4010,7 @@
           "immutable": true,
           "locationInModule": {
             "filename": "src/neuronx-compile.ts",
-            "line": 22
+            "line": 29
           },
           "name": "image",
           "type": {
@@ -3958,7 +4026,7 @@
           "immutable": true,
           "locationInModule": {
             "filename": "src/neuronx-compile.ts",
-            "line": 26
+            "line": 33
           },
           "name": "neuronxVersion",
           "type": {
@@ -3968,6 +4036,96 @@
       ],
       "symbolId": "src/neuronx-compile:CompileRuntime"
     },
+    "aws-cdk-neuronx-patterns.CompiledModelOptions": {
+      "assembly": "aws-cdk-neuronx-patterns",
+      "datatype": true,
+      "docs": {
+        "stability": "stable",
+        "summary": "Precompiled model options."
+      },
+      "fqn": "aws-cdk-neuronx-patterns.CompiledModelOptions",
+      "kind": "interface",
+      "locationInModule": {
+        "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+        "line": 21
+      },
+      "name": "CompiledModelOptions",
+      "properties": [
+        {
+          "abstract": true,
+          "docs": {
+            "default": "- using the predefined code",
+            "stability": "stable",
+            "summary": "Code used for inference."
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 31
+          },
+          "name": "code",
+          "optional": true,
+          "type": {
+            "fqn": "aws-cdk-lib.aws_s3_deployment.ISource"
+          }
+        },
+        {
+          "abstract": true,
+          "docs": {
+            "default": "\"./compiled\"",
+            "stability": "stable",
+            "summary": "The path where compiled artifacts (i.e. xxx.neff) are stored."
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 41
+          },
+          "name": "compiledArtifactPath",
+          "optional": true,
+          "type": {
+            "primitive": "string"
+          }
+        },
+        {
+          "abstract": true,
+          "docs": {
+            "default": "- Each properties are set default.",
+            "stability": "stable",
+            "summary": "Neuronx compile options."
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 26
+          },
+          "name": "compileOptions",
+          "optional": true,
+          "type": {
+            "fqn": "aws-cdk-neuronx-patterns.CompileOptions"
+          }
+        },
+        {
+          "abstract": true,
+          "docs": {
+            "default": "\"./model\"",
+            "stability": "stable",
+            "summary": "Model ID or saved path."
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 36
+          },
+          "name": "modelIdOrPath",
+          "optional": true,
+          "type": {
+            "primitive": "string"
+          }
+        }
+      ],
+      "symbolId": "src/transformers-neuronx-sagemaker-realtime-inference:CompiledModelOptions"
+    },
     "aws-cdk-neuronx-patterns.IAcceleratorChips": {
       "assembly": "aws-cdk-neuronx-patterns",
       "docs": {
@@ -4119,10 +4277,57 @@
       "fqn": "aws-cdk-neuronx-patterns.Model",
       "kind": "class",
       "locationInModule": {
-        "filename": "src/neuronx-compile.ts",
-        "line": 110
+        "filename": "src/model.ts",
+        "line": 84
       },
       "methods": [
+        {
+          "docs": {
+            "returns": "model instance",
+            "stability": "stable",
+            "summary": "model informations at S3 Bucket."
+          },
+          "locationInModule": {
+            "filename": "src/model.ts",
+            "line": 101
+          },
+          "name": "fromBucket",
+          "parameters": [
+            {
+              "docs": {
+                "summary": "Model stored S3 Bucket."
+              },
+              "name": "bucket",
+              "type": {
+                "fqn": "aws-cdk-lib.aws_s3.IBucket"
+              }
+            },
+            {
+              "docs": {
+                "summary": "Model stored objects prefix."
+              },
+              "name": "prefix",
+              "type": {
+                "primitive": "string"
+              }
+            },
+            {
+              "docs": {
+                "summary": "model basic infromation."
+              },
+              "name": "options",
+              "type": {
+                "fqn": "aws-cdk-neuronx-patterns.ModelOptions"
+              }
+            }
+          ],
+          "returns": {
+            "type": {
+              "fqn": "aws-cdk-neuronx-patterns.Model"
+            }
+          },
+          "static": true
+        },
         {
           "docs": {
             "returns": "model instance",
@@ -4130,8 +4335,8 @@
             "summary": "model informations at HuggingFace."
           },
           "locationInModule": {
-            "filename": "src/neuronx-compile.ts",
-            "line": 117
+            "filename": "src/model.ts",
+            "line": 91
           },
           "name": "fromHuggingFace",
           "parameters": [
@@ -4170,8 +4375,8 @@
           },
           "immutable": true,
           "locationInModule": {
-            "filename": "src/neuronx-compile.ts",
-            "line": 121
+            "filename": "src/model.ts",
+            "line": 105
           },
           "name": "modelId",
           "type": {
@@ -4184,16 +4389,46 @@
           },
           "immutable": true,
           "locationInModule": {
-            "filename": "src/neuronx-compile.ts",
-            "line": 122
+            "filename": "src/model.ts",
+            "line": 106
           },
           "name": "options",
           "type": {
             "fqn": "aws-cdk-neuronx-patterns.ModelOptions"
           }
+        },
+        {
+          "docs": {
+            "stability": "stable"
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/model.ts",
+            "line": 107
+          },
+          "name": "bucket",
+          "optional": true,
+          "type": {
+            "fqn": "aws-cdk-lib.aws_s3.IBucket"
+          }
+        },
+        {
+          "docs": {
+            "stability": "stable"
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/model.ts",
+            "line": 108
+          },
+          "name": "prefix",
+          "optional": true,
+          "type": {
+            "primitive": "string"
+          }
         }
       ],
-      "symbolId": "src/neuronx-compile:Model"
+      "symbolId": "src/model:Model"
     },
     "aws-cdk-neuronx-patterns.ModelOptions": {
       "assembly": "aws-cdk-neuronx-patterns",
@@ -4205,8 +4440,8 @@
       "fqn": "aws-cdk-neuronx-patterns.ModelOptions",
       "kind": "interface",
       "locationInModule": {
-        "filename": "src/neuronx-compile.ts",
-        "line": 104
+        "filename": "src/model.ts",
+        "line": 78
       },
       "name": "ModelOptions",
       "properties": [
@@ -4217,8 +4452,8 @@
           },
           "immutable": true,
           "locationInModule": {
-            "filename": "src/neuronx-compile.ts",
-            "line": 105
+            "filename": "src/model.ts",
+            "line": 79
           },
           "name": "parameters",
           "type": {
@@ -4226,7 +4461,7 @@
           }
         }
       ],
-      "symbolId": "src/neuronx-compile:ModelOptions"
+      "symbolId": "src/model:ModelOptions"
     },
     "aws-cdk-neuronx-patterns.NeuronxCompile": {
       "assembly": "aws-cdk-neuronx-patterns",
@@ -4243,7 +4478,7 @@
         },
         "locationInModule": {
           "filename": "src/neuronx-compile.ts",
-          "line": 182
+          "line": 102
         },
         "parameters": [
           {
@@ -4269,124 +4504,224 @@
       "kind": "class",
       "locationInModule": {
         "filename": "src/neuronx-compile.ts",
-        "line": 177
+        "line": 87
       },
       "name": "NeuronxCompile",
       "properties": [
         {
           "docs": {
-            "stability": "stable",
-            "summary": "S3 URL that compiled artifact uploaded."
+            "stability": "stable"
           },
           "immutable": true,
           "locationInModule": {
             "filename": "src/neuronx-compile.ts",
-            "line": 181
+            "line": 88
           },
-          "name": "compiledArtifactS3Url",
+          "name": "compiledArtifactS3Bucket",
           "type": {
-            "primitive": "string"
+            "fqn": "aws-cdk-lib.aws_s3.IBucket"
           }
-        }
-      ],
-      "symbolId": "src/neuronx-compile:NeuronxCompile"
-    },
-    "aws-cdk-neuronx-patterns.NeuronxCompileProps": {
-      "assembly": "aws-cdk-neuronx-patterns",
-      "datatype": true,
-      "docs": {
-        "stability": "stable",
-        "summary": "Props of NeuronxCompile."
-      },
-      "fqn": "aws-cdk-neuronx-patterns.NeuronxCompileProps",
-      "kind": "interface",
-      "locationInModule": {
-        "filename": "src/neuronx-compile.ts",
-        "line": 128
-      },
-      "name": "NeuronxCompileProps",
-      "properties": [
+        },
         {
-          "abstract": true,
           "docs": {
             "stability": "stable",
-            "summary": "The bucket to upload compiled artifacts."
+            "summary": "S3 Prefix that compiled artifact uploaded."
           },
           "immutable": true,
           "locationInModule": {
             "filename": "src/neuronx-compile.ts",
-            "line": 140
+            "line": 96
           },
-          "name": "bucket",
+          "name": "compiledArtifactS3Prefix",
           "type": {
-            "fqn": "aws-cdk-lib.aws_s3.IBucket"
+            "primitive": "string"
           }
         },
         {
-          "abstract": true,
           "docs": {
             "stability": "stable",
-            "summary": "The model to be compiled."
+            "summary": "S3 URL that compiled artifact uploaded."
           },
           "immutable": true,
           "locationInModule": {
             "filename": "src/neuronx-compile.ts",
-            "line": 144
+            "line": 92
           },
-          "name": "model",
+          "name": "compiledArtifactS3Url",
           "type": {
-            "fqn": "aws-cdk-neuronx-patterns.Model"
+            "primitive": "string"
           }
         },
         {
-          "abstract": true,
           "docs": {
-            "stability": "stable",
-            "summary": "VPC in which this will launch compile worker instance."
+            "stability": "stable"
           },
           "immutable": true,
           "locationInModule": {
             "filename": "src/neuronx-compile.ts",
-            "line": 132
+            "line": 99
           },
-          "name": "vpc",
+          "name": "nPositions",
           "type": {
-            "fqn": "aws-cdk-lib.aws_ec2.IVpc"
+            "primitive": "number"
           }
         },
         {
-          "abstract": true,
           "docs": {
-            "default": "- Each properties are set default.",
-            "stability": "stable",
-            "summary": "Neuronx compile options."
+            "stability": "stable"
           },
           "immutable": true,
           "locationInModule": {
             "filename": "src/neuronx-compile.ts",
-            "line": 159
+            "line": 100
           },
-          "name": "compileOptions",
-          "optional": true,
+          "name": "optLevel",
           "type": {
-            "fqn": "aws-cdk-neuronx-patterns.CompileOptions"
+            "fqn": "aws-cdk-neuronx-patterns.OptLevel"
           }
         },
         {
-          "abstract": true,
           "docs": {
-            "stability": "stable",
-            "summary": "The instance type of compile worker instance."
+            "stability": "stable"
           },
           "immutable": true,
           "locationInModule": {
             "filename": "src/neuronx-compile.ts",
-            "line": 136
+            "line": 101
           },
-          "name": "instanceType",
-          "optional": true,
+          "name": "parameters",
           "type": {
-            "fqn": "aws-cdk-neuronx-patterns.NeuronxInstanceType"
+            "fqn": "aws-cdk-neuronx-patterns.Parameters"
+          }
+        },
+        {
+          "docs": {
+            "stability": "stable"
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/neuronx-compile.ts",
+            "line": 97
+          },
+          "name": "tpDegree",
+          "type": {
+            "primitive": "number"
+          }
+        },
+        {
+          "docs": {
+            "stability": "stable"
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/neuronx-compile.ts",
+            "line": 98
+          },
+          "name": "quantDtype",
+          "optional": true,
+          "type": {
+            "fqn": "aws-cdk-neuronx-patterns.QuantDtype"
+          }
+        }
+      ],
+      "symbolId": "src/neuronx-compile:NeuronxCompile"
+    },
+    "aws-cdk-neuronx-patterns.NeuronxCompileProps": {
+      "assembly": "aws-cdk-neuronx-patterns",
+      "datatype": true,
+      "docs": {
+        "stability": "stable",
+        "summary": "Props of NeuronxCompile."
+      },
+      "fqn": "aws-cdk-neuronx-patterns.NeuronxCompileProps",
+      "kind": "interface",
+      "locationInModule": {
+        "filename": "src/neuronx-compile.ts",
+        "line": 38
+      },
+      "name": "NeuronxCompileProps",
+      "properties": [
+        {
+          "abstract": true,
+          "docs": {
+            "stability": "stable",
+            "summary": "The bucket to upload compiled artifacts."
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/neuronx-compile.ts",
+            "line": 46
+          },
+          "name": "bucket",
+          "type": {
+            "fqn": "aws-cdk-lib.aws_s3.IBucket"
+          }
+        },
+        {
+          "abstract": true,
+          "docs": {
+            "stability": "stable",
+            "summary": "The model to be compiled."
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/neuronx-compile.ts",
+            "line": 50
+          },
+          "name": "model",
+          "type": {
+            "fqn": "aws-cdk-neuronx-patterns.Model"
+          }
+        },
+        {
+          "abstract": true,
+          "docs": {
+            "stability": "stable",
+            "summary": "VPC in which this will launch compile worker instance."
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/neuronx-compile.ts",
+            "line": 42
+          },
+          "name": "vpc",
+          "type": {
+            "fqn": "aws-cdk-lib.aws_ec2.IVpc"
+          }
+        },
+        {
+          "abstract": true,
+          "docs": {
+            "default": "- Each properties are set default.",
+            "stability": "stable",
+            "summary": "Neuronx compile options."
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/neuronx-compile.ts",
+            "line": 69
+          },
+          "name": "compileOptions",
+          "optional": true,
+          "type": {
+            "fqn": "aws-cdk-neuronx-patterns.CompileOptions"
+          }
+        },
+        {
+          "abstract": true,
+          "docs": {
+            "stability": "stable",
+            "summary": "The instance type of compile worker instance."
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/neuronx-compile.ts",
+            "line": 54
+          },
+          "name": "instanceType",
+          "optional": true,
+          "type": {
+            "fqn": "aws-cdk-neuronx-patterns.NeuronxInstanceType"
           }
         },
         {
@@ -4399,7 +4734,7 @@
           "immutable": true,
           "locationInModule": {
             "filename": "src/neuronx-compile.ts",
-            "line": 154
+            "line": 64
           },
           "name": "runtime",
           "optional": true,
@@ -4418,7 +4753,7 @@
           "immutable": true,
           "locationInModule": {
             "filename": "src/neuronx-compile.ts",
-            "line": 165
+            "line": 75
           },
           "name": "spot",
           "optional": true,
@@ -4436,7 +4771,7 @@
           "immutable": true,
           "locationInModule": {
             "filename": "src/neuronx-compile.ts",
-            "line": 149
+            "line": 59
           },
           "name": "volumeSize",
           "optional": true,
@@ -4454,7 +4789,7 @@
           "immutable": true,
           "locationInModule": {
             "filename": "src/neuronx-compile.ts",
-            "line": 171
+            "line": 81
           },
           "name": "vpcSubnets",
           "optional": true,
@@ -4633,8 +4968,8 @@
       "fqn": "aws-cdk-neuronx-patterns.OptLevel",
       "kind": "enum",
       "locationInModule": {
-        "filename": "src/neuronx-compile.ts",
-        "line": 42
+        "filename": "src/model.ts",
+        "line": 16
       },
       "members": [
         {
@@ -4660,7 +4995,7 @@
         }
       ],
       "name": "OptLevel",
-      "symbolId": "src/neuronx-compile:OptLevel"
+      "symbolId": "src/model:OptLevel"
     },
     "aws-cdk-neuronx-patterns.Parameters": {
       "assembly": "aws-cdk-neuronx-patterns",
@@ -4671,8 +5006,8 @@
       "fqn": "aws-cdk-neuronx-patterns.Parameters",
       "kind": "class",
       "locationInModule": {
-        "filename": "src/neuronx-compile.ts",
-        "line": 82
+        "filename": "src/model.ts",
+        "line": 56
       },
       "methods": [
         {
@@ -4682,8 +5017,8 @@
             "summary": "Create a Parameters representing an amount bilion."
           },
           "locationInModule": {
-            "filename": "src/neuronx-compile.ts",
-            "line": 88
+            "filename": "src/model.ts",
+            "line": 62
           },
           "name": "billion",
           "parameters": [
@@ -4711,8 +5046,8 @@
             "summary": "Return this number of parameters as bilion."
           },
           "locationInModule": {
-            "filename": "src/neuronx-compile.ts",
-            "line": 96
+            "filename": "src/model.ts",
+            "line": 70
           },
           "name": "toBilion",
           "returns": {
@@ -4723,7 +5058,7 @@
         }
       ],
       "name": "Parameters",
-      "symbolId": "src/neuronx-compile:Parameters"
+      "symbolId": "src/model:Parameters"
     },
     "aws-cdk-neuronx-patterns.QuantDtype": {
       "assembly": "aws-cdk-neuronx-patterns",
@@ -4734,8 +5069,8 @@
       "fqn": "aws-cdk-neuronx-patterns.QuantDtype",
       "kind": "enum",
       "locationInModule": {
-        "filename": "src/neuronx-compile.ts",
-        "line": 32
+        "filename": "src/model.ts",
+        "line": 6
       },
       "members": [
         {
@@ -4747,9 +5082,493 @@
         }
       ],
       "name": "QuantDtype",
-      "symbolId": "src/neuronx-compile:QuantDtype"
+      "symbolId": "src/model:QuantDtype"
+    },
+    "aws-cdk-neuronx-patterns.TransformersNeuronxSageMakerInferenceModelData": {
+      "assembly": "aws-cdk-neuronx-patterns",
+      "docs": {
+        "stability": "stable"
+      },
+      "fqn": "aws-cdk-neuronx-patterns.TransformersNeuronxSageMakerInferenceModelData",
+      "kind": "class",
+      "locationInModule": {
+        "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+        "line": 51
+      },
+      "methods": [
+        {
+          "docs": {
+            "stability": "stable"
+          },
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 52
+          },
+          "name": "fromBucket",
+          "parameters": [
+            {
+              "name": "bucket",
+              "type": {
+                "fqn": "aws-cdk-lib.aws_s3.IBucket"
+              }
+            },
+            {
+              "name": "prefix",
+              "type": {
+                "primitive": "string"
+              }
+            },
+            {
+              "name": "options",
+              "type": {
+                "fqn": "aws-cdk-neuronx-patterns.BucketCompiledModelOptions"
+              }
+            }
+          ],
+          "returns": {
+            "type": {
+              "fqn": "aws-cdk-neuronx-patterns.TransformersNeuronxSageMakerInferenceModelData"
+            }
+          },
+          "static": true
+        },
+        {
+          "docs": {
+            "stability": "stable"
+          },
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 78
+          },
+          "name": "fromNeuronxCompile",
+          "parameters": [
+            {
+              "name": "compile",
+              "type": {
+                "fqn": "aws-cdk-neuronx-patterns.NeuronxCompile"
+              }
+            },
+            {
+              "name": "code",
+              "optional": true,
+              "type": {
+                "fqn": "aws-cdk-lib.aws_s3_deployment.ISource"
+              }
+            }
+          ],
+          "returns": {
+            "type": {
+              "fqn": "aws-cdk-neuronx-patterns.TransformersNeuronxSageMakerInferenceModelData"
+            }
+          },
+          "static": true
+        }
+      ],
+      "name": "TransformersNeuronxSageMakerInferenceModelData",
+      "properties": [
+        {
+          "docs": {
+            "stability": "stable"
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 86
+          },
+          "name": "bucket",
+          "type": {
+            "fqn": "aws-cdk-lib.aws_s3.IBucket"
+          }
+        },
+        {
+          "docs": {
+            "stability": "stable"
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 88
+          },
+          "name": "code",
+          "type": {
+            "fqn": "aws-cdk-lib.aws_s3_deployment.ISource"
+          }
+        },
+        {
+          "docs": {
+            "stability": "stable"
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 87
+          },
+          "name": "compiledArtifactS3Prefix",
+          "type": {
+            "primitive": "string"
+          }
+        },
+        {
+          "docs": {
+            "stability": "stable"
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 91
+          },
+          "name": "nPositions",
+          "type": {
+            "primitive": "number"
+          }
+        },
+        {
+          "docs": {
+            "stability": "stable"
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 92
+          },
+          "name": "optLevel",
+          "type": {
+            "fqn": "aws-cdk-neuronx-patterns.OptLevel"
+          }
+        },
+        {
+          "docs": {
+            "stability": "stable"
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 95
+          },
+          "name": "parameters",
+          "type": {
+            "fqn": "aws-cdk-neuronx-patterns.Parameters"
+          }
+        },
+        {
+          "docs": {
+            "stability": "stable"
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 89
+          },
+          "name": "tpDegree",
+          "type": {
+            "primitive": "number"
+          }
+        },
+        {
+          "docs": {
+            "stability": "stable"
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 94
+          },
+          "name": "compiledArtifactPath",
+          "optional": true,
+          "type": {
+            "primitive": "string"
+          }
+        },
+        {
+          "docs": {
+            "stability": "stable"
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 93
+          },
+          "name": "modelIdOrPath",
+          "optional": true,
+          "type": {
+            "primitive": "string"
+          }
+        },
+        {
+          "docs": {
+            "stability": "stable"
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 90
+          },
+          "name": "quantDtype",
+          "optional": true,
+          "type": {
+            "fqn": "aws-cdk-neuronx-patterns.QuantDtype"
+          }
+        }
+      ],
+      "symbolId": "src/transformers-neuronx-sagemaker-realtime-inference:TransformersNeuronxSageMakerInferenceModelData"
+    },
+    "aws-cdk-neuronx-patterns.TransformersNeuronxSageMakerRealtimeInferenceEndpoint": {
+      "assembly": "aws-cdk-neuronx-patterns",
+      "base": "constructs.Construct",
+      "docs": {
+        "stability": "stable"
+      },
+      "fqn": "aws-cdk-neuronx-patterns.TransformersNeuronxSageMakerRealtimeInferenceEndpoint",
+      "initializer": {
+        "docs": {
+          "stability": "stable"
+        },
+        "locationInModule": {
+          "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+          "line": 179
+        },
+        "parameters": [
+          {
+            "name": "scope",
+            "type": {
+              "fqn": "constructs.Construct"
+            }
+          },
+          {
+            "name": "id",
+            "type": {
+              "primitive": "string"
+            }
+          },
+          {
+            "name": "props",
+            "type": {
+              "fqn": "aws-cdk-neuronx-patterns.TransformersNeuronxSageMakerRealtimeInferenceEndpointProps"
+            }
+          }
+        ]
+      },
+      "kind": "class",
+      "locationInModule": {
+        "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+        "line": 167
+      },
+      "methods": [
+        {
+          "docs": {
+            "stability": "stable"
+          },
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 287
+          },
+          "name": "grantInvoke",
+          "parameters": [
+            {
+              "name": "grantee",
+              "type": {
+                "fqn": "aws-cdk-lib.aws_iam.IGrantable"
+              }
+            }
+          ],
+          "returns": {
+            "type": {
+              "fqn": "aws-cdk-lib.aws_iam.Grant"
+            }
+          }
+        }
+      ],
+      "name": "TransformersNeuronxSageMakerRealtimeInferenceEndpoint",
+      "properties": [
+        {
+          "docs": {
+            "custom": {
+              "attribute": "true"
+            },
+            "stability": "stable",
+            "summary": "The ARN of the endpoint."
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 172
+          },
+          "name": "endpointArn",
+          "type": {
+            "primitive": "string"
+          }
+        },
+        {
+          "docs": {
+            "custom": {
+              "attribute": "true"
+            },
+            "stability": "stable",
+            "summary": "The name of the endpoint."
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 177
+          },
+          "name": "endpointName",
+          "type": {
+            "primitive": "string"
+          }
+        }
+      ],
+      "symbolId": "src/transformers-neuronx-sagemaker-realtime-inference:TransformersNeuronxSageMakerRealtimeInferenceEndpoint"
+    },
+    "aws-cdk-neuronx-patterns.TransformersNeuronxSageMakerRealtimeInferenceEndpointProps": {
+      "assembly": "aws-cdk-neuronx-patterns",
+      "datatype": true,
+      "docs": {
+        "stability": "stable"
+      },
+      "fqn": "aws-cdk-neuronx-patterns.TransformersNeuronxSageMakerRealtimeInferenceEndpointProps",
+      "kind": "interface",
+      "locationInModule": {
+        "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+        "line": 126
+      },
+      "name": "TransformersNeuronxSageMakerRealtimeInferenceEndpointProps",
+      "properties": [
+        {
+          "abstract": true,
+          "docs": {
+            "remarks": "The model data requires at least compiled artifacts.",
+            "stability": "stable",
+            "summary": "Model data for SageMaker inference."
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 131
+          },
+          "name": "modelData",
+          "type": {
+            "fqn": "aws-cdk-neuronx-patterns.TransformersNeuronxSageMakerInferenceModelData"
+          }
+        },
+        {
+          "abstract": true,
+          "docs": {
+            "default": "- 60 seconds, when set the `modelDataDownloadTimeout` then use same value (max 60 minutes)",
+            "see": "https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-inference-code.html#your-algorithms-inference-algo-ping-requests",
+            "stability": "stable",
+            "summary": "The timeout value, for your inference container to pass health check by SageMaker Hosting."
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 164
+          },
+          "name": "containerStartupHealthCheckTimeout",
+          "optional": true,
+          "type": {
+            "fqn": "aws-cdk-lib.Duration"
+          }
+        },
+        {
+          "abstract": true,
+          "docs": {
+            "default": "- Only the predefined environment variables required to use Neuronx have been set.",
+            "stability": "stable",
+            "summary": "A map of environment variables to pass into the container."
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 140
+          },
+          "name": "environment",
+          "optional": true,
+          "type": {
+            "collection": {
+              "elementtype": {
+                "primitive": "string"
+              },
+              "kind": "map"
+            }
+          }
+        },
+        {
+          "abstract": true,
+          "docs": {
+            "stability": "stable",
+            "summary": "An image of the container where the inference job is executed."
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 135
+          },
+          "name": "image",
+          "optional": true,
+          "type": {
+            "fqn": "@aws-cdk/aws-sagemaker-alpha.ContainerImage"
+          }
+        },
+        {
+          "abstract": true,
+          "docs": {
+            "default": "- It is determined automatically according to the number of model parameters and compilation options.",
+            "stability": "stable",
+            "summary": "The instance type of compile worker instance."
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 145
+          },
+          "name": "instanceType",
+          "optional": true,
+          "type": {
+            "fqn": "aws-cdk-neuronx-patterns.NeuronxInstanceType"
+          }
+        },
+        {
+          "abstract": true,
+          "docs": {
+            "default": "- 60 seconds, when `volumeSize` larger than 30GB then 1GB x 15 seconds (max 60 minutes)",
+            "stability": "stable",
+            "summary": "The timeout value, to download and extract the model that you want to host from Amazon S3 to the individual inference instance associated with this production variant."
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 158
+          },
+          "name": "modelDataDownloadTimeout",
+          "optional": true,
+          "type": {
+            "fqn": "aws-cdk-lib.Duration"
+          }
+        },
+        {
+          "abstract": true,
+          "docs": {
+            "default": "- 2.5 GB per billion parameter (Max 512 GB)",
+            "remarks": "Currently only Amazon EBS gp2 storage volumes are supported.",
+            "see": "https://aws.amazon.com/jp/releasenotes/host-instance-storage-volumes-table",
+            "stability": "stable",
+            "summary": "The size, of the ML storage volume attached to individual inference instance associated with the production variant."
+          },
+          "immutable": true,
+          "locationInModule": {
+            "filename": "src/transformers-neuronx-sagemaker-realtime-inference.ts",
+            "line": 152
+          },
+          "name": "volumeSize",
+          "optional": true,
+          "type": {
+            "fqn": "aws-cdk-lib.Size"
+          }
+        }
+      ],
+      "symbolId": "src/transformers-neuronx-sagemaker-realtime-inference:TransformersNeuronxSageMakerRealtimeInferenceEndpointProps"
     }
   },
-  "version": "0.0.2",
-  "fingerprint": "p6gVUQb1hvNV7Eh7GaBS3XMbGzlWxaqBx9dV/sS1KX0="
+  "version": "0.0.4",
+  "fingerprint": "XO2QlDqW3bojQGczF18xCsjcRr+vPLQOWOpbuoVbGs4="
 }