llama-stack 0.4.3__py3-none-any.whl → 0.5.0rc1__py3-none-any.whl

llama_stack/models/llama/llama3_2/vision_prompt_format.md

@@ -1,141 +0,0 @@
-## User and assistant conversation
-
-Here is a regular multi-turn user assistant conversation and how its formatted.
-
-##### Input Prompt Format
-```
-<|begin_of_text|><|start_header_id|>system<|end_header_id|>
-
-You are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>
-
-Who are you?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
-
-
-```
-
-##### Model Response Format
-```
-I'm a helpful assistant, here to provide information, answer questions, and assist with tasks to the best of my abilities. I'm a large language model, which means I can understand and respond to natural language inputs, and I'm constantly learning and improving to provide more accurate and helpful responses.
-
-I can help with a wide range of topics, from general knowledge and trivia to more specific areas like science, history, technology, and more. I can also assist with tasks like language translation, text summarization, and even generating creative content like stories or dialogues.
-
-What can I help you with today?<|eot_id|>
-```
-
-
-##### Notes
-This format is unchanged from Llama3.1
-
-## User and assistant conversation with Images
-
-This example shows how to pass and image to the model as part of the messages.
-
-##### Input Prompt Format
-```
-<|begin_of_text|><|start_header_id|>user<|end_header_id|>
-
-<|image|>Describe this image in two sentences<|eot_id|><|start_header_id|>assistant<|end_header_id|>
-
-
-```
-
-##### Model Response Format
-```
-The image depicts a small dog standing on a skateboard, with its front paws firmly planted on the board and its back paws slightly raised. The dog's fur is predominantly brown and white, with a distinctive black stripe running down its back, and it is wearing a black collar around its neck.<|eot_id|>
-```
-
-
-##### Notes
-
-- The `<|image|>` tag is used to indicate presence of the image
-- The model isn't an early fusion model so doesn't actually translate an image into several tokens. Instead the cross-attention layers take input "on the side" from a vision encoder
-![Image](mm-model.png)
-- Its important to postion the <|image|> tag appropriately in the prompt. Image will only attend to the subsequent text tokens
-- The <|image|> tag is part of the user message body, implying that it should only come after the header `<|start_header_id|>{role}<|end_header_id|>` in the message body
-- We recommend using a single image in one prompt
-
-
-## Builtin and Zero Shot Tool Calling
-
-
-Llama3.2 vision models follow the same tool calling format as Llama3.1 models when inputs are text only.
-Use `Environment: ipython` to enable tools.
-Add `Tools: {{tool_name1}},{{tool_name2}}` for each of the builtin tools.
-The same builtin tools as Llama3.1 are available,
-- code_interpreter (for executing python code)
-- brave_search (to search the web)
-- wolfram_alpha (for querying wolfram alpha for mathematical questions)
-
-
-##### Input Prompt Format
-```
-<|begin_of_text|><|start_header_id|>system<|end_header_id|>
-
-Environment: ipython
-Tools: brave_search, wolfram_alpha
-Cutting Knowledge Date: December 2023
-Today Date: 23 September 2024
-
-You are a helpful assistant.
-<|eot_id|><|start_header_id|>user<|end_header_id|>
-
-Search the web for the latest price of 1oz gold?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
-
-
-```
-
-##### Model Response Format
-```
-<|python_tag|>brave_search.call(query="latest price of 1oz gold")<|eom_id|>
-```
-
-
-##### Notes
-
-- Note the `<|python_tag|>` before `brave_search` function call.
-- The `<|eom_id|>` tag is used to indicate the end of the message.
-- Similar to Llama3.1, code_interpreter is not explicitly mentioned but is enabled via `Environment: ipython`.
-- Tool Calling does NOT work with images in the prompt as of now.
-
-
-## Prompt format for base models
-
-
-For base models (Llama3.2-11B-Vision and Llama3.2-90B-Vision), the prompt format for a simple completion is as follows
-
-
-##### Input Prompt Format
-```
-<|begin_of_text|>The color of the sky is blue but sometimes it can also be
-```
-
-##### Model Response Format
-```
- red, orange, pink, purple, and even black. The color of the sky is determined by the amount of sunlight that is scattered by the atmosphere and the amount of dust and water vapor present in the atmosphere. During sunrise and sunset, the sky can take on a range of colors due to the scattering of light by
-```
-
-
-##### Notes
-- Same as Llama3.1
-
-## Prompt format for base models with Image
-
-
-For base models (Llama3.2-11B-Vision and Llama3.2-90B-Vision), here is an example of how the text completion format looks with an image,
-
-
-##### Input Prompt Format
-```
-<|begin_of_text|><|image|>If I had to write a haiku for this one
-```
-
-##### Model Response Format
-```
-, it would be: A skateboarder's delight, a puppy on a board, a furry little thrill-seeker. This puppy is a true skateboarding enthusiast, always eager to hit the streets and show off his skills. He's a master of the board, gliding effortlessly across the pavement with grace and style.
-```
-
-
-##### Notes
-- Note the placement of the special tags <|begin_of_text|> and <|image|>
-
-Thank You!

llama_stack/models/llama/llama3_3/__init__.py

@@ -1,5 +0,0 @@
-# Copyright (c) Meta Platforms, Inc. and affiliates.
-# All rights reserved.
-#
-# This source code is licensed under the terms described in the LICENSE file in
-# the root directory of this source tree.

llama_stack/models/llama/llama3_3/prompts.py

@@ -1,259 +0,0 @@
-# Copyright (c) Meta Platforms, Inc. and affiliates.
-# All rights reserved.
-#
-# This source code is licensed under the terms described in the LICENSE file in
-# the root directory of this source tree.
-
-# Copyright (c) Meta Platforms, Inc. and affiliates.
-# All rights reserved.
-#
-# This source code is licensed under the terms described in the LICENSE file in
-# top-level folder for each specific model found within the models/ directory at
-# the top-level of this source tree.
-
-import json
-import textwrap
-
-from llama_stack.models.llama.datatypes import (
-    BuiltinTool,
-    RawMessage,
-    StopReason,
-    ToolCall,
-    ToolPromptFormat,
-)
-
-from ..prompt_format import (
-    # llama3_1_e2e_tool_call_dialog,
-    TextCompletionContent,
-    UseCase,
-    llama3_1_builtin_tool_call_dialog,
-    llama3_1_custom_tool_call_dialog,
-)
-
-
-def wolfram_alpha_response():
-    return textwrap.dedent(
-        """
-        {
-            "queryresult": {
-                "success": true,
-                "inputstring": "100th decimal of pi",
-                "pods": [
-                    {
-                        "title": "Input interpretation",
-                        "subpods": [
-                            {
-                                "title": "",
-                                "plaintext": "100th digit | \u03c0"
-                            }
-                        ]
-                    },
-                    {
-                        "title": "Nearby digits",
-                        "subpods": [
-                            {
-                                "title": "",
-                                "plaintext": "...86208998628034825342117067982148086513282306647093..."
-                            }
-                        ]
-                    },
-                    {
-                        "title": "Result",
-                        "primary": true,
-                        "subpods": [
-                            {
-                                "title": "",
-                                "plaintext": "7"
-                            }
-                        ]
-                    }
-                ]
-            }
-        }
-        """
-    )
-
-
-def usecases() -> list[UseCase | str]:
-    return [
-        textwrap.dedent(
-            """
-            # Llama 3.1 - Prompt Formats
-            ## Tokens
-            Here is a list of special tokens that are supported by Llama 3.1:
-            - `<|begin_of_text|>`: Specifies the start of the prompt
-            - `<|end_of_text|>`: Model will cease to generate more tokens. This token is generated only by the base models.
-            - `<|finetune_right_pad_id|>`: This token is used for padding text sequences to the same length in a batch.
-            - `<|start_header_id|>` and `<|end_header_id|>`: These tokens enclose the role for a particular message. The possible roles are: [system, user, assistant and tool]
-            - `<|eom_id|>`: End of message. A message represents a possible stopping point for execution where the model can inform the executor that a tool call needs to be made. This is used for multi-step interactions between the model and any available tools. This token is emitted by the model when the Environment: ipython instruction is used in the system prompt, or if the model calls for a built-in tool.
-            - `<|eot_id|>`: End of turn. Represents when the model has determined that it has finished interacting with the user message that initiated its response. This is used in two scenarios:
-                - at the end of a direct interaction between the model and the user
-                - at the end of multiple interactions between the model and any available tools
-                This token signals to the executor that the model has finished generating a response.
-            - `<|python_tag|>`: Is a special tag used in the model's response to signify a tool call.
-            """
-        ),
-        textwrap.dedent(
-            """
-            There are 4 different roles that are supported by Llama 3.1
-            - `system`: Sets the context in which to interact with the AI model. It typically includes rules, guidelines, or necessary information that helps the model respond effectively.
-            - `user`: Represents the human interacting with the model. It includes the inputs, commands, and questions to the model.
-            - `tool`: A new role introduced in Llama 3.1. This role is used to mark messages with the output of a tool call when sent back to the model from the executor. (The actual token used by the model for this role is "ipython".)
-            - `assistant`: Represents the response generated by the AI model based on the context provided in the `system`, `tool` and `user` prompts.
-            """
-        ),
-        UseCase(
-            title="Llama 3.1 Base Model",
-            description="Text completion for Llama 3.1 base model uses this format.",
-            dialogs=[TextCompletionContent(content="Color of sky is blue but sometimes can also be")],
-            notes="Note start special tag",
-        ),
-        "## Llama 3.1 Instruct Model",
-        UseCase(
-            title="User and assistant conversation",
-            description="Here is a regular multi-turn user assistant conversation and how its formatted.",
-            dialogs=[
-                [
-                    RawMessage(role="system", content="You are a helpful assistant"),
-                    RawMessage(
-                        role="user",
-                        content="Answer who are you in the form of jeopardy?",
-                    ),
-                ]
-            ],
-            notes="",
-        ),
-        "## Tool Calling Formats",
-        textwrap.dedent(
-            """
-            The three built-in tools (brave_search, wolfram_alpha, and code interpreter) can be turned on using the system prompt:
-            - Brave Search: Tool call to perform web searches.
-            - Wolfram Alpha: Tool call to perform complex mathematical calculations.
-            - Code Interpreter: Enables the model to output python code.
-            """
-        ),
-        UseCase(
-            title="Builtin Tool Calling",
-            description=textwrap.dedent(
-                """
-                Here is an example of a conversation using brave search
-                """
-            ),
-            dialogs=[llama3_1_builtin_tool_call_dialog()],
-            notes=textwrap.dedent(
-                """
-                - Just including Environment: ipython turns on code interpreter; therefore, you don't need to specify code interpretation on the Tools: line. The model can generate python code which is interpreted by the executor, with the result provided back to the model.
-                - The message body of the assistant response starts with a special tag <|python_tag|>
-                - As alluded to above, in such an environment, the model can generate <|eom_id|> instead of just the standard <|eot_id|> . The latter indicates the turn is finished, while the former indicates continued multi-step reasoning. That is, the model is expecting a continuation message with the output of the tool call.
-                - The model tool call response is of the form `tool.call(query="...")` wher tool is `brave_search` or `wolfram_alpha`
-                """
-            ),
-        ),
-        UseCase(
-            title="Builtin Code Interpreter",
-            description="Here is an actual example of model responding with code",
-            dialogs=[
-                [
-                    RawMessage(role="system", content="Environment: ipython"),
-                    RawMessage(
-                        role="user",
-                        content="Write code to check if number is prime, use that to see if the number 7 is prime",
-                    ),
-                ],
-            ],
-            notes=textwrap.dedent(
-                """
-                - Model starts with <|python_tag|> and continues writing python code that it needs to be executed
-                - No explicit mention of code_interpreter in system prompt. `Environment: ipython` implicitly enables it.
-                """
-            ),
-        ),
-        UseCase(
-            title="Built-in tools full interaction",
-            description="Here is a full interaction with the built-in tools including the tool response and the final assistant response.",
-            dialogs=[
-                [
-                    RawMessage(
-                        role="system",
-                        content="Environment: ipython\nTools: brave_search, wolfram_alpha\n",
-                    ),
-                    RawMessage(role="user", content="What is the 100th decimal of pi?"),
-                    RawMessage(
-                        role="assistant",
-                        content="",
-                        stop_reason=StopReason.end_of_message,
-                        tool_calls=[
-                            ToolCall(
-                                call_id="tool_call_id",
-                                tool_name=BuiltinTool.wolfram_alpha,
-                                arguments=json.dumps({"query": "100th decimal of pi"}),
-                            )
-                        ],
-                    ),
-                    RawMessage(
-                        role="tool",
-                        content=wolfram_alpha_response(),
-                    ),
-                ],
-            ],
-            notes=textwrap.dedent(
-                """
-                - Note the `<|python_tag|>` in the assistant response.
-                - Role is `tool` for the wolfram alpha response that is passed back to the model.
-                - Final message from assistant has <|eot_id|> tag.
-                """
-            ),
-        ),
-        "## Zero shot tool calling",
-        UseCase(
-            title="JSON based tool calling",
-            description=textwrap.dedent(
-                """
-                Llama models can now output custom tool calls from a single message to allow easier tool calling.
-                The following prompts provide an example of how custom tools can be called from the output of the model.
-                It's important to note that the model itself does not execute the calls; it provides structured output to facilitate calling by an executor.
-                """
-            ),
-            dialogs=[llama3_1_custom_tool_call_dialog()],
-            notes=textwrap.dedent(
-                """
-                - JSON format for providing tools needs name, description and parameters
-                - Model responds with `<|python_tag|>` and `<|eom_id|>` as `Environment: ipython` was in the system prompt
-                - Instructions for tools added as a user message
-                - Only single tool calls are supported as of now
-                """
-            ),
-        ),
-        # FIXME: This is not working yet as expected
-        # UseCase(
-        #     title="E2E tool call example",
-        #     description=textwrap.dedent(
-        #         """
-        #         Here is an example showing the whole multi-step turn by taking custom tool outputs and passing back to the model.
-        #         """
-        #     ),
-        #     dialogs=[
-        #         llama3_1_e2e_tool_call_dialog(
-        #             tool_prompt_format=ToolPromptFormat.function_tag
-        #         )
-        #     ],
-        #     notes="",
-        # ),
-        "## Example of a user defined tool calling",
-        UseCase(
-            title="`<function>` based tool calling",
-            description=textwrap.dedent(
-                """
-                Here is an example of how you could also write custom instructions for model to do zero shot tool calling.
-                In this example, we define a custom tool calling format using the `<function>` tag.
-                """
-            ),
-            dialogs=[llama3_1_custom_tool_call_dialog(ToolPromptFormat.function_tag)],
-            notes=textwrap.dedent(
-                """
-                - In this case, model does NOT respond with `<|python_tag|>` and ends with `<|eot_id|>`
-                - Instructions for tools added as a user message
-                """
-            ),
-        ),
-    ]

llama_stack/models/llama/llama4/args.py

@@ -1,107 +0,0 @@
-# Copyright (c) Meta Platforms, Inc. and affiliates.
-# All rights reserved.
-#
-# This source code is licensed under the terms described in the LICENSE file in
-# the root directory of this source tree.
-
-from enum import Enum
-
-from pydantic import BaseModel, model_validator
-
-
-class QuantizationScheme(Enum):
-    int4_weight_int8_dynamic_activation = "int4_weight_int8_dynamic_activation"
-
-
-class QuantizationArgs(BaseModel):
-    scheme: QuantizationScheme | None = None
-    group_size: int | None = None
-    spinquant: bool = False
-
-
-class LoRAArgs(BaseModel):
-    rank: int
-    scale: float
-
-
-class MoEArgs(BaseModel):
-    num_experts: int = -1
-    capacity_factor: float = 1.0  # capacity factor determines how many tokens each expert can choose
-    auto_scale_F: bool = (  # noqa: N815
-        True  # if true, rescales hidden_dim such that number of activated params is same as equivalent dense layer
-    )
-    top_k: int = 1
-    interleave_moe_layer_step: int = 1
-
-
-class Size(BaseModel):
-    height: int
-    width: int
-
-
-class VisionArgs(BaseModel):
-    image_size: Size
-    patch_size: Size
-
-    # parameters for the encoder transformer
-    dim: int
-    n_layers: int
-    n_heads: int
-    mlp_ratio: float
-    output_dim: int
-
-    pixel_shuffle_ratio: float
-
-
-class ModelArgs(BaseModel):
-    dim: int = -1
-    n_layers: int = -1
-    n_heads: int = -1
-    n_kv_heads: int | None = None
-    head_dim: int | None = None
-
-    vocab_size: int = -1
-    multiple_of: int = 256  # make SwiGLU hidden layer size multiple of large power of 2
-    ffn_dim_multiplier: float | None = None
-    ffn_exp: float | None = None
-    norm_eps: float = 1e-5
-
-    attention_chunk_size: int | None = None
-    rope_theta: float = 500000
-    use_scaled_rope: bool = False
-    rope_scaling_factor: float | None = None
-    rope_high_freq_factor: float | None = None
-
-    nope_layer_interval: int | None = None  # No position encoding in every n layers
-    use_qk_norm: bool = False
-    # Set to True to enable inference-time temperature tuning (useful for very long context)
-    attn_temperature_tuning: bool = False
-    floor_scale: float = 8192.0
-    attn_scale: float = 0.1
-
-    vision_args: VisionArgs | None = None
-    moe_args: MoEArgs | None = None
-    quantization_args: QuantizationArgs | None = None
-    lora_args: LoRAArgs | None = None
-
-    max_batch_size: int = 32
-    max_seq_len: int = 2048
-
-    @model_validator(mode="after")
-    def validate(self) -> "ModelArgs":
-        assert self.n_kv_heads <= self.n_heads, f"n_kv_heads ({self.n_kv_heads}) must be <= n_heads ({self.n_heads})"
-        assert self.n_heads % self.n_kv_heads == 0, (
-            f"n_heads ({self.n_heads}) must be divisible by n_kv_heads ({self.n_kv_heads})"
-        )
-        assert self.dim % self.n_heads == 0, f"dim ({self.dim}) must be divisible by n_heads ({self.n_heads})"
-
-        if self.use_scaled_rope:
-            # NOTE: ideally these values should have come from params.json. However, we have
-            # shipped the models everywhere. Only Llama-4-Scout uses scaled rope and needs these
-            # specific values.
-            if self.rope_scaling_factor is None:
-                self.rope_scaling_factor = 16
-            if self.rope_high_freq_factor is None:
-                self.rope_high_freq_factor = 1
-
-        return self

llama_stack/models/llama/llama4/ffn.py

@@ -1,58 +0,0 @@
-# Copyright (c) Meta Platforms, Inc. and affiliates.
-# All rights reserved.
-#
-# This source code is licensed under the terms described in the LICENSE file in
-# the root directory of this source tree.
-
-# Copyright (c) Meta Platforms, Inc. and affiliates.
-# All rights reserved.
-#
-# This source code is licensed under the terms described in the LICENSE file in
-# top-level folder for each specific model found within the models/ directory at
-# the top-level of this source tree.
-
-from typing import Any
-
-from fairscale.nn.model_parallel.layers import ColumnParallelLinear, RowParallelLinear
-from fairscale.nn.model_parallel.mappings import reduce_from_model_parallel_region
-from torch import nn
-from torch.nn import functional as F
-
-
-class FeedForward(nn.Module):
-    def __init__(
-        self,
-        dim: int,
-        hidden_dim: int,
-        do_reduce: bool = True,
-    ):
-        super().__init__()
-        self.do_reduce = do_reduce
-
-        self.w1 = ColumnParallelLinear(dim, hidden_dim, bias=False, gather_output=False, init_method=lambda x: x)
-        self.w2 = RowParallelLinear(hidden_dim, dim, bias=False, input_is_parallel=True, init_method=lambda x: x)
-        self.w3 = ColumnParallelLinear(dim, hidden_dim, bias=False, gather_output=False, init_method=lambda x: x)
-        self._register_load_state_dict_pre_hook(self.load_hook)
-
-    def load_hook(
-        self,
-        state_dict: dict[str, Any],
-        prefix: str,
-        local_metadata: dict[str, Any],
-        strict: bool,
-        missing_keys: list[str],
-        unexpected_keys: list[str],
-        error_msgs: list[str],
-    ) -> None:
-        if prefix + "mlp.fc1_weight" in state_dict:
-            w1, w3 = state_dict.pop(prefix + "mlp.fc1_weight").chunk(2, dim=0)
-            state_dict[prefix + "w1.weight"] = w1
-            state_dict[prefix + "w3.weight"] = w3
-            state_dict[prefix + "w2.weight"] = state_dict.pop(prefix + "mlp.fc2_weight")
-
-    def forward(self, x):
-        x = F.silu(F.linear(x, self.w1.weight)) * F.linear(x, self.w3.weight)
-        out = F.linear(x, self.w2.weight)
-        if self.do_reduce:
-            return reduce_from_model_parallel_region(out)
-        return out