@stephen-lord/other2 1.0.8 → 1.0.10

package/dist/docs/manus/Tech-Understanding-Logit-Bias-in-LLMs-Medium.md

@@ -0,0 +1,1354 @@
+# Understanding Logit Bias in LLMs | Medium
+
+**Source:** https://medium.com/@serhatcck/token-level-control-in-openai-models-a-developers-guide-to-logit-bias-6fcc04a8a41f
+
+---
+
+URL: https://medium.com/@serhatcck/token-level-control-in-openai-models-a-developers-guide-to-logit-bias-6fcc04a8a41f
+Author: Serhat ÇİÇEK
+
+Understanding Logit Bias in LLMs | Medium
+
+Sitemap
+
+Open in app
+
+Sign up
+
+Sign in
+
+Medium Logo
+
+Get app
+
+Write
+
+Search
+
+Sign up
+
+Sign in
+
+# Token-Level Control in OpenAI Models: A Developer’s Guide to Logit Bias
+
+Serhat ÇİÇEK
+
+6 min read
+
+Dec 11, 2025
+
+--
+
+Listen
+
+Share
+
+Press enter or click to view image in full size
+
+Logit bias is one of the least discussed yet most influential parameters in modern LLM development. By directly manipulating token-level probabilities, it allows developers to shape model outputs with a precision that traditional prompting cannot achieve. Whether you need deterministic text generation, safer content boundaries, or fine-tuned behavioral constraints without model retraining, logit bias provides a powerful mechanism for controlling how an LLM thinks and responds. Understanding this feature is essential for anyone building reliable, predictable, and production-grade AI systems.
+
+## What Is Bias in Large Language Models?
+
+Bias in Large Language Models refers to systematic tendencies in how a model generates text. Instead of producing purely neutral or balanced outputs, an LLM may favor certain tokens, ideas, or associations because of the statistical patterns it learned during training. This means the model’s responses can shift — not due to correctness, but due to patterns embedded in the data or architecture.
+
+A useful way to understand this is to compare LLM bias with human cognitive bias. For example, if a developer spends years writing code mostly in Python, they naturally develop a positive bias toward that language. When starting a new project, choosing Python becomes more likely — not necessarily because it is the best option, but because past experience shaped their preference.
+
+LLMs behave in a similar way: if the training data heavily features certain topics, styles, or associations, the model becomes more inclined to reproduce them.
+
+In practical terms, bias is simply a distortion in token probabilities. Some tokens become more likely, others less likely, shaping the model’s tone, content, and reasoning. For AI developers, understanding this is crucial: bias affects predictability, alignment, safety, and overall output quality.
+
+### Common Sources of Bias in LLMs
+
+- Instruction-Tuning Bias — Reinforcement from human feedback shaping preferred behaviors.
+- Decoding-Time Bias — Sampling techniques and parameters (e.g., temperature, logit bias) that shift token probabilities.
+- Objective & Loss Function Bias — Optimization that favors certain patterns over others.
+- Representational Bias — Embeddings forming unequal relationships between concepts.
+- Training Data Imbalance — Overrepresented topics, sentiments, or cultural viewpoints.
+
+## Bias and Tokens in OpenAI Models
+
+OpenAI’s bias controls — such as logit_bias — operate entirely at the token level. Tokens are the smallest units a model uses to understand and generate text. Instead of reading characters or whole words, LLMs break text into tokens using a tokenizer. This means that any bias applied to a model directly alters the probability distribution of individual tokens, not entire sentences.
+
+For example:
+
+- [“Hello”, “ world”, “!”]
+- A simple sentence like “Hello world!” could be encoded into:
+- The word “JavaScript” may be split into multiple tokens depending on the tokenizer.
+- The word “Python” might be a single token.
+
+Because bias works at this granular level, developers must understand how tokenization affects outcomes. Applying a positive logit bias to the token representing “Python” increases the chance of the model using that word in its response. Conversely, applying a strong negative bias to a token like “no” can drastically reduce its appearance, even if it makes the response grammatically awkward.
+
+### What Do +100 and −100 Mean in Logit Bias?
+
+In OpenAI models, logit bias values such as +100 and −100 represent extremely strong adjustments to a token’s probability. A +100 bias forces the model to strongly prefer a specific token — effectively pushing its probability close to 100% whenever it’s a valid next token. Conversely, a −100 bias nearly eliminates a token from the generation process, reducing its probability to almost zero. These extreme values behave like “hard constraints” for token selection. For example, applying +100 to the token for “Python” almost guarantees the model will mention it, while assigning −100 to the token “Java” will cause the model to avoid it entirely — even when it would normally be a reasonable choice. Although smaller values can create more nuanced nudges, ±100 is commonly used when developers need deterministic control over model output.
+
+## PoC
+
+To experiment with token-level biasing in real time, you can download and run the project available at LLM-Bias on GitHub. Simply clone the repository from
+
+https://github.com/Serhatcck/LLM-Bias, install the dependencies, and start the application. The interface allows you to enter a prompt, choose a logit bias value, and test how different token biases affect the model’s output.
+
+Press enter or click to view image in full size
+
+### -100 Logit Bias
+
+We start with a simple baseline prompt:
+
+“Explain quantum mechanics with two sentences.”
+
+Without any bias applied, the model generates a neutral, standard explanation based on its learned distribution of scientific terminology.
+
+Press enter or click to view image in full size
+
+Now we repeat the exact same prompt, but this time we introduce a negative logit bias for the token(s) representing “physics” (for example “physics” or its token ID).
+
+Assigning a strong negative bias such as −80 to −100 almost completely suppresses the appearance of that word.
+
+Press enter or click to view image in full size
+
+After running both prompts, the difference becomes clear:
+
+- The negatively biased output avoids using the word physics entirely, even when it logically fits the explanation. The model instead substitutes more generic or indirect descriptions to compensate.
+- The unbiased output may reference physics naturally, since quantum mechanics is part of the field.
+
+## Insights From Our Logit Bias Experiments
+
+While testing both positive and negative logit bias across different prompts, we observed several practical behaviors that developers should be aware of:
+
+### Negative Logit Bias Observations
+
+- Logit bias values above –80 tend to be weak. The model may still generate the unwanted token, especially in contexts where it is highly probable.
+- To reliably suppress a word, you must include all of its variations in the logit bias list — such as lowercase/uppercase versions, plural forms, and common morphological variants.
+
+### Positive Logit Bias Observations
+
+- Positive bias values of +80 or lower typically do not create strong enough pressure. The model fails to consistently insert the target word unless the word already aligns with the prompt.
+- When a high positive value (e.g., +100) is applied to a token unrelated to the prompt, the model often takes significantly longer to produce a response. This happens because the LLM struggles to justify the forced token within the context.
+
+## Summary
+
+Negative logit bias is far more practical for production environments, especially when filtering unwanted words, profanity, or brand names. However, developers must carefully select and tokenize all relevant variations of the terms they want to block. Positive logit bias, while useful for controlled generation, often introduces latency and instability when forcing unrelated tokens into the output.
+
+## References
+
+### OpenAI Documentation
+
+- https://platform.openai.com/docs/guides/text-generation/tokenization
+- GPT Tokenization Guide
+- https://platform.openai.com/docs/api-reference/chat/create#chat-create-logit_bias
+- OpenAI API Reference — Logit Bias
+
+### LLM Tokenization & Probability
+
+- https://huggingface.co/docs/tokenizers/index
+- Tokenizers by HuggingFace (general tokenizer behavior)
+- https://github.com/openai/tiktoken
+- OpenAI Tiktoken Library (tokenizer implementation)
+
+### Research Papers on LLM Bias
+
+- https://arxiv.org/abs/2005.14050
+- Assessing Social and Linguistic Bias in Language Models
+- https://arxiv.org/abs/1906.07337
+- Bias in Language Models: A Comprehensive Review
+- https://arxiv.org/abs/2312.01708
+- A Survey on Bias in Large Language Models (arXiv)
+
+LLM
+
+NLP
+
+AI Agent
+
+Bias In Ai
+
+Llm Tokens
+
+## Written by Serhat ÇİÇEK
+
+119 followers
+
+85 following
+
+Security Researcher & AI Researcher & Software Engineer
+
+## No responses yet
+
+Help
+
+Status
+
+About
+
+Careers
+
+Press
+
+Blog
+
+Privacy
+
+Rules
+
+Terms
+
+Text to speech
+
+# Logit Bias - LLM Parameter Guide - Vellum
+URL: https://vellum.ai/llm-parameters/logit-bias
+
+Logit Bias - LLM Parameter Guide - Vellum
+
+LLM ParametersLogit Bias
+
+← All Parameters
+
+# Logit Bias
+
+OpenAIAnthropic
+
+# What is Logit Bias
+
+The logit bias parameter lets you control whether the model is more or less likely to generate a specific word.
+
+# How does it work behind the scenes
+
+The model is always deciding which word (or tokens) to pick next. All these tokens have their own IDs, and using logit bias we can forbid the model to use some of these IDs.
+
+But how can we actually find these IDs?
+
+The simplest way is to use OpenAI’s tokenizer tool . Just type in your words, toggle the “Text-Token ID” option at the bottom, and you’ll get the IDs for your words. In some cases you’ll get more tokens for one word.
+
+It’s important to note here that different models may produce different tokens for the same input, so you should always check with the model provider to learn about their tokenization process.
+
+There are couple of important things to note here:
+
+You can use OpenAI’s tokenizer tool to find out the tokens for GPT-3.5 and GPT-4 models, but there is still no data for GPT-4o and GPT-4o mini. One word can have two tokens. Characters before a word (e.g. a space or underscore) can produce different tokens for the same word. Capitalization and no-capitalization versions of same word might result in different tokens.
+
+# How to set this parameter correctly
+
+In the API, this parameter accepts a JSON object that maps token IDs to bias value. This bias value can vary between -100 to 100. The parameter takes tokens, not text, so you’d use the tokenizer we mentioned above to take the token ids for the words that you’d want to “bias”.
+
+# How to experiment with Logit Bias
+
+The closer the value is to -100, the more likely that token will be blocked from being generated. The closer it is to 100, the more the model is encouraged to use that token.
+
+To test this parameter, try adjusting the values gradually and analyze the impact. Using small values like 1 or -1 won’t make much difference, but values like 5 or -5 can have a much stronger effect.
+
+# When to use Logit Bias
+
+Use logit bias when know specifically which words you want to ban or encourage repetitive use.
+
+### Example 1: Ban offensive words
+
+One example where you’d want to ban some words (tokens) from appearing in the results is for moderation purposes.
+
+Suppose you’re building a guardrail that will capture offensive content in your chatbot. Now, you may want to ban words like “stupid”. The word “stupid” tokenizes to two IDs [267, 16263] , and the same word with a space before “ stupid” tokenizes to another ID [18754] . To ban them from appearing in the results we can add the logit bias like so:
+
+Example 2: Encourage neutral answers in a chatbot If you’re using a customer support chatbot, you’ll likely want it to maintain a calm, neutral tone. To help with that, you can encourage the model to use more neutral words like “understand,” “assist,” and “resolve”. To make the model output “understand,” you need to map it to two token IDs and add a bias of 5:
+
+
+
+# Constrained Decoding and Structured Output for Agent Reliability - Engineering Notes
+URL: https://notes.muthu.co/2025/11/constrained-decoding-and-structured-output-for-agent-reliability/
+
+Constrained Decoding and Structured Output for Agent Reliability - Engineering Notes
+
+# Engineering Notes
+
+Thoughts and Ideas on AI by Muthukrishnan
+
+# Constrained Decoding and Structured Output for Agent Reliability
+
+When building production AI agents, one of the most persistent problems is unpredictable output formats. An agent needs to call a tool with precise JSON parameters, but the LLM wraps the output in markdown code blocks, adds explanatory text, or hallucinate invalid field names. This breaks the entire agent pipeline.
+
+Constrained decoding solves this by restricting what tokens an LLM can generate, ensuring outputs always conform to specified formats like JSON schemas, regular expressions, or context-free grammars. It’s the difference between hoping your agent produces valid JSON and guaranteeing it.
+
+## Concept Introduction
+
+During text generation, an LLM samples tokens from a probability distribution at each step. Constrained decoding modifies this process by masking invalid tokens, setting their probability to zero before sampling. The result: tool calls always have correct parameter names and types, database queries never contain invalid SQL syntax, API requests match OpenAPI specifications, and decision outputs are always parseable by downstream systems.
+
+```
+Standard Decoding:
+P(next_token | context) → Sample from all vocabulary
+
+Constrained Decoding:
+P(next_token | context, grammar) → Sample only from valid tokens
+
+```
+
+The constraint can be:
+
+- A JSON schema (only generate valid JSON matching the schema)
+- A regular expression (output must match the regex)
+- A context-free grammar (follow specific syntax rules)
+- A finite-state machine (transition through defined states)
+
+Modern implementations use:
+
+- Token masking at inference time
+- Incremental parsing to track valid next tokens
+- Beam search with grammar-aware scoring
+- Logit bias to steer generation probabilistically
+
+## Historical & Theoretical Context
+
+The concept emerged from multiple research threads:
+
+1. Semantic Parsing (1990s–2000s) Early NLP systems used grammar-based parsers to convert natural language to formal representations (SQL, logic). These were rigid but guaranteed valid output.
+
+2. Constrained Generation in NLG (2010s) Neural text generation models began incorporating hard constraints:
+
+- Hokamp & Liu (2017): Grid Beam Search for lexically constrained generation
+- Forcing specific phrases to appear in translations or summaries
+
+3. Structured Prediction (2015–2020) Seq2seq models for code generation, semantic parsing, and structured data extraction needed format guarantees. Early solutions used post-processing and re-ranking.
+
+4. LLM Function Calling Era (2020–present) As LLMs became agents with tool use, reliable structured output became critical:
+
+- OpenAI Function Calling (2023): Proprietary constrained decoding for JSON tool calls
+- Guidance (2023): Microsoft’s grammar-based generation library
+- Outlines (2023): Fast regex and JSON schema constraints using FSMs
+- LM Format Enforcer (2023): Token masking for various formats
+
+### Theoretical Foundation
+
+Constrained decoding connects to:
+
+- Formal Language Theory: Using automata to define valid sequences
+- Parsing Theory: Incremental parsing to determine next valid tokens
+- Probabilistic Inference: Conditioning probability distributions on constraints
+- Program Synthesis: Generating code that compiles/type-checks
+
+## Algorithms & Math
+
+### Core Algorithm: FSM-Guided Token Masking
+
+The most efficient modern approach uses finite-state machines:
+
+Pseudocode:
+
+```
+def constrained_decode(prompt, schema, max_tokens):
+    # Convert schema to FSM
+    fsm = schema_to_fsm(schema)
+    state = fsm.initial_state
+    tokens = []
+
+    for _ in range(max_tokens):
+        # Get next token logits from LLM
+        logits = llm.forward(prompt + tokens)
+
+        # Mask invalid tokens based on current FSM state
+        valid_tokens = fsm.get_valid_tokens(state)
+        masked_logits = mask_logits(logits, valid_tokens)
+
+        # Sample next token
+        next_token = sample(masked_logits)
+        tokens.append(next_token)
+
+        # Update FSM state
+        state = fsm.transition(state, next_token)
+
+        # Check if reached accept state
+        if fsm.is_terminal(state):
+            break
+
+    return tokens
+
+```
+
+Mathematical Formulation:
+
+Let $\mathcal{G}$ be a grammar defining valid outputs, and $\mathcal{L}(\mathcal{G})$ the language it accepts.
+
+Standard decoding samples:
+
+$$ t_i \sim \text{softmax}(\mathbf{z}_i) $$
+
+Constrained decoding samples:
+
+$$ t_i \sim \text{softmax}(\mathbf{z}_i + \mathbf{m}_i) $$
+
+where the mask $\mathbf{m}_i$ is:
+
+$$ m_i^{(j)} = \begin{cases} 0 & \text{if } t_1, \ldots, t_{i-1}, j \in \mathcal{L}(\mathcal{G}) \\ -\infty & \text{otherwise} \end{cases} $$
+
+### JSON Schema to FSM Conversion
+
+Converting a JSON schema to an FSM involves:
+
+1. Tokenize the schema structure:`{`,`"field"`,`:`,`[`, numbers, strings, etc.
+2. Build states for each schema element: object start, field names, value types
+3. Define transitions: Valid next tokens from each state
+4. Handle recursion: For nested objects/arrays
+
+Example:
+
+```
+{
+  "type": "object",
+  "properties": {
+    "action": {"type": "string", "enum": ["search", "calculate"]},
+    "value": {"type": "number"}
+  },
+  "required": ["action", "value"]
+}
+
+```
+
+FSM states:
+
+```
+START → "{" → "action" → ":" → ("search"|"calculate") → "," → "value" → ":" → NUMBER → "}" → END
+
+```
+
+At each state, only specific tokens are valid (e.g., after`"action":`, only`"search"` or`"calculate"`).
+
+## Design Patterns & Architectures
+
+Schema-First Agent Design
+
+Define schemas before implementing agents:
+
+```
+from pydantic import BaseModel
+
+class SearchTool(BaseModel):
+    query: str
+    max_results: int = 10
+    filters: dict[str, str] = {}
+
+class CalculateTool(BaseModel):
+    expression: str
+    precision: int = 2
+
+# Agent now MUST output one of these
+
+```
+
+Layered Validation
+
+Combine multiple constraint layers:
+
+1. Token-level: Constrained decoding ensures valid syntax
+2. Type-level: Schema validation checks types
+3. Semantic-level: Business logic validates values
+
+```
+# Layer 1: Constrained decoding produces valid JSON
+output = constrained_generate(prompt, json_schema)
+
+# Layer 2: Validate against Pydantic model
+tool_call = SearchTool.parse_raw(output)
+
+# Layer 3: Business logic
+if tool_call.max_results > 100:
+    raise ValueError("max_results too high")
+
+```
+
+Progressive Refinement
+
+For complex outputs, chain constrained generations:
+
+```
+graph LR
+    A[User Query] --> B[Generate Tool Choice<br/>Constrained: tool names only]
+    B --> C[Generate Parameters<br/>Constrained: specific schema]
+    C --> D[Execute Tool]
+    D --> E[Generate Response<br/>Constrained: response format]
+  
+```
+
+This reduces error accumulation compared to generating everything at once.
+
+### Integration with Agent Architectures
+
+Planner-Executor-Memory Loop:
+
+```
+class ConstrainedAgent:
+    def plan(self, goal: str) -> Plan:
+        # Constrained to Plan schema
+        return constrained_generate(
+            f"Create plan for: {goal}",
+            schema=Plan.schema()
+        )
+
+    def execute(self, step: PlanStep) -> ActionResult:
+        # Constrained to ActionResult schema
+        return constrained_generate(
+            f"Execute: {step.description}",
+            schema=ActionResult.schema()
+        )
+
+```
+
+ReAct Loop:
+
+```
+def react_step(observation: str) -> ThoughtActionObservation:
+    # Force exactly: Thought: <text>\nAction: <json>\n
+    return constrained_generate(
+        prompt=observation,
+        grammar=react_grammar  # CFG for ReAct format
+    )
+
+```
+
+## Practical Application
+
+### Small Coding Example: Weather Agent with Outlines
+
+```
+from outlines import models, generate
+from pydantic import BaseModel
+
+# Define tool schema
+class WeatherQuery(BaseModel):
+    location: str
+    unit: str  # "celsius" or "fahrenheit"
+
+# Load model
+model = models.transformers("mistralai/Mistral-7B-Instruct-v0.2")
+
+# Create constrained generator
+generator = generate.json(model, WeatherQuery)
+
+# Generate - GUARANTEED to be valid WeatherQuery
+prompt = """User: What's the weather in Tokyo?
+Assistant: I'll check the weather. Tool call:
+"""
+
+result = generator(prompt)
+# result = WeatherQuery(location="Tokyo", unit="celsius")
+
+print(f"Location: {result.location}")
+print(f"Unit: {result.unit}")
+
+```
+
+### Integration with LangGraph
+
+```
+from langgraph.graph import StateGraph, END
+from outlines import models, generate
+from pydantic import BaseModel
+
+class AgentState(BaseModel):
+    messages: list[str]
+    next_action: str | None = None
+
+class ActionSchema(BaseModel):
+    action: str  # "search" | "calculate" | "finish"
+    parameters: dict
+
+# Constrained action generator
+model = models.transformers("meta-llama/Llama-3-8B-Instruct")
+action_generator = generate.json(model, ActionSchema)
+
+def decide_action(state: AgentState) -> AgentState:
+    prompt = format_prompt(state.messages)
+    action = action_generator(prompt)  # Always valid ActionSchema
+    state.next_action = action.action
+    return state
+
+def execute_action(state: AgentState) -> AgentState:
+    # Execute the action
+    result = execute(state.next_action)
+    state.messages.append(result)
+    return state
+
+# Build graph
+workflow = StateGraph(AgentState)
+workflow.add_node("decide", decide_action)
+workflow.add_node("execute", execute_action)
+workflow.add_conditional_edges("decide",
+    lambda s: END if s.next_action == "finish" else "execute")
+workflow.set_entry_point("decide")
+
+app = workflow.compile()
+
+```
+
+### OpenAI Structured Outputs
+
+OpenAI provides native support:
+
+```
+from openai import OpenAI
+from pydantic import BaseModel
+
+class ResearchResult(BaseModel):
+    summary: str
+    key_findings: list[str]
+    confidence_score: float
+
+client = OpenAI()
+
+completion = client.chat.completions.create(
+    model="gpt-4o-2024-08-06",
+    messages=[
+        {"role": "user", "content": "Analyze this research paper: ..."}
+    ],
+    response_format=ResearchResult  # Structured output mode
+)
+
+result = ResearchResult.model_validate_json(completion.choices[0].message.content)
+# Guaranteed to be valid ResearchResult
+
+```
+
+## Latest Developments & Research
+
+### Recent Breakthroughs (2023-2025)
+
+1. Efficient FSM Construction
+
+“Faster Constrained Decoding for Open-Domain Generation” (2024)
+
+- Converts JSON schemas to minimal FSMs in milliseconds
+- Reduces overhead to <5% latency increase
+- Open-sourced in Outlines 0.1+ library
+
+2. Grammar-Based Prompting
+
+“Guidance: A Faster, More Efficient Programming Paradigm for Constrained Generation” (Microsoft, 2023)
+
+- Interleaves constraints with generation
+- Allows complex patterns like “generate code that compiles”
+- Used in Microsoft Copilot
+
+3. Type-Aware Constrained Decoding
+
+“TypeT5: Seq2seq Type Inference using Static Analysis” (2024)
+
+- Combines static type checking with constrained decoding
+- Ensures generated code is type-safe
+- 99.7% type correctness on HumanEval
+
+4. Semantic Constraints
+
+“NeuroLogic Aesque Decoding”* (2024)
+
+- Combines probabilistic constraints (logit bias) with hard constraints
+- Balances fluency and correctness
+- Used for constrained dialogue generation
+
+### Benchmarks
+
+JSON Schema Compliance (SchemaBench, 2024):
+
+- Prompt engineering: 82% valid
+- Post-processing: 89% valid
+- Constrained decoding: 99.8% valid
+
+Tool Calling Reliability (FunctionHub, 2024):
+
+- Standard generation: 76% executable calls
+- OpenAI function calling: 94% executable
+- Outlines JSON mode: 99.2% executable
+
+### Open Problems
+
+1. Multi-modal constraints: Extending to image/audio generation
+2. Soft constraints: Probabilistic preferences vs. hard rules
+3. Constraint learning: Inferring schemas from examples
+4. Distributed decoding: Constraints across multi-agent systems
+5. Constraint debugging: Tools to visualize why constraints fail
+
+### Ongoing Research
+
+- Adaptive masking: Learning which tokens to mask based on context
+- Constraint synthesis: Automatically generating schemas from documentation
+- Probabilistic grammars: Weighted FSMs for soft guidance
+- Cross-lingual constraints: Applying constraints to multilingual models
+
+## Cross-Disciplinary Insight
+
+### Connections to Compiler Theory
+
+Constrained decoding is essentially parsing in reverse:
+
+- Parser: Valid string → Abstract syntax tree
+- Constrained decoder: AST (schema) → Valid string
+
+Modern compilers use LR parsers that incrementally determine valid next tokens, which is exactly what constrained decoders do. The FSM used is analogous to a parse table in compiler design.
+
+### Links to Control Theory
+
+The constraint can be seen as a controller in a feedback loop:
+
+```
+graph LR
+    A[LLM<br/>Plant] -->|Token probabilities| B[Constraint<br/>Controller]
+    B -->|Masked probabilities| C[Sampler]
+    C -->|Next token| A
+    B -.->|Desired output format| B
+  
+```
+
+This mirrors model predictive control where future states are constrained to a safe/desired region.
+
+### Cognitive Science Parallel
+
+Human language production involves monitoring: we catch ourselves mid-sentence if about to say something incorrect. Constrained decoding is an artificial form of this executive control, filtering invalid “thoughts” before they’re expressed.
+
+## Daily Challenge: Build a Constrained SQL Generator
+
+Goal: Create an AI agent that generates valid SQL queries using constrained decoding.
+
+Requirements:
+
+1. Define a simple SQL grammar (SELECT, FROM, WHERE with basic conditions)
+2. Implement constrained decoding to ensure syntactic validity
+3. Test on natural language queries
+
+Starter Code:
+
+```
+from outlines import models, generate
+
+# Define SQL grammar (simplified)
+sql_grammar = r"""
+    start: select_stmt
+    select_stmt: "SELECT" columns "FROM" table where_clause?
+    columns: COLUMN ("," COLUMN)*
+    table: WORD
+    where_clause: "WHERE" condition
+    condition: COLUMN OPERATOR VALUE
+
+    COLUMN: /[a-z_]+/
+    OPERATOR: "=" | ">" | "<" | "!="
+    VALUE: /"[^"]*"/ | /[0-9]+/
+    WORD: /[a-z_]+/
+
+    %import common.WS
+    %ignore WS
+"""
+
+model = models.transformers("your-model")
+sql_generator = generate.cfg(model, sql_grammar)
+
+# Test queries
+queries = [
+    "Find all users with age greater than 30",
+    "Get product names where price is less than 100",
+]
+
+for query in queries:
+    prompt = f"Natural language: {query}\nSQL: "
+    sql = sql_generator(prompt)
+    print(f"Query: {query}")
+    print(f"SQL: {sql}\n")
+
+```
+
+Extension Challenges:
+
+1. Add support for JOIN operations
+2. Validate against actual table schemas
+3. Measure how often unconstrained models produce invalid SQL
+4. Compare generation time with/without constraints
+
+Time estimate: 20-30 minutes
+
+## References & Further Reading
+
+### Key Papers
+
+“Guidance: A Faster Programming Paradigm for Constrained LLM Generation” Microsoft Research, 2023 https://github.com/microsoft/guidance
+
+“Outlines: Fast and Flexible Structured Generation” Normal Computing, 2023 https://github.com/outlines-dev/outlines Paper: https://arxiv.org/abs/2307.09702
+
+“Grammar-Constrained Decoding for Structured NLP Tasks” Shin et al., EMNLP 2021 https://arxiv.org/abs/2106.08462
+
+“Constrained Decoding for Neural NLG from Compositional Representations” Balakrishnan et al., ACL 2019 https://arxiv.org/abs/1906.07220
+
+“A Guided Constrained Decoding for Faithful Text Generation”* Lu et al., NeurIPS 2022 https://arxiv.org/abs/2210.05097
+
+### Tools & Libraries
+
+- Outlines (Python): https://github.com/outlines-dev/outlines
+- Guidance (Python): https://github.com/microsoft/guidance
+- LM Format Enforcer (Python): https://github.com/noamgat/lm-format-enforcer
+- LMQL (Query language): https://lmql.ai/
+- OpenAI Structured Outputs: https://platform.openai.com/docs/guides/structured-outputs
+
+### Blog Posts & Tutorials
+
+“Structured Generation with Outlines” https://outlines-dev.github.io/outlines/welcome/
+
+“How OpenAI’s Structured Outputs Work” https://cookbook.openai.com/examples/structured_outputs_intro
+
+“Building Reliable Agents with Constrained Decoding” LangChain blog, 2024 https://blog.langchain.dev/constrained-decoding/
+
+### Frameworks Supporting Constrained Outputs
+
+- LangGraph: Via custom parsers + retry logic
+- CrewAI: Via Pydantic models + validation
+- AutoGen: Via response format specifications
+- LlamaIndex: Via output parsers + constrained generation
+
+### Advanced Topics
+
+- Incremental Parsing: Earley parsers for CFGs
+- Efficient FSM Minimization: Hopcroft’s algorithm
+- Probabilistic Context-Free Grammars: Soft constraints
+- Constraint Propagation: SAT solvers for complex constraints
+
+---
+
+Next Steps:
+
+1. Complete the daily challenge to internalize the concepts
+2. Experiment with Outlines or Guidance on your own prompts
+3. Profile the latency impact of constraints on your use case
+4. Design schemas for your agent’s tool calls
+5. Read the Outlines paper for implementation details
+
+Constrained decoding transforms AI agents from unpredictable text generators into reliable system components.
+
+---
+
+# How to build function calling and JSON mode for open-source and fine-tuned LLMs
+URL: https://baseten.co/blog/how-to-build-function-calling-and-json-mode-for-open-source-and-fine-tuned-llms
+
+How to build function calling and JSON mode for open-source and fine-tuned LLMs
+
+"Inference Engineering" is now available. Get your copy here
+
+Model performance
+
+# How to build function calling and JSON mode for open-source and fine-tuned LLMs
+
+Use a state machine to generate token masks for logit biasing to enable function calling and structured output at the model server level.
+
+### Authors
+
+Bryce Dubayah
+
+Philip Kiely
+
+### Last updated
+
+May 16, 2025
+
+### Share
+
+Today, we announced support for function calling and structured output for LLMs deployed with our TensorRT-LLM Engine Builder. This adds support at the model server level for two key features:
+
+Function calling: also known as “tool use,” this feature lets you pass a set of defined tools to a LLM as part of the request body. Based on the prompt, the model selects and returns the most appropriate function/tool from the provided options.
+
+Structured output: an evolution of “JSON mode,” this feature enforces an output schema defined as part of the LLM input. The LLM output is guaranteed to adhere to the provided schema, with full Pydantic support.
+
+To introduce these features, we build new capabilities into our customized version of NVIDIA’s Triton inference server. This engineering deep dive explains how the implementation works under the hood: defining schemas and tools, building a state machine, and using logit biasing to force valid output.
+
+And the best part? Thanks to pre-computed token masks, there’s minimal latency impact from using either feature after the first call with a given schema is completed. You can expect the same tokens per second when generating JSON as when generating ordinary text.
+
+If you’re looking to get started quickly with these new features, check out our launch announcement and docs for function calling and structured output. For implementation details, keep reading!
+
+## How structured output is generated
+
+To understand how it’s possible to guarantee structured output, we need to dive into the details of how a token is generated during LLM inference. If you’re familiar with LLM inference, you’ll know that a new token is generated on each forward pass through the model. During that forward pass:
+
+A vector of logits is outputted from the final layer of the LLM’s neural network.
+
+A normalization function like softmax is applied to turn the logits into probabilities.
+
+Using these probabilities, a token is selected. Depending on settings like`top_p`,`top_k`,`beam_width`, and`temperature`, this may not always be the highest-probability token.
+
+Structured output uses logit biasing in the first step to guarantee valid tokens are generated.
+
+### Logit biasing ensures token validity
+
+The length of the logit vector outputted in the first step is equal to the number of tokens in the model’s vocabulary. For example, Llama 3 LLMs have a vocabulary of ~128,000 tokens. Thus, the logit vector will have about 128K values. Each logit in the vector is a score representing how much the LLM thinks that the given token from the vocabulary could be the next token in the output sequence.
+
+For structured output, we only want to generate valid tokens. For example, an array in JSON must have both an opening and closing bracket:`[1, 2, 3]`. If we already have generated`[1, 2, 3` then the valid options are:
+
+A comma, space, and another value such as four: ,`4`.
+
+A closing bracket to end the array:`]`.
+
+From the model’s vocabulary, most of the possible tokens will not be valid at certain points when generating structured output. Logit biasing guarantees valid output structure by identifying every invalid token and setting its score to negative infinity, ensuring that the invalid tokens cannot be generated.
+
+✕
+
+This discussion of logit biasing raises a natural question: how do we know where we are in the output schema and which tokens are valid?
+
+### State machine provides token requirements
+
+The model server running beneath the inference process is responsible for tracking output format using a state machine. This model server is a modified version of NVIDIA Triton with extra capabilities that we call “Briton” (Baseten + Triton = Briton).
+
+Using an industry standard library Outlines, which also powers VLLM, the Briton model server takes the schema passed as model output, transforms it into a regular expression, then generates a state machine from that regex. We chose Outlines for its robust feature set and reliability.
+
+However, Outlines is written in Python, while TensorRT-LLM and Triton run in C++ for speed and efficiency. To handle this, we first generate the state machine in Python, then serialize it to Protocol Buffers and load it into the model server.
+
+Once loaded into the model server, the state machine makes the logit biasing process incredibly efficient. The state machine is cached in memory, and an appropriate token mask – a list of 1s and 0s corresponding to valid and invalid tokens – is created for each node of the state machine for logit biasing. This means that these calculations aren’t made during inference time, rather, existing masks are applied based on which state is active.
+
+With no token mask calculations happening during token generation, this approach to logit biasing has a negligible effect on model performance, so you’ll get the same high tokens per second that you’re used to from TensorRT-LLM while also ensuring that every token is valid for the provided output schema.
+
+## How to use function calling
+
+Function calling works by providing LLMs with a structured description of a set of tools. Based on the prompt, the model selects the most appropriate tool or tools for the task described. Functions can be anything: API calls, ORM access, SQL queries, or just a script.
+
+✕
+
+A function written to be passed to an LLM — note the descriptive docstring.
+
+It’s essential to understand that function calling does not give the LLM the capability to execute code. Instead, the function calling asks the LLM to choose the most appropriate function from the list of available tools. The actual function execution needs to happen in the same environment that made the LLM call.
+
+Our function calling implementation follows the OpenAI API spec for compatibility, but applies to any model served with TensorRT-LLM via the Engine Builder that has built-in function calling capabilities (e.g. Llama 3.1 Instruct, but not Llama 3). Using the same logit biasing process that creates structured output, Briton (the modified Triton inference server) guarantees schematically correct tool responses.
+
+✕
+
+Example payload with function calling via the "tools" key
+
+Function calling is critical for building agentic workflows and other advanced Compound AI systems. To use function calling for yourself, check out our function calling example in the documentation.
+
+## How to use structured output
+
+The more general structured output feature forces LLMs to return output that adheres to a Pydantic schema. Structured output is valid JSON, but goes beyond JSON mode with support for required and optional fields, multiple data types, and additional validations like maximum length.
+
+To start, define your output schema as a Pydantic model.
+
+✕
+
+Pydantic model for a "Person" object. The schema can be passed to an LLM to structure output.
+
+Then, when you add the schema to the LLM call, the model server will build the schema into a state machine and use it for token masking as described above. The LLM inference arguments match the OpenAI API spec for structured output to ensure maximum compatibility.
+
+✕
+
+Example LLM request payload with a response schema.
+
+Structured output is useful for a wide range of Compound AI applications as the guaranteed schema adherence means you can integrate LLMs into larger systems without worrying about type errors. To try structured output for your application, start with our structured output example in the documentation.
+
+## What to build with function calling and structured output
+
+While the implementation behind these new features is interesting, what’s even more exciting is the use cases they enable.
+
+Function calling unlocks a wide range of agentic use cases for open source LLMs. With function calling, you can give agents access to a set of tools to accomplish tasks. As we saw above, the LLM is only able to select the best tool, not actually execute the API call or run the function, so that’s where multi-step AI systems are needed.
+
+These multi-step, often multi-model systems are commonly known as Compound AI. When building multi-stage Compound AI systems, structured output is critical. With structured output, each component of the system can communicate in valid JSON, preventing errors and avoiding parsing overhead.
+
+As you build with function calling and structured output, remember that the model server changes don’t enhance quality, they only enforce format. Clear prompting and techniques like few-shot prompting still have their place for getting quality output within the enforced structure.
+
+Get started building:
+
+First, deploy Llama 3.1 8B with the TensorRT-LLM Engine Builder
+
+Then try function calling with an accurate LLM math demo
+
+And get JSON-mode output with in a document parsing demo
+
+Subscribe to our newsletter
+
+Stay up to date on model performance, inference infrastructure, and more.
+
+‌
+
+## Explore Baseten today
+
+Start deploying Talk to an engineer
+
+# Custom Logits Processors - vLLM
+URL: https://docs.vllm.ai/en/latest/features/custom_logitsprocs/
+
+You are viewing the latest developer preview docs. Click here to view docs for the latest stable release.
+
+-  Examples 
+- Offline Inference
+- Online Serving
+- Others
+- Pooling
+- RL
+- General
+- Inference and Serving
+- Deployment
+- Integrations
+- Training
+-  Configuration 
+- Models
+- Extensions
+- Hardware Supported Models
+-  Features 
+-  Speculative Decoding 
+-  Developer Guide 
+-  Model Implementation 
+- CI
+- Design Documents
+-  Architecture Overview 
+-  Attention Backend Feature Support 
+-  CUDA Graphs 
+-  Vision Encoder (ViT) CUDA Graphs 
+-  CustomOp 
+-  Dual Batch Overlap 
+-  How to debug the vLLM-torch.compile integration 
+-  Fused MoE Modular Kernel 
+-  Fusion torch.compile passes 
+-  Integration with Hugging Face 
+-  Hybrid KV Cache Manager 
+-  Logits Processors 
+-  Metrics 
+-  Multi-Modal Data Processing 
+-  Model Runner V2 Design Document 
+-  Fused MoE Kernel Features 
+-  Python Multiprocessing 
+-  Optimization Levels 
+-  P2P NCCL Connector 
+-  Paged Attention 
+-  Automatic Prefix Caching 
+-  torch.compile integration 
+-  torch.compile with Multimodal Encoders 
+-  Benchmarking 
+-  API Reference 
+-  benchmarks 
+-  sweep 
+-  compilation 
+-  utility 
+-  config 
+-  device_allocator 
+-  distributed 
+-  ec_transfer 
+-  elastic_ep 
+-  eplb 
+-  kv_transfer 
+-  mooncake 
+-  moriio 
+-  offloading 
+-  p2p 
+-  weight_transfer 
+-  engine 
+-  entrypoints 
+-  cli 
+-  mcp 
+-  openai 
+-  completion 
+-  engine 
+-  generate 
+-  models 
+-  parser 
+-  realtime 
+-  responses 
+-  speech_to_text 
+-  pooling 
+-  classify 
+-  embed 
+-  pooling 
+-  score 
+-  sagemaker 
+-  serve 
+-  disagg 
+-  elastic_ep 
+-  instrumentator 
+-  lora 
+-  profile 
+-  render 
+-  rlhf 
+-  rpc 
+-  sleep 
+-  tokenize 
+-  inputs 
+-  kernels 
+-  logging_utils 
+-  lora 
+-  ops 
+-  triton_ops 
+-  xpu_ops 
+-  punica_wrapper 
+-  model_executor 
+-  scaled_mm 
+-  layers 
+-  fla 
+-  fused_moe 
+-  oracle 
+-  prepare_finalize 
+-  router 
+-  runner 
+-  mamba 
+-  pooler 
+-  tokwise 
+-  quantization 
+-  transform 
+-  quark 
+-  utils 
+-  rotary_embedding 
+-  model_loader 
+-  models 
+-  offloader 
+-  warmup 
+-  multimodal 
+-  processing 
+-  parser 
+-  platforms 
+-  plugins 
+-  lora_resolvers 
+-  profiler 
+-  ray 
+-  reasoning 
+-  renderers 
+-  tokenizers 
+-  tool_parsers 
+-  tracing 
+-  transformers_utils 
+-  configs 
+-  processors 
+-  triton_utils 
+-  usage 
+-  utils 
+-  v1 
+-  ops 
+-  core 
+-  engine 
+-  executor 
+-  kv_offload 
+-  worker 
+-  metrics 
+-  pool 
+-  sample 
+-  ops 
+-  spec_decode 
+-  structured_output 
+-  worker 
+-  mm 
+-  model_states 
+-  pool 
+-  sample 
+-  spec_decode 
+-  CLI Reference 
+- Community
+-  Blog 
+-  Forum 
+-  Slack 
+
+# 404 - Not found
+
+Back to top
+
+# GUEST POST - Crafting Unique AI Personas: Harnessing the Power of Logit Bias in Large Language Models | Microsoft Agent Framework
+URL: https://devblogs.microsoft.com/semantic-kernel/guest-post-crafting-unique-ai-personas-harnessing-the-power-of-logit-bias-in-large-language-models/
+Author: Anthony Puppo
+
+GUEST POST - Crafting Unique AI Personas: Harnessing the Power of Logit Bias in Large Language Models | Microsoft Agent Framework 
+
+ 
+
+ 
+
+ 
+
+ 
+
+ 
+
+ 
+
+ 
+
+Anthony Puppo
+
+Software Engineer
+
+ 
+
+Large Language Models (LLMs) have revolutionized our interaction with software. However, there’s a catch – their responses can be monotonous and impersonal. This is where ‘personas’ come in. They add a human touch to LLMs, transforming generic outputs into customized responses that resonate with users. This is particularly handy in applications like customer service bots and virtual assistants. But how do we create these personas without hefty costs or time investments? The good news is, we can tweak a set of common parameters in most LLMs to influence their output, and that’s what we’ll explore today.
+
+The examples in this blog post utilize C#, Semantic Kernel, SharpToken and the OpenAI API. If you’d like to follow along and experiment yourself, first create a new console project:
+
+```default
+dotnet new console --framework net7.0
+```
+
+Then install dependencies using NuGet:
+
+```default
+dotnet add package Microsoft.SemanticKernel --prerelease
+dotnet add package SharpToken
+```
+
+Additionally, I’ve written a small demo application that utilizes some of the techniques discussed in this blog. It is open-source and available on GitHub.
+
+### Introduction To How LLMs Work
+
+In simplified terms, LLMs function a bit like a predictive text engine. They interpret a sentence with each word or part of a word as a ‘token’. The tokens are then transformed into a numerical format that the model can process.
+
+```csharp
+var encoding = GptEncoding.GetEncoding("gpt-3.5-turbo");
+var rawTokens = encoding.Encode("Wonderful day we're having!");
+var textTokens = rawTokens.Select((x) => $""{encoding.Decode(new() { x })}"").ToList();
+
+Console.WriteLine($"Raw tokens: {string.Join(", ", rawTokens)}");
+Console.WriteLine($"Tokenized text: {string.Join(", ", textTokens)}");
+
+// Output:
+// Raw tokens: 62372, 1285, 1938, 584, 2351, 3515, 0
+// Tokenized text: "Wonder", "ful", " day", " we", "'re", " having", "!"
+```
+
+Based on its prior training, the model predicts what should come next in the sequence. For example, after “I don’t like”, the model might suggest “apples”. This prediction can be thought of as the model’s first draft — it’s good, but can we make it better?
+
+### LLM Parameters
+
+Consumers of LLMs have the ability to adjust certain parameters. This can yield creative, varied, and engaging results.
+
+##### Temperature
+
+This parameter adjusts the entropy of our model’s output. A high temperature makes the model’s output diverse and creative, while a lower temperature results in more focused and predictable responses.
+
+##### Top P
+
+Top P (also known as nucleus sampling) guides the selection of next tokens based on cumulated probabilities. This is a more nuanced measure of controlling randomness that can often lead to more diverse outputs. For example, if the model is predicting the next word in “The cat climbed up the ___”, and the options tree, roof, and wall add up to around 90% probability, Top P, if set at 90%, restricts the model to select among those three options.
+
+##### Frequency and Presence Penalties
+
+Frequency penalty discourages the overuse of specific tokens, and presence penalty penalizes tokens previously used in the output, irrespective of frequency. These mechanisms can be instrumental in quelling repetition and promoting diversity.
+
+##### Logit Bias
+
+Logit bias directly manipulates the logits (the raw, unnormalized scores predicted by the model) for specific tokens before they are passed through the softmax function for probability distribution. By adjusting the logit bias, one can promote or demote particular tokens. For instance, if we want the model to avoid using a certain token, we can assign a negative logit bias to it, making it less likely to be chosen. Likewise, assigning a higher bias will have the model favor the token.
+
+### Persona Generation Using Logit Bias
+
+For a practical demonstration, let’s consider a scenario where we desire our model to generate shorter sentences. To achieve this, we can manipulate the bias for common punctuation such as “.”, “!”, and “?”.
+
+First, setup the kernel so we can interact with the model:
+
+```csharp
+var kernel = new KernelBuilder()
+    .WithOpenAIChatCompletionService("gpt-3.5-turbo", "<your-openai-api-key>")
+    .Build();
+```
+
+Then call it with our custom settings:
+
+```csharp
+var result = await kernel.InvokeSemanticFunctionAsync(
+    "Describe a rainbow.",
+    requestSettings: new OpenAIRequestSettings()
+    {
+        Temperature = 0,
+        TopP = 1,
+        FrequencyPenalty = 0,
+        PresencePenalty = 0,
+        TokenSelectionBiases = new[] { ".", "!", "?" }
+            .SelectMany((x) => encoding.Encode(x))
+            .ToDictionary((x) => x, (x) => 10)
+    });
+
+Console.WriteLine(result);
+```
+
+And we get the following:
+
+A rainbow is a beautiful and natural phenomenon. It appears as a circular arc of colors in the sky. It is formed when sunlight is refracted, or bent, as it passes through raindrops. The sunlight is then reflected inside the raindrop and refracted again. This process causes the light to separate into its component colors. The colors of a rainbow, from top to bottom, are red, orange, yellow, green, blue, indigo, and violet. The colors are vibrant and distinct. The rainbow usually appears after rain showers when the sun is still shining. It can also be seen near waterfalls or fountains. The sight of a rainbow is often associated with joy, hope, and wonder. It is a mesmerizing display of nature’s beauty.
+
+Conversely, if we make the bias negative:
+
+```csharp
+// Surrounding code omitted for brevity...
+TokenSelectionBiases = new[] { ".", "!", "?" }
+    .SelectMany((x) => encoding.Encode(x))
+    .ToDictionary((x) => x, (x) => -10)
+```
+
+We then get something like this:
+
+A rainbow is a beautiful and natural phenomenon that occurs when sunlight is refracted, or bent, by water droplets in the air, creating a spectrum of colors in the sky. Typically, a rainbow appears as a semi-circular arc of vibrant colors, with red being the outermost color and violet being the innermost color, although sometimes a full circle can be seen in certain conditions. The colors of a rainbow, in order, are red, orange, yellow, green, blue, indigo, and violet, often remembered by the acronym ROYGBIV. Each color of the rainbow is distinct and blends seamlessly into the next, creating a stunning display of hues that can be seen against a backdrop of dark clouds or a clear blue sky. Rainbows are often seen after rain showers when the sun emerges from behind the clouds, casting its rays onto the raindrops in the air, causing them to act as tiny prisms that refract the sunlight and create the colorful spectrum. The sight of a rainbow is often associated with feelings of joy, wonder, and hope, as it is a symbol of beauty and harmony in nature. Rainbows are not physical objects that can be touched or approached, but rather optical illusions that appear to be located at a specific distance from the observer, making them seem elusive and magical. Overall, a rainbow is a breathtaking and ephemeral display of colors that captivates the imagination and reminds us of the wonders of the natural world around us.
+
+The first response is more concise and straightforward, providing a clear and simple explanation of a rainbow. It uses a more casual and conversational tone, making it easier to understand for a general audience.
+
+The second is more detailed and comprehensive, providing a more scientific explanation of a rainbow. It uses a more formal and academic tone, making it suitable for a more knowledgeable audience or someone seeking a deeper understanding. The language is more descriptive and the sentences are longer, contributing to a more elaborate and thorough explanation.
+
+The effects of tweaking logit_bias are evident in the given examples, and these modifications show how we can mold the model’s responses to be more in line with a specific persona. By amplifying or diminishing this bias, we can guide the model to generate responses that are concise or verbose, casual or formal, simple or detailed, depending on the desired personality. However, the key lies in balance. Overdoing it might result in an overbearing or inconsistent persona, while underdoing it might make the persona feel generic.
+
+### Next Steps
+
+So, what are some of the options for putting the topics discussed here into practice?
+
+1. Experiment with Logit Bias: Get hands-on experience with this feature. Start with simple tweaks to the bias values and observe how the output changes. As you gain familiarity, attempt to create a more complex persona by adjusting the bias for a wider range of tokens.
+2. Dive into Stylometry: Learn more about stylometry. This field of study can provide insights into how writing styles can be quantified and analyzed, which can be helpful in creating more nuanced personas.
+3. Implement Part-of-Speech Tagging: Incorporate part-of-speech tagging. It can be useful in understanding the grammatical structure of the sentences generated by the model (or text from a pre-existing persona you are attempting to emulate). This understanding can help you tune the logit bias more effectively.
+4. Randomize Character Creation: Create a corpus of words relevant to the desired persona. Use this corpus to randomly assign attributes to the model’s persona. This can add an element of unpredictability to the model, making it more engaging.
+5. Explore Token Frequencies and TF-IDF: Rather than merely looking at the plain text, consider tokenizing the text. This approach can be combined with Term Frequency-Inverse Document Frequency (TF-IDF) to assess the frequency of model tokens. This insight can guide the adjustment of logit bias values more appropriately (since models are operating at the token level).
+6. Combine Parameters: Don’t limit yourself to logit bias. Try combining it with other parameters like temperature and top_p for more nuanced control over the output. Remember, the aim is to create a persona that is consistent, engaging, and believable.
+
+### Closing Thoughts
+
+Crafting unique personas can be a tricky pursuit. Logit bias, however, offers a promising starting point. It’s a tool to help you steer your models outputs towards a more personalized touch. Yet, it’s important to note, it’s but one piece of the puzzle. While other parameters might not singularly make a huge impact in persona development, their combined use could unlock more possibilities. The journey to mastering persona creation in LLMs is an intriguing one, and hopefully, this has given you a useful compass to navigate it.
+
+ 
+
+ 
+
+Category
+
+Semantic Kernel
+
+Topics
+
+Personas Semantic Kernel
+
+Share
+
+ 
+
+## Author
+
+Anthony Puppo
+
+Software Engineer
+
+ 
+
+ 
+
+ 
+
+November 1, 2023
+
+### What to expect from v1 and beyond for Semantic Kernel.
+
+Matthew Bolanos
+
+November 2, 2023
+
+### AutoGen Agents Meet Semantic Kernel
+
+John Maeda
+
+ 
+
+## Stay informed
+
+Get notified when new posts are published.
+
+Email *
+
+Country/Region * Select...United StatesAfghanistanÅland IslandsAlbaniaAlgeriaAmerican SamoaAndorraAngolaAnguillaAntarcticaAntigua and BarbudaArgentinaArmeniaArubaAustraliaAustriaAzerbaijanBahamasBahrainBangladeshBarbadosBelarusBelgiumBelizeBeninBermudaBhutanBoliviaBonaireBosnia and HerzegovinaBotswanaBouvet IslandBrazilBritish Indian Ocean TerritoryBritish Virgin IslandsBruneiBulgariaBurkina FasoBurundiCabo VerdeCambodiaCameroonCanadaCayman IslandsCentral African RepublicChadChileChinaChristmas IslandCocos (Keeling) IslandsColombiaComorosCongoCongo (DRC)Cook IslandsCosta RicaCôte dIvoireCroatiaCuraçaoCyprusCzechiaDenmarkDjiboutiDominicaDominican RepublicEcuadorEgyptEl SalvadorEquatorial GuineaEritreaEstoniaEswatiniEthiopiaFalkland IslandsFaroe IslandsFijiFinlandFranceFrench GuianaFrench PolynesiaFrench Southern TerritoriesGabonGambiaGeorgiaGermanyGhanaGibraltarGreeceGreenlandGrenadaGuadeloupeGuamGuatemalaGuernseyGuineaGuinea-BissauGuyanaHaitiHeard Island and McDonald IslandsHondurasHong Kong SARHungaryIcelandIndiaIndonesiaIraqIrelandIsle of ManIsraelItalyJamaicaJan MayenJapanJerseyJordanKazakhstanKenyaKiribatiKoreaKosovoKuwaitKyrgyzstanLaosLatviaLebanonLesothoLiberiaLibyaLiechtensteinLithuaniaLuxembourgMacau SARMadagascarMalawiMalaysiaMaldivesMaliMaltaMarshall IslandsMartiniqueMauritaniaMauritiusMayotteMexicoMicronesiaMoldovaMonacoMongoliaMontenegroMontserratMoroccoMozambiqueMyanmarNamibiaNauruNepalNetherlandsNew CaledoniaNew ZealandNicaraguaNigerNigeriaNiueNorfolk IslandNorth MacedoniaNorthern Mariana IslandsNorwayOmanPakistanPalauPalestinian AuthorityPanamaPapua New GuineaParaguayPeruPhilippinesPitcairn IslandsPolandPortugalPuerto RicoQatarRéunionRomaniaRwandaSabaSaint BarthélemySaint Kitts and NevisSaint LuciaSaint MartinSaint Pierre and MiquelonSaint Vincent and the GrenadinesSamoaSan MarinoSão Tomé and PríncipeSaudi ArabiaSenegalSerbiaSeychellesSierra LeoneSingaporeSint EustatiusSint MaartenSlovakiaSloveniaSolomon IslandsSomaliaSouth AfricaSouth Georgia and South Sandwich IslandsSouth SudanSpainSri LankaSt HelenaAscensionTristan da CunhaSurinameSvalbardSwedenSwitzerlandTaiwanTajikistanTanzaniaThailandTimor-LesteTogoTokelauTongaTrinidad and TobagoTunisiaTurkeyTurkmenistanTurks and Caicos IslandsTuvaluU.S. Outlying IslandsU.S. Virgin IslandsUgandaUkraineUnited Arab EmiratesUnited KingdomUruguayUzbekistanVanuatuVatican CityVenezuelaVietnamWallis and FutunaYemenZambiaZimbabwe
+
+I would like to receive the Microsoft Agent Framework Newsletter. Privacy Statement.
+
+Subscribe
+
+Follow this blog
+
+ 
+
+ 
+
+ 
+
+ 
+
+ 
+
+ 
+
+ 
+
+Sign in
+
+Theme