Introduction xix

Chapter 1: Introduction 1

Overview of GenAI Applications and Large Language Models 1

The Rise of Large Language Models 1

Neural Networks, Transformers, and Beyond 2

GenAI vs. LLMs: What’s the Difference? 5

The Three-Layer GenAI Application Stack 6

The Infrastructure Layer 6

The Model Layer 7

The Application Layer 8

Paths to Productionizing GenAI Applications 9

Sample LLM-Powered Chat Application 11

The Importance of Cost Optimization 12

Cost Assessment of the Model Inference Component 12

Cost Assessment of the Vector Database Component 19

Benchmarking Setup and Results 20

Other Factors to Consider 23

Cost Assessment of the Large Language Model Component 24

Summary 27

Chapter 2: Tuning Techniques for Cost Optimization 29

Fine-Tuning and Customizability 29

Basic Scaling Laws You Should Know 30

Parameter-Efficient Fine-Tuning Methods 32

Adapters Under the Hood 33

Prompt Tuning 34

Prefix Tuning 36

P-tuning 39

IA3 40

Low-Rank Adaptation 44

Cost and Performance Implications of PEFT Methods 46

Summary 48

Chapter 3: Inference Techniques for Cost Optimization 49

Introduction to Inference Techniques 49

Prompt Engineering 50

Impact of Prompt Engineering on Cost 50

Estimating Costs for Other Models 52

Clear and Direct Prompts 53

Adding Qualifying Words for Brief Responses 53

Breaking Down the Request 54

Example of Using Claude for PII Removal 55

Conclusion 59

Providing Context 59

Examples of Providing Context 60

RAG and Long Context Models 60

Recent Work Comparing RAG with Long Content Models 61

Conclusion 62

Context and Model Limitations 62

Indicating a Desired Format 63

Example of Formatted Extraction with Claude 63

Trade-Off Between Verbosity and Clarity 66

Caching with Vector Stores 66

What Is a Vector Store? 66

How to Implement Caching Using Vector Stores 66

Conclusion 69

Chains for Long Documents 69

What Is Chaining? 69

Implementing Chains 69

Example Use Case 70

Common Components 70

Tools That Implement Chains 72

Comparing Results 76

Conclusion 76

Summarization 77

Summarization in the Context of Cost and Performance 77

Efficiency in Data Processing 77

Cost-Effective Storage 77

Enhanced Downstream Applications 77

Improved Cache Utilization 77

Summarization as a Preprocessing Step 77

Enhanced User Experience 77

Conclusion 77

Batch Prompting for Efficient Inference 78

Batch Inference 78

Experimental Results 80

Using the accelerate Library 81

Using the DeepSpeed Library 81

Batch Prompting 82

Example of Using Batch Prompting 83

Model Optimization Methods 83

Quantization 83

Code Example 84

Recent Advancements: GPTQ 85

Parameter-Efficient Fine-Tuning Methods 85

Recap of PEFT Methods 85

Code Example 86

Cost and Performance Implications 87

Summary 88

References 88

Chapter 4: Model Selection and Alternatives 89

Introduction to Model Selection 89

Motivating Example: The Tale of Two Models 89

The Role of Compact and Nimble Models 90

Examples of Successful Smaller Models 91

Quantization for Powerful but Smaller Models 91

Text Generation with Mistral 7B 93

Zephyr 7B and Aligned Smaller Models 94

CogVLM for Language-Vision Multimodality 95

Prometheus for Fine-Grained Text Evaluation 96

Orca 2 and Teaching Smaller Models to Reason 98

Breaking Traditional Scaling Laws with Gemini and Phi 99

Phi 1,
1.5, and 2 B Models 100

Gemini Models 102

Domain-Specific Models 104

Step 1 - Training Your Own Tokenizer 105

Step 2 - Training Your Own Domain-Specific Model 107

More References for Fine-Tuning 114

Evaluating Domain-Specific Models vs. Generic Models 115

The Power of Prompting with General-Purpose Models 120

Summary 122

Chapter 5: Infrastructure and Deployment Tuning Strategies 123

Introduction to Tuning Strategies 123

Hardware Utilization and Batch Tuning 124

Memory Occupancy 126

Strategies to Fit Larger Models in Memory 128

KV Caching 130

PagedAttention 131

How Does PagedAttention Work? 131

Comparisons, Limitations, and Cost Considerations 131

AlphaServe 133

How Does AlphaServe Work? 133

Impact of Batching 134

Cost and Performance Considerations 134

S3: Scheduling Sequences with Speculation 134

How Does S3 Work? 135

Performance and Cost 135

Streaming LLMs with Attention Sinks 136

Fixed to Sliding Window Attention 137

Extending the Context Length 137

Working with Infinite Length Context 137

How Does StreamingLLM Work? 138

Performance and Results 139

Cost Considerations 139

Batch Size Tuning 140

Frameworks for Deployment Configuration Testing 141

Cloud-Native Inference Frameworks 142

Deep Dive into Serving Stack Choices 142

Batching Options 143

Options in DJL Serving 144

High-Level Guidance for Selecting Serving Parameters 146

Automatically Finding Good Inference Configurations 146

Creating a Generic Template 148

Defining a HPO Space 149

Searching the Space for Optimal Configurations 151

Results of Inference HPO 153

Inference Acceleration Tools 155

TensorRT and GPU Acceleration Tools 156

CPU Acceleration Tools 156

Monitoring and Observability 157

LLMOps and Monitoring 157

Why Is Monitoring Important for LLMs? 159

Monitoring and Updating Guardrails 160

Summary 161

Conclusion 163

Index 181