TorchServe

TorchServe is a performant, flexible and easy to use tool for serving PyTorch models in production.

What’s going on in TorchServe?

• High performance Llama 2 deployments with AWS Inferentia2 using TorchServe

• Naver Case Study: Transition From High-Cost GPUs to Intel CPUs and oneAPI powered Software with performance

• Run multiple generative AI models on GPU using Amazon SageMaker multi-model endpoints with TorchServe and save up to 75% in inference costs

• Deploying your Generative AI model in only four steps with Vertex AI and PyTorch

• PyTorch Model Serving on Google Cloud TPUv5

• Monitoring using Datadog

• Torchserve Performance Tuning, Animated Drawings Case-Study

• Walmart Search: Serving Models at a Scale on TorchServe

• Scaling inference on CPU with TorchServe

• TorchServe C++ backend

• TorchServe with Intel® Extension for PyTorch*

• Grokking Intel CPU PyTorch performance from first principles: a TorchServe case study

• Grokking Intel CPU PyTorch performance from first principles( Part 2): a TorchServe case study

• Case Study: Amazon Ads Uses PyTorch and AWS Inferentia to Scale Models for Ads Processing

• Optimize your inference jobs using dynamic batch inference with TorchServe on Amazon SageMaker

• Using AI to bring children’s drawings to life

• Model Serving in PyTorch

• Evolution of Cresta’s machine learning architecture: Migration to AWS and PyTorch

• Explain Like I’m 5: TorchServe

• How to Serve PyTorch Models with TorchServe

• How to deploy PyTorch models on Vertex AI

• Quantitative Comparison of Serving Platforms

---

TorchServe Quick Start

Topics: Quick Start

Learn how to install TorchServe and serve models.

Running TorchServe

Topics: Running TorchServe

Indepth explanation of how to run TorchServe

Why TorchServe

Topics: Examples

Various TorchServe use cases

TorchServe GenAI Use Cases

Topics: Use Cases

Showcasing GenAI deployment scenarios and use cases

Performance

Topics: Performance,Troubleshooting

Guides and best practices on how to improve perfromance when working with TorchServe

Metrics

Topics: Metrics,Performance,Troubleshooting

Collecting and viewing Torcherve metrics

Large Model Inference

Topics: Large-Models,Performance

Serving Large Models with TorchServe

Troubleshooting

Topics: Troubleshooting,Performance

Various updates on Torcherve and use cases.

TorchServe Security Policy

Topics: Security

Security Policy

FAQs

Topics: FAQS

Various frequently asked questions.