TorchServe on linux aarch64 - Experimental

TorchServe has been tested to be working on linux aarch64 for some of the examples.

• Tested this on Amazon Graviton 3 instance(m7g.4x.large)

Installation

Currently installation from PyPi or installing from source works

python ts_scripts/install_dependencies.py
pip install torchserve torch-model-archiver torch-workflow-archiver

Optimizations

You can also enable this optimizations for Graviton 3 to get an improved performance. More details can be found in this blog

export DNNL_DEFAULT_FPMATH_MODE=BF16
export LRU_CACHE_CAPACITY=1024

Example

This example on Text to Speech synthesis was verified to be working on Graviton 3

To Dos

• CI

• Regression tests