Skip to content

Engine loading Performance Numbers

These numbers were run on a Mac Pro 2019 (2.8 Ghz Intel Core i7)

Because of the warning above, here are some benchmarking numbers to get an idea of the performance of loading in multiple engines vs loading in only one engine.

Benchmarking Numbers for only having one engine loaded in.

Engine Name Model Throughput Total p50 Total p90 inference p50 inference p90
1 Engine MXNet resnet 50 10.19, 5000 iteration / 490602 ms. 95.520 ms 103.606 ms 90.120 ms 97.954 ms
MultiEngine MXNet resnet 50 10.17, 5000 iteration / 491656 ms. 96.084 ms 104.198 ms 90.696 ms 98.692 ms
1 Engine TensorFlow resnet 50 7.15, 2500 iteration / 349777 ms. 123.493 ms 142.029 ms 110.592 ms 126.268 ms
MultiEngine TensorFlow resnet 50 6.84, 2500 iteration / 365355 ms. 124.743 ms 160.793 ms 110.935 ms 144.142 ms
1 Engine PyTorch resnet 50 6.87, 5000 iteration / 728242 ms. 140.246 ms 155.400 ms 133.463 ms 147.721 ms
MultiEngine PyTorch resnet 50 6.59, 5000 iteration / 759261 ms. 144.358 ms 168.535 ms 137.336 ms 160.346 ms
1 Engine MXNet resnet 18 22.04, 5000 iteration / 226893 ms. 43.487 ms 52.077 ms 38.105 ms 45.660 ms
MultiEngine MXNet resnet 18 22.64, 5000 iteration / 220807 ms. 42.750 ms 47.923 ms 37.482 ms 42.109 ms
1 Engine Tensorflow MobileNet 1.0 30.45, 5000 iteration / 164225 ms. 31.190 ms 34.780 ms 19.983 ms 22.836 ms
MultiEngine Tensorflow MobileNet 1.0 29.17, 5000 iteration / 171395 ms. 31.860 ms 38.047 ms 20.631 ms 24.798 ms
1 Engine PyTorch resnet 18 14.90, 5000 iteration / 335509 ms. 64.268 ms 73.352 ms 57.620 ms 65.733 ms
MultiEngine PyTorch resnet 18 13.54, 5000 iteration / 369255 ms. 71.959 ms 81.688 ms 64.580 ms 73.439 ms