Recently, at my company, we changed our CI/CD EC2 instances from t2 family to t3, mainly because it’s cheaper and more performant. Little did we know, this change made some of our apps crash immediately when deployed to AWS Fargate.

TL;DR: New AWS EC2 instances have a different CPU than the one used by Fargate, which doesn’t guarantee that your app will run correctly.

It’s fair to say that most of the apps will run fine even if built on a different infrastructure, but in our case, we had Python apps that were using some well-known data science Python libraries (hnswlib, nmslib, spacy, …) that depend on the C++ compiler. Which means that the typical pip install hnswlib was actually compiling code. You can check out this file https://github.com/nmslib/hnswlib/blob/master/python_bindings/setup.py and this file https://github.com/explosion/spaCy/blob/master/setup.py for more info.

The problem is that the CPU on newer EC2 instances support AVX-512 instructions which are designed to accelerate convolutional neural network-based algorithms. AWS Fargate, on the other hand, does not. The C++ compiler takes advantage of these instructions to get the maximum performance of the CPU. Installing the aforementioned Python libraries, have compiled C++ code that was unexecutable on Fargate’s CPU. The result is a single pretty error message when running the AWS Fargate tasks: Illegal instruction (core dumped).

It is possible to disable AVX-512 instructions during C++ code compilation but only on the compilation-command level, to do so, we would need to write our own setup.py for as many data science libraries as we use, which was impossible.

So, until AWS Fargate starts supporting AVX-512 while using the same new Intel Skylake CPU, we’re stuck with using T2, C4 or M4 EC2 instances in our CI/CD system.

Bonus: This is an interesting article about Linus Torvalds “hoping a painful death to AVX-512”.