Skip to content

Securing TEE Apps: A Developer's Guide

Published: at 02:00 PM

Written by: prateek, roshan, siddhartha and linguine (Marlin), krane (Asula)

Thanks to jesse, jim, lamby, nick, ren, rajiv and yangwao for their thoughtful feedback and comments.


Trusted Execution Environments (or TEEs) have become increasingly popular since Apple announced its Private Cloud and NVIDIA made Confidential Computing available in GPUs. Their confidentiality guarantees help secure user data (which may include private keys) while the isolation properties ensure that the execution of programs deployed on them can’t be tampered with–whether by a human, another program or the operating system. It should thus be no surprise that they have also found product-market fit in the fast-evolving crypto x AI space.

As with any new technology, TEEs are arguably going through a period of unchecked optimism and experimentation. This piece hopes to serve as a practical guide for the discerning developer to understand what TEEs are, their security model, common pitfalls and best practices to use them safely. Although fairly technical and builder-oriented, we hope that the larger community can use this guide to ask developers the right questions and promote a safer ecosystem.

NB: In order to make the text accessible, we knowingly replace well-known TEE terminologies with simpler equivalents and take similar liberty for the explanation of domain-specific concepts.

What are TEEs?

TEEs are an isolated environment in a processor or data center where programs can run without any interference from the rest of the system. This is achieved through careful system design which enforces strict access control policies on how other external programs can interact with applications running in them. TEEs are ubiquitous in mobile phones, servers, PCs and cloud environments making them very accessible and affordable.

The above paragraph may sound vague and abstract, but it is intentionally so. Different server and cloud vendors implement TEEs differently.

TEEs Illustrated

TEEs Illustrated. Source

Most readers are likely using devices that they logged into using biometric data. Ever wondered why no malicious app, website or jailbroken OS has been able to leak a database of such private data (stolen from the device, not a government/corporate server)? It is because in addition to the data being encrypted, the circuitry in your device simply wouldn’t allow programs to access the areas of memory and processor that process such data! Biometrics on phones and computers are stored on TEEs.

Hardware ledgers are another example crypto users would be familiar with. A ledger connects to a machine for power and sandboxed communication, but stores seed phrases itself.

In both the above cases, users trust the device manufacturer to correctly design chips and provide appropriate firmware updates to prevent confidential data from being exported or viewed.

The security model

Unfortunately, the large variety of TEE implementations means each implementation (Intel SGX, Intel TDX, AMD SEV, AWS Nitro Enclaves, ARM TrustZone) requires its own analysis. We’ll largely talk about Intel SGX, TDX and AWS Nitro for the rest of this piece as they are most relevant when it comes to web3 for their popularity, extent of available dev tooling and ease to work with.

A typical workflow for an application deployed in a TEE is as follows:

App Deployment Workflow with TEEs

App Deployment Workflow with TEEs.

Clearly, there are 3 potential points of trust:

Thankfully, eliminating the developer and host as points of failure are exactly what well-designed TEE applications are intended to do. They do so using Reproducible Builds and Remote Attestations.

Reproducible Builds: A typical program today includes a number of packages. These packages change with time as new updates and patches are released. If the identifier of a program is its entire code’s hash, any change in the version number of a package used or the way the dependency tree is created before calculating the measurement (code hashes or Platform Configuration Register values), alters the measurement and thus, the identifier of the program.

As the name suggests, the concept of reproducible builds involves following a set of standards such that a certain version of code when built using the standardized pipeline always returns the same code measurement.

Nix is a popular tool to build reproducible executables. Once the source code is public, anyone can inspect or audit the code to ensure the developer has not inserted anything untoward and then build it to arrive at the same code measurements that the developer would have deployed in production. But how do we know the measurement of the binary that’s running in the TEE? Enter Remote Attestations!

Remote Attestations: Remote Attestations are signed messages from the TEE platform (a trusted party) that contain measurements of the program, the TEE platform version, etc. It lets an external observer know that a certain program (with specified measurements) is being executed in a secure location (real TEE with a secure platform version) that no one can access.

Attestation Verification Workflow with TEEs

Attestation Verification Workflow with TEEs.

Together, Reproducible Builds and Remote Attestations let an external observer know the precise code that’s being run within a TEE along with the TEE platform version, preventing the developer or the host from lying about the code being run or where it is running.

However, the vendor is the ultimate root of trust in the case of TEEs. If it wishes to be malicious (or creates vulnerable hardware), it can always sign on to attestations for programs running outside a TEE (which may or may not be the program the user thinks they are giinteracting with). Either by itself or in collusion with the developer or host, it can pass off the execution of altered code to be validly attested. Threat models which consider the vendor a possible attack vector should thus avoid relying solely on TEEs and are better off coupling them with zero-knowledge proofs or consensus mechanisms.

The allure of TEEs

crypto x AI loves TEEs

crypto x AI loves TEEs.

The properties that have made TEEs especially popular, especially for the deployment of AI agents, in our opinion are:

For better or worse, there simply exists no alternative to TEEs that satisfy the above requirements of a web3 AI developer. Similar to how it is believed that higher throughput enables new use-cases for blockchains, TEEs expand the surface area of what onchain applications can do, potentially unlocking new use cases.

TEEs aren’t a magic bullet that can fix broken system design

Programs operating in a TEE are still vulnerable to a range of attacks and bugs. Just like smart contracts, they are susceptible to a range of issues. For simplicity, we categorize possible loopholes as below:

Developer negligence

Whether intentional or unintentional, developers can weaken the security guarantees of programs running in TEEs through their choices, actions and/or omissions. These include:

Runtime vulnerabilities

Even a careful developer with a robust codebase may fall prey to attacks that can be executed at runtime. Developers have to think through whether any of the following can affect the security guarantees of their program.

Architectural flaws

Zooming out, even before starting to write any code, developers should think through the stack they plan to use to build the enclave application. Decisions are much easier to make when priorities are clear from the start!

Operational oversight

Last but not the least, there are some practical considerations on how the TEE is operated by the host:

Until now, we’ve primarily described features that make TEEs exciting, especially for crypto x AI. We’ve also laid out a laundry list of security issues that can arise with TEE deployments due to a multitude of factors. In the next section, we’ll describe different levels of security developers can impart to their TEEs when deploying them.

Building enclaves securely

We divide our recommendations as follows:

Most secure scenario: No external dependencies

Creating highly secure applications can involve eliminating external dependencies like external inputs, APIs, or services, reducing the attack surface. This approach ensures that the application operates in a self-contained manner, with no external interactions that could compromise its integrity or security. While this strategy may limit functionality, it offers high security in scenarios where trust and confidentiality are preferred.

This level of security is achievable for several crypto x AI use cases if the models are run locally.

Necessary precautions to take

Irrespective of whether or not the application has external dependencies, the following are a must!

Use-case dependent suggestions

Based on the use case the application targets and how it is structured, the following tips could be useful in making your application safer.

Missing pieces

There exists an inherent tension between the agile development process developers expect when using TEEs and the security measures that are required to build safe applications using them.

While this post tries to lay out the most common issues and precautions when working with TEEs, there still remain several open questions the developer community could think about:

Looking ahead

Enclaves and TEE have clearly become an extremely exciting area of computing. As previously mentioned, the ubiquity of AI and its constant access to users’ sensitive data has meant large tech companies like Apple and NVIDIA are using enclaves in their products and offering enclaves as part of their product offerings.

On the other hand, the crypto community has always been very security conscious. As developers try to expand the suite of onchain applications and use-cases, we’ve seen TEEs become popular as a solution that offers the right trade-off of features and trust assumptions. While, TEEs are not as trust-minimized as full ZK solutions, we expect they will be the avenue that will slowly meld offerings from crypto companies and big tech for the first time.