Eleos: Exit-Less OS Services for SGX Enclaves Meni Orenbach Marina Minkin Pavel Lifshits Mark Silberstein Accelerated Computing Systems Lab Belgrade, Serbia April 23-26, 2017

What do we do? Improve performance: I/O intensive & memory demanding SGX enclaves

Why? Cost of SGX execution for these applications is high

How? In-enclave System Calls & User Managed Virtual Memory

Results Eleos vs vanilla SGX

2x

Throughput: memcached & face verification servers Even for 5x available enclave memory Available for Linux, Windows*

(*) Without Eleos, these applications crash in Windows enclaves 24 April@Eurosys' 2017

Meni Orenbach, Technion

2

●

Background

●

Motivation

●

Overhead analysis

●

Eleos design

●

Evaluation

24 April@Eurosys' 2017

Meni Orenbach, Technion

3

SGX enclaves are already here! ●

Secured execution environment

●

Reversed sandbox

●

Small TCB

●

Private code & data –

●

Application Enclave Enclave

Confidentiality

–

Integrity

–

Freshness

Operating system

Only CPU is trusted

24 April@Eurosys' 2017

Meni Orenbach, Technion

4

SGX enclaves are already here! ●

Secured execution environment

●

Reversed sandbox

●

Small TCB

●

Private code & data –

●

Application Enclave Enclave

Confidentiality

–

Integrity

–

Freshness

Operating system

Only CPU is trusted

24 April@Eurosys' 2017

Meni Orenbach, Technion

5

SGX enclaves are already here! ●

Secured execution environment

●

Reversed sandbox

●

Small TCB

●

Private code & data –

●

Application Enclave Enclave

Confidentiality

–

Integrity

–

Freshness

Operating system

Only CPU is trusted

24 April@Eurosys' 2017

Meni Orenbach, Technion

6

SGX enclaves are already here! ●

Secured execution environment

●

Reversed sandbox

●

Small TCB

●

Private code & data –

●

Application Enclave Enclave

Confidentiality

–

Integrity

–

Freshness

Operating system

Only CPU is trusted

24 April@Eurosys' 2017

Meni Orenbach, Technion

7

SGX enclaves are already here! ●

Secured execution environment

●

Reversed sandbox

●

Small TCB

●

Private code & data –

●

Application Enclave Enclave

Confidentiality

–

Integrity

–

Freshness

Operating system

Only CPU is trusted

24 April@Eurosys' 2017

Meni Orenbach, Technion

8

SGX enclaves are already here! ●

Secured execution environment

●

Reversed sandbox

●

Small TCB

●

Private code & data –

●

Application Enclave Enclave

Confidentiality

–

Integrity

–

Freshness

Operating system

Only CPU is trustedLets look at

How to secure server applications with enclaves

24 April@Eurosys' 2017

Meni Orenbach, Technion

9

Background: Lifetime of a secured server Untrusted (Host & OS)

24 April@Eurosys' 2017

Trusted (Enclave)

Meni Orenbach, Technion

10

Background: Lifetime of a secured server Untrusted (Host & OS)

Trusted (Enclave)

Untrusted memory Unsecured access

24 April@Eurosys' 2017

Meni Orenbach, Technion

11

Background: Lifetime of a secured server Untrusted (Host & OS)

Trusted (Enclave)

Untrusted memory Unsecured access Dedicated SGX mem Limited to: 128 MB Secured access

24 April@Eurosys' 2017

Meni Orenbach, Technion

12

Background: Lifetime of a secured server Untrusted (Host & OS) Host app

Trusted (Enclave)

Wait for network requests

24 April@Eurosys' 2017

Meni Orenbach, Technion

13

Background: Lifetime of a secured server Untrusted (Host & OS) Host app

Trusted (Enclave)

Wait for network requests

24 April@Eurosys' 2017

Meni Orenbach, Technion

14

Background: Lifetime of a secured server Untrusted (Host & OS) Host app

Wait for network requests

24 April@Eurosys' 2017

Trusted (Enclave) Enter enclave

Meni Orenbach, Technion

Decrypt requests

15

Background: Lifetime of a secured server Untrusted (Host & OS) Host app

Wait for network requests

Trusted (Enclave) Enter enclave

Decrypt requests

Process requests

24 April@Eurosys' 2017

Meni Orenbach, Technion

16

Background: Lifetime of a secured server Untrusted (Host & OS) Host app

Wait for network requests

Trusted (Enclave) Enter enclave

Decrypt requests

Process requests

Encrypt responses 24 April@Eurosys' 2017

Meni Orenbach, Technion

17

Background: Lifetime of a secured server Untrusted (Host & OS) Host app

Wait for network requests

Trusted (Enclave) Enter enclave

Decrypt requests

Process requests

Send responses 24 April@Eurosys' 2017

Exit enclave Meni Orenbach, Technion

Encrypt responses 18

SGX enclaves should be fast ●

ISA extensions

●

Implemented in HW & Firmware

●

Same CPU HW

●

In-cache execution suffers no overheads

24 April@Eurosys' 2017

Meni Orenbach, Technion

19

SGX enclaves should be fast ●

ISA extensions

●

Implemented in HW & Firmware

●

Same CPU HW

●

In-cache execution suffers no overheads

However... 24 April@Eurosys' 2017

Meni Orenbach, Technion

20

Executing a Key-Value Store in enclave is slower

24 April@Eurosys' 2017

Meni Orenbach, Technion

21

Executing a Key-Value Store in enclave is slower Throughput: Slowdown factor 40 35

34X

30 25 20 15

11X

10 5 0 24 April@Eurosys' 2017

64 MB

512 MB Meni Orenbach, Technion

Memory footprint

22

Executing a Key-Value Store in enclave is slower Throughput: Slowdown factor 40 35

Crashes in Windows

30 25

34X

20 15

11X

10 5 0 24 April@Eurosys' 2017

64 MB

512 MB Meni Orenbach, Technion

Memory footprint

23

●

Background

●

Motivation

●

Overhead analysis

●

Eleos design

●

Evaluation

24 April@Eurosys' 2017

Meni Orenbach, Technion

24

Overhead analysis Untrusted (Host & OS) Host app

Wait for network requests

Trusted (Enclave) Enter enclave

Decrypt requests

150 cycles/32B

Process requests

*100 cycles/32B

Send responses 24 April@Eurosys' 2017

Exit enclave Meni Orenbach, Technion

Encrypt responses

*150 cycles/32B 25

Overhead analysis Untrusted (Host & OS) Host app

Wait for network requests

Trusted (Enclave) Enter enclave ~3,300 cycles

Decrypt requests

150 cycles/32B

Process requests

*100 cycles/32B

Send responses 24 April@Eurosys' 2017

Exit enclave Meni Orenbach, Technion

Encrypt responses

*150 cycles/32B 26

Overhead analysis Untrusted (Host & OS) Host app

Wait for network requests

Trusted (Enclave) Enter enclave ~3,300 cycles

Decrypt requests

150 cycles/32B

Process requests

*100 cycles/32B

Send responses 24 April@Eurosys' 2017

Exit enclave ~3,800 Menicycles Orenbach, Technion

Encrypt responses

*150 cycles/32B 27

Overhead analysis Untrusted (Host & OS) Host app

Wait for network requests

Trusted (Enclave) Enter enclave ~3,300 cycles

Exits causes indirect costs: 1.5X – 5X slower execution FlexSC [OSDI'10] syscall analysis Exit enclave Send responses ~3,800 24 April@Eurosys' 2017 Menicycles Orenbach, Technion

Decrypt requests

150 cycles/32B

Process requests

*100 cycles/32B

Encrypt responses

*150 cycles/32B 28

Overhead analysis Untrusted (Host & OS) Host app

Wait for network requests

Trusted (Enclave) Enter enclave ~3,300 cycles

Exits causes indirect costs: 1.5X – 5X slower execution FlexSC [OSDI'10] syscall analysis Exit enclave Send responses ~3,800 24 April@Eurosys' 2017 Menicycles Orenbach, Technion

Decrypt requests

150 cycles/32B

Process requests

*100 cycles/32B

Encrypt responses

*150 cycles/32B 29

Eleos does better! Throughput: Slowdown factor 40

SGX

35

Eleos

30 25 20

5x

15 10 5 0 24 April@Eurosys' 2017

3.5x

64 MB

512 MB Meni Orenbach, Technion

Memory footprint

30

Eleos does better! Throughput: Slowdown factor 40

SGX

35

Eleos

30 25 20

5x

15 10 5 0 24 April@Eurosys' 2017

3.5x

64 MB 512 MB How does Eleos achieve this? Meni Orenbach, Technion

Memory footprint

31

Eleos: Exit-less services Exit-less system calls with RPC infrastructure Exit-less SGX paging

24 April@Eurosys' 2017

Meni Orenbach, Technion

32

Eleos: Exit-less services Exit-less system calls with RPC infrastructure Exit-less SGX paging

24 April@Eurosys' 2017

Meni Orenbach, Technion

33

Background: SGX paging System mem SGX mem

Dedicated memory Enclave code & data Limited to 128 MB

24 April@Eurosys' 2017

Meni Orenbach, Technion

34

Background: SGX paging Enclave Trusted

secret_foo(): ... *p = 1;

System mem SGX mem

Untrusted 24 April@Eurosys' 2017

Meni Orenbach, Technion

35

Background: SGX paging Enclave Trusted

secret_foo(): ... *p = 1;

System mem SGX mem

Hardware Address translation

Untrusted 24 April@Eurosys' 2017

Meni Orenbach, Technion

36

Background: SGX paging Enclave Trusted

secret_foo(): ... *p = 1;

System mem SGX mem

Hardware Address translation Page table Encrypted Untrusted 24 April@Eurosys' 2017

Meni Orenbach, Technion

37

Background: SGX paging Enclave Trusted

secret_foo(): ... *p = 1;

System mem SGX mem

Hardware Address translation Page table Encrypted Untrusted 24 April@Eurosys' 2017

Meni Orenbach, Technion

Swapped-out

38

Background: SGX paging Enclave Trusted

secret_foo(): ... *p = 1;

System mem SGX mem

Hardware Address translation Page table SGX-driver Untrusted 24 April@Eurosys' 2017

Fault handler Meni Orenbach, Technion

Encrypted Swapped-out

39

Background: SGX paging Enclave Trusted

secret_foo(): ... *p = 1;

Hardware Address translation

System mem SGX mem Decrypted Integrity validation

Page table SGX-driver Untrusted 24 April@Eurosys' 2017

Fault handler Meni Orenbach, Technion

Encrypted Swapped-out

40

Background: SGX paging Enclave Trusted

secret_foo(): ... *p = 1; *(++p) = 2;

Hardware Address translation

System mem SGX mem Decrypted

Page table

SGX driver Untrusted 24 April@Eurosys' 2017

Fault handler Meni Orenbach, Technion

Encrypted

41

Background: SGX paging Enclave Trusted

secret_foo(): ... *p = 1; *(++p) = 2;

Hardware Address translation Fast path

SGX driver Untrusted 24 April@Eurosys' 2017

System mem SGX mem Decrypted

Page table Fault handler Meni Orenbach, Technion

Encrypted

42

Background: SGX paging Enclave Trusted

secret_foo(): ... *p = 1; *(++p) = 2;

Hardware Address translation Fast path

System mem SGX mem Decrypted

Page table

Fault Encrypted SGX driver Since SGX memory is small handler Untrusted paging is not as rare as in native applications 24 April@Eurosys' 2017

What are the overheads? Meni Orenbach, Technion

43

Background: SGX paging Enclave Trusted

secret_foo(): ... *p = 1; *(++p) = 2;

Hardware Address translation

System mem SGX mem Decrypted

Page table

SGX driver Untrusted 24 April@Eurosys' 2017

Fault handler Meni Orenbach, Technion

Encrypted

44

SGX paging overheads Enclave Trusted

System mem

secret_foo(): ... *p = 1; *(++p) = 2;

SGX mem

Hardware Address translation Enclave exit SGX driver Untrusted 24 April@Eurosys' 2017

Page table

Decrypted

Enclave resume

Fault handler Meni Orenbach, Technion

Encrypted

45

SGX paging overheads Enclave Trusted

Indirect costs

Enclave exit SGX driver Untrusted 24 April@Eurosys' 2017

System mem

secret_foo(): ... *p = 1; *(++p) = 2;

SGX mem

Hardware Address translation Page table

Decrypted

Enclave resume

Fault handler Meni Orenbach, Technion

Encrypted

46

SGX paging overheads Enclave Trusted

Indirect costs

Enclave exit

System mem

secret_foo(): ... *p = 1; *(++p) = 2;

SGX mem

Hardware Address translation Page table

Decrypted

Enclave resume

Fault SGX driver handler Untrusted Overaheads: Untrusted software 24 April@Eurosys' 2017 Meni Orenbach, Technion manages enclave memory

Encrypted

47

Wanted: In-enclave virtual memory management

24 April@Eurosys' 2017

No more exits! Meni Orenbach, Technion

48

Ideal in-enclave VM management Enclave Trusted

secret_foo(): ... *p = 1; *(++p) = 2;

System mem SGX mem

Hardware Address translation Page table

SGX driver Untrusted 24 April@Eurosys' 2017

Fault handler Meni Orenbach, Technion

49

Ideal in-enclave VM management Enclave Trusted

secret_foo(): ... *p = 1; *(++p) = 2;

System mem SGX mem

Hardware Address translation Page table Fault handler 24 April@Eurosys' 2017

Meni Orenbach, Technion

50

Ideal in-enclave VM management Enclave Trusted

secret_foo(): ... *p = 1; *(++p) = 2;

System mem SGX mem

Hardware Address translation No available hardware

Page table Fault handler

24 April@Eurosys' 2017

Meni Orenbach, Technion

51

Ideal in-enclave VM management Enclave Trusted

secret_foo(): ... *p = 1; *(++p) = 2;

System mem SGX mem

Hardware Software Address Address translation translation Page table Fault handler 24 April@Eurosys' 2017

Meni Orenbach, Technion

52

SUVM: Secured user-space VM Enclave Trusted

secret_foo(): s_ptr p = suvm_malloc(1024); ... *p = 1;

System mem SGX mem

Software Address translation Page table Fault handler 24 April@Eurosys' 2017

Meni Orenbach, Technion

53

SUVM: Secured user-space VM Enclave Trusted

secret_foo(): s_ptr p = suvm_malloc(1024); ... *p = 1;

System mem SGX mem

Software Address translation Template class: SecuredPointer.

Page table Fault handler

24 April@Eurosys' 2017

Meni Orenbach, Technion

54

SUVM: Secured user-space VM Enclave Trusted

secret_foo(): s_ptr p = suvm_malloc(1024); ... *p = 1;

System mem SGX mem

Software Address translation Template class: SecuredPointer.

Page table Fault handler

24 April@Eurosys' 2017

Meni Orenbach, Technion

Encrypted

55

SUVM: Secured user-space VM Enclave Trusted

secret_foo(): s_ptr p = suvm_malloc(1024); ... *p = 1;

System mem SGX mem

Software Address translation Template class: SecuredPointer.

Page table Fault handler

24 April@Eurosys' 2017

Meni Orenbach, Technion

Encrypted Swapped-out

56

SUVM: Secured user-space VM Enclave Trusted

secret_foo(): s_ptr p = suvm_malloc(1024); ... *p = 1;

System mem SGX mem

Software Address translation Template class: SecuredPointer.

Page table Fault handler

24 April@Eurosys' 2017

Meni Orenbach, Technion

Encrypted Swapped-out

57

SUVM: Secured user-space VM Enclave Trusted

secret_foo(): s_ptr p = suvm_malloc(1024); ... *p = 1;

Software Address translation Template class: SecuredPointer.

SGX mem Decrypted Integrity validation

Page table Fault handler

24 April@Eurosys' 2017

System mem

Meni Orenbach, Technion

Encrypted Swapped-out

58

SUVM: Secured user-space VM Enclave Trusted

secret_foo(): s_ptr p = suvm_malloc(1024); ... *p = 1;

Software Address translation Template class: SecuredPointer.

SGX mem Decrypted Integrity validation

Page table Fault handler

Control path 24 April@Eurosys' 2017in-enclave

System mem

Meni Orenbach, Technion

Encrypted Swapped-out

59

SUVM: Secured user-space VM Enclave Trusted

secret_foo(): s_ptr p = suvm_malloc(1024); ... *p = 1; *(++p) = 2;

Software Address translation

System mem SGX mem Decrypted

Page table Fault handler 24 April@Eurosys' 2017

Meni Orenbach, Technion

Encrypted

60

SUVM: Secured user-space VM Enclave Trusted

secret_foo(): s_ptr p = suvm_malloc(1024); ... *p = 1; *(++p) = 2;

Software Address translation

System mem SGX mem Decrypted

Page table Fault handler 24 April@Eurosys' 2017

Meni Orenbach, Technion

Encrypted

61

SUVM: Secured user-space VM Enclave Trusted

secret_foo(): s_ptr p = suvm_malloc(1024); ... *p = 1; *(++p) = 2;

Software Address translation Fast path No page table Lookup!

SGX mem Decrypted

Page table Fault handler

24 April@Eurosys' 2017

System mem

Meni Orenbach, Technion

Encrypted

62

Wait...Software based VM management?

Based on software address translation on GPUs, ActivePointers [ISCA'2016] 24 April@Eurosys' 2017 Meni Orenbach, Technion

63

SUVM key contributions ●

Multi-threaded Compared to SGX: Fast path: up to 20% overheads Slow path: Eliminates costs of exits READ WRITE

1 Thread 5.5x 3.5x

4 Threads 7x 5.9x

Throughput speedup 24 April@Eurosys' 2017

Meni Orenbach, Technion

64

Software address translation offers new optimizations ●

Customized page size

●

Customized eviction policy

●

Multi-enclave memory coordination

●

Write-back only dirty pages

●

Sub-page direct access to backing store

24 April@Eurosys' 2017

Meni Orenbach, Technion

65

Software address translation offers new optimizations ●

Customized page size

●

Customized eviction policy

●

Multi-enclave memory coordination

●

Write-back only dirty pages

●

Sub-page direct access to backing store

24 April@Eurosys' 2017

Meni Orenbach, Technion

Virtual Machine ballooning

66

Software address translation offers new optimizations ●

Customized page size

●

Customized eviction policy

●

Multi-enclave memory coordination

●

Write-back only dirty pages

●

Sub-page direct access to backing store

24 April@Eurosys' 2017

Meni Orenbach, Technion

Virtual Machine ballooning

67

●

Background

●

Motivation

●

Overhead analysis

●

Eleos design

●

Evaluation

24 April@Eurosys' 2017

Meni Orenbach, Technion

68

Biometric Identity checking server Workload generator

Face verification server

10Gb NIC

+ ID 24 April@Eurosys' 2017

? = 450MB DB Meni Orenbach, (5X Technion SGX mem)

69

Biometric Identity validating server Speedup compared to vanilla SGX Eleos

Native

3.5 3 2.5 2 1.5 1 0.5 0

1

24 April@Eurosys' 2017

2 Meni Orenbach, Technion

Server threads

4 70

Biometric Identity validating server Speedup compared to vanilla SGX Eleos

Native

3.5 3 2.5 2 1.5 1 0.5 0

1

24 April@Eurosys' 2017

2 Meni Orenbach, Technion

Server threads

4 71

Biometric Identity validating server Speedup compared to vanilla SGX Eleos 3.5

Native

Eleos scales better than vanilla-SGX: Saves inter-processor-interrupts

3 2.5 2 1.5 1 0.5 0

1

24 April@Eurosys' 2017

2 Meni Orenbach, Technion

Server threads

4 72

Biometric Identity validating server Speedup compared to vanilla SGX Eleos 3.5

Native

Eleos scales better than vanilla-SGX: Saves inter-processor-interrupts

3 2.5 2 1.5

Saturate 10Gb network

1 0.5 0

1

24 April@Eurosys' 2017

2 Meni Orenbach, Technion

Server threads

4 73

Memcached Workload Generator (memaslap)

Memcached Graphene LibOS [Eurosys'2014]

10Gb NIC ~75 LOC modification for SUVM 24 April@Eurosys' 2017

GET(

)

500MB DB (5.5X SGX mem) Meni Orenbach, Technion

74

Memcached Speedup compared to vanilla SGX (500 MB) Eleos (500MB DB)

vanilla SGX (20MB DB)

3 2.5 2 1.5 1

No SGX Faults

No SGX Faults

0.5 0 24 April@Eurosys' 2017

1 Thread

4 Threads Meni Orenbach, Technion

Server threads

75

Memcached Speedup compared to vanilla SGX (500 MB) Eleos (500MB DB)

vanilla SGX (20MB DB)

3 2.5 2 1.5 1

No SGX Faults

No SGX Faults

0.5 0

1 Thread 4 Threads Eleos+Graphene is 3x slower than native 24 April@Eurosys'Disclaimer: 2017 Meni Orenbach, Technion Server threads

76

Take aways ●

Eleos eliminates enclave exits costs

●

Eleos available for Windows and Linux –

●

Makes memory demanding applications available on Windows today

Eleos takes a modularize approach –

Memory demanding app? Link to SUVM

–

I/O intensive app? Link to RPC

–

Maintaining small TCB

24 April@Eurosys' 2017

Meni Orenbach, Technion

77

Traditional SGX: Host-centric OS services Enclave

Operating System 24 April@Eurosys' 2017

Meni Orenbach, Technion

78

Traditional SGX: Host-centric OS services Enclave

Get data

Operating System 24 April@Eurosys' 2017

Meni Orenbach, Technion

79

Traditional SGX: Host-centric OS services Enclave

Get data

Data Unavailable

Operating System 24 April@Eurosys' 2017

Meni Orenbach, Technion

80

Traditional SGX: Host-centric OS services Enclave

Get data

Data Unavailable

Fetch data

Operating System 24 April@Eurosys' 2017

Meni Orenbach, Technion

81

Traditional SGX: Host-centric OS services Enclave

Get data

Data Unavailable

Fetch data

Operating System 24 April@Eurosys' 2017

Meni Orenbach, Technion

82

Eleos Insight: Enclave-centric OS services Enclave

Get data

Fetch data

In-enclave Services

24 April@Eurosys' 2017

Meni Orenbach, Technion

83

Take aways (2) ●

●

Eleos adapts 'accelerator-centric management' –

System calls: GPUfs [ASPLOS'13], GPUnet [OSDI'14]

–

Virtual memory: ActivePointers [ISCA'16]

We can do more! –

Asynchronous DMA host copies

–

Non-blocking enclave launches More information at: “SGX Enclaves as Accelerators" [Systex'16]

24 April@Eurosys' 2017

Meni Orenbach, Technion

84

Thank you

Code is available at: https://github.com/acsl-technion/eleos

[email protected] 24 April@Eurosys' 2017

Meni Orenbach, Technion

85

Backup slides

24 April@Eurosys' 2017

Meni Orenbach, Technion

86