Balancing CoreXL and SecureXL Michael Endrizzi Director of Services and Training [email protected]

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

1

Who Is Instructor? • • • • • • • • • •

•

Michael Endrizzi – Midpoint Tech Director of Training and Services Age: 56 CCSA,CCSE, CCSMA, CCISP, ITIL Information security since 1982 Developer on Secure Computing Sidewinder firewall in 1993 Worked with NSA Owned information security businesses Independent security consultant for 16 years Working with Check Point since 1996. 10 year hiatus into auditing now back Oh yeah, I like rock climbing

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

2

Balancing Check Point Systems

• • • • •

Overview Linux Review SecureXL CoreXL Balancing CoreXL Tips

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

3

Danger Will Robinson • • • • • • • • • •

This class will teach you how to tune BUSY/Critical firewalls Commands you learn here will take affect IMMEDIATELY These commands are to be used delicately. By issuing these commands without fully testing them, you may negatively impact performance on CRITICAL firewalls. R77.10 introduced different behavior on some commands. Read the docs. Most commands do not allow you do easily undo themselves. You have to use your notes and photo clips to record the current configuration in case you wish to revert! If you change multiple items at once, you may not be able to undo the changes in case of failure. Backups will not save this information. You have to do a snapshot/restore in order to recover. DISCLAIMER: By reading this line you are totally responsible for all changes to your environment.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

4

What is SecureXL?? SecureXL (aka Secure Network Distributor) is a way of speeding up rule processing by analyzing traffic patterns and handing off known and previously approved traffic to high throughput traffic handlers.

VS1 Secure Network Distributor SND

• • •

Processing incoming traffic from the network interfaces Securely accelerating authorized packets (if Performance Pack is running) - SecureXL Distributing non-accelerated packets among kernel instances - CoreXL

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

5

What is CoreXL? CoreXL allows you to increase the throughput capacity of your firewall platform thru the utilization of multi-processors concurrently processing firewall requests.

• Cost efficient • Easier to manage • Easier to debug

• More expensive to purchase and maintain

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

6

What is NGF/VSX? NGF (Next Generation Firewall) is a standalone gateway running 1 instance of the firewall module AND has the ability to run several threat prevention blades IPS/AV/AB/Threat-Emulation

FWK

VSX is a physical chassis that runs multiple instances of firewall gateways. Think of VMware that runs firewalls as guests.

VS0

VS1

VS2

Linux 2.6.18-XXcp All based on Linux © 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

7

What is SPLAT/GAIA? • Secure Platform (SPLAT) is the commands and kernel modules added to Linux which transforms the Linux platform into a Check Point firewall • GAIA was a self-contained command environment created to simplify administration. Looks like Cisco command environment.

GAIA

GAIA command shell – Self contained shell looks like Cisco CLI SPLAT command set added to Linux command set

SPLAT

Linux command set Linux Kernel

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

8

Goal NGF/VSX + CoreXL + SecureXL = Tuned System Describes many topics in these articles. After this course you will be able to understand these SK’s

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

9

Course Take Away Slide This is the take-away slide, the value behind the whole course. If I bore you to death with 200+ slides, this slide #190(approx) puts to practice what the whole course is trying to teach: Allocate CPUs in the following priority order:

1. 2. 3. 4. 5.

Share cache for common data Allocate CPUs to busy Internal interfaces Allocate CPUs to slower less busy External interfaces Allocate CPUs to FW instances Remaining threads are usually idle so distribute evenly and let kernel find idle processor

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

10

What Do I Need To Know? Basic Linux CLI experience, What is operating system, What are processes, What is a processor, What is a cache, . Basic programming: code, data, variables. Unix ‘top’

Unix ps

/proc file system

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

11

Secure/CoreXL Tuning Areas 5 areas of tuning 3) CP Process Affinity

2) Fw kernel Instance Affinity

VS1

VS2

FWD logging

syslogd

1) Interface Affinity SND

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

4) Linux Process

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

SecureXL 0) Rule Processing Secure Speedup Network Distributor

12

Firewall Management SmartCenter Components Smart Dashboard POLICY User Space

Edit Policy

FWD (logging)

CPD (mgt server communication)

Push Policy

FWM (Management Server)

Kernel Space

Linux TCP/IP

NIC

NIC 1/24/2015

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

13

NGF Components Smart Center MGT Station POLICY User Space

FWSSD (spawns security servers like SMTP filtering)

FWD (logging) CPD (mgt server communication)

Kernel Space

1/24/2015

VPND (VPN)

CPWD (watchdog for dead processes)

SPLAT/GAIA Kernel (fwk) (Security Enforcement)

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

14

VSX/VS Components

Smart Center MGT Station

User mode

cpd

cpd

cpd

fwk

fwk

fwk

fwd

fwd

fwd

vpnd

vpnd

vpnd

VS0

VS1

VS2

Kernel mode

1/24/2015

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

15

R75.40 VSX – FWK in User Mode

cpd

• Note: fwk was moved to user mode • With large number of VS’s, kernel was getting too big

fwk

fwd

vpnd

User mode

VS0

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

Our Lab Eth0: 10.2.1.253/24

Eth1: 172.17.1.2/24

Eth0: 10.2.1.101/24

Eth2: 10.2.2.253/24

Eth0:1: 10.2.2.101/24

Eth3: 172.17.2.2/24

Eth0 : 172.17.1.111/24

Eth0:2 : 172.17.2.111/24

Eth0:2: 10.2.0.101/24

Eth0:1 : 172.17.0.111/24 Eth0: 10.2.1.153/24

Eth0:1 : 10.2.0.153/24 VB: Host/Host#1

Eth1:1 : 172.17.0.1/24 VB: internal/internet

20

1/24/2015 © 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

20

Balancing Check Point Systems

• • • • •

Overview Linux Review SecureXL CoreXL Balancing CoreXL Tips

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

21

Linux Kernel Basics

• Linux Overview • Threads • Network Processing

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

22

Linux History

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

23

Evolution of the Kernel Multics-Unix

Computer Day 1 The Code Blob

User

User

User

User User

User

User

Kernel

•

•

Programs are one compiled binary all superuser PROBLEM: hacking, bad code corrupted whole system

1/24/2015

• • •

Users have their own process resources (memory, files..) Monolithic kernel has its own resources (memory, code base). Kernel could multiprocess user processes but not self

User

User

Kernel

Monolithic Kernel

•

Linux

NT - Mach

•

•

Kernel

Some tried to simplify the kernel…. Kernel broke self into smaller processes, some run in user space Processes used Inter Process Communication to work together

• PROBLEM: SLOW In-efficient PROBLEM: too big expensive because of context switches © 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected] for hardware at time Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

Mod ule

• • •

Mod ule

Mod ule

BACK to monolithic kernel (one memory space). Minimal context switch and no slow IPC Dynamic added/subtracted modules. User processes don’t call kernel process, user processes go into kernel mode with shared data structures! Minimize context switch and data passing (more later on this)

24

Linux Differentiators • Original kernels were monolithic: • Single binary process • Single processing thread • Cooperative Multi-tasking – Could not pre-empt kernel processing • Single Address space • Linux • Linux Tervald – Still heavily involved vs design by committee vs free-for-all • Pre-emptive kernel – Most kernel tasks can be pre-empted for higher priority tasks • Modular – Kernel functions can be dynamically created/removed Check Point implements firewall subsystem in these modules • Multi-processor support • Threads = Processes (Unique to Linux – will explain later) • Users can see internal kernel data in sysfs file system. Looking glass into kernel internals

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

25

Linux 2.4 to 2.6 Introduced to reduce the time a kernel task held on to the processor locking out other tasks. Overall increase in efficiency and multi-processing support.

1. Scheduler – Improved fair scheduling with 100’s of processors 2. Threads – Process = thread. No special handling for threads 3. Interrupts – Can be pre-empted and no locking out all CPUs while processing interrupts 4. Pre-emptive Kernel – The whole kernel is pre-emptive. Can be interrupted at any point

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

26

Linux Kernel Structure An application process consists of: • User Identity & Permissions • Code and Execution Pointer • Data Scratch Pad (Stack and Hash) • Check Point VSX Firewall Kernel and Helpers

When applications call the kernel, the kernel does not “take over” as a separate entity. Applications transform into the “The Hulk”. The application processing thread takes on kernel permissions, resources and code base to complete a task. If the thread does not go to sleep, processing returns to application mode (Bruce Banner) almost as if it made an internal function call.

Most kernel work done here. • Resource allocation • Scheduling • Security • Memory Mgt • File System Mgt • Communication • Check Point NGF FirewallKernel

Talks to hardware, handles hardware interrupts. In Check Point, drivers are right from manufacturer. Some appliances have modifications

http://www.amazon.com/Linux-Kernel-Development-3rd-Edition/dp/0672329468 © 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

27

Linux Kernel Basics

• Linux Overview • Threads • Network Processing

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

28

What is a program? Simple terms: A program is made up of: 1. CPU Instructions – Does the work 2. Data – Scratch pad area 3. Security attributes - restrictions

• CPU Instructions • Data • Security attributes

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

29

What is a process? A program is a compiled binary sitting on a disk in a file. A process is when the program is executed on a processor, assigned memory, and is managed by the OS kernel.

Program becomes a Process • CPU Instructions • Data • Security attributes

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

30

What Does 32-bit User Program Data look like? All 32-bit programs have 4 gig of memory available to them. This is how it is allocated.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

31

Why Care?: CPU vs. Memory problem Are you chasing a CPU or memory problem? Need to know how lack of memory will slow a system and make it feel like it is a CPU problem. ‘top’->’f’->’u’

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

VM Why do you care? If you ever want to figure out why a process is swapping, you need to be able to know what parts of a process are taking too much space.

Virtual Size

Physical Memory

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

33

Stack Frame A program is broken into functions. When a function is called, the program has to save information about where it was and pass information to the new function. This placeholder is called a stack frame. Stack frames are To High Memory Stack like a track of cookie crumbs to help you go back to where External Environment where. Parameters GROW

Program Counter

Saved Frame Pointer Local Variables

To 0 memory

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

35

Heap space and Memory Leaks Programs sometimes need dynamically allocated and sized chunks of memory for maybe network packets. The program uses the malloc call to get that memory. It came off the heap space. If the programmer forgets to free the memory, then you have a memory leak.

Sometimes heap space becomes so fragmented with mixed free and allocated memory, that programs slow down because its hard for them to claim and release memory efficiently. Thus ‘reboot!!!’.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

36

Buffer Overflow

Stack

Code

Length: 1000 bytes 301 LOGIN: + username +

Length: 256 bytes

Packets Fragmented Mail Program Running as Super-User

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

IMAPD Attack Authentication Attempt Backfires To 0 memory

Stack Local Variables

Attack Login String

UserName[256]

Mike Password

Saved Frame Pointer

New PC

Program Counter Parameters

EndOfBuffer NULL

Environment

To High Memory © 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

38

Virtual Memory 4GB Process A

4GB Process B

NOTE: Most programs don’t use all 4 gig at once. Only use small portion.

4GB Process C

4GB Process D

How much total memory is needed to store all these processes - MAX???

NOTE: This is for 32-bit systems. 64-bit can see 264 = 16 exibytes of memory

Virtual memory is 3 things: 1) Allows processes to think they own all memory 2) Allows processes to ignore physical memory limitations 3) Paging system: That uses disk to swap out sleeping data to temp storage area to free up physical memory for active processes.

Not needed right now so swap out

+

10GB Disk Swap space

Assuming all processes needed 3GB User space We would need at a minimum 12GB (3GB+3GB+3GB+3GB)User + 1GB kernel = 13GB Physical and Swap space to hold all these programs.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

Demo: Using VIRT Memory Here in my ‘memgrow’ program I allocated 1GB of HEAP memory, but didn’t read/write to it. Notice how the kernel allocated the VM space to my process but did not actually map it into physical memory or SWAP it out because we are running out of space (I only have 1 gig of physical memory)

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

DEMO: Writing to VIRT Memory Here the same ‘memgrow’ program BUT I wrote to every byte so the kernel had to map the page to my process and bring it into physical memory. And when physical memory fills up…then swap it out… WHAT IS VIRTUAL MEMORY???? 1) Process thinks it owns all 32-bit 3GB by itself 2) Process doesn’t understand physical memory constraints of 1GB (only 32/64 max addressable memory 3) Paging swapped out data that the kernel could not keep in physical memory

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

User Program Security Attributes Computers Day 1 allowed a program to see and do everything. But programs started to overrun each other. So Day 2 a kernel was developed that was the traffic cop between programs so they wouldn’t bump into each other OR corrupt the kernel itself. NOTE: On many embedded systems there is no kernel/user/security separation. All processes can see and do everything. For example: SCADA controllers that run machinery. Programs that monitor the brakes in your car.

User Level • CPU Instructions • Data • Security attributes

• CPU Instructions • Data • Security attributes

Kernel Level

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

43

What is the Kernel? The ‘kernel’ is responsible for resource management.

User Level

• • • • • • • • •

Access to physical devices Security separation Arbitrator of shared resources – who gets what first Virtual memory management Clock ticks and time slices Process management Scheduling processes File system access Network routing

Kernel Level

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

User Program Entering The Kernel Linux TODAY: When a process makes a system call, the process does not hand over processing to another entity. The processing thread gains access to kernel resources (after appropriate security checks) so the kernel is actually executing “on behalf of a specific process”. The process running in kernel mode now has access to both user and kernel memory. Its like a program making a function call and the function has enhanced magic powers, and then returns to the main user process. Just like “The Hulk”

User Space

Kernel Space

SYSTEM CALL (security checks)

+

• Kernel Data Structures • Superuser privileges My friend Arah becoming the HULK

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

45

3 Processing Contexts In Linux a processor can be in 1 of 3 processing contexts at any time….period 1. In user-space, executing user code in a user process 2. In kernel-space, in process context, executing on behalf of a specific user process 3. In kernel-space, in interrupt context, not associated with a process, handling an interrupt

Why do you care?

http://www.amazon.com/Linux-Kernel-Development-3rd-Edition/dp/0672329468

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

46

Pre-Emption and Context Switch In order to avoid one process from dominating a processor pre-emptive multi-tasking was introduced. This is where one process B can interrupt another process A at any time (preempt) resulting in a context switch to the process B. Prior to mid-1990’s context switches (stopping one process and starting another) were CPU/Memory intensive. Kernels had to copy the process state into internal kernel Process Control Blocks (PCBs) to save the state and then copy out a PCB to the new process. Process B

Process A • Virtual Memory Space • Stack and Hash Tables (Scratch Pad) • CPU Instructions • Processor registers • Security attributes

Context Switch

• Virtual Memory Space • Stack and Hash Tables (Scratch Pad) • CPU Instructions • Processor registers • Security attributes

Kernel Process Control Block

Process Control Block

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

47

Old Interprocess Communication Copy Data In the old days communication between processes and to/from the kernel data was copied. Very inefficient. Process B

Process A Copy this to Process B

Kernel

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

48

New Interprocess Communication Share Data So a shared memory mechanism was developed using virtual memory tricks that allowed memory spaces to be shared between processes and the kernel. Process A

Process B You can see some of my data

Kernel © 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

49

What is a Multi-Processor Multi-processors come in several different varieties • Multi-Processor: Multiple physical CPUs in a single chassis • Multi-Core : Multiple processor cores on a single physical CPU • HyperThreading: Simulated multiple processors with threads on a single physical core • Combination of the above • No matter which of the above configurations, the OS sees logical processors. The implementation is opaque. • Note where the caches are! Remember this when you assign interfaces so you keep the cache hot OS sees 4 ‘logical’ processors

Logical Processors Threads Cores Physical CPU

http://blogs.msdn.com/b/gauravseth/archive/2006/03/20/555519.aspx © 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

50

What are CPU Threads Enabling Symmetric Multi-Threading (SMT) or HyperThreading (HT) doubles the number of logical processors. • • • • •

Works just like Linux thread processing where a process (web server) has 2 threads (2 clients requesting pages) and the kernel can preemptively multi-task the two threads so it seems like they are parallel processing. Without HT. Each Linux thread gets a time-slice by the Linux kernel but only 1 thread runs at a time. With HT: At the hardware level there is a mini-Linux like kernel that can multiplex/task 2 threads concurrently. So the two threads could conceptually start and finish within 1 kernel time slice instead of 2 separate time slices. Performance improvement 30%??? on CPU intensive items. I/O intensive theoretically could slower. Shares a cache

Without Hyperthreading, Managed by Kernel Thread 1

Thread 2

Thread 1

Thread 2

WithHyperthreading, Managed by HW

TIME http://blogs.msdn.com/b/gauravseth/archive/2006/03/20/555519.aspx © 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

51

Example:HP-DL380p Multi-Core No HyperThreading HP DL380p – 2 Physical Processors 8 Logical Processors

CPU0

CACHE

CPU1 CACHE

HP DL380p – 8 Logical Processors No Hyper Threading

OS sees 8 ‘logical’ processors

Logical Processors Threads

Cache

Cores Physical CPU © 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

52

Example: CheckPoint 12600 12 core

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

53

What is a multi-processing? Day 3 – As loads increased computer designers decided to add multiple processors to the system. Processes lent themselves nicely to the paradigm, each process could float to a free processor and execute.

Process A • Virtual Memory Space • Stack and Hash Tables (Scratch Pad) • CPU Instructions • Processor registers • Security attributes

Process B

• Virtual Memory Space • Stack and Hash Tables (Scratch Pad) • CPU Instructions • Processor registers • Security attributes

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

54

Shared Data Contention LOCKUP!! The biggest problem with concurrency is access to shared data. If not programmed correctly two processes can fight over updating a shared piece of data and they fight to the death. What you see is a frozen monitor!!!

• Virtual Memory Space • Stack and Hash Tables (Scratch Pad) • CPU Instructions • Processor registers • Security attributes

• Virtual Memory Space • Stack and Hash Tables (Scratch Pad) • CPU Instructions • Processor registers • Security attributes

I’m going to write to variable X first!!!

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

NO! I’m going to write to variable X first!!!

Threads To The Rescue If a process wanted to enable concurrent processing (web server serving up pages, word processor with multiple open documents), running multiple processes was inefficient. Threads were created to support concurrent processing using SHARED data and NOT copying data between processes or resource heavy context switching. SHARED DATA • • • • • • •

Virtual Memory Space CPU Instructions Processor state Kernel threads Security attributes File system Signals

Thread C Thread B Thread A

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

56

Light Weight/Quick Context Switches Thread context switches are very fast because the kernel only has to save a several registers (TCB) (approx 96 bytes vs. approx 1K+ with heavy process switches). This is because all the non-saved data is shared between the threads so its live data and no need to save it and restore it. In addition, threads can see into each others address space because remember everything in the process is shared.

• • •

Thread registers Program counter Stack pointer

Thread B

Thread A

Mini- Context Switch

Kernel Thread Control Block(TCB)

Thread Control Block(TCB)

https://courses.cs.washington.edu/courses/cse451/11sp/section/kim_section4.pdf © 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

57

Threads and Multi-Processing Threads naturally lend themselves to multi-processing…you can concurrently run several threads on different processors. Of course there are shared data contentions that must be dealt internally by the threads

SHARED DATA • • • • • • •

Virtual Memory Space CPU Instructions Processor state Kernel threads Security attributes File system Signals

Thread C Thread B

Thread A

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

58

Threads and Scheduling Linux 2.6.18 changed its default scheduling to favor low latency processing for more realtime applications. It is called Completely Fair Scheduling (CSF). Its simple, when the kernel schedules a process it chooses a process that has used the least amount of its time slice. That way CPU-lite processes are favored. These are usually ones that are blocking waiting for input like network interrupts, keyboard, graphics, etc. When a process is ready to run, the kernel looks for: 1. What CPU is free on the affinity list for the process. 2. Did it run on the CPU before so the cache is fresh with my data? 3. Am I still in default mode where no autobalance occurred and so choose CPU 0 (mostly interrupts)

Thread B Which CPU?

Thread A

Thread C Which CPU?

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

59

Processor Affinity If you have a thread that is usually very busy, you can wire it to a single CPU (or subset of CPUs). That way all the data is cached locally on the single CPU and it can usually run less interrupted on that CPU. This is called Processor Affinity. A process/thread has an ‘affinity’ for a CPU. Can be done in Unix and Windows Oss Check Point leverages this in their CoreXL technology.

• • • • • • •

Virtual Memory Space CPU Instructions Processor state Kernel threads Security attributes File system Signals

Thread C

Thread B Thread A

CPU1

CPU2

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

CPU3 60

Types of Threads 3 types of threads: 1. User threads: Totally internal to process, kernel cannot see them limited multi-processing capabilities for example they couldn’t block to sleep or I/O. 2. User threads mapped onto kernel threads. Full multi-processing capabilities for user process 3. Internal kernel threads: Used only by kernel for internal kernel processing not visible to user processes http://linuxgazette.net/23/flower/threads.html http://www.thegeekstuff.com/2012/03/linux-threads-intro/

Internal kernel threads

User Level • • • • •

Kernel Level

• • • •

Scheduler Blocking I/O handling Thread table Monitoring

Virtual Memory Space CPU Instructions Processor state Kernel threads Security attributes

User Threads mapped to Kernel Threads

Internal Kernel Threads

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

61

Thread Demo Internal kernel threads have a [] around them. User threads, usually do NOT have [] on them. NOTE: sometimes user processes do, but its because the kernel can’t find its command parameters and notates it with [] which is confusing. Other commands can filter this out. “ps –ef”

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

62

Thread Demo: Windows Threads Even windows has threads

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

63

Demo: Kernel Thread Kernel threads have no virtual memory size because they all share the VM of the kernel and no allocated extra memory like user threads can be allocated. Remember this: will be similar when we look at CP NGX

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

64

Thread Demo: Multi-Processing Threads You can watch threads switching between processors to see which processor is busy

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

65

Linux Unique Thread Architecture Other OS’s have a concept of Light Weight Processes (LWP) (threads) where the parent process spawns owns, and manages the threads. Think mamma black bear protecting her cubs

Linux 2.6 everything is a thread, and they are self managed. They keep track of their own state and don’t rely upon a parent to manage them. Linux CAN simulate LWP to the outside world and this is what you will see in the demo/labs. Think busy bees – once born they are off on their own

Main Process Parent Code

Thread A Thread A

Thread B

Thread C

Thread C

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Thread B

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

66

Linux Process to Thread Relationship So a Linux process is like a beehive that keeps all the common data/honey. The threads are like the bees that go off and do their work and bring back data/honey where it is all shared between them.

• • • • • • •

Virtual Memory Space CPU Instructions Processor state Kernel threads Security attributes File system Signals

Thread A

Thread B

Thread C

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

67

Clone() – Threads and Sharing This system call is the heart of threads and sharing. Depending on what the programmer tells the thread to share, the thread could act like a full blown HEAVY process (old days), or like a little lite worker bee that doesn’t carry any baggage with them (Linux today). Other Unix’s are starting to implement this in some fashion but Linux was first. int __clone(int (*fn) (void *arg), void *child_stack, int flags, void *args)

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

68

Thread Summary

• Threads can have 3 contexts: • User • Kernel • Interrupt (very special type of thread, very limited) • Threads are the workers and the data is ‘usually’ stored in a shared common process space • Threads enter kernel space and take on kernel context: virtual memory space, and security context.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

69

Linux Kernel Basics

• Linux Overview • Threads • Network Processing

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

70

Interrupts The kernel interacts with external devices such as network cards in two ways: 1. Interrupts – Only interrupt the kernel when there is something to do can be more efficient 2. Polling – Continuously see if there is any activity on the device. Very inefficient and slows down the kernel Modern Linux device drivers use a combination of both (New API - NAPI). They wait for an interrupt to occur and begin processing. They will then disable the interrupt and go into a polling mode until there is nothing more to do (no more data). They will then re-enable the interrupt to be alerted for more activity. Interrupt Handler

Kernel Poll

Hey WAKE UP

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

71

Interrupt Priority Interrupts have a priority system. Interrupts such as the system clock have to be processed right away and are a higher priority. They will stop all processor activity (even lower level interrupt handlers) to be handled. Lower priority interrupts will disable its own interrupt line (eg. Network card) so that it doesn’t get interrupted from the same source and run uninterrupted (unless a higher priority interrupt comes along).

Kernel LOW PRIORITY

HIGH PRIORITY System Clock

http://www.amazon.com/Essential-Device-Drivers-Sreekrishnan-Venkateswaran/dp/0132396556 Page 493 NAPI © 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

72

Advanced Programmable Interrupt Controller (APIC) APICs are the hardware that manage interrupts. A motherboard had one I/O APIC that interfaces with the hardware and talks to LOCAL APIC controllers embedded within the CPU.

In SMP environments where IRQs can be handled by multiple CPUs APICs can be dynamically programmed by the kernel to direct IRQs to a specific CPU for balancing out the handling of IRQs from external devices.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

73

Top/Bottom Half Interrupts Interrupt processing is split into 2 parts: 1. Hardware Interrupt (Top): Device specific, stops processor, has to be quick 2. Software Interrupt (Bottom): Thread-like, heavy lifting when CPU has time Software Iinterrupt (Bottom ½) Heavy lifting

Stop processor (Top ½) Quick and get out

Schedule SW Interrupt: When processor has time do generic processing of packets. HW Interrupt: Run device specific handler

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

74

‘top’ view of interrupts

‘top’ – ‘1’ for CPUs

Hardware interrupt handles device drivers

Software interrupt handles

http://www.amazon.com/Linux-Kernel-Development-Robert-Love/dp/0672329468/ref=sr_sp-atf_title_1_1?s=books&ie=UTF8&qid=1389542475&sr=11&keywords=linux+kernel+development Page: 3322 © 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

75

Interrupts are not Process/Threads This is why you can’t see interrupts in a ‘ps’ or a ‘top’, different data structure than processes/threads. Very much like them but CANNOT SLEEP/BLOCK! Soft Interrupt Table array Softirq[0]

Softirq[1]

………

Softirq[31]

Process/Thread Table (used for ‘ps’ and ‘top’)

Hard Interrupt Table linked list

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

76

Device Drivers A network device driver runs within the kernel and has two primary functions: User Process

Kernel Driver Transmit Functions

Interrupt Handler

Device Driver

1) Organize data so NIC can grab it

2) DMA data to kernel memory and send interrupt when ready © 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

77

DMA vs CPU copy Note that a Direct Memory Access (DMA) processor is responsible for transferring data between the NIC and the host. This allows the CPU to parallel process other activities while the transfer occurs. There my be some bus contention between the CPU and DMA, but not as bad as if the CPU had to perform the transfer.

Kernel

User Process

DMA Interface

Initiate Data Transfer!!!

Host DMA chip

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

78

Packet Journey – User to Device Linux Kernel Send device interrupt when data is ready to be copied by DMA transfer

Net I/O Kernel Thread On Behalf Of User Copies into DMA space and massages data through TCP/IP stack

Call to device driver to put data into the right structures for the NIC and tell the NIC it can copy.

hard_start_xmit()

Linux User Space

send(socket, msg, strlen(msg), 0)

Application

eth0 TCP/IP Stack DMA Space

http://www.ece.rice.edu/~willmann/teng_nics_overview.html#overview http://www.amazon.com/Essential-Device-Drivers-Sreekrishnan-Venkateswaran/dp/0132396556 Page 505 http://www.linuxfoundation.org/collaborate/workgroups/networking/kernel_flow © 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

79

Packet Journey – Device To User Linux Kernel Hardware Interrupt (Top ½) (Stops processor)

Linux User Space Software Iinterrupt (Bottom ½) Massage packet through TCP/IP Processing

User process waits for data on socket

User thread Inside kernel context

int recv(int s, void *buf, size_t len, int flags);

DMA Space

eth0

Application

Signal to continue

User Space

TCP/IP Stack

IRQ 177

Soft IRQ copies data to user space

Device DMAs data to kernel DMA memory

Big job! Schedule software interrupt

http://www.ece.rice.edu/~willmann/teng_nics_overview.html#overview http://www.amazon.com/Essential-Device-Drivers-Sreekrishnan-Venkateswaran/dp/0132396556 Page 505 http://www.linuxfoundation.org/collaborate/workgroups/networking/kernel_flow © 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

80

Monitor # of Interrupts per Device /proc/interrupts keeps track of # of hw interrupts per interface since boot. Linux will use eth0 as default for network cards until system gets busy then it tries and re-balance between CPUs (see eth0)

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

81

Interrupt Affinity In Linux, you are able to wire an interrupt to a specific CPU. This is called Interrupt Affinity. Once again this allows data to be cached locally on a single (set) CPU for the interrupt handler. Linux will start with CPU 0 handling all interrupts. (?? Does Linux auto balance??) Interrupt affinity is used by Check Point CoreXL as we will see in the next section. https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide/s-cpu-irq.html

IRQ 32 CPU 1

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

82

Linux Network Summary

• Interrupts come in 2 halves • Top – Hardware device driver – Quick and get out • Bottom – Heavy lifting of data through kernel • Interrupts are NOT threads or processes. They have many things in common but you can’t view them and are internal to the kernel. • Device drivers come in 2 halves • Transmit – send data to device • Receive – Usually interrupt driven to pick up packets

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

83

Linux Evolution wrt SPLAT/GAIA Date

Linux Release

CP Release

2000

Linux 2.2

R 4.0

Nov 2001

Linux 2.4

NG FP1

Jan 2008

Linux 2.6

R65 HF02

Today

Linux 2.6-18cp

R77.10

http://en.wikipedia.org/wiki/Check_Point_VPN-1

Linux Tidbits: GPL (GNU Public License) – • No custom mods to Linux, they have to be shared with Linux community as source code. There is a version 2.6.18cp (not sure status) • Can compile code with GNU compiler and keep source private. CP uses internal kernel modules to do this.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

84

Balancing Check Point Systems

• • • • •

Overview Linux Review SecureXL CoreXL Balancing CoreXL Tips

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

85

Why SecureXL?? The classic problem has been at noon when everyone having lunch at their desk, everyone starts browsing the Internet and thus slowing down business traffic. In addition as more applications become web-oriented HTTP traffic is dominating the network. SecureXL was primarily created to address the web-browsing-at-noon-problem.

Symptom

Verify

Highly utilized system

‘top’, ps –o psr,command

High network traffic on 2 interfaces

/proc/interfaces

High HTTP traffic

SmartLog

Ifconfig is dropping packets, retransmits

Ifconfig, netstat -s

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

86

HTTP 1.0 HTTP 1.0 uses multiple concurrent requests to retrieve a multi-part web page for the user.

Text

Picture Text

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

87

HTTP 1.1 HTTP 1.1 uses a single request to retrieve a multi-part web page for the user.

Text

Picture Text

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

88

HTTP processing SecureXL is smart enough to know that all packets after packet #1 are all going to the same site and will hit the same rule. So SecureXL speeds up the processing of these packets by bypassing the full rule processing and just relying on state tables built by packet #1 to send the packets through the firewall.

Text

Picture Text

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

89

What does SecureXL accelerate? Secure XL can accelerate (bypass full rule processing) in two instances: 1) Subsequent connections to the same dest and port 2) On a single connection, packets 2 thru N.

Text

Picture Text

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

90

NGF Process Mapping SND does the acceleration and packet distribution. NGF has 1 SecureXL module to accelerate network packets. Firewall dispatcher sends packets to the right firewall instance (there can be many..next section) https://downloads.Check Point.com/fileserver/SOURCE/direct/ID/7513/FILE/CoreXL_Advanced_Configuration_Guide.pdf

User mode Kernel mode

fwk

Firewall Dispatcher(fwkdrvr)

SND

Performance Pack Packet Handler (SecureXL acceleration)

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

91

VSX Process Mapping In VSX, SND is still one module but SND is aware of individual VSs. SecureXL can be turned off/on per VS but SND shares info about all VSs when making acceleration decisions for the whole chassis. When Check Point moved the fw kernel from the Linux kernel to User Mode, they left only a little bit of code to work with the firewall dispatcher in place. Other than that it was a clean compile of the User mode kernel…This was not a massive rewrite

VS0

VS1

VS2

User mode Kernel mode Firewall Dispatcher (fwkdrvr)

SND

VS0 SecureXL

VS1 SecureXL

VS2 SecureXL

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

92

NGF Firewall Kernel Modules Here you can see the kernel modules for NGF vs VSX. Basically the same firewall module is used for both. But…probably…VSX only uses the bottom half processing because the firewall itself is in user space.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

93

SecureXL Acceleration There use to be a hardware acceleration device by Nokia, now its in software. CP recently put it back in hardware

SecureXL acceleration 1) 2)

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

Subsequent packets from a single connection Subsequent packets from the same source IP, same dest IP and same dest port (multiple HTTP requests to same dest) 94

Packet Journey Thru SecureXL Linux Kernel

Linux User Space

Hardware Interrupt (stops whole processor single thread)

Device driver hogs processor. Can’t be interrupted. Just transfer data.

Software Interrupt

Core 0 IRQ 177

Eth0 needs service

FW Instance 1

Core 0

Signal ‘Continue’

Can’t accelerate this, send on to a specific FW Instance

Big job! Schedule software interrupt

Standard Linux Processing

Core 1

FW Instance 2

Core 2

SND picks instance 1 to process packet

Core XL © 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Core 3

FW Instance 3

SND Interface eth0

Device Driver

eth0

Concurrent processing of SI’s is possible unlike HW interrupts. Can be interrupted.

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

95

Firewall Chain – SecureXL Modules Once packets make it through SecureXL, these are the modules in the firewall chain that build the SecureXL connection tables and sync with the SecureXL module itself.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

96

SecureXL and IPS/AV/Bot Integration When IPS/AV/Bot is enabled, not all traffic can be accelerated because it has to be inspected by the IPS engine.

PSL is the Packet Streaming Library for re-assembly of IP packets so the IPS/AV/Bot can look for signatures. SecureXL can forward packets directly to the PSL and bypass firewall processing. This is called ‘Medium Path’ because it bypasses rule checking on the 2nd+ packets

Fwaccel stats – SecureXL statistics for Medium Path

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

97

VSX Detailed Internals VS2

Packet stays in pool Should I Accelerate?

VS1 VS0

Packet Data Pool

HW Interrupt

Firewall in user mode inspects and creates pass/fail

Construct The Packet Packet Meta-Data Queue

SND

Pointer to packet is what moves

fwk

Outbound to another VS

Fwkdrv out F2F

Accelerate

Fwkdrv in F2F

Write pass/drop to msg q

Outbound to another VS

Dispatches to right VS

IP Stack Inbound

Message Queue Fwdrv deq

SND Outbound Device Driver

Implements action from fwk to pass/drop

Legend Kernel Code User Space Code Shared Memory

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

98

SecureXL Demo - VSX

• • • • •

Pass 2 gig file thru a firewall R75.40VS VSX Traffic going through fw-vsx1 a virtual firewall All run inside Virtual Box No other traffic

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

99

VSX – SecureXL On SecureXL runs as a software interrupt within the Linux kernel. You can see the %si get higher when it is busy.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

100

VSX – SecureXL off When SecureXL is OFF, you can see the fwk1_dev thread handle the work.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

101

AutoBalance Interfaces Note how the interrupt handling got shifted from CP0 to CP1. More on this later

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

102

SecureXL Demo - NGF

• • • •

Pass 2 gig file thru a firewall R75.40VS NGF Standalone gateway All run inside Virtual Box No other traffic

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

103

SecureXL OFF – FW busy Worker threads are idle, SND and FW sharing the work Busy kernel threads

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

104

SecureXL OFF – FW sees packets Firewall sees all packets in fw monitor

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

105

SecureXL ON– FW idle Worker threads are idle, SND is doing all the work Work done in SND

Work done in SND

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

106

SecureXL ON– FW no packets The first packet may/not be seen if its in the state table already. But no others

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

107

4 types of acceleration 1. Standard state table We talk about these 2. Accept connection templates (e.g. like HTTP) 3. NAT templates – Perform NAT in SecureXL and not in the firewall kernel (sk71200) [Expert@HostName]# echo 'cphwd_nat_templates_support=1' >> $FWDIR/boot/modules/fwkern.conf [Expert@HostName]# echo 'cphwd_nat_templates_enabled=1' >> $FWDIR/boot/modules/fwkern.conf [Expert@HostName]# cat $FWDIR/boot/modules/fwkern.conf [Expert@HostName]# reboot

4. Drop templates (sk66402)(R76- HF/R76+Standard) – Drop packets in SecureXL, negation of rules. 5.

Other drops (have to test) SAM, DDOS, SmartEvent/IPS triggered. CONFLICT: sk33781, sk66402, sk98348 all conflict with one another on this topic. Have to test. Sk98348 says drop templates are the negation of the security policy and drops are accelerated if it does not match a rule.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

108

Standard State Table Standard state table tuples will be accelerated. Here you can see a state table entry. Subsequent packets will get accelerated.

fwaccel conns

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

109

Accept Connection Temples Accept connection templates that will be accelerated. Here you can see where subsequent connections from the same source IP will be accelerated. We created multiple SSH Sessions through the firewall same Client->Server

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

Our demo had 1 accept template

110

fwaccel stat This command provides the status of SecureXL

State table Accept connections DOS drops NAT in SecureXL

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

111

fwaccel stats This commands provides you statistics on which packets are accelerated and which are forwarded To Firewall(F2F) for rule processing.

Total acceleration “C” Current Counts # from accept templates NAT performed by SecureXL PXL: PSL + SecureXL IPS packets

Connections sent to firewall, NOT XL/SLOW PATH

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

112

Debug Tips • If you have any random issues, immediately turn off SecureXL to determine if there is a difference • Using ‘top’ to monitor performance, turn SecureXL on/off and see what %SI is doing • Might have to distribute SecureXL across multiple cores if %SI is busy and doesn’t autobalance. See next section. • Monitor stats to make sure both state table and connection templates are being used • Move HTTP 1.0 type protocols to the top of the rulebase so they get hit • Avoid protocols that disable connection template acceleration (more on this at end)

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

113

SecureXL Licensing • ADN – Advanced Data Networking & Clustering (formerly ACCL) • SecureXL • Dynamic Routing • CoreXL???????? • ClusterXL • QoS,load balancing, ISP redundancy • Looks like CPSB-ADNC • Platform • Appliances – All inclusive • Open Platform • Ala Carte List - $1500

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

114

SecureXL Summary

• • • •

Definitely required for high usage gateways Easy to administrate (on/off) Understand the difference between state table and connection templates All takes place in the %SI under ‘top’

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

115

Balancing Check Point Systems

• • • • •

Overview Linux Review SecureXL CoreXL Balancing CoreXL Tips

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

116

Why CoreXL CoreXL allows you to utilize multiple processors on a single chassis for concurrent processing of firewall requests in order to expand capacity and reduce latency on your existing platform. Cheaper to expand capacity on a single bigger chassis than to cluster multiple smaller chassis (ClusterXL)

• Cost efficient • Easier to manage • Easier to debug

• ClusterXL: More expensive to purchase and maintain

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

117

CoreXL Manages 4 Affinity Types 3) CP Process Affinity

2) Fw kernel Instance Affinity

VS1

VS2

FWD logging

syslogd

1) Interface Affinity

SND

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

4) Linux Process

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

Secure Network Distributor

118

CoreXL

• • • •

Interface Affinity Instance Affinity Process Affinity Linux Process Affinity

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

119

The LIE • Interface affinity is grey zone. Could be included in both SecureXL and CoreXL • Interface affinity can be used with SecureXL license and no CORExl license • Interface affinity can also be used without CoreXL or SecureXL license, it is a Linux function • Here - Interface affinity is grouped with CoreXL for completeness and topic flow

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

120

CoreXL Interface Affinity Interface ‘Affinity’ is the mapping of processors to interfaces to handle network packets. Default is ALL, which in reality is Core 0. SND is responsible for managing interfaces assigned to that core. If there are multiple CPUs handling different interfaces, then each CPU has a different SND.

1) Interface Affinity

SND

Secure Network Distributor • Process network traffic • Accelerate • Distribute to firewall instances

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

121

MultiQueue (sk80940) MQ enables assigning multiple interrupts per interface. Certain interface cards have multiple TxRx queues per interface. Src/Dst flows are tied to a queue. Then queues are assigned IRQs and tied to specific processors. This technique optimizes CPU cache utilization.

https://greenhost.nl/2013/04/10/multiqueue-network-interfaces-with-smp-on-linux/

Standard on R76 and R77 Previous 71.50

IRQ 1

IRQ 1

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

IRQ 2

IRQ 3

122

MultiQ Restrictions • • • •

R77.10 Only on appliances…needs the right hardware and drivers Supports increased throughput, not so much increased number of sessions Based on src/dst assigned to a CPU. So a single high throughput src/dst will only use 1 CPU and not take advantage of multiple CPUs.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

123

Display Interface Affinities

In this complex environment, SND can concurrently run on cores 0-7 © 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

124

Monitor Hits on SND /proc/interrupts keeps track of # of interrupts per interface since boot. We can use it to monitor invokations of SND In this simple environment SND has ability to concurrently run on any core (that isn’t running a fw instance), but by Linux default it chooses to run on CPU 0. (Probably not good because all interfaces and processes will use CPU 0). So if you have interfaces that are dropping packets, you might want to check this if CPU 0 is busy. POINT: Even though the configuration seems balanced, you need to verify!

SND on core 0 is doing ALL the work for all interfaces

SND has never autobalanced, all on Core 0 © 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

126

IRQ to CPU mapping Linux maps IRQs to CPUs in the proc/irq directory in the smp_affinity file These are used to program the APIC (interrupt controller chip). By default a IRQ can run on any processor but

Linux chooses Core 0 Why core 0? Well the kernel doesn’t want to have the interrupt context float between CPUs. This way it keeps interface interrupt data in local cache. R77.10 changed from ‘all’ to CPU 0. © 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

127

Setting affinity command

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

128

Setting interface affinity sim affinity –s sets the interface affinity by changing the values in the smp_affinity file. This in turns programs the APIC to send interrupts to a different processor.

NOTE: Interface settings will survive reboot (BUT not CoreXL settings (next)).

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

129

Impact Immediate No reboot, sim affinity –s takes effect immediately

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

130

Distribute SND processing This is how you distribute SND processing on interfaces that are overloaded and dropping packets. Give those interfaces/SND their own processor that does not have other FW components on it

VS1 VS2

SND

SND

Busy

SND

Busy

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

Secure Network Distributor

131

SND Chooses non-fwk CPUs When an interface is set to All, it will attempt to use a CPU that is NOT being used by a firewall instance. But it will try to use a CPU that is being used by another interface…in order to keep the the local CPU cache fresh. So when under low CPU usage, most interfaces will default to ALL (below) and be autobalanced as CPU and interface activity picks up. The default for ALL is CPU 0…until CPU activity picks up R77, interfaces will be set to the default CPU0

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

132

Interface Auto - Balance Look what happens when straining a CPU with a 3 gig SCP transfer between members directly connected. Every 60 seconds, CoreXL examines the CPUs to see if they are busy. If they are busy it will rebalance interfaces to non-busy CPUs. (fwkernel and Linux processes rebalance every 1-2 seconds). (NOTE: I do NOT know how to set back to autobalance once you hard set the interfaces except by reboot on NGF, or factor defaults on VSX) Below the fw kernel saw the CPU going to 80 si% and rebalanced the interfaces from ‘all’ to ‘eth1:1’, gave eth1 its own CPU

Before After

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

133

Busy box? How to tell if a box is ‘busy’??? Here is an example of an Internet gateway with 10gig interfaces. This is the /proc/interrupts table. You can see that only CPU 0 has been used to process network packets. This tells us that the CPUs have not become busy since reboot and the fw kernel has not done any rebalancing. If the ifconfig shows packet drops, then you have different issues than CPU not being able to handle the load.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

134

Set Interface Affinity – No Performance Pack

$FWDIR/conf

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

135

CoreXL

• • • •

Interface Affinity Instance Affinity Process Affinity Linux Process Affinity

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

136

CoreXL Manages 4 Affinity Types 2) Firewall Instances

VS1

VS2

FWD logging

syslogd

SND

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

Secure Network Distributor

137

HyperThread Review Enabling Symmetric Multi-Threading (SMT) or Hyper Threading (HT) doubles the number of logical processors. • • • • •

Note the difference between HT, Dual Core and Dual Processor. Note where the caches are! Remember this when you assign interfaces so you keep the cache hot Works just like Linux thread processing where a process (web server) has 2 threads (2 clients requesting pages) and the kernel can preemptively multi-task the two threads so it seems like they are parallel processing. Without HT. Each Linux thread gets a time-slice by the Linux kernel but only 1 thread runs at a time. With HT: At the hardware level there is a mini-Linux like kernel that can multiplex/task 2 threads concurrently. So the two threads could conceptually start and finish within 1 kernel time slice instead of 2 separate time slices. Performance improvement 30%???

Without Hyperthreading, Managed by Kernel Thread 1

Thread 2

Thread 1

Thread 2

WithHyperthreading, Managed by HW

http://blogs.msdn.com/b/gauravseth/archive/2006/03/20/555519.aspx

TIME © 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

138

HT and Cache Sharing Remember that the kernel instances share a state tables. So when allocating instances keep similar data flows on the same cache so that portion of the connection table is always in cache.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

139

R77.10 Uses SMT/HT Implemented in R77+ Restrictions:

• Only enhances performance of IPS/AB/AV CPU intensive functions and NOT I/O operations. Too many interrupts may actually slow it down. • Supported only on R77+ GAIA • Only on Check Point Appliances • Has to be enabled in the BIOS • Does not work with large number of HIDE NAT connections. Each CPU has pre-allocated # of HIDE NAT slots. If one CPU uses all its HIDE NAT slots then it can’t handle new HIDE NAT connections.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

140

Assigning Instances to CPUs

• NGF Firewall Affinity • VSX Firewall Affinity

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

141

NGF Standalone Instance Affinity In NGF there is only 1 firewall. With CoreXL, the kernel will replicate itself X times, depending on how many firewall instances you setup. Each instance will parallel process network traffic with the SAME shared rulebase and state tables. Each instance has an ‘affinity’ for a specific processor.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

142

NGF CoreXL

Affinity

In a NGF gateway, CoreXL generates X copies of the kernel into individual Linux kernel threads

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

143

NGF Process/Thread Mapping Before we assign affinity, we should look at what types of processes/threads we are dealing with. When multiple processes have the same PID, that means they are threads sharing the VM of the parent (CPD and FWD below). Here you can see the firewalls are individual KERNEL threads inside the kernel with a parent of PID 1 – ‘init’. Kernel threads usually come in thread groups of size 1, unlike user space.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

144

NGF Process Spawn Tree Here you can see how the firewall processes were spawned in what order

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

145

Assigning Instances to CPUs

• NGF Firewall Affinity • VSX Firewall Affinity

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

147

VSX CoreXL Affinity Where NGF has 1 firewall on a chassis, VSX has multiple firewalls running on a single physical chassis. Each firewall is represented by a Virtual System (VS).

VSX VS’s are a different than NGF. Each VS has a 1 firewall instance that is executed by 1 corresponding Linux OS User mode process (not totally true, but not lying and still making the point).

VS

VS

VS

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

VS

148

R75.40 VSX Process Architecture VS instances are implemented with Linux threads (user mode but mapped onto kernel threads so they can be scheduled by the kernel)

• • • • •

Virtual Memory Space CPU Instructions Processor state Kernel threads Security attributes

=

fw kernel instance

VS instance VS instance

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

VS instance

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

149

VS instances The terminology gets a little confusing because CP is not explicit nor consistent in defining what an instance is for VSX. A firewall kernel instance which runs on behalf of a VS, can also be further subdivided into VS instances. These VS instances can be assigned to individual processors.

VS

fw kernel instance

VS instance

VS instance VS instance

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

150

VSX VS0 CoreXL Affinity Here you can see we are configured 3 VS instances within VS0 (very important!!). I have 3 VS instances running. So you can see that VSi0-VSi2 are allowed to run on any of the 1-3 CPUs. CPU 0 is reserved for eth0 traffic (hold on).

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

151

VS0 Processes and Threads So VS0 has 4 VS instances (VSi) generated for it that are watched over by the watcher daemon.

Processes

Threads

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

152

VS0 CoreXL Config You can only configure CoreXL VS0 from the command line….

Trying to configure more VS instances for VS1 from the command line and look….

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

153

Change non-VS0 CoreXL OK. We’ll listen to directions…

1= CoreXL OFF

2+ = CoreXL ON

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

154

Look Ma Babies!!! • • •

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

VS1 now has 2 instances. CPD has 1 main and 2 instances FWD has 4 daemons (not sure why)

155

Linux view of VSX instances VSX uses USER threads for all components

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

156

VSX Process Spawn Tree Here you can see the parent-child relationship of how firewall instances are spawned. Note: the process names change, like fw could be fwd – Linux command line issue.

pstree -p

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

157

Assigning Instances to CPUs

• NGF Firewall Affinity Assignment • VSX Firewall Affinity Assignment • Monitor Affinity

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

159

Default Affinity Assignment The default affinity for firewall instances is to assign CPUs from high to low. There can never be more instances than CPUs so that mapping is 1:1. The default affinity assignments for interfaces is to CPUs that are NOT running firewall instances…if possible. Note that the default assignment is probably adequate for 90% of the cases….unless you have a lot of busy network interfaces, help processes, Linux processes that interfere with the firewall instances. For example, if eth0 ran 100% of CPU0, then you might want to move the firewall instance. From sk98348

# of Cores

# of FW Instances

# of SNDs

1

1

0 (Corexl disabled)

2

2

2

4

3

1

6-20

# cores -2

2

21-30

# cores -4

4

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

Reserved for SND

160

NGF: Assign Affinity When you re-assign an instance to a CPU, you are telling the instance to only use THAT CPU…when the CPU is free. So double-edge sword: 1. GOOD: Guarantee the cache will be always hot on that CPU 2. BAD: What if that CPU is busy with other assigned processes…Has to wait till end of the other process timeslice to get CPU time. You could up its priority. So make sure you choose a CPU that is NOT assigned to any other process if possible.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

162

Assign ALL CPUs You can set affinity to ALL CPUs by assigning all CPUs to the instance. You can obviously also set the affinity to a subset of CPUs. Note: that you forfeit hot caching

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

163

Affinity Set via Linux Note that instead of fw ctl affinity command, you could just use the regular Linux affinity command ‘taskset’. Does the same thing.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

164

Permanent Affinity Custom firewall instance affinity are not permanent (but INTERFACE affinity is). On reboot you have to re-assign affinity. You can make firewall instance affinity permanent configuring it in $FWDIR/conf/fwaffinity.conf. NOTE: auto vs. all

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

165

Stickiness State table tuples are ‘sticky’ to a single core. Once you start a network conversation through a specific core, the associated tuple will always use the same CPU to process that network conversation. So a backup process between backup server A and client B for example could monopolize a single CPU. <1.1.1.1, 1111, 2.2.2.2, 2222, TCP>

2.2.2.2

1.1.1.1 CORE 0

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

166

Assigning Instances to CPUs

• NGF Firewall Affinity Assignment • VSX Firewall Affinity Assignment • Monitor Affinity

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

167

VSX Default Affinity The default config for VSX is firewall instances are assigned to all but CPU 0 (reserved for interfaces). Default varies depending on components, but it won’t be “ALL”.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

168

VS Affinity Command Hierarchy In VSX there is a hierarchy how VS affinity commands work. You can apply the instance command to the ‘V’ entire VS, ‘P’, only the firewall processes or ‘I’ a specific VS instance. The ‘I’ then inherits from the ‘P’ which inherits from the ‘VS’.

‘VS’ for Virtual System

‘P’ for Firewall Processes ‘I’ for instance

cpd

fwk VS instance

fwd

vpnd

VS0 © 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

169

Impact of Setting Affinity The SRC column shows at what level the affinity command for the process was issued. ‘V’ means the command was issued to ALL the components of a VS. ‘P’ means only to the firewall instances. ‘I’ means a single fw instance.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

170

VSX set affinity command Here is the command for setting affinity for the 3 levels of processes. Note that if you do not set a affinity for a level, the level will inherit the affinity from the previous level.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

171

VS affinity config files After you set affinity you can see the impact in these configuration files. Because of these files, VSX affinity is permanent…unlike NGF!!!

Here are the affinity configurations that the VSs use to set their affinities. As you set affinities at the different levels, these files will begin to appear. This is how a VS instance knows what affinity to use. If there is no config file at the I instance level, then it goes up to the P Process level config file, etc.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

172

Set VS affinity Here we set the VS affinity. Note that it sets the affinity for ALL processes in VS0. Firewalls, Firewall Helpers, Linux processes. Note how the SRC column is “V” for the VS affinity configuration file

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

173

Set P Process Affinity This will set the P Process affinity for JUST the fwk processes in a VS. NOTE: the ‘P’ means to set affinity to what is found in the ‘P’ config file

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

174

Set P Process Affinity This will set the P Process affinity is for JUST the fwk processes and VS instances. NOTE: -fwkall will set the fwk VS instances for ALL the VS’s.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

175

Set ‘I’ VS Affinity Here you can see we set he ‘I” VS Instance affinity The “I” in the SRC column means the affinity config comes from the ‘I’ config file

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

176

Now let’s set everything to ALL As with NGF, if you set affinity with all CPUs, it will be set to ‘ALL’. Note it only sets the affinity for VS0, and not the other VSs

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

177

VSX: CoreXL set per VS CoreXL can be enabled/disabled per VS instance, just like SecureXL VS0 use cpconfig, VS1+ use SmartDashboard to set CoreXL to 1

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

178

Reset to Defaults REBOOT!!!!!

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

179

Changing Cluster CoreXL config At boot - Cluster membership is tested to ensure same # of CPUs in CoreXL. When changing CoreXL cpconfig CPU configuration in a cluster: 1. 2. 3. 4. 5. 6. 7.

Start on standby member B, bring it down Change number of processors - cpconfig Reboot Member B Comes up in the READY state Fail over Active member A to the Ready member B (Stateless) Modify formerly Active member A Reboot member A

SETTING AFFINITY (not CoreXL Firewall Instance count) must be done manually in both members and does not impact cluster status

Member A - Active

Member B - Ready

https://sc1.Check Point.com/documents/R76/CP_R76_ClusterXL_AdminGuide/7298.htm © 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

180

Cluster XL Hints 1. Keep SecureXL and CoreXL configuration the same on both members 2. On upgrades, config will carry over but not necessarily on from scratch migrations. ( e.g. process assignment)

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

181

Summary: NGF to VSX CoreXL NGF can delegate its firewall processing into multiple fw instances. Each fw instance is represented by a Linux kernel thread and can be assigned a core affinity.

Linux Process /Firewall Instance

VSX has each VS represented by a Linux process with 1 or more internal VS instances mapped to Linux threads. The whole Linux process OR the VS instance (thread) can be mapped to a processor

Linux Process /Firewall Instance Linux Process /Firewall Instance

Firewall Instance

Firewall Instance

Firewall Instance

Linux Process /Firewall Instance

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

182

Assigning Instances to CPUs

• NGF Firewall Affinity Assignment • VSX Firewall Affinity Assignment • Monitor Affinity

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

183

‘top’ Tricks for CoreXL F

SORT j u

f

Processor Number of page faults

Add columns j u

Processor Number of page faults

1

Show all processors

i,z

Just show running processes

c

Command vs process name

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

184

Important TOP Stats with ‘f’ command

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

185

Unix ‘top’ Show all processors and see what process is running on what processor

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

186

Monitor Running Processes With ‘f’ ->‘i’ and ‘j’ and ‘u’ you can monitor running processes that might be faulting or jumping processors

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

187

CoreXL ‘ps’ arguments

-e

All processes

-L

All threads

-o

Specific columns (below) pid

Pid of process

psr

Processor id

cmd,comm,com mand,args

Command/Process names in long-short formats. with (args) arguments

lwp

Thread ID

majflt

Page faults

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

188

Unix ‘ps’ to monitor processor ‘watch –d –n1’ will execute a command every 1 second and highlight changes ‘ps’ can be used to monitor processor usage by a specific process

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

189

‘watch’ Interrupt table

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

191

CoreXL

• • • •

Interface Affinity Instance Affinity Process Affinity Linux Process Affinity

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

192

CoreXL Manages 4 Affinity Types 3) CP Process Affinity

VS1

VS2

FWD logging

syslogd

SND

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

4) Linux Process

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

Secure Network Distributor

193

List firewall helper daemons Firewall helper daemons (cpd, fwd, vpnd) are running in the VS0-VSX VS’s. Linux processes are running under the control of VS0. Many times you will see they are able to run on ALL processors

Linux only processes

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

194

Set Process Affinity Let’s say your fwd logging process is monopolizing the system with heavy logging. You an assign a whole processor to it to handle the load. NOTE: /opt/CPsuite-R75.40VS/fw1/conf/vsaffinity_exception.conf are a list of LINUX processes that are not impacted by the affinity command. You have to edit this list to modify LINUX process affinity

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

196

Process to CPU Busy processes such as VPND or FWD (logging) can be dedicated to a single CPU because they might dominate a core or swap cores frequently thus losing their cache freshness. Note in VSX it is specific to your VS context

FWD Logging/HA

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

197

CoreXL Licensing • Appliances include all the cores • Open Servers • You can select how many cores to buy • CPSG-PCC • CC = # of cores • CPSG-PCCBB • BB = # of blades • CPSG-P1207 • Security Gateway • 12 core • 7 blades • NOTE: You only have to license the cores that FWK run on. You can buy a 2 core license and have a 12 core gateway. ..10 of the cores will run Linux, SND, CP helper programs. 2 cores run FWK • Example: Ala carte 2 core to 4 core upgrade $7500 list

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

198

Balancing Check Point Systems

• • • • •

Overview Linux Review SecureXL CoreXL Balancing CoreXL Tips

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

199

Tuning Tips • • • •

Traffic to Rules to SecureXL to CoreXL tuning VSX: Keep traffic flows on same system VLAN resources CPU or Memory

•

IPS Integration???

• Suggested CoreXL Configurations • Command summary

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

200

Classic Target 2 interface Internet firewall, busy at noon with HTTP traffic interrupting business traffic

Symptom

Verify

Highly utilized system

‘top’, ps –o psr,command

High network traffic on 2 interfaces

/proc/interfaces

High HTTP traffic

SmartLog

Ifconfig is dropping packets, retransmits

Ifconfig, netstat -s

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

201

Determine Your Traffic Patterns eth0 to eth3 Use SmartLog to determine your major traffic flows between interfaces. Also look for busy tuples, remember they are sticky to processor!!

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

202

SecureXL Restrictions The following traffic is not throughput (state tables), nor connection-rate (accept templates)accelerated by SecureXL: Traffic types other than TCP, UDP, PIM, GRE, ESP First packets of any new TCP session, unless a "template" exists First packet in a UDP session Traffic matching certain Firewall rules: rules with a service that uses a resource rules for dropping or rejecting traffic rules where the source or destination is the gateway itself rules with a Security Server rules with user authentication rules with session authentication The following traffic is not connection-rate (accept templates) accelerated by SecureXL and will stop building templates in the rulebase if they are found: Non-TCP/UDP connections such as PIM, GRE, ESP ---- ICMP Protocols that are not connection intensive such as SMTP, FTP, RPC, NFS, NNTP, NTP Complex connections such as IPSec VPN, FTP, H.323, etc. Traffic in environment using NAT (for security, NAT addresses can change and can be shared) © 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

203

Check SecureXL and Rulebase ICMP prevents SecureXL Accept Connection templates from accelerating HTTP 1.0/type connections. Need to move to end of rulebase.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

204

HTTPS and CoreXL tuning Use /proc/interrupts and ifconfig to see if any of your interfaces were struggling to keep up. If the interfaces are constantly rebalancing or dropping packets, then you might have a problem.

All interfaces on eth0!

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

205

Processor Utilization Look to see if CPU 0 is busy. Busy FWD

Busy core 0

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

206

Balance CoreXL So investigation shows Core 0 is busy. This means you should dedicate a ‘free’ processor to eth0 because eth0 is doing a lot of work with regular processes AND interface handling

Give eth0 and eth3 their own core

Then give the offending process its own processor.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

207

Drop Rule If SmartTracker or SmartEvent shows a Denial of Service or heavy traffic from a malicious source, consider using drop templates or Suspicous Activity Monitor instead of the rulebase to drop the traffic. The drop templates are enforced by SecureXL and done before you have to go through the rulebase. https://supportcenter.Check Point.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk67861

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

208

Tuning Tips • • • • • •

Traffic to Rules to SecureXL to CoreXL tuning VSX: Keep traffic flows on same system VLAN resources CPU or Memory? Suggested CoreXL Configurations Command summary

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

209

VSX Archtecture With VSX which is the most efficient way to route packets between two networks and why?

Router

VSX Chassis

Router

VSX Chassis

VRouter

VSX Chassis

11

VSX Chassis

VRouter

1

VRouter

1

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

210

VSX Architecture Design Balance VSLS VSs based on traffic flow w/ SecureXL caching If you have lots of traffic between two VSs, then keep them on the same gateway because SecureXL /CoreXL will cache states and accelerate locally

Traffic states cached locally and SecureXL will not send it through user mode kernel. CoreXL will cache states in processor CPU dedicated to those interfaces

VSX Chassis

SND

VSX Chassis

SND

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

Tuning Tips • • • • • •

Traffic to Rules to SecureXL to CoreXL tuning VSX: Keep traffic flows on same system VLAN resources CPU or Memory? Suggested CoreXL Configurations Command summary

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

212

Bonded(VLAN) Interfaces

Core 0

VLAN

VLAN

VLAN

VLAN

When you have a bonded VLAN that trunks multiple networks over a single logical interface, (assuming it will be heavily utilized) considering assigning a dedicated CPU to each interface to handle the load and keep the caches ‘fresh’.

Core 1

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

Tuning Tips • • • • • •

Traffic to Rules to SecureXL to CoreXL tuning VSX: Keep traffic flows on same system VLAN resources CPU or Memory Suggested CoreXL Configurations Command summary

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

214

CPU or Memory How do you know if you need more CPU or more memory or both?

?

?

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

215

‘top’ is WRONG Linux 2.6.18 ‘top’ is just WRONG. It does not compute SWAP space correctly because it does not take into account (e.g.) that the processes is sharing 2GB of space from a shared library along with 10 other processes. So each of the 10 processes thinks it alone is using the 2GB of shared space. Below you can see that nautilus VIRT is 433MB, and SWAP is 427Mb, but so are most the other processes. It just doesn’t add up to the ‘used’ SWAP. So its hard to tell what processes are using what memory and what is using SWAP space.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

216

#1 Mo Memory Because you can’t trust ‘top’ too much, a better measure is to look at the major page faults for running processes. If they are faulting a lot, you have a problemo. Using ‘top’ watch the running processes by typing ‘i’. This will only list the running processes. Then use ‘f’ and ‘u’ for listing page faults. Below is a real live MLM that is faulting heavily on one of the logging daemons. You can have a fast CPU and add more CPUs, but fwd will spend most of its time swapping pages.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

217

#2 More CPU If you have a lot of active processes that are NOT faulting, then you have 3 options: 1. More CPUs 2. Faster CPUs 3. Set affinity on the most busiest processes so they are using their hot caches and not spending time flushing caches 4. Set the affinity of the busiest processes to -15 so that they get a bigger chunk of the timeslice

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

218

#3 Memory Leak 1. If ‘Swap used” is > 0 and growing, you may have a memory leak. I 2. If “VIRT” is growing over time on a process, you may have a memory leak.

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

219

Tuning Tips • • • • • •

Traffic to Rules to SecureXL to CoreXL tuning VSX: Keep traffic flows on same system VLAN resources CPU or Memory Suggested CoreXL Configurations Command summary

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

220

Classic Target - NGF 2 interface Internet firewall, busy at noon with HTTP traffic interrupting business trafficAssumes that SecureXL is working on ALL rules and SND is doing most the work and not the FWK. NO IPS/AV/ThreatPrevention,etc. Goal is to allocate evenly. Obviously this is only a guess, but you’d have to evaluate your system with the following commands to have more accurate measurement Symptom

Verify

Highly utilized system

‘top’, ps –o psr,command

High network traffic on 2 interfaces

/proc/interfaces

High HTTP traffic

SmartLog

Ifconfig is dropping packets, retransmits

Ifconfig, netstat -s

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

221

Balancing Goals This is the take-away slide, the value behind the whole course. Allocate CPUs in the following priority order: 1. 2. 3. 4. 5.

Share cache for common data Allocate CPUs to busy Internal interfaces Allocate CPUs to slower less busy External interfaces Allocate CPUs to FW instances Remaining threads are usually idle so distribute evenly and let kernel find idle processor

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

222

Reminder: Share Cache!!

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

223

Suggested CoreXL Distributions-NGF Type

CPU/Item

1

2

4

8 (2 core)

12 (2 core)

FW Processes

cpd

0

All

All

3

3

fwd

0

All

All

3

4

Linux

Linux processes

All

All

All

All

All

Interfaces (SND)

External Interface

0

0

0

0 (cache 0)

0 (cache 0)

Internal Interface

0

1

1

1 (cache 0)

1 (cache 0)

Sync

0

All

All

2

2

fw_0

0

0

2

4(cache 1)

6(cache 1)

1

3

5(cache 1)

7(cache 1)

fw_2

6(cache 1)

8(cache 1)

fw_3

7(cache 1)

9(cache 1)

Fw kernel instances

fw_1

fw_4

10(cache 1)

fw_5

11 (cache 1)

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

224

Classic Target - VSX VSX with multiple business silos to Internet. Assumes that SecureXL is working on ALL rules and SND is doing most the work and not the FWK. NO IPS/AV,etc NOTE: fw0 is not handling any traffic aside from sync. Only mgt traffic Goal is to allocate evenly. Obviously this is only a guess, but you’d have to evaluate your system with the following commands to have more accurate measurement

Symptom

Verify

Highly utilized system

‘top’, ps –o psr,command, vsx resctrl, vsx memstat

High network traffic on 2 interfaces

/proc/interfaces

High HTTP traffic

SmartLog

Ifconfig is dropping packets, retransmits

Ifconfig, netstat -s

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

225

Suggested CoreXL Distributions-VSX Type

CPU/Item

1

2

4

8 (dual core)

12(dual core)

Interfaces (SND)

Ext VS0

0

All

0

4

6

Ext VS1

0

0

1

1(cache0)

0 (cache0)

Ext VS2

0

1

2

1(cache0)

1 (cache0)

Ext VS3

0

1

3

3(cache0)

2 (cache0)

Int VS0

0

All

0

4

6

Int VS1

0

0

1

0(cache0)

3 (cache0)

Int VS2

0

1

2

1(cache0)

4 (cache0)

Int VS3

0

1

3

2(cache0)

5 (cache0)

Sync

0

All

0

4

7

fw_0

0

All

0

4

8

fw_1 (VS1)

0

All

1

5

9

fw_2 (VS2)

0

All

2

6

10

fw_3 (VS3)

0

All

3

7

11

Fw kernel instances

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

226

Suggested CoreXL Distributions-VSX

Type

Linux

CPU/Item

1

2

4

8 (dual core)

12(dual core)

cpd(VS1)

All

All

All but 0-3

6,7,8

cpd(VS2)

All

All

All but 0-3

6,7,8

cpd(VS3)

All

All

All but 0-3

6,7,8

fwd(VS0)

All

All

All but 0-3

6,7,8

fwd(VS1)

All

All

All but 0-3

6,7,8

fwd(VS2)

All

All

All but 0-3

6,7,8

fwd(VS3)

All

All

All but 0-3

6,7,8

Linux processes

All

All

All but 0-3

All

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

227

Tuning Tips • • • • •

Traffic to Rules to SecureXL to CoreXL tuning VSX: Keep traffic flows on same system VLAN resources Suggested CoreXL Configurations Command summary

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

228

SecureXL Cheat Sheet sim affinity –l

List interface affinities

NGF: fw ctl affinity –l –v -a

List all affinities including interfaces

VSX: fw ctl affinity –l -x

List all affinities including interfaces

sim affinity -s

Set interface affinities

fwaccel stat

Review SecureXL status

fwaccel stats

Review SecureXL stats

fwaccel conns

Review SecureXL state table

fwaccel templates

Review SecureXL accept connection templates

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

229

CoreXL Commands List interface affinities NGF: fw ctl affinity –l –v –a fw ctl multik stat

List all affinities including interfaces

VSX: fw ctl affinity –l –x fw ctl affinity –l -x –flags tkn fw ctl multik stat

List all affinities including interfaces. -flags tkn lists internal threads

cat /proc/interrupts

List IRQ table and interrupts handled by CPU

watch –d –n1 “ps –ef | fgrep syslogd” top -------- F->j fw ctl affinity –s –d –fwkall 3

Watch how a process changes CPU

Set all process affinities to ALL (auto)

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

230

Thanks!!!!

© 2013 Midpoint Technology, Inc. 952-837-6206 – [email protected]

1/24/2015

Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc

231