Balancing CoreXL and SecureXL Michael Endrizzi Director of Services and Training
[email protected]
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
1
Who Is Instructor? • • • • • • • • • •
•
Michael Endrizzi – Midpoint Tech Director of Training and Services Age: 56 CCSA,CCSE, CCSMA, CCISP, ITIL Information security since 1982 Developer on Secure Computing Sidewinder firewall in 1993 Worked with NSA Owned information security businesses Independent security consultant for 16 years Working with Check Point since 1996. 10 year hiatus into auditing now back Oh yeah, I like rock climbing
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
2
Balancing Check Point Systems
• • • • •
Overview Linux Review SecureXL CoreXL Balancing CoreXL Tips
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
3
Danger Will Robinson • • • • • • • • • •
This class will teach you how to tune BUSY/Critical firewalls Commands you learn here will take affect IMMEDIATELY These commands are to be used delicately. By issuing these commands without fully testing them, you may negatively impact performance on CRITICAL firewalls. R77.10 introduced different behavior on some commands. Read the docs. Most commands do not allow you do easily undo themselves. You have to use your notes and photo clips to record the current configuration in case you wish to revert! If you change multiple items at once, you may not be able to undo the changes in case of failure. Backups will not save this information. You have to do a snapshot/restore in order to recover. DISCLAIMER: By reading this line you are totally responsible for all changes to your environment.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
4
What is SecureXL?? SecureXL (aka Secure Network Distributor) is a way of speeding up rule processing by analyzing traffic patterns and handing off known and previously approved traffic to high throughput traffic handlers.
VS1 Secure Network Distributor SND
• • •
Processing incoming traffic from the network interfaces Securely accelerating authorized packets (if Performance Pack is running) - SecureXL Distributing non-accelerated packets among kernel instances - CoreXL
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
5
What is CoreXL? CoreXL allows you to increase the throughput capacity of your firewall platform thru the utilization of multi-processors concurrently processing firewall requests.
• Cost efficient • Easier to manage • Easier to debug
• More expensive to purchase and maintain
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
6
What is NGF/VSX? NGF (Next Generation Firewall) is a standalone gateway running 1 instance of the firewall module AND has the ability to run several threat prevention blades IPS/AV/AB/Threat-Emulation
FWK
VSX is a physical chassis that runs multiple instances of firewall gateways. Think of VMware that runs firewalls as guests.
VS0
VS1
VS2
Linux 2.6.18-XXcp All based on Linux © 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
7
What is SPLAT/GAIA? • Secure Platform (SPLAT) is the commands and kernel modules added to Linux which transforms the Linux platform into a Check Point firewall • GAIA was a self-contained command environment created to simplify administration. Looks like Cisco command environment.
GAIA
GAIA command shell – Self contained shell looks like Cisco CLI SPLAT command set added to Linux command set
SPLAT
Linux command set Linux Kernel
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
8
Goal NGF/VSX + CoreXL + SecureXL = Tuned System Describes many topics in these articles. After this course you will be able to understand these SK’s
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
9
Course Take Away Slide This is the take-away slide, the value behind the whole course. If I bore you to death with 200+ slides, this slide #190(approx) puts to practice what the whole course is trying to teach: Allocate CPUs in the following priority order:
1. 2. 3. 4. 5.
Share cache for common data Allocate CPUs to busy Internal interfaces Allocate CPUs to slower less busy External interfaces Allocate CPUs to FW instances Remaining threads are usually idle so distribute evenly and let kernel find idle processor
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
10
What Do I Need To Know? Basic Linux CLI experience, What is operating system, What are processes, What is a processor, What is a cache, . Basic programming: code, data, variables. Unix ‘top’
Unix ps
/proc file system
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
11
Secure/CoreXL Tuning Areas 5 areas of tuning 3) CP Process Affinity
2) Fw kernel Instance Affinity
VS1
VS2
FWD logging
syslogd
1) Interface Affinity SND
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
4) Linux Process
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
SecureXL 0) Rule Processing Secure Speedup Network Distributor
12
Firewall Management SmartCenter Components Smart Dashboard POLICY User Space
Edit Policy
FWD (logging)
CPD (mgt server communication)
Push Policy
FWM (Management Server)
Kernel Space
Linux TCP/IP
NIC
NIC 1/24/2015
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
13
NGF Components Smart Center MGT Station POLICY User Space
FWSSD (spawns security servers like SMTP filtering)
FWD (logging) CPD (mgt server communication)
Kernel Space
1/24/2015
VPND (VPN)
CPWD (watchdog for dead processes)
SPLAT/GAIA Kernel (fwk) (Security Enforcement)
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
14
VSX/VS Components
Smart Center MGT Station
User mode
cpd
cpd
cpd
fwk
fwk
fwk
fwd
fwd
fwd
vpnd
vpnd
vpnd
VS0
VS1
VS2
Kernel mode
1/24/2015
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
15
R75.40 VSX – FWK in User Mode
cpd
• Note: fwk was moved to user mode • With large number of VS’s, kernel was getting too big
fwk
fwd
vpnd
User mode
VS0
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
Our Lab Eth0: 10.2.1.253/24
Eth1: 172.17.1.2/24
Eth0: 10.2.1.101/24
Eth2: 10.2.2.253/24
Eth0:1: 10.2.2.101/24
Eth3: 172.17.2.2/24
Eth0 : 172.17.1.111/24
Eth0:2 : 172.17.2.111/24
Eth0:2: 10.2.0.101/24
Eth0:1 : 172.17.0.111/24 Eth0: 10.2.1.153/24
Eth0:1 : 10.2.0.153/24 VB: Host/Host#1
Eth1:1 : 172.17.0.1/24 VB: internal/internet
20
1/24/2015 © 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
20
Balancing Check Point Systems
• • • • •
Overview Linux Review SecureXL CoreXL Balancing CoreXL Tips
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
21
Linux Kernel Basics
• Linux Overview • Threads • Network Processing
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
22
Linux History
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
23
Evolution of the Kernel Multics-Unix
Computer Day 1 The Code Blob
User
User
User
User User
User
User
Kernel
•
•
Programs are one compiled binary all superuser PROBLEM: hacking, bad code corrupted whole system
1/24/2015
• • •
Users have their own process resources (memory, files..) Monolithic kernel has its own resources (memory, code base). Kernel could multiprocess user processes but not self
User
User
Kernel
Monolithic Kernel
•
Linux
NT - Mach
•
•
Kernel
Some tried to simplify the kernel…. Kernel broke self into smaller processes, some run in user space Processes used Inter Process Communication to work together
• PROBLEM: SLOW In-efficient PROBLEM: too big expensive because of context switches © 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected] for hardware at time Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
Mod ule
• • •
Mod ule
Mod ule
BACK to monolithic kernel (one memory space). Minimal context switch and no slow IPC Dynamic added/subtracted modules. User processes don’t call kernel process, user processes go into kernel mode with shared data structures! Minimize context switch and data passing (more later on this)
24
Linux Differentiators • Original kernels were monolithic: • Single binary process • Single processing thread • Cooperative Multi-tasking – Could not pre-empt kernel processing • Single Address space • Linux • Linux Tervald – Still heavily involved vs design by committee vs free-for-all • Pre-emptive kernel – Most kernel tasks can be pre-empted for higher priority tasks • Modular – Kernel functions can be dynamically created/removed Check Point implements firewall subsystem in these modules • Multi-processor support • Threads = Processes (Unique to Linux – will explain later) • Users can see internal kernel data in sysfs file system. Looking glass into kernel internals
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
25
Linux 2.4 to 2.6 Introduced to reduce the time a kernel task held on to the processor locking out other tasks. Overall increase in efficiency and multi-processing support.
1. Scheduler – Improved fair scheduling with 100’s of processors 2. Threads – Process = thread. No special handling for threads 3. Interrupts – Can be pre-empted and no locking out all CPUs while processing interrupts 4. Pre-emptive Kernel – The whole kernel is pre-emptive. Can be interrupted at any point
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
26
Linux Kernel Structure An application process consists of: • User Identity & Permissions • Code and Execution Pointer • Data Scratch Pad (Stack and Hash) • Check Point VSX Firewall Kernel and Helpers
When applications call the kernel, the kernel does not “take over” as a separate entity. Applications transform into the “The Hulk”. The application processing thread takes on kernel permissions, resources and code base to complete a task. If the thread does not go to sleep, processing returns to application mode (Bruce Banner) almost as if it made an internal function call.
Most kernel work done here. • Resource allocation • Scheduling • Security • Memory Mgt • File System Mgt • Communication • Check Point NGF FirewallKernel
Talks to hardware, handles hardware interrupts. In Check Point, drivers are right from manufacturer. Some appliances have modifications
http://www.amazon.com/Linux-Kernel-Development-3rd-Edition/dp/0672329468 © 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
27
Linux Kernel Basics
• Linux Overview • Threads • Network Processing
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
28
What is a program? Simple terms: A program is made up of: 1. CPU Instructions – Does the work 2. Data – Scratch pad area 3. Security attributes - restrictions
• CPU Instructions • Data • Security attributes
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
29
What is a process? A program is a compiled binary sitting on a disk in a file. A process is when the program is executed on a processor, assigned memory, and is managed by the OS kernel.
Program becomes a Process • CPU Instructions • Data • Security attributes
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
30
What Does 32-bit User Program Data look like? All 32-bit programs have 4 gig of memory available to them. This is how it is allocated.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
31
Why Care?: CPU vs. Memory problem Are you chasing a CPU or memory problem? Need to know how lack of memory will slow a system and make it feel like it is a CPU problem. ‘top’->’f’->’u’
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
VM Why do you care? If you ever want to figure out why a process is swapping, you need to be able to know what parts of a process are taking too much space.
Virtual Size
Physical Memory
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
33
Stack Frame A program is broken into functions. When a function is called, the program has to save information about where it was and pass information to the new function. This placeholder is called a stack frame. Stack frames are To High Memory Stack like a track of cookie crumbs to help you go back to where External Environment where. Parameters GROW
Program Counter
Saved Frame Pointer Local Variables
To 0 memory
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
35
Heap space and Memory Leaks Programs sometimes need dynamically allocated and sized chunks of memory for maybe network packets. The program uses the malloc call to get that memory. It came off the heap space. If the programmer forgets to free the memory, then you have a memory leak.
Sometimes heap space becomes so fragmented with mixed free and allocated memory, that programs slow down because its hard for them to claim and release memory efficiently. Thus ‘reboot!!!’.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
36
Buffer Overflow
Stack
Code
Length: 1000 bytes 301 LOGIN: + username +
Length: 256 bytes
Packets Fragmented Mail Program Running as Super-User
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
IMAPD Attack Authentication Attempt Backfires To 0 memory
Stack Local Variables
Attack Login String
UserName[256]
Mike Password
Saved Frame Pointer
New PC
Program Counter Parameters
EndOfBuffer NULL
Environment
To High Memory © 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
38
Virtual Memory 4GB Process A
4GB Process B
NOTE: Most programs don’t use all 4 gig at once. Only use small portion.
4GB Process C
4GB Process D
How much total memory is needed to store all these processes - MAX???
NOTE: This is for 32-bit systems. 64-bit can see 264 = 16 exibytes of memory
Virtual memory is 3 things: 1) Allows processes to think they own all memory 2) Allows processes to ignore physical memory limitations 3) Paging system: That uses disk to swap out sleeping data to temp storage area to free up physical memory for active processes.
Not needed right now so swap out
+
10GB Disk Swap space
Assuming all processes needed 3GB User space We would need at a minimum 12GB (3GB+3GB+3GB+3GB)User + 1GB kernel = 13GB Physical and Swap space to hold all these programs.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
Demo: Using VIRT Memory Here in my ‘memgrow’ program I allocated 1GB of HEAP memory, but didn’t read/write to it. Notice how the kernel allocated the VM space to my process but did not actually map it into physical memory or SWAP it out because we are running out of space (I only have 1 gig of physical memory)
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
DEMO: Writing to VIRT Memory Here the same ‘memgrow’ program BUT I wrote to every byte so the kernel had to map the page to my process and bring it into physical memory. And when physical memory fills up…then swap it out… WHAT IS VIRTUAL MEMORY???? 1) Process thinks it owns all 32-bit 3GB by itself 2) Process doesn’t understand physical memory constraints of 1GB (only 32/64 max addressable memory 3) Paging swapped out data that the kernel could not keep in physical memory
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
User Program Security Attributes Computers Day 1 allowed a program to see and do everything. But programs started to overrun each other. So Day 2 a kernel was developed that was the traffic cop between programs so they wouldn’t bump into each other OR corrupt the kernel itself. NOTE: On many embedded systems there is no kernel/user/security separation. All processes can see and do everything. For example: SCADA controllers that run machinery. Programs that monitor the brakes in your car.
User Level • CPU Instructions • Data • Security attributes
• CPU Instructions • Data • Security attributes
Kernel Level
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
43
What is the Kernel? The ‘kernel’ is responsible for resource management.
User Level
• • • • • • • • •
Access to physical devices Security separation Arbitrator of shared resources – who gets what first Virtual memory management Clock ticks and time slices Process management Scheduling processes File system access Network routing
Kernel Level
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
User Program Entering The Kernel Linux TODAY: When a process makes a system call, the process does not hand over processing to another entity. The processing thread gains access to kernel resources (after appropriate security checks) so the kernel is actually executing “on behalf of a specific process”. The process running in kernel mode now has access to both user and kernel memory. Its like a program making a function call and the function has enhanced magic powers, and then returns to the main user process. Just like “The Hulk”
User Space
Kernel Space
SYSTEM CALL (security checks)
+
• Kernel Data Structures • Superuser privileges My friend Arah becoming the HULK
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
45
3 Processing Contexts In Linux a processor can be in 1 of 3 processing contexts at any time….period 1. In user-space, executing user code in a user process 2. In kernel-space, in process context, executing on behalf of a specific user process 3. In kernel-space, in interrupt context, not associated with a process, handling an interrupt
Why do you care?
http://www.amazon.com/Linux-Kernel-Development-3rd-Edition/dp/0672329468
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
46
Pre-Emption and Context Switch In order to avoid one process from dominating a processor pre-emptive multi-tasking was introduced. This is where one process B can interrupt another process A at any time (preempt) resulting in a context switch to the process B. Prior to mid-1990’s context switches (stopping one process and starting another) were CPU/Memory intensive. Kernels had to copy the process state into internal kernel Process Control Blocks (PCBs) to save the state and then copy out a PCB to the new process. Process B
Process A • Virtual Memory Space • Stack and Hash Tables (Scratch Pad) • CPU Instructions • Processor registers • Security attributes
Context Switch
• Virtual Memory Space • Stack and Hash Tables (Scratch Pad) • CPU Instructions • Processor registers • Security attributes
Kernel Process Control Block
Process Control Block
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
47
Old Interprocess Communication Copy Data In the old days communication between processes and to/from the kernel data was copied. Very inefficient. Process B
Process A Copy this to Process B
Kernel
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
48
New Interprocess Communication Share Data So a shared memory mechanism was developed using virtual memory tricks that allowed memory spaces to be shared between processes and the kernel. Process A
Process B You can see some of my data
Kernel © 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
49
What is a Multi-Processor Multi-processors come in several different varieties • Multi-Processor: Multiple physical CPUs in a single chassis • Multi-Core : Multiple processor cores on a single physical CPU • HyperThreading: Simulated multiple processors with threads on a single physical core • Combination of the above • No matter which of the above configurations, the OS sees logical processors. The implementation is opaque. • Note where the caches are! Remember this when you assign interfaces so you keep the cache hot OS sees 4 ‘logical’ processors
Logical Processors Threads Cores Physical CPU
http://blogs.msdn.com/b/gauravseth/archive/2006/03/20/555519.aspx © 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
50
What are CPU Threads Enabling Symmetric Multi-Threading (SMT) or HyperThreading (HT) doubles the number of logical processors. • • • • •
Works just like Linux thread processing where a process (web server) has 2 threads (2 clients requesting pages) and the kernel can preemptively multi-task the two threads so it seems like they are parallel processing. Without HT. Each Linux thread gets a time-slice by the Linux kernel but only 1 thread runs at a time. With HT: At the hardware level there is a mini-Linux like kernel that can multiplex/task 2 threads concurrently. So the two threads could conceptually start and finish within 1 kernel time slice instead of 2 separate time slices. Performance improvement 30%??? on CPU intensive items. I/O intensive theoretically could slower. Shares a cache
Without Hyperthreading, Managed by Kernel Thread 1
Thread 2
Thread 1
Thread 2
WithHyperthreading, Managed by HW
TIME http://blogs.msdn.com/b/gauravseth/archive/2006/03/20/555519.aspx © 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
51
Example:HP-DL380p Multi-Core No HyperThreading HP DL380p – 2 Physical Processors 8 Logical Processors
CPU0
CACHE
CPU1 CACHE
HP DL380p – 8 Logical Processors No Hyper Threading
OS sees 8 ‘logical’ processors
Logical Processors Threads
Cache
Cores Physical CPU © 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
52
Example: CheckPoint 12600 12 core
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
53
What is a multi-processing? Day 3 – As loads increased computer designers decided to add multiple processors to the system. Processes lent themselves nicely to the paradigm, each process could float to a free processor and execute.
Process A • Virtual Memory Space • Stack and Hash Tables (Scratch Pad) • CPU Instructions • Processor registers • Security attributes
Process B
• Virtual Memory Space • Stack and Hash Tables (Scratch Pad) • CPU Instructions • Processor registers • Security attributes
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
54
Shared Data Contention LOCKUP!! The biggest problem with concurrency is access to shared data. If not programmed correctly two processes can fight over updating a shared piece of data and they fight to the death. What you see is a frozen monitor!!!
• Virtual Memory Space • Stack and Hash Tables (Scratch Pad) • CPU Instructions • Processor registers • Security attributes
• Virtual Memory Space • Stack and Hash Tables (Scratch Pad) • CPU Instructions • Processor registers • Security attributes
I’m going to write to variable X first!!!
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
NO! I’m going to write to variable X first!!!
Threads To The Rescue If a process wanted to enable concurrent processing (web server serving up pages, word processor with multiple open documents), running multiple processes was inefficient. Threads were created to support concurrent processing using SHARED data and NOT copying data between processes or resource heavy context switching. SHARED DATA • • • • • • •
Virtual Memory Space CPU Instructions Processor state Kernel threads Security attributes File system Signals
Thread C Thread B Thread A
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
56
Light Weight/Quick Context Switches Thread context switches are very fast because the kernel only has to save a several registers (TCB) (approx 96 bytes vs. approx 1K+ with heavy process switches). This is because all the non-saved data is shared between the threads so its live data and no need to save it and restore it. In addition, threads can see into each others address space because remember everything in the process is shared.
• • •
Thread registers Program counter Stack pointer
Thread B
Thread A
Mini- Context Switch
Kernel Thread Control Block(TCB)
Thread Control Block(TCB)
https://courses.cs.washington.edu/courses/cse451/11sp/section/kim_section4.pdf © 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
57
Threads and Multi-Processing Threads naturally lend themselves to multi-processing…you can concurrently run several threads on different processors. Of course there are shared data contentions that must be dealt internally by the threads
SHARED DATA • • • • • • •
Virtual Memory Space CPU Instructions Processor state Kernel threads Security attributes File system Signals
Thread C Thread B
Thread A
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
58
Threads and Scheduling Linux 2.6.18 changed its default scheduling to favor low latency processing for more realtime applications. It is called Completely Fair Scheduling (CSF). Its simple, when the kernel schedules a process it chooses a process that has used the least amount of its time slice. That way CPU-lite processes are favored. These are usually ones that are blocking waiting for input like network interrupts, keyboard, graphics, etc. When a process is ready to run, the kernel looks for: 1. What CPU is free on the affinity list for the process. 2. Did it run on the CPU before so the cache is fresh with my data? 3. Am I still in default mode where no autobalance occurred and so choose CPU 0 (mostly interrupts)
Thread B Which CPU?
Thread A
Thread C Which CPU?
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
59
Processor Affinity If you have a thread that is usually very busy, you can wire it to a single CPU (or subset of CPUs). That way all the data is cached locally on the single CPU and it can usually run less interrupted on that CPU. This is called Processor Affinity. A process/thread has an ‘affinity’ for a CPU. Can be done in Unix and Windows Oss Check Point leverages this in their CoreXL technology.
• • • • • • •
Virtual Memory Space CPU Instructions Processor state Kernel threads Security attributes File system Signals
Thread C
Thread B Thread A
CPU1
CPU2
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
CPU3 60
Types of Threads 3 types of threads: 1. User threads: Totally internal to process, kernel cannot see them limited multi-processing capabilities for example they couldn’t block to sleep or I/O. 2. User threads mapped onto kernel threads. Full multi-processing capabilities for user process 3. Internal kernel threads: Used only by kernel for internal kernel processing not visible to user processes http://linuxgazette.net/23/flower/threads.html http://www.thegeekstuff.com/2012/03/linux-threads-intro/
Internal kernel threads
User Level • • • • •
Kernel Level
• • • •
Scheduler Blocking I/O handling Thread table Monitoring
Virtual Memory Space CPU Instructions Processor state Kernel threads Security attributes
User Threads mapped to Kernel Threads
Internal Kernel Threads
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
61
Thread Demo Internal kernel threads have a [] around them. User threads, usually do NOT have [] on them. NOTE: sometimes user processes do, but its because the kernel can’t find its command parameters and notates it with [] which is confusing. Other commands can filter this out. “ps –ef”
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
62
Thread Demo: Windows Threads Even windows has threads
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
63
Demo: Kernel Thread Kernel threads have no virtual memory size because they all share the VM of the kernel and no allocated extra memory like user threads can be allocated. Remember this: will be similar when we look at CP NGX
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
64
Thread Demo: Multi-Processing Threads You can watch threads switching between processors to see which processor is busy
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
65
Linux Unique Thread Architecture Other OS’s have a concept of Light Weight Processes (LWP) (threads) where the parent process spawns owns, and manages the threads. Think mamma black bear protecting her cubs
Linux 2.6 everything is a thread, and they are self managed. They keep track of their own state and don’t rely upon a parent to manage them. Linux CAN simulate LWP to the outside world and this is what you will see in the demo/labs. Think busy bees – once born they are off on their own
Main Process Parent Code
Thread A Thread A
Thread B
Thread C
Thread C
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Thread B
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
66
Linux Process to Thread Relationship So a Linux process is like a beehive that keeps all the common data/honey. The threads are like the bees that go off and do their work and bring back data/honey where it is all shared between them.
• • • • • • •
Virtual Memory Space CPU Instructions Processor state Kernel threads Security attributes File system Signals
Thread A
Thread B
Thread C
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
67
Clone() – Threads and Sharing This system call is the heart of threads and sharing. Depending on what the programmer tells the thread to share, the thread could act like a full blown HEAVY process (old days), or like a little lite worker bee that doesn’t carry any baggage with them (Linux today). Other Unix’s are starting to implement this in some fashion but Linux was first. int __clone(int (*fn) (void *arg), void *child_stack, int flags, void *args)
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
68
Thread Summary
• Threads can have 3 contexts: • User • Kernel • Interrupt (very special type of thread, very limited) • Threads are the workers and the data is ‘usually’ stored in a shared common process space • Threads enter kernel space and take on kernel context: virtual memory space, and security context.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
69
Linux Kernel Basics
• Linux Overview • Threads • Network Processing
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
70
Interrupts The kernel interacts with external devices such as network cards in two ways: 1. Interrupts – Only interrupt the kernel when there is something to do can be more efficient 2. Polling – Continuously see if there is any activity on the device. Very inefficient and slows down the kernel Modern Linux device drivers use a combination of both (New API - NAPI). They wait for an interrupt to occur and begin processing. They will then disable the interrupt and go into a polling mode until there is nothing more to do (no more data). They will then re-enable the interrupt to be alerted for more activity. Interrupt Handler
Kernel Poll
Hey WAKE UP
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
71
Interrupt Priority Interrupts have a priority system. Interrupts such as the system clock have to be processed right away and are a higher priority. They will stop all processor activity (even lower level interrupt handlers) to be handled. Lower priority interrupts will disable its own interrupt line (eg. Network card) so that it doesn’t get interrupted from the same source and run uninterrupted (unless a higher priority interrupt comes along).
Kernel LOW PRIORITY
HIGH PRIORITY System Clock
http://www.amazon.com/Essential-Device-Drivers-Sreekrishnan-Venkateswaran/dp/0132396556 Page 493 NAPI © 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
72
Advanced Programmable Interrupt Controller (APIC) APICs are the hardware that manage interrupts. A motherboard had one I/O APIC that interfaces with the hardware and talks to LOCAL APIC controllers embedded within the CPU.
In SMP environments where IRQs can be handled by multiple CPUs APICs can be dynamically programmed by the kernel to direct IRQs to a specific CPU for balancing out the handling of IRQs from external devices.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
73
Top/Bottom Half Interrupts Interrupt processing is split into 2 parts: 1. Hardware Interrupt (Top): Device specific, stops processor, has to be quick 2. Software Interrupt (Bottom): Thread-like, heavy lifting when CPU has time Software Iinterrupt (Bottom ½) Heavy lifting
Stop processor (Top ½) Quick and get out
Schedule SW Interrupt: When processor has time do generic processing of packets. HW Interrupt: Run device specific handler
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
74
‘top’ view of interrupts
‘top’ – ‘1’ for CPUs
Hardware interrupt handles device drivers
Software interrupt handles
http://www.amazon.com/Linux-Kernel-Development-Robert-Love/dp/0672329468/ref=sr_sp-atf_title_1_1?s=books&ie=UTF8&qid=1389542475&sr=11&keywords=linux+kernel+development Page: 3322 © 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
75
Interrupts are not Process/Threads This is why you can’t see interrupts in a ‘ps’ or a ‘top’, different data structure than processes/threads. Very much like them but CANNOT SLEEP/BLOCK! Soft Interrupt Table array Softirq[0]
Softirq[1]
………
Softirq[31]
Process/Thread Table (used for ‘ps’ and ‘top’)
Hard Interrupt Table linked list
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
76
Device Drivers A network device driver runs within the kernel and has two primary functions: User Process
Kernel Driver Transmit Functions
Interrupt Handler
Device Driver
1) Organize data so NIC can grab it
2) DMA data to kernel memory and send interrupt when ready © 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
77
DMA vs CPU copy Note that a Direct Memory Access (DMA) processor is responsible for transferring data between the NIC and the host. This allows the CPU to parallel process other activities while the transfer occurs. There my be some bus contention between the CPU and DMA, but not as bad as if the CPU had to perform the transfer.
Kernel
User Process
DMA Interface
Initiate Data Transfer!!!
Host DMA chip
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
78
Packet Journey – User to Device Linux Kernel Send device interrupt when data is ready to be copied by DMA transfer
Net I/O Kernel Thread On Behalf Of User Copies into DMA space and massages data through TCP/IP stack
Call to device driver to put data into the right structures for the NIC and tell the NIC it can copy.
hard_start_xmit()
Linux User Space
send(socket, msg, strlen(msg), 0)
Application
eth0 TCP/IP Stack DMA Space
http://www.ece.rice.edu/~willmann/teng_nics_overview.html#overview http://www.amazon.com/Essential-Device-Drivers-Sreekrishnan-Venkateswaran/dp/0132396556 Page 505 http://www.linuxfoundation.org/collaborate/workgroups/networking/kernel_flow © 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
79
Packet Journey – Device To User Linux Kernel Hardware Interrupt (Top ½) (Stops processor)
Linux User Space Software Iinterrupt (Bottom ½) Massage packet through TCP/IP Processing
User process waits for data on socket
User thread Inside kernel context
int recv(int s, void *buf, size_t len, int flags);
DMA Space
eth0
Application
Signal to continue
User Space
TCP/IP Stack
IRQ 177
Soft IRQ copies data to user space
Device DMAs data to kernel DMA memory
Big job! Schedule software interrupt
http://www.ece.rice.edu/~willmann/teng_nics_overview.html#overview http://www.amazon.com/Essential-Device-Drivers-Sreekrishnan-Venkateswaran/dp/0132396556 Page 505 http://www.linuxfoundation.org/collaborate/workgroups/networking/kernel_flow © 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
80
Monitor # of Interrupts per Device /proc/interrupts keeps track of # of hw interrupts per interface since boot. Linux will use eth0 as default for network cards until system gets busy then it tries and re-balance between CPUs (see eth0)
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
81
Interrupt Affinity In Linux, you are able to wire an interrupt to a specific CPU. This is called Interrupt Affinity. Once again this allows data to be cached locally on a single (set) CPU for the interrupt handler. Linux will start with CPU 0 handling all interrupts. (?? Does Linux auto balance??) Interrupt affinity is used by Check Point CoreXL as we will see in the next section. https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide/s-cpu-irq.html
IRQ 32 CPU 1
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
82
Linux Network Summary
• Interrupts come in 2 halves • Top – Hardware device driver – Quick and get out • Bottom – Heavy lifting of data through kernel • Interrupts are NOT threads or processes. They have many things in common but you can’t view them and are internal to the kernel. • Device drivers come in 2 halves • Transmit – send data to device • Receive – Usually interrupt driven to pick up packets
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
83
Linux Evolution wrt SPLAT/GAIA Date
Linux Release
CP Release
2000
Linux 2.2
R 4.0
Nov 2001
Linux 2.4
NG FP1
Jan 2008
Linux 2.6
R65 HF02
Today
Linux 2.6-18cp
R77.10
http://en.wikipedia.org/wiki/Check_Point_VPN-1
Linux Tidbits: GPL (GNU Public License) – • No custom mods to Linux, they have to be shared with Linux community as source code. There is a version 2.6.18cp (not sure status) • Can compile code with GNU compiler and keep source private. CP uses internal kernel modules to do this.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
84
Balancing Check Point Systems
• • • • •
Overview Linux Review SecureXL CoreXL Balancing CoreXL Tips
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
85
Why SecureXL?? The classic problem has been at noon when everyone having lunch at their desk, everyone starts browsing the Internet and thus slowing down business traffic. In addition as more applications become web-oriented HTTP traffic is dominating the network. SecureXL was primarily created to address the web-browsing-at-noon-problem.
Symptom
Verify
Highly utilized system
‘top’, ps –o psr,command
High network traffic on 2 interfaces
/proc/interfaces
High HTTP traffic
SmartLog
Ifconfig is dropping packets, retransmits
Ifconfig, netstat -s
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
86
HTTP 1.0 HTTP 1.0 uses multiple concurrent requests to retrieve a multi-part web page for the user.
Text
Picture Text
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
87
HTTP 1.1 HTTP 1.1 uses a single request to retrieve a multi-part web page for the user.
Text
Picture Text
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
88
HTTP processing SecureXL is smart enough to know that all packets after packet #1 are all going to the same site and will hit the same rule. So SecureXL speeds up the processing of these packets by bypassing the full rule processing and just relying on state tables built by packet #1 to send the packets through the firewall.
Text
Picture Text
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
89
What does SecureXL accelerate? Secure XL can accelerate (bypass full rule processing) in two instances: 1) Subsequent connections to the same dest and port 2) On a single connection, packets 2 thru N.
Text
Picture Text
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
90
NGF Process Mapping SND does the acceleration and packet distribution. NGF has 1 SecureXL module to accelerate network packets. Firewall dispatcher sends packets to the right firewall instance (there can be many..next section) https://downloads.Check Point.com/fileserver/SOURCE/direct/ID/7513/FILE/CoreXL_Advanced_Configuration_Guide.pdf
User mode Kernel mode
fwk
Firewall Dispatcher(fwkdrvr)
SND
Performance Pack Packet Handler (SecureXL acceleration)
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
91
VSX Process Mapping In VSX, SND is still one module but SND is aware of individual VSs. SecureXL can be turned off/on per VS but SND shares info about all VSs when making acceleration decisions for the whole chassis. When Check Point moved the fw kernel from the Linux kernel to User Mode, they left only a little bit of code to work with the firewall dispatcher in place. Other than that it was a clean compile of the User mode kernel…This was not a massive rewrite
VS0
VS1
VS2
User mode Kernel mode Firewall Dispatcher (fwkdrvr)
SND
VS0 SecureXL
VS1 SecureXL
VS2 SecureXL
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
92
NGF Firewall Kernel Modules Here you can see the kernel modules for NGF vs VSX. Basically the same firewall module is used for both. But…probably…VSX only uses the bottom half processing because the firewall itself is in user space.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
93
SecureXL Acceleration There use to be a hardware acceleration device by Nokia, now its in software. CP recently put it back in hardware
SecureXL acceleration 1) 2)
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
Subsequent packets from a single connection Subsequent packets from the same source IP, same dest IP and same dest port (multiple HTTP requests to same dest) 94
Packet Journey Thru SecureXL Linux Kernel
Linux User Space
Hardware Interrupt (stops whole processor single thread)
Device driver hogs processor. Can’t be interrupted. Just transfer data.
Software Interrupt
Core 0 IRQ 177
Eth0 needs service
FW Instance 1
Core 0
Signal ‘Continue’
Can’t accelerate this, send on to a specific FW Instance
Big job! Schedule software interrupt
Standard Linux Processing
Core 1
FW Instance 2
Core 2
SND picks instance 1 to process packet
Core XL © 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Core 3
FW Instance 3
SND Interface eth0
Device Driver
eth0
Concurrent processing of SI’s is possible unlike HW interrupts. Can be interrupted.
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
95
Firewall Chain – SecureXL Modules Once packets make it through SecureXL, these are the modules in the firewall chain that build the SecureXL connection tables and sync with the SecureXL module itself.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
96
SecureXL and IPS/AV/Bot Integration When IPS/AV/Bot is enabled, not all traffic can be accelerated because it has to be inspected by the IPS engine.
PSL is the Packet Streaming Library for re-assembly of IP packets so the IPS/AV/Bot can look for signatures. SecureXL can forward packets directly to the PSL and bypass firewall processing. This is called ‘Medium Path’ because it bypasses rule checking on the 2nd+ packets
Fwaccel stats – SecureXL statistics for Medium Path
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
97
VSX Detailed Internals VS2
Packet stays in pool Should I Accelerate?
VS1 VS0
Packet Data Pool
HW Interrupt
Firewall in user mode inspects and creates pass/fail
Construct The Packet Packet Meta-Data Queue
SND
Pointer to packet is what moves
fwk
Outbound to another VS
Fwkdrv out F2F
Accelerate
Fwkdrv in F2F
Write pass/drop to msg q
Outbound to another VS
Dispatches to right VS
IP Stack Inbound
Message Queue Fwdrv deq
SND Outbound Device Driver
Implements action from fwk to pass/drop
Legend Kernel Code User Space Code Shared Memory
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
98
SecureXL Demo - VSX
• • • • •
Pass 2 gig file thru a firewall R75.40VS VSX Traffic going through fw-vsx1 a virtual firewall All run inside Virtual Box No other traffic
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
99
VSX – SecureXL On SecureXL runs as a software interrupt within the Linux kernel. You can see the %si get higher when it is busy.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
100
VSX – SecureXL off When SecureXL is OFF, you can see the fwk1_dev thread handle the work.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
101
AutoBalance Interfaces Note how the interrupt handling got shifted from CP0 to CP1. More on this later
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
102
SecureXL Demo - NGF
• • • •
Pass 2 gig file thru a firewall R75.40VS NGF Standalone gateway All run inside Virtual Box No other traffic
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
103
SecureXL OFF – FW busy Worker threads are idle, SND and FW sharing the work Busy kernel threads
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
104
SecureXL OFF – FW sees packets Firewall sees all packets in fw monitor
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
105
SecureXL ON– FW idle Worker threads are idle, SND is doing all the work Work done in SND
Work done in SND
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
106
SecureXL ON– FW no packets The first packet may/not be seen if its in the state table already. But no others
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
107
4 types of acceleration 1. Standard state table We talk about these 2. Accept connection templates (e.g. like HTTP) 3. NAT templates – Perform NAT in SecureXL and not in the firewall kernel (sk71200) [Expert@HostName]# echo 'cphwd_nat_templates_support=1' >> $FWDIR/boot/modules/fwkern.conf [Expert@HostName]# echo 'cphwd_nat_templates_enabled=1' >> $FWDIR/boot/modules/fwkern.conf [Expert@HostName]# cat $FWDIR/boot/modules/fwkern.conf [Expert@HostName]# reboot
4. Drop templates (sk66402)(R76- HF/R76+Standard) – Drop packets in SecureXL, negation of rules. 5.
Other drops (have to test) SAM, DDOS, SmartEvent/IPS triggered. CONFLICT: sk33781, sk66402, sk98348 all conflict with one another on this topic. Have to test. Sk98348 says drop templates are the negation of the security policy and drops are accelerated if it does not match a rule.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
108
Standard State Table Standard state table tuples will be accelerated. Here you can see a state table entry. Subsequent packets will get accelerated.
fwaccel conns
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
109
Accept Connection Temples Accept connection templates that will be accelerated. Here you can see where subsequent connections from the same source IP will be accelerated. We created multiple SSH Sessions through the firewall same Client->Server
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
Our demo had 1 accept template
110
fwaccel stat This command provides the status of SecureXL
State table Accept connections DOS drops NAT in SecureXL
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
111
fwaccel stats This commands provides you statistics on which packets are accelerated and which are forwarded To Firewall(F2F) for rule processing.
Total acceleration “C” Current Counts # from accept templates NAT performed by SecureXL PXL: PSL + SecureXL IPS packets
Connections sent to firewall, NOT XL/SLOW PATH
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
112
Debug Tips • If you have any random issues, immediately turn off SecureXL to determine if there is a difference • Using ‘top’ to monitor performance, turn SecureXL on/off and see what %SI is doing • Might have to distribute SecureXL across multiple cores if %SI is busy and doesn’t autobalance. See next section. • Monitor stats to make sure both state table and connection templates are being used • Move HTTP 1.0 type protocols to the top of the rulebase so they get hit • Avoid protocols that disable connection template acceleration (more on this at end)
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
113
SecureXL Licensing • ADN – Advanced Data Networking & Clustering (formerly ACCL) • SecureXL • Dynamic Routing • CoreXL???????? • ClusterXL • QoS,load balancing, ISP redundancy • Looks like CPSB-ADNC • Platform • Appliances – All inclusive • Open Platform • Ala Carte List - $1500
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
114
SecureXL Summary
• • • •
Definitely required for high usage gateways Easy to administrate (on/off) Understand the difference between state table and connection templates All takes place in the %SI under ‘top’
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
115
Balancing Check Point Systems
• • • • •
Overview Linux Review SecureXL CoreXL Balancing CoreXL Tips
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
116
Why CoreXL CoreXL allows you to utilize multiple processors on a single chassis for concurrent processing of firewall requests in order to expand capacity and reduce latency on your existing platform. Cheaper to expand capacity on a single bigger chassis than to cluster multiple smaller chassis (ClusterXL)
• Cost efficient • Easier to manage • Easier to debug
• ClusterXL: More expensive to purchase and maintain
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
117
CoreXL Manages 4 Affinity Types 3) CP Process Affinity
2) Fw kernel Instance Affinity
VS1
VS2
FWD logging
syslogd
1) Interface Affinity
SND
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
4) Linux Process
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
Secure Network Distributor
118
CoreXL
• • • •
Interface Affinity Instance Affinity Process Affinity Linux Process Affinity
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
119
The LIE • Interface affinity is grey zone. Could be included in both SecureXL and CoreXL • Interface affinity can be used with SecureXL license and no CORExl license • Interface affinity can also be used without CoreXL or SecureXL license, it is a Linux function • Here - Interface affinity is grouped with CoreXL for completeness and topic flow
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
120
CoreXL Interface Affinity Interface ‘Affinity’ is the mapping of processors to interfaces to handle network packets. Default is ALL, which in reality is Core 0. SND is responsible for managing interfaces assigned to that core. If there are multiple CPUs handling different interfaces, then each CPU has a different SND.
1) Interface Affinity
SND
Secure Network Distributor • Process network traffic • Accelerate • Distribute to firewall instances
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
121
MultiQueue (sk80940) MQ enables assigning multiple interrupts per interface. Certain interface cards have multiple TxRx queues per interface. Src/Dst flows are tied to a queue. Then queues are assigned IRQs and tied to specific processors. This technique optimizes CPU cache utilization.
https://greenhost.nl/2013/04/10/multiqueue-network-interfaces-with-smp-on-linux/
Standard on R76 and R77 Previous 71.50
IRQ 1
IRQ 1
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
IRQ 2
IRQ 3
122
MultiQ Restrictions • • • •
R77.10 Only on appliances…needs the right hardware and drivers Supports increased throughput, not so much increased number of sessions Based on src/dst assigned to a CPU. So a single high throughput src/dst will only use 1 CPU and not take advantage of multiple CPUs.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
123
Display Interface Affinities
In this complex environment, SND can concurrently run on cores 0-7 © 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
124
Monitor Hits on SND /proc/interrupts keeps track of # of interrupts per interface since boot. We can use it to monitor invokations of SND In this simple environment SND has ability to concurrently run on any core (that isn’t running a fw instance), but by Linux default it chooses to run on CPU 0. (Probably not good because all interfaces and processes will use CPU 0). So if you have interfaces that are dropping packets, you might want to check this if CPU 0 is busy. POINT: Even though the configuration seems balanced, you need to verify!
SND on core 0 is doing ALL the work for all interfaces
SND has never autobalanced, all on Core 0 © 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
126
IRQ to CPU mapping Linux maps IRQs to CPUs in the proc/irq directory in the smp_affinity file These are used to program the APIC (interrupt controller chip). By default a IRQ can run on any processor but
Linux chooses Core 0 Why core 0? Well the kernel doesn’t want to have the interrupt context float between CPUs. This way it keeps interface interrupt data in local cache. R77.10 changed from ‘all’ to CPU 0. © 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
127
Setting affinity command
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
128
Setting interface affinity sim affinity –s sets the interface affinity by changing the values in the smp_affinity file. This in turns programs the APIC to send interrupts to a different processor.
NOTE: Interface settings will survive reboot (BUT not CoreXL settings (next)).
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
129
Impact Immediate No reboot, sim affinity –s takes effect immediately
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
130
Distribute SND processing This is how you distribute SND processing on interfaces that are overloaded and dropping packets. Give those interfaces/SND their own processor that does not have other FW components on it
VS1 VS2
SND
SND
Busy
SND
Busy
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
Secure Network Distributor
131
SND Chooses non-fwk CPUs When an interface is set to All, it will attempt to use a CPU that is NOT being used by a firewall instance. But it will try to use a CPU that is being used by another interface…in order to keep the the local CPU cache fresh. So when under low CPU usage, most interfaces will default to ALL (below) and be autobalanced as CPU and interface activity picks up. The default for ALL is CPU 0…until CPU activity picks up R77, interfaces will be set to the default CPU0
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
132
Interface Auto - Balance Look what happens when straining a CPU with a 3 gig SCP transfer between members directly connected. Every 60 seconds, CoreXL examines the CPUs to see if they are busy. If they are busy it will rebalance interfaces to non-busy CPUs. (fwkernel and Linux processes rebalance every 1-2 seconds). (NOTE: I do NOT know how to set back to autobalance once you hard set the interfaces except by reboot on NGF, or factor defaults on VSX) Below the fw kernel saw the CPU going to 80 si% and rebalanced the interfaces from ‘all’ to ‘eth1:1’, gave eth1 its own CPU
Before After
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
133
Busy box? How to tell if a box is ‘busy’??? Here is an example of an Internet gateway with 10gig interfaces. This is the /proc/interrupts table. You can see that only CPU 0 has been used to process network packets. This tells us that the CPUs have not become busy since reboot and the fw kernel has not done any rebalancing. If the ifconfig shows packet drops, then you have different issues than CPU not being able to handle the load.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
134
Set Interface Affinity – No Performance Pack
$FWDIR/conf
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
135
CoreXL
• • • •
Interface Affinity Instance Affinity Process Affinity Linux Process Affinity
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
136
CoreXL Manages 4 Affinity Types 2) Firewall Instances
VS1
VS2
FWD logging
syslogd
SND
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
Secure Network Distributor
137
HyperThread Review Enabling Symmetric Multi-Threading (SMT) or Hyper Threading (HT) doubles the number of logical processors. • • • • •
Note the difference between HT, Dual Core and Dual Processor. Note where the caches are! Remember this when you assign interfaces so you keep the cache hot Works just like Linux thread processing where a process (web server) has 2 threads (2 clients requesting pages) and the kernel can preemptively multi-task the two threads so it seems like they are parallel processing. Without HT. Each Linux thread gets a time-slice by the Linux kernel but only 1 thread runs at a time. With HT: At the hardware level there is a mini-Linux like kernel that can multiplex/task 2 threads concurrently. So the two threads could conceptually start and finish within 1 kernel time slice instead of 2 separate time slices. Performance improvement 30%???
Without Hyperthreading, Managed by Kernel Thread 1
Thread 2
Thread 1
Thread 2
WithHyperthreading, Managed by HW
http://blogs.msdn.com/b/gauravseth/archive/2006/03/20/555519.aspx
TIME © 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
138
HT and Cache Sharing Remember that the kernel instances share a state tables. So when allocating instances keep similar data flows on the same cache so that portion of the connection table is always in cache.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
139
R77.10 Uses SMT/HT Implemented in R77+ Restrictions:
• Only enhances performance of IPS/AB/AV CPU intensive functions and NOT I/O operations. Too many interrupts may actually slow it down. • Supported only on R77+ GAIA • Only on Check Point Appliances • Has to be enabled in the BIOS • Does not work with large number of HIDE NAT connections. Each CPU has pre-allocated # of HIDE NAT slots. If one CPU uses all its HIDE NAT slots then it can’t handle new HIDE NAT connections.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
140
Assigning Instances to CPUs
• NGF Firewall Affinity • VSX Firewall Affinity
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
141
NGF Standalone Instance Affinity In NGF there is only 1 firewall. With CoreXL, the kernel will replicate itself X times, depending on how many firewall instances you setup. Each instance will parallel process network traffic with the SAME shared rulebase and state tables. Each instance has an ‘affinity’ for a specific processor.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
142
NGF CoreXL
Affinity
In a NGF gateway, CoreXL generates X copies of the kernel into individual Linux kernel threads
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
143
NGF Process/Thread Mapping Before we assign affinity, we should look at what types of processes/threads we are dealing with. When multiple processes have the same PID, that means they are threads sharing the VM of the parent (CPD and FWD below). Here you can see the firewalls are individual KERNEL threads inside the kernel with a parent of PID 1 – ‘init’. Kernel threads usually come in thread groups of size 1, unlike user space.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
144
NGF Process Spawn Tree Here you can see how the firewall processes were spawned in what order
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
145
Assigning Instances to CPUs
• NGF Firewall Affinity • VSX Firewall Affinity
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
147
VSX CoreXL Affinity Where NGF has 1 firewall on a chassis, VSX has multiple firewalls running on a single physical chassis. Each firewall is represented by a Virtual System (VS).
VSX VS’s are a different than NGF. Each VS has a 1 firewall instance that is executed by 1 corresponding Linux OS User mode process (not totally true, but not lying and still making the point).
VS
VS
VS
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
VS
148
R75.40 VSX Process Architecture VS instances are implemented with Linux threads (user mode but mapped onto kernel threads so they can be scheduled by the kernel)
• • • • •
Virtual Memory Space CPU Instructions Processor state Kernel threads Security attributes
=
fw kernel instance
VS instance VS instance
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
VS instance
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
149
VS instances The terminology gets a little confusing because CP is not explicit nor consistent in defining what an instance is for VSX. A firewall kernel instance which runs on behalf of a VS, can also be further subdivided into VS instances. These VS instances can be assigned to individual processors.
VS
fw kernel instance
VS instance
VS instance VS instance
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
150
VSX VS0 CoreXL Affinity Here you can see we are configured 3 VS instances within VS0 (very important!!). I have 3 VS instances running. So you can see that VSi0-VSi2 are allowed to run on any of the 1-3 CPUs. CPU 0 is reserved for eth0 traffic (hold on).
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
151
VS0 Processes and Threads So VS0 has 4 VS instances (VSi) generated for it that are watched over by the watcher daemon.
Processes
Threads
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
152
VS0 CoreXL Config You can only configure CoreXL VS0 from the command line….
Trying to configure more VS instances for VS1 from the command line and look….
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
153
Change non-VS0 CoreXL OK. We’ll listen to directions…
1= CoreXL OFF
2+ = CoreXL ON
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
154
Look Ma Babies!!! • • •
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
VS1 now has 2 instances. CPD has 1 main and 2 instances FWD has 4 daemons (not sure why)
155
Linux view of VSX instances VSX uses USER threads for all components
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
156
VSX Process Spawn Tree Here you can see the parent-child relationship of how firewall instances are spawned. Note: the process names change, like fw could be fwd – Linux command line issue.
pstree -p
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
157
Assigning Instances to CPUs
• NGF Firewall Affinity Assignment • VSX Firewall Affinity Assignment • Monitor Affinity
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
159
Default Affinity Assignment The default affinity for firewall instances is to assign CPUs from high to low. There can never be more instances than CPUs so that mapping is 1:1. The default affinity assignments for interfaces is to CPUs that are NOT running firewall instances…if possible. Note that the default assignment is probably adequate for 90% of the cases….unless you have a lot of busy network interfaces, help processes, Linux processes that interfere with the firewall instances. For example, if eth0 ran 100% of CPU0, then you might want to move the firewall instance. From sk98348
# of Cores
# of FW Instances
# of SNDs
1
1
0 (Corexl disabled)
2
2
2
4
3
1
6-20
# cores -2
2
21-30
# cores -4
4
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
Reserved for SND
160
NGF: Assign Affinity When you re-assign an instance to a CPU, you are telling the instance to only use THAT CPU…when the CPU is free. So double-edge sword: 1. GOOD: Guarantee the cache will be always hot on that CPU 2. BAD: What if that CPU is busy with other assigned processes…Has to wait till end of the other process timeslice to get CPU time. You could up its priority. So make sure you choose a CPU that is NOT assigned to any other process if possible.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
162
Assign ALL CPUs You can set affinity to ALL CPUs by assigning all CPUs to the instance. You can obviously also set the affinity to a subset of CPUs. Note: that you forfeit hot caching
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
163
Affinity Set via Linux Note that instead of fw ctl affinity command, you could just use the regular Linux affinity command ‘taskset’. Does the same thing.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
164
Permanent Affinity Custom firewall instance affinity are not permanent (but INTERFACE affinity is). On reboot you have to re-assign affinity. You can make firewall instance affinity permanent configuring it in $FWDIR/conf/fwaffinity.conf. NOTE: auto vs. all
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
165
Stickiness State table tuples are ‘sticky’ to a single core. Once you start a network conversation through a specific core, the associated tuple will always use the same CPU to process that network conversation. So a backup process between backup server A and client B for example could monopolize a single CPU. <1.1.1.1, 1111, 2.2.2.2, 2222, TCP>
2.2.2.2
1.1.1.1 CORE 0
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
166
Assigning Instances to CPUs
• NGF Firewall Affinity Assignment • VSX Firewall Affinity Assignment • Monitor Affinity
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
167
VSX Default Affinity The default config for VSX is firewall instances are assigned to all but CPU 0 (reserved for interfaces). Default varies depending on components, but it won’t be “ALL”.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
168
VS Affinity Command Hierarchy In VSX there is a hierarchy how VS affinity commands work. You can apply the instance command to the ‘V’ entire VS, ‘P’, only the firewall processes or ‘I’ a specific VS instance. The ‘I’ then inherits from the ‘P’ which inherits from the ‘VS’.
‘VS’ for Virtual System
‘P’ for Firewall Processes ‘I’ for instance
cpd
fwk VS instance
fwd
vpnd
VS0 © 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
169
Impact of Setting Affinity The SRC column shows at what level the affinity command for the process was issued. ‘V’ means the command was issued to ALL the components of a VS. ‘P’ means only to the firewall instances. ‘I’ means a single fw instance.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
170
VSX set affinity command Here is the command for setting affinity for the 3 levels of processes. Note that if you do not set a affinity for a level, the level will inherit the affinity from the previous level.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
171
VS affinity config files After you set affinity you can see the impact in these configuration files. Because of these files, VSX affinity is permanent…unlike NGF!!!
Here are the affinity configurations that the VSs use to set their affinities. As you set affinities at the different levels, these files will begin to appear. This is how a VS instance knows what affinity to use. If there is no config file at the I instance level, then it goes up to the P Process level config file, etc.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
172
Set VS affinity Here we set the VS affinity. Note that it sets the affinity for ALL processes in VS0. Firewalls, Firewall Helpers, Linux processes. Note how the SRC column is “V” for the VS affinity configuration file
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
173
Set P Process Affinity This will set the P Process affinity for JUST the fwk processes in a VS. NOTE: the ‘P’ means to set affinity to what is found in the ‘P’ config file
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
174
Set P Process Affinity This will set the P Process affinity is for JUST the fwk processes and VS instances. NOTE: -fwkall will set the fwk VS instances for ALL the VS’s.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
175
Set ‘I’ VS Affinity Here you can see we set he ‘I” VS Instance affinity The “I” in the SRC column means the affinity config comes from the ‘I’ config file
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
176
Now let’s set everything to ALL As with NGF, if you set affinity with all CPUs, it will be set to ‘ALL’. Note it only sets the affinity for VS0, and not the other VSs
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
177
VSX: CoreXL set per VS CoreXL can be enabled/disabled per VS instance, just like SecureXL VS0 use cpconfig, VS1+ use SmartDashboard to set CoreXL to 1
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
178
Reset to Defaults REBOOT!!!!!
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
179
Changing Cluster CoreXL config At boot - Cluster membership is tested to ensure same # of CPUs in CoreXL. When changing CoreXL cpconfig CPU configuration in a cluster: 1. 2. 3. 4. 5. 6. 7.
Start on standby member B, bring it down Change number of processors - cpconfig Reboot Member B Comes up in the READY state Fail over Active member A to the Ready member B (Stateless) Modify formerly Active member A Reboot member A
SETTING AFFINITY (not CoreXL Firewall Instance count) must be done manually in both members and does not impact cluster status
Member A - Active
Member B - Ready
https://sc1.Check Point.com/documents/R76/CP_R76_ClusterXL_AdminGuide/7298.htm © 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
180
Cluster XL Hints 1. Keep SecureXL and CoreXL configuration the same on both members 2. On upgrades, config will carry over but not necessarily on from scratch migrations. ( e.g. process assignment)
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
181
Summary: NGF to VSX CoreXL NGF can delegate its firewall processing into multiple fw instances. Each fw instance is represented by a Linux kernel thread and can be assigned a core affinity.
Linux Process /Firewall Instance
VSX has each VS represented by a Linux process with 1 or more internal VS instances mapped to Linux threads. The whole Linux process OR the VS instance (thread) can be mapped to a processor
Linux Process /Firewall Instance Linux Process /Firewall Instance
Firewall Instance
Firewall Instance
Firewall Instance
Linux Process /Firewall Instance
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
182
Assigning Instances to CPUs
• NGF Firewall Affinity Assignment • VSX Firewall Affinity Assignment • Monitor Affinity
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
183
‘top’ Tricks for CoreXL F
SORT j u
f
Processor Number of page faults
Add columns j u
Processor Number of page faults
1
Show all processors
i,z
Just show running processes
c
Command vs process name
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
184
Important TOP Stats with ‘f’ command
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
185
Unix ‘top’ Show all processors and see what process is running on what processor
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
186
Monitor Running Processes With ‘f’ ->‘i’ and ‘j’ and ‘u’ you can monitor running processes that might be faulting or jumping processors
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
187
CoreXL ‘ps’ arguments
-e
All processes
-L
All threads
-o
Specific columns (below) pid
Pid of process
psr
Processor id
cmd,comm,com mand,args
Command/Process names in long-short formats. with (args) arguments
lwp
Thread ID
majflt
Page faults
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
188
Unix ‘ps’ to monitor processor ‘watch –d –n1’ will execute a command every 1 second and highlight changes ‘ps’ can be used to monitor processor usage by a specific process
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
189
‘watch’ Interrupt table
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
191
CoreXL
• • • •
Interface Affinity Instance Affinity Process Affinity Linux Process Affinity
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
192
CoreXL Manages 4 Affinity Types 3) CP Process Affinity
VS1
VS2
FWD logging
syslogd
SND
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
4) Linux Process
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
Secure Network Distributor
193
List firewall helper daemons Firewall helper daemons (cpd, fwd, vpnd) are running in the VS0-VSX VS’s. Linux processes are running under the control of VS0. Many times you will see they are able to run on ALL processors
Linux only processes
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
194
Set Process Affinity Let’s say your fwd logging process is monopolizing the system with heavy logging. You an assign a whole processor to it to handle the load. NOTE: /opt/CPsuite-R75.40VS/fw1/conf/vsaffinity_exception.conf are a list of LINUX processes that are not impacted by the affinity command. You have to edit this list to modify LINUX process affinity
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
196
Process to CPU Busy processes such as VPND or FWD (logging) can be dedicated to a single CPU because they might dominate a core or swap cores frequently thus losing their cache freshness. Note in VSX it is specific to your VS context
FWD Logging/HA
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
197
CoreXL Licensing • Appliances include all the cores • Open Servers • You can select how many cores to buy • CPSG-PCC • CC = # of cores • CPSG-PCCBB • BB = # of blades • CPSG-P1207 • Security Gateway • 12 core • 7 blades • NOTE: You only have to license the cores that FWK run on. You can buy a 2 core license and have a 12 core gateway. ..10 of the cores will run Linux, SND, CP helper programs. 2 cores run FWK • Example: Ala carte 2 core to 4 core upgrade $7500 list
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
198
Balancing Check Point Systems
• • • • •
Overview Linux Review SecureXL CoreXL Balancing CoreXL Tips
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
199
Tuning Tips • • • •
Traffic to Rules to SecureXL to CoreXL tuning VSX: Keep traffic flows on same system VLAN resources CPU or Memory
•
IPS Integration???
• Suggested CoreXL Configurations • Command summary
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
200
Classic Target 2 interface Internet firewall, busy at noon with HTTP traffic interrupting business traffic
Symptom
Verify
Highly utilized system
‘top’, ps –o psr,command
High network traffic on 2 interfaces
/proc/interfaces
High HTTP traffic
SmartLog
Ifconfig is dropping packets, retransmits
Ifconfig, netstat -s
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
201
Determine Your Traffic Patterns eth0 to eth3 Use SmartLog to determine your major traffic flows between interfaces. Also look for busy tuples, remember they are sticky to processor!!
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
202
SecureXL Restrictions The following traffic is not throughput (state tables), nor connection-rate (accept templates)accelerated by SecureXL: Traffic types other than TCP, UDP, PIM, GRE, ESP First packets of any new TCP session, unless a "template" exists First packet in a UDP session Traffic matching certain Firewall rules: rules with a service that uses a resource rules for dropping or rejecting traffic rules where the source or destination is the gateway itself rules with a Security Server rules with user authentication rules with session authentication The following traffic is not connection-rate (accept templates) accelerated by SecureXL and will stop building templates in the rulebase if they are found: Non-TCP/UDP connections such as PIM, GRE, ESP ---- ICMP Protocols that are not connection intensive such as SMTP, FTP, RPC, NFS, NNTP, NTP Complex connections such as IPSec VPN, FTP, H.323, etc. Traffic in environment using NAT (for security, NAT addresses can change and can be shared) © 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
203
Check SecureXL and Rulebase ICMP prevents SecureXL Accept Connection templates from accelerating HTTP 1.0/type connections. Need to move to end of rulebase.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
204
HTTPS and CoreXL tuning Use /proc/interrupts and ifconfig to see if any of your interfaces were struggling to keep up. If the interfaces are constantly rebalancing or dropping packets, then you might have a problem.
All interfaces on eth0!
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
205
Processor Utilization Look to see if CPU 0 is busy. Busy FWD
Busy core 0
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
206
Balance CoreXL So investigation shows Core 0 is busy. This means you should dedicate a ‘free’ processor to eth0 because eth0 is doing a lot of work with regular processes AND interface handling
Give eth0 and eth3 their own core
Then give the offending process its own processor.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
207
Drop Rule If SmartTracker or SmartEvent shows a Denial of Service or heavy traffic from a malicious source, consider using drop templates or Suspicous Activity Monitor instead of the rulebase to drop the traffic. The drop templates are enforced by SecureXL and done before you have to go through the rulebase. https://supportcenter.Check Point.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk67861
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
208
Tuning Tips • • • • • •
Traffic to Rules to SecureXL to CoreXL tuning VSX: Keep traffic flows on same system VLAN resources CPU or Memory? Suggested CoreXL Configurations Command summary
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
209
VSX Archtecture With VSX which is the most efficient way to route packets between two networks and why?
Router
VSX Chassis
Router
VSX Chassis
VRouter
VSX Chassis
11
VSX Chassis
VRouter
1
VRouter
1
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
210
VSX Architecture Design Balance VSLS VSs based on traffic flow w/ SecureXL caching If you have lots of traffic between two VSs, then keep them on the same gateway because SecureXL /CoreXL will cache states and accelerate locally
Traffic states cached locally and SecureXL will not send it through user mode kernel. CoreXL will cache states in processor CPU dedicated to those interfaces
VSX Chassis
SND
VSX Chassis
SND
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
Tuning Tips • • • • • •
Traffic to Rules to SecureXL to CoreXL tuning VSX: Keep traffic flows on same system VLAN resources CPU or Memory? Suggested CoreXL Configurations Command summary
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
212
Bonded(VLAN) Interfaces
Core 0
VLAN
VLAN
VLAN
VLAN
When you have a bonded VLAN that trunks multiple networks over a single logical interface, (assuming it will be heavily utilized) considering assigning a dedicated CPU to each interface to handle the load and keep the caches ‘fresh’.
Core 1
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
Tuning Tips • • • • • •
Traffic to Rules to SecureXL to CoreXL tuning VSX: Keep traffic flows on same system VLAN resources CPU or Memory Suggested CoreXL Configurations Command summary
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
214
CPU or Memory How do you know if you need more CPU or more memory or both?
?
?
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
215
‘top’ is WRONG Linux 2.6.18 ‘top’ is just WRONG. It does not compute SWAP space correctly because it does not take into account (e.g.) that the processes is sharing 2GB of space from a shared library along with 10 other processes. So each of the 10 processes thinks it alone is using the 2GB of shared space. Below you can see that nautilus VIRT is 433MB, and SWAP is 427Mb, but so are most the other processes. It just doesn’t add up to the ‘used’ SWAP. So its hard to tell what processes are using what memory and what is using SWAP space.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
216
#1 Mo Memory Because you can’t trust ‘top’ too much, a better measure is to look at the major page faults for running processes. If they are faulting a lot, you have a problemo. Using ‘top’ watch the running processes by typing ‘i’. This will only list the running processes. Then use ‘f’ and ‘u’ for listing page faults. Below is a real live MLM that is faulting heavily on one of the logging daemons. You can have a fast CPU and add more CPUs, but fwd will spend most of its time swapping pages.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
217
#2 More CPU If you have a lot of active processes that are NOT faulting, then you have 3 options: 1. More CPUs 2. Faster CPUs 3. Set affinity on the most busiest processes so they are using their hot caches and not spending time flushing caches 4. Set the affinity of the busiest processes to -15 so that they get a bigger chunk of the timeslice
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
218
#3 Memory Leak 1. If ‘Swap used” is > 0 and growing, you may have a memory leak. I 2. If “VIRT” is growing over time on a process, you may have a memory leak.
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
219
Tuning Tips • • • • • •
Traffic to Rules to SecureXL to CoreXL tuning VSX: Keep traffic flows on same system VLAN resources CPU or Memory Suggested CoreXL Configurations Command summary
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
220
Classic Target - NGF 2 interface Internet firewall, busy at noon with HTTP traffic interrupting business trafficAssumes that SecureXL is working on ALL rules and SND is doing most the work and not the FWK. NO IPS/AV/ThreatPrevention,etc. Goal is to allocate evenly. Obviously this is only a guess, but you’d have to evaluate your system with the following commands to have more accurate measurement Symptom
Verify
Highly utilized system
‘top’, ps –o psr,command
High network traffic on 2 interfaces
/proc/interfaces
High HTTP traffic
SmartLog
Ifconfig is dropping packets, retransmits
Ifconfig, netstat -s
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
221
Balancing Goals This is the take-away slide, the value behind the whole course. Allocate CPUs in the following priority order: 1. 2. 3. 4. 5.
Share cache for common data Allocate CPUs to busy Internal interfaces Allocate CPUs to slower less busy External interfaces Allocate CPUs to FW instances Remaining threads are usually idle so distribute evenly and let kernel find idle processor
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
222
Reminder: Share Cache!!
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
223
Suggested CoreXL Distributions-NGF Type
CPU/Item
1
2
4
8 (2 core)
12 (2 core)
FW Processes
cpd
0
All
All
3
3
fwd
0
All
All
3
4
Linux
Linux processes
All
All
All
All
All
Interfaces (SND)
External Interface
0
0
0
0 (cache 0)
0 (cache 0)
Internal Interface
0
1
1
1 (cache 0)
1 (cache 0)
Sync
0
All
All
2
2
fw_0
0
0
2
4(cache 1)
6(cache 1)
1
3
5(cache 1)
7(cache 1)
fw_2
6(cache 1)
8(cache 1)
fw_3
7(cache 1)
9(cache 1)
Fw kernel instances
fw_1
fw_4
10(cache 1)
fw_5
11 (cache 1)
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
224
Classic Target - VSX VSX with multiple business silos to Internet. Assumes that SecureXL is working on ALL rules and SND is doing most the work and not the FWK. NO IPS/AV,etc NOTE: fw0 is not handling any traffic aside from sync. Only mgt traffic Goal is to allocate evenly. Obviously this is only a guess, but you’d have to evaluate your system with the following commands to have more accurate measurement
Symptom
Verify
Highly utilized system
‘top’, ps –o psr,command, vsx resctrl, vsx memstat
High network traffic on 2 interfaces
/proc/interfaces
High HTTP traffic
SmartLog
Ifconfig is dropping packets, retransmits
Ifconfig, netstat -s
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
225
Suggested CoreXL Distributions-VSX Type
CPU/Item
1
2
4
8 (dual core)
12(dual core)
Interfaces (SND)
Ext VS0
0
All
0
4
6
Ext VS1
0
0
1
1(cache0)
0 (cache0)
Ext VS2
0
1
2
1(cache0)
1 (cache0)
Ext VS3
0
1
3
3(cache0)
2 (cache0)
Int VS0
0
All
0
4
6
Int VS1
0
0
1
0(cache0)
3 (cache0)
Int VS2
0
1
2
1(cache0)
4 (cache0)
Int VS3
0
1
3
2(cache0)
5 (cache0)
Sync
0
All
0
4
7
fw_0
0
All
0
4
8
fw_1 (VS1)
0
All
1
5
9
fw_2 (VS2)
0
All
2
6
10
fw_3 (VS3)
0
All
3
7
11
Fw kernel instances
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
226
Suggested CoreXL Distributions-VSX
Type
Linux
CPU/Item
1
2
4
8 (dual core)
12(dual core)
cpd(VS1)
All
All
All but 0-3
6,7,8
cpd(VS2)
All
All
All but 0-3
6,7,8
cpd(VS3)
All
All
All but 0-3
6,7,8
fwd(VS0)
All
All
All but 0-3
6,7,8
fwd(VS1)
All
All
All but 0-3
6,7,8
fwd(VS2)
All
All
All but 0-3
6,7,8
fwd(VS3)
All
All
All but 0-3
6,7,8
Linux processes
All
All
All but 0-3
All
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
227
Tuning Tips • • • • •
Traffic to Rules to SecureXL to CoreXL tuning VSX: Keep traffic flows on same system VLAN resources Suggested CoreXL Configurations Command summary
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
228
SecureXL Cheat Sheet sim affinity –l
List interface affinities
NGF: fw ctl affinity –l –v -a
List all affinities including interfaces
VSX: fw ctl affinity –l -x
List all affinities including interfaces
sim affinity -s
Set interface affinities
fwaccel stat
Review SecureXL status
fwaccel stats
Review SecureXL stats
fwaccel conns
Review SecureXL state table
fwaccel templates
Review SecureXL accept connection templates
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
229
CoreXL Commands List interface affinities NGF: fw ctl affinity –l –v –a fw ctl multik stat
List all affinities including interfaces
VSX: fw ctl affinity –l –x fw ctl affinity –l -x –flags tkn fw ctl multik stat
List all affinities including interfaces. -flags tkn lists internal threads
cat /proc/interrupts
List IRQ table and interrupts handled by CPU
watch –d –n1 “ps –ef | fgrep syslogd” top -------- F->j fw ctl affinity –s –d –fwkall 3
Watch how a process changes CPU
Set all process affinities to ALL (auto)
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
230
Thanks!!!!
© 2013 Midpoint Technology, Inc. 952-837-6206 –
[email protected]
1/24/2015
Proprietary and Confidential: No part of this document may be reproduced without permission from Midpoint Technology, Inc
231