Unit No: 1 Foundation of UNIX Operating Systems Pavan R Jaiswal
Objectives ◦ ◦ ◦ ◦
To To To To
learn basics of operating system understand kernel & its types thoroughly understand GRUB-I, GRUB-II & their significance expose in-depth internals of UNIX file system
Outcomes ◦ Ability to Install Linux OS (Ubuntu/Fedora) successfully ◦ Ability to change legacy GRUB to GRUB-II ◦ Ability to configure Linux kernel ◦ Ability to manage file system effectively & efficiently Foundation of UNIX OS
2
Basics of operating systems Kernel & its types OS booting process, GRUB-I, GRUB-II Buffer cache Internal representation of files, file management, file concept System calls Free space management, disk management Swap space
Foundation of UNIX OS
3
Computer = set of resources ◦ Processor(s), memory, I/O & communication devices
OS ◦ Enables use of resources ◦ Manages resources
Resources not limited to hardware Shift from ◦ Pure efficient use of resources ◦ Enhance user experience
Foundation of UNIX OS
to
4
Fig 1 Standard OS model Foundation of UNIX OS
5
Kernel ◦ Substance, core, essence, centre, heart, heart & soul, marrow, etc.
Pieces of software that performs OS tasks
Has privileged access to resources
Foundation of UNIX OS
6
Kernel mode or kernel space
User mode or user space
System call ◦ User mode functionality
program
invokes
kernel
Foundation of UNIX OS
mode
7
Processor scheduler
Memory manager
I/O manager
Inter-process communication manager
File system manager
Foundation of UNIX OS
8
OS tends to be complex because ◦ Provides many services ◦ Support variety of hardware and software
Architectures ◦ ◦ ◦ ◦ ◦
Monolithic Layered Micro Distributed Exo
Foundation of UNIX OS
9
Fig 2 Traditional UNIX architecture Foundation of UNIX OS
10
Fig 3 Traditional UNIX kernel Foundation of UNIX OS
11
Fig 4 General UNIX architecture Foundation of UNIX OS
12
Part of UNIX OS that contains code for ◦ Controlling execution of process ◦ Scheduling process fairly
◦ Allocating main memory ◦ Allocating secondary memory
◦ Handling peripherals
Foundation of UNIX OS
13
Loading and existence into main memory
Mostly written in C and assembly language
User programs accesses kernel services via system call interface Provides its services transparently
Foundation of UNIX OS
14
File system
Process management
Input/output
Inter-process communication
Memory management
◦ Directory hierarchy, regular files, peripherals ◦ Multiple file system ◦ How processes share CPU, memory and signals ◦ How processes access files, terminals
Foundation of UNIX OS
15
Kernel services are implemented with several data structures ◦ Process table ◦ Open file table
When process executes a system call, execution mode of process changes from user mode to kernel mode
In kernel mode, process access system data structures
Foundation of UNIX OS
16
Fig 5 Monolithic architecture Foundation of UNIX OS
17
Main function
OS services
Utility functions
Fig 6 Monolithic architecture Foundation of UNIX OS
18
All OS functionality is included in single address
space
Strong points – well understood, good performance
Problems ◦ Kernel components aren’t protected from each other ◦ Not easily extended / modified ◦ May be unclear structure
Eg – UNIX, Linux, most commercial system
Foundation of UNIX OS
19
Fig 7 Microkernel architecture Foundation of UNIX OS
20
Fig 8 Microkernel architecture Foundation of UNIX OS
21
Provides set of minimal core services
Interface to hardware layer
Max functionality is moved to user space
Essential functions in kernel ◦ Primitive memory management ◦ I/O & interrupt management ◦ Inter-process communication ◦ Basic scheduling
Message passing is a communication mean
Eg – Win NT, Amoeba, Chorus, Mach, L4
Foundation of UNIX OS
22
Benefits ◦ Extensibility/reliability Easier to extend More reliable More secure Rigorously tested
◦ Portability Easier to port
◦ Distributed system support Messages are sent without knowing target machine
◦ Object oriented OS
Foundation of UNIX OS
23
Motivation ◦ Traditional centralized resource management cannot be specialized, extended or replaced ◦ Privileged applications applications
must
be
used
by
all
◦ Fixed high level abstractions to costly for good efficiency
Foundation of UNIX OS
24
Goals ◦ Implement traditional abstraction entirely at application level ◦ Focus on managing security not resources
Design principles ◦ Track resource ownership ◦ Ensure protection by guarding resource usage
◦ Revoke access to resources ◦ Expose hardware allocation and revocation
Foundation of UNIX OS
25
Separates resource allocation & protection from resource management
Relies on application specific library OS to provide user modification
Incomplete without library OS
Multiplexes hardware directly
Provides primitives for secure management of physical resources
Applications use them to develop appropriate abstraction
Foundation of UNIX OS
26
Fig 9 Exokernel architecture Foundation of UNIX OS
27
Terms ◦ Booting is a bootstrapping process that starts OS when the user turns on computer system ◦ Boot sequence is set of operations the computer performs when it is switched on that loads an operating system
Booting sequence 1.
Turn on
2.
CPU jumps to address of BIOS (0xFFFF0)
3.
BIOS runs POST
4.
Find bootable device
5.
Load & execute boot sector from MBR
6.
Load OS
Foundation of UNIX OS
28
Fig 10 UNIX boot process Foundation of UNIX OS
29
OS is boot from a hard disk where MBR contains
primary boot loader
MBR is 512 byte sector
First 446 byte = primary boot loader, next 64 byte =
partition table
Located in first sector of disk (sector 1 of cylinder 0, head 0)
After MBR is loaded into RAM, BIOS yields control to it
Foundation of UNIX OS
30
Fig 11 Master Boot Record Foundation of UNIX OS
31
Also referred as kernel loader
Task at this stage is to load kernel
GRUB and LILO are most popular kernels used
Other boot loader ◦ Bootman ◦ NTLDR ◦ XOSL ◦ BootX ◦ Loadin ◦ BootCamp ◦ Syslinux ◦ GAG
Foundation of UNIX OS
32
OS independent boot loader
Multi boot software package from GNU
Flexible CLI
File system access
Support multiple executable format
Support diskless system
Foundation of UNIX OS
33
1.
BIOS finds bootable device & transfers control
to MBR 2.
MBR contains stage 1, which loads next stage of GRUB
3.
After receiving control, GRUB 2 displays boot menu
4.
In case of no selection, default kernel is loaded into RAM and then kernel takes control Foundation of UNIX OS
34
Foundation of UNIX OS
35
Derived from PUPA (Published Unexamined Patent
Application) research project, Japan
Next generation of GNU-GRUB
Rewritten from scratch to clean up everything for
modularity & portability
Goals ◦ Scripting support ◦ Graphical Interface ◦ Dynamic loading of modules
Foundation of UNIX OS
36
◦ Portability for various architecture
◦ Internationalization ◦ Real memory management ◦ Object-oriented framework for filesystem ◦ Cross platform installation ◦ Rescue mode
◦ Fix design mistakes in GRUB legacy
Foundation of UNIX OS
37
At run time environment, minimizing the
frequency of disk access is expected
Kernel does it by keeping pool of internal data buffers
Buffer cache is internal data buffer
It contains data in recently used disk blocks
Kernel first look up data into buffer cache
Foundation of UNIX OS
38
A buffer consists of two parts ◦ A memory array ◦ Buffer header
Disk block : buffer 1:1 device num block num
ptr to data area
status ptr to previous buf on hash queue ptr to previous buf on free list
ptr to next buf on hash queue ptr to next buf on free list
Fig 12 Buffer header Foundation of UNIX OS
39
Device num ◦ Logical file system number
Block num ◦ Block number of the data on disk
Status ◦ Buffer is currently locked ◦ Buffer contains valid data ◦ Delayed-write ◦ Kernel is reading/writing from/to disk ◦ Process is currently waiting for buffer to become free
Foundation of UNIX OS
40
Buffer pool according to LRU
Kernel maintains a free list of buffer ◦ Doubly linked list ◦ Take buffer from the head of the free list ◦ When returning a buffer, attaches the buffer to the tail
Fig 13 Free list of buffers Foundation of UNIX OS
41
When kernel accesses a disk block ◦ Separate queue (doubly linked circular list) ◦ Hashed as a function of the device and block num ◦ Every disk block exists on one and only on hash queue and only once on the queue
Fig 14 Buffers on hash queue Foundation of UNIX OS
42
Determine the logical device num and block num Algorithm getblk is used while reading and writing disk blocks Scenarios ◦ Kernel finds block on hash queue The buffer is free. [Scenario 1] The buffer is currently busy. [Scenario 5] ◦ Block not present on hash queue The kernel allocates a buffer from the free list. [Scenario 2] In attempting to allocate a buffer from the free list, finds a buffer on the free list that has been marked “delayed write”. [Scenario 3] The free list of buffers is empty. [Scenario 4]
Foundation of UNIX OS
43
Hash queue headers
blkno0 mod 4
blkno1 mod 4 blkno2 mod 4
blkno3 mod 4
28
4
64
17
5
97
98
50
10
3
35
99
freelist header
Search for block 4
Foundation of UNIX OS
44
blkno0 mod 4 blkno1 mod 4
blkno2 mod 4 blkno3 mod 4
28
4
64
17
5
97
98
50
10
3
35
99
freelist header
Search for block 4
Foundation of UNIX OS
45
Hash queue headers
blkno0 mod 4 blkno1 mod 4 blkno2 mod 4 blkno3 mod 4
28
4
17
5
97
98
50
10
3
35
99
64
freelist header
Search for block 18: not in cache
Foundation of UNIX OS
46
Hash queue headers
blkno0 mod 4 blkno1 mod 4 blkno2 mod 4
blkno3 mod 4
28
4
17
5
97
98
50
10
35
99
64
18
freelist header
Remove 1st block from free list: assign to 18
Foundation of UNIX OS
47
Hash queue headers
blkno0 mod 4 blkno1 mod 4 blkno2 mod 4 blkno3 mod 4
28
4
64
17
5
97
98
50
10
3
35
99
freelist header
Search for block 18, Delayed write block in free list
Foundation of UNIX OS
48
Writing blocks 3,5, reassign 4 to 18
Foundation of UNIX OS
49
Hash queue headers
blkno0 mod 4
28
4
64
17
5
97
blkno2 mod 4
98
50
10
blkno3 mod 4
3
35
99
blkno1 mod 4
freelist header
Search for block 18: free list empty
Foundation of UNIX OS
50
Hash queue headers
blkno0 mod 4 blkno1 mod 4 blkno2 mod 4 blkno3 mod 4
28
4
17
5
97
98
50
10
3
35
99
64
freelist header
Search for block 99: block busy
Foundation of UNIX OS
51
algorithm getblk (file_system_no,block_no) while (buffer not found) if (buffer in hash queue)
if (buffer busy) sleep (event buffer becomes free) continue mark buffer busy remove buffer from free list return buffer
else
if (there is no buffer on free list) sleep (event any buffer becomes free) continue remove buffer from free list if (buffer marked as delayed write) asyschronous white buffer to disk continue remove buffer from hash queue put buffer onto hash queue return buffer
Foundation of UNIX OS
52
algorithm brelse (locked buffer)
wakeup all process event, waiting for any buffer to become free wakeup all process event, waiting for this buffer to become free raise processor execution level to block interrupt if (buffer content valid and buffer not old) enqueue buffer at the end of free list
else
enqueue buffer at the beginning of free list
lower processor execution level to allow interrupt unlock (buffer)
Foundation of UNIX OS
53
algorithm bread(file system block no) get buffer for block if (buffer data valid) return buffer initiate disk read sleep(event disk read complete) return buffer
Foundation of UNIX OS
54
algorithm bwrite(buffer) initiate disk write if (I/O asynchronous) sleep (event I/O complete) release buffer (algorithm brelse) elseif (buffer marked for delayed write) mark buffer to put at head of free list
Foundation of UNIX OS
55
File is remarkable feature of UNIX
File is a sequence of bytes of data that reside in semi permanent form on some stable medium
Network interface, disk drive, keyboard, printer, etc are treated as file
File management is flexible and powerful
Provides hierarchical directory scheme
Foundation of UNIX OS
56
Simple/ordinary file ◦ Stores data on secondary storage ◦ Contains executable program, DB tools, pictures, audio, graphics, etc
Directory file ◦ Contains name of other file &/or directory ◦ Have names just like other file ◦ Holds record for each file &/or directory it contains ◦ Record contains file inode number and name
Foundation of UNIX OS
57
Link File ◦ Points to existing file ◦ Allows existing file to be renamed or shared without duplicating its contents ◦ Link can be soft or hard
Special file (device file) ◦ Mean of accessing hardware device ◦ Each h/w device is associated with atleast one device file ◦ Types – character special and block special file
Foundation of UNIX OS
58
It is structured hierarchical upside down treelike
Starts with root and can have any number of files or subdirectories
Fig 15 UNIX file system structure Foundation of UNIX OS
59
After logon, UNIX put you in home/logon directory
Home directory can be specified with ‘~’ sign, provided C/Korn shell is used
PWD is denoted by ‘.’ and its parent by ‘..’
Pathnames can be specified in 3 ways ◦ Absolute pathname (starting with ‘/’) ◦ Relative pathname (starting with pwd)
◦ Relative pathname (starting with user home directory)
Foundation of UNIX OS
60
Root directory (‘/’) ◦ Top of file system hierarchy ◦ It is a master cabinet which contains all ◦ Try command
$ls /
/bin ◦ Contains binary executables ◦ Usually it is symbolic link to /usr/bin
/dev ◦ Contains files corresponding to devices ◦ Eg. lp (line printer), tty (terminal), hd (hard drive)
Foundation of UNIX OS
61
/etc ◦ Contains commands and files used in system administration ◦ Typical user is not allowed to use these files ◦ Eg. login, passwd, profile
/lib ◦ Contains archive files ◦ Contains libraries for C, C++ & other languages ◦ Some libraries can be found in /usr/lib
/lost+found ◦ Contains file which are not connected to any directory ◦ fsck tool can be used to see these files
Foundation of UNIX OS
62
/tmp ◦ Contains temporary files
/users ◦ Holds home directories of all users ◦ May be located in many various parts of file system
/usr ◦ Holds directories which contains utilities, tools, languages, libraries, manual pages ◦ Two important sub directory - /bin & /lib
/etc/passwd ◦ Contains logon details of each user
Foundation of UNIX OS
63
1.
OPEN
2.
CLOSE
3.
READ
4.
WRITE
5.
CHOMOD, CHOWN
6.
RENAME
7.
MKDIR
8.
RMDIR
9.
CHDIR
10.
CHROOT
11.
LINK
Foundation of UNIX OS
64
namei alloc free iget
ialloc ifree
iput buffer allocation algorithms getblk
brelse
bread
bwrite
Fig. 16 Lower level filesystem algorithms
Foundation of UNIX OS
65
Contains the information necessary for a process to access a file Exits in a static form on disk and the kernel reads them into an in-core inode Consists of ◦ file owner identifier ◦ file type ◦ file access permissions ◦ file access times ◦ number of links to the file ◦ file size ◦ table of contents for the disk address of data in a file
Foundation of UNIX OS
66
In-core copy of inode consists of
◦ status of the in-core inode ◦ logical device number of file system ◦ inode number ◦ pointers to other in-core inodes ◦ reference count
In nutshell inode holds the metadata of file
Foundation of UNIX OS
67
while (not done)
if (inode in inode cache) if (inode locked)
sleep(event inode becomes unlocked) continue
if (inode on inode free list) remove from free list return locked inode
if (no inode on free list) return error remove new inode from free list set inode number remove inode from old hash queue and place on new one read inode from disk set reference count 1 return locked indoe
Foundation of UNIX OS
68
lock inode if not locked decrement inode refernece count if (refernce count==0) if (inode link==0) free disk block set file type to 0 free inode
if (file accessed or inode changed or file changed) update disk inode
put inode on free list
Release inode lock
Foundation of UNIX OS
69
Since disk is limited, we need to reuse it
To keep track of free disk space, system maintains a free space list
Free space list records all free disk blocks
New file allocation is done amongst the free disk block
When file is deleted, its disk space is added to free space list
Foundation of UNIX OS
70
Frequently, free space list is maintained as a bit map or bit vector
Each block is represented as 1 bit ◦ If block is free, bit is 1 ◦ If block is allocated, bit is 0
For eg. consider disk where blocks 2,3,5,7,8,10 are free and rest are allocated. The free space bit map would be 0110 1011 01
Advantage – relatively simple and efficient to find first free block
Foundation of UNIX OS
71
Free blocks are linked with each other
First block contains pointer to next free block and so on
Scheme is not efficient as list traversing needs substantial I/O time
Fig 17 Linked list approach Foundation of UNIX OS
72
Grouping ◦ Store address of n free block in the first free block ◦ Address of large number of free blocks can now be found quickly
Counting ◦ Generally several contiguous blocks are allocated or freed
simultaneously ◦ So idea is to keep address of first block and number n of free contiguous blocks that follow first block ◦ Now each entry in free space list then consists of disk address and
a count.
Foundation of UNIX OS
73
In UNIX/Linux, every thing is considered as file
For
eg.
if
you
have
two
hard
disks
then
representation will be ◦ /dev/hda (primary master HDD) & ◦ /dev/hdb, (primary slave HDD)
For floppy drive - /dev/fd0
For CD ROM - /dev/cdrom
For DVD-writer - /dev/dvdwriter
SATA, USB mass storage - /dev/sda
Foundation of UNIX OS
74
Partition on HDD - /dev/hda0 (1st partition)
Things to be remembered before partition creation ◦ What is the purpose of partition creation ◦ Check free space left with command: fdisk –l
Steps: 1.
fdisk –l
2.
fdisk /dev/hda
3.
Create new partition - Specify size
4.
Update partition table - partprob /dev/hda
Foundation of UNIX OS
75
Swap device is a block device in configurable section of a disk
Kernel allocates contiguous space on the swap device without fragmentation
Free space of swap device is maintained in core table – map
Kernel treats each unit of swap map as group of disk blocks
As kernel allocates and frees resources, it updates map accordingly
Foundation of UNIX OS
76
Physical Addresses
Virtual Addresses
Text
Swap device 684
0
278k
1k
432k
Data 65k
:
573k
66k
Stack 128k
595k
690
: 401k :
Fig 18 Mapping process to swap device Foundation of UNIX OS
77
Physical Addresses
Virtual Addresses
Text
Swap device 684
0
278k
1k
432k :
Data 65k
573k
66k
595k
690
:
Stack 128k
401k :
Fig 19 Mapping process to swap device Foundation of UNIX OS
78
1.
Draw and explain UNIX architecture.
2.
Differentiate monolithic, micro and exo kernel.
3.
Explain booting process in detail.
4.
Compare and contrast GRUB-1 and GRUB-2.
5.
Explain with neat diagram buffer header.
6.
Write and explain algorithm getblk().
Foundation of UNIX OS
79
7.
Explain algorithm bread() and bwrite()
8.
Explain algorithm iget() and iput()
9.
Draw and explain UNIX file system architecture.
10.
What
do
you
understand
by
free
space
management? Explain its different approaches. 11.
What is the importance of swapping in OS?
Foundation of UNIX OS
80
[1] Maurice J. Bach, “The Design of UNIX Operating System”,
PHI, ISBN 978-81-203-0516-8 [2] Evi Nemeth, Garth Snyder, Tren Hein, Ben Whaley, “Unix and Linux System Administration Handbook”, Fourth
Edition, ISBN: 978-81-317-6177-9, 2011 [3] http://www.linuxquestions.org/questions/linux-general1/lilo-vs-grub1-vs-grub2-847495/
Foundation of UNIX OS
81
Thank You Foundation of UNIX OS
82
