Lecture 007 - NFS, AFS

Distributed File System

Why Distributed File System

Mission of andrew.cmu.edu:

• any person can use distributed file system on their computer.

• it should behave the same as traditional file system (even interface)

• support all interface: open, status, close, read, write, create, delete, directory, rename, symlink, lock, ....

// QUESTION: why IPFS is successful while AFS is not

Usage:

• sharing files

• user mobility, location transparency (don't have to carry harddrive)

• backup, centralized management

Challenges:

• heterogeneity: different OS

• scale: support many people

• security: private files

• failures: single-point failure (network failure)

• concurrency

Building Distributed File System

Assumptions Driven Design: user-centric

• most files are owned by single person

• no much concurrency access

• common sequential and read access

• locality: likely to edit the same file again

We need to modify:

• kenrel: for interface, we need to virtualize file system

• server: serving client

• communication layer: request protocol

Remote Procedure Calls

Assuming one centralized server.

Instead of kernel calls, we send RPC to server for every file operation. Files are completely stored on centralized server, and softwares that invoke file instructions are stored on client.

Tradeoff:

• easy to implement

• terrible network latency

• very big server load

Caching

Client-side Caching: store copy on client

• on demand: when "open"

• pre-fetching: cache in advance for latency benifit

Update Visibility Issue: when write instruction can't be updated immediately to server (or lost when client crash) Cache Staleness Issue: when read instruction is after a remote write

Ideal Model: one copy semantics: no difference if in "no cache" case in user perspective

Broadcast Invalidation

Broadcast Invalidation: every update is broadcasted to every client that uses specific file.

• network redundancy

• somewhat consistency

• fast reads

Check on Use: client check with server before use

• prevent network redundancy

• somewhat consistency

• slow reads (because it need to ask server)

• server load too high

Callbacks: client register with server when they got a copy, server suppose to multicast to all client who registered when there is an update to file.

• server crash: lost all registered state

• client crash: re-register and get a copy (might miss callback)

Leases: client is granted with exclusive access to server's file with cache

• Lease Period: duration of control (in case client crash)

• Lease Renewal: client request renewal

• Read / Write Separate:

  • multiple client can obtain read lease
  • only one client can obtain read lease

NFSv2

NFSv2

• client cache in memory, with callback

• file attribute expire in 60s

• data never expire, but will be updated

• dirty data update every 30s or when file close

NFSv2 do all file instruction locally, but data consistency is poor. (Unacceptable in distributed applications)

Failure Handling:

• stateless server (no pending transactions or leases, let client figure out version number): no history of every operation

• duplicated request: keep a version number (or operation unique id) in server (making operations idempotent)

• write-through caching: client can close() successful only if updated successfully to server

AFS: - also caches on disk - prefetching (good for sequential access) - callback

File Access Consistency

UNIX: sequential consistency (atomic operation) NFS: 30 seconds window AFS: session semantics

AFSv2:

• file write not visible to other machine until closed

• server send "break callback" to all clients when file change

AFS benifits vs NFS:

• AFS lower server load than NFS

  • file cached on client
  • server is not busy if file is read only

• (but sometimes access from local disk is slower than another machine's memory over LAN)

Authentication

Say Hanke logs onto hankec@yukisa machine and want to access [email protected], the question is:

• How do hankec@yukisa know who is accessing the machine

• Why should [email protected] believe it is really Hanke who want to access the file after listing to what hankec@yukisa says.

So we need authentication at application level.

Symmetic Crypto (Private Key)

• shared secret between parties

• fast speed (password)

• e.g.: AES

Asymmetric Crypto (Public Key)

• no shared secret, only one side generate key for other

• slow speed (ssh)

• e.g.: RSA

Key Distribution Center (KDC)

Key Distribution Center (KDC): whoever stores your password and log you in. And plus it can encode message to let you talk to others (who also registered to KDC) secretly.

Key distribution center uses Symmetic Crypto, but it provide single point of failure: KDC knows the key for everybody, which is bad.

CODA's Replica Control

CODA: successor of AFS that allow clients to operate in disconnected mode

• situration: network is slow, but machines are powerful (compared to 80s), and mobile device appears

Pessimistic Replica Control: require C$C$ to obtain a lock before being offline

• client $C$ can acquire: two types of locks

  • Exclusive (Read+Write): no other clients can read or write while client $C$ is offline
  • Shared (Read): other clients can only read while client $C$ is offline

• Cannot handel non-voluntary disconnection (can't require lock before crash)

• Disconnect in long period of time cause troubles

Optimistic Replica Control: .git

• higher avaliability: always allow access of replica on server

• inconsistencies: assume you can tolerate consistency

• detect inconsistency and do a merge

  • when merge, notify user
  • or let the user manage how to merge

Examples of Version Control System:

• Revision Control System (RCS): Relies on locks, similar to "pessimistic control", very simple

• Subversion (SVN): scalable, with merges and parallel branch, fully centralized

• Git (Global Information Tracking): using better branches, decentralized, with automatic merging