Mission of andrew.cmu.edu
:
any person can use distributed file system on their computer.
it should behave the same as traditional file system (even interface)
support all interface: open, status, close, read, write, create, delete, directory, rename, symlink, lock, ....
// QUESTION: why IPFS is successful while AFS is not
Usage:
sharing files
user mobility, location transparency (don't have to carry harddrive)
backup, centralized management
Challenges:
heterogeneity: different OS
scale: support many people
security: private files
failures: single-point failure (network failure)
concurrency
Assumptions Driven Design: user-centric
most files are owned by single person
no much concurrency access
common sequential and read access
locality: likely to edit the same file again
We need to modify:
kenrel: for interface, we need to virtualize file system
server: serving client
communication layer: request protocol
Assuming one centralized server.
Instead of kernel calls, we send RPC to server for every file operation. Files are completely stored on centralized server, and softwares that invoke file instructions are stored on client.
Tradeoff:
easy to implement
terrible network latency
very big server load
Client-side Caching: store copy on client
on demand: when "open"
pre-fetching: cache in advance for latency benifit
Update Visibility Issue: when write
instruction can't be updated immediately to server (or lost when client crash)
Cache Staleness Issue: when read
instruction is after a remote write
Ideal Model: one copy semantics: no difference if in "no cache" case in user perspective
Broadcast Invalidation: every update is broadcasted to every client that uses specific file.
network redundancy
somewhat consistency
fast reads
Check on Use: client check with server before use
prevent network redundancy
somewhat consistency
slow reads (because it need to ask server)
server load too high
Callbacks: client register with server when they got a copy, server suppose to multicast to all client who registered when there is an update to file.
server crash: lost all registered state
client crash: re-register and get a copy (might miss callback)
Leases: client is granted with exclusive access to server's file with cache
Lease Period: duration of control (in case client crash)
Lease Renewal: client request renewal
Read / Write Separate:
NFSv2
client cache in memory, with callback
file attribute expire in 60s
data never expire, but will be updated
dirty data update every 30s or when file close
NFSv2 do all file instruction locally, but data consistency is poor. (Unacceptable in distributed applications)
Failure Handling:
stateless server (no pending transactions or leases, let client figure out version number): no history of every operation
duplicated request: keep a version number (or operation unique id) in server (making operations idempotent)
write-through caching: client can close()
successful only if updated successfully to server
AFS: - also caches on disk - prefetching (good for sequential access) - callback
UNIX: sequential consistency (atomic operation) NFS: 30 seconds window AFS: session semantics
AFSv2:
file write not visible to other machine until closed
server send "break callback" to all clients when file change
AFS benifits vs NFS:
AFS lower server load than NFS
(but sometimes access from local disk is slower than another machine's memory over LAN)
Say Hanke
logs onto hankec@yukisa
machine and want to access [email protected]
, the question is:
How do hankec@yukisa
know who is accessing the machine
Why should [email protected]
believe it is really Hanke
who want to access the file after listing to what hankec@yukisa
says.
So we need authentication at application level.
Symmetic Crypto (Private Key)
shared secret between parties
fast speed (password)
e.g.: AES
Asymmetric Crypto (Public Key)
no shared secret, only one side generate key for other
slow speed (ssh)
e.g.: RSA
Key Distribution Center (KDC): whoever stores your password and log you in. And plus it can encode message to let you talk to others (who also registered to KDC) secretly.
Key distribution center uses Symmetic Crypto, but it provide single point of failure: KDC knows the key for everybody, which is bad.
CODA: successor of AFS that allow clients to operate in disconnected mode
Pessimistic Replica Control: require C to obtain a lock before being offline
client C can acquire: two types of locks
Cannot handel non-voluntary disconnection (can't require lock before crash)
Disconnect in long period of time cause troubles
Optimistic Replica Control: .git
higher avaliability: always allow access of replica on server
inconsistencies: assume you can tolerate consistency
detect inconsistency and do a merge
Examples of Version Control System:
Revision Control System (RCS): Relies on locks, similar to "pessimistic control", very simple
Subversion (SVN): scalable, with merges and parallel branch, fully centralized
Git (Global Information Tracking): using better branches, decentralized, with automatic merging
Table of Content