[LWN Logo]

From: Philipp Reisner <kde@ist.org>
To: Linux-HA mailing list <linux-ha@muc.de>
Subject: drbd (was nmbd)
Date: Sat, 6 Nov 1999 12:13:42 +0100


You can find it at http://www.complang.tuwien.ac.at/reisner/drbd/


				 DRBD
				======
		   by Philipp Reisner <kde@ist.org>


What is it ?
------------

Drbd is a block device which is designed to build high availability clusters.
This is done by mirroring a whole block device via ( a dedicated ) network.
You could see it as a network raid 1.



What is the scope of drbd, what else do I need to build a HA cluster ?
----------------------------------------------------------------------

Drbd takes over the data, writes it to the local disk and sends
it to the other host. On the other host, it takes it to the disk there.

The other components needed are a cluster membership service, which is
supposed to be _heartbeat_, and some kind of application that works
on top of a block device. 
 Examples:
  *) A filesystem & fsck.
  *) A journaling FS.
  *) A database with recovery capabilities.



How does it work ?
------------------

Each device (drbd provides more than one of these devices) has a state,
which can be 'primary' or 'secondary'. On the node with the primary device
the application is supposed to run and to access the device (/dev/nbX).
Every write is sent to the local 'lower level block device' and to the
node with the device in 'secondary' state. The secondary device simply
writes the data to its lower level block device. Reads are always carried 
out locally. 

If the primary node fails, heartbeat is switching the secondary device
into primary state and starts the application there. (If you are using
it with a non-journaling FS this involves running fsck)

If the failed node comes up again, it is a new secondary node and has
to synchronise its content to the primary. This, of course, will happen
whithout interruption of service in the background. 



Howto !
-------

Let's assume that your two machines are named 'node1' and 'node2', and
that you want to use /dev/loop0 as the lower level block device on both,
then these commands are needed on node1 to setup drbd:

insmod drbd.o
drbdsetup /dev/nb0 /dev/loop0 node1 node2

On node2 you need:

insmod drbd.o
drbdsetup /dev/nb0 /dev/loop0 node2 node1

And finally your cluster membership service needs to tell one of 
the two to become primary by running:

drbdsetup /dev/nb0 p 



Implementation details
----------------------

At first I wanted to use UDP so that I could benefit from UDP's multicasting
abilities to implement clusters of more than two nodes. But after I
finished the first experimental UDP implementation it turned out that
the kernel is only storing up to 64 kb of incoming data on an UDP socket.
And with faster networks it happens quite easily that your intel box 
(timer interrupt @ 100 times per second) is not scheduling your receiving
process often enough and you get an enormous amount of lost packets.
(Alpha did a *lot* better, there you have 1024 interrupts per second).

So for now I am using TCP.

Accessing the lower level block device is done with a temporary copy
of the buffer_head and a call to ll_rw_block. Thus you should never
access the lower level block device directly when you have a drbd
running on top of it!

I am using /dev/nb0 (and major 43) because of 

~linux-2.2.7/drivers/block/ll_rw_blk.c  line ~447 :
--snip--
     /* Loop uses two requests, 1 for loop and 1 for the real device.
      * Cut max_req in half to avoid running out and deadlocking. */
        if ((major == LOOP_MAJOR) || (major == NBD_MAJOR))
            max_req >>= 1;
--snap--

It _will_ _deadlock_ if you use another major number than one of these two!!



Status of drbd-0.1.tar.gz
-------------------------

This is a proof-it-is-no-vapo-ware-release and not more!
It is the first time it is somehow working without immediately crashing
my machine (you can "even" unload the module). It is not able to 
handle blocksized other than 1024 and there is not a single line of code
of the sync-a-new-secondary stuff yet.
The used port is hardcoded into drbdsetup (7788) for now.
I have tested it on an intel box and an alpha machine.


------------------------------------------------------------------------
     Want to try something new?            Are you a Linux hacker?
		    Volunteer in testing mergemem!
	      (Get it from http://das.ist.org/mergemem)
-----
  Philipp Reisner                PGP: http://der.ist.org/~kde/pgp.asc

------------------------------------------------------------------------------
Linux HA Web Site:
  http://linux-ha.org/
Linux HA HOWTO:
  http://metalab.unc.edu/pub/Linux/ALPHA/linux-ha/High-Availability-HOWTO.html
------------------------------------------------------------------------------