Merge branch 'mhop'

* mhop: (28 commits)
  runmirrors
  ftpsync, runmirrors
  runmirrors
  ftpsync
  runmirrors
  common
  runmirrors
  ftpsync tracefile
  fix whitespace indenting to minimize diff
  quote some more arguments
  fix whitespace to minimize diff between master and mhop branches
  readme
  More comments in the mirror sample
  common
  set delay to 600 to be what we tell users in our config file would be default
  runmirrors getopts
  common
  common
  runmirrors
  runmirrors
  ...

5 files changed, 241 insertions, 86 deletions

diff --git a/README b/README
index 8541eaf..e85af02 100644
--- a/README
+++ b/README
@@ -11,7 +11,7 @@ Currently the following scripts are available:
 
   * ftpsync     - Used to sync an archive using rsync
   * runmirrors  - Used to notify leaf nodes of available updates
-  * dircombine  - Internal script to manage the mirror user's $HOME 
+  * dircombine  - Internal script to manage the mirror user's $HOME
                   on debian.org machines
   * typicalsync - Generates a typical Debian mirror
   * udh         - We are lazy, just a shorthand to avoid typing the
@@ -23,14 +23,14 @@ For impatient people, short usage instruction:
 
  - Create a dedicated user for the whole mirror.
  - Create a seperate directory for the mirror, writeable by the new user.
- - Place the ftpsync script in the mirror user's $HOME/ (or $HOME/bin)
+ - Place the ftpsync script in the mirror user's $HOME/bin (or just $HOME)
  - Place the ftpsync.conf.sample into $HOME/etc as ftpsync.conf and edit
-   it to suit your system.  You should at the very least change the TO= 
+   it to suit your system.  You should at the very least change the TO=
    and RSYNC_HOST lines.
  - Create $HOME/log (or wherever you point $LOGDIR to)
  - Setup the .ssh/authorized_keys for the mirror user and place the public key of
    your upstream mirror into it. Preface it with
-no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,command="~/bin/ftpsync",from="IPADDRESS" 
+no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,command="~/bin/ftpsync",from="IPADDRESS"
    and replace $IPADDRESS with that of your upstream mirror.
  - You are finished
 
@@ -53,7 +53,7 @@ functionality:
 
  - Must perform a 2-stage sync
    The archive mirroring must be done in 2 stages. The first rsync run
-   must ignore the index files.  The correct exclude options for the 
+   must ignore the index files.  The correct exclude options for the
    first rsync run are:
   --exclude Packages* --exclude Sources* --exclude Release* --exclude ls-lR*
    The first stage must not delete any files.
@@ -76,7 +76,7 @@ functionality:
 
  - Should understand multi-stage pushes.
    The script should parse the arguments it gets via ssh, and if they
-   contain a hint to only sync stage1 or stage2, then ONLY those steps 
+   contain a hint to only sync stage1 or stage2, then ONLY those steps
    SHOULD be performed.
 
    Rationale: This enables us to coordinate the timing of the first
@@ -106,51 +106,67 @@ Some of the advantages of the new version are:
  - Support for multi-stage archive synchronisations
  - Support for hook scripts at various points
  - Support for multiple archives, even if they are pushed using one ssh key
+ - Support for multi-hop, multi-stage archive synchronisations
 
   Correct support for multiple pushes
   -----------------------------------
   When the script receives a second push while it is running and syncing
-  the archive it won't ignore it. Instead it will rerun the 
+  the archive it won't ignore it. Instead it will rerun the
   synchronisation step to ensure the archive is correctly synchronised.
-   
+
   Scripts that fail to do that risk ending up with an inconsistent archive.
-   
-   
+
+
   Can do multi-stage archive synchronisations
   -------------------------------------------
   The script can be told to only perform the first or second stage of the
-  archive synchronisation. 
-   
+  archive synchronisation.
+
   This enables us to send all the binary packages and sources to a
   number of mirrors, and then tell all of them to sync the
   Packages/Release files at once. This will keep the timeframe in which
   the mirrors are out of sync very small and will greatly help things like
   DNS RR entries or even the planned GeoDNS setup.
-   
-   
+
+
+  Multi-hop, multi-stage archive synchronisations
+  -----------------------------------------------
+  The script can be told to perform a multi-hop multi-stage archive
+  synchronisation.
+
+  This is basically the same as the multi-stage synchronisation
+  explained above, but enables the downstream mirror to push his own
+  staged/multi-hop downstreams before returning. This has the same
+  advantage than the multi-stage synchronisation but allows us to do
+  this over multiple level of mirrors. (Imagine one push going from
+  Europe to Australia, where then locally 3 others get updated before
+  stage2 is sent out. Instead of 4times transferring data from Europe to
+  Australia, just to have them all updated near instantly).
+
+
   Can run hook scripts
   --------------------
   ftpsync currently allows 5 hook scripts to run at various points of the
   mirror sync run.
-   
+
   Hook1: After lock is acquired, before first rsync
   Hook2: After first rsync, if successful
   Hook3: After second rsync, if successful
   Hook4: Right before leaf mirror triggering
   Hook5: After leaf mirror trigger (only if we have slave mirrors; HUB=true)
-   
+
   Note that Hook3 and Hook4 are likely to be called directly after each other.
   The difference is that Hook3 is called *every* time the second rsync
   succeeds even if the mirroring needs to re-run due to a second push.
   Hook4 is only executed if mirroring is completed.
-   
-   
+
+
   Support for multiple archives, even if they are pushed using one ssh key
   ------------------------------------------------------------------------
   If you get multiple archives from your upstream mirror (say Debian,
-  Debian-Backports and Volatile), previously you had to use 3 different ssh 
+  Debian-Backports and Volatile), previously you had to use 3 different ssh
   keys to be able to automagically synchronize them.  This script can do it
-  all with just one key, if your upstream mirror tells you which archive. 
+  all with just one key, if your upstream mirror tells you which archive.
   See "Commandline/SSH options" below for further details.
 
 
@@ -171,6 +187,7 @@ Option         Behaviour
 stage1         Only do stage1 sync
 stage2         Only do stage2 sync
 all            Do a complete sync (default)
+mhop           Do a multi-hop sync
 archive:foo    Sync archive foo (if the file $HOME/etc/ftpsync-foo.conf
                exists and is configured)
 callback       Call back when done (needs proper ssh setup for this to
@@ -223,9 +240,9 @@ as well as the newer multi-stage push.
 The normal push, as described above, will simply push the leaf node and
 then go on with the other nodes.
 
-The multi-staged push first pushes a mirror and tells it to only do a 
+The multi-staged push first pushes a mirror and tells it to only do a
 stage1 sync run. Then it waits for the mirror (and all others being pushed
-in the same run) to finish that run, before it tells all of the staged 
+in the same run) to finish that run, before it tells all of the staged
 mirrors to do the stage2 sync.
 
 This way you can do a nearly-simultaneous update of multiple hosts.
diff --git a/bin/ftpsync b/bin/ftpsync
index 73b6604..98f80d4 100755
--- a/bin/ftpsync
+++ b/bin/ftpsync
@@ -56,6 +56,7 @@ VERSION="80286"
 # stage1		 Only do stage1 sync
 # stage2		 Only do stage2 sync
 # all			 Do a complete sync
+# mhop           Do a mhop sync, usually additionally to stage1
 # archive:foo    Sync archive foo (if config for foo is available)
 # callback       Call back when done (needs proper ssh setup for this to
 #                work). It will always use the "command" callback:$HOSTNAME
@@ -89,6 +90,9 @@ check_commandline() {
 			sync:all)
 				SYNCALL="true"
 				;;
+			sync:mhop)
+				SYNCMHOP="true"
+				;;
 			*)
 				echo "Unknown option ${1} ignored"
 				;;
@@ -287,6 +291,8 @@ SYNCSTAGE1=${SYNCSTAGE1:-"false"}
 SYNCSTAGE2=${SYNCSTAGE2:-"false"}
 # Do we sync all?
 SYNCALL=${SYNCALL:-"true"}
+# Do we have a mhop sync?
+SYNCMHOP=${SYNCMHOP:-"false"}
 # Do we callback?
 SYNCCALLBACK=${SYNCCALLBACK:-"false"}
 # If we call back we need some more options defined in the config file.
@@ -442,10 +448,14 @@ function mainroutine() {
 	    hook $HOOK
     done
 
-    if [ -d "`dirname "${TO}/${TRACE}"`" ]; then
-        LC_ALL=POSIX LANG=POSIX date -u > "${TO}/${TRACE}"
-        echo "Used ftpsync version: ${VERSION}" >> "${TO}/${TRACE}"
-	    echo "Running on host: $(hostname -f)" >> "${TO}/${TRACE}"
+    # We only update our tracefile when we had a stage2 or an all sync.
+    # Otherwise we would update it after stage1 already, which is wrong.
+	if [ "xtruex" = "x${SYNCSTAGE2}x" ] || [ "xtruex" = "x${SYNCALL}x" ]; then
+        if [ -d "`dirname "${TO}/${TRACE}"`" ]; then
+            LC_ALL=POSIX LANG=POSIX date -u > "${TO}/${TRACE}"
+            echo "Used ftpsync version: ${VERSION}" >> "${TO}/${TRACE}"
+	        echo "Running on host: $(hostname -f)" >> "${TO}/${TRACE}"
+        fi
     fi
 
     HOOK=(
@@ -464,10 +474,29 @@ function mainroutine() {
 	rm -f "${LOCK}"
 
     if [ x${HUB} = "xtrue" ]; then
-		# Only trigger slave mirrors if we had a push for stage2 or all
-	    if [ "xtruex" = "x${SYNCSTAGE2}x" ] || [ "xtruex" = "x${SYNCALL}x" ]; then
+		# Trigger slave mirrors if we had a push for stage2 or all, or if its mhop
+	    if [ "xtruex" = "x${SYNCSTAGE2}x" ] || [ "xtruex" = "x${SYNCALL}x" ] || [ "xtruex" = "x${SYNCMHOP}x" ]; then
+            RUNMIRRORARGS=""
+            if [ -n "${ARCHIVE}" ]; then
+                # We tell runmirrors about the archive we are running on.
+			    RUNMIRRORARGS="-a ${ARCHIVE}"
+            fi
+			# We also tell runmirrors that we are running it from within ftpsync, so it can change
+			# the way it works with mhop based on that.
+			RUNMIRRORARGS="${RUNMIRRORARGS} -f"
+
+			if [ "xtruex" = "x${SYNCSTAGE1}x" ]; then
+				# This is true when we have a mhop sync. A normal multi-stage push sending stage1 will
+				# not get to this point.
+				# So if that happens, tell runmirrors we are doing mhop
+				RUNMIRRORARGS="${RUNMIRRORARGS} -k mhop"
+			elif [ "xtruex" = "x${SYNCSTAGE2}x" ]; then
+				RUNMIRRORARGS="${RUNMIRRORARGS} -k stage2"
+			elif [ "xtruex" = "x${SYNCALL}x" ]; then
+				RUNMIRRORARGS="${RUNMIRRORARGS} -k all"
+			fi
 		    log "Trigger slave mirrors"
-		    ${BASEDIR}/bin/runmirrors "${ARCHIVE}"
+		    ${BASEDIR}/bin/runmirrors ${RUNMIRRORARGS}
 		    log "Trigger slave done"
 
 		    HOOK=(
diff --git a/bin/runmirrors b/bin/runmirrors
index a79e33d..752e351 100755
--- a/bin/runmirrors
+++ b/bin/runmirrors
@@ -7,7 +7,7 @@ set -u
 # Based losely on existing scripts, written by an unknown number of
 # different people over the years.
 #
-# Copyright (C) 2008 Joerg Jaspert <joerg@debian.org>
+# Copyright (C) 2008, 2009 Joerg Jaspert <joerg@debian.org>
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License as
@@ -28,12 +28,68 @@ BASEDIR=${BASEDIR:-"${HOME}"}
 
 NAME="`basename $0`"
 
-# In case we are called with an argument we look for a different configuration.
-CONF=${1:-""}
-if [ -n "${CONF}" ]; then
-	NAME="${NAME}-${CONF}"
+HELP="$0, (C) 2008, 2009 by Joerg Jaspert <joerg@debian.org>\n
+Usage:\n\n
+
+1.) a single parameter with NO leading -.\n
+\t  This will will then be used as the addition for our configfile. Ie. \`$0 security\` will\n
+\t  have us look for ${NAME}-security.{conf,mirror} files.\n\n
+
+2.) using getopt style parameters:\n
+\t -a [NAME]   - Same as 1.) above, used for the config files. Default empty.\n
+\t -k [TYPE]   - Type of push. all, stage2, mhop. Default mhop.\n
+\t -f          - Run from within the mirrorscript ftpsync. Don't use from commandline!\n
+\t -h          - Print this help and exit
+"
+# If we got options, lets see if we use newstyle options, or oldstyle. If oldstyle
+# it will not start with a -. If we find oldstyle we assume its only one, the config
+# name we run on.
+if [ $# -gt 0 ]; then
+    if [ "x${1:0:1}x" != "x-x" ]; then
+        # Yes, does not start with a -, so use it for the config name.
+        CONF=${1:-""}
+        if [ -n "${CONF}" ]; then
+            NAME="${NAME}-${CONF}"
+        fi
+    else
+        # Yeah well, new style, starting with - for getopts
+        while getopts ':a:k:fh' OPTION ; do
+            case $OPTION in
+                a)	CONF="${OPTARG}"
+                    if [ -n "${CONF}" ]; then
+                        NAME="${NAME}-${CONF}"
+                    fi
+                    ;;
+                k)	PUSHKIND="${OPTARG}"
+                    ;;
+                f)	FROMFTPSYNC="true"
+                    ;;
+                h)	echo -e $HELP
+                    exit 0
+                    ;;
+
+                *)  echo "Invalid usage"
+		            echo -e $HELP 
+                    exit 1
+                    ;;
+            esac
+        done
+    fi
 fi
+# Make sure the values are always defined, even if there was no commandline option
+# for them
+# Default config is empty
+CONF=${CONF:-""}
+
+# Set the default to all, if we didnt get told about it. Currently
+# valid: all - normal push. mhop - multi-hop multi-stage push, this is stage1,
+# stage2 - staged push, second phase. Default is mhop.
+PUSHKIND=${PUSHKIND:-"mhop"}
 
+# If we are pushed from within ftpsync. Default false.
+FROMFTPSYNC=${FROMFTPSYNC:-"false"}
+
+########################################################################
 # Read our config file
 . "${BASEDIR}/etc/${NAME}.conf"
 
@@ -80,6 +136,8 @@ HOOK1=${HOOK1:-""}
 HOOK2=${HOOK2:-""}
 HOOK3=${HOOK3:-""}
 
+########################################################################
+
 # Some sane defaults
 cd ${BASEDIR}
 umask 022
@@ -90,7 +148,7 @@ mkdir -p "${LOCKDIR}"
 
 trap 'log "Mirrorpush done" >> ${LOG}; savelog "${LOG}" > /dev/null' EXIT
 
-log "Pushing leaf mirrors" >> ${LOG}
+log "Pushing leaf mirrors. Inside ftpsync: ${FROMFTPSYNC}. Pushkind: ${PUSHKIND}" >> ${LOG}
 
 HOOK=(
 	HOOKNR=1
@@ -116,13 +174,21 @@ while read MTYPE MLNAME MHOSTNAME MUSER MSSHOPT; do
 	if [ "x${MTYPE}x" = "xDELAYx" ]; then
 		# We should wait a bit.
 		if [ -z ${MLNAME} ]; then
-			MLNAME=60
+			MLNAME=600
 		fi
 		log "Delay of ${MLNAME} requested, sleeping" >> "${LOG}"
 		sleep ${MLNAME}
 		continue
 	fi
 
+    # If we are told we have a mhop sync to do and are called from within ftpsync,
+    # we will only look at staged/mhop entries and ignore the rest.
+    if [ "x${PUSHKIND}x" = "xmhopx" ] && [ "x${FROMFTPSYNC}x" = "xtruex" ]; then
+        if [ "x${MTYPE}x" != "xstagedx" ] && [ "x${MTYPE}x" != "xmhopx" ]; then
+            continue
+        fi
+    fi
+
     # Now, MSSHOPT may start with a -. In that case the whole rest of the line is taken
     # as a set of options to give to ssh, we pass it without doing anything with it.
     # If it starts with a 1 or 2 then it will tell us about the ssh protocol version to use,
@@ -172,6 +238,8 @@ while read MTYPE MLNAME MHOSTNAME MUSER MSSHOPT; do
 		PUSHLOCKOWN="${LOCKDIR}/${MLNAME}.stage1"
 		PUSHTYPE="${MTYPE}"
 		PUSHARCHIVE=${PUSHARCHIVE}
+		PUSHKIND=${PUSHKIND}
+		FROMFTPSYNC=${FROMFTPSYNC}
 	)
 
 	# And finally, push the mirror
diff --git a/etc/common b/etc/common
index 465ab73..2f6af01 100644
--- a/etc/common
+++ b/etc/common
@@ -17,6 +17,8 @@
 #                         then push stage2
 # $PUSHARCHIVE - what archive to sync? (Multiple mirrors behind one ssh key!)
 # $PUSHCB      - do we want a callback?
+# $PUSHKIND    - whats going on? are we doing mhop push or already stage2?
+# $FROMFTPSYNC - set to true if we run from within ftpsync.
 #
 # This function assumes that the variable LOG is set to a directory where
 # logfiles can be written to.
@@ -38,6 +40,8 @@ signal () {
     PUSHTYPE=${PUSHTYPE:-"all"}
     PUSHARCHIVE=${PUSHARCHIVE:-""}
     PUSHCB=${PUSHCB:-""}
+    PUSHKIND=${PUSHKIND:-"all"}
+    FROMFTPSYNC=${FROMFTPSYNC:-"false"}
 
     # And now get # back to space...
     SSHOPTS=${SSHOPTS/\#/ }
@@ -45,23 +49,26 @@ signal () {
     # Defaults we always want, no matter what
     SSH_OPTIONS="-o user=${USERNAME} -o BatchMode=yes -o ServerAliveInterval=45 -o ConnectTimeout=45 -o PasswordAuthentication=no"
 
+    # If there are userdefined ssh options, add them.
     if [ -n "${SSH_OPTS}" ]; then
         SSH_OPTIONS="${SSH_OPTIONS} ${SSH_OPTS}"
     fi
 
+    # Does this machine need a special key?
     if [ -n "${SSHKEY}" ]; then
         SSH_OPTIONS="${SSH_OPTIONS} -i ${SSHKEY}"
     fi
 
+    # Does this machine have an extra own set of ssh options?
     if [ -n "${SSHOPTS}" ]; then
         SSH_OPTIONS="${SSH_OPTIONS} ${SSHOPTS}"
     fi
 
+    # Set the protocol version
     if [ ${SSHPROTO} -ne 1 ] && [ ${SSHPROTO} -ne 2 ] && [ ${SSHPROTO} -ne 99 ]; then
         # Idiots, we only want 1 or 2. Cant decide? Lets force 2.
         SSHPROTO=2
     fi
-
     if [ -n "${SSHPROTO}" ] && [ ${SSHPROTO} -ne 99 ]; then
         SSH_OPTIONS="${SSH_OPTIONS} -${SSHPROTO}"
     fi
@@ -69,64 +76,82 @@ signal () {
     date -u >> "${LOGDIR}/${MIRROR}.log"
 
     PUSHARGS=""
+    # Archive is non-empty, so tell the downstreams
     if [ -n "${PUSHARCHIVE}" ]; then
         PUSHARGS="${PUSHARGS} sync:archive:${PUSHARCHIVE}"
     fi
+    # We have a callback wish, tell downstreams
     if [ -n "${PUSHCB}" ]; then
         PUSHARGS="${PUSHARGS} sync:callback"
     fi
+    # If we are running an mhop push AND our downstream is one to receive it, tell it.
+    if [ "xmhopx" = "x${PUSHKIND}x" ] && [ "xmhopx" = "x${PUSHTYPE}x" ]; then
+        PUSHARGS="${PUSHARGS} sync:mhop"
+    fi
 
     if [ "xallx" = "x${PUSHTYPE}x" ]; then
-        # Default normal "fire and forget" push
+        # Default normal "fire and forget" push. We background that, we do not care about the mirrors doings
         echo "Sending normal push" >> "${LOGDIR}/${MIRROR}.log"
         PUSHARGS1="sync:all"
         ssh $SSH_OPTIONS "${HOSTNAME}" "${PUSHARGS} ${PUSHARGS1}" >>"${LOGDIR}/${MIRROR}.log" 2>&1 &
-    elif [ "xstagedx" = "x${PUSHTYPE}x" ]; then
-        # Want a staged push. Fine, lets do that
+    elif [ "xstagedx" = "x${PUSHTYPE}x" ] || [ "xmhopx" = "x${PUSHTYPE}x" ]; then
+        # Want a staged push. Fine, lets do that. Not backgrounded. We care about the mirrors doings.
         echo "Sending staged push" >> "${LOGDIR}/${MIRROR}.log"
 
-        # Step1: Do a push to only sync stage1, do not background
-        PUSHARGS1="sync:stage1"
-        ssh $SSH_OPTIONS "${HOSTNAME}" "${PUSHARGS} ${PUSHARGS1}" >>"${LOGDIR}/${MIRROR}.log" 2>&1
-        touch "${PUSHLOCKOWN}"
-
-        # Step2: Wait for all the other "lock"files to appear.
-        tries=0
-        # We do not wait forever
-        while [ ${tries} -lt ${PUSHDELAY} ]; do
-            total=0
-            found=0
-            for file in ${PUSHLOCKS}; do
-                total=$((total + 1))
-                if [ -f ${file} ]; then
-                    found=$((found + 1))
+        # Only send stage1 if we havent already send it. When called with stage2, we already did.
+        if [ "xstage2x" != "x${PUSHKIND}x" ]; then
+            # Step1: Do a push to only sync stage1, do not background
+            PUSHARGS1="sync:stage1"
+            ssh $SSH_OPTIONS "${HOSTNAME}" "${PUSHARGS} ${PUSHARGS1}" >>"${LOGDIR}/${MIRROR}.log" 2>&1
+            touch "${PUSHLOCKOWN}"
+
+            # Step2: Wait for all the other "lock"files to appear.
+            tries=0
+            # We do not wait forever
+            while [ ${tries} -lt ${PUSHDELAY} ]; do
+                total=0
+                found=0
+                for file in ${PUSHLOCKS}; do
+                    total=$((total + 1))
+                    if [ -f ${file} ]; then
+                        found=$((found + 1))
+                    fi
+                done
+                if [ ${total} -eq ${found} ] || [ -f "${LOCKDIR}/all_stage1" ]; then
+                    touch "${LOCKDIR}/all_stage1"
+                    break
                 fi
+                tries=$((tries + 5))
+                sleep 5
             done
-            if [ ${total} -eq ${found} ] || [ -f "${LOCKDIR}/all_stage1" ]; then
-                touch "${LOCKDIR}/all_stage1"
-                break
+            # In case we did not have all PUSHLOCKS and still continued, note it
+            # This is a little racy, especially if the other parts decide to do this
+            # at the same time, but it wont hurt more than a mail too much, so I don't care much
+            if [ ${tries} -ge ${PUSHDELAY} ]; then
+            echo "Failed to wait for all other mirrors. Failed ones are:" >> "${LOGDIR}/${MIRROR}.log"
+                for file in ${PUSHLOCKS}; do
+                    if [ ! -f ${file} ]; then
+                        echo "${file}" >> "${LOGDIR}/${MIRROR}.log"
+                        error "Missing Pushlockfile ${file} after waiting ${tries} second, continuing"
+                    fi
+                done
             fi
-            tries=$((tries + 5))
-            sleep 5
-        done
-        # In case we did not have all PUSHLOCKS and still continued, note it
-        # This is a little racy, especially if the other parts decide to do this
-        # at the same time, but it wont hurt more than a mail too much, so I don't care much
-        if [ ${tries} -ge ${PUSHDELAY} ]; then
-        echo "Failed to wait for all other mirrors. Failed ones are:" >> "${LOGDIR}/${MIRROR}.log"
-            for file in ${PUSHLOCKS}; do
-                if [ ! -f ${file} ]; then
-                    echo "${file}" >> "${LOGDIR}/${MIRROR}.log"
-                    error "Missing Pushlockfile ${file} after waiting ${tries} second, continuing"
-                fi
-            done
+            rm -f "${PUSHLOCKOWN}"
         fi
-        rm -f "${PUSHLOCKOWN}"
 
-        # Step3: It either timed out or we have all the "lock"files, sync stage2
-        PUSHARGS2="sync:stage2"
-        echo "Now doing the second stage push" >> "${LOGDIR}/${MIRROR}.log"
-        ssh $SSH_OPTIONS "${HOSTNAME}" "${PUSHARGS} ${PUSHARGS2}" >>"${LOGDIR}/${MIRROR}.log" 2>&1
+        # Step3: It either timed out or we have all the "lock"files, do the rest
+        # If we are doing mhop AND are called from ftpsync - we now exit.
+        # That way we notify our uplink that we and all our clients are done with their
+        # stage1. It can then finish its own, and if all our upstreams downlinks are done,
+        # it will send us stage2.
+        # If we are not doing mhop or are not called from ftpsync, we start stage2
+        if [ "xtruex" = "x${FROMFTPSYNC}x" ] && [ "xmhopx" = "x${PUSHKIND}x" ]; then
+            return
+        else
+            PUSHARGS2="sync:stage2"
+            echo "Now doing the second stage push" >> "${LOGDIR}/${MIRROR}.log"
+            ssh $SSH_OPTIONS "${HOSTNAME}" "${PUSHARGS} ${PUSHARGS2}" >>"${LOGDIR}/${MIRROR}.log" 2>&1
+        fi
     else
         # Can't decide? Then you get nothing.
         return
@@ -181,7 +206,7 @@ hook () {
 
 # Return the list of 2-stage mirrors.
 get2stage() {
-    egrep '^staged' "${MIRRORS}" | {
+    egrep '^(staged|mhop)' "${MIRRORS}" | {
         while read MTYPE MLNAME MHOSTNAME MUSER MPROTO MKEYFILE; do
             PUSHLOCKS="${LOCKDIR}/${MLNAME}.stage1 ${PUSHLOCKS}"
         done
diff --git a/etc/runmirrors.mirror.sample b/etc/runmirrors.mirror.sample
index 2f841bb..744ad98 100644
--- a/etc/runmirrors.mirror.sample
+++ b/etc/runmirrors.mirror.sample
@@ -7,11 +7,15 @@
 #
 # Type ShortName HostName User -$SOMESSHOPTION
 #
+# The fields Type, ShortName, HostName and User are *mandantory*.
 #
-# Type is either all or staged, meaning:
+# Type is either all, staged or mhop, meaning:
 # all    - do a "normal" push. Trigger them, go on.
 # staged - do a two-stage push, waiting for them after stage 2(and all others that
 #          are staged) before doing stage2
+# mhop   - send a multi-hop staged push. This will tell the mirror to initiate
+#          a mhop/stage1 push to its staged/mhop mirrors and then exit.
+#          When all mhop got back we then send the stage2 through to them.
 #
 # ShortName will be used as a shorthand in logfile outputs and for the logfile
 # where every ssh output gets redirected to.
@@ -33,21 +37,33 @@
 # want. There is currently no way around this, as that would mean
 # dropping backward compatibility.
 #
+# Backwards compatibility:
+# An older runmirrors script will NOT run with a newer runmirrors.mirror file, but
+# a new runmirrors can run with an old runmirrors.mirror file. This should make updates
+# painless.
+#
 # Examples:
 # all eu.puccini puccini.debian.org archvsync 2
 #
-# will push puccini.debian.org, user archvsync, using ssh protocol 2 and the specified ssh key.
-#
+# -> will push puccini.debian.org, user archvsync, using ssh protocol 2
+#    and the globally configured ssh key.
 #
 # all eu.puccini puccini.debian.org archvsync -p 2222
 #
-# which will do the same as above, but use port 2222 to connect to.
+# -> will do the same as above, but use port 2222 to connect to.
+#
+# staged eu.puccini puccini.debian.org archvsync
+# staged eu.powell powell.debian.org archvsync
+#
+# -> will push both puccini and powell in stage1, waiting for both to
+#    finish stage1 before stage2 gets pushed. The username will be archvsync.
 #
-# staged eu.puccini puccini.debian.org
-# staged eu.powell powell.debian.org
+# staged eu.puccini puccini.debian.org archvsync
+# mhop eu.powell powell.debian.org archvsync
 #
-# will push both puccini and powell in stage1, waiting for both to
-# finish stage1 before stage2 gets pushed.
+# -> will do the same as above, but powell gets told about mhop and can then
+#    push its own staged/mhop mirrors before returning. When both returned
+#    then stage2 is sent to both.
 #
 # One special value is allowed: DELAY
 # This word has to be on a line itself, followed by a space and a number.