Back when we started moving backups to different servers instead of the main server we had written a script to clone sites one at a time using a PHP instance that would load the server (picking one at random) and if a file is not in the local system it would clone the zip from the server over to the backup server. The problem with this is because if network bugs when they happen a file would stop and not finish. So we added a .temp check for new transfers not finished but found that they would finish on network drop and just finish the zip even if not complete.
Now we do things a little different and have better logging, and today we added a bunch of new checks including one that does a byte for byte check when the file is finished before marking as a good .zip and removing the .temp flag. We also unpack the zip in a folder to see if it will open and then clear that new folders contents as its not needed anymore. This allows us to keep backups as of tonight (some may be from before the changes) and should be more stable.
This was really important as we now restore from backups and need to trust the backup file is sound.
If you have questions on how any of this works just comment and we would be happy to answer any questions.