Using zorder to pack files in repeatable order
Scenario
A customer has a large number of small files that do not change often. The nature of the application causes the files to be regularly rewritten, without the content being changed. This causes the file modification dates to change.
Issues posed by scenario
AhsayOBS may use the modification date to determine whether a file has changed. In this scenario, a change will be detected in every file; but because there is a minimum delta size cutoff, every small unchanged file will be reuploaded, incurring an unexpectedly large upload.
You may attempt to work around this by packing the source folder into a zip file or similar as part of a pre-backup command. However, most zip tools (tar, 7z etc) do not make any attempt to consistently order the input files. If the source files are regenerated periodically without any actual data change, the file inodes may differ, the operating system will provide the files in a different order, and the generated zip file may be ordered entirely differently, once again incurring an unexpectedly large upload.
Introducing zorder
zorder is a tool that zips a folder using a consistent file order, to alleviate the above issues. MyClient makes the zorder tool freely available for Windows; versions for OS X and Linux are available upon request.
To maximise the efficiency of AhsayOBS's own delta and compression systems, zorder does not compress the generated zip file.
Download
zorder-1.1.zip (2015-12-14)
- Feature: Support zip files larger than 2GB
- Enhancement: More user-friendly help message
zorder-1.0.zip (2014-04-30)
- Initial public release
Command-line Usage
ABOUT
zorder 1.1, Copyright (2014-2015) MyClientGlobal.com Ltd
Pack a directory in deterministic order, to minimise incremental deltas.
USAGE
zorder.exe source-directory output-file.zip