[solved] Strange mksquashfs4 issue

So I nailed it.

First I tracked it down to be openwrt-specific as if I build squashfs tools without openwrt patches mksquashfs create byte-to-byte identical FS archives on cygwin and linux if fed with the same source set of files. Then I applied OpenWrt-supplied patches and compared the differences in the images created under cygwin vs linux. The only differences were in the 12 bytes header known as COMPRESSOR OPTIONS. Rest of the file was bit-to-bit identical. Looking and headers differences it was easy to spot that for some yet unknown reason on cygwin compressor options header was generated in big endian while on linux is was in expected little endian format.

From here it was a quick and easy ride to find the problem. It is a bug in OpenWrt's 160-expose_lzma_xz_options.patch patch. Here is a fix for it:

diff --git a/tools/squashfs4/patches/160-expose_lzma_xz_options.patch b/tools/squashfs4/patches/160-expose_lzma_xz_options.patch
index 9e1c1fbb..209a91c8 100644
--- a/tools/squashfs4/patches/160-expose_lzma_xz_options.patch
+++ b/tools/squashfs4/patches/160-expose_lzma_xz_options.patch
@@ -26,15 +26,15 @@
 +
 +#include <stdint.h>
 +
-+#ifndef linux
++#if defined(linux) || defined(__CYGWIN__)
++#include <endian.h>
++#else
 +#ifdef __FreeBSD__
 +#include <machine/endian.h>
 +#endif
 +#define __BYTE_ORDER BYTE_ORDER
 +#define __BIG_ENDIAN BIG_ENDIAN
 +#define __LITTLE_ENDIAN LITTLE_ENDIAN
-+#else
-+#include <endian.h>
 +#endif
 +
 +

As typical for these cases it wasn't a cygwin's fault, it was software fault treating cygwin differently compared to linux. About 90% of cases I've seen with problems getting some linux software working under cygwin were due to software's build configuration system trying to do something special for cygwin while it was appropriate just to leave it alone and behave the same was as under linux.

With the above patch for the patch in place filesystem images produced under cygwin are bit to bit identical to images produced under linux which is the result I expect to get from a properly written software. Case closed.