CAKE w/ Adaptive Bandwidth [October 2021 to September 2022]

moeller0 · 2021-12-07T14:38:29.919Z

I wonder whether https://subversion.xray.aps.anl.gov/trac/dbw/browser/trunk/Xalert/stevens/tcpipi/icmptime/icmptime.c might not be helpful, instead of going for the full hping3 we might just get a small binary that does get us ICMP timestamps, not sure about the license/usability of that code for our purpose though. Piggy-backing on an existing maintained project is probably better

Lynx · 2021-12-07T14:42:42.106Z

icmptime.c:20:10: fatal error: netinet/ip_var.h: No such file or directory
   20 | #include <netinet/ip_var.h>
      |          ^~~~~~~~~~~~~~~~~~
compilation terminated.

This is on WSL2. Not sure how it would fare for OpenWrt.

moeller0 · 2021-12-07T14:54:04.489Z

Yeah the code is ancient and does not actually assume Linux to be a thing (effortlessly explained by the code's age), I mostly linked that because the whole thing seems rather short and some of us wanted parts of this project to move to C-code .

_FailSafe · 2021-12-07T15:08:56.496Z

Lynx:

then if we can get hping3 made an Official OpenWrt package

I'm working on this now, but slow going. Will update if/when I make some headway...

Lynx · 2021-12-07T15:10:01.220Z

Ace - thanks for trying. Don't forget to post issues on this thread because help is pretty fast. It's like placing a match next to a tinderbox. So much eagerness from many different parties. It's quite amazing really.

moeller0 · 2021-12-07T15:12:29.802Z

For fun, here is @tievolu's complete list of type13 reflectors from a well connected place:

  Pings done = 114    RTT/2 min =   3  |                     Request  |                       Reply  |                  Round Trip  |  
-------------------------------------------------------------------------------------------------------------------------------------
| Ping Target      |           Offset  |  Current | Average ? jitter  |  Current | Average ? jitter  |  Current | Average ? jitter  |  
-------------------------------------------------------------------------------------------------------------------------------------
| 27.112.79.80     |                0  |    ERROR |     N/A ? N/A     |    ERROR |     N/A ? N/A     |    ERROR |     N/A ? N/A     |  
| 79.110.170.43    |                0  |    ERROR |     N/A ? N/A     |    ERROR |     N/A ? N/A     |    ERROR |     N/A ? N/A     |  
| 146.112.61.106   |              -60  |        3 |    3.95 ? 0.75    |        4 |    3.67 ? 0.55    |        7 |    7.62 ? 0.70    |  
| 149.112.112.112  |              -40  |        3 |    3.71 ? 0.80    |        4 |    4.05 ? 0.95    |        7 |    7.76 ? 0.35    |  
| 208.67.220.220   |              -59  |        4 |    3.76 ? 0.70    |        3 |    4.00 ? 0.70    |        7 |    7.76 ? 0.40    |  
| 208.67.222.222   |              -63  |        4 |    3.62 ? 0.95    |        3 |    4.14 ? 1.00    |        7 |    7.76 ? 0.75    |  
| 9.9.9.10         |              -60  |        3 |    3.76 ? 0.95    |        5 |    4.05 ? 1.05    |        8 |    7.81 ? 0.20    |  
| 149.112.112.11   |              -60  |        4 |    3.67 ? 1.25    |        4 |    4.14 ? 1.20    |        8 |    7.81 ? 0.25    |  
| 146.112.41.3     |              -62  |        5 |    3.71 ? 0.95    |        3 |    4.10 ? 1.15    |        8 |    7.81 ? 0.40    |  
| 9.9.9.9          |              -62  |        3 |    3.67 ? 1.20    |        5 |    4.14 ? 1.30    |        8 |    7.81 ? 0.50    |  
| 94.140.15.15     |              -63  |        5 |    3.81 ? 0.90    |        3 |    4.05 ? 0.90    |        8 |    7.86 ? 0.20    |  
| 94.140.15.16     |              -62  |        3 |    3.62 ? 0.95    |        5 |    4.29 ? 0.95    |        8 |    7.90 ? 0.20    |  
| 149.112.112.10   |              -59  |        3 |    3.52 ? 0.90    |        5 |    4.38 ? 1.10    |        8 |    7.90 ? 0.50    |  
| 94.140.14.14     |              -63  |        3 |    3.86 ? 1.05    |        5 |    4.14 ? 1.05    |        8 |    8.00 ? 0.00    |  
| 94.140.14.15     |              -63  |        3 |    3.67 ? 1.10    |        5 |    4.43 ? 1.10    |        8 |    8.10 ? 0.20    |  
| 146.112.41.2     |              -59  |        3 |    3.57 ? 0.80    |        5 |    4.95 ? 2.50    |        8 |    8.52 ? 2.10    |  
| 9.9.9.11         |              -61  |        3 |    5.05 ? 2.95    |        4 |    4.00 ? 1.20    |        7 |    9.05 ? 2.75    |  
| 185.228.168.10   |              321  |        4 |    5.71 ? 4.95    |        4 |    4.48 ? 0.95    |        8 |   10.19 ? 4.20    |  
| 185.233.106.232  |            26088  |        3 |    4.57 ? 1.40    |        7 |    6.48 ? 1.35    |       10 |   11.05 ? 0.25    |  
| 185.233.107.4    |            12251  |        4 |    4.19 ? 1.70    |        7 |    7.00 ? 1.80    |       11 |   11.19 ? 0.40    |  
| 185.228.168.168  |              321  |        3 |    6.81 ? 6.00    |        5 |    4.62 ? 1.75    |        8 |   11.43 ? 6.85    |  
| 185.235.81.1     |              -59  |        3 |    3.90 ? 1.50    |        8 |    8.05 ? 1.60    |       11 |   11.95 ? 0.40    |  
| 176.9.1.117      |              -61  |        7 |    5.10 ? 2.55    |        6 |    7.29 ? 2.50    |       13 |   12.38 ? 0.55    |  
| 74.82.42.42      |              -60  |        9 |    5.71 ? 3.80    |        6 |    6.71 ? 2.15    |       15 |   12.43 ? 2.25    |  
| 176.9.93.198     |              -59  |        6 |    4.71 ? 2.60    |        6 |    8.43 ? 3.80    |       12 |   13.14 ? 2.40    |  
| 185.95.218.42    |              -62  |        5 |    5.38 ? 2.40    |        9 |    8.76 ? 2.40    |       14 |   14.14 ? 0.20    |  
| 185.95.218.43    |              -63  |        3 |    4.57 ? 2.30    |       11 |    9.71 ? 1.90    |       14 |   14.29 ? 0.60    |  
| 193.17.47.1      |              -62  |        7 |    4.81 ? 2.60    |        8 |    9.62 ? 2.65    |       15 |   14.43 ? 0.45    |  
| 185.43.135.1     |              -63  |        4 |    4.62 ? 2.05    |       10 |   10.05 ? 2.50    |       14 |   14.67 ? 1.05    |  
| 93.104.213.190   |              -63  |        3 |    5.05 ? 2.20    |       11 |   10.24 ? 2.05    |       14 |   15.29 ? 2.05    |  
| 130.59.31.248    |              -63  |        8 |    5.52 ? 3.45    |        9 |   11.48 ? 3.45    |       17 |   17.00 ? 0.00    |  
| 130.59.31.251    |              -64  |        4 |    4.57 ? 1.65    |       13 |   12.52 ? 1.75    |       17 |   17.10 ? 0.20    |  
| 185.150.99.255   |              -65  |        3 |    6.38 ? 3.55    |       12 |   11.10 ? 4.55    |       15 |   17.48 ? 3.60    |  
| 213.196.191.96   |              -63  |    ERROR |    6.33 ? 4.90    |    ERROR |   12.38 ? 5.20    |    ERROR |   18.71 ? 7.30    |  
| 193.19.108.3     |          -228670  |        5 |    5.62 ? 3.40    |       15 |   13.81 ? 3.50    |       20 |   19.43 ? 0.40    |  
| 5.1.66.255       |              -64  |        4 |    4.86 ? 1.85    |       15 |   14.86 ? 1.95    |       19 |   19.71 ? 0.40    |  
| 194.242.2.3      |          -228670  |        3 |    5.38 ? 4.00    |       16 |   14.43 ? 2.70    |       19 |   19.81 ? 1.60    |  
| 146.255.56.98    |              223  |        3 |    6.57 ? 3.90    |       17 |   13.57 ? 4.10    |       20 |   20.14 ? 0.20    |  
| 212.78.94.40     |              -66  |        4 |    4.52 ? 2.25    |       16 |   15.76 ? 2.35    |       20 |   20.29 ? 0.50    |  
| 193.19.108.2     |          -228671  |        4 |    4.24 ? 1.65    |       18 |   18.05 ? 2.05    |       22 |   22.29 ? 0.60    |  
| 185.216.27.142   |              -63  |       10 |    7.48 ? 5.95    |        7 |   14.81 ? 5.75    |       17 |   22.29 ? 8.00    |  
| 194.242.2.2      |          -228670  |        5 |    4.95 ? 2.20    |       18 |   17.76 ? 2.25    |       23 |   22.71 ? 0.45    |  
| 185.64.79.5      |              -66  |        3 |    5.38 ? 3.70    |       24 |   17.81 ? 4.00    |       27 |   23.19 ? 0.40    |  
| 37.252.230.153   |              -67  |        3 |    5.67 ? 4.45    |       21 |   18.05 ? 4.25    |       24 |   23.71 ? 0.40    |  
| 209.250.226.191  |              -70  |        6 |    4.43 ? 2.40    |       18 |   19.76 ? 2.30    |       24 |   24.19 ? 0.30    |  
| 46.227.200.55    |              -67  |        5 |    6.05 ? 3.65    |       20 |   19.10 ? 3.65    |       25 |   25.14 ? 0.20    |  
| 46.227.200.54    |              -67  |        5 |    4.95 ? 2.70    |       20 |   20.52 ? 1.70    |       25 |   25.48 ? 1.00    |  
| 37.252.232.95    |              -67  |        9 |    5.10 ? 2.95    |       17 |   20.43 ? 2.80    |       26 |   25.52 ? 0.65    |  
| 95.216.229.153   |              -75  |        4 |    5.10 ? 1.75    |       28 |   26.24 ? 2.25    |       32 |   31.33 ? 1.10    |  
| 185.134.196.55   |              -63  |        3 |    4.62 ? 2.40    |       12 |   30.52 ? 12.50   |       15 |   35.14 ? 10.60   |  
| 185.134.196.54   |              -63  |        8 |    4.86 ? 2.35    |        7 |   35.05 ? 12.65   |       15 |   39.90 ? 10.50   |  
| 46.239.223.80    |              -74  |        4 |    5.00 ? 1.90    |       51 |   49.62 ? 2.20    |       55 |   54.62 ? 0.40    |  
| 35.245.234.132   |              -95  |        9 |   11.57 ? 1.85    |       82 |   79.62 ? 1.95    |       91 |   91.19 ? 0.30    |  
| 35.230.160.38    |             -119  |       26 |   16.95 ? 2.15    |       66 |   74.81 ? 2.05    |       92 |   91.76 ? 0.40    |  
| 149.112.122.10   |              -99  |        7 |    8.67 ? 2.85    |       87 |   85.19 ? 2.75    |       94 |   93.86 ? 0.30    |  
| 149.112.121.20   |             -101  |        5 |    6.81 ? 2.50    |       89 |   87.14 ? 2.50    |       94 |   93.95 ? 0.10    |  
| 149.112.122.20   |              -99  |        5 |    8.38 ? 2.20    |       89 |   85.62 ? 2.40    |       94 |   94.00 ? 0.20    |  
| 149.112.121.10   |              -98  |        6 |    9.33 ? 2.25    |       88 |   84.71 ? 2.15    |       94 |   94.05 ? 0.30    |  
| 149.112.122.30   |             -101  |        4 |    6.14 ? 4.70    |       90 |   88.10 ? 4.80    |       94 |   94.24 ? 0.50    |  
| 149.112.121.30   |             -101  |        3 |    4.19 ? 1.70    |       92 |   90.10 ? 1.95    |       95 |   94.29 ? 0.45    |  
| 35.237.220.84    |              -98  |       21 |   21.76 ? 2.80    |       81 |   80.14 ? 2.90    |      102 |  101.90 ? 0.50    |  
| 35.231.247.227   |              -95  |        9 |   13.52 ? 3.05    |       92 |   88.43 ? 3.15    |      101 |  101.95 ? 0.90    |  
| 174.138.29.175   |              -61  |       85 |   87.10 ? 2.75    |       88 |   85.33 ? 2.90    |      173 |  172.43 ? 0.35    |  
| 139.162.112.47   |              -14  |    ERROR |  168.90 ? 3.00    |    ERROR |   84.05 ? 3.05    |    ERROR |  252.95 ? 0.45    |  
| 172.104.93.80    |              -15  |      178 |  179.19 ? 1.65    |       88 |   86.38 ? 1.45    |      266 |  265.57 ? 0.60    |  
| 101.102.103.104  |               -1  |      195 |  200.62 ? 5.15    |       88 |   82.38 ? 5.15    |      283 |  283.00 ? 0.00    |  
| 101.101.101.101  |               -8  |      200 |  203.05 ? 2.65    |       87 |   84.86 ? 3.00    |      287 |  287.90 ? 0.65    |  
-------------------------------------------------------------------------------------------------------------------------------------

note how small the offsets are to most of these....

Lynx · 2021-12-07T15:14:08.899Z

Do the offsets seem to be staying the same roughly between runs?

dlakelan · 2021-12-07T15:27:04.258Z

moeller0:

note how small the offsets are to most of these....

I wonder if systemd is responsible for this. By default it comes with an ntp client whereas before that you had to actually install ntpd

_FailSafe · 2021-12-07T15:33:12.059Z

Not working yet, but posting this here in case anyone wants to work in parallel toward getting this to compile: https://github.com/Fail-Safe/hping3/blob/main/Makefile

(I'm not a Makefile expert, so please don't take anything already in the file as correct or even necessary.)

moeller0 · 2021-12-07T15:39:49.727Z

Lynx:

Do the offsets seem to be staying the same roughly between runs

On first glance they look relatively stable, but the script prints out one of these tables (always ordered by Round Trip I believe, so the exact sequence is not fixed) per iteration making it tedious to track the offset per IP-address.

dlakelan:

I wonder if systemd is responsible for this. By default it comes with an ntp client whereas before that you had to actually install ntpd

Since I was running this on my macos laptop outside of my home network, systemd would need to be active at the reflector sites, and I do have my reservations about systemd's prevalence on servers...

dlakelan · 2021-12-07T16:03:21.831Z

moeller0:

systemd's prevalence on servers

it's the standard for Debian, which is I think a very common server distribution.

tievolu · 2021-12-07T16:30:21.876Z

Lynx:

Do the offsets seem to be staying the same roughly between runs?

In my experience with my own scripts, the offsets are generally very small (mostly <5ms) and stable once ntpd has settled on an accurate time for your system clock (i.e. about 24 hours after a reboot).

In terms of offset size there are some outliers with much larger offsets, as shown in @moeller0's output above (presumably these hosts aren't sending back timestamps based on milliseconds-since-midnight-UTC), but even these offsets are pretty stable.

moeller0:

On first glance they look relatively stable, but the script prints out one of these tables (always ordered by Round Trip I believe, so the exact sequence is not fixed) per iteration making it tedious to track the offset per IP-address.

Just comment out this line to remove the sorting by RTT:

my @results_to_print = sort {&get_sort_string($a) cmp &get_sort_string($b)} @results_to_print;

BlueKingMuch · 2021-12-07T16:45:52.211Z

just for update on compiling i give up. (at least for today).

I tried first with the arm compiler, then with the toolkit (seems to be the makefile which does problems, but i don't understand it completely), in both ways I am differently stuck.

Hope you or another person has more luck or knowledge. or both.

moeller0 · 2021-12-07T16:48:45.319Z

tievolu:

(presumably these hosts aren't sending back timestamps based on milliseconds-since-midnight-UTC)

The bunch of hosts with offset around -228670 all resolve to the same owner in Sweden via whois, well possible that they have a 3-4 minute offset of their main clock....

Then, thank you very much for sharing your code and the timestamp reflector list.

tievolu:

Just comment out this line to remove the sorting by RTT:

my @results_to_print = sort {&get_sort_string($a) cmp &get_sort_string($b)} @results_to_print;

Thanks, I guess I need to export the data in a way that is easier to parse for post processing....

tievolu · 2021-12-07T17:09:34.523Z

Lynx:

Excited by the prospect of being able to show proof of concept by saturating upload and showing the upload time increase (whilst download time remains the same), then later saturating download and showing the download time increase (whilst upload time remains the same).

That would be super rewarding / cool!

It works well in my monster perl script. For example, here's the script detecting bufferbloat in the upload direction:

Wed Dec  1 01:06:50 2021 [2034-00056166]: ---------------------------------------------------------------------------
Wed Dec  1 01:06:50 2021 [2034-00056166]: |  Bandwidth usage  |      8.26.56.26 |  129.250.35.251 |  208.67.222.222 |
Wed Dec  1 01:06:50 2021 [2034-00056166]: ---------------------------------------------------------------------------
Wed Dec  1 01:06:50 2021 [2034-00056166]: | TX Mbps | RX Mbps |  Rqst  |  Rply  |  Rqst  |  Rply  |  Rqst  |  Rply  |
Wed Dec  1 01:06:50 2021 [2034-00056166]: ---------------------------------------------------------------------------
...
Wed Dec  1 01:07:40 2021 [2034-00056189]: |  34.578 |   0.774 |   13   |   14   |   16   |   12   |   15   |   14   |
Wed Dec  1 01:07:42 2021 [2034-00056190]: |  32.304 |   0.695 |   15   |   13   |   11   |   13   |   11   |   14   |
Wed Dec  1 01:07:44 2021 [2034-00056191]: |  29.493 |   0.638 |   13   |   13   |   14   |   13   |   12   |   14   |
Wed Dec  1 01:07:46 2021 [2034-00056192]: |  34.485 |   1.725 |   11   |   14   |   14   |   13   |   10   |   14   |
Wed Dec  1 01:07:48 2021 [2034-00056193]: |  33.479 |   0.721 |   14   |   13   |   13   |   13   |   13   |   13   |
Wed Dec  1 01:07:50 2021 [2034-00056194]: |  26.218 |   0.545 |   50 X |   14   |   55 X |   13   |   52 X |   14   |
Wed Dec  1 01:07:52 2021 [2034-00056195]: |  26.952 |   0.566 |   35 X |   13   |   32 X |   14   |   36 X |   14   |
Wed Dec  1 01:07:54 2021 [2034-00056196]: |  29.268 |   0.687 |   68 X |   13   |   66 X |   14   |   69 X |   14   |

Timestamped pings are used to separate out request (upload) and reply (download) latency in exactly the same way as the example script I posted a few weeks back.

Lynx · 2021-12-07T17:14:15.766Z

You are right. I forgot one of the main sources of a working technical solution to the problem of ascertaining direction of bufferbloat. Your contribution is so much appreciated in this respect, @tievolu! You cracked one of the major problems.

From memory you use nping, right? So did you face the issues identified above with nping? That it is a little slow? I have found results are improved by getting the tick duration really small, which seems impossible with nping.

tievolu · 2021-12-07T17:36:05.530Z

Lynx:

From memory you use nping, right? So did you face the issues identified above with nping? That it is a little slow? I have found results are improved by getting the tick duration really small, which seems impossible with nping.

Yeah, I use nping. I ping three reflectors once every two seconds, with the ping requests set to time out after one second. Now that you mention it I did find that things would break if I tried to do it more frequently than that, but for my own purposes two seconds is sufficient.

My script still uses the tactic of resting at max bandwidth and reducing it temporarily if latency gets bad, with "bad latency" defined as at least 3 out of the last 10 requests/replies exceeding 30ms for all reflectors.

Usually bad latency looks like the example above, with all three reflectors returning bad results at the same time, all back to back, and only one bandwidth reduction is needed to address it, so the script typically mitigates bufferbloat after about six seconds.

My script is very much tailored to my own connection and I haven't made any attempt to implement a moving ping baseline or any of the other clever stuff you guys are working on. Another thing to note is that I'm doing all this on a Qotom x86 box, so results might be different on more typical router hardware.

Lochnair · 2021-12-07T18:53:30.724Z

Lynx:

Also @Lochnair did you manage to get sprout working on your VPS setup?

Had a feeling this question would be waiting for me when I got home
But no not yet. I wanted to test on the router, and for that I need to break out a cross-compilation toolchain.

Safest way to get the dependencies right is to make a package for it I'd assume, so I'd gotta remember how that works again as well. Or I could always compile a static binary so there's no dependency issues.

And I'm not entirely sure how to use the thing, the notes in the fork helps, but I still don't know what the MAC addresses are for.

_FailSafe · 2021-12-07T20:26:43.996Z

Here's where I sit with compiling it. Not great to be honest...

Summary

user@4fbfcf8066d8:~/openwrt$ make package/network/utils/hping3/compile V=sc | tee build.log
make[1]: Entering directory '/home/user/openwrt'
make[2]: Entering directory '/home/user/openwrt/package/libs/toolchain'
echo "libc" >> /home/user/openwrt/staging_dir/target-x86_64_musl/pkginfo/toolchain.default.install
echo "libgcc" >> /home/user/openwrt/staging_dir/target-x86_64_musl/pkginfo/toolchain.default.install
echo "libatomic" >> /home/user/openwrt/staging_dir/target-x86_64_musl/pkginfo/toolchain.default.install
echo "libstdcpp" >> /home/user/openwrt/staging_dir/target-x86_64_musl/pkginfo/toolchain.default.install
echo "libpthread" >> /home/user/openwrt/staging_dir/target-x86_64_musl/pkginfo/toolchain.default.install
echo "librt" >> /home/user/openwrt/staging_dir/target-x86_64_musl/pkginfo/toolchain.default.install
make[2]: Leaving directory '/home/user/openwrt/package/libs/toolchain'
time: package/libs/toolchain/compile#0.14#0.18#0.31
make[2]: Entering directory '/home/user/openwrt/package/network/utils/hping3'
rm -f /home/user/openwrt/build_dir/target-x86_64_musl/hping3-2021-12-07-3547c769/.built
touch /home/user/openwrt/build_dir/target-x86_64_musl/hping3-2021-12-07-3547c769/.built_check
ccache_cc -O2 -ftree-vectorize -march=ivybridge -mfpmath=sse -mabi=sysv -pipe -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -fmacro-prefix-map=/home/user/openwrt/build_dir/target-x86_64_musl/hping3-2021-12-07-3547c769=hping3-2021-12-07-3547c769 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro -I/home/user/openwrt/staging_dir/target-x86_64_musl/usr/include -ffunction-sections -fdata-sections -o /home/user/openwrt/build_dir/target-x86_64_musl/hping3-2021-12-07-3547c769/hping3
/home/user/openwrt/staging_dir/toolchain-x86_64_gcc-11.2.0_musl/lib/gcc/x86_64-openwrt-linux-musl/11.2.0/../../../../x86_64-openwrt-linux-musl/bin/ld: /home/user/openwrt/staging_dir/toolchain-x86_64_gcc-11.2.0_musl/lib/gcc/x86_64-openwrt-linux-musl/11.2.0/../../../../x86_64-openwrt-linux-musl/lib/crt1.o: in function `_start_c':
/home/user/openwrt/build_dir/toolchain-x86_64_gcc-11.2.0_musl/musl-1.2.2/crt/crt1.c:18: undefined reference to `main'
collect2: error: ld returned 1 exit status
make[2]: *** [Makefile:60: /home/user/openwrt/build_dir/target-x86_64_musl/hping3-2021-12-07-3547c769/.built] Error 1
make[2]: Leaving directory '/home/user/openwrt/package/network/utils/hping3'
time: package/network/utils/hping3/compile#1.12#0.19#1.29
    ERROR: package/network/utils/hping3 failed to build.
make[1]: *** [package/Makefile:116: package/network/utils/hping3/compile] Error 1
make[1]: Leaving directory '/home/user/openwrt'
make: *** [/home/user/openwrt/include/toplevel.mk:230: package/network/utils/hping3/compile] Error 2

Lochnair · 2021-12-07T21:05:36.903Z

I think you need to run a make make package/network/utils/hping3/clean first to get all the errors.

Tried here myself and it's complaining about some byteorder file. The ./configure script does some weird stuff. It doesn't seem to handle cross compilation well.

This part is the one causing me grief:

CC=${CC:=cc}

echo build byteorder.c...
$CC byteorder.c -o byteorder || exit 1

INSTALL_MANPATH=`echo $MANPATH|cut -f1 -d:`
if [ "$INSTALL_MANPATH" = "" ]; then
        INSTALL_MANPATH="/usr/local/man"
fi
BYTEORDER=`./byteorder -m`

echo create byteorder.h...
cat > byteorder.h <<EOF
#ifndef __BYTEORDER_H
#define __BYTEORDER_H

EOF
echo \#ifndef $BYTEORDER >> byteorder.h
echo \#define $BYTEORDER >> byteorder.h
echo \#endif /\* $BYTEORDER \*/ >> byteorder.h
cat >> byteorder.h <<EOF

#endif /* __BYTEORDER_H */
EOF

It compiles byteorder with the cross compiler, and then tries to run it after. If you don't QEMU setup with binfmt, that's gonna fail hard. Gonna look at what the byteorder program does and see if we can be rid of it