I have a customer who needs a router with LTE support, and want to buy something modern for them. First step: go to OpenWRT table of Hardware, then, under "View the Table of Hardware other ways", click Dataclouds. I end up at https://openwrt.org/toh/views/dataclouds, great. Now, under "Modem", open LTE, and... https://openwrt.org/toh/views/toh_admin_modem_wlan?dataflt[0]=modem_%3DLTE which is an admin interface, not a view. In my opinion, this is a mistake, as the path to it clearly suggests that it is supposed to be a way to view the data.
The link does suit my needs, but the complaint is that the path to it is not obvious.
You can easily reach this page via https://openwrt.org/toh/views/start
The reason why https://openwrt.org/toh/views/toh_admin_modem_wlan is only accessible with wiki account: Search engine bots are dumb.
Example:
66.249.66.212 - - [15/May/2018:14:36:09 -0400] "GET /toh/views/toh_admin_modem_wlan?dataflt%5B74%5D=vlan_%3DNo&dataflt%5B148%5D=lede%20unsupported_%3DFailsafe%20mode%2C%20WiFi%205GHz%2C%20WiFi%202.4GHz%20partly&dataflt%5B158%5D=wlan%20hardware_%3DTI%20ACX111%20%28integrated%29&dataflt%5B159%5D=switch_%3DQualcomm%20Atheros%20TP9343&dataflt%5B160%5D=modem_%3DADSL&datasrt=wlan%20hardware HTTP/1.1" 200 21828 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.146 - - [15/May/2018:14:36:09 -0400] "GET /toh/views/toh_admin_modem_wlan?dataflt%5B74%5D=vlan_%3DNo&dataflt%5B145%5D=lede%20unsupported_%3DWiFi%202.4GHz%2C%20WiFi%205GHz%2C%20USB&dataflt%5B161%5D=modem_%3DMobile%20Wireless%2FCellular%20network&dataflt%5B164%5D=wlan%20hardware_%3DAtheros%20AR5416%20%28mini-PCI%29&dataflt%5B165%5D=switch_%3DRalink%20RT3050&datasrt=wlan%20hardware HTTP/1.1" 200 22054 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.148 - - [15/May/2018:14:36:09 -0400] "GET /toh/views/toh_admin_modem_wlan?dataflt%5B31%5D=modem_%3DDOCSIS&dataflt%5B74%5D=vlan_%3DNo&dataflt%5B155%5D=lede%20unsupported_%3DBluetooth&dataflt%5B156%5D=switch_%3Dyes&dataflt%5B160%5D=lede%20supported%20current%20rel_%3Dsnapshot&dataflt%5B161%5D=wlan%20hardware_%3DTI%20TNETW1130GVF%20%28VLYNQ%29&dataflt%5B162%5D=unsupported%20functions_%3Dmust%20be%20forced%20into%20booting%20from%20NOR%20flash&datasrt=oem%20device%20homepage%20url HTTP/1.1" 200 22495 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
Dumb bots can get stuck on certain ToH pages, going on forever with silly filtering and sorting.
googlebot, back in 2018, created 7 million hits and 115GB of traffic in one month.
robots.txt does exist and is effective for googlebot and others, but there are other searchengines (and people using wget) out there that just do not care about robots.txt.
The only way to stop searchengines getting stuck on those pages and creating useless traffic: Make them accessible only with wiki account.
This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.