(Better and faster) Table of Hardware

richb-hanover-priv · October 23, 2024, 9:00pm

Yes, there do seem to be a lot of variations in the Flash field. I worked with the fellow who set up the TechData pages, so I can give this background:

The template for TechData pages allows/recommends that people control-click to multi-select from a list. See, for example, the TechData page for this old, discontinued (not very popular) router: https://openwrt.org/toh/hwdata/mqmaker/mqmaker_witi_board_256mb
Click on the "Edit" tab at the lower left of the box listing the attributes to see the instructions.
IIRC, this flexibility was added to cater to routers that have multiple flash/storage devices.

So the trick will be to intuit what the person who created the Techdata page was thinking, and how to represent the device in the best/proper light.

Earlier this year, I sucked up all the data from the ToH (I don't remember how). Here is the contents of the Flash column (below - sorted, removing duplicates). You can see the data at: https://docs.google.com/spreadsheets/d/1j_Yv62tdYzvA4iU216W3B0XHswXv2U2LrxQ0QOB5-vE/edit?usp=sharing

I am going to review several of these entries to see what the original author might have been thinking when they selected multiple Flash values...

PS One saving grace in all this is that I bet these multiple-entry devices are for older devices that are no longer useful/interesting. Consequently, it may not matter as much how they are displayed...

"" (empty string)
1, 128NAND
1, 512NAND
2
2, 128NAND
2, microSD
4
4, 8
4, 8, 16
4, 128NAND
4, 4096
4, 4096, eMMC
4, 8192, eMMC, microSD
4, microSDXC
8
8, 16
8, 16, microSD
8, 32
8, 128NAND
8, 512NAND
8, 4096, eMMC
8, 8192, eMMC
8, microSD
8, more, than, 8GB, eMMC
8, SD
16
16, 32
16, 32, 64
16, 32, microSD
16, 32NAND, SDHC
16, 128NAND
16, 128NAND, microSDXC
16, 256
16, 256NAND
16, 512NAND, microSD
16, 1024NAND
16, 4096, eMMC
16, 4096NAND
16, 8192, eMMC
16, 8192, eMMC, microSDXC
16, eMMC, microSD
16, microSD
16, microSDHC
16, microSDXC
16, SD
32
32, 128
32, 128NAND
32, 128NAND, 8192, eMMC, microSD
32, 128NAND, microSD
32, 256NAND
32, 2048, eMMC
32, microSD
32, microSD, microSDHC, microSDXC
32, SD, SDHC, SDXC
32NAND
64
64NAND
64NAND, 128NAND
64NAND, microSD
128
128, 2048NAND
128, 4096
128, 4096, 512NAND
128, 8192, eMMC
128, SD
128NAND
128NAND, 512NAND
128NAND, microSD
128NAND, SD
256
256, SD
256NAND
256NAND, 512NAND
256NAND, SD
512
512NAND
512NAND, microSD
512NAND, SD
1024
1024, eMMC
1024, SD, SDHC
1024NAND
1024NAND, SD, microSD
2048NAND, microSD
2048NAND, microSDHC
4096
4096, eMMC
4096, eMMC, microSDHC
4096, eMMC, more, than, 8GB, eMMC, microSDHC
4096, eMMC, SDHC
4096NAND
4096NAND, microSD
4096NAND, SD, microSD
4096NAND, SDHC
8192
8192, eMMC
8192, eMMC, microSD
8192, eMMC, microSDHC
8192, eMMC, microSDXC
8192, eMMC, SD
8192, microSDHC
8192NAND, microSD
CF, card
eMMC
eMMC, microSD
Flash MB
microSD
microSD, microSDHC
microSD, microSDHC, microSDXC
microSDHC
microSDXC
more, than, 8GB
more, than, 8GB, eMMC
more, than, 8GB, eMMC, microSD, microSDHC, microSDXC
more, than, 8GB, NAND
SD
SDHC
SDHC, microSDHC

evs · October 23, 2024, 9:12pm

I think this is great!

If we're working on table of hardware.....

A better way to add a new device would be handy. I recently screwed up adding the JG927A for example....

Being able to handle something rather than "more than 20" gigabit ethernet ports would be nice?
(For example when sorting for the switches....)

Similarly, how will you handling PoE powered devices (PD) vs PoE sources (PSE's?)? What happens if it has multiple power supplies? (i.e. fortiap 421 is dual PoE input which is redundant/resilient and it has a 12v input too. Whilst hiveap 330 has PoE & 12v, but doesn't feature redundancy/resiliency?)

How are you handle "switches": we don't have a category in ToH. From what I understand JG924A gets picked up as a switch and it doesn't have the category filled. Whilst if it's put under "other" it doesn't end up in the switch field at the moment?

Only other thing I guess I can think of in terms of fields is "rack mount" and "dimensions". (i guess to generalise the rack mount: mounting options?)

evs · October 23, 2024, 9:37pm

Whilst I like the idea. It's fraught with danger if you get parsing text fields wrong....

Might be easier in the openwrt context. But part selectors regularly get the units and parsing incorrect.

For example I recently ran into PPM and PPB which gets you out by a factor of 1000!

richb-hanover-priv · October 23, 2024, 9:40pm

I think we need to be cautious about considering changes to the data in the TechData pages that underlies the Table of Hardware. This is a huge database, compiled over a decade, and with carefully collected and mildly inconsistent data.

There would be a ton of work involved in making any changes, primarily to avoid any backward incompatibilities.

If you think this is a project that you'd like to work on, I would encourage you to open a new topic to collect further feedback.

richb-hanover-priv · October 23, 2024, 9:46pm

I got curious: how many devices have multiple Flash values? And are they relevant to recent OpenWrt users? (The answers to these questions help us decide how hard to work to present an accurate picture in the ToH. As an (invalid) example, if all the devices with multiple values could only run 20.xx, we don't have to work very hard. But it's not going to be that simple.)

Toward that end, I pulled the data from that spreadsheet into a SQLite database, then wrote a query that pulled out only the routers that had multiple Flash values. I then sorted by Current Release firmware to see how "modern" they are. The results are in the Routers that have multiple Flash values tab of the spreadsheet above.

(For people who are curious, here's the PRQL query for the SQL database. Ask more if you want to get the SQLite database file, or to learn more about the query language.)

# PRQL query to retrieve routers that have "," in their Flash MB 
from Routers
select {
    `Device Type`,
    Brand,
    Model, 
    Availability,
    `Supported Current Rel`,
    `Flash MB`,
    `RAM MB`, 
}
filter (text.contains "," `Flash MB`)
sort {-`Supported Current Rel`}

slh · October 24, 2024, 12:14am

For system like the OpenWrt One this distinction could matter, because you can install OpenWrt to either of the different flash options. For devices like the nbg6817, where the 4 MB NOR flash is only used for the bootloader(-stages) and calibration, but in no capacity for anything touched by OpenWrt, it's more straight forward to ignore it (as given, at least for the device database) and to concentrate on the eMMC that actually gets used by OpenWrt (everything, kernel, rootfs, overlay) instead).

There are several quite recent devices in the former bracket though, NOR+NAND, NAND+eMMC, …

evs · October 24, 2024, 12:15am

Thanks for your fast response.
I think i ended up a bit ranty, so I put it under a hidden part. I tried to already edit for brevity but it ended up pretty long. Apologies.

I guess it's these points in summary:

valid input data is essential to have the front end work.
Parsing user string input is hard
valid data is also hard to add if we don't have the required drop downs for the wiki interface
Is it worthwhile investing logic in string parsing existing fields in the database, or is that better served on on improving database itself
how are PoE PSE's handled? I see issues with parsing power supplies especially
I already made a topic asking for assistance from a wiki admin regarding updating the wiki forms. I offered assistance in updating documentation on how to add to ToH hwdata.
Is tmomas available by forum still? I'd appreciate it if you could help get me in contact with a wiki admin. edit: looking at the new wiki account forum post, need to go for IRC? So I've tried that today...

Further discussion & images

No worries. Wasn't implying saying we need to go and modify existing columns.
I just think that changing the front end to do string parsing on text fields is also hard.
I don't know whether that's better than the other existing infrastructure where you can control click
options.

Bad input / not well defined data plus adding string parsing on top will result in inconsistent results even if we revamp the front end.

At least I believe It's worthwhile to discuss whether parsing the existing database is a "workaround" or the best path forward?
Or is doing improvements on the source data set better?

mm.
There is existing documentation on the wiki that if you want a field added to an existing column, you need to contact the wiki admin and create a new forum topic in the documentation section.

https://openwrt.org/meta/create_new_dataentry_page

Already made a thread:
HPE (or adding a new) brand on create new device entry page?

I already did that regarding HPE as a drop down for brand, as well as asking how the existing data ended up correct without a switch drop down and "hpe" brand drop down....

Your feedback regarding "if you have a problem with it you should do it yourself" is valid haha =)
Already offered in the other forum post....
Plus I gave it a go and ended up with causing an inconsistent data problem....

My experience with dokuwiki as anything but a user approximates zero.
I've hosted other web applicaions before, but not my strong suit.
I could certainly give it a go.

I think the approach there is to minimise
the cost of failure. Hence ask for help first....

Comes down to time IMO, I'd rather work on hardware than websites though =P I don't want to leave a mess for other people to clean up though.

I also don't want to break the wiki rules.
It appears the existing instructions for getting things into ToH can create issues if it's a switch and if it's not an existing brand.

Hence I did as instructed and posted on the forum asking the existing wiki admins for columns/forms to be updated?

At the moment when trying to add data, you can end up with inconsistent data.
I would suggest a parallel path to a better table of hardware is to make sure that data is easy to enter/correct as well as consistent?

I understand well that this is obviously volunteer time......
Not trying to say what "should" or "must" be changed. Nor saying "just" do this or "just" do that =P.

OP asking for feedback and questions and adding my unsolicited feedback haha.

Regarding backwards incompatibilities.
We currently don't have mounting options, nor dimensions as a column?
We do have an "outdoor" / indoor column?
I fail to see how you can get inconsistent data if the data doesn't exist.
(other than the fact that there will be empty fields, which is of course a problem with existing/all fields).

I do see the issue where if you add new columns without deprecating or doing a huge effort in manual/automatic parsing to move to a new schema you could just add to the inconsistent data problem in the short term....

My existing question still remains, how should/is this implementation going to handle multiple power supply types?
Looks like there's no OR or AND functionality like the statement above regarding parametric search.
(Better and faster) Table of Hardware - #23 by Search

I think that the parametric search and other types like that can be frustrating as you need to control click
lots of things and less than greater than on the string fields with units can give bad/inconsistent results. (i.e. power supplies would be a good example here, many terms including VAC, ~, AC, VDC, DC, "V", VRMS, mah, Wh, USB, PoE, PD, PoE Powered Device, IEC C13, IEC C10, IEC C15, IEC C8, IEC C7)

I would say yes regarding inconsistent data haha.

So it comes down to if you're willing to do string parsing to improve the front end. One still need good input data, or you end up having to do lots of exceptions etc.

I believe this power supply approach yields inconsistent results at the moment?
I can get the HPE PoE switches coming up as a PoE powered device for example?

Please see screen captures:

wxop · October 24, 2024, 7:22am

Thanks for your help, guys!

Here comes version 1.3

allows to correctly (I hope) sort/filter/search thru the 'flashmb' column
preloads images
changes clearSort button position
adds some example presets
some minor CSS tweaks

Regarding the 'flashmb' messy column, I used the following logic (function "_getFlashArrayBestValue"):

github.com

soif/OpenWrtTOH/blob/afa8c232d23e9d7f1d447b64729469928164cb29/src/www/static/js/toh_conf.js#L229


      
          // get the best value to use in sort/filter of the 'flashmb' column ----------------------
          function _getFlashArrayBestValue(arr){
          	// ignore these, just in case (because JS is SOOOOOOO sensitive)
          	if(arr == null){
          		return '';
          	}
          	if( typeof(arr) !="object" ){
          		return arr;
          	}
          
          	var target=0;	
          	// now we can walk into the array of values, without throwing a fatal error (did I ever said I love JS ?) .....
          	arr.forEach((v) => {
          		// 'microSD' or 'SD' means(?) we have Gigas available, so we rank as 128G
          		if(v.match(/microsd/i) || v.match(/^SD$/)){
          			//console.log('SD found in :'+v);
          			v=128*1024;
          		}
          		// eMMC size is unknown, but if it is alone, it maybe(?) means at least 1M(?)
          		else if(v.match(/eMMC/i) && arr.length==1){
          			v=1;

Please examine the function and tell me if it is a good compromise, and if something need to be improved/changed.

For non JS coders, this function is called on each cell that is to be sorted or filtered and returns the number to use, either from a member of the cell's array (after some cleanup), or by building one , ie for SD or eMMC. By simply reading the comments, you should be able to understand it.

wxop · October 25, 2024, 12:30pm

Can someone help to explain me what means "" and "-" values in columns (ie Yes/NO columns like "VLAN", or number/text columns like "SFP")
Does it means "Unknown", or "Not set", or something else?

more details in GH issue #2 , (if you're able to, please prefer to answer there.)

navossoc · October 25, 2024, 8:30pm

I just come here to say: nice work dude

's

wxop · October 26, 2024, 2:57am

Thank you.

HTH

wxop · October 26, 2024, 1:31pm

Enhanced TOH v1.4 has just been released:

thanks to @richb-hanover-priv suggestion in #3, the selected filter, features, view and columns are now populated in the URL, so you can bookmark/share your custom TOH views.
code has been refactored

Enjoy!

BTW: I desperately miss some stars on GH, and/or some likes in this topic .

bob.aitah · October 27, 2024, 6:21pm

Just tried a few sample searches; worked great, no errors.

+1 for this going official (when data checks are done).

Flori · October 28, 2024, 12:55pm

A good idea. If the PCI filter is selected, the models with mPCIe and M.2 ports should also appear. Or separate filters.

wxop · October 28, 2024, 1:22pm

Currently the PCI feature searches the "PCI" pattern in the comments column, thus it already returns PCI as well as PCIe or mPCIe....

Where is this "M.2 ports" information located ?

wxop · October 28, 2024, 3:56pm

To implement filters like these, the existing "database" needs to be thoroughly cleaned and better organized. This is not the case at the moment.

This is "slightly" beyond my attempt to make it more convenient using the current data.

wxop · October 28, 2024, 4:31pm

I've just published Better TOH v1.5!

Changes include:

ability to store/recall the currents filters or columns in User Presets (stored in cookies )
Links filter/column immediately applies (in 'replace' mode)
enhanced/more features & presets , thanks to @hrw contribution!
Fixes the filter "not cleared" bug
minor CSS design improvements

We are near "production ready", what you think?

ed8 · October 28, 2024, 9:39pm

I would recommend to increase the minimum RAM size from 64 MB to 128 MB
(in theory 64Mb is the bare minimum but 128 MB as minimum is recommended)
see also https://openwrt.org/supported_devices/864_warning

ps: I noticed some strange behavior in this scenario:
When using following link https://soif.github.io/OpenWrtTOH/?features=available,eth_1g,memory_minimum,power_poe,wifi_ax&view=normal
the ZyXEL NWA50AX Pro does not show up.
However as soon as I uncheck Ethernet 1G it does show up

wxop · October 29, 2024, 11:48am

Nice catch @ed8 .

I think i've found the culprit (Tabulator has a random bug when adding/removing filter).
I should be able to post a fix soon.

wxop · October 30, 2024, 7:02pm

Here is my better OpenWRT Table Of Hardware v1.52 :

Fix the features add/remove Tabulator filtering bug
Correctly update browser URL after clearing filters
Update "Availability" filters
More "loading" icon displayed on long tasks
Add a CHANGELOG file (thanks to @richb-hanover-priv )
Deluxe console logs (for developers)

enjoy !