Script: Convert device information to YAML

Community Builds, Projects & Packages
1

Hi all, based on this openwrt-devel mail thread I started a little script scraping some device specific information. This should eventually help creating and maintaining a device wiki with hardware information.

The output should be some YAML file which is then rendered into a fancy website.

I just figured out a few properties, I'd appreciate if somebody else could join in!

source /etc/os-release

#flash:
#  - size_mb: 128
#    type: SPI-NAND (Macronix)
#  - size_mb: 4
#    type: SPI-NOR (Macronix MX25R3235F)
#usb:
#  - version: 2
#    ports: 1
#  - version: 3
#    ports: 2
#ethernet:
#  - mbit: 100
#    ports: 4
#  - mbit: 1000
#    ports: 1
#vendor: Foobar
#model: Lorem Ipsum
#variant: v4

echo "soc: \"$(cat /proc/cpuinfo | grep 'system type' | cut -d ":" -f2 | sort | uniq | xargs)\""
echo "ram_mb: $(($(cat /proc/meminfo | grep MemTotal | xargs | cut -d " " -f 2) / 1024))"
echo "arch: $OPENWRT_ARCH"
echo "target: $OPENWRT_BOARD"
echo "id: \"$(cat /tmp/sysinfo/board_name)\""
echo "leds:"
for led in $(ls /sys/devices/platform/leds/leds/); do
    echo "  - \"$led\""
done
3 Likes
2

Tracked this on the mailing list.
I like that idea.
However, I would suggest that we pick it up during the build.
This has the advantage that for every build the information is up to date.
Everything else will stop simming after a while!

I found a project in python that parses a dtb file

The needed value can then be written into a YAML file and after that could be pushed to the web.

A played a little bit and it is working as expected.

3

Hi, thanks for the quick response. Could you please add an example output?
I don't see it as a big problem that the information is not continuously updated, as a device hardware unlikely change over time. A TP-Link 4300 will always keep the same SoC and amount of memory.

4

@feckert ping?

5

Sorry for the delay I am just somewhere else at the moment. My boss is breathing down my neck :wink:

If I still have time today, I will put together a small script.
First of all, a question:
Since I am based on the python libaries fdt and yaml, is that a problem? I am thinking about the dependencies.
For testing python I always use pipenv. Is this an option in openwrt if we want to build this information during build? This is my pipfile

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
fdt = ""
pyyaml = ""

[requires]
python_version = "3.7"

6

this one is hit and miss @aparcar... perhaps deriving from;

/etc/os-release >>>
OPENWRT_ARCH... |+ OPENWRT_BOARD

etc. etc. might be yield more frequent results... ( readelf? )

7

I'm fine with Python 3.7, however current master seem to use Python 3.8.1.

Hopefully the scripts wont get to complicated, so we can adapt this later.

I'll be mostly offline until Sunday, sorry for rushing and now being offline :roll_eyes:

8

Sorry I don't full understand your post.... can you please elaborate

9 
[root@synology-001132 /]# cat /proc/cpuinfo | grep 'system type' | cut -d 
":" -f2 | sort | uniq | xargs

#versus
[root@synology-001132 /]# cat /etc/os-release | grep 'OPENWRT_ARCH=' | grep -q 'arm' && echo armlike
armlike ( not exactly but you get the picture, case or something else entirely et. al. )

the rest all ok here...

10

This script converts the device tree files (dtb) coming from the openwrt build into yml files.
So far only the model and the soc is written. I have tested only with the lantiq target.
The generated dtb files for this target are located in the directory <openwrt>/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200

So i called my script from my pipenv directory with the following command line.
./device-info.py -i <openwrt>/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200 -o ./info

And I got in the output directory the yml files for all dtb files in the input directory.

For example:

model: BT OpenReach VDSL Modem
soc: alphanetworks,asl56026

Perhaps we can continue working on this basis

11

Not to take anything away from the OP or fellow contribs, but I use and recommend

which is an ansible-driven configuration collector ("gather facts") and can output to csv json markdown sql html or plain txt. The results are printed in a nice html clickable page for all hosts in your inventory file. Seems to work with openwrt, but not perfectly. ( I dont have pip3/python3 running yet).

Maybe this can give you ideas or combine bits of code to give you something even more Deluxe. Good luck yaml'ing and such.

12

Looks good to me, however we still have the problem that some values are auto detected... I guess a combination of both is the only way...

13

Thank you the tool looks good. However Python is a bit of a big package so I'd rather not have it as a dependency for the initial device info. Can you evaluate how well it works without Python on OpenWrt devices?

14

Where do we go from here?
If we want to automate the whole thing, then I think it's best to get the data from the devicetree binary blob with this script, since this is automatable during the build or doring a postprocess step.
Which information do we need from a running system?

15

Let's try to get all the data from the initial mail thread via device tree. Once done we can look for additional things scraped during run time

16

@feckert Is your plan to use the script on dtb files? If so we have to compile as those files before parsing - just a comment.

Looking at the dts (using the to_dts() function) there is quite some content in the file, however I don't really see number of CPU cores or storage size. Do you have an idea for that?

Parse dtb: image-ar7161_ubnt_routerstation-pro.dtb
/dts-v1/;
// version: 17
// last_comp_version: 16
// boot_cpuid_phys: 0x0

/ {
    #address-cells = <0x1>;
    #size-cells = <0x1>;
    compatible = "ubnt,routerstation-pro", "qca,ar7161";
    model = "Ubiquiti RouterStation Pro";
    interrupt-controller {
        compatible = "qca,ar7100-cpu-intc";
        interrupt-controller;
        #interrupt-cells = <0x1>;
        qca,ddr-wb-channel-interrupts = <0x2 0x3 0x4 0x5>;
        qca,ddr-wb-channels = <0x1 0x3 0x1 0x2 0x1 0x0 0x1 0x1>;
        phandle = <0x2>;
    };
    ahb {
        compatible = "simple-bus";
        ranges;
        #address-cells = <0x1>;
        #size-cells = <0x1>;
        interrupt-parent = <0x2>;
        apb {
            compatible = "simple-bus";
            ranges;
            #address-cells = <0x1>;
            #size-cells = <0x1>;
            interrupt-parent = <0x3>;
            interrupt-controller@18060010 {
                compatible = "qca,ar7100-misc-intc";
                reg = <0x18060010 0x4>;
                interrupt-parent = <0x2>;
                interrupts = <0x6>;
                interrupt-controller;
                #interrupt-cells = <0x1>;
                phandle = <0x3>;
            };
            memory-controller@18000000 {
                compatible = "qca,ar7100-ddr-controller";
                reg = <0x18000000 0x100>;
                #qca,ddr-wb-channel-cells = <0x1>;
                phandle = <0x1>;
            };
            uart@18020000 {
                compatible = "ns16550a";
                reg = <0x18020000 0x20>;
                interrupts = <0x3>;
                clocks = <0x4 0x2>;
                clock-names = "uart";
                reg-io-width = <0x4>;
                reg-shift = <0x2>;
                no-loopback-test;
                status = "okay";
            };
            usb-phy@18030000 {
                compatible = "qca,ar7100-usb-phy";
                reg = <0x18030000 0x10>;
                reset-names = "usb-phy", "usb-host", "usb-ohci-dll";
                resets = <0x5 0x4 0x5 0x5 0x5 0x6>;
                #phy-cells = <0x0>;
                status = "okay";
                phandle = <0xB>;
            };
            gpio@18040000 {
                compatible = "qca,ar7100-gpio";
                reg = <0x18040000 0x30>;
                interrupts = <0x2>;
                ngpios = <0x10>;
                gpio-controller;
                #gpio-cells = <0x2>;
                interrupt-controller;
                #interrupt-cells = <0x2>;
                phandle = <0xC>;
            };
            pll-controller@18050000 {
                compatible = "qca,ar7100-pll", "syscon";
                reg = <0x18050000 0x20>;
                clock-names = "ref";
                #clock-cells = <0x1>;
                clock-output-names = "cpu", "ddr", "ahb";
                phandle = <0x4>;
            };
            wdt@18060008 {
                compatible = "qca,ar7130-wdt";
                reg = <0x18060008 0x8>;
                interrupts = <0x4>;
                clocks = <0x4 0x2>;
                clock-names = "wdt";
            };
            interrupt-controller@18060018 {
                compatible = "qca,ar7100-misc-intc";
                reg = <0x18060018 0x4>;
                interrupt-parent = <0x2>;
                interrupts = <0x2>;
                interrupt-controller;
                #interrupt-cells = <0x1>;
                phandle = <0x6>;
            };
            reset-controller@18060024 {
                compatible = "qca,ar7100-reset";
                reg = <0x18060024 0x4>;
                #reset-cells = <0x1>;
                phandle = <0x5>;
            };
            pcie-controller@17010000 {
                compatible = "qca,ar7100-pci";
                #address-cells = <0x3>;
                #size-cells = <0x2>;
                bus-range = <0x0 0x0>;
                reg = <0x17010000 0x100>;
                reg-names = "cfg_base";
                ranges = <0x2000000 0x0 0x10000000 0x10000000 0x0 0x7000000 0x1000000 0x0 0x0 0x0 0x0 0x1>;
                interrupt-parent = <0x6>;
                interrupts = <0x4>;
                #interrupt-cells = <0x1>;
                interrupt-map-mask = <0xF800 0x0 0x0 0x0>;
                interrupt-map = <0x8800 0x0 0x0 0x0 0x6 0x0 0x9000 0x0 0x0 0x0 0x6 0x1 0x9800 0x0 0x0 0x0 0x6 0x2>;
                status = "okay";
            };
        };
        eth@19000000 {
            status = "okay";
            compatible = "qca,ar7100-eth", "syscon";
            reg = <0x19000000 0x200 0x18070000 0x4>;
            interrupts = <0x4>;
            phy-mode = "rgmii";
            pll-data = <0x110000 0x1099 0x991099>;
            pll-reg = <0x4 0x10 0x11>;
            pll-handle = <0x4>;
            resets = <0x5 0x9>;
            reset-names = "mac";
            qca,mac-idx = <0x0>;
            phy-handle = <0x7>;
            phandle = <0x8>;
            mdio {
                status = "okay";
                compatible = "qca,ath79-mdio";
                #address-cells = <0x1>;
                #size-cells = <0x0>;
                regmap = <0x8>;
                clocks = <0x4 0x4>;
                clock-names = "ref";
                ethernet-phy@0 {
                    reg = <0x0>;
                    phandle = <0x9>;
                };
                ethernet-phy@4 {
                    reg = <0x4>;
                    phandle = <0x7>;
                };
            };
        };
        eth@1a000000 {
            status = "okay";
            compatible = "qca,ar7100-eth", "syscon";
            reg = <0x1A000000 0x200 0x18070004 0x4>;
            interrupts = <0x5>;
            phy-mode = "rgmii";
            pll-data = <0x110000 0x1099 0x991099>;
            pll-reg = <0x4 0x14 0x13>;
            pll-handle = <0x4>;
            resets = <0x5 0xD>;
            reset-names = "mac";
            qca,mac-idx = <0x1>;
            phy-handle = <0x9>;
            phandle = <0xA>;
            mdio {
                status = "disabled";
                compatible = "qca,ath79-mdio";
                #address-cells = <0x1>;
                #size-cells = <0x0>;
                regmap = <0xA>;
                clocks = <0x4 0x4>;
                clock-names = "ref";
                builtin-switch;
            };
        };
    };
    cpus {
        #address-cells = <0x1>;
        #size-cells = <0x0>;
        cpu@0 {
            device_type = "cpu";
            compatible = "mips,mips24Kc";
            clocks = <0x4 0x0>;
            reg = <0x0>;
        };
    };
    usb@1b000000 {
        compatible = "generic-ehci";
        reg = <0x1B000000 0x1000>;
        interrupt-parent = <0x2>;
        interrupts = <0x3>;
        phy-names = "usb-phy";
        phys = <0xB>;
        has-synopsys-hc-bug;
        status = "okay";
        #address-cells = <0x1>;
        #size-cells = <0x0>;
        port@1 {
            reg = <0x1>;
            #trigger-source-cells = <0x0>;
        };
    };
    usb@1c000000 {
        compatible = "generic-ohci";
        reg = <0x1C000000 0x1000>;
        interrupt-parent = <0x3>;
        interrupts = <0x6>;
        phy-names = "usb-phy";
        phys = <0xB>;
        status = "okay";
        #address-cells = <0x1>;
        #size-cells = <0x0>;
        port@1 {
            reg = <0x1>;
            #trigger-source-cells = <0x0>;
        };
    };
    spi@1f000000 {
        compatible = "qca,ar7100-spi";
        reg = <0x1F000000 0x10>;
        clocks = <0x4 0x2>;
        clock-names = "ahb";
        #address-cells = <0x1>;
        #size-cells = <0x0>;
        status = "okay";
        num-cs = <0x1>;
        flash@0 {
            compatible = "jedec,spi-nor";
            reg = <0x0>;
            spi-max-frequency = <0x17D7840>;
            partitions {
                compatible = "ecoscentric,redboot-fis-partitions";
            };
        };
    };
    chosen {
        bootargs = "console=ttyS0,115200";
    };
    aliases {
        led-boot = "/leds/rf_green";
        led-failsafe = "/leds/rf_green";
        led-running = "/leds/rf_green";
        led-upgrade = "/leds/rf_green";
    };
    ref {
        compatible = "fixed-clock";
        #clock-cells = <0x0>;
        clock-output-names = "ref";
        clock-frequency = <0x2625A00>;
    };
    leds {
        compatible = "gpio-leds";
        rf_green {
            label = "ubnt:green:rf";
            gpios = <0xC 0x2 0x0>;
        };
    };
    keys {
        compatible = "gpio-keys";
        wps {
            label = "sw4";
            linux,code = <0x198>;
            gpios = <0xC 0x8 0x1>;
            debounce-interval = <0x3C>;
        };
    };
};
17

@feckert Is your plan to use the script on dtb files? If so we have to compile as those files before parsing - just a comment.

I would say that we use the dtb files as a basis for parsing the information's.
There is everything inside including the includes like the dts files.
So we have one file per device with all information's in binary from (dtb).
With every openWrt build run these dtb files are generated when the images are built.

Looking at the dts (using the to_dts() function) there is quite some content in the file, however I don't really see number of CPU cores or storage size. Do you have an idea for that?

I am not quite sure whether this has to be defined in the dtb or whether it will be probed! I think we should start small and then expand. If the information is not defined, it can be added later by hand or added when building the YAML file. So that we have a semi-automatic process

18

I mentioned some required information in the first openwrt-devel mail and in the first forum post, can you help implementing these?

19

Sorry for the delay, because of the corona everything is a bit confused in the normal workflow!

Sure. That would be a good starting point

How are we supposed to proceed?
I have a few more comments.

  • Do we want to integrate this into the openwrt buildsystem?
  • Is it alright that we run the conversion tool to be written in a pipenv? (Because we want to use the dtb python lib. If not then it would be difficult because of the python package dependency)
  • Should I set up a repository in my github account where I implement the source?
  • I Could only do test with the lantiq dtb. (Thats my device. I also have x86 but they does not have a dtb)
20

No problem at all! Please don't be stressed by this issue it has no priority!

We could add a make target that prints/stores the information

Yes let's use a Python venv

Yes please

I have a random selection of other devices here and we can compile for whatever else anyway. Your approach is based on DTB files and not on running devices, isn't it?