Port pr #1003 (...enlarge r7800 flash...) to r7500v2

For Developers
1

@blogic @dissent1 @mkresin @pkgadd @hnyman

hi,

requests to extend the netgear r7500v2 flash in a similar fashion as done for the r7800 in PR #1003 (also described in this tutorial) seems to come up periodically.

In short, is it really realistic that some random user with a r7500v2 build and test a similar change for the r7500v2 and then have a PR merged? If not the dev's above, who should I put this request to?

The response so far is along the lines of "make the changes and test in a similar fashion as described by @hnyman man in PR #1003." i.e.

@hnyman
Tested with Openwrt master r7093-4fdc6ca31b and OEM V1.0.2.52:
I flashed a modified Openwrt r7093-4fdc6ca31b build and filled overlay with random data, flashed the newest OEM firmware V1.0.2.52, played with OEM settings and let it re-create the certificate for VPN, and then flashed vanilla Openwrt to see the outcome. Both OEM and Openwrt worked ok.

I'd add that someone will need to edit the install instructions/docs/wiki explaining that the next stable version for the r7500v2 will have to be flashed via tftp again as I don't think the change will take via a sysupgrade.

So I have a r7500v2, made the changes to the r7500v2 partition tables described in the tutorial, built and installed a working factory.img (in branch 18.06.1, not master yet). FWIW, I've also modified the script in the afore mentioned tutorial to extend the flash in the official 18.06.1 r7500v2 image which is what i'm using now.

I am willing to build a modified version against (a recommended) point in master and test (fill up overlay, reflash stock and test a little, reflash official 18.06.1) and report back as described above, even submit the PR.

A little feedback would be appreciated.

Regards,

nmrh.

1 Like
2

Disclaimer: I'm not an OpenWrt developer and don't speak for the project, what I mention below is just my own opinion and personal interpretation of common best principles.

Basically there are two orthogonal considerations here:

  • try to avoid disrupting sysupgrades from previous LEDE or OpenWrt releases
    • coming from 17.01.x to $newer will need tftp based 'debricking', as the kernel partition really had to be extended (currents kernels are ~2.4 MB in size, there's no way to keep them below 2 MB - even less going further); the same applies when coming from the OEM firmware.
    • introducing another breakage point between 18.06.x and 19.01.0 would be unfortunate, especially as users tend not to notice documentation until they fall victim to it. On the other hand there is another potential disruptive change upcoming (switching from swconfig to qca8k/ DSA based switch configurations), with which this change could be synchronized (breaking it once, not twice) - however qca8k might not make it for 19.01.x.
  • leave the door open to reinstall the stock OEM firmware
    this is the really crucial part, your changes shouldn't affect reverting back to the stock firmware, meaning that doing a tftp based debricking with the stock firmware is supposed to result in a fully functional router again (including the little details like VPN certificates being regenerated, language switching working, etc.). For the r7800, @hnyman has looked into this problem and confirmed that extending the rootfs is safe, namely that the overwritten partitions are getting recreated while doing tftp based debricking. While it's reasonable to assume that r7500v1 and r7500v2 behave in a similar way, this needs to be actually confirmed (not that you can upgrade from previous OpenWrt versions, but that reverting to the stock OEM firmware remains possible) before such a change can be implemented in OpenWrt.
    If you look at the referenced thread, this checking went two-fold, first by backing up the untouched OEM partitions, installing an oversized OpenWrt image overwriting their contents and then checking what happens when trying to revert to the OEM firmware (do all partitions get recreated with sane values), but then also by looking into the upgrade/ firstboot code of the OEM firmware to confirm that there's actually the code necessary for recreating and re-initializing them.
2 Likes
3

@slh thank you for responding

  • try to avoid disrupting sysupgrades from previous LEDE or OpenWrt releases

I understand. I'll willing to do the work if there is a dev that will review, merge or reject the PR. If r7500v2 is an orphan then I'll just continue making do...

  • leave the door open to reinstall the stock OEM firmware

i had the presence of mind to take a backup of mtd7 from stock 18.06.1 before expanding the ubi partition - I can always compare sha sums of the netgear partitions after attempting reverting to stock OEM and going back to stock 18.06.1 to get the partition again.

I found what might be the preinit script to rebuild the netgear partition for the r7500v2 here @dissent1 will have to confirm that this is relevant. A cursory inspection indicates that it is similar in function to the r7800 script.

Beyond repeating making a VPN as @hnyman did on stock OEM, I'll need some guidance if the PR reviewer wants something more to verify that the stock OEM functionality is still present on the rebuilt netgear partition...

A somewhat capable user with little credibility like myself can do all this... if that's acceptable for the openwrt project.

4

The checksums will differ, as they contain variable data (i18n downloaded from the web, vpn certificates, etc.), recreated/ regenerated data will inevitably differ - but be fully functional, if the recreation works correctly - and that's exactly what needs to be confirmed.

5

R7500v2 is apparently so close to the R7800 that the same process should work okay.

You should be able to find the relevant OEM source files, like you already linked one. Those files pretty much provide the basis for the assumption that it is similar.

My test of flashing modified openwrt, filling overlay with garbage, flashing OEM and testing basic functionality, flashing normal openwrt and looking closer into partition contents like VPN cert, should be sufficient proof.

The long forum discussions linked in 1003 show the commands that I used.
E.g. Netgear R7800 exploration (IPQ8065, QCA9984) - #1147 by hnyman

Ps. Note that for looking into recreated OEM certs etc., You will need to use unmodified vanilla openwrt, so that Netgear partition is visible.

1 Like
6

Also note that the change should be done in master, not 18.06

Current head of master should be ok. (I compiled yesterday r8466-251c350727 for R7800)

1 Like
7

@hnyman, thankyou for responding and your contributions on the r7800 that make this possible. It will take me a few days to read, comprehend, build and test. I'll report back here in a bit.

8

Looks promising. I'll do the tests below again after building an img against master with a proper fix to the dts file.

Since I already had an extended overlay partition from a moded stock 18.06.1 img, I went ahead and filled it up with with a some files about 3-10M in size created via:

root@OpenWrt:/overlay# head -c 10485760 </dev/urandom >/overlay/10.dat

root@OpenWrt:/overlay# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 3.0M      3.0M         0 100% /rom
tmpfs                   233.6M     80.0K    233.5M   0% /tmp
/dev/ubi0_1              79.2M     74.9M    184.0K 100% /overlay
overlayfs:/overlay       79.2M     74.9M    184.0K 100% /
tmpfs                   512.0K         0    512.0K   0% /dev

I assume 184 K free is sufficient...

Again, I used a modified version of the script in the tutorial to modify the stock 18.06.1 img file; hence,

root@OpenWrt:/overlay# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00c80000 00020000 "qcadata"
mtd1: 00500000 00020000 "APPSBL"
mtd2: 00080000 00020000 "APPSBLENV"
mtd3: 00140000 00020000 "art"
mtd4: 00140000 00020000 "artbak"
mtd5: 00400000 00020000 "kernel"
mtd6: 06080000 00020000 "ubi"
mtd7: 04480000 00020000 "netgear"
mtd8: 00700000 00020000 "reserve"

still shows the mtd "netgear" partition. The netgear partition will go away after compiling with a modified dts file.

Shut off the router, flashed stock OEM (R7500v2-V1.0.3.38.img), played around for a min (including turning on VPN) - no issues.

flashed stock 18.06.1 and then:

root@OpenWrt:~# ubiattach -m 7
UBI device number 1, total 548 LEBs (69582848 bytes, 66.3 MiB), available 18 LEBs (2285568 bytes, 2.1 MiB), LEB size 126976 bytes (124.0 KiB)
ubinfo /dev/ubi1
ubi1
Volumes count:                           6
Logical eraseblock size:                 126976 bytes, 124.0 KiB
Total amount of logical eraseblocks:     548 (69582848 bytes, 66.3 MiB)
Amount of available logical eraseblocks: 18 (2285568 bytes, 2.1 MiB)
Maximum count of volumes                 128
Count of bad physical eraseblocks:       0
Count of reserved physical eraseblocks:  20
Current maximum erase counter value:     2
Minimum input/output unit size:          2048 bytes
Character device major/minor:            247:0
Present volumes:                         0, 1, 2, 3, 4, 5

root@OpenWrt:~# ubinfo /dev/ubi1_0
Volume ID:   0 (on ubi1)
Type:        dynamic
Alignment:   1
Size:        1 LEBs (126976 bytes, 124.0 KiB)
State:       OK
Name:        cert
Character device major/minor: 247:1

root@OpenWrt:~# cat /dev/ubi1_0 | gunzip -c | head -n 50
cert.info0000644000000000000000000000001613326020725011363 0ustar  rootroot4ES66C5X000F4
ca.crt0000644000000000000000000000245213326020625010653 0ustar  rootroot-----BEGIN CERTIFICATE-----
MIIDojCCAwugAwIBAgIJAKRnQ7Z+6z6dMA0GCSqGSIb3DQEBCwUAMIGTMQswCQYD
VQQGEwJUVzELMAkGA1UECBMCVFcxDzANBgNVBAcTBlRhaXBlaTEQMA4GA1UEChMH
bmV0Z2VhcjEQMA4GA1UECxMHbmV0Z2VhcjETMBEGA1UEAxMKbmV0Z2VhciBDQTEQ
MA4GA1UEKRMHRWFzeVJTQTEbMBkGCSqGSIb3DQEJARYMbWFpbEBuZXRnZWFyMB4X
DTE4MDcyNTA3MDcwMVoXDTM4MDcyMDA3MDcwMVowgZMxCzAJBgNVBAYTAlRXMQsw
CQYDVQQIEwJUVzEPMA0GA1UEBxMGVGFpcGVpMRAwDgYDVQQKEwduZXRnZWFyMRAw
DgYDVQQLEwduZXRnZWFyMRMwEQYDVQQDEwpuZXRnZWFyIENBMRAwDgYDVQQpEwdF
YXN5UlNBMRswGQYJKoZIhvcNAQkBFgxtYWlsQG5ldGdlYXIwgZ8wDQYJKoZIhvcN
AQEBBQADgY0AMIGJAoGBAP2VDx1Mus4Aftp48nAmi2C+i6gqPOZGdqYuYM2QDDhx
39k+PGFVZKth+8ixSUMJ6x64VF/yjUs7G/xuERfjlrOelmBt4JofekLuSOUaz1Bm
ZvaHi5rF9mF6x4IS1hw0jxeHyst4wFEsmsJmr9sGRsKu30IPyTtVMGWsIJwFiKgL
AgMBAAGjgfswgfgwHQYDVR0OBBYEFMEXgte8VIFPejBJ+Td/+GT+cj8+MIHIBgNV
HSMEgcAwgb2AFMEXgte8VIFPejBJ+Td/+GT+cj8+oYGZpIGWMIGTMQswCQYDVQQG
EwJUVzELMAkGA1UECBMCVFcxDzANBgNVBAcTBlRhaXBlaTEQMA4GA1UEChMHbmV0
Z2VhcjEQMA4GA1UECxMHbmV0Z2VhcjETMBEGA1UEAxMKbmV0Z2VhciBDQTEQMA4G
A1UEKRMHRWFzeVJTQTEbMBkGCSqGSIb3DQEJARYMbWFpbEBuZXRnZWFyggkApGdD
tn7rPp0wDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOBgQA4DTT7+C3zr7e6
v1+Am2nbWXSEEsDD1AqYJ7WYY8efONUyL1j3DSVXpqkVc2OCJtQEI6b1guk+CL5J
Q/PNLPhH/iEsItpSlQPVSMXUqMMKNrwOvIM8pf2w+0ccZ240OWXrPw1EcpoGk9hm
F6+NtEaHTqIeAl99Mb/7C1tckznmhw==
-----END CERTIFICATE-----
dh1024.pem0000644000000000000000000000036513326020724011164 0ustar  rootroot-----BEGIN DH PARAMETERS-----
MIGHAoGBALnDNf8fw2dxcC24/CcTFW1SSjKd/aztBY3QkUH6Hfn7cLYUl5Vyq2R3
SaINONN7RKUYawgA1mynVYInR9GERyzQkMjyR0uMuV0H3SiEBFZqCbqeXNqGW3p9
9iTQ/AJbSQzrz1d6SRJEQeN4rGSwJdGOrHBeb+d7TtLhiRMpvKFbAgEC
-----END DH PARAMETERS-----
client.crt0000644000000000000000000000746213326020725011555 0ustar  rootrootCertificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 2 (0x2)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=TW, ST=TW, L=Taipei, O=netgear, OU=netgear, CN=netgear CA/name=EasyRSA/emailAddress=mail@netgear
        Validity
            Not Before: Jul 25 07:08:05 2018 GMT
            Not After : Jul 20 07:08:05 2038 GMT
        Subject: C=TW, ST=TW, L=Taipei, O=netgear, OU=netgear, CN=client/name=EasyRSA/emailAddress=mail@netgear
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (1024 bit)
                Modulus:
                    00:af:0f:57:33:9e:21:bc:b2:65:b8:89:3d:82:fd:
                    fc:75:f2:6b:e1:db:d7:03:0d:8a:4b:f2:37:b1:2b:
                    a0:01:00:c9:b1:ab:17:c6:20:6a:0c:03:58:d3:c3:
                    f3:c4:33:d3:56:4d:6a:ce:0b:63:e8:4c:79:63:de:
                    cf:85:f9:ed:c8:45:4b:d4:cd:77:88:15:c5:30:3e:
                    bb:0c:1d:ec:38:10:16:ab:7e:85:6f:63:1d:21:4f:
                    e9:fc:97:40:5d:54:bf:e0:0e:0d:82:ee:5f:0d:6b:
                    f2:a2:4b:49:40:eb:c9:96:ad:64:b1:51:fc:94:ef:
1 Like
9

In short, it works to the extent I've tested. Details below.

Assuming this is good enough for a pr and there is flexibility in the openwrt contributor identity requirements I can submit the PR. However, it seems to me that @hnyman and associates did the real work for the r7500v2 and should have the credit of the PR.

Details:
OpenWrt SNAPSHOT, r8467-dd02a19ff5 built with PR #1554.

I built all packages but had to disable kmod-openvswitch* mentioned here to complete the build.

Flashing the factory.img and filling /overlay as described above:

root@OpenWrt:/overlay# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 2.8M      2.8M         0 100% /rom
tmpfs                   233.1M      1.1M    231.9M   0% /tmp
/dev/ubi0_1              79.3M     75.0M    188.0K 100% /overlay
overlayfs:/overlay       79.3M     75.0M    188.0K 100% /
tmpfs                   512.0K         0    512.0K   0% /dev
root@OpenWrt:/overlay# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00c80000 00020000 "qcadata"
mtd1: 00500000 00020000 "APPSBL"
mtd2: 00080000 00020000 "APPSBLENV"
mtd3: 00140000 00020000 "art"
mtd4: 00140000 00020000 "artbak"
mtd5: 00400000 00020000 "kernel"
mtd6: 06080000 00020000 "ubi"
mtd7: 00700000 00020000 "reserve"

No "netgear" partition this time.

Turning off the router and flashing stock OEM (R7500v2-V1.0.3.38.img) works. Basic router functionality works.

I'll add that my settings from the last time I did this (see above) persisted (in "reserve" based on @hnyman's post in here). I had to enter the password I used earlier and the VPN service set up above was enabled - I did not test the VPN so I don't know if it would have worked given the certs in "netgear" were regenerated - I'd guess (hope) not. Doing a reset while on stock OEM cleared out the password/VPN enabled settings.

Turning off the router and flashing back to stock 18.06.1:

 -----------------------------------------------------
 OpenWrt 18.06.1, r7258-5eb055306f
 -----------------------------------------------------
root@OpenWrt:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 3.0M      3.0M         0 100% /rom
tmpfs                   233.6M    556.0K    233.0M   0% /tmp
/dev/ubi0_1              17.9M     52.0K     16.9M   0% /overlay
overlayfs:/overlay       17.9M     52.0K     16.9M   0% /
tmpfs                   512.0K         0    512.0K   0% /dev

root@OpenWrt:~# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00c80000 00020000 "qcadata"
mtd1: 00500000 00020000 "APPSBL"
mtd2: 00080000 00020000 "APPSBLENV"
mtd3: 00140000 00020000 "art"
mtd4: 00140000 00020000 "artbak"
mtd5: 00400000 00020000 "kernel"
mtd6: 01c00000 00020000 "ubi"
mtd7: 04480000 00020000 "netgear"
mtd8: 00700000 00020000 "reserve"

root@OpenWrt:~# ubiattach -m 7
UBI device number 1, total 548 LEBs (69582848 bytes, 66.3 MiB), available 18 LEBs (2285568 bytes, 2.1 MiB), LEB size 126976 bytes (124.0 KiB)

root@OpenWrt:~# ubinfo /dev/ubi1
ubi1
Volumes count:                           6
Logical eraseblock size:                 126976 bytes, 124.0 KiB
Total amount of logical eraseblocks:     548 (69582848 bytes, 66.3 MiB)
Amount of available logical eraseblocks: 18 (2285568 bytes, 2.1 MiB)
Maximum count of volumes                 128
Count of bad physical eraseblocks:       0
Count of reserved physical eraseblocks:  20
Current maximum erase counter value:     2
Minimum input/output unit size:          2048 bytes
Character device major/minor:            247:0
Present volumes:                         0, 1, 2, 3, 4, 5

root@OpenWrt:~# ubinfo /dev/ubi1_0
Volume ID:   0 (on ubi1)
Type:        dynamic
Alignment:   1
Size:        1 LEBs (126976 bytes, 124.0 KiB)
State:       OK
Name:        cert
Character device major/minor: 247:1

root@OpenWrt:~# cat /dev/ubi1_0 | gunzip -c | head -n 50
cert.info0000644000000000000000000000001613374347054011375 0ustar  rootroot4ES66C5X000F4
ca.crt0000644000000000000000000000245213374346653010672 0ustar  rootroot-----BEGIN CERTIFICATE-----
MIIDojCCAwugAwIBAgIJAMEl9VRMqrerMA0GCSqGSIb3DQEBCwUAMIGTMQswCQYD
VQQGEwJUVzELMAkGA1UECBMCVFcxDzANBgNVBAcTBlRhaXBlaTEQMA4GA1UEChMH
bmV0Z2VhcjEQMA4GA1UECxMHbmV0Z2VhcjETMBEGA1UEAxMKbmV0Z2VhciBDQTEQ
MA4GA1UEKRMHRWFzeVJTQTEbMBkGCSqGSIb3DQEJARYMbWFpbEBuZXRnZWFyMB4X
DTE4MTExODIwMzgwM1oXDTM4MTExMzIwMzgwM1owgZMxCzAJBgNVBAYTAlRXMQsw
CQYDVQQIEwJUVzEPMA0GA1UEBxMGVGFpcGVpMRAwDgYDVQQKEwduZXRnZWFyMRAw
DgYDVQQLEwduZXRnZWFyMRMwEQYDVQQDEwpuZXRnZWFyIENBMRAwDgYDVQQpEwdF
YXN5UlNBMRswGQYJKoZIhvcNAQkBFgxtYWlsQG5ldGdlYXIwgZ8wDQYJKoZIhvcN
AQEBBQADgY0AMIGJAoGBAJ/1tdiKhAyp9et2qeb9J5BizpnXZHf+DKoZ5J/7S5BY
Hw4JeK4ZBRn+DAkJOA3hGZIJTaRX2FWNN2mHcNgKhOlOnQvXeZD+boSNDfxZctrn
ghIu7JSregaQGHC7Zq/J4qU/sFrJW2YRpCM2zdeJvhHj9P7kLOJPbvlwJNAMlbQJ
AgMBAAGjgfswgfgwHQYDVR0OBBYEFIboPgSXbJdqAeGNZicu5buT8FEVMIHIBgNV
HSMEgcAwgb2AFIboPgSXbJdqAeGNZicu5buT8FEVoYGZpIGWMIGTMQswCQYDVQQG
EwJUVzELMAkGA1UECBMCVFcxDzANBgNVBAcTBlRhaXBlaTEQMA4GA1UEChMHbmV0
Z2VhcjEQMA4GA1UECxMHbmV0Z2VhcjETMBEGA1UEAxMKbmV0Z2VhciBDQTEQMA4G
A1UEKRMHRWFzeVJTQTEbMBkGCSqGSIb3DQEJARYMbWFpbEBuZXRnZWFyggkAwSX1
VEyqt6swDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOBgQAp1uWucCCopY1f
5XdFETxe8tfKdYRxY7j/6+hhe/upY6f3miiKVYsQXnG1f4qEGcEDQDCY8267TEyl
e6HQ2d9hLASvSWC7rpvY+izxAy0ad/8RIMa8Ojh/NP7luvAr8NeZPhwG7pfuW8Wb
si68TpGX4I8B/6cHmCXsWqM97l+NyA==
-----END CERTIFICATE-----
dh1024.pem0000644000000000000000000000036513374347053011176 0ustar  rootroot-----BEGIN DH PARAMETERS-----
MIGHAoGBAJmln8bQKEkcZVV3hTrR8AxB0TjuLekDe0oMImb4ZO3zFLstEOFda/QP
JjZljxR+k+9n4DdXhdr6T2hjdFej+L4+/HfoecqxjHKn7DIXOMoEMv/l3vNg9JVJ
coCGPNY5ZkEmjIOjWzgHA3y0qNTeT6YNoqwXxHxCH65F2V9RTWTjAgEC
-----END DH PARAMETERS-----
client.crt0000644000000000000000000000746213374347054011567 0ustar  rootrootCertificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 2 (0x2)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=TW, ST=TW, L=Taipei, O=netgear, OU=netgear, CN=netgear CA/name=EasyRSA/emailAddress=mail@netgear
        Validity
            Not Before: Nov 18 20:40:12 2018 GMT
            Not After : Nov 13 20:40:12 2038 GMT
        Subject: C=TW, ST=TW, L=Taipei, O=netgear, OU=netgear, CN=client/name=EasyRSA/emailAddress=mail@netgear
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (1024 bit)
                Modulus:
                    00:c9:7c:05:bc:70:3d:8b:06:94:80:af:9b:66:9b:
                    18:4b:04:35:75:b9:71:1a:3e:61:b2:c3:c0:c0:5a:
                    eb:32:21:ba:7e:b9:c5:be:8c:63:01:8d:ee:f2:97:
                    6a:41:55:91:61:3e:64:a7:57:f2:8b:1f:36:72:bb:
                    4d:ba:a6:be:7d:d7:63:93:c6:eb:6f:f8:80:a9:a0:
                    fa:38:78:b9:86:24:22:ee:e8:79:72:50:9d:d4:d5:
                    8e:21:5b:61:47:24:15:9e:23:61:fe:61:13:16:14:
                    4d:64:91:94:12:79:92:f8:47:9c:69:9b:0e:fa:81:

Attempting to flash the sysupgrade.bin file from my OpenWrt SNAPSHOT, r8467-dd02a19ff5 with the dts mod (from luci, not tftp) while on stock 18.06.1 puts the router into a boot loop. TFTP flashing an openwrt factory.img (or presumably the stock OEM img - I didn't test stock OEM) recovers from this.

10

See PR #1554 here.

EDIT: Due to OpenWRT's identity requirements for a PR, I have closed this PR. At this time, this is as far as I can take this. Another contributor will need to move this forward.

11

Hello nmrh, would you mind I re-create this PR by amend your commit ? I have 7500v2 too and hopes this patch upstream.

2 Likes
12

as stated in my comment above, this is as far as I'm willing to take it. Please feel free to move the PR forward yourself.

Regards,

nmrh

2 Likes
13

thank you for moving this forward

2 Likes
14

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.