@ilyas, so copy to root folder worked (instead of /tmp, that said the file is too big) and we get back to pandora without issues!
1 Like
@andreykiselev @pellmen @lolyinseo
actually those aren't error messages. they're "happy messages" telling us USB is working (it doesn't mount automatically... there might be a package for doing that.. don't know what it is ;))
the "error" you were all seeing is because /mnt/sda1 doesn't exist 
(because you didn't create it)
mkdir /mnt/sda1
mount /dev/sda1 /mnt/sda1
1 Like
@andreykiselev about settings not persisting after reboot.. yes this is very interesting to me...
- can you check you can write to the filesystem? ie:
mount
cat /proc/mounts
touch /root/a
dd if=/dev/mtd1 of=/root/mtd1
ls -l /root
if the files aren't there then there's something seriously wrong... if they are, but they disappear after reboot, then there should be something in your bootlog (post it here... i'll probably notice something you didn't )
@pellmen you also have a Micron chip... do you also see the "settings are not saved" problem (someone on 4pda was also saying the same)?
@pellmen great! i suppose the file is too big, but that seems risky to me (to read from a partition you're writing to at the same time.............. seems like a stupid idea).
the really lame thing is that there's an openwrt "optimization" that apparently makes things 5% faster but it means it wastes half of our RAM as "high memory" that the kernel can't use directly... i've also noticed problems writing to /tmp (which is RAM) and "out of space" errors although there's plenty of free memory.... i mean, our NAND is 256 and our RAM is 512... there's no way it shouldn't all fit in /tmp... but it says it's "full" that's retarded. we bought a 512MB router and we can only use 256MB? unfortunately that "optimization" seems to be baked into the kernel... i'm going to try to undo it (but not today...)
anyway @pellmen... the short comment is:
that may work but it seems pretty risky... especially the bit that says mtd erase ubi... if that stuff is cached (which it probably was in your case because you were trying to read it or something) maybe it will be OK but in general it's the same as cutting off the branch you're sitting on...
so, please, keep pandorabox-backup.img on a USB stick and do it from there (ie, don't copy to /tmp)... that seems saner
@ilyas Here is the log of file creation:
root@OpenWrt:/mnt/sda1# mount
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
tmpfs on /tmp/root type tmpfs (rw,noatime,mode=755)
overlayfs:/tmp/root on / type overlay (rw,noatime,lowerdir=/,upperdir=/tmp/root/upper,workdir=/tmp/root/work)
tmpfs on /dev type tmpfs (rw,nosuid,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)
/dev/sda1 on /mnt/sda1 type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
root@OpenWrt:/mnt/sda1# cat /proroot@OpenWrt:/mnt/sda1# cat /proc/moaroot@OpenWrt:/mnt/sda1# cat /proc/mouroot@OpenWrt:/mnt/sda1# cat /proc/mounts a
/dev/root /rom squashfs ro,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,noatime 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,noatime 0 0
tmpfs /tmp tmpfs rw,nosuid,nodev,noatime 0 0
tmpfs /tmp/root tmpfs rw,noatime,mode=755 0 0
overlayfs:/tmp/root / overlay rw,noatime,lowerdir=/,upperdir=/tmp/root/upper,workdir=/tmp/root/work 0 0
tmpfs /dev tmpfs rw,nosuid,relatime,size=512k,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,mode=600,ptmxmode=000 0 0
debugfs /sys/kernel/debug debugfs rw,noatime 0 0
/dev/sda1 /mnt/sda1 vfat rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 0 0
root@OpenWrt:/mnt/sda1# toaroot@OpenWrt:/mnt/sda1# touroot@OpenWrt:/mnt/sda1# touch /roaroot@OpenWrt:/mnt/sda1# touch /ro
rom/ root/
root@OpenWrt:/mnt/sda1# touch /rooroot@OpenWrt:/mnt/sda1# touch /root/a
root@OpenWrt:/mnt/sda1# dd if =/deva/mtatd1 of=/root/mtd1
512+0 records in
512+0 records out
root@OpenWrt:/mnt/sda1# ll /rooroot@OpenWrt:/mnt/sda1# ll /root/
drwxr-xr-x 1 root root 80 Mar 8 07:35 ./
drwxr-xr-x 1 root root 120 Jan 1 1970 ../
-rw-r--r-- 1 root root 0 Mar 8 07:35 a
-rw-r--r-- 1 root root 262144 Mar 8 07:35 mtd1
root@OpenWrt:/mnt/sda1# reboot
And they disappeared after the reboot. The boot log is available here: https://ufile.io/wc36h.
[Update] Looking at my own log, could this be a problem?
# MTK NAND # : Use HW ECC
I actually disabled internal ECC in the NAND chip when flashing raw bin image and did not enable it back. So, U-Boot can mistakenly think that it should rely on internal NAND ECC generator and then expect to get correct verification. Do you know if it's possible to write a command sequence to NAND chip from u-boot or from the running system? I don't really want to desolder NAND chip again to set the flag.
1 Like
Maybe it will help?) I think maybe you'll compile the firmware with all packages preinstalled - it would be better, than install every package manualy.
@andreykiselev sorry i'm the wrong person to ask about anything hardware related if you know the magic sequence (what memory address, whatever) to enable ECC for your chip, yes it can be done (presumably from the kernel)... but you'd have to find that. i'll help you with the code if you need it
but it looks like your system is freaking out in general...
Data buffer not 16 bytes aligned: 8ffe16c8
...
[ 3.271742] compare signature failed 1f840.
...
[ 3.280011] load_fact_bbt failed
...
[ 12.660959] UBIFS error (ubi0:1 pid 491): ubifs_recover_master_node: failed to recover master node
[ 12.669897] UBIFS error (ubi0:1 pid 491): ubifs_recover_master_node: dumping first master node
...
[ 12.787364] UBIFS (ubi0:1): background thread "ubifs_bgt0_1" stops
[ 12.794761] mount_root: failed to mount -t ubifs /dev/ubi0_1 /tmp/overlay: Invalid argument
[ 12.803384] mount_root: overlay filesystem has not been fully initialized yet
[ 12.810868] mount_root: switching to ubifs overlay
[ 12.815730] mount_root: switching to ubifs failed - fallback to ramoverlay
I guess you're rather the "exception" in that you've seriously messed around with your hardware... the average user may not have the same experience.... @souk do you think you can provide similar boot logs? your problem appears to be the same... (failure to mount ubifs partition after boot)
@andreykiselev how about stock xiaomi? any problems with that (with your hacked NAND chip i mean).... i saw the bbt messages on your log but otherwise it looks sane
could you revert to stock and try again, this time with a mtd erase of roofs0 and rootfs1 and overlay?
cd /tmp
mv openwrt-ramips-mt7621-xiaomi_mir3p-squashfs-factory.bin factory.bin
dd if=factory.bin bs=1M count=4 | mtd write - kernel1
mtd erase rootfs0
mtd erase rootfs1
mtd erase overlay
dd if=factory.bin bs=1M skip=4 | mtd write - rootfs0
@pellmen i think those mtd erases are a good idea in general...
also, @andreykiselev, if you don't mind (and especially if the mtd erase lines don't make a difference)
can you setup a tftpd server, tftp boot initramfs-kernel.bin (latest), then
ubiformat -v -y /dev/mtd9
ubiattach -p /dev/mtd9
then copy over sysupgrade.bin (to /tmp !) and:
sysupgrade sysupgrade.bin
and reboot.
I will still try to track down the "read only root" problem you're having (how annoying) but I want to see if the above give better results....
EDIT: i didn't say it very clearly, but from your boot log it's obvious that your overlay read/write partition failed to mount, so it switched to "ramoverlay" which obviously isn't persistent storage (RAM... cleared on power-off). the "why" is more interesting...
EDIT2: @andreykiselev can you also give me the output of
mtdinfo -a
2 Likes
@ilyas Thanks for looking at this! Regarding the NAND, well, I had to bitbang a little to flash it, so I think I have pretty good idea of its command set. I found that there should be MTD API in 4.14 kernel: https://www.kernel.org/doc/html/v4.14/driver-api/mtdnand.html, but I did not use kernel API before. Do you have any idea how to link a small C program with this API?
I don't have a router with me now, so will follow your instructions tomorrow...
[UPDATE 2019-03-14]
So, just reverted back to dev firmware and everything looks good with the persistence. Here is the boot log: https://ufile.io/zd72r. The /root is ro, so I cloned mtd1 partition into etc and it was still there after reboot. Trying to re-flash OpenWRT now...
[UPDATE]
No luck with mtd erase 
No persistence between reboots. Will do sysupgrade later...
[UPDATE]
root@OpenWrt:~# mtdinfo -a
Count of MTD devices: 10
Present MTD devices: mtd0, mtd1, mtd2, mtd3, mtd4, mtd5, mtd6, mtd7, mtd8, mtd9
Sysfs interface supported: yes
mtd0
Name: Bootloader
Type: nand
Eraseblock size: 131072 bytes, 128.0 KiB
Amount of eraseblocks: 2 (262144 bytes, 256.0 KiB)
Minimum input/output unit size: 2048 bytes
Sub-page size: 2048 bytes
OOB size: 64 bytes
Character device major/minor: 90:0
Bad blocks are allowed: true
Device is writable: false
mtd1
Name: Config
Type: nand
Eraseblock size: 131072 bytes, 128.0 KiB
Amount of eraseblocks: 2 (262144 bytes, 256.0 KiB)
Minimum input/output unit size: 2048 bytes
Sub-page size: 2048 bytes
OOB size: 64 bytes
Character device major/minor: 90:2
Bad blocks are allowed: true
Device is writable: true
mtd2
Name: Bdata
Type: nand
Eraseblock size: 131072 bytes, 128.0 KiB
Amount of eraseblocks: 2 (262144 bytes, 256.0 KiB)
Minimum input/output unit size: 2048 bytes
Sub-page size: 2048 bytes
OOB size: 64 bytes
Character device major/minor: 90:4
Bad blocks are allowed: true
Device is writable: false
mtd3
Name: Factory
Type: nand
Eraseblock size: 131072 bytes, 128.0 KiB
Amount of eraseblocks: 2 (262144 bytes, 256.0 KiB)
Minimum input/output unit size: 2048 bytes
Sub-page size: 2048 bytes
OOB size: 64 bytes
Character device major/minor: 90:6
Bad blocks are allowed: true
Device is writable: false
mtd4
Name: crash
Type: nand
Eraseblock size: 131072 bytes, 128.0 KiB
Amount of eraseblocks: 2 (262144 bytes, 256.0 KiB)
Minimum input/output unit size: 2048 bytes
Sub-page size: 2048 bytes
OOB size: 64 bytes
Character device major/minor: 90:8
Bad blocks are allowed: true
Device is writable: true
mtd5
Name: crash_syslog
Type: nand
Eraseblock size: 131072 bytes, 128.0 KiB
Amount of eraseblocks: 4 (524288 bytes, 512.0 KiB)
Minimum input/output unit size: 2048 bytes
Sub-page size: 2048 bytes
OOB size: 64 bytes
Character device major/minor: 90:10
Bad blocks are allowed: true
Device is writable: true
mtd6
Name: reserved0
Type: nand
Eraseblock size: 131072 bytes, 128.0 KiB
Amount of eraseblocks: 2 (262144 bytes, 256.0 KiB)
Minimum input/output unit size: 2048 bytes
Sub-page size: 2048 bytes
OOB size: 64 bytes
Character device major/minor: 90:12
Bad blocks are allowed: true
Device is writable: false
mtd7
Name: kernel_stock
Type: nand
Eraseblock size: 131072 bytes, 128.0 KiB
Amount of eraseblocks: 32 (4194304 bytes, 4.0 MiB)
Minimum input/output unit size: 2048 bytes
Sub-page size: 2048 bytes
OOB size: 64 bytes
Character device major/minor: 90:14
Bad blocks are allowed: true
Device is writable: true
mtd8
Name: kernel
Type: nand
Eraseblock size: 131072 bytes, 128.0 KiB
Amount of eraseblocks: 32 (4194304 bytes, 4.0 MiB)
Minimum input/output unit size: 2048 bytes
Sub-page size: 2048 bytes
OOB size: 64 bytes
Character device major/minor: 90:16
Bad blocks are allowed: true
Device is writable: true
mtd9
Name: ubi
Type: nand
Eraseblock size: 131072 bytes, 128.0 KiB
Amount of eraseblocks: 1964 (257425408 bytes, 245.5 MiB)
Minimum input/output unit size: 2048 bytes
Sub-page size: 2048 bytes
OOB size: 64 bytes
Character device major/minor: 90:18
Bad blocks are allowed: true
Device is writable: true
root@OpenWrt:~#
1 Like
On the last release there's no problems with it. I've tried it many times.
1 Like
ubinfo -a
?
and, if it wouldn't be too much trouble, could you send me your ubi partition (PM is fine)
dd if=/dev/mtd9 of=ubi.img
maybe by comparing it to mine I can see what's going wrong (hacking, really, but it might work...)
I'll ask my friend with micron do this too. Maybe something more?
1 Like
@ilyas Hm... it seems to work after sysupgrade... Here is the bootlog: https://ufile.io/gps1x. And here is the output of ubiinfo, but I'm not sure if there is any use of it now since everything seems to be okay:
root@OpenWrt:/# ubinfo -a
UBI version: 1
Count of UBI devices: 1
UBI control device major/minor: 10:59
Present UBI devices: ubi0
ubi0
Volumes count: 2
Logical eraseblock size: 126976 bytes, 124.0 KiB
Total amount of logical eraseblocks: 1964 (249380864 bytes, 237.8 MiB)
Amount of available logical eraseblocks: 0 (0 bytes)
Maximum count of volumes 128
Count of bad physical eraseblocks: 0
Count of reserved physical eraseblocks: 40
Current maximum erase counter value: 3
Minimum input/output unit size: 2048 bytes
Character device major/minor: 251:0
Present volumes: 0, 1
Volume ID: 0 (on ubi0)
Type: dynamic
Alignment: 1
Size: 45 LEBs (5713920 bytes, 5.4 MiB)
State: OK
Name: rootfs
Character device major/minor: 251:1
-----------------------------------
Volume ID: 1 (on ubi0)
Type: dynamic
Alignment: 1
Size: 1875 LEBs (238080000 bytes, 227.0 MiB)
State: OK
Name: rootfs_data
Character device major/minor: 251:2
[UPDATE] Just checked couple of times, and the files created on /root stay after reboot. Also, changes made with the interface are now persistent. Is there anything special with sysupgrade or flashing with tftp? I can keep testing if needed.
1 Like
crap....
did you tftpboot to do sysupgrade? or just sysupgrade from the (flash booted) system that wasn't working? did you do ubiformat first?
i want the broken system! 
if we assume that something about the installation is broken we can just have it run "ubiformat" on first boot, but that's not the same as fixing it (it's a hack)... need to figure out what's going wrong in the installation...
could you re-install (mtd erase and all) and give me ubinfo -a and your mtd9? in fact, can you give me your current (working, hopefully clean, no new data) 'mtd9' and your 'broken' (fails to mount) mtd9? (please? )
[EDIT in response to UPDATE|: yeah well now that you're properly mounting ubifs as root (not RAM-backed anymore) your changes would be persistent, and you'd be a happy customer i'm still in the process of figuring out just exactly what's going on during installation (my initial impulse is to always copy/paste code or treat it as a black box... sometimes unfortunately that doesn't work out so well ;))... but it's probably that the way we're laying down the bits initially (mtd write etc) isn't really such a good idea... and it works sometimes and doesn't work all the time (at least so far i know that "ubiformat" is the recommended way of laying down UBIFS systems... we directly write the headers and expect it to fix things up by itself later...)
anyway, i don't want to speak too soon, but it looks like we're getting somewhere (finally... i hope...)
thanks!
2 Likes
@ilyas Will test it as soon as I can!
Side note: Because I used raw flash image from other guy, I want to flash a tx calibration table (originally mtd4, now it's Factory) from my recovered image. How can I set Factory block writable to flash it?
1 Like
@andreykiselev i'll send you a kernel module to make read-only partitions writable... (in a bit...)
[UPDATED] here's the kernel module https://github.com/iscilyas/openwrt-r3p/releases/download/v0.1-rc1%2Bmt7615e-5/kmod-mtd-rw_4.14.104+git-20160214-1_mipsel_24kc.ipk
and here's how to use it:
PS: obviously I didn't test it... but it's a standard part of openwrt so it probably works
@pellmen if you could get your friend with Micron to install stock/install openwrt (using the instructions on the TOH), give full bootlogs, reboot, (full logs) then do sysupgrade and reboot (again full logs)... and possibly (depending on how much time he wants to invest) tftpboot and ubiformat and sysupgrade... that would give me at least 2 data points to work with... since @andreykiselev 's NAND is known to be in a rather strange state, it would be unsafe to draw general conclusions from only his experiences...
PS: i could swear there was someone on 4pda who was saying his configuration changes were disappearing after reboot (or was that here....) ... if that person reads this post, please also give me full bootlogs (at least)
Sorry, reached the limit of replies...
Here is ubinfo of a good block:
UBI version: 1
Count of UBI devices: 1
UBI control device major/minor: 10:59
Present UBI devices: ubi0
ubi0
Volumes count: 2
Logical eraseblock size: 126976 bytes, 124.0 KiB
Total amount of logical eraseblocks: 1964 (249380864 bytes, 237.8 MiB)
Amount of available logical eraseblocks: 0 (0 bytes)
Maximum count of volumes 128
Count of bad physical eraseblocks: 0
Count of reserved physical eraseblocks: 40
Current maximum erase counter value: 3
Minimum input/output unit size: 2048 bytes
Character device major/minor: 251:0
Present volumes: 0, 1
Volume ID: 0 (on ubi0)
Type: dynamic
Alignment: 1
Size: 45 LEBs (5713920 bytes, 5.4 MiB)
State: OK
Name: rootfs
Character device major/minor: 251:1
-----------------------------------
Volume ID: 1 (on ubi0)
Type: dynamic
Alignment: 1
Size: 1875 LEBs (238080000 bytes, 227.0 MiB)
State: OK
Name: rootfs_data
Character device major/minor: 251:2
And here is the block itself: https://ufile.io/d9rn3
Now the problem is that I cannot revert back to stock firmware to replicate standard failing installation... I run:
fw_setenv flag_try_sys1_failed 0
fw_setenv flag_try_sys2_failed 1
reboot
with dev firmware on a usb drive, but then it just reboots into OpenWRT and there is no red led when reset should be pressed... Is there anything I do wrong to recover stock?
P.S. Thanks for the write-enabler!
perhaps you accidentally write openwrt kernel to kernel_stock(kernel0) ??
& try setenv via uboot
@lolyinseo Just recovered stock with tftp... Bulletproof way
@ilyas So, after recovering original dev and then flashing OpenWRT in an easy way, the symptom is back again! Here is the ubinfo:
UBI version: 1
Count of UBI devices: 1
UBI control device major/minor: 10:59
Present UBI devices: ubi0
ubi0
Volumes count: 2
Logical eraseblock size: 126976 bytes, 124.0 KiB
Total amount of logical eraseblocks: 1964 (249380864 bytes, 237.8 MiB)
Amount of available logical eraseblocks: 0 (0 bytes)
Maximum count of volumes 128
Count of bad physical eraseblocks: 0
Count of reserved physical eraseblocks: 40
Current maximum erase counter value: 1
Minimum input/output unit size: 2048 bytes
Character device major/minor: 251:0
Present volumes: 0, 1
Volume ID: 0 (on ubi0)
Type: dynamic
Alignment: 1
Size: 45 LEBs (5713920 bytes, 5.4 MiB)
State: OK
Name: rootfs
Character device major/minor: 251:1
-----------------------------------
Volume ID: 1 (on ubi0)
Type: dynamic
Alignment: 1
Size: 1875 LEBs (238080000 bytes, 227.0 MiB)
State: OK
Name: rootfs_data
Character device major/minor: 251:2
And here is the bad ubi block: https://ufile.io/v5azb
So, no persistence, as it used to be. Should I keep it running for now?
Also, here is a boot log of the latest (bad) system: https://ufile.io/19pgx
P.S. Write enabler worked like a charm and I got my serial and MACs back!