Help to fix OpenWrt builds for Mofi3500

Context: I'm new to building OpenWrt (and using git) though I have experience using other version control and building other stuff... kernel for custom video drivers, other distros, etc. I would like to improve the install experience for one particular model (USB port does not work and WAN port does not work).

I have several questions but first of all...

I followed Build system - Usage. It says "it is recommended that you work with the latest sources" and suggests git pull. But for initial testing (e.g. so I can still use opkg pointed at the 18.06.1 download server), I want a build matching the 18.06.1 release - kernel 4.14.63, so I did git clone https://github.com/openwrt/openwrt.git -b openwrt-18.06 (as this was the only 18.06 branch name I could find; I forget how I was looking for branch names though).

My router is Mofi3500 so I set (and did no other changes in make menuconfig):
target: "MediaTek Ralink MIPS"
sub-target: "RT3x5x/RT5350 based boards"
target profile: "MoFi Network MOFI3500-3GN"

The build went fine and installed to the router fine, but in the firmware made from this was labelled (in /etc/banner) 18.06-SNAPSHOT r7390-dd6d554 and the kernel was 4.14.81.

So, how do I point git at the exact 18.06.1 release?

(and maybe this should be a little more described on the Build system - Usage page?)

For the WAN issue, to get a fix directly into the firmware image (not just fix it manually after install) I've experimented with two ways of fixing this:

  1. target/linux/ramips/dts/MOFI3500-3GN.dts (mediatek,portmap = <0x3e> instead of 2f as it is now), this seemed to fix which port would be used as WAN, but had other issues, probably firewall changes would be needed to match.
  2. files/etc/config/network copying back from my installed image, then changing the vlan 1 section port from "0 1 2 3 5 6t" to "1 2 3 4 5 6t" (removed 0, added 4), and the vlan 2 section port from "4 6t" to "0 6t" (removed 4, added 0). But it seems like this is just a fix in an overlay file put into the final image, and should be fixed "higher up" in the make process (but specific to this target). How is the /etc/config/network file built from various parts, customized by various features enabled in menuconfig, customized by target, and/or customized by various packages being enabled? Or is it even put together on firstboot, as I don't see a config/network file in my build_dir that matches the end result on the router?

Where is the "correct" place to fix this?

I have a similar question for the USB port issue, and others, but I'll leave it at that for now.

The release has a git tag, as is common convention.

~/devel/openwrt$ git tag | fgrep 18.06
v18.06.0
v18.06.0-rc1
v18.06.0-rc2
v18.06.1

~/devel/openwrt$ git checkout v18.06.1

There are also branches for development and the "backports" to the 18.06 line

  remotes/origin/master
  remotes/origin/openwrt-18.06
1 Like

okay great, on doing this... (plus make distclean, feeds update, feeds install, menuconfig again changing only target/subtarget/profile, defconfig, make download, diffconfig, and finally make) ...I get a firmware image labelled in /etc/banner as "18.06.1, r7258-5eb055306f" and with kernel 4.14.63.

Next issue (and this was the same before when I was on 18.06-SNAPSHOT) is that I don't have the same set of packages as the project build for this model's ...sysupgrade.bin. In particular no web GUI until I opkg update, opkg install luci.

I understand this is controlled in my make menuconfig choices but how can I fetch the same choices that the overall project used for my router model?

It's been a while since I've had LuCI in a build, but you should be able to add the basic LuCI functionality using make menuconfig with

LuCI > Collections > luci

or, as I suggest TLS support, either luci-ssl, luci-ssl-openssl, or (at least on master!) luci-ssl-nginx

I don't know how much flash and RAM your device has, so I'd take a first cut without nginx.

Sorry I'm a noob at this, I may be asking something wrong here. Maybe using wrong jargon.

When I put this sysupgrade.bin firmware on my router, I can surf to the router at 192.168.1.1 and see the admin pages.

When I run this build locally (now with the source as of the correct v18.06.1 tag - thanks for that) and put the resulting file from bin/targets/ramips/rt305x/openwrt-ramips-rt305x-mofi3500-3gn-squashfs-sysupgrade.bin on my router, I cannot surf to it at 192.168.1.1, and ps -w |grep http shows no web server running.

This is just the first difference I noticed. There could be lots of other packages "missing" that I just haven't noticed yet. So that's why I'm wondering how do I make my build (and package list) the same. Is the menuconfig options available anywhere (or also version controlled?) that were used for the project-provided downloadable builds?

The "release" builds include LuCI, by default. The "snapshot" (and self-compiled) builds do not.

The "release" contents can be found through files on http://downloads.openwrt.org/releases/18.06.1/targets/ramips/rt305x/
(look at the small list at the bottom of the page)

config.seed is generally comparable to the output from ./scripts/diffconfig.sh and that can help you identify which packages that are part of a release build that you may want to add to your own build. config.seed isn't a "minimal" list in that adding something like one of the LuCI Collections will bring in many packages.

(Also, config.seed is a good starting point, but may not be completely compatible with builds off master, or even on the same release branch at a later point in time.)

Edit:

Looking at "your" config.seed, building all kernel modules for all variants and the SDK is probably is not what you want. Looking further down, it looks like the main things in the build that are added is a basic LuCI set up (non-encrypted):

CONFIG_PACKAGE_libiwinfo-lua=y
CONFIG_PACKAGE_liblua=y
CONFIG_PACKAGE_liblucihttp=y
CONFIG_PACKAGE_liblucihttp-lua=y
CONFIG_PACKAGE_libubus-lua=y
CONFIG_PACKAGE_lua=y
CONFIG_PACKAGE_luci=y
CONFIG_PACKAGE_luci-app-firewall=y
CONFIG_PACKAGE_luci-base=y
CONFIG_PACKAGE_luci-lib-ip=y
CONFIG_PACKAGE_luci-lib-jsonc=y
CONFIG_PACKAGE_luci-lib-nixio=y
CONFIG_PACKAGE_luci-mod-admin-full=y
CONFIG_PACKAGE_luci-proto-ipv6=y
CONFIG_PACKAGE_luci-proto-ppp=y
CONFIG_PACKAGE_luci-theme-bootstrap=y
CONFIG_PACKAGE_rpcd=y
CONFIG_PACKAGE_rpcd-mod-rrdns=y
CONFIG_PACKAGE_uhttpd=y

I would guess that adding the "luci" Collection as described above would pull all of this in with one change in make menuconfig.

From my other issues threads (linked in OP), it seems to be 16 MB flash, 32 MB RAM. Not 100% sure what's a "correct" way to verify that but I see from dmesg and free:

[    0.592706] 1f000000.cfi: Found 1 x16 devices at 0x0 in 16-bit bank. Manufacturer ID 0x0000c2 Chip ID 0x0022cb
...
             total       used       free     shared    buffers     cached
Mem:         28160      18260       9900        124       1780       5596
-/+ buffers/cache:      10884      17276
Swap:            0          0          0

With 16 MB, you should be able to build in a "standard" LuCI install with TLS enabled. I personally use OpenSSL as I use the OpenSSH clients and servers, but the mBED library (which I believe to be the "default" TLS if you just select luci-ssl) is very good as well.

Okay thanks. I really appreciate the build-noob help.

I used the online config.seed, copied it over my .config, did make defconfig (per the Build system - Usage page this will expand it to full config), then make menuconfig (to, as you said, remove all the other variants), make clean, make download, and finally make. This is in progress now.

Aside: why does make -j9 give me an error this time around? whereas make -j1 (so far) is working. Do I have to go back to make distclean (and repeat make download) for every config change? or git change? or when?

In my experience, the download phase sometimes doesn't complete in time for the world phase, so that the needed files haven't been downloaded yet. My "mantra" is

make -jN clean download world

which completes each phase ("target" in make parlance) in order.

The first build that needs to build the toolchain takes a while. In my experience a "plain" clean does not require you to rebuild the toolchain.

Enabling ccache in the developer/advanced options can help speed subsequent builds -- ccache looks at the sources and, if they haven't changed, re-uses the previous result. At least for projects of the complexity of OpenWrt and how its build system works, I haven't seen any ill effects of using ccache for personal builds.

Edit:

distclean is generally more than needed, as it wipes "everything". I'd use it if changing branches in the same root directory. I find a better way to work with multiple branches is using the git worktree feature to check out the "other" branch into a different directory.

1 Like

Okay great. Now I have a way to more or less replicate the downloadable project-provided 18.06.1 firmware for my router. And replicate the problems.

So now I am back to my first post... where is the "correct" place to fix the WAN port issue, target/linux/ramips/dts/MOFI3500-3GN.dts, or files/etc/config/network (which seems like just an overlay solution), or somewhere else? I have grep'd and find'd all around the source tree and I can't see a template of the /etc/config/network file or where there is a build step that generates it by some formula.

(i.e. to fix this for all users for future builds, not just fix it on my router here after installing the firmware.)

And I suppose if I am hoping to contribute a fix back to the project, I should switch my git back to the 18.06 branch, not my current v18.06.1 tag, right? or master? (switch to git checkout openwrt-18.06 rather than git checkout v18.06.1 as discussed in post 2)

In addition to that WAN issue (which, if I can fix, would be committed back to the project), in my own build I want OpenVPN but had this issue, so is there a way in git to have most of my openwrt tree pointing at the v18.06.1 tag, but have one subfolder (feeds/luci/applications/luci-app-openvpn) pointing to a later tag (or the moving-target openwrt-18.06 branch even)?

Okay I still don't quite get what I'm doing wrong with this. Let's say after a successful -j1 build, I *-enable one or two additional modules in make menuconfig. Then (with or without make clean here I seem to have the same problem) I do, as separate consecutive commands, make download, and ionice -c 3 nice -n19 make -j9 as suggested on the Build system - Usage page. I still get an error (in the latest example):

...
 make[3] -C feeds/luci/libs/luci-lib-nixio clean-build
 make[3] -C feeds/luci/libs/luci-lib-nixio compile
 make[3] -C feeds/luci/contrib/package/lucihttp compile
 make[2] diffconfig
make -r world: build failed. Please re-run make with -j1 V=s to see what's going on
/home/mmuc/programming/mofi/openwrt/include/toplevel.mk:216: recipe for target 'world' failed
make: *** [world] Error 1

But following that, ionice -c 3 nice -n19 make -j1 gives a successful build, firmware .bin file, and all.

So after a successful build, am I supposed to do make clean if changing only a few modules in make menuconfig?

And why has -j9 never worked for me yet despite doing make download first as a separate command?

In my experience, either

make download
make -jN

or

make -jN download world

is often required if there have been changes to the config, feeds, or source, to ensure that all the files have been obtained prior to beginning parallel compilation.

I always clean as it doesn't cost me much build time (I use ccache) and it gives me more confidence that dependency changes or changes in dependencies don't result in an "improper" build.


As long as I ensure that download completes before any compilation begins, parallel compilation has been very reliable for me.

Other than source errors on my part, the only other time I have run into "failed" builds is on switching branches without fully synchronizing things, especially when the cross-compile tool chain changes significantly.

Okay I wiped my build directory and started fresh (seems like maybe my git status was in the "detached head" state), and it has been mostly better. Sometimes, even doing (all as separate consecutive commands) make clean, make download, and ionice -c 3 nice -n19 make -j9, I still get a build error. However rather than drop it back to -j1 as the error suggests, I just repeat the -j9 command and the second time it works.

Anyway that is working close enough, but I now have an Out of memory error : - (

So my next hurdle "to fix OpenWrt builds for Mofi3500" (as subject line states ; - ) is that AFTER I successfully used config.seed to make a firmware very similar to the project-downloadable 18.06.1 firmware (the same 97 packages present after installation), I started adding some additional packages into the firmware so it will be immediately useful after flashing for certain tasks that I need, and now I am getting "Out of memory" errors on my first few commands after first boot. E.g.

root@openwrt:~# opkg update
Downloading http://downloads.openwrt.org/releases/18.06.1/targets/ramips/rt305x/packages/Packages.gz
*** Failed to download the package list from http://downloads.openwrt.org/releases/18.06.1/targets/ramips/rt305x/packages/Packages.gz
...
[plus 4 similar lines]
...
Collected errors:
 * xsystem: wget: vfork: Out of memory.
 * opkg_download: Failed to download http://downloads.openwrt.org/releases/18.06.1/targets/ramips/rt305x/packages/Packages.gz, wget returned -1.
...
[plus 4 similar lines]
...
root@openwrt:~# free
             total       used       free     shared    buffers     cached
Mem:         28160      19912       8248       1024       1632       4028
-/+ buffers/cache:      14252      13908
Swap:            0          0          0

The LuCI status page seems to show lots of memory available although granted I don't know how to interpret it:

Total Available: 7968 kB / 28160 kB (28%)
Free: 6336 kB / 28160 kB (22%)
Buffered: 1632 kB / 28160 kB (5%)

Are there any docs on checking if we're going to be over flash or RAM size limits or docs with tips on how to stay under such limits?

(by "adding packages" I mean with make menuconfig changing from "m" to "*" to be built-in instead of modules.)
(and by "useful for certain tasks that I need" I mean that I added usbutils, kmod-usb-serial-ftdi, coreutils-stty, luci-app-openvpn, and about 15 others that are dependencies of these - I can provide exact diffs of the build config or opkg list-installed before and after if the specifics are helpful.)

You're not going to like the answer, but devices with 32 MB of RAM have been "on life support" for a while now.

Devices with ≤4MB flash and/or ≤32MB ram will work but they will be very limited (usually they can't install or run additional packages) because they have low RAM and flash space.

Insufficient RAM for stable operation

  • 32 MB can work for minimal router/AP functions, but may repeatedly “crash”, depending on your hardware and use case
  • 64 MB may still have issues with stability, depending on your hardware and use cases
  • 128 MB or more is recommended if software past basic router/AP functionality is to be used

Despite the title, the RAM size is actually a much harder limit, affecting much more than just 4 MB flash devices. If you look through the commit log, you will notice quite some efforts to get opkg to use less RAM, but it's still a problem with just 32 MB RAM (even for normal operations, before touching opkg). Likewise you already are in trouble with sysupgrade and trying to flash a (larger) 6-8 MB firmware on a device with just 32 MB RAM, unless you really clean up services and loaded kernel modules manually beforehand, there is a high risk that you'll oom during the sysupgrade and brick your device for good (requiring bootloader/ tftp assistance to recover in the best case).

Okay thanks, yes I found that same page shortly after my post.

So as I try to remove some unneeded packages from my build, how do I know when I've improved? is it more about the size of the ...sysupgrade.bin file (currently 4980913 bytes)? or more about free or some other command on the running router after installing (and which numbers need to be under what limits)?

There isn't a way to tell from "by inspection" from the build artifacts how much memory it will consume while running.

Yes, free memory at run time is a huge concern and free is one way to look at that.

My Archer C7s, running only APs, routing, NTP, crond, sshd (with a login session and free), and logging "idles" at ~23 Mbytes of RAM right now. (Note that there is no LuCI or any non-critical services or applications in that.)

jeff@office:~$ free
              total        used        free      shared  buff/cache   available
Mem:         124500       23112       77440         380       23948       64648
Swap:             0           0           0

As stated in that link:

32 MB can work for minimal router/AP functions, but may repeatedly “crash”, depending on your hardware and use case

The sysupgrade problem is not to be trivialized either. During the upgrade process, in RAM you need to have:

  • Running kernel
  • Any other running services
  • The image itself (that's 5 MB right there, from your previous post)
  • The "backup" of config
  • The sysupgrade scripts
  • All utilities and libraries required to run the sysupgrade scripts
  • Enough free RAM to run those scripts

Failure at any point when the ROM image is being written will likely result in at least a soft-brick of the router, perhaps worse, depending on the router's flash layout and what is written during sysupgrade.