Netgear WAC510 repair "can't create '/var/config': No space left on device"

Hello! I'm a first-time contributor on this forum. I'm not sure where else I should put this so I guess I'll put it here. Thanks in advance!

I recently purchased a Netgear WAC510 AP (that runs OpenWrt!) with a boot problem. I opened up the AP and connected the UART header to a PL-2303HX TTL-USB adapter. Now I can interface with ipq40xx and SDK as root.

I grabbed the bootlog from the WAC510 and compared it to the OEM bootlog (see link and diff output, below). I think I can see what the problem is: I get a “cp: can't create '/var/config': No space left on device” error which leads to a slew of “No such file or directory” errors.

So I checked the available space:

[root@SDK ~]# df -a
Filesystem                Size      Used Available Use% Mounted on
mtd:ubi_rootfs           24.5M     24.5M         0 100% /
proc                         0         0         0   0% /proc
tmpfs                   120.9M    100.0K    120.8M   0% /tmp
devpts                       0         0         0   0% /dev/pts
none                    120.9M      2.4M    118.5M   2% /etc
none                         0         0         0   0% /sys
/dev/mtdblock11          15.0M     14.8M    248.0K  98% /var

/var is 98% full! So my guess is a bad upgrade caused junk data to get stuck in the memory that isn’t getting cleared? I’d like clear the memory and attempt a clean boot to factory condition and hopefully sidestep having to flash new firmware over TFTP. But I don’t know what commands to use to wipe the memory. Can anyone point me in the right direction? Thank you!

Netgear WAC510 specs, OEM firmware, UART pinouts, OEM bootlog: https://openwrt.org/toh/netgear/wac510

Below is part of the output of a diff command comparing bootlog0 (OEM) to bootlog1 (mine). The entire diff output is here: https://pastebin.com/31nEy6hu

My full bootlog with all errors is here: https://pastebin.com/hA0NbJ5t

$ diff -aw bootlog0.txt bootlog1.txt

333c317
< e[1;29mMounting var to jffs2.      e[1;0m[   12.396526] jffs2: notice: (123) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
---
> e[1;29mMounting var to jffs2.      e[1;0m[   11.480949] jffs2: notice: (123) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
348a333
> sh: /var/log/core.node.2149-WAC510_V1.3.0.10.gz: unknown operand
355d339
< Watching /var/log
359c343,348
< e[1;29mChecking database.          e[1;0me[1;32m[DONE]e[1;0m
---
> e[1;29mChecking database.          e[1;0me[1;33m[COPYING DEFAULT]e[1;0m
> 
> e[1;29m                            e[1;0mroot: Restoring configuration from file /etc/default-config.
> cp: can't create '/var/config': No space left on device
> root: Error copying config file /etc/default-config.
> e[1;32m[DONE]e[1;0m
361c350,368
< e[1;29mVerifing checksum.          e[1;0me[1;32m[DONE]e[1;0m
---
> e[1;29mVerifing checksum.          e[1;0mmd5sum: can't open '/var/config': No such file or directory
> md5sum: WARNING: 1 of 1 computed checksums did NOT match
> e[5me[1;33m[WRONG CHECKSUM]e[0m
> 
> e[1;29m                            e[1;0mroot: Restoring configuration from file /etc/default-config.
> cp: can't create '/var/config': No space left on device
> root: Error copying config file /etc/default-config.
> e[1;32m[DONE]e[1;0m
> cat: can't open '/var/config': No such file or directory
> sh: 1: unknown operand
> /usr/local/bin/check_config: line 7: can't open /var/config: no such file
> sh: 1275: unknown operand
> md5sum: can't open '/var/config': No such file or directory
> md5sum: WARNING: 1 of 1 computed checksums did NOT match
> grep: /var/config: No such file or directory
> sh: 5000: unknown operand
> sh: 5001: unknown operand
> sh: 156: unknown operand
> grep: /var/config: No such file or directory
363a371
> Watching /var/log
366c374
< 6 bytes (6B) copied, 0.009219 seconds, 650B/s
---
> 6 bytes (6B) copied, 0.006182 seconds, 970B/s
369c377
< 6 bytes (6B) copied, 0.007613 seconds, 788B/s
---
> 6 bytes (6B) copied, 0.006015 seconds, 997B/s
372c380
< 12064 bytes (11.8KB) copied, 0.116933 seconds, 100.8KB/s
---
> 12064 bytes (11.8KB) copied, 0.071392 seconds, 165.0KB/s
375,376c383,387
< 12064 bytes (11.8KB) copied, 0.101445 seconds, 116.1KB/s
< e[1;32m[DONE]e[1;0m
---
> 12064 bytes (11.8KB) copied, 0.060612 seconds, 194.4KB/s
> The command setCountryID needs exactly 1 argument(s)...
> e[1;31m[FAILED]e[1;0m
> grep: /var/config: No such file or directory
> cat: can't open '/var/config': No such file or directory
378c389,391
< e[1;29mCreating vap interface.     e[1;0me[1;32m[DONE]e[1;0m
---
> e[1;29mCreating vap interface.     e[1;0msh: 1: unknown operand
> sh: 1: unknown operand
> e[1;32m[DONE]e[1;0m
389a403
> grep: /var/config: No such file or directory
391c405
< e[1;29mStarting Translator...      e[1;0mstart-stop-daemon: can't execute '/usr/bin/log_ro': No such file or directory
---
> e[1;29mStarting Translator...      e[1;0m/etc/init.d/rcS: /etc/init.d/S090translators.sh: line 24: can't open /var/config: no such file
394c408,409
< e[1;29mStarting Translator...      e[1;0me[1;32m[password]e[1;0m
---
> e[1;29mStarting Translator...      e[1;0m/etc/init.d/rcS: /etc/init.d/S090translators.sh: line 24: can't open /var/config: no such file
> e[1;32m[password]e[1;0m
396c411,412
< e[1;29mStarting Translator...      e[1;0me[1;32m[ssh]e[1;0m
---
> e[1;29mStarting Translator...      e[1;0m/etc/init.d/rcS: /etc/init.d/S090translators.sh: line 24: can't open /var/config: no such file
> e[1;32m[ssh]e[1;0m
398c414,415
< e[1;29mStarting Translator...      e[1;0me[1;32m[snmp]e[1;0m
---
> e[1;29mStarting Translator...      e[1;0m/etc/init.d/rcS: /etc/init.d/S090translators.sh: line 24: can't open /var/config: no such file
> e[1;32m[snmp]e[1;0m
400c417,418
< e[1;29mStarting Translator...      e[1;0me[1;32m[telnet]e[1;0m
---
> e[1;29mStarting Translator...      e[1;0m/etc/init.d/rcS: /etc/init.d/S090translators.sh: line 24: can't open /var/config: no such file
> e[1;32m[telnet]e[1;0m
402,403c420
< e[1;29mStarting Translator...      e[1;0mawk: /tmp/bridge_vlan_script: No such file or directory
< awk: /tmp/bridge_dhcpc_script: No such file or directory
---
> e[1;29mStarting Translator...      e[1;0m/etc/init.d/rcS: /etc/init.d/S090translators.sh: line 24: can't open /var/config: no such file
406,411c423
< e[1;29mStarting Translator...      e[1;0mtap-mac: eth0 A2-40-A0-7B-9E-0Fset_dot3_power: Dot3 POE-MDI successfully set for eth0
< set_dot3_power: Dot3 POE-MDI successfully set for eth1
< mv: can't rename '/tmp/dibbler-client.conf_tm': No such file or directory
< ip: RTNETLINK answers: Numerical result out of range
< ash: ::0: unknown operand
< /bin/sh: /bin/ip: not found
---
> e[1;29mStarting Translator...      e[1;0m/etc/init.d/rcS: /etc/init.d/S090translators.sh: line 24: can't open /var/config: no such file
414c426,427
< e[1;29mStarting Translator...      e[1;0me[1;32m[country_tr]e[1;0m
---
> e[1;29mStarting Translator...      e[1;0m/etc/init.d/rcS: /etc/init.d/S090translators.sh: line 24: can't open /var/config: no such file
> e[1;32m[country_tr]e[1;0m
416c429,430
< e[1;29mStarting Translator...      e[1;0me[1;32m[hostapd_tr]e[1;0m
---
> e[1;29mStarting Translator...      e[1;0m/etc/init.d/rcS: /etc/init.d/S090translators.sh: line 24: can't open /var/config: no such file
> e[1;32m[hostapd_tr]e[1;0m
418c432,433
< e[1;29mStarting Translator...      e[1;0me[1;32m[qos_translator]e[1;0m
---
> e[1;29mStarting Translator...      e[1;0m/etc/init.d/rcS: /etc/init.d/S090translators.sh: line 24: can't open /var/config: no such file
> e[1;32m[qos_translator]e[1;0m
420c435,436
< e[1;29mStarting Translator...      e[1;0me[1;32m[captive_conf_tr]e[1;0m
---
> e[1;29mStarting Translator...      e[1;0m/etc/init.d/rcS: /etc/init.d/S090translators.sh: line 24: can't open /var/config: no such file
> e[1;32m[captive_conf_tr]e[1;0m
422c438,439
< e[1;29mStarting Translator...      e[1;0me[1;32m[dhcp]e[1;0m
---
> e[1;29mStarting Translator...      e[1;0m/etc/init.d/rcS: /etc/init.d/S090translators.sh: line 24: can't open /var/config: no such file
> e[1;32m[dhcp]e[1;0m
424c441,442
< e[1;29mStarting Translator...      e[1;0me[1;32m[ntp]e[1;0m
---
> e[1;29mStarting Translator...      e[1;0m/etc/init.d/rcS: /etc/init.d/S090translators.sh: line 24: can't open /var/config: no such file
> e[1;32m[ntp]e[1;0m
426c444,445
< e[1;29mStarting Translator...      e[1;0me[1;32m[timezone]e[1;0m
---
> e[1;29mStarting Translator...      e[1;0m/etc/init.d/rcS: /etc/init.d/S090translators.sh: line 24: can't open /var/config: no such file
> e[1;32m[timezone]e[1;0m
428c447,448
< e[1;29mStarting Translator...      e[1;0me[1;32m[dump_config_logs_tr]e[1;0m
---
> e[1;29mStarting Translator...      e[1;0m/etc/init.d/rcS: /etc/init.d/S090translators.sh: line 24: can't open /var/config: no such file
> e[1;32m[dump_config_logs_tr]e[1;0m
430c450,451
< e[1;29mStarting Translator...      e[1;0me[1;32m[urlfile_block_domain]e[1;0m
---
> e[1;29mStarting Translator...      e[1;0m/etc/init.d/rcS: /etc/init.d/S090translators.sh: line 24: can't open /var/config: no such file
> e[1;32m[urlfile_block_domain]e[1;0m
432c453,454
< e[1;29mStarting Translator...      e[1;0me[1;32m[urlfile_block_host]e[1;0m
---
> e[1;29mStarting Translator...      e[1;0m/etc/init.d/rcS: /etc/init.d/S090translators.sh: line 24: can't open /var/config: no such file
> e[1;32m[urlfile_block_host]e[1;0m
434c456,457
< e[1;29mStarting Translator...      e[1;0me[1;32m[urlfile_block_expr]e[1;0m
---
> e[1;29mStarting Translator...      e[1;0m/etc/init.d/rcS: /etc/init.d/S090translators.sh: line 24: can't open /var/config: no such file
> e[1;32m[urlfile_block_expr]e[1;0m
436c459,460
< e[1;29mStarting Translator...      e[1;0me[1;32m[urlenable]e[1;0m
---
> e[1;29mStarting Translator...      e[1;0m/etc/init.d/rcS: /etc/init.d/S090translators.sh: line 24: can't open /var/config: no such file
> e[1;32m[urlenable]e[1;0m
438c462,463
< e[1;29mStarting Translator...      e[1;0me[1;32m[devicemode]e[1;0m
---
> e[1;29mStarting Translator...      e[1;0m/etc/init.d/rcS: /etc/init.d/S090translators.sh: line 24: can't open /var/config: no such file
> e[1;32m[devicemode]e[1;0m
440c465,469
< e[1;29mStarting Translator...      e[1;0me[1;32m[miniupnpd_tr]e[1;0m
---
> e[1;29mStarting Translator...      e[1;0m/etc/init.d/rcS: /etc/init.d/S090translators.sh: line 24: can't open /var/config: no such file
> e[1;32m[miniupnpd_tr]e[1;0m
> cat: can't open '/var/config': No such file or directory
> ifconfig: brtrunk: error fetching interface information: Device not found
> cat: can't open '/var/config': No such file or directory
444a474
> ifconfig: brtrunk: error fetching interface information: Device not found
451,452c481,485
< updating from update1 server
< Internet is Disconnected
---
> ifconfig: brtrunk: error fetching interface information: Device not found
> grep: /var/config: No such file or directory
> grep: /var/config: No such file or directory
> sh: 1: unknown operand
> sh: 1: unknown operand
457,458d489
< 
< 
461,462d491
< 
< 
465,467c494,496
< 
< 
< Netgear7B9E0F login: 
---
> 
SDK login: ifconfig: brtrunk: error fetching interface information: Device not found
> ifconfig: brtrunk: error fetching interface information: Device not found
> ifconfig: brtrunk: error fetching interface information: Device not found

When you say "upgrade" -- can you please specify what you mean? Did you do an opkg upgrade (or the LuCI based equivalent) of packages on your router? Or did you upgrade using sysupgrade?

1 Like

What worries me more, it /var/ apparently being its own mtd, rather than a symlink to a tmpfs backed /tmp/ - as that doesn't look like OpenWrt at all (and if it isn't, we have no idea how it's supposed to look like).

3 Likes

very good point!

@beni9n - what is the output of the following:

ubus call system board
1 Like

Thank you for getting back to me!

When you say "upgrade" -- can you please specify what you mean?

I shouldn't have added that. I'm speculating that the error happened during an upgrade. I bought it off of eBay as-is so I don't know what happened to it. The previous owner said they were unable to test it.

Did you do an opkg upgrade (or the LuCI based equivalent) of packages on your router? Or did you upgrade using sysupgrade?

I have not attempted either. My only other attempt to fix it was to hard-reset it. For the WAC510, you hold the reset button for about 6 seconds until the power light starts flashing amber. But when I reset it the power light flashes amber permanently. I could try and post the bootlog from a hard reset if you want.

So we don't even know if there is OpenWrt on the device in any form whatsoever??
Let's start with trying to establish what is on there:

1 Like

Thank you for the idea. I tried your command in IPQ40xx but it doesn't recognize the command:

(IPQ40xx) # ubus call system board
Unknown command 'ubus' - try 'help'

I tried it in SDK too but it wasn't recognized:

[root@SDK ~]# ubus call system board
-sh: ubus: not found

Here's the list of available IPQ40xx commands:

(IPQ40xx) # help
?       - alias for 'help'
base    - print or set address offset
board_mini_parameters_set- set <SN> <DEFAULT_SSID> <lan address> <wlan address>
board_parameters_set- board_parameters_set <SN> <PRID> <HWID> <SUBHWID> <REGION_INFO> <BOARD_TYPE> <DEV_TYPE> <DEF_SSID> <DEF_WIFI_PASSWD> <DEF_IPADDR>         <DEF_ADMIN_USER> <DEF_ADMIN_PASSWD> <LAN_BASEMAC> <WLAN_BASEMAC>
board_parameters_show- shows <SN> <PRID> <HWID> <SUBHWID> <REGION_INFO> <BOARD_TYPE> <DEV_TYPE> <DEF_SSID> <DEF_WIFI_PASSWD> <DEF_IPADDR>         <DEF_ADMIN_USER> <DEF_ADMIN_PASSWD> <LAN_BASEMAC> <WLAN_BASEMAC>
bootipq - bootipq from flash device
bootm   - boot application image from memory
bootp   - boot image via network using BOOTP/TFTP protocol
chpart  - change active partition
cmp     - memory compare
cp      - memory copy
crc32   - checksum calculation
dhcp    - boot image via network using DHCP/TFTP protocol
echo    - echo args to console
env     - environment handling commands
exit    - exit script
false   - do nothing, unsuccessfully
fdt     - flattened device tree utility commands
fuseipq - fuse QFPROM registers from memory
go      - start application at address 'addr'
help    - print command description/usage
i2c     - I2C sub-system
imxtract- extract a part of a multi-image
loop    - infinite loop on address range
md      - memory display
mii     - MII utility commands
mm      - memory modify (auto-incrementing address)
mmc     - MMC sub system
mmcinfo - display MMC info
mtdparts- define flash/nand partitions
mtest   - simple RAM read/write test
mw      - memory write (fill)
nand    - NAND sub-system
nboot   - boot from NAND device
nm      - memory modify (constant address)
pci     - list and access PCI Configuration Space
ping    - send ICMP ECHO_REQUEST to network host
printenv- print environment variables
reset   - Perform RESET of the CPU
run     - run commands in an environment variable
saveenv - save environment variables to persistent storage
setenv  - set environment variables
sf      - SPI flash sub-system
showvar - print local hushshell variables
smeminfo- print SMEM FLASH information
source  - run script from memory
test    - minimal test like /bin/sh
tftpboot- boot image via network using TFTP protocol
tftpput - TFTP put command, for uploading files to a server
true    - do nothing, successfully
uartrd  - uartrd read from second UART
uartwr  - uartwr to second UART
ubi     - ubi commands
usb     - USB sub-system
usbboot - boot from USB device
version - print monitor, compiler and linker version

So it is not OpenWrt at all.

I'd recommend that you start by trying to install OpenWrt per the instructions on the device info page

2 Likes

My bootlog (https://pastebin.com/hA0NbJ5t) refers to OpenWrt three times:

121. Description:  ARM OpenWrt Linux-3.14.43
138. Description:  ARM OpenWrt qcom-ipq40xx-ap.dkxx device tree blob
157. [    0.000000] Linux version 3.14.43-V1.3.0.10 (root@cbuap-build2.netgear.com) (gcc version 4.8.3 (OpenWrt/Linaro GCC 4.8-2014.01 r43761) ) #1 SMP PREEMPT Mon Jun 26 01:25:18 PDT 2017

I took that to mean that it's running some version of OpenWrt. Perhaps a version modified by Netgear?

By the way, I ran the 'version' command to get the print monitor, compiler and linker version:

(IPQ40xx) # version 
U-Boot 2012.07-V1.2.0.0 [local,local] (Oct 26 2016 - 02:39:01)
arm-openwrt-linux-gcc (OpenWrt/Linaro GCC 4.8-2014.01 r43761) 4.8.3
GNU ld (GNU Binutils) 2.22

It might look like OpenWrt to you, but it very obviously isn't - on the contrary, it has all the tell-tale signs of an OEM firmware and has pretty much nothing in common with the OpenWrt we'd know.

Think about it, no (normal-) user is ever going to (attach a serial console and-) see those OpenWrt references, so why spend time and money to get rid of them. Follow the vendor's documentation to recover you device via push-button tftp, that 'should' do the right thing.

I'm a bit lazy and just insert a canned reply here, summarizing the situation, it doesn't completely fit the question, but still answers a lot of it on a high(er) level:

It appears you are using firmware that is not from the official OpenWrt project.

When using forks/offshoots/vendor-specific builds that are "based on OpenWrt", there may be many differences compared to the official versions (hosted by OpenWrt.org). Some of these customizations may fundamentally change the way that OpenWrt works. You might need help from people with specific/specialized knowledge about the firmware you are using, so it is possible that advice you get here may not be useful.

You may find that the best options are:

  1. Install an official version of OpenWrt, if your device is supported (see https://firmware-selector.openwrt.org).
  2. Ask for help from the maintainer(s) or user community of the specific firmware that you are using.
  3. Provide the source code for the firmware so that users on this forum can understand how your firmware works (OpenWrt forum users are volunteers, so somebody might look at the code if they have time and are interested in your issue).

If you believe that this specific issue is common to generic/official OpenWrt and/or the maintainers of your build have indicated as such, please feel free to clarify.

4 Likes

A lot of hardware vendors will take OpenWrt and create a custom/proprietary fork for their own firmware development. This is highly customized and without a doubt is fundamentally different in how it operates, even though it has a common ancestor (think of it like genetics -- human embryos actually develop gill structures because of common genes shared with fish, but we're about as similar to fish as the custom firmware is to OpenWrt).

3 Likes

It isn't. The prompt (and OP) clearly states it's OEM stuff. Look at the 'SDK' in the prompt. And the kernel version this thing uses.

Welcome to the wonderful world of GPL violations. 3.14 is a kernel OpenWrt never used. OpenWrt gained IPQ40xx support in 2018. The first release supporting IPQ40xx was 18.06 and it used a 4.14 kernel.

1 Like

@Borromini - Yes... you have a point. The fact is that the boot log and filesystem info info really doesn't look like OpenWrt at all (and I had not gone through the file on pastebin), but...

they also stated that the device runs OpenWrt in the OP.

As we know with certainty now, the vendor firmware on the device was clearly developed against a branch off an ancient version of OpenWrt and then highly customized (including the kernel version which I didn't check at the time of my first response).

However, we've seen strange things happen when people have installed some non-vendor/community branch (beyond the reach of the standard community builds that are pseudo-supported here), or the wrong image (or initramfs), or if they have done package upgrades, so I thought we'd start there.

I probably should have outright asked for the ubus call info or read the full log for more details and I'm sure we would have arrived at the answer more quickly :-). So, yes, point taken :slight_smile: -- there were many clues!

2 Likes

@psherman My apologies, I didn't mean to be harsh. (And I learned why people are asking for ubus call system board output).

3 Likes

no worries @Borromini

1 Like