Meraki MX60 (APM821XX PowerPC 44x) Bugs

For Developers
1

I bought a few of these for cheap on ebay. These problems have occurred both with 22.03.2 and yesterday's snapshot.

  1. "Illegal instruction" in mpg123.
    Running mpg123 on any file (even with -t for no output) crashes with a report of an illegal instruction:
[33679.076678] mpg123[2667]: illegal instruction (4) at b7d5ffa8 nip b7d5ffa8 lr b7d5ff74 code 1 in libmpg123.so.0.47.0[b7d3d000+5f000]
[33679.088684] mpg123[2667]: code: be810050 7fc802a6 83256d50 2c090000 83456d58 3fde0004 90010084 3bde4d98
[33679.098160] mpg123[2667]: code: 7c751b78 7c982378 7cbc2b78 7f79d214 <7fe131ce> 41e2003c 3be5244c 38800000
Tue Nov 22 02:31:07 2022 kern.info kernel: [33679.076678] mpg123[2667]: illegal instruction (4) at b7d5ffa8 nip b7d5ffa8 lr b7d5ff74 code 1 in libmpg123.so.0.47.0[b7d3d000+5f000]
Tue Nov 22 02:31:07 2022 kern.info kernel: [33679.088684] mpg123[2667]: code: be810050 7fc802a6 83256d50 2c090000 83456d58 3fde0004 90010084 3bde4d98
Tue Nov 22 02:31:07 2022 kern.info kernel: [33679.098160] mpg123[2667]: code: 7c751b78 7c982378 7cbc2b78 7f79d214 <7fe131ce> 41e2003c 3be5244c 38800000
  1. Kernel crash while establishing IPSec tunnel.
    Using strongswan-swanctl / vici framework with a configuration which works on an x86 installation. Only after reaching the server and negotiating IKE successfully does this occur:
...
Tue Nov 22 02:46:37 2022 daemon.info ipsec: 16[IKE] authentication of 'REDACTED' with ECDSA-384 signature successful
Tue Nov 22 02:46:37 2022 daemon.info ipsec: 16[IKE] IKE_SA dublin[1] established between REDACTED ... REDACTED
Tue Nov 22 02:46:37 2022 daemon.info ipsec: 16[IKE] scheduling rekeying in 3272s
Tue Nov 22 02:46:37 2022 daemon.info ipsec: 16[IKE] maximum IKE_SA lifetime 3632s
Tue Nov 22 02:46:37 2022 daemon.info ipsec: 16[IKE] received ESP_TFC_PADDING_NOT_SUPPORTED, not using ESPv3 TFC padding
Tue Nov 22 02:46:37 2022 daemon.info ipsec: 16[CFG] selected proposal: ESP:AES_CBC_256/HMAC_SHA2_256_128/NO_EXT_SEQ
[34609.091339] BUG: Kernel NULL pointer dereference on read at 0x00000010
[34609.097872] Faulting instruction address: 0xe53b71d0
[34609.102852] Oops: Kernel access of bad area, sig: 11 [#1]
[34609.108237] BE PAGE_SIZE=4K PowerPC 44x Platform
[34609.112842] Modules linked in: ath9k ath9k_common pppoe ppp_async nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet ath9k_hw ath snd_usb_audio pppox ppp_generic nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_counter nft_chain_nat nf_tables nf_nat nf_flow_table nf_conntrack mac80211 ipt_REJECT cfg80211 xt_time xt_tcpudp xt_policy xt_multiport xt_mark xt_mac xt_limit xt_esp xt_comment xt_TCPMSS xt_LOG xfrm_interface spi_gpio spi_bitbang snd_usbmidi_lib slhc nfnetlink nf_reject_ipv4 nf_log_syslog nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c iptable_mangle iptable_filter ipt_ah ip_tables crc_ccitt compat ledtrig_usbport ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ipcomp6 xfrm6_tunnel esp6 ah6 xfrm4_tunnel ipcomp esp4 ah4 tunnel6 tunnel4 snd_rawmidi snd_seq_device snd_pcm_oss
[34609.113192]  snd_mixer_oss snd_hwdep snd_compress snd_pcm snd_timer snd soundcore input_core xfrm_user xfrm_ipcomp af_key xfrm_algo sha512_generic sha256_generic libsha256 sha1_generic seqiv drbg md5 kpp hmac echainiv des_generic libdes cmac cbc authencesn authenc usb_storage leds_gpio dwc2 roles gpio_button_hotplug usbcore nls_base usb_common crc32c_generic
[34609.230969] CPU: 0 PID: 2746 Comm: charon Not tainted 5.15.79 #0
[34609.236956] NIP:  e53b71d0 LR: e53b71c4 CTR: c0349db0
[34609.241992] REGS: c2cc19d0 TRAP: 0300   Not tainted  (5.15.79)
[34609.247806] MSR:  00029000 <CE,EE,ME>  CR: 84882202  XER: 20000000
[34609.253984] DEAR: 00000010 ESR: 00000000
[34609.253984] GPR00: e53b71c4 c2cc1ab0 c1193700 e529bdf4 e529bce0 00000008 00000100 c2ca6a48
[34609.253984] GPR08: 00000000 00000080 00000000 00000004 24882208 00000000 b7094d40 b7094d40
[34609.253984] GPR16: b7094d40 b7094d40 b7094d40 e53b9424 88008808 00000002 c08158b0 00000032
[34609.253984] GPR24: c08158c0 00000100 c2ca6a28 c210d080 00000010 c2ca6a20 00000048 c2cd4240
[34609.291292] NIP [e53b71d0] 0xe53b71d0
[34609.294973] LR [e53b71c4] 0xe53b71c4
[34609.298559] Call Trace:
[34609.300995] [c2cc1ab0] [e53b71c4] 0xe53b71c4 (unreliable)
[34609.306403] [c2cc1b70] [e53b75a8] 0xe53b75a8
[34609.310689] [c2cc1b90] [c063be40] __xfrm_init_state+0x354/0x5c0
[34609.316608] [c2cc1be0] [e535a148] 0xe535a148
[34609.320901] [c2cc1c30] [e53584e8] 0xe53584e8
[34609.325187] [c2cc1cf0] [c0586db8] netlink_rcv_skb+0x60/0x154
[34609.330846] [c2cc1d40] [e5357724] 0xe5357724
[34609.335132] [c2cc1d60] [c058650c] netlink_unicast+0x2a0/0x330
[34609.340860] [c2cc1db0] [c0586784] netlink_sendmsg+0x1e8/0x444
[34609.346588] [c2cc1e10] [c04dad38] __sys_sendto+0xe4/0x158
[34609.351988] [c2cc1f10] [c000c0a8] ret_from_syscall+0x0/0x28
[34609.357543] --- interrupt: c00 at 0xb791b3c8
[34609.361795] NIP:  b791b3c8 LR: b791b370 CTR: b7065834
[34609.366822] REGS: c2cc1f20 TRAP: 0c00   Not tainted  (5.15.79)
[34609.372628] MSR:  0002d000 <CE,EE,PR,ME>  CR: 20002202  XER: 00000000
[34609.379066]
[34609.379066] GPR00: 0000014f b6f47240 b6f4edf4 00000008 b6f474cc 000001e8 00000000 b6f472d4
[34609.379066] GPR08: 0000000c b7065834 00000020 00000000 f0b16905 00000000 b7094d40 b7094d40
[34609.379066] GPR16: b7094d40 b7094d40 b7094d40 b7094d40 b707052a b6f472d4 00000080 b6f4748c
[34609.379066] GPR24: 00000000 00000008 b6f474cc 000001e8 00000000 b6f47d84 b795bb74 0000014f
[34609.413867] NIP [b791b3c8] 0xb791b3c8
[34609.417514] LR [b791b370] 0xb791b370
[34609.421073] --- interrupt: c00
[34609.424115] Instruction dump:
[34609.427086] 7f43d378 80bb0040 38a50007 54a5e8fe 48002009 38800000 7f63db78 833b0040
[34609.434917] 4bee42c9 2c030000 41820190 a0c3000e <80bc0000> 54c6e8fe 809f0110 7c062800
[34609.442911] ---[ end trace 79c678ad609cb5bb ]---
[34609.447513]
[34610.449084] Kernel panic - not syncing: Fatal exception
[34610.454302] Rebooting in 3 seconds..

Any advice on what this could be or how I should proceed to investigate and debug further will be appreciated.

2

Disassembling that code online shows an Altivec PPC insn, which I suppose isn't supported on the APM821XX/MX60 and hence "illegal instruction".

You could test by rebuilding the mpg123 package after forcing Altivec disabled in the Makefile, if possible. This below may provide more context, and @chunkeey might have more suggestions.

1 Like
3

OK that did fix mpg123. The Makefile for mpg123 (feeds/packages/sound/mpg123/Makefile) explicitly enables an altivec build, which does not work on this chip. I think that all the supported targets in powerpc464 are the same chip APM82181. I haven't been able to find a document of the exact capabilities of this particular chip.

4

Yeah, I suspect this a "tip of the iceberg" problem for all those targets and some review is in order.

You might also try the ffmpeg package to check for the same error, and the ffmpeg Makefile has some Altivec logic worth looking at.

Out of curiosity, what do you see in /proc/cpuinfo? I'm guessing it has hard float but no altivec.

5

cpuinfo reveals very little:

root@mx60-4:~# cat /proc/cpuinfo
processor	: 0
cpu		: APM821XX
clock		: 800.000008MHz
revision	: 28.131 (pvr 12c4 1c83)
bogomips	: 1600.00

timebase	: 800000008
platform	: PowerPC 44x Platform
model		: Meraki MX60/MX60W Security Appliance
Memory		: 512 MB

I found a two page sales brochure about the chip it definitely says it has a FPU and crypto unit but doesn't say the PowerPC ABI level or any other instruction details.

6

No worries, It's on wikipedia.

"The 460 core adheres to Power ISA v.2.03 using the Book III-E specification."

The kernel also has an overview:
https://www.kernel.org/doc/html/v6.0/powerpc/isa-versions.html

Note: the PPC464 supports some extra DSP instructions (FMAC) but there's no support for AltiVec(R)/Velocity Engine/VMX instructions.