Porting guide ar71xx to ath79?


#647

And what was your conclusion about those stats?
I mentioned the shared interrupt 12 already two weeks ago and wondered about that.

For me the interrupt usage looks more normal. (note that wan on this dumb AP router is unused.) eth0 and ath9k counts are pretty close, so packets have flowed to both direction in balance. (router was rebooted and then several stressing speed tests with flent were run)

 -----------------------------------------------------
 OpenWrt SNAPSHOT, r7866-8c91807214
 -----------------------------------------------------
root@router2:~# cat /proc/interrupts
           CPU0
  3:          0      MIPS   3  ehci_hcd:usb1
  4:    1095750      MIPS   4  19000000.eth
  5:          0      MIPS   5  1a000000.eth
  7:     461618      MIPS   7  timer
  8:         12      MISC   3  ttyS0
 11:          0      MISC   6  ohci_hcd:usb2
 12:     834189  AR71XX PCI   1  ath9k, ath9k
ERR:        566
root@router2:~#

#648

We need to figure out how to configure interrupt-map in ar7100.dtsi


#649

i spent a considerable amount of time looking into the shared irq issue in pci-ar71xx.c before i submitted the current "fix" (https://github.com/openwrt/openwrt/pull/1225).

here is what i found:

  • ar71xx_pci_irq_init() doesn't seem to take into account ATH79_PCI_IRQ_BASE which is 40
  • using irq_domain_add_simple() rather than irq_domain_add_linear() would allow specifying ATH79_PCI_IRQ_BASE during the irq domain setup
  • in testing, however, the above change caused both ath9k cards on my device (rspro, ar7161) to be assigned the same irq - 41
  • another possibility seemed to be adding a full interrupt-map property to the pcie-controller node to assign the correct irqs to the 3 pci slots (17, 18, 19)
  • the openfirmware interrupt-controller parsing code in drivers/of/irq.c (of_irq_parse_raw) sees that pcie-controller@180c0000 node is itself an interrupt-controller, so it never attempts to parse the interrupt-map property
  • many permutations of adding a child interrupt-controller node to the pcie-controller node also did not work. the intent here was to separate the pcie-controller and interrupt-controller, so that of_irq_parse_raw would parse the interrupt-map property

i raised this issue on irc before submitting the original PR and was told that the reason was (or might be):
"irq cascade on a cascade so mips IRQ maps to the intc and a intc irq maps to the cascade inside the pci core".
"irq mapping between static ar71xx and ath79 OF is very different"
"the virtual -> physical numbers no longer correlate"

so, basically, i don't know :wink:


#650

I can easily map 2 devices to different IRQ 12 and 13, or 40 and 41.

But it causes crash (noone responds on the second IRQ whatever it is) and Wi-Fi doesn't work at all.
To get it to 40 and 41 it is using irq_domain_add_simple()

and add interrupts = <0>; to the first ath9k node and interrupts = <1>; to the second.

In this case I have exactly same output of /proc/interrupts as on ar71xx. but Wi-Fi doesn't work at all.

I see interfaces in ifconfig, but nothing is transmitted.

So we have a bug somewhere.


#652

there are a couple options i found helpful when looking into this (besides tons of added printk(), lol):

CONFIG_IRQ_DOMAIN_DEBUG=y which creates /sys/kernel/debug/irq_domain_mapping
CONFIG_GENERIC_IRQ_DEBUGFS=y which adds more irq stuff (i forget exactly what...)

i added them by running "make kernel_menuconfig" and rebuilding and reflashing.

you can also add '#define DEBUG' to the top of drivers/of/irq.c for some additional output.


#653

I think we need to remove

interrupt-controller property from pcie and then configure it properly.
If we have a correct map, interrupts will be translated.


#654

And ranges property looks to be wrong

https://elinux.org/Device_Tree_Usage#How_Interrupts_Work


#655

So after some research I am quite sure that pcie0 shouldn't have an interrupt-controller property.

This is done by default if interrupt-map property is there. In this case it will be read properly.

The question is how to set it up correctly.


#656

Nope. pcie0 should have a interrupt-controller. There is only one IRQ for PCI and this IRQ needs a dispatcher. When that interrupt is triggered, driver should read corresponding interrupt status register to determine which PCI device triggered the current interrupt. Part of the PCI driver is actually a cascade driver like ar9340-intc in ath79 and the pcie0 node itself acts as a interrupt controller node.
BTW the PCI on ar7100 isn't a PCIE. It's only a PCI controller. It seems that the pcie0 node is directly copied from other PCIE devices.


#657

According to another part of the link posted by @Pilot6 I think the interrupt-map in ar7100.dtsi is wrong.
PS: I'm using my phone now and I currently can't figure out the correct interrupt map. And due to lack of a ar7161 device I think I'm not able to fix this problem.


#658

It has a controller, but the property shouldn't be there. If we set the "interrupt-controller" property, the "interrupt-map" is ignored.
I built without the "interrupt-controller" and now changing values in "interrupt-map" really affect things.
But the problem is that it looks like the driver code can't handle two interrupts.

The second is not translated anywhere.


#659

It is something like this

interrupt-map-mask = <0xf800 0 0 7>;
				interrupt-map = <0x8800 0 0 1 &pcie0 1 3
						 0x8800 0 0 2 &pcie0 2 3
						 0x8800 0 0 3 &pcie0 3 3
						 0x8800 0 0 4 &pcie0 4 3	

						 0x9000 0 0 1 &pcie0 2 3
						 0x9000 0 0 2 &pcie0 3 3
						 0x9000 0 0 3 &pcie0 4 3
						 0x9000 0 0 4 &pcie0 1 3>;

Or even simpler

interrupt-map-mask = <0xf800 0 0 7>;
				interrupt-map = <0x8800 0 0 1 &pcie0 1 3
						 0x9000 0 0 1 &pcie0 2 3>;

I can't figure out what is the last field.

It works with those maps, but still only one IRQ, If I add "interrupts" property to an ath9k node, a second is mapped to MIPS instead of PCI.
@981213

#660

If I keep "interrupt-controller" property and add "interrupts = <0> and <1> to ath9k nodes,
/proc/interrupts looks exactly like in ar71xx, but with IRQ 12 and 13.

If I change the driver code to add an IRQ domain not linear, but simple and use 40 as base, then I have it 100% same. But I get errors that noone does ann interrupt on the second IRQ.


#661

I haven't check how those interrupt related nodes are parsed, but if interrupt-map isn't parsed after interrupt-controller defined we need to write another IRQ cascade driver for PCIE controller.:frowning:


#662

Only IRQ numbers from cpuintc are the same between ar71xx and ath79. Other numbers are just software defined interrupt map between IRQ cascade driver and devices and those are replaced by interrupts defined in dts

PS: I remembered to see the same low-throughput problem occurs on other QCA chips somewhere but I never tested it myself.
It's midnight now (UTC+8 here) and I may run a test tomorrow.


#663

Look into drivers/of/irq.c

/* Now check if cursor is an interrupt-controller and if it is
		 * then we are done
		 */
		if (of_get_property(ipar, "interrupt-controller", NULL) !=
				NULL) {
			pr_debug(" -> got it !\n");
			return 0;
		}

#664

I've changed the driver to use a separated subnode as interrupt controller.
I currently don't have any router with me and the code only passed compile test.
You guys could test this if you are free: https://github.com/981213/openwrt/tree/ath79_pci
PS: I only updated ar7100.dtsi and current dts for other chips are currently broken.


#665

Thanks, that looks good. I will test it. I will take care of the dts file, it needs some more changes.
interrupt-mask, etc. But this way it may move forward.


#666

@981213 This way the interrupt-controller is completely ignored.

root@Router:~# cat /proc/interrupts
           CPU0       
  0:          0      MIPS   0  ath9k, ath9k
  3:          0      MIPS   3  ehci_hcd:usb1
  4:       1046      MIPS   4  19000000.eth
  5:        655      MIPS   5  1a000000.eth
  7:       8538      MIPS   7  timer
  8:         14      MISC   3  ttyS0
 11:          0      MISC   6  ohci_hcd:usb2
ERR:          0

I tried to add interrupt-parent = <&pci0_intc>; to the ath9k nodes, but still the interrupts go to cpuintc and don't work.

irq: no irq domain found for /ahb/apb/pcie-controller@180c0000 !

I can provide any debug what you say.


#667

BTW, the correct map settings are

interrupt-map-mask = <0xf800 0 0 7>;
				interrupt-map = <0x8800 0 0 1 &pci0_intc 1
						 0x9000 0 0 1 &pci0_intc 2>;

The mask is for 0x8800 and 0x9000 and this sort of ids, the 7 is for {1,2,3,4} if we use all INT#. It can be 1 if we use only INT#A.

The 1's in the map are for INT#A, that are used in this case I guess.

So we map INT#A from device @0x8800 to &pci_intc IRQ1...

But first we need to get PCI to cascade IRQs, that doesn't happen.