Qualcomm Fast Path For LEDE


#630

Right now my next try is going to be re-implementing the qca-ssdk-hnat that is on the broken folder.
Am I correct into thinking that's a driver for hardware NAT? @gwlim what is the ssdk?
Googling for "ISIS NAT" is not really giving me the results I want....

On the TP-Link Stock firmware kernel log (V4) I see both entries for both fast-classifier and for ssdk hnat.
Maybe that will help me out.


#631

Thats right with full duplex throughput, but tcp isn´t fire and forget... Every package needs to be acknowledged...

How did you test your throughput? iperf with udp?

My thoughts are that the tp link fw use the hw nat feature of the switch and therefore the single cpu port is not important. Possibly your mentioned dd-wrt beta uses also hw nat feature. This are probably proprietary drivers for hw nat and dd-wrt uses a lot of proprietary code...

OpenWrt is fully opensource apart from some firmware.

The fast path mod only takes a shorter path through the kernel network stack and use the cpu port(s)...


#632

Top and htop show only one CPU in usage.
Only one CPU is working.
If two CPU are working at command top should show CPU 0 and CPU 1.
I think this version with fast path work only one CPU.
This is problem, I think.


#633

Wait, the QCA9563 has more than one core? Are you sure @jgvega?


#634

QCA9563 is single core.


#635

Excuse me. I think 2 cores


#636

if its taking a shorter path through the stack will things like ipsec or macsec etc cause any issues or am i totally off base? i also compile my kernel smash stack to aggressive security...


#637

I dont think that anything gets broken with sfe.
The question is what can utilized with sfe...
I think ipsec and macsec dont profit from sfe...

Openvpn routing traffic can be utilized on my ar71xx devices.


#638

Juppin, You should compile version lede with support USB and to habilite writeable u-boot for overclock.
2 in 1, overclock and fast path.
I have a overclock at my router wrd3600 with pepe2k's solution and after write your firmware.
It's fantastic. Better performance possible is muy router.
It's a idea.
Other theme is IPsec and macsec. I think a user un this forum did a solution modifing function at qualcomm extension for openwrt.
Regards.

Excuse my english.


#639

have you noticed performance differences in browsers? im currently using fast path along side cake, im not sure if the core qdisc has much to do with it, but i notice a stabler connection on internet explorer compared to firefox or chrome. Any ideas if this has anything to do with fast classifier or any ideas at all?


#640

With the patch Juppin provided it seems to work, thanks.


#641

I also tried fastpath along ‘cake’ with the ‘a piece of cake’ script.

I noticed two things:

  • On the wifi I noticed that resolving websites is a little bit quicker.
  • When I do a copy job of files from the pc to the nas it buffers more quickly at a higher throughput.

I’m running a WNDR4300.


#642

Hello everyone.

A very strange thing happens to me. I have tried to use the fast path version of juppin and also the gwlim version. In both tests it happens to me that the fast path module starts correctly.

root@LEDE:/sys/fast_classifier# lsmod | grep fast
fast_classifier        44768  0
nf_conntrack           50800 14 fast_classifier,nf_nat_ipv4,nf_conntrack_ipv6,nf_conntrack_ipv4,xt_state,xt_helper,xt_conntrack,xt_connmark,xt_connlimit,xt_connbytes,nf_nat_masquerade_ipv4,nf_nat,nf_conntrack_rtcache,sch_cake
shortcut_fe            51408  1 fast_classifier
shortcut_fe_ipv6       52560  1 fast_classifier

When I see the debug_info file it seems that it is not working.

root@LEDE:/sys/fast_classifier# cat debug_info
size=25 offload=0 offload_no_match=0 offloaded=0 done=0 offloaded_fail=72 done_fail=70

My ISP operator uses a coaxial cable with an analog signal and connects to a modem in my house. This modem connects to my router through an ethernet cable.

My router is connected to my ISP by the DHCP client mode in the eth0.2 interface.

My kernel log is:

[    0.000000] Linux version 4.4.87 (gwlim@lede-project.org) (gcc version 6.3.0 (LEDE GCC 6.3.0 r2993+516-b9a408c) ) #0 Fri Sep 8 19:53:16 2017
[    0.000000] MyLoader: sysp=29cc4571, boardp=3e8b20a0, parts=1f2321dc
[    0.000000] bootconsole [early0] enabled
[    0.000000] CPU0 revision is: 0001974c (MIPS 74Kc)
[    0.000000] SoC: Atheros AR9344 rev 2
[    0.000000] Determined physical RAM map:
[    0.000000]  memory: 08000000 @ 00000000 (usable)
[    0.000000] Initrd not found or empty - disabling initrd
[    0.000000] No valid device tree found, continuing without
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000] On node 0 totalpages: 32768
[    0.000000] free_area_init_node: node 0, pgdat 803c2370, node_mem_map 81000000
[    0.000000]   Normal zone: 256 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 32768 pages, LIFO batch:7
[    0.000000] Primary instruction cache 64kB, VIPT, 4-way, linesize 32 bytes.
[    0.000000] Primary data cache 32kB, 4-way, VIPT, cache aliases, linesize 32 bytes
[    0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
[    0.000000] pcpu-alloc: [0] 0 
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 32512
[    0.000000] Kernel command line:  board=TL-WDR4300  console=ttyS0,115200 rootfstype=squashfs,jffs2 noinitrd
[    0.000000] PID hash table entries: 512 (order: -1, 2048 bytes)
[    0.000000] Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
[    0.000000] Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
[    0.000000] Writing ErrCtl register=00000000
[    0.000000] Readback ErrCtl register=00000000
[    0.000000] Memory: 125420K/131072K available (3045K kernel code, 146K rwdata, 380K rodata, 324K init, 192K bss, 5652K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS:51
[    0.000000] Clocks: CPU:680.000MHz, DDR:450.000MHz, AHB:225.000MHz, Ref:40.000MHz
[    0.000000] clocksource: MIPS: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 5621354254 ns
[    0.000008] sched_clock: 32 bits at 340MHz, resolution 2ns, wraps every 6316128254ns
[    0.008677] Calibrating delay loop... 338.94 BogoMIPS (lpj=677888)
[    0.039607] pid_max: default: 32768 minimum: 301
[    0.044858] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.052232] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.062055] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.072953] futex hash table entries: 256 (order: -1, 3072 bytes)
[    0.080265] NET: Registered protocol family 16
[    0.086305] MIPS: machine is TP-LINK TL-WDR3600/4300/4310
[    0.095580] registering PCI controller with io_map_base unset
[    0.314560] Can't analyze schedule() prologue at 8006714c
[    0.327544] PCI host bridge to bus 0000:00
[    0.332164] pci_bus 0000:00: root bus resource [mem 0x10000000-0x13ffffff]
[    0.339834] pci_bus 0000:00: root bus resource [io  0x0000]
[    0.346047] pci_bus 0000:00: root bus resource [??? 0x00000000 flags 0x0]
[    0.353614] pci_bus 0000:00: No busn resource found for root bus, will use [bus 00-ff]
[    0.362483] pci 0000:00:00.0: [168c:0033] type 00 class 0x028000
[    0.362514] pci 0000:00:00.0: invalid calibration data
[    0.368276] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x0001ffff 64bit]
[    0.368333] pci 0000:00:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
[    0.368397] pci 0000:00:00.0: supports D1
[    0.368411] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
[    0.368609] pci_bus 0000:00: busn_res: [bus 00-ff] end is updated to 00
[    0.368640] pci 0000:00:00.0: BAR 0: assigned [mem 0x10000000-0x1001ffff 64bit]
[    0.376829] pci 0000:00:00.0: BAR 6: assigned [mem 0x10020000-0x1002ffff pref]
[    0.384890] pci 0000:00:00.0: using irq 40 for pin 1
[    0.390977] clocksource: Switched to clocksource MIPS
[    0.397575] NET: Registered protocol family 2
[    0.403172] TCP established hash table entries: 1024 (order: 0, 4096 bytes)
[    0.410985] TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
[    0.418087] TCP: Hash tables configured (established 1024 bind 1024)
[    0.425256] UDP hash table entries: 256 (order: 0, 4096 bytes)
[    0.431803] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[    0.439022] NET: Registered protocol family 1
[    0.443926] PCI: CLS 0 bytes, default 32
[    0.458493] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.465031] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    0.477996] io scheduler noop registered
[    0.482403] io scheduler deadline registered (default)
[    0.488279] Serial: 8250/16550 driver, 1 ports, IRQ sharing disabled
[    0.495744] console [ttyS0] disabled
[    0.519808] serial8250.0: ttyS0 at MMIO 0x18020000 (irq = 11, base_baud = 2500000) is a 16550A
[    0.529432] console [ttyS0] enabled
[    0.536856] bootconsole [early0] disabled
[    0.547677] m25p80 spi0.0: found s25fl064k, expected m25p80
[    0.553392] m25p80 spi0.0: s25fl064k (8192 Kbytes)
[    0.559284] 5 tp-link partitions found on MTD device spi0.0
[    0.564971] Creating 5 MTD partitions on "spi0.0":
[    0.569846] 0x000000000000-0x000000020000 : "u-boot"
[    0.576139] 0x000000020000-0x000000159164 : "kernel"
[    0.582574] 0x000000159164-0x0000007f0000 : "rootfs"
[    0.588999] mtd: device 2 (rootfs) set to be root filesystem
[    0.594815] 1 squashfs-split partitions found on MTD device rootfs
[    0.601104] 0x0000006b0000-0x0000007f0000 : "rootfs_data"
[    0.607963] 0x0000007f0000-0x000000800000 : "art"
[    0.614149] 0x000000020000-0x0000007f0000 : "firmware"
[    0.624370] switch0: Atheros AR8327 rev. 4 switch registered on ag71xx-mdio.0
[    1.199319] libphy: ag71xx_mdio: probed
[    1.772308] ag71xx ag71xx.0: connected to PHY at ag71xx-mdio.0:00 [uid=004dd034, driver=Atheros AR8216/AR8236/AR8316]
[    1.783670] eth0: Atheros AG71xx at 0xb9000000, irq 4, mode:RGMII
[    1.791679] NET: Registered protocol family 10
[    1.799351] NET: Registered protocol family 17
_[    1.803960] bridge: automatic filtering via arp/ip/ip6tables has been deprecated. Update your scripts to load br_netfilter if you need this. _  **This is normal?**
[    1.816770] 8021q: 802.1Q VLAN Support v1.8
[    1.825884] VFS: Mounted root (squashfs filesystem) readonly on device 31:2.
[    1.834414] Freeing unused kernel memory: 324K
[    3.330065] init: Console is alive
[    3.333757] init: - watchdog -
[    4.953656] kmodloader: loading kernel modules from /etc/modules-boot.d/*
[    5.070017] usbcore: registered new interface driver usbfs
[    5.075717] usbcore: registered new interface driver hub
[    5.081217] usbcore: registered new device driver usb
[    5.091693] exFAT: Version 1.2.9
[    5.129213] SCSI subsystem initialized
[    5.138454] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    5.146472] ehci-platform: EHCI generic platform driver
[    5.151906] ehci-platform ehci-platform: EHCI Host Controller
[    5.157788] ehci-platform ehci-platform: new USB bus registered, assigned bus number 1
[    5.167934] ehci-platform ehci-platform: irq 3, io mem 0x1b000000
[    5.182995] ehci-platform ehci-platform: USB 2.0 started, EHCI 1.00
[    5.190119] hub 1-0:1.0: USB hub found
[    5.194316] hub 1-0:1.0: 1 port detected
[    5.202236] usbcore: registered new interface driver usb-storage
[    5.215834] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
[    5.225664] init: - preinit -
[    5.727398] usb 1-1: new high-speed USB device number 2 using ehci-platform
[    5.971335] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    5.981747] random: procd: uninitialized urandom read (4 bytes read, 23 bits of entropy available)
[    5.991263] hub 1-1:1.0: USB hub found
[    5.997915] hub 1-1:1.0: 4 ports detected
[    6.278993] usb 1-1.1: new high-speed USB device number 3 using ehci-platform
[    6.382184] usb-storage 1-1.1:1.0: USB Mass Storage device detected
[    6.389237] scsi host0: usb-storage 1-1.1:1.0
[    7.948224] eth0: link up (1000Mbps/Full duplex)
[    7.953007] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    9.254933] mount_root: loading kmods from internal overlay
[    9.292243] kmodloader: loading kernel modules from //etc/modules-boot.d/*
[    9.300915] kmodloader: done loading kernel modules from //etc/modules-boot.d/*
[    9.670791] jffs2: notice: (390) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
[    9.686877] block: attempting to load /tmp/jffs_cfg/upper/etc/config/fstab
[    9.697173] block: extroot: device not present, retrying in 5 seconds
[   10.904938] scsi 0:0:0:0: Direct-Access     TOSHIBA  External USB 3.0 5438 PQ: 0 ANSI: 6
[   10.916499] sd 0:0:0:0: [sda] 732566645 4096-byte logical blocks: (3.00 TB/2.73 TiB)
[   10.925097] sd 0:0:0:0: [sda] Write Protect is off
[   10.930008] sd 0:0:0:0: [sda] Mode Sense: 23 00 00 00
[   10.930596] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   10.970348]  sda: sda1 sda2
[   10.977719] sd 0:0:0:0: [sda] Attached SCSI disk
[   14.830485] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: 
[   14.858601] mount_root: switched to extroot
[   14.865928] urandom-seed: Seeding with /etc/urandom.seed
[   15.065197] eth0: link down
[   15.078926] procd: - early -
[   15.082029] procd: - watchdog -
[   15.844974] EXT4-fs (sda1): mounting ext3 file system using the ext4 subsystem
[   15.852610] procd: - watchdog -
[   15.856306] procd: - ubus -
[   15.859286] random: nonblocking pool is initialized
[   16.086700] procd: - init -
[   16.941943] kmodloader: loading kernel modules from /etc/modules.d/*
[   16.988068] ntfs: driver 2.1.32 [Flags: R/O MODULE].
[   17.002622] tun: Universal TUN/TAP device driver, 1.6
[   17.007802] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[   17.023994] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: 
[   17.044213] l2tp_core: L2TP core driver, V2.0
[   17.056181] l2tp_netlink: L2TP netlink interface
[   17.068856] sit: IPv6 over IPv4 tunneling driver
[   17.082731] nat46: module (version 683fbd2b765506332a1af141545652bf58f03166) loaded.
[   17.107243] gre: GRE over IPv4 demultiplexor driver
[   17.124559] ip_gre: GRE over IPv4 tunneling driver
[   17.148544] ip6_tables: (C) 2000-2006 Netfilter Core Team
[   17.196129] u32 classifier
[   17.198880]     input device check on
[   17.202638]     Actions configured
[   17.220085] Mirror/redirect action on
[   17.235706] nf_conntrack version 0.5.0 (1964 buckets, 7856 max)
[   17.263299] usbcore: registered new interface driver cdc_acm
[   17.269100] cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
[   17.295571] Loading modules backported from Linux version wt-2017-01-31-0-ge882dff19e7f
[   17.303738] Backport generated by backports.git backports-20160324-13-g24da7d3c
[   17.327824] ip_tables: (C) 2000-2006 Netfilter Core Team
[   17.382528] usbcore: registered new interface driver usblp
[   17.451967] xt_time: kernel timezone is -0000
[   17.574100] PPP generic driver version 2.4.2
[   17.584580] PPP MPPE Compression module registered
[   17.595582] NET: Registered protocol family 24
[   17.612202] PPTP driver version 0.8.5
[   17.690885] l2tp_ppp: PPPoL2TP kernel driver, V2.0
[   17.727622] ath: EEPROM regdomain: 0x0
[   17.727641] ath: EEPROM indicates default country code should be used
[   17.727649] ath: doing EEPROM country->regdmn map search
[   17.727667] ath: country maps to regdmn code: 0x3a
[   17.727676] ath: Country alpha2 being used: US
[   17.727684] ath: Regpair used: 0x3a
[   17.739607] ieee80211 phy0: Selected rate control algorithm 'minstrel_ht'
[   17.744044] ieee80211 phy0: Atheros AR9340 Rev:2 mem=0xb8100000, irq=47
[   17.750953] PCI: Enabling device 0000:00:00.0 (0000 -> 0002)
[   17.762193] ath: EEPROM regdomain: 0x0
[   17.762209] ath: EEPROM indicates default country code should be used
[   17.762217] ath: doing EEPROM country->regdmn map search
[   17.762235] ath: country maps to regdmn code: 0x3a
[   17.762244] ath: Country alpha2 being used: US
[   17.762252] ath: Regpair used: 0x3a
[   17.771568] ieee80211 phy1: Selected rate control algorithm 'minstrel_ht'
[   17.776018] ieee80211 phy1: Atheros AR9300 Rev:4 mem=0xb0000000, irq=40
[   17.875387] kmodloader: done loading kernel modules from /etc/modules.d/*
[   26.616980] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   26.635483] device eth0.1 entered promiscuous mode
[   26.640402] device eth0 entered promiscuous mode
[   26.666889] IPv6: ADDRCONF(NETDEV_UP): br-lan: link is not ready
[   26.718561] IPv6: ADDRCONF(NETDEV_UP): eth0.2: link is not ready
[   27.212396] eth0: link up (1000Mbps/Full duplex)
[   27.229614] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   27.280791] br-lan: port 1(eth0.1) entered forwarding state
[   27.286564] br-lan: port 1(eth0.1) entered forwarding state
[   27.292444] IPv6: ADDRCONF(NETDEV_CHANGE): eth0.2: link becomes ready
[   27.434726] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready
[   29.287047] br-lan: port 1(eth0.1) entered forwarding state
[   32.062149] IPv6: ADDRCONF(NETDEV_UP): wlan1: link is not ready
[   32.250089] device wlan1 entered promiscuous mode
[   32.341992] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
[   32.409341] device wlan0 entered promiscuous mode
[   32.414261] br-lan: port 3(wlan0) entered forwarding state
[   32.419886] br-lan: port 3(wlan0) entered forwarding state
[   33.067075] br-lan: port 3(wlan0) entered disabled state
[   34.014863] fast-classifier: starting up
[   34.019109] fast-classifier: registered
[   34.659050] IPv6: ADDRCONF(NETDEV_CHANGE): wlan1: link becomes ready
[   34.665718] br-lan: port 2(wlan1) entered forwarding state
[   34.671372] br-lan: port 2(wlan1) entered forwarding state
[   35.458948] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[   35.465625] br-lan: port 3(wlan0) entered forwarding state
[   35.471300] br-lan: port 3(wlan0) entered forwarding state
[   36.675012] br-lan: port 2(wlan1) entered forwarding state
[   37.475021] br-lan: port 3(wlan0) entered forwarding state

I do not know what's going on.

Can someone help me solve the problem?

Regards.


#643

i believe i have the same issue, its loaded and registered but no offload, maybe i need a firewall rule?


#645

i was looking at the latest patch and it seems like there were a couple other things i was missing for classifier, so i basically just followed this for my own image. ill let you know what i get, im compiling it right now, i noticed i had to enable a couple more modules but im still missing kmod-ipv6 and libfastclassifier.


#646

Tell me if you solve the problem.


#647

135/5000
Hi.
I have used the images compiled by juppin. fast path They are working perfectly.

I leave the link to the images.
https://github.com/juppin/openwrt_custom_builds_trunk/tree/master/ar71xx_8mb_luci-ssl_nano_ddns_sqm_sfe_openvpn_wpad_openssl_dnsmasq-full_extroot-f2fs-ext4

Juppin, thank you very much for your work. You are a crack.


#648

@juppin
I am getting error in image build.
I did following:
git clone https://github.com/lede-project/source.git
patch -p1 < 666-shortcut-fe_trunk.patch (succeeded without error)
copied contents of hack-4.x to pending-4.x
make menuconfig, select fast-path classifier.
make world

Error Log:

Applying /home/coug/lede_trunk/target/linux/generic/hack-4.9/202-reduce_module_size.patch using plaintext: 
patching file Makefile
Hunk #1 FAILED at 403.
1 out of 1 hunk FAILED -- saving rejects to file Makefile.rej
Patch failed!  Please fix /home/coug/lede_trunk/target/linux/generic/hack-4.9/202-reduce_module_size.patch!
make[3]: *** [/home/coug/lede_trunk/build_dir/toolchain-mips_24kc_gcc-7.3.0_musl/linux-4.9.100/.prepared] Error 1
make[3]: Leaving directory `/home/coug/lede_trunk/toolchain/kernel-headers'
Command exited with non-zero status 2
time: toolchain/kernel-headers/compile#9.00#4.99#42.05
make[2]: *** [toolchain/kernel-headers/compile] Error 2
make[2]: Leaving directory `/home/coug/lede_trunk'
make[1]: *** [/home/coug/lede_trunk/staging_dir/toolchain-mips_24kc_gcc-7.3.0_musl/stamp/.toolchain_compile] Error 2
make[1]: Leaving directory `/home/coug/lede_trunk'
make: *** [world] Error 2

#649

Please use openwrt source and not lede...

No need to copy patches in another directory...

Please do a "make -j1 V=s" and post the detailed error log here. Last 100 lines should be sufficient...


#650

Build succeeds!.. Thank you.