[RETRACTED] compressed-memory: this project is officially retracted, take note about it's future

For Developers
1

[RETRACTED] OpenWrt rejected this project, and I will not make the needless effort to convince them it is worth it.

However, at least one brand, at least 6 individual developers, and 2 projects (at the moment of writing this retraction letter), besides my own projects, actively use this code in production. I use this code myself for my custom builds, so I will keep this code updated.

However, I expect exponentially more usage of this project in the future, even when OpenWrt officially rejected it. For any brand, individual, or project interested in this project, I strongly suggest you not relying on the pull request submitted to OpenWrt, it is officially retracted now.

The best way to get the latest patches for compressed-memory and luci-app-compressed-memory is by cherry-picking the commits from my official builds, which you can grab from here:

MT7621A: https://github.com/NoTengoBattery/openwrt-mt7621/commits/mt7621-upstreamtrack
IPQ40xx: https://github.com/NoTengoBattery/openwrt/commits/linksys-ea6350v3-mastertrack

As an alternative, I will update my IPQ40xx master branch exclusively with the patches against the master branch from OpenWrt itself. This means that you can download the patches directly, and apply them using git or patch by downloading them from the following link:

Patches against OpenWrt master branch: https://github.com/NoTengoBattery/openwrt/compare/openwrt:master...master.patch

For any concern about this code, please contact me directly via email, or by the preferred way which is opening an issue in any of my OpenWrt repositories.

8 Likes
2

[RETRACTED] This post is officially retracted. Please take note that compressed-memory and luci-app-compressed-memory will be actively maintained, but I will not longer provide them to OpenWrt.

3

[RETRACTED] This post is officially retracted. Please take note that compressed-memory and luci-app-compressed-memory will be actively maintained, but I will not longer provide them to OpenWrt.

3 Likes
5

I'm gonna try your packages on my next build. Nice work, man.

1 Like
6

[RETRACTED] This post is officially retracted. Please take note that compressed-memory and luci-app-compressed-memory will be actively maintained, but I will not longer provide them to OpenWrt.

1 Like
8

small possibly offtopic... ~3 weeks ago~... tried

edited likely deprecated feedback 
############### your community git kopts at the time
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_CMDLINE="rcutree.kthread_prio=5"
CONFIG_CMDLINE_EXTEND=y
CONFIG_CRYPTO_LZ4HC=y
CONFIG_CRYPTO_ZSTD=y
CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=n
CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_SCHEDUTIL=y
CONFIG_THERMAL_DEFAULT_GOV_POWER_ALLOCATOR=y
CONFIG_THERMAL_DEFAULT_GOV_STEP_WISE=n
CONFIG_DEFAULT_CFQ=y
CONFIG_DEFAULT_DEADLINE=n
CONFIG_IOSCHED_BFQ=y
CONFIG_IOSCHED_CFQ=y
CONFIG_MQ_IOSCHED_DEADLINE=y
CONFIG_MQ_IOSCHED_KYBER=y
CONFIG_ARM_CPU_TOPOLOGY=y
CONFIG_SCHED_AUTOGROUP=y
CONFIG_SCHED_MC=y
CONFIG_DM_MQ_DEFAULT=y
CONFIG_SCSI_MQ_DEFAULT=y
CONFIG_DEFAULT_BBR=y
CONFIG_DEFAULT_CUBIC=n
CONFIG_PM=y
CONFIG_WQ_POWER_EFFICIENT_DEFAULT=y
CONFIG_FRONTSWAP=y
CONFIG_KSM=y
CONFIG_UKSM=y
CONFIG_Z3FOLD=y
CONFIG_ZSWAP=y
CONFIG_HZ_100=n
CONFIG_HZ_500=y
CONFIG_JUMP_LABEL=y
CONFIG_PREEMPT=y
CONFIG_PREEMPT_NONE=n
CONFIG_RCU_BOOST=y
CONFIG_RCU_BOOST_DELAY=251
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_IO_ACCOUNTING=y
CONFIG_TASK_XACCT=y

CONFIG_IOSCHED_BFQ=y

gave me trouble... did not spend too long on it... likely just need to track down the "parent option"...

( unless you know it by any chance :female_detective: )

( happy to make this part of my builds and test ~ add 806x specific feedback )

ok... so had time to redo this and turns out it wants...

BFQ_GROUP_IOSCHED
( also wants PM_DEBUG )

NOTES1:

  1. Needs block-mount ( base-system )
  2. /etc/init.d/zram enable | start = "zram: warning: zram was not allowed to run"
  3. zram status ( prints absolutely nothing )

NOTES2:

  1. adding block-mount seems to have helped zram...
  2. zswap status command not referenced in help / additional commands
  3. the start|stop of zram seems a tad glitchy... refuses to start intermittently without adequate feedback ( appears related to luci default selections )
################### zswap status
root@OpenWrt:/# zswap status
Gathering stats info for zswap subsystem

ZSWAP
-----
Currently enabled                   - Y
Compression algorithm               - lz4
Memory allocator (zpool)            - z3fold
Max. memory pool                    - 12.00 % (56.00 MiB)

STATS
-----
Duplicate entry                     - 0
Pool limit hit (times)              - 0
Rejected due to alloc. fail         - 0
Rejected due to bad compression     - 0
Rejected due to kmemcache fail      - 0
Rejected due to reclaim failed      - 0
Same filled pages (identical)       - 0
Pages currently in the pool         - 0
Written-back pages to storage       - 0

DATA
----
System memory                       - 477.90 MiB (489.37 MB)
Pool total size                     - 0.00 MiB (0.00 %)
Original data size                  - 0.00 MiB (0.00 %)
Estimated savings                   - 0.00 MiB (0.00 %)
Compression ratio                   - 1.00 

################### zram status
root@OpenWrt:/# zram status
Gathering stats info for zram device <zram0>

ZRAM
----
Block device                        - /sys/block/zram0
Backing loop device                 - none
Backing file of loop dev            - 
Compression algorithm               - lz4hc
Compression driver                  - lzo lz4 [lz4hc] zstd 
Max. compression streams            - 2

DATA
----
Original data size                  - 0.00 MiB (0.00 %)
Compressed data size                - 0.00 MiB (0.00 %)
Estimated savings                   - -0.01 MiB (-0.00 %)
Compression ratio (absolute)        - 0.33 
Compression ratio                   - 97.52 
Expected compression ratio          - 1.58 

MEMORY
------
System memory                       - 477.90 MiB (489.37 MB)
Memory used (current)               - 0.01 MiB (0.00 %)
Allocator overhead                  - 0.01 MiB (0.00 %)
Max. memory ever used               - 0.01 MiB (0.00 %)
Allocator efficiency                - 0.34 %

PAGES
-----
Same pages count                    - 0 
Compacted pages count               - 0 
Uncompressible pages count          - 0 
Pages discarded count               - 66091 

############## zswap restart
root@OpenWrt:/# zswap restart
zswap: info: successfully disabled
zswap: zpool for zswap: 'z3fold'
zswap: driver for zswap: 'lz4'
zswap: maximum pool for zswap: 30% (146812109 bytes)
zswap: zswap was successfully enabled
zswap: system swapiness is vm.swappiness = 75
zram: deactivating swap device '/dev/zram0'
zram: swap disabled for '/dev/zram0'
zram: resetting device 'zram0' to defaults
[ 2146.531301] zram0: detected capacity change from 270712832 to 0
zswap: trying to enable zram on success...
zram: detected RAM size: 489373.696 kb
zram: compression algorithm: 'lz4hc', driver: 'lz4hc'
zram: expected compression ratio for 'lz4hc': 2.180
zram: zswap factor for compression ratio: 0.72
zram: scaled compression ratio': 1.580500000
zram: memory pool base: 35.00%
zram: calculated memory pool: 270709294 bytes
zram: memory pool limit for zram: 72.45%
zram: worst expected compression ratio: 1.035
zram: zswap memory pool: 50.00% uncompressed of zram (12%)
zram: maximum memory pool allowed for zram: 308235693 bytes
zram: maximum pool size for zram: 270709294 bytes (55.32%)
zram: info: total memory for compression: 55.32% +  12.00%
zram: resetting device 'zram0' to defaults
zram: compression algorithm for 'zram0': lzo lz4 [lz4hc] zstd 
zram: available cores/CPUs for compression: 2
zram: CPU streams factor for 'lz4hc': 45.835
zram: calculated compression streams: 1
zram: compression streams for 'zram0': 2
zram: backing file for 'zram0' is not set or does not exist
[ 2146.821849] zram0: detected capacity change from 0 to 270712832
Setting up swapspace version 1, size = 270708736 bytes
[ 2146.851640] Adding 264364k swap on /dev/zram0.  Priority:1000 extents:1 across:264364k SSDscFS
zram: activated '/dev/zram0' for swapping (258.17 MiB)
zswap: pool limit for zram: 72.45%
zswap: worst case compression ratio for zram: 1.035
zswap: absolute maximum memory pool: 74.99%
zswap: expected compression ratio for z3fold: 3.000
zswap: zswap scale for zram pool: 50.00%
zswap: adjusted zswap pool: 12%
zswap: maximum pool for zswap: 12% (58724844 bytes)

ipq806x-preempt>7500v2|ea8500|nbg6817|r7800|rt2600ac usb / initramfs only

Summary...

  1. Fixes needed re: not-starting and why verbosity... / logix < luci parameter validation
  2. Needs testing specifically on systems with high application / fs workloads ( NAS )... i have a feeling it offers alot in such setups...
1 Like
9

[RETRACTED] This post is officially retracted. Please take note that compressed-memory and luci-app-compressed-memory will be actively maintained, but I will not longer provide them to OpenWrt.

1 Like
10

thanks for the explanations... if i ever have a need to build again from scratch... I will omit the kernel options as advised.

11

[RETRACTED] This post is officially retracted. Please take note that compressed-memory and luci-app-compressed-memory will be actively maintained, but I will not longer provide them to OpenWrt.

1 Like
12

Any news on getting this merged? Excited to give it a try.

Also do you think this’ll help with 4/32 MB devices?

13

[RETRACTED] This post is officially retracted. Please take note that compressed-memory and luci-app-compressed-memory will be actively maintained, but I will not longer provide them to OpenWrt.

3 Likes
14

I don’t think you should get discouraged. Not having a readily installable package for people to quickly install and test is also a barrier to entry. If you and others in this tested it then I think it’s reasonable to create a pull request on the packages repo on Github (and the separate LuCi repo) and see if the developers there have more feedback. Once it’s accepted you’ll definitely get more people to try it.

Thanks for this BTW!

1 Like
15

[RETRACTED] This post is officially retracted. Please take note that compressed-memory and luci-app-compressed-memory will be actively maintained, but I will not longer provide them to OpenWrt.

16

How come?
In OpenWrt packages that are just kernel mods are widely used

17

[RETRACTED] This post is officially retracted. Please take note that compressed-memory and luci-app-compressed-memory will be actively maintained, but I will not longer provide them to OpenWrt.

18

Are you patching the kernel or just requiring addition kmods? If it’s the latter then I think it should be fine. I agree that you have to use the kmods that match your kernel but that’s not really an issue.

19

[RETRACTED] This post is officially retracted. Please take note that compressed-memory and luci-app-compressed-memory will be actively maintained, but I will not longer provide them to OpenWrt.

20

Are the patches just to build new kmods? Doesn’t seem like that have any side effects if it was approved.

Is it possible for this to work if they’re missing?

21

[RETRACTED] This post is officially retracted. Please take note that compressed-memory and luci-app-compressed-memory will be actively maintained, but I will not longer provide them to OpenWrt.

1 Like
22

Hi,

Nice work. A couple of suggestions:

  1. Choosing lz4 as default might be better. It's faster and most OpenWRT devices do not have a powerful cpu. If the memory situation is so dire that zstd over lz4 is absolutely required, upgrading hardware is probably a better solution. And do not use lz4-hc, yes it decompress as fast as lz4, but the compression is much slower than even zstd.

  2. zram writeback and zswap have much smaller use case, think of routers that have 16/128 configuration. Not saying it's useless, but I'd focus on zram first. Also I've seen some raspberry folks claim that zswap introduces crashes up until 5.0 for them. Not sure how it affects openwrt though.

Lastly, this won't help most 4/32 devices for newer releases as the 4mb is the bigger trouble and the rom is already lzma compressed. However, if this could be push into 18.04/17.10, some 32m devices may benifit.

Cheers,
Lun

1 Like