Decrease git repo size localy

First of did not know were to ask. I decided it somehow belongs here. Sorry if i was wrong.
I have WNDR3700v4 and i have cloned openwrt repo. I build firmware with some patches and include many programs and config files in it so that i just upgrade my router and it all works. I usualy just git pull and update and install new feeds with "./scripts/feeds update -a && ./scripts/feeds install -a" But my openwrt git folder is 23 GB in size. That's too big. All my other files combined are 45 GB in size. Can i somehow reduce it's size? I just want to build openwrt for WNDR 3700v4. I like to have 4 old full backups and anything that will reduce overall system size is good. I think building openwrt for just one router won't need 23 GBs of source code and makefiles.

Really? My openwrt/.git is 183MB. Have you tried git gc? BTW adding feeds etc shouldn't affect the OpenWrt .git directory since feeds are excluded in .gitignore.

make dirclean
1 Like

Seems far too large. The feeds repos are generally already cloned as --shallow

jeff@deb-devel:~/devel$ git clone https://git.openwrt.org/openwrt/openwrt.git _openwrt
Cloning into '_openwrt'...
remote: Enumerating objects: 481791, done.
remote: Counting objects: 100% (481791/481791), done.
remote: Compressing objects: 100% (131537/131537), done.
remote: Total 481791 (delta 334330), reused 476798 (delta 330017)
Receiving objects: 100% (481791/481791), 148.68 MiB | 14.24 MiB/s, done.
Resolving deltas: 100% (334330/334330), done.
jeff@deb-devel:~/devel$ du -h -d1 _openwrt/
39K	    _openwrt/config
739K	_openwrt/scripts
1.2M	_openwrt/toolchain
161M	_openwrt/.git
307K	_openwrt/include
9.5K	_openwrt/.github
35M 	_openwrt/target
13M 	_openwrt/package
2.1M	_openwrt/tools
213M	_openwrt/

After ./scripts/feeds update -a

jeff@deb-devel:~/devel$ du -h -d1 _openwrt/
5.5K	_openwrt/tmp
39K 	_openwrt/config
56M 	_openwrt/feeds
739K	_openwrt/scripts
38K 	_openwrt/staging_dir
1.2M	_openwrt/toolchain
161M	_openwrt/.git
307K	_openwrt/include
9.5K	_openwrt/.github
35M 	_openwrt/target
13M 	_openwrt/package
2.1M	_openwrt/tools
268M	_openwrt/

I think we are talking build_dir here, which can be quite sizable...
Running the same du calls as Jeff should show the answer very quickly whether it is expected or borked

2 Likes

The toolchains take up a lot of space. Each time you build for a new target a new toolchain is built and installed.

Not .git i mean openwrt repo itself is 23 GB in size. Openwrt/.git is 211 MB.

What's the point after build it's same size, but after that building takes much more time.

Exactly. build_dir takes 18 GB

 ~/Desktop/git du -h -d1 openwrt/
1.2M    openwrt/toolchain
12K     openwrt/.github
52K     openwrt/config
95M     openwrt/bin
372K    openwrt/include
992M    openwrt/dl
53M     openwrt/tmp
180M    openwrt/feeds
18G     openwrt/build_dir
211M    openwrt/.git
58M     openwrt/target
1.9M    openwrt/scripts
21M     openwrt/package
3.0M    openwrt/tools
3.8G    openwrt/staging_dir
24G     openwrt/

So is there a way to reduce build_dir size? 18 GB are needed to build 25 MB Firmware for only one router?

Edit:
Did not realized there would be different toolchains in build_dir and staging_dir for different gcc versions, so now i deleted toolchains for gcc 7 & 8 in both dirs and now openwrt/ folder size is 14 GB.

1 Like

That seems consistent with my memory that a 16 MB drive for a VM was not sufficient to install command-line Debian and build OpenWrt

Seems rather pointless to backup reproducible binaries. But then again I would say the only thing worth backing up from ones buildroot would be the output of:

./scripts/diffconfig.sh

assuming the other bits from the OP (patches...) are available.

You can also use Advanced configuration options (for developers) --> [*] Automatic removal of build directories to keep the build directory size down.

The large build_dir may be an indication that you build all packages by default (possibly by copying the diffconfig from the buildbots?) - it usually makes no sense to build all packages locally. In general it is enough to build with all kmods enabled and to fetch the rest from repositories.

I do not build all of them. I build whatever i choose in menuconfig. Usually firmware comes out about 25 MB in size. I'll try this but what i noticed is after cleaning it takes much more time to build. It will take longer if i remove build directories right?

...and you can also use a filesystem which uses compression on the fly such as ZFS

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.