OpenWrt Forum Archive

Topic: squid 3.4.8 with sslbump and dynamic cert generation for chaos calmer

The content of this topic has been archived on 27 Mar 2018. There are no obvious gaps in this topic, but there may still be some posts missing at the end.

With a lot of patience mixed with cursing I managed to build a working Squid 3.4.8 with SSL bump, dynamic certificate generation and "disk" caching for chaos calmer trunk (r43084). It requires libstdcpp, libpthread, libnettle, librt, libnetfilter-conntrack and openssl. the ca-certificates package is also needed to run, though not to build. Easy-RSA is recommended since it makes the creation of your root certificate easy.

BE FOREWARNED - Sniffing HTTPS traffic and re-signing via your own certificate authority can be a dangerous thing.  If you don't have a clear understanding of the risks and consequences, please educate yourself on HTTPS and how certificates are used in PKI.

The Good News: Once you import your root certificate into the clients, things just work. The first time you load a particularly big SSL site (especially one that pulls HTTPS content from many different domains) you'll have a delay (2-3 seconds) as the certs are retrieved and generated, but subsequent loads are fast as ever.

The Bad News: The binary is over 5 MB! There has to be a way to trim this down, I just don't know how to do it. Facebook's authentication breaks, so most apps that use your Facebook login will break. Other OpenID-powered sites worked fine for me, so I'm not precisely sure what the issue is. Regardless, that's a squid-dev problem to discuss. Google, Amazon and other federated identity providers worked perfectly throughout my testing.


Here's a tar.gz to drop into the openwrt source build tree. In "make menuconfig" it will appear under Network -> Web Servers/Proxies.

Here's a sample configuration for performing transparent ssl bumping. Assumes the use of easy-rsa to generate the CA cert. Also implements caching to an SD card (/mnt/sdcard).

/etc/squid3/squid.conf:

acl localnet src 10.0.0.0/8     # RFC1918 possible internal network
acl localnet src 172.16.0.0/12  # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl localnet src fc00::/7       # RFC 4193 local private network range
acl localnet src fe80::/10      # RFC 4291 link-local (directly plugged) machines

acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT

# HTTP Stuff
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow localnet
http_access allow localhost
http_access deny all
http_port 3128 intercept

# SSL Stuff
acl no_bump dstdomain .pressganey.com
ssl_bump        none            localhost
ssl_bump        none            no_bump
ssl_bump        server-first    all
sslcrtd_program /usr/lib/squid3/ssl_crtd -s /mnt/sdcard/ssl_db -M 4MB
https_port 3129 intercept ssl-bump cert=/etc/easy-rsa/keys/ca.crt key=/etc/easy-rsa/keys/ca.key generate-host-certificates=on
#The following port is required for overflow/outbound negotiation of server-first connections
http_port 3127

#Logging
access_log      /mnt/sdcard/cache/access.log
cache_log       /mnt/sdcard/cache/cache.log
cache_store_log /mnt/sdcard/cache/store.log
pid_filename    /mnt/sdcard/cache/squid.pid
netdb_filename  /mnt/sdcard/cache/netdb.state
coredump_dir    /mnt/sdcard/cache

#Caching
cache_mem 2 MB
maximum_object_size_in_memory 128 KB
cache_dir ufs /mnt/sdcard/cache 5000 16 256
maximum_object_size 200 MB
cache allow all

You'll also need two rules in /etc/config/firewall to auto-forward HTTP and HTTPS traffic to squid (Change the subnet and router IP to suit):

config redirect
    option src 'lan'
    option proto 'tcp'
    option src_ip '192.168.1.0/24'
    option src_dport '80'
    option dest_ip '192.168.1.1'
    option dest_port '3128'
    option target 'DNAT'

config redirect
    option src 'lan'
    option proto 'tcp'
    option src_ip '192.168.1.0/24'
    option src_dport '443'
    option dest_ip '192.168.1.1'
    option dest_port '3129'
    option target 'DNAT'

In case Google Drive sharing decides not to work here are the 3 files you need:

package/network/squid3/Makefile:

#
# Copyright (C) 2006-2012 OpenWrt.org
#
# This is free software, licensed under the GNU General Public License v2.
# See /LICENSE for more information.
#

include $(TOPDIR)/rules.mk

PKG_NAME:=squid
PKG_VERSION:=3.4.8
PKG_RELEASE:=1

PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.bz2
PKG_SOURCE_URL:=http://www.squid-cache.org/Versions/v3/3.4/
PKG_MD5SUM:=094bd5f974d13485d51d02e93ec6027b

include $(INCLUDE_DIR)/package.mk

define Package/squid3/Default
  SECTION:=net
  CATEGORY:=Network
  SUBMENU:=Web Servers/Proxies
  URL:=http://www.squid-cache.org/
endef

define Package/squid3
  $(call Package/squid3/Default)
  MENU:=1
  DEPENDS:=+libpthread +libopenssl +libnettle +librt +libnetfilter-conntrack +libstdcpp
  TITLE:=full-featured Web proxy cache
endef

define Package/squid3/description
    Squid is a high-performance proxy caching server for web clients,
    supporting FTP, gopher, and HTTP data objects. Unlike traditional
    caching software, Squid handles all requests in a single,
    non-blocking, I/O-driven process.
    This implementation is geared specifically toward transparent
    proxying and caching of HTTP and HTTPS traffic, including sslbump
    and dynamic certificate generation.
endef

define Package/squid3/conffiles
/etc/squid3/cachemgr.conf
/etc/squid3/errorpage.css
/etc/squid3/mime.conf
/etc/squid3/squid.conf
endef

CONFIGURE_ARGS += \
    --datadir=/usr/share/squid3 \
    --libexecdir=/usr/lib/squid3 \
    --sysconfdir=/etc/squid3 \
    --enable-shared \
    --enable-static \
    --enable-x-accelerator-vary \
    --with-pthreads \
    --with-dl \
    --enable-icmp \
    --enable-kill-parent-hack \
    --enable-arp-acl \
    --enable-err-languages=English \
    --enable-default-err-language=English \
    --enable-linux-netfilter \
    --enable-icmp \
    --disable-external-acl-helpers \
    --disable-auth-negotiate \
    --disable-auth-ntlm \
    --disable-auth-digest \
    --disable-auth-basic \
    --disable-wccp \
    --disable-wccpv2 \
    --disable-snmp \
    --disable-htcp \
    --enable-underscores \
    --enable-cache-digests \
    --enable-referer-log \
    --enable-delay-pools \
    --enable-useragent-log \
    --enable-storeio \
    --enable-epoll \
    --with-maxfd=4096 \
    --without-libcap \
    --enable-icap-client \
    --enable-ssl \
    --enable-ssl-crtd

CONFIGURE_VARS += \
    ac_cv_header_linux_netfilter_ipv4_h=yes \
    ac_cv_epoll_works=yes \

define Build/Compile
    # pass INCLUDES to compile host sources against our OpenSSL, not the host one
    $(MAKE) -C $(PKG_BUILD_DIR)/lib \
        all
    $(MAKE) -C $(PKG_BUILD_DIR) \
        DESTDIR="$(PKG_INSTALL_DIR)" \
        install
endef

define BuildPlugin
  define Package/$(1)/install
    $(INSTALL_DIR) $$(1)/usr/lib/squid3
    $(CP) $(PKG_INSTALL_DIR)/usr/lib/squid3/$(2) $$(1)/usr/lib/squid3/
  endef

  $$(eval $$(call BuildPackage,$(1)))
endef

define Package/squid3/install
    $(INSTALL_DIR) $(1)/etc/squid3
    $(CP) $(PKG_INSTALL_DIR)/etc/squid3/* $(1)/etc/squid3/
    $(INSTALL_DIR) $(1)/usr/bin
    $(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/bin/purge $(1)/usr/bin/
    $(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/bin/squidclient $(1)/usr/bin/
    $(INSTALL_DIR) $(1)/usr/share/squid3
    $(CP) $(PKG_INSTALL_DIR)/usr/share/squid3/* $(1)/usr/share/squid3/
    $(INSTALL_DIR) $(1)/usr/sbin
    $(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/sbin/squid $(1)/usr/sbin/
    $(INSTALL_DIR) $(1)/usr/lib/squid3
    $(CP) $(PKG_INSTALL_DIR)/usr/lib/squid3/* $(1)/usr/lib/squid3/
    /bin/chmod 4755 $(1)/usr/lib/squid3/pinger
endef

$(eval $(call BuildPackage,squid3))

package/network/squid3/patches/001-cross_compile.patch:

--- a/src/Makefile.in
+++ b/src/Makefile.in
@@ -7295,7 +7295,7 @@
 
 # cf_gen builds the configuration files.
 cf_gen$(EXEEXT): $(cf_gen_SOURCES) $(cf_gen_DEPENDENCIES) cf_gen_defines.cci
-    $(HOSTCXX) -o $@ $(srcdir)/cf_gen.cc -I$(srcdir) -I$(top_builddir)/include/ -I$(top_builddir)/src
+    g++ -o $@ $(srcdir)/cf_gen.cc -I$(srcdir) -I$(top_builddir)/include/ -I$(top_builddir)/src
 
 # squid.conf.default is built by cf_gen when making cf_parser.cci
 squid.conf.default squid.conf.documented: cf_parser.cci

package/network/squid3/patches/002-skip-configure-ssl-tests.patch:

--- a/configure
+++ b/configure
@@ -22221,12 +22221,10 @@
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether the SSL_get_certificate is buggy" >&5
 $as_echo_n "checking whether the SSL_get_certificate is buggy... " >&6; }
   if test "$cross_compiling" = yes; then :
-  { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
-$as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
-as_fn_error $? "cannot run test program while cross compiling
-See \`config.log' for more details" "$LINENO" 5; }
+  { $as_echo "$as_me:${as_lineno-$LINENO}: cross-compiler, cannot test, assume no" >&5
+$as_echo "$as_me: cross-compiler, cannot test, assume no" >&2;} 
 else
-  cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+{  cat confdefs.h - <<_ACEOF >conftest.$ac_ext
 /* end confdefs.h.  */
 
 
@@ -22263,17 +22261,17 @@
 fi
 rm -f core *.core core.conftest.* gmon.out bb.out conftest$ac_exeext \
   conftest.$ac_objext conftest.beam conftest.$ac_ext
+}
 fi
 
 
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether the workaround for SSL_get_certificate works" >&5
 $as_echo_n "checking whether the workaround for SSL_get_certificate works... " >&6; }
   if test "$cross_compiling" = yes; then :
-  { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
-$as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
-as_fn_error $? "cannot run test program while cross compiling
-See \`config.log' for more details" "$LINENO" 5; }
+  { $as_echo "$as_me:${as_lineno-$LINENO}: cross-compiler, cannot test, assume no" >&5
+$as_echo "$as_me: cross-compiler, cannot test, assume no" >&2;} 
 else
+{
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
 /* end confdefs.h.  */
 
@@ -22313,6 +22311,7 @@
 fi
 rm -f core *.core core.conftest.* gmon.out bb.out conftest$ac_exeext \
   conftest.$ac_objext conftest.beam conftest.$ac_ext
+}
 fi
 

Much thanks.  I was able to get this packaged for r41815 with one minor addition to Depends:=+libltdl.

Have you submitted that as patch to the dev team?

Thank you for tackling this package.

I tried compiling the squid3 package on Ubuntu 14.10 for the x86_64 platform on a fresh install of trunk (r43868). This was not successfully unfortuncately (see below) using a normal 'mkae' (no parallel processing using -j x).

I understand libltdl is built as dependency of libtool. It is certainly marked as to be built in my .config and the build actually happens. I can also find libltdl.so.7 (it is a symlink pointing at libltdl.so in the same directory). Could Chnageset 43793
https://dev.openwrt.org/changeset/43793 "tools: build libtool after autoconf, not before (fixes #18659)" be the reason? I don't think anything else chnaged much since you posted to this forum.

PS: Once fixed I am more than happy to submit a patch to devs (crediting laird.bedore of course)

Cheers




...
make[3]: Entering directory '/home/hanno/apu-trunk/feeds/packages/net/squid3'
mkdir -p /home/hanno/apu-trunk/bin/x86_64/packages /home/hanno/apu-trunk/build_dir/target-x86_64_uClibc-0.9.33.2/squid-3.4.8/ipkg-x86_64/squid3/CONTROL /home/hanno/apu-trunk/staging_dir/target-x86_64_uClibc-0.9.33.2/pkginfo
install -d -m0755 /home/hanno/apu-trunk/build_dir/target-x86_64_uClibc-0.9.33.2/squid-3.4.8/ipkg-x86_64/squid3/etc/squid3
cp -fpR /home/hanno/apu-trunk/build_dir/target-x86_64_uClibc-0.9.33.2/squid-3.4.8/ipkg-install/etc/squid3/* /home/hanno/apu-trunk/build_dir/target-x86_64_uClibc-0.9.33.2/squid-3.4.8/ipkg-x86_64/squid3/etc/squid3/
install -d -m0755 /home/hanno/apu-trunk/build_dir/target-x86_64_uClibc-0.9.33.2/squid-3.4.8/ipkg-x86_64/squid3/usr/bin
install -m0755 /home/hanno/apu-trunk/build_dir/target-x86_64_uClibc-0.9.33.2/squid-3.4.8/ipkg-install/usr/bin/purge /home/hanno/apu-trunk/build_dir/target-x86_64_uClibc-0.9.33.2/squid-3.4.8/ipkg-x86_64/squid3/usr/bin/
install -m0755 /home/hanno/apu-trunk/build_dir/target-x86_64_uClibc-0.9.33.2/squid-3.4.8/ipkg-install/usr/bin/squidclient /home/hanno/apu-trunk/build_dir/target-x86_64_uClibc-0.9.33.2/squid-3.4.8/ipkg-x86_64/squid3/usr/bin/
install -d -m0755 /home/hanno/apu-trunk/build_dir/target-x86_64_uClibc-0.9.33.2/squid-3.4.8/ipkg-x86_64/squid3/usr/share/squid3
cp -fpR /home/hanno/apu-trunk/build_dir/target-x86_64_uClibc-0.9.33.2/squid-3.4.8/ipkg-install/usr/share/squid3/* /home/hanno/apu-trunk/build_dir/target-x86_64_uClibc-0.9.33.2/squid-3.4.8/ipkg-x86_64/squid3/usr/share/squid3/
install -d -m0755 /home/hanno/apu-trunk/build_dir/target-x86_64_uClibc-0.9.33.2/squid-3.4.8/ipkg-x86_64/squid3/usr/sbin
install -m0755 /home/hanno/apu-trunk/build_dir/target-x86_64_uClibc-0.9.33.2/squid-3.4.8/ipkg-install/usr/sbin/squid /home/hanno/apu-trunk/build_dir/target-x86_64_uClibc-0.9.33.2/squid-3.4.8/ipkg-x86_64/squid3/usr/sbin/
install -d -m0755 /home/hanno/apu-trunk/build_dir/target-x86_64_uClibc-0.9.33.2/squid-3.4.8/ipkg-x86_64/squid3/usr/lib/squid3
cp -fpR /home/hanno/apu-trunk/build_dir/target-x86_64_uClibc-0.9.33.2/squid-3.4.8/ipkg-install/usr/lib/squid3/* /home/hanno/apu-trunk/build_dir/target-x86_64_uClibc-0.9.33.2/squid-3.4.8/ipkg-x86_64/squid3/usr/lib/squid3/
/bin/chmod 4755 /home/hanno/apu-trunk/build_dir/target-x86_64_uClibc-0.9.33.2/squid-3.4.8/ipkg-x86_64/squid3/usr/lib/squid3/pinger
find /home/hanno/apu-trunk/build_dir/target-x86_64_uClibc-0.9.33.2/squid-3.4.8/ipkg-x86_64/squid3 -name 'CVS' -o -name '.svn' -o -name '.#*' -o -name '*~'| xargs -r rm -rf
Package squid3 is missing dependencies for the following libraries:
libltdl.so.7
Makefile:126: recipe for target '/home/hanno/apu-trunk/bin/x86_64/packages/packages/squid3_3.4.8-1_x86_64.ipk' failed
make[3]: *** [/home/hanno/apu-trunk/bin/x86_64/packages/packages/squid3_3.4.8-1_x86_64.ipk] Error 1
make[3]: Leaving directory '/home/hanno/apu-trunk/feeds/packages/net/squid3'
package/Makefile:174: recipe for target 'package/feeds/packages/squid3/compile' failed
make[2]: *** [package/feeds/packages/squid3/compile] Error 2
make[2]: Leaving directory '/home/hanno/apu-trunk'
package/Makefile:171: recipe for target '/home/hanno/apu-trunk/staging_dir/target-x86_64_uClibc-0.9.33.2/stamp/.package_compile' failed
make[1]: *** [/home/hanno/apu-trunk/staging_dir/target-x86_64_uClibc-0.9.33.2/stamp/.package_compile] Error 2
make[1]: Leaving directory '/home/hanno/apu-trunk'
/home/hanno/apu-trunk/include/toplevel.mk:173: recipe for target 'world' failed
make: *** [world] Error 2

Hey Chillfire,
Like otte_1981 mentioned, the quick fix appears to be to add +libltdl to the DEPENDS: line in the Makefile.

It seems that libltdl is not always required, but when it's enabled it takes ownership of certain libtool functions. I've been doing my builds and tests against a few different MIPS-based platforms, all without libltdl. What should probably work for all platforms/situations would be to add something like this to the DEPENDS:

+PACKAGE_libltdl:libltdl

That way, libltdl will be included if it's being built, otherwise it will be ignored.

-Laird

(Last edited by laird.bedore on 9 Jan 2015, 15:57)

I have posted a new, improved, and updated package for Squid 3.4.10. See this post for all the details.

-Laird

The discussion might have continued from here.