XHR request timed out (restore backup)

Hi;

I have made some additions to the luci backup / restore script, mainly for databases , dramatically increasing execution time resulting in page time-outs during backup / restore.

Because of openwrt updates, this issue has resurfaced:

which was solved by following @Ansuel recomendation (poll result file):

<script type="text/javascript">//<![CDATA[
        function refresh_status(x, data)
        {
                var file = "/backup.txt?rand=" + Math.random();;
                $( "#contain" ).load( file , function (data)
                        {
                                var status = document.getElementById('status');
                                status.innerHTML = data;
                        }
                );
        }

	XHR.poll(5, '<%=url('admin/system/flashops')%>', null,
                function(x, data)
                {
                        refresh_status(x, data);
                }
        );
//]]></script>

Since then, I have updated to a more recent trunk:

from:
commit 1b937cb14184b5ff9a7a75fbc5d226032f931c35
Author: Rafał Miłecki rafal@milecki.pl
Date: Wed Jul 3 11:16:22 2019 +0200

to:
commit 7cb721c03fdc163818f8114692229d0097d2f26b
Author: Adrian Schmutzler freifunk@adrianschmutzler.de
Date: Tue Jul 7 11:49:36 2020 +0200

where luci, cgi... have dramatically changed.

nginx.conf:

# Note: external servers (vhosts) should not redirect http to https, else redirect loop (eg: sme-server)

user root;
worker_processes auto;

pid /var/run/nginx.pid;

events {
	worker_connections 1024;
}
#events {}

http {
	#access_log off;
	log_format openwrt
		'$request_method $scheme://$host$request_uri => $status'
		' (${body_bytes_sent}B in ${request_time}s) <- $http_referer';
	include	mime.types;
	default_type application/octet-stream;
	sendfile on;
	keepalive_timeout 0;
	# client_body_buffer_size 10K;
	# client_header_buffer_size 1k;
	# client_max_body_size 1G;
	#large_client_header_buffers 2 1k;
	client_max_body_size 128M;
	#large_client_header_buffers 2 1k;
	gzip on;
	#gzip_http_version 1.1;
	gzip_vary on;
	#gzip_comp_level 1;
	gzip_proxied any;
	root /www;

	server {
		listen [::]:80 default_server;
		listen 80 default_server;
		server_name _;
		return 301 https://$host$request_uri;
	}
	server {
		listen 443 ssl default_server;
		listen [::]:443 ssl default_server;
		server_name localhost;
		ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
		ssl_prefer_server_ciphers on;
		ssl_ciphers "EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH:DHE+AESGCM:DHE:!RSA!aNULL:!eNULL:!LOW:!RC4:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!CAMELLIA:!SEED";
		ssl_session_tickets off;
                ssl_certificate /etc/ssl/domains/domain.crt;
                ssl_certificate_key /etc/ssl/domains/domain.key;
		fastcgi_connect_timeout 300;
		fastcgi_send_timeout 300;
		fastcgi_read_timeout 300;
		fastcgi_buffer_size 32k;
		fastcgi_buffers 4 32k;
		fastcgi_busy_buffers_size 32k;
		fastcgi_temp_file_write_size 32k;
		client_body_timeout 10;
		client_header_timeout 10;
		send_timeout 120;		# 60 sec should be enough, if experiencing a lot of timeouts, increase this.
		output_buffers 1 32k;
		postpone_output 1460;
		location ~* .(jpg|jpeg|png|gif|ico|css|js)$ {
			expires 365d;
		}
		location /cgi-bin/luci {
			index  index.html;
			include uwsgi_params;
			uwsgi_param SERVER_ADDR $server_addr;
			uwsgi_modifier1 9;
			uwsgi_pass unix:////var/run/luci-webui.socket;
		}
		location ~ /cgi-bin/cgi-(backup|download|upload|exec) {
			include uwsgi_params;
			uwsgi_param SERVER_ADDR $server_addr;
			uwsgi_modifier1 9;
			uwsgi_pass unix:////var/run/luci-cgi_io.socket;
		}
		location /luci-static {
			error_log stderr crit;
		}
		location /ubus {
			ubus_interpreter;
			ubus_socket_path /var/run/ubus.sock;
			ubus_parallel_req 2;
		}
		location ~ \.php$ {
			fastcgi_index index.php;
			include fastcgi_params;
			fastcgi_param  SCRIPT_FILENAME $document_root$fastcgi_script_name;

			if (-f $request_filename) {
				# Only throw it at PHP-FPM if the file exists (prevents some PHP exploits)
				#fastcgi_pass 127.0.0.1:1026;     # The upstream determined above
				fastcgi_pass unix:/var/run/php7-fpm.sock;
			}
		}
		include locations/*.conf;
	}
	include vhosts/*.conf;
}

I am unsure (given openwrt updates) what is the correct way to approach / solve this.

Questions:

  • How identify what is timing out (approx 20 seconds)
  • Is there a timeout I can increase
  • If polling a result file is still the correct approach, how do I alter flash.js to achieve this.
  • General suggestion: If there is no way to control script_timeout on a page basis, can it be added?

Thanks;
Bill

I think the best you can do is move all of your custom function to a ubus call and use a status to check for the progress instead of checking the file directly (and wait)

This way you can change your script to pool for the status (much faster and you will solve your timeout problem) and ONLY IF the status is for example (completed) request the file...

Also using the ubus call luci api will clean your code

Hi Ansuel;
Current approach is using a wrapper around /sbin/sysupgrade:

SYSUPGRADE=/sbin/sysupgrade_real # renamed original sysupgrade
CMD=$1
FILE=$2

case $CMD in
        --create-backup )
                # Backup additional stuff
                do_save
                if [ "$FILE" == "-" ]; then
                        tar --no-recursion -zcf - -T config.files 2> /dev/null
                fi
        ;;
        --restore-backup )
                # Restore additional stuff
                do_restore $FILE
        ;;
        *)
                # Do non backup, restore commands
                $SYSUPGRADE $*
        ;;
esac

to avoid changing luci code
Are you recommending altering flash.js to use ubus calls as opposed to fs.exec? or, modify previous result polling to use ubus API?
Is there a package that does something similar you can point me to?
Is the new luci/ubus API/Interface documented anywhere?

Thanks;
Bill

After exhaustive googling, forced to conclude:

my rudimentary js coding skills are not a saleable job skill
luci client side javascript too new for useful examples on 'net
examples in openwrt code "beyond me"

For the record; Here's what needs to be modified (excerpt, /www/luci-static/resources/view/system/flash.js):

    handleRestoreConfirm: function(btn, ev) {
        return fs.exec('/sbin/sysupgrade', ['--restore-backup', '/tmp/backup.tar.gz']).then(L.bind(function(btn, res) {
            if (res.code != 0) {
                ui.addNotification(null, [E('p', _('The restore command failed with code %d').format(res.code)), res.stderr ? E('pre', {}, [res.stderr]) : '']);
                L.raise('Error', 'Unpack failed');
            }
            btn.firstChild.data = _('Rebooting…');
            return fs.exec('/sbin/reboot');
        }, this, ev.target)).then(L.bind(function(res) {
            if (res.code != 0) {
                ui.addNotification(null, E('p', _('The reboot command failed with code %d').format(res.code)));
                L.raise('Error', 'Reboot failed');
            }
            ui.showModal(_('Rebooting…'), [E('p', {
                'class': 'spinning'
            }, _('The system is rebooting now. If the restored configuration changed the current LAN IP address, you might need to reconnect manually.'))]);
            ui.awaitReconnect(window.location.host, '192.168.1.1', 'openwrt.lan');
        }, this)).catch(function(e) {
            ui.addNotification(null, E('p', e.message))
        }).finally(function() {
            btn.firstChild.data = _('Upload archive...')
        });
    },

As @Ansuel helpfully opined, need to add some sorta ubus call to prevent XHR request timeout. In general, I think that adding a timeout parameter to fs.exec* classes is architecturally the correct approach.

I am still open to help on this, but, the hack / "solution" below is working.

Since I'm not about to be stopped by lack of skills, I opted for "Plan B" (A hack) for the following reasons

  • restore does not happen often and is not performance critical
  • assumes OpenWrt will always reboot after restore
  • luci API is moving target. Do not want to have to revisit this issue
  • I am adding a lot of packages to OpenWrt (www.rossco.org) and try to touch the core as little as possible.

Solution: delay restore (and attendant timing issues) until post restore reboot.

Updated /sbin/sysupgrade (snippet, wrapper):

SYSUPGRADE=/sbin/sysupgrade_real # renamed original sysupgrade
CMD=$1
FILE=$2

case $CMD in
        --create-backup )
                # Backup additional stuff
                do_save
                if [ "$FILE" == "-" ]; then
                        tar --no-recursion -zcf - -T config.files 2> /dev/null
                fi
        ;;
        --restore-backup )
                # Restore additional stuff
                #do_restore $FILE
                CALLER=$(ps -o comm= $PPID)
                if [ "$CALLER" == "rpcd" ]; then
                        # Called by luci, at reboot, rc.local will restore and reboot
                        # Hacky "solution" to "XHR request timed out"
                        mv $FILE /etc/backup.tar.gz
                else
                        # assume called by rc.local
                        do_restore /etc/backup.tar.gz
                fi
        ;;
        *)
                # Do non backup, restore commands
                $SYSUPGRADE $*
        ;;
esac

/etc/rc.local (snippet):

if [ -f /etc/backup.tar.gz ]; then
        # Wait for network
        until ping -c 1 google.com; do sleep 1; done
        /sbin/sysupgrade --restore-backup /etc/backup.tar.gz
        rm -f /etc/backup.tar.gz
        reboot
fi

1 Like

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.