Package table: re-indexing

Talk about Documentation
1

@bobafetthotmail I noticed that the package update script and the re-indexing is running with nice 19.
Since we delete the complete searchindex before the re-indexing, the wiki search and all pagequerys do not show any data during this time, which is baaaad (I stumbled over an effect of this this morning: The "complete lists" section of https://openwrt.org/toh/views/start was completely empty, making me quite nervous... It worked hours ago, why not now?. I was relieved when I realized that the re-indexing was causing this.)

In order to keep the searchindex "downtime" as low as possible, I suggest to let the indexer run with at least nice 0. I think the server is beefy enough to handle this without major impact on webserver performance.

In addition to this, 00:00 server time is right where traffics is ramping up again.
20:00 to 22:00 would be a better time to start the update.

From todays stats:
grafik

Could you please implement these two changes for the next update run?

They will certainly not lead to 50% less update time, but if we can get 5% faster, why not?

2

I'll give the wiki reindexing command (which is the culprit as you said) more priority as you requested, since it has to finish the job ASAP.

Currently yeah it runs at the same low priority of the package indexing script that does not have a deadline per-se

The 0:00 is the time the package indexing script starts, that's fine.

I'll have it wait until it's 20:00 to actually do the part that requires the wiki reindex, which is only this block at the end of the script:

    #deleting all files in there as we are going to just replace them to avoid leaving ghosts in the database
    rm -R /var/www/dokuwiki/data/pages/packages/pkgdata
    rm -R /var/www/dokuwiki/data/pages/packages/index

    cp -av "$newpackages_folder/$package_release/." /var/www/dokuwiki/data/pages/packages/ || exit 1

    echo files copied

    #-c option here should clear the index before reindexing and since the above does not work decently i am using it.
    sudo -u www-data /var/www/dokuwiki/bin/indexer.php -c > /home/bobafetthotmail/indexer_package_add.log 2>&1

fi
1 Like