Migrating luci-app-https-dns-proxy to javascript -- seeking suggestions for better design

While mulling over how to migrate existing lua app to javascript I came up with a few options/questions/roadblocks on which I'd very much appreciate the feedback from people doing development for living, especially people who tremendously helped me before, like @aparcar @dibdot @feckert @jow @hnyman @slh.

My goal is to keep the existing config file where instances directly translate to the binary's CLI parameters. I don't want to add anything else there.

With the lua app, I've had lua files for each resolver in the supported providers directory and I had the localizable labels so the names could be translated to other languages.

The first hurdle is transitioning this to json.

I'm considering 3 options for this:

  1. One large file on the router and pulling this file into WebUI with javascript.
  2. Keeping a single file per provider and serving the combined json object to javascript thru RPCD.
  3. Checking for file from GitHub repo and if it fails fall back to option 1.
  4. Option 1 + button in WebUI to pull and save updated list from Github.

Obviously the benefit of options 3 and 4 is that the already installed package does not need to be updated to update the list of resolvers. I would welcome feedback on which option I should pursue.

Second hurdle is the actual structure/storing/loading elements in a json/javascript file. I was thinking of something like:

const resolvers = [
  {
    name: "FooDNS",
    template: "https://dh-{country}.foodns.com/{username}/dns-query",
    regex:
      "https://dh-(?<country>[a-z]*).foodns.com/(?<username>.*?)/dns-query",
    params: {
      country: {
        type: "select",
        options: ["jp", "ch", "sg"],
        value: "jp",
      },
      username: {
        value: "user",
        type: "text",
      },
    },
  },
  {
    name: "Custom",
    template: "{url}",
    regex: ".*",
    params: {
      url: {
        type: "text",
      },
    },
  },
];

code on runkit

With the template used for building the final URL to be stored in config and regex for matching resolver URLs from config and the type in the params being used to display the relevant WebUI element (like a drop-down for list of supported countries or options, like private, family, ad-blocking and the text field for non-predetermined user-customizable options, like the username for NextDNS.io or custom lists for blitz.ahadns.com)

So the questions are:

  1. How to implement customizable labels similar to those in lua files?
  2. Being new to javascript/json any feedback for general organization/naming of objects in that?

Thanks in advance gentlemen!

2 Likes

The i18n string extraction script (https://github.com/openwrt/luci/blob/master/build/i18n-scan.pl) already considers title and description properties to be translatable strings, we just need to extend the find arguments to include your resolver JSON files.

In your JavaScript code you would then do something like:
let resolver_title = _(resolver_data.title);

As for the organization of the JSON spec files, I think implementing an rpcd call which renders out one single array of resolver dictionaries is the way to go.

Your regex matching with named capture groups that are mapped to parameter inputs makes sense too, but it entails duplicate work which might be prone to errors. You would need to both maintain the regexp pattern and the template string. Maybe you should just use the template string and move the validation patterns into the parameter descriptions:

const resolvers = [
  {
    name: "FooDNS",
    template: "https://dh-{country}.foodns.com/{username}/dns-query",
    params: {
      country: {
        type: "select",
        regex: "[a-z]*",
        options: ["jp", "ch", "sg"],
        value: "jp",
      },
      username: {
        value: "user",
        type: "text",
      },
    },
  },
  {
    name: "Custom",
    template: "{url}",
    params: {
      url: {
        regex: ".*",  // ".*" would be implicit default
        type: "text",
      },
    },
  },
];

To turn the URL template string into a regex instance for matching, you can use a function like this:

let template = 'https://dh-{country}.foodns.com/{username}/dns-query';

function templateToRegexp(template) {
    return RegExp('^' + template.split(/(\{\w+\})/g).map(part => {
        let placeholder = part.match(/^\{(\w+)\}$/);

        if (placeholder)
            return `(?<${placeholder[1]}>.*?)`;
        else
            return part.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
    }).join('') + '$');
}

let regexp = templateToRegexp(template);
/* 
results in: 
/^https:\/\/dh-(?<country>.*?)\.foodns\.com\/(?<username>.*?)\/dns-query$/
*/

As an additional idea:

You could even let the user paste a complete URL and select the appropriate resolver plus extract the relevant parameters automatically by comparing the user entered URL against each providers generated regexp.

The same approach could likely be adopted for DDNS-Scripts, where a user could simply enter the completely formatted URL to fully configure a provider entry.

1 Like

Thank you for your prompt reply, the quoted brilliant suggestion and the code sample.

I've made good progress converting the status.js part to javascript and while I haven't tested it with many providers the current js code seems to work well:

The overview.js part where users could add/configure resolvers is yet to be written tho. Given your experience with luci, do you have a suggestion how to implement it?

I was thinking of using the form.GridSection with resolver as a first column, custom options second and then address and port. Similar to this screenshot but with additional column for customizations. However some providers would have no customization options, some would have a drop-down for customization option and some would have a text entry. Also would be nice to have a customization option title and they may be different for different providers (some have regions, some have filtering options, some have user names).

I'm struggling to reconcile it with the current luci framework and I'd like to use built-ins as much as possible and not code/render whole section from scratch.

Any suggestions @jow?

Maybe take a look at luci-app-statistics, in particular the plugin configuration. It uses GridSections as well and overrides the row action render function to customize (in this case show/hide) the Edit button.

The different nature of the additional customization options should be no issue for the GridSection table view if they're only rendered in the modal popup. Hiding the edit button for those providers without customization option should be doable by overriding the row action render function.

1 Like

The main thing is not to greatly increase the size of the luci-app-https-dns-proxy package, a huge request to you.
I set it up via PuTTY and everything works fine, no matter how much I need this function, the main thing is that the weight does not increase, so that you can install an eight megabit router and not only one luci-app-https-dns-proxy, but you also need to install other packages, you need at least flash memory priority, and so thank you very much, great program!

So far, the js IPK is 2Kb smaller than existing lua IPK. When all said and done, it will probably still be about 1Kb smaller. And it's definitely going to be less strenuous on the low resource routers as most processing is done in the browser, not on the router itself.

Hi,

What are the advantages of the rewrite to javascript?

Smaller package, less required dependencies. This is not about the resolver functionality itself but about the configuration ui part.

Thank you for the info

Is there a documentation/code I could peruse to learn more about modal popup customization options?

Customizing options within the modal popup should work similar to ordinary options in TpyedSections, NamedSections etc. There's some additional properties to control whether options are also shown in the table preview, whether they're editable there etc.

See http://openwrt.github.io/luci/jsapi/LuCI.form.GridSection.html

You can override the GridSections addModalOptions() function to modify the CBI map displayed in the modal popup before it is rendered. There's also a number of per-option properties to control behavior in GridSection context, namely .editable (control whether option is rendered as input or text preview within grid section row), .modalonly (control whether an option is shown only in the gridsection table, only in the modal, or both) and textvalue() to override how a widget is displayed in text form in the GridSection preview.

Then there's the undocumented internal GridSection.renderRowActions() method which is responsible for rendering the per-row action buttons. An implementation could look like this:

s = m.section(form.GridSection, ...);
s.renderRowActions = function(section_id) {
    let provider = uci.get('https-dns-proxy', section_id, 'provider');
    let tdEl = E('td', {
        'class': 'td cbi-section-table-cell nowrap cbi-section-actions'
    }, E('div'));

    if (has_additional_options(my_provider_data[provider])) {
        dom.append(tdEl.lastElementChild,
            E('button', {
                'class': 'cbi-button cbi-button-edit',
                'click': ui.createHandlerFn(this, 'renderMoreOptionsModal', section_id)
            }, [ _('Customize provider…') ])
        );
    }

    return tdEl;
}

It would show or not show a "Customize provider…" button next to each GridSection row, depending on whether the provider configured for the given row has additional options or not.

1 Like

Hello @jow!

Thank you for the invaluable advice in the previous reply. I've been making progress towards final steps of the new luci app, but hit a wall again. So there's one uci option (resolver_url) that would not be shown to a user and two other options would be shown instead (a Provider option and a Parameter option). On load I'll need to parse the resolver_url from config into Provider/Parameter, but then if these are changed, but not saved, I will no longer need to get the resolver_url from config.

I wonder how to best approach it. I've discovered the https://openwrt.github.io/luci/jsapi/LuCI.form.HiddenValue.html which says it's not the most efficient way of storing hidden data now -- what would be the efficient way nowdays (what code can I peruse for examples)?

Also, when the form is being rendered within the GridSection -- how do I get a reference to the section_id of the currently rendered row?

Finally, would this: https://github.com/openwrt/luci/blob/b3d661cd84760a0cdf084a25f21556a07e369d33/modules/luci-mod-network/htdocs/luci-static/resources/tools/network.js#L549-L555 be a good example to look at of how can I update/re-render one option within modal window when another option is changed?

I don't understand your requirement here. Do you wish to present a single underlying uci option as multiple widgets which are then written back into a single uci option on save?

Since it's all client side JS now you can simply store your data as property directly within the option object, like this:

o = s.option(...);
o.mydata = something;

Or utilize closures, bound functions or other native JavaScript facilities.

I guess one example would be the zone selector widget, which queries a bunch of data in it's load() callback, stores it as custom properties and then uses this data throughout the other functions in the rendering and event handling logic.

Depends on the context. Within addModalOptions() you'll receive the currently rendered section_id as second argument. Additionally the section specific Map() instance rendered within the modal popup will have a .section string property set, containing the uci section_id the map refers to. So if you're within an option or section load() callback you can refer to this.map.section to obtain the UCI section ID value the map is bound to.

Thanks again for your prompt reply!

Yes.

Do I make 'something' different for different typed instances with the load() function?

2 posts were split to a new topic: How to setup DoH server failover

With the sectiontitle function is it possible to override the 'Name' heading in the grid? I want to change Name to Size and I'm not sure how.

Why can't you just change the title from Name to Size? Did you implement the option in a way that it shows a size in the grid overview and a provider name in the modal popup?

If so I suggest to simply declare two different options instead, one for size and one for the name, then set the first to modalonly = false and the second to modalonly = true:

o = s.option(form.DummyValue, '_size', 'Size');
o.modalonly = false; // only show in grid view
...

o = s.option(form.DummyValue, '_name', 'Name');
o.modalonly = true; // only show in modal
...
1 Like

Thanks, didn't realize I can override cfgvalue on a dummy option!

Makes me wonder why was the sectiontitle introduced? What's the difference between utilizing sectiontitle and using a dummy option with custom cfgvalue?