Uclient-fetch: add more options from wget and create uclient-curl clone of curl

For Developers
1

I would like to contribute to uclient-fetch to extend it's functionality and add options that wget have:

  • --header option to pass an additional header e.g. Authorization: Bearer token or Accept: application/json
  • -method=HTTPMethod to able to send DELETE, PUT and other HTTP methods.
  • --body-data=STRING to be able to send raw JSON

This few options covers most of use cases when you calling some API. And it should be really easy to implement them without big overhead in size or at least I can make them conditional on build.
In the same time this can help to other packages to use the uclient-fetch which now is about 28Kb instead of curl which is about 280kb.
We can also easily add mapping for curl options and create a uclient-curl as a link to uclient-fetch and thus OpenWrt will have some limited curl out of the box. This is very cool because almost all API documentation uses the curl in examples. This is not a new idea https://curl.haxx.se/mail/archive-2017-09/0020.html and can resolve some issues like this https://github.com/Neilpang/acme.sh/issues/1768

The first patch with the --header option I already sent to dev list http://lists.infradead.org/pipermail/openwrt-devel/2020-January/021439.html

But it seems to be ignored and also I just don't want to be subscribed to such dev list because it's like drinking from fire hydrant: I received dozen of emails which are not related to uclient-fetch.
I spent my time and wanted to contribute but my patch is lost and nobody will ever remember of it.

That's why I want to discuss this change here because at least I'm sure that core developers will see it.
Why not just create uclient-fetch mirror on github so I can create a usual PR and watch changes?
I made a quick search on github and there is already a lot of clones of uclient-fetch and some with unique commits:

So why not just use GitHub or GitLab? It's 2020 year then why to make life of contributors harder and use mailing list full of garbage?

Ok, sorry for complaining, I have also few things to discuss.
In fact there is some functionality that I propose to remove.
First of all support of Digest Auth: for all my entire life I never saw that is was used anywhere. It makes no sense when using HTTPS and gives just a small security when used over raw HTTP.
So my proposition is to deprecate it and make it conditional on build.

Next thing to remove is Auth challange which was copied from wget: when used basic auth then wget makes two calls: first without credentials, then it checks that server returned 401 and only then it makes the second call with credentials.
This is absolutely useless functionality. I created a ticket in wget bugtracker where descibed why they should remove it: 57686 Basic Auth: make auth-no-challenge mode by default.

I was affected by this functionality I wanted to disable it but uclient-fetch doesn't have the --auth-no-challenge option. Other OpenWrt users were also confused by this behavior Uclient-fetch and basic auth

So my proposition is just remove this auth challenge functionality. This will make uclient-fetch code much clearer and size a little bit smaller.

Also I'll try to use base64 functions from libubox and remove base64.c file from uclient-fetch so this will make it's size also a little bit smaller.

By removing this unused functionality we can make the uclient-fetch with similar size that it have now while extend it's functionality.

Please tell me if you are ok with my proposed changes and what do you think about them. If you are fine with them then I'll continue my work on uclient-fetch.

Thank you

2

Sorry, don't have much time these days, so excuse my sparse reply.

  • You should read https://openwrt.org/submitting-patches as your patch is very likely going to be rejected because it doesn't contain necessary formalities, like subject, description, signed-off-by etc.
  • AFAIK you don't need to be subscribed to openwrt-devel@ to send patches or get responses to your patches/emails
  • Your patch is not lost, you need to be patient as the review resources are scarce, feel free to reply to that email with "ping" if you don't get response in more then 30-45 days, it's OK
3

ok, instead of removing I can just add the option --auth-no-challenge

4

Very appreciated. This "feature" of uclient, to unconditionally first try a non-authed access, is a waste of bandwidth and time. BTW: I filed a bug report, too, some time ago.
Also, some new features of a curl_light would be appreciated, send file via http. No SSL.

5

uclient-wget: Did you do a patch already regarding "--auth-no-challenge" , or, even better, simply to remove the first GET without credentials ?
If yes, pls provide the sources.

6

No I don't have any patch for this and won't do it until my previous patch with --header param will be accepted

7

Do I understand correctly that basic authorization with some kind of widely-acceptable header using uclient-fetch is impossible for now?

Here's my example.

I'm trying to get dynamic DNS working with YDNS and OpenWrt's uclient-fetch. The shell script of updater provided by YDNS at GitHub uses curl. I've got it working fine, the command script executes is:
curl --basic --insecure -u "myydnsAPIusername:myydnsAPIpassword" https://ydns.io/api/v1/update/?host=my-custom-name.ydns.eu&ip=999.888.777.666
(I dropped --silent from GitHub example because DDNS "Last update / Next update" fields in LuCI didn't update with this option for some reason.)

But curl requires libcurl4, ca-bundle packages to work (all 3 packages require extra disk space). I would like to use OpenWrt's built-in uclient-fetch to keep it simple. But I always get HTTP status code 400 error (uclient-fetch error 8) when I execute my uclient-fetch commands, i.e.:
uclient-fetch -O - --no-check-certificate "https://myydnsAPIusername:myydnsAPIpassword@ydns.io/api/v1/update/?host=my-custom-name.ydns.eu&ip=999.888.777.666"
or
uclient-fetch -O - --no-check-certificate --user="myydnsAPIusername" --password="myydnsAPIpassword" "https://ydns.io/api/v1/update/?host=my-custom-name.ydns.eu&ip=999.888.777.666"

According to YDNS API documentation, HTTP status code 400 indicates The action could not be performed due to invalid input parameters.

I have no idea how curl works and what it's request to API looks like exactly. However I believe I should use some kind of --basic option for uclient-fetch to get it working with YDNS or YDNS blocks uclient-fetch requests because it lacks --auth-no-challenge. Or maybe both things, but according to this post they are impossible right now.

I couldn't find anything about this except this thread and other thread mentioned in first post. That's why I'm asking here for clarification.

8

Please follow the guide for submitting patches mentioned above by ynezz.
Reviewers' time is a scarce resource, you should make their job as easy as possible.

2 Likes
9

I updated and resent the patch https://patchwork.ozlabs.org/project/openwrt/patch/20210508192343.66941-1-stokito@gmail.com/

Also the same patch on GitLab https://gitlab.com/openwrt/project/uclient/-/merge_requests/1

I wasn't able to figure out how to run unit tests, please create some documentation.

The problems with the patchwork that it's very inconvenient to use and hard to understand.
I tried to send a new version of my patch but instead a created a new patch that seems unrelated to previous. I don't know any status of the patch and can't subscribe to it.

The situation is so bad that it's even mentioned in documentation that the only way to do something is to repeat your email:

Reviewers' time is a scarce resource

totally agree, and we lost the resource because of inefficient system. Please, just switch to Github or self hosted Gitlab

Also do we really need the uclient-fetch or why we can't just use BusyBox wget? As far I understood the only one reason was because it didn't supported https. But looks like now it may be a good replacement. It's looks not good to reinvent the wheel of BustBox's reinvented wheel.

P.S. @mpa are you that guy that made a talk about shrinking OpenWrt images?

10

I checked and got the same. Not sure that it's related to the Digest auth (they should return 403 instead of 400). You can report a bug to them (for example here https://github.com/ydns/bash-updater)
I can recommend you to use duckdns.org. It's supported by dynds https://openwrt.org/docs/guide-user/services/ddns/client

But to keep place I just created a /etc/crontab/root file:

43 */4 * * * wget -4 -q -O /tmp/ddns wget -4 -q -O /tmp/ddns http://www.duckdns.org/update/stokito/YOURKEY

The 43 minute I set just to avoid requests waves to their server.

Also you can send PR to add ydns to the OpenWRT ddns client

11

FYI I sent few patched to add wget --header --method and --body-data
https://lists.openwrt.org/pipermail/openwrt-devel/2022-May/038628.html

Please review

12

A short comparisson with BB wget:

  • MUCH smaller
  • Already has --header with overriding
  • Already has --server-response to print headers
  • Patches for --method, --user --password. http://lists.busybox.net/pipermail/busybox/2022-May/089712.html
  • Doesn't have -4 -6 unlike uclient. This may be needed for DynDNS
  • Have some limitations with TLS. This can be improved.
  • Code is tricky and less clear
  • Doesn't support Digest Auth unlike uclient. The DA is almost not used and not needed on HTTPS.
14

@Daniel can you comment on this? I'd be curious to know why we have a client-fetch if bb already supports such thing :slight_smile:

1 Like
16

The biggest draw-back of busybox wget is that it doesn't support libustream-ssl to dynamically link the SSL/TLS backend of the users choice (or none). Also, busybox wget is just a minimalistic wget drop-in and not a library like libuclient.

Regarding support for --header, not knowing about this thread or the MR on gitlab, I've implemented this from scratch in a more libubox'y way and without making any changes to libuclient.

1 Like