Uci -m import is broken?

Installing and Using OpenWrt
1

Am I doing something wrong or is there a bug in uci?

To reproduce the issue:

mkdir -p /tmp/ucitest
# create test files
printf '%s\n%s\n' "config conf1 'conf1_name'" "option opt1_name 'opt1_val'" > /tmp/ucitest/test1
printf '%s\n%s\n' "config conf2 'conf2_name'" "option opt2_name 'opt2_val'" > /tmp/ucitest/test2
# use 1st test file as source
cp /tmp/ucitest/test1 /tmp/ucitest/test-merge
# merge 2nd test file
uci -m -c /tmp/ucitest import test-merge < /tmp/ucitest/test2
cat /tmp/ucitest/test-merge

Output:

config conf1 'conf1_name'
option opt1_name 'opt1_val'

uci import (without -m) works fine:

uci -c /tmp/ucitest import test-merge < /tmp/ucitest/test2
cat /tmp/ucitest/test-merge

Output:

config conf2 'conf2_name'
	option opt2_name 'opt2_val'

OpenWrt 24.10.5

2

Looks like it only works when complete path is specified. Filed an issue:

3

@antonk
Uci creates the merged config in ram.
This is intentional as it allows dynamic, "runtime" alterations of configs without continual writes to flash.

Run your test again, then instead of doing cat to read the file, do:

uci -c /tmp/ucitest show test-merge

You should then get the desired result. Here is what I get:

root@meshnode-8ecb:/tmp/ucitest# uci -c /tmp/ucitest show test-merge
test-merge.conf1_name=conf1
test-merge.conf1_name.opt1_name='opt1_val'
test-merge.conf2_name=conf2
test-merge.conf2_name.opt2_name='opt2_val'
root@meshnode-8ecb:/tmp/ucitest#

If you want this in your ouput file, do:
uci -c /tmp/ucitest commit test-merge

After this cat will show the file has been updated.

4

@bluewavenet I'd like to move the discussion from the Github issue back here since IMO it has drifted away from the issue itself into "how does uci work?".

Quoting your last reply:

If the buffer and file are not in sync, it [the command uci import < <file_path>] does write the file. If you copy the file from somewhere else, as you were doing, then the buffer and file will of course be out of sync. The buffer will be empty.

I'm not sure what you are referring to as 'buffer' and what you mean by 'in sync'. It seems like in your mind, this is some theoretical construction. What I've been referring to is an actual buffer of yet-not-committed config changes, which has an actual path in the filesystem: /tmp/.uci. The uci changes command essentially prints what's stored in that location. There is no other config buffer, as far as I'm aware.

Let me demonstrate this:

grep -r '.' /tmp/.uci/

This command currently outputs nothing, meaning simply that no uncommitted changes exist.

touch /etc/config/test
grep -r '.' /tmp/.uci/

Still outputs nothing.

uci set test.ttt=ttt
grep -r '.' /tmp/.uci/

Output:

/tmp/.uci/test:test.ttt='ttt'

Now there is an unsaved config change, which you can also see by issuing the command:

uci changes

which outputs:

test.ttt='ttt'

Now if I run uci commmit test, this will make the changes to file /etc/config/test and remove the above line from /tmp/.uci/test:

uci commit test
grep -r '.' /tmp/.uci/

(no output)

If I run uci show test, the newly added line is printed:

test.ttt=ttt

Rather than doing this, exact same result will be achieved by manually creating the file /etc/config/test:

rm -f /etc/config/test
uci show test

Output:

uci: Entry not found

No changes are registered in /tmp/.uci:

uci changes

(no output)

printf '%s\n' "config uuu 'uuu'" > /etc/config/test
uci changes

(no output)

uci show test

Output:

test.uuu=uuu

In light of the fact that this is how uci config buffer actually works, and of the fact that there is no other buffer or hidden magic, you should be able to see that manually creating files in /etc/config/ does not make uci go 'out-of-sync'.

If you disagree with these statements, I'm asking you to actually read the above explanation and evidence, as I'm honestly getting tired of repeating the same thing over and over (in the issue, I had to write 3 times (each time with evidence) that uci import [config_name] < [some_file] writes the file immediately, before you half-sorta-accepted the claim, while at the same time making up ideas of 'out-of-sync buffer' and 'manually creating uci config files breaks things'. You are making me work hard explaining things and bringing evidence, only to either ignore it altogether or quasi-explain the obvious and simple facts by some fictional constructs which do not correspond to reality - with no actual evidence.

I'm sorry if this sounds harsh but IMO the way you are conducting this discussion justifies this response. I had to decide between this response and just bailing out.

5

The merge import will not commit the merged changes. It will only "save" them (visible with uci changes). The exception is when the target configuration is deemed to be outside the default config location.

But even specifying the same /etc/config/ directory in the package name makes uci assume it's outside the default.

2 Likes
6

The "buffer" as I describe it, consists of the files kept in tmpfs as I am well aware. Perhaps for clarity I should use the term "buffer files".

By this I mean the /etc/config file is in sync with the buffer file.

Very much not a theoretical construction, more of an understanding of what happens gleaned from many years of building and manipulating OpenWrt packages.

Yes, but it is more than that, it is the dynamic config data used at run time by uci that can be read and written by packages, as well as users.

No, to be more exact it shows the current changes in the dynamic uci data compared to the static config date stored in /etc/config

This should really be grep -r '.' /tmp/.uci/test as there may well be, and often are, other config files in there, but for this example it does not matter.

Yes, there is no output because /etc/config/test contains nothing and nothing has happened to trigger a sync.

Correct. UCI checked the existence of the (currently empty) /etc/config/test file, then created the dynamic, runtime "buffer file". The static file is still empty at this stage.

The commit writes the buffer file to /etc/config making both files the same (in sync).

No, this is where we disagree. By manually creating the static file, uci does not know it exists, so the buffer file remains empty.

The static file is deleted.
uci show test makes uci check the static file exists, it no longer does so deletes the buffer file if it exists.

No, this means the static file does not exist, so uci has deleted the dynamic buffer-file if it existed.

The uci changes command is equivalent of doing a diff between the dynamic and static files.
(install diffutils if you want to try it).

You are missing the very powerful functionality uci provides. It is not hidden magic, it is just hidden in plain sight. Most people are unaware of this because of the historical use of config files - write a config file and your package reads it - end of story.

I have actually read your evidence. But have you read my explanation? Probably, but as you have not yet seen the "hidden magic", waht I have been saying does not connect.

All OpenWrt packages should use uci (or uci libraries) in their procd scripts and, as required at runtime, to read (or set) config options.

Most packages do these days, but some do not. For a simple stand alone package, technically it does not matter, but for complex packages it can be essential.

Here is an example, the Mesh11sd package.

root@meshnode-8ecb:~# cat /etc/config/mesh11sd

config mesh11sd 'setup'
	option debuglevel '3'
	option auto_config '1'
	option portal_detect '4'

config mesh11sd 'mesh_params'

root@meshnode-8ecb:~#

The dynamic configuration:

root@meshnode-8ecb:~# uci export mesh11sd
package mesh11sd

config mesh11sd 'setup'
	option debuglevel '3'
	option auto_config '1'
	option portal_detect '4'

config mesh11sd 'mesh_params'
	option mesh_fwding '1'
	option mesh_retry_timeout '255'
	option mesh_confirm_timeout '255'
	option mesh_holding_timeout '255'
	option mesh_rssi_threshold '-68'
	option mesh_gate_announcements '1'
	option mesh_hwmp_rootmode '4'
	option mesh_hwmp_root_interval '5000'
	option mesh_hwmp_active_path_to_root_timeout '6000'
	option mesh_max_peer_links '16'
	option mesh_plink_timeout '500'
	option mesh_hwmp_rann_interval '1953'
	option mesh_hwmp_preq_min_interval '586'
	option mesh_hwmp_active_path_timeout '1465'
	option mesh_hwmp_max_preq_retries '4'
	option mesh_connected_to_gate '1'

root@meshnode-8ecb:~#

All of these options can, and often do, get changed dynamically by the mesh daemon.
If the service daemon is stopped, all the dynamic changes are reverted using uci revert mesh11sd

The daemon also makes dynamic changes to network, wireless, dhcp, firewall and system as part of what it does.
This is all made simple by utilising the full power of uci.

There are no fictional constructs here. It is real and in use.

These are not made up ideas, they are what happens in reality.

Your examples and "evidence", in the light of missing knowledge of how uci works, has drawn you to a conclusion that is only a subset of how it works.

7

So where is this "dynamic data" located? Can you actually demonstrate its existence in any way?

What does it mean "uci does not know"? Where is uci's knowledge stored? I just demonstrated that uci is perfectly happy to print out a completely new file, manually created by me. Any normal uci commands, when applied to that file, produce exact same results as with any other config file. If some "knowledge" was required, wouldn't UCI throw an error or misbehave in some way? And if there is no error and no misbehavior then what leads you to think that any such knowledge exists? I understand that you have your subjective experience with uci but do you actually believe that this experience is good enough to make claims about a thing in a physical world? And you expect me to accept those claims without any evidence, based on your subjective experience?

Again, what do you actually mean by "in sync"? Do you believe that UCI keeps a copy of config files somewhere and that copy needs to exactly match the "static" config file, sans the changes? If so then where is that copy stored? If no then why would you call this "sync"?

8

Here it is in the code:

You see these 3 paths? This is where UCI's "knowledge" is stored. If you think that there is something else somewhere else then how about providing actual evidence?

9

I will explain it again.
If you create a file in /etc/config, it is just a file, uci does not know anything about it until it is told.

If you run uci show uci export uci get, specifying the config name, YOU trigger the process I have detailed previously. Here it is again:

  1. A static (written to flash) config file is created - in your example you create it manually. Usually though it is installed/created by a package when the package itself is installed or first run.
  2. So far the UCI utility is unaware of it. You do for exampleuci show testconfig or the package procd script does uci get testconfig.test . Now, uci checks if the static config file (/etc/config/testconfig) and the dynamic config file (in what I call the buffer file - located in /tmp/.uci) exist.
  3. If the static file exists but the dynamic file does not exist, the static is copied to create the initial version of the dynamic.
  4. Now the static and dynamic files are in sync ie there are no differences.
  5. From now on, uci will only read or write the dynamic file, except for when a commit is requested with uci commit testconfig
  6. If you, or the package or for that matter any other uci aware package does eg uci set testconfig.test='test_option', this will be written ONLY to the dynamic file.
  7. If you now do uci get testconfig.test, you will see the value "test_option" returned.
  8. BUT if you do cat /etc/config/testconfig nothing (or the original value) will be present.
  9. If you do uci changes testconfig, you will see only the value you just set with the uci set command. ie uci is showing the differences between the two files.
  10. If you then do uci commit testconfig, the dynamic file will overwrite the static file in flash.

UCI remembers what it is told about (by writing to files).
These paths have nothing to do with the actual config data.
They are the default locations of where the config data is stored.
They can be changed at any time on the uci command line:

Options:
	-c <path>  set the search path for config files (default: /etc/config)
	-C <path>  set the search path for config override files (default: /var/run/uci)
	-d <str>   set the delimiter for list values in uci show
	-f <file>  use <file> as input instead of stdin
	-m         when importing, merge data into an existing package
	-n         name unnamed sections on export (default)
	-N         don't name unnamed sections
	-p <path>  add a search path for config change files
	-P <path>  add a search path for config change files and use as default
	-t <path>  set save path for config change files
	-q         quiet mode (don't print error messages)
	-s         force strict mode (stop on parser errors, default)
	-S         disable strict mode
	-X         do not use extended syntax on 'show'

Why do you think that I think there is something else stored somewhere else?

In short, uci or its library scripts, use the dynamic files at runtime, for both reading and writing.
If the requirement is to make any changes non-volatile (ie able to survive a reboot), then the dynamic file(s) must be committed (eg with uci commit).

Far from a fictional construct, this is actual reality.
It works this way as intended.
ACTUAL packages use it in this way.

Your examples give correct results, BUT your conclusions are incorrect as you have not been aware of the full functionality.

If you start with the classic mindset of "just read the config file, nothing else is needed", then you are missing entirely the purpose of UCI.

To you uci -m import seemed broken because the result you saw did not match your perception of how it works.

1 Like
10

How is this different from any other config file? Again, you believe in some "awareness" but UCI doesn't have brain cells. It only has data. So where is this "awareness" stored? If you can not show the data then it doesn't exist (and if you think that it's stored in the RAM then no - UCI only runs on-demand, it doesn't stay in RAM - just run ps | grep uci and see by yourself).

Then what's the problem to show where is the copied file?

Please don't go into mindsets and perceptions. I'm only interested in facts. Not beliefs, not "from experience". Simple: you make a claim that the file is copied - show me where the copy is. If you can not show the copy then the logical thing to do is drop the claim, not repeat it over and over.

11

This isn’t supported by facts. uci reads the file from /etc/config/ when requested and merges in any uncommitted changes from /tmp/.uci. Otherwise, the /tmp/.uci/ is empty or has 0 byte files if all earlier changes are committed.

I’m not sure what you guys are debating anymore.

1 Like
12

In condensed form, I claim that the import command's behavior is inconsistent because uci import <config_name> writes to file immediately while uci -m import <config_name> doesn't. @bluewavenet claims (as far as I understand) that uci import <config_name> behaves in this way because I created the config file in /etc/config manually. So I'm to blame because I have the 'wrong mindset'. This claim is apparently not supported by any evidence but @bluewavenet is perfectly happy to double- and triple-down on it anyway. That's, in short, the debate.

13

Where did I say that?

In the data.

I have shown you where it is stored, you chose to ignore what I said, then repeated what I said in your own words.

No, but it keeps its data in volatile RAM, in the form of tmpfs storage.

It is copied to where I said it is copied to, the same place you said it is kept.

Me too. I have never said "i believe it is like this" or implied it.

It reads both.
It writes to just /tmp/.uci when doing a uci set or uci add_list
It overwrites /etc/config when a commit is done, setting thr /tmp/.uci file to empty.

We are debating "uci -m import is broken?"
In my opinion, supported by use over many years, is that it is not broken.

14

The only actual path you mentioned was /tmp/.uci. I have already shown you (via multiple grep '.' /tmp/.uci commands run under different conditions) that no copied data is ever written there, it's only the changes. So let's make things clear: are you claiming that this is where uci copies files to? If so then please give an example uci command which creates this copied file, and an output of cat <copied_file>. This should be very easy. If no then how about you retract the claim that uci copies data anywhere, and, as a direct consequence, you acknowledge that uci does not do any "syncing"?

15

I have double and triple presented the evidence that supports the fact that -m import is not broken.
The problem is that you refuse to accept the evidence either because you have not read it, or you do not understand it, or it does not match how you "believe" it works.

It is most likely the last one - supported by your assertion that it it is all a fictional construct.

16

Without finding any history or commit comments to back this up, it seems reasonable for uci to not commit merge imports in case the merge creates unexpected results. It can be easily reverted.

17

I agree, this does seem reasonable. If this was documented, I would have no issue with this behavior. This thread was started because no such documentation exists, and I just assumed that the second import command behaves similarly to the first one, and when this didn't work, I thought the command was broken.

18

That is how I have always found that it works and this is indeed consistent with only commit actually writing to the /etc/config file.

But if the /etc/config file does not exist then uci import (without the merge) does create the file.

I will add this to the wiki if we are agreed.

19

I don't know what we agreed on but this debate was too much for me, so whatever :slight_smile: