How do I cleanup my configs. Should I delete the non "apk-new"-files and then rename the "apk-new"-files by removing "apk-new"?
If I look at System/startup there is only one service named *.apk.new and that is disabled.
You may start by comparing each pair and pick new defaults if they got one. The new file is a reference file because your local file differs. That's the same on every Linux package manger I'm aware of.
I don't understand. Is my approach about right to delete and then rename the other file. After that I would upload it as a restored firmware.
If you delete your file and rename the new file.... You then loose all your options set....
Again. Compare the content and then decide what to do. The new file is the default file from the package. And the package manager sees the local file and the package file differ so your file stays untouched. The user needs to decide what to do.
So you are saying that [non "apk-new"-files] are the ones in use, I thought it is the other way around.
No. Your config is in use. Try this - you may need to DL
diffutils:
cd /etc/config
diff -y dhcp.apk-new dhcp
this will list the two files side by side. Vertical bars | denote your options that differ from the new files compared to your file, while right pointer > shows additions.
What?
I only want to get ride of the extra files, I don't want to change anything else or in them. I don't need to compare them, unless they are loaded and used both at the same time.
Only your config file is parsed. The New files as explained by @_bernd are reference files. They potentially indicate changes to the configuration changes to the package.
Here’s an example: I used a package a couple of years ago that used:
list Processes 'uhttpd'
list Processes ‘dnsmasq’
The current config now relies on: option 'Processes uhttpd dnsmasq'
Where is it?
Based on your screenshot:
/etc/config/dhcp
/etc/config/luci
are your files that your current setup is using/parsing and /etc/config/dhcp.apk-new and /etc/config/luci.apk-new are the two reference files that correspond.
Ok, thank you both. Took me some time to get it. 
New and old file wasn't that clear because that differed from date. Not from name though, now I get how this was meant by bernd.
Again. If you are not yet familiar with how Linux package manager works since the mid 1990ies.... Take your take and read about APK in general and maybe even how dpkg from Debian works. It's more or less the same regarding the handling of files which are provided by a package (and are marked! As config files) and the file on the file system which got modified.
Maybe the OpenWrt docs are not that verbose because the target audience should be aware of such a (basic) fact. But in this spacial case the OpenWrt docs point in the end to the APK upstream documentation.
@efahl weren't you involved with the transition to APK? What do you think, should this be more verbose explained in the wiki or in some Lucy subpage? (If you don't mind I pull you in)
From time to time... Either a config option default behavior changes in the upstream source tree, or maybe just in the configuration provided by the OpenWrt package. But especially in the case when your local files differs you should compare from time to time the settings in both files. Otherwise it could hit you by surprise. Most changes are mentioned in the release files... But as always. Take your time, and take care.
Not because of me, I even didn't read any doc.
Yeah, that seems like a good idea. It should also be noted that apk behaves exactly like opkg did in this regard, you'd get a bunch of *-opkg files (identical content as the *.apk-new files) when you upgrade packages in place.
It should also be noted that you don't get them when you do an ASU upgrade, as that does a fresh install of all packages into the image, which are then overwritten by sysupgrade when it restores your backed up config.