Need to copy many files yes skip existing ones - CP not working as advertised

I have 2 dissimilar USB drives attached to my OpenWRT router. (Plenty of RAM so this is not an issue). I started with the CP command with -r and -u options but the copy updates all files when the date is newer (today's date is being used instead of original file date), so that is effectively copying every file from one drive to the other without exception.

I downloaded the package RSYNC and started it, it ran a couple of hours (have about 2TB of files to copy) and now every time I start the rsync command, my OpenWRT system crashes and I don't have any indication of the error.
I am using ... rsync -rv --ignore-existing /mnt/sda2/source/ /mnt/sdc2/destination
Now this exact rsync command ran for 2 hours and eventually took the openwrt server down, now it crashes the server every time I start it.
Is there another copy option that will copy a mass of files and will work by only copying a file if it does not exist on the destination? (Both RSYNC and CP options seem different that other distributions) Thanks!

  • Avoid using non-native Linux file systems as much as possible.
  • Use a separate power cable for each drive if possible and check SMART to make sure the drives are not faulty.
  • Stop Adblock, VPN, Tor, Torrent, DNS encryption and other CPU, RAM, I/O resource-demanding services while running rsync.
  • If nothing helps, then probably your router is not suited for the task.

See also:

3 Likes

I don't know excatly what you are trying to do. But it sounds like you don't want to use "-u" but "-n" (no-clobber"). From the manpage:

-n, --no-clobber: do not overwrite an existing file

A lot of the switches on OpenWrt appear to be different than other Linux distributions. (Same is true for rsync). In this case, here is what I receive for "cp":

OpenWrt 19.07.2, r10947-65030d81f3
 -----------------------------------------------------
root@OpenWrt:~# cp
BusyBox v1.30.1 () multi-call binary.

Usage: cp [OPTIONS] SOURCE... DEST

Copy SOURCE(s) to DEST

        -a      Same as -dpR
        -R,-r   Recurse
        -d,-P   Preserve symlinks (default if -R)
        -L      Follow all symlinks
        -H      Follow symlinks on command line
        -p      Preserve file attributes if possible
        -f      Overwrite
        -i      Prompt before overwrite
        -l,-s   Create (sym)links
        -T      Treat DEST as a normal file
        -u      Copy only newer files
root@OpenWrt:~#
 OpenWrt 19.07.2, r10947-65030d81f3
 -----------------------------------------------------
root@OpenWrt:~# rsync
rsync  version 3.1.3  protocol version 31
Copyright (C) 1996-2018 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    no socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, no ACLs, no xattrs, iconv, symtimes, prealloc

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

rsync is a file transfer program capable of efficient remote update
via a fast differencing algorithm.

Usage: rsync [OPTION]... SRC [SRC]... DEST
  or   rsync [OPTION]... SRC [SRC]... [USER@]HOST:DEST
  or   rsync [OPTION]... SRC [SRC]... [USER@]HOST::DEST
  or   rsync [OPTION]... SRC [SRC]... rsync://[USER@]HOST[:PORT]/DEST
  or   rsync [OPTION]... [USER@]HOST:SRC [DEST]
  or   rsync [OPTION]... [USER@]HOST::SRC [DEST]
  or   rsync [OPTION]... rsync://[USER@]HOST[:PORT]/SRC [DEST]
The ':' usages connect via remote shell, while '::' & 'rsync://' usages connect
to an rsync daemon, and require SRC or DEST to start with a module name.

Options
 -v, --verbose               increase verbosity
     --info=FLAGS            fine-grained informational verbosity
     --debug=FLAGS           fine-grained debug verbosity
     --msgs2stderr           special output handling for debugging
 -q, --quiet                 suppress non-error messages
     --no-motd               suppress daemon-mode MOTD (see manpage caveat)
 -c, --checksum              skip based on checksum, not mod-time & size
 -a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
     --no-OPTION             turn off an implied OPTION (e.g. --no-D)
 -r, --recursive             recurse into directories
 -R, --relative              use relative path names
     --no-implied-dirs       don't send implied dirs with --relative
 -b, --backup                make backups (see --suffix & --backup-dir)
     --backup-dir=DIR        make backups into hierarchy based in DIR
     --suffix=SUFFIX         set backup suffix (default ~ w/o --backup-dir)
 -u, --update                skip files that are newer on the receiver
     --inplace               update destination files in-place (SEE MAN PAGE)
     --append                append data onto shorter files
     --append-verify         like --append, but with old data in file checksum
 -d, --dirs                  transfer directories without recursing
 -l, --links                 copy symlinks as symlinks
 -L, --copy-links            transform symlink into referent file/dir
     --copy-unsafe-links     only "unsafe" symlinks are transformed
     --safe-links            ignore symlinks that point outside the source tree
     --munge-links           munge symlinks to make them safer (but unusable)
 -k, --copy-dirlinks         transform symlink to a dir into referent dir
 -K, --keep-dirlinks         treat symlinked dir on receiver as dir
 -H, --hard-links            preserve hard links
 -p, --perms                 preserve permissions
 -E, --executability         preserve the file's executability
     --chmod=CHMOD           affect file and/or directory permissions
 -o, --owner                 preserve owner (super-user only)
 -g, --group                 preserve group
     --devices               preserve device files (super-user only)
     --specials              preserve special files
 -D                          same as --devices --specials
 -t, --times                 preserve modification times
 -O, --omit-dir-times        omit directories from --times
 -J, --omit-link-times       omit symlinks from --times
     --super                 receiver attempts super-user activities
 -S, --sparse                turn sequences of nulls into sparse blocks
     --preallocate           allocate dest files before writing them
 -n, --dry-run               perform a trial run with no changes made
 -W, --whole-file            copy files whole (without delta-xfer algorithm)
     --checksum-choice=STR   choose the checksum algorithms
 -x, --one-file-system       don't cross filesystem boundaries
 -B, --block-size=SIZE       force a fixed checksum block-size
 -e, --rsh=COMMAND           specify the remote shell to use
     --rsync-path=PROGRAM    specify the rsync to run on the remote machine
     --existing              skip creating new files on receiver
     --ignore-existing       skip updating files that already exist on receiver
     --remove-source-files   sender removes synchronized files (non-dirs)
     --del                   an alias for --delete-during
     --delete                delete extraneous files from destination dirs
     --delete-before         receiver deletes before transfer, not during
     --delete-during         receiver deletes during the transfer
     --delete-delay          find deletions during, delete after
     --delete-after          receiver deletes after transfer, not during
     --delete-excluded       also delete excluded files from destination dirs
     --ignore-missing-args   ignore missing source args without error
     --delete-missing-args   delete missing source args from destination
     --ignore-errors         delete even if there are I/O errors
     --force                 force deletion of directories even if not empty
     --max-delete=NUM        don't delete more than NUM files
     --max-size=SIZE         don't transfer any file larger than SIZE
     --min-size=SIZE         don't transfer any file smaller than SIZE
     --partial               keep partially transferred files
     --partial-dir=DIR       put a partially transferred file into DIR
     --delay-updates         put all updated files into place at transfer's end
 -m, --prune-empty-dirs      prune empty directory chains from the file-list
     --numeric-ids           don't map uid/gid values by user/group name
     --usermap=STRING        custom username mapping
     --groupmap=STRING       custom groupname mapping
     --chown=USER:GROUP      simple username/groupname mapping
     --timeout=SECONDS       set I/O timeout in seconds
     --contimeout=SECONDS    set daemon connection timeout in seconds
 -I, --ignore-times          don't skip files that match in size and mod-time
 -M, --remote-option=OPTION  send OPTION to the remote side only
     --size-only             skip files that match in size
 -@, --modify-window=NUM     set the accuracy for mod-time comparisons
 -T, --temp-dir=DIR          create temporary files in directory DIR
 -y, --fuzzy                 find similar file for basis if no dest file
     --compare-dest=DIR      also compare destination files relative to DIR
     --copy-dest=DIR         ... and include copies of unchanged files
     --link-dest=DIR         hardlink to files in DIR when unchanged
 -z, --compress              compress file data during the transfer
     --compress-level=NUM    explicitly set compression level
     --skip-compress=LIST    skip compressing files with a suffix in LIST
 -C, --cvs-exclude           auto-ignore files the same way CVS does
 -f, --filter=RULE           add a file-filtering RULE
 -F                          same as --filter='dir-merge /.rsync-filter'
                             repeated: --filter='- .rsync-filter'
     --exclude=PATTERN       exclude files matching PATTERN
     --exclude-from=FILE     read exclude patterns from FILE
     --include=PATTERN       don't exclude files matching PATTERN
     --include-from=FILE     read include patterns from FILE
     --files-from=FILE       read list of source-file names from FILE
 -0, --from0                 all *-from/filter files are delimited by 0s
 -s, --protect-args          no space-splitting; only wildcard special-chars
     --address=ADDRESS       bind address for outgoing socket to daemon
     --port=PORT             specify double-colon alternate port number
     --sockopts=OPTIONS      specify custom TCP options
     --blocking-io           use blocking I/O for the remote shell
     --stats                 give some file-transfer stats
 -8, --8-bit-output          leave high-bit chars unescaped in output
 -h, --human-readable        output numbers in a human-readable format
     --progress              show progress during transfer
 -P                          same as --partial --progress
 -i, --itemize-changes       output a change-summary for all updates
     --out-format=FORMAT     output updates using the specified FORMAT
     --log-file=FILE         log what we're doing to the specified FILE
     --log-file-format=FMT   log updates using the specified FMT
     --password-file=FILE    read daemon-access password from FILE
     --list-only             list the files instead of copying them
     --bwlimit=RATE          limit socket I/O bandwidth
     --outbuf=N|L|B          set output buffering to None, Line, or Block
     --write-batch=FILE      write a batched update to FILE
     --only-write-batch=FILE like --write-batch but w/o updating destination
     --read-batch=FILE       read a batched update from FILE
     --protocol=NUM          force an older protocol version to be used
     --iconv=CONVERT_SPEC    request charset conversion of filenames
     --checksum-seed=NUM     set block/file checksum seed (advanced)
 -4, --ipv4                  prefer IPv4
 -6, --ipv6                  prefer IPv6
     --version               print version number
(-h) --help                  show this help (-h is --help only if used alone)

Use "rsync --daemon --help" to see the daemon-mode command-line options.
Please see the rsync(1) and rsyncd.conf(5) man pages for full documentation.
See http://rsync.samba.org/ for updates, bug reports, and answers
rsync error: syntax or usage error (code 1) at main.c(1562) [client=3.1.3]
root@OpenWrt:~#

Yes that is pretty normal for embedded systems with limited ressources like ram, rom and cpu. Its use case is limited. Especially if you use it wrongly. Your router is not a NAS with a powerfull PSU for Harddrives and a CPU/RAM for heavy I/O usage. Beside x86 versions and racks. But that will not mean they would rewrite OpenWrt with regular binaries for such a usecase.

I don't know what all goes into building this distribution, other than a lot. I just installed "linux file system for windows" which was able to provide a port for my drive I use on the Espressobin running OpenWRT. While it does make sense that systems like the Espressobin with a full 1 Gb of flash would seem to be able to perform mass copies, it does make even more sense that the scaled down capabilities of the various commands provided in the typical 32M framework of traditional routers via OpenWRT could not be as robust as from a larger system.

My problem is that OpenWRT was my introduction to Linux and I haven't yet gotten beyond the basics nor am I yet brave enough to move to something more robust to serve as the NAS base you speak of (would also need FTP and Transmission on the distribution I would switch to). I guess what I need really for tasks like copying from dissimilar FS based drives is a Linux desktop that has full support for every kind of Windows drive. This would be better than trying to get by with some make-shift windows app that provides ext4 file support, which is what I was finally successful using along with the Windows app "FastCopy" with the "no overwrite" option to copy the 2Tb drive.

If you have any suggestion what I might use instead of OpenWRT for my NAS and with that include FTP and Transmission, and either wile using the Espressobin hardware or even a Pi 4, or something with low power drain and no need for a fan, and also support spin-down as my drives are not used constantly, then I'm open for suggestions for trying to migrate this direction in the future for my NAS project. I love OpenWRT but the performance of Samba4 is not that great I am seeing.

DS118

everything but fanless, no config hassles either

2 Likes

That is the most important point. Even a RPi =< v4 is very limited in this scenario. With a RPi4 you could think about to run "one" harddrive as NAS. But not for heavy usage. It might fit your needs. Research the internet for benchmarks.

With those NAS stations like @anon50098793 recommended you can fullfill all your tasks you mentioned above. Even an older used device for low budget is better than your router when it comes to application support compared to OpenWrt. Synology and Qnap are the big players in this business. They have their own App shops for almost any usecase you can imagine. Beside that you can use open source apps ofc.

2 Likes

Especially with NAS purposes in mind, a low-end[0] Atom Board with 2-6 SATA ports easily pays off, actually the most expensive part of this would be a decent case. It's no coincidence that many modern NAS devices are x86_64 based.

--
[0] obviously the sky is the limit, you can reach higher if you need/ want to.

2 Likes