How to transfer a part of a file from remote system via SCP?

I am trying to fetch a log file which is repeatedly updated. I want that updated file, but don’t want to download the entire file again and again from the remote system via SCP.

Any idea to achieve this in minimal network usage?

Here is Solutions:

We have many solutions to this problem, But we recommend you to use the first solution because it is tested & true solution that will 100% work for you.

Solution 1

With SCP alone, there isn’t really any solution. The protocol only allows for full file copies, not partial transfers.

If you’re willing to look beyond just SCP, there are a couple of options:

  1. Use a command like the following to generate a file containing just the new lines of the log file and transfer that:

    ssh [email protected] tail -n +`wc -l /local/file.log` /remote/file.log >> /local/file.log
    

    That will count the lines in the local file, and tell the remote system to print all lines after that line number over the SSH connection, which will then be appended to the local file by shell redirection. This method may be a bit finicky in some shells, and is slightly racy (a write arriving at the file while it’s being copied may end up only partially transferred, but won’t get completely transferred by the next invocation).

  2. Use rsync with the --checksum option. This enables incremental transfers of data, and because of the rolling-hash checksumming it uses should transfer an absolute minimum of data. Unlike the first method, this isn’t racy at all, but it also requires rsync being present on both ends.

  3. Set up a (secured) web server that supports byte-range requests. You can then use an HTTP HEAD request to get the new file size (optionally with an If-Modified-Since header to turn it into a no-op if the file hasn’t changed), and request the range extending from where your copy ends to the total length reported by the HEAD request. This has the most setup involved, and can transfer partial lines, but will guarantee you can’t lose any data.

  4. Figure out a way to set up remote logging for the application generating the log file, and set up your local system as a log host. This only works of course if your local system is on all the time, but it also guarantees that you have the most up-to-date version of the log possible without you needing to do anything by hand to update it.

Note: Use and implement solution 1 because this method fully tested our system.
Thank you 🙂

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0

Leave a Reply