About the difference in folder size as a result of copying folders using rsync

Asked 2 years ago, Updated 2 years ago, 117 views

When I copied the entire folder to another server using rsync, there was a difference in size and I am troubled.If anyone knows how to solve this problem, I would like to ask you to teach me a question.

Let me explain a little bit about the situation.

I am currently using two servers (assuming A and B).
Both operating systems are set to ubuntu 16.04.

I would like to copy the folder on server A to server B.
The folder contains several files as follows:

Nov132018
├-- test.AQ863.1.1.h264
├-- test.AQ863.1.2.h264
├-- test.AQ863.1.3.h264
├-- test.AQ863.1.4.h264
└-- test.AQ863.1.5.h264

We copied this entire folder to Server B using the following command:

rsync-av-essh/media/user1/1C3C-F8AC/Nov132018amori@ipaddress:/tank/user1/tmpNAS/NAS1/movies

I checked the folder transferred on server B side and found that all files, including all folders, were transferred.

However, the problem was that the size of the folder was 20 times larger.

Specifically,
On the server A side,

$ls-lh Nov132018/
total 504M
-rw -r --r --1 user 1 user 1504M Nov 1304:16 test.AQ863.1.1.h264
-rw-r--r--1 user1 user10 Nov 1304:26 test.AQ863.1.2.h264
-rw-r--r--1 user1 user10 Nov 1304:36 test.AQ863.1.3.h264
-rw -r --r --1 user 1 user 10 Nov 1304:46 test.AQ863.1.4.h264
-rw-r--r--1 user1 user10 Nov 1304:56 test.AQ863.1.5.h264

But

On the server B side,

$ls-lh Nov132018/
total 10G
-rwxr-xr-x1 amori 504M Nov 1304:16 test.AQ863.1.1.h264
-rwxr -x1 amori amori 0 Nov 1304:26 test.AQ863.1.2.h264
-rwxr -x1 amori amori 0 Nov 1304:36 test.AQ863.1.3.h264
-rwxr-x1 amori amori 0 Nov 1304:46 test.AQ863.1.4.h264
-rwxr-x1 amori amori 0 Nov 1304:56 test.AQ863.1.5.h264

That was it.

The size of the transferred file is correct, but the folder size is very different from 500M and 10G.

I don't know why this kind of problem is occurring.
I would like to keep it to 500M on server B side.
If anyone knows how to do that, please let me know.

Now that I know the cause, I will fill in a little more information.

It seems that the cause is the block size.
After using the df command,
Server A has a block size (1K-blocks) of 60049056
Server B's block size (1K-blocks) is 10978791680
That was it.

Server B also has other data, so I don't know what will happen if I change the block size, so I stopped here.If you have any advice, please let me know.

linux ubuntu rsync

2022-09-30 17:43

2 Answers

Maybe it's because you copied the sparse file. Try copying it again by giving --sparse option to rsync.

Delete the destination file before you try again, as rsync does not see the sparse state difference in synchronization (re-copy) determination.


2022-09-30 17:43

I feel that what was hard-linked on server A is a different file on server B.
If so, I think it would be better to add an additional "-H" option to rsync.

rsync-avH-esh (omitted) 


2022-09-30 17:43

If you have any answers or tips


© 2024 OneMinuteCode. All rights reserved.