NSYNC still not working - 3 months

[phildog33]

Been trying to sync two N7700s together for 3 months now... Tried rsync, couldnt figure it out.
Tried NSYNC ran into problems with their 2TB file limitation

Broke up my main N7700 (source) into Many folders which are less than 2TB... one folder is only 100gb but has lots of system files, maybe 150,000 small files...

This folder also wont sync...

I dont understand... Is there anyway to sync a heavily used N7700 to another one???

Please help.. I was so close to getting rsync to work, with the modules, script, editing it.. etc... But couldnt initiliaze it in crontab

Phillip

[Nimmnenkeks]

Hi,

same Probs at my both 7700.
Have done the setting for nsync to "another FTP server" and in the moment I try to sync 3,9TB from one Folder to my second 7700.

In the moment it has copied 1,8TB (all my BD's -> not ripped) and 30% of my DVD's.

The Settigs were:
- FTP Service on both 7700 -> enabled
- nsync on Target-Server -> enabled
- nsync on Source new Task (same like the tasks for "normal nsync", but don't activate Thecus NAS, take "another"
- IP for the target-server
- Login & Pass
- Test it

an it seems it works

Tomorrow i will post my results

regards

[phildog33]

thank you so much for this reply.. im trying it now with LEGACY server instead of THECUS selected, on the nsync task.. ill report back if it works

ps... is there still a 2tb file size limitation with this though...

if my folder is 3.7TB will it NSYNC task the WHOLE folder or stall again... because it has to zip and send over that unzipped file...

maybe using FTP its file by file in the folder, instead of the whole folder
p

[Nimmnenkeks]

Hi Phillip,

please search for "Thecus_N4100_Nsync_Quick_Guide_2008-08-18.pdf" if it wont work.

Sorry for my bad english

My task shows at the moment 276322 of 323368 in progress ( my folder Media) with BD/DVD/MP3/Photo.

The log-Folder on the nsync-target hat now no entry, I think a log-file will be written after the job.

After the job is done I'll test the nsync-data (ex. the BD-files -open them with Arcsoft from my VMC).

regards

Thorsten

[Nimmnenkeks]

Yeah,

it's done.
The only difference between the two folder (source & target), were the old temp-files from the task: Nsyync to a Thecus NAS.

These files begin with a .pxxxxxxxxxxx -> delete them and both N7700 were in sync.

Today I try another folder an try the Backup-Function

regards

[phildog33]

this is working PERFECTLY now.. thank you so much. Ive unchecked Thecus NAS from nsync, and instead have Legacy Server and its working PERFECTLY

thank you thank you.. nsync option is not working correctly
p

[Nimmnenkeks]

The act. Firmware 2.01.08 is changed for NSync.

Now all files form A would be copied to B

No incremental-function any more

Why ???

This will take about 26h for syncing my 5 TB Media-Folder fron A to B evry time,and it's growing

Is there a module for N7700 to get a nsync-function with incremental-option

regards

[peterfu]

I use rsync to backup my N7700 data to my N5200 - that works perfectly and does a great job - if needed I can write here a little howto.
br
Peter

[Ruediger]

I use rsync to backup my N7700 data to my N5200 - that works perfectly and does a great job - if needed I can write here a little howto.
br
Peter

Hello Peter,
do you use a PC doing the rsync-job, or have you succeeded doing the rsync from Thecus-to-Thecus?

Greetings,
Ruediger

[phildog33]

yes.. if you have done rsync from thecus n7700 to another thecus..like the n5200 pro or another n7700

please post steps , so we can follow them
p

[peterfu]

OK, heres a little HowTo:
You need at least following modules: SSHD, SYSUSER for ssh access and RSYNC for the rsync stuff.
Install and enable SSHD and SYSUSER module. Change immediate the password of the user sys via the module property page of the SYSUSER module.

Install, but don't enable the RSYNC module.
Now select the source of Your data - in my case it is the N7700.
Login via ssh to the N7700 and modify the file /raid/data/module/RSYNC/system/etc/rsyncd.conf (the file is created with default values and all Your shares in it)
Here how the file looks on my N7700 (at least a part of it):

#######################################################################
##
## /raid/data/module/RSYNC/system/etc/rsyncd.conf - RSYNCD config file
##
#######################################################################

gid = users
read only = true
transfer logging = false
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
hosts allow = 192.168.7.6


[Public]
comment = Public Files
path = /raid/data/Public
read only = no
list = yes
uid = sys
gid = nogroup

The only thing I have modified is the hosts allow to the IP adress of my N5200 - the target.

Now enable the RSYNC module on Your source - in my case the N7700.

Logout from the source and log on via ssh to the target.
If You wish to have a exact copy of Your source on the target, then You have to create all top level folders which You want to sync via the Web Interface - I have decided to sync all source folders to a folder called NAS_BACKUP.

It's time for a first test now. With following command I sync my Public folder from the source N7700 to the target N5200

/raid/data/module/RSYNC/system/bin/rsync -a --delete --stats 192.168.7.5::Public /raid/data/NAS_BACKUP/Public/

The IP adress in the above commandline is the IP adress of my N7700 and I sync the folder Public (which is defined in the rsyncd.conf file of the source N7700) to the top level folder NAS_BACKUP and there into a subfolder Public.

If You test it I would choose a Test folder or a folder with a small number of files.

If it works it's time to enter this in the cron to run it automatically.
For this modify the file /app/cfg/crond.conf with Your entry. My entry looks following:

00 */6 * * * /raid/data/module/RSYNC/system/etc/copy_all.sh >> /raid/data/NAS_BACKUP/rsync.log 2>&1

The cron calls every 6 hours the script copy_all.sh (this is a simple script with all rsync commands and some echo commands) and redirects all outputs to a log file - here a part of the script:

#!/bin/bash
#
# called by cron to backup the N7700 to the N5200 via rsync
echo "start backup via rsync : `date`"
echo "syncing folder : Public"
/raid/data/module/RSYNC/system/bin/rsync -a --delete --stats 192.168.7.5::Public /raid/data/NAS_BACKUP/Public/
echo "syncing folder : Special"
/raid/data/module/RSYNC/system/bin/rsync -a --delete --stats 192.168.7.5::Special /raid/data/NAS_BACKUP/Special/
echo "end backup via rsync : `date`"
echo "+++++++++++++++++++++++++++++++++++++++++++++++++"

For updating the real cron entries do following command:

crontab /app/cfg/crond.conf -u root

The log file looks following:

start backup via rsync : Mon May 25 18:00:02 CEST 2009
syncing folder : Public

Number of files: 15535
Number of files transferred: 0
Total file size: 804895522908 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 505518
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 1073
Total bytes received: 509421

sent 1073 bytes received 509421 bytes 340329.33 bytes/sec
total size is 804895522908 speedup is 1576699.28
syncing folder : Special

Number of files: 57
Number of files transferred: 0
Total file size: 17383696900 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 1494
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 35
Total bytes received: 1556

sent 35 bytes received 1556 bytes 3182.00 bytes/sec
total size is 17383696900 speedup is 10926270.84

end backup via rsync : Mon May 25 18:00:04 CEST 2009
+++++++++++++++++++++++++++++++++++++++++++++++++

Thats it - from now on Your source is synced every 6 hour to the target. Be aware that deleted files on the source are also deleted on the target with the above rsync options.

br
Peter

[Ruediger]

Hello Peter,
thank you for your comprehensive HowTo. Helped a lot.

Ruediger

[Nimmnenkeks]

Hall Peter,

gibt es noch die 2TB-Einschränkung beim aktuellen rsync-Modul?

In der Firmware 2.01.09 (N7700) ist das Problem immer noch nicht gefixt.

Jetzt gibt es zwar die Option "Synchronize" und "Incremental", jedoch wird immer der komplette Inhalt von N7700A nach N7700 B kopiert.

Wieso ist das so schwer "nur die veränderten Daten" zu kopieren und die "nicht veränderten Dateien" zu übergehen?

Mein Media-Ordner braucht bei 4TB jetzt (und seit 2.01.08) 2 Tage.

In der Firmware 2.01.06/7 hat das noch gefunzt, jedoch kein XFS....

Gruß

T.

[peterfu]

Hall Peter,

gibt es noch die 2TB-Einschränkung beim aktuellen rsync-Modul?

In der Firmware 2.01.09 (N7700) ist das Problem immer noch nicht gefixt.

Jetzt gibt es zwar die Option "Synchronize" und "Incremental", jedoch wird immer der komplette Inhalt von N7700A nach N7700 B kopiert.

Wieso ist das so schwer "nur die veränderten Daten" zu kopieren und die "nicht veränderten Dateien" zu übergehen?

Mein Media-Ordner braucht bei 4TB jetzt (und seit 2.01.08) 2 Tage.

In der Firmware 2.01.06/7 hat das noch gefunzt, jedoch kein XFS....

Gruß

T.

There is no 2TB limit in the rsync module - I think You are talking about the build in function nsync. As I don't use nsync I have no idea. If they had changed it I assume that it copies everything the first time You run it and then it is doing the incremental copy the following times. I would also do a setup of the nsync jobs completly new with the new options.
br
Peter

[Nimmnenkeks]

Hello Peter,
thought I've read tere is a 2TB linitation on the rsync-module.

I dont't mean the built in in Thecus.

I've tested 2.01.09 with both options (sync & incremental). They both copies all Files from N7700A to N7700 B, anyway the files were changed or not.

I only want to make a Backup from my 8TB N7700 A without copying the whole Files from Source to Target.

My BluRay ISO's and Folders were up to 44GB, so it takes very long to sync my media by the Thecus built in.

Is your "How to" to use for my problem?

Please PM in german

Thx

br
T.

[peterfu]

Can't PM to You, seems You have disabled it - and Yes You can use my HowTo to sync Your 2 N7700.

German:
Kann Die keine PM schicken, schaut aus als ob Du es deaktiviert hast - und ja, Du kannst mein Howto dazu verwenden Deine 2 7700 zu synchronisieren

br
Peter