NSYNC still not working - 3 months

Been trying to sync two N7700s together for 3 months now... Tried rsync, couldnt figure it out.
Tried NSYNC ran into problems with their 2TB file limitation

Broke up my main N7700 (source) into Many folders which are less than 2TB... one folder is only 100gb but has lots of system files, maybe 150,000 small files...

This folder also wont sync...

I dont understand... Is there anyway to sync a heavily used N7700 to another one???

Please help.. I was so close to getting rsync to work, with the modules, script, editing it.. etc... But couldnt initiliaze it in crontab

Phillip

Hi,

same Probs at my both 7700.
Have done the setting for nsync to "another FTP server" and in the moment I try to sync 3,9TB from one Folder to my second 7700.

In the moment it has copied 1,8TB (all my BD's -> not ripped) and 30% of my DVD's.

The Settigs were:
- FTP Service on both 7700 -> enabled
- nsync on Target-Server -> enabled
- nsync on Source new Task (same like the tasks for "normal nsync", but don't activate Thecus NAS, take "another"
- IP for the target-server
- Login & Pass
- Test it

an it seems it works

Tomorrow i will post my results

regards

thank you so much for this reply.. im trying it now with LEGACY server instead of THECUS selected, on the nsync task.. ill report back if it works

ps... is there still a 2tb file size limitation with this though...

if my folder is 3.7TB will it NSYNC task the WHOLE folder or stall again... because it has to zip and send over that unzipped file...

maybe using FTP its file by file in the folder, instead of the whole folder
p

Hi Phillip,

please search for "Thecus_N4100_Nsync_Quick_Guide_2008-08-18.pdf" if it wont work.

Sorry for my bad english

My task shows at the moment 276322 of 323368 in progress ( my folder Media) with BD/DVD/MP3/Photo.

The log-Folder on the nsync-target hat now no entry, I think a log-file will be written after the job.

After the job is done I'll test the nsync-data (ex. the BD-files -open them with Arcsoft from my VMC).

regards

Thorsten

Yeah,

it's done.
The only difference between the two folder (source & target), were the old temp-files from the task: Nsyync to a Thecus NAS.

These files begin with a .pxxxxxxxxxxx -> delete them and both N7700 were in sync.

Today I try another folder an try the Backup-Function

regards

this is working PERFECTLY now.. thank you so much. Ive unchecked Thecus NAS from nsync, and instead have Legacy Server and its working PERFECTLY

thank you thank you.. nsync option is not working correctly
p

The act. Firmware 2.01.08 is changed for NSync.:surprised:

Now all files form A would be copied to B

No incremental-function any more:angst:

Why ???:confused:

This will take about 26h for syncing my 5 TB Media-Folder fron A to B evry time,and it's growing

Is there a module for N7700 to get a nsync-function with incremental-option

regards

I use rsync to backup my N7700 data to my N5200 - that works perfectly and does a great job - if needed I can write here a little howto.
br
Peter

Quote:

---

Originally Posted by peterfu

I use rsync to backup my N7700 data to my N5200 - that works perfectly and does a great job - if needed I can write here a little howto.
br
Peter

---

Hello Peter,
do you use a PC doing the rsync-job, or have you succeeded doing the rsync from Thecus-to-Thecus?

Greetings,
Ruediger

yes.. if you have done rsync from thecus n7700 to another thecus..like the n5200 pro or another n7700

please post steps , so we can follow them
p

OK, heres a little HowTo:
You need at least following modules: SSHD, SYSUSER for ssh access and RSYNC for the rsync stuff.
Install and enable SSHD and SYSUSER module. Change immediate the password of the user sys via the module property page of the SYSUSER module.

Install, but don't enable the RSYNC module.
Now select the source of Your data - in my case it is the N7700.
Login via ssh to the N7700 and modify the file /raid/data/module/RSYNC/system/etc/rsyncd.conf (the file is created with default values and all Your shares in it)
Here how the file looks on my N7700 (at least a part of it):

Quote:

---

#######################################################################
##
## /raid/data/module/RSYNC/system/etc/rsyncd.conf - RSYNCD config file
##
#######################################################################

gid = users
read only = true
transfer logging = false
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
hosts allow = 192.168.7.6


[Public]
comment = Public Files
path = /raid/data/Public
read only = no
list = yes
uid = sys
gid = nogroup

---

The only thing I have modified is the hosts allow to the IP adress of my N5200 - the target.

Now enable the RSYNC module on Your source - in my case the N7700.

Logout from the source and log on via ssh to the target.
If You wish to have a exact copy of Your source on the target, then You have to create all top level folders which You want to sync via the Web Interface - I have decided to sync all source folders to a folder called NAS_BACKUP.

It's time for a first test now. With following command I sync my Public folder from the source N7700 to the target N5200

Quote:

---

/raid/data/module/RSYNC/system/bin/rsync -a --delete --stats 192.168.7.5::Public /raid/data/NAS_BACKUP/Public/

---

The IP adress in the above commandline is the IP adress of my N7700 and I sync the folder Public (which is defined in the rsyncd.conf file of the source N7700) to the top level folder NAS_BACKUP and there into a subfolder Public.

If You test it I would choose a Test folder or a folder with a small number of files.

If it works it's time to enter this in the cron to run it automatically.
For this modify the file /app/cfg/crond.conf with Your entry. My entry looks following:

Quote:

---

00 */6 * * * /raid/data/module/RSYNC/system/etc/copy_all.sh >> /raid/data/NAS_BACKUP/rsync.log 2>&1

---

The cron calls every 6 hours the script copy_all.sh (this is a simple script with all rsync commands and some echo commands) and redirects all outputs to a log file - here a part of the script:

Quote:

---

#!/bin/bash
#
# called by cron to backup the N7700 to the N5200 via rsync
echo "start backup via rsync : `date`"
echo "syncing folder : Public"
/raid/data/module/RSYNC/system/bin/rsync -a --delete --stats 192.168.7.5::Public /raid/data/NAS_BACKUP/Public/
echo "syncing folder : Special"
/raid/data/module/RSYNC/system/bin/rsync -a --delete --stats 192.168.7.5::Special /raid/data/NAS_BACKUP/Special/
echo "end backup via rsync : `date`"
echo "+++++++++++++++++++++++++++++++++++++++++++++++++"

---

For updating the real cron entries do following command:

Quote:

---

crontab /app/cfg/crond.conf -u root

---

The log file looks following:

Quote:

---

start backup via rsync : Mon May 25 18:00:02 CEST 2009
syncing folder : Public

Number of files: 15535
Number of files transferred: 0
Total file size: 804895522908 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 505518
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 1073
Total bytes received: 509421

sent 1073 bytes received 509421 bytes 340329.33 bytes/sec
total size is 804895522908 speedup is 1576699.28
syncing folder : Special

Number of files: 57
Number of files transferred: 0
Total file size: 17383696900 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 1494
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 35
Total bytes received: 1556

sent 35 bytes received 1556 bytes 3182.00 bytes/sec
total size is 17383696900 speedup is 10926270.84

end backup via rsync : Mon May 25 18:00:04 CEST 2009
+++++++++++++++++++++++++++++++++++++++++++++++++

---

Thats it - from now on Your source is synced every 6 hour to the target. Be aware that deleted files on the source are also deleted on the target with the above rsync options.

br
Peter

Hello Peter,
thank you for your comprehensive HowTo. Helped a lot.

Ruediger

Hall Peter,

gibt es noch die 2TB-Einschränkung beim aktuellen rsync-Modul?

In der Firmware 2.01.09 (N7700) ist das Problem immer noch nicht gefixt.

Jetzt gibt es zwar die Option "Synchronize" und "Incremental", jedoch wird immer der komplette Inhalt von N7700A nach N7700 B kopiert.

Wieso ist das so schwer "nur die veränderten Daten" zu kopieren und die "nicht veränderten Dateien" zu übergehen?

Mein Media-Ordner braucht bei 4TB jetzt (und seit 2.01.08) 2 Tage.

In der Firmware 2.01.06/7 hat das noch gefunzt, jedoch kein XFS....

Gruß

T.

Quote:

---

Originally Posted by Nimmnenkeks

Hall Peter,

gibt es noch die 2TB-Einschränkung beim aktuellen rsync-Modul?

In der Firmware 2.01.09 (N7700) ist das Problem immer noch nicht gefixt.

Jetzt gibt es zwar die Option "Synchronize" und "Incremental", jedoch wird immer der komplette Inhalt von N7700A nach N7700 B kopiert.

Wieso ist das so schwer "nur die veränderten Daten" zu kopieren und die "nicht veränderten Dateien" zu übergehen?

Mein Media-Ordner braucht bei 4TB jetzt (und seit 2.01.08) 2 Tage.

In der Firmware 2.01.06/7 hat das noch gefunzt, jedoch kein XFS....

Gruß

T.

---

There is no 2TB limit in the rsync module - I think You are talking about the build in function nsync. As I don't use nsync I have no idea. If they had changed it I assume that it copies everything the first time You run it and then it is doing the incremental copy the following times. I would also do a setup of the nsync jobs completly new with the new options.
br
Peter

Hello Peter,
thought I've read tere is a 2TB linitation on the rsync-module.

I dont't mean the built in in Thecus.

I've tested 2.01.09 with both options (sync & incremental). They both copies all Files from N7700A to N7700 B, anyway the files were changed or not.

I only want to make a Backup from my 8TB N7700 A without copying the whole Files from Source to Target.

My BluRay ISO's and Folders were up to 44GB, so it takes very long to sync my media by the Thecus built in.

Is your "How to" to use for my problem?

Please PM in german

Thx

br
T.

Can't PM to You, seems You have disabled it - and Yes You can use my HowTo to sync Your 2 N7700.

German:
Kann Die keine PM schicken, schaut aus als ob Du es deaktiviert hast - und ja, Du kannst mein Howto dazu verwenden Deine 2 7700 zu synchronisieren

br
Peter

he cant use the pm system until he has a few more posts :)

Thanks Peter!

In German:
Danke Peter!

Das mit PM finde ich auch nicht zum freischalten.

Werde am WE mal Deine Anleitung ausprobieren. Es macht doch keinen Sinn die BD's und DVD's immer und immer wieder erneut zu schreiben (wie es Thecus seit 2.01..08 tut).

Ein schönes Wochenende

Gruß
T.

I have been dealing with Thecus for months now about NSYNC -- incremental, synchronize, etc... They are very slow.. almost 24 hours for one folder for me. And have the 2TB file limit since it syncs my folder into one ZIP file, and then unzip on the other end

Anyways... I have 2 of the N7700.. With lots of data 8TBs.. in Photos, Videos, Documents folders.

I really would like to get R-SYNC to work.. and the tutorial on page one was great. Thank you.

I setup SSH, SYSUSER, RSYNC on both 2x N7700 -- Source and Target.

The only thing.. I dont know how to RUN the command to test it.. I can SSH in and modify the RSYNC.config folder with my proper IP address.. so I am good here

I just cant activate it or run it... Im just learning
Please help if you can

The RSYNC module have to be enabled on the source and also the config file has to be modified on the source.
On the target N7700 - thats the one which holds the backups, theres no need to enable the module, just install it.

Let's assume You have defined on Your source N7700 - thast the one You want to backup a Folder called [Test] in the configfile (don't forget to disabel/enable the module after modifying the onfig file and You want to sync it to a Folder called "Test" on Your target N7700 (the one which holds the backups) the You have to login in Your target N7700 via ssh and issue following command:

Quote:

---

/raid/data/module/RSYNC/system/bin/rsync -a --delete --stats 192.168.7.5::Test /raid/data/Test

---

please replace 192.168.7.5 with the IP adress of Your source N7700.

rsync is always triggered from the target and the source listens for requests.

Hope this helps.

br
Peter

It helps so much Peter thank you. I understand the command, the modules , etc.

The only thing I need help on, is very basic... Where do I issue the command? Is it in the chron tab?

I'm sorry, just this last step,, how to issue the command

p

Oh sorry - first enter it on the commandline - if it works, then put in in cron.
For this modify the file /app/cfg/crond.conf with Your entry.

Quote:

---

00 */6 * * * /raid/data/module/RSYNC/system/bin/rsync -a --delete --stats 192.168.7.5::Test /raid/data/Test >> /raid/data/logfiles/rsync.log

---

After modyfing the above mentioned file You have to run following command:

Quote:

---

crontab /app/cfg/crond.conf -u root

---

to update cron.
From now on Your command runs every 6 hours and writes to a log file named rsync.log in a folder called "logfiles" - this folder should have been created before.

br
Peter

I can modify the crontab and put it back on the Thecus using SSH, I'm good there

but I can't figure out how to RUN the Command..
Do I use SSH again or use the terminal app on my MAV

sorry I'm such a novice, but I'm learning

Running a command means:
log in via ssh to the command line. Put there the command in and press return.
br
Peter

Hi Peter,

I login to the Thecus N7700 using cyberduck.
In my case: 192.168.0.191
when i get in.. I jsut see the list of folders/files... I can get the cron config.. download it.. edit it.. and put it back

but i dont see anywhere to do a command line interface


should i be using the Terminal app in my Mac instead
p

ok its super easy.. just open up terminal and type

ssh sys@192.168.0.191 in my case
then
password

and then the command
/raid/data/module/RSYNC/system/bin/rsync -a --delete --stats 192.168.0.191::DROPBOX /raid/data/STUDIO_BACKUP/

wooo --- i got it
thanks peter.. super easy... ive just never used the commandline to ssh in like this

had one last question peter --- i think im moving along great now , thank you --

I have these big folders

Photography -- 2TBs
Video - 2TBs
Music - 1TB
Documents - 1 TB

i was hoping to stagger their syncing from Source- to - Target
Because they are so big and take so much time to sync new data

I had nysnc setup that Photos did on Monday at 8am, Video folder on Wednesday at 11am, etc... Just so the syncs wouldnt run into eachother

What do you think is it possible?

Or can i just spread out the big master sync (the big BACKUP_FOLDER) once per week

P

just wanted to add -- how much BETTER rsync is than either of the nsync options

after getting rsync to work --- for me, on a 50gB folder of Documents

1. NSYNC -- synchronize --- puts the whole folder each time, no changes -- Hour each time

2. NSYNC -- incremental -- puts only the changes, but takes forever to 'look' at each file -- 30 mins each time

3. RSYNC -- looking only for changes --- once synced up, the next time it ran, took only --- 5 seconds ! amazing

thanks again PETER, you saved us here in the studio
p

Your welcome phildog33 - the first time You run rsync takes a long time cause everything is copied, the next times it's very quick (as You have already recognized), so You can do it more often.
Check also the rsync options - the example I have given makes a real 1:1 copy, so files deleted on the source are also deleted on the target - if You leave the option "--delete" away, then Yor are also protected towards accidently deleted files, but then You have to manually clean up the target box for not needed files - better would be to switch on the recycle option of the source.
br
Peter

Peter


Is ther anyway to sync just once a week like the nsync module .. Or does it have to be in increments of 1-24 hours on the command line

p

Of course, that depends on the entries of the crontab entry - the crontab entry is this one:

Quote:

---

00 */6 * * * /raid/data/module/RSYNC/system/bin/rsync -a --delete --stats 192.168.7.5::Test /raid/data/Test >> /raid/data/logfiles/rsync.log

---

what You have put in the /app/cfg/crond.conf file - for the time definition the first enries are important:

Quote:

---

00 */6 * * *

---

a monthly entry could look like:

Quote:

---

15 14 1 * *

---

which means : every first day of the month at 14:15
a weekly entry could look like:

Quote:

---

5 4 * * 7

---

which means, runs every sunday at 04:05
You have a lot of freedom to define at what time the job runs - see here for a description of the entries:
http://linux.die.net/man/5/crontab
br
Peter

perfecto --- and to stagger those updates...

just put one for


00 */6 * * * /raid/data/module/RSYNC/system/bin/rsync -a --delete --stats 192.168.7.5::Test /raid/data/Test >> /raid/data/logfiles/rsync.log


and the other folder (Test2 folder) for

15 14 1 * * /raid/data/module/RSYNC/system/bin/rsync -a --delete --stats 192.168.7.5::Test2 /raid/data/Test2 >> /raid/data/logfiles/rsync.log

..

In the same cron?

P

You can put as many as You like in the cron file - the cron file is a list of jobs that are executed at the given times.
b
Peter