Well done Veeam. <slow clap>

[badass]

I'd just like to say a giant well done to Veeam. I have been singing your praises since Veeam B&R 4.x due to the excellent reliability of your software.
Now with Veeam 6.5 it seems that you finally gave in to the pressure to make your software chase features over reliability. Just like everything else.
So far, despite applying your latest patch with this specific issue listed, backups are not working. I suppose that's a bit much to expect when I am running a software setup that's listed on your compatibility matrix.
I guess I'll just file Veeam in my mind next to average (as in ****e) software.

So incredibly disappointed.

P.S. kick your support department into touch.

[wasabi]

We've had no issues with (patched) 6.5. What issues you got?

[shaithis]

I did find 6.5 build 106 to be bad, 109 is certainly better...but I've had to re-create jobs and revoke licenses a number of times to resolve issues.

Wouldn't be so bad if some of the backup jobs weren't so complex :S

[wasabi]

I did find 6.5 build 106 to be bad, 109 is certainly better...but I've had to re-create jobs and revoke licenses a number of times to resolve issues.

Wouldn't be so bad if some of the backup jobs weren't so complex :S

We've not been on Veeam taht long so went straight from 6.1 to 6.5 version 109. It would be nice if they had some sort of scripting tool where you could import / export configs - the wizard is nice and friendly but with a lot fo jobs to configure with different service credentials, CBT requirements and so on, it gets cumbersome.

[DeludedGuy]

We are on 6.5 and we don't have that many issues on system which is a little complicated with backups of backups running all of the time.

[benmilligan]

I'd just like to say a giant well done to Veeam. I have been singing your praises since Veeam B&R 4.x due to the excellent reliability of your software.
Now with Veeam 6.5 it seems that you finally gave in to the pressure to make your software chase features over reliability. Just like everything else.
So far, despite applying your latest patch with this specific issue listed, backups are not working. I suppose that's a bit much to expect when I am running a software setup that's listed on your compatibility matrix.
I guess I'll just file Veeam in my mind next to average (as in ****e) software.

So incredibly disappointed.

P.S. kick your support department into touch.

I wanted to reach out to you to see if you have a case number to reference? Wanted to see if I can help.

[badass]

We've had no issues with (patched) 6.5. What issues you got?

It's not dismounting the snapshots when in hot add mode and it's getting stuck on removing snapshots in both hot add mode and network mode. Running VCenter 5 and ESXi5
Since the Vcenter is the last remnant on the virtual infrastructure built by the bunch of knuckle dragging idiots that originally set up the virtual infrastructure that's getting replaced to emiminate it as a possible cause.

[wasabi]

It's not dismounting the snapshots when in hot add mode and it's getting stuck on removing snapshots in both hot add mode and network mode. Running VCenter 5 and ESXi5
Since the Vcenter is the last remnant on the virtual infrastructure built by the bunch of knuckle dragging idiots that originally set up the virtual infrastructure that's getting replaced to emiminate it as a possible cause.

Sounds more like a vCenter issue than Veeam on the surface, although the Veeam version change would obviously make that seem 'odd'. Is the removing snapshot command showing up in vSphere client with the Veeam user account credentials? Tried re-adding your vCenter to the Veeam infrastructure with different administrator credentials?

[shaithis]

There is the obvious safe removal of snapshots in the vcenter options page of the job as well.

[badass]

Both your offers of assistance are appreciated but this post was me venting at yet another piece of software that was originally reliable and required minimal supervision to keep it working to one where they chase implementing features before they are ready over reliability.

To troubleshoot, each possible cause is being eliminated one by one.

The snapshots are not being removed because Veeam is not releasing the disks in hot add mode. I know this because I couldn't get the vcenter to remove the snapshots until I had a look at the Veeam proxy and saw it still had the snapshots of the disks mounted. Removing the mounted disks (that are infact snapshots) then allowed me to remove the snapshots in Vsphere. Despite telling this to the support engineer, the only suggestion he could come up with was that after going through the logs, the snapshots could not be removed because they are locked by another process Politely I told the engineer that perhaps that process was the Veeam proxy that had the effing snapshots mounted.

I will get this fixed - I have no doubt about that. What I don't like is the fact that I have wasted so much of my time troubleshooting this when my time would be better spent managing my engineers better and planning and implementing fixes and improvements to the IT infrastructure.

[Moby-Dick]

*Veeam hat on*
I've seen the devs at work , they do a lot of QC , trust me

Back when I was an end user of the product, I've seen cases where the hot-add process wont release the discs correctly, In earlier versions it was a bit of a feature of the API , but I believe quite a few additional measures are taken to improve said process in the current version. However if you have had a job fail without releasing the discs , then you can get into a situation where you get a nasty snapshot chain due to those locked discs. Obviously changing to network mode wouldn't help at this stage as the discs are still hotadded.

Not sure how far your log chain goes back, but I would hazard a guess that there will be a "patient zero" moment that started the problems all off.

As a side note I get inundated with requests to add features , its quite rare I get ones for not adding any

*Veeam hat off*

shai, the safe snapshots removal is a bit of a hangover from the ESX 3.5 days before VMware started using the consolidation helper snapshots. Its not an option I'd advocate on vSphere 4 and above.

[badass]

*Veeam hat on*
I've seen the devs at work , they do a lot of QC , trust me

Back when I was an end user of the product, I've seen cases where the hot-add process wont release the discs correctly, In earlier versions it was a bit of a feature of the API , but I believe quite a few additional measures are taken to improve said process in the current version. However if you have had a job fail without releasing the discs , then you can get into a situation where you get a nasty snapshot chain due to those locked discs. Obviously changing to network mode wouldn't help at this stage as the discs are still hotadded.

Not sure how far your log chain goes back, but I would hazard a guess that there will be a "patient zero" moment that started the problems all off.

As a side note I get inundated with requests to add features , its quite rare I get ones for not adding any

*Veeam hat off*

shai, the safe snapshots removal is a bit of a hangover from the ESX 3.5 days before VMware started using the consolidation helper snapshots. Its not an option I'd advocate on vSphere 4 and above.

Fortunately, having had problems with Vsphere backups in the past, I regularly check the snapshot space usage and clean up any snapshots related to failed backups etc.
I believe we not have reliable backups running, though the root cause is currently elusive. One VM that we don't care about was removed from one job and another job was set to start 10 minutes earlier. This worked fine with Veeam 6.1 with the original schedule. I'll post more info later in case anyone else gets similar problems.

[badass]

I'm not going to jump to conclusions here but it looks like removing an NT4 VM from any backup jobs has stopped breaking all subsequent backup jobs.
Another change has been made to one of the affected VM's today so I'll have to wait till tomorrow to confirm it the problem was resolved by changing the timing of one job or removing the NT4 VM.

[wasabi]

I'm not going to jump to conclusions here but it looks like removing an NT4 VM from any backup jobs has stopped breaking all subsequent backup jobs.
Another change has been made to one of the affected VM's today so I'll have to wait till tomorrow to confirm it the problem was resolved by changing the timing of one job or removing the NT4 VM.

Interesting. Luckily i havn't seen an NT4 box in about 5 years Any idea of the underlying cause? We you using VMware tools to quiesce?

[badass]

Interesting. Luckily i havn't seen an NT4 box in about 5 years Any idea of the underlying cause? We you using VMware tools to quiesce?

Still don't know the root cause. The NT4 VM was shut down at some points and it still had problems. I don't care that much any more as the VM is effectively decommissioned now.
Since the changes Veeam has gone back to its good old near faultless self.

[Moby-Dick]

Glad to hear it !