Veeam Hardened Linux Repository

Just a note- this is a rewrite of my original post. Taking feedback from the wonderful tech community I am lucky to be plugged into- I have added setup which should add to a much more secure solution.

As a Veeam SE who is excited about the upcoming release of Veeam Backup and Replication V11 -  I  decided to take a look at some of the features that I think will truly make a difference to our customers. In recent conversations with customers and prospects alike, the topic of ransomware always comes up. The conversations always include the question of how to build a solution which provides a fail-safe way to provide recovery of their data. No matter what the circumstance. Hardware failure, fire, meteor, or Ransomware attack!

Having been in charge of the IT security for several organizations myself in the past, I know it was a matter of when there is a attempt to breach my networks. Not If. With that I also knew the job of keeping the threat at bay was first priority. And I knew that this takes a security in-depth approach, a multi-layered defense if you will. Along with a solid recovery plan should these defenses get by-passed.

Though, there are several ways to provide this with Veeam. In this post I will focus on an upcoming feature in Veeam B&R V11. Coming to a datacenter near you soon. This feature is simple, taking advantage of our love for Linux.

By utilizing attributes added at the files level, along a new approach in Veeam v11 when setting up repositories on Linux deployments. This solution provides locking at the file level, along an ACL for limited access to backup data, even for the backup admin.  So, in true Veeam fashion running a solution that fits your hardware and that makes sense for your datacenter.  Following Veeam’s truly software defined approach- allowing you you to utilize your hardware and not have to manage another storage platform in your Datacenter!

Just a note. I wrote this post with my good friend and fellow NW technologist JD Wallace. Being a colleague at Veeam, before taking his current position as Principal SE with PURE Storage, JD and I have worked together for some time. As such, we like to keep in touch. Popular topic of discussion- innovation in technology and how it benefits the customers we talk to on a daily basis. It was exactly one of these discussions that lead to this post.

This post will be the first in a series to demonstrate this security in depth strategy. Here, highlighting the Hardened Linux repo with the immutability flag we will discuss.  Check out JD’s Post where he demonstrates how selecting the hardware chosen for your repository deployment can provide an additional  level in the layered security approach.

NOTE- The description on what needs to be setup to support the hardened Repository based ont he above setup is below. At the end, of this post there is a quick and separate walkthrough of the setup in my home lab

- Let’s get into it!

Configure the Linux Server

We'll start by creating a new Linux repository with a data volume. For production use, be sure to configure your Linux repository according to Veeam's best practices. In this lab that was done on a FlashArray connected via iSCSI.
JD has a great write up on setting up an Ubuntu Machine utilizing a ISCSI volume. See previous post for step-by-step instructions on setting up this VM. This feature will work fine on either an EXT4 or XFS formatted FlashArray volume, however since FlashArray natively offers great data efficiency will skip XFS and just stick with EXT4 for this lab.

Linux Machine Setup was as follows

Ubuntu 20.04 VM

4 CPU, 8GB RAM

16GB OS volume

2TB FlashArray volume for backup repo mounted at /mnt/Veeam11Repo formatted with ext4

XATTR is the tool which will allow us to set the immutabile file system attribute (or flag) on the individual files. These attributes are associated permanently with the files and directory - providing additional permissions at the backup file level. We had to install this tool to be used by the Veeam Backup Server later. That was done by running the following command.

  sudo apt install xattr -y

We also set up a directory named /backups on the Data Volume to be utilized as storage for my backup files. And enabled a strong Root password.

Create special VBR user for Repo

One of the key pieces of this immutable repository is the ability in Veeam V11 to setup a hardened Linux Repository. First order of business is to setup a user for the single use of setting up setup of the repository on the Veeam Server. This new method differs from the current Linux Repo setup, as it deploys a persistent transport service on the Linux machine. Once this is completed, there is no further communication with this account, over SSH. Meaning even a compromised Veeam Server, a privileged user, an inside actor if you will, can delete or change data on the repository. So, configure the repository, turn off SSH, and be confident there is no unauthorized  access to your repository. All while maintaining a persistent backup target.

In order to utilkize this feature, we need to set it up so that there is a user account to add the persistent service . I don’t like the idea of adding a user to the sudoers file- as it just adds another user I may forget about, I am going to temporarily enable the Root User.

Enable Root User

sudo passwd root

Then Enter and Retype the new password

Creating a new user account on the Linux VM

During setup a folder named Veeam11Repo was created on the data volume.  And this account will be used by the VBR server to add the persistent agent. This way we have a specific account that need only be used temporarily, to setup the Linux Repository. Think of it as a service account that need not be shared.

Here we create a new user and set a password. This account will be utilized once to add the transport service, so go ahead and make it complicated.

   sudo useradd -m linuxrepo

   sudo passwd linuxrepo

cat /etc/passwd | grep linuxrepo

Now change repository  ownership to our new user account. This is in the directory created previously named, backups.

   sudo chown -R linuxrepo:linuxrepo /backups

sudo chmod 700 /backups

Now confirm the settings.

    ls -alh /backups

Create Veeam Linux Repo

Now that our linux server is configured let’s head over to the VBR console to add it as a new Immutable Linux Repo. Under Backup Repositories- Add Attached Disk - Linux.

Add New Server from the next window

Adding account- Here we will use the new feature- Hardened credentials. This will provide access One-Time only. The account will not be stored passed setup.

Add the username and password setup on the Linux Machine. Now as I enabled the Root password on the Linux Machine I use “Elevate account privledges” and insert the Root passwd below.

Hit Apply, Trust the Certificate- and wait for the status below.

Now we have the service installed on the repo. Let’s complete the setup. The Linux server will now show up in the Repository List. Now click Populate from the right. And select the directory we set up during the configuration. In this example it is /mnt/Veeam11 Repo directory. Then click Next

Next configure the repository to utilize what we have configured on the Linux Machine. In this example I will utilize both the XFS file system, and the ability to apply the immutability flag to all backup files written by Veeam. Click next.

On the next menu we need to select the mount server and the Cache for instant VM recovery. The choice here will need to be assigned to a Windows server. In this example I utilize the Veeam server itself. This may differ in your production environment. Couple of examples- remote repositories, where you don’t want mount points across the WAN. My favorite design, Environments where the Veeam Server is just a management server and does not run multiple roles, such as proxy or repo.

Select Next, and Next again. Then look for the following success screen. After this is compelte Click Next and then finish.

That’s it! You are ready to setup a backup job to your newly minted, hardened and Immutable Repository.

File Immutability Test

Fast-forward. I have a backup job configured on the new Linux repository, which has been completed. I now go into the properties for the backup under disks. Where, I have the ability to Delete Backup from Disks. Choosing this and then clicking yes- I am met with the following dialogue box.

Give this a try with the Veeam File Browser as well- and you will see even the files associated with the immutable chain cannot be removed,

Further, navigating to the directory directly on the Linux machine under my standard login will still not allow me to delete these file

Can I manually delete a backup file from the linux shell before the immutability retention as linuxrepo?

 It is not until I have root access and remove the immutability attribute can I delete the file. (This is where FlashArray SafeMode comes in. Look for Part II of this blog series.)

So, that concludes this part. Hope you can see the power of this setup in the backup architecture design. Remember, this is no replacement for the 3-2-1 rule, just simply another tool in providing a reliable and resilient backup solution for your datacenter. Also, this is just one of the features version 11 will provide to the future of your backup solution. So, please come back and look for further posts on the features on their way. And as discussed in the beginning, I present Part 2 - where we provide further security in-depth, utilizing a newly announced hardware snapshot feature on Flash Array.

Linux Hardened Repo walkthrough from home lab