Difference between revisions of "QuantaStor Local Update Mirror"

From OSNEXUS Online Documentation Site
Jump to: navigation, search
m
m (Nginx + RSync Package Mirror Method)
(14 intermediate revisions by the same user not shown)
Line 3: Line 3:
 
== Overview ==
 
== Overview ==
  
If your QuantaStor appliance cannot connect to the internet to update packages, then you can use a local update mirror to update QuantaStor.  
+
Often times QuantaStor systems are deployed in sites where there's no public internet access.  On these deployments the QuantaStor systems are unable to directly access packages.osnexus.com and in order for the systems to run upgrades they will need a local package mirror.  The local package mirror is basically a VM which has a copy of the QuantaStor and other package files from packages.osnexus.com so that the local DNS server can be configured to resolve packages.osnexus.com to the local mirror within the datacenter.
  
== Systems with Ubuntu 14.04 Base ==
+
There are a couple of methods by which a package mirror repository can be setup, those are:
  
Follow the steps below to create a local mirror of the OSNEXUS QuantaStor update repository using apt-mirror for systems with Ubuntu 14.04 as its base operating system. The below instructions assume you already have an local Ubuntu server currently installed. For each instruction, the label [QS] will be instructions for the QuantaStor appliance, and the label [MIR] will be instructions for the mirror server.
+
* Nginx + rsync of packages.osnexus.com
 +
* Apache + apt-mirror of packages.osnexus.com
 +
 
 +
== Nginx + RSync Package Mirror Method ==
 +
 
 +
The Nginx + RSync method is the simplest and most common way to mirror the OSNexus package repository at packages.osnexus.com.
 +
 
 +
Follow the steps below to create a local mirror of the OSNEXUS QuantaStor update repository.
 +
 
 +
1. Contact support@osnexus.com to request that your SSH public key be added to the OSNEXUS QuantaStor replication mirror for rsync access. Support can also assist with the below process and any questions you may have.
 +
 
 +
2. Setup a Linux VM with a web server, below is an example config file using nginx on Ubuntu 18.04 or newer:
 +
 
 +
Edit the /etc/nginx/sites-enabled/osnexus-mirror file. '''Please note. The file may be different depending on your nginx version or Linux OS. '''
 +
 
 +
'''Replace''' yourserver.example.net with your web server DNS FQDN.
 +
 
 +
<pre>
 +
 
 +
#
 +
# A virtual host entry for an OSNEXUS QuantaStor mirror
 +
#
 +
 
 +
server {
 +
    listen      80;
 +
    server_name  yourserver.example.net packages;
 +
    location / {
 +
        root /var/lib/www/packages-root;
 +
        autoindex on;
 +
    }
 +
}
 +
</pre>
 +
 
 +
Now restart and ensure it is enabled to startup automatically.
 +
 
 +
<pre>
 +
systemctl enable nginx
 +
systemctl restart nginx
 +
</pre>
 +
 
 +
3. ensure your ssh key is available for the root user as the id_rsa file under /root/.ssh/
 +
 
 +
4. Run an initial rsync to synchronize the QuantaStor update repo using the credentials and SSH key setup with the OSNEXUS support team in step 1.
 +
 
 +
<pre>
 +
rsync -aPv rsync://YOURUSER@pkgmirror.osnexus.com/packages /var/lib/www/packages-root/
 +
</pre>
 +
 
 +
5. Add a cron entry that will run every day to synchronize your local mirror with the QuantaStor repo. Note that this requires ssh access from your mirror server to pkgmirror.osnexus.com
 +
 
 +
<pre>
 +
edit the /etc/cron.d/osnexus-mirror-sync file and add the below contents:
 +
0 02 * * * root /usr/bin/rsync -aPv rsync://YOURUSER@pkgmirror.osnexus.com/packages /var/lib/www/packages-root/
 +
</pre>
 +
 
 +
6. Add your new local mirror to your QuantaStor systems for all updates:
 +
 
 +
'''Replace''' yourserver.example.net with your web server DNS FQDN.
 +
 
 +
<pre>
 +
echo 'deb http://yourserver.example.net quantastor-bionic main' > /etc/apt/sources.list.d/osnexus.list
 +
</pre>
 +
 
 +
If you need any further assistance, please contact support@osnexus.com.
 +
 
 +
== Apache + apt-mirror Package Mirror Method ==
 +
 
 +
Follow the steps below to create a local mirror of the OSNEXUS QuantaStor update repository using apt-mirror for systems with Ubuntu 14.04 as its base operating system. The below instructions assume you already have an local Ubuntu server currently installed.  
 +
 
 +
[QS] label indicates commands will to run on your QuantaStor systems
 +
 
 +
[MIR] label indicates commands to run on your package mirror server / VM
  
 
=== Instructions ===
 
=== Instructions ===
Line 25: Line 96:
 
[MIR] 7. Edit /etc/apt/mirror.list to be like the below:
 
[MIR] 7. Edit /etc/apt/mirror.list to be like the below:
  
 +
==== Ubuntu 18.04 (Bionic) mirror.list Configuration ====
 +
<pre>
 +
set base_path    /var/www/html/repo
 +
set nthreads    20
 +
set _tilde 0
 +
 +
deb http://packages.osnexus.com/packages bionic main
 +
deb http://archive.ubuntu.com/ubuntu bionic main restricted universe multiverse
 +
deb http://archive.ubuntu.com/ubuntu bionic-security main restricted universe multiverse
 +
deb http://archive.ubuntu.com/ubuntu bionic-updates main restricted universe multiverse
 +
deb-src http://archive.ubuntu.com/ubuntu bionic main restricted universe multiverse
 +
deb-src http://archive.ubuntu.com/ubuntu bionic-security main restricted universe multiverse
 +
deb-src http://archive.ubuntu.com/ubuntu bionic-updates main restricted universe multiverse
 +
clean http://archive.ubuntu.com/ubuntu
 +
clean http://packages.osnexus.com/packages
 +
</pre>
 +
 +
==== Ubuntu 16.04 (Xenial) mirror.list Configuration ====
 +
<pre>
 +
set base_path    /var/www/html/repo
 +
set nthreads    20
 +
set _tilde 0
 +
 +
deb http://packages.osnexus.com/packages xenial main
 +
deb http://archive.ubuntu.com/ubuntu xenial main restricted universe multiverse
 +
deb http://archive.ubuntu.com/ubuntu xenial-security main restricted universe multiverse
 +
deb http://archive.ubuntu.com/ubuntu xenial-updates main restricted universe multiverse
 +
deb-src http://archive.ubuntu.com/ubuntu xenial main restricted universe multiverse
 +
deb-src http://archive.ubuntu.com/ubuntu xenial-security main restricted universe multiverse
 +
deb-src http://archive.ubuntu.com/ubuntu xenial-updates main restricted universe multiverse
 +
clean http://archive.ubuntu.com/ubuntu
 +
clean http://packages.osnexus.com/packages
 +
</pre>
 +
 +
==== Ubuntu 14.04 (Trusty) mirror.list Configuration ====
 
<pre>
 
<pre>
 
############# config ##################
 
############# config ##################
Line 52: Line 158:
 
clean http://packages.osnexus.com/packages
 
clean http://packages.osnexus.com/packages
 
</pre>
 
</pre>
 +
 +
=== Running the Package Mirroring ===
  
 
[MIR] 8. Create a var folder in the local repo (<code>mkdir -p /var/www/html/repo/var</code>)
 
[MIR] 8. Create a var folder in the local repo (<code>mkdir -p /var/www/html/repo/var</code>)
Line 67: Line 175:
 
[QS] 13. Edit /etc/apt/sources.list to be like the below:
 
[QS] 13. Edit /etc/apt/sources.list to be like the below:
  
 +
 +
==== Ubuntu 18.04 (Bionic) mirror.list Configuration ====
 +
<pre>
 +
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ bionic-updates main
 +
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ bionic universe
 +
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ bionic-updates universe
 +
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ bionic multiverse
 +
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ bionic-updates multiverse
 +
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ bionic-security main restricted universe multiverse
 +
</pre>
 +
 +
==== Ubuntu 16.04 (Xenial) mirror.list Configuration ====
 +
<pre>
 +
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ xenial-updates main
 +
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ xenial universe
 +
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ xenial-updates universe
 +
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ xenial multiverse
 +
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ xenial-updates multiverse
 +
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ xenial-security main restricted universe multiverse
 +
</pre>
 +
 +
==== Ubuntu 14.04 (Trusty) mirror.list Configuration ====
 
<pre>
 
<pre>
 
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ trusty-updates main
 
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ trusty-updates main
Line 77: Line 207:
  
 
'''Note:''' Due to some oddities with apt-mirror, we have to specify using 64-bit with <code>[arch=amd64]</code>. If you don't, you'll see errors related to not being able to find 32-bit (i386) packages.
 
'''Note:''' Due to some oddities with apt-mirror, we have to specify using 64-bit with <code>[arch=amd64]</code>. If you don't, you'll see errors related to not being able to find 32-bit (i386) packages.
 +
 +
=== Upgrading QuantaStor ===
  
 
[QS] 14. Edit /etc/apt/sources.list.d/osnexus.list to be like the below:
 
[QS] 14. Edit /etc/apt/sources.list.d/osnexus.list to be like the below:
Line 87: Line 219:
  
 
'''Note:''' You may get stuck not being able to update the virt-what package. If this happens, you may either download (<code>wget</code>) and install (<code>dpkg -i</code>) the .deb package from a trusted source on the QuantaStor appliance, or download it to /var/www/html/repo/mirror/archive.ubuntu.com/ubuntu/pool/universe/v/virt-what.
 
'''Note:''' You may get stuck not being able to update the virt-what package. If this happens, you may either download (<code>wget</code>) and install (<code>dpkg -i</code>) the .deb package from a trusted source on the QuantaStor appliance, or download it to /var/www/html/repo/mirror/archive.ubuntu.com/ubuntu/pool/universe/v/virt-what.
 
== Alternative Methods ==
 
 
Follow the steps below to create a local mirror of the OSNEXUS QuantaStor update repository.
 
 
1. Contact support@osnexus.com to request that your SSH public key be added to the OSNEXUS QuantaStor replication mirror for rsync access. Support can also assist with the below process and any questions you may have.
 
 
2. Setup a Linux VM with a web server, below is an example config file using nginx on Ubuntu 18.04 or newer:
 
 
Edit the /etc/nginx/sites-enabled/osnexus-mirror file. '''Please note. The file may be different depending on your nginx version or Linux OS. '''
 
 
'''Replace''' yourserver.example.net with your web server DNS FQDN.
 
 
<pre>
 
 
#
 
# A virtual host entry for an OSNEXUS QuantaStor mirror
 
#
 
 
server {
 
    listen      80;
 
    server_name  yourserver.example.net packages;
 
    location / {
 
        root /var/lib/www/packages-root;
 
        autoindex on;
 
    }
 
}
 
</pre>
 
 
Now restart and ensure it is enabled to startup automatically.
 
 
<pre>
 
systemctl enable nginx
 
systemctl restart nginx
 
</pre>
 
 
3. ensure your ssh key is available for the root user as the id_rsa file under /root/.ssh/
 
 
4. Run an initial rsync to synchronize the QuantaStor update repo using the credentials and SSH key setup with the OSNEXUS support team in step 1.
 
 
<pre>
 
rsync -aPv rsync://YOURUSER@pkgmirror.osnexus.com/packages /var/lib/www/packages-root/
 
</pre>
 
 
5. Add a cron entry that will run every day to synchronize your local mirror with the QuantaStor repo. Note that this requires ssh access from your mirror server to pkgmirror.osnexus.com
 
 
<pre>
 
edit the /etc/cron.d/osnexus-mirror-sync file and add the below contents:
 
0 02 * * * root /usr/bin/rsync -aPv rsync://YOURUSER@pkgmirror.osnexus.com/packages /var/lib/www/packages-root/
 
</pre>
 
 
6. Add your new local mirror to your QuantaStor systems for all updates:
 
 
'''Replace''' yourserver.example.net with your web server DNS FQDN.
 
 
<pre>
 
echo 'deb http://yourserver.example.net/packages quantastor-bionic main' > /etc/apt/sources.list.d/osnexus.list
 
</pre>
 
 
If you need any further assistance, please contact support@osnexus.com.
 

Revision as of 09:25, 23 May 2022


Overview

Often times QuantaStor systems are deployed in sites where there's no public internet access. On these deployments the QuantaStor systems are unable to directly access packages.osnexus.com and in order for the systems to run upgrades they will need a local package mirror. The local package mirror is basically a VM which has a copy of the QuantaStor and other package files from packages.osnexus.com so that the local DNS server can be configured to resolve packages.osnexus.com to the local mirror within the datacenter.

There are a couple of methods by which a package mirror repository can be setup, those are:

  • Nginx + rsync of packages.osnexus.com
  • Apache + apt-mirror of packages.osnexus.com

Nginx + RSync Package Mirror Method

The Nginx + RSync method is the simplest and most common way to mirror the OSNexus package repository at packages.osnexus.com.

Follow the steps below to create a local mirror of the OSNEXUS QuantaStor update repository.

1. Contact support@osnexus.com to request that your SSH public key be added to the OSNEXUS QuantaStor replication mirror for rsync access. Support can also assist with the below process and any questions you may have.

2. Setup a Linux VM with a web server, below is an example config file using nginx on Ubuntu 18.04 or newer:

Edit the /etc/nginx/sites-enabled/osnexus-mirror file. Please note. The file may be different depending on your nginx version or Linux OS.

Replace yourserver.example.net with your web server DNS FQDN.


#
# A virtual host entry for an OSNEXUS QuantaStor mirror
#

server {
    listen       80;
    server_name  yourserver.example.net packages;
    location / {
        root /var/lib/www/packages-root;
        autoindex on;
    }
}

Now restart and ensure it is enabled to startup automatically.

systemctl enable nginx
systemctl restart nginx

3. ensure your ssh key is available for the root user as the id_rsa file under /root/.ssh/

4. Run an initial rsync to synchronize the QuantaStor update repo using the credentials and SSH key setup with the OSNEXUS support team in step 1.

rsync -aPv rsync://YOURUSER@pkgmirror.osnexus.com/packages /var/lib/www/packages-root/

5. Add a cron entry that will run every day to synchronize your local mirror with the QuantaStor repo. Note that this requires ssh access from your mirror server to pkgmirror.osnexus.com

edit the /etc/cron.d/osnexus-mirror-sync file and add the below contents:
0 02 * * * root /usr/bin/rsync -aPv rsync://YOURUSER@pkgmirror.osnexus.com/packages /var/lib/www/packages-root/

6. Add your new local mirror to your QuantaStor systems for all updates:

Replace yourserver.example.net with your web server DNS FQDN.

echo 'deb http://yourserver.example.net quantastor-bionic main' > /etc/apt/sources.list.d/osnexus.list

If you need any further assistance, please contact support@osnexus.com.

Apache + apt-mirror Package Mirror Method

Follow the steps below to create a local mirror of the OSNEXUS QuantaStor update repository using apt-mirror for systems with Ubuntu 14.04 as its base operating system. The below instructions assume you already have an local Ubuntu server currently installed.

[QS] label indicates commands will to run on your QuantaStor systems

[MIR] label indicates commands to run on your package mirror server / VM

Instructions

[MIR] 1. Elevate as root (sudo -i)

[MIR] 2. Update packages (apt-get update)

[MIR] 3. Install and enable Apache Web Server (apt install -y apache2 && systemctl enable apache2)

[MIR] 4. Create a package directory and set the correct permissions (mkdir -p /var/www/html/repo && chown www-data:www-data /var/www/html/repo)

[MIR] 5. Install apt-mirror (apt install -y apt-mirror)

[MIR] 6. Make a backup of /etc/apt/mirror.list (cp /etc/apt/mirror.list /etc/apt/mirror.list-bak)

[MIR] 7. Edit /etc/apt/mirror.list to be like the below:

Ubuntu 18.04 (Bionic) mirror.list Configuration

set base_path    /var/www/html/repo
set nthreads     20
set _tilde 0

deb http://packages.osnexus.com/packages bionic main
deb http://archive.ubuntu.com/ubuntu bionic main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu bionic-security main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu bionic-updates main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu bionic main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu bionic-security main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu bionic-updates main restricted universe multiverse
clean http://archive.ubuntu.com/ubuntu
clean http://packages.osnexus.com/packages

Ubuntu 16.04 (Xenial) mirror.list Configuration

set base_path    /var/www/html/repo
set nthreads     20
set _tilde 0

deb http://packages.osnexus.com/packages xenial main
deb http://archive.ubuntu.com/ubuntu xenial main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu xenial-security main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu xenial-updates main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu xenial main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu xenial-security main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu xenial-updates main restricted universe multiverse
clean http://archive.ubuntu.com/ubuntu
clean http://packages.osnexus.com/packages

Ubuntu 14.04 (Trusty) mirror.list Configuration

############# config ##################
#
set base_path    /var/www/html/repo
#
# set mirror_path  $base_path/mirror
# set skel_path    $base_path/skel
# set var_path     $base_path/var
# set cleanscript $var_path/clean.sh
# set defaultarch  <running host architecture>
# set postmirror_script $var_path/postmirror.sh
# set run_postmirror 0
set nthreads     20
set _tilde 0
#
############# end config ##############

deb http://packages.osnexus.com/packages trusty main
deb http://archive.ubuntu.com/ubuntu trusty main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu trusty-security main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu trusty-updates main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu trusty main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu trusty-security main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu trusty-updates main restricted universe multiverse
clean http://archive.ubuntu.com/ubuntu
clean http://packages.osnexus.com/packages

Running the Package Mirroring

[MIR] 8. Create a var folder in the local repo (mkdir -p /var/www/html/repo/var)

[MIR] 9. Copy /var/spool/apt-mirror/var/postmirror.sh to the var folder in the local repo (mkdir -p cp /var/spool/apt-mirror/var/postmirror.sh /var/www/html/repo/var)

[MIR] 10. Start the mirroring process (nohup apt-mirror &)

Note: Depending on your internet speed, this may take some time, but you can monitor the process with tail nohup.out. When it's finished, continue with step 11.

[QS] 11. Elevate as root (sudo -i)

[QS] 12. Make a copy of /etc/apt/sources.list and /etc/apt/sources.list.d/osnexus.list (cp /etc/apt/sources.list /etc/apt/sources.list.bk && cp /etc/apt/sources.list.d/osnexus.list /etc/apt/sources.list.d/osnexus.list.bk

[QS] 13. Edit /etc/apt/sources.list to be like the below:


Ubuntu 18.04 (Bionic) mirror.list Configuration

deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ bionic-updates main
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ bionic universe
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ bionic-updates universe
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ bionic multiverse
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ bionic-updates multiverse
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ bionic-security main restricted universe multiverse

Ubuntu 16.04 (Xenial) mirror.list Configuration

deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ xenial-updates main
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ xenial universe
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ xenial-updates universe
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ xenial multiverse
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ xenial-updates multiverse
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ xenial-security main restricted universe multiverse

Ubuntu 14.04 (Trusty) mirror.list Configuration

deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ trusty-updates main
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ trusty universe
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ trusty-updates universe
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ trusty multiverse
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ trusty-updates multiverse
deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/archive.ubuntu.com/ubuntu/ trusty-security main restricted universe multiverse

Note: Due to some oddities with apt-mirror, we have to specify using 64-bit with [arch=amd64]. If you don't, you'll see errors related to not being able to find 32-bit (i386) packages.

Upgrading QuantaStor

[QS] 14. Edit /etc/apt/sources.list.d/osnexus.list to be like the below:

deb [arch=amd64] http://<YOUR_MIR_IP>/repo/mirror/packages.osnexus.com/packages/ trusty main

[QS] 15. Update packages and upgrade QuantaStor (apt-get update && qs-upgrade)

Note: You may get stuck not being able to update the virt-what package. If this happens, you may either download (wget) and install (dpkg -i) the .deb package from a trusted source on the QuantaStor appliance, or download it to /var/www/html/repo/mirror/archive.ubuntu.com/ubuntu/pool/universe/v/virt-what.