Hi, everyone

Today in this article, I will show you how to create and configure “Tagged Vlan / Vlan Tagging” in a virtualized Oracle Database Appliance (OVM / ODA_BASE).

1. Introduction

The goal of this document is to provide the necessary steps describing how to add a new VLAN in the Oracle Database Appliance. This document covers version ODA X4-2 (ODA_BASE).
The document explains how to go through the different steps and show errors encountered during the addition of new VLAN.

2. OVM Network Overview

Every new Vlan created in the ODA, it will create:

  • A new virtual network interface on top of bond0 with the vlan Id: (bond0.199 and bond0.200).
  • A new bridge network interface (vlan199, DEV).
PS: We are using BOND0 as our public interface and you should adapt the ethernet interface to your needs during this procedure, e.g: (bond0, bond1, bond2, eth1 or eth2 and etc).
Also inside of the ODA_BASE it will be added all the new virtual interfaces (eth3, eth4 and etc.) related to the bridge network interfaces in the DOM0.
To identify this relationship, from DOM0:
  • xm list -l 

It will display a lot of configuration for the DOM0 and you can get the MAC Address for the bridge interface and make the relationship to the ODA_BASE.

DOM0:

  (device
(vif
(bridge vlan199)
(handle 3)
(uuid 79b1e8cb-45f4-b03d-ea75-3b682932edb9)
(script /etc/xen/scripts/vif-bridge)
(mac 00:16:3e:43:18:eb)
(type netfront)
(backend 0)
)
)
(device
(vif
(bridge DEV)
(handle 4)
(uuid b2c14fd8-6aca-4d55-36fd-4168d1aba017)
(script /etc/xen/scripts/vif-bridge)
(mac 00:16:3e:4f:4d:82)
(type netfront)
(backend 0)
)
)

ODA_BASE:

ifconfig | grep -i “00:16:3e:43:18:eb“

eth3      Link encap:Ethernet  HWaddr 00:16:3E:43:18:EB

ifconfig | grep -i “00:16:3e:4f:4d:82“

eth4      Link encap:Ethernet  HWaddr 00:16:3E:4F:4D:82 

3. Checks Before Add Vlan

Before starting adding/changing anything in the environment. We must be sure to backup all configuration files related to our procedure.

3.1 Backup Network Configuration  File – DOM0 / ODA_BASE (All Nodes):

DOM0 / Node1: 

mkdir -p /root/ODA_DOM0_VLAN_BKP_`hostname`/network-scripts 

cp -p /etc/sysconfig/network /root/ODA_DOM0_VLAN_BKP_`hostname`   

cp -rp /etc/sysconfig/network-scripts/* /root/ODA_DOM0_VLAN_BKP_`hostname`/network-scripts

 DOM0 / Node2: 

mkdir -p /root/ODA_DOM0_VLAN_BKP_`hostname`/network-scripts 

cp -p /etc/sysconfig/network /root/ODA_DOM0_VLAN_BKP_`hostname`   

cp -rp /etc/sysconfig/network-scripts/* /root/ODA_DOM0_VLAN_BKP_`hostname`/network-scripts

ODA_BASE / Node1: 

mkdir -p /root/ODA_BASE_VLAN_BKP_`hostname`/network-scripts 

cp -p /etc/sysconfig/network /root/ODA_BASE_VLAN_BKP_`hostname` 

cp -rp /etc/sysconfig/network-scripts/* /root/ODA_BASE_VLAN_BKP_`hostname`/network-scripts  

ODA_BASE / Node2: 

mkdir -p /root/ODA_BASE_VLAN_BKP_`hostname`/network-scripts 

cp -p /etc/sysconfig/network /root/ODA_BASE_VLAN_BKP_`hostname` 

cp -rp /etc/sysconfig/network-scripts/* /root/ODA_BASE_VLAN_BKP_`hostname`/network-scripts  

 PS: Also is possible to shutdown the ODA_BASE and perform a backup/snapshot of the VM.

4. Steps Create New Vlan, ODA_BASE

 4.1 From ODA_BASE as root user:

oakcli create vlan vlan199 -vlanid 199 -if bond0 -node 0

oakcli create vlan vlan199 -vlanid 199 -if bond0 -node 1 

oakcli create vlan DEV -vlanid 200 -if bond0 -node 0

oakcli create vlan DEV -vlanid 200 -if bond0 -node 1 

PS: Is not required to run in each node, since we are passing the parameter “-node 0 | -node 1”.

4.2 Setup new Vlan’s in the ODA_BASE:

  •  oakcli configure oda_base
Core Licensing Options:
        1. 2 CPU Cores
        2. 4 CPU Cores
        3. 6 CPU Cores
        4. 8 CPU Cores
        5. 10 CPU Cores
        6. 12 CPU Cores
        7. 14 CPU Cores
        8. 16 CPU Cores
        9. 24 CPU Cores
        Current CPU Cores       :24
        Selection[1 .. 9](default 24 CPU Cores) : 9
        ODA base domain memory in GB(min 16, max 244)(Current Memory 236G)[default 244] : 236
Additional vlan networks to be assigned to oda_base ? (y/n) [n]: y
Select the network to assign (vlan199,DEV,vlan199,DEV): vlan199
Additional vlan networks to be assigned to oda_base ? (y/n) [n]: y
Select the network to assign (DEV,DEV): DEV
Additional vlan networks to be assigned to oda_base ? (y/n) [n]: n
Vlan network to be removed from oda_base ? (y/n) [n]: n
INFO: Node 0:Configured oda base pool
INFO: Node 1:Configured oda base pool
INFO: Node 0:ODA Base configured with new memory
INFO: Node 0:ODA Base configured with new vcpus
INFO: Node 0:New vlans added in ODA Base
INFO: Changes will be incorporated after the domain is restarted on Node 0.
INFO: Node 1:ODA Base configured with new memory
INFO: Node 1:ODA Base configured with new vcpus
INFO: Node 1:New vlans added in ODA Base
INFO: Changes will be incorporated after the domain is restarted on Node 1.
INFO: Updating /etc/sysctl.conf in oda_base domain with parameter “vm.nr_hugepages=63453”

INFO: Updating /etc/security/limits.conf in oda_base domain with new memlock value 177000000

4.3 Show New Vlan’s Added:

  • oakcli show oda_base 
ODA base domain
ODA base CPU cores      :24
ODA base domain memory  :236
ODA base template       :/OVS/oda_base_12.1.2.0.gz
ODA base vlans          :[‘priv1’, ‘net1’, ‘net2’, ‘vlan199’, ‘DEV’]

ODA base current status :Running

4.4 Restart ODA_BASE

After add all new Vlan’s, restart ODA_BASE from DOM0 on both nodes:

  • oakcli stop oda_base 
  • oakcli start oda_base 

PS: It is possible also to run: oakcli restart oda_base. 
(Personally I prefer run stop/start, since I already had problems with the oakd using restart command)
I would advise to perform a reboot also of the DOM0, just to “get the house organized”.

5. Changing IP’s DOM0 / ODA_BASE

To be able to keep connecting to the DOM0 after change the switch in trunk mode, is necessary to change the IP from NET1 (Bridge Interface) to VLAN199 (Bridge Interface).
All configurations in the network of DOM0 are done, node per node and not in the same time.
To achieve it, is necessary to connect from one DOM0 node through interconnecting to the other DOM0 node (Crossed connection).

PS: This step is required only if you do not have access to the DOM0 ILOM to use the console (web-console).
Unfortunately it was my case.

5.1 SSH Connect DOM0 (Crossed Connection)

 From node2-dom0, connect to the node1-dom0 using the interconnect IP from PRIV1 network interface:

  • node2-dom0: ssh root@192.168.10.20 (Interconnect IP node1-dom0)

5.2 SSH Connect ODA_BASE (Crossed Connection)

 From node2-oda_base, connect to the node1-oda_base using the interconnect IP from eth0 network interface:

  • node2-oda_base: ssh root@192.168.10.22 (Interconnect IP node1-oda_base)

DOM0: Crossed connection from NODE2-DOM0 to NODE1-DOM0

 At this point we can start to change the network configuration files and validate if trunk mode is working fine.

Put a comment in the configurations of NET1 (Bridge Interface), as below:
  •  vi /etc/sysconfig/network-scripts/ifcfg-net1

#This file was dynamically created by OAK
DEVICE=net1
TYPE=Bridge
ONBOOT=yes
DELAY=0
BOOTPROTO=none
#IPADDR=10.10.10.10
#NETMASK=255.255.255.0
#GATEWAY=10.10.10.1
BRIDGING_OPTS=”multicast_snooping=0″

 Copy the IP configurations from NET1 (Bridge Interface) and past it, in the VLAN199 configuration file:

  • vi /etc/sysconfig/network-scripts/ifcfg-vlan199

#This file was dynamically created by OAK
DEVICE=vlan199
TYPE=Bridge
ONBOOT=yes
DELAY=0
BOOTPROTO=none
IPADDR=10.10.10.10
NETMASK=255.255.255.0
GATEWAY=10.10.10.1
BRIDGING_OPTS=”multicast_snooping=0″

Also is required to change the network interface used by the default gateway, from ETH1 to ETH3:

  • vi /etc/sysconfig/network 

NETWORKING=yes
HOSTNAME=node1-oda_base
NOZEROCONF=yes
GATEWAY=10.10.10.1
GATEWAYDEV=eth3

ODA_BASE: Connection from NODE1-ODA_BASE to NODE1-ODA_BASE

Put a comment in the configurations of eth1, as below:

  • vi /etc/sysconfig/network-scripts/ifcfg-eth1

  #Comet Auto generated config file. Bond interface eth1
DEVICE=eth1
ONBOOT=yes
BOOTPROTO=none
#IPADDR=10.10.10.12
#NETMASK=255.255.255.0
#GATEWAY=10.10.10.1

Copy the IP configuration from ETH1 to ETH3 network interface:

  • vi /etc/sysconfig/network-scripts/ifcfg-eth3

  #Comet Auto generated config file. Bond interface eth3
DEVICE=eth3
ONBOOT=yes
BOOTPROTO=none
IPADDR=10.10.10.12
NETMASK=255.255.255.0
GATEWAY=10.10.10.1

Also is required to change the network interface used by the default gateway, from ETH1 to ETH3:

  • vi /etc/sysconfig/network 

NETWORKING=yes
HOSTNAME=node1-oda_base
NOZEROCONF=yes
GATEWAY=10.10.10.1
GATEWAYDEV=eth3

5.3 Shutdown/Startup ETH1 and ETH3 (NODE1-ODA_BASE), as root user:

  • ifdown eth1 
  • ifdown eth3
  • ifup eth3
  • ifup eth1  

5.4 Shutdown/Startup NET1 and VLAN199 (NODE1-DOM0), as root user:

  • ifdown net1 
  • ifdown vlan199
  • ifup vlan199
  • ifup net1

PS: After these steps ask to the network team to Active the “Trunk Mode” in their switch.
Every network connection should work properly after this change.