Working with Virtual IPs in Oracle Cloud

Recently I needed to use a Virtual IP (VIP) in Oracle Cloud (OCI) but unfortunately I have realized that there is a problem. The problem is that OCI can recognize only IP addresses directly assigned to VNIC of an instance. But that is not the case of VIP. Let’s have a look how to cope with this problem.

Here is the situation (at the beginning I’m describing a sample scenario so if you are not interested you can skip down directly to the steps).

We have two compute instances, each with one private IP which is assigned to a VNIC.

Instance 1 (vm1):

Instance 2 (vm2):

Both of the instances are within the same subnet with mask /24 hence there is no issue with reaching each other.

The problem appears when you need to assign another IP address to the same VNIC and the IP has to be randomly on one or the other compute instance’s VNIC.

Let me clarify this in more detail. Imagine you have two private IPs assigned to one compute instance. In addition to existing one you have added another IP from the same subnet. There is no problem with that if you have it assigned also on a VNIC.

In such a case you can easily reach new IP form the other instance.

But what if you need to have a floating IP or VIP which is randomly on one or the other instance?

If you attempt to assign the same IP address to the other instance you’ll get an error that IP address is already in use. This is correct and expected behavior. So this is not the way to go.

Solution for this is to unassigned the IP from first instance’s VNIC.

Now we successfully assigned the IP address to second instance but here comes the problem again. If you have to the same IP addresses on more than one instance than it will be used only for that instance. In other words packets will never leave the instance where the IP is assigned (similarly to localhost). So you can simply remove the IP from the first instance to ensure that new IP is only on the second instance.

When you try to reach the IP on the second instance you’ll get Host Unreachable error. Which is the main problem here.

So finally we get into the problem with floating IPs or VIPs. If the floating IP is NOT directly assigned to VNIC it is not reachable. But two same IP address can not be assigned to two different VNICs of two different instances. So how to solve this problem?

As of now it is not resolvable directly in OCI. This is simply unsupported state at the moment. Oracle is working on resolution of such a state but if you need to use it right now you have to probably do something like I did.

My solution simply ensures dynamic reassignment of IP addresses on VNICs. Indeed I just utilized OCI CLI for assignment of the IP only on VNIC where it suppose to be. Steps are as follows:

1. Make sure you have working OCI CLI on all involved instances. https://docs.cloud.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm
2. There is a switch “ unassign-if-already-assigne” which force unassignment of an IP from existing VNIC while assigning to new one.
3. Create a bash script which will iterate through assigned IP addresses on the instance and ensures that all of the listed IP address are assigned to VNIC of the instance.
4. Setup a cron job (or any other trigger) which will run the bash script on regular bases or in case of new IP is assigned to the instance.

My bash script looks as follows:

#!/bin/bash
set -x
CURR_COUNT=./curr_count
if [ -f "$CURR_COUNT" ]; then
  ETH=$(/usr/sbin/ip addr show | awk '/inet.*brd/{print $NF; exit}')
  CURR_IP_COUNT=$(ip addr  |  grep $ETH | grep inet |wc -l)
  if [ $CURR_IP_COUTN -ne $(read line < $CURR_COUNT) ]; then
     echo $CURR_IP_COUNT > curr_count
     LIST_OF_IPS=$(/usr/sbin/ifconfig | grep inet | grep -v "127.0.0.1" | awk '{print $2}')
     for i in $LIST_OF_IPS
     do
      VNIC=$(curl -s http://169.254.169.254/opc/v1/vnics/ | jq  '.[].vnicId'  | sed -r 's/"//g')
      oci network vnic assign-private-ip --vnic-id ${VNIC} --ip-address $i --unassign-if-already-assigned
     done
  fi
else
  ETH=$(/usr/sbin/ip addr show | awk '/inet.*brd/{print $NF; exit}')
  CURR_IP_COUNT=$(ip addr  |  grep $ETH | grep inet |wc -l)
  echo $CURR_IP_COUNT > curr_count
fi
set +x

The same code but in a screenshot for better readability.

Then I setup simple cron job to run every minute which was sufficient for my scenario.

Limitations: I presume that assigning a VIP to the instance is done automatically or by some side job. In my case I did Weblogic server migration hence Nodemanger of Weblogic did the migration of VIP between instances.