• Home
  • Deploy OpenShift with NSX-T Antrea Network CNI

NSX-T Antrea is a Container Network Interface plugin for Kubernetes. It can be used to replace OVNKubernetes as default CNI.


NSX-T Antrea is a Container Network Interface (CNI) plugin for Kubernetes.

It is designed to provide networking and security capabilities for K8S clusters. Developed by VMware, NSX-T Antrea integrates with NSX-T Data Center to extend its networking and security features to Kubernetes environments.

Key features of NSX-T Antrea include:

1 - Network Connectivity: NSX-T Antrea facilitates communication between pods (containers) within Kubernetes clusters, ensuring seamless connectivity across the cluster.

2 - Security Policies: It allows administrators to define fine-grained security policies to control traffic between pods, enhancing security within the Kubernetes environment.

3 - Policy Enforcement: NSX-T Antrea enforces network policies at the pod level, ensuring that only authorized communication occurs between pods, thus helping to prevent unauthorized access and potential security breaches.

4 - Integration with NSX-T Data Center: As part of the NSX-T suite, Antrea seamlessly integrates with NSX-T Data Center, leveraging its advanced networking and security capabilities to enhance Kubernetes environments.

Overall, NSX-T Antrea CNI simplifies the management of networking and security within Kubernetes clusters, providing administrators with the tools they need to build robust and secure containerized environments.

About This Laboratory

In this lab, we’ll be using OpenShift 4.12.45 and Antrea 1.8.0 (VMware Container Networking with Antrea 1.8.0).

* Antrea 1.9.0 requires additional components (Agent and Controller) which will not be covered in this post. We will therefore continue with version 1.8.0.

VMWare Documentation

VMWare provided instructions about Container Networking with Antrea.

They can be found here.

- There is also an old post about this here.

Obtaining Antrea

Antrea files can be downloaded from A valid user with access to an active subscription must be used.

For the installation, you will need the Antrea container image and the Antrea Kubernetes Operator container image.

These images must be made available in a Container Registry of your choice.

πŸ’‘ You must upload the downloaded images to a docker compatible registry.

⚠️ If you don’t know how to upload Antrea images to a docker registry, take a look at this post.

After you upload the images to a registry, check pulling them before proceeding. Example:

 $ podman pull

 $ podman pull

Using Antrea

You can use Antrea as a network CNI for your OpenShift or Kubernetes cluster, regardless of whether you have NSX running in your environment.

⛔️ Please note that after the cluster is installed, the CNI cannot be changed.

You may now be wondering: Why would I do that? πŸ€”

Well, you shouldn’t, unless you want to learn something different, or you’re working on an OpenShit integration with NSX-T.

Deploy OpenShift with Antrea Network CNI

This task consists of 2 phases:

1) Deploy the OCP cluster using the Antrea “networkType” instead of OVNKubermetes.

2) Integrating the OCP cluster with NSX-T.

βœ… You can integrate the OpenShift cluster with Antrea as the CNI with NSX-T at any time after the OpenShift cluster is created.

NOTE: In this lab, we’ll only cover the first phase, which is usually more complex. Phase 2 can be completed with the reading indicated at the beginning of this post.

Preparing OCP Deployment

Create a subdirectory named “antrea-ocp” and place the necessary files in it.

  $ pwd

I’ve organized my directories in this way to make the process easier to understand.

  $ tree

  β”œβ”€β”€ 1-ocp-deploy-with-nsx-antrea
  β”‚Β Β  β”œβ”€β”€ operator_image
  β”‚Β Β  └── operator_manifests
  β”œβ”€β”€ 2-ocp-nsx-integration
  β”‚Β Β  └── interworking
  β”œβ”€β”€ auth
  β”œβ”€β”€ deploy
  β”‚Β Β  β”œβ”€β”€ kubernetes
  β”‚Β Β  └── openshift
  └── tls

First, create the install-config.yaml file, as we need to customize it a bit.

 $ openshift-install create install-config --dir .

On your install-config.yaml, look for networkType and replace OVNKubernetes by antrea.

 $ cat install-config.yaml.tmp
   networkType: antrea

Backup your install-config.yaml:

  $ cp install-config.yaml install-config.yaml.tmp

Create the default openshift manifests. We’ll need to copy the antrea operator manifests to the openshift default manifest directory.

πŸ”Έ Your install-config.yaml must be correctly configured for this step to work.

  $ openshift-install create manifests --dir .

Now we have several files on our antrea-ocp directory:

  $ tree

  β”œβ”€β”€ 1-ocp-deploy-with-nsx-antrea
  β”‚Β Β  β”œβ”€β”€ operator_image
  β”‚Β Β  β”‚Β Β  β”œβ”€β”€ antrea-operator-v1.13.1_vmware.1.tar.gz
  β”‚Β Β  └── operator_manifests
  β”‚Β Β      β”œβ”€β”€ deploy.tar.gz
  β”œβ”€β”€ 2-ocp-nsx-integration
  β”‚Β Β  └── interworking
  β”‚Β Β      β”œβ”€β”€ interworking-ubi-0.13.0_vmware.1.tar
  β”œβ”€β”€ install-config.yaml.tmp
  β”œβ”€β”€ manifests
  β”‚Β Β  β”œβ”€β”€ cloud-provider-config.yaml
  β”‚Β Β  β”œβ”€β”€ cluster-config.yaml
  β”‚Β Β  β”œβ”€β”€ cluster-dns-02-config.yml
  β”‚Β Β  β”œβ”€β”€ cluster-infrastructure-02-config.yml
  β”‚Β Β  β”œβ”€β”€ cluster-ingress-02-config.yml
  β”‚Β Β  β”œβ”€β”€ cluster-network-01-crd.yml
  β”‚Β Β  β”œβ”€β”€ cluster-network-02-config.yml
  β”‚Β Β  β”œβ”€β”€ cluster-proxy-01-config.yaml
  β”‚Β Β  β”œβ”€β”€ cluster-scheduler-02-config.yml
  β”‚Β Β  β”œβ”€β”€ cvo-overrides.yaml
  β”‚Β Β  β”œβ”€β”€ kube-cloud-config.yaml
  β”‚Β Β  β”œβ”€β”€ kube-system-configmap-root-ca.yaml
  β”‚Β Β  β”œβ”€β”€ machine-config-server-tls-secret.yaml
  β”‚Β Β  └── openshift-config-secret-pull-secret.yaml
  └── openshift
      β”œβ”€β”€ 99_cloud-creds-secret.yaml
      β”œβ”€β”€ 99_kubeadmin-password-secret.yaml
      β”œβ”€β”€ 99_openshift-cluster-api_master-machines-0.yaml
      β”œβ”€β”€ 99_openshift-cluster-api_master-machines-1.yaml
      β”œβ”€β”€ 99_openshift-cluster-api_master-machines-2.yaml
      β”œβ”€β”€ 99_openshift-cluster-api_master-user-data-secret.yaml
      β”œβ”€β”€ 99_openshift-cluster-api_worker-machineset-0.yaml
      β”œβ”€β”€ 99_openshift-cluster-api_worker-user-data-secret.yaml
      β”œβ”€β”€ 99_openshift-machineconfig_99-master-ssh.yaml
      β”œβ”€β”€ 99_openshift-machineconfig_99-worker-ssh.yaml
      β”œβ”€β”€ 99_role-cloud-creds-secret-reader.yaml
      └── openshift-install-manifests.yaml

πŸ”΄ Pay attention because this is the most sensible part. We’ll need to place some Antrea operator files inside the OpenShift manifest directory.

Uncompress the Antrea Operator deployment manifests:

  $ tar xzvf 1-ocp-deploy-with-nsx-antrea/operator_manifests/deploy.tar.gz


Now our directory should look like this:

  $ tree -d
  β”œβ”€β”€ 1-ocp-deploy-with-nsx-antrea
  β”‚Β Β  β”œβ”€β”€ operator_image
  β”‚Β Β  └── operator_manifests
  β”œβ”€β”€ 2-ocp-nsx-integration
  β”‚Β Β  └── interworking
  β”œβ”€β”€ deploy    <----- ANTREA OPERATOR MANIFESTS
  β”‚Β Β  β”œβ”€β”€ kubernetes
  β”‚Β Β  └── openshift
  β”œβ”€β”€ manifests
  └── openshift

  10 directories

2 files must be customized:

- operator.yaml
- operator.antrea.vmware.com_v1_antreainstall_cr.yaml

In these files, enter the URL of the docker registry where the images were uploaded.

  $ grep -A3 containers deploy/openshift/operator.yaml
          - name: antrea-operator
            # Replace this with the built image name
            image: ReplaceOperatorImage

  $ grep antreaImage deploy/openshift/operator.antrea.vmware.com_v1_antreainstall_cr.yaml
    antreaImage: ReplaceAntreaImage

They should look like this:

  $ grep -A3 containers deploy/openshift/operator.yaml
          - name: antrea-operator
            # Replace this with the built image name

  $ grep antreaImage deploy/openshift/operator.antrea.vmware.com_v1_antreainstall_cr.yaml

Copy the Antrea operator manifest files to OpenShift’s manifests directory.

  $ pwd

  $ ls -l
  total 16
  drwxr-xr-x. 4 root root   71 Apr 19 11:14 1-ocp-deploy-with-nsx-antrea
  drwxr-xr-x. 3 root root   43 Apr 19 11:18 2-ocp-nsx-integration
  drwxr-xr-x. 4  201  201   41 Mar 25 23:29 deploy
  -rw-r--r--. 1 root root 4747 Apr 19 14:03 install-config.yaml.tmp
  drwxr-x---. 2 root root 4096 Apr 19 14:12 manifests
  drwxr-x---. 2 root root 4096 Apr 19 14:12 openshift

 $ cp deploy/openshift/* manifests/

It may be a good idea to make a backup of the “antrea-ocp” directory before attempting the cluster installation.

  $ cd ..
  $ tar cpzvf antrea-ocp.tar.gz antrea-ocp/deploy \
    antrea-ocp/manifests antrea-ocp/install-config.yaml.tmp

OpenShift Deployment

Now all you have to do is install the OpenShift cluster, taking care to point it to the directory where all the files are located:

  $ cd antrea-ocp
  $ openshift-install create cluster --dir . --log-level debug

And the OpenShift cluster is live using CNI Antrea. Nice! 😎

  INFO Checking to see if there is a route at openshift-console/console...
  DEBUG Route found in openshift-console namespace: console
  DEBUG OpenShift console route is admitted
  INFO Install complete!
  INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/root/antrea-ocp/auth/kubeconfig'
  INFO Access the OpenShift web-console here:
  INFO Login to the console with user: "kubeadmin", and password: "BgitS-SrrMS-qehQz-haqjx"
  DEBUG Time elapsed per stage:
  DEBUG      pre-bootstrap: 27s
  DEBUG          bootstrap: 13s
  DEBUG             master: 18s
  DEBUG Bootstrap Complete: 14m29s
  DEBUG                API: 2m7s
  DEBUG  Bootstrap Destroy: 43s
  DEBUG  Cluster Operators: 12m52s
  INFO Time elapsed: 29m10s

Antrea creates several resources on the OpenShift cluster.

Antrea Services

Antrea PODs

Antrea CRDs


Antrea makes the internal OpenShift networks routable. πŸ”₯

β˜‘οΈ This behavior is already known to those who use MetalLB in Baremetal implementations.

Internal Access

As for an example, in this lab the OpenShift cluster uses the following networks:


  - cidr:


    - cidr:

Your firewall will probably block these communications (like mine).

Firewall Blocks

And your cluster will have trouble communicating on the local network.


Because the internal networks were routed through Antrea, the machineNetwork will try to reach clusterNetwork on TCP ports:

* 8443
* 8080
* 8181

They must be allowed on firewall.

Alowed TCP ports

⚠️ Probably other ports needs to be allowed. Monitor your firewall logs to spot them.

That’s it. I don’t know if anyone will want to use this solution. But that’s how it’s done.

Good luck with your implementation! πŸ˜†

May The Force Be With You

Did you like the content? Check out these other interesting articles! πŸ”₯

Could you help?

Please support this content by clicking on one of our advertisers’ banners. ❀️

comments powered by Disqus