The 6th release of ONAP, Frankfurt, got released today. Four things jump out at me:

• Greater commercial activity
• Increased stability/security
• Comprehensive 5G support
• Expanded Kubernetes (K8s) cloud support

Greater Commercial Activity

The biggest danger for an early stage open source project is whether it reaches escape velocity before the novelty wears off. For ONAP, this has frankly been something I have been worried about the most. The current release has been quite encouraging on this front. This table shows the commercial activity around all or a subset of ONAP projects:

Customers in production/ pre-production7Customers in PoC/Pilot7Customers Contributing5Vendors with a Pure-Play Product3 (including Aarna)Vendors Using Parts of ONAP for a Product7Vendors Providing ONAP Services7

Are we at escape velocity with ONAP? You tell me :).

A related point, Bell’s use of ONAP has thus far been shrouded in mystery. As of today this has changed. Now you can now find out exactly what Bell is using ONAP for, how they put it into production, and what value they are getting in this recent customer case study by LF. More in perhaps a subsequent blog.

Increased Stability/Security

The Orange OPNFV crew took over the ONAP Integration project. They have years of deep experience on CI, automated testing, test tooling, and more. This experience has directly benefited ONAP. ONAP now has something called patch gating, where every path submission triggers an ONAP deployment followed by a set of automated tests. This increases the velocity of the project as people are not second guessing each other’s contributions. And it clearly increases stability. There are numerous other security enhancements also. Net-net the confidence you should have in putting ONAP into production continues to go up.

Comprehensive 5G Support

With support for full 5G orchestration, end-to-end slicing, Self Organizing Network (SON) enhancements, Fault Management/Performance Management (FM/PM) collection, close cooperation with 3GPP and the O-RAN Alliance Software Community, Configuration Management (CM) over NETCONF/RESTCONF/REST, a Configuration & Persistency service, and robust PNF support, is there a reason to not use ONAP for 5G?

Expanded Cloud Native Support

As a company we think 5G and edge will exclusively be Kubernetes based (you can still have a few VMs supported via technologies such as Kubevirt and Virtlet). For this reason, I’m super excited to see the various enhancements in the K8s plugin of the MultiCloud project that make it easier to register multiple K8s cloud, define composite applications (built using CNFs and Cloud Native Applications or CNAs) that span clouds, and orchestrate these composite applications with a single click based on intent. Orchestration, day 0 configuration, networking aspects are all taken care of. Then for ongoing lifecycle management and day 1, 2 configuration, we can cut over to existing ONAP controllers and CDS. For slicing, we can use SO+OOF+UUI and the new slicing GUI/models/workflows.

You are probably getting a sense for our roadmap — we are furiously working on ANOD 5.0 that will be our first cloud native 5G/edge orchestration, lifecycle management, and automation product. We will also build a network slice manager on top.

In the meantime:

• End users can try out Aarna.ml ONAP Distribution (ANOD) 4.0 based on the previous El Alto release. You can use it to start your disaggregated open source cloud native 5G journey. In fact, we are a part of the 5G Cloud Native Demo where we used ONAP to onboard a cloud native 5GC and orchestrate it onto a Kubernetes cloud. See webinar recording here.
• xNF vendors can try interop testing with our OVP-in-a-Box offering. Oftentimes, we find that it’s too much friction for a xNF vendor to install ONAP in-house to simply try out their product with ONAP. For such organizations, we have a SaaS offering of ONAP on GCP. It’s a hassle-free way to try out ONAP. If things go well, you can always bring that instance in-house.
• You can also take our ONAP training. With COVID, these are all virtual. But we’ve been working hard to ensure that the learning outcomes are the same as before. Contact us if you want to take advantage of ONAP training. We have a 15% discount if you purchase before the quarter end or June 30th.

For more information on ONAP Frankfurt, check out the official page or the “What’s New in ONAP Frankfurt” webinar (I’m a panelist). Anything else on your mind? Feel free to contact us.

‍

Aarna.ml ONAP Distribution 4.0 and OVP-in-a-Box.

We announced two new products this week. The first is Aarna.ml ONAP Distribution (or ANOD) 4.0, a commercial distribution of the ONAP El Alto release. If you need more background on ONAP El Alto, view our slides from the "What's New in ONAP El Alto" webinar. In a nutshell, ONAP El Alto is a non-functional release with significant work done on security and stability. Our ONAP distribution provides three value-adds:

• Aarna OOM, a value-add version of the community ONAP Operations Manager (OOM) includes automation via Ansible scripts and a simple GUI
• Commercial support at two levels: 8x5 or 24x7
• 100% pure play open source, so that you never have to worry about vendor lock in

Unlike previous releases, we waited to deploy ANOD 4.0 with real paying customers before making it generally available. All three initial customers are happy with the results so far. One of them is John Hopkins University Applied Physics Lab that is doing research on 5G. More on that collaboration in a future blog. See the product page for more details.

________________________________________________________________________________________________________________________________________________________________________________

We also announced another product called OVP-in-a-Box. This is a hosted (or a SaaS) version of ONAP meant to help xNF vendors establish interop with ONAP.

A bit of background first. The Linux Foundation has created an interop program called the OPNFV Verification Program or OVP that gives out badges to products that pass the program's compliance and validation tests. There are two versions of OVP — NFVI and VNF. Here we are talking about the VNF OVP program which is used to establish interop with ONAP.

The issue with OVP is that xNF (xNF signifies VNF, PNF, or CNF) vendors need to have a running instance of ONAP for OVP testing. We have found that many xNF vendors do not have the internal knowledge or skills to deploy ONAP, Moreover, even if they got deployment help from someone like us, they initially want to try out ONAP without actually installing it in-house. For these customers, our OVP-in-a-Box offering will reduce the friction of trying out ONAP by providing xNF vendors:

• A running instance of ONAP on Google Cloud
• Training and support for running OVP tests

If you are interested in getting a demo or trying out either of these products, feel free to contact us!

I'm a presenter for a Linux Foundation webinar next week on the topic of "Integrating ONAP with a 5G Cloud Native Network" on May 19, 2020.

First some history — multiple members of the OPNFV community put together the original end-to-end 5G Cloud Native demo which was shown at Kubecon in November 2019. The demo was received with tremendous excitement as it was the first truly cloud native 5G demo.

‍

One key component missing from that demo was automation. To help with this gap, we (Aarna.ml) decided to contribute to the effort by integrating ONAP with the demo. By using a subset of ONAP, we will be able to fill gaps around orchestration, lifecycle management, monitoring, and closed loop automation over time. See details on this effort here. However, since the vRAN portion of the original demo had some stringent hardware requirements in terms of FPGA boards and the need for a faraday cage to not interfere with licensed spectrum, we decided to simplify the problem and focus only on the 5G Core (5GC) instead for now.

We are happy to show the first version of this demo next week. We will show ONAP onboarding the Altran 5GC and then orchestrating it onto Red Hat OpenShift. Instead of a full 5G vRAN, we will use a gNB emulator instead.

Please join us for this exciting webinar and demo!

As one of the main contributors, I'm thrilled to state that the Private LTE/5G Integrated Cloud Native NFV/App Stack blueprint in the Linux Foundation Edge Akraino project got approved today.

Given the opening up of unlicensed/licensed private spectrum all around the world (e.g. CBRS in the US), Private LTE/5G promises to be very exciting market. Six end users (Airbus, Globe, Orange, Tata Communications, T-Mobile, and Verizon), a number of vendors (such as us), and individuals are collaborating on this blueprint demo which  will be created in a completely open manner and will contain, to the degree possible, open source components.

The key components of this blueprint are:

• NFVI hardware: Standard server, switch, storage components
• NFVI software: Kubernetes with OVN (SDN), Virtlet (to run VMs), Multus (for multiple CNI), Istio, and SD-EWAN (to connect an app across clouds). A main component in the NFVI software will also be an open source 5G UPF CNF.
• Orchestrator: ONAP with AF integration, OpenNESS
• Workloads (CNFs): Facebook Magma for vEPC, TIP OCN and Polaris for 5GC
• Workloads (CNAs): We are starting with the applications in the original ICN blueprint, viz. 360° video, EdgeX Foundry, video AI/ML. However, we might change things around to collaborate with other Akraino blueprints such as the 5G MEC blueprint that is working on cloud gaming, HD video, and live broadcasting.

We will first start with Private LTE over CBRS but then quickly move over to Private 5G and edge computing.

As an open source effort, we could always use more help, Please join us if this is interesting!

I'm excited to announce that we have joined the Industry Consortium for the Platforms for Advanced Wireless Research (PAWR) program to both contribute our ONAP expertise and to learn about cutting edge 5G research. The PAWR program is a critical organization for the US government's research initiatives in the field of advanced wireless technologies including 5G networks and beyond.

The PAWR Project Office (PPO) manages the $100 million public-private partnership and oversees the research platforms. The PPO is co-led by US Ignite and Northeastern University. It collaborates closely with the National Science Foundation, the wireless research community, local communities, and industry, in part through the Industry Consortium, in the design, development, deployment, and initial operations of the research platforms. PAWR has one testbed in Salt Lake City that's generally available, and two under construction — COSMOS in New York City and AERPAW in Research Triangle, North Carolina.

The Linux Foundation open source Open Network Automation Platform (ONAP) project is a popular automation software used by PAWR projects. As the leading ONAP distribution vendor, this is the main rationale for us joining PAWR. In addition to working on orchestrating, managing, and automating 5G Core and RAN projects, we are also looking forward to some advanced work around network slicing, O-RAN integration, AI/ML analytics, and management of edge computing applications — all using ONAP. We will contribute and learn at the same time!

If you are involved with PAWR and wish to collaborate with us, feel free to contact us. In the meantime, please join me on May 19th for a Linux Foundation webinar titled, "Integrating ONAP with a 5G Cloud Native Network". We will share the latest progress and a demo on the community based 5G Cloud Native Network + ONAP demo that is being developed in the OPNFV community.

ONAP Components SSL certificate expiry root cause analysis and approaches to resolve it

Recently, ONAP Dublin has been failing due to SSL certificate expiration. This blog discusses the issue, looks at how the community is addressing it, and then offers some possible alternative solutions.

ONAP SSL Certificate expiry issue

‍SSL certificates, generated at the time of building the Docker image, are generally bundled together by ONAP components. These build time SSL certificates (with private & public keys) have default one (1) year validity from the date and time that they are generated. And it is this validity that breaks any and all  running ONAP deployments past the expiration date. Past this date, most components in Dublin throw the following exception.

‍javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

‍Few instances, demonstrating this today are:

• Policy portal console web app fails to authenticate with the Portal application
• DMaaP certificate expiry impacts A&AI, SO and other components

Why can’t we overwrite these SSL certificate files in ONAP?

‍
Overwriting these SSL certificates sounds like an easy fix. However, while we can overwrite these files in case of a simple docker container, we can not do the same  in case of a K8s cluster. This is primarily because of how ConfigMap/Secrets and Volume mount works in the K8s cluster infrastructure. Simply put, K8s does not support binary file mounting or overwriting of the file using ConfiMap.

Below  are the 2 file formats adapted by ONAP components which indicates  the key reason for bundling of these files within Docker image.

• Java Key Store (JKS) format is a binary Java Key Store (JKS) format (Till Java 8)
• PKCS#12 or PFX format is a binary format for storing the server certificates  (From Java 9)

‍

Frankfurt Approach: Deployment time keystore file generation

‍

Adapted by DCAE components, ONAP Frankfurt generates keystore files at the time of deployment with 1 year certificate validity. Consequently, one runs into issues especially when one wants to  run a component for more than a year. In which case, the only way out is to redeploy the affected components.

Does ONAP support importing enterprise specific SSL certificates?

‍
At present there is no direct solution, since all SSL certificates use ONAP domain name
as a key alias and this requires additional work for each customer.

Potential Solutions

‍
K8s persistent volumes
We can consider leveraging K8s persistent volumes over NFS share. For each ONAP application, we can pre-create the NFS share path and appropriately mount it with the local application path.

For example, ONAP policy PODs store the SSL keystore under a local folder /opt/app/policy/etc/ssl,. We should mount it to an NFS share /onap-ssl-cert-store/policy/ssl to avoid the certificate expiry issue.

The flip side of this approach is that it opens up security holes because if someone were to gain unauthorized access to this NFS share, it would compromise the entire ONAP system.

Base64 encoded ConfigMaps/Secrets
In this approach, we can base64 encode the SSL keystore files and mount it as a K8s volume. The base64 data can be decoded to generate the keystore file at the time of deployment:

• Identify and overwrite the Docker entry point scripts to decode the base64 ConfigMap/Secrets volume.
• Install base64 decoding CLI or scripts, in case the docker image does not have one already.

Build docker container images and host them under separate tagsIn this approach, we need to set up a CI/CD pipe-line to build the target K8s service Docker images to inject the SSL keystore with custom or customer provided keys.

Pros of this approach are:

• Will protect and secure ONAP components by locally hosting a Docker registry within the private network.
• Make code security scanning and auditing more transparent and compliant with enterprise wide security principals

The only caveat with this approach is that the Docker image maintainability becomes very challenging when we want to roll out upstream fixes.

Conclusion

‍
Addressing ONAP SSL certificates is one part of the puzzle and having an enterprise ready ONAP deployment to get early adopters to put ONAP into production. We at Aarna.ml are working actively on this topic to greatly simplify ONAP enterprise adoption in addition to topics such as backup, restore and upgrade capability.