Debugging ONAP Series: SSL Certificates

ONAP Components SSL certificate expiry root cause analysis and approaches to resolve it

Recently, ONAP Dublin has been failing due to SSL certificate expiration. This blog discusses the issue, looks at how the community is addressing it, and then offers some possible alternative solutions.

ONAP SSL Certificate expiry issue

‍SSL certificates, generated at the time of building the Docker image, are generally bundled together by ONAP components. These build time SSL certificates (with private & public keys) have default one (1) year validity from the date and time that they are generated. And it is this validity that breaks any and all  running ONAP deployments past the expiration date. Past this date, most components in Dublin throw the following exception.

‍javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

‍Few instances, demonstrating this today are:

• Policy portal console web app fails to authenticate with the Portal application
• DMaaP certificate expiry impacts A&AI, SO and other components

Why can’t we overwrite these SSL certificate files in ONAP?

‍
Overwriting these SSL certificates sounds like an easy fix. However, while we can overwrite these files in case of a simple docker container, we can not do the same  in case of a K8s cluster. This is primarily because of how ConfigMap/Secrets and Volume mount works in the K8s cluster infrastructure. Simply put, K8s does not support binary file mounting or overwriting of the file using ConfiMap.

Below  are the 2 file formats adapted by ONAP components which indicates  the key reason for bundling of these files within Docker image.

• Java Key Store (JKS) format is a binary Java Key Store (JKS) format (Till Java 8)
• PKCS#12 or PFX format is a binary format for storing the server certificates  (From Java 9)

‍

Frankfurt Approach: Deployment time keystore file generation

‍

Adapted by DCAE components, ONAP Frankfurt generates keystore files at the time of deployment with 1 year certificate validity. Consequently, one runs into issues especially when one wants to  run a component for more than a year. In which case, the only way out is to redeploy the affected components.

Does ONAP support importing enterprise specific SSL certificates?

‍
At present there is no direct solution, since all SSL certificates use ONAP domain name
as a key alias and this requires additional work for each customer.

Potential Solutions

‍
K8s persistent volumes
We can consider leveraging K8s persistent volumes over NFS share. For each ONAP application, we can pre-create the NFS share path and appropriately mount it with the local application path.

For example, ONAP policy PODs store the SSL keystore under a local folder /opt/app/policy/etc/ssl,. We should mount it to an NFS share /onap-ssl-cert-store/policy/ssl to avoid the certificate expiry issue.

The flip side of this approach is that it opens up security holes because if someone were to gain unauthorized access to this NFS share, it would compromise the entire ONAP system.

Base64 encoded ConfigMaps/Secrets
In this approach, we can base64 encode the SSL keystore files and mount it as a K8s volume. The base64 data can be decoded to generate the keystore file at the time of deployment:

• Identify and overwrite the Docker entry point scripts to decode the base64 ConfigMap/Secrets volume.
• Install base64 decoding CLI or scripts, in case the docker image does not have one already.

Build docker container images and host them under separate tagsIn this approach, we need to set up a CI/CD pipe-line to build the target K8s service Docker images to inject the SSL keystore with custom or customer provided keys.

Pros of this approach are:

• Will protect and secure ONAP components by locally hosting a Docker registry within the private network.
• Make code security scanning and auditing more transparent and compliant with enterprise wide security principals

The only caveat with this approach is that the Docker image maintainability becomes very challenging when we want to roll out upstream fixes.

Conclusion

‍
Addressing ONAP SSL certificates is one part of the puzzle and having an enterprise ready ONAP deployment to get early adopters to put ONAP into production. We at Aarna.ml are working actively on this topic to greatly simplify ONAP enterprise adoption in addition to topics such as backup, restore and upgrade capability.