Acentury

The typical mobile network is more complicated today than ever before. More frequency bands are in use with different propagation characteristics, multiple radio access technologies used simultaneously, all to deliver an expanding list of innovative user services. This means combinations to test are expanding. How can operators adopt their testing and deployment processes to manage this change? First, we need to look at how things are generally done today. Mobile network operators spend a great deal of time and money testing their networks and services to ensure that they meet their customers expectations. One of the ways they do this is with an RF lab.

When it comes to effective network and device testing, the first thing to recognize is that the end user is the ultimate tester, and their perception of the service is the ultimate measure. That in turn depends on the correct configuration and operation of the entire network.

It’s the role of the operator to make sure that each element of the network contributes towards a positive customer experience. Many factors determine the proper operation of the network. These include:

• Split possibilities to the RRUs

• Propagation characteristics of the RF channel

• The user equipment itself

The operator must perform many tests in a controlled environment before releasing a given configuration into production. The key question is: How do we balance time cost, resources, and quality in testing to ensure the best end-user experience?

The Problem with Wireless Lab Solutions Today

Many operators rely simply on testing parts of the network in isolation. This is useful for certain cases where problems can typically occur (i.e., key interface points), but doesn't really represent the reality of the service being delivered. Some operators rely heavily on emulation, which is also part of a reasonable testing approach. In essence, this is simply a UE talking to something pretending to be a network, or vice versa. However, emulation doesn't reflect the actual real-world environment, even when done at scale. There are simply too many things still missing from the model.

More and more operators are now creating testing environments that mirror the real world to evaluate what the end user will experience. This requires the most gear and human resources, but results in the most accurate testing. This process is also typically repeated, with more time invested in the changeover between tests before engineers deploy proposed changes onto the network. This method is therefore only effective in cases where the number of potential permutations is small, and the rate of change is low.

However, telecommunication networks are becoming more like IT networks for almost everything except the physical layer. The potential for virtualization means that there are vastly greater numbers of potential configurations, and the rate at which these configurations change is much faster. This leads to a huge increase in the number of possible scenarios to test and a much shorter time frame in which to test them.

Prior to Open RAN, the operator could rely on their infrastructure vendor partner(s) to manage much of this complexity with purpose-built highly integrated network projections. With Open RAN, the operator becomes the integrator and is therefore responsible for ensuring that different vendors hardware and software work together to deliver their service. MNOs may not have the skills necessary to handle this new reality.

To thrive today, mobile operators will have to change how they test and deploy their networks. They will need new tools and new processes to do it. At Acentury, we think there are lessons to be learned from cloud native companies who have already faced these challenges in the past.

DevOps and CI/CD for Carriers

In the world of IT, a practice known as DevOps is used to closer associate the roles of the development and operation to produce quality outputs more effectively. The idea is that if these two groups are closely integrated, communications will be further streamlined and there will be less opportunity for mistakes. Within DevOps there's related processes known as “Continuous Integration” (CI), which refers to the automation of the creation and testing processes, and “Continuous Deployment” (CD) which refers to the automated packaging and delivery of the product. How can they be applied to the world of wireless networks with real bits of infrastructure, and how can traditional mobile operators benefit from them?

CI/CD bridges the gaps between development and operation activities by using automation in the building, testing, and deployment processes. It compiles the incremental changes made by developers, packages them into deliverable automated tests, verifies the functionality, and automated deployment services deliver them to the end users. The aim of CI/CD is to increase early defect discovery, increase productivity, and provide faster release cycles. This process contrasts with traditional methods where a collection of updates are integrated into one large batch before being tested and deployed together in a new version.

Organizations that use DevOps have clearly seen significant quality improvements. This results from a stronger testing regimen that catches more defects and a short cycle time between releases. However, the real advantage is the close collaboration between the development and operation functions; there are simply fewer opportunities of mistake to arise when people work closely together. The continuous nature of the process means that the development and deployment cycle time is much faster.

The automation of the more mundane tasks leaves more time for value-added work. Staff can focus their time on innovation and customer response instead of manually running test scripts, which is a significant upside in a wireless market.

How can we apply this concept to testing and deployment of things that are not virtual? How does it apply to testing and deploying of 5G RAN?

The 5G core and upper layers of the RAN can be virtualized or containerized - the DevOps concept applies well there, but what about physical things like RRUs and the mobile devices at the other end of the mobile network? There is a need for a tool that can tie all the elements in the lab together to automate the entire process.

Automating Your Wireless Lab: The Mirror Lab

To address this need, we propose the concept of the “Mirror Lab”. The Mirror Lab reflects the production network in the lab environment, and the equipment configured the same way. This means that any testing done in the lab is directly applicable to the production environment. A properly equipped Mirror Lab includes an orchestration layer to manage and control the testing setup execution, test result valuation processes, and resources.

That’s not all. The other essential part of a complete Mirror Lab is a link to the production network to enable a deployment mechanism for automatic rollout of any validated changes once testing is complete and the results are ideal. The Mirror Lab will release the validated configuration up to the production network, where it is automatically deployed.

In summary, the mirror lab blends the lab and production environments together isolating them for tests and integrating them for deployment. This way, it models the real world. The subscriber device sees in the lab what it will see in the field. It enables more sophisticated test scenarios like roaming and handover but with the actual radio access technologies, frequency bands, vendor equipment, and rf site designs that exist in the field including specific MIMO and carrier aggregation configurations.

A Mirror Lab allows for the optimal use of test resources as well, whether those are physical equipment like RRUs, software containers running various network functions, or even people. Everything is coordinated and automated, it is efficiently shared so there is no wasted time caused by resource blocking. The automation aspect reduces the time-to-market because testing is much quicker. The tests can be run faster, more consistently, and more often by software than any human could ever execute them. The tests are perfectly repeatable thousands of times in a way that isn't possible manually, so the confidence in the result is also much higher.

The automation of the deployment function takes the more time-consuming manual steps out of that practice as well so there is a reduced opportunity for human error in the deployment process. The odds of a mismatch between lab and production therefore become virtually zero.

The Elements of a Mirror Lab

End-to-End Network: This can be a subset of the product production environment, but it should be a representative such as every combination of RRU vendor and/or frequency allocation.

The concept here is simple: use your own network to test your network. Use actual RAN and subscriber equipment because it is the best way to understand what will really happen over a period. Focus on end-to-end testing of the real thing as much as possible.

The Orchestration Layer: This is a key feature of the Mirror Lab. This layer controls and coordinates all the equipment and resources in the lab. The Orchestration Layer ensures all the resources work together seamlessly to test and deploy. It is also used to share resources simultaneously among multiple testers, who can utilize the resources from virtually anywhere.

Test Automation: The ability to run tests and collect logs at scale with limited human involvement is a key element of the CI/CD process.

To facilitate test automation, each resource in the system is controlled by the orchestration layer and configured according to the scenario being tested. Such configurations can include higher levels of RAN and CUD splits, single signal generators, or emulators. The RAN is configured just like the production network and the devices under test are set up to make various call types.

The Orchestration Layer then controls the rf switch matrix to execute the tests without human involvement. The various call types are automatically started on the device under test and the RAN equipment is broadcasting signals into the rf switch matrix by controlling the arrow switch matrix. The various call types are made under different rf scenarios, each with different orders of MIMO and carrier aggregation. Changing which signals are combined duplicates MIMO and carrier aggregation and changing the attenuation of the signal paths mimics handover between sites and roaming. These can be perfectly repeated as many times as necessary, and each test is just like the previous.

Deployment Automation: The ability to copy a successfully tested configuration from the test environment to the production environment is what differentiates this approach from other testing concepts. This saves time and minimizes the chances of production being out of sync with the lab.

Once tests have been completed and passed the orchestration layer can send a validated configuration to the production network via an API (an API for each vendor is required). This closes the CI/CD loop so the process can begin again. This saves the effort and time of manually uploading the new configuration and reduces the opportunity for human error if there are multiple infrastructure vendors. Another key benefit of this API link to the production environment is that configurations can be recalled from the real world to the lab for further testing. Scenarios can be recalled and automatically set up in the lab for evaluation. Hundreds or even thousands of regressions to be run to reproduce, and root cause any issues.

How LAMTA Can Automate Your Wireless Lab

When we come across this need for a Mirror Lab at Acentury, we use our LAMTA solution. It is used by multiple operators and equipment vendors to run their rf testing operations. LAMTA is an orchestration platform that applies the concept of DevOps and CI/CD to the testing and deployment of 5g RAN. It enables end-to-end testing of the entire system using a combination of RAN devices under test and emulation. With LAMTA, lab resources can be shared among multiple users simultaneously in different locations, it provides a management platform to plan and track testing activities and provides the automation platform test setup. The latter feature takes minutes instead of hours or even days and testing can be repeated hundreds to thousands of times since there's no human involvement in the testing. The tests themselves are perfectly repeatable which means higher confidence in the results.

Once the testing is complete, LAMTA can deploy the validated configuration directly to the production network closing the CI/CD loop. This eliminates a lot of tedious manual work, reduces the chance of human error, and reduces the chance that production doesn't match what was validated.

Overall, LAMTA integrates the lab and production environments together to allow seamless testing and deployment as part of the same system workflow. It enables faster and more sophisticated testing and rapid deployment at scale.

Interested in learning more about how LAMTA can augment your wireless lab? Take a look at LAMTA's features, contact us directly, or have us contact you.