Network Capacity Management Using Proactive Network Maintenance
You probably know that Proactive Network Maintenance (PNM) is about finding and fixing problems before they impact the customer. But the other side of PNM is about managing the capacity or bandwidth available in the network. PNM may have started with the former concept in mind, but the latter is becoming more important as we rely on higher amounts of capacity at the edge. As the world adjusts to life under the COVID-19 pandemic, access network capacity is becoming even more critical. PNM is an important tool set for network capacity management, and CableLabs is helping operators manage network quality and capacity together.
Network condition impacts network capacity. Network impairments, a broad class of failures and flaws in the ability of a network to carry data, have to be addressed before they lead to service failure. The DOCSIS® protocol is a method for sending data over multiple radio frequencies in hybrid fiber coax networks, and comes with several resiliency mechanisms that help service continue in spite of impairments, to a point. These impairments in the cable plant may impact a few or all frequencies. Impairments that impact specific frequencies may or may not be able to be compensated for, on those frequencies. If severe, the impairment may impact the data carried on those frequencies entirely, leading to correctable or even uncorrectable data errors. If not severe, profiles may be able to adjust to lower modulation orders to allow less data to be reliably carried than otherwise. Impairments that impact a larger amount of frequencies of course have a greater impact on the bandwidth the network can carry. In any case, impairments impact the capacity that the operator can get from the access network.
This is why PNM, which is an important set of tools for network operators to manage network condition, becomes even more important as we depend more on our network capacity and move toward higher utilization of the access network capacity. As demand for bandwidth increases at the edge, PNM becomes an important network capacity management tool for network providers. The difference between a perfect network and one with flaws felt by customers begins to shrink. PNM begins to be an imperative; it is “table stakes” for maintaining communications services and managing the capacity of the network.
CableLabs has been working with these issues in mind for some time. In July of 2019, I wrote on the subject of 10G and reliability, pointing out that higher bandwidth solutions closer to the customer will be required for 10G. Then, in August, I wrote on the subject of reliability from a cable perspective and pointed out that the impairments addressed through PNM impact capacity. So, we see that reliability and network capacity are closely coupled. As we move toward higher bandwidth services, expand the utilization of frequencies and further push the limits of technology, reliable and sufficient bandwidth become highly coupled. Therefore, so do the tools that network providers use to manage these service qualities. CableLabs is working on solutions to help operators succeed in this reality.
Recently, CableLabs announced the release of a new capability in our Proactive Operations (ProOps) platform that uses RxMER per subcarrier and profile information to inform the selection of PNM opportunities. Also, our PNM working group announced the release of our DOCSIS 3.1 PNM primer of engineering practices, which we intend to develop toward best practices for the industry. If you are an operator or vendor interested in this subject, contact us for more information and to help us develop this solution for the industry.
 Because of the resiliency of DOCSIS® technology, impairments in the network have an impact on the capacity available in the network for serving customers, even when service remains functional, and even when customers may not notice right away or always. Without resiliency, an impairment leads to failure. Network resiliency is what keeps service running over impairments, which lets operators fix problems before they become severe and provide highly reliable services.
Proactive Network Maintenance (PNM): Are You InGeNeOS?
I love a good acronym! InGeNeOS™ is an acronym built from Intelligent General Next Operations Systems. It’s the name of a CableLabs working group that solves Proactive Network Maintenance (PNM) issues for the cable industry, and it might be for you.
What’s So InGeNeOS about PNM?
The InGeNeOS group focuses on discussing, inventing, building and sharing network operations tools and techniques from the data made available from DOCSIS® systems, including the CM, CMTS and test devices. Other CableLabs working groups focus on DOCSIS specifications, and the SCTE Network Operations Subcommittee Working Group 7 focuses on network operations training material. The InGeNeOS group connects these two worlds and turns the network information into capabilities that engineers and technicians can use to maintain services. We turn DOCSIS system information into solutions that identify, diagnose and sometimes automatically correct network problems—often before the customer notices. When these tools get good enough, they can become proactive. Thus, we often refer to this group as the PNM Working Group (WG). See why we put it into an acronym?
Don’t Just Think—Do!
This group doesn’t merely ponder PNM solutions; it is very active in several ways:
- Developing best practices for PNM solutions—We just started an effort to document PNM best practices in a DOCSIS 3.1 environment.
- Guiding specifications development for emerging technologies—For example, although Full Duplex (FDX) DOCSIS technology is not yet deployed, we know it must be fully ready when it is, and that includes being operationally supportable.
- Sharing experiences, both problems and solutions—Many working group participants work maintenance problems at operator companies, or for operators, so they bring problems to the working group to get ideas for causes and resolutions.
- Testing theories in the lab—Once we develop theories about the causes of problems in the field, we reproduce the theorized conditions in the lab to confirm the cause. We can also calibrate measurements, test methods for detection and develop new PNM tools and methods based on these tests.
This developing, defining, knowledge sharing and testing help operators reduce costs and improve service reliability by improving their network maintenance operations. All these are just examples of what we do. If you have ideas that might fit within this framework, keep reading.
So You Think You’re InGeNeOS?
Operators in—and vendors supporting—the cable industry can easily benefit from joining the InGeNeOS group:
- If you are a cable operator and a CableLabs member, consider this your invitation to join.
- If you are a cable operator but not a member, this is a very good reason to become a member.
- If you are a vendor, all you need to do is sign the NDA and IPR.
In any case, contact Jason Rupe to join the InGeNeOS group.
Proactive Network Maintenance: Where Is Common Data Collection Going Next?
Welcome to 2019, and the future of Proactive Network Maintenance (PNM)! As CableLabs’ PNM solutions mature, we look toward developing a framework for PNM that everybody can use.
On December 21, 2018, CableLabs released the Combined Common Collection Framework (XCCF) version 2.0. This new version includes several enhancements, including refactoring to make it easier to work with and tuning to support new use cases. If you are in the PNM club and receive the monthly reports, you’ve seen some of these enhancements over the past 6 months:
- In July, the XCCF team solved many encountered cable modem (CM) stability issues associated with polling. We also created methods to purge and maintain files in the cache, and to allow better load management.
- In August, we added additional scalability enhancements, multiple formats for outputs, listing abilities, read or write (RW) and read-only (RO) strings, and we introduced code to support the new 10th PNM measurement.
- As of September, the XCCF includes a data archiver function to allow users to manage and save space on the application program interface (API).
- As of October, the XCCF includes a method to manage host machine performance, as well as the ability to separately manage read and write strings. We have also significantly optimized query time and further hardened and enhanced it for greater performance as part of the Gridmetrics™ project.
- In November, the XCCF team added improved CM polling, a few back-end improvements, a CM events driver and better spectrum analysis handling in a new driver.
- In December, final touches to version 2.0 of the XCCF included an ability to receive traps from CMs and converged cable access platforms (CCAPs), and store them. A considerable amount of code was refactored to decouple data storage from the rest of the code so that users can swap out the data storage with their favorite database or alternative method.
CableLabs is using the XCCF for several use cases, including PNM, Gridmetrics™, Profile Management, LTE ingress detection and general network monitoring needs. We envision several use cases beyond those already identified and have demonstrated effectiveness for many of these:
- CM testing: This is the purpose of the Cable Modem Validation Application (CMVA). We found it useful for validating chip and CM builds for PNM, for modem testing in an operator’s lab setting, and for streamlining testing such as the acceptance test plan (ATP) for certification. We also found it useful for the next use case.
- Developing PNM methods: The convenience of the CMVA, coupled with XCCF, makes it useful for a lot of basic data visualization needs, including experimenting with detecting impairments. The PNM working group used it to capture output from testing while reproducing observations from the field and discovering the effectiveness of some measurement methods. CableLabs is using it to guide our development of the next use case.
- PNM ApplicatioN Environment (PNMANE): CableLabs will be developing an environment for PNM applications that sit on top of XCCF.
- Gridmetrics™: XCCF collects machine data, so why not collect voltage measurements from power supply modems? We did exactly that, and it works very well. Now, operators can use their network to monitor power levels they receive, and thus provide out-of-band measurement of the power grid to support power companies.
- Profile management: The profile that a CM uses dictates much of the quality of experience that a customer enjoys. XCCF collects the information needed from the network to enable optimizing the profiles that the CMs use, thus enhancing service performance and experience.
- LTE ingress detection: Spectrum analysis and receive modulation error ratio (RxMER) data, as well as other PNM measures, can reveal ingress, including LTE. CableLabs has a method for detecting these issues, which can inform not only PNM activity but also the placement of small cells, and report on overall network health for such an endeavor.
- General network monitoring and operations: Northbound data collection on any network device is possible in the same manner as supported by XCCF. A whole world of possibilities exists here. Fortunately, the XCCF is suited to evolve to new methods of machine data collection and is set to protect the network from over-polling, too.
- Service protection automation in Software Defined Networking (SDN): Many SDN solutions lack access to the network data required to adjust network conditions for an automated response to network impairments and other conditions. The XCCF can provide useful information northbound to applications that can protect against failures, develop traffic engineering solutions or analyze and define solutions for network risk.
We expect use cases to drive how we support the XCCF into 2019, and this is where you all come in.
Please comment below or contact us to share your use cases and experiences with using the XCCF and CMVA, or where your PNM needs are headed. Many operators and vendors have requested access to download and install the XCCF, as well as CMVA, and we hope and expect that everyone has been getting good use out of them.
CableLabs needs feedback to continue supporting projects, so we need your input. Please let us know how you’ve been using the XCCF or CMVA, particularly how you’ve gained value for your company or the industry overall. Anything you can share is helpful, no matter how brief or minor you might think it is. Your input will help drive future development of XCCF, CMVA and our current efforts, including PNMANE.
PNM Series: The Business Case for a Common Collection Framework
This is the second in our series on Proactive Network Maintenance (PNM). If you missed our introduction to PNM, you can check out the first entry which explains some background on the subject.
PNM is our CableLabs project focused on assuring cable service provider companies can maintain the network at a level of quality so that major impacts to service are avoided. The proactive part means the maintenance happens before the customer’s service is impacted. But, to do this well, a service provider must collect data from the network. However, collecting data from the network in a way that doesn’t impact service is not easy.
What is the Common Collection Framework?
The Common Collection Framework is a set of Python software modules that handle the task of collecting PNM data from the network elements and presenting the data to PNM applications. It provides the data in a common form so that software applications don’t have to talk network language to get the data it needs. It also protects the network from overly frequent data requests, which can impact service.
CableLabs created a DOCSIS® Common Collection Framework (DCCF) and a Wi-Fi Common Collection Framework (WCCF). We have also started the creation of an optical-centered collection framework. We may even create an in-home wired (MoCA) framework if members express the need. To keep the usage model simple, CableLabs intends to join these frameworks into a combined Common Collection Framework (XCCF). Because cable services are provided over a network comprised of many different technologies, CableLabs is making it easy for members to use the right mix of collection frameworks to get data from the right network elements for their needs.
CableLabs recently released an architecture document to the public that describes the DCCF in detail. You can obtain a copy at this link and reference it in your work. The document describes what the DCCF is, as well as the intended architecture for XCCF. There is also a partner document reporting on the Wireless Common Collection Framework, available here.
What’s Under the Hood?
Because the PNM data are presented in the formats presented by the network, existing applications shouldn’t have trouble connecting to the XCCF to obtain its data. Translator software takes the output from the network and gets it ready for applications to use.
Why did CableLabs build it?
A PNM application or program needs data to drive it - obtaining the data required can be a significant request to network elements. Service providers need to know that the network isn’t impacted by PNM requests, so they need some level of control to assure service is the priority. Further, there are potentially numerous PNM applications that need the same data, so having every application impact the network in uncoordinated ways is not efficient, and not necessarily customer friendly. A PNM program that utilizes multiple applications needs a common collection capability to support the applications and relieve the network.
There are clear advantages to using the XCCF to support network operations:
- It provides one polling mechanism to manage, serving all applications.
- Building your own applications, and supporting purchased applications, becomes easier with the XCCF.
- The network isn’t overly taxed with data requests, so it can be ruled out as a cause when there is a problem.
- You get clear separation from the network and the applications, which fits the way operations are usually organized.
- Updating is easy between the applications and the network when you have the XCCF as the point to manage those changes, and XCCF is built to support that.
- XCCF is extensible, and we have loads of great ideas to consider on the roadmap.
- Because XCCF is based on SDN architecture concepts, scaling is understood, and high reliability is supported.
- Because it is accessible by all CableLabs members, any member can use it to test out a PNM capability in a field trial to learn about its benefits to their business.
There is quantifiable business value here too!
- Testing a new PNM capability within operations is easier and more realistic when the data are already presented to the applications in a common way, reducing the uncertainty in the payback of a PNM business case.
- Using the XCCF can streamline implementation of PNM applications in a PNM program, making the business case for PNM pay back faster.
- CAPEX is lower because simpler, cheaper PNM solutions can enter operations and scale better when small applications can be pointed to existing XCCF instances.
- OPEX is lower because applications are separate from the network, and the XCCF interface can be rapidly, easily maintained.
- PNM advantages are achievable because a significant effort in any PNM program is solved with the XCCF. It avoids scaling risks that could otherwise increase the OPEX of a PNM program.
For all these reasons, CableLabs heard from our members that an XCCF capability was needed, so we responded.
Where do I get a copy?
CableLabs members can obtain a copy here. Vendors who are willing to sign the necessary CableLabs agreements can also obtain a copy. We hope our community can contribute feedback, and potentially contribute code as well, to the XCCF. We also look toward the community to drive our roadmap for the XCCF, providing input to what capabilities need to be supported with the highest priorities.
Don't forget to subscribe to our blog to read more about PNM in the future.
Introduction to Proactive Network Maintenance (PNM): The Importance of Broadband
This is the introduction for our upcoming series on Proactive Network Maintenance (PNM).
The advent of the Internet has had a profound impact on American life. Broadband is a foundation for economic growth, job creation, global competitiveness and a better way of life. The internet is enabling entire new industries and unlocking vast new possibilities for existing ones. It is changing how we educate children, deliver health care, manage energy, ensure public safety, engage government and access, organize and disseminate knowledge.
There is a lot riding on broadband service which places a focus on customer service; to create both a faster and more reliable broadband experience that delight customers. Recent technological advancements in systems and solutions, as well as agile development, have enabled new cloud-based tools to enhance customer experience.
Over the past decade, CableLabs has been inventing and refining tools to improve the experience of broadband. CableLabs is providing both specifications and reference designs to interested parties to improve how customers experience their broadband service. Proactive Network Maintenance (PNM) is one of these innovations.
What is Proactive Network Maintenance and Why Should You Care?
Proactive network maintenance (PNM) is a revolutionary philosophy. Unlike predictive, or preventive maintenance, proactive maintenance depends on a constant and rigorous inspection of the network to look for the causes of a failure, before that failure occurs, and not treating network failures as routine or normal. PNM is about detecting impending failure conditions followed by remediation before problems become evident to users.
In 2008 the first instantiation of PNM was pioneered at CableLabs. This powerful innovation used information available in each cable modem and mathematically analyzed it to identify impairments in the coax portion of the cable network. From this time forward, every cable modem in the network is a troubleshooting device and could be used as a preventive diagnostic tool.
This is important when trying to track down transient issues related to the time of day, temperature, and other environmental variables, which can play a huge role in the performance of the cable system. With transient issues, it is important to have sensors continually monitoring the network. Since then, with improvements in technology, more sophisticated tools have been added giving operators unprecedented amounts of information about the state of the network.
Problems are solved quickly and efficiently because we can pinpoint where the problems are. Technicians like PNM because they become empowered to find and fix issues. An impairment originating from within a customer’s home can be dispatched to a service technician. While impairments originating on the cable plant itself can be dispatched to line technicians. Customer service agents also like the tools because they create actionable service requests. Lastly, impairments that can be attributed to headend alignment issues can be routed to a headend technician. All of this can be done before the customer is even aware there is a problem!
So, what does CableLabs have to do with all this?
The PNM project continues to innovate. Because of the success of PNM for the cable network, capabilities have been added to investigate in-home coax, WiFi and soon fiber optic networks. Monitoring is key, and by using powerful cloud-based predictive algorithms and analytics, networks can be monitored 24 x 7 to provide insights, follow trends and detect important clues with the goal to identify, diagnose and fix issues before customers notice any impact.
CableLabs provides a toolkit of technical capabilities and reference designs that interested parties can use to create and customize tools fitting specific business needs. Operators can get started with reference designs, build expertise and their own solutions, and integrate the tools into their own systems. In addition, suppliers have licensed the technology and are creating a turn-key solution that operators can choose to work with.
In my upcoming series, I will cover DOCSIS PNM, MoCA PNM, Optical PNM, Common Collection Framework and explore in greater depth how PNM enhances the customer experience. Be sure to subscribe to our blog to find out more.
Proactive Network Maintenance (PNM): Smarter Technology for Smarter Maintenance
We have always wanted more information about our world: Where is our next meal coming from; what’s the weather like; are my kids OK; did my team win? Past information delivery systems, such as the telegraph, the telephone, and broadcast television, have all been supplemented with the Internet. At a basic level, the Internet exists, in large part, to satisfy our appetite for information.
As Internet providers we need to keep that in mind, and determine when our service is not getting the job done. The term proactive network maintenance (PNM) describes the activities that assure the job of providing information is getting done – hopefully, before the customer becomes aware of any problem.
In the cable space the networks are diverse, expansive and non-obvious. I once heard someone state that anytime a subscriber complained, his company rushed to get a technician out to the house right away. Ouch! Without a network maintenance system, it’s a guessing-game where in the network the fault occurred. Chances are the wrong technician went to a wrong location with a wrong set of skills and wrong tools, and failed to get the repair job done. This is trouble for everybody (except maybe your competition). There will be a call-back, the subscriber is going to be dissatisfied, the technician is going to be judged as a failure by his supervisor, and the network fault continues to cause even more problems with other subscribers.
Cable services are delivered by systems, such as the video system, the DOCSIS system, and the IP phone system. Video systems are somewhat difficult to mine for data because of their proprietary nature, many generations of legacy STBs with differing capabilities, and bandwidth-limited data channels, called Out Of Band (OOB). But there are some good opportunities here.
For example, if a customer is watching a digital TV channel, and the box determines there are uncorrected code blocks, it “knows” that the customer is watching an impaired picture. A higher code block error rate means a worse picture. The customer should not have to get up and make a phone call when a connected box can do that automatically.
The DOCSIS system, on the other hand, has a pretty good set of handles to determine what is going on in the network. Most of the assurance traffic uses SNMP (simple network management protocol) and a fat pipe to relay all that data. Generally the assurance process starts with a lie: The operator asks what the cable modem’s health is, but the cable modem is usually fine – it’s the network that may be sick. And a sick network can affect many cable modems.
CableLabs has a proactive network maintenance effort that is developing systems to report if the network is getting the job done. The effort focuses on developing methods and tools around the existing data, developing new sources of data, and transferring information to member cable operators.
Proactive network maintenance results in faster and more accurate diagnosis of problems, faster repairs, happier customers and lower costs for cable operators.
Tom Williams is Principle Architect with the Network Technologies group at CableLabs. Tom’s main interest is in the physical layer of cable networks, and he enjoys digital signal processing (DSP), RF network design, and testing. He has been a cable engineer for 34 years.
DOCSIS 3.1: Where We Are Today
The NCTA recently announced Gigasphere, which encompasses the new experiences customers will come to enjoy based on the DOCSIS 3.1 technology developed by CableLabs, Inc.
There is a lot of excitement around DOCSIS 3.1, and the new capabilities it will bring to cable networks for cable operators and their customers. Customers will be able to enjoy gigabit speed access on the downstream and upstream, allowing them to download 4K video and ultra-high definition movies faster than ever. Reduced network delay will also take the online gaming experience to a new height, increasing responsiveness and allowing higher resolution graphics. Also, the widespread availability of cable high-speed internet will make the benefits of DOCSIS 3.1 available to over 90% of U.S. households.
Cable service providers, in addition to being able to provide richer experiences to their customers, will have an easier time maintaining their networks due to the Proactive Network Maintenance functionalities embedded into DOCSIS 3.1; thus being able to provide a more consistent and reliable experience to their customers.
Looking back, it has been just over 18 months of intense work, collaboration and innovation across technology vendors and MSOs since we started the DOCSIS 3.1 project, reaching our first public milestone in November 2013 by publishing the first official release of the DOCSIS 3.1 specification – and we are not done yet! We are planning for our first round of equipment interoperability testing later this year, and we expect to see devices ready for certification by the middle of 2015.
Also, CableLabs is working with several MSOs to to make sure the networks are ready when DOCSIS 3.1 devices hit the market. We are engaged in a wide range of HFC network testing activities to prime the network to support the higher capacities enabled by DOCSIS 3.1.
Finally, to increase the ecosystem technical expertise around DOCSIS 3.1, CableLabs will be hosting a DOCSIS 3.1 training session June 2- June 5 for attendees to gain an in-depth understanding of DOCSIS 3.1.
These are exciting times at CableLabs, so much has been accomplished, and there is much more to look forward to!
Stay tuned for further updates ....
For more information, please contact Belal Hamzeh.
Belal Hamzeh is the Director of Broadband Evolution at CableLabs, leading the DOCSIS 3.1 activities, in addition to being the Principle Architect for the DOCSIS 3.1 Physical and RF Layers. He has been heavily involved in developing broadband technologies for wired and wireless networks and variety of applications for over 10 years.