Redundancy Elimination via the Network Edge and Pushes

Timeframe: October 2017 – September 2020

The focus of this project is to explore how we can leverage the intelligent network edge to improve network performance.  In short, this work explores the extent to which we can leverage pro-active pushes, be those pushes being done locally (D2D) or remotely (trusted third party) to a mobile device or the network edge (ex. Mobile Edge Computing) to reduce network consumption. Specifically, this work aims to explore several key approaches that include:

  • Provider Accessible Storage Subsystem (PASS): What happens if we take the spare storage on a device and make it writable (accessible) to a trusted third party?  What if a content provider or middleware could push content to our device or a nearby MEC node to pre-stage content when the network is less busy?  Rather than reacting to content swings, could we trigger rules or choose when to push to the device.  Could we allow the usage to be highly malleable or to operate in a less than best effort manner whereby client usage always takes precedence and the allowed storage is as available?  What clever constructs could we make depending on the extent to which the mobile client / user is aware of such an approach? Are there interesting economic models that could emerge from such approaches?
  • Whirlwind (D2D Exchanges): On the other extreme, what if we allow devices to pro-actively exchange previously seen content?  Could we exchange content via approaches such as Bluetooth, WiFi Direct, or LTE Direct whereby clients agree to share popular content when in proximity?  Leveraging our past work on opportunistic networking that shows reasonable availability, prevalence, and reciprocity, what should be exchanged in terms of content?  Should it simply be hashes which then has rules to trigger when to exchange or should the client pro-actively grab / fetch the objects or streaming blocks?  Should there be a manager or external entity to help coordinate these actions?
  • Hybrid Approach: To what extent could we blend such strategies between the extremes of the push from external parties to the localized D2D exchange?  What might effective strategies be with respect to information exchanges that yield a net positive in terms of network performance versus mobile device energy consumption?


The following individuals are supported / have been supported in efforts related to this effort.

  • Prof. Aaron Striegel, Principal Investigator
  • Shangyue Zhu, Graduate Student – Data Collection, Software Construction, Simulation
  • Alamin Mohammed, Graduate Student – Data Collection, Software Construction
  • Poorna Talkad Sukumar – Graduate Student – Data Collection and Visualization
  • Lixing Song, Graduated with PhD – Data Collection [Rose-Hulman]
  • Xueheng Hu, Graduated with PhD – Data Collection and Analysis [Amazon Lab 126]
  • Zhongying Qiao, Graduate with Masters – Simulation + Data Analysis [Bay Area]

Publications / Datasets

  • We have three datasets (Fall 17, Fall 18, Fall 19) available from captures taken from the WiFi that we provide that the University Development Tent.  The tent serves up WiFi from 4+ 802.11ac access points to several hundred users over the course of several hours.  Taps and instrumentation were set up at various points and the following datasets can be requested for analysis by researchers following the signing of a DUA (Data Usage Agreement).
    • Packet timing / metadata (Layers 2-4) as captured from the wireless controller (inbound / outbound – Internet) which serves as the gateway for the WiFi service via NAT.
    • Accompanying SNMP data (all MAC addresses hashed) for many of the runs taken periodically via snmpwalk across the service period including details with regards to transmit / receive and varying phone models
    • The ability to run redundancy elimination algorithms on payload data (send us a Docker container and we can give you back the output / results)

Major Findings / Contributions

  • Over the course of the last three years, we have conducted redundancy elimination studies on data going to / from the wireless controller gateway at the University Relations (Development) tent.  This tent hosts several hundred individuals to which we offer courtesy high-speed WiFi (802.11ac) during a multi-hour tailgate for the purpose of data analysis as related to redundancy elimination and other research topics.  In earlier work analyzing redundancy as observed both in prior tailgates as well as a commuter train to / from Chicago, we have observed relatively lower levels than expected in terms of redundancy. Most importantly, we have observed further decreases as both video as well as HTTPS prevalence have dramatically redundancy.  Our general conclusion is that passive extraction of redundancy is unlikely to be effective by either in-line elimination or for any gains from D2D-based exchanges.  We are currently prepping a journalized version of that earlier work taking advantage of our multi-year dataset.
  • Generally, D2D-based exchanges are unlikely to be significantly advantageous in practice for the purposes of significant reductions in bandwidth.  The sparsity of content makes it difficult to discern what to share (from a D2D perspective) and further complications from Apple / Google in terms of simple longitudinal app execution make it unlikely to this to be feasibly deployable / realizable.  As such, we conclude that push-based architectures with an eye towards pre-staging (MEC-driven or cloud-driven) for content likely makes more sense than on-demand content.  The sole exception remains synchronous content best done by multicast but experiences from our stadium venue point to even that synchrony being push-able (replays versus live watching).
  • We are in the process of re-architecting our ScaleBox code for which we do redundancy elimination to a new / more flexible platform that will offer PASS services in a portable / easy to stand up context (Docker container for servers / MEC deployment).  We plan to release the PASS components in mid-to-late June (2020) via GitHub with the code released via the MS-PL (Microsoft Public License).


PASS Updates

[January 29th, 2020] A few interesting updates on what is going on with the work on PASS (Provider Accessible Storage Subsystem) which fits under the broader umbrella of our Redundancy Elimination at the Edge work that is funded by NSF.  A bit more under the hood work but hopefully some fairly neat work down below […]

Posted in Research | Tagged , , , | Comments Off on PASS Updates

Journal Paper – A game-theoretic analysis on the economic viability of mobile content pre-staging

[January 23rd, 2020] Our journal paper entitled “A game-theoretic analysis on the economic viability of mobile content pre-staging” is now live via the Wireless Networks journal.  The paper focuses on mobile content pre-staging with an eye towards whether or not said pre-staging is solely beneficial to the provider or pre-staging gains are shared with the […]

Posted in Research | Tagged , , , , , | Comments Off on Journal Paper – A game-theoretic analysis on the economic viability of mobile content pre-staging

Sabbatical Update

[November 11th, 2019] Brief update as I am well over halfway through my sabbatical this fall.  Some interesting new projects in the hopper that we will highlight the projects bake a bit more but I can give a bit of a preview of some of the efforts. WiFi Leaf Detection: We are looking how WiFi […]

Posted in Research | Tagged , , , , , , , | Comments Off on Sabbatical Update


This work has been supported in part by the National Science Foundation through grant CNS-1718400.