Taming the transient while reconfiguring BGP

BGP reconfigurations are a daily occurrence for most network operators, especially in large networks. Yet, performing safe and robust BGP reconfiguration changes is still an open problem. Few BGP reconfiguration techniques exist, and they are either (i) unsafe , because they ignore transient states, which can easily lead to invariant violations; or (ii) impractical , as they duplicate the entire routing and forwarding states, and require special hardware. In this paper, we introduce Chameleon, the first BGP reconfiguration framework capable of maintaining correctness throughout a reconfiguration campaign while relying on standard BGP functionalities and minimizing state duplication. Akin to concurrency coordination in distributed systems, Chameleon models the reconfiguration process with happens-before relations. This modeling allows us to capture the safety properties of transient BGP states. We then use this knowledge to precisely control the BGP route propagation and convergence, so that input invariants are provably preserved at any time during the reconfiguration. We fully implement Chameleon and evaluate it in both testbeds and simulations, on real-world topologies and large-scale reconfiguration scenarios. In most experiments, our system computes reconfiguration plans within a minute, and performs them from start to finish in a few minutes, with minimal overhead.

Figure 1: BGP reconfigurations often lead to disruptions, even when using recent reconfiguration frameworks such as Snowcap [28].Here, Snowcap transiently violates two invariants (reachability and waypointing).In contrast, Chameleon reconfigures the network without violating any.
Among all reconfiguration scenarios, BGP ones are special because they are both particularly frequent and potentially highly disruptive.In large networks, for instance, operators reconfigure BGP up to 20 times a day on average [36].Also, since BGP controls routing to (and from) remote destinations, reconfiguring it can have Internet-wide consequences.The recent Microsoft outage in January 2023 illustrates this perfectly: Microsoft Azure services were indeed unavailable for 90 minutes due to a BGP reconfiguration [4].
Perhaps surprisingly, no existing network reconfiguration framework enables both safe (in a way that preserves network invariants) and practical (in a way that works operationally) BGP reconfigurations.Most previous works targeted other reconfiguration scenarios, such as networks running purely intra-domain routing protocols (OSPF or IS-IS) or SDN/OpenFlow [7].A few techniques [2,28,36] did focus on BGP reconfiguration, but they suffer from fundamental limitations.Specifically, "Shadow Configurations" [2] and BGP Ships-in-the-Night [36] perform the reconfiguration by duplicating the routing and forwarding states on every network device.Doing so is impractical though as it comes with significant overhead and is not supported by most routers [7].In contrast, Snowcap [28] gradually modifies the BGP configuration in-place while ensuring that the network converges to a correct state.Snowcap, however, does not guarantee the correctness of any of the transient states explored as the network converges during the reconfiguration.
We illustrate the limitations of Snowcap in Fig. 1, which depicts the evolution of the throughput during a simple reconfiguration of the 11-router Abilene network (see §6).The reconfiguration should maintain two invariants: (i) reachability (i.e., traffic should not be dropped); and (ii) waypointing (i.e., traffic should always cross a firewall).Snowcap transiently violates both invariants during the reconfiguration for almost two seconds.For critical (e.g., SLA-or security-related) requirements, transient violations are problematic.Additional experiments for different scenarios show a similar trend, and even longer violations in some cases (cf.Fig. 12).
We present Chameleon, the first in-place BGP reconfiguration framework that guarantees correctness during the entire reconfiguration process for both the steady and transient states.
Intuitively, building a framework like Chameleon requires solving a concurrency problem.It indeed entails reasoning about the asynchronous computations and non-deterministic message exchanges of all BGP routers in the network.This is challenging, for at least three reasons.First, the problem requires a formal concurrency model of a converging BGP network which exceeds the existing BGP models-existing network verification models only reason about the steady state [10,29] or whether a steady state will be reached [14].Second, the safe BGP reconfiguration problem requires to design concurrency control mechanisms (i.e., synchronization techniques) that provably maintain network-wide correctness without controlling individual BGP message timings.Finally, it requires to design a runtime controller that can efficiently orchestrate the entire reconfiguration process.
We address the challenges in the following ways.First, we introduce a happens-before model which captures BGP-specific concurrency.Using this model, we show how to constrain BGP route propagation through synchronization barriers that provably maintain invariants for all possible concurrent executions.We then compile this schedule into a reconfiguration plan.This plan enforces the computed synchronization barriers by pushing standard BGP commands to the routers and checking their execution.Finally, Chameleon's runtime controller applies this reconfiguration plan to the live network.
Ensuring network correctness throughout the entire reconfiguration process does not come for free: the need to coordinate some of the BGP computation means that the reconfiguration will typically last longer.This cost is exemplified in Fig. 1, where Chameleon took 58 sec to complete instead of 1.7 sec for Snowcap.All in all, we think that such an increase in duration is a small price to pay for the increased correctness.
We implement Chameleon and evaluate it in both testbeds and simulations on real-world topologies and large reconfiguration scenarios.In most experiments, Chameleon computes reconfiguration plans in <1 min and performs them from start to finish in a few minutes, with minimal overhead.Figure 2: Grammar for constructing a specification  for a single destination .Chameleon guarantees that  is satisfied during the entire reconfiguration, i.e., the sequence of forwarding states satisfies .
To sum up, our main contributions are: • The first practical BGP reconfiguration technique capable of preserving correctness invariants throughout the entire reconfiguration campaign, including transient states.
• A concurrency model for BGP reconfigurations and its convergence process using happens-before relations.
• A complete implementation of Chameleon in Rust alongside an online application to explore reconfiguration scenarios (available at https://bgpsim.github.io?s=example).
• An extensive evaluation of Chameleon and its overhead.

OVERVIEW
We now provide an overview of Chameleon.We start with the problem statement and illustrate the challenges using a running example before describing Chameleon's high-level workflow.

Problem Statement
Reconfiguration.We refer to the problem of adapting the BGP configuration of one or more routers as a BGP reconfiguration.Examples of reconfigurations include: (i) local changes like adding or removing a BGP session or changing the local preference of routes learned from a specific neighbor; and (ii) global changes such as switching from an iBGP full-mesh to route reflection network-wide.
Specification.We refer to a specification  as a set of network invariants that describe important forwarding properties for the network operators.Our specification language (cf.Fig. 2) combines individual properties with boolean operators and Linear Temporal Logic (LTL).Boolean operators allow users to express intricate properties on the forwarding state, such as isolating traffic from parts of the network.In contrast, temporal operators formulate constraints on the sequences of transient states explored during reconfigurations.For example, if a reconfiguration involves changing the egress router of a given destination prefix, operators can require routers to switch from the initial egress router to the final egress once, without switching back and forth multiple times.
We assume the initial and final configurations are correct and comply with .Otherwise, there is little point in maintaining correctness during the reconfiguration.Similarly, we assume that the initial and final configurations eventually converge to a stable state [14,15] to ensure that the semantics of input specifications is always sound.Goals.We aim to perform both local and global BGP reconfigurations in-place while guaranteeing the correctness of a given specification  throughout the entire process.Specifically, we aim to achieve the following properties: 1. (safety) the reconfiguration process must guarantee that the input specification  is satisfied throughout any transient state explored during the BGP convergence process; 2. (practicality) the reconfiguration process must limit the overhead it imposes on routers' resources, including routing and forwarding table sizes and reconfiguration time.
Running example.As an example BGP reconfiguration scenario, we consider the network in Fig. 3, which consists of six BGP routers, two of which ( 2 and  5 ) operate as route reflectors [3]. 1 Routers  1 and  6 each receive one BGP route for the same prefix.We refer to those routes as  1 and  6 , respectively.The reconfiguration involves decreasing the local preference of  1 from 200 to 50 so that all routers switch to using  6 .We suppose the operators aim at preserving reachability throughout the reconfiguration, which they can express as  = G  reach(  ) in our specification language.
Preferring  6 will cause all routers to forward "to the right" instead of "to the left." Intuitively, reachability is violated whenever routers on the left-hand side of the network ( 1 ,  2 or  4 ) update their state before routers on the right ( 3 ,  5 or  6 ).Depending on the timing of the BGP messages, this can easily happen, especially since  3 only learns  6 from one of the two route reflectors,  2 or  5 , i.e., after they have already selected  6 .

Chameleon
In a nutshell, Chameleon achieves safety by coordinating the forwarding state updates BGP makes during the reconfiguration.To practically coordinate the updates, Chameleon gradually introduces a temporary BGP configuration (which differs from both the initial and final configuration).This temporary configuration enforces a particular ordering of BGP messages that satisfies the specification.
Finding and implementing specific message orderings is far from trivial.The first challenge is capturing achievable orderings by introducing temporary BGP configurations.This is hard as many BGP-specific mechanisms-like route reflection [15]-limit route visibility.The second challenge is implementing a specific execution using temporary configurations; this requires both reasoning about the distributed routing state and synchronizing BGP computations.
Workflow.Chameleon solves those challenges by following the following three consecutive steps.Chameleon's workflow is visualized in Fig. 4 using the running example from Fig. 3.
1.The analyzer describes the space of concurrent convergence processes by analyzing the initial and final configuration (the input to Chameleon) and computing happens-before relations between routing states of different routers.2. The scheduler explores the space of convergence processes spanned by the happens-before relations to find one that satisfies the specification.It describes this convergence process as a node schedule that captures which routes are selected at which time. 3. The compiler transforms this node schedule into a reconfiguration plan, that is, a sequence of temporary configuration commands and local conditions for synchronization.
Finally, Chameleon's runtime controller performs the reconfiguration on the live network by checking the local conditions and applying the commands, precisely following the compiled reconfiguration plan.In the following, we describe all three steps of Chameleon in more detail, using the running example from Fig. 3.
Step 1: Analyzer §3.The analyzer extracts happens-before relations between selected routes in the network, encoding the propagation path of routes.These relations define the space of convergence processes that are realizable just using temporary BGP configurations.We obtain these relations by simulating the network and analyzing the resulting network state in a way that generalizes to any BGP configuration.
In our example from Fig. 3, router  1 propagates the initial route  1 to both route reflectors  2 and  5 , who then announce  1 towards  3 ,  4 , and  6 .Therefore,  4 can choose  1 as long as  2 or  5 select  1 .We repeat the same for  6 .
Step 2: Scheduler §4.We express the happens-before relations together with the specification as an Integer Linear Program (ILP).A solution to the ILP is a node schedule describing a BGP convergence process that meets the specification, in both steady and transient states.The scheduler allows concurrent updates if their relative order does not affect the specification.Further, Chameleon allows additional propagation paths to increase route visibility and ensure a solution exists.We maximize the number of concurrent updates (thus, minimizing the reconfiguration time) as a primary objective while reducing additional propagation paths as a secondary goal.
Chameleon schedules the example reconfiguration using four rounds, updating nodes from right to left.It also computes the rounds in which a router selects its old or new route.In Fig. 4, orange arrows represent the propagation of the old route  1 , whereas blue arrows depict the new one  6 .
Step 3: Compiler §5.Chameleon computes a reconfiguration plan to implement the calculated schedule, along with local conditions to synchronize the update and guarantee correctness.Each temporary command only rewrites routing preferences by modifying route attributes such as weight [25] or local preference.The conditions assert a router knows or selects a specific route and can be checked locally by inspecting that router's routing table.Figure 4: Chameleon computes a reconfiguration plan to transition from the old to the new configuration while satisfying the specification.The illustrated instance corresponds to the example from Fig. 3. Explore this reconfiguration plan interactively at https://bgpsim.github.io?s=example.We provide a tutorial in App.E.
Chameleon's reconfiguration plan for our example first updates the egress router  6 to make  6 available in the network.It then makes both  5 and  3 select  6 using a temporary session between  3 and  6 .In Round 3, it updates both  2 and  4 .Chameleon introduces a temporary session between  4 and  1 to break  4 's dependency on  2 .Finally, Chameleon finishes the reconfiguration by updating the local-pref on  1 , which causes  1 to select  6 .Notice, that this reconfiguration can be performed without using temporary BGP sessions by reducing concurrent updates.
If the reconfiguration updates multiple destinations at a time, Chameleon runs the scheduler and the compiler for each destination separately and combines the generated schedules.This is only possible if all prefixes converge independently, which is valid case for most networks.However, there might exist networks for which some prefixes are dependent.We explain in §8 how the scheduler can be extended to support such networks.

ANALYZER
Chameleon models the routing state of the network as propagation paths of BGP routes.It then describes the reconfiguration process in terms of how these routes change over time and formulates happensbefore relations capturing safety conditions on BGP transient states.
Network & Routing Model.We model the network as a graph  = ( , ) with nodes  connected by edges .External networks advertise BGP routes towards a destination  to multiple nodes; the egress routers of .Nodes propagate routes over BGP sessions that form an overlay signaling graph, different from the physical topology.We define routes in terms of their propagation path rather than modeling a route's specific BGP attributes.If  = [,  1 , . . .,   , ] is a route received at node , then  was first received by the egress router  () =  1 , and advertised to  by its BGP neighbor   = pre().The network's configuration determines: (i) which route  a node  selects, (ii) to which neighbors  announces its selected route , (iii) how the egress  () is mapped to the next hop.Observe that the routing state of any converged network, that is, a network in a fixed routing state with no more messages in flight, is always consistent.However, the consistency of a routing state only ensures that there exists a configuration such that the network converges to this routing state.
Using these terms, we define a BGP reconfiguration as the transition from an initial routing state P old to the final routing state P new .A reconfiguration process [P old , P 1 , . . ., P new ] is a sequence of routing states which contains all the intermediate routing states of the network during a reconfiguration.A reconfiguration process is safe if its corresponding sequence of forwarding states satisfies the specification that the operator expresses [nh old , nh 1 , . . ., nh new ] |= .
Our routing model considers a single destination  at a time.However, one destination can represent an entire class of equivalent prefixes for which the network computes the same routing and forwarding state.If a BGP reconfiguration affects multiple destinations, Chameleon treats each prefix equivalence class separately: this is safe because BGP separately processes individual destinations.
Methodology & Framework.Chameleon controls each intermediate routing state during the reconfiguration to safely transition from P old to P new .To that end, Chameleon synthesizes a sequence of routing states such that each state is consistent.Consequently, the BGP reconfiguration process never affects the forwarding state.Instead, we trigger each routing and forwarding state update by reconfiguring nodes.A naive approach to synthesizing a sequence of routing state updates is to make each node  switch its selected route directly from the initial route P old () to its final one P new ().This technique, however, does not work for most reconfiguration scenarios.Assume node  learns both its initial and final route from   = pre(P old ()) = pre(P new ()).The naive approach cannot ensure the consistency of every routing state, as   must switch before .While   is using P new () and  still selects P old (), the routing state is inconsistent.Chameleon uses two techniques to resolve this issue.
The first technique is to establish a temporary BGP session directly with the egress router,  (P old ()) or  (P new ()).This session allows  to select the route [,  1 , ] directly.This technique has two significant downsides.First, the additional session causes state duplication.Second, the new route may have different BGP attributes that affect the route propagation.Thus, we must ensure no other node selects a route from  while  selects [,  1 , ].
The second technique to ensure each routing state's consistency is to leverage BGP redundancy.Most BGP configurations using route reflection have multiple reflectors.As a result, node  may receive multiple equivalent routes that share the same BGP attributes.Choosing one equivalent route over another does not impact the forwarding state, but it affects the routing state.Therefore, simply switching between equivalent routes can violate routing state consistency while the update propagates through the network.We introduce a setup and cleanup phase, during which Chameleon chooses between equivalent routes, ensuring the forwarding state remains unchanged.Chameleon still enforces routing state consistency during the main update phase.
Our methodology results in each node  updating its next hop exactly once from ℎ old () to ℎ new ().Thus, Chameleon never exports any transient routes towards inter-domain neighbors.However, there may exist highly constraining specifications that Chameleon cannot enforce.We prove in App.B that our methodology is sufficient to perform any reconfiguration preserving reachability.
Happens-Before Relations.We model a BGP reconfiguration process within our framework using happens-before relations.First, we capture the set of all neighbors that advertise identical routes as the initial and the final route towards node  using D old () and D new (), respectively. cannot use its initial route P old () for longer than any of D old () also select their initial route.Likewise,  can only select P new () as soon as any of D old () select their final route.The resulting happens-before relations for node  are: Fig. 5 illustrates the happens-before relations for the initial state using the running example introduced in Fig. 3. Node  4 receives equivalent routes for  with propagation paths   and   from route reflectors  2 and  5 .Thus,  4 can select route P old ( 4 ) for as long as either  2 or  4 advertise it.

SCHEDULER
We now detail Chameleon's scheduler.The goal of the scheduler is to generate a node schedule, that is, the order in which routers switch from selecting their old route in P old to their new route P new .We encode this schedule by assigning each node a round in which it will change its routing (and forwarding) decision.We formalize the problem of finding a schedule with at most  rounds as an ILP.We increment  in a loop as long as no solution exists, thus minimizing the number of rounds  and, consequently, the reconfiguration time.Our ILP captures these three high-level properties: 1. Happens-before relations: The scheduler transforms the happensbefore relations from §3 into constraints.We design the objective function to minimize the number of temporary BGP sessions.2. Concurrent Updates: Allowing nodes to update concurrently reduces the number of required rounds.However, this concurrency can lead to different update orderings within a single round.Therefore, we require all routers to change their next hop in a round where no other router along its path also changes its forwarding decision.This constraint guarantees updates in the same round are independent concerning the forwarding state.3. Specification: The resulting sequence of forwarding states must satisfy the specification , expressed as LTL.

Happens-Before Relations
We enforce BGP propagation rules in the ILP model by using three symbolic integer variables for each node  ∈  : We call the 3-tuple   old ,   nh ,   new the schedule of node .Here,   nh ∈ N describes the round in which  changes its next hop.Further,   old ∈ N capture the last round when  receives the old route P old (), and   new ∈ N is first round for  to receive P new ().Next, we enforce BGP route propagation as described by the happens-before model in §3.A neighbor  ∈ D old () must exist that selects its old route P old () for longer than  selects P old (), i.e.,   old <   old .Similarly, a neighbor  ∈ D new () must exist that selects its new route before  does, i.e.,   new >   new .This yields Whenever   old <   nh , node  no longer knows its initial route P old () as long as it must use its old next hop.Similarly, if   nh <   new , node  does not know route P new () when it should use its new next hop.In both cases, we must introduce a temporary BGP session.We minimize the number of such sessions as follows: We encode all non-linear min, max, and if-then-else operators with linear constraints using the big- method [16].

Concurrent Updates
Our ILP allows multiple nodes to update their next hop in the same round.As a result, individual updates of the same round may occur in any order.Hence, the ILP must guarantee the specification is satisfied in all possible orderings for each round.To that end, Chameleon ensures all forwarding state updates in the same round are independent.We define two forwarding state updates of node  1 and  2 to be independent if the forwarding path of  1 before and after the update does not traverse  2 , and vice versa-consequently, every forwarding path in the network experiences at most one change in each round.
We enforce the independence of all forwarding state updates in the same round recursively.For each node  ∈  and round , we introduce the boolean variable    ∈ B to capture whether either  or a different node along the forwarding path of  changes its next hop in round .We express    in terms of its next hop: where  = nh old () and  = nh new () In case   nh ≠ ,    directly depends on either the initial or the final next hop.However, in case  changes its routing decision in round , that there is no other update along either the old or the new forwarding path of .

Specification
To encode the specification  in the ILP, Chameleon generates a syntax tree according to the specification language.Each node in that graph is labeled with its production rule, as shown in Fig. 2. 2 We then generate a directed acyclic graph   = (Φ,   ) by combining equivalent nodes with identical descendants.In other words, we simplify the tree by combining branches that result in redundant constraints.In the following, we call each node   ∈ Φ in this syntax graph an expression.For each expression   ∈ Φ, we introduce symbolic boolean variables  , ∈ B.  , is true if and only if the expression   is satisfied in round .The following explains the constraints we add to the ILP for each kind of expression.
Reachability.Chameleon creates recursive constraints to ensure reachability.Essentially, we say a node  satisfies reachability reach() if its next hop does so.This recursive statement requires not only checking  for reachability but also every other node  ∈  in the network.More formally, for each node  ∈  and each round  ∈ {1, . . ., }, we introduce a boolean variable   reach, ∈ B and constrain it in terms of 's next hop : 2 Our implementation ensures the uniqueness of this tree by surrounding each production rule with parenthesis.
Here, nh old and nh new are the initial and final forwarding states, and  is the next hop of node  at round .Notice that these constraints are sufficient if and only if there are no forwarding loops.We explain in §4.4 how we ensure the absence of loops.
Waypointing.Waypoints are treated similarly to reachability.Node  satisfies wp(, ) if its next hop does so or if its next hop is .For each waypoint target  ∈  in the specification, we introduce a boolean variable   wp( ), for each node  ∈  and round  ∈ {1, . . ., }, constrained in terms of 's next hop : where Logical and Temporal Modal Operators.Chameleon encodes a logical or temporal modal expression   by following its production rule and refering to the symbolic boolean variables of its subexpressions.We constrain the variables  , ∈ B as follows: • We implement the conjunction rule   =   ∧   at round  using the big- method [16]:  , =  , ∧  , .The other logical operators (¬ and ∨) are constructed identically.
• We encode the globally operator   = G   by asserting   is satisfied indefinitely, starting from :  , =  ≥  , .
• To encode the until operator   =   U   , we assert the existence of a round  ≥  until which   is satisfied, and at which   holds.To that end, we unroll the ∃ quantifier: • We implement all other temporal modal operators similarly to U and G by unrolling logical quantifiers.
Ultimately, we assert the reconfiguration plan satisfies  by asserting the root of the syntax tree   holds in the first round  = 1:

Absence of loops
The recursive constraints of reachability and waypoint require the absence of forwarding loops.To illustrate, let us consider a forwarding loop between  1 and  2 .While generating the reachability constraints, Chameleon only requires   1 reach =   2 reach , thus allowing the model to choose an arbitrary value for   1  reach .We ensure the absence of forwarding loops by enumerating all possible forwarding loops.To that end, we build a graph  nh = ( ,  nh ) by combining the old and the new forwarding states.For each node  ∈  ,  nh contains two edges from  to its old and new next hop, nh old () and nh new (), i.e., (, nh old ()) ∈  nh and (, nh new ()) ∈  nh .Each simple cycle in  nh represents one possible forwarding loop.We enumerate all simple cycles  of  nh .For each cycle  = ( 1 , . . .,   ,  1 ) ∈  of length  and each round , we assert that not all nodes   along the cycle can choose  +1 as next hop simultaneously: The constraints on concurrent updates (cf.§4.2 and Eq. ( 2)) implicitly ensure the absence of loops if the initial forwarding state is loop-free.We provide a proof in App.D. However, we notice the variance of the scheduling time of Chameleon decreasing when explicitly enumerating each possible loop, while the average time is unaffected.We, therefore, keep the explicit loop constraints in the model even though they are not strictly necessary.

COMPILER
Chameleon's compiler transforms the schedule of the routing and forwarding state into a reconfiguration plan.This reconfiguration plan consists of a sequence of commands and conditions, constituting synchronization to implement the schedule.Each command will only affect a specific entry of a single router's routing and forwarding table.Similarly, each conditions only need to check a particular entry of the routing or forwarding tables.The following explains how the compiler transforms the 3-tuples (  old ,   nh ,   new ), computed by the scheduler, into a reconfiguration plan.
To gradually transition the routing state as computed by the scheduler, Chameleon changes route attributes that stay local to that device without affecting other routers in the network.To that end, the compiled commands modify a route's weight [25] using route maps.
Reconfiguration Phases.The reconfiguration plan consists of three phases: setup, update, and cleanup.During the update phase, we forbid Chameleon to switch between equivalent routes from different neighbors (cf.§3).Instead, we ensure each node appropriately changes its BGP neighbor in the setup phase.Based on the happensbefore relations, there must exist a neighbor   old ∈ D old () of node  that advertises P old () to  for longer than   old .Similarly, a neighbor   new ∈ D new () advertises P new () to  before round   new .The setup phase ensures each node  prefers the route learned from   old .Further, the setup phase establishes all temporary BGP sessions.Similarly, the cleanup phase removes all route preferences and temporary sessions.
We divide the update phase into  rounds.Chameleon ensures synchronization by (i) using pre-and post-conditions for commands and (ii) transitioning between rounds when all post-conditions are satisfied.Pre-conditions check a route's availability, while a post-condition ensures it is selected.Chameleon generates those commands and conditions using rules shown in old , make  prefer the route from  P old () using a temporary BGP session.
• post-condition:  selects the route from  P old () .
• In round   nh , make  prefer the route from  P new () using a temporary BGP session.
• post-condition:  selects the route from  P new () .
• In round   new , make  prefer P new () from   new .• pre-condition:  knows P new ().
Original Reconfiguration Commands.Applying the reconfiguration plan must transition the network from the initial to the final configuration.Chameleon interleaves its temporary commands with the original reconfiguration commands.Let  * ⊆  contain all nodes subject to those commands.For each node  ∈  * , we apply the original commands  * for  immediately before or after   nh .If  * makes  deny route P old (), then we execute  * after   nh .Otherwise, we schedule  * before   nh .Thus, we guarantee  knows P old () until round   nh , and P new () starting from   nh .Chameleon treats each destination separately.To support reconfigurations that affect multiple destinations, Chameleon performs the update phase for all destinations in parallel and aligns their execution along the original commands.For instance, assume that the original command  * is applied before round 3 for destination  1 and after round 4 for  2 .We perform the update phase for  1 until round 2 and  2 until round 4.Then, we apply the  * and proceed with the update phase of both  1 and  2 .However, such an alignment may not exist if multiple nodes are subject to the original reconfiguration.In fact, the schedule for  1 could require the original command  * 1 to be applied after  * 2 , whereas  2 expects the reverse order for  * 1 and  * 2 .In that case, we use Snowcap to split the reconfiguration into commands that targeting individual nodes.We then apply each of them one by one in an ordering computed by Snowcap.

CASE STUDY
We demonstrate the effectiveness of Chameleon in a testbed.The testbed consists of three Cisco Nexus 7000 devices, each running four virtual routers.We connect the resulting 12 virtual routers to a Barefoot Tofino programmable switch, enabling us to emulate any network with 12 routers.We simulate the physical distances between routers using a server that delays all packets all link.We also peer the egress routers with external BGP speakers.We use the above setup to emulate the Abilene network from Topology Zoo [21], a network with 11 nodes.We configure the routers to run OSPF [24] and connect them in an iBGP route reflection topology with three route reflectors.Then, we generate three external networks, each injecting routes towards 1024 individual prefixes at nodes  1 ,  2 , and  3 .All nodes prefer routes from  1 over  2 and  3 and decide between  2 and  3 based on the shortest IGP path.The emulated reconfiguration removes the BGP session between  1 and its external peer, forcing all routers to change their routing decision during the reconfiguration.The network treats all prefixes identically; thus, Chameleon considers only a single prefix equivalence class.
During the reconfiguration, we require that all destinations remain reachable.Furthermore, each router must first use  1 =  (P old ()) as egress and then switch, only once, to its final egress   =  (P new ()) (either   =  2 or   =  3 ).More precisely, To validate if Chameleon satisfies the specification, we inject traffic at a constant rate at each node towards  and measure the egress router where they leave the network.
Comparison with Snowcap.Snowcap applies the reconfiguration command directly to the network as the reconfiguration affects only a single line in the configuration.Fig. 1 in the introduction shows the comparison between Chameleon and Snowcap.Snowcap' reconfiguration takes 1.7 seconds to finish.However, during this time, the network drops around 15 k packets for almost one second of traffic.Further, around 1.3 k packets violate the waypoint constraints.In contrast, Chameleon performs the reconfiguration without dropping any packets and permanently preserving waypoint requirements.Chameleon reconfigures the network in around 112 seconds, of which 72 seconds are spent in the main update phase.Fig. 6 shows Chameleon's phases and rounds.We run the same experiment on five different topologies and show the results in the Appendix (cf.Fig. 12).All experiments show a similar outcome: reconfiguring the network with Snowcap causes transient black holes and violations of waypoint constraints, while Chameleon performs all reconfigurations safely.

EVALUATION
In this section, we evaluate the overhead of Chameleon in three dimensions.First, we analyze the scheduling time ( §7.1) and show that it typically takes Chameleon a few minutes to schedule challenging BGP reconfigurations that affect the routing state of the entire network.We additionally capture the two major factors impacting Chameleon's scheduling time with metrics that we call reconfiguration and specification complexity; we show that the former has a heavier impact than the latter.Second, we analyze the reconfiguration time ( §7.2) and show that Chameleon can perform large reconfigurations in a few minutes while guaranteeing safety during the entire process.Third, we measure routing table sizes ( §7.3) and compare the memory requirement of Chameleon with previous work: Snowcap does not need any extra memory, SITN roughly double, and Chameleon only around 10%.
Implementation.We implement Chameleon using around 7 k lines of Rust code 3 that includes the analyzer, scheduler, compiler, and runtime controller, and that supports specifications as defined in Fig. 2. We use our BGP simulator (≈ 30 k lines of Rust code) to obtain the initial and final routing state and to validate Chameleon's reconfiguration plan for experiments that are too large to run directly on our testbed.We use COIN-OR CBC [8] to solve the ILP.We run all experiments on a server with 32 CPU cores and 64 GB of memory.COIN-OR CBC is allowed to use all available cores.
Methodology and Reconfiguration Scenario.We use 106 topologies from Topology Zoo [21]: their size ranges from a few routers to 754.For each topology, we randomly select three different nodes  1 ,  2 , and  3 , and connect them to three external networks.All external networks advertise the same destination , and all internal nodes prefer routes received by  1 over those from  2 and  3 .We then elect three random nodes to be BGP route reflectors; each other router is a client of all three reflectors.
We choose a reconfiguration scenario that affects the routing state of all routers in the network to evaluate Chameleon in a challenging setting.Adding or removing individual BGP sessions are reported to be the most common BGP reconfigurations [36], but they rarely affect all routers in the network.Instead, we simulate adding a route map to the initially most preferred egress point  1 : the route map denies the route towards  from its external peer, forcing all routers in the network to change their selected route during the reconfiguration.Chameleon usually finds a schedule in a few minutes, and up to 8 hours for the largest scenario, in which 660 routers change their next hop.A point corresponds to one of 106 scenarios, and both the -and -axis have a logarithmic scale.

Scheduling Time
We first evaluate Chameleon's scheduling time.Perhaps surprisingly, the time spent solving the ILP model is not directly proportional to topological metrics such as the network size (see Table 2 in the Appendix).We investigate the most important factors impacting Chameleon's scheduling time.We identify two orthogonal dimensions: reconfiguration complexity and specification complexity.Intuitively, the reconfiguration complexity relates to the number of nodes each router can reach during a reconfiguration, irrespective of the invariants to be preserved (i.e., the specification).In contrast, the specification complexity models the complexity of invariants to be preserved during a reconfiguration, irrespectively of the network topology, routing, and forwarding.
In the following, we separately evaluate how Chameleon's scheduling time varies with respect to reconfiguration and specification complexity.To do so, we evaluate the impact of reconfiguration complexity by keeping the same specification and simulating different reconfigurations across the 106 networks in our dataset (Figure 7).We then assess the impact of specification complexity by considering specifications of increasing complexity for the same network and reconfiguration scenario (Figure 8).
Reconfiguration Complexity.Some reconfigurations only affect a few nodes, while some affect nodes located far away from each other.Other scenarios, however, can update multiple nodes along a single forwarding path, thus introducing dependencies.As a result, Chameleon may schedule one scenario in a few milliseconds, while it needs seconds or even minutes to schedule a different scenario, even if both reconfigure the same network.
We measure the number of dependencies between nodes by defining the reconfiguration complexity C  .Intuitively, C  counts for each node  the number of different nodes that  could reach during the reconfiguration.To determine C  , we first construct a directed graph  ℎ = ( ,  ℎ ) as the union of the initial and final forwarding state ℎ old and ℎ new (as described in §4.4).For all nodes   ∈  ,  ℎ contains edges (, ℎ old ()) and (, ℎ new ()).Further, let  ℎ = { ∈  | ℎ old () ≠ ℎ new ()} be the set of all nodes that eventually change their next hop, and let reachable( ℎ , ) be the set of all reachable nodes in  ℎ from .The reconfiguration complexity C  is given as: Note that C  only depends on the initial and final forwarding states nh old and nh new and does not depend on the specification .Fig. 7 displays a log-log plot that shows the strong correlation between the reconfiguration complexity C  and the scheduling time of Chameleon.Each point represents one reconfiguration scenario.Chameleon finds a solution in a few minutes despite the non-trivial specification  from Eq. ( 4) and the network-wide reconfiguration scenario.In the worst case, it takes our system less than 8 hours to schedule a reconfiguration in which 754 nodes update their routing state, and 660 nodes change their next hop.For the vast majority of scenarios, Chameleon finds a schedule in less than a minute.
We explain the strong correlation between C  and the scheduling time based on the semantics of constraints in our ILP.Constraints related to the specification ( §4.3) and concurrent updates ( §4.2) are recursive: a variable    associated with node  at round  depends on the next hop of  at round .Consequently,    depends on the next hop of all nodes along all possible forwarding paths of  that also change their next hop, which C  precisely captures.Specification Complexity.The specification language from Fig. 2 captures specifications ranging from reachability to complex waypoint constraints that change over time.We evaluate how the complexity of the specification influences the scheduling time to determine the impact of (i) recursive properties like waypoints and (ii) temporal operators.
To this end, we define two different specifications:   does contain temporal operators in the form U G, and   asserts the same properties throughout the reconfiguration.We construct   and   as follows: Both specifications require reachability for all nodes and waypoint constraints for some nodes   .Increasing the number of nodes in   influences the complexity of the specification.Each node  ∈   must use either its initial egress router  1 =  (P old ()) or its final one   =  (P old ()) (cf.§6).In addition, the temporal specification   further requires that node  ∈   switches once from  1 to   .More precisely, Fig. 8 shows the scheduling time for a varying number of nodes in   .The figure refers to reconfigurations on the CogentCo topology, the network with 197 nodes for which both C  and the scheduling time are the second largest of all scenarios (i.e., the scenario with a scheduling time of 18 minutes in Fig. 7).We run every experiment 20 times for each   and plot the median scheduling time, as well as the 10th and 90th percentiles for both   (orange) and   (blue).
The figure highlights a significant difference between the nontemporal specification   and the temporal one   : While adding more waypoints has a small impact on the scheduling time of Chameleon, increasing the number of temporal operators affects the scheduling time significantly.This disparity is because waypoint constraints are recursive; constraints for a single waypoint target are added for all nodes, even if only one node must use that waypoint target.In contrast, the temporal operators in   demand variables and constraints for each additional node in   .
Takeaways.Chameleon's scheduling time depends on both reconfiguration and specification complexity, two orthogonal dimensions.However, the scheduling time is much more sensitive to the reconfiguration complexity.In fact, the specification complexity only affects scheduling time by 20× at most, while the reconfiguration complexity has an impact of over four orders of magnitude.

Reconfiguration Time
We show that Chameleon applies large-scale reconfigurations in a few minutes, while satisfying the specification in all transient states.
Quantifying Reconfiguration Time.Our testbed includes only 12 virtual routers.To estimate the reconfiguration time of Chameleon in larger networks, we run simulations.
The time for Chameleon to apply a single round depends on two factors: the time to apply the reconfiguration and the convergence time itself.From our measurements in Fig. 6, we determine that routers in our testbed take between 8 and 12 seconds to modify BGP route maps (presumably to anticipate additional changes, such that updates can be applied in batches).In contrast, the convergence time is negligible.This is because Chameleon enforces the happensbefore relations: updating the selected route of one node will never change the selected route of a different node.Consequently, any BGP update will be ignored, and the convergence time depends on the distance between nodes rather than on the network size.We approximate the running time  of a reconfiguration plan by counting its number of rounds , including the setup and cleanup phase.To that end, we define   as the approximation of the time to apply a single round of the reconfiguration plan.We approximate the reconfiguration time as The reconfiguration time heavily depends on the specific hardware.Our case study suggests   ≈ 12 s for Cisco Nexus 7000 routers.Further, we measure no significant impact of routing table size on   , as we have compared   for table sizes between 1 and 16 k.In the following, we use   = 12 s.
Results.We estimate the running time for all 106 scenarios and find that Chameleon's approximate reconfiguration time is less than 2 minutes for 85% of the scenarios while preserving the specification from Eq. (4).For the largest reconfiguration scenario, we estimate that Chameleon takes 5 minutes to reconfigure all 754 nodes.We provide a cumulative distribution function of the approximated reconfiguration time across all 106 scenarios in Fig. 9.

Routing Table Size
In the following, we compare Chameleon with related work regarding additional routing table entries needed during the reconfiguration process.On average, Chameleon replicates about 11% of the routing state to preserve invariants during the transient state, whereas related work duplicates the entire routing process to achieve similar guarantees.For Chameleon, the routing table size increases depending directly on the number of temporary BGP sessions-the scheduler can minimize the required number of temporary sessions to account for networks or routers with specific memory limitations.
Methodology.While simulating a reconfiguration, we measure the maximum number of routing table entries in the network at any given time during the process.We then compare this distributed routing table size of Chameleon with the baseline of directly reconfiguring the network (as Snowcap would also do) and SITN [36].Results.Chameleon requires, on average, 11% more routing table entries than the baseline, whereas SITN requires 96% more entries.Fig. 10 shows detailed statistics for all 106 scenarios.
We note that temporary BGP sessions exclusively cause state replication in Chameleon's reconfigurations.Hence, Chameleon can customize its reconfiguration plans according to specific memory limitations of operational networks.For example, if a given node  is operating at its memory's limit, Chameleon can generate a reconfiguration plan in which  only uses existing BGP sessions by including a constraint   old =   nh =   new .

LIMITATIONS AND DISCUSSION
We now discuss the main limitations of Chameleon.They pertain to how Chameleon manages constraining specifications, dependencies between prefixes, and unplanned external events.
Specifications.We design Chameleon to guarantee that for each BGP destination, every router always selects its best BGP route in the initial configuration or the final one.We choose to avoid transiently leaking intermediate routes to neighboring networks, preventing unnecessary interdomain churn.
The drawback of this choice is that Chameleon may not be able to compute a safe reconfiguration for highly constrained specifications.We prove that Chameleon always guarantees reachability (App.B).However, it may be unable to carry out a safe reconfiguration in the presence of many waypointing invariants.This is because no safe reconfiguration actually exists in some settings if routers are only allowed to use their initial and final BGP routes.In those cases, Chameleon notifies the user that it cannot perform the reconfiguration safely.
We expect that in practice, Chameleon cannot perform reconfigurations rarely.In all our experiments ( §7), we only encountered two examples of unsolvable specifications: they try to enforce waypointing requirements in a star-shaped topology.
We plan to extend Chameleon so that operators can trade consistency of routes announced to neighboring networks for the feasibility of highly constrained reconfigurations.This would require explicitly supporting routing invariants, that is, constraints on the BGP routes selected by routers during reconfigurations.Supporting routing invariants within Chameleon's design should be straightforward: in addition to updating the specification language (see Fig. 2), it would involve formulating new happens-before relations for routing invariants and translating them into ILP constraints.
Dependencies between prefixes.Chameleon treats all prefixes independently.If a BGP reconfiguration impacts multiple prefixes, Chameleon computes reconfiguration plans for each prefix, solving the resulting smaller problems in parallel and performing per-prefix reconfiguration concurrently.The above approach is consistent and, hence, correct whenever eBGP routes are not modified in iBGP, as is often suggested by current best practices.
More advanced BGP configurations can, however, create dependencies between prefixes, such as route aggregation/de-aggregation or iBGP policies.
• Most BGP speakers support route aggregation, where nodes aggregate multiple routes into a single one and advertise the summary to neighboring routers.We note that only routes aggregated within a network create dependencies between prefixes.Typically, routes are aggregated only at the network border to either reduce the number of routes handled in iBGP or to announce a single eBGP route to external networks [5].In those common cases, it is still correct to treat prefixes independently since all internal routers know either all individual routes or only the summarized ones.Similar considerations hold when prefixes are de-aggregated in sub-prefixes, for example, for traffic engineering [26].
• Routers may apply arbitrary iBGP policies using route maps, i.e., modifications of BGP routes when crossing internal BGP sessions.Some Internet networks do configure iBGP policies [35].
Depending on the configured route maps, this practice can create dependencies between prefixes.For example, if route maps discard a route on internal BGP sessions, different routers may forward the same packet by matching it to more or less specific prefixes, depending on which routes they receive.
Chameleon's scheduler can be extended to support these cases by jointly considering multiple prefixes.Supporting route aggregation requires generalizing our current happens-before relations.Additionally, modeling the absence of some routes at specific nodes can be encoded in the next hop computation of the ILP model.
External Events.Chameleon guarantees network correctness throughout the reconfiguration process in the absence of external events, such as link failures or external route changes.Stated differently, Chameleon might (transiently) violate some of the invariants if external events happen during the reconfiguration.
Because of the commands applied by Chameleon, only the withdrawals of the best BGP routes can break Chameleon's correctness guarantees.By definition, non-best routes are not used by any router and are, therefore, irrelevant.New best routes that appear during the reconfiguration are also not problematic since transient states installed by Chameleon make routers ignore the new routes until the end of the reconfiguration process.In contrast, Snowcap is vulnerable to all kinds of BGP events.Figure 11: Chameleon is resilient against many events.
As an illustration, we simulate the effect of unplanned external events during reconfiguration in our testbed (cf.§6).Fig. 11a shows the effect of a link failure that causes OSPF to reconverge during the reconfiguration.OSPF takes around half a second to reconverge, causing routers to drop packets during that time.The effect of the link failure would have been similar if it had occurred before or after the reconfiguration.Fig. 11b shows that the network reacts to the appearance of a new, more preferred route advertised to  4 only after Chameleon restores the original route preferences.That is, routers choose a sub-optimal path for about 70 seconds, but safety is preserved throughout the reconfiguration.
We envision that Chameleon's runtime controller can additionally track the network state and external events, especially BGP withdrawals, and react to them as follows: 1.If the event does not impact correctness, we can delay its effect by simply ignoring it, similar to Fig. 11. 2. If the event causes a suboptimal routing state, we can quickly compute a new reconfiguration plan to reduce the time spent in a suboptimal routing state.This is enabled by Chameleon's scheduling efficiency (cf.§7.1). 3.If the event triggers long-term anomalies, Chameleon immediately commits to the final configuration so that it can restore connectivity as quickly as possible.

RELATED WORK
An extended volume of proceeding work from academia and industry has studied the general network update problem [7].While some operate on traditional (distributed) networks, the majority of systems [19,23,27] extend SDN, in which a central controller modifies the forwarding tables of all nodes.They can give strong guarantees like per packet consistency or congestion avoidance by leveraging direct control of the forwarding decision.
As for traditional network update systems, we identify two broad categories.The first category of tools [2,18] duplicates the control and data planes on all routers, allowing them to run both the initial and the final configurations in parallel.They then gradually instruct routers to forward traffic according to the final configuration in an order that avoids forwarding loops [34,36].While useful, these tools tend not to be used in practice due to their overhead.
The second category of tools [6,9,28] performs the reconfiguration "in-place" by partitioning the configuration changes into smaller units and gradually applying them to the network while preserving correctness.Compared to the first category, "in-place" reconfiguration imposes virtually no overhead, nor does it require router support.Prior techniques like [6,9] enable avoiding forwarding loops during IGP reconfigurations but are not applicable to BGP.Snowcap [28] is the only in-place system to reconfigure BGP.However, it provides correctness guarantees only in steady states.
The literature has proposed many models for distributed routing protocols, particularly BGP, to verify properties of the network's steady state [10,15,29].Most notably, the Stable Paths Problem [14] formulates conditions for a network to converge to a unique state.However, they cannot describe how the network converges.In contrast, Chameleon models transient states during convergence.
Based on previous network models, many promising verification systems have emerged [1,11,13,30].These systems are complementary to Chameleon by verifying the final configuration.Similarly, network management tools [22,31,33] can use our system safely reconfigure the network.
Finally, research has proposed extensions to BGP [17] to guarantee the absence of forwarding loops during convergence.Unfortunately, they have seen little adoption due to additional control logic introduced to BGP speakers.

CONCLUSION
We presented Chameleon, the first system to perform BGP reconfiguration while maintaining correctness throughout the entire reconfiguration process.We proposed a framework to describe the convergence process of BGP and a technique to generate a reconfiguration plan that seamlessly transitions the network from the old to the new configuration.We demonstrated how Chameleon schedules and performs large-scale reconfigurations in real networks within a few minutes while satisfying complex specifications.
Ethical issues.This work does not raise any ethical issues.

APPENDIX
In the following, we provide supporting material that has not been peer-reviewed.

A NETWORK SIZE VS. RECONFIGURATION COMPLEXITY
Table 2 shows the network size and reconfiguration complexity for seven different topologies from TopologyZoo, together with Chameleon's time to schedule the same reconfiguration as in §7.1 (Table 2 shares the same data as Fig. 7).We find stronger correlation between the reconfiguration complexity C and scheduling time than with the number of nodes | | in the network.For instance, the topology Pern has more nodes than Ion, but its scheduling time is smaller by almost two orders of magnitude.However, Ion has a reconfiguration complexity that is around 20 times larger than the one of Pern.

B EXISTENCE OF A SOLUTION
The scheduler presented in §4 only allows a router to change its forwarding decision once from the initial to the final next hop.In the following, we prove that this technique is sufficient for maintaining reachability throughout the reconfiguration.
Theorem 1.Let  be a set of nodes and  = ∈ reach() be the specification.Further, let nh old and nh new :  ↦ →  ∪ {∅,  } be the initial and final forwarding state.Let  :  ↦ → {1, . . ., } be a node schedule, and let nh  be the forwarding state at round  according to .If nh old |=  and nh new |= , then there exists a node schedule  such such that ∀ ∈ {1, . . ., } : nh  |= .
Proof.We proof Theorem 1 by constructing a , such that ∀ : nh  |= .We do so using a breath-first traversal of nh new , as shown in Alg. 1.For each round , the algorithm keeps a set of nodes   ⊆  ∪ { } have updated its state at round , i.e., ∀ ∈   ,  () ≤ .In each round , the algorithm picks one node  ∈   that has  ∉   −1 and nh new () ∈   to migrate at this round  () = .
Notice, that   is always constructed from   −1 by adding a single node  ∈   for which nh new () ∈   −1 .Therefore, at each step , nodes within   form a tree rooted at  within ℎ  .Thus, in every round , ∀ ∈   : nh new () ∈   , and that all nodes  ∈   eventually reach  using nh  .
Algorithm 1 Building a schedule  for transitioning from nh old to nh new .This algorithm is used to proof Theorem 1.

C SUPPLEMENTARY CASE STUDIES
The case study in §6 only considers the Abilene network from Topology Zoo [21].We also run the same scenario for 5 additional topologies: Compuserve, Hibernia Canada, Sprint, JGN2plus, and EEnet that contain 11 or 12 internal routers.Similar to §6, we configure each topology with random OSPF link weights, and we elect three random routers as BGP route reflectors.We simulate three external networks  1 ,  2 ,  3 , advertising the same 1024 prefixes to three internal routers, while the routes towards  1 have a shorter AS path length than the ones from  2 and  3 .The reconfiguration involves removing the eBGP session towards  1 , causing all routers to select either the routes from  2 or  3 , depending on the shorter IGP path.We use the specification from Eq. ( 4) that requires reachability and contains temporal waypoints.
We reconfigure the network both with Snowcap [28] (which simply applies the reconfiguration command) and Chameleon.We find that Snowcap causes black holes for one to two seconds in all scenarios.Snowcap additionally violates waypoint invariants in four out of five scenarios.Chameleon, however, reconfigures the networks while satisfying the specification.It takes Chameleon less than 1 minute to perform the reconfiguration in all five scenarios.

D ABSENCE OF LOOPS
In the following, we proof that the constraints from Eq. ( 2) also ensure the absence of forwarding loops in all rounds.We further evaluate the impact of the explicit constraints from Eq. (3) on the scheduling time of Chameleon.
Theorem 2. Let  be a set of nodes (routers), and let nh old :  ↦ →  ∪ {∅,  } and nh new :  ↦ →  ∪ {∅,  } be the initial and final forwarding state.For each node  ∈  , let   nh be the round in which  changes from its old next hop nh old () to nh new ().If nh old is free of forwarding loops, and if Eq. ( 2) is satisfied for each node  and round  (i.e., all updates of the same round are independent), then the resulting forwarding state in each round  is free of forwarding loops.
Proof.We prove this statement by contradiction.Let  be the first round in which a forwarding loop  = ( 1 , . . .,   , . . .,   ,  1 ) appears.Further, let   ∈  be a node in the loop that changes its next hop to  +1 in round .Node   must exist, since round  − 1 is loop-free.Further, since all updates in round  are independent, no other node   ∈  \ {  } updates its next hop in round .Thus,  .This is equivalent to 0 ≥ 1, and, by contradiction, proves that Eq. ( 2) ensures the absence of forwarding loops.□ Even though the constraints from Eq. ( 2) implicitly ensure loopfreedom, we explicitly add constraints as described in §4.4.We do this because we notice the variance of the scheduling time decreasing when using explicit loop constraints.Fig. 13 shows the difference of solving the same problem with and without explicit constraints.In the most extreme case, the scheduling time with implicit constraints is around 40× larger than solving the same problem with explicit constraints.

E.1 Visualization
The application visualizes the network using four different layers, two of which show the current state of the network while the other two show the configuration.The button 1 on the top left switches between those layers.3) significantly reduces the variance of the scheduling time of Chameleon.This plot shows the scheduling time of Chameleon for the same scenario using different specifications arranged on the -axis (as for Fig. 8).The red points show the scheduling time without explicit loop constraints, while the green points show the scheduling time with explicit constraints.The -axis has a logarithmic scale.
The Data Plane layer displays the forwarding state.Hovering over nodes highlights their forwarding path to reach the destination.Similarly, the Control Plane layer visualizes the propagation of BGP routes as arrows.Hovering over a yellow arrow will display the propagated route attributes, while hovering over a node will show the current routing table of that node, along with its selected route.
On the other hand, both the IGP Config and BGP Config layers visualize parts of the current configuration.Selecting a node will show its complete BGP configuration on the right side of the screen.

E.2 Event-Based Simulation
The application is based on an event-based simulator.Each BGP message is an event, enqueued in a FIFO queue.Once executed, the event has an immediate effect and might enqueue new events.The simulator does not execute events automatically, but when the user clicks on 3 .Clicking the button on the top right 2 displays all currently enqueued events on the right side.This list allows the user to trigger an event explicitly.Finally, 4 executes the entire queue until a violation occurs or the queue is empty.

E.3 Reconfiguration Plan
Clicking the button 5 shows the current reconfiguration plan on the right, grouped into the three phases (collapsed by default).Expanding a phase (and a round of the update phase) reveals the pre-condition 6 , the command 7 , and its post-condition 8 .The left side indicates whether the conditions are satisfied and if the command is already applied.Clicking on a command will execute it, assuming that the pre-conditions are satisfied.
The application continuously verifies the conditions.The current status is shown left of the reconfiguration plan 9 Clicking the icon reveals a list of all satisfied or violated invariants.

Figure 3 :
Figure3: Example network with six internal routers.Two routers  1 and  6 receive a route  1 and  6 , respectively, for the same prefix.The depicted reconfiguration lowers the local preference of  1 from 200 to 50, causing the network to shift from using  1 to  6 .

Figure 5 :
Figure 5: Illustration of the BGP convergence model based on the example from Fig. 3 in the initial state.

Figure 7 :
Figure 7: This figure shows a strong correlation between reconfiguration complexity C  and the scheduling time.Chameleon usually finds a schedule in a few minutes, and up to 8 hours for the largest scenario, in which 660 routers change their next hop.A point corresponds to one of 106 scenarios, and both the -and -axis have a logarithmic scale.

Figure 8 :
Figure8: Temporal expressions have a more significant impact on Chameleon's scheduling time than non-temporal constraints.This plot measures the scheduling time for the same using specifications of different complexity.At   = ∅, the specification only includes reachability, while the specification at   =  contains temporal (blue) or nontemporal (orange) waypoint constraints for all nodes.

Figure 9 :
Figure9: The Cumulative Distribution Function (CDF) of the reconfiguration time  for all 106 scenarios.We approximate Chameleon to reconfigure 85% of scenarios in less than two minutes.

𝑒 3 (𝑒 4 (
a) Link failure after 7 s.Chameleon causes no disruption.Packets are lost for ≈ 0.5 s because of OSPF convergence.b) New route announced at 10s is ignored during the update phase.At 81 s, when we restore the original preferences, all nodes converge to the new route from  4 .

Figure 12 :
Figure12: Chameleon consistently and safely reconfigures a network, while Snowcap[28] triggers black holes, and violates waypoint specifications.We repeat a similar experiment to Fig.1on five additional topologies from Topology Zoo[21], all containing 11 or 12 routers.The reconfiguration causes routers to change their selected route from  1 to either  2 or  3 .The left column shows the total traffic (and violations) for Snowcap, while the right shows the throughput for Chameleon

𝑁 𝜙 = ∅ 𝑁 𝜙 = 𝑁 1 10 100Figure 13 :
Figure 13: Adding explicit constraints to avoid forwarding loops with Eq. (3) significantly reduces the variance of the scheduling time of Chameleon.This plot shows the scheduling time of Chameleon for the same scenario using different specifications arranged on the -axis (as for Fig.8).The red points show the scheduling time without explicit loop constraints, while the green points show the scheduling time with explicit constraints.The -axis has a logarithmic scale.

Table 1
based on the 3-tuple (  old ,   nh ,   new ) of each node .These rules guarantee node  selects P old () until round   old , updates its next hop in round   nh , and then selects P new () starting from round   new .
Figure 6: Most rounds take between 10 and 12 seconds to execute.The highlighted regions visualize the setup or cleanup phase, as well as each round  in the update phase.Most time is spent waiting for the routers to update their route maps.Explore the reconfiguration plan interactively at https://bgpsim.github.io?s=abilene.
In the median case, Chameleon requires 8% more routing table entries compared with the baseline, while SITN requires 96%.This figure shows the CDF of the additional routing table entries required by Chameleon and SITN, normalized by the maximum table size used for the baseline.

Table 2 :
The reconfiguration complexity C  shows stronger correlation with scheduling time than the number of nodes in the network.

1 :
procedure Solve( , nh old , nh new )  ← { ∈  \   | nh new () ∈   }   ← { ∈  \   | nh new () ∈   }Next, we show that we eventually schedule all nodes, i.e., that ∀ ∈  ,  () > 0. For the sake of contradiction, let  * be the set of nodes for which  () = 0. Since nh new |= , each node  ∈  * can reach  in the final state.Let  ∈  * be the node with the shortest path towards  in nh new .Since  () = 0 we know that  ∉   at Line 14.However, this is impossible since nh new () ∈   as  is the node in  * with the shortest path towards .We have already shown that all nodes in   eventually reach  in nh  .Thus, let  ∉   , and let  be the path of  in nh  .If  ∪   = ∅, then  is the same as the path of  in nh old , which must end at the destination since nh old |= .Otherwise, if  ∪   ≠ ∅, then the path  eventually enters   , and thus, nh  |= .This concludes the proof of Theorem 1.□ 6:We now prove that ∀ : nh  |= .To that end, consider any node  ∈  and any round .Let   = { ∈  |  () ≤  } be the set of all nodes already updated in round .