Your organization has 30 VPCs across multiple AWS accounts and regions. You need all VPCs to communicate with each other for shared services (databases, cache, APIs). Currently, you have 30x29/2 = 435 VPC peering connections (each pair must peer directly). This is unmaintainable. Design a scalable topology using Transit Gateway that reduces complexity and improves security.
Transit Gateway (TGW) is a hub-and-spoke architecture that solves this: instead of 435 peering connections, all VPCs connect to one TGW, resulting in 30 attachments. Steps: (1) Create a Transit Gateway: `aws ec2 create-transit-gateway --description "Hub for 30 VPCs" --options AmazonSideAsn=64512,DefaultRouteTableAssociation=enable,DefaultRouteTablePropagation=enable`. (2) Attach each VPC to the TGW: `aws ec2 create-transit-gateway-vpc-attachment --transit-gateway-id tgw-xxxxx --vpc-id vpc-1 --subnet-ids subnet-xxxxx subnet-yyyyy` (repeat for all 30 VPCs). Use multiple subnets for multi-AZ resilience. (3) Enable route propagation for each attachment — this automatically adds routes in the TGW route table: `aws ec2 enable-transit-gateway-route-table-propagation --transit-gateway-route-table-id tgw-rtb-xxxxx --transit-gateway-attachment-id tgw-attach-xxxxx`. (4) Update VPC route tables to send cross-VPC traffic to the TGW: `aws ec2 create-route --route-table-id rtb-xxxxx --destination-cidr-block 0.0.0.0/0 --transit-gateway-id tgw-xxxxx` (or more specific CIDR ranges). (5) Use security groups and NACLs to control traffic between VPCs — TGW doesn't provide security boundaries, so enforce them at the VPC level. For multi-account deployments, use AWS Resource Access Manager (RAM): `aws ram create-resource-share --name "TGW Share" --resource-arns arn:aws:ec2:region:account:transit-gateway/tgw-xxxxx --principals arn:aws:organizations::root:organization/o-xxxxx --allow-external-principals`. This shares the TGW across accounts. Benefits: (1) 30 attachments instead of 435 connections. (2) Centralized management — security policies enforced at TGW. (3) Multi-region support — TGW peering connects TGWs in different regions. (4) Reduced operational overhead. For 30 VPCs, TGW reduces complexity from N*(N-1)/2 = 435 to 30 + management overhead. Cost: TGW costs ~$0.32/hour + $0.02/GB data processed. For organizations with 30+ VPCs, TGW is cost-effective despite the hourly charge.
Follow-up: You deploy Transit Gateway with 30 VPC attachments. Traffic flows correctly, but you want to isolate departments — Accounting VPCs should only communicate with each other and the Shared Services VPC, not with Engineering VPCs. How do you configure this isolation within the same TGW?
You're using AWS PrivateLink to expose a service (running on EC2 in VPC-A) to other VPCs (VPC-B, VPC-C) without routing traffic through the internet. You set up a Network Load Balancer (NLB) in VPC-A, created a VPC Endpoint Service, and VPC-B created a VPC Endpoint to connect. Traffic works. But you want to add multi-region redundancy — the service should also be accessible via PrivateLink from a different region. Design this.
PrivateLink doesn't natively support multi-region, but you can implement redundancy with VPC peering + PrivateLink or use multiple regional PrivateLink endpoints. Approaches: (1) Regional PrivateLink endpoints — create a separate VPC Endpoint Service in each region (e.g., us-east-1 and eu-west-1) pointing to regional NLBs. VPCs in each region connect to the regional endpoint. Limitation: VPCs in us-east-1 cannot directly access eu-west-1 PrivateLink endpoint; they must use a different connection method. (2) VPC peering between regions + single PrivateLink endpoint — peer the regional VPCs to VPC-A (region 1), then route all PrivateLink traffic through the peering connection. However, PrivateLink endpoints are region-specific and cannot route through peering transparently. (3) Most practical: Run the service (NLB) in multiple regions and use Route 53 private hosted zones with latency-based or geoproximity routing to direct traffic to the nearest regional endpoint. Each region has its own VPC Endpoint Service. VPC-B connects to both regional endpoints and Route 53 returns the nearest one. Setup: (1) In region 1 (us-east-1): Create NLB → VPC Endpoint Service → VPC Endpoint in consumer VPCs. (2) In region 2 (eu-west-1): Create NLB → VPC Endpoint Service → VPC Endpoint in consumer VPCs. (3) Create Route 53 private hosted zone (e.g., `service.internal`): `aws route53 create-hosted-zone --name service.internal --vpc VPCRegion=us-east-1,VPCId=vpc-xxxxx`. (4) Add A records for both regional endpoints with latency-based routing: `aws route53 change-resource-record-sets --hosted-zone-id zone-id --change-batch file://routing-policy.json` with latency routing to each region. Limitation: PrivateLink endpoints are private DNS names; Route 53 must resolve them within the VPC. For production multi-region: use AppMesh, Global Accelerator, or DNS-based failover rather than PrivateLink. PrivateLink is designed for single-region service exposure. If you need true multi-region redundancy with PrivateLink, run the full service stack in each region and use DNS failover or Global Load Balancer (not PrivateLink) for cross-region traffic.
Follow-up: You set up regional PrivateLink endpoints with Route 53 latency-based routing. VPCs in both regions can access the service. But during a region failover (us-east-1 endpoint unavailable), Route 53 continues to route some traffic to the failed endpoint. Why?
You connect 10 customer VPCs to your shared service VPC using AWS PrivateLink. Each customer has their own VPC Endpoint connecting to your VPC Endpoint Service. Traffic works, but you want to implement per-customer rate limiting and access control without managing 10 different NLBs or firewalls. How do you implement this?
PrivateLink endpoints preserve the source IP (Elastic Network Interface IP from the customer VPC), which you can use for access control and rate limiting. Implementation: (1) Use VPC Endpoint principals — configure the VPC Endpoint Service to allow only specific AWS principals (accounts or roles). `aws ec2 modify-vpc-endpoint-service-permissions --service-name com.amazonaws.vpce.region.vpce-svc-xxxxx --add-permitted-principals '["arn:aws:iam::ACCOUNT:root"]'`. This restricts which customers can connect. (2) Behind the NLB, run EC2 instances or Lambda (via ALB) that implement per-customer logic: (a) Extract source IP from the request or connection logs. (b) Validate against a customer whitelist. (c) Apply rate limiting per customer using a token bucket (e.g., Redis cache). (3) Use Target Group attributes to implement stickiness — route requests from the same customer VPC endpoint to the same backend instance, enabling per-customer session tracking. `aws elbv2 modify-target-group-attributes --target-group-arn arn:aws:elasticloadbalancing:... --attributes Key=stickiness.enabled,Value=true Key=stickiness.type,Value=source_ip`. (4) Log and monitor traffic per customer using VPC Flow Logs or NLB access logs: `aws ec2 create-flow-logs --resource-type VPC --resource-ids vpc-xxxxx --traffic-type ALL --log-destination-type cloud-watch-logs --log-group-name /aws/vpc/flowlogs`. (5) For rate limiting, use AWS WAF (if ALB is in front of NLB) or implement rate limiting in application code: `from limits import key_func; limiter = Limiter(key_func=lambda: get_customer_id()); @limiter.limit("100/minute")`. Most practical: Log endpoint connections, extract source IP + customer ID, apply rate limiting in application code or WAF rules. For multi-tenant PrivateLink, logging and rate limiting at the application layer (Layer 7) is most flexible. Alternatively, use AWS Network Access Analyzer to map and audit per-customer access.
Follow-up: You implement per-customer rate limiting (100 requests/minute per customer) in your backend application. Most customers respect this, but one customer is sending 500 requests/min and hitting the limit. They claim they have legitimate use cases. How do you handle this?
Your organization uses both VPC Peering (for a few critical VPCs) and Transit Gateway (for 20+ other VPCs). A VPC-A (peered to VPC-B) needs to communicate with VPC-C (connected via TGW). Currently, traffic from VPC-A to VPC-C requires routing through VPC-B, which adds latency and complexity. Design a unified network topology.
The issue is that VPC peering and TGW are separate connectivity mechanisms. To unify them: (1) Simplest approach: Migrate all VPCs from peering to TGW. Remove peering connections: `aws ec2 delete-vpc-peering-connection --vpc-peering-connection-id pcx-xxxxx`. Attach VPC-A and VPC-B to the same TGW and disable peering. This centralizes all connectivity in TGW. (2) Hybrid approach (if migration is risky): Connect the peered VPCs to TGW as well. VPC-A connects via both peering (to VPC-B) and TGW (to VPC-C). Route traffic to VPC-B through peering (shorter, more direct), and traffic to VPC-C through TGW. (3) For gradual migration: Create TGW attachments for critical VPCs first, then migrate others. During transition, use dual-stack routing: route tables include both peering and TGW targets, allowing gradual cutover. Implementation: (1) Attach VPC-A, VPC-B, VPC-C to a single TGW: `aws ec2 create-transit-gateway-vpc-attachment --transit-gateway-id tgw-xxxxx --vpc-id vpc-A --subnet-ids ...` (repeat for B and C). (2) Update route tables: In VPC-A, add route for VPC-C CIDR → TGW. Keep route for VPC-B CIDR → Peering (or switch to TGW). (3) For VPC-B and VPC-C, add routes → TGW for all other VPCs. (4) If peering and TGW both exist, prefer TGW by setting longer-prefix routes or lower metric in peering. BGP advertises more specific prefixes, so TGW can advertise /25 while peering is /24. Most practical: Complete migration to TGW. Peering is deprecated for multi-VPC deployments. TGW provides centralized management, network segmentation (using route tables), and scales to 5000+ attachments. For VPC-A ↔ VPC-B ↔ VPC-C, TGW eliminates the need for routing through intermediaries.
Follow-up: You migrate all VPCs to TGW, removing peering. Connectivity works, but latency between VPC-A and VPC-B increases from 1ms (with peering) to 3ms (with TGW). Why the increase?
You have a core shared services VPC (VPC-Services) that houses databases, caches, and APIs. 50 application VPCs need access to VPC-Services. You use Transit Gateway to connect all 51 VPCs (1 core + 50 app). Traffic from app VPCs to VPC-Services works. But traffic between app VPCs should not be allowed (each app team manages their own VPC). How do you enforce this isolation while using a single TGW?
Transit Gateway route tables allow fine-grained control over which attachments can communicate. By default, all attachments can reach each other (if route propagation is enabled). To enforce isolation: (1) Create separate TGW route tables: (a) Route table for VPC-Services — propagate all app VPC attachments so services can see all apps. (b) Route tables for each app VPC — include only VPC-Services attachment + its own CIDR, no other app VPCs. (2) Disable default route propagation: `aws ec2 modify-transit-gateway --transit-gateway-id tgw-xxxxx --options DefaultRouteTablePropagation=disable`. (3) Create route tables: `aws ec2 create-transit-gateway-route-table --transit-gateway-id tgw-xxxxx --tag-specifications 'ResourceType=transit-gateway-route-table,Tags=[{Key=Name,Value=app-rt},{Key=Type,Value=application}]'`. Do this 51 times (1 for services, 50 for apps). (4) For each app VPC attachment, associate it with its own route table: `aws ec2 associate-transit-gateway-route-table --transit-gateway-route-table-id tgw-rtb-app-1 --transit-gateway-attachment-id tgw-attach-app-1`. (5) Create static routes in app route tables to reach VPC-Services: `aws ec2 create-transit-gateway-route --transit-gateway-route-table-id tgw-rtb-app-1 --destination-cidr-block 10.1.0.0/16 --transit-gateway-attachment-id tgw-attach-services`. (6) In VPC-Services route table, enable propagation for all 50 app attachments: `aws ec2 enable-transit-gateway-route-table-propagation --transit-gateway-route-table-id tgw-rtb-services --transit-gateway-attachment-id tgw-attach-app-1` (repeat 50 times). Result: App VPCs can only reach VPC-Services and themselves; they cannot reach other app VPCs. VPC-Services can reach all app VPCs. This is a hub-and-spoke topology with isolation. Alternative (simpler for small N): Use network ACLs or security groups to deny traffic between app VPCs at the VPC level, even if TGW allows routing. This is less efficient but simpler to manage. Most practical for 50 VPCs: TGW route tables with the isolation described above. Provides infrastructure-level security and scales well.
Follow-up: You configure TGW route tables for hub-and-spoke with isolation. Each app VPC has its own route table. Traffic works, but management is complex — 51 route tables and 100 static routes to maintain manually. How do you automate this?
Your organization is using both Transit Gateway and PrivateLink for different workloads. Some services are exposed via TGW (within AWS), others via PrivateLink (to external partners). A service in VPC-A (connected via TGW) needs to call an API in VPC-B that's exposed only via PrivateLink. VPC-A can't directly reach the PrivateLink endpoint. Design a solution.
PrivateLink endpoints are private to the VPCs that create them; TGW attachments don't have built-in access to PrivateLink endpoints in peered VPCs. Solutions: (1) Expose the API via both TGW and PrivateLink — keep the PrivateLink endpoint for external partners, and add an internal NLB accessible via TGW for internal VPCs. This requires running two NLBs but provides flexibility. (2) Use PrivateLink with TGW — create VPC Endpoint for the PrivateLink service in VPC-A, then route TGW traffic to the local endpoint. However, this requires endpoints in every VPC that needs access. (3) Most practical: Use a service proxy in the service VPC (VPC-B). The proxy exposes both: (a) PrivateLink endpoint to external partners, (b) Internal NLB accessible via TGW to VPC-A and other internal VPCs. Diagram: External Partner → PrivateLink → Proxy (in VPC-B) ← TGW ← VPC-A. The proxy can route traffic to the backend service. (4) Alternative: If the API is stateless, duplicate it — run the API in both VPC-B (for internal access via TGW) and expose one instance via PrivateLink for external partners. Both point to the same backend database. Implementation: (1) In VPC-B, run an internal NLB (not exposed via PrivateLink): `aws elbv2 create-load-balancer --name internal-nlb --subnets subnet-xxxxx --scheme internal --type network`. (2) Register the API backend to the internal NLB: `aws elbv2 register-targets --target-group-arn tg-arn --targets Id=i-xxxxx Port=8080`. (3) Attach NLB to TGW or advertise its IP via TGW: `aws ec2 create-transit-gateway-route --transit-gateway-route-table-id tgw-rtb --destination-cidr-block 10.2.0.0/16 --transit-gateway-attachment-id tgw-attach-b`. (4) In VPC-A's route table, add: `10.2.0.0/16 → TGW`. Now VPC-A can reach the internal NLB via TGW. (5) For external partners, keep the PrivateLink endpoint pointing to the same backend or a proxy. This design separates internal and external access, allowing different networking patterns.
Follow-up: You run an internal NLB in VPC-B accessible via TGW, and a separate PrivateLink endpoint for external partners. Both route to the same backend service. During traffic spikes, the backend is CPU-bound. You're considering auto-scaling, but you have both TGW and PrivateLink traffic. How do you ensure both can scale together?
You're migrating from a complex VPC peering mesh (30 VPCs, 435 peering connections) to Transit Gateway. During cutover, you need zero downtime. Traffic must switch from peering to TGW without interruption. Design the cutover plan.
Zero-downtime cutover from peering to TGW requires careful sequencing. Plan: (1) Preparation phase (day 1-3): (a) Create TGW and attach all 30 VPCs: `aws ec2 create-transit-gateway` and `create-transit-gateway-vpc-attachment` for each. (b) Enable route propagation or add static routes to TGW. (c) Test connectivity in parallel environments (route traffic to TGW on test VPCs first). (d) Do NOT delete peering connections yet. (2) Cutover phase (day 4): (a) Update route tables to prefer TGW over peering (lower cost/metric). In AWS, routing prefers longer-prefix matches; if both peering and TGW have the same prefix, use route priority or local preference. (b) Gradually shift traffic by updating route tables on a few VPCs at a time: start with non-critical VPCs, monitor for errors, then move to critical ones. (c) For each VPC, update its route table: delete the peering route, add the TGW route. Or keep both routes and let longest-prefix-match decide (if TGW is more specific). Example: Peering route 10.0.0.0/8 → Peering, TGW route 10.1.0.0/16 → TGW. Traffic to 10.1.0.0/16 uses TGW (longer prefix), others use peering. (d) Gradually remove peering connections as traffic migrates. (3) Post-cutover phase (day 5+): (a) Verify all traffic flows through TGW. (b) Remove remaining peering connections: `aws ec2 delete-vpc-peering-connection`. (c) Clean up peering routes from VPC route tables. (4) Monitoring: (a) Use VPC Flow Logs to track traffic before and after: `aws ec2 create-flow-logs --resource-type VPC --resource-ids vpc-xxxxx`. (b) Compare packet loss, latency, and error rates. (c) Set CloudWatch alarms for connection errors. (5) Rollback plan: If issues occur, revert route tables to peering: `aws ec2 replace-route --route-table-id rtb-xxxxx --destination-cidr-block 10.0.0.0/8 --vpc-peering-connection-id pcx-xxxxx`. Keep peering connections active for 1-2 weeks after TGW cutover in case you need to rollback. Key: Dual-stack routing during cutover. Keep both peering and TGW active, gradually shift traffic by updating route priorities, then remove peering after TGW is proven stable. This minimizes blast radius and allows quick rollback.
Follow-up: You complete the TGW migration and remove peering connections. Two weeks later, an application reports intermittent "destination unreachable" errors for a specific peer VPC (one they historically communicated with directly via peering). Peering is already deleted. How do you troubleshoot?