Understand GCP Cloud Traffic with VPC Flow Logs

Google Cloud Platform is adding more tools to help organizations build and manage software-defined virtual networks; the latest is logging for performance analysis and network forensics.

Google Virtual Private Cloud provides a private software-defined network layer on top of GCP’s public cloud. It includes tools for managing IP addresses, routing, security, and integration with on-premises resources and other public clouds; think of it as a control plane for your cloud virtual infrastructure.

Modern networks need tooling to capture and analyze network traffic to ensure secure and stable network operations. You need to be able to record IP traffic across your VMs and export the resulting data in a format that can be analyzed by your choice of network management tooling. The new VPC Flow Logs are tools for capturing this information without needing to install agents for specific VPC networks and subnets down to individual VMs and virtual NICs. Captured near real time, you can work with it in Google’s native logging tools or third-party applications.

Each Compute Engine VM captures its own flow logs; the data is collected and delivered to your logging endpoints every five seconds. There’s a lot of information in a VPC Flow Logs record, above and beyond connection details. You get throughput and RTT details as well as GCP-specific information about connections inside VPC networks and connections to Google services.

Beyond simple monitoring, this helps you to map out network performance, choose how to rebalance connections and - when possible - where you can begin to reconfigure and refactor services. Flow Logs data will help architects looking at how to break down a monolithic application into microservices or planning how to deploy containerized workloads using tools like Kubernetes.

This level of detail can also help you understand traffic patterns and map growth, simplifying how you scale applications and services or enabling automation of deployment of additional resources. Cloud services like GCP charge for network connections between zones and regions, with different rates for network egress in different geographies. Getting a clearer picture of your network flows and the cost implications can help you rebalance resources and optimize network traffic to minimize transit costs.

You could send a real-time feed from Flow Logs to a machine learning model to analyze when traffic patterns show there’s a problem with the WAN connection from your data center to processing resources running on GCP, for example.

This scale of network logging is also important for security analytics. You can use Flow Logs with security analysis tools when you’re investigating patterns of network activity that indicate intrusion or compromise rather than network issues or a spike in customer demand. As well as investigating after the fact, logs can trigger alerts for suspicious activity. You could use machine learning tools to build a model of normal operations and use it to spot possible breaches early, detecting what could be the exfiltration of stolen data from your network; or you could pipe the logs into pre-built services that can perform the analysis and generate alerts.

Logs are initially stored in Google’s Stackdriver logging platform, which can be used to bring log data in from other sources, including AWS. Logs in Stackdriver can include data from inside VMs, as well as from VPC Flow Logs, and the Stackdriver monitoring API has also just been updated with new endpoints that should simplify managing alerting policies and notification channels. You can also bring in on-premises sources so you can analyze traffic across your data centers and your cloud infrastructures. Logs can also be exported to BigQuery or delivered to Google’s Cloud Pub/Sub service to export to real-time analytics and SIEM platforms. The same tool can deliver data to third-party tooling with the option of filtering outputs as required. Initially, this supports Cisco Stealthwatch and Sumo Logic.

Azure and AWS have already had network flow logging and analysis options for some time, with Azure’s Network Watcher and AWS VPC Flow Logs, which track all inbound and outbound traffic to instances in your AWS VPC (including traffic that’s rejected as well as accepted). Google is working through the list of enterprise-grade tooling needed to make its cloud a viable option for larger organizations, and VPC Flow Logs adds a much need level of visibility to GPC cloud infrastructures.