Google Kubernetes Engine (GKE) is a popular container orchestration platform that allows developers to deploy and manage containerized applications at scale. However, a recent security vulnerability has been discovered in GKE that could allow attackers to gain access to clusters and steal data or launch denial-of-service attacks.

The vulnerability is caused by a misunderstanding about the system:authenticated group, which includes any Google account with a valid login. This group can be assigned overly permissive roles, such as cluster-admin, which gives attackers full control over a GKE cluster.

AWS App Runner and Google Cloud Run are two serverless computing platforms that can help you deploy and run containerized applications without having to worry about servers. Both platforms are relatively new, but they have quickly become popular choices for developers.

What are the similarities?

Both platforms are serverless, meaning that you don’t have to provision or manage servers. The platforms will automatically scale your application up or down based on demand, so you only pay for the resources that you use. Both platforms support containerized applications. This means that you can package your application code and dependencies into a container and deploy it to the platform. Both platforms are easy to use. You can deploy your application with a few clicks or a few commands. Both platforms are scalable. They can automatically scale your application up or down based on demand, so you can handle even the most unpredictable traffic spikes.

Azure networking and GCP networking are both comprehensive cloud networking services that offer a wide range of features and capabilities. However, there are some key differences between the two platforms.

Azure networking offers a more traditional networking model, with a focus on virtual networks (VNets), subnets, and network security groups (NSGs). VNets are isolated networks that can be used to group together resources, such as virtual machines (VMs), storage, and applications. Subnets are smaller subdivisions of a VNet, and they can be used to further isolate resources. NSGs are used to control traffic flow within and between VNets.

It can be tough to try to reverse engineer an existing project that has never used terraform. Terraformer can look at an existing project and generate the corresponding terraform code for you.  I tried it out on an existing legacy project which used Google Cloud Storage, BigQuery and various service accounts. The setup was a little tricky so I put together a script to simply things. The script assumes you have gcloud setup or a service account key/impersonation and you may need to adjust the –resources parameter.

One hour ago:

bq cp mydataset.table@-3600000 mydataset.table_restored

Absolute (ms since UNIX epoch) GMT: Wednesday, 26 May 2021 13:41:53 = 1622036513000 https://www.epochconverter.com/

bq cp mydataset.table@1622036513000 mydataset.table_restored

More on Bigquery time travel

A great way to split up your pipeline based on the urgency of results aggregate-data-with-dataflow

After the recent GCP outage related to IAM, I found some odd behaviour with gsutil/gcloud. A script that had faithfully run for many months stopped working with:

ServiceException: 401 Anonymous caller does not have storage.buckets.list access to project xxxx
I tried recreating the service account key used for the operation with no luck. To fix the problem, I had to create a new bucket!

gsutil mb -b on -l us-east1 gs://my-awesome-bucket123ed321/
Creating gs://my-awesome-bucket123ed321/…
gsutil ls
gs://my-awesome-bucket123ed321/
gs://<my_original_bucket_I_couldnt_see>/

You can finally set budgets via the API in GCP. This is a huge relief to all those org admins out there who have had to do this manually.

AND, hold on to your hats, there’s terraform support as well! 

Looks like Christmas came late….

data "google_billing_account" "account" {  
  provider = google-beta  
  billing_account = "000000-0000000-0000000-000000"  
}  
  
resource "google_billing_budget" "budget" {  
  provider = google-beta  
  billing_account = data.google_billing_account.account.id  
  display_name = "Example Billing Budget"  
  amount {  
    specified_amount {  
      currency_code = "USD"  
      units = "100000"  
    }  
  }  
  threshold_rules {  
      threshold_percent =  0.5  
  }  
}

An invaluable start on how to start managing GCP projects with Terraform. I wish I’d found this a year ago.

Rather than fully configuring your backend.tf in a file.```

terraform {
backend “gcs” {
bucket = “my-bucket-123”
prefix = “terraform/state”
}

I prefer to use the command line in order avoid polluting the code with any environment specific names.

terraform init \
-backend-config=“bucket=my-bucket-123” \
-backend-config=“prefix=terraform/state

I’m glad Google are finally starting to embrace Terraform by creating their own modules. Version 0.1.0 of the project-factory looks really promising.

I’m currently bench-marking Flink against Google Cloud Dataflow using the same Apache Beam pipeline for quantitative analytics. One observation I’ve seen with Flink is the tail latency associated with some shards. 

Google Cloud Dataflow can optimise away stragglers in large jobs using “Dynamic Workload Rebalancing".  As far as I know, Flink is currently unable to perform similar optimisations.

This one is better explained with the presentation below. If you want to learn how to run quantitative analytics at scale, it’s well worth a watch.

Our team recently completed a challenging yet rewarding project: building a scalable and portable risk engine using Apache Beam and Google Cloud Dataflow. This project allowed us to delve deeper into distributed computing and explore the practical application of these technologies in the financial domain.