An invaluable start on how to start managing GCP projects with Terraform. I wish I’d found this a year ago.

Rather than fully configuring your backend.tf in a file.``` terraform { backend “gcs” { bucket = “my-bucket-123” prefix = “terraform/state” } I prefer to use the command line in order avoid polluting the code with any environment specific names. terraform init \ -backend-config=“bucket=my-bucket-123” \ -backend-config=“prefix=terraform/state

I’m glad Google are finally starting to embrace Terraform by creating their own modules. Version 0.1.0 of the project-factory looks really promising.

The DA Platform is a huge step forward for running Flink at scale. I was lucky enough to see a demo and was really impressed. Far more advanced that the what can be achieved with Dataflow at the moment.

Here are some tips on how to create an effective SRE culture: Start with the right mindset. SRE is a mindset that sees reliability as everyone’s responsibility, not just the responsibility of the SRE team. It is important to create a culture where everyone is empowered to take ownership of reliability and to make decisions that will improve the reliability of the systems they work on. Embrace failure. Failure is inevitable, so it is important to create a culture where failure is seen as an opportunity to learn and improve.

Here are some tips on how to deliver reliable, high-throughput, low-latency (micro)services: Design your services for reliability. This means designing your services to be fault-tolerant, scalable, and resilient. You can do this by using techniques such as redundancy, load balancing, and caching. Use the right tools and technologies. There are a number of tools and technologies that can help you to deliver reliable, high-throughput, low-latency microservices. These include messaging systems, load balancers, and caching solutions.

The premise was simple. Use “big” data analytics and machine learning models to predict the movement of stock prices. However, we had really “dirty” data and our Data Scientists were stuggling to seperate the noise from the signals. We spent a lot of time cleaning the data and introducing good old principles like “how can I run the model somewhere over than a laptop?”. This was a true startup, a bunch of people in a room trying to get stuff working.

I’m currently bench-marking Flink against Google Cloud Dataflow using the same Apache Beam pipeline for quantitative analytics. One observation I’ve seen with Flink is the tail latency associated with some shards. Google Cloud Dataflow can optimise away stragglers in large jobs using “Dynamic Workload Rebalancing". As far as I know, Flink is currently unable to perform similar optimisations.

The point of cryptocurrency is to provide a decentralized, secure, and efficient way to transfer value. Cryptocurrencies are not issued by any central authority, such as a government or bank, and they are not backed by any physical asset. Instead, they are created and maintained by a network of computers that are running a special software program. This software program is designed to verify and record cryptocurrency transactions, and to prevent fraud.

Great talk by by Peter Lawrey regarding latency in micro-services. https://www.infoq.com/presentations/latency-sensitive-microservices/