Posts
Monitor Costs in Azure
There are a few ways to monitor costs in Azure. One way is to use the Azure Cost Management + Billing portal. This portal provides a graphical interface that you can use to view your costs over time, track your spending against budgets, and identify areas where you can save money.
Another way to monitor costs is to use the Azure Cost Management API. This API allows you to programmatically access your cost data and integrate it with other systems.
read morePosts
Chronicle Queue and Aeron
Chronicle Queue and Aeron are both high-performance messaging systems, but they have different strengths and weaknesses.
Chronicle Queue is designed for low latency and high throughput messaging within a single machine or cluster. It uses a shared memory ring buffer to store messages, which can achieve very low latency (<1 microsecond) for messages that are sent and received on the same machine. Chronicle Queue also supports persistence, so messages can be written to disk and recovered in the event of a crash.
read morePosts
MLOps with Kubeflow
Kubeflow is an open-source platform for machine learning and MLOps on Kubernetes. It provides a set of tools and components that make it easy to deploy, manage, and scale machine learning workflows on Kubernetes.
Kubeflow includes a variety of components, including:
Notebooks: A Jupyter notebook service that allows data scientists to develop and experiment with machine learning models.
Pipelines: A tool for building and deploying machine learning pipelines.
Experimentation: A tool for tracking and managing machine learning experiments.
read morePosts
confluent kafka vs apache beam
Confluent Kafka and Apache Beam are both open-source platforms for streaming data. However, they have different strengths and weaknesses.
Confluent Kafka is a distributed streaming platform that is used to store and process large amounts of data in real time. It is a good choice for applications that require high throughput and low latency. Kafka is also a good choice for applications that need to be fault-tolerant and scalable.
Apache Beam is a unified programming model for batch and streaming data processing.
read morePosts
AWS Lambda and GCP Cloud
AWS Lambda and Google Cloud Run are both serverless computing platforms that allow you to run code without provisioning or managing servers. However, there are some key differences between the two platforms:
Supported languages: AWS Lambda supports a wide range of programming languages including Node.js, Java, Python, Go, Ruby, and C#. Cloud Run supports Docker images, which can be written in any language. Cold start: When a Lambda function is first invoked, it takes a few milliseconds to start up.
read morePosts
Cloud gotchas 2
Serverless Serverless is great. You create your services and hand them over to AWS Lambda/GCP Cloud Run/Azure Functions and let them rip. Your system can scale up to hundreds of instances and quickly service your clients. However, you must consider
how will your downstream clients respond to such peaks in volume? Will they be able to cope? how must will auto-scaling cost? how portable is your code between serverless platforms? how will you handle bugs in the serverless platform?
read morePosts
Azure create K8 cluster
Here is a Terraform file that you can use to create a Kubernetes cluster in Azure:
provider "azurerm" { version = "~> 3.70.0" subscription_id = var.azure_subscription_id client_id = var.azure_client_id client_secret = var.azure_client_secret tenant_id = var.azure_tenant_id } resource "azurerm_resource_group" "aks_cluster" { name = var.resource_group_name location = var.location } resource "azurerm_kubernetes_cluster" "aks_cluster" { name = var.aks_cluster_name location = azurerm_resource_group.aks_cluster.location resource_group_name = azurerm_resource_group.aks_cluster.name node_count = 3 vm_size = "Standard_D2s_v3" network_profile { kubernetes_network_interface_id = azurerm_network_interface.
read morePosts
AWS vs Azure vs GCP
AWS, Azure, and GCP are the three leading cloud computing platforms in the market. They offer a wide range of services, including compute, storage, databases, networking, machine learning, and artificial intelligence.
Here are some of the key differences between the three platforms:
Market share: AWS is the market leader, with a 33% market share in 2022. Azure is second with a 22% market share, and GCP is third with a 9% market share.
read morePosts
Cloud gotchas 1
Since 2017 I’ve been involved in a wide variety of “cloud” projects and there’s some common myths I’ve observed.
Migrations are just containers Change is hard and unless you’re working for a startup, most cloud transformations start as lift and shift exercises. Contracts have been signed and everyone has been sold the myth that all you need to do is “dockerise” your containers and away you go.
Unfortunately, most of the hyperscalers (cloud provider - GCP, AWS, Azure, etc) will dazzle you with the way they’ve been doing things for years and just tell you and will instruct you to “do as they say”.
read morePosts
Reverse engineering an existing GCP project with terraformer
It can be tough to try to reverse engineer an existing project that has never used terraform. Terraformer can look at an existing project and generate the corresponding terraform code for you. I tried it out on an existing legacy project which used Google Cloud Storage, BigQuery and various service accounts. The setup was a little tricky so I put together a script to simply things. The script assumes you have gcloud setup or a service account key/impersonation and you may need to adjust the –resources parameter.
read more