All posts in category

My Choice of Programming Languages

When I was a kid, I used to spend days tinkering with woodworking tools. I was lucky enough to have a wide set of tools at my disposal. However, there was no one around to give me a hint about what tool to use when. So, I quickly came up with a heuristic: if my fingers and a tool survived an exercise, I've used the right tool; if either the fingers or the tool got damaged, I'd try other tools for the same task until I find the right one. And it worked quite well for me! Since then, I'm an apologist of the idea that every tool is good only for a certain set of tasks.

A programming language is yet another kind of tool. When I became a software developer, I adapted my heuristic to the new reality: if, while solving a task using a certain language, I suffer too much (fingers damage) or I need to hack things more often than not (tool damage), it's a wrong choice of a language.

Since the language is just a tool, my programming toolbox is defined by the tasks I work on the most often. Since 2010, I've worked in many domains, starting from web UI development and ending with writing code for infrastructure components. I find pleasure in being a generalist (jack of all trades), but there is always a pitfall of spreading yourself too thin (master of none). So, for the past few years, I've been trying to limit my sphere of expertise with the server-side, distributed systems, and infrastructure. Hence, the following choice of languages.

Language logos

Read more

Rust - Writing Parsers With nom Parser Combinator Framework

I've been working on my new Rust side-project for several months now, and I've got some learnings to share. The project is called pq - it's a command-line tool to parse and query log files as time series. It comes with its own domain-specific language that is highly influenced by PromQL. A typical pq usage may look like this:

tail -f /var/log/nginx/access.log | pq '
/...some fancy regex.../
| map {
    .0 as ip,
    .1:ts,
    .2 as method,
    .3:str as status_code,
    .4 as content_len
  }
| select topk(
      10,
      sum(
          count_over_time(
              __line__{method="GET", status_code="200"}[1s]
          )
      ) by (ip)
  )
| to_json
'

pq has many components, including various log parsing strategies and a pretty sophisticated query execution engine. But surprisingly or not, about half of the time I've put into this project so far was dedicated to writing the parser of the pq's own query language. To be honest, when I was starting the project, I didn't see that coming...

nom logo

Luckily, writing a parser in Rust was mostly a pleasant experience, thanks to a crate concisely named nom. Although learning how to write parsers with nom wasn't completely seamless. So here is my journey.

Read more

pq - parse and query log files as time series

I often find myself staring at Nginx or Envoy access logs flooding my screens with real-time data. My only wish at such moments is to be able to aggregate these lines somehow and analyze the output at a slower pace, ideally, with some familiar and concise query language. And to my surprise, I haven't met a tool satisfying all my requirements yet. Well, I should be honest here - I haven't done thorough research. But if there would be a tool as widely known as jq for JSON, I wouldn't miss it probably.

So, here we go - my attempt to write a full-fledged parsing and query engine and master Rust at the same time. Yes, I know, it's a bad idea. But who has time for good ones?

First things first - a usage preview:

Read more

Understanding Rust Privacy and Visibility Model

I spent the last couple of months writing code in Rust. It was probably my third or fourth attempt to write something substantial in this language. And every time my level of understanding of things deepened. I'm by no means a Rust expert so probably I'll be extremely inaccurate in the terminology here. And likely I'll get lots of technical details wrong too. But I had this epiphany moment of how the visibility and privacy model works in Rust so I can't help but think of sharing it with someone else.

Read more

Go, HTTP handlers, panic, and deadlocks

Maybe the scenario I'm going to describe is just a silly bug no seasoned Go developer would ever make, but it is what it is.

I'm not an expert in Go but I do write code in this language from time to time. My cumulative number of LOC is probably still below 100 000 but it's definitely not just a few hundred lines of code. Go always looked like a simple language to me. But also it looked safe. Apparently, it's not as simple and safe as I've thought...

Here is a synthetic piece of code illustrating the erroneous logic I stumbled upon recently:

// main.go
package main

import (
    "fmt"
    "sync"
)

func main() {
    mutex := &sync.Mutex{}

    f := func() {
        fmt.Println("In f()")

        defer func() {
            if r := recover(); r != nil {
                fmt.Println("Recovered", r)
            }
        }()

        dogs := []string{"Lucky"}

        mutex.Lock()
        fmt.Println("Last dog's name is", dogs[len(dogs)])
        mutex.Unlock()
    }

    f()

    fmt.Println("About to get a deadlock in main()")
    mutex.Lock()
}

Read more

Writing Web Server in Python: sockets

What is a web server?

Let's start by answering the question: What is a web server?

First off, it's a server (no pun intended). A server is a process [sic] serving clients. Surprisingly or not, a server has nothing to do with hardware. It's just a regular piece of software run by an operating system. Like most other programs around, a server gets some data on its input, transforms data in accordance with some business logic, and then produces some output data. In the case of a web server, the input and output happen over the network via Hypertext Transfer Protocol (HTTP). For a web server, the input consists of HTTP requests from its clients - web browsers, mobile applications, IoT devices, or even other web services. And the output consists of HTTP responses, oftentimes in form of HTML pages, but other formats are also supported.

Read more

Working with container images in Go

I've been working on adding basic images support to my experimental container manager and to my surprise, the task turned to be more complex than I initially expected. I spent some time looking for ways to manage container images directly from my application code. There is plenty of tools out there (docker, containerd, podman, buildah, cri-o, etc) providing image management capabilities. However, if you don't want to have a dependency on an external daemon running in your system, or you don't feel like shelling out for exec-ing a command-line tool from the code, the options are at best limited.

I've reviewed a bunch of the said tools focusing on the underlying means they use to deal with images and at last, I found two appealing libraries. The first one is github.com/containers/image library "[...] aimed at working in various way with containers' images and container image registries". The second one is github.com/containers/storage "[...] which aims to provide methods for storing filesystem layers, container images, and containers". The libraries are meant to be used in conjunction and form a very powerful image management tandem. But unfortunately, I could not find a sufficient amount of documentation, especially how to get started kind of it.

Without the docs the only way to learn how to use the libraries for me was to analyze the code of their dependants (most prominently - buildah and cri-o). It took me a while to forge a working example which is capable of:

  • pulling images from remote repositories;
  • storing images locally;
  • creating and mounting containers (i.e. writable instances of images).

In the rest of the article, I'll try to show how to use the libraries to perform the said task and highlight the most interesting parts of this journey.

Disclaimer: This is by no means an attempt to fully or even partially document the libraries!

Read more

Master Go While Learning Containers

I spent half a year deep-diving into the world of containers and their orchestration. I have been enjoying it very much and learned a lot. On my journey, I need to tackle lots of interesting and specific concepts. But there is one commonality almost every project in this area possesses. When it comes to containers - the Go programming language is ubiquitous!

Read more

My 10 Years of Programming Experience

Regardless of whether it's the end of the calendar decade or not it's the end of a programming decade for me. I started early in 2010 and since then I've been programming almost every day, including weekends and vacations. This was a really exciting period in my life and I realized that it's been a while since 2010 only recently. So, I decided to put into words some of my learnings from that time. Warning: the content of this article is highly opinionated and extremely subjective.

Read more

From Callback Hell to async/await Heaven

In the previous article, we learned how to implement a simple but workable event loop. However, programs which are supposed to be run by the event loop are full of callbacks. This is the usual problem of event-loop-driven environments. When business logic becomes reasonably complicated, callbacks make program's code hardly readable and painfully maintainable. And the callback hell begins! There is plenty of ways to deal with the artificial complexity arose due to callbacks, but the most impressive one is to make the code great flat again. And by flat, I mean callback-less and synchronous-like. Usually, it's done by introducing async/await syntactic feature. But every high-level abstraction is built on top of some basic and fundamental ideas. Let's check the async/await sugar out and see what exactly happens under the hood.



Read more

Explaining event loop in 100 lines of code

There is plenty of articles out there about the event loop. However, as a software engineer, I prefer to read code, not text. And there is no better way of learning a new concept than implementing it yourself. So, let's try to grasp the idea of the event loop by coding a new and shiny one.

NB: In the article, we will try to describe the idea of the event loop in general, not a specific implementation of the event loop in Node.js or Python, or some other language/library.

Read more

Save the day with gevent

One day you were hired by a startup to build yet another web site service. The problem sounded similar to numerous back-ends you had been doing before. So, you decided to move with python to be conservative enough and pick flask to be lightweight but yet powerful enough. Nobody was fired for buying IBM starting things in python, wise man's choice.

Read more

Truly optional scalar types in protobuf3 (with Go examples)

In contrast to protobuf2 there is no way in protobuf3 to mark some fields as optional and some other fields as required. Instead, any field might be omitted leading this field to be set to its default zero-value. I believe there were many good reasons for such a design decision. However, while this behavior might be superior to the proto2's explicit distinction between required and optional fields, it also has some unfortunate implications.

Read more

Node.js Writable streams distilled

Writable streams are an abstraction for a destination to which data is written... And this time it's a concise abstraction! Compared to vague readable streams (multiple operation modes behind single abstraction) writable streams implement only one mode and hence expose only one essential method write(chunk). Nevertheless, the idea of writable streams is not trivial and it's due to one nice feature it provides - backpressure. Let's try to wrap our mind around writable stream abstraction.

Read more

Node.js Readable streams distilled

Readable stream is an abstraction for some data source. Which could be hard to grasp and even harder to use...

Everybody knows that readable streams support two modes of operating (flowing and paused) and piping to writable streams. It's not that easy to understand the purposes of these mechanisms and behavioral differences though. Since one readable stream abstraction stands for multiple usage modes its public interface (i.e. the set of methods and events) is a bit inconsistent. Usage of readable streams might be totally confusing without the understanding of the underlying ideas. In this article, we will make an attempt to justify the abstraction of readable streams trying to implement our own file reader. Also, we will have a look at some nicer ways to consume readable streams.

Read more