Vagiz Duseev

Getting started with Wutch

2021-04-04T20:00:00+00:00

Wutch is a python based live server that observes changes in the given directories and executes a shell command on each change. It also, optionally, renders the results in the default web browser, automatically refreshing each web page after every change.

You can use wutch with Sphinx, Jekyll, and other static site generators. On the GIF above you can see how wutch builds its own documentation. It behaves just like a live server. Adding any change causes a subprocess shell to run Sphinx (doc site generator used by wutch) to rebuild its docs in real time.

Why wutch?

If you’ve ever worked with live servers popular among frontend developers you know how convenient they are. Problem is you should either rely on the Live Server extension such as this one or create a build pipeline using something like gulp.

We wanted to create a command line tool that is able to rebuild documentation or website on each change and automatically refresh browser’s webpage afterwards.

Features

Watches multiple directories and glob file patters for changes
Able to ignore multiple directories and glob file patters
Runs given shell command on each change
Starts a live server (with an option to bind to a given port and address)
(optionally) Opens a browser when Wutch starts pointing to the build directory
Automatically refreshes web page on each build
Configurable timeout between rebuilds
Configurable binging port and host (can be used as a development web server)

Hot to install wutch?

At the moment, wutch is distributed as a Python package for Python >= 3.8. In the future a Homebrew formula, as well as .deb and .rpm distributions will be added.

You can install wutch like this:

pip install wutch

Or, to install for the current user only:

pip install --user wutch

How to use wutch?

After installation wutch binary becomes available. Now you can start a wutch watcher process. By default, it will do the following

watch current directory ./ for changes
start a live server
open a web browser pointing to the ./_build/index.html
run sphinx-build shell command on each change
auto-refresh web page on each build

$ wutch

How to configure wutch to perform different actions? Wutch supports multiple sources of configuration settings. They are loaded in a priority given below. Let’s take a command parameter that specifies what shell command to run as an example here.

command line options

$ wutch --command "bundle exec jekyll build"

environment variables starting with WUTCH_

export WUTCH_COMMAND="bundle exec jekyll build"

configuration file wutch.cfg present in the current directory

{ "command": "bundle exec jekyll build" }

default variables hardcoded into wutch itself

sphinx-build

Below is the full list of supported parameters:

-h, --help            show this help message and exit
-c COMMAND, --command COMMAND
                      Shell command executed in response to file changes. Defaults to: sphinx-build.
-p [PATTERNS ...], --patterns [PATTERNS ...]
                      Matches paths with these patterns (separated by ' '). Defaults to: ['*'].
-P [IGNORE_PATTERNS ...], --ignore-patterns [IGNORE_PATTERNS ...]
                      Ignores file changes in these patterns (separated by ' '). Defaults to: [].
-d [DIRS ...], --dirs [DIRS ...]
                      Directories to watch (separated by ' '). Defaults to: ['.'].
-D [IGNORE_DIRS ...], --ignore-dirs [IGNORE_DIRS ...]
                      Ignore file changes in these directories (separated by ' '). Defaults to: ['_build', 'build'].
-w WAIT, --wait WAIT  Wait N seconds after the command is finished before refreshing the web page. Defaults to: 3.
-b BUILD, --build BUILD
                      Build directory containing files to render in the browser. Defaults to: _build.
-I [INJECT_PATTERNS ...], --inject-patterns [INJECT_PATTERNS ...]
                      Patterns of files to inject with JS code that refreshes them on rebuild (separated by ' '). Defaults to: ['*.htm*'].
-i INDEX, --index INDEX
                      File that will be opened in the browser with the start of the watcher. Defaults to: index.html.
--host HOST           Host to bind internal HTTP server to. Defaults to: localhost.
--port PORT           TCP port to bind internal HTTP server to. Defaults to: 5010.
-B NO_BROWSER, --no-browser NO_BROWSER
                      Do not open browser at wutch launch. Defaults to: False.
-S NO_SERVER, --no-server NO_SERVER
                      Do not start the webserver, just launch the shell command. Defaults to: False.

Building Sphinx documentation with Wutch

Wutch relies on itself when building and rendering its own documentation written with Sphinx (see wutch docs sources).

In order to set up a Sphinx doc development using wutch’s live server it’s enough to place a wutch.cfg file into the project folder and then simply run wutch binary.

{
    "dirs": ["docs"],
    "patterns": ["*.rst", "*.py"],
    "command": "make -C docs build",
    "build": "docs/_build/html",
}

With these settings wutch will look for the changes happening in *.rst and *.py files in the ./docs directory. For each change, wutch will run make -C docs build command from its Makefile. It will then open the ./docs/_build/html/index.html file in the default browser and will auto-refresh the browser page every rebuild.

Building Jekyll with Wutch

Jekyll already comes with the jekyll serve command that creates a server which rebuilds the website for every change. However, jekyll is not able to refresh the page in the browser automatically.

Imagine you have a pretty standard Jekyll project structure. Below is a structure of this blog.

drwxr-xr-x  _drafts
drwxr-xr-x  _includes
drwxr-xr-x  _layouts
drwxr-xr-x  _posts
drwxr-xr-x  _site
drwxr-xr-x  assets
-rwxr-xr-x  index.md
drwxr-xr-x  pages
-rw-r--r--  robots.txt
-rwxr-xr-x  404.html
-rwxr-xr-x  Gemfile
-rw-r--r--  Gemfile.lock
-rwxr-xr-x  _config.yml

Here is how to build and run a live server for it with Wutch. For this example we will be using command line options instead of the config file.

$ wutch --ignore-dirs _site --build _site --command "bundle exec jekyll build"

Using Wutch as live server for frontend builds

Even though there are plenty of options in frontend to implement a live server that watches for changes and rebuilds CSS and HTML, still, wutch is capable of that too.

Imagine you have a Gulp based build pipeline for your website and you can run a build using gulp build command. Then all results of the build are dropped into the build directory of the project.

In that case we’d need to look for the changes in the current directory, but ignore the changes in the build folder.

$ wutch --ignore-dirs "build" --build "build" --command "gulp build"

More about Wutch

Designing Twitter

2018-06-05T20:00:00+00:00

This article describes several different approaches to designing a system like Twitter. The kind of task you could be asked to perform during a technical interview. Each presented architecture describes the system from a very high level, focusing on the main idea of the optimization or design rather than small details.

Features
Architectural approaches

Features

First of all, let’s specify a list of features that will be present in the system. Typically, a system like Twitter consists of user accounts that can follow each other and something called feed/dashboard, or timeline, a place where all posts made by user’s follow-pull are gathered and rendered in a reversed time manner.

Tweet

Each user is able to tweet or post a simple textual message. For the sake of simplicity we don’t consider retweets, likes, or comments.
Follow

Any user can follow any other user. When user follows someone else a subscription is created. Whenever followed user is posting something, the follower will see these tweets in his feed.
Feed

Collection of tweets made by followed users organized in a time-reversed fashion. An important addition to mention is that a system like twitter prefers eventual consistence over immediate consistency. It is better to get fast read rather than immediate write.

Architectural approaches

We will review several approaches, starting with the simplest one, like storing everything in a relational database. But eventually the system is going to target millions of users making hundreds of thousands tweets a day.

Take-1: Relational database

Description

When taking the most direct approach, the obvious thing is to have a set of relational tables: users, tweets, and followers. Users table is nothing but a storage of active/registered users. The Followers table stores a row for each existing subscription of one user to another. This will be required when calculating the feed for each user. Finally, Tweets table contains all tweets with a foreign key specifying the author of each tweet.

Upsides 👍

The obvious strong side of this approach is its indisputable simplicity. Just one database to manage, several tables, everything is stored in one place. As less overhead as possible. To be fair, many start-ups rely on this exact approach when building their product at the very beginning. Tumblr is a good example of that.

Downsides 👎

Well, while being the easiest to implement, this system is not very scalable. Of course, it can handle a hundred users. If everyone would subscribe to everyone the followers table would grow to be 10,000 rows, which is fine unless the user base starts to really grow. You could imagine that the select statement to calculate the timeline of each user would eventually take forever to complete or would hit a memory boundary.

Technologies

Database: MySQL/PostgreSQL
Backend: Python (Django)/Ruby on Rails/PHP

Take-2: Sharded relational database

Description

A different approach could be evolved out of an original single database design. It is possible to significantly improve scalability of the database in this particular case by shadring actual tweet storage and indexes. Since the feed is based on chronologically ordered collection of tweets, the sharding might be done on a time basis. Separating database indexes from the actual storage is an attempt to speed up query execution. There would still be one master in each of the sections. However, by introducing slave replicas and shards we can significantly improve response times.

Upsides 👍

Indexes are now stored in separate shards. The actual calculation of the timeline will be performed only by scanning index shards.

Downsides 👎

A design like this would eventually hit the read operation ceiling when talking to index shards.

Technologies

Database: MySQL (no PostgreSQL due to the lack of decent logical replication)
Backend: Same

Take-3: Pre-calculated feeds

Description

If we bring the idea of trading off write speeds for read speed to its absolute, we should eventually take a look at the approach that pre-calculates the feeds (also called materialized timelines). The immediate consistency of the write operation is traded off for a faster feed read operation for each user. In order to implement this, each time someone posts a new tweet the timeline of each user must be updated. It seems obvious that some caching solution is in order here. Memcached might be an option, but it stores the data in a blob format. Binary blob format will require us to pull out the whole timeline of a particular user, append a new tweet to it, and then put it pack to the cache.

Contrary, Redis has a notion of list data structure, which allows us to append tweets without extracting whole timeline from the cache. With this approach, each time some user requests a feed, a pre-calculated ready-to-read feed will be obtained from the Redis.

To make things even better, it would be nice to introduce a load balancer which has been deliberately ignored in the previous considerations. A load balancer would choose a proper Redis instance to perform a read or write operation.

The backend application would request a list of followers whenever some user posts a new tweet. Based on this list the backed will add new tweet to all of the required lists in Redis.

Upsides 👍

The time complexity of read operation is O(1). Write complexity, however, is O(n). We must append a new tweet to every timeline that belongs to a follower.

Downsides 👎

The amount of stored data might become quite great in size. Redis will require a replication in order to keep serving requests even when one instance with pre-calculated feed dies. Since Redis is an in-memory database, each instance would also need a great amount of RAM to operate properly, which is an obvious price we pay for an instant access to user’s timeline.

Another dangerous case is presented by the users with millions of followers. Any update they post to their followers must be written to millions of Redis lists for pre-calculated timelines. When taking replicas into account, this could take an unpredictable amount of time.

A possible solution to that is to keep the tweets of popular users somewhere separate and avoid the whole on-write timeline update story completely. Instead, whenever any of their followers requests a tweet feed, the backend app will manually merge the tweets of the popular user inside the timeline. Doing this in runtime during read operation is obviously not a O(1) complexity. However, this prevents us from generating a write avalanche each time they post something.

Technologies

Database: Redis (cached feed), MySQL (followers table)
Backend: Same

Make Vim detect Pipenv based Python environment

2018-04-12T06:00:00+00:00

Vim can be a great IDE. If I could you use just one word to describe it, that word would be “fast”. Vim can be easily configured to be a powerful IDE for Python development. However, as times change, as does the official recommended packaging tool for Python – it was Pip before, now it’s Pipenv, a high level wrapper around pip and virtualenv.

Setting up a proper integration between Vim and Pipenv may seem like a cumbersome task. It entirely depends on what you want to achieve. This article focuses on integration between Vim, Pipenv, and YouCompleteMe – a fast, fuzzy-search code completion engine for Vim. But some of the options described in the article are suitable for other setups as well.

Vim as a Python IDE

I could talk about Vim all day, if I had a chance. Vim requires less resources to run than most of the modern IDEs. It’s faster in many ways, even when it comes to syntax verification, file reading, and handling of large repositories. Vim has a tremendously large amount of commands, plugins, and shortcuts which are all configurable and allow you to navigate and edit in a blink of an eye. And last but not least, you can usually find an instance of Vim (or at least vi) on any Unix based OS, which proves to be especially useful for DevOps engineers. Same powerful editor on a server and on your workstation.

Here is a small list of plugins you could install to turn Vim into a Python IDE:

" Smart auto-indentation for Python
Plugin 'vim-scripts/indentpython.vim'
" Auto-completing engine
Plugin 'Valloric/YouCompleteMe'
" Syntax checker
Plugin 'vim-syntastic/syntastic'
" Python backend for 'syntastic'
Plugin 'nvie/vim-flake8'
" Status bar (powerline)
Plugin 'vim-airline/vim-airline'
" Awesome staring screen for Vim
Plugin 'mhinz/vim-startify'
" File manager
Plugin 'scrooloose/nerdtree'
" Search bar
Plugin 'kien/ctrlp.vim'
" Theme
Plugin 'crusoexia/vim-monokai'
" Powerful commenting utility
Plugin 'scrooloose/nerdcommenter'
" Rich python syntax highlighting
Plugin 'kh3phr3n/python-syntax'

Obviously, there is so much more to Vim than just a list of plugins. Vim’s configuration file – .vimrc – can become quite large, just because there are so many things you can configure in Vim. However, complete configuration of Vim for Python development is a topic of a different article.

VirtualEnv Support

One of the issues you can discover is that by default Vim has no notion of a virtual environment in which you are supposed to run. In order to perform any auto-completion or syntax checking tasks Vim utilizes the default python interpreter found in the PATH environment variable.

The classic way to make Vim aware of project’s virtual environment is to check the list of environment variables to detect whether we are running inside a virtual environment and activate it. This approach is documented on the realpython.com:

"python with virtualenv support
py << EOF
import os
import sys
if 'VIRTUAL_ENV' in os.environ:
    project_base_dir = os.environ['VIRTUAL_ENV']
    activate_this = os.path.join(project_base_dir, 'bin/activate_this.py')
    execfile(activate_this, dict(__file__=activate_this))
EOF

Unfortunately, this no longer works if we deal with Pipenv based virtual environments. The environment itself might not even be in the same directory, or not even under default path ~/.local/share/virtualenvs/.

However, we can take advantage of Pipenv’s own commands to determine if we are supposed to be running in a virtual environment.

pipenv --venv

When we have a PyEnv installed it will detect our call to pipenv. Let’s take a look at possible situations:

Directory is not configured for PyEnv or Pipenv

In this case both python interpreter and Pipenv are not configured for the current directory. PyEnv reports that it could, theoretically, run pipenv, which is available in the 3.6.5 installation, but would need you to indicate that explicitly.

# Remove PyEnv configuration file for experiment
$ rm .python-version
$ pipenv --venv
pyenv: pipenv: command not found

The `pipenv` command exists in these Python versions:
3.6.5

The same behavior will be observed if pipenv is not installed as a package in the interpreter of choice. Let’s say, we specify a python version 3.4.3 in a .python-verison file, which has no pipenv installed. Same error will be thrown out by PyEnv in that case.

Directory is configured for PyEnv, but no virtual environment is created yet

Here, .python-version file exists for PyEnv to detect. And the python interpreter that we specified in the file has Pipenv installed as a package. In that case Pipenv will indicate that no virtual environment has been created yet.

# Create PyEnv configuration file
$ echo "3.6.5" > .python-version
$ pipenv --venv
No virtualenv has been created for this project yet!

Directory is configured for PyEnv and virtual environment exists

In this case pyenv --venv returns full path to the virtual environment utilized in this project.

$ pipenv --venv
/Users/user/.local/share/virtualenvs/.aws-MRQhgJfv

Making it work in `.vimrc`

We can take advantage of the fact that whenever we call pipenv --venv its return code will indicate successful (sys.exit(0)) or unsuccessful (sys.exit(1)) run.

There is a variable named shell_error in Vim, and, as you probably figured out, it contains the last exit code of the executed shell command.

If we make a call to pipenv --venv from .vimrc, we will either get a correct path, or one of the error messages we observed earlier. In order to make a call we can use the system() function of Vim:

let pipenv_venv_path = system('pipenv --venv')

Then, simply by checking the value of shell_error variable, we can find out if the call was successful.

if shell_error = 0
  " do one thing
else
  " do another thing
endif

Since in this article we set up Python’s auto-completion for YouCompleteMe engine, we will use YCM’s global variable g:ycm_python_binary_path to point it to the correct executable of proper virtual environment.

let venv_path = substitute(pipenv_venv_path, '\n', '', '')
let g:ycm_python_binary_path = venv_path . '/bin/python'

The final script that we’ll add to the .vimrc will lok like this:

" Point YCM to the Pipenv created virtualenv, if possible
" At first, get the output of 'pipenv --venv' command.
let pipenv_venv_path = system('pipenv --venv')
" The above system() call produces a non zero exit code whenever
" a proper virtual environment has not been found.
" So, second, we only point YCM to the virtual environment when
" the call to 'pipenv --venv' was successful.
" Remember, that 'pipenv --venv' only points to the root directory
" of the virtual environment, so we have to append a full path to
" the python executable.
if shell_error == 0
  let venv_path = substitute(pipenv_venv_path, '\n', '', '')
  let g:ycm_python_binary_path = venv_path . '/bin/python'
else
  let g:ycm_python_binary_path = 'python'
endif

This script can be further modified to support different completion engines or even other types of plugins, but the essence remains the same – determine whether Vim is running in the correctly configured Pipenv directory.

Perfect AWS CLI setup

2018-04-07T08:00:00+00:00

I deploy to AWS a lot. I believe it’s fine to use AWS GUI when you explore things, but otherwise it is better to write scripts to achieve results. Be it Bash scripts that use AWS CLI or Python scripts that use boto3 library. Writing scripts guarantees that when you forget how to properly deploy a cluster of ElasticSearch instances and shards you will just use your script instead of researching AWS documentation again. AWS CLI is a Python library installed via pip.

I try to keep the installation of AWS CLI isolated from everything else. Making it possible to have multiple installations with different versions. Here is how I achieve that on MacOS.

~/.aws/ directory. This directory will be automatically created when you configure your AWS CLI with aws configure command. However, I prefer to create it ahead of time, because the installation will be kept here.
~/.aws/.python-version file that contains Python version that will be used for AWS CLI. As of now, this file contains 3.6.5. This, of course, assumes that you use Pyenv to manage Python installations. Also the version that you chose must have Pipenv installed in it.
awscli library. Once you are in ~/.aws/ directory just run
```
pipenv install awscli
```
This will create a dedicated virtual environment and install AWS CLI there so that it’s exclusively available only from that place. Pipenv will know which virtual environment to use in this directory thanks to the Pipfile.lock file.
We have an isolated and clean installation of AWS CLI. Now, how can we make it available across the system? At the moment the only way to start AWS CLI is to fire up Pipenv from the directory with Pipfile.
```
pipenv run aws
```
Instead, let’s make a symlink that will redirect any requests to aws command into our dedicated virtual environment. It’s pretty easy.
- Create ~/.aws/bin/ directory.
- Create ~/.aws/bin/aws file there with the following script inside:

    #!/usr/bin/env bash
    # The line above ensures cross compatibility in MacOS

    # Set ENV variable for PyEnv to know which interpreter to use
    # This will not work if you no longer have 3.6.5 version in Pyenv!
    export PYENV_VERSION=3.6.5

    # Set PIPENV location ENV variable to tell Pipenv where to look for virtual environment.
    export PIPENV_PIPFILE=~/.aws/Pipfile

    # When this command is executed following happens:
    # 1. Pyenv starts and uses shims to look for the pipenv module in the
    #    3.6.5 installation of Python. Then it starts pipenv
    # 2. Pipenv reads Pipfile location from environment variable of the
    #    shell that we just set and finds the aws executable
    #    in the dedicated virtual evnironment.
    # 3. We pass all the arguments in our script to aws executable using
    #    "$@" bash directive.
    pipenv run aws "$@"

Make the script executable
```
chmod a+x ~/.aws/bin/aws
```
Create a symlink to this executable script

    sudo ln -s ~/.aws/bin/aws /usr/local/bin

Voilà, the aws executable is now available from any directory.

  $ pwd
  /Users/user/

  $ aws --version
  aws-cli/1.15.0 Python/3.6.5 Darwin/17.4.0 botocore/1.10.0

Choosing between Ruby environment managers: rbenv vs. RVM

2018-02-05T00:29:01+00:00

Using a system’s default Ruby interpreter to develop projects and install required gemsets is a sure path to dependency nightmare. Each project needs its own environment independent from others. At the same time, the system’s installation of Ruby should be left intact.

This article gives a quick overview of the popular Ruby environment managers and provides a concise installation instruction for a painless setup on MacOS X.

To quickly jump to the environment manager installation click here.

Overview

There are two main options for environment management when it comes to Ruby. First one is the RVM. RVM was the first decent environment manager which could manage multiple interpreters and per-project based gemsets, it is still very popular among Ruby developers. Although, not only the installation process of it is somewhat unusual for these days, but is also messes up with a lot of things in the system in a magic and incomprehensible way. For instance, RVM ensures that a correct version of Ruby will be utilized in a project by replacing original cd command in *nix systems, which is way too much for nothing else but Ruby environment manager.

The other known option is called rbenv. It’s a much more simpler and reliable solution, and it does not mess the system up, keeping everything local and manageable.

RVM and its downsides

So, what’s wrong with the RVM? The installation of RVM is quite bizarre for anyone without a background in Linux systems administration, and project’s homepage leaves even more for a reader to guess regarding RVM’s internal logic and implementation. RVM gives an impression of the tool that mingles with the system in an irreversible way. Obviously, it’s a tested, trusted, and community verified tool that does not perform any malicious actions with your system, but the first impression it gives doesn’t look like something you would expect from a user friendly environment manager.

The installation requires you to add a pair of public GPG keys of project’s maintainers to your GPG keychain and then download and run a 1000 lines long bash script that will do the installation for you.

First, adding GPG keys:

# Add public GPG key of the RVM's authors to your GPG keychain
gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB

# Result:
gpg: key 105BD0E739499BDB: public key "Piotr Kuczynski <piotr.kuczynski@gmail.com>" imported
gpg: key 3804BB82D39DC0E3: 82 signatures not checked due to missing keys
gpg: key 3804BB82D39DC0E3: public key "Michal Papis (RVM signing) <mpapis@gmail.com>" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 2
gpg:               imported: 2

Next thing that RMV’s website will suggest is to run the following command to continue installation:

\curl -sSL https://get.rvm.io | bash -s stable

This line downloads and runs RVM’s bash installation script.

Before running any unknown script I usually stop and suspend the installation process. You never know what that script is going to do to your system unless your read through it. RVM skips the “read” part right away. I’d call such installation process a bad practice. How difficult will it be to remove RVM from the system and clean up the traces? Was it to hard for creators to enable a Homebrew based installation? You can’t tell until you spend a considerable amount of time searching for the answers. And, again, doing a deep research of what’s supposed to be a simple Ruby environment manager is not something you would expect from a user friendly tool.

Interestingly enough, if prior to going all in and running an unknown script you think ahead and decide to check the official website for the uninstalling instructions, you won’t find them. There is a single mention of implode command on the RVM’s CLI usage page. But according to that page:

implode   - removes all ruby installations it manages, everything in ~/.rvm

implode does not uninstall the RVM itself. Basically, there is no way to automatically uninstall RVM other than manually cleaning up everything that the installation did to your system. And that’s a huge downside.

The only way to uninstall RVM is to manually clean up everything it did to your system.

If you proceed with the installation and run the second step from RVM’s homepage, the bash script will be downloaded and executed.

# Download RVM's installation script and install it
\curl -sSL https://get.rvm.io | bash -s stable

# Result:
Downloading https://github.com/rvm/rvm/archive/1.29.3.tar.gz
Downloading https://github.com/rvm/rvm/releases/download/1.29.3/1.29.3.tar.gz.asc
gpg: Signature made Sun Sep 10 22:59:21 2017 CEST
# ...
Installing RVM to /Users/user/.rvm/
    RVM PATH line found in /Users/user/.mkshrc.
    RVM PATH line not found for Bash or Zsh, rerun this command with '--auto-dotfiles' flag to fix it.
    Adding rvm loading line to /Users/user/.profile /Users/user/.bash_profile /Users/user/.zlogin.
Installation of RVM in /Users/user/.rvm/ is almost complete:
# ...
Searching for binary rubies, this might take some time.
No binary rubies available for: osx/10.13/x86_64/ruby-2.4.1.
# ...
Installing Ruby from source to: /Users/user/.rvm/rubies/ruby-2.4.1, this may take a while depending on your cpu(s)...
# ...
  * To start using RVM you need to run `source /Users/user/.rvm/scripts/rvm`
    in all your open shell windows, in rare cases you need to reopen all shell windows.

If you are as impatient as me, you would most likely hope that at least the usage of RVM is clear and simple after all that unknown compilation-installation-mess. I would argue that it’s not. Let’s think about a first time Ruby user who managed to install RVM. After spending around 10 minutes reading, trying to understand, and installing RVM the user will search the homepage for the link that describes some usage examples or the basics:

Did you find a page that explains the usage? Or maybe, CLI usage would be a better page to look at for the first time user? You would certainly hope that at least the basics are short and concise, because an environment manager is not the final goal of the whole process. The final goal is to develop using Ruby in a clean and isolated fashion.

The usage instruction of RVM is muddy and unclear. The understanding of correct way to use the tool might require even more time than its installation. So, if you have installed RVM by mistake, here is the way to clean everything up and uninstall RVM completely.

Cleaning up after RVM

RVM can be uninstalled in a several steps. Here are two topics on StackOverflow covering the uninstallation of RVM: How can I remove RVM (Ruby Version Manager) from my system? and Howto Uninstall RVM [duplicate].

0. Delete RVM installed rubies

If you have already installed and integrated RVM into your shell, you will need to start with the rvm implode command.

# Check that RVM exists in your path
echo $PATH | grep rvm

# If grep does not return anything, then RVM is not integrated into your shell
# yet. Otherwise, run "rvm implode" to uninstall ruby interpreters
# installed by RVM
rvm implode

1. Removing the ~/.rvm directory

Some size stats before removing the directory

# Check disk usage of the ~/.rvm directory
# -s displays an entry for each specified file or directory
# -h prints the result in a "Human-readable" format.
#    e.g., Byte, Kilobyte, Megabyte, Gigabyte, Terabyte and Petabyte.
du -sh ~/.rvm
# Result:
181M	/Users/user/.rvm

The pure default installation of RVM on MacOS takes 181 Mb.

Remove RVM’s directory

rm -rf ~/.rvm

2. Cleaning shell configuration files

Check the following files for the references to RVM and remove the RVM line from each of them:

~/.bashrc
~/.bash_profile
~/.profile
~/.zshrc
~/.zlogin

References look like this:

[[ -s "$HOME/.rvm/scripts/rvm" ]] && source "$HOME/.rvm/scripts/rvm" # Load RVM into a shell session *as a function*

We can do a quick grep to check if RVM put itself there.

cat ~/.bashrc | grep rvm
# Result:
[[ -s "$HOME/.rvm/scripts/rvm" ]] && source "$HOME/.rvm/scripts/rvm" # Load RVM into a shell session *as a function*

3. Removing unnecessary GPG keys

To remove public GPG keys of RVM maintainers we will need their user names. We can find them in the full list of public keys stored in GPG.

gpg --list-keys
# Result:
# ...
pub   rsa4096 2016-11-11 [SC]
      7D2BAF1CF37B13E2069D6956105BD0E739499BDB
uid           [ unknown] Piotr Kuczynski <piotr.kuczynski@gmail.com>
sub   rsa4096 2016-11-11 [E]

pub   rsa4096 2014-10-28 [SC]
      409B6B1796C275462A1703113804BB82D39DC0E3
uid           [ unknown] Michal Papis (RVM signing) <mpapis@gmail.com>
uid           [ unknown] Michal Papis <michal.papis@toptal.com>
uid           [ unknown] [jpeg image of size 5015]
sub   rsa2048 2015-11-02 [E]
sub   rsa4096 2014-10-28 [S] [expires: 2019-03-09]
# ...

You can use any representation of user name in any uid row listed in a key to delete it. I will use email address.

# Delete key
gpg --delete-key piotr.kuczynski@gmail.com
# Result:
gpg (GnuPG) 2.2.3; Copyright (C) 2017 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

pub  rsa4096/105BD0E739499BDB 2016-11-11 Piotr Kuczynski <piotr.kuczynski@gmail.com>

Delete this key from the keyring? (y/N) y
# Piotr's key is removed now

# Delete Michal's key
gpg --delete-key mpapis@gmail.com
gpg (GnuPG) 2.2.3; Copyright (C) 2017 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

pub  rsa4096/3804BB82D39DC0E3 2014-10-28 Michal Papis (RVM signing) <mpapis@gmail.com>

Delete this key from the keyring? (y/N) y
# Both public keys are now removed

rbenv

rbenv is short for Ruby Environment. It’s a different command line tool that enables you to quickly and easily switch between different rubies installed on your system.

What I like about rbenv is it’s simplicity and obviousness. It might look especially familiar for those coming from Python background, because rbenv, in fact, looks a lot like pyenv — a python version manager forked from rbenv and modified for Python.

rbenv is easy in development. All you need is to create a .ruby-version file in a directory and from that moment the version specified in this file will be used in this particular directory.

It bundles greatly with Bundler. In fact, bundler takes care of gem management completely. Each installation of Ruby gets its own installation of bundler. Installation can be configured so that it reuses gems for projects that utilize same Ruby version. Otherwise, gemsets are installed on per-project basis.

rbenv works by introducing a directory full of small executables called shims into your path. The path to this directory looks like this: /Users/user/.rbenv/shims. Each shim is a tiny Bash script that has the exact same name as any Ruby interpreter based tool in your system.

For example, before calling the actual Jekyll executable, a shim will check the .ruby-version files and RBENV_VERSION environmental variable as well as global ~/.rbenv/version file if other options did not work. Shim will then pull out a correct version of Jekyll according to your specified Ruby version. To be able to do that, the directory with shims is placed first in the $PATH. This way any call to Jekyll in our case will be at first directed to a shim.

Let’s take a look at how a user would use rbenv.

Checking what Ruby versions are available

rbenv install --list

# Alternatively: rbenv install -l

# Result:
Available versions:
  1.8.5-p52
  1.8.5-p113
  1.8.5-p114
  1.8.5-p115
  1.8.5-p231
  1.8.6
  1.8.6-p36
# ...
  2.5.0-rc1
  2.5.0
  2.6.0-dev
  2.6.0-preview1
# ...
  jruby-9.1.15.0
  jruby-9.1.16.0
  jruby-9.2.0.0-dev
# ...
  ree-1.8.7-2012.01
  ree-1.8.7-2012.02
  topaz-dev

The whole list is much longer. You can see that beside the versions of the standard interpreter there is a whole bunch of other implementations.

Installing Ruby versions

Any version is installed into the ~/.rbenv directory. You can later specify which installed version you want to use in your project.

rbenv install 2.5.0

Install bundler

gem install bundler

Install gems

Now, if you have a Gemfile ready, you can perform an installation of all required gems.

bundle install

Choose Ruby version for project

The order in which Ruby version is checked by rbenv is perfectly described in the docs. It is hard to rephrase it better:

When you execute a shim, rbenv determines which Ruby version to use by reading it from the following sources, in this order:

The RBENV_VERSION environment variable, if specified. You can use the rbenv shell command to set this environment variable in your current shell session.

The first .ruby-version file found by searching the directory of the script you are executing and each of its parent directories until reaching the root of your filesystem.

The first .ruby-version file found by searching the current working directory and each of its parent directories until reaching the root of your filesystem. You can modify the .ruby-version file in the current working directory with the rbenv local command.

The global ~/.rbenv/version file. You can modify this file using the rbenv global command. If the global version file is not present, rbenv assumes you want to use the “system” Ruby—i.e. whatever version would be run if rbenv weren’t in your path.

Rbenv installation on MacOS X

A proper installation of ruby environment manager on MacOS X requires only few steps. Most importantly, rbenv has to be installed through Homebrew, and rbenv init command has to be called.

Step 1

Install rbenv

brew install rbenv

Note that this also installs ruby-build, so you’ll be ready to install other Ruby versions out of the box.

Step 2

Initialize rbenv on your OS.

rbenv init

That’s it!

Don’t forget to call rbenv rehash after installing new gems to give rbenv a change to generate new shims.

Writing an IntelliJ IDEA plugin to manage window layouts

2017-10-20T11:29:01+00:00

There are many IntelliJ plugins out there. Still, existing SDK documentation does not cover every development step. Here I describe how a basic IntelliJ plugin with menu and import/export functionality can be developed.

Plugin’s purpose

Sometimes, after working on some project for a long enough period, you end up with a particular window layout that is convenient for you, e.g., console on the right side, source code on the left, debug window at the top.

However, if you open the same project on a different machine, the layout you’ve so delicately configured will be lost. Same situation will happen if you want to reuse a layout from some other project.

Preserve Layout Plugin allows you to export the layout of any IntelliJ project and then import it back. Export is done using the XML format.

About this blog post

I wrote this post to address the issues I encountered during development of such plugin. As of current date (Oct 20, 2017), IntelliJ SDK documentation is way far from being complete. Below is my collection of findings regarding missing pieces.

Initial Setup

For initial setup, if you never developed a JetBrains’ plugin, I highly recommend to go through the Getting Started section of IntelliJ Platform SDK.

You will notice that some sections are missing and greyed out. However, just by following these initial setup steps you will get just enough to create your own plugin. It is important to mention that in order to debug a plugin you will need the source code of IntelliJ Community Edition checked out and attached to the project.

Prototyping

If you are familiar with plugin development for IntelliJ, it is safe to skip this section.

To play around a little bit, let’s create a very simple plugin with one action. Just to test the things and confirm that setup is working.

First thing you might look in to is the plugin.xml file that contains the basic configuration of your plugin.

After filling in the standard information fields like <vendor> or <version> we can jump straight to the <actions> section.
Here we can statically position and initialize our actions.

plugin.xml is sufficient enough for most needs. However, when doing complex stuff, we can initialize actions dynamically in Java.

Let’s position our action in the Project View popup menu, the one that appears when you right-click the project name in IDEA.

Information tags:

<id>com.duseev.intellij.preservelayout</id>
<name>Preserve Layout Plugin</name>
<version>1.0</version>
<vendor email="vagiz@duseev.com" url="http://duseev.com">Duseev.com</vendor>

Decribing actions:

<actions>
  <action id="ExportLayout"
          class="com.duseev.intellij.preservelayout.LayoutExporter"
          text="LayoutExporter"
          description="LayoutExporter">
  <add-to-group group-id="ProjectViewPopupMenu"
                anchor="after"
                relative-to-action="ExternalToolsGroup"/>
  </action>
</actions>

You can see that we created a unique id for this action. A class for the should also be specified. A name and description will be LayoutExporter. The <add-to-group ...> tag is the one responsible for tying our action to the specific menu in IDEA.

Obviously, we will need a class responsible for this action. Let’s make it super simple: it will print project’s name to the console. You can notice that we inherit from AnAction class and override the actionPerformed method.

package com.duseev.intellij.preservelayout;

import com.intellij.openapi.actionSystem.AnAction;
import com.intellij.openapi.actionSystem.AnActionEvent;
import com.intellij.openapi.project.*;

public class LayoutExporter extends AnAction {

   @Override
   public void actionPerformed(AnActionEvent e) {
       // TODO: insert action logic here
       Project project = e.getProject();
       System.out.println(project.getName());
   }
}

After we run this code, IDEA will spawn a new instance of itself with our plugin installed by default. After we right click on the project (you will need a project to be created or open) and invoke context menu, we will see that LayoutExporter action appeared in this menu. Clicking this action will print a project name to the console output.

Of course, we can assign actions to a different menu. For example, in preserve-layout-plugin actions are added to the WindowMenu.

Window layout in IDEA

Stepping into the actual logic, we might ask ourselves where is the layout of current project stored in IntelliJ IDEA? Turns out it can be stored in two ways:

directory based project: in workspace.xml file under .idea directory
file based project: in .iws file in a project directory

JetBrains is planning to deprecate the file based format. But we still have to support both options.

Getting current window layout: To obtain current layout state we obtain an instance of the ToolWindowManagerImpl:

ToolWindowManagerImpl mgr = (ToolWindowManagerImpl) ToolWindowManager.getInstance(e.getProject());

Having the manager, we can now ask it for a current layout. An instance of DesktopLayout will be returned. Then we cas use DesktopLayout to return org.jdom.Element representation of itself.

DesktopLayout dl = mgr.getLayout();
Element layout = dl.writeExternal("layout");

Next step is to save this XML/DOM Element somewhere.

File Saver dialog & Saving/writing a file

File Saving dialog: Somehow, file saving dialog in IntelliJ is invoked in a different way, comparing to File Chooser. It is also not mentioned yet in documentation. However, you can come across a FileSaverDescriptor class, which when instantiated and passed to FileChooserFactory as a parameter will produce a correct dialog window.

Writing to a File in IntelliJ: Actual data writing is simple, but not obvious. It is required to execute “write” operations outside of the main thread. To address this, IntelliJ provides a very convenient WriteCommandAction abstraction.

Besides that, best approach to write a file to a disk is through the VirtualFile abstraction layer.

Let’s take a look at a composed example of logic required to save an XML to a file in IntelliJ.

// Use XML to covert Element to String
XMLOutputter outputter = new XMLOutputter(Format.getPrettyFormat());
String exportContent = outputter.outputString(doc);

// Describe file saving dialog
FileSaverDescriptor descriptor = new FileSaverDescriptor(
        "Save Layout",
        "Choose path for layout export...",
        "xml");

// Obtain a file wrapper from FileChooserFactory
VirtualFileWrapper wrapper = FileChooserFactory
        .getInstance()
        .createSaveFileDialog(descriptor, e.getProject())
        .save(e.getProject().getBaseDir(), "layout.xml");

if (wrapper == null) return;
VirtualFile file = wrapper.getVirtualFile(true);

if (file == null) throw new Exception("Couldn't create new file");

// Write to disk outside of main GUI thread
new WriteCommandAction.Simple(e.getProject()) {
    @Override
    protected void run() throws Throwable {
        VfsUtil.saveText(file, exportContent);
    }
}.execute();

File Chooser dialog & importing layout

The best way to select a file in IntelliJ is to use the FileChooser dialog and the overloaded function chooseFile that accepts a callback parameter to be called after the file is selected. When using this approach IntelliJ is able to invoke native file chooser dialog window on every platform.

FileChooser.chooseFile(
    FileChooserDescriptorFactory.createSingleFileDescriptor(),
    e.getProject(),
    null,
    file -> importLayoutFileToProject(file.getCanonicalPath(), e.getProject())
);

Now we need a callback function importLayoutFileToProject().

private void importLayoutFileToProject(String path, Project project) {
    if (path == null) return;
    if (project == null) return;

    try {
        Element layout = parseLayoutFile(path);
        applyLayoutToProject(layout, project);
    } catch (Exception ex) {
        // notify user
    }
}

As you can notice, importLayoutFileToProject makes 2 calls to the helping methods. One to parse the imported XML file, and the other to apply the parsed layout to IDE. Let’s implement them as well.

private Element parseLayoutFile(String path) throws Exception {
    Document doc = new SAXBuilder().build(new File(path));
    return doc.getRootElement();
}

private void applyLayoutToProject(Element layout, Project project) {
    ToolWindowManagerImpl mgr = (ToolWindowManagerImpl) ToolWindowManager.getInstance(project);
    DesktopLayout dl = mgr.getLayout();
    dl.readExternal(layout);
}

Having all that we can now import and export window layout’s of any project. However, it would be nice to notify a user about results of the export/import process.

IntelliJ Notifications

Basic notifications are very easy to fire in IntelliJ:

Notifications.Bus.notify(new Notification(
    "Preserve Layout",
    "Successful Export",
    "Saved to " + file.getCanonicalPath(),
    NotificationType.INFORMATION
), e.getProject());

Here we can change the icon to emphasize the ERROR status.

Notifications.Bus.notify(new Notification(
    "Preserve Layout",
    "Export Failed",
    ex.getMessage(),
    NotificationType.ERROR
), e.getProject());

Make plugin runnable in every JetBrains IDE

Last thing but not the least important. To support every JetBrains IDE it is required to specify the <depends> tag in <plugin>. More details about plugin compatibility are available at SDK docs.

<depends>com.intellij.modules.lang</depends>

Special thanks

This plugin has been developed thanks to Alexander Zolotov from JetBrains team. He provided tons of invaluable tips and advices during the development of this plugin.

Vagiz Duseev

Getting started with Wutch

Why wutch?

Features

Hot to install wutch?

How to use wutch?

Building Sphinx documentation with Wutch

Building Jekyll with Wutch

Using Wutch as live server for frontend builds

More about Wutch

Designing Twitter

Table of contents

Features

Architectural approaches

Take-1: Relational database

Description

Upsides 👍

Downsides 👎

Technologies

Take-2: Sharded relational database

Description

Upsides 👍

Downsides 👎

Technologies

Take-3: Pre-calculated feeds

Description

Upsides 👍

Downsides 👎

Technologies

Make Vim detect Pipenv based Python environment

Vim as a Python IDE

VirtualEnv Support

Directory is not configured for PyEnv or Pipenv

Directory is configured for PyEnv, but no virtual environment is created yet

Directory is configured for PyEnv and virtual environment exists

Making it work in .vimrc

Perfect AWS CLI setup

Choosing between Ruby environment managers: rbenv vs. RVM

Overview

RVM and its downsides

Cleaning up after RVM

0. Delete RVM installed rubies

1. Removing the ~/.rvm directory

2. Cleaning shell configuration files

3. Removing unnecessary GPG keys

rbenv

Checking what Ruby versions are available

Installing Ruby versions

Install bundler

Install gems

Choose Ruby version for project

Rbenv installation on MacOS X

Step 1

Step 2

That’s it!

Writing an IntelliJ IDEA plugin to manage window layouts

Plugin’s purpose

About this blog post

Initial Setup

Prototyping

Window layout in IDEA

File Saver dialog & Saving/writing a file

File Chooser dialog & importing layout

IntelliJ Notifications

Make plugin runnable in every JetBrains IDE

Special thanks

Making it work in `.vimrc`