mixedmath

It is no secret that I love vim. I've written and helped maintain a variety of vim plugins, but I've mostly restricted my attention to direct use of vimscript or to call external tools and redirect their output to a vim buffer.

Only recently did I learn how easy it is to use python to interact directly with vim objects through the vim-python package. And it was so easy that this felt a bit like writing a quick shell script to facilitate some immediate task — it allows quick and dirty work.

In this note, I'll review how I recently wrote a quick python function (to be used in vim) to facilitate conversion of python f-strings (which I love) to python 3.5 (which I also love, but which is admittedly a bit behind).

Of course the real reason is to remind future me just how easy this is.

Description of the Problem

In python 3.7+, one can use f-strings to format strings. This syntax looks like

x = 4
print(f"Python has at least {x} many ways to format strings now.")

I love them, they're great. But they are not backportable since they are a change to fundamental syntax. Thus systems relying on previous versions of python (be it python2 or even python3.5) can't parse python files containing f-strings.

The goal is this: given a line containing an f-string, write an equivalent line containing no f-string.

The formula I choose is simple: use .format() instead. Thus given

s = f"string with {some} {variables}"

we will convert this to

s = "string with {} {}".format(some, variables)

I will assume that the string takes place on one line for simplicity (and because in my application this was almost always true), but that there are any number of variables included in the line.

Background on python in vim

Some good insight is gained through reading :h python-vim. In fact, this is essentially the only documentation on the python-vim interface (though it is pretty good documentation — perhaps the actual best source of learning this interface is to read some plugins which leverage the python connection).

I will assume that vim has been compiled with python3 support (which can be checked by looking for +python3 in vim –version). Then the key idea is that one can use :py3 to call python functions. For example,

:py3 print("Hello")

Being restricted to one line is inconvenient, and it is far more useful to use the form

:py3 << EOF
def say_hello():
  print("Sup doc.")
EOF
:py3 say_hello()

In this way, we can define and use python functions. Right now, the output of these functions simply appears in the status lines at the bottom of the screen. This is of limited value. Often, we will really want to interact directly with the vim buffer. The key to do this is the vim module for python.^{1 1}The vim module for python is included in the vim source code, but it's written in C. In fact, I do not understand the module code itself almost at all. Fortunately it's pretty easy to use even if I have to explore the intricacies through trial and error.

The vim module for python can be used through

import vim

# do vim things

This is what :h python-vim actually documents, and I defer to the list of methods there. The important thing is that it provides direct access to vim buffers.

Today, we will use only the direct access to the current line, vim.current.line.

A Solution

I very routinely open scratch buffers (i.e. buffers that I call scratch.tmp). Sometimes I momentary notes, or I edit vim macros in place, or I define vim functions for short-term use. I will assume that we have opened up a scratch buffer and our pythonfile withfstrings.py. (In fact, I usually open it in a vsplit). I also very routinely assign Q to do whatever convenient function I have written in my scratch buffer (or whatever macro I want to repeat most simply). Of course, these are my idiosyncratic habits — the ideas are easy enough to translate.

In our scratch buffer, we can write a vim function, which is internally a python function, and map the symbol Q to call it.

function! Convert()
py3 << EOF

import vim

import re
bracketed = re.compile("{.*?}")

def convert():
    line = vim.current.line
    single = line.find("'")
    double = line.find('"')
    if single == -1:
        sep = '"'
    elif double == -1 or single < double:
        sep = "'"
    else:
        sep = '"'

    # l sep inner sep post
    l, _, m = line.partition(sep)
    inner, _, post = m.partition(sep)

    #get rid of f
    l = l[:-1]

    ret = ""
    var_names = bracketed.findall(inner)
    var_names = list(map(lambda x: x[1:-1], var_names))

    ret += l + sep + inner + sep
    ret += ".format("
    for var in var_names:
        ret += "{}={}, ".format(var, var)
    ret = ret[:-2]
    ret += ")" + post
    vim.current.line = ret

convert()
EOF

endfunction

com! Conv call Convert()
nnoremap Q :Conv<CR>

To use, one sources the scratch buffer (either directly, like :source scratch.tmp, or through navigating to its split and using :so %). Then one can find a line containing an f-string and hit Q (or alternately use :Conv).

Why is this good?

Although nontrivial, writing a simple python script like this takes only a few minutes. And then converting f-strings became the least important part of my backporting project.^{2 2}Of course this isn't the only way. I suspect that a perl regex guru could write a single s:///g map to handle this all immediately — but I can't do that quickly. I can do very simply python regex quickly though.

And more broadly, this is such an easy process. Off-loading the mental burden of repeating annoying tasks is valuable. The most revelatory aspect of this experience for me is that it's easy enough to allow throwaway functions for momentary ease.

In my recent application, I converted 482 f-strings using this function. A few of them were multiline strings and my (admittedly simple) function failed — and so I did those by hand. But it took me less than 10 minutes to write the function (maybe even less than 5, though my first function expected double quotes only). Overall, this was a great gain.

Keeping the Script

Having come this far, it's worth considering how we might keep the script around if we wanted to.

One good approach is to separate the python from the vim (allowing, for instance, testing on the python itself) and have a thin vim wrapper. We'll look at this in two possible levels of encapsulation.

1. Separate the python from the vim.
2. Make the function a plugin. (Presumably for slightly more meaningful functions than this one).

Separate the Files

Let's separate the python from the vim. In this was we can approach the python as proper little python script, merely waiting for a thin vim wrapper around it at the end.

We can separate the python out into the following file, convert_fstring.py.

# convert_fstring.py
import re
bracketed = re.compile("{.*?}")

def convert(line):
    single = line.find("'")
    double = line.find('"')
    if single == -1:
        sep = '"'
    elif double == -1 or single < double:
        sep = "'"
    else:
        sep = '"'

    # l sep inner sep post
    l, _, m = line.partition(sep)
    inner, _, post = m.partition(sep)

    # get rid of f
    l = l[:-1]

    ret = ""
    var_names = bracketed.findall(inner)
    var_names = list(map(lambda x: x[1:-1], var_names))

    ret += l + sep + inner + sep
    ret += ".format("
    for var in var_names:
        ret += "{}={}, ".format(var, var)
    ret = ret[:-2]
    ret += ")" + post
    return ret

if __name__ == "__main__":
    test_input = 'mystr = f"This {is} an {f}-string".reverse()'
    expected  = 'mystr = "This {is} an {f}-string".format(
      is=is, f=f).reverse()'
    assert expected == convert(test_input)

This is a raw python file. I included a mock unit-test (in case one was testing the function while writing it... I didn't compose the function like this because I didn't think about it, but in the future I probably will).

Now let's write a thin vim wrapper around it.

" convert_string.vim

let s:script_dir = fnamemodify(resolve(expand('<sfile>', ':p')), ':h')

function! Convert()
py3 << EOF

import sys
import vim

script_dir = vim.eval('s:script_dir')
sys.path.insert(0, script_dir)

import convert_fstring

def convert():
  out = convert_fstring.convert(vim.current.line)
  vim.current.line = out

convert()
EOF

endfunction

com! Conv call Convert()
nnoremap Q :Conv<CR>

This is mostly pretty clear, with (I think) two exceptions concerning s:script_dir.

The problem comes from trying to import convert_fstring — it's not in our PYTHONPATH, and the current working directory for python from within vim is a bit weird. It would be possible to directly code in the current directory into the PYTHONPATH, but I think it is more elegant to have convert_fstring.vim add the location of convert_fstring.py to python's path.^{3 3}But admittedly this is a new habit. Perhaps this will change in the future. One can do that through

There are two additions of note that address this import. The first is the line

let s:script_dir = fnamemodify(resolve(expand('<sfile>', ':p')), ':h')

This sets a script-local variable (the prefix s:) containing the directory in which the script was sourced from. (This is where we use the assumption that the vimfile is in the same directory as the python file. If they're in relative directories, then one much change the path addition to python appropriately).

The string <sfile> is populated automatically by vim to contain the filename of the sourcefile. Calling expand with the :p filemodifier returns the complete path of <sfile>. Calling resolve follows symlinks so that the path is absolute. And the final fnamemodify with the :h modifier returns the head, or rather the directory, of the file. Thus the result is to get an absolute path to the directory containing the script (and in this case also the python file).

Then in the python function, we add this location to the path through

script_dir = vim.eval('s:script_dir')
sys.path.insert(0, script_dir)

The first line has vim convert the vim variable into a python string, and the second inserts this directory into the PYTHONPATH.

I have a collection of loosely organized helper scripts in a (creatively named) ~/scripts directory. Some of these were written as pure python shellscripts that I've added some vim wrappers around.

Writing a Plugin

To reiterate, I consider this excessive for my sample application. But we've covered so many of the ideas that we should touch on it now.

Any plugin should follow Tim Pope's Pathogen design model, with the ultimate goal of having a structure that looks like

myplugin/
  README.markdown
  LICENSE
  autoload/
    myplugin.vim
    ...
  plugin/
  ...

Pathogen (or Pathogen-compatible plugin managers, which as far as I know means all of them... like Vundle or vim-plug) will allow plugins to be installed in ~/.vim/bundle, and each plugin there will be added to vim's rtp. ^{4 4}For more on plugin design, I suggest reading Tim Pope's plugins and googling around. There are lots of good ideas out there. Each plugin manager installs plugins (i.e. adds to the runtimepath) in its own way. For Vundle, I include an extra line in my .vimrc.

Suppose I want to package my script as a plugin called fstring_converter. Then I'll create the directory structure

fstring_converter/
  ftplugin/
    python.vim
    convert_fstring.py

where python.vim is what I called convert_fstring.vim before. Placing fstring_converter in ~/.vim/bundle (or symlinking it, or installing it there through e.g. Vundle) will "install it". Then opening a python file will load it, and one can either Conv a line or (as it's currently written) use Q. (Although I think it's a bit unkind and certainly unsustainable to create this mapping for users).

I note that one should generically use autoload/ when available, but I don't touch that now.

As an additional note, vim does add some directories to the PYTHONPATH under the hood. For each directory in vim's runtimepath, vim adds the subdirectory python3 (and also pythonx) to the python module search path. As a result, we could omit the explicit path addition to the python.vim (aka convert_fstring.vim) file if we organized the plugin like

fstring_converter/
  ftplugin/
    python.vim
  python3/
    convert_fstring.py

Regardless, this describes a few ways to make a vim plugin out of our python script.

Leave a comment

Comment via email