Saturday, 18 January 2014

Starting with Elisp and external processes

1 Getting started with Elisp

This post will describe my first steps with Emacs Lisp and how to use it to communicate with an external process.

The little 'script' I implemented will:

  • use the inotifywait tools on Linux to monitor filesystem changes1.
  • call a function to react to them.

As for all things Emacs, the Emacs documentation is incredibly detailed and very, very good and all informations in this post can be found (with a lot more information) in the documentation.

This post is simply a little write up of the most basic steps to get started.

It will cover the following steps:

  1. How to start a external program from Emacs.
  2. How to react to output generated from the started process.

2 Starting a process in Emacs

When deciding to start an external program there are two possibilities: either start a synchronous process or an asynchronous one.

As I want inotifywait to keep running in the background, while reacting to the output I have to start an external process using the start-process function provided by emacs:

(start-process "some-name" ;; A name for the process
        "*buffer-to-write-output-into*" ;; A buffer the process can write to
        "inotifywait" ;; The command to run
        ;; Following are any arguments that are required by the command:
        "-m"
        "-e"
        "create"
        "<file_to_watch>")
      

It is important that the command-line arguments are passed as variadic arguments, so if you want to apply this command to a list of arguments you already have stored in a list you can simply use the apply-function 2.

(process (apply 'start-process "some-name"
        "*buffer-to-write-output-into*"
        "inotifywait" argument-list))
      

That is all there is to starting an asynchronous process in Emacs.

3 Reacting to output from the process

Reacting to output generated by the process is done using classic callbacks.

To register the callback-function with a process, you have to store the return value of the start-process call and register a callback using the set-process-filter function:

(defun callback-func (process msg)
        "Callback function need to accept two arguments:
        - the process that created the output.
        - the message containing the currently produced output."
        (message msg))

        (let* ((arguments (list "-m" "-e" "create" "<filepath>"))
        (process (apply 'start-process "NOTIFYWAIT" "*notifywait*" "inotifywait" arguments)))
        (set-process-filter process 'callback-func))
      

Now all output generated by the inotifywait-command will be send to the callback and can be processed in whatever fashion required.

I wrote a little callback that will run all nosetests for a Python-project for example, as soon as a file under the source-directory changes. You can find the source-code on Github.

Footnotes:

1

If you are using Emacs ≥ 24.4 then cross-platform inotify-functionality is included in Emacs. As both Debian and OpenSUSE both still provide Emacs 24.3 however, I worked around this with this small script.

2

If you know Python then you can think of apply as being the same as calling a function with func(*a_list).

Wednesday, 8 January 2014

Automatically install packages in new virtual environments

1 Automatically install packages into a virtual environment

When using multiple virtual environments in Python I happen to always require the same basic packages in the environments I use:

# Default python tools
pyflakes
pep8
nose
# Emacs requirements
jedi
epc
sexpdata
# Ipython and its requirements
ipython
Jinja2
tornado
pyzmq

When creating new environments often to try out a few things it becomes quite cumbersome to always install the packages by hand.

So instead I was looking for a quick and easy solution to the problem and found it via virtualenvwrapper.

Installing and configuring virtualenvwrapper

As virtualenvwrapper is just a wrapper around virtualenv it is required to have virtualenv installed first1.

After that you can just use pip to install virtualenvwrapper.

sudo pip install virtualenvwrapper

Now add the following line to your shell-initialization file (probably ~/.bashrc or ~/.zshrc) and re-source the file:

export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
# Only add the previous export if you want to use virtualenvwrapper
# with Python3
export WORKON_HOME=$HOME/venv
source /usr/local/bin/virtualenvwrapper.sh

Now everything is ready to define the required hooks for automatically installing the default packages: by default the hooks are defined in the WORKON_HOME directory.

The hook we want to modify is the postmkvirtualenv-hook that is called after the successful creation of the environment.

I just created a file holding all requirements inside the WORKON_HOME directory that I can then install via this simple hook:

pip install -r "$WORKON_HOME/default_requirements.txt"

Footnotes:

1

Except if you are using a Python ≥ 3.3 which includes the pyvenv command. You can set virtualenvwrapper to use the venv module by also adding the following export to your shell-initialization:

export VIRTUALENVWRAPPER_VIRTUALENV=pyvenv-3.3

This command will not install pip inside the new environment however.

Thursday, 25 July 2013

Org-Mode (version 8) and pdf export with syntax highlighting

This post is will describe how to get a nice syntax-highlighted PDF file by using Emacs' org-mode with LaTeX and Python.

Why?

I like to keep my notes in org-mode files as those are plain-text and thus can be edited using Emacs (or any other good text editor) while remaining easy to search.

However, sometimes these notes tend to become big enough to warrant the production of a small document. One of the easiest ways to produce a pretty output remains LaTeX. And it is equally easy to make the source code exports look really nice!

How?

This guide assumes that org-mode >= version 8 is used and LaTeX (including the minted package) is already installed.1

Next a installed version of pygments is needed2. The easiest way to obtain that is via pip:

pip install pygments

Next we have to setup the org-mode LaTeX-export to use the minted package:

;; Include the latex-exporter
(require 'ox-latex)
;; Add minted to the defaults packages to include when exporting.
(add-to-list 'org-latex-packages-alist '("" "minted"))
;; Tell the latex export to use the minted package for source
;; code coloration.
(setq org-latex-listings 'minted)
;; Let the exporter use the -shell-escape option to let latex
;; execute external programs.
;; This obviously and can be dangerous to activate!
(setq org-latex-pdf-process
      '("xelatex -shell-escape -interaction nonstopmode -output-directory %o %f"))

To test the process you can simply evaluate these commands in the *scratch*-buffer.

If you want to toggle these options in org-mode on a per-file basis you can simple put the following snippet at the top of your org-file:

#+BEGIN_SRC emacs-lisp :exports results :results silent
  (require 'ox-latex)
  (add-to-list 'org-latex-packages-alist '("" "minted"))
  (setq org-latex-listings 'minted)
  (setq org-latex-pdf-process
        '("xelatex -shell-escape -interaction nonstopmode -output-directory %o %f"))
#+END_SRC

No matter which way you choose the following command should produce a PDF-file with syntax highlighted source-code:

org-latex-export-to-pdf

Footnotes:

1

Installing new packages in modern LaTeX distributions like texlive or miktex is really simple and explained: here (texlive) and here (miktex).

2

I recommend using a virtualenv if you do not want to pollute your global Python installation.

Sunday, 11 November 2012

Install matplotlib on Ubuntu 12.10 for python 3

Install matplotlib Ubuntu 12.10

This is a follow-up post for installing matplotlib for Python 3, now that matplotlib 1.2 has been released and officially supports Python 3: it will describe how to install all tools necessary to setup a virtualenvironment, and how to create one that will host the installation of matplotlib and numpy.

Install distribute & pip

Installation instructions for the current version can be found at the Python Package Index.

Normally the following code will suffice:


wget http://python-distribute.org/distribute_setup.py
sudo python3 distribute_setup.py

This will finish installing distribute which will allow us to install pip.


# Careful to use the correct easy_install if you have it installed
# for python2 as well!
sudo easy_install pip

Now you have pip installed and can proceed to install virtualenv and virtualenvwrapper (a number of scripts that make using virtual environments a lot easier).

Install virtualenv & virtualenvwrapper

To install these two packages make sure to use the correct version of pip if you have it installed for Python 2 and Python 3:


sudo pip install virtualenv
sudo pip install virtualenvwrapper

Now you have to make sure virtualenvwrapper is loaded and is using Python 3. To do this just add the following two lines to your bashrc (or zshrc) file:


VIRTUALENVWRAPPER_PYTHON='/usr/bin/python3'
source /usr/local/bin/virtualenvwrapper.sh

Installing matplotlib

First create a new virtual environment for matplotlib:


mkvirtualenv <name_of_environment>

The virtual environment will normally be automatically activated (you can normally see this when the shell prompt is prepended with (<name_of_environment>)<rest_of_your_prompt>:).

If it is not activated simply issue the following command:


workon <name_of_environment>

Now before installing matplotlib we need to install a few more dependencies: to install numpy we need Python header files and to compile numpy we need a c-compiler.

To be able to display plots using the TkAgg back-end we will also need the Tk-libraries, as well as the tk-headers and libpng headers.

All these dependencies can be installed with apt-get:


sudo apt-get install python3-dev build-essential python3-tk tk-dev libpng12-dev

Then you can install the dependencies for matplotlib and matplotlib itself:


pip install numpy
# using <package>==<version> syntax makes sure to install exactly the
# specified version.
pip install matplotlib==1.2

This should allow you to run the following script and produce a simple linear graph:


import matplotlib.pyplot as plt
plt.plot([1,2,3,4])
plt.ylabel('some numbers')
plt.show()

Wednesday, 22 August 2012

Installing Matplotlib on Python3

Having recently tried to install matplotlib in a virtual environment for Python 3, I encountered some problems and decided to document the process I had to take to make sure it works:

In retrospective it actually was really easy and can be achieved with the following few steps [1]:


sudo apt-get install python3-tk tk tk-dev
pip install numpy
pip install git+https://github.com/matplotlib/matplotlib.git

This will install the necessary packages for matplotlib:

  • numpy for fast mathematical computations.
  • tk for displaying the generated plots.

After these steps the following small program should display a very simple linear graph:


import matplotlib
matplotlib.use("TkAgg")
import matplotlib.pyplot as plt
plt.plot([1,2,3,4])
plt.ylabel('some numbers')
plt.show()

[1]Assuming you have already installed pip and are running Ubuntu or Linux Mint.

Monday, 9 April 2012

Populating the vim error-list from a file

When using vim to compile code it is possible to let it parse the messages from the built tool and create a error-list that can be used to jump to the correct files (and most often the correct line).

When using a tool such as grep or the Windows Powershell Select-Pattern it is possible to obtain output that resembles a certain form:

filename:linenumber: found-string
filename2:linenumber2: found-string

This can be used in vim to create an error-list as well, that can be used to jump between these matches incredibly fast - the only things that need configuration are the grep-command, the makeprg-command and the errorformat 1.

Adjusting grep

To obtain all the necessary information from grep it is necessary to issue the command with the following parameters:

grep -Hn pattern file
-H

Will make grep print the file the match was found in.

-n

Will print the line-number of the match.

Adjusting makeprg

Instead of compiling a program, the makeprg command should simply echo back the current buffer that should be used to create the error-list - this can easily be achieved by simply using the echo command.

Therefore the following command must be used in vim.

set makeprg=cat\ %

When running the command make in vim now it will simply echo the currently active buffer and try to match the pre-defined errorformats against it 2.

Adjusting the errorformat

Running make after adjusting the makeprg-command might already work - if not a new errorformat can be set with the following text:

set errorformat+=%f:%l:%m
%f

Will identify the token that represents the file-name.

%l

Will identify the token that represents the line-number.

%m

Will identify the token that represents the error-message 3.

Due to the += the new errorformat will simply be appended to the already defined ones and not overwrite former lists.


  1. It is also possible to use the vimgrep-command - especially on Windows that seemed to be very slow and using the built-in commands and this method proved much faster.

  2. To see the pre-defined formats the command set errorformat? can be used.

  3. To see more possible tokens that can be used check :h errorformat.

Monday, 12 March 2012

Closing file handles on Windows

While using Visual Studio 2010 I ran into a really annoying issue that prevented the IDE from copying files to the output directory, because the files were locked:

Unable to copy file '...bin\Debug\[ProjectName].config'. Access to the path '...bin\Debug\[ProjectName].config' is denied."

This behaviour seems to be a not so rare bug in Visual Studio, as can be seen from this Stackoverflow post.

Having tried many of the suggested solutions and still getting the error occasionally, I decided wasting my time by manually finding out what process is using the directory and killing it via the Sysinternal Process Explorer was too much of a hassle.

So I wrote a small one-liner that uses the handle.exe program from the Sysinternal Suite to automatically find and kill all handles on the project-dir.

To run this "script" you need the Linux utilities awk and sed on your path. (which you can either get via Cygwin or from GnuWin32). Moreover it writes the commands to really close the handles to a Powershell-script that will be executed to close the handles.

Warning: This script requires administrative privileges to run (otherwise handle.exe will not work), and forcefully closing handles can cause system instability. If you are unsure about closing the handles just delete the | .\kill_handles.ps1-part in the first listing - this will store all commands that will really close the handles in a file called .\kill_handles.ps1 but will not immediately run that file.

Because of some peculiarities of awk on Windows three files are needed, as directly using the awk-parts did not work on Windows.

This code is the main script that just runs all commands in order - it should be safest to replace [path_to_project_dir] with the full path - otherwise handle.exe might pick up directories whose handles you do not want to close. The same goes is true for the [project_name] - this command is used to just extract the lines needed from the handle output - using the folder name of the solution should normally suffice - to check if the output looks good just delete everything behind that command and analyze the output to make sure.

handle.exe [path_to_project_dir]\bin | awk '/[project_name]/' | awk -f extract_pid_hex.awk | sed 's/://g' | awk -f write_handle_commands.awk ] kill_handles.ps1 | .\kill_handles.ps1

This snippet should be put into an awk-file called extract_pid_hex.awk - it extracts the process- and handle-ids from the handle.exe output.

{print $3 " " $6}

The following snippet is an awk-file that writes the commands to close handles. This file should be named write_handle_commands.awk.

{print "handle.exe -p " $1 " -c " $2 " -y"}

After running these commands I at least could always delete the bin directory of the project and rebuild it again without any errors.

If anyone knows a way of doing this only using powershell I would be glad to know how - it might be a good way to finally getting to know it.