Page Contents

2020-10-12 #rake #ruby #static_site_generation

I’m a big fan of Ruby’s rake program, which provides a convenient way to run tasks from the command line. More importantly, these tasks are written in Ruby. I use rake everywhere:

  • in ruby projects, to run/test/package code

  • in java projects, to download dependencies, run, test and package code into self-contained zip files

  • in writing projects, to turn asciidoc or latex into html or pdf and display them

  • etc

Here, I want to talk about how (and why) I use rake as a static-site generator (SSG) for my website. An SSG takes a collection of files, usually written in some kind of markup language, and produces a coherent set of html files ready to place on a web server.

There are many popular SSGs and I have previously used jekyll, and then hugo. However, when I wanted to add something to my website, I started to get caught up in hugo’s template mechanisms, the assumptions of the theme I was using, etc, …​ and soon realised this was becoming too much to learn. It didn’t help that most hugo/jekyll themes seem targetted at blogging, rather than to create a static website.

What is an SSG at heart? I use asciidoc to write my files. asciidoctor is a program which converts asciidoc files into html. What else do I need? A common template to provide a menu, some inter-page links, etc. Most of this is just pushing files around and adding text.

I decided to try some ruby code and a rake file to create my own (admittedly small and simple) websites. The following steps walk through the creation of a simple SSG using rake tasks and some ruby coding.

To follow this, you will need:

  • ruby and rake (see https://www.ruby-lang.org)

  • asciidoctor, which is a ruby program, and can be installed using gem install asciidoctor

  • to be able to code using text editor and terminal

Step 1: creating and displaying pages

We will start with some conventions:

  • the "pages" directory will contain the asciidoc documents we want to display, with the file extension "adoc". Call this the "source" directory.

  • the "site" directory will contain the built website.

  • "rakefile.rb" sits at the root, and contains all our ruby code and rake tasks.

First, put some files into "pages" - there must be at least one file named "index.adoc" to act as the home page.

e.g.

pages/index.adoc
  = Home Page

  This is the home page.
pages/about.adoc
  = About

  This is the about page.

  Some code:

  [source,ruby]
  ----
  def say_hello
    puts "Hello"
  end
  ----

To build our website, we need to convert each .adoc file into .html using asciidoctor, and save it to the "site" directory. The following code goes into "rakefile.rb"

CWD = File.expand_path(File.dirname(__FILE__))
SRC = File.join(CWD, 'pages')   # location of adoc files    1
SITE = File.join(CWD, 'site')   # location of final html files

def make_pages
  Dir.mkdir(SITE) unless Dir.exist?(SITE)                   2
  Dir.foreach(SRC) do |file|
    next unless file.end_with? 'adoc'                       3
    `asciidoctor -D site #{File.join(SRC,file)}`            4
  end
end
  1. Work out the absolute path to the source directory, "pages", for easy reference.

  2. Create the target directory, unless it exists.

  3. Make sure we only process .adoc files in the source directory.

  4. Use asciidoctor to do the processing. Notice the -D option to save the result in the "site" directory.

To call this, we create a rake task (add this to "rakefile.rb"):

desc 'build the web site into "site"'
task :build do
  make_pages
end

Ruby has its own built-in webserver, and you can show the pages from the "sites" directory as simply as:

desc 'show web site - port 8000'
task :show do
  Dir.chdir('site') do
    `ruby -run -ehttpd . -p8000`
  end
end

We can now run our simple SSG. First, let’s see which tasks are available:

$ rake -T
rake build  # build the web site into "site"
rake show   # show web site - port 8000

Now we can build and view the pages:

$ rake build
Converting file: index.adoc
Converting file: about.adoc
make page: index.html
make page: about.html
$ rake show
[2020-04-27 18:24:57] INFO  WEBrick 1.6.0
[2020-04-27 18:24:57] INFO  ruby 2.7.0 (2019-12-25) [x86_64-linux]
[2020-04-27 18:24:57] INFO  WEBrick::HTTPServer#start: pid=6232 port=8000

Point browser at "localhost:8000" to see the index page, and "localhost:8000/about.html" to see the about page.

Success!

Step 2: adding a common menu

The next step is to tie the site together with common header/footer content. We divide this into two steps:

  1. create the "shell" of our page, without a header or footer. This is done by calling asciidoctor with the "-s" (or "--no-header-footer") switch. For convenience, the output is left in the source directory.

  2. for each page, create a complete html file by combining our custom header, the shell code for our page, and our custom footer.

The make_shells method is not much different from make_pages above, except we work in the source directory and use different options when calling asciidoctor.

Notice the selection of source-highlighter - I am using rouge.

def make_shells
  Dir.chdir(SRC) do
    Dir.foreach('.') do |file|
      next unless file.end_with? 'adoc'
      puts "Converting file: #{file}"
      `asciidoctor -a source-highlighter=rouge -s #{file}`
    end
  end
end

The make_pages method must now construct the complete page.

Something we will find useful later is to provide information about each page. In jekyll or hugo this information is provided as a kind of header on the source asciidoc file. Here, I will make this simple, and provide information in the rakefile so it can be looked for when needed. (I like to put this at the top of the rakefile, so it’s easy to see and edit when I add new pages.)

To start, we want a page title for each file:

Page = Struct.new(:filename, :title)          1

PAGES = [ Page.new('index', 'Home'),          2
          Page.new('about', 'About')
]

def page(filename)                            3
  PAGES.find {|p| p.filename == filename }
end
  1. A structure is used to make it easy to retrieve information.

  2. An array PAGES holds information about each page.

  3. A useful method to look up the Page information for a given filename - this is the filename of the adoc file without the .adoc ending.

Then we need a method to return the header for our pages:

def header(page_name)
  title = page(page_name).title

  return <<END
  <html><head>
    <title>#{title} :: Rakefile SSG</title>
  </head>
  <body>
    <a href="/index.html">Home</a> |
    <a href="/about.html">About</a>
    <hr>
    <div>
    <h1>#{title}</h1>
END
end

Notice that:

  1. We retrieve the title from the page information, created above.

  2. The title field in the html header uses this title - output whatever information you need.

  3. At the start of the body, we output our simple menu.

  4. The title of the page is given as a "h1" header, ready for the page content to follow.

The footer information:

def footer
  "</body></html>"
end

Naturally, you can expand on the header/footer information as much as you wish. For example, the footer could contain contact or copyright information.

Next, we need to rewrite the make_pages method so it will put the header/footer information around the content shell:

def make_pages
  Dir.mkdir(SITE) unless Dir.exist?(SITE)
  Dir.foreach(SRC) do |file|
    next unless file.end_with? 'html'                     1
    page_name = file.sub('.html', '')                     2
    puts "make page: #{file}"

    File.open(File.join(SITE, file), "w") do |site_file|  3
      # output header                                     4
      site_file.puts header(page_name)
      # output shell
      IO.foreach(File.join(SRC, file)) do |line|
        site_file.puts line
      end
      site_file.puts "</div>"
      # output footer
      site_file.puts footer
    end
  end
end
  1. We will treat each .html file in the source directory.

  2. Get the filename without extension, to retrieve its page information.

  3. Create a new file in the "site" directory, with the same name.

  4. Output the header, copy in the shell, and add the footer.

Finally, add a call to the make_shells method at the top of the "build" task, and your website will now use your html template.

Step 3: adding some style

A CSS style file controls the layout, colours and general format of your website. I have a directory "styles" in which I have a "style.css" stylesheet for the main pages.

Here’s a simple style for illustration:

  • sets white text on a black background.

  • green h1-level headings.

  • underlined yellow links, blue when hovered over.

style.css
body {
  background-color: #000000;
  color: #ffffff;
}

h1 {font-size: 1.6em; color: #ffff00;}
a:link { color: #ffff00; text-decoration: underline; }
a:visited { color: #ff0000; text-decoration: underline; }
a:hover { color: #ff00ff; text-decoration: underline; }

Also needed is the style sheet for the source highlighter - you can create one from rouge using:

$ rougify style molokai > styles/syntax.css

Add the following lines to the "HEAD" section of your HTML header:

    <link rel="stylesheet" type="text/css" href="/styles/style.css">
    <link rel="stylesheet" type="text/css" href="/styles/syntax.css">

In your "build" task, add a line to copy the styles directory into site.

desc 'build the web site into "site"'
task :build do
  make_shells
  make_pages
  `cp -r styles site`
end

Check that rebuilding and showing your website uses the custom header and style sheets. (The Ruby code on the "about" page should now be highlighted.)

Step 4: tagging and a sidebar

We added a simple menu to our website, but another useful way to index pages is through some kind of "tag" system. We will optionally provide a list of tags for our pages, and have a sidebar listing the tags, with a page created for each tag listing the pages which contain that tag.

Adding tag information to pages

First, we need to add tag information to our pages. We do this by adding a "tags" field to the Page struct, and including some tags to our pages. For pages without tags, the "tags" field should contain the empty list.

e.g. make a page:

pages/ssg-generator.adoc
= Static Site Generator in Rake

A static-site generator (SSG) takes a collection of files, usually written in
some kind of markup language, and produces a coherent set of html files ready
for a web server.

Edit the PAGES descriptions to include the tags:

Page = Struct.new(:filename, :title, :tags)

PAGES = [ Page.new('index', 'Home', []),
          Page.new('about', 'About', []),
          Page.new('ssg-generator', 'Static Site Generator', [:ruby, :ssg])
]

def tags                       1
  PAGES.collect(&:tags).flatten.compact.uniq.sort
end

def pages_for_tag(tag)         2
  PAGES.select {|p| p.tags.include?(tag)}.sort {|a,b| a.title.upcase <=> b.title.upcase}
end
  1. Method returns a sorted list of all the tags used in our website.

  2. Method returns a list of the pages which include the given tag.

Second, for each tag, we need to create a page with a list of the pages under that tag. These new pages are placed in "site/tags".

def make_tag_pages
  Dir.chdir(SITE) do
    Dir.mkdir('tags') unless Dir.exist?('tags')
    tags.each do |tag|                          1
      File.open(File.join('tags', "#{tag}.html"), "w") do |f|
        f.puts header("Tag: #{tag}")
        f.puts "<ul>"
        pages_for_tag(tag).each do |page|       2
          f.puts "<li><a href=\"/#{page.filename}.html\">#{page.title}</a></li>"
        end
        f.puts "</ul>"
        f.puts "</div>"
        f.puts footer
      end
    end
  end
end
  1. For each tag in the website, we create a new web page. Notice the header/footer etc uses the same code as above, for making a page. This way, all the pages have a consistent template.

  2. The method pages_for_tag returns a list of all the pages using this tag. A simple link to the page is placed within a list on the page.

Call make_tag_pages in the build task.

One point to tidyup here is that our header method is now being called for pages created by our code - these pages are not listed in PAGES. So the first part of the header method must be changed, to provide a title for the tag pages:

def header(page_name)
  title = if page(page_name)
            page(page_name).title
          else
            page_name
          end

  # ... rest is the same as above
end

Create the sidebar listing all tags

Third, a sidebar should be created, with each tag listed and linking to its respective page. We create this sidebar in its own section of each webpage:

def sidebar(file)
  file.puts "<div>"               1
  file.puts "<h3>Tags</h3>"       2
  file.puts "<ul>"
  tags.each do |tag|              3
    file.puts "<li><a href=\"#{File.join('/tags', "#{tag}.html")}\">#{tag}</a></li>"
  end
  file.puts "</ul>"
  file.puts "</div>"
end
  1. Creates the new division (section) for the tag sidebar

  2. Outputs a simple header

  3. Loops through each tag, and links to its page in the "tags" directory.

This method must be called before adding the footer, in both the make_pages and the make_tags_pages methods: its input is the file being written to.

If you try building the website now, you will see a list of tags underneath the main content of each page. This is not how we want things laid out, but the links should work.

Sidebar positioning

Fourth, the sidebar must be placed correctly, and we do this using style sheets.

The main content and sidebar are currently in their own div sections: these sections must be given distinct names.

def sidebar(file)
  file.puts "<div class=\"content tags\">"

  # ...
end

def header(page_name)
  # ...

    <hr>
    <div class=\"content main\">
    <h1>#{title}</h1>
END
end

Finally, we specify the position of the two sections in css. Add the following to "styles/style.css":

.content.tags {
  width: 15%;
  float: right;
}

.content.main {
  width: 85%;
  float: left;
}

Step 5: speedier updates

The slowest part of building the website is waiting for asciidoctor to convert each individual file to html. The second part, putting together the "site" directory, is almost instantaneous. A solution is to have an "update" task which will only rebuild files which are new or have changed.

For this, we use ruby’s exist? and mtime methods. We adapt the make_shells method so it takes a parameter update, which we set to true when using the "update" task. This will then check if the target html file exists and, if it does, whether its last modification time is before that of the source adoc file. Asciidoctor is then only called on files that need creating or updating.

      if update
        target = file.sub(/adoc\Z/, 'html')
        if !File.exist?(target) or File.mtime(target) < File.mtime(file)
          puts "Updating file: #{file}"
          `asciidoctor -a source-highlighter=rouge -s #{file}`
        end
      else
        puts "Converting file: #{file}"
        `asciidoctor -a source-highlighter=rouge -s #{file}`
      end

The "update" task then looks just like the "build" task, except make_shells is called with the update parameter set to true.

You can then keep your website showing in the browser while editing your latest page, and call "update" before refreshing the page in your browser to view the changes.

Conclusion

This website was built in this way, but I added an optional date and tags for each page, and I changed my mind about showing tags in a sidebar. Which is ironic, because that was the "something" I was trying to add in the first place which led me away from jekyll/hugo to this approach!