Livetext: A smart processor for text

This README is currently mangled. Fixes coming soon!

Livetext is simply a tool for transforming text from one format into another. The source file has commands embedded in it, and the output is dependent on those commands.

Why is this special? It's very flexible, very extensible, and it's extensible in Ruby.

Livetext grew out of several motivations. One was a desire for a markup language that would permit me to write articles (and even books) in my own way and on my own terms. I've done this more than once (and I know others who have, as well).

I liked Softcover, but I found it to be very complex. I never liked Markdown much -- I find it very dumb, and it's not extensible at all. (In fairness to Markdown, it does serve a different purpose in everyday life.)

I wanted something that had the basic functionality of all my ad hoc solutions but allowed extensions. Then my old solutions would be like subsets of the new format. This was a generalization similar to the way we began several years ago to view HTML as a subset of XML.

Here goes:

• It's a text transformer
• It's Ruby-based (later on, more language agnostic)
• It's (potentially) agnostic about output format
• It's designed to be flexible, extensible, and easy
• It's designed to be "plugin" oriented
• It's like an old-fashioned text formatter (but extensible)
• It's like a macro processor (but not)
• It's like markdown and others (but not)
• It's like erb or HAML (but not)
• It's powerful but not too dangerous
• It's not necesarily a markdown replacement
• It's definitely not a softcover replacement
• It could possibly augment markdown, softcover, others

A Livetext file is simply a text file which may have commands interspersed. A command is simply a period followed by a name and optional parameters (at the beginning of a line).

The period will be configurable later if you want to use another character. The names are (for now) actual Ruby method names, so names such as to_s and inspect are currently not allowed.

At present, I am mostly emitting "dumb HTML" or Markdown as output. In theory, you can write code (or use someone else's) to manipulate text in any way and output any format. Technically, you could even emit PDF, PNG, or SVG formats.

It's possible to embed comments in the text. Later it will be possible to pass them through to the output in commented form.

The command .end is special, marking the end of a body of text. Some commands may operate on a block of lines rather than just a few parameters. (A text block is like a here-document.) There is no method name corresponding to the .end command.

The file extension I've chosen is .lt3 (though this may change). Note: The source for this README is a .lt3 file which uses its own little ad hoc library (called tutorial.rb). Refer to the repo to see these.

At first, my idea was to provide predefined commands and allow user-defined commands (to be distinguished by a leading . or .. marker). So the single and double dots were both legal.

However, my concept at present is that the double dots (currently unused) may be used for subcommmands.

User-defined commands may be added to the standard namespace. There are plans to permit commands beginning with a specified character other than the period (to be stored in their own namespace.

When a leading period is followed by a space, that line is a comment. When it is follwed by a name, that name is typically understood to be a method name. Any remaining text on the line is treated as a parameter list to be accessed by that method. Some methods accept a text block (multiple lines of text terminated by a .end tag).

Very commonly we want to format short words or phrases in italics, boldface, or a monospaced (fixed width) font. The Markdown spec provides ways to do this that are fairly intuitive; but I personally don't like them. My own notation works a different way.

First of all, note that these don't work across source lines; they're strictly intra-line. You may need (for example) an italicized phrase that spans across a newline; at present, you'll need a workaround for that.

I find that most short items I want to format are single tokens. Therefore I use a prefixed character in front of such a token: Underscore for italics, asterisk for boldface, and backtick for "code font." The formatting ends when the first blank space is encountered, without any kind of suffixed character.

I also find it's common to want to terminate such a string with some kind of naturally-occurring punctuation mark. If we double the initial delimiter, it will be understood to terminate at the first period, comma, or right parenthesis.

Of course, there are cases where this won't work; a formatted string may contain spaces, or it may exclude characters before the blank space. In this case, we can use an opening bracket after the prefix and a closing bracket at the end of the string.

This means that it can be difficult to include brackets inside a formatted token. The solution is simply to escape with a backslash.

A delimiter character sitting by itself need not be escaped. It will be output as a literal.

A delimiter character that is already inside another string need not be escaped. These cannot be nested (though there is a way to accomplish this using functions).

Most of this is summarized in this example (taken from one of the testcases):

Test: basic_formatting

 Here are examples of *boldface
and _italics
and code as well as *[more complex] examples of \_[italicized text] and [code font].

Here are some random punctuation marks:
; # . * _ ` : @ % ^ & $
No need to escape these:  * _ `

 Here are examples of boldface
and italics
and code
as well as more complex examples
of italicized text
and code font.

 

Here are some random punctuation marks:
; # . * _ ` : @ % ^ & $
 

No need to escape these:  * _ `

The module Livetext::Standard contains the set of standard or predefined methods. Their names are essentially the same as the names of the dot-commands, with occasional exceptions. (For example, it is impractical to use the name def as a method name, so the module has a _def method instead.) Here is the current list:

Here are some tests from the suite. The file name reflects the general purpose of the test.

Test: hello_world

 Hello,
world!

 Hello,
world!

Test: comments_ignored_1

 . Comments are ignored
abc 123
this is a test
. whether at beginning, middle, or
more stuff
still more stuff
. end of the file

 abc 123
this is a test
more stuff
still more stuff

Test: block_comment

 .comment
This is
a comment
.end
abc 123
xyz
.comment
And so is this.
.end

one
more
time
.comment
And so
is
this
.end

 abc 123
 xyz
 one
 more
 time

Test: def_method

 .backtrace
abc
123
.def foobar     # yet another experimental comment
::STDERR.puts "This is the"
::STDERR.puts "foobar method"
.end
xyz
.foobar     # and still another
xyzzy
123

 abc
123
xyz
xyzzy
123

Test: simple_vars

 Just
some text.
.set name=GulliverFoyle, nation=Terra
Hi, there.
$name is my name, and $nation is my nation.
I'm $name, from $nation.
That's all.

 Just
some text.
Hi, there.
GulliverFoyle is my name, and Terra is my nation.
I'm GulliverFoyle, from Terra.
That's all.

Test: simple_include

 Here I am
trying to
include
.include simplefile.inc
I hope that
worked.

 Here I am
trying to
include
a simple
include file.
I hope that
worked.

Test: simple_mixin

 Here I am
testing a simple mixin
.mixin simple_mixin
Now call a method:
.hello_world
That's all.

 Here I am
testing a simple mixin
Now call a method:
Hello, world.
That's all.

Test: simple_copy

 The copy command
copies any file
without interpretation,
such as:
.copy simplefile.inc
That is all.

 The copy command
copies any file
without interpretation,
such as:
a simple
include file.
That is all.

Test: copy_is_raw

 A copy command
does not interpret its input:
.copy rawtext.inc     # another useless comment
That's all.

 A copy command
does not interpret its input:
This is not a comment:
.comment woohoo!
This is not a method:
.no_such_method
That's all.

Test: raw_text_block

 .backtrace
This text block will be passed thru
with no interpretation or processing:
.raw
.comment
This isn't a
real comment.
.end  This isn't picked up.

.not_a_method
And this stuff won't be munged:
`alpha _beta *gamma
Or these:
`(alpha male)
_(beta max)
*(gamma rays)
__EOF__

 This text block will be passed thru
with no interpretation or processing:
.comment
This isn't a
real comment.
.end  This isn't picked up.

.not_a_method
And this stuff won't be munged:
`alpha _beta *gamma
Or these:
`(alpha male)
_(beta max)
*(gamma rays)

Suppose you wanted to write a method called chapter that would simply output a chapter number and title with certain heading tags and a horizontal rule following. There is more than one way to do this.

The simplest way is just to define a method inline with the rest of the text. Here's an example.

     .comment
     This example shows how to define
     a simple method "chapter" inline
     .end
   
     . This is also a comment, by the way.
     .def chapter
        params = <i>args</i>
        raise "chapter: expecting at least two args" unless params.size > 1
        num, <b>title</b> = params     # Chapter number + title
        title = title.join(" ")  # Join all words into one string
        text = <<-HTML
        <h3>Chapter #{num}</h3>
        <h2>#{title}</h2>
        <hr>
        HTML
        <i>puts</i> text
     .end
     . Now let's invoke it...
     .chapter 1 Why I Went to the Woods
     It was the best of times, and you can call me Ishmael. The clocks
     were striking thirteen.

What can we see from this example? First of all, notice that the part between .def and .end (the body of the method) really is just Ruby code. The method takes no parameters because parameter passing is handled inside the Livetext engine and the instance variable @_args is initialized to the contents of this array. We usually refer to the @_args array only through the method _args which returns it.

The _args method is also an iterator. If a block is attached, that block will be called for every argument.

We then create a string using these parameters and call it using the _puts method. This really does do a puts call, but it applies it to wherever the output is currently being sent (defaulting to STDOUT).

All the "helper" methods start with an underscore so as to avoid name collisions. These are all stored in the Livetext::UserAPI module (which also has some methods you will never use).

Here is the HTML output of the previous example:

     <h3>Chapter 1</h3>
     <h2>Why I Went to the Woods</h2>
     <hr>
     It was the best of times, and you can call me Ishmael. The clocks
     were striking thirteen.

What are some other helper methods? Here's a list.

Suppose you wanted to take a list of words, more than one per line, and alphabetize them. Let's write a method called alpha for that. This exercise and the next one are implemented in the test suite.

Test: example_alpha

 .def alpha
text = _body.to_a.join(" ")
words = text.split.sort
words.each {|w| _out "    #{w}" }
.end
Here is an alphabetized list:

.alpha
fishmonger anarchist aardvark glyph gryphon
halcyon zymurgy mataeotechny zootrope
pareidolia manicotti quark bellicose anamorphic
cytology fusillade ectomorph
.end
That's all.

 Here is an alphabetized list:

 

 aardvark
 anamorphic
 anarchist
 bellicose
 cytology
 ectomorph
 fishmonger
 fusillade
 glyph
 gryphon
 halcyon
 manicotti
 mataeotechny
 pareidolia
 quark
 zootrope
 zymurgy

 

That's all.

Test: example_alpha2

 .def alpha
cols = _args.first
cols = "1" if cols == ""
cols = cols.to_i
raise "Columns must be 1-5" unless cols.between?(1,5)
text = _body.join("\n")
text.gsub!(/\n/, " ")
words = text.split.sort
words.each_slice(cols) do |row|
row.each {|w| _out! '%-15s' % w }
_out
end
.end
Here is an alphabetized list:

.alpha 3
fishmonger anarchist aardvark glyph gryphon
halcyon zymurgy mataeotechny zootrope
pareidolia manicotti quark bellicose anamorphic
cytology fusillade ectomorph
.end
And that is all.

 Here is an alphabetized list:

 

aardvark       anamorphic     anarchist
bellicose      cytology       ectomorph
fishmonger     fusillade      glyph
gryphon        halcyon        manicotti
mataeotechny   pareidolia     quark
zootrope       zymurgy
 

And that is all.

Let's assume we have a file called mylib.rb in the same directory as the file we're processing. (Issues such as paths and security have not been addressed yet.) We'll stick the actual Ruby code in here (and nothing else).

   # File: mylib.rb
   
   def alpha
     cols = <i>args.first</i>
     cols = "1" if cols == ""
     cols = cols.to_i
     raise "Columns must be 1-5" unless cols.between?(1,5)
     text = <i>body.join</i>
     text.gsub!(/n/, " ")
     words = text.split.sort
     words.each_slice(cols) do |row| 
       row.each {|w| <i>print</i> '%-15s' % w }
       <i>puts</i> 
     end
   end

Now the .lt3 file can be written this way:

    .mixin mylib
    Here is an alphabetized list:
   
    .alpha 3
    fishmonger anarchist aardvark glyph gryphon
    halcyon zymurgy mataeotechny zootrope
    pareidolia manicotti quark bellicose anamorphic
    cytology fusillade ectomorph
    .end
   
    And that is all.

The output, of course, is the same.

You can define variables in Livetext, defined with .set and referenced with a $$. Later there will be a few predefined variables. Variables are just string values.

Test: simple_vars

 Just
some text.
.set name=GulliverFoyle, nation=Terra
Hi, there.
$name is my name, and $nation is my nation.
I'm $name, from $nation.
That's all.

 Just
some text.
Hi, there.
GulliverFoyle is my name, and Terra is my nation.
I'm GulliverFoyle, from Terra.
That's all.

Test: more_complex_vars

 Just some more text.
.set bday="May 31", date="5/31"
My birthday is $bday, so they tell me.
That's $date if you're American.
That's all.

 Just some more text.
My birthday is May 31, so they tell me.
That's 5/31 if you're American.
That's all.

Test: functions

 Testing out
some functions
here...

.func myfunc
"Eureka!"
.end
I am calling $$myfunc here...
Let's see
what happens.
Functions can take a parameter in brackets
(which it can parse itself as needed):
.func mean
list = param.split(",").map(&:to_f)
sum = list.inject(0, :+)
avg = sum / list.size.to_f
avg.to_s
.end
The result is $$mean[1,2,3,4,5,6,7] as I see it.
Some functions like date and time are predefined.

 Testing out
 some functions
 here...
 

I am calling Eureka! here...
Let's see
what happens.
 

Functions can take a parameter in brackets
(which it can parse itself as needed):
 

The result is 4.0 as I see it.
 

Some functions like date and time are predefined.
 

This list is not prioritized yet.

• Add versioning information
• Clean up code structure
• Add RDoc
• Think about command line executable
• Write as pure library in addition to executable
• Package as gem
• Document: require include copy mixin errout and others
• Need much better error checking and corresponding tests
• Worry about nesting of elements (probably mostly disallow)
• Think about UTF-8
• Document API fully
• Add _raw_args and let _args honor quotes
• Support quotes in .set values
• Support "namespaced" variables (`[.set code.font="whatever"])
• Support functions (``[func is undefined]$c)
• Support function namespacing
• Create predefined variables (e.g., [source_file is undefined]$, $_line])
• Create predefined functions (e.g., $_date)
• More support for markdown
• Allow turning on/off: formatting, variable interpolation, function interpolation?
• .require with file and sigil parameters
• Investigate "common intermediate format" - output renderers all read it
• Comments passed through (e.g. as HTML comments)
• .run to execute arbitrary Ruby code inline?
• Concept of .proc (guaranteed to return no value, produce no output)?
• Exceptions??
• Ruby [AFE is undefined]$ levels?
• Warn when overriding existing names?
• Think about passing data in (erb replacement)
• ]Allow custom ending tag on raw method
• Ignore first blank line after `[.end? (and after raw-tag?)
• Allow/encourage custom passthru method?
• Must have sane support for CSS
• Support for Pygments and/or other code processors
• Support for gists? arbitrary links? other remote resouces?
• Small libraries for special purposes (books? special Softcover support? blogs? PDF? RMagick?)
• Experiment with idea of special libraries having pluggable output formats (via Ruby mixin?)
• Imagining a lib that can run/test code fragments as part of document generation
• Create vim (emacs?) syntax files
• Someday: Support other languages (Elixir, Python, ...)
• .pry method?
• .irb method?
• Other debugging features
• Feature to "break" to EOF?
• .meth? method ending in ? takes a block that may be processed or thrown away (`.else perhaps?)
• .dump to dump all variables and their values
• .if and .else?
• Make any/all delimiters configurable
• HTML helper? (in their own library?)