diff --git a/index.html b/index.html index e98061e..158400f 100644 --- a/index.html +++ b/index.html @@ -12,6 +12,7 @@

simpet

diff --git a/posts/awk_for_static_site_generation.html b/posts/awk_for_static_site_generation.html index 8fa26f2..27be97c 100644 --- a/posts/awk_for_static_site_generation.html +++ b/posts/awk_for_static_site_generation.html @@ -30,33 +30,34 @@

AWK works as follow : it takes an optional regex and execute some code between bracket, as a function, at each line of the text input.

For example :

/^#/ {
-    print "<h1>" $0 "</h1>"
+    print "{{article}}lt;h1{{article}}gt;" $0 "{{article}}lt;/h1{{article}}gt;"
 }
-

Although `$n` refers to the n-th records in the line (according to a delimiter, like in a csv), the special `$0` refers to the whole line.

-

In this case, for each line starting with `#`, awk will print (to the standard output), `<h1> [content of the line] </h1>`.

+

Although $n refers to the n-th records in the line (according to a delimiter, like in a csv), the special $0 refers to the whole line.

+

In this case, for each line starting with #, awk will print (to the standard output), {{article}}lt;h1{{article}}gt; [content of the line] {{article}}lt;/h1{{article}}gt;.

This is the beginning to parse headers in markdown.

-

However, by trying this, we immediatly see that `#` is part of the whole line, hence it also appear in the html whereas it sould not.

+

However, by trying this, we immediatly see that # is part of the whole line, hence it also appear in the html whereas it sould not.

AWK has a way to prevent this, as it is a complete scripting language, with built-in functions, that enable further manipulations.

+

substr acts as its name indicates, it return a substring of its argument.

/^#/ {
-    print "<h1>" substr($0, 3) "</h1>"
+    print "{{article}}lt;h1{{article}}gt;" substr($0, 3) "{{article}}lt;/h1{{article}}gt;"
 }

In the example above, as per the documentation

-

it returns the subtring of `$0` starting at 3 (1 being `#` and 2 the whitespace following it) to the end of the line.

-

Now this is better, but we now are able to generalized it to all headers. Another function, `match` can return the number of char matched by a regex,

-

and allows the script to dynamically determine which depth of header it parses. This length is stored is the global variable `RLENGTH`:

+

it returns the subtring of $0 starting at 3 (1 being # and 2 the whitespace following it) to the end of the line.

+

Now this is better, but we now are able to generalized it to all headers. Another function, match can return the number of char matched by a regex,

+

and allows the script to dynamically determine which depth of header it parses. This length is stored is the global variable RLENGTH:

/^#+ / {
     match($0, /#+ /);
     n = RLENGTH;
-    print "<h" n-1 ">" substr($0, n + 1) "</h" n-1 ">" 
+    print "{{article}}lt;h" n-1 "{{article}}gt;" substr($0, n + 1) "{{article}}lt;/h" n-1 "{{article}}gt;" 
 }

Reproducing this technique to parse the rest proves to be difficult, as lists for example, are not contained in a single line, hence

-

how to know when to close it with `</ul>` or `</ol>`

+

how to know when to close it with {{article}}lt;/ul{{article}}gt; or {{article}}lt;/ol{{article}}gt;

Introducing a LIFO stack

Since according to the markown syntax, it is possible to have nested blocks such as headers and lists withing blockquotes, or lists withing lists, I came with the simple idea to track to current environnement in a stack in AWK.

Turns out it came out to be easy, I only needed a pointer to track the size of the lifo, a fonction to push an element, an another one to pop one out :

@@ -88,8 +89,8 @@ function pop() {

The stack does not have to be strictly declared. The value of inside the LIFO correspond to the current markdown environment.

-

This is a clever trick, because when I need to close an html tag, I use the poped element between a `</` and a `>` instead of having a matching table.

-

I also used a simple `last()` function to return the last pushed value in the stack without popping it out :

+

This is a clever trick, because when I need to close an html tag, I use the poped element between a {{article}}lt;/ and a {{article}}gt; instead of having a matching table.

+

I also used a simple last() function to return the last pushed value in the stack without popping it out :

# Function to get last value in LIFO
 function last() {
     return stack[stack_pointer]
@@ -102,17 +103,17 @@ function last() {
     env = last()
     if (env == "ul" ) {
         # In a unordered list block, print a new item 
-        print "<li>" substr($0, 3) "</li>" 
+        print "{{article}}lt;li{{article}}gt;" substr($0, 3) "{{article}}lt;/li{{article}}gt;" 
     } else {
         # Otherwise, init the unordered list block 
         push("ul")
-        print "<ul>
-<li>" substr($0, 3) "</li>"
+        print "{{article}}lt;ul{{article}}gt;
+{{article}}lt;li{{article}}gt;" substr($0, 3) "{{article}}lt;/li{{article}}gt;"
     }
 }
-

I believe the code is pretty self explanatory, but when the last environement is not `ul`, then we enter this environement.

+

I believe the code is pretty self explanatory, but when the last environement is not ul, then we enter this environement.

This translates as pushing it to the stack.

Otherwise, it means we are already reading a list, and we only need to add a new element to it.

Parsing the simple paragraph and ending the parser

@@ -120,84 +121,85 @@ function last() {

it does not start with a specific caracter. That is, to match it, we match everything that is not a special character.

I have no idea if this is the best solution, but so far it proved to work:

# Matching a simple paragraph
-!/^(#|\*|-|\+|>|`|$|	|    )/ {
+!/^(#|*|-|+|>|`|$|	|    )/ {
     env = last() 
     if (env == "none") {
         # If no block, print a paragraph
-        print "<p>" $0 "</p>"
+        print "{{article}}lt;p{{article}}gt;" $0 "{{article}}lt;/p{{article}}gt;"
     } else if (env == "blockquote") {
         print $0
     }
 }
-

AS `BEGIN`, AWK provide the possibilty to execute code at the very end of the file, with the `END` keyword.

+

AS BEGIN, AWK provide the possibilty to execute code at the very end of the file, with the END keyword.

Naturally we need to empty the stack and close all html tags that might have been opened during the parsing.

It only is a while loop, until the last environement is "none", as it way initiated : 

END {
     env = last()
     while (env != "none") {
         env = pop()
-        print "</" env ">"
+        print "{{article}}lt;/" env "{{article}}gt;"
         env = last()
     }
 }

This way we are able to simply parse markdown and turn it into an HTML file.

-

Of course I am aware that is lacks emphasis, strong and code within a line of text.

-

However I did implement it, but maybe it will be explained in another edit of this post.

-

Nonetheless the code can still be consulted on github.

Parsing in-line fonctionnalities

For now we have seen a way to parse blocks, but markdown also handles strong, emphasis and links. However, these tags can appear anywhere in a line.

Hence we need to be able to parse these lines apart from the block itself : indeed a header can container a strong and a link.

-

A very useful function in awk is `match` : it literally is a regex engine, looking for a pattern in a string.

+

The previously introduced but very useful function match fits this need : it literally is a regex engine, looking for a pattern in a string.

Whenever the pattern is found, two global variables are filled :

-

For the following, `line` represents the line processed by the function, as the following `while` loops are actually part of a single function.

-

This way `match(line, /*([^*]+)*/)` matches a string surrounded by two `*`, corresponding to an emphasis text.

-

The `` are espaced are thez are special characters, and the group* is inside the parenthesis.

-

To matche several instances of emphasis text within a line, a simple `while` will do the trick.

-

We now only have to insert html tags `<em>` are the right space around the matched text, and we are good to go.

-

We can save the global variables `RSTART` and `RLENGTH` for further use, in case they were to be change. Using them we also can extract the

+

For the following, line represents the line processed by the function, as the following while loops are actually part of a single function.

+

This way match(line, /*([^{{article}}#42;]+)*/) matches a string (that does not start with a {{article}}#42) surrounded by two {{article}}#42;, corresponding to an emphasis text.

+

The {{article}}#42; are espaced as they are special characters, and the group is delimited by the parenthesis.

+

To match several instances of emphasis text within a line, a simple while will do the trick.

+

We now only have to insert html tags {{article}}lt;em{{article}}gt; are the right space around the matched text, and we are good to go.

+

We can save the global variables RSTART and RLENGTH for further use, in case they were to be change. Using them we also can extract the

matched substrings and reconstruct the actual html string :

-
while (match(line, /*([^*]+)*/)) {
+
while (match(line, /*([^{{article}}#42;]+)*/)) {
     start = RSTART
     end = RSTART + RLENGTH - 1
-    # Build the result: before match, <em>, content, </em>, after match
-    line = substr(line, 1, start-1) "<em>" substr(line, start+1, RLENGTH-2) "</em>" substr(line, end+1)
+    # Build the result: before match, {{article}}lt;em{{article}}gt;, content, {{article}}lt;/em{{article}}gt;, after match
+    line = substr(line, 1, start-1) "{{article}}lt;em{{article}}gt;" substr(line, start+1, RLENGTH-2) "{{article}}lt;/em{{article}}gt;" substr(line, end+1)
 }
 
 

+
The while loop enables us to repeat this process as many times as this pattern is encountered within the line.

+
  

+
We now can repeat the pattern for all inline fonctionnalities, e.g. strong and code.

 
The case of url is a bit more deep as we need to match two groups : the actual text and the url itself.

-
No real issue here, the naïve way is to match thd whole, and looking for both the link and the url within the matched whole.

-
This way `match(line, /\[([^\]]+)\]\([^\)]+\)/)` matches a text between `[]` followed by a text between `()` : the markdown representation of links.

-
As above, we store the `start` and `end` and also the whole match :

+
No real issue here, the naïve way is to match the whole, and looking for both the link and the url within the matched whole.

+
This way match(line, /[([^]]+)]([^)]+)/) matches a text between [] followed by a text between () : the markdown representation of links.

+
As above, we store the start and end and also the whole match :

 
 

 
start = RSTART
 end = RSTART + RLENGTH - 1
 matched = substr($0, RSTART, RLENGTH)
 
 

-
It is possible to apply the match fonction on this `matched` string, and extract, first, the text in `[]`, and last the text in `()`

-
if (match(matched, /\[([^\]]+)\]/)) {
+
It is possible to apply the match fonction on this matched string, and extract, first, the text in [], and last the text in ()

+
if (match(matched, /[([^]]+)]/)) {
     matched_link = substr(matched, RSTART+1, RLENGTH-2) 
 }
-if (match(matched, /\([^\)]+\)/)) {
+if (match(matched, /([^)]+)/)) {
     matched_url = substr(matched, RSTART+1, RLENGTH-2)
 }
 
 

-
As the link text and the url are stored, using the variables `start` and `end`, it is easy to reconstruct the html line :

-
line = substr(line, 1, start-1) "<a href="" matched_url "">" matched_link "</a>" substr(line, end+1)
+
As the link text and the url are stored, using the variables start and end, it is easy to reconstruct the html line :

+
line = substr(line, 1, start-1) "{{article}}lt;a href="" matched_url ""{{article}}gt;" matched_link "{{article}}lt;/a{{article}}gt;" substr(line, end+1)
 
 

 
The inline parsing function is now complete, all we have to do it apply is systematically on the text within html tags and this finished the markdown parser.

 
This, of course, is the first brick of a static site generator, maybe the most complexe one. 

 
We shall see up next how to orchestrate this parser to make is a actual site generator.

+
The code is available in the repo.

     
 
 
diff --git a/posts/markdown_testing_suite.html b/posts/markdown_testing_suite.html
new file mode 100644
index 0000000..db28940
--- /dev/null
+++ b/posts/markdown_testing_suite.html
@@ -0,0 +1,75 @@
+
+
+
+
+    
+    simpet
+    
+    
+    
+
+
+
+    
simpet

+    

+        
A test suite for markdown parser

+
As I implemented my own markdown parser for bob, my static site (blog) generator, I also wanted to make sure it was parsing markdown correctly.

+
Hence I thought about a custom testing suite. After all this blog is also to make things from scratch to grasp a better understanding of how things work overall.

+
The concept 

+
Also this is a custom script I still wanted to make it somehow generic. In others words I wanted it to be used against any markdown parser (assuming it follows a certain input/ouput constraints).

+
For example, if the test script is test_md_parser, the parser parser, then actually testing the parser shall be the command :

+
test_md_parser parser
+
+

+
The one condition is that parser take the markdown string as a standard input, and print the rendered html in the standard output.

+
This way the markdown parser can feed custom markdown, of which it known the outputed html, and directly compare it with the output of the parser.

+
One more thing is, my markdown parser is written as an awk script, but some may be bash scripts or even executables. This means I need to add an argument to precise the interpreter (if needed).

+
In my case this would look like this :

+
test_md_parser parser.awk awk
+
+

+
and for a bash script, it will be as such :

+
test_md_parser parser.sh bash
+
+

+
Unit testing

+
The purpose of the testing suite is to confront an expect output with the actual outputs from typical markdown syntax.

+
I started by making an array of size 3n, n being the number of tests. Indeed for display purposes each test has 

+
    
+
  • a title : quickly defining what kind of syntax is being tested
    • 
+
  • a markdown input: a legal markdown syntax text
    • 
+
  • an expected output: the corresponding html output
    • 
+

+
This approach has flaws, obviously, and the biggest one being the consistence of html. Indeed this html :

+
{{article}}lt;h1{{article}}gt;Title{{article}}lt;/h1{{article}}gt;
+
+

+
is strictly equivalent to :

+
{{article}}lt;h1{{article}}gt;
+    Title
+{{article}}lt;/h1{{article}}gt;
+
+

+
whereas the strings are not equal.

+
The most naive approach I came with (and because I wanted a quick prototype so I didn't think much about it) was to remove all carriage return from the parser output, using tr -d '{{article}}#92;n'.

+
The best solution would be to implement an html minimizer, and apply this minimization on the output of the parser (and maybe the expected result as well) to ensure perfect equality no matter how many carriage return and trailing spaces there would be. Most likely this could be done in a further version.

+
All tests are hard coded within the script. I am aware this might not be the best solution, but on the other hand as this is a script, and not a compiled program, it is as easy as changing "hardcoded" tests as it would be on a separate config file, so its does not bother me right now.

+
Implementation of the testing suite

+
As mentionned earlier, all tests are defined in a array of size 3n as such : 

+

+declare -a tests=(
+    "Test header"
+    "# Header1"
+    "{{article}}lt;h1{{article}}gt;Header1{{article}}lt;/h1{{article}}gt;é
+)
+
+

+
Running a single test would take an input, run the parser against it, store the output and compare it to the expected. It would return 0 for success and 1 in case of failure.

+
Looping over the whole array introduced above, with the input being all 3n+1 elements of the array and expected being all 3n+2 ensures all tests are executed in order and as they should.

+
To know whether or not all tests were successful, I simply make the sum of all the returned status of the run_test() function : if the sum equals 0, logically all tests are ok.

+
However I added a nice console output that prints all tests as they are being executed, which also prints in green succesful tests, and in red failed tests.

+
This very simple testing suite would be part of the bob blog generator.

+    

+
+
+