This commit is contained in:
parent
1d289989af
commit
7d5da2b981
@ -12,6 +12,7 @@
|
||||
<body>
|
||||
<h1 class='title'>simpet</h1>
|
||||
<ul>
|
||||
<li><a href="./posts/markdown_testing_suite.html">markdown testing suite</a></li>
|
||||
<li><a href="./posts/awk_for_static_site_generation.html">awk for static site generation</a></li>
|
||||
</ul>
|
||||
</body>
|
||||
|
||||
@ -30,33 +30,34 @@
|
||||
<p>AWK works as follow : it takes an optional regex and execute some code between bracket, as a function, at each line of the text input.</p>
|
||||
<p>For example :</p>
|
||||
<pre><code>/^#/ {
|
||||
print "<h1>" $0 "</h1>"
|
||||
print "{{article}}lt;h1{{article}}gt;" $0 "{{article}}lt;/h1{{article}}gt;"
|
||||
}
|
||||
</code>
|
||||
</pre>
|
||||
<p>Although `$n` refers to the n-th records in the line (according to a delimiter, like in a csv), the special `$0` refers to the whole line.</p>
|
||||
<p>In this case, for each line starting with `#`, awk will print (to the standard output), `<h1> [content of the line] </h1>`.</p>
|
||||
<p>Although <code>$n<code> refers to the n-th records in the line (according to a delimiter, like in a csv), the special </code>$0</code> refers to the whole line.</p>
|
||||
<p>In this case, for each line starting with <code>#<code>, awk will print (to the standard output), </code>{{article}}lt;h1{{article}}gt; [content of the line] {{article}}lt;/h1{{article}}gt;</code>.</p>
|
||||
<p>This is the beginning to parse headers in markdown.</p>
|
||||
<p>However, by trying this, we immediatly see that `#` is part of the whole line, hence it also appear in the html whereas it sould not.</p>
|
||||
<p>However, by trying this, we immediatly see that <code>#</code> is part of the whole line, hence it also appear in the html whereas it sould not.</p>
|
||||
<p>AWK has a way to prevent this, as it is a complete scripting language, with built-in functions, that enable further manipulations.</p>
|
||||
<p><code>substr</code> acts as its name indicates, it return a substring of its argument.</p>
|
||||
<pre><code>/^#/ {
|
||||
print "<h1>" substr($0, 3) "</h1>"
|
||||
print "{{article}}lt;h1{{article}}gt;" substr($0, 3) "{{article}}lt;/h1{{article}}gt;"
|
||||
}
|
||||
</code>
|
||||
</pre>
|
||||
<p>In the example above, as per the <a href="https://www.gnu.org/software/gawk/manual/html_node/String-Functions.html#index-substr_0028_0029-function">documentation</a> </p>
|
||||
<p>it returns the subtring of `$0` starting at 3 (1 being `#` and 2 the whitespace following it) to the end of the line.</p>
|
||||
<p>Now this is better, but we now are able to generalized it to all headers. Another function, `match` can return the number of char matched by a regex,</p>
|
||||
<p>and allows the script to dynamically determine which depth of header it parses. This length is stored is the global variable `RLENGTH`:</p>
|
||||
<p>it returns the subtring of <code>$0<code> starting at 3 (1 being </code>#</code> and 2 the whitespace following it) to the end of the line.</p>
|
||||
<p>Now this is better, but we now are able to generalized it to all headers. Another function, <code>match</code> can return the number of char matched by a regex,</p>
|
||||
<p>and allows the script to dynamically determine which depth of header it parses. This length is stored is the global variable <code>RLENGTH</code>:</p>
|
||||
<pre><code>/^#+ / {
|
||||
match($0, /#+ /);
|
||||
n = RLENGTH;
|
||||
print "<h" n-1 ">" substr($0, n + 1) "</h" n-1 ">"
|
||||
print "{{article}}lt;h" n-1 "{{article}}gt;" substr($0, n + 1) "{{article}}lt;/h" n-1 "{{article}}gt;"
|
||||
}
|
||||
</code>
|
||||
</pre>
|
||||
<p>Reproducing this technique to parse the rest proves to be difficult, as lists for example, are not contained in a single line, hence </p>
|
||||
<p>how to know when to close it with `</ul>` or `</ol>`</p>
|
||||
<p>how to know when to close it with <code>{{article}}lt;/ul{{article}}gt;<code> or </code>{{article}}lt;/ol{{article}}gt;</code></p>
|
||||
<h2>Introducing a LIFO stack</h2>
|
||||
<p>Since according to the markown syntax, it is possible to have nested blocks such as headers and lists withing blockquotes, or lists withing lists, I came with the simple idea to track to current environnement in a stack in AWK.</p>
|
||||
<p>Turns out it came out to be easy, I only needed a pointer to track the size of the lifo, a fonction to push an element, an another one to pop one out :</p>
|
||||
@ -88,8 +89,8 @@ function pop() {
|
||||
</code>
|
||||
</pre>
|
||||
<p>The stack does not have to be strictly declared. The value of inside the LIFO correspond to the current markdown environment.</p>
|
||||
<p>This is a clever trick, because when I need to close an html tag, I use the poped element between a `</` and a `>` instead of having a matching table.</p>
|
||||
<p>I also used a simple `last()` function to return the last pushed value in the stack without popping it out :</p>
|
||||
<p>This is a clever trick, because when I need to close an html tag, I use the poped element between a <code>{{article}}lt;/<code> and a </code>{{article}}gt;</code> instead of having a matching table.</p>
|
||||
<p>I also used a simple <code>last()</code> function to return the last pushed value in the stack without popping it out :</p>
|
||||
<pre><code># Function to get last value in LIFO
|
||||
function last() {
|
||||
return stack[stack_pointer]
|
||||
@ -102,17 +103,17 @@ function last() {
|
||||
env = last()
|
||||
if (env == "ul" ) {
|
||||
# In a unordered list block, print a new item
|
||||
print "<li>" substr($0, 3) "</li>"
|
||||
print "{{article}}lt;li{{article}}gt;" substr($0, 3) "{{article}}lt;/li{{article}}gt;"
|
||||
} else {
|
||||
# Otherwise, init the unordered list block
|
||||
push("ul")
|
||||
print "<ul>
|
||||
<li>" substr($0, 3) "</li>"
|
||||
print "{{article}}lt;ul{{article}}gt;
|
||||
{{article}}lt;li{{article}}gt;" substr($0, 3) "{{article}}lt;/li{{article}}gt;"
|
||||
}
|
||||
}
|
||||
</code>
|
||||
</pre>
|
||||
<p>I believe the code is pretty self explanatory, but when the last environement is not `ul`, then we enter this environement.</p>
|
||||
<p>I believe the code is pretty self explanatory, but when the last environement is not <code>ul</code>, then we enter this environement.</p>
|
||||
<p>This translates as pushing it to the stack.</p>
|
||||
<p>Otherwise, it means we are already reading a list, and we only need to add a new element to it.</p>
|
||||
<h2>Parsing the simple paragraph and ending the parser</h2>
|
||||
@ -120,84 +121,85 @@ function last() {
|
||||
<p>it does not start with a specific caracter. That is, to match it, we match everything that is not a special character.</p>
|
||||
<p>I have no idea if this is the best solution, but so far it proved to work:</p>
|
||||
<pre><code># Matching a simple paragraph
|
||||
!/^(#|\*|-|\+|>|`|$| | )/ {
|
||||
!/^(#|*|-|+|>|`|$| | )/ {
|
||||
env = last()
|
||||
if (env == "none") {
|
||||
# If no block, print a paragraph
|
||||
print "<p>" $0 "</p>"
|
||||
print "{{article}}lt;p{{article}}gt;" $0 "{{article}}lt;/p{{article}}gt;"
|
||||
} else if (env == "blockquote") {
|
||||
print $0
|
||||
}
|
||||
}
|
||||
</code>
|
||||
</pre>
|
||||
<p>AS `BEGIN`, AWK provide the possibilty to execute code at the very end of the file, with the `END` keyword.</p>
|
||||
<p>AS <code>BEGIN<code>, AWK provide the possibilty to execute code at the very end of the file, with the </code>END</code> keyword.</p>
|
||||
<p>Naturally we need to empty the stack and close all html tags that might have been opened during the parsing.</p>
|
||||
<p>It only is a while loop, until the last environement is "none", as it way initiated : </p>
|
||||
<pre><code>END {
|
||||
env = last()
|
||||
while (env != "none") {
|
||||
env = pop()
|
||||
print "</" env ">"
|
||||
print "{{article}}lt;/" env "{{article}}gt;"
|
||||
env = last()
|
||||
}
|
||||
}
|
||||
</code>
|
||||
</pre>
|
||||
<p>This way we are able to simply parse markdown and turn it into an HTML file.</p>
|
||||
<p>Of course I am aware that is lacks emphasis, strong and code within a line of text. </p>
|
||||
<p>However I did implement it, but maybe it will be explained in another edit of this post.</p>
|
||||
<p>Nonetheless the code can still be consulted on <a href="https://github.com/SiwonP/bob">github</a>.</p>
|
||||
<h2>Parsing in-line fonctionnalities</h2>
|
||||
<p>For now we have seen a way to parse blocks, but markdown also handles strong, emphasis and links. However, these tags can appear anywhere in a line.</p>
|
||||
<p>Hence we need to be able to parse these lines apart from the block itself : indeed a header can container a strong and a link.</p>
|
||||
<p>A very useful function in awk is `match` : it literally is a regex engine, looking for a pattern in a string.</p>
|
||||
<p>The previously introduced but very useful function <code>match</code> fits this need : it literally is a regex engine, looking for a pattern in a string.</p>
|
||||
<p>Whenever the pattern is found, two global variables are filled :</p>
|
||||
<ul>
|
||||
<li>RSTART : the index of the first character matching the *group*</li>
|
||||
<li>RLENGTH: the length of the matched *group*</li>
|
||||
<li>RSTART : the index of the first character matching the <em>group</em></li>
|
||||
<li>RLENGTH: the length of the matched <em>group</em></li>
|
||||
</ul>
|
||||
<p>For the following, `line` represents the line processed by the function, as the following `while` loops are actually part of a single function.</p>
|
||||
<p>This way `match(line, /*([^*]+)*/)` matches a string surrounded by two `*`, corresponding to an emphasis text.</p>
|
||||
<p>The `<em>` are espaced are thez are special characters, and the </em>group* is inside the parenthesis.</p>
|
||||
<p>To matche several instances of emphasis text within a line, a simple `while` will do the trick.</p>
|
||||
<p>We now only have to insert html tags `<em>` are the right space around the matched text, and we are good to go.</p>
|
||||
<p>We can save the global variables `RSTART` and `RLENGTH` for further use, in case they were to be change. Using them we also can extract the </p>
|
||||
<p>For the following, <code>line<code> represents the line processed by the function, as the following </code>while</code> loops are actually part of a single function.</p>
|
||||
<p>This way <code>match(line, /*([^{{article}}#42;]+)*/)<code> matches a string (that does not start with a <code>{{article}}#42</code>) surrounded by two </code>{{article}}#42;</code>, corresponding to an emphasis text.</p>
|
||||
<p>The <code>{{article}}#42;</code> are espaced as they are special characters, and the <em>group</em> is delimited by the parenthesis.</p>
|
||||
<p>To match several instances of emphasis text within a line, a simple <code>while</code> will do the trick.</p>
|
||||
<p>We now only have to insert html tags <code>{{article}}lt;em{{article}}gt;</code> are the right space around the matched text, and we are good to go.</p>
|
||||
<p>We can save the global variables <code>RSTART<code> and </code>RLENGTH</code> for further use, in case they were to be change. Using them we also can extract the </p>
|
||||
<p>matched substrings and reconstruct the actual html string :</p>
|
||||
<pre><code>while (match(line, /*([^*]+)*/)) {
|
||||
<pre><code>while (match(line, /*([^{{article}}#42;]+)*/)) {
|
||||
start = RSTART
|
||||
end = RSTART + RLENGTH - 1
|
||||
# Build the result: before match, <em>, content, </em>, after match
|
||||
line = substr(line, 1, start-1) "<em>" substr(line, start+1, RLENGTH-2) "</em>" substr(line, end+1)
|
||||
# Build the result: before match, {{article}}lt;em{{article}}gt;, content, {{article}}lt;/em{{article}}gt;, after match
|
||||
line = substr(line, 1, start-1) "{{article}}lt;em{{article}}gt;" substr(line, start+1, RLENGTH-2) "{{article}}lt;/em{{article}}gt;" substr(line, end+1)
|
||||
}
|
||||
</code>
|
||||
</pre>
|
||||
<p>The while loop enables us to repeat this process as many times as this pattern is encountered within the line.</p>
|
||||
<p> </p>
|
||||
<p>We now can repeat the pattern for all inline fonctionnalities, e.g. strong and code.</p>
|
||||
<p>The case of url is a bit more deep as we need to match two groups : the actual text and the url itself.</p>
|
||||
<p>No real issue here, the naïve way is to match thd whole, and looking for both the link and the url within the matched whole.</p>
|
||||
<p>This way `match(line, /\[([^\]]+)\]\([^\)]+\)/)` matches a text between `[]` followed by a text between `()` : the markdown representation of links.</p>
|
||||
<p>As above, we store the `start` and `end` and also the whole match :</p>
|
||||
<p>No real issue here, the naïve way is to match the whole, and looking for both the link and the url within the matched whole.</p>
|
||||
<p>This way <code>match(line, /[([^]]+)]([^)]+)/)<code> matches a text between <code>[]</code> followed by a text between </code>()</code> : the markdown representation of links.</p>
|
||||
<p>As above, we store the <code>start<code> and </code>end</code> and also the whole match :</p>
|
||||
<p> </p>
|
||||
<pre><code>start = RSTART
|
||||
end = RSTART + RLENGTH - 1
|
||||
matched = substr($0, RSTART, RLENGTH)
|
||||
</code>
|
||||
</pre>
|
||||
<p>It is possible to apply the match fonction on this `matched` string, and extract, first, the text in `[]`, and last the text in `()`</p>
|
||||
<pre><code>if (match(matched, /\[([^\]]+)\]/)) {
|
||||
<p>It is possible to apply the match fonction on this <code>matched<code> string, and extract, first, the text in <code>[]</code>, and last the text in </code>()</code></p>
|
||||
<pre><code>if (match(matched, /[([^]]+)]/)) {
|
||||
matched_link = substr(matched, RSTART+1, RLENGTH-2)
|
||||
}
|
||||
if (match(matched, /\([^\)]+\)/)) {
|
||||
if (match(matched, /([^)]+)/)) {
|
||||
matched_url = substr(matched, RSTART+1, RLENGTH-2)
|
||||
}
|
||||
</code>
|
||||
</pre>
|
||||
<p>As the link text and the url are stored, using the variables `start` and `end`, it is easy to reconstruct the html line :</p>
|
||||
<pre><code>line = substr(line, 1, start-1) "<a href="" matched_url "">" matched_link "</a>" substr(line, end+1)
|
||||
<p>As the link text and the url are stored, using the variables <code>start<code> and </code>end</code>, it is easy to reconstruct the html line :</p>
|
||||
<pre><code>line = substr(line, 1, start-1) "{{article}}lt;a href="" matched_url ""{{article}}gt;" matched_link "{{article}}lt;/a{{article}}gt;" substr(line, end+1)
|
||||
</code>
|
||||
</pre>
|
||||
<p>The inline parsing function is now complete, all we have to do it apply is systematically on the text within html tags and this finished the markdown parser.</p>
|
||||
<p>This, of course, is the first brick of a static site generator, maybe the most complexe one. </p>
|
||||
<p>We shall see up next how to orchestrate this parser to make is a actual site generator.</p>
|
||||
<p>The code is available in the <a href="https://git.simonpetit.top/simonpetit/top">repo</a>.</p>
|
||||
</article>
|
||||
</body>
|
||||
|
||||
|
||||
75
posts/markdown_testing_suite.html
Normal file
75
posts/markdown_testing_suite.html
Normal file
@ -0,0 +1,75 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="fr" dir="ltr">
|
||||
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<title>simpet</title>
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover">
|
||||
<link href="https://fonts.googleapis.com/css?family=Cutive+Mono|IBM+Plex+Mono&display=swap" rel="stylesheet">
|
||||
<link rel="stylesheet" type="text/css" href="../css/poststyle.css">
|
||||
</head>
|
||||
|
||||
<body>
|
||||
<h1 class='title'><a href="../index.html">simpet</a></h1>
|
||||
<article>
|
||||
<h1>A test suite for markdown parser</h1>
|
||||
<p>As I implemented my own markdown parser for <a href="https://git.simonpetit.top/simonpetit/bob">bob</a>, my static site (blog) generator, I also wanted to make sure it was parsing markdown correctly.</p>
|
||||
<p>Hence I thought about a custom testing suite. After all this blog is also to make things from scratch to grasp a better understanding of how things work overall.</p>
|
||||
<h2>The concept </h2>
|
||||
<p>Also this is a custom script I still wanted to make it somehow generic. In others words I wanted it to be used against any markdown parser (assuming it follows a certain input/ouput constraints).</p>
|
||||
<p>For example, if the test script is <code>test_md_parser<code>, the parser </code>parser</code>, then actually testing the parser shall be the command :</p>
|
||||
<pre><code>test_md_parser parser
|
||||
</code>
|
||||
</pre>
|
||||
<p>The one condition is that <code>parser</code> take the markdown string as a standard input, and print the rendered html in the standard output.</p>
|
||||
<p>This way the markdown parser can feed custom markdown, of which it known the outputed html, and directly compare it with the output of the parser.</p>
|
||||
<p>One more thing is, my markdown parser is written as an <code>awk<code> script, but some may be </code>bash</code> scripts or even executables. This means I need to add an argument to precise the interpreter (if needed).</p>
|
||||
<p>In my case this would look like this :</p>
|
||||
<pre><code>test_md_parser parser.awk awk
|
||||
</code>
|
||||
</pre>
|
||||
<p>and for a bash script, it will be as such :</p>
|
||||
<pre><code>test_md_parser parser.sh bash
|
||||
</code>
|
||||
</pre>
|
||||
<h2>Unit testing</h2>
|
||||
<p>The purpose of the testing suite is to confront an expect output with the actual outputs from typical markdown syntax.</p>
|
||||
<p>I started by making an array of size <code>3n<code>, </code>n</code> being the number of tests. Indeed for display purposes each test has </p>
|
||||
<ul>
|
||||
<li>a title : quickly defining what kind of syntax is being tested</li>
|
||||
<li>a markdown input: a legal markdown syntax text</li>
|
||||
<li>an expected output: the corresponding html output</li>
|
||||
</ul>
|
||||
<p>This approach has flaws, obviously, and the biggest one being the consistence of html. Indeed this html :</p>
|
||||
<pre><code>{{article}}lt;h1{{article}}gt;Title{{article}}lt;/h1{{article}}gt;
|
||||
</code>
|
||||
</pre>
|
||||
<p>is strictly equivalent to :</p>
|
||||
<pre><code>{{article}}lt;h1{{article}}gt;
|
||||
Title
|
||||
{{article}}lt;/h1{{article}}gt;
|
||||
</code>
|
||||
</pre>
|
||||
<p>whereas the strings are not equal.</p>
|
||||
<p>The most naive approach I came with (and because I wanted a quick prototype so I didn't think much about it) was to remove all carriage return from the parser output, using <code>tr -d '{{article}}#92;n'</code>.</p>
|
||||
<p>The best solution would be to implement an html minimizer, and apply this minimization on the output of the parser (and maybe the expected result as well) to ensure perfect equality no matter how many carriage return and trailing spaces there would be. Most likely this could be done in a further version.</p>
|
||||
<p>All tests are hard coded within the script. I am aware this might not be the best solution, but on the other hand as this is a script, and not a compiled program, it is as easy as changing "hardcoded" tests as it would be on a separate config file, so its does not bother me right now.</p>
|
||||
<h2>Implementation of the testing suite</h2>
|
||||
<p>As mentionned earlier, all tests are defined in a array of size <code>3n</code> as such : </p>
|
||||
<pre><code>
|
||||
declare -a tests=(
|
||||
"Test header"
|
||||
"# Header1"
|
||||
"{{article}}lt;h1{{article}}gt;Header1{{article}}lt;/h1{{article}}gt;é
|
||||
)
|
||||
</code>
|
||||
</pre>
|
||||
<p>Running a single test would take an <code>input<code>, run the parser against it, store the <code>output<code> and compare it to the <code>expected</code>. It would return </code>0</code> for success and </code>1</code> in case of failure.</p>
|
||||
<p>Looping over the whole array introduced above, with the <code>input<code> being all <code>3n+1<code> elements of the array and </code>expected</code> being all </code>3n+2</code> ensures all tests are executed in order and as they should.</p>
|
||||
<p>To know whether or not all tests were successful, I simply make the sum of all the returned status of the <code>run_test()<code> function : if the sum equals </code>0</code>, logically all tests are ok.</p>
|
||||
<p>However I added a nice console output that prints all tests as they are being executed, which also prints in green succesful tests, and in red failed tests.</p>
|
||||
<p>This very simple testing suite would be part of the <a href="https://git.simonpetit.top/simonpetit/bob">bob</a> blog generator.</p>
|
||||
</article>
|
||||
</body>
|
||||
|
||||
</html>
|
||||
Loading…
Reference in New Issue
Block a user