Foswiki Blog Foswiki and the JAMstack

On 13 June 2020 Ray Lutz asked a question on the Foswiki mailing list: JAMstack for foswiki? Is there a migration path? The discussion continued aboutJAMstack and PublishPlugin was suggested as an option.Not knowing what JAMstack was I read a few documents:Modern Web Development on the JAMstack (O'Reilly 2019) suggested by RayProvides lots of "why" and "when", but no "what" or "how". The article certainly peeked my interestAdding Dynamic And Async Functionality To JAMstack Sites (18 Dec2019)Googling for examples gave me the above reference, which I found enlightening. It also illustrated that to configure/build the JAMstack, you will write a lot of Javascript and css. Nothing wrong with that, but if you are not into either of these, you are left with writing the content only. No searching, no commenting, no formatting of search results.Netlify (Wikipedia 3 May 2020)This article states that "On March 4, 2020, Netlify secured $53 million". No wonder we are hearing more about Netlify and JAMstack.State of the Jamstack Survey 2020: First results (27 May 2020)This survey shows that JAMstack has a real audience for people building sites with many users. And "fantastic scaling, high performance, and workflows and tooling that developers love." are the drivers for its adoption.After reading these and other articles I thought I put my thoughts in some detail in a blog post. JAMstack has a place. It's first and foremost an idea that integrates the development of content with a workflow and delivery on the cloud platform. JAMstack provides the platform for integration of a lot of moving parts. And JAMstack requires developers to do the integration. It fits right in with the fashionable DevOps model extended to Content development. The idea is worth exploring further.My experienceI managed a document management system for 8 years. The system supported the company ISO 9000 standard accreditation with around 400 topics and about 200 documents. I used the Publish plugin, originally because I supported sites on CD that had no access to the internet. And later because it was easier to manage the publication of a static site than restrict the access for non-developers to "approved product" only. The site eventually died an unceremonious death when the company got bought out.Two lessons stand out.A static site is just that: Static. As soon as I published the first CD, there was an overwhelming response "I can't search!". Or for those that were smart enough to use grep or other file search tools: "Can't you get a better search?". That was resolved when I found a tool that would index a collection of files and provide a sensible web interface. After some fiddling, I managed to get it to work inside a topic, so it did not look too much out of place in the rest of the web site.Broken links. The first time I provided access to the published site via the internet, I got a huge response. Once you have a lot (several 100) users, you find they look in places that you have not visited for years. And find that some links you publish for further reading do no longer exist. Or you find that a topic has been refactored, but some references to that topic are now broken. So we started using a test regime. Before every publication we checked all links for existence.It should be clear from these lessons that we were well on our way to a primitive DevOps workflow. Once we decided the Foswiki site was ready for publication we ran the following steps:Publish the siteCheck the links... repeat until all errors fixedBuild the index for the searchBurn the CDShip the CD to the web serverUpgradeWe did this twice a year. The full process would take a day or two and it was pretty well fool proof. The result was also very well accepted. Most users were not interested in contributing to the text. We accepted update requests when people improved processes or started using new tools. It was not intended to be a very dynamic site.Foswiki and JAMstackIn my experience, Foswiki supported the notion of a static site well. But a static site does not function as a wiki. A wiki has a search facility available. A static site needs to be provided with one. A wiki allows one to add content. A static site does not. What if you want users to annotate pages? What if you want users to place orders? You do need access to a separate system that will store the content that is added. The JAMstack idea recognises this, but is also acutely aware that the dynamic use of a web site is a minor part of the content to be maintained.For my site above, I had to add one topic (page) that would reference an external component to search, using a prepared index against the static site. The comment plugin would need to change to accept a target in a Foswiki site. And if one succeeded to get that to work, one would need some clever tricks to display the updated target in the static frame. Not impossible, but not totally straight foreward either. Where Foswiki has a real advantage is that it has an infrastructure that could support this. For instance, The comments in the topic are placed within a START/ENDSECTION and the COMMENT macro with an extended target parameter will make sure the whole section is returned after the comment is inserted. Ahhh... In your dreams!As an alternative one could set up a separate comment server. And write some Javascript thatwill retrieve and display all relevant comments for the pagewill add comments to the comment server and refresh the content.That would be the JAMstack way. Foswiki would then package this code in a PageComment plugin and make it available as a PAGECOMMENT macro, so that the javascript, html and css would be included when the topic was rendered/published. Content developers can add comments to any topic. From what I read, Netlify could do this with a function and access it through an extension of Markdown.The same principle can be applied to any other dynamic content. Want to buy something? Use the ordering and payment server. Want to display data from a rain gauge? Extract the data from the metrics server. With some thought I expect Foswiki can support all these things. But should it?KatacodaI am in the process of writing some course material for Foswiki. The motivation is provided in the Feature proposal. Katacoda course development is using the JAMstack principles. Content is written in Markdown with extenstions. In addition to the content, one must provide a description of the course structure in JSON. The JSON groups and sequences the markdown. In addition one can make other components available as assets. All these parts are pushed to github. From there a hook takes care of the building of the website inside a docker container that is deployed to a cloud server. And then the course is available. You can see the result here ( WIP).The result is worth thinking about. There are three components:The Katacoda application container. This deals with the github hook, builds the web site in a container that is associated with my username.The course container. This is deployed to a cloud server when I log in and request the course. It provides Ubuntu.The Foswiki container (courtesy Tim Legge). This is deployed in the setup script of the course.The course itself uses the course content provided in Markdown. Links from the course will open Foswiki pages using the web server in the docker image or show script content using cat in the bash shell inside the Foswiki docker container. So you see the structure of services using other services to achieve an overall result: The Course.I started out creating the course structure and content using Markdown. I stored the components in git, pushing the result to github and waiting for the course to appear in my container. Usually that happened within 5 minutes, but sometimes there were longer delays. After a month I found that no longer satisfactory. I was learning Markdown, so there were many mistakes and the absence of an INCLUDE made for some very repetitive updates across multiple scenarios.And since I was using Foswiki at the same time, I was tempted to create the course content in Foswiki. Once I did, I could get a quicker turn around on the content edits,because I could see how the content rendered in the browser after save. When I was happy with the content I would paste it in a flat file and do global changes to translate the TML to Markdown. That too became tedious, so I scripted the translation. That was easy, because I had already restricted my TML to fit in the Katacoda Markdown flavour.Once that was done, a few steps in a makefile did the rest:translate the Foswiki topics to Markdown in the git structure with the correct filenamegit add *git commitwait for the site to be prepared and test.That was much faster overall, because the slow step (wait for the site to be prepared) operates on an (almost) perfect product.In my dreams I have a Foswiki application and plugin that will support these steps, so that all the construction is done in Foswiki. But that is in my dreams....Why not FoswikiThe survey referenced above asserts that developers like the JAMstack approach. If you are not using Foswiki regularly, it is yet another tool that you need to learn. If you spent your days using an IDE to support your activities, Foswiki is a distraction. It would be really nice if you could write the TML in your development environment, click preview and see it rendered in a tab. Save would actually save the text (as a revision) in the topic. But I can only dream.On the other hand, Content developers would find Foswiki a helpful environment. Links navigate. Topics are developed in context and the support of the extensive MACRO environment is of great benefit. Their rendered product could be saved in git and then pushed when deemed complete. Test could be hooked in and the whole DevOps approach would follow the JAMstack philosophy. So integration is possible. And Publish plugin is a first step.Tags: JAMstack