Zettelstore: Change Log

Changes for Version 0.2.0 (pending)

Remove support for ;;small text;; Zettelmarkup. (breaking: Zettelmarkup)
Selecting zettel based on their metadata allows the same syntax as searching for zettel content. For example, you can list all zettel that have an identifier not ending with 00 by using the query id=!<00. (minor: api, webui)
Show unlinked references in info page of each zettel. Unlinked references are phrases within zettel content that might reference another zettel with the same title as the phase. (major: webui)
Add endpoint /u/{ID} to retrieve unlinked references. (major: api)
Remove support for //deprecated emphasized// Zettelmarkup. (minor: Zettelmarkup)
Add options to profile the software. Profiling can be enabled at the command line or via the administrator console. (minor)
Add computed zettel that lists all supported parser / recognized zettel syntaxes. (minor)

Changes for Version 0.1 (2021-11-11)

v0.1.2 (2021-11-18) fixes a bug when selecting zettel from a list when more than one comparison is negated.
v0.1.1 (2021-11-12) updates the documentation, mostly related to the deprecation of the // markup.
Remove visual Zettelmarkup (italic, underline). Semantic Zettelmarkup (emphasize, insert) is still allowed, but got a different syntax. The new syntax for inserted text is >>inserted>>, while its previous syntax now denotes emphasized text: __emphasized__. The previous syntax for emphasized text is now deprecated: //deprecated emphasized//. Starting with Version 0.2.0, the deprecated syntax will not be supported. The reason is the collision with URLs that also contain the characters //. The ZMK encoding of a zettel may help with the transition (/v/{ZettelID}?_part=zettel&_enc=zmk, on the Info page of each zettel in the WebUI). Additionally, all deprecated uses of // will be rendered with a dashed box within the WebUI. (breaking: Zettelmarkup).
API client software is now a separate project. (breaking)
Initial support for HTTP security headers (Content-Security-Policy, Permissions-Policy, Referrer-Policy, X-Content-Type-Options, X-Frame-Options). Header values are currently some constant values. (possibly breaking: api, webui)
Remove visual Zettelmarkup (bold, striketrough). Semantic Zettelmarkup (strong, delete) is still allowed and replaces the visual elements syntactically. The visual appearance should not change (depends on your changes / additions to CSS zettel). (possibly breaking: Zettelmarkup).
Add API endpoint POST /v to retrieve HTMl and text encoded strings from given ZettelMarkup encoded values. This will be used to render a HTML page from a given zettel: in many cases the title of a zettel must be treated separately. (minor: api)
Add API endpoint /m to retrieve only the metadata of a zettel. (minor: api)
New metadata value content-tags contains the tags that were given in the zettel content. To put it simply, all-tags = tags + content-tags. (minor)
Calculating the context of a zettel stops at the home zettel. (minor: api, webui)
When renaming or deleting a zettel, a warning will be given, if other zettel references the given zettel, or when “deleting” will uncover zettel in overlay box. (minor: webui)
Fix: do not allow control characters in JSON-based creating/updating API. Otherwise, the created / updated zettel might not be parseable by the software (but still by a human). In certain cases, even the WebUI might be affected. (minor: api, webui)
Fix: when a very long word (longer than width of browser window) is given, still allow to scroll horizontally. (minor: webui)
Separate repository for contributed software. First entry is a software for creating a presentation by using zettel. (info)
Many smaller bug fixes and inprovements, to the software and to the documentation.

Changes for Version 0.0.15 (2021-09-17)

Move again endpoint characters for authentication to make room for future features. WebUI authentication moves from /a to /i (login) and /i?logout (logout). API authentication moves from /v to /a. JSON-based basic zettel handling moves from /z to /j and /z/{ID} to /j/{ID}. Since the API client is updated too, this should not be a breaking change for most users. (minor: api, webui; possibly breaking)
Add API endpoint /v/{ID} to retrieve an evaluated zettel in various encodings. Mostly replaces endpoint /z/{ID} for other encodings except “json” and “raw”. Endpoint /j/{ID} now only returns JSON data, endpoint /z/{ID} is used to retrieve plain zettel data (previously called “raw”). See documentation for details. (major: api; breaking)
Metadata values of type tag set (the metadata with key tags is its most prominent example), are now compared in a case-insensitive manner. Tags that only differ in upper / lower case character are now treated identical. This might break your workflow, if you depend on case-sensitive comparison of tag values. Tag values are translated to their lower case equivalent before comparing them and when you edit a zettel through Zettelstore. If you just modify the zettel files, your tag values remain unchanged. (major; breaking)
Endpoint /z/{ID} allows the same methods as endpoint /j/{ID}: GET retrieves zettel (see above), PUT updates a zettel, DELETE deletes a zettel, MOVE renames a zettel. In addtion, POST /z will create a new zettel. When zettel data must be given, the format is plain text, with metadata separated from content by an empty line. See documentation for more details. (major: api (plus WebUI for some details))
Allows to transclude / expand the content of another zettel into a target zettel when the zettel is rendered. By using the syntax of embedding an image (which is some kind of expansion too), the first top-level paragraph of a zettel may be transcluded into the target zettel. Endless recursion is checked, as well as a possible “transclusion bomb ” (similar to a XML bomb). See manual for details. (major: zettelmarkup)
The endpoint /z allows to list zettel in a simpler format than endpoint /j: one line per zettel, and only zettel identifier plus zettel title. (minor: api)
Folgezettel are now displayed with full title at the bottom of a page. (minor: webui)
Add API endpoint /p/{ID} to retrieve a parsed, but not evaluated zettel in various encodings. (minor: api)
Fix: do not list a shadowed zettel that matches the select criteria. (minor)
Many smaller bug fixes and inprovements, to the software and to the documentation.

Changes for Version 0.0.14 (2021-07-23)

Rename “place” into “box”. This also affects the configuration keys to specify boxes box-uriX (previously place-uri-X. Older changes documented here are renamed too. (breaking)
Add API for creating, updating, renaming, and deleting zettel. (major: api)
Initial API client for Go. (major: api)
Remove support for paging of WebUI list. Runtime configuration key list-page-size is removed. If you still specify it, it will be ignored. (major: webui)
Use endpoint /v for user authentication via API. Endpoint /a is now used for the web user interface only. Similar, endpoint /y (“zettel context”) is renamed to /x. (minor, possibly breaking)
Type of used-defined metadata is determined by suffix of key: -number, -url, -zid will result the values to be interpreted as a number, an URL, or a zettel identifier. (minor, but possibly breaking if you already used a metadata key with above suffixes, but as a string type)
New user-role “creator”, which is only allowed to create new zettel (except user zettel). This role may only read and update public zettel or its own user zettel. Added to support future client software (e.g. on a mobile device) that automatically creates new zettel but, in case of a password loss, should not allow to read existing zettel. (minor, possibly breaking, because new zettel template zettel must always prepend the string new- before metdata keys that should be transferred to the new zettel)
New suported metadata key box-number, which gives an indication from which box the zettel was loaded. (minor)
New supported syntax html. (minor)
New predefined zettel “User CSS” that can be used to redefine some predefined CSS (without modifying the base CSS zettel). (minor: webui)
When a user moves a zettel file with additional characters into the box directory, these characters are preserved when zettel is updated. (bug)
The phase “filtering a zettel list” is more precise “selecting zettel” (documentation)
Many smaller bug fixes and inprovements, to the software and to the documentation.

Changes for Version 0.0.13 (2021-06-01)

Startup configuration box-X-uri (where X is a number greater than zero) has been renamed to box-uri-X. (breaking)
Web server processes startup configuration url-prefix. There is no need for stripping the prefix by a front-end web server any more. (breaking: webui, api)
Administrator console (only optional accessible locally). Enable it only on systems with a single user or with trusted users. It is disabled by default. (major: core)
Remove visibility value “simple-expert” introduced in version 0.0.8. It was too complicated, esp. authorization. There was a name collision with the “simple” directory box sub-type. (major)
For security reasons, HTML blocks are not encoded as HTML if they contain certain snippets, such as <script or <iframe. These may be caused by using CommonMark as a zettel syntax. (major)
Full-text search can be a prefix search or a search for equal words, in addition to the search whether a word just contains word of the search term. (minor: api, webui)
Full-text search for URLs, with above additional operators. (minor: api, webui)
Add system zettel about license, contributors, and dependencies (and their license). For a nicer layout of zettel identifier, the zettel about environment values and about runtime metrics got new zettel identifier. This affects only user that referenced those zettel. (minor)
Local images that cannot be read (not found or no access rights) are substituted with the new default image, a spinning emoji. See /file?name=box/constbox/emoji_spin.gif. (minor: webui)
Add zettelmarkup syntax for a table row that should be ignored: |%. This allows to paste output of the administrator console into a zettel. (minor: zmk)
Many smaller bug fixes and inprovements, to the software and to the documentation.

Changes for Version 0.0.12 (2021-04-16)

Raise the per-process limit of open files on macOS to 1.048.576. This allows most macOS users to use at least 500.000 zettel. That should be enough for the near future. (major)
Mitigate the shortcomings of the macOS version by introducing types of directory boxes. The original directory box type is now called "notify" (the default value). There is a new type called "simple". This new type does not notify Zettelstore when some of the underlying Zettel files change. (major)
Add new startup configuration default-dir-box-type, which gives the default value for specifying a directory box type. The default value is “notify”. On macOS, the default value may be changed “simple” if some errors occur while raising the per-process limit of open files. (minor)

Changes for Version 0.0.11 (2021-04-05)

New box schema "file" allows to read zettel from a ZIP file. A zettel collection can now be packaged and distributed easier. (major: server)
Non-restricted search is a full-text search. The search string will be normalized according to Unicode NFKD. Every character that is not a letter or a number will be ignored for the search. It is sufficient if the words to be searched are part of words inside a zettel, both content and metadata. (major: api, webui)
A zettel can be excluded from being indexed (and excluded from being found in a search) if it contains the metadata no-index: true. (minor: api, webui)
Menu bar is shown when displaying error messages. (minor: webui)
When selecting zettel, it can be specified that a given value should not match. Previously, only the whole select criteria could be negated (which is still possible). (minor: api, webui)
You can select a zettel by specifying that specific metadata keys must (or must not) be present. (minor: api, webui)
Context of a zettel (introduced in version 0.0.10) does not take tags into account any more. Using some tags for determining the context resulted into erratic, non-deterministic context lists. (minor: api, webui)
Selecting zettel depending on tag values can be both by comparing only the prefix or the whole string. If a search value begins with '#', only zettel with the exact tag will be returned. Otherwise a zettel will be returned if the search string just matches the prefix of only one of its tags. (minor: api, webui)
Many smaller bug fixes and inprovements, to the software and to the documentation.

A note for users of macOS: in the current release and with macOS's default values, a zettel directory must not contain more than approx. 250 files. There are three options to mitigate this limitation temporarily:

You update the per-process limit of open files on macOS.
You setup a virtualization environment to run Zettelstore on Linux or Windows.
You wait for version 0.0.12 which addresses this issue.

Changes for Version 0.0.10 (2021-02-26)

Menu item “Home” now redirects to a home zettel. Its default identifier is 000100000000. The identifier can be changed with configuration key home-zettel, which supersedes key start. The default home zettel contains some welcoming information for the new user. (major: webui)
Show context of a zettel by following all backward and/or forward reference up to a defined depth and list the resulting zettel. Additionally, some zettel with similar tags as the initial zettel are also taken into account. (major: api, webui)
A zettel that references other zettel within first-level list items, can act as a “table of contents” zettel. The API endpoint /o/{ID} allows to retrieve the referenced zettel in the same order as they occur in the zettel. (major: api)
The zettel “New Menu” with identifier 00000000090000 contains a list of all zettel that should act as a template for new zettel. They are listed in the WebUIs ”New“ menu. This is an application of the previous item. It supersedes the usage of a role new-template introduced in version 0.0.6. Please update your zettel if you make use of the now deprecated feature. (major: webui)
A reference that starts with two slash characters (“//”) it will be interpreted relative to the value of url-prefix. For example, if url-prefix has the value /manual/, the reference [[Zettel list|//h]] will render as <a href="/manual/h">Zettel list</a>. (minor: syntax)
Searching/selecting ignores the leading '#' character of tags. (minor: api, webui)
When result of selecting or searching is presented, the query is written as the page heading. (minor: webui)
A reference to a zettel that contains a URL fragment, will now be processed by the indexer. (bug: server)
Runtime configuration key marker-external now defaults to “➚” (“➚”). It is more beautiful than the previous “↗︎” (“↗︎”), which also needed the additional “︎” to disable the conversion to an emoji on iPadOS. (minor: webui)
A pre-build binary for macOS ARM64 (also known as Apple silicon) is available. (minor: infrastructure)
Many smaller bug fixes and inprovements, to the software and to the documentation.

Changes for Version 0.0.9 (2021-01-29)

This is the first version that is managed by Fossil instead of GitHub. To access older versions, use the Git repository under zettelstore-github.

Server / API

(major) Support for property metadata. Metadata key published is the first example of such a property.
(major) A background activity (called indexer) continuously monitors zettel changes to establish the reverse direction of found internal links. This affects the new metadata keys precursor and folge. A user specifies the precursor of a zettel and the indexer computes the property metadata for Folgezettel. Metadata keys with type “Identifier” or “IdentifierSet” that have no inverse key (like precursor and folge with add to the key forward that also collects all internal links within the content. The computed inverse is backward, which provides all backlinks. The key back is computed as the value of backward, but without forward links. Therefore, back is something like the list of “smart backlinks”.
(minor) If Zettelstore is being stopped, an appropriate message is written in the console log.
(minor) New computed zettel with environmental data, the list of supported meta data keys, and statistics about all configured zettel boxes. Some other computed zettel got a new identifier (to make room for other variant).
(minor) Remove zettel 00000000000004, which contained the Go version that produced the Zettelstore executable. It was too specific to the current implementation. This information is now included in zettel 00000000000006 (Zettelstore Environment Values).
(minor) Predefined templates for new zettel do not contain any value for attribute visibility any more.
(minor) Add a new metadata key type called “Zettelmarkup”. It is a non-empty string, that will be formatted with Zettelmarkup. title and default-title have this type.
(major) Rename zettel syntax “meta” to “none”. Please update the Zettelstore Runtime Configuration and all other zettel that previously used the value “meta”. Other zettel are typically user zettel, used for authentication. However, there is no real harm, if you do not update these zettel. In this case, the metadata is just not presented when rendered. Zettelstore will still work.
(minor) Login will take at least 500 milliseconds to mitigate login attacks. This affects both the API and the WebUI.
(minor) Add a sort option “_random” to produce a zettel list in random order. _order / order are now an aliases for the query parameters _sort / sort.

WebUI

(major) HTML template zettel for WebUI now use Mustache syntax instead of previously used Go template syntax. This allows these zettel to be used, even when there is another Zettelstore implementation, in another programming language. Mustache is available for approx. 48 programming languages, instead of only one for Go templates. If you modified your templates, you must adapt them to the new syntax. Otherwise the WebUI will not work.
(major) Show zettel identifier of folgezettel and precursor zettel in the header of a rendered zettel. If a zettel has real backlinks, they are shown at the botton of the page (“Additional links to this zettel”).
(minor) All property metadata, even computed metadata is shown in the info page of a zettel.
(minor) Rendering of metadata keys title and default-title in info page changed to a full HTML output for these Zettelmarkup encoded values.
(minor) Always show the zettel identifier on the zettel detail view. Previously, the identifier was not shown if the zettel was not editable.
(minor) Do not show computed metadata in edit forms anymore.

Changes for Version 0.0.8 (2020-12-23)

Server / API

(bug) Zettel files with extension .jpg and without metadata will get a syntax value “jpg”. The internal data structure got the same value internally, instead of “jpeg”. This has been fixed for all possible alternative syntax values.
(bug) If a file, e.g. an image file like 20201130190200.jpg, is added to the directory box, its metadata are just calculated from the information available. Updated metadata did not find its way into the zettel box, because the .meta file was not written.
(bug) If just the .meta file was deleted manually, the zettel was assumed to be missing. A workaround is to restart the software. If the .meta file is deleted, metadata is now calculated in the same way when the .meta file is non-existing at the start of the software.
(bug) A link to the current zettel, only using a fragment (e.g. [[Title|#title]]) is now handled correctly as a zettel link (and not as a link to external material).
(minor) Allow zettel to be marked as “read only”. This is done through the metadata key read-only.
(bug) When renaming a zettel, check all boxes for the new zettel identifier, not just the first one. Otherwise it will be possible to shadow a read-only zettel from a next box, effectively modifying it.
(minor) Add support for a configurable default value for metadata key visibility.
(bug) If list-page-size is set to a relatively small value and the authenticated user is not the owner, some zettel were not shown in the list of zettel or were not returned by the API.
(minor) Add support for new visibility “expert”. An owner becomes an expert, if the runtime configuration key expert-mode is set to true.
(major) Add support for computed zettel. These zettel have an identifier less than 0000000000100. Most of them are only visible, if expert-mode is enabled.
(bug) Fixes a memory leak that results in too many open files after approx. 125 reload operations.
(major) Predefined templates for new zettel got an explicit value for visibility: “login”. Please update these zettel if you modified them.
(major) Rename key readonly of Zettelstore Startup Configuration to read-only-mode. This was done to avoid some confusion with the the zettel metadata key read-only. Please adapt your startup configuration. Otherwise your Zettelstore will be accidentally writable.
(minor) References starting with “./” and “../” are treated as a local reference. Previously, only the prefix “/” was treated as a local reference.
(major) Metadata key modified will be set automatically to the current local time if a zettel is updated through Zettelstore. If you used that key previously for your own, you should rename it before you upgrade.
(minor) The new visibility value “simple-expert” ensures that many computed zettel are shown for new users. This is to enable them to send useful bug reports.
(minor) When a zettel is stored as a file, its identifier is additionally stored within the metadata. This helps for better robustness in case the file names were corrupted. In addition, there could be a tool that compares the identifier with the file name.

WebUI

(minor) Remove list of tags in “List Zettel” and search results. There was some feedback that the additional tags were not helpful.
(minor) Move zettel field "role" above "tags" and move "syntax" more to "content".
(minor) Rename zettel operation “clone” to “copy”.
(major) All predefined HTML templates have now a visibility value “expert”. If you want to see them as an non-expert owner, you must temporary enable expert-mode and change the visibility metadata value.
(minor) Initial support for Folgezettel. If you click on “Folge” (detail view or info view), a new zettel is created with a reference (precursor) to the original zettel. Title, role, tags, and syntax are copied from the original zettel.
(major) Most predefined zettel have a title prefix of “Zettelstore”.
(minor) If started in simple mode, e.g. via double click or without any command, some information for the new user is presented. In the terminal, there is a hint about opening the web browser and use a specific URL. A Welcome zettel is created, to give some more information. (This change also applies to the server itself, but it is more suited to the WebUI user.)

Changes for Version 0.0.7 (2020-11-24)

With this version, Zettelstore and this manual got a new license, the European Union Public Licence (EUPL), version 1.2 or later. Nothing else changed. If you want to stay with the old licenses (AGPLv3+, CC BY-SA 4.0), you are free to fork from the previous version.

Changes for Version 0.0.6 (2020-11-23)

Server

(major) Rename identifier of Zettelstore Runtime Configuration to 00000000000100 (previously 00000000000001). This is done to gain some free identifier with smaller number to be used internally. If you customized this zettel, please make sure to rename it to the new identifier.
(major) Rename the two essential metadata keys of a user zettel to credential and user-id. The previous values were cred and ident. If you enabled user authentication and added some user zettel, make sure to change them accordingly. Otherwise these users will not authenticated any more.
(minor) Rename the scheme of the box URL where predefined zettel are stored to “const”. The previous value was “globals”.

Zettelmarkup

(bug) Allow to specify a fragment in a reference to a zettel. Used to link to an internal position within a zettel. This applies to CommonMark too.

API

(bug) Encoding binary content in format “json” now results in valid JSON content.
(bug) All query parameters of selecting zettel must be true, regardless if a specific key occurs more than one or not.
(minor) Encode all inherited meta values in all formats except “raw”. A meta value is called inherited if there is a key starting with default- in the Zettelstore Runtime Configuration. Applies to WebUI also.
(minor) Automatic calculated identifier for headings (only for “html”, “djson”, “native” format and for the Web user interface). You can use this to provide a zettel reference that links to the heading, without specifying an explicit mark ([!mark]).
(major) Allow to retrieve all references of a given zettel.

Web user interface (WebUI)

(minor) Focus on the first text field on some forms (new zettel, edit zettel, rename zettel, login)
(major) Adapt all HTML templates to a simpler structure.
(bug) Rendered wrong URLs for internal links on info page.
(bug) If a zettel contains binary content it cannot be cloned. For such a zettel only the metadata can be changed.
(minor) Non-zettel references that neither have an URL scheme, user info, nor host name, are considered “local references” (in contrast to “zettel references” and “external references”). When a local reference is displayed as an URL on the WebUI, it will not opened in a new window/tab. They will receive a local marker, when encoded as “djson” or “native”. Local references are listed on the Info page of each zettel.
(minor) Change the default value for some visual sugar putd after an external URL to &\#8599;&\#xfe0e; (“↗︎”). This affects the former key icon-material of the Zettelstore Runtime Configuration, which is renamed to marker-external.
(major) Allow multiple zettel to act as templates for creating new zettel. All zettel with a role value “new-template” act as a template to create a new zettel. The WebUI menu item “New” changed to a drop-down list with all those zettel, ordered by their identifier. All metadata keys with the prefix new- will be translated to a new or updated keys/value without that prefix. You can use this mechanism to specify a role for the new zettel, or a different title. The title of the template zettel is used in the drop-down list. The initial template zettel “New Zettel” has now a different zettel identifier (now: 00000000091001, was: 00000000040001). Please update it, if you changed that zettel.
Note: this feature was superseded in version 0.0.10 by the “New Menu” zettel.
(minor) When a page should be opened in a new windows (e.g. for external references), the web browser is instructed to decouple the new page from the previous one for privacy and security reasons. In detail, the web browser is instructed to omit referrer information and to omit a JS object linking to the page that contained the external link.
(minor) If the value of the Zettelstore Runtime Configuration key list-page-size is greater than zero, the number of WebUI list elements will be restricted and it is possible to change to the next/previous page to list more elements.
(minor) Change CSS to enhance reading: make line-height a little smaller (previous: 1.6, now 1.4) and move list items to the left.

Changes for Version 0.0.5 (2020-10-22)

Application Programming Interface (API) to allow external software to retrieve zettel data from the Zettelstore.
Specify boxes, where zettel are stored, via an URL.
Add support for a custom footer.

Changes for Version 0.0.4 (2020-09-11)

Optional user authentication/authorization.
New sub-commands file (use Zettelstore as a command line filter), password (for authentication), and config.

Changes for Version 0.0.3 (2020-08-31)

Starting Zettelstore has been changed by introducing sub-commands. This change is also reflected on the server installation procedures.
Limitations on renaming zettel has been relaxed.

Changes for Version 0.0.2 (2020-08-28)

Configuration zettel now has ID 00000000000001 (previously: 00000000000000).
The zettel with ID 00000000000000 is no longer shown in any zettel list. If you changed the configuration zettel, you should rename it manually in its file directory.
Creating a new zettel is now done by cloning an existing zettel. To mimic the previous behaviour, a zettel with ID 00000000040001 is introduced. You can change it if you need a different template zettel.

Changes for Version 0.0.1 (2020-08-21)

Initial public release.