Add README for internal data representation

Added first informal parser interface, only requiring a method to parse
events and one to parse trackers.

Theoretically we only *require* a method to parse events since, through
their contained activities, they would also come with trackers.
But this would 1) be much more opaque and a lot of work to then extract
the trackers again and 2) leave out trackers which do not yet have any
activities associated with them (i.e. trackers never once accomplished).

We can still turn the informal parser into a formal interface if need
arises: https://realpython.com/python-interface/

.gitignore

@@ -1,5 +1,5 @@
-data/
+/data/
-output.db
+/output.db
 
 # Created by https://www.toptal.com/developers/gitignore/api/vim,linux,python,pandas
 # Edit at https://www.toptal.com/developers/gitignore?templates=vim,linux,python,pandas

.woodpecker.yml

@@ -1,86 +0,0 @@
-branches: main
-
-pipeline:
-  code_lint:
-    image: python
-    commands:
-      - pip install poetry
-      - poetry install
-      - pip install black
-      - echo "----------------- running lint ------------------"
-      - python --version && poetry --version && black --version
-      - poetry run black .
-
-  unit_tests:
-    image: thekevjames/nox
-    commands:
-      - pip install poetry
-      - poetry install
-      - echo "----------------- running tests ------------------"
-      - python --version && poetry --version && nox --version
-      - poetry run nox
-
-  static_analysis:
-    image: python
-    commands:
-      - pip install poetry
-      - poetry install
-      - pip install mypy
-      - echo "----------------- running analysis ------------------"
-      - python --version && poetry --version && mypy --version
-      - poetry run mypy .
-
-  build_dist:
-    image: python
-    commands:
-      - pip install poetry
-      - poetry install
-      - echo "----------------- running analysis ------------------"
-      - python --version && poetry --version
-      - poetry build
-    when:
-      branch: main
-
-  release_prep:
-    image: python
-    commands:
-      - echo "----------------- preparing release ------------------"
-      - python tools/extract-changelog.py
-
-  gitea_release:
-    image: plugins/gitea-release
-    settings:
-      api_key: 
-        from_secret: gitea_release_token
-      base_url: https://git.martyoeh.me
-      files: dist/*
-      title: NEWEST_VERSION.md
-      note: NEWEST_CHANGES.md
-    when:
-      event: tag
-      tag: v*
-
-  pypi_release:
-    image: python
-    commands:
-      - pip install poetry
-      - poetry install
-      - echo "----------------- publishing to pypi ------------------"
-      - poetry publish --username "$PYPI_USERNAME" --password "$PYPI_PASSWORD"
-    when:
-      event: tag
-      tag: v*
-
-  notify_matrix:
-    image: plugins/matrix
-    settings:
-      homeserver: https://matrix.org
-      roomid:
-        from_secret: matrix_roomid
-      userid:
-        from_secret: matrix_userid
-      accesstoken:
-        from_secret: matrix_token
-    when:
-      status: [ success, failure ]
-

CHANGELOG.md

@@ -5,19 +5,13 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project tries to adhere to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-<!-- ## [Unreleased] -->
+## [Unreleased]
-
-## [0.4.1] - 2022-01-05
-
-### Added
-
-* Added pypi release publication
 
 ### Changed
 
 * Compatible with Python stretching back to version 3.7
 
-## [0.4.0] - 2021-12-06
+## [0.4] - 2021-12-06
 
 ### Added 
 

README.md

@@ -1,34 +1,15 @@
-# habitmove
+# habit-migrate
 
-Takes habit in one habit-tracking application and transforms them ready to use for another.
+Can take an export of [nomie](https://nomie.app/) habits in json format and convert it to be importable in [Loop Habit Tracker](https://loophabits.org/).
-
-Currently can take an export of nomie habits in json format and convert it to be importable in Loop Habit Tracker.
-Plans for reverse migration are on the roadmap, and ultimately this tool ideally understands more and more habit formats to prevent app lock-in.
 
 Confirmed working for nomie version 5.6.4 and Loop Habit Tracker version 2.0.2 and 2.0.3. 
 Presumably works for other nomie 5.x versions and other Loop 2.x versions as well, 
 but that is untested.
 
-## Installation
-
-Installation can be accomplished through *pip*:
-
-```bash
-pip install habitmove
-```
-
-Requirements:
-
-`habitmove` requires at least Python 3.7. 
-It has only been tested on GNU/Linux (amd64) though it should work on other platforms.
-
 ## Usage 
 
-Run as a cli utility `habitmove` currently takes a single argument: the nomie database `.json` file to import habits from.
+Run as a commandline utility habit migrate currently takes a single argument, the nomie database `.json` file.
-
+The output as importable Loop Habit Tracker database will be written to `output.db` in present working directory.
-Invoked like: `habitmove nomie-export.json`.
-
-The output as a Loop Habit Tracker database will be written to `output.db` in the present working directory.
 
 Can also take an existing Loop Habit database (exported from the application), 
 and add the nomie exported habits and checkmarks to it.
@@ -37,28 +18,36 @@ it will not (should not™️) overwrite anything.
 If there are any duplicated habits however, 
 it will add duplications of the existing repetitions into the database.
 
-## Development
+Invoked like: `python run.py nomie-export.json`.
+Note, however, that -- until a packaged version is released -- you will need to have some packages in your environment.
+If you wish to run it un-packaged, install [poetry](https://python-poetry.org/) and let it do all dependency management by doing:
 
-To enable easy development on the app, 
+```
-install [poetry](https://python-poetry.org/) and let it do all dependency management for you by doing:
-
-```bash
 poetry install
 poetry run habitmove <nomie-json>
 ```
 
-To see a set up more closely resembling the final cli environment, 
+In the future there might be an easier road to using this package but that's the way it is for now.
-with its libraries loaded as environmental dependencies enter the poetry shell:
 
-```bash
+The package can also be used as a library to load nomie data
-poetry shell 
+or move data into Loop Habit Tracker.
-```
 
-The package can eventually also be used as a library to load nomie data to work with in Python,
+## Internal representation
-or to move data into Loop Habit Tracker.
-Take a look at the `Parser` and `Transformer` interfaces respectively.
 
-To run tests for the app, simply invoke `pytest` through `poetry run pytest` or from within the `poetry shell`.
+Internally, the data gets represented within three concepts: 
-To run larger scale test automation, make sure you habe nox installed and run `poetry run ` or again through the shell.
+Events, Activities and Trackers.
 
-You can exclude integration tests that take longer and inspect the complete database output of the program through the parameters `-m "not e2e"` for both `pytest` and `nox` (which also does it automatically).
+Events are simple entries or logs of, basically, anything and represent *qualitative* data. 
+At their most basic, they only describe 'something' at a certain point in time.
+For that, they have to contain a time and they may contain prose text (i.e. an arbitrary text string).
+Additionally, an event can contain a list of one or more activities.
+
+Activities are the changes to whatever is tracked *quantitatively*.
+They always belong to an event and thus the moment in time the event took place.
+They might even be the only interesting thing that took place in the event,
+but not necessarily.
+Lastly, they contain a single tracker which they belong to.
+
+Trackers are the meta-data of whatever is being tracked quantitatively through activities.
+They define a name, label, scores, descriptions, reminders and so on.
+All data being imported is transformed into this model and output from it again.

noxfile.py

@@ -3,7 +3,7 @@ import nox
 
 @nox.session(python=["3.7", "3.8", "3.9"])
 def tests(session):
-    args = session.posargs or ["--cov"]
+    args = session.posargs or ["--cov", "-m", "not e2e"]
     session.run("poetry", "install", external=True)
     session.run("pytest", *args)
     pass

pyproject.toml

@@ -1,10 +1,7 @@
 [tool.poetry]
 name = "habitmove"
-version = "0.4.1"
+version = "0.4.0"
 description = "migrate nomie data to loop habits tracker"
-license="GPL-3.0-only"
-readme="README.md"
-repository="https://git.martyoeh.me/Marty/habit-migrate"
 authors = ["Marty Oehme <marty.oehme@gmail.com>"]
 packages = [
     { include = "habitmove", from = "src"},
@@ -38,13 +35,3 @@ show_missing = true
 [build-system]
 requires = ["poetry-core>=1.0.0"]
 build-backend = "poetry.core.masonry.api"
-
-[[tool.mypy.overrides]]
-module = [
-    "click",
-    "click.testing",
-    "pytest",
-    "nox",
-    "importlib-metadata"
-]
-ignore_missing_imports = true

src/habitmove/__init__.py

@@ -4,6 +4,6 @@ import sys
 try:
     from importlib.metadata import version as metadata_version
 except ImportError:
-    from importlib_metadata import version as metadata_version  # type: ignore
+    from importlib_metadata import version as metadata_version
 
 __version__ = str(metadata_version(__name__))

src/habitmove/cli.py

@@ -3,13 +3,13 @@ import habitmove.schema as schema
 import habitmove.habits as habits
 import habitmove.repetitions as rep
 import habitmove.nomie as nomie
-from habitmove.nomiedata import NomieImport
+from habitmove.nomiedata import ImportData
 
 import click
 from . import __version__
 
 
-def migrate(data: NomieImport):
+def migrate(data: ImportData):
     db = schema.migrate("output.db")
     if not db:
         raise ConnectionError
@@ -29,7 +29,7 @@ def migrate(data: NomieImport):
 @click.version_option(version=__version__)
 @click.argument("inputfile")
 def main(inputfile):
-    data = nomie.get_data(inputfile)
+    data = nomie.get_data(inputfile, False)
     migrate(data)
 
 

src/habitmove/nomie.py

@@ -3,7 +3,7 @@
 import json
 import re
 from click import secho, echo
-from habitmove.nomiedata import Tracker, Event, Activity, NomieImport
+from habitmove.nomiedata import Tracker, Event, Activity, ImportData
 
 
 def load_file(filename):
@@ -26,7 +26,7 @@ def confirmation_question(question, default_no=True):
 
 
 # display stats and ask user to confirm if they seem okay
-def verify_continue(data: NomieImport):
+def verify_continue(data: ImportData):
     trackers = ""
     for t in data.trackers:
         trackers += t.label + ", "
@@ -94,14 +94,14 @@ def get_activities_for_event(event_text, tracker_list):
 
 
 # return the data belonging to nomie
-def get_data(file, interactive=True):
+def get_data(file: str, interactive: bool = True):
     raw_data = load_file(file)
     nomie_version = raw_data["nomie"]["number"]
 
     tracker_list = get_trackers(raw_data["trackers"])
     event_list = get_events(raw_data["events"], tracker_list)
 
-    data = NomieImport(nomie_version, tracker_list, event_list)
+    data = ImportData(nomie_version, tracker_list, event_list)
     if interactive:
         verify_continue(data)
 

src/habitmove/nomie_parser.py

@@ -0,0 +1,13 @@
+from __future__ import annotations
+from json import loads as jsonloads
+
+from habitmove.parser import Parser
+
+
+class NomieParser(Parser):
+    def __init__(self, data="{}") -> None:
+        """Load a data set and prepare parser data"""
+        self.data = jsonloads(data)
+
+    def extract_version(self) -> str:
+        return self.data["nomie"]["number"]

src/habitmove/nomiedata.py

@@ -47,7 +47,7 @@ class Event:
     id: str
     start: int
     end: int
-    text: str
+    text: str = ""
     activities: list[Activity] = field(default_factory=lambda: [])
     score: int = 0
     lat: float = 0.0
@@ -59,7 +59,7 @@ class Event:
 
 
 @dataclass(frozen=True)
-class NomieImport:
+class ImportData:
     version: str
     trackers: list[Tracker]
     events: list[Event]

src/habitmove/parser.py

@@ -0,0 +1,32 @@
+from __future__ import annotations
+from pathlib import Path
+
+from habitmove.nomiedata import Event, ImportData, Tracker
+
+
+class Parser:
+    def __init__(self, data="") -> None:
+        """Load a data set and prepare parser data"""
+        self.data = data
+
+    @classmethod
+    def from_file(cls, path: str) -> Parser:
+        """Load in a data set"""
+        txt = Path(path).read_text()
+        return cls(data=txt)
+
+    def parse(self) -> ImportData:
+        """Extract all data from a data set"""
+        raise NotImplementedError
+
+    def extract_version(self) -> str:
+        """Extract import dataset version from the data set"""
+        raise NotImplementedError
+
+    def extract_trackers(self) -> list[Tracker]:
+        """Extract trackers from the data set"""
+        raise NotImplementedError
+
+    def extract_events(self) -> list[Event]:
+        """Extract events from the data set"""
+        raise NotImplementedError

src/habitmove/repetitions.py

@@ -55,10 +55,9 @@ def habit_list_add_ids(c: sqlite3.Cursor, habitlist: list[Habit]) -> dict[int, H
     :return habit_id_dict: The habit collection as a dict with the keys
                                  consisting of the habit's sqlite database ID.
     """
-    with_id: dict[int, Habit] = {}
+    with_id = {}
     for h in habitlist:
         sql_id = fetch_habit_id(c, h.uuid or "")
-        if sql_id is not None:
         with_id[sql_id] = h
 
     return with_id
@@ -75,8 +74,6 @@ def fetch_habit_id(cursor: sqlite3.Cursor, uuid: str) -> Optional[int]:
     if id is not None:
         return id[0]
 
-    return None
-
 
 def add_to_database(
     cursor: sqlite3.Cursor, habits: dict[int, Habit], repetition: Repetition

tests/conftest.py

@@ -0,0 +1,2 @@
+def pytest_configure(config):
+    config.addinivalue_line("markers", "e2e: mark as end to end test.")

tests/test_cli.py

@@ -1,15 +1,46 @@
-import click.testing
+from click.testing import CliRunner
 import pytest
 
+# for integration tests
+from pathlib import Path
+from shutil import copyfile
+from subprocess import run
+
 from habitmove import cli
 
 
 @pytest.fixture
 def runner():
-    return click.testing.CliRunner()
+    return CliRunner()
+
+
+# Create an isolated environment to test the output file in
+@pytest.fixture
+def runner_with_nomie_input(tmp_path):
+    runner = CliRunner()
+    fname_input_data = Path("tests/data/nomie/input.json").resolve()
+    fname_target_data = Path("tests/data/loop/output.db").resolve()
+    with runner.isolated_filesystem(temp_dir=tmp_path):
+        copyfile(fname_input_data.resolve(), f"input.json")
+        copyfile(fname_target_data.resolve(), f"target")
+        yield runner
 
 
 def test_cli_fails_without_file(runner):
     result = runner.invoke(cli.main)
     assert result.exit_code == 2
     assert "Missing argument" in result.output
+
+
+@pytest.mark.e2e
+def test_produces_output_file(runner_with_nomie_input):
+    result = runner_with_nomie_input.invoke(cli.main, "input.json")
+    assert result.exit_code == 0
+    assert Path("output.db").exists()
+
+
+@pytest.mark.e2e
+def test_produces_correct_output(runner_with_nomie_input):
+    runner_with_nomie_input.invoke(cli.main, "input.json")
+    result = run(["sqldiff", "output.db", "target"], capture_output=True)
+    assert result.stdout == b""

tests/test_nomie_parser.py

@@ -0,0 +1,28 @@
+from habitmove.nomie_parser import NomieParser
+import json
+import pytest
+
+
+@pytest.fixture
+def sample_data():
+    return '{ "nomie": { "number": "5.6.4", "created": "2021-08-26T08:15:36.898Z", "startDate": "2021-08-26T08:15:36.898Z", "endDate": "2021-08-26T08:15:36.898Z" }}'
+
+
+def test_nomie_parser_exists():
+    sut = NomieParser()
+    assert type(sut) == NomieParser
+
+
+def test_nomie_parser_errors_on_invalid_data():
+    with pytest.raises(json.decoder.JSONDecodeError):
+        NomieParser(data="invalid_test_data")
+
+
+def test_nomie_parser_saves_data():
+    sut = NomieParser(data='{"test": "entry"}')
+    assert sut.data == {"test": "entry"}
+
+
+def test_nomie_parser_extracts_version(sample_data):
+    sut = NomieParser(data=sample_data)
+    assert sut.extract_version() == "5.6.4"

tests/test_parser.py

@@ -0,0 +1,41 @@
+from habitmove.parser import Parser
+from inspect import signature
+
+
+def test_parser_interface_exists():
+    sut = Parser()
+    assert type(sut) == Parser
+
+
+def test_parser_interface_contains_methods():
+    sut = Parser()
+    assert sut.__getattribute__("parse") != None
+    assert sut.__getattribute__("from_file") != None
+    assert sut.__getattribute__("extract_version") != None
+    assert sut.__getattribute__("extract_trackers") != None
+    assert sut.__getattribute__("extract_events") != None
+
+
+def test_parser_from_file_returns_parser():
+    sut = Parser().from_file
+    assert signature(sut).return_annotation == "Parser"
+
+
+def test_parser_parse_returns_Import_Data():
+    sut = Parser().parse
+    assert signature(sut).return_annotation == "ImportData"
+
+
+def test_parser_version_returns_String():
+    sut = Parser().extract_version
+    assert signature(sut).return_annotation == "str"
+
+
+def test_parser_extract_trackers_returns_tracker_list():
+    sut = Parser().extract_trackers
+    assert signature(sut).return_annotation == "list[Tracker]"
+
+
+def test_parser_extract_events_returns_event_list():
+    sut = Parser().extract_events
+    assert signature(sut).return_annotation == "list[Event]"

tools/extract-changelog.py

@@ -1,56 +0,0 @@
-import re
-
-## Extracts the version and newest changes from a semantic changelog.
-#
-# Important, it only works with three-parted version numbers
-# a-la 1.2.3 or 313.01.1888 -- needs \d.\d.\d to work.
-#
-# The version number and changeset will be put in `NEWEST_VERSION.md`
-# and `NEWEST_CHANGES.md` respectively, for further use in releases.
-OUTPUT_FILE_VERSION = "NEWEST_VERSION.md"
-OUTPUT_FILE_CHANGES = "NEWEST_CHANGES.md"
-
-
-def getVersion(file):
-    for line in file:
-        m = re.match(r"^## \[(\d+\.\d+\.\d+)\]", line)
-        if m and m.group(1):
-            return m.group(1)
-
-
-def getSection(file):
-    inRecordingMode = False
-    for line in file:
-        if not inRecordingMode:
-            if re.match(r"^## \[\d+\.\d+\.\d+\]", line):
-                inRecordingMode = True
-        elif re.match(r"^## \[\d+\.\d+\.\d+\]", line):
-            inRecordingMode = False
-            break
-        elif re.match(r"^$", line):
-            pass
-        else:
-            yield line
-
-
-def toFile(fname, content):
-    file = open(fname, "w")
-    file.write(content)
-    file.close()
-
-
-with open("CHANGELOG.md") as file:
-    title = getVersion(file)
-    print(title)
-    toFile(OUTPUT_FILE_VERSION, title)
-
-with open("CHANGELOG.md") as file:
-    newest_changes_gen = getSection(file)
-    newest_changes = ""
-    for line in newest_changes_gen:
-        newest_changes += line
-    print("[Extracted Changelog]")
-    print(newest_changes)
-    toFile(OUTPUT_FILE_CHANGES, newest_changes)
-
-file.close()