Changelog for hasklepias

0.20.0

• Adds the Hasklepias.CohortCollection module which is exposed as the collector application. This application can be used to combine cohorts that were derived from different input data (e.g. different partitions of data). However, cohorts must be derived from the same specification using the same shape (e.g. rowWise or colWise). The application can be installed from the asclepias repository using cabal install. It is also available to download from download.novisci.com/hasklepias/collector-0.20.0-linux.tar.gz (currently only available for linux (not sure of architecture)). The following gives an example of using the application on local files:

$ cat collector-test/tests/manifestrw.txt 
testrw1.json
testrw2.json
testrw3.json
$ collector -f collector-test/tests/manifestrw.txt -d collector-test/tests
{"example":[{"totalProcessed":7,"attritionInfo":[{"attritionCount":2,"attritionLevel":{"contents":[1,"dummy"],"tag":"ExcludedBy"}},{"attritionCount":5,"attritionLevel":{"tag":"Included"}}]},{"contents":{"rowAttributes":[{"name":"myVar1","attrs":{"getPurpose":{"getTags":[],"getRole":["Outcome"]},"getDerivation":"","getLongLabel":"another label","getShortLabel":"somelabel"},"type":"Count"},{"name":"myVar2","attrs":{"getPurpose":{"getTags":[],"getRole":[]},"getDerivation":"","getLongLabel":"","getShortLabel":""},"type":"Bool"}],"rowData":[["a",[5,true]],["b",[5,true]],["c",[10,false]],["d",[99,true]],["f",[86,true]]]},"tag":"RW"}]}

0.19.0

• Overhauls the way that cohorts are written to JSON, mostly in the Cohort.Output module. The important bit is that intermediate types were added that can hold both row-wise and column-wise cohort data as list of Values (Data.Aeson internal representation of JSON). These intermediate types were made Semigroup instances, which means that cohorts can be combined. Note that you should only combine cohorts (and set of cohorts) evaluated from the same set of cohort specifications. Since all types of the the underlying data are masked by Value, you technically can combine any values of these intermediate types, but the results would be generally be nonsensical. Use this feature of combining cohorts to (e.g.) collect the same cohort data processed on different partitions of population data. While users should not need to worry about these details, here is the semigroup behavior for two cohorts (i.e. cohort1 <> cohort2):
  • In both row-wise and column-wise formats, attrition information is added as one would expect.
  • In both row-wise and column-wise formats, feature attributes are kept from the first cohort (again, you should only combine two compatible cohorts).
  • In row-wise format, row data is stacked.
  • In column-wise format, data is stacked by column.
  • If you try to combine a row-wise cohort with a column-wise cohort, the cohort in the second position is dropped altogether.
• Adds a CohortSet type (which is technically not a Set but a Map) as a container for Cohorts, in this way each cohort can be named. Correspondingly, adds a CohortSetSpec type which is a Map of CohortSpecs.
• Adds the constructor function makeCohortSpecs for easily making a CohortSetSpec from a list.
• Adds the evalCohortSet function which evaluates a CohortSetSpec from Population into a CohortSet.
• Moves the F type synonym for Feature to the Feature module.
• Removes any the usage of Data.List.NonEmpty.fromList in Cohort.Output which was throwing an error when the input data is empty.
• Adds the empty file exampleData/emptyData.jsonl for testing the app on empty data.

0.18.2

• Mostly internal work to clean up AttributionInfo values. Now counts for all the criteria are included in the information, including 0 counts. Also added a total processed field. Adds a Semigroup instance for AttributionInfo so that attrition information can be later combined -- this will be useful, for example, when combining the same cohort across partitions.
• Adds FromJSON instance for AttributionInfo (and its components), so that attrition info can be read back into a Haskell program.

0.18.1

• Fixes incorrect type on buildNofConceptsBinaryConcurBaseline which returned Bool instead of Binary.

0.18.0

• Refactors the Features.Compose module a bit. Adds constructors for composing Definition. For example, D1C :: (a2 -> a1 -> a) -> Definition (F n1 b -> F n02 a2) -> Definition (F n1 b -> F n01 a1) -> Definition (F n1 b -> F n0 a ). Such constructors allow one to build definitions from other definitions, provided the input types are the same. Note: there is not a single interface to these constructors like is provided by the define and defineA functions. The poorly designed Eval typeclass is now replaced with a single eval function which simply pattern matches on the various shapes of Definitions. I fully expect this module to get another refactor in the future, but it is shaping up OK.
• Examples are updated according to the changes define/eval.
• Adds the buildNofXOrNofYWithGapBool template, which can be used for the common feature or 2-outpatient events with at least some gap between them or 1 inpatient event.

0.17.1

• Fixes silly mistake with git.

0.17.0

• Adds the Cohort.AssessmentIntervals modules, which provides types and safe constructors for intervals during which features can be evaluated. The module currently provides the BaselineInterval type, with constructors baseline and baselineBefore. These two constructors guarantee that the resulting BaselineInterval will meet or precede (respectively) the provided Index. Use baseline if you want a BaselineInterval that ends at the beginning of the Index; use baselineBefore if you need space between the end of the baseline interval and Index. Similarly, there is a FollowupInterval type, with constructors followup, followupMetBy, and followupAfter. Note that the followup function always returns a FollowupInterval such that end index < end (followup duration index) for any provided duration. The baseline and followup functions were not named with their associated relation to Index (meets and startedBy, resp.), since they are most likely the most common use case. The AssessmentInterval type is a sum type with (currently) two variants: one containing a BaselineInterval and the other a FollowupInterval. The following functions create AssesmentmentIntervals using the corresponding function:
  • makeBaselineFromIndex: baseline
  • makeBaselineBeforeIndex: baselineBefore
  • makeFollowupFromIndex: followup
  • makeFollowupMetByIndex: followupMetBy
  • makeFollowupAfterIndex: followupAfter
• Modifies the continuous enrollment template to take a function Index i a -> AssessmentInterval a as an argument to enforce that functions that create valid assessment interval are used. Updates ExampleCohort1 accordingly.
• Updates interval-algebra dependency to 0.10.2, and updates functions as needed.
• Adds the EventData.Predicate module which exposes Predicates on Events. For example, isEnrollmentEvent has the type Predicate (Event a) (so getPredicate isEnrollmentEvent :: (Event a -> Bool)). This module also includes two utilities for composing Predicates: (&&&) and (|||) for conjunction and disjunction respectively. For example, running isEnrollmentEvent ||| isBirthYearEvent would return True if an event either has the Enrollment domain or has the Demographic domain with BirthYear as its field. You can also form Predicates on the interval part of an Event. Something like Predicate (\x -> before index x) &&& isBirthYearEvent is a predicate that returns True when an event is before index and the event contains a BirthYear fact
• Adds the EventData.Accessors module and moves functions such as viewBirthYears from Hasklepias.FeatureEvents to this new module.
• Adds an initial framework for feature definition builders (i.e templates for functions that define features). These are found in the Hasklepias.Templates.Features module. Adds a basic set of builders including:
  • buildIsEnrolled: identifies whether there is an event concurring with index
  • buildContinuousEnrollment: identifies whether a set of events meets continuous enrollment criteria given an allowable gap between enrollment intervals
  • buildNofX: a template that finds whether there are N events of some predicate X
  • buildNofXBool: buildNofX specialized to return Bool
  • buildNofXBinary: buildNofX specialized to return Binary
  • and several more

0.16.2

• Reexports ToJSON typeclass so users can export data as needed.

0.16.1

• Updates FromJSON instance for Domain, so that a JSON event with "domain" = "Enrollment" is deserialized into Event whose Domain is Enrollment.

0.16.0

• Adds a basic framework for Feature definition templates. Initially, this includes two templates for enrollment related features:
  • defIsEnrolled is a definition that maps an Index and a container of Events to a Status (i.e. Include or Exclude). This template takes no arguments.
  • defContinuousEnrollment is a definition that maps an Index, a container of Events, and a Status to a Status. This template takes two arguments: a function that creates the interval from the index during which enrollment is assessed and an allowable gap between any enrollment intervals. The input Status is used so that continuous enrollment may depend on other statuses. For example, you may want to have continuous enrollment depend on being enrolled.
• Adds a framework for testing the templates. E.g. templates can be tested using cabal test templates.
• Updates the ExampleCohort1 to use the defIsEnrolled and defContinuousEnrollment templates.
• Makes the Index i a type an instance of Intervallic, so you can use methods like begin, end, and interval algebra functions directly on an Index without having to unpack the interval first.
• Adds stripped down Enrollment Domain. This does type is not faithful to the EDM, as it does not include the event data model's plan fact.
• Adds the isEnrollment predicate function for identifying Domains that are Enrollment. This can be used with filterByDomain to filter a container of Events to those that are enrollment events.

0.15.2

• Updates viewBirthYears utility to filter a list of events to those with BirthYear demographic facts. In this way, one doesn't need to prefilter the input list by, e.g., a concept.
• Adds viewStates and viewGenders utilities for extracting a list of Demographic States and Gender (resp.) values from a collection of events. Note that these functions (like viewBirthYears) return a List, as the source data may contain be 0 or more values for a given subject. You probably only want one value for a given demographic, so you may need a function like headMay if you want the first element of the list (if it exists). Note too that the API of these accessor functions for facts in a Context need a careful design review and may be changed in the future.
• Adds yearFromDay, monthFromDay, and dayOfMonthFromDay utilities to get the year, month, and day of month, respectively from a Day.

0.15.1

• Tinkers with package version dependencies in .cabal file.

0.15.0

• Adds Occurrence type which is a simply a pair a reason and an EventTime. That is, an Occurrence captures what and when something occurred. Adds the CensoredOccurrence type which is similar to an Occurrence, except that the reason is of type CensoringReason cr or, where data CensoringReason cr or = AdminCensor | C cr | O or. The time of a CensoredOccurrence is a MaybeCensored (EventTime a) (not simply can EventTime). See the examples in examples/ExampleFeatures4 for usage.
• A number of the utility functions in the Hasklepias.FeatureEvents module are generalized to operate on data structures other than lists of Events.
• Exports type synonyms F n a and Def d for Feature n a and Definiton d to save a bit of typing.
• Adds Hasklepias.Misc module as a location to collect miscellaneous types and functions for the time-being, until better locations are found or created.
• Adds examples/ExampleFeatures4.hs which is an extensive example of assessing exposure protocols and censored outcomes.
• Moves the Cohort module to the top-level. Moves the FeatureEvents module within the Hasklepias module.

0.14.0

• Adds the ability to modify the output shape of cohorts. The makeCohortApp now takes a "shape" function as an argument of type Cohort d -> CohortShape shape. Currently, two functions of this type are provided: rowWise and colWise. The rowWise functions presents the output feature data in a row-wise format where each subject's data is its own array; whereas colWise presents the feature data where all the data of a given feature are in a single array.
• Updates interval-algebra dependency to 0.9.0. Updates code accordingly.
• Reorganizes modules to simplify imports.

0.13.2

• Adds Binary Stype data type.
• Adds ToJSON instance for most Stype types.
• Adds Featureset module and type, which is simply a list of Featureables. This is just a step towards being able to define the shape of JSON output.

0.13.1

• Adds the Featureable type, which allows users to put features into a heterogenous list. A drawback of Featureable is that two Featurable values cannot be tested for equality, so testing will need to occur before a Feature n d is packed into a Featureable (by the packFeature function) or after the Featureable is encoded to JSON. See examples/ExampleCohort1.hs for example usage.
• Adds a "type" field to JSON output Features, which is a string representing the type d in a Feature n d. E.g., for a Feature "x" Bool, the result would be : {..., "type": "Bool", ...}.
• Adds the Role and Purpose types, which now are included as part of the Attributes type. The Roles mostly align with stype's valid_roles, with the exception that "identifier", "index" and "censoring" are not included and "other" corresponds to Unspecified.

0.13.0

• Adds a rudimentary Attributes type and HasAttributes typeclass for adding attributes to Features. This interfaces will likely change in the future, but for now users have the ability to add information like labels to a Feature. In fact, Features which are encoded to JSON are required to have attributes.
• Adds rudimentary Stype module in order to (ultimately) interface with the R stype package. For now, this module simply creates a few of the types, some of which are not an appropriate implementation. For example, the Nominal type is simply newtype Nominal a = Nominal a. Essentially, it's just a way to label something as nominal at this point.

0.12.0

• Converts AttritionStatus from a List to a NonEmpty container.
• Adds the MakeApp module with a single function makeCohortApp which takes a list of cohort specifications and returns an application (an IO () function). Currently, the application is bare bones, printing the resulting cohorts (one per specification) to stdout and any parsing errors to stderr. For example usage, see the code in exampleApp.

0.11.1

• Modifies a Context so that its _facts are no longer Maybe Domain and now just Domain.

0.11.0

• Refactors the FeatureCompose module. FeatureSpec and FeatureDefinition types are dropped, and now there is a single Definition type with two related typeclasses: Define (with function define) and DefineA (with function defineA). Both of these typeclasses can lift functions to functions of either Features or FeatureData. For example define can take a function c -> b -> a, and, depending on the type annotation gives back a definition Definition (FeatureData c -> FeatureData b -> FeatureData a) or Definition (Feature name2 c -> Feature name1 b -> Feature name0 a). The defineA works similarly for a function of type c -> b -> f a, where f is either FeatureData or Feature. The eval function takes any Definition and an appropriate argument to give back the desired return type. For example, to evaluate def of type Definition (FeatureData c -> FeatureData b -> FeatureData a), you would call eval def (x, y), where (x, y) :: FeatureData c, FeatureData b). At this time, one can define Definitions with up to 3 inputs.
• For now, the Attributes component of Features is dropped. This will be rethought and added back in at later time.

0.10.0

• Refactors FeatureSpecs and Features to have a Symbol as its name, rather than Text.
• Updates parseEventLines and related functions to keep parse errors. Similarly for parseSubjectLines which keeps the error message as well as the line number.

0.9.0

• Refactors cohort building types and functions. Still aways to go, but the basic ideas are there now.
• Adds the ExampleCohort1 module to demonstrate the updates on calendar based cohorts.
• Adds several new functions and modules to Reexports including the Test.Tasty and Test.Tasty.HUnit testing modules for testing cohort building.

0.8.3

• Modifies FromJSON instance for events to use parseEvent as well as create a moment from the provided begin in the case that end is missing.
• Updates FromJSON instance for events to parse the facts object. Currently the only implemented domain is demographics.
• Adds FromJSON instances for Population, so now data from multiple subjects can be marshaled into Hasklepias programs.

0.8.2

• Adds preliminary Subject, Population, ObsUnit, and Cohort types to the Hasklepias module, along with the makeObsUnitFeatures which takes a function that maps a Subject into a ObsUnit. Currently, this function only supports 1-1 mapping between subjects and observational units. There is also the makeCohort function which maps a Population to a Cohort. The types get preliminary FromJSON (for Subject and Population) and ToJSON (for ObsUnit and Cohort) instances as well. Example 1 demonstrates use of these two functions.

0.8.1

• Adds an initial pass at the Domain type; simply including the Demographics type for now.

0.8.0

• Reorganizes modules into a more "vertical" structure that reflects the decoupling of the various components that make up hasklepias:
  • EventData: types and functions related to the event data model
  • FeatureCompose: types and functions for composing new features
  • FeatureEvents: various utilities for composing features from events specifically
  • Hasklepias: at this point, just reexporting the above modules and other Haskell functions

0.7.3

• Refactors the eval* function for Features so that there is a single eval not two.

0.7.2

• Adds Criteria module which provides specialized functions and types for working with boolean Features which are meant to be used to determine whether a subject is included or excluded from cohorts.

0.7.1

• Fixes bug in FeatureData Monad instance, so you don't get infinite recursion.

0.7.0

• Modifies the way that Features are defined and evaluated. For one, the dependency between Events and Features is eliminated, thus decoupling the defining of Features from the input data type. There are now 4 functions for defining features: define0, define1, define2, and define2d, and 3 functions for evaluating eval0, eval1, and eval2 (and corresponding evalSpec*). These functions will hopefully be combined into a single interface at later time. See the function signatures for the types of functions passed to a define* function. Examples will be forthcoming as the API stabilizes.

0.6.1

• Adds derving Eq to Feature type.

0.6.0

• Adds PolyKinds extension to Feature module to enable poly-kind inputs to FeatureDefinitions. Adds a related Defineable typeclass with define and eval functions as a common interface for defining new definitions and evaluating them.
• Removes defineEF and applyEF function (and other similar functions). The functionality is now handled by the Defineable class.

0.5.1

• Adds Show, Functor, and Generic to Reexports.
• Updates interval-algebra to 0.8.2.

0.5.0

• Changes what was the Feature type into FeatureData. The Feature type becomes a container for FeatureData with a name and attributes.
• Adds the FeatureSpec type which contains FeatureDefinitions plus a name and attributes. The name and attributes are mapped directly into the resulting Feature when a FeatureSpec is evaluated, while the FeatureDefinition is evaluated into FeatureData. The evalEFFeature, evalFEFFeature, and evalFFFFeature are provided for evaluating a FeatureSpec according the corresponding FeatureDefinition.
• Adds additional functions to reexports.
• Adds witherable dependency to use a more general filter function.

0.4.4

• Adds the FFF option to FeatureDefinition to define (Feature f -> Feature e -> Feature d) along with corresponding defineFFF and applyFFF.
• Adds zipWith, id, and Integer to re-exports.

0.4.3

• Exports Feature constructor.
• Adds defineFEF2 function for creating a feature definition where the provided function returns a Feature d rather than just a d.
• Generalizes allPairs from type [a] -> [a] -> [(a, a)] to [a] -> [b] -> [(a, b)].
• Reexports a few functions and types from Data.Time.Calendar. Also reexports const from Data.Function.

0.4.2

• Updates interval-algebra to 0.8.0.

0.4.1

• Modifies the example in example/ExampleFeatures3 to use the pipe |> operator.
• Adds the hasAllConcepts function to the HasConcepts class.
• Adds a Reexports module with the goal to re-export everything one might need from other Haskell libraries to build a cohort.
• Removes a number of unneeded/unused functions from the Functions module.
• Adds the Safe language extension to modules where possible.

0.4.0

• Adds the FeatureDefinition to represent common patterns for building Features:

data FeatureDefinition e a d =
    EF  (Events a -> Feature d)
  | FEF (Feature e -> Events a -> Feature d)

• Provides an initial set of functions designed to make defining Features easier, namely defineEF and defineFEF. These functions construct FeatureDefinitions of using EF and FEF constructors, respectively. The example features in examples/ExampleFeatures1 demonstrate their use.
• Adds the allPairs function to form all pairs of elements of two lists.
• Adds the splitByConcepts to split a container of events into a pair such that first element contains events have any of the first argument's concepts, and similarly for the second element.
• Demonstrates how allPairs and splitByConcepts might be used in the examples/ExampleFeatures3 module.
• Adds a rudimentary ToJSON instance for Features so that data can be encoded and output from the software. This is pretty rough; e.g. encoding an Interval Int feature produces: "{\"end\":10,\"begin\":0}".
• Removes the Transformations module and transformToMeetingSequence function. The same functionality is available by using the formMeetingSequence function from interval-algebra. See examples/ExampleFeatures2 for the updated example.
• Adds the toConceptEventOf function which creates a ConceptEvent but takes the intersection of Concepts in the first argument and concepts in the context of the Event in the second argument to form the new ConceptEvent. This is a way to keep only those concepts you need in the event.

0.3.0

• Updates code as needed to work with interval-algebra v0.6.2. In particular, the Event a is now a synonym for PairedInterval Context a, hence any methods that work on the PairedInterval also work for the Event type.
• Adds the ConceptEvent a type which is a synonym for PairedInterval Concept a; i.e, this is an event without facts or a source.
• Adds the toConceptEvent function for dropping from an Event a to a ConceptEvent a, and mkConceptEvent function for directly making a ConceptEvent from concepts and an interval.
• Adds generators for lists of arbitrary events. The generator for Concepts is limited at this point; it simply takes a subsample of the first 10 letters of the alphabet. Currently, only generators for Event Int are provided by the generateEventsInt. For example, in the repl generateEventsInt 2 produces two randomly generated events:

*Hasklepias> generateEventsInt 2
[{(-33, -16), Context {getConcepts = fromList ["G","I"], getFacts = Nothing, getSource = Nothing}},{(12, 13), Context {getConcepts = fromList ["A","C","D","E","G","I"], getFacts = Nothing, getSource = Nothing}}]

• Adds the transformToMeetingSequence function which takes a set of concepts and a list of possibly non-disjoint ConceptEventss and returns a list of ConceptEvents, where each consecutive event meets the next. Moreover, only those concepts selected (in the first argument) are kept in the output list of events. In the case that none of the events have the chosen concepts during an interval, an ConceptEvent with an empty set of concept is returned. A few examples might make this more clear.

*Hasklepias> :set -XOverloadedStrings
*Hasklepias> x <- fmap (map toConceptEvent) (generateEventsInt 1)
*Hasklepias> x
[{(3, 4), fromList ["B","C"]}]
*Hasklepias> transformToMeetingSequence (map packConcept ["A"]) x
[{(3, 4), fromList []}]
*Hasklepias> transformToMeetingSequence (map packConcept ["B"]) x
[{(3, 4), fromList ["B"]}]
*Hasklepias> x <- fmap (map toConceptEvent) (generateEventsInt 10)
*Hasklepias> x
[{(-44, 7), fromList ["C","D","E","F","H","J"]},{(-30, -29), fromList ["A","B","F","G","H","I","J"]},{(-25, 5), fromList ["C","D","E","I"]},{(-20, -19), fromList ["A","C","E","G","I","J"]},{(-17, -16), fromList ["B","D","F","J"]},{(-6, -5), fromList ["E","F","H","J"]},{(2, 21), fromList ["A","F","J"]},{(18, 19), fromList ["D","F","G","H","I"]},{(19, 20), fromList ["B","C","D","E","F","H"]},{(30, 31), fromList ["B","C","D","H","J"]}]
*Hasklepias> transformToMeetingSequence (map packConcept ["B", "I"]) x
[{(-44, -30), fromList []},{(-30, -29), fromList ["B","I"]},{(-29, -25), fromList []},{(-25, -17), fromList ["I"]},{(-17, -16), fromList ["B","I"]},{(-16, 5), fromList ["I"]},{(5, 18), fromList []},{(18, 19), fromList ["I"]},{(19, 20), fromList ["B"]},{(20, 30), fromList []},{(30, 31), fromList ["B"]}]

• Adds an example of transformToMeetingSequence could be used to derive a feature that is the list of durations that a subject was both hospitalized and on antibiotics at the same time in the examples/ExampleFeatures2 module.