DiagrammeR::grViz('
digraph EventStream {
CD [label="Context Determination"];
EC [label="Evidence Capture"];
EF [label="Evidence Filtering", style="dashed"];
EI [label="Evidence Identification"];
EA [label="Evidence Accumulation"];
CD -> EC -> EF -> EI -> EA;
EA -> CD [style="dashed", constraint=false];
}')Event Stream Scoring System
Event stream is an open architecture for scoring assessments, particularly assessments based on games and simulations which generate complex event logs as a work product. Also, the assumption is that data will be collected over time, possibly from different client assessment systems and accumulated on a central server.
The Event Stream Model
(EvidenceSteam?) proposes a streaming model for scoring an assessment which uses the following structure:
Context Determination is the process by which the assessment activity (task) is chosen. This could be entirely scripted in advance, or it could be an adaptive process that changes depending on the evidence gathered by the rest of the system.
Evidence Capture: This is the collection of student work and logging of interactions with the system. It generally consists of a series of events with timestamps. The timestamps include the context under which the data was collected. It also contains some kind of data, which could be a work product.
Evidence Filtering: Not all of the captured log information is used to draw inferences about student proficiencies. This step reduces the size of the data stream by filtering out events which will not be processed.
Evidence Identification: Not all of the raw events in the evidence stream provide evidence. The goal of this step is to summarize the observable events within a given scoring epoch (task or part of a task) as a collection of observable variables.
Evidence Accumulation: Presumably the evidence is captured to make inferences about certain competencies the student may or may not possess. The evidence accumulation process takes the observables from each scoring epoch and updates the beliefs about where the student exists in the space of measured competencies.
The statistics summarizing the information about student performance are the most important output of this process. These may be passed back to the context determination process to make an adaptive system.
The Proc4 Message Schema
The Evidence Stream architecture is based on an older 4 process architecture of an idealized assessment system ((proc4?)). In particular, the Proc4 message format was developed for use in this system and is still applicable to the evidence stream architecture. Like an email message, it has a number of header fields which provide information about where the information comes from and what it is, and then the body of the message is an arbitrary data object, whose fields and expected values depends on the type of the message.
The current implement uses json (javascript object notation), although a previous incarnations of the Proc 4 system used a similar XML schema.
message = {
app: "ecd://producer.org/Assessment/Version:,
uid: "Student1",
context: "Task or scoring window",
sender: "Sending Process",
mess: "Message subject",
timestamp: "Date time of event",
processed: "Logical flag telling if message was processed.",
data: {
"field1":"value1",
"field2":"value2"
}
errorMessages: ["Exception during processing",
"Could be more than one"],
watermarks: {"Tag1":"timestamp1",
"Tag2":"timestamp2"},
}