↖ contents

5 — Reading and writing data

Scientific data arrives as text files with conventions, and the conventions are facts about the FILE, not properties a reader should guess. M9's CSV reader refuses to infer: the caller declares each column's kind — real, integer, timestamp with a named layout, text, or skip — and declares the file's missing-value convention. For comparison: handed a real flux-network file, a fast modern dataframe library scanned the first thousand rows, decided a column was integer, and stopped a hundred kilobytes in when -2.03 arrived. The column was never integer; the file just opens with round numbers. Inference is a bet about the rows you have not read yet.

The data for this chapter is a half-hour temperature series with one gap, marked -9999 — the file's own convention, stated in its documentation, declared by the caller:

TIMESTAMP,TA
202512010000,3.2
202512010030,3.0
202512010100,2.7
202512010130,-9999
202512010200,2.1
202512010230,1.8
202512010300,1.6
202512010330,1.7

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83

expected output:

rows 8, valid 7
window 2025-12-01T00:00:00Z .. 2025-12-01T03:30:00Z
mean TA 2.300 degC

Walk the pipeline:

• opt.missing := -9999.0 — the declared gap value becomes NaN on parse. NaN is the honest in-band representation for "no measurement": floats carry it, every statistic refuses it (chapter 6), and the one comparison it satisfies — x # x, since NaN is the only value not equal to itself — makes the valid-data filter a single visible line.
• Csv.SetStamp (t, 0, Csv.StampYmdHm) — the timestamp layout is named, not sniffed, and parses into Time.Instant: seconds since the Unix epoch, UTC. A time axis that is secretly local time is an hour of silent error for every user who assumed otherwise; time zones are facts, so the API makes them explicit.
• F64 (temp [i]) — the file's reals are F32 (they have seven significant digits at most); statistics run in F64. The widening is written, because chapter 2.
• Fmt.Fixed (pool, …, 3) — formatting allocates, so it takes the pool. Which brings us to the last idea.

The pool answers "who frees this?"

Every M9 allocation is carved from a named POOL, and a procedure that keeps storage beyond its own frame takes the pool as a parameter. The rule of thumb, measured across this repository's whole library: the pool parameter appears exactly when the allocation outlives the call. A signature with a pool is telling you the result lives on and who owns it; a signature without one is a promise that nothing was kept. This program declares one pool at the top; everything — the parsed table, the formatted strings — dies with the program, and there is no free() to forget and no garbage collector to wonder about.

On the writing side, Io deals in whole files (WriteFile, ReadFile): a partial-read API is an invitation to the truncation bugs this repository has already paid for, so the boundary is one call, checked.

← Previous: memory, pools and strings · Next: simple math and statistics →