↖ contents

8 — Data through zarr

Large scientific arrays live in chunked stores: the array is cut into fixed-size chunks, each compressed and addressable by its coordinates, with small JSON metadata describing shape, dtype and fill. Zarr is the convention the ICOS Carbon Portal serves its data with, over plain HTTPS — and ZarrStore is a complete reader for it: metadata parsed and validated at open, blosc decompression, a chunk cache, and every index checked against the shape the store itself declared.

The store in this chapter is a local replica of the repository's test store, and the sandbox that runs these examples online has no network on purpose -- so the replica is served *inside* the sandbox, on its own private loopback, started beside your cell and gone when it exits. Real chunked data, zero egress, and a tutorial that cannot fail with the portal's next maintenance window. Run locally, serve the store yourself: python3 -m http.server 18931 --directory <dir-holding-co2.zarr>. Against a live portal the only change is the URL — the TLS client underneath verifies certificate and hostname, proven against the standard set of deliberately broken hosts.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73

expected output:

shape 100 x 50
co2[0,0]   = 394.9816047539
co2[99,49] = 403.8924951332
co2[10,5]  is a gap (deleted chunk, NaN fill)

Three things are new here, and each is the language keeping a promise from an earlier chapter:

• The gap is DATA. Chunk (2,1) of this store was deleted; zarr's rule is that a missing chunk reads as the declared fill value, NaN here. The read succeeds, the gap arrives as NaN, and chapter 5's x = x test spots it.
• Ownership is in the signature. Open answers a SHARED PTR Store — a counted handle — and Close (OWN s) CONSUMES it: after the Close line, any further use of st is a compile error naming what killed it and where. The use-after-close bug class is not tested away here; it is uncompilable.
• The foreign boundary is declared. Blosc and the socket layer are C libraries, bound in FOR "C" units where every procedure states its C name and its thread discipline ([SERIAL] / [REENTRANT]) — the compiler emits the serialisation, so a global-state C library cannot be raced by accident. The honest consequence: linking this program names those libraries. With the store served locally (python3 -m http.server 18931 --directory /tmp/m9stores), the build is:

  m9c --make -c C8Zarr.m9
  gcc C8Zarr.o ZarrStore.o Json.o Http.o DynStr.o Io.o Fmt.o \
      m9rt.c tcpshim.c tlsshim.c -l:libblosc.so.1 \
      -lssl -lcrypto -lm -o co2

m9c is deliberately not a build system; what it DOES supply by default (include paths, the module closure, the runtime) it will show you with -v.

← Previous: timeseries · Next: preparing data for plotting →