Lesson 05 of 05 · published

Cameras: One Dataset, One Writer

~17 min · open-data, provenance, spatial-index, geometry, swift

Level 0Lost

0 XP0/36 lessons0/12 achievements

0/100 XP to next level100 XP to go0% complete

A public dataset, read by one tool

Speed-camera warnings are the one piece of road knowledge Pathfinder brings along itself, and it takes them from a public source: the Korean National Police Agency's standard dataset of fixed unmanned traffic-enforcement cameras (전국무인교통단속카메라표준데이터, dataset 15028200 on data.go.kr). The portal describes it as fixed cameras only, mobile units excluded, with the police supplying speed, signal, passage and other cameras and local governments supplying bus-lane and parking ones, refreshed about twice a year.

The app never calls that API. One executable in PathfinderKit, pathfinder-cameras, is the only writer of the camera file the app bundles. fetch pages through the API a thousand rows at a time with a key read from the environment or from cwkPippa's env file, writes the compact bundle, and prints a table; verify prints the same table from a saved raw pull. No key ships in the app. A refresh is one fetch and one new build.

Why only one writer

The dataset says what each camera enforces with a code, regltSe, and Pathfinder's meaning for those codes is inferred, not quoted. The full pull answered the question with its own numbers: codes 1 and 2 carry real limits (speed, and signal cameras that mostly enforce speed as well), code 4 is mostly limit 0 (parking), code 3 is limit 0 and stands at bus stops (bus lane), and code 99 carries expressway section cameras at 100-110. Bus lanes and parking are dropped as not driving-relevant.

An inference can go stale without making a sound. If the police renumber a code, a hand-edited JSON or a second import script would carry the old meaning forward silently. So the tool that writes the file is also the tool that re-checks the inference: every fetch prints the code-by-limit table, and a change of meaning shows up as a row that no longer looks like its label. One writer means one place where that check cannot be skipped.

The same writer cleans what the source mixes. Section lengths arrive in kilometres on most rows and in metres on some, so a value under 100 is read as kilometres (no Korean average-speed section is 100 km long, and none is under 100 m). Rows outside Korea's latitude and longitude box are dropped. Management numbers repeat between institutions and even within one, so a camera's id is the institution code plus the number, with a suffix when it repeats.

The pull that ships had 43,724 rows. 32,184 of them are driving-relevant and bundled, and 1,100 of those are section-enforcement cameras: 551 where a section starts and 549 where one ends.

The provenance the doc promises and the data does not carry

Invariant 8 says the bundled set names its source, its dataset modified date, when it was fetched and the range of per-row reference dates. The model agrees: CameraDataset has a datasetModified field. But the code that builds the bundle passes datasetModified: nil every time, and the encoder leaves a nil optional out, so the shipped header carries the source, the fetch time and the newest and oldest reference dates, and no dataset modified date at all. The portal page shows one; the tool never asks for it.

Teach the code, and keep the lesson it carries: a provenance field that its only writer always fills with nil is a promise in a document, not a fact in the data. The fix, when it comes, belongs in the writer, which is exactly why there is only one.

Which cameras are on this road

CameraIndex files the cameras in a grid of 0.01° cells, about 1.1 km north to south and 0.9 km east to west in Korea, so "what is near this road" never scans the country. It samples the route every 400 m, collects the cameras in each sample's cell and the eight around it, and projects each candidate only onto the stretch that runs from 1,000 m before the first sample that saw it to 1,000 m past the last. Projecting every candidate onto the whole line had cost tens of millions of segment tests on the main actor for a long route, every time a route was chosen or previewed.

A camera counts when it lands within 30 m of the route on the driving side. The dataset's direction code is not geometry, so a rule stands in for it: traffic in Korea keeps right, so a pole for this direction stands to the right of the line or overhead, and a camera more than 10 m to the left belongs to the opposite carriageway. The side is a signed cross-track distance, positive to the right of travel.

What she says, and where

CameraWatch warns in two modes. On a guided drive it places the cameras on the route once and warns by distance along it. In the free camera drive, with no route at all, it watches a 25° cone ahead of the car's course, at most 35 m off the course line, above 5 m/s. The first warning comes at 500 m, or at 800 m when the car is doing 19.4 m/s or more (about 70 km/h): "500미터 앞에 과속 단속 카메라 제한속도 60". Within 200 m, if the car is more than 3 km/h over the limit, an urgent "아빠, 속도 줄여" follows. Each camera is warned at most once per stage. Section cameras say where enforcement begins and ends: "구간 단속 시작" and "구간 단속 끝".

A walk hears no camera warnings, and a trip Google guides has none, because Google shows its own limits on its own map. Since #401 the free camera drive is Korea's alone. The button appears only when the phone's own fix is in South Korea, never by the map's centre or a simulated position; the resolved region is consulted only before the first fix, and a drive already running always stays stoppable. In Seville a button that only ever answered "Korea only", and looked like the talk's camera, was noise. Its glyph is now a speedometer.

An inferred meaning needs a single writer that proves it again on every write. Data you interpret rather than read has a meaning that can drift under you. Put the interpretation, the evidence for it and the output in one tool, and make the evidence print every time, so a change of meaning is something you see rather than something you ship.

Code

cwkPathfinder, PathfinderKit/Sources/PathfinderKit/Cameras/PoliceCameraRow.swift: CameraDataset.build, where datasetModified is always nil (verbatim, lines 81-104)·swift

extension CameraDataset {
  public static let policeSource = "National Police Agency standard dataset 15028200 (data.go.kr)"

  /// The bundled dataset from a full pull: driving-relevant cameras only,
  /// each with an id unique across providers (the management number repeats
  /// between and even within institutions).
  public static func build(rows: [PoliceCameraRow], fetchedAt: String) -> CameraDataset {
    var seen: [String: Int] = [:]
    var cameras: [SpeedCamera] = []
    for row in rows {
      let base = [row.insttCode, row.mnlssRegltCameraManageNo].compactMap(PoliceCameraRow.text).joined(separator: "-")
      let count = seen[base, default: 0]
      seen[base] = count + 1
      let id = count == 0 ? base : "\(base)-\(count)"
      guard let camera = row.camera(id: id), camera.isDrivingRelevant else { continue }
      cameras.append(camera)
    }
    let dates = cameras.compactMap(\.referenceDate).sorted()
    return CameraDataset(
      source: "\(policeSource): \(cameras.count) driving-relevant of \(rows.count) rows",
      datasetModified: nil, fetchedAt: fetchedAt, newestReferenceDate: dates.last,
      oldestReferenceDate: dates.first, cameras: cameras)
  }
}

cwkPathfinder, PathfinderKit/Sources/PathfinderKit/Cameras/CameraIndex.swift: the corridor, the driving side and the signed side (verbatim, lines 94-118)·swift

    var found: [CameraOnRoute] = []
    for (index, seen) in candidates {
      let camera = cameras[index]
      let window = (seen.lowerBound - Self.projectionReach)...(seen.upperBound + Self.projectionReach)
      guard camera.isDrivingRelevant,
        let match = polyline.project(camera.coordinate, within: window), match.lateral <= corridor
      else { continue }
      let side = Self.signedSide(of: camera.coordinate, from: match)
      if side < -leftTolerance { continue }
      found.append(CameraOnRoute(camera: camera, along: match.distanceAlong, lateral: match.lateral))
    }
    return found.sorted { $0.along < $1.along }
  }

  /// Signed cross-track distance: positive to the right of the direction of
  /// travel, negative to the left.
  static func signedSide(of point: Coordinate, from match: Polyline.Match) -> Double {
    let frame = Geo.LocalFrame(origin: match.coordinate)
    let (px, py) = frame.project(point)
    let heading = match.bearing * .pi / 180
    // Unit vector of travel (east = sin, north = cos); right-hand normal.
    let rightX = cos(heading)
    let rightY = -sin(heading)
    return px * rightX + py * rightY
  }

A minimal sketch: the driving-side rule on synthetic points in a flat frame (compiled with swiftc and run; its output is at the bottom)·swift

import Foundation

// A minimal sketch of Pathfinder's driving-side rule in a flat local frame
// (x east, y north, metres). The points are synthetic, not real cameras.
struct Point { var x: Double; var y: Double }

/// Signed cross-track distance: positive to the right of travel, negative to
/// the left (the same right-hand normal CameraIndex.signedSide uses).
func signedSide(of p: Point, from foot: Point, bearing degrees: Double) -> Double {
  let heading = degrees * .pi / 180
  let (px, py) = (p.x - foot.x, p.y - foot.y)
  return px * cos(heading) + py * -sin(heading)
}

func verdict(_ camera: Point, foot: Point, bearing: Double,
             corridor: Double = 30, leftTolerance: Double = 10) -> String {
  let lateral = hypot(camera.x - foot.x, camera.y - foot.y)
  guard lateral <= corridor else { return "not on this road (\(Int(lateral)) m off)" }
  let side = signedSide(of: camera, from: foot, bearing: bearing)
  if side < -leftTolerance { return "other carriageway (\(Int(side)) m)" }
  return "on the route (side \(Int(side)) m)"
}

// A road running due north (bearing 0) along x = 0; the foot of each camera
// is the closest point on the road.
let northbound: [(String, Point)] = [
  ("pole on the right shoulder", Point(x: 7, y: 120)),
  ("gantry over the median", Point(x: -4, y: 310)),
  ("pole across the median", Point(x: -18, y: 540)),
  ("camera on a side street", Point(x: 55, y: 800)),
]
for (name, camera) in northbound {
  print("north", name.padding(toLength: 28, withPad: " ", startingAt: 0),
        verdict(camera, foot: Point(x: 0, y: camera.y), bearing: 0))
}

// The same pole seen from the opposite direction (bearing 180) is on the right.
let pole = Point(x: -18, y: 540)
print("south", "pole across the median".padding(toLength: 28, withPad: " ", startingAt: 0),
      verdict(pole, foot: Point(x: 0, y: 540), bearing: 180))

// Output:
// north pole on the right shoulder   on the route (side 7 m)
// north gantry over the median       on the route (side -4 m)
// north pole across the median       other carriageway (-18 m)
// north camera on a side street      not on this road (55 m off)
// south pole across the median       on the route (side 18 m)

External links

Exercise

Find a file in your own project that was generated from outside data and committed: a country list, a price table, a timezone map, a set of codes. Find out who writes it. If the answer is "a script once, plus some hand edits", write the single writer: a command that fetches, transforms, writes the file, and prints the evidence for every interpretation it makes (counts per code, units it converted, rows it dropped). Then check whether every provenance field the file claims to carry is actually filled.

Hint

Start by diffing what the current file says against what a fresh fetch would say. Before you call a differing row a hand edit, rule out the boring causes: the source may have changed since the file was written, your transformer may not be the version that wrote it, and its output may not be deterministic (row order, timestamps, number formatting). Re-running the old transformer on a saved raw pull settles most of that. What still differs is most likely a hand edit, and each one is either a rule your writer should learn or a mistake it should stop repeating. A field that is always empty is a cheap thing to find and an easy thing to fix once you look.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousOnly Dad Cuts Her Off, When the Ear Can Tell Next →Quiz · 4 questions

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.