Latitude, Longitude and the Nautical Mile: Distance and Bearing Without a Chart

Musings of a former maths teacher learning to sail

Brass dividers stepping off a distance on the latitude scale at the edge of a paper chart

A nautical mile is one minute of latitude. I say that with some feeling, because for about a fortnight I had it filed in my head as one minute of longitude, and it took a Day Skipper exercise to knock it out.

The distinction sounds pedantic. It isn’t. It’s the reason you measure distance off the side of a chart and never off the top or bottom, and it’s the first thing that has to be right before anything else works.

The mile that is really an angle

A nautical mile isn’t a length that someone chose. It’s an angle, subtended at the centre of the Earth and projected onto the surface. Stand at the centre of the planet, open your arms by one minute of arc — one sixtieth of a degree — and the gap between your fingertips at the surface is one nautical mile.

That gives you the number that makes navigation tolerable: one minute of latitude equals one nautical mile, everywhere. A degree of latitude is sixty miles. The quarter-circle from the equator to the pole is 90 × 60 = 5,400 miles.

It also has a peculiar side effect. Because the measurement is angular, it only counts the horizontal component. Walk from the bottom of a mountain to the top, and unless you have also moved sideways across the face of the Earth, you have travelled no distance at all. All that effort, and the angle at the centre of the planet has barely twitched. The mountaineer’s mile and the sailor’s mile are different animals, and only one of them cares how tired you are.

At sea that’s exactly the simplification you want, because the sea is helpfully flat.

If you did GCSE maths, you’ve already met the piece that matters here: the angle at the centre decides what fraction of the circle an arc takes up, and the circle’s own size then scales that fraction into an actual length — arc length = angle/360 × 2πr. A nautical mile is that theorem run in reverse: fix the angle at one arcminute, and whatever length that same fraction works out to, on a planet of this particular radius, is the mile.

Earth's centre with two radius lines meeting the surface one arcminute apart, the arc between them highlighted

Diagram: Lidingo, VineetKumar and Yodin, via Wikimedia Commons (CC BY-SA 4.0).

Why longitude won’t behave

Meridians of longitude are great circles: all pole to pole, all the same size, all converging at the top and bottom. So the distance between two meridians isn’t fixed — widest at the equator, zero at the poles.

Parallels of latitude are not great circles, the equator excepted. Their radius shrinks as you go north or south, by a factor of the cosine of the latitude:

distance along a parallel = difference in longitude (minutes) x cos(latitude)

That east–west distance has a name: departure. It’s what a difference in longitude is actually worth in miles, at the latitude you happen to be.

Some concrete numbers, because a formula on its own doesn’t stick:

So sail due east from 50°N for a whole degree of longitude — 60 minutes — and you have covered 60 × 0.6428 = 38.6 miles, not 60. Get that wrong on a passage plan and you arrive hours late.

It’s also why the latitude scale runs down the sides of a chart and the longitude scale along the top, and why every instructor tells you to step your dividers off the side. The side scale is in miles. The top scale is in something that merely looks like miles.

Former maths teacher’s aside: I used to introduce the cosine as “adjacent over hypotenuse” and then spend weeks trying to make it mean something. Here it means something immediately. It is the factor by which the world narrows as you go north.

Draw the actual triangle and the SOHCAHTOA you learned for a GCSE exam becomes a fact about the planet. Take a point on a parallel of latitude, drop a line from it at right angles to the Earth’s axis, and join it to the planet’s centre: that gives a right-angled triangle, with the right angle where the perpendicular meets the axis. The angle at the surface point — between the line back to the centre and the line across to the axis — is exactly your latitude, and its adjacent side is the parallel’s own radius, its hypotenuse the Earth’s radius. Cosine of latitude, adjacent over hypotenuse, is exactly the shrinking factor the formula uses.

Picture it: the Earth’s axis running vertically through the middle, your surface point out on the rim, one short line running sideways from that point to the axis, and the long diagonal line running from the same point down to the planet’s centre. Three lines, one right angle, one triangle — and the angle sitting at your own position on the surface is the latitude you’d read off a chart.

Distance

Two positions, no chart, no plotter. How far apart are they?

Which method to use for distance and bearing between two positions

The two easy cases

Same meridian. If both positions share a longitude, subtract the latitudes and convert to minutes. That’s the distance in miles, exactly.

Same parallel. If both share a latitude, take the difference in longitude in minutes and multiply by the cosine of that latitude.

The general case: plane sailing

For anything short enough that the curvature of the Earth doesn’t matter much — essentially every coastal passage — you can pretend the patch of ocean you’re crossing is flat and use Pythagoras. Navigators call this plane sailing, and yes, that’s where the phrase comes from.

Take a real passage, from the Needles open water buoy, off the western end of the Isle of Wight, to the harbour entrance at Alderney. Chartplotters and GPS give positions in decimal degrees; paper charts give degrees and minutes. Converting between them is multiplication or division by sixty, and in the Python script it is two small functions:

def to_decimal_degrees(sign, degrees, minutes):
    """Degrees and decimal minutes to decimal degrees: deg + min / 60."""
    return sign * (degrees + minutes / MINUTES_PER_DEGREE)


def to_degrees_minutes(decimal_degrees):
    """Decimal degrees back to {"sign", "degrees", "minutes"}."""
    sign = -1 if decimal_degrees < 0 else 1
    whole = int(abs(decimal_degrees))
    minutes = (abs(decimal_degrees) - whole) * MINUTES_PER_DEGREE
    return {"sign": sign, "degrees": whole, "minutes": minutes}

Here are both forms of our two positions:

A   Needles open water buoy   50.637168° N   1.649183° W   =   50° 38.230' N   001° 38.951' W
B   Alderney harbour entrance 49.734654° N   2.186306° W   =   49° 44.079' N   002° 11.178' W

The conversion is only multiplication by sixty: the 0.637168 of a degree after the decimal point is 0.637168 × 60 = 38.230 minutes.

Step one, the difference in latitude (d.lat). One minute of latitude is one mile, so the answer in minutes is also the answer in miles:

def difference_in_latitude(lat1, lat2):
    """d.lat in minutes, north positive. One minute of latitude is one mile."""
    return (lat2 - lat1) * MINUTES_PER_DEGREE

50° 38.230' - 49° 44.079' = 54.151' south   ->  54.15 miles

Step two, the difference in longitude (d.long). The extra line takes the short way round, so a passage across the 180° meridian doesn’t come out as 358°:

def difference_in_longitude(lon1, lon2):
    """d.long in minutes, east positive, taking the short way round."""
    delta = (lon2 - lon1 + 180) % 360 - 180
    return delta * MINUTES_PER_DEGREE

002° 11.178' - 001° 38.951' = 32.227' west

Step three, convert that to departure using the mean latitude of the two points. This is the cosine factor from earlier, applied at the latitude halfway along the leg:

def mean_latitude(lat1, lat2):
    """The latitude halfway between the two points, in degrees."""
    return (lat1 + lat2) / 2


def departure(d_long_minutes, lat):
    """East-west distance in miles: d.long (minutes) x cos(latitude)."""
    return d_long_minutes * math.cos(math.radians(lat))

mean lat  = (50.637168° + 49.734654°) / 2 = 50.1859°
cos 50.1859° = 0.6403
departure  = 32.227 x 0.6403 = 20.64 miles

Step four, Pythagoras. math.hypot is the square root of the two squares added together:

def plane_sailing_distance(d_lat_minutes, departure_miles):
    """Pythagoras on the flat triangle formed by d.lat and departure."""
    return math.hypot(d_lat_minutes, departure_miles)

distance = sqrt(54.15^2 + 20.64^2)
         = sqrt(2932.3 + 425.8)
         = sqrt(3358.1)
         = 57.95 miles

Almost 58 miles to Alderney.

Right-angled triangle with d.lat and departure as the two short sides and distance as the hypotenuse

Diagram: BoH, via Wikimedia Commons (CC BY-SA 3.0).

The accurate version

Before the formula, the shape of it. Higher-tier GCSE covers the cosine rule for an ordinary flat triangle — c² = a² + b² − 2ab·cos C, letting you find a side when you know the other two sides and the angle between them. The spherical version below is the same rule, lifted onto the surface of a sphere: sides become arcs, lengths become angles, and sin/cos of those angles stand in for the side lengths. It looks unrecognisable until you’ve seen the family resemblance once.

An ordinary flat triangle, sides a, b, c, with the angle between two of them marked

Diagram: David Weisman, via Wikimedia Commons (public domain).

The same triangle redrawn with its sides as arcs on the surface of a sphere, for the spherical law of cosines

Diagram: TheOtherJesse, via Wikimedia Commons (public domain).

Plane sailing is an approximation. The exact answer needs spherical trigonometry, traditionally the spherical law of cosines:

cos d = sin(lat1) x sin(lat2) + cos(lat1) x cos(lat2) x cos(d.long)

with d coming out as an angle, which you multiply by 60 to get miles. In Python:

def spherical_cosine_distance(wp1, wp2):
    """Spherical law of cosines, in miles. Loses precision on short legs."""
    lat1, lat2 = math.radians(wp1.lat), math.radians(wp2.lat)
    d_lon = math.radians(difference_in_longitude(wp1.lon, wp2.lon) / 60)
    cos_d = math.sin(lat1) * math.sin(lat2) + math.cos(lat1) * math.cos(lat2) * math.cos(d_lon)
    cos_d = max(-1.0, min(1.0, cos_d))  # rounding can push it just past 1
    return math.degrees(math.acos(cos_d)) * MINUTES_PER_DEGREE

The max/min line is a guard: on a very short leg, rounding can push the cosine a hair above 1.0, and math.acos would raise an error. For our two points:

sin 50.6372°  = 0.773145       sin 49.7347° = 0.763059
cos 50.6372°  = 0.634229       cos 49.7347° = 0.646328
cos 0.5371°   = 0.999956

cos d = (0.773145 x 0.763059) + (0.634229 x 0.646328 x 0.999956)
      = 0.589956 + 0.409902
      = 0.999858

d = arccos(0.999858) = 0.96581°
distance = 0.96581 x 60 = 57.95 miles

Plane sailing gave 57.949; the exact answer is 57.949 too, and the two differ by less than a metre. That is several orders of magnitude better than your ability to steer.

Notice something uncomfortable, though. The answer hinged on cos d = 0.999858. Round that to five decimal places, 0.99986, and the distance becomes 57.5 miles. Round to four, 0.9999, and it’s 48.6 miles, nine miles short. Round to three and it’s nonsense: zero. Short distances push the cosine formula right up against the top of its range, where it loses precision. That is the entire reason the haversine formula exists:

a = sin^2(d.lat / 2) + cos(lat1) x cos(lat2) x sin^2(d.long / 2)
c = 2 x arcsin(sqrt(a))
distance = c (in degrees) x 60

In Python, it is the same shape, with asin and a square root in place of the acos:

def haversine_distance(wp1, wp2):
    """Haversine, in miles. Stays accurate for short distances."""
    lat1, lat2 = math.radians(wp1.lat), math.radians(wp2.lat)
    d_lat = lat2 - lat1
    d_lon = math.radians(difference_in_longitude(wp1.lon, wp2.lon) / 60)
    a = math.sin(d_lat / 2) ** 2 + math.cos(lat1) * math.cos(lat2) * math.sin(d_lon / 2) ** 2
    c = 2 * math.asin(math.sqrt(a))
    return math.degrees(c) * MINUTES_PER_DEGREE

Running the same numbers:

sin(0.45126°) = 0.0078758   squared = 0.00006203
sin(0.26856°) = 0.0046873   squared = 0.00002197
cos(lat1) x cos(lat2) = 0.409920

a = 0.00006203 + (0.409920 x 0.00002197)
  = 0.00006203 + 0.00000901
  = 0.00007103

sqrt(a) = 0.0084282
c = 2 x arcsin(0.0084282) = 0.96581°
distance = 57.95 miles

Same answer, but every quantity in it is small and well away from the awkward end of its range. Haversine is what’s inside the chartplotter.

Here’s the intuition, no calculus required: picture the graph of cosine near zero. It’s nearly flat there — the curve barely dips as the angle grows because you’re right at its peak. That makes recovering a short distance from a rounded cosine unreliable: many small distances map to values crowded close to 1. Haversine avoids that crowded end of the range by computing sin²(d/2), a small positive value whose relative differences are retained, then recovering the angle with a square root and arcsine.

Graphs of sine, cosine and tangent near zero, showing cosine flat near its peak and sine steep near zero

Diagram: Qniemiec, via Wikimedia Commons (CC BY-SA 3.0).

Bearing

Bearing is the messier half of the problem.

The quick way

Plane sailing gives it for free, out of the same triangle, as the angle from the meridian:

def plane_sailing_bearing(d_lat_minutes, departure_miles):
    """True bearing 0-360 from the same triangle: atan2(departure, d.lat)."""
    return math.degrees(math.atan2(departure_miles, d_lat_minutes)) % 360

The % 360 is there because atan2 returns a number between −180° and +180°, and a navigator wants a three-figure bearing from 000° to 360°. Without it, our south-westerly leg would come out as −159°, which is correct and useless. By hand, the same calculation is:

tan(angle) = departure / d.lat
           = 20.64 / 54.15
           = 0.3811
angle = 20.9°

We’re going south and west, so that’s S 20.9° W, or 180 + 20.9 = 200.9° in the three-figure notation everyone actually uses.

The proper way

The exact initial great-circle bearing is the next function. The y and x variables are the east-west and north-south parts of the first step of the track, and atan2 turns them into an angle:

def great_circle_initial_bearing(wp1, wp2):
    """True bearing to set off on along the great circle, 0-360."""
    lat1, lat2 = math.radians(wp1.lat), math.radians(wp2.lat)
    d_lon = math.radians(difference_in_longitude(wp1.lon, wp2.lon) / 60)
    y = math.sin(d_lon) * math.cos(lat2)
    x = math.cos(lat1) * math.sin(lat2) - math.sin(lat1) * math.cos(lat2) * math.cos(d_lon)
    return math.degrees(math.atan2(y, x)) % 360

The same formulas, written out:

y = sin(d.long) x cos(lat2)
x = cos(lat1) x sin(lat2) - sin(lat1) x cos(lat2) x cos(d.long)
bearing = atan2(y, x)

Signs matter: east and north positive, west and south negative, so our d.long is −0.5371°.

y = sin(-0.5371°) x cos(lat2) = -0.009374 x 0.646328 = -0.006059
x = (0.634229 x 0.763059) - (0.773145 x 0.646328 x 0.999956)
  = 0.483954 - 0.499684
  = -0.015730

atan2(-0.006059, -0.015730) = 201.1°

So 201.1° true, against plane sailing’s 200.9°. Two tenths of a degree apart, which is finer than any tiller will hold.

The atan2 matters. Both x and y are negative, putting the answer in the third quadrant; a plain arctangent would divide the two, get a positive number, and confidently send you north-east.

This is the same quadrant-sign-reading you’ll have done for GCSE trigonometry beyond the first 90° — just dressed up in navigation’s own variables. Positive x and positive y put you in the quadrant you’d expect; once either sign flips, a plain arctangent loses track of which quadrant you’re actually in, because tan repeats every 180° but your position on the full circle doesn’t. atan2 looks at the sign of both x and y together and places you in the right one of the four.

The four quadrants of the Cartesian plane with the sign convention for x and y in each

Diagram: Gustavb, via Wikimedia Commons (CC BY-SA 3.0).

The bearing doesn’t stay put

Here’s the thing that surprised me. The bearing on arrival is the reverse leg’s initial bearing, turned through 180°:

def great_circle_final_bearing(wp1, wp2):
    """True bearing on arrival: the reverse leg's initial bearing, turned round."""
    return (great_circle_initial_bearing(wp2, wp1) + 180) % 360

Sail that great circle and the bearing changes under you: the same track arrives at Alderney on 200.7°, not 201.1°. A shortest path over a sphere crosses each meridian at a slightly different angle, because the meridians aren’t parallel.

The alternative is a rhumb line: a track of constant bearing, which on a Mercator chart is a straight line. It’s the one you can actually steer. It is also always longer, because it isn’t the shortest path. The maths stretches the latitude scale the way a Mercator chart does (d_psi) and then uses that to scale the east-west leg of the triangle:

def rhumb_line(wp1, wp2):
    """Constant-bearing track: {"distance_nm", "bearing_true"}. Straight on a Mercator chart."""
    lat1, lat2 = math.radians(wp1.lat), math.radians(wp2.lat)
    d_lat = lat2 - lat1
    d_lon = math.radians(difference_in_longitude(wp1.lon, wp2.lon) / 60)
    d_psi = math.log(math.tan(math.pi / 4 + lat2 / 2) / math.tan(math.pi / 4 + lat1 / 2))
    # Along an east-west track d_psi is zero, so fall back to cos(latitude).
    q = d_lat / d_psi if abs(d_psi) > 1e-12 else math.cos(lat1)
    distance = math.hypot(d_lat, q * d_lon) * EARTH_RADIUS_NM
    bearing = math.degrees(math.atan2(d_lon, d_psi)) % 360
    return {"distance_nm": distance, "bearing_true": bearing}

For our passage it runs 200.9° the whole way and measures 57.9490 miles against the great circle’s 57.9489 — a penalty of about twenty centimetres.

So for coastal work the two are the same thing to any accuracy you can steer. The difference only bites on ocean crossings, which is why a transatlantic route is planned as a series of rhumb-line legs chasing a great circle, and a hop to Alderney is one straight line.

True, magnetic, and what you actually steer

201.1° is a true bearing, measured from the geographic North Pole. Your steering compass points at the magnetic pole, which is somewhere else entirely and moving. The difference is variation: small in the central Channel at the moment, well under a degree west, but tens of degrees in parts of the world, and changing year on year. On top of that sits deviation, your own boat’s steel and electronics pulling the card around, differently on every heading.

true  ->  (apply variation)  ->  magnetic  ->  (apply deviation)  ->  compass

In code, the westerly errors are simply added:

def true_to_compass(true_bearing, variation_west=0.0, deviation_west=0.0):
    """Add westerly errors, subtract easterly ones (give easterly as negative).

    "Error west, compass best": true -> magnetic -> compass.
    """
    return (true_bearing + variation_west + deviation_west) % 360

Going that way, from what you calculated to what you steer, you add westerly errors and subtract easterly ones. Hence “error west, compass best”: the compass number is the larger of the two when the error is westerly. With 201.1° true and, for illustration, variation 2°W and deviation 1°W, you steer 204.1°.

The number you computed is never the number you put on the binnacle. And none of that has touched tidal stream, which on a Channel Islands passage will shove you sideways harder than every correction above put together. Different calculation, different post.

Teaching a computer the same sums

Working one passage through by hand is the right way to learn this. It is the wrong way to plan a trip with six waypoints, because every leg repeats the same ten steps and every step is a chance to drop a sign. So I turned the whole article into a Python script. The complete source is in a gist: passage_plan.py and an example journey.

How the script was made

I didn’t type it. I described it to an AI coding assistant, Claude Code, and the interesting part was the description, because a vague request gets you a vague script. The brief had six requirements:

That last point is the one that matters if you are going to trust code you didn’t write. The tests check the figures worked through in this post: 57.95 miles, 201.1° initial bearing, 200.7° on arrival, and a rhumb line a few centimetres longer. They also cover the awkward cases, which I’ll describe below.

The two waypoints are now the Needles open water buoy and the Alderney harbour entrance. Changing the end points meant redoing every worked example in the article. I didn’t redo them by hand: the script printed each intermediate value, and the numbers in the sections above come from that output.

What is in it

Every function above is in the script, and so are the pieces around them. In the order the post uses them:

Writing the functions meant handling several things a person with a chart does without thinking:

Running it

The journey file for this passage is five columns, with a header row. Lines beginning with # are comments:

# Needles open water buoy to the Alderney harbour entrance
name,lat_deg,lat_min,lon_deg,lon_min
Needles open water buoy,50N,38.230,1W,38.951
Alderney harbour entrance,49N,44.079,2W,11.178

Run it with the journey file and a name for the plan:

python3 passage_plan.py needles-to-alderney.csv plan.csv

It writes one line per leg, using the haversine distance and the great-circle initial bearing:

leg,from,to,distance_nm,bearing_true
1,Needles open water buoy,Alderney harbour entrance,57.95,201.1

Add more waypoints, in the order you intend to sail them, and you get more legs, plus a total distance printed on screen. The bearings are all true. Variation and deviation depend on your boat and the date, so true_to_compass is there to call, but the plan doesn’t apply it for you.

What the plan means on the water

The plan says 57.95 miles on 201.1° true. That is a straight line between two points, and it is where the sailing starts, not where it ends.

At a steady five knots, 57.95 miles is just under twelve hours (57.95 ÷ 5 = 11.6), so this is an overnight passage for most boats, and the tide will change direction more than once. The bearing is the line on the chart, not the course to steer. The stream will push the boat off the line, so the heading you hold to stay on it will differ from 201.1° and will change through the passage. Correct for variation and deviation as above to get the compass number, then allow for tide, which is a calculation for another post.

A straight line also ignores everything else the chart shows. Before a leg goes into a real plan, I’d lay it over the chart and check for shoals, traffic separation schemes and tidal gates. The script is arithmetic. It doesn’t know what is between the waypoints.

So why do any of this by hand?

Because the chartplotter will give you the answer in under a second, and it will be right. The script will give you the answer for six legs in the same time.

The same argument applies here as with the Rule of Twelfths for tidal heights: the value of the pencil-and-paper method isn’t the answer, it’s knowing roughly what the answer should be before the box tells you. If the plotter claims Alderney is 78 miles off, I now know without thinking that 54 minutes of latitude is 54 miles and the rest is a modest departure, so something is misconfigured. That check costs ten seconds.

The script adds a second check, in two parts. Writing it explains the process: you can’t write departure or plane_sailing_bearing without knowing why each step is there and what order the steps go in, which is a harder test of understanding than reading the method in a book. Running it then checks your understanding of what the chartplotter is doing. Give the script the same two waypoints you gave the plotter. If the distance and bearing agree, you know what the box is computing. If they don’t, either the script or your understanding has a hole, and finding it is where the learning happens.

The same goes for trusting the script. I only trust it because I know what it should say: roughly 58 miles, a little west of south. If I hadn’t worked the passage through by hand first, I’d have no way to tell a good script from a plausible-looking one.

It also changes how you read a chart. Once you’ve felt the cosine shrinking the parallels, the converging meridians stop being a memorised fact and become something you can see in the grid.

I got the definition wrong for a fortnight. I’m fairly confident I’ll never get it wrong again, which is the one genuine advantage of learning a thing the hard way.

Fridays are for sailing on this blog. Last week: the Rule of Twelfths. Next week: more from the Day Skipper course. Follow along if you’d like it in your feed.